SA-4448

Summary

We are experiencing frequent (~every few days) "java.lang.OutOfMemoryError" on our newly deployed PRODUCTION system. The JVM nodes need to be restarted.

Error Messages

java.lang.OutOfMemoryError: occuring in several contexts.

Steps to Reproduce

This is a PRODUCTION system: we have ~200 users per node - these users are running their normal day-to-day work: (which involves running reports).

Root Cause

It was found (from the ALERTS logs) that several 'unbounded' (no WHERE CLAUSE) SQL statements were being generated from a couple of places in our application.
Local Changes (code-fixes to activities - to limit the number of rows returned) were made to reduce the likelyhood of OOM occuring. (In fact, we found other contributing factors to high memory usage : these also have been addressed by making Local Changes to the Application.

Resolution

By checking the ALERT logs: we saw PEGA0005 alerts which contains SQL statements which did not have a WHERE CLAUSE; the underlying tables it accessed had many (million) rows in them. Therefore the SQL run 'unbounded' - and downstream this caused the system to experience low memory conditions - sometimes severe enough to cause the OOM exception.

By cross-checking the GC logs for the same node and ~ the same time, we were able to confirm that the 'Free Nursey' (in particular) section of the HEAP were being rapidly used up during the running on these unbounded SQL, like this: