Skip to main content

This content has been archived and is no longer being updated. Links may not function; however, this content may be relevant to outdated versions of the product.

Support Article

Production system receives OOM errors

SA-4934

Summary



Administrators observe OOM errors on 4 Production server nodes, all hosted from the same machine.


Root Cause



Following observed in the native_stderr.log:
 
VMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2014/12/24 11:33:08 - please wait.
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2014/12/24 11:33:08 - please wait.
JVMDUMP007I JVM Requesting Heap dump using '/dumps/was85LV3_K10/heapdump.20141224.113308.7114.0001.txt'
JVMDUMP010I Heap dump written to /dumps/was85LV3_K10/heapdump.20141224.113308.7114.0001.txt
JVMDUMP007I JVM Requesting Heap dump using '/dumps/was85LV3_K10/heapdump.20141224.113308.7114.0002.txt'
JVMDUMP010I Heap dump written to /dumps/was85LV3_K10/heapdump.20141224.113308.7114.0002.txt
JVMDUMP032I JVM requested Java dump using '/dumps/was85LV3_K10/javacore.20141224.113308.7114.0003.txt' in response to an event
JVMDUMP010I Java dump written to /dumps/was85LV3_K10/javacore.20141224.113308.7114.0003.txt
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
JVMDUMP032I JVM requested Java dump using '/dumps/was85LV3_K10/javacore.20141224.113308.7114.0004.txt' in response to an event
 
 
Normally, the system information, such as “Free memory”, is added to the PegaRULES log every 10 minutes.
 
2014-12-24 11:28:48,573 [RULESLV03,maxpri=10]] [  STANDARD] [                    ] (          internal.async.Agent) INFO    - System date: Wed Dec 24 11:28:48 GMT 2014 Total memory: 2,150,957,056 Free memory: 954,846,072 Requestor Count: 26 Shared Pages memory usage: 0%
2014-12-24 11:32:45,811 [  WebContainer : 209] [  STANDARD] [     LTSB-GPMT:03.01] (PMT_Work_RiskReviewCase.Action) INFO  customer-pega.intranet.group|127.0.0.1 4106321 - GPMT: Success - Case creation. Feed: . Parent case ID (feed): GPMT-PR-241214-14. Child case ID: GPMT-CC-241214-43
2014-12-24 11:57:11,173 [RULESLV03,maxpri=10]] [  STANDARD] [                    ] (.access.DatabaseConnectionImpl) ERROR File.GWCSFircoSoftInput|initialize your_operatorid - Couldn't obtain a connection. Refresh the DataSource, and try again
2014-12-24 11:57:13,652 [RULESLV03,maxpri=10]] [  STANDARD] [                    ] (          internal.async.Agent) INFO    - System date: Wed Dec 24 11:57:13 GMT 2014 Total memory: 1,691,353,088 Free memory: 1,023,355,912 Requestor Count: 27 Shared Pages memory usage: 0%
2014-12-24 11:57:16,107 [  WebContainer : 197] [          ] [                    ] (ngineinterface.service.HttpAPI) ERROR customer-pega.intranet.group|127.0.0.1  - 127.0.0.1: com.pega.pegarules.pub.context.RequestorLockException
 
However, in the above extract (PegaRULES_lv03_c1.log) more than 10 minutes is observed, which could be an indication that the system has suspended that thread while it is performing the heap dump.
No OutOfMemoryError in the PegaRULES log files that were sent. So the original OOM was reported at 11:30 whereas the same error does not appear in the PegaRULES log until 12:25/12:26.
There appears to be ~300 occurrences of the OOM in each of the two log files. 
This appears to be happening when the JVM is trying to allocate memory in the native heap. See IBM support article entitled java.lang.OutOfMemoryError while creating new threads. Also, increase the process limit for your user. Furthermore, verify that the AES server is up and running.
Suggest Edit

Published January 31, 2016 - Updated October 8, 2020

Did you find this content helpful? Yes No

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.

We'd prefer it if you saw us at our best.

Pega Community has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice
Contact us