SA-31148

Summary

One node of total 9 nodes crashed with an out of memory error and server needed to be restarted to fix the issue.

Error Messages

Log File Monitor - Java OutOfMemoryError – PermGen Space Availability:Solutions Center

Steps to Reproduce

This is an issue observed in runtime, and exact steps are unknown. 

Root Cause

Developers recently implemented a change to the dynamic system setting (DSS) to control prconfig/timeout/browser/default, increasing the value to 12000 (which means that idle requestors will remain in heap for 200 minutes before being passivated and removed).  With the requestor volume (and requestor sizes which can reach up to 90MB), the larger collection consumed enough memory to get very near to max heap (8GB) and to trigger full gcs which slowed down all running threads. 

Resolution

System Administrators have updated the DSS back to 1200 seconds (20 minutes), and are performing a rolling restart of servers to complete this change.