Node crashed with OutOfMemory error
One node of total 9 nodes crashed with an out of memory error and server needed to be restarted to fix the issue.
Error MessagesLog File Monitor - Java OutOfMemoryError – PermGen Space Availability:Solutions Center
Steps to Reproduce
This is an issue observed in runtime, and exact steps are unknown.
Developers recently implemented a change to the dynamic system setting (DSS) to control prconfig/timeout/browser/default, increasing the value to 12000 (which means that idle requestors will remain in heap for 200 minutes before being passivated and removed). With the requestor volume (and requestor sizes which can reach up to 90MB), the larger collection consumed enough memory to get very near to max heap (8GB) and to trigger full gcs which slowed down all running threads.
System Administrators have updated the DSS back to 1200 seconds (20 minutes), and are performing a rolling restart of servers to complete this change.
Published December 6, 2016 - Updated December 9, 2016