SA-10516

Summary

RequestorLockExceptions begin occurring with regularity on one node. For the next hour, the logs report specific threads may be hung due to having run for long periods of time, and there are many CPU starvation messages.

Error Messages

Various, from the SystemOut logfile:

[4/28/15 15:23:18:042 CDT] CPU Starvation detected. Current thread scheduling delay is 228 seconds.
[4/28/15 15:28:08:112 CDT] CPU Starvation detected. Current thread scheduling delay is 19 seconds.

[4/28/15 15:22:51:846 CDT] 00000f6a ThreadMonitor: Thread "WebContainer : 21" (000000f2) has been active for 816184 milliseconds and may be hung. There is/are 11 thread(s) in total in the server that may be hung.

[4/28/15 15:33:05:495 CDT] 0000000b AlarmThreadMo : Alarm Thread "Non-deferrable Alarm : 2" (0000000b) previously reported to be delayed has now completed. It was active for approximately 183882 milliseconds.
[4/28/15 15:33:05:495 CDT] 00000009 AlarmThreadMo : Alarm Thread "Non-deferrable Alarm : 0" (00000009) previously reported to be delayed has now completed. It was active for approximately 183888 milliseconds.

Steps to Reproduce

Not Applicable

Root Cause

File System Exhaustion (out of disk space) and Heap Exhaustion (insufficient memory in heap for normal operations) can both contribute to this type of system hang. Heavy Garbage collection and insufficient disk space can cause CPU exhaustion, which causes the errors.

Resolution

Two changes were made to prevent future issues - ensuring that sufficient disk space is available for normal server operations (e.g. swap space, writing to files, etc.) and configuring the JVM to have a larger (4 GB) heap.