Threads hung in JVM, system unresponsive
RequestorLockExceptions begin occurring with regularity on one node. For the next hour, the logs report specific threads may be hung due to having run for long periods of time, and there are many CPU starvation messages.
Various, from the SystemOut logfile:
[4/28/15 15:23:18:042 CDT] CPU Starvation detected. Current thread scheduling delay is 228 seconds.
[4/28/15 15:28:08:112 CDT] CPU Starvation detected. Current thread scheduling delay is 19 seconds.
[4/28/15 15:22:51:846 CDT] 00000f6a ThreadMonitor: Thread "WebContainer : 21" (000000f2) has been active for 816184 milliseconds and may be hung. There is/are 11 thread(s) in total in the server that may be hung.
[4/28/15 15:33:05:495 CDT] 0000000b AlarmThreadMo : Alarm Thread "Non-deferrable Alarm : 2" (0000000b) previously reported to be delayed has now completed. It was active for approximately 183882 milliseconds.
[4/28/15 15:33:05:495 CDT] 00000009 AlarmThreadMo : Alarm Thread "Non-deferrable Alarm : 0" (00000009) previously reported to be delayed has now completed. It was active for approximately 183888 milliseconds.
Steps to Reproduce
File System Exhaustion (out of disk space) and Heap Exhaustion (insufficient memory in heap for normal operations) can both contribute to this type of system hang. Heavy Garbage collection and insufficient disk space can cause CPU exhaustion, which causes the errors.
Two changes were made to prevent future issues - ensuring that sufficient disk space is available for normal server operations (e.g. swap space, writing to files, etc.) and configuring the JVM to have a larger (4 GB) heap.
0% found this useful