Support Article
JVM hangs on Conclusion cache read or update
SA-3300
Summary
JVM hangs with users unable to login. A javacore/thread dump shows most of the Webcontainer user threads are in a park state executing either a call stack containing ConclusionCacheImpl.noteChange() or ConclusionCacheImpl.getOrCreate() methods. These methods wait indefinitely trying to get a ReentrantReadWriteLock$ReadLock or ReentrantReadWriteLock$WriteLock via a wrapper PRPC class "ReadWriteLockUpgrade". Below are example thread call stacks in parked state in a thread dump generated after JVM freeze
"ajp-0.0.0.0-8109-79" daemon prio=10 tid=0x00007f9d6011e800 nid=0x5685 waiting on condition [0x00007f9cc122f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000078bcfb9e0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
at com.pega.pegarules.util.ReadWriteLockUpgrade.acquireWriteLock(ReadWriteLockUpgrade.java:132)
at com.pega.pegarules.exec.internal.basic.dictionary.ConclusionCacheImpl.noteChange(ConclusionCacheImpl.java:1239)
"ajp-0.0.0.0-8109-101" daemon prio=10 tid=0x00007f9d60130800 nid=0x433 waiting on condition [0x00007f9cbf00c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000078bcfb9e0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
at com.pega.pegarules.util.ReadWriteLockUpgrade.readLock(ReadWriteLockUpgrade.java:90)
at com.pega.pegarules.exec.internal.basic.dictionary.ConclusionCacheImpl.getOrCreate(ConclusionCacheImpl.java:945)
at com.pega.pegarules.exec.internal.basic.dictionary.ConclusionCacheImpl.getOrCreate(ConclusionCacheImpl.java:1032)
Error Messages
N/A
Steps to Reproduce
This only happens on PRPC instances that have HFix-8969 installed.
Root Cause
The root cause of this problem is a defect in Pegasystems’ code/rules.
Resolution
This issue is resolved by hotfix item Hfix-10223
Additional details can be found on PDN KB article :
https://pdn.pega.com/performance/troubleshooting-jvm-hangs-on-conclusion-cache-read-or-update-hfix-10223-replaces-hfix-8969
Published January 31, 2016 - Updated October 8, 2020
Have a question? Get answers now.
Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.