SA-9467

Summary

Issue with thread locks and system hang occurs almost every day at heavy loading hours.

Stack trace of blocked threads from Thread dump shows that most of threads are blocked or waiting (parked) on loading readonly data pages.

Server restart does not help to resolve the issue in the long term. On the next day at peak hours issue occurs again.

Error Messages

Snippet from log thread dump:

"WebContainer : 0" Id=185 in WAITING on lock=com.pega.jsr166backport.java.util.concurrent.locks.ReentrantLock$NonfairSync@8dea7e41 (running in native)
owned by WebContainer : 12 Id=836
BlockedCount : 48, BlockedTime : -1, WaitedCount : 60756, WaitedTime : -1
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:182)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:822)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:853)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1189)
at com.pega.jsr166backport.java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:190)
at com.pega.jsr166backport.java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:268)
at com.pega.pegarules.priv.web.ProcessingValve$RequestToken.lock(ProcessingValve.java:146)
at com.pega.pegarules.priv.web.ProcessingValve.waitOnPendingRequests(ProcessingValve.java:93)
at com.pega.pegarules.exec.internal.declare.infengine.pages.DeclarativePageDirectoryImpl.loadAndReturnDeclarePage(DeclarativePageDirectoryImpl.java:734)
at com.pega.pegarules.session.internal.mgmt.PRNodeImpl.loadAndReturnDeclarePage(PRNodeImpl.java:1618)
at com.pega.pegarules.session.internal.mgmt.PRNodeImpl.getRODeclarePage(PRNodeImpl.java:1607)
at com.pega.pegarules.session.internal.mgmt.base.ThreadPageDir.getOrLoadDeclarePage(ThreadPageDir.java:884)
at com.pega.pegarules.session.internal.mgmt.base.ThreadPageDir.getDirectPage(ThreadPageDir.java:786)
at com.pega.pegarules.session.internal.mgmt.base.ThreadPageDir.getDirectPage(ThreadPageDir.java:719)
at com.pega.pegarules.session.internal.mgmt.base.ThreadPageDir.getPageForAutoPopulate(ThreadPageDir.java:508)
at com.pega.pegarules.session.internal.mgmt.PRThreadImpl.getPageForAutoPopulate(PRThreadImpl.java:550)
<etc>

Steps to Reproduce

Unknown.

Root Cause

Initial performance issue is due to the application which is using a Declare page at a node level and there is contention to access it.
Only one thread at time can access a data pages with a node scope. It turns out this declare page is used by activity “applicationprofilesetup” which is used often by PRPC. The issue is more complex due to a Declare page which tries to load itself. 
Also an auto-populated property tries to refer to that data page and it does so passing an empty paramater value.

I remind here what are auto-populated property:
https://pdn.pega.com/sites/default/files/help_v71/procomhelpmain.htm#definitions/a/autopopulatedproperty.htm

This happened when attempted to display a custom harness.

Also, we reported we noticed a large number of Pega0050 alert
There are about a Declare page being copied into another and appear to be a source of contention.

Resolution

There are multiple properties, that are autopopulated during the loading of this custom harness. 
Remove page copy from a Declare page to another one in order to avoid contention and set the parameter as required, so that it does not executed without this parameter. 
Also for another of our suspicous Declare  page we found that there was a tick at"Limit to a single data page". We have removed it and set "Clear pages after non-use" and "RELOAD IF OLDER THAN" = 1H. 
With these changes, the issue disappeared.