Support Article
Servers are unstable and facing latency issues
Summary
User reports that their servers are unstable and are facing latency issues. User screen gets frozen and does not respond to any actions.
Error Messages
Seeing exceptions below exceptions before --- Thread Dump Starts ---
com.pega.pegarules.pub.context.RequestorLockException: Unable to synchronize on requestor H63B57088E0FF19CAFF71F2D1EE384D26 within 120 seconds:
Alert logs showed OOM condition due to pulling back a large number of rows from declarative table:
2016-11-08 01:49:51,393 GMT 7 PEGA0028 4625820672 4348654387
Collection usage threshold exceeded for long-lived object memory pool (Java heap) INFO:MemoryPool=Java heap;PoolType=long-lived object;Event=java.management.memory.collection.threshold.exceeded;TimeStamp=2016-11-08 01:49:51,382 GMT;Count=1;NotificationSequenceNo=0;Threshold=4348654387;Used=4625820672;Max=4831838208;Committed=4633067520;Init=4831838208;
2016-11-08 01:51:26,473 GMT 7 PEGA0027 512575 25000
Database RowSet list returned more than 25,000 rows: 512,575. SQL:
2016-11-08 01:51:26,474 GMT 7 PEGA0042 212279 2000
"Packaging of database query took more than threshold of 2,000 ms: 212,279 ms SQL: select pxInsName, pxChangeClass, pxDefIndex, pxRuleAppliesTo, pxChangeObjClass, pxChangePurpose,
pxRuleObjClass, pzInsKey, pxChangeType, pxRuleKey, pxRuleSet, pxObjClass, pxRuleSetVersion, pxChangeProperty from PEGA.pr_sys_decchg where pxChangeClass = ? "
2016-11-08 01:51:26,474 GMT 7 PEGA0027 512577 25000
Database ResultSet list returned more than 25,000 rows: 512,577. SQL: select pxInsName, pxChangeClass, pxDefIndex, pxRuleAppliesTo, pxChangeObjClass, pxChangePurpose, pxRuleObjClass,
pzInsKey, pxChangeType, pxRuleKey, pxRuleSet, pxObjClass, pxRuleSetVersion, pxChangeProperty from PEGA.pr_sys_decchg where pxChangeClass = ?
Steps to Reproduce
Regularly happening on the environment.
Root Cause
A defect in Pegasystems’ code or rules.
ExecuteRDB in com.pega.pegarules.exec.internal.declare.infengine.cbi.GlobalDeclarativeClass.listChangeDefinitions does not scale here, because it tries to load all the results into memory first causing OOM. So the alternative is to use BasicListSpecification which uses page size by default.
Resolution
Apply HFix-30490 and verify index on pxChangeClass column in table pr_sys_decchg.
Published December 15, 2016 - Updated October 8, 2020
Have a question? Get answers now.
Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.