Support Article

Servers are unstable and facing latency issues

SA-30963

Summary



User reports that their servers are unstable and are facing latency issues. User screen gets frozen and does not respond to any actions.

Error Messages


Seeing exceptions below exceptions before --- Thread Dump Starts ---
com.pega.pegarules.pub.context.RequestorLockException: Unable to synchronize on requestor H63B57088E0FF19CAFF71F2D1EE384D26 within 120 seconds:

Alert logs showed OOM condition due to pulling back a large number of rows from declarative table:

2016-11-08 01:49:51,393 GMT    7    PEGA0028    4625820672    4348654387 
Collection usage threshold exceeded for long-lived object memory pool (Java heap) INFO:MemoryPool=Java heap;PoolType=long-lived object;Event=java.management.memory.collection.threshold.exceeded;TimeStamp=2016-11-08 01:49:51,382 GMT;Count=1;NotificationSequenceNo=0;Threshold=4348654387;Used=4625820672;Max=4831838208;Committed=4633067520;Init=4831838208; 

2016-11-08 01:51:26,473 GMT    7    PEGA0027    512575    25000 
Database RowSet list returned more than 25,000 rows: 512,575. SQL: 

2016-11-08 01:51:26,474 GMT    7    PEGA0042    212279    2000 
"Packaging of database query took more than threshold of 2,000 ms: 212,279 ms    SQL: select pxInsName, pxChangeClass, pxDefIndex, pxRuleAppliesTo, pxChangeObjClass, pxChangePurpose, 

pxRuleObjClass, pzInsKey, pxChangeType, pxRuleKey, pxRuleSet, pxObjClass, pxRuleSetVersion, pxChangeProperty from PEGA.pr_sys_decchg where pxChangeClass = ? " 

2016-11-08 01:51:26,474 GMT    7    PEGA0027    512577    25000 
Database ResultSet list returned more than 25,000 rows: 512,577. SQL: select pxInsName, pxChangeClass, pxDefIndex, pxRuleAppliesTo, pxChangeObjClass, pxChangePurpose, pxRuleObjClass, 
pzInsKey, pxChangeType, pxRuleKey, pxRuleSet, pxObjClass, pxRuleSetVersion, pxChangeProperty from PEGA.pr_sys_decchg where pxChangeClass = ?


Steps to Reproduce



Regularly happening on the environment.


Root Cause



A defect in Pegasystems’ code or rules.
E
xecuteRDB in com.pega.pegarules.exec.internal.declare.infengine.cbi.GlobalDeclarativeClass.listChangeDefinitions does not scale here, because it tries to load all the results into memory first causing OOM. So the alternative is to use BasicListSpecification which uses page size by default.

Resolution



Apply HFix-30490 and verify index on pxChangeClass column in table pr_sys_decchg.

 

Published November 30, 2016 - Updated December 15, 2016

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.