Support Article
Load test performance failure for PRPC 6.3SP1 application
SA-8288
Summary
Load test performance reached 50% for PRPC 6.3 SP1 Application and crashed due to DB dead locks. The DB table dead locks resulted in the websphere JVM hung threads occurring. This lead to the crash of the performance.
Error Messages
system crash down due to dead locks of Db tables - from database log:
DBU1 -913 : DEADLOCK
update pr_sys_queue_sla set pxProcessingNodeId = ?,
pxLastExecutionDateTime = ?, pyItemStatus = 'Now-Processing'
where pyItemId = ? and pyItemStatus != 'Now-Processing'
Tuesday 3/17/2015: Starting at 12:21 thru 12:58 ....
DEADLOCK: : REASON 00C90088 TYPE 00000304
on .... NAME DPE001U1.PGTSP026.X'00000238'.X'01'
Table name: PR_SYS_QUEUE_SLA (DPE001U1.PGTSP026)
12.21.45 STC10896 DSNT375I +DBU1 PLAN=DISTSERV WITH 064
064 CORRELATION-ID=db2jcc_appli
064 CONNECTION-ID=SERVER
064 LUW-ID=GA68C9B5.AB1F.CEA8EA0EDD77=154263
064
064 THREAD-INFO=FPETPEG1:10.104.201.181:FPETPEG1:db2jcc_applicat
064 IC:33811:*:*
064 IS DEADLOCKED WITH PLAN=DISTSERV WITH
064 CORRELATION-ID=db2jcc_appli
064 CONNECTION-ID=SERVER
064 LUW-ID=GA68C9B5.AB1C.CEA8EA0ECAEC=154260
064
064 THREAD-INFO=FPETPEG1:10.104.201.181:FPETPEG1:db2jcc_applicat
064 IC:33811:*:*
064 ON MEMBER DBU1
12.21.55 STC10896 DSNT375I +DBU1 PLAN=DISTSERV WITH 577
577 CORRELATION-ID=db2jcc_appli
577 CONNECTION-ID=SERVER
577 LUW-ID=GA68C9B5.AB6A.CEA8EA533EE6=154371
577
577 THREAD-INFO=FPETPEG1:10.104.201.181:FPETPEG1:db2jcc_applicat
577 IC:33811:*:*
577 IS DEADLOCKED WITH PLAN=DISTSERV WITH
577 CORRELATION-ID=db2jcc_appli
577 CONNECTION-ID=SERVER
577 LUW-ID=GA68C9B5.AB1C.CEA8EA0ECAEC=154260
Steps to Reproduce
Dead locks are occurring during performance testing.
Root Cause
The root cause of this problem is a defect in customer application code/rules.
Resolution
Missing indexes for pr_sys_queue_sla table was making DB escalate locks on this table, which led to resource contention and deadlocks.
Update indexes for the pr_sys_queue_sla to match the indexes that exist in PRPC 6.3SP1 install.
Published January 31, 2016 - Updated October 8, 2020
Have a question? Get answers now.
Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.