Skip to main content

This content has been archived and is no longer being updated. Links may not function; however, this content may be relevant to outdated versions of the product.

Support Article

JMS MDB listener locks

SA-31403

Summary



There are a lot of database locks and lock related errors in Pega logs, which are related to JMS MDB listener messages. The error is shown as "ORA-02049: timeout: distributed transaction waiting for lock".

Also, sometimes there are "ORA-00060: deadlock detected while waiting for resource" in similar conditions. Suspect, that these two may be related and have one cause.

Locking affects system performance and user do not see any reasons for such long locks.

JMS handler start transaction in the database during commit phase (flushing differed operations) at the very end of the message processing and this should be fast (sub-second or seconds).

At the same time, user see errors in the log which say that connection was waiting for lock for 60 seconds. User database monitoring also confirms that there are row-level locks in Oracle at this time. What are the reasons for such long locks?


Error Messages



"ORA-02049: timeout: distributed transaction waiting for lock", "ORA-00060: deadlock detected while waiting for resource", see in the attachment.

2016-11-21 13:16:36,267 [stenerThreadPool : 0] [ STANDARD] [ ] [ AN_APP:01.01] ( internal.services.ServiceAPI) ERROR JMS|EQNotification|Services|ProcessMsg|A4C950867117CFB2A15D80C44B6B9F1AF - JMS service [EQNotification][Services][ProcessMsg] failed:Database-Saver-InsertUpdatefail AAA-BB-CCC-DDDD-EEEEE 2049 42000 ORA-02049: timeout: distributed transaction waiting for lock
:com.pega.pegarules.pub.database.DatabaseException: Database-Saver-InsertUpdatefail AAA-BB-CCC-DDDD-EEEEE 2049 42000 ORA-02049: timeout: distributed transaction waiting for lock

From: (A4C950867117CFB2A15D80C44B6B9F1AF:EQNotification)
SQL: update DATA.ABRFS_PARTY set BIC = ? , BIRTHDATE = ? , BLOCKCODE = ? , BUSINESSTYPECODE = ? , BUSINESSTYPEID = ? , CLUSTERCODE = ? , CLUSTERPRIORITY = ? , COMMENTS = ? , CONTACTSOURCECODE = ? , CUSTOMERCLOSEDDATE = ? , DATEFOUNDED = ? , DATEOFFIRSTCONTACT = ? , DISABLEREASONCODE = ? , FIRSTACCOPENDATE = ? , INCEPTIONBRANCHSRCCODE = ? , INDUSTRYCODE = ? , ISDISABLED = ? , ISDUPLICATE = ? , ISOUTFLOW = ? , KPP = ? , MANAGERID = ? , MARKETINGSEGMENTCODE = ? , MONTHLYSALARYFUND = ? , NAME = ? , NUMBEROFEMPLOYEES = ? , OFFICIALNAME = ? , OGRN = ? , OKVEDCODE = ? , OPENDATE = ? , OPFCODE = ? , OUTFLOWPOSITION = ? , OUTFLOWTRAFFICLIGHT = ? , PARENTPARTYID = ? , PINEQ = ? , PXCREATEDATETIME = ? , PXCREATEOPERATOR = ? , PXCREATEOPNAME = ? , PXCREATESYSTEMID = ? , PXINSNAME = ? , PXOBJCLASS = ? , PXUPDATEDATETIME = ? , PXUPDATEOPERATOR = ? , PXUPDATEOPNAME = ? , PXUPDATESYSTEMID = ? , PYFIRSTNAME = ? , PYFULLNAME = ? , PYID = ? , PYLABEL = ? , PYLASTNAME = ? , PYRULESETNAME = ? , REVENUE = ? , SECONDNAME = ? , SERVICEBRANCHSRCCODE = ? , SERVICECITYID = ? , SOURCECODE = ? , SOURCEID = ? , SRCSYSCODE = ? , STATUSCODE = ? , STATUSID = ? , SWIFT = ? , TIN = ? , WEBADDRESS = ? , pzPVStream = ? where pzInsKey = ?
SQL Inserts: <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <2014-11-08 04:21:40.0> <Riskadmin> <Riskadmin> <<null>> <D-3400335> <AAA-BB-CCC-DDDD-EEEEE> <2016-11-21 13:15:36.16> <<null>> <<null>> <CRMBMB> <<null>> <??????? ?????? ??????????> <D-3400335> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<null>> <<stream>> <AAA-BB-CCC-DDDD-EEEEE D-3400335>
Secondary Exception Info: ORA-00001: unique constraint (DATA.ABRFS_PARTY_PK) violated

Caused by SQL Problems.
Problem #1, SQLState 42000, Error code 2049: java.sql.SQLSyntaxErrorException: ORA-02049: timeout: distributed transaction waiting for lock

at com.pega.pegarules.data.internal.access.ExceptionInformation.createExceptionDueToDBFailure(ExceptionInformation.java:227)
at com.pega.pegarules.data.internal.access.DatabasePreparedStatementImpl.addBatch(DatabasePreparedStatementImpl.java:672)
at com.pega.pegarules.data.internal.access.Saver.saveAltOp(Saver.java:1374)
at com.pega.pegarules.data.internal.access.StatementOperationStore.performAltOps(StatementOperationStore.java:136)
at com.pega.pegarules.data.internal.access.DatabasePreparedStatementImpl.addBatch(DatabasePreparedStatementImpl.java:703)
at com.pega.pegarules.data.internal.access.Saver.saveNewInstanceWithNoDeletedColumnNotOnlyIfNew(Saver.java:1112)
at com.pega.pegarules.data.internal.access.Saver.saveInstance(Saver.java:840)
at com.pega.pegarules.data.internal.access.DatabaseImpl.performOps(DatabaseImpl.java:3028)
at com.pega.pegarules.data.internal.access.DatabaseImpl.attemptToProcessUpdates(DatabaseImpl.java:2433)
at com.pega.pegarules.data.internal.access.DatabaseImpl.processUpdates(DatabaseImpl.java:2283)
at com.pega.pegarules.data.internal.access.DatabaseImpl.flushDeferredOps(DatabaseImpl.java:3903)
at com.pega.pegarules.session.external.engineinterface.service.EngineAPI.processRequestInner(EngineAPI.java:470)
at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1203)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:930)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:815)
at com.pega.pegarules.session.external.engineinterface.service.EngineAPI.processRequest(EngineAPI.java:348)
at com.pega.pegarules.integration.engine.internal.services.jms.JMSListener.handleRequest(JMSListener.java:1287)
at com.pega.pegarules.integration.engine.internal.services.jms.JMSListener.runOnce(JMSListener.java:716)
at com.pega.pegarules.integration.engine.internal.services.jms.JMSListenerServant.invoke(JMSListenerServant.java:111)
at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._invokeEngine_privact(EngineImpl.java:312)
at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.invokeEngine(EngineImpl.java:263)
at com.pega.pegarules.session.internal.engineinterface.etier.ejb.EngineBean.invokeEngine(EngineBean.java:225)
at sun.reflect.GeneratedMethodAccessor78.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:419)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:460)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingException(PRBootstrap.java:482)
at com.pega.pegarules.internal.etier.ejb.EngineBeanBoot.invokeEngine(EngineBeanBoot.java:168)
at com.pega.pegarules.internal.etier.interfaces.EJSLocalStatelessEngineCMT_021cb84d.invokeEngine(Unknown Source)
at com.pega.pegarules.session.internal.engineinterface.etier.mdb.PRJMSListener.callEngine(PRJMSListener.java:207)
at com.pega.pegarules.session.internal.engineinterface.etier.mdb.PRJMSListener.onMessage(PRJMSListener.java:129)
at sun.reflect.GeneratedMethodAccessor180.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:419)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:460)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:509)
at com.pega.pegarules.internal.etier.mdb.PRJMSListenerBoot.onMessage(PRJMSListenerBoot.java:91)
at com.ibm.ejs.jms.listener.MDBWrapper$PriviledgedOnMessage.run(MDBWrapper.java:309)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:63)
at com.ibm.ejs.jms.listener.MDBWrapper.callOnMessage(MDBWrapper.java:266)
at com.ibm.ejs.jms.listener.MDBWrapper.onMessage(MDBWrapper.java:235)
at com.ibm.mq.jms.MQSession$FacadeMessageListener.onMessage(MQSession.java:133)
at com.ibm.msg.client.jms.internal.JmsSessionImpl.run(JmsSessionImpl.java:2911)
at com.ibm.msg.client.jms.internal.JmsXAQueueSessionImpl$1.run(JmsXAQueueSessionImpl.java:1007)
at com.ibm.mq.jms.MQSession.run(MQSession.java:958)
at com.ibm.ejs.jms.JMSSessionHandle.run(JMSSessionHandle.java:1056)
at com.ibm.ejs.jms.listener.ServerSession.connectionConsumerOnMessage(ServerSession.java:1090)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:760)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:726)
at sun.reflect.GeneratedMethodAccessor179.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:47)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:100)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:132)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:581)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)
Caused by: java.sql.SQLSyntaxErrorException: ORA-02049: timeout: distributed transaction waiting for lock

at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:399)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1059)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:522)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:587)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:225)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:53)
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:943)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1150)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:4798)
at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:4875)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1361)
at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.pmiExecuteUpdate(WSJdbcPreparedStatement.java:1187)
at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.executeUpdate(WSJdbcPreparedStatement.java:804)
at com.pega.pegarules.data.internal.access.DatabasePreparedStatementImpl.executeUpdate(DatabasePreparedStatementImpl.java:483)
at com.pega.pegarules.data.internal.access.DatabasePreparedStatementImpl.addBatch(DatabasePreparedStatementImpl.java:609)
... 60 more

Steps to Reproduce



Not sure what conditions trigger the ORA-02049.


Root Cause



An issue in the custom application code or rules.

Resolution



Issue appears to match the scenario described in the below Support Article:

https://pdn.pega.com/support-articles/lockgoneexception-container-managed-transaction-processing

As documented, transaction error can occurs when the system has been configured using Container Managed Transaction option with XA database driver if the JMSMDB activity has been implemented with "Commit" or "Write-Now".

 

Suggest Edit

Published December 15, 2016 - Updated October 8, 2020

Did you find this content helpful? Yes No

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.

We'd prefer it if you saw us at our best.

Pega Community has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice
Contact us