Support Article
MDB Listener stop processing messages
SA-37452
Summary
JMS messages, received by mean of custom MDB Listener deployed on WebSphere, all of sudden are not being processed.
There are no messages queued for processing.
According to the infrastructure team MQ messages are passed to the application for processing.
Error Messages
5/2/17 9:12:17:085 CEST] 0000006b TimeoutManage I WTRN0006W: Transaction 0000015BC7FECFFD00000001181223B42F2B154608BD04E539C2DEE5A712D3E71B634F840000015BC7FECFFD00000001181223B42F2B154608BD04E539C2DEE5A712D3E71B634F8400000001 has timed out after 120 seconds.
[5/2/17 9:12:17:090 CEST] 0000006b TimeoutManage I WTRN0124I: When the timeout occurred the thread with which the transaction is, or was most recently, associated was Thread[WMQJCAResourceAdapter : 3,5,main]. The stack trace of this thread when the timeout occurred was:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:164)
java.net.SocketInputStream.read(SocketInputStream.java:134)
oracle.net.ns.Packet.receive(Packet.java:282)
oracle.net.ns.DataPacket.receive(DataPacket.java:103)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:230)
oracle.net.ns.NetInputStream.read(NetInputStream.java:175)
oracle.net.ns.NetInputStream.read(NetInputStream.java:100)
oracle.net.ns.NetInputStream.read(NetInputStream.java:85)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:122)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:78)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1179)
oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1155)
oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:279)
oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186)
oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:366)
oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:752)
oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:359)
oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:531)
oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:221)
oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:503)
oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:280)
oracle.jdbc.xa.client.OracleXADataSource.getPooledConnection(OracleXADataSource.java:466)
oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(OracleXADataSource.java:154)
oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(OracleXADataSource.java:128)
com.ibm.ws.rsadapter.spi.InternalGenericDataStoreHelper$1.run(InternalGenericDataStoreHelper.java:1365)
com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
com.ibm.ws.rsadapter.spi.InternalGenericDataStoreHelper.getPooledConnection(InternalGenericDataStoreHelper.java:1384)
com.ibm.ws.rsadapter.spi.WSRdbDataSource.getPooledConnection(WSRdbDataSource.java:2154)
com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.getConnection(WSManagedConnectionFactoryImpl.java:1787)
com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.createManagedConnection(WSManagedConnectionFactoryImpl.java:1557)
com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.createManagedConnection(WSManagedConnectionFactoryImpl.java:1113)
com.ibm.ejs.j2c.FreePool.createManagedConnectionWithMCWrapper(FreePool.java:2162)
com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1840)
com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3818)
com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3094)
com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1548)
com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:1031)
com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:644)
com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:611)
com.pega.pegarules.data.internal.access.DataSourceManager.getConnection(DataSourceManager.java:221)
com.pega.pegarules.data.internal.access.DatabaseConnectionImpl.activate(DatabaseConnectionImpl.java:1471)
com.pega.pegarules.data.internal.access.ConnectionManagerImpl.getConnectionFromDataSourceIdentity(ConnectionManagerImpl.java:1008)
com.pega.pegarules.data.internal.access.ConnectionManagerImpl.getConnection(ConnectionManagerImpl.java:1189)
com.pega.pegarules.data.internal.access.ConnectionManagerImpl.getConnection(ConnectionManagerImpl.java:1129)
com.pega.pegarules.data.internal.access.ConnectionManagerImpl.getConnection(ConnectionManagerImpl.java:343)
com.pega.pegarules.data.internal.access.ThreadConnectionStoreImpl.getConnection(ThreadConnectionStoreImpl.java:424)
com.pega.pegarules.data.internal.access.RDBPageResultPackager.getConnection(RDBPageResultPackager.java:923)
com.pega.pegarules.data.internal.access.DatabaseImpl.list(DatabaseImpl.java:6035)
com.pega.pegarules.data.internal.access.DatabaseImpl.executeRDB(DatabaseImpl.java:6676)
com.pega.pegarules.data.internal.access.DatabaseImpl.executeRDB(DatabaseImpl.java:6632)
com.pega.pegarules.data.internal.access.DatabaseImpl.executeRDB(DatabaseImpl.java:6613)
com.pega.pegarules.session.internal.authorization.context.AppCentricRulesetIndexWriter.doesTopAppHashNeedToBeInsertedIntoFlattenedHierarchy(AppCentricRulesetIndexWriter.java:562)
com.pega.pegarules.session.internal.authorization.context.AppCentricRulesetIndexWriter.populateApplicationRuleHierarchy(AppCentricRulesetIndexWriter.java:501)
com.pega.pegarules.session.internal.authorization.context.AppCentricRulesetIndexWriter.populateApplicationRuleHierarchy(AppCentricRulesetIndexWriter.java:613)
com.pega.pegarules.session.internal.authorization.context.AppCentricRulesetIndexWriter.populateApplicationRuleHierarchy(AppCentricRulesetIndexWriter.java:634)
com.pega.pegarules.session.internal.authorization.context.AppCentricRulesetIndexWriter.writeApplicationInfoToDatabase(AppCentricRulesetIndexWriter.java:193)
com.pega.pegarules.session.internal.authorization.context.custom.ManagedApplicationContext.updateDBRSLAndAppStack(ManagedApplicationContext.java:847)
com.pega.pegarules.session.internal.authorization.context.custom.ManagedApplicationContext.updateToParent(ManagedApplicationContext.java:430)
com.pega.pegarules.session.internal.authorization.context.custom.ManagedApplicationContext.getChangeCount(ManagedApplicationContext.java:390)
com.pega.pegarules.session.internal.authorization.context.AbstractApplicationContext.getAppInfo(AbstractApplicationContext.java:52)
com.pega.pegarules.session.internal.authorization.context.AppCentricRulesetIndexWriter.populateApplicationRuleHierarchy(AppCentricRulesetIndexWriter.java:629)
com.pega.pegarules.session.internal.authorization.context.AppCentricRulesetIndexWriter.writeApplicationInfoToDatabase(AppCentricRulesetIndexWriter.java:193)
com.pega.pegarules.session.internal.authorization.context.custom.ManagedApplicationContext.updateDBRSLAndAppStack(ManagedApplicationContext.java:847)
com.pega.pegarules.session.internal.authorization.context.custom.ManagedApplicationContext.<init>(ManagedApplicationContext.java:107)
com.pega.pegarules.session.internal.authorization.context.custom.CustomApplicationContext.<init>(CustomApplicationContext.java:75)
com.pega.pegarules.session.internal.authorization.context.ContextUtils.ensureCustomizable(ContextUtils.java:890)
com.pega.pegarules.session.internal.authorization.Authorization.alterRuleSetList(Authorization.java:1035)
com.pega.pegarules.exec.internal.declare.infengine.pages.DeclarativePageDirectoryImpl.runRule(DeclarativePageDirectoryImpl.java:985)
com.pega.pegarules.exec.internal.declare.infengine.pages.DeclarativePageDirectoryImpl.runLoadActivity(DeclarativePageDirectoryImpl.java:452)
com.pega.pegarules.session.internal.mgmt.base.handler.ReadOnlyDataPageHandler.loadDataPage(ReadOnlyDataPageHandler.java:218)
com.pega.pegarules.session.internal.mgmt.base.handler.ReadOnlyDataPageHandler.findDataPage(ReadOnlyDataPageHandler.java:399)
com.pega.pegarules.session.internal.mgmt.base.AbstractPageDirectory.findDataPageInCurrentDir(AbstractPageDirectory.java:920)
com.pega.pegarules.session.internal.mgmt.base.AbstractPageDirectory.getDirectPage(AbstractPageDirectory.java:713)
com.pega.pegarules.session.internal.mgmt.base.ThreadPageDir.getDirectPage(ThreadPageDir.java:469)
com.pega.pegarules.session.internal.mgmt.base.ThreadPageDir.getPage(ThreadPageDir.java:427)
com.pega.pegarules.session.internal.mgmt.PRThreadImpl.getPage(PRThreadImpl.java:539)
com.pega.pegarules.session.internal.mgmt.Executable.findPage(Executable.java:2086)
com.pega.pegarules.session.internal.mgmt.Executable.findPage(Executable.java:2039)
com.pega.pegarules.session.internal.mgmt.Executable.findPage(Executable.java:2021)
com.pega.pegarules.session.internal.mgmt.authentication.Authentication.getSecuritySettingDeclarePage(Authentication.java:652)
com.pega.pegarules.session.internal.mgmt.authentication.Authentication.getSecuritySettingAsBoolean(Authentication.java:661)
com.pega.pegarules.session.internal.mgmt.authentication.Authentication.doAuthentication(Authentication.java:441)
com.pega.pegarules.integration.engine.internal.services.ServiceAPI.handleAuthentication(ServiceAPI.java:1578)
com.pega.pegarules.session.external.engineinterface.service.EngineAPI.activityExecutionProlog(EngineAPI.java:548)
com.pega.pegarules.session.external.engineinterface.service.EngineAPI.processRequestInner(EngineAPI.java:383)
sun.reflect.GeneratedMethodAccessor214.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
java.lang.reflect.Method.invoke(Method.java:620)
com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1270)
com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:1008)
com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:841)
com.pega.pegarules.session.external.engineinterface.service.EngineAPI.processRequest(EngineAPI.java:331)
com.pega.pegarules.integration.engine.internal.services.jms.JMSListener.handleRequest(JMSListener.java:1266)
com.pega.pegarules.integration.engine.internal.services.jms.JMSListener.runOnce(JMSListener.java:708)
com.pega.pegarules.integration.engine.internal.services.jms.JMSListenerServant.invoke(JMSListenerServant.java:112)
com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._invokeEngine_privact(EngineImpl.java:315)
com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.invokeEngine(EngineImpl.java:263)
com.pega.pegarules.session.internal.engineinterface.etier.ejb.EngineBean.invokeEngine(EngineBean.java:225)
sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
java.lang.reflect.Method.invoke(Method.java:620)
com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:370)
com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:411)
com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:223)
com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingException(AppServerBridgeToPega.java:245)
com.pega.pegarules.internal.etier.ejb.EngineBeanBoot.invokeEngine(EngineBeanBoot.java:168)
com.pega.pegarules.internal.etier.interfaces.EJSLocalStatelessEngineCMT_021cb84d.invokeEngine(Unknown Source)
com.pega.pegarules.session.internal.engineinterface.etier.mdb.PRJMSListener.callEngine(PRJMSListener.java:208)
com.pega.pegarules.session.internal.engineinterface.etier.mdb.PRJMSListener.onMessage(PRJMSListener.java:130)
sun.reflect.GeneratedMethodAccessor212.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
java.lang.reflect.Method.invoke(Method.java:620)
com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:370)
com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:411)
com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:223)
com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:272)
com.pega.pegarules.internal.etier.mdb.PRJMSListenerBoot.onMessage(PRJMSListenerBoot.java:91)
com.ibm.ejs.container.WASMessageEndpointHandler.invokeJMSMethod(WASMessageEndpointHandler.java:138)
com.ibm.ws.ejbcontainer.mdb.MessageEndpointHandler.invokeMdbMethod(MessageEndpointHandler.java:1146)
com.ibm.ws.ejbcontainer.mdb.MessageEndpointHandler.invoke(MessageEndpointHandler.java:844)
com.sun.proxy.$Proxy26.onMessage(Unknown Source)
com.ibm.mq.connector.inbound.MessageEndpointWrapper.onMessage(MessageEndpointWrapper.java:131)
com.ibm.mq.jms.MQSession$FacadeMessageListener.onMessage(MQSession.java:133)
com.ibm.msg.client.jms.internal.JmsSessionImpl.run(JmsSessionImpl.java:2911)
com.ibm.mq.jms.MQSession.run(MQSession.java:958)
com.ibm.mq.connector.inbound.ASFWorkImpl.doDelivery(ASFWorkImpl.java:97)
com.ibm.mq.connector.inbound.AbstractWorkImpl.run(AbstractWorkImpl.java:235)
com.ibm.ejs.j2c.work.WorkProxy$RunWork.run(WorkProxy.java:281)
java.security.AccessController.doPrivileged(AccessController.java:426)
javax.security.auth.Subject.doAs(Subject.java:494)
com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java:133)
com.ibm.ejs.j2c.work.WorkProxy$RunWork.run(WorkProxy.java:284)
com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:63)
com.ibm.ejs.j2c.work.WorkProxy.run(WorkProxy.java:666)
com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892).
Steps to Reproduce
Not Applicable
Root Cause
A defect or configuration issue in the operating environment.
The former error caused the message binding to stop on the application server due to the default setting being used for the associated Activation Specification.
Activation Specs are used in latest WebSphere versions to configure inbound message delivery to message-driven beans (MDBs) running inside the Application Server.
This is because activation specification advanced properties "Stop endpoint if message delivery fails" and "Number of sequential delivery failures before suspending endpoint" are using by default respectively the values of "true" and 0.
This causes the messaging endpoint to stop at the first occurrence of an unhandled exception to protect from the risk of incurring in the poison message syndrome.
For more details refer to IBM Knowledge Center page "WebSphere MQ messaging provider activation specification advanced properties"
Resolution
Make the following change to the operating environment:
The issue has been resolved resuming (restart) the Activation Specification associated with the stopped endpoint.
More advanced and reliable redelivery policies can be put in place engaging the Broker and the Application Server Administrators to configure a backout queue where to route messages after a certain amount of failed delivery attempts.
This queue can be monitored to take corrective actions on pending messages allowing admistrators to look for common patterns and potential software problems.
Published May 6, 2017 - Updated October 8, 2020
Have a question? Get answers now.
Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.