Skip to main content

This content has been archived and is no longer being updated. Links may not function; however, this content may be relevant to outdated versions of the product.

Support Article

Deadlock while deploying a custom MDB on JBoss EAP 6.4

SA-24926

Summary



Pega 7.2 EAR installed on JBoss EAP 6.4 has been customized to deploy a custom MDB connecting to an external JMS destination using MQ resource adapter.
The deployment process hangs after many exceptions appeared in the logs.
A deadlock is found in the thread dump generated using jstack.

 

Error Messages



2016-06-24 13:54:16,475 [default-threads - 37] (       etier.mdb.PRJMSListener) ERROR   - PRPC Engine invocation failed, javax.ejb.EJBException: Unable to locate implementation of PegaRULES Engine class for EngineBMT 
2016-06-24 13:54:54,710 [default-threads - 37] (ervices.jms.JMSListenerServant) ERROR   - PegaRULES unavailable, discarding function initfor JMS Listener, name=unknown with id=unknown
2016-06-24 13:54:54,710 [default-threads - 37] (       etier.mdb.PRJMSListener) ERROR   - Failed to obtain Listener ID for JMS MDB Listener: YOUR_MDB

Found one Java-level deadlock:
=============================
"default-threads - 49":
  waiting to lock monitor 0x00000000012297d8 (object 0x00000006a7c9f438, a com.pega.pegarules.integration.engine.internal.services.listener.ListenerStateManagerImpl),
  which is held by "default-threads - 36"
"default-threads - 36":
  waiting to lock monitor 0x00002aaac89170e8 (object 0x00000006a0eb4ec0, a com.pega.pegarules.cluster.internal.PRClusterProviderImpl),
  which is held by "your_hostname.fqdn"
"your_hostname.fqdn":
  waiting to lock monitor 0x00002aaac8783d58 (object 0x00000006830f59d0, a java.lang.Class),
  which is held by "default-threads - 36"


Steps to Reproduce

  1. Customize the Pega 7.2 EAR adding the deployment descriptors for a custom MDB JMS Listener that is connecting to MQ using a resource adapter.
  2. Deploy the modified EAR file and check the logs.
  3. Observe that after initial exceptions the deployment process hangs.


Root Cause



A defect or configuration issue in the operating environment:
Since there are pending messages in the JMS destination the MDB is activated immediately and start receiving messages before the Stateless Session Bean EngineBMT is completely deployed.
The MDB should wait for the PRPC Engine to complete the bootstrap before start receiving messages. So this is deployment order issue.
This cause a race condition that leads to deadlock situation.

Resolution



In similar scenarios it is recommended to deploy the MDB disconnected from the JMS destination to prevent messages from being delivered to the MDB (which is still starting) during server startup.
In JBoss this can be achieved leveraging the Delivery Active JBoss extension.
In the jboss-ejb3.xml file, configure the value of active as false to mark that the MDB will not be receiving messages as soon as it is deployed:


<jboss:ejb-jar xmlns:jboss="http://www.jboss.com/xml/ns/javaee"
                xmlns="http://java.sun.com/xml/ns/javaee"
                xmlns:s="urn:security:1.1"
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                xmlns:d="urn:delivery-active:1.0"
                xmlns:mdb="urn:resource-adapter-binding"
                xsi:schemaLocation="http://www.jboss.com/xml/ns/javaee 
                                    http://www.jboss.org/j2ee/schema/jboss-ejb3-2_0.xsd
                                    http://java.sun.com/xml/ns/javaee 
                                    http://java.sun.com/xml/ns/javaee/ejb-jar_3_1.xsd"
                version="3.1"
                impl-version="2.0">
  <enterprise-beans>
  ...
  <message-driven>
     <ejb-name>YourMDB</ejb-name>
         <activation-config>
             <activation-config-property>
                 <activation-config-property-name>destination</activation-config-property-name>
                 <activation-config-property-value>java:/YOUR_DESTINATION_JNDI_NAME</activation-config-property-value>
             </activation-config-property>
         </activation-config>
  </message-driven>
  </enterprise-beans>
  <assembly-descriptor>
     <d:deliveryType>
        <ejb-name>YourMDB</ejb-name>
        <d:active>false</d:active>
     </d:deliveryType>
  </assembly-descriptor>
</jboss:ejb-jar>

The message delivery can be started and stopped using the JBoss CLI.

The following management operations can be used to enable or disable the delivery for the MDB at runtime
To use them, connect to the JBoss instance using the JBoss CLI, then enter the path of the MDB you want to manage delivery for: 

/bin/jboss-cli.sh
connect <host>:<port>
[[email protected]_host:your_port /] /deployment=prpc_j2ee14_jboss61JBM.ear/subdeployment=prbeans.jar/subsystem=ejb3/message-driven-bean=<your MDB name>:stop-delivery()
{"outcome" => "success"}

[[email protected]your_host:your_port /] /deployment=prpc_j2ee14_jboss61JBM.ear/subdeployment=prbeans.jar/subsystem=ejb3/message-driven-bean=<your MDB name>:start-delivery()
{"outcome" => "success"}

Suggest Edit

Published July 2, 2016 - Updated October 8, 2020

Did you find this content helpful? Yes No

100% found this useful

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.

We'd prefer it if you saw us at our best.

Pega Community has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice
Contact us