Support Article

Cluster nodes are not in sync -agent running issue

SA-36739

Summary



User is having Pega 7.1.7 2x2 cluster node, where application is working fine in one node but not working in second node.

Mainly the inboundsmsflow agent is not working properly in all nodes.

Application functionality not work properly in one node where as in another node it is working fine in a cluster node environment.

User can see errors in the logs related to agents.



Error Messages



Some error messages are present.
PegaMKT-Engine 14 InboundSMSOfferFlow Advanced Every 30 s Exception at 20170326T201836.257 GMT: com.pega.pegarules.pub.PRRuntimeException: caught exception during autocommit check at com.pega.pegarules.data.internal.access.ConnectionManagerImpl.returnConnection(ConnectionManagerImpl.java:2228) at com.pega.pegarules.data.internal.access.ThreadConnectionStoreImpl.returnConnection(ThreadConnectionStoreImpl.java:504) at com.pega.pegarules.data.internal.access.RDBPageResultPackager.close(RDBPageResultPackager.java:897) at com.pega.pegarules.data.internal.access.DatabaseImpl.executeRDB(DatabaseImpl.java:6496) at com.pega.pegarules.data.internal.access.DatabaseImpl.executeRDB(DatabaseImpl.java:6429) at com.pega.pegarules.data.internal.access.DatabaseImpl.listRDB(DatabaseImpl.java:6377) at com.pegarules.generated.activity.ra_action_associatecustomer_7993a9d9ff7d7b37c3d6f51aa007b5b1.step4_circum0(ra_action_associatecustomer_7993a9d9ff7d7b37c3d6f51aa007b5b1.java:406) at com.pegarules.generated.activity.ra_action_associatecustomer_7993a9d9ff7d7b37c3d6f51aa007b5b1.perform(ra_action_associatecustomer_7993a9d9ff7d7b37c3d6f51aa007b5b1.java:133) at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3375) at com.pega.pegarules.session.internal.mgmt.Executable.invokeActivity(Executable.java:10450) at com.pegarules.generated.activity.ra_action_initiateoffer_dacbe3c0f55262b970926f56eb699275.step5_4_circum0(ra_action_initiateoffer_dacbe3c0f55262b970926f56eb699275.java:1397) at com.pegarules.generated.activity.ra_action_initiateoffer_dacbe3c0f55262b970926f56eb699275.step5_0_circum0_PegaMKT_Data_Inbound_SMS(ra_action_initiateoffer_dacbe3c0f55262b970926f56eb699275.java:684) at com.pegarules.generated.activity.ra_action_initiateoffer_dacbe3c0f55262b970926f56eb699275.perform(ra_action_initiateoffer_dacbe3c0f55262b970926f56eb699275.java:163) at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3375) at com.pega.pegarules.session.internal.async.agent.QueueProcessor.runActivity(QueueProcessor.java:719) at com.pega.pegarules.session.internal.async.agent.QueueProcessor.execute(QueueProcessor.java:600) at com.pega.pegarules.session.internal.async.BatchRequestorTask.run(BatchRequestorTask.java:885) at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1177) at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:926) at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:825) at com.pega.pegarules.session.internal.async.BatchRequestorTask.run(BatchRequestorTask.java:618) at com.pega.jsr166backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:991) at com.pega.jsr166backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:532) at java.lang.Thread.run(Thread.java:744) Caused by: java.sql.SQLRecoverableException: Closed Connection at oracle.jdbc.driver.PhysicalConnection.getAutoCommit(PhysicalConnection.java:2254) at com.pega.pegarules.data.internal.access.DatabaseConnectionImpl.getAutoCommit(DatabaseConnectionImpl.java:806) at com.pega.pegarules.data.internal.access.ConnectionManagerImpl.returnConnection(ConnectionManagerImpl.java:2221) ... 23 more


2017-03-30 10:20:39,494 [ saryppapp1] [ STANDARD] [ ] [ ZainNBA:01.01.01] (onse.PegaMKT_Work_Offer.Action) ERROR - Error in Obj-Open
com.pega.pegarules.pub.database.BadInputException: Cannot open or save instances of class Data-BatchOutPR89--the class might be abstract, or it might have no defined keys, or it might descend from Code-.
From: (REQUESTOR_ID)
at com.pega.pegarules.pub.database.BadInputException.createClassCannotHaveInstancesException(BadInputException.java:167)

2017-03-30 12:21:41,692 [ pool-18-thread-1] [ STANDARD] [ ] [ ] (dnode.api.DNodeServiceListener) ERROR - Cannot initialize DNode context and start Cassandra, PRPC continuing as non-DNode
com.pega.dsm.dnode.api.DNodeException: Cannot bootstrap cassandra
at com.pega.dsm.dnode.impl.cassandra.Cassandra.bootstrap(Cassandra.java:102)
at com.pega.dsm.dnode.api.DNodeBootstrap.bootstrap(DNodeBootstrap.java:34)
at com.pega.dsm.dnode.api.DNodeServiceListener$1.run(DNodeServiceListener.java:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IllegalStateException: Dnode ports are in use, looks like another instance of dnode is running on this machine
at com.pega.dsm.dnode.impl.cassandra.Cassandra.bootstrap(Cassandra.java:97)
... 7 more


Steps to Reproduce



Not Applicable


Root Cause



A defect or configuration issue in the operating environment.

The multinode environment has same IP and temporary directory for multiple nodes, causing PRPC to generate same nodeid for multiple prpc jvms.

Resolution



Here’s the explanation for the reported behavior:

1. Setup only one dnode per system.
2. Update weblogic startup scripts to define separate Pega Temp dir location for every managed servers.
3. Update prconfig.xml to add env variable "cluster/hazelcast/interfaces" to set external IP that is accessible from other systems in the cluster.

Published April 19, 2017 - Updated May 5, 2017

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.