Support Article
Cluster nodes are not in sync -agent running issue
SA-36739
Summary
User is having Pega 7.1.7 2x2 cluster node, where application is working fine in one node but not working in second node.
Mainly the inboundsmsflow agent is not working properly in all nodes.
Application functionality not work properly in one node where as in another node it is working fine in a cluster node environment.
User can see errors in the logs related to agents.
Error Messages
Some error messages are present.
PegaMKT-Engine 14 InboundSMSOfferFlow Advanced Every 30 s Exception at 20170326T201836.257 GMT: com.pega.pegarules.pub.PRRuntimeException: caught exception during autocommit check at com.pega.pegarules.data.internal.access.ConnectionManagerImpl.returnConnection(ConnectionManagerImpl.java:2228) at com.pega.pegarules.data.internal.access.ThreadConnectionStoreImpl.returnConnection(ThreadConnectionStoreImpl.java:504) at com.pega.pegarules.data.internal.access.RDBPageResultPackager.close(RDBPageResultPackager.java:897) at com.pega.pegarules.data.internal.access.DatabaseImpl.executeRDB(DatabaseImpl.java:6496) at com.pega.pegarules.data.internal.access.DatabaseImpl.executeRDB(DatabaseImpl.java:6429) at com.pega.pegarules.data.internal.access.DatabaseImpl.listRDB(DatabaseImpl.java:6377) at com.pegarules.generated.activity.ra_action_associatecustomer_7993a9d9ff7d7b37c3d6f51aa007b5b1.step4_circum0(ra_action_associatecustomer_7993a9d9ff7d7b37c3d6f51aa007b5b1.java:406) at com.pegarules.generated.activity.ra_action_associatecustomer_7993a9d9ff7d7b37c3d6f51aa007b5b1.perform(ra_action_associatecustomer_7993a9d9ff7d7b37c3d6f51aa007b5b1.java:133) at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3375) at com.pega.pegarules.session.internal.mgmt.Executable.invokeActivity(Executable.java:10450) at com.pegarules.generated.activity.ra_action_initiateoffer_dacbe3c0f55262b970926f56eb699275.step5_4_circum0(ra_action_initiateoffer_dacbe3c0f55262b970926f56eb699275.java:1397) at com.pegarules.generated.activity.ra_action_initiateoffer_dacbe3c0f55262b970926f56eb699275.step5_0_circum0_PegaMKT_Data_Inbound_SMS(ra_action_initiateoffer_dacbe3c0f55262b970926f56eb699275.java:684) at com.pegarules.generated.activity.ra_action_initiateoffer_dacbe3c0f55262b970926f56eb699275.perform(ra_action_initiateoffer_dacbe3c0f55262b970926f56eb699275.java:163) at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3375) at com.pega.pegarules.session.internal.async.agent.QueueProcessor.runActivity(QueueProcessor.java:719) at com.pega.pegarules.session.internal.async.agent.QueueProcessor.execute(QueueProcessor.java:600) at com.pega.pegarules.session.internal.async.BatchRequestorTask.run(BatchRequestorTask.java:885) at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1177) at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:926) at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:825) at com.pega.pegarules.session.internal.async.BatchRequestorTask.run(BatchRequestorTask.java:618) at com.pega.jsr166backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:991) at com.pega.jsr166backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:532) at java.lang.Thread.run(Thread.java:744) Caused by: java.sql.SQLRecoverableException: Closed Connection at oracle.jdbc.driver.PhysicalConnection.getAutoCommit(PhysicalConnection.java:2254) at com.pega.pegarules.data.internal.access.DatabaseConnectionImpl.getAutoCommit(DatabaseConnectionImpl.java:806) at com.pega.pegarules.data.internal.access.ConnectionManagerImpl.returnConnection(ConnectionManagerImpl.java:2221) ... 23 more
2017-03-30 10:20:39,494 [ saryppapp1] [ STANDARD] [ ] [ ZainNBA:01.01.01] (onse.PegaMKT_Work_Offer.Action) ERROR - Error in Obj-Open
com.pega.pegarules.pub.database.BadInputException: Cannot open or save instances of class Data-BatchOutPR89--the class might be abstract, or it might have no defined keys, or it might descend from Code-.
From: (REQUESTOR_ID)
at com.pega.pegarules.pub.database.BadInputException.createClassCannotHaveInstancesException(BadInputException.java:167)
2017-03-30 12:21:41,692 [ pool-18-thread-1] [ STANDARD] [ ] [ ] (dnode.api.DNodeServiceListener) ERROR - Cannot initialize DNode context and start Cassandra, PRPC continuing as non-DNode
com.pega.dsm.dnode.api.DNodeException: Cannot bootstrap cassandra
at com.pega.dsm.dnode.impl.cassandra.Cassandra.bootstrap(Cassandra.java:102)
at com.pega.dsm.dnode.api.DNodeBootstrap.bootstrap(DNodeBootstrap.java:34)
at com.pega.dsm.dnode.api.DNodeServiceListener$1.run(DNodeServiceListener.java:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IllegalStateException: Dnode ports are in use, looks like another instance of dnode is running on this machine
at com.pega.dsm.dnode.impl.cassandra.Cassandra.bootstrap(Cassandra.java:97)
... 7 more
Steps to Reproduce
Not Applicable
Root Cause
A defect or configuration issue in the operating environment.
The multinode environment has same IP and temporary directory for multiple nodes, causing PRPC to generate same nodeid for multiple prpc jvms.
Resolution
Here’s the explanation for the reported behavior:
1. Setup only one dnode per system.
2. Update weblogic startup scripts to define separate Pega Temp dir location for every managed servers.
3. Update prconfig.xml to add env variable "cluster/hazelcast/interfaces" to set external IP that is accessible from other systems in the cluster.
Published May 6, 2017 - Updated October 8, 2020
Have a question? Get answers now.
Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.