Skip to main content

This content has been archived and is no longer being updated. Links may not function; however, this content may be relevant to outdated versions of the product.

Support Article

Nodes do not join cluster post update to Pega 8.1.1

SA-81830

Summary



After upgrading to Pega Platform 8.1.1, the nodes do not join the Hazelcast cluster.


Error Messages



[7e5.cached.thread-10] [STANDARD] [ ] [ ] (bs.AbstractSystemOperationsJob) ERROR - Error during job execution: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:498) 
at com.pega.pegarules.management.internal.system.operations.jobs.AbstractSystemOperationsJob$SystemOperationsJobRunnable.invokeMethodOnClass(AbstractSystemOperationsJob.java:114) 
at com.pega.pegarules.management.internal.system.operations.jobs.AbstractSystemOperationsJob$SystemOperationsJobRunnable.run(AbstractSystemOperationsJob.java:107) 
at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1366) 
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:1109) 
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:916) 
at com.pega.pegarules.management.internal.system.operations.jobs.AbstractSystemOperationsJob.invoke(AbstractSystemOperationsJob.java:89) 
at com.pega.pegarules.management.internal.system.operations.jobs.AbstractSystemOperationsJob.execute(AbstractSystemOperationsJob.java:58)

at com.pega.pegarules.management.internal.system.operations.jobs.AbstractSystemOperationsJob.execute(AbstractSystemOperationsJob.java:38)

at com.pega.pegarules.cluster.internal.PRPCTask.call(PRPCTask.java:130)

at com.pega.pegarules.cluster.internal.PRPCTask.call(PRPCTask.java:53)

at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
.
.
.
com.pega.pegarules.management.internal.system.operations.jobs.ClusterManagementJob.getCurrentNodeTypes(ClusterManagementJob.java:132)

at com.pega.pegarules.management.internal.system.operations.jobs.ClusterManagementJob.getConcatenatedNodetypesString(ClusterManagementJob.java:97) 
at com.pega.pegarules.management.internal.system.operations.jobs.ClusterManagementJob.getHighLevelNodeInfoAsJSON(ClusterManagementJob.java:126) 
at com.pega.pegarules.management.internal.system.operations.jobs.ClusterManagementJob.getHighLevelNodeInfo(ClusterManagementJob.java:155) 
at com.pega.pegarules.management.internal.system.operations.jobs.ClusterManagementJob.invokeOperation(ClusterManagementJob.java:316) 
... 22 more


Steps to Reproduce



Upgrade Pega Platform 7.4 to Pega Platform 8.1.1.


Root Cause



A defect in Pegasystems’ code or rules.

Every node must discover the cluster topology on startup. In the current implementation, remote execution is used to communicate between nodes in the cluster.

On node startup, the following occurred:
  1. A node requests information from all the currently running nodes in the cluster (Hazelcast call to all nodes).
  2. Every other node will know that a new node joined and will try to build a new topology picture by enquiring all nodes about their status
  3. At this point, all the nodes are trying to communicate with each other and are waiting for a response.
A race condition occurs leading to a distributed dead-lock. Multiple nodes are stuck in the topology discovery indefinitely. This race condition occurs when the cluster size expands.


Resolution



Perform the following local-change:

Add the below Dynamic System Settings (DSS),
  • Owning Ruleset: Pega-Engine
  • Setting Purpose: prconfig/systemoperations/job/timeoutinsecond/default
  • Value: 120
This timing change avoids the race condition.

The recommended change is to install the current Pega Platform 8.1 patch release.

Alternatively, update to Pega Platform 8.1.4.

Published August 15, 2019 - Updated December 2, 2021

Was this useful?

0% found this useful

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.

Did you find this content helpful?

Want to help us improve this content?

We'd prefer it if you saw us at our best.

Pega Community has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice
Contact us