Summary

Environment has 16 nodes in the cluster.  Post upgrading to Pega 8.1.3 all the 16 nodes start and join the cluster. Long time is required for the Admin studio and the Cluster management landing page to become available though all nodes are present in the cluster. An error occurs on the User Interface (UI) and it provides an incorrect state of the environment though the page becomes available. 

Error Messages

Unable to update browser requestor details.  Failure on ajax.

java.util.concurrent.TimeoutException: MemberCallableTaskOperation failed to complete within 30 SECONDS. Invocation{op=.hazelcast.executor.impl.operations.MemberCallableTaskOperation{serviceName='hz:impl:executorService', identityHash=1098918858, partitionId=-1, replicaIndex=0, callId=14818, invocationTime=1558523845765, waitTimeout=-1, callTimeout=60000, name=/pega/system/mgmt/exec}, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=60000, firstInvocationTimeMs=1558523848154, firstInvocationTime, lastHeartbeatMillis=1558523875431, lastHeartbeatTime, pendingResponse={VOID}, backupsAcksExpected=0, backupsAcksReceived=0, connection=Connection[id=4,5701 50316, endpoint=5701, alive=true, type=MEMBER]}
at hazelcast.spi.impl.operationservice.impl.InvocationFuture.newTimeoutException(InvocationFuture.java:68) ~[hazelcast-enterprise]
at hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:201) ~[hazelcast-enterprise]
at hazelcast.util.executor.DelegatingFuture.get(DelegatingFuture.java:88) ~[hazelcast-enterprise]
at util.concurrent.ForwardingFuture.get(ForwardingFuture.java:69) ~[guava-jar:?]
pega.pegarules.management.internal.system.operations.SystemOperationsDistributedExecutorImpl.createResultFromOneFutureObject(SystemOperationsDistributedExecutorImpl.java:156) ~[prprivate.jar:?]
pega.pegarules.management.internal.system.operations.SystemOperationsDistributedExecutorImpl.getResultsFromFutureObjects(SystemOperationsDistributedExecutorImpl.java:146) ~[prprivate.jar:?] at pega.pegarules.management.internal.system.operations.SystemOperationsDistributedExecutorImpl.executeOnAllNodesInCluster(SystemOperationsDistributedExecutorImpl.java:90) ~[prprivate.jar:?]
pega.pegarules.management.internal.system.operations.AbstractManagementAPI.executeOnAllNodes(AbstractManagementAPI.java:178) ~[prprivate.jar:?]
at pega.pegarules.management.internal.system.operations.AbstractManagementAPI.executeOnAllNodes(AbstractManagementAPI.java:164) at pega.pegarules.management.internal.system.operations.ClusterManagementAPI.getClusterInformation(ClusterManagementAPI.java:89) ~[prprivate.jar:?]
pegarules.generated.activity.ra_action_pzgetclustermembersinformation_4383efb57ebfe77c6b87b3367eb5d888.step1_circum0(ra_action_pzgetclustermembersinformation_4383efb57ebfe77c6b87b3367eb5d888.java:195) ~[?:?]
pegarules.generated.activity.ra_action_pzgetclustermembersinformation_4383efb57ebfe77c6b87b3367eb5d888.perform(ra_action_pzgetclustermembersinformation_4383efb57ebfe77c6b87b3367eb5d888.java:70) ~[?:?]
 

Steps to Reproduce

Root Cause

Resolution

1. Apply HFix-51612
2. Bring all the tiers down
3. Start the Stream and Decision Data Store (DDS) tier nodes one by one
4. Start the other nodes