Support Article

Rule checkout fails with HazelcastInstanceNotActiveException

SA-37107

Summary



Rule check-out and check-in fails occasionally. Tracer shows error:



Error Messages



com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.AbstractDistributedObject.throwNotActiveException(AbstractDistributedObject.java:81)
at com.hazelcast.spi.AbstractDistributedObject.lifecycleCheck(AbstractDistributedObject.java:76)
at com.hazelcast.spi.AbstractDistributedObject.getNodeEngine(AbstractDistributedObject.java:70)
at com.hazelcast.topic.impl.TopicProxy.publish(TopicProxy.java:33)
at com.pega.pegarules.cluster.internal.PRClusterHazelcastImpl.broadcastSystemPulse(PRClusterHazelcastImpl.java:275)
at com.pega.pegarules.data.internal.access.DatabaseImpl.addToUpdatesCache(DatabaseImpl.java:7292)
at com.pega.pegarules.data.internal.access.DatabaseImpl.addToUpdatesCache(DatabaseImpl.java:7240)

Steps to Reproduce



Not Applicable


Root Cause



An issue in the custom application code or rules :
Rule checkout fails due to "Hazelcast instance is not active" error which is the result of a FATAL error in the Hazelcast stack:
Cannot send response: com.hazelcast.spi.impl.CallTimeoutResponse@2f872cef to Address[Your_IP]:5701

This FATAL error is caused by java.lang.OutOfMemoryError: Java heap space.

ALERT log shows this OOM error, is caused by an Agent that calls the your_Activity on your_class Class which results in running a report definition bringing large amount of data on the clipboard.

Resolution

A defect or configuration issue in the operating environment. Disable the Application Agent temporarily and restart the JVM.

Also, have developers check the report definition that the Agent calls and make sure that the number of records returned by the report definition is restricted to prevent an unbounded SQL query from pulling large amounts of data onto the Clipboard.

Published April 26, 2017 - Updated April 27, 2017


0% found this useful

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.