SA-37107

Summary

Rule check-out and check-in fails occasionally. Tracer shows error:

Error Messages

com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.AbstractDistributedObject.throwNotActiveException(AbstractDistributedObject.java:81)
at com.hazelcast.spi.AbstractDistributedObject.lifecycleCheck(AbstractDistributedObject.java:76)
at com.hazelcast.spi.AbstractDistributedObject.getNodeEngine(AbstractDistributedObject.java:70)
at com.hazelcast.topic.impl.TopicProxy.publish(TopicProxy.java:33)
at com.pega.pegarules.cluster.internal.PRClusterHazelcastImpl.broadcastSystemPulse(PRClusterHazelcastImpl.java:275)
at com.pega.pegarules.data.internal.access.DatabaseImpl.addToUpdatesCache(DatabaseImpl.java:7292)
at com.pega.pegarules.data.internal.access.DatabaseImpl.addToUpdatesCache(DatabaseImpl.java:7240)

Steps to Reproduce

Not Applicable

Root Cause

An issue in the custom application code or rules :
Rule checkout fails due to "Hazelcast instance is not active" error which is the result of a FATAL error in the Hazelcast stack:
Cannot send response: com.hazelcast.spi.impl.CallTimeoutResponse@2f872cef to Address[Your_IP]:5701

This FATAL error is caused by java.lang.OutOfMemoryError: Java heap space.

ALERT log shows this OOM error, is caused by an Agent that calls the your_Activity on your_class Class which results in running a report definition bringing large amount of data on the clipboard.

Resolution

A defect or configuration issue in the operating environment. Disable the Application Agent temporarily and restart the JVM.

Also, have developers check the report definition that the Agent calls and make sure that the number of records returned by the report definition is restricted to prevent an unbounded SQL query from pulling large amounts of data onto the Clipboard.