SA-28736

Summary

User observing Hazlecast Exceptions in logs which are being followed by Out of Memory Errors, Hung Threads and finally impacting the Pega Application as the application is going down abruptly.

Error Messages

[9/18/16 5:17:50:833 MST] 00000038 BasicInvocati W com.hazelcast.spi.impl.BasicInvocation [IP_Address]:port_number [4783232062e4a86cf91e45eedfb08505] [3.2] While asking 'is-executing': InvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=GetOperation{}, partitionId=240, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[IP_Address]:Port_Number}, done=false}
java.util.concurrent.TimeoutException: Call BasicInvocation{ serviceName='hz:impl:mapService', op=com.hazelcast.spi.impl.BasicInvocation$IsStillExecuting@117d117d, partitionId=-1, replicaIndex=0, tryCount=0, tryPauseMillis=0, invokeCount=1, callTimeout=5000, target=Address[IP_Address]:Port_Number} encountered a timeout
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponse(BasicInvocation.java:801)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponseOrThrowException(BasicInvocation.java:769)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:696)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.isOperationExecuting(BasicInvocation.java:875)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.waitForResponse(BasicInvocation.java:753)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:695)


[9/19/16 23:11:02:522 MST] 00000037 PRBootstrap E com.pega.pegarules.internal.bootstrap.PRBootstrap Problem during method invocation (invokeEngine)
com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.AbstractDistributedObject.getService(AbstractDistributedObject.java:78)
at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:50)
at com.pega.pegarules.cluster.internal.PRClusterHazelcastImpl.get(PRClusterHazelcastImpl.java:710)
at com.pega.pegarules.session.internal.async.AgentInfoManager.getAgentInfo(AgentInfoManager.java:32)
at com.pega.pegarules.session.internal.async.AgentQueue.updateAgentInfo(AgentQueue.java:777)


[9/20/16 7:51:12:352 MST] 0000004f NotificationD W CWWME0006W: An exception occurred sending notification javax.management.Notification[source=WebSphere:name=wsfpp1lppza186wspbreusServer,process=wsfpp1lppza186wspbreusServer,platform=proxy,node=wsfpp1lppza186,j2eeType=J2EEServer,version=7.0.0.39,type=Server,mbeanIdentifier=cells/ZB20
/nodes/wsfpp1lppza186/servers/wsfpp1lppza186wspbreusServer/server.xml#Server_1380574405439,cell=ZB20,spec=1.0,processType=ManagedProcess][type=websphere.thread.hung][message=Thread "WebContainer : 41" has been active for 666,231 milliseconds and may be hung. There is/are 1 thread(s) in total that may be hung.] to LocalNotificationService@113247936: java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11



[9/20/16 7:39:54:838 MST] 0000003a webapp E com.ibm.ws.webcontainer.webapp.WebApp logError CWSRV0293E: [Servlet Error]-[java.lang.OutOfMemoryError]: java.lang.Ou
tOfMemoryError
at java.util.zip.Inflater.init(Native Method)


[9/20/16 7:39:57:371 MST] 00000053 webapp E com.ibm.ws.webcontainer.webapp.WebApp logServletError CWSRV0293E: [Servlet Error]-[WebStandard]: com.ibm.ws.webcontai
ner.webapp.WebAppErrorReport: Internal Error



[9/20/16 7:51:12:352 MST] 0000004f NotificationD W CWWME0006W: An exception occurred sending notification javax.management.Notification[source=WebSphere:name=wsfpp1lp
pza186wspbreusServer,process=wsfpp1lppza186wspbreusServer,platform=proxy,node=wsfpp1lppza186,j2eeType=J2EEServer,version=7.0.0.39,type=Server,mbeanIdentifier=cells/ZB20
/nodes/wsfpp1lppza186/servers/wsfpp1lppza186wspbreusServer/server.xml#Server_1380574405439,cell=ZB20,spec=1.0,processType=ManagedProcess][type=websphere.thread.hung][me
ssage=Thread "WebContainer : 41" has been active for 666,231 milliseconds and may be hung. There is/are 1 thread(s) in total that may be hung.] to LocalNotificationSer
vice@113247936: java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11

Steps to Reproduce

Not applicable

Root Cause

Whenever classes are being loaded from the extracted jar files like praxis, prrampart, which will happen if there is a SOAP call or a REST call due to inadequate native memory, it is resulting in an OutOfMemory error, which could be observed from Pega RULES logs.

This behavior can be associated with the use of Compressed References with 64-bit JVM, which will encourage the JVM to use 32-bit addressing which will also reduce the amount of heap space available to the JVM, which ultimates result in less native memory for creation of threads.

The above behaviour is a knwon issue with IBM JVMs.

The use of compressed references can be found from the java core dump, a snippet of the same has been added below for reference.

2CIUSERARG               -Dibm.websphere.internalClassAccessMode=allow
2CIUSERARG               -verbose:gc
2CIUSERARG               -Xms4096m
2CIUSERARG               -Xmx4096m
2CIUSERARG               -Xcompressedrefs
2CIUSERARG               

 

Resolution

Add the following setting in your environment to start the Heap at the 4GB mark., re-start the JVM.

Setting:  -Xgc:preferredHeapBase=0x100000000

Issue is with using Compressed References with IBM JVM, which could be overcome by using the above setting.

Refer the link provided below:

https://www-304.ibm.com/support/docview.wss?uid=swg21660890