Skip to main content

This content has been archived and is no longer being updated. Links may not function; however, this content may be relevant to outdated versions of the product.

Support Article

Pega BRE Hazlecast and Memory Errors

SA-28736

Summary



User observing Hazlecast Exceptions in logs which are being followed by Out of Memory Errors, Hung Threads and finally impacting the Pega Application as the application is going down abruptly.


Error Messages



[9/18/16 5:17:50:833 MST] 00000038 BasicInvocati W com.hazelcast.spi.impl.BasicInvocation [IP_Address]:port_number [4783232062e4a86cf91e45eedfb08505] [3.2] While asking 'is-executing': InvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=GetOperation{}, partitionId=240, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[IP_Address]:Port_Number}, done=false}
java.util.concurrent.TimeoutException: Call BasicInvocation{ serviceName='hz:impl:mapService', [email protected]17d, partitionId=-1, replicaIndex=0, tryCount=0, tryPauseMillis=0, invokeCount=1, callTimeout=5000, target=Address[IP_Address]:Port_Number} encountered a timeout
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponse(BasicInvocation.java:801)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponseOrThrowException(BasicInvocation.java:769)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:696)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.isOperationExecuting(BasicInvocation.java:875)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.waitForResponse(BasicInvocation.java:753)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:695)


[9/19/16 23:11:02:522 MST] 00000037 PRBootstrap E com.pega.pegarules.internal.bootstrap.PRBootstrap Problem during method invocation (invokeEngine)
com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.AbstractDistributedObject.getService(AbstractDistributedObject.java:78)
at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:50)
at com.pega.pegarules.cluster.internal.PRClusterHazelcastImpl.get(PRClusterHazelcastImpl.java:710)
at com.pega.pegarules.session.internal.async.AgentInfoManager.getAgentInfo(AgentInfoManager.java:32)
at com.pega.pegarules.session.internal.async.AgentQueue.updateAgentInfo(AgentQueue.java:777)


[9/20/16 7:51:12:352 MST] 0000004f NotificationD W CWWME0006W: An exception occurred sending notification javax.management.Notification[source=WebSphere:name=wsfpp1lppza186wspbreusServer,process=wsfpp1lppza186wspbreusServer,platform=proxy,node=wsfpp1lppza186,j2eeType=J2EEServer,version=7.0.0.39,type=Server,mbeanIdentifier=cells/ZB20
/nodes/wsfpp1lppza186/servers/wsfpp1lppza186wspbreusServer/server.xml#Server_1380574405439,cell=ZB20,spec=1.0,processType=ManagedProcess][type=websphere.thread.hung][message=Thread "WebContainer : 41" has been active for 666,231 milliseconds and may be hung. There is/are 1 thread(s) in total that may be hung.] to [email protected]: java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11



[9/20/16 7:39:54:838 MST] 0000003a webapp E com.ibm.ws.webcontainer.webapp.WebApp logError CWSRV0293E: [Servlet Error]-[java.lang.OutOfMemoryError]: java.lang.Ou
tOfMemoryError
at java.util.zip.Inflater.init(Native Method)


[9/20/16 7:39:57:371 MST] 00000053 webapp E com.ibm.ws.webcontainer.webapp.WebApp logServletError CWSRV0293E: [Servlet Error]-[WebStandard]: com.ibm.ws.webcontai
ner.webapp.WebAppErrorReport: Internal Error



[9/20/16 7:51:12:352 MST] 0000004f NotificationD W CWWME0006W: An exception occurred sending notification javax.management.Notification[source=WebSphere:name=wsfpp1lp
pza186wspbreusServer,process=wsfpp1lppza186wspbreusServer,platform=proxy,node=wsfpp1lppza186,j2eeType=J2EEServer,version=7.0.0.39,type=Server,mbeanIdentifier=cells/ZB20
/nodes/wsfpp1lppza186/servers/wsfpp1lppza186wspbreusServer/server.xml#Server_1380574405439,cell=ZB20,spec=1.0,processType=ManagedProcess][type=websphere.thread.hung][me
ssage=Thread "WebContainer : 41" has been active for 666,231 milliseconds and may be hung. There is/are 1 thread(s) in total that may be hung.] to LocalNotificationSer
[email protected]: java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11


Steps to Reproduce



Not applicable


Root Cause



Whenever classes are being loaded from the extracted jar files like praxis, prrampart, which will happen if there is a SOAP call or a REST call due to inadequate native memory, it is resulting in an OutOfMemory error, which could be observed from Pega RULES logs.

This behavior can be associated with the use of Compressed References with 64-bit JVM, which will encourage the JVM to use 32-bit addressing which will also reduce the amount of heap space available to the JVM, which ultimates result in less native memory for creation of threads.

The above behaviour is a knwon issue with IBM JVMs.

The use of compressed references can be found from the java core dump, a snippet of the same has been added below for reference.

2CIUSERARG               -Dibm.websphere.internalClassAccessMode=allow
2CIUSERARG               -verbose:gc
2CIUSERARG               -Xms4096m
2CIUSERARG               -Xmx4096m
2CIUSERARG               -Xcompressedrefs
2CIUSERARG               

 



Resolution



Add the following setting in your environment to start the Heap at the 4GB mark., re-start the JVM.

Setting:  -Xgc:preferredHeapBase=0x100000000

Issue is with using Compressed References with IBM JVM, which could be overcome by using the above setting.

Refer the link provided below:

https://www-304.ibm.com/support/docview.wss?uid=swg21660890

 

 

Suggest Edit

Published December 15, 2016 - Updated October 8, 2020

Did you find this content helpful? Yes No

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.

We'd prefer it if you saw us at our best.

Pega Community has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice
Contact us