Support Article
Pega 7 High Availability configuration issues with JBoss EAP 6.3
Summary
User is trying to configure a multi-node Pega 7.1.8 system with High Availability following the Pega 7.1.8 High Availability Administration Guide.
The quiesce and crash recovery scenario tests are failing as passivated sessions are not correctly restored in the surviving nodes.
The UI is blocked displaying the wait icon and an exception appears in the logs.
User environment details:
- Pega 7.1.8
- Apache Front web with mod_proxy_balancer module for load balancing
- JBoss EAP 6.3.3 GA
Error Messages
The following errror message appears in the PegaRULES log:
WARN [org.jboss.as.clustering.web.infinispan] (http-/0.0.0.0:8080-5) JBAS010322: Failed to load session UXUjDfFxwlxqJeBdqitofM1o: java.lang.RuntimeException: JBAS010333: Failed to load session attributes for session: UXUjDfFxwlxqJeBdqitofM1o
at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:229)
at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:212)
at org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34)
at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:48)
at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:85)
at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:550)
at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getData(DistributedCacheManager.java:238)
at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getSessionData(DistributedCacheManager.java:196)
at org.jboss.as.web.session.DistributableSessionManager.loadSession(DistributableSessionManager.java:1429) [jboss-as-web-7.4.3.Final-redhat-2.jar:7.4.3.Final-redhat-2]
at org.jboss.as.web.session.DistributableSessionManager.findSession(DistributableSessionManager.java:688) [jboss-as-web-7.4.3.Final-redhat-2.jar:7.4.3.Final-redhat-2]
at org.jboss.as.web.session.DistributableSessionManager.findSession(DistributableSessionManager.java:84) [jboss-as-web-7.4.3.Final-redhat-2.jar:7.4.3.Final-redhat-2]
at org.apache.catalina.connector.Request.doGetSession(Request.java:2629) [jbossweb-7.4.10.Final-redhat-1.jar:7.4.10.Final-redhat-1]
at org.apache.catalina.connector.Request.getSession(Request.java:2381) [jbossweb-7.4.10.Final-redhat-1.jar:7.4.10.Final-redhat-1]
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:99) [jboss-as-web-7.4.3.Final-redhat-2.jar:7.4.3.Final-redhat-2]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.4.10.Final-redhat-1.jar:7.4.10.Final-redhat-1]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.4.10.Final-redhat-1.jar:7.4.10.Final-redhat-1]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.4.10.Final-redhat-1.jar:7.4.10.Final-redhat-1]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb-7.4.10.Final-redhat-1.jar:7.4.10.Final-redhat-1]
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) [jbossweb-7.4.10.Final-redhat-1.jar:7.4.10.Final-redhat-1]
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.4.10.Final-redhat-1.jar:7.4.10.Final-redhat-1]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.4.10.Final-redhat-1.jar:7.4.10.Final-redhat-1]
at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
Caused by: java.lang.ClassNotFoundException: com.pega.pegarules.session.internal.authorization.ContextMapDiagCallback
at java.net.URLClassLoader$1.run(URLClassLoader.java:366) [rt.jar:1.7.0_25]
at java.net.URLClassLoader$1.run(URLClassLoader.java:355) [rt.jar:1.7.0_25]
at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.7.0_25]
at java.net.URLClassLoader.findClass(URLClassLoader.java:354) [rt.jar:1.7.0_25]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) [rt.jar:1.7.0_25]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) [rt.jar:1.7.0_25]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) [rt.jar:1.7.0_25]
at java.lang.Class.forName0(Native Method) [rt.jar:1.7.0_25]
at java.lang.Class.forName(Class.java:270) [rt.jar:1.7.0_25]
at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:97)
at org.jboss.as.clustering.ClassLoaderAwareClassResolver.resolveClass(ClassLoaderAwareClassResolver.java:92)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:948)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1255)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:224)
at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1745)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1658)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1285)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:149)
at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:135)
at org.jboss.marshalling.MarshallerObjectInputStream.readObjectOverride(MarshallerObjectInputStream.java:53)
at org.jboss.marshalling.river.RiverObjectInputStream.readObjectOverride(RiverObjectInputStream.java:307)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:364) [rt.jar:1.7.0_25]
at java.util.HashMap.readObject(HashMap.java:1155) [rt.jar:1.7.0_25]
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) [:1.7.0_25]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
at org.jboss.marshalling.reflect.SerializableClass.callReadObject(SerializableClass.java:307)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1637)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1285)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
at org.jboss.as.clustering.SimpleMarshalledValue.get(SimpleMarshalledValue.java:122)
at org.jboss.as.clustering.web.impl.SessionAttributeMarshallerImpl.unmarshal(SessionAttributeMarshallerImpl.java:72)
at org.jboss.as.clustering.web.infinispan.CoarseSessionAttributeStorage.load(CoarseSessionAttributeStorage.java:64)
at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$2.invoke(DistributedCacheManager.java:227)
... 21 more
Steps to Reproduce
- Configure the system according to the Pega 7.1.8 High Availability Administration Guide available here:
https://pdn.pega.com/documents/pega-718-high-availability-administration-guide - Open the User Portal and open or create a new work item.
- Check from the Load Balancer logs for the node which is serving the request.
- Open the Developer Portal and navigate to High Availability Landing Page (Designer studio > System > High Availability > HA Cluster Management)
- Select the node found in step 3 and click 'Quiesce'
- Disable the the node found in step 3 in the Load Balancer
- Return to the User Portal browser window and resume the work by performing an operation in perform harness
Root Cause
The issue is caused by session replication in JBoss EAP cluster which tries to restore the session anticipating the Pega 7 built-in activation.
Resolution
Perform the following local-change:
Comment out or remove the <distributable /> tag from web.xml deployment descriptor file in the prweb.war web application.
Published January 31, 2016 - Updated October 8, 2020
Have a question? Get answers now.
Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.