SA-17602

Summary

System crashes after multiple build up. Stuck threads can be viewed in the JVM logs, most of these stuck threads trying to access log4j class.

This issue occurs when Pega7 is deployed on weblogic.

Error Messages

NoClassDefFoundError, message='com/pega/apache/log4j/spi/ThrowableInformation', stacktrace=java.lang.NoClassDefFoundError: com/pega/apache/log4j/spi/ThrowableInformation
at com.pega.apache.log4j.spi.LoggingEvent.<init>(LoggingEvent.java:155)
at com.pega.apache.log4j.Category.forcedLog(Category.java:391)
...
 
2015-11-03 08:40:07,236 [fault (self-tuning)'] [ STANDARD] [ ] ( services.http.HTTPService) ERROR pegaserver.domain.com |10.255.255.255 - Caught unexpected java.lang.NoClassDefFoundError, message='com/pega/apache/log4j/spi/ThrowableInformation', stacktrace=java.lang.NoClassDefFoundError: com/pega/apache/log4j/spi/ThrowableInformation
 at com.pega.apache.log4j.spi.LoggingEvent.<init>(LoggingEvent.java:155)
 at com.pega.apache.log4j.Category.forcedLog(Category.java:391)
 at com.pega.apache.log4j.Category.error(Category.java:322)
 at com.pega.pegarules.priv.LogHelper.error(LogHelper.java:1345)
 at com.pega.pegarules.session.internal.PRSessionProviderImpl.borrowRequestor(PRSessionProviderImpl.java:612)
 at com.pega.pegarules.session.internal.PRSessionProviderImpl.obtainRequestor(PRSessionProviderImpl.java:1381)
 at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:898)
 at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:811)
 at com.pega.pegarules.integration.engine.internal.services.ServiceAPI.getServiceMethod(ServiceAPI.java:2978)
 at com.pega.pegarules.integration.engine.internal.services.http.HTTPService.invoke(HTTPService.java:251)
 at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._invokeEngine_privact(EngineImpl.java:315)
 at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.invokeEngine(EngineImpl.java:263)
 at com.pega.pegarules.session.internal.engineinterface.etier.ejb.EngineBean.invokeEngine(EngineBean.java:225)
 at sun.reflect.GeneratedMethodAccessor226.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:338)
 at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:379)
 at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:216)
 at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingException(AppServerBridgeToPega.java:238)
 at com.pega.pegarules.internal.et

Steps to Reproduce

Unknown.

Root Cause

This occurs when an exception is thrown from a remote service. (in this particular case, from a JMS Service).

The Pega Engine tries to log it via log4j.  As log4j classes are in database, Pega tries to load the classes from the database.  However, because it is a remote invocation, while trying to connect to database via weblogic datasource, the database connection is intercepted by weblogic security.  As such, the class is not loaded from the database and the NoClassDefFound error occurs.

Resolution

This can be addresed by changing the weblogic settings in (CHOOSE ONE) one of the following ways:

1. Enable "global trust between domains".
or
2. Enable "Cross Domain Security Between WebLogic Server Domains".
or
3. Set downgrade-untrusted-principal to false.

These are weblogic settings and more information on these settings can be found in the Weblogic documentationb