SA-19494

Summary

User is observing exceptions related to the OOTB Batch activity "System-Queue-ServiceLevel.ProcessEvent" on production server on regular basis for all the applications. There are more than 5000 Exceptions from different applications logged every day.
Although applications do not see any functional impact application responsiveness is effected because of this logging.
User is looking for root cause of this logging.

Error Messages

2016-01-18 03:48:09,592 [  PegaRULES-Batch-14] [  STANDARD] [GSSC_Agents:01.01.01] (Queue_ExecutionRequest_.Action) ERROR   - Failed to process queue: System-Queue-ExecutionRequest-Connect-Default;com.pega.pegarules.pub.context.PRQueueException: Database-General: code: Problem encountered when getting connection for database pegadata SQLState: 0 Message: <none>;    at com.pega.pegarules.session.internal.async.agent.QueueManagerImpl$QueueIteratorImpl.next(QueueManagerImpl.java:2117);    at com.pega.pegarules.session.internal.async.agent.QueueManagerImpl$QueueIteratorImpl.first(QueueManagerImpl.java:1865);    at com.pega.pegarules.session.internal.engineinterface.service.RequestProcessorImpl.processQueue(RequestProcessorImpl.java:655);    at com.pegarules.generated.activity.ra_action_processqueue_90bab9d8d8f981f8da8c556128761aab.step1_circum0(ra_action_processqueue_90bab9d8d8f981f8da8c556128761aab.java:160);    at com.pegarules.generated.activity.ra_action_processqueue_90bab9d8d8f981f8da8c556128761aab.perform(ra_action_processqueue_90bab9d8d8f981f8da8c556128761aab.java:69);    at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3275);    at com.pega.pegarules.session.internal.async.agent.QueueProcessor.runActivity(QueueProcessor.java:721);    at com.pega.pegarules.session.internal.async.agent.QueueProcessor.execute(QueueProcessor.java:602);    at com.pega.pegarules.session.internal.async.BatchRequestorTask.run(BatchRequestorTask.java:884);    at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1192);    at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:930);    at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:829);    at com.pega.pegarules.session.internal.async.BatchRequestorTask.run(BatchRequestorTask.java:617);    at com.pega.jsr166backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:991);    at com.pega.jsr166backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:532);    at java.lang.Thread.run(Thread.java:662);
Caused by: com.pega.pegarules.pub.database.ConnectionException: Database-General    Problem encountered when getting connection for database pegadata    0        weblogic.common.resourcepool.ResourceDisabledException: Pool PegaRules is Suspended, cannot allocate resources to applications..;From: (B0A4A8D97FB52AA05966BE76497DC6544) ;
Caused by SQL Problems.;Problem #1, SQLState null, Error code 0: weblogic.jdbc.extensions.PoolDisabledSQLException: weblogic.common.resourcepool.ResourceDisabledException: Pool PegaRules is Suspended, cannot allocate resources to applications..;Problem #2, SQLState null, Error code 0: java.sql.SQLException: Cannot obtain connection: driverURL = jdbc:weblogic:pool:PegaRules, props = {EmulateTwoPhaseCommit=false, connectionPoolID=PegaRules, jdbcTxDataSource=true, LoggingLastResource=false, dataSourceName=PegaRules};    at com.pega.pegarules.data.internal.access.ExceptionInformation.createExceptionDueToDBFailure(ExceptionInformation.java:163);    at com.pega.pegarules.data.internal.access.ConnectionManagerImpl.getConnectionFromDataSourceIdentity(ConnectionManagerImpl.java:1053);    at com.pega.pegarules.data.internal.access.ConnectionManagerImpl.getConnection(ConnectionManagerImpl.java:1188);    at com.pega.pegarules.data.internal.access.ConnectionManagerImpl.getConnection(ConnectionManagerImpl.java:1128);    at com.pega.pegarules.data.internal.access.ConnectionManagerImpl.getConnection(ConnectionManagerImpl.java:343);    at com.pega.pegarules.data.internal.access.ThreadConnectionStoreImpl.getConnection(ThreadConnectionStoreImpl.java:408);    at com.pega.pegarules.data.internal.access.RDBPageResultPackager.getConnection(RDBPageResultPackager.java:923);    at com.pega.pegarules.data.internal.access.DatabaseImpl.list(DatabaseImpl.java:6076);    at com.pega.pegarules.data.internal.access.DatabaseImpl.executeRDB(DatabaseImpl.java:6691);    at com.pega.pegarules.data.internal.access.DatabaseImpl.executeRDB(DatabaseImpl.java:6647);    at com.pega.pegarules.data.internal.access.DatabaseImpl.executeRDB(DatabaseImpl.java:6628);    at com.pega.pegarules.session.internal.async.agent.QueueManagerImpl$QueueIteratorImpl.next(QueueManagerImpl.java:2020);    ... 15 more;
Caused by: weblogic.jdbc.extensions.PoolDisabledSQLException: weblogic.common.resourcepool.ResourceDisabledException: Pool PegaRules is Suspended, cannot allocate resources to applications..;    at weblogic.jdbc.common.internal.JDBCUtil.wrapAndThrowResourceException(JDBCUtil.java:251);    at weblogic.jdbc.pool.Driver.connect(Driver.java:161);    at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:651);    at weblogic.jdbc.jts.Driver.connect(Driver.java:127);    at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:364);    at com.pega.pegarules.data.internal.access.DataSourceManager.getConnection(DataSourceManager.java:221);    at com.pega.pegarules.data.internal.access.DatabaseConnectionImpl.activate(DatabaseConnectionImpl.java:1436);    at com.pega.pegarules.data.internal.access.ConnectionManagerImpl.getConnectionFromDataSourceIdentity(ConnectionManagerImpl.java:1007);    ... 25 more;

Steps to Reproduce

Not Applicable.

Root Cause

WebLogic JVMs has a process in place to test connections for consistency, if this happens during Garbage Collection (GC), even the tests get halted, taking them more time to complete the test, hence disabling the pool as it did not get response for the connection tests within a stipulated time.
 

Resolution

Issue is with Weblogic server and suggested the user to refer the below articles and based on the version of Weblogic server, user must apply the patch accordingly.

https://community.oracle.com/message/10407326#10407326
https://community.oracle.com/message/4444239#4444239