Support Article
Outage experienced
SA-32892
Summary
A systems administrator (SA) has received information from the business indicating general application slowness.
The SA has confirmed these observations with a cursory inspection of the Pega logfiles, but is unable to determine root cause.
Error Messages
No error messages are displayed on screen; however there are multiple thread dumps (x16) contained in the Pega Log, and the Pega ALERT logfile contains numerous EXCP0001 alerts, and general performance alerts.
However, of great interest is a database error found in the log:
[12/13/16 15:39:51:036 EST] 00152122 SystemErr R Caught SQLException while reading prpublic.jar!com/pega/pegarules/pub/database/BadInstanceDataException.class
[12/13/16 15:39:51:036 EST] 00152122 SystemErr R java.sql.SQLException: Listener refused the connection with the following error:
ORA-12516, TNS:listener could not find available handler with matching protocol stack
DSRA0010E: SQL State = 66000, Error Code = 12,516
[12/13/16 15:39:51:036 EST] 00152122 SystemErr R at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:489)
[12/13/16 15:39:51:036 EST] 00152122 SystemErr R at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:553)
[12/13/16 15:39:51:036 EST] 00152122 SystemErr R at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:254)
...
Steps to Reproduce
Unknown. No specific steps were identified to reproduce this issue.
Root Cause
An analysis of the thread dump data revealed two distinct causes of requestor slowness:
- slow database performance in replies to queries made by PRPC
- PRPC First Use Assembly slowness trying to retrieve rule data from the database
Drilling down into the database slowness, the above SQL Error was found in the logs.
Resolution
The resolution to this problem is found in the SQL Error.
The error indicates that the WebSphere JDBC Pooled Connection Manager is configured to enable more JDBC Connections to the Oracle Database, than what the Oracle Database Listener is configured to manage.
In other words, the Oracle PROCESSES and SESSIONS configuration is defined smaller than the MAXCONNECTIONS that can be created by the WebSphere JDBC Pooled Connection Manager.
To resolve this issue, the DBA and WebSphere Admin must jointly examine the configuration, and tune the two configurations in such a way as to prevent the JDBC Pooled Connection Manager from overrunning the Oracle Listener Process.
Published February 27, 2017 - Updated October 8, 2020
Have a question? Get answers now.
Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.