Support Article
Server going down with Not returning connection error
SA-29561
Summary
Production server down with SQLServerException need root cause analysis.
Error Messages
We see the below error repeatedly whenever the server is going down.
2016-10-07 07:14:14,133 [ WorkManager(2)-78] [ STANDARD] [ ] (rnal.async.MasterForRequestors) ERROR - Page pyReportPageLocale in Thread STANDARD in Requestor H698778EE622328A1577343E15A19A6E0 is missing meta data. Skipping this page for passivation
2016-10-07 07:15:28,584 [ WorkManager(2)-28] [ STANDARD] [ PegaRULES:07.10] (rnal.async.MasterForRequestors) ERROR - Page pyReportPageLocale in Thread STANDARD in Requestor H698778EE622328A1577343E15A19A6E0 is missing meta data. Skipping this page for passivation
2016-10-07 07:15:28,753 [ WorkManager(2)-98] [ STANDARD] [ PegaRULES:07.10] (l.access.ConnectionManagerImpl) ERROR - Not returning connection 1994255 for database "pegarules" to the pool as it previously encountered the following error
User ID: System
Last SQL: select pzClearCacheType, pzObjClass, pzKeyString, pzParameters from dbo.pr_sys_updatescache where (pzCreateDateTime >= ?) and (pzCreateDateTime < ?) order by pzCreateDateTime asc
com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset
at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1352)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1339)
at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1654)
at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:3686)
at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:5022)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:388)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:340)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4575)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1400)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:179)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:154)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery(SQLServerPreparedStatement.java)
at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:342)
at com.pega.pegarules.data.internal.access.DatabasePreparedStatementImpl.executeQuery(DatabasePreparedStatementImpl.java:404)
at com.pega.pegarules.data.internal.access.DatabaseImpl.performSystemPulse(DatabaseImpl.java:8682)
at com.pega.pegarules.data.internal.access.DatabaseImpl.performSystemIndex(DatabaseImpl.java:8500)
at com.pegarules.generated.activity.ra_action_systemindexer_e9bb95a86a643fb648babf3085010547.step2_circum0(ra_action_systemindexer_e9bb95a86a643fb648babf3085010547.java:292)
at com.pegarules.generated.activity.ra_action_systemindexer_e9bb95a86a643fb648babf3085010547.perform(ra_action_systemindexer_e9bb95a86a643fb648babf3085010547.java:86)
at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3275)
at com.pega.pegarules.session.internal.async.agent.QueueProcessor.runActivity(QueueProcessor.java:721)
at com.pega.pegarules.session.internal.async.agent.QueueProcessor.execute(QueueProcessor.java:602)
at com.pega.pegarules.session.internal.async.BatchRequestorTask.run(BatchRequestorTask.java:884)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1192)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:930)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:829)
at com.pega.pegarules.session.internal.async.BatchRequestorTask.run(BatchRequestorTask.java:617)
Steps to Reproduce
1. Shutdown/Not available database server without stopping application server
Root Cause
A defect or configuration issue in the operating environment causes this issue. The database is taken offline not able to ping from the application server. This led to failed data source connections reset.
Resolution
Perform the following local-change:
Update the datasource to add connection pre-testing and failing connection clean-up. This will further safeguard against any temporary database outages by adding below tag in sql datasource.
<check-valid-connection-sql>Select 1</check-valid-connection-sql>
Published October 27, 2016 - Updated October 8, 2020
Have a question? Get answers now.
Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.