SA-28557

Summary

A PRPC71ML6 system running on Websphere Application Server against Oracle database suffered from performance issue. The administrator noticed the system freeze and the database resources are running out.

Error Messages

In the Thread Dumps, there are many entries of the below trace: 

"WebContainer : 45" Id=124435 in WAITING on lock=java.util.concurrent.locks.ReentrantLock$NonfairSync@c8abb9f5 (running in native) 
BlockedCount : 370, BlockedTime : -1, WaitedCount : 94739, WaitedTime : -1 
at sun.misc.Unsafe.park(Native Method) 
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:182) 
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:822) 
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:853) 
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1189) 
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:197) 
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:273) 
at com.pega.pegarules.priv.factory.AbstractContainerFactory.releaseObject(AbstractContainerFactory.java:326) 
at com.pega.pegarules.priv.factory.AbstractContainerFactory.releaseObject(AbstractContainerFactory.java:294) 

In Websphere Application Server log, many occurrences of the below error has been found. 

java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:140)
oracle.net.ns.Packet.receive(Packet.java:311)
oracle.net.ns.DataPacket.receive(DataPacket.java:105)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:305)
oracle.net.ns.NetInputStream.read(NetInputStream.java:249)
oracle.net.ns.NetInputStream.read(NetInputStream.java:171)
oracle.net.ns.NetInputStream.read(NetInputStream.java:89)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:429)
 

Steps to Reproduce

The system hangs occasionally and system resources are running out. 

Root Cause

The root cause has not been identified. A few configuration changes have been made and since then the issue has not reoccurred. 

Resolution

The below changes has been applied:

1) Apply prconfig setting of <env name="compiler/classloadCompilerOutput" value="false"/>.

This setting will address a Thread deadlock issue. See the below PDN article:

See https://pdn.pega.com/support-articles/prpcworkmanager-thread-deadlock

2) Install HFix-29456 overcome NullPointerException which occurs after applying <env name="compiler/classloadCompilerOutput" value="false"/>

3) Turned on "Pass by Reference" in WebSphere to improve the performance.  In the Websphere Administrative console:

i) Select Servers -> Server Types -> WebSphere application servers to display the Application servers page.
ii) Click the name of your Pega 7 server to display the Configuration page.
iii) Under Container Settings, expand Container Services, and then click ORB service.

4) The error of java.net.SocketInputStream.socketRead0(Native Method) appears to be a known Websphere Application Server issue as documented in the below IBM knowledge base article:

http://www-01.ibm.com/support/docview.wss?uid=swg1PM91941

To overcome this issue either increase oracle.jdbc.ReadTimeout parameter greater than the server's "total transaction lifetime timeout" or applying socket keep alive parameters providing Websphere Application Server and JDK have been upgraded to support this setting.