SA-5404

Summary

During performance testing, the following issues were discovered:

400 requests are handled normally by the system.
On 10% loading increase, performance decreases (time spent per one stage of a request increases) drastically and continues to fall exponentially.
Thus, further loading increase seems impossible - though CPU and Memory resources utilization stays under 70%.
With maximum loading level reached, Thread dumps contain large number of blocked threads (around 50%). Most of those are being blocked by synchronized blocks from com.pega.pegarules.priv.factory.AbstractContainerFactory class.

Thread Dump records relate blocks on the following methods:

"WebContainer : 175" daemon prio=7 tid=600000000a75dc00 nid=1770 lwp_id=2989052 waiting for monitor entry [9fffffffbe34d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.pega.pegarules.priv.factory.AbstractContainerFactory.acquireObject(AbstractContainerFactory.java:206)
- waiting to lock <9ffffffcedba7000> (a java.util.ArrayList)

"WebContainer : 127" daemon prio=7 tid=6000000005e9ec00 nid=1543 lwp_id=2988524 waiting for monitor entry [9fffffffc134e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.pega.pegarules.priv.factory.AbstractContainerFactory.releaseObject(AbstractContainerFactory.java:297)
- waiting to lock <9ffffffcedba7000> (a java.util.ArrayList)

Customer needs help to analyze the work of stated class under heavy load (conditions alike to those described above) and plan corrections and adjustments accordingly.

Error Messages

ThreadDump

"WebContainer : 175" daemon prio=7 tid=600000000a75dc00 nid=1770 lwp_id=2989052 waiting for monitor entry [9fffffffbe34d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.pega.pegarules.priv.factory.AbstractContainerFactory.acquireObject(AbstractContainerFactory.java:206)
- waiting to lock <9ffffffcedba7000> (a java.util.ArrayList)

"WebContainer : 127" daemon prio=7 tid=6000000005e9ec00 nid=1543 lwp_id=2988524 waiting for monitor entry [9fffffffc134e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.pega.pegarules.priv.factory.AbstractContainerFactory.releaseObject(AbstractContainerFactory.java:297)
- waiting to lock <9ffffffcedba7000> (a java.util.ArrayList)

Steps to Reproduce

During performance testing, the following issues were discovered:

400 requests are handled normally by the system.
On 10% loading increase, performance decreases (time spent per one stage of a request increases) drastically and continues to fall exponentially.
Thus, further loading increase seems impossible - though CPU and Memory resources utilization stays under 70%.
With maximum loading level reached, Thread dumps contain large number of blocked threads (around 50%). Most of those are being blocked by synchronized blocks from com.pega.pegarules.priv.factory.AbstractContainerFactory class.

Root Cause

The root cause of this problem is software use/operation error.
Customer had agent which were intensively working with the string pool.

Resolution

This issue is resolved through the following local change:
Customer made some changes in application (how agents work with BLOBs) which significally reduce CPU load on application server, Then the  contention on the string pool has gone.

We notice a significant different in terms of http request between the resource estimate guide and their current application.
On their sizing guide, the “Server interactions per case” is 38. Server interaction corresponds to HTTP requests (including Ajax requests).
However, in pratice a total of 290 HTTP request are usedd for 1 request (or 1 case).
So it appears there is much more server interaction in the current application (290) compared to the resource estimate (38).
This explains why the performance is not in line with their resource estimate guide.

The following extract from the PDN article below is discussing it:
At a business level, another important measure of performance is throughput — how many completed units of work (orders, customer service requests, investigations) can be processed per minute (or hour, or workday). A system that provides users with excellent response time, but is designed with too many required interactions (for each unit of completed work) may have unacceptably low throughput. For example, in an initial design, assume that completing a certain business transaction that requires 24 interactions (each averaging 30 seconds). If this transaction can be redesigned to require only 18 interactions (with comparable response and user input times), the new design improves user productivity and throughput.
 
https://pdn.pega.com/performance/performance-overview