SA-5638

Summary

PRPC uses a Requestor Pool to process incoming Service requests. This Requestor Pool provides PRPC with a list of 'pre-created' Requestors which can be 'leased' out to run PRPC activities and then returned to the Requestor Pool. Requestor Pools are resized (dynamically, based on configuration) to react to incoming demand. If an incoming service request finds that the Requestor Pool is fully leased-out; this request will be queued by PRPC until such time as the Requestor Pool has capacity. Incoming Requests are NOT queued indefiniately - they will be timed-out (configurable) if they need to wait too long.
It was observed that during a period of time, many timeouts were occuring. This is a normal fail-safe mechanism that PRPC employs to avoid the PRPC system being overloaded by incoming requests

Error Messages

2014-10-04 17:29:12,601 [ http-8080-16] [ ] [ ] ( internal.mgmt.PREnvironment) ERROR hostname.gcs.pega.com |127.0.0.1 - Problem obtaining requestor from ServiceRequestorPool
com.pega.pegarules.pub.PRException: Timed out borrowing service requestor from requestor pool for service package: GCSPackage
From: (unknown)
at com.pega.pegarules.session.internal.mgmt.RequestorPool.borrowRequestor(RequestorPool.java:476)
at com.pega.pegarules.session.internal.mgmt.RequestorPoolManager.borrowRequestor(RequestorPoolManager.java:74)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.borrowRequestor(PRSessionProviderImpl.java:608)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.obtainRequestor(PRSessionProviderImpl.java:1403)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:902)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:815)
at com.pega.pegarules.integration.engine.internal.services.ServiceAPI.getServiceMethod(ServiceAPI.java:2971)
at com.pega.pegarules.integration.engine.internal.services.http.HTTPService.invoke(HTTPService.java:231)
.
.
.

Steps to Reproduce

All Service Requestor Pools having a point (based on incoming demand) were queued requests and will be timed out. Therefore, a load test is a way of 'forcing' this condition.

Root Cause

The Root Cause in this particular case, is that a particular activity was running for an unexpectedly long time and the Requestor Pool capacity was full. Demand and processing time was sufficient in this case that requests were queued long enough for them to be automatically timed-out, hence, the error condition.

Resolution

There are many reasons why this situation could occur: in this particular case, the time taken by an activity was taking many minutes to complete (normally would take seconds). It was suspected that this was due to Database Contention. An AWR report (Oracle) or equivalent native DBA tools is needed to investigate DB contention.

(If the service activities are performing DB operations (such as Obj-browse), then if mulitple copies of the same activity are running in parallel: then it is incumbant on the Application Designer and DBA to ensure that excessive Table locks are not taken. Otherwise, the incoming requests may be sufficient to trigger such a situation.

The Timeout setting may be increased to allow requests to queue longer. Requestor Pool sizes can also be increased. But the latter step should only be performed after careful analysis (increasing Requestor Pool sizes arbitrarily may cause Resource (usually memory) Exhaustion).