Support Article
Agent is not starting regularly
SA-28932
Summary
In a Pega 7.2 application deployed on WebLogic 12c the User implemented a large set of custom agents using advanced scheduling to perform long running tasks.
User noticed that those agents were falling behind the expected schedule.
Also the "Next run time" reported in System Management Application (SMA) Agents Management page was not respected sometimes with some agents that were never executed and other that were executed far behind the expected run time.
Using the WebLogic Administration Console the user noticed pending messages accumulating in the PRAsyncTopic JMS Destination.
No errors were present in the logs.
Error Messages
No error messages found but pending messages are reported by the WebLogic Administration Console for the PRAsyncTopic JMS destination .
Steps to Reproduce
- Create a custom activity that is supposed to run for many minutes.
- Create a large number of agents (more than 40) to run the activity every 30 seconds using Advanced Mode scheduling
- Observe that the next run time reported in SMA Agent Management Page is not respected.
- Notice pending messages piling up in the PRAsyncTopic JMS Destination.
Root Cause
A defect or configuration issue in the operating environment.
Some of the agents were not configured correctly hence they were not executed as supposed. The "Next run time" shown in SMA Agent Management page shows the next expected run time in case the agent is not running at the moment but it cannot take into account any delay introduced by a preceding execution that overlaps the next run time instant.
In that case the effective next execution will be determined by the activity end time plus the interval defined in the agent schedule in case of periodic agents or the next scheduled time in case of recurring agents.
Also in WebLogic deployments PRPC uses the message-driven bean (MDB) PRAsync to process all the background batch tasks through WebLogic’s PRAsyncTopic JMS topic.
Given that the default WebLogic Work Manager allows for a maximum of 16 concurrent MDB sessions a custom work manager is recommended when a large number of concurrent tasks is expected to run in background.
A pending message remains in the PRAsyncTopic destination for every missed execution caused by agent activity delays. Those messages will be consumed correctly once the activity eventually complete.
Resolution
Perform the following local-change:
Apply the changes recommended in PDN Article https://pdn.pega.com/how-increase-concurrent-prasync-mdb-sessions-prpc-ear-file-deployments-weblogic-server
For example:
- Define a global Work-Manager for PRPC background tasks indicating a Min and a Max thread constraint.
- Edit the weblogic-ejb-jar.xml deployment descriptor contained in prbeans.jar to add the following elements to the PRAsync bean definition:
a. the dispatch-policy element pointing to the global work-manager defined in one of the step.
b. the pool node with initial-beans-in-free-pool and max-beans-in-free-pool elements as described in the PDN article.
b. the pool node with initial-beans-in-free-pool and max-beans-in-free-pool elements as described in the PDN article.
Published October 21, 2016 - Updated October 8, 2020
Have a question? Get answers now.
Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.