Queue processor FAQ

A Queue Processor rule is an internal background process on the server that you can use for queue management and asynchronous processing instead of using standard agents. For higher scaling throughput, use standard queue processors for simple queue management or dedicated queue processors for customized or delayed processing of messages.

Do I need to make any configuration changes to make my queue processors work?

Yes. Queue processors and the supporting DSM Stream service require node classification at startup.

  • If you have a single Pega node, set it to be Universal to support both Stream service and queue processors by using the following setting: -DNodeType=Universal.
  • If you have more than one node and you want the Stream service to run separately from the queue processors, set the Stream service node by using -DNodeType=Stream and set the queue processor node by using  -DNodeType=BackgroundProcessing settings.

For more information about node classification, see Node classification.

How do I queue a message to a queue processor?

You can queue a message to a queue processor by performing either of the following actions:

  • Use the Queue-for-Processing activity method.
  • Use the Run in Background smart shape when you create a case in the Case Designer.

For more information, see Queue-for-Processing method, Running a background process.

Can I queue a message to be processed at a scheduled time?

Yes. When you define a queue processor, set the When to process parameter to Delayed. Then, when you queue the message, you can specify the time to process the message.

For more information, see Queue-for-Processing method.

Do I have to create a queue processor to use this feature?

No. By default, the following standard queue processors are included with Pega Platform™:

  • pyProcessNotification
  • pzStandardProcessor
  • pyFTSIncrementalIndexer

If your use case does not generate a huge volume of queue items and does not have high scaling and throughput requirements, queue the page to the existing standard queue processor and specify the activity that you want this queue processor to run.

For more information, see Queue Processor rules.

How does a queue processor work and where do I find information about how it runs?

The Queue Processor rule automatically generates a stream data set and a corresponding data flow. The stream data set sends messages to and receives messages from the stream service. The data flow manages the subscription of messages to ensure message processing. You can view data flows that correspond with the queue processors in your system on the QueueProcessors landing page in Admin Studio.

For more information see, Queue processor management.

How do I configure the scaling requirements for my queue processors?

Queue processors can scale both horizontally and vertically. You can increase the number of processing nodes, or in the rule configuration, you can increase the number of threads.

For more information, see Creating a Queue Processor rule.

Do the Queue Processor rules and Stream service run on all approved operating systems and Java Development Kits?

Yes, however, there are some differences, for example:

  • For all Windows-based Pega Platform installations, specify the log retention time for 720 hours by using the following setting:

      <env name="dsm/services/stream/server_properties/log.retention.hours" value=“720" />.

  • For IBM z/OS installations, the total number of Kafka partitions that you can create is limited. Therefore, reduce the default number of partitions per Kafka topic from 20 to a lower number by using the num.partitions setting. For example, use the following setting to reduce the number of partitions to five:

      <env name="dsm/services/stream/server_properties/num.partitions" value="5"/>. 

For more information, see Stream node configurations for a Queue Processor rule.

Why is my new queue processor not working?

If your queue processor does not work, consider the following situations:

  • As part of Pega Infinity™, a new AsyncProcessor requestor type was introduced. The access group that is configured in this requestor type is used to resolve queue processors. You need to ensure that the newly created queue processor is resolvable using this context. For more information, see AsyncProcessor requestor type.
  • Queue Processor rules internally rely on the DSM service. Ensure that you define at least one node in a cluster to be a Stream node by using the setting: -DNodeType=Stream. For more information, see Node classification for Queue Processor rules.
In Pega Cloud, a stream node is automatically configured and ready to use.

What happens when I queue messages with no stream node running?

When a stream node is not available and you queue a message, the system saves the message in a database and pushes it to Kafka when the stream node is running. The system does not process the messages in a queue until at least one stream mode is available.

For more information, see Node classification for Queue Processor rules.

When I create a queue processor, what happens if I enter a large number of threads per node?

Most threads do not process any messages. The maximum number of Kafka partitions is 20 for one Pega Platform cluster, which means that the maximum number of threads that can process messages across all nodes in a cluster is 20.

How do I monitor and diagnose issues with queue processors?

In Admin Studio, the QueueProcessors landing page lists the queue processors that currently run on your system. From this landing page, you can also trace, enable, and disable your queue processors. You can also use REST APIs to perform these actions.

For more information, see Tracing Queue Processor rules.

What agents have been converted into queue processors?

The following agents have been converted into queue processors:

  • ProcessEngine-ProcessNotification
  • ProcessEngine-AgentCaseBulkProcessing
  • FTSIncrementalIndexer

These agents are no longer available.

For more information, see Queue Processor rules.

How do I increase the performance of my queue processors?

Performance has two dimensions: time to process a message and total message throughput. You can increase total message throughput by performing one of the following actions:

  • Time to process a message depends on the amount of work done by the processing activity. Optimize the activity to reduce the time to process a message.
  • Enhance message throughput in the following ways:
    • Scale out by increasing the number of processing nodes.
      This setting applies only to on-premises users. For Pega Cloud, a bigger sandbox is required.
    • Scale up by increasing the number of threads per node up to 20 threads per cluster.

For more information, see Queue Processor rules.

88% found this useful

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.