Skip to main content

This content has been archived and is no longer being updated. Links may not function; however, this content may be relevant to outdated versions of the product.

Support Article

Partitioning does not work for data flows

SA-56764

Summary



On setting the partition key and running the data flow (based on a data set), data flows are not partitioned.


Error Messages



Not Applicable


Steps to Reproduce

  1. Create a data base table.
  2. Map the data set to a database table.
  3. Create a data flow based to read from the data set.


Root Cause



An issue in the custom application code or rules.


Resolution



Here’s the explanation for the reported behavior:

Partitioning does not depend just on the number of nodes present but also on how powerful the hardware (CPU, memory, and disk IO) is to run multiple threads per node.
The data flow execution splits the amount of work to multiple assignments, and one of the factor is partition key. It is better to have more number of partitions.

The below formula limits the number of assignments created for better parallelism.


Number of Assignments = Number of nodes in the data flow cluster * Number of Configured Threads * 2.

Ensure the configuration (such as, number of assignments to process, number of nodes and threads) meets the above factors.


 
Suggest Edit

Published August 24, 2018 - Updated December 2, 2021

Did you find this content helpful? Yes No

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.

We'd prefer it if you saw us at our best.

Pega Community has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice
Contact us