Support Article

Multinode Support for Stream datasets

SA-37188

Summary



Provide feature to able to use the Stream datasets in a multinode set-up to scale out the processing. Following are the criterias:

1. Records saved in Stream datasets should be consistently routed to the same nodes so that the state maintained in each and every node is consistent.
2. To be able to indicate which property, or combination of properties, will be used for partitioning so as to consistently route records to nodes. By default, the class keys should be used as partitioning keys.
3. In case a node fails, redistribute the partitions across the remaining nodes to continue the processing.
4. Data flow to validate at activation time, in case it uses event strategies, if the partitioning of the stream shape in the event strategy is the same or a subset of the ones in the stream data set. If the validation fails than the activation should fail.


Error Messages



Not Applicable


Steps to Reproduce



Not Applicable


Root Cause



A defect in Pegasystems’ code or rules.

Resolution



Apply HFix-33801.

Instructions:
  1. In order to utilise the feature, specify a partition key on a newly created stream dataset.
  2. After setting that key, records saved to this dataset will be directed to nodes that are part of the Data Flow cluster.
  3. The algorithm uses a simple hash calculation and only forwards records that run Data Flow services.
  4. Once a node is removed or added to the cluster the algorithm will be adjusted to rebalance partitions.

Published April 27, 2017 - Updated May 14, 2017

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.