Multinode Support for Stream datasets
SummaryProvide feature to able to use the Stream datasets in a multinode set-up to scale out the processing. Following are the criterias:
1. Records saved in Stream datasets should be consistently routed to the same nodes so that the state maintained in each and every node is consistent.
2. To be able to indicate which property, or combination of properties, will be used for partitioning so as to consistently route records to nodes. By default, the class keys should be used as partitioning keys.
3. In case a node fails, redistribute the partitions across the remaining nodes to continue the processing.
4. Data flow to validate at activation time, in case it uses event strategies, if the partitioning of the stream shape in the event strategy is the same or a subset of the ones in the stream data set. If the validation fails than the activation should fail.
Error MessagesNot Applicable
Steps to ReproduceNot Applicable
Root CauseA defect in Pegasystems’ code or rules.
- In order to utilise the feature, specify a partition key on a newly created stream dataset.
- After setting that key, records saved to this dataset will be directed to nodes that are part of the Data Flow cluster.
- The algorithm uses a simple hash calculation and only forwards records that run Data Flow services.
- Once a node is removed or added to the cluster the algorithm will be adjusted to rebalance partitions.
Published April 27, 2017 - Updated May 14, 2017