INC-126129 · Issue 569666
PropertyToColumnMap made more robust
Resolved in Pega Version 8.1.9
The DF_ProcessEmails dataflow was intermittently failing with a StageException error. This was traced to schema changes being propagated asynchronously by system pulse, which seem to have caused PropertyToColumnMap to cache stale schema. To resolve this, if the property mapping is not found the first time, the system will make a second attempt to get the mapping. Additional logging has also been added for better diagnostics.
INC-128385 · Issue 564521
Behavior made consistent between SSA and legacy engines
Resolved in Pega Version 8.1.9
There was a behavioral disparity between the legacy execution engine and the SSA engine where the latter was not creating a new page when the index was one above the size of the page list. This has now been corrected in order to make the SSA behavior fully backward compatible with the legacy engine, i.e. a new blank page will be added to the list if the index is one above the size of the list.
INC-129222 · Issue 568530
Handling improvements for commit logs
Resolved in Pega Version 8.1.9
The ADM commit log logs the number of unconsumed messages that are going to expire. In certain circumstances, it can include unconsumed messages that are not going to expire in the count. Because they are not expired and removed, the environment was running out of disk space due to the ADM commitlogs table growing larger than expected and performance issues were seen. To resolve this, a new adm_commitlog.adm_responses_commit_log_date_tiered table has been created, with a default_time_to_live of 24 hours. DateTieredCompactionStrategy has been set with max_window_size_seconds as 24hrs and tombstone_compaction_interval as 24hrs.
INC-132976 · Issue 580685
Performance improvements for Test Strategy data flow
Resolved in Pega Version 8.1.9
In the Test Strategy panel under Single case -> "Settings", selecting the "Data flow" option and choosing CustomerData dataflow was taking an excessive amount of time to run on a system with an extremely large database. To improve performance, two areas have been addressed: 1) the default behavior for record key suggestions in the test panel has been modified to collect only the ID as the additional data is not necessary at that time; 2) a DSS has been added that will opt out of reading and collecting the customer IDs in order to minimize data stored on the clipboard.
INC-138037 · Issue 586593
Strategy handling updated for very large systems using IH summary
Resolved in Pega Version 8.1.9
When a Strategy in a Real-time dataflow used IH Summary on a system with more than 5000 groups for one eventKey, the message "Error retrieving aggregates from Cassandra KVS" intermittently appeared. Investigation showed that if the number of result rows was greater than the FETCH_SIZE (set to 5000), it meant another read to Cassandra was required and an exception was generated. To resolve this, updates have been made so that instead of returning maps, the system will return iterators and change them to map on the calling thread.
SR-D92734 · Issue 553412
Simulation can take Data flow type as destination
Resolved in Pega Version 8.1.9
Support has been added for Data flow functionality as simulation target and data transform in simulation input.
INC-126796 · Issue 561533
Modifications to getFunctionalServiceNodes process
Resolved in Pega Version 8.2.7
The count of the Interaction History write related threads was increasing rapidly and a stack trace indicated "waiting on condition" and "java.lang.Thread.State: WAITING (parking)" errors. Investigation showed that this was due to getFunctionalServiceNodes using Hazelcast to determine node status by making a service request on an installation with a very large number of nodes, causing thread locking. To resolve this, the implementation has been updated to avoid calling getFunctionalServiceNodes on save of Interaction History, instead using Cassandra and only calling getFunctionalServiceNodes on the master node, not on all nodes.
INC-128385 · Issue 564519
Behavior made consistent between SSA and legacy engines
Resolved in Pega Version 8.2.7
There was a behavioral disparity between the legacy execution engine and the SSA engine where the latter was not creating a new page when the index was one above the size of the page list. This has now been corrected in order to make the SSA behavior fully backward compatible with the legacy engine, i.e. a new blank page will be added to the list if the index is one above the size of the list.
INC-128898 · Issue 564690
Updated precondition checks for Tumbling Time data in event strategies
Resolved in Pega Version 8.2.7
Tumbling Time data keys in event strategies were not being properly executed when certain window configurations were used. This was has been resolved by turning off the optimalization of Cassandra reads for small windows when window size is not known upfront and set dynamically (set size by property). In addition, an issue with Cassandra timeouts after the dataflow run had been running for several months has been resolved by adding a 'time to live' value for tumbling time windows, and event strategies has been switched to use leveled compaction by default.
SR-D82727 · Issue 547723
Improved management for table pr_log_dataflow_events
Resolved in Pega Version 8.2.7
The Lifecycle event table was sometimes growing too large. This additional strain of database transaction volume caused poor performance on the Dataflow tier and lead to cluster instability and time-consuming cluster restarts. Due to problems in one of the Pulse tasks, the Pulse thread was not processing single case metrics properly and causing the unbounded queue for single case to grow. This has been addressed by switching to a fixed queue size, which is configurable with the DSS: dnode/single_case_queue_size. The default value of the DSS is 4000, and if changed a system restart is required. An error will be logged each 1000 queue misses, and metrics will be dropped if the queue is full. In addition, the Pulse task frequency has been improved and managed to prevent interference with other Pulse tasks and will be triggered only if a run is system-paused for a long interval. Rebalances now have a failsafe if something unexpected happens during the Pausing of the run, and If the cluster becomes unstable, the life cycle events logs may be disabled with dataflow/run/events/persist .