SA-28294

Summary

The user is seeing errors while truncating data set. Followed the steps outlined in the Support Article and this  does fix the issue temporarily - only for one truncation. The second time tried to perform a truncation, it failed with the error message "TruncateException: Error during truncate: Cannot achieve consistency level ALL".

The error message is happening because ALL Cassandra nodes need to be up in order to perform a truncation. However, one of them is down now. Cannot see this when using the DDS landing pages in the product - DSM thinks that Cassandra is completely up and running. However, by using the Cassandra NodeTool utility, we see one node was down 

Error Messages

Caused by: com.datastax.driver.core.exceptions.TruncateException: Error during truncate: Cannot achieve consistency level ALL

Steps to Reproduce

 

1. Designer Studio -> Decisioning -> Decisions -> Data Flows.
2. Click on DF1 – SampleDataFlows.
3. Double click on "SampleDataSet" data set.
4.  Open the Data Set using the magnifier.
5.  Once the Data set rule is opened click action and select RUN and select Truncate under operation and select execute.

Root Cause

While reviewing the Cassandra logs, in most of the DNodes noticed following exception:
java.lang.IllegalArgumentException: Mutation of 41875212 bytes is too large for the maximum size of 16777216
 
On DNode-5, we observed OutOfMemoryError. In order to identify the root cause of the out-of-memory, we applied a diagnostic patch on 'dnode-7.2.1.jar' and observed that there were enormous records being accessed from Cassandra, each record taking 303 MB.

Resolution

Filtered out customers who have more than a pre-defined number of accounts.