TruncateException: Cannot achieve consistency level ALL
The user is seeing errors while truncating data set. Followed the steps outlined in the Support Article and this does fix the issue temporarily - only for one truncation. The second time tried to perform a truncation, it failed with the error message "TruncateException: Error during truncate: Cannot achieve consistency level ALL".
The error message is happening because ALL Cassandra nodes need to be up in order to perform a truncation. However, one of them is down now. Cannot see this when using the DDS landing pages in the product - DSM thinks that Cassandra is completely up and running. However, by using the Cassandra NodeTool utility, we see one node was down
Caused by: com.datastax.driver.core.exceptions.TruncateException: Error during truncate: Cannot achieve consistency level ALL
Steps to Reproduce
- Designer Studio -> Decisioning -> Decisions -> Data Flows.
- Click on DF1 – SampleDataFlows.
- Double click on "SampleDataSet" data set.
- Open the Data Set using the magnifier.
- Once the Data set rule is opened click action and select RUN and select Truncate under operation and select execute.
While reviewing the Cassandra logs, in most of the DNodes noticed following exception:
java.lang.IllegalArgumentException: Mutation of 41875212 bytes is too large for the maximum size of 16777216
On DNode-5, we observed OutOfMemoryError. In order to identify the root cause of the out-of-memory, we applied a diagnostic patch on 'dnode-7.2.1.jar' and observed that there were enormous records being accessed from Cassandra, each record taking 303 MB.
Filtered out customers who have more than a pre-defined number of accounts.
Published September 22, 2016 - Updated October 9, 2016