Skip to main content

This content has been archived and is no longer being updated. Links may not function; however, this content may be relevant to outdated versions of the product.

Support Article

Cassandra Write Failing on Batch Run

SA-90998

Summary



Cassandra Write fails on multiple nodes when executing the data flow to write records to the Cassandra data set.
The limits configuration is updated to allow 128k open files for the Tomcat user (owner of Cassandra JVM process).


Error Messages



IdentifyStateForParty data flow fails due to Cassandra failure –

JVMStabilityInspector.java:117 - JVM state determined to be unstable. 
Exiting forcefully due to: 
java.io.FileNotFoundException: /opt/tomcat/apache-tomcat-9.0.21/cassandra_data/data/data/partydataforbat_c3e8b610a4ad15b787d1d1ee66a6ccd6-581a3ea0cfc611e998dd5d958be5ecab/data-partydataforbat_c3e8b610a4ad15b787d1d1ee66a6ccd6-ka-1043-Data.db (Too many open files)

        at java.io.RandomAccessFile.open0(Native Method) ~[na:1.8.0_222]
        at java.io.RandomAccessFile.open(RandomAccessFile.java:316) ~[na:1.8.0_222]
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) ~[na:1.8.0_222]


Steps to Reproduce



Execute the Batch  data flow.


Root Cause



A defect or configuration issue in the operating environment. 

The system's hard limit on open files was not relaxed as per the recommended Cassandra settings:


root - nofile 100000


Resolution



Make the following change to the operating environment: 

Set cat /proc/pid/limits

where pid is the actual pid of the Cassandra process

When the actual number of Max open files is lower than the requested limit, set the limit in /etc/systemd/system/cassandra.service as below:

[Service]
LimitNOFILE=100000

For this to take effect, execute the below:

sudo systemctl daemon-reload
Suggest Edit

Published November 6, 2019 - Updated December 2, 2021

Did you find this content helpful? Yes No

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.

We'd prefer it if you saw us at our best.

Pega Community has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice
Contact us