Support Article
Cassandra Write Failing on Batch Run
SA-90998
Summary
Cassandra Write fails on multiple nodes when executing the data flow to write records to the Cassandra data set.
The limits configuration is updated to allow 128k open files for the Tomcat user (owner of Cassandra JVM process).
Error Messages
IdentifyStateForParty data flow fails due to Cassandra failure –
JVMStabilityInspector.java:117 - JVM state determined to be unstable.
Exiting forcefully due to: java.io.FileNotFoundException: /opt/tomcat/apache-tomcat-9.0.21/cassandra_data/data/data/partydataforbat_c3e8b610a4ad15b787d1d1ee66a6ccd6-581a3ea0cfc611e998dd5d958be5ecab/data-partydataforbat_c3e8b610a4ad15b787d1d1ee66a6ccd6-ka-1043-Data.db (Too many open files)
at java.io.RandomAccessFile.open0(Native Method) ~[na:1.8.0_222]
at java.io.RandomAccessFile.open(RandomAccessFile.java:316) ~[na:1.8.0_222]
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) ~[na:1.8.0_222]
Steps to Reproduce
Execute the Batch data flow.
Root Cause
A defect or configuration issue in the operating environment.
The system's hard limit on open files was not relaxed as per the recommended Cassandra settings:
root - nofile 100000
Resolution
Make the following change to the operating environment:
Set cat /proc/pid/limits
where pid is the actual pid of the Cassandra process
When the actual number of Max open files is lower than the requested limit, set the limit in /etc/systemd/system/cassandra.service as below:
[Service] LimitNOFILE=100000 For this to take effect, execute the below: sudo systemctl daemon-reload
Published November 6, 2019 - Updated December 2, 2021
Have a question? Get answers now.
Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.