Cassandra Write Failing on Batch Run
Cassandra Write fails on multiple nodes when executing the data flow to write records to the Cassandra data set.
The limits configuration is updated to allow 128k open files for the Tomcat user (owner of Cassandra JVM process).
IdentifyStateForParty data flow fails due to Cassandra failure –
JVMStabilityInspector.java:117 - JVM state determined to be unstable.
Exiting forcefully due to: java.io.FileNotFoundException: /opt/tomcat/apache-tomcat-9.0.21/cassandra_data/data/data/partydataforbat_c3e8b610a4ad15b787d1d1ee66a6ccd6-581a3ea0cfc611e998dd5d958be5ecab/data-partydataforbat_c3e8b610a4ad15b787d1d1ee66a6ccd6-ka-1043-Data.db (Too many open files)
at java.io.RandomAccessFile.open0(Native Method) ~[na:1.8.0_222]
at java.io.RandomAccessFile.open(RandomAccessFile.java:316) ~[na:1.8.0_222]
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) ~[na:1.8.0_222]
Steps to Reproduce
Execute the Batch data flow.
A defect or configuration issue in the operating environment.
The system's hard limit on open files was not relaxed as per the recommended Cassandra settings:
root - nofile 100000
Make the following change to the operating environment:
Set cat /proc/pid/limits
where pid is the actual pid of the Cassandra process
When the actual number of Max open files is lower than the requested limit, set the limit in /etc/systemd/system/cassandra.service as below:
[Service] LimitNOFILE=100000 For this to take effect, execute the below: sudo systemctl daemon-reload
Published November 6, 2019 - Updated December 2, 2021
Was this useful?
0% found this useful
Have a question? Get answers now.
Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.