Support Article

7.2 DNode for this cluster is stuck in Joining state

SA-27581

Summary



D-Node server is showing as being stuck in state of 'Online-Joining'.

Error Messages



2016-08-23 12:20:12,922 [xx.xx.xx.xx] [ STANDARD] [ ] [ ] (dnode.api.DNodeServiceListener) ERROR - Cannot initialize DNode context and start Cassandra, PRPC continuing as non-DNode
com.pega.dsm.dnode.api.DNodeException: Cannot bootstrap cassandra
at com.pega.dsm.dnode.impl.cassandra.Cassandra.bootstrap(Cassandra.java:109)
at com.pega.dsm.dnode.api.DNodeBootstrap.bootstrap(DNodeBootstrap.java:41)
at com.pega.dsm.dnode.api.DNodeServiceListener$2.run(DNodeServiceListener.java:169)
[...]
Caused by:
java.lang.RuntimeException: Unable to create thrift socket to /xx.xx.xx.xx:9160
at org.apache.cassandra.thrift.CustomTThreadPoolServer$Factory.buildTServer(CustomTThreadPoolServer.java:268)
at org.apache.cassandra.thrift.TServerCustomFactory.buildTServer(TServerCustomFactory.java:46)
at org.apache.cassandra.thrift.ThriftServer$ThriftServerThread.<init>(ThriftServer.java:105)
at org.apache.cassandra.thrift.ThriftServer.start(ThriftServer.java:52)
at org.apache.cassandra.service.CassandraDaemon.start(CassandraDaemon.java:400)
at com.pega.dsm.dnode.impl.cassandra.Cassandra.startCassandra(Cassandra.java:183)
at com.pega.dsm.dnode.impl.cassandra.Cassandra.initCassandraDaemon(Cassandra.java:146)
at com.pega.dsm.dnode.impl.cassandra.Cassandra.bootstrap(Cassandra.java:102)
[...]
Caused by:
org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address /xx.xx.xx.xx:9160.
at org.apache.cassandra.thrift.TCustomServerSocket.<init>(TCustomServerSocket.java:75)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$Factory.buildTServer(CustomTThreadPoolServer.java:263)


Steps to Reproduce



Not applicable

Root Cause



Another process (not related to PRPC) was already using the Port that the Cassandra process was trying to use.

Resolution



Make sure that no two programs are attempting to start a Server Socket on the same port number (on the same interface).

On a Linux system , the following command (run as root) can be used to identify the PID and name of any process which is using particular port.

    netstat -tulpn | grep <port> 

For instance, the following command confirms that 'lighttpd' is running on port 8080 (0.0.0.0=on all interfaces); and the process id is 3972:

    > netstat -tulpn | grep 8080
    tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      3972/lighttpd




 

Published September 2, 2016 - Updated September 13, 2016

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.