SA-31332

Summary

The second node of a multi-node system is not communicating with the Pega Cluster, and is indicating "Node failed to start" in the logfile when attempting to boot.

Error Messages

2016-11-18 13:16:12,517 [ ip-10-255-25xxx] [ STANDARD] [ ] [ ] ( hazelcast.instance.Node) FATAL - [10.255.xxx]:5701 [bd387c6b4d0465b247bdb4f35669b73f] [3.4.1] Could not join cluster in 300000 ms. Shutting down now!
2016-11-18 13:16:12,837 [ ip-10-255-2yyy] [ STANDARD] [ ] [ ] ( internal.access.DatabaseImpl) ERROR - Encountered problem when performing a Commit
java.lang.IllegalStateException: Node failed to start!

Steps to Reproduce

Attempt to start node.

Root Cause

A defect or configuration issue in the operating environment:  

The server machines have more than one NIC installed.

During the server start up, some server instances chose to use an IP address associated to a NIC that does not map the machine's hostname and thus caused the reported hazelcast start-up failure. 

Resolution

In systems with multiple network interfaces, one can can also specify which network interface Hazelcast should use.

Servers often have more than one network interface so you may want to list the valid IPs. Range characters ('*' and '-') can be used for simplicity.

So 10.255.255.*, for instance, refers to IPs between 10.255.255.0 and 10.255.255.255.

Interface 10.255.255.4-18 refers to IPs between 10.255.255.4 and 10.255.255.18 (4 and 18 included).

If Hazelcast cannot find a matching interface, then it will print a message to the console and will not start on that node.

The setting for Hazelcast to use a specified network interface is "cluster/hazelcast/interface".

For example:

<env name="cluster/hazelcast/interface" value="10.255.255.*" />

The setting can be added to the prconfig.xml file or applied through Dynamic System Setting.