Hazelcast uses wrong IP
SummaryHazelcast is picking up the wrong IP addresses. The user has a Pega environment with multi nodes on multiple physical servers with multiple network interface cards. When the user brings up the nodes, the Hazelcast as well as search nodes is picking wrong IP address. The server is not starting with some port conflict with the Weblogic server ports. Pega spawns process which tries to use the Weblogic default ports and ultimately fails to start.
Error MessagesSep 02, 2016 4:48:27 PM org.jboss.netty.channel.DefaultChannelPipeline
WARNING: An exception was thrown by a user handler while handling an exception event ([id: 0x1f031ab6, /IP:PORT :> /IP:PORT] EXCEPTION: java.io.IOException: Broken pipe)
java.lang.NoClassDefFoundError: Could not initialize class org.jboss.netty.channel.ChannelFutureListener
Steps to ReproduceStart the Pega nodes.
Root CauseIn a multi node, server and network interface cards environment, if we don't specify node level, IPs and ports, Pega will notify the user when nodes become available.
Secondly when DNODE is enable, by default, the port 7000 is used for storage 7003 stream communication. These ports are already used by Weblogic and the conflict is causing crash.
ResolutionSet the following prconfig settings in each node to specify the node IP, port range and cluster members. This will fix the Hazelcast picking up wrong IP issue
<env name="identification/cluster/protocol" value="Hazelcast" />
<env name="cluster/Hazelcast/interface" value="IP address of this node" />
<env name="cluster/Hazelcast/ports" value="port range" />
<env name="cluster/Hazelcast/members" value="members IP of the clusters" />This prconfig setting will fix the issue of picking up wrong search node IP
<env name="indexing/distributed/network/host" value="IP address of this node" />
Disable the Dnode using this prconfig
<env name="initServices/dnode" value="false" />If you want to use DSM in future use the following prconfig settings, to avoid Dnode port conflict with Weblogic.
Note: If you running Pega 72, to effect the dnode settings, you need HFix-28313(https://pdn.pega.com/support-articles/compegadsmdnodeapidnodeexception-unable-enable-dnode)
Published September 12, 2016 - Updated October 6, 2016