Support Article
Hazelcast uses wrong IP
SA-27874
Summary
Hazelcast is picking up the wrong IP addresses. The user has a Pega environment with multi nodes on multiple physical servers with multiple network interface cards. When the user brings up the nodes, the Hazelcast as well as search nodes is picking wrong IP address. The server is not starting with some port conflict with the Weblogic server ports. Pega spawns process which tries to use the Weblogic default ports and ultimately fails to start.
Error Messages
Sep 02, 2016 4:48:27 PM org.jboss.netty.channel.DefaultChannelPipeline
WARNING: An exception was thrown by a user handler while handling an exception event ([id: 0x1f031ab6, /IP:PORT :> /IP:PORT] EXCEPTION: java.io.IOException: Broken pipe)
java.lang.NoClassDefFoundError: Could not initialize class org.jboss.netty.channel.ChannelFutureListener
at com.pega.dsm.dnode.impl.server.BasicAuthenticationHandler.sendHttpResponse(BasicAuthenticationHandler.java:135)
at com.pega.dsm.dnode.impl.server.BasicAuthenticationHandler.exceptionCaught(BasicAuthenticationHandler.java:62)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
Steps to Reproduce
Start the Pega nodes.
Root Cause
In a multi node, server and network interface cards environment, if we don't specify node level, IPs and ports, Pega will notify the user when nodes become available.
Secondly when DNODE is enable, by default, the port 7000 is used for storage 7003 stream communication. These ports are already used by Weblogic and the conflict is causing crash.
Resolution
Set the following prconfig settings in each node to specify the node IP, port range and cluster members. This will fix the Hazelcast picking up wrong IP issue
<env name="identification/cluster/protocol" value="Hazelcast" />
<env name="cluster/Hazelcast/interface" value="IP address of this node" />
<env name="cluster/Hazelcast/ports" value="port range" />
<env name="cluster/Hazelcast/members" value="members IP of the clusters" />
This prconfig setting will fix the issue of picking up wrong search node IP
<env name="indexing/distributed/network/host" value="IP address of this node" />
Disable the Dnode using this prconfig
<env name="initServices/dnode" value="false" />
If you want to use DSM in future use the following prconfig settings, to avoid Dnode port conflict with Weblogic.
Note: If you running Pega 72, to effect the dnode settings, you need HFix-28313(https://pdn.pega.com/support-articles/compegadsmdnodeapidnodeexception-unable-enable-dnode)
Published October 7, 2016 - Updated October 8, 2020
Have a question? Get answers now.
Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.