HazelcastException after upgrading from 7.1.6 to 7.1.7
PDN article https://pdn.pega.com/release-note/open-ports-required-for-cluster-communication describes HazelcastException in Pega 7.1. The setup has 2 JVMs on six hosts (12 in total); two ports are open on each host. The hardware has load balancing. Can this cluster feature be turned off, and if so how?
2015-04-13 16:30:50,196 [ WEBK124] [ STANDARD] [ ] [ ] ( etier.impl.EngineStartup) ERROR - PegaRULES initialization failed. Server: WEBK124
com.pega.pegarules.pub.context.InitializationFailedError: PRNodeImpl init failed
2015-04-13 16:30:50,196 [ WEBK124] [ STANDARD] [ ] [ ] ( etier.impl.EngineStartup) INFO - PegaRULES initialization failed. Server: WEBK124
2015-04-13 16:30:50,337 [ WEBK124] [ STANDARD] [ ] [ ] (ervlet.WebAppLifeCycleListener) ERROR - Enterprise tier failed to initialize properly, PegaRULES not available
2015-04-13 16:30:50,352 [ WEBK124] [ STANDARD] [ ] [ ] (ervlet.WebAppLifeCycleListener) INFO - Web Tier initialization is complete.
Steps to Reproduce
This issue occurs in production.
The setup has ports required for Hazelcast disabled.
Use the following line in prconfig.xml to disable Hazelcast.
<env name="identification/cluster/protocol" value="none" />
However this is not recommended as the use of Hazelcast is the way forward for the Pega product in future releases a lot of internal functionality will be switched to use this framework for their functionality.
Overview of Hazelcast:
In Pega 7.1.7 engine auto discovers the nodes in the cluster and provide the info to Hazelcast in a form that it can use to establish its cluster membership.
When each node starts, it already updates the pr_sys_statusnodes table with its start datetime and its last pulse time. An additional column pyClusterAddress is added to the pr_sys_statusnodes table to contain the IP Address and Port number that Hazelcast used when each node starts. When a new Hazelcast node is being initialized, the pr_sys_statusnodes table can be queried (based on their last pulse times) to get a list of potential active members to use as the seed for Hazelcast's node discovery protocol.
Process for cluster startup:
• Prior to starting the cluster, look for a list of potential members in cluster/hazelcast/members (ie. the members are specified in prconfig.xml)
•If no member list is found in prconfig.xml, query pr_sys_statusnodes to get a list of nodes, ordered by last pulse time, most recent first. This is the list of potential members to use for discovery purposes.
•Attempt to join the cluster based on the list of potential members
•Store this nodes IP/Port into pr_sys_statusnodes - Any nodes that start up after this point should find me.
•If, after startup is complete, this node believes it's the only node in the cluster:
-Check pr_sys_statusnodes for nodes that have updated since this node started and attempt to rejoin the cluster if there are.
- If there aren't any new nodes, I can be assured that I'm the only one in the cluster. I don't have to look anymore because other nodes starting should now auto-
discover me.•If we got the members of the cluster from the prconfig.xml file, ensure that the members in my cluster are also in pr_sys_statusnodes. If not, we need to shut down since we have a DB/Cluster misconfiguration
The default port range for Hazelcast communication between nodes in a cluster is 5701 through 5800. This can be overridden by setting the value of "cluster/hazelcast/ports" in the prconfig.xml file. For instance, "5701-5800".
Auto discovery of nodes by Pega to seed for Hazelcast's node discovery protocol can be overridden by following setting in the prconfig.xml file:
Enable Hazelcast as the PRPC Clustering Protocol:
< env name="identification/cluster/protocol" value="hazelcast" />
This loads the appropriate runtime support for Hazelcast.
Enable multicast discovery support:
< env name="cluster/hazelcast/multicast/enabled" value="true" />
This enables Hazelcast's own default discovery protocol (NOT PRPCs default) to simplify quick setup of a cluster.
TCP/IP discovery (when multicast discovery is disabled (which is the PRPC default)):
First, specify one or more member addresses - these are machine addresses that should/will contain PRPC clustered nodes.
There are a number of ways to provide this info - it can be a hostname, localhost, or actual TCP/IP address, or a list of these (see the HZ doc above).
< env name="cluster/hazelcast/members" value="lislakw7eu" /> -- single value, hostname
< env name="cluster/hazelcast/members" value="10.88.29.85,10.88.29.99" /> -- list of values, ip addresses
< env name="cluster/hazelcast/members" value="10.88.29.85-99" /> -- range of values (.85-.99 inclusive)
Second, specify the ports Hazelcast can use - a range is best here.
< env name="cluster/hazelcast/ports" value="5701-5800" />
Expose Hazelcast's logging for debug purposes:
< env name="identification/cluster/logging" value="" /> -- yes, an empty string