SA-42260

Summary

Search is working only on the Primary indexing node of a multiple node application across multiple Pega servers. The Secondary indexing node(s) will either display "unavailable" or when running "Check Index diagnostic", the message returned is "index file unavailable or corrupted".

Error Messages

The following error appears in the PegaRULES log:

[8/1/17 10:38:50:107 EDT] 0000007a SystemOut O 2017-08-01 10:38:50,104 [ your_host] [ STANDARD] [ ] [ ] (.internal.PRSearchProviderImpl) ERROR - Failed to initialize full text search functionality for this node.
com.pega.pegarules.search.internal.es.FTSInitializationException: Failed to initialize full text search for this node.
at com.pega.pegarules.search.internal.PRSearchProviderImpl._initializeFTS(PRSearchProviderImpl.java:701)
at com.pega.pegarules.search.internal.PRSearchProviderImpl.initializeFTS(PRSearchProviderImpl.java:246)
at com.pega.pegarules.search.internal.PRSearchProviderImpl.initializeFTS(PRSearchProviderImpl.java:189)
at com.pega.pegarules.session.internal.async.startup.SearchInitializationTask.runTask(SearchInitializationTask.java:38)
at com.pega.pegarules.session.internal.async.startup.StartupTaskUtil.runTask(StartupTaskUtil.java:68)
at com.pega.pegarules.session.internal.mgmt.PREnvironment.finishInit(PREnvironment.java:537)
...
Caused by:
java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:594)
at sun.nio.ch.Net.bind(Net.java:574)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:260)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:87)
at com.pega.elasticsearch.common.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
at com.pega.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372)
at com.pega.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296)
at com.pega.elasticsearch.common.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at com.pega.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at com.pega.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:627)
at java.lang.Thread.run(Thread.java:809)

Steps to Reproduce

Not Applicable

Root Cause

A defect or configuration issue in the operating environment. The machines that the primary and secondary Pega Index nodes are running on have only one IP address and default Hazelcast (5701-5708) and Indexing ports (9300-9400) are open and available for Pega engine to use. If the machines are not multi-homed (multiple IPs), Pega engine can automatically detect correct IP and port ranges. The Configuration Management screen on SMA shows the primary and secondary nodes had specific IP address values for prconfig

Dynamic System Settings:

cluster/hazelcast/interface and indexing/distributed/network/host

There were also entries for cluster/hazelcast/ports and it was set to default 5701-5708.

Resolution

Perform the following local-change:

Remove all IP settings from prconfig.xml and Dynamic System Settings and restart these nodes. Also remove the port setting if default port ranges are available. If using single IP machines, these settings are not needed and Pega engine automatically picks up the right IPs and ports.