SA-15

Summary

PRPC Environment is composed of three PRPC instances.  One PRPC instance hosts the LUCENE Index functionality, while the other two PRPC instances refer to the INDEX node to obtain search results.  Users have complained that if they are logged into a REMOTE node (not the Indexing Node), and they attempt to search for PRPC RULES or DATA, they always receive "limited" results - never complete results.  However, if the users log into the INDEX node, they are able to receive complete search results. 

Error Messages

Tracing the search functionality from a remote node reveals the following error message: "Invalid clipboard stream detected in module com.pega.pegarules.data.internal.clipboard.XMLStream.newStream(String, StorageStream). Reason: sax parse error: Expecting root element as PRPC "pagedata", instead found: HTML"

This error message is also found in the log files of the remote nodes.  No error messages are found in the logfile of the INDEX node.

Steps to Reproduce

Peform lucene search of rules from the Remote nodes.

Root Cause

To investigate, enable DEBUG tracing from the SMA:

- On the REMOTE nodes, enable DEBUG switches:

Rule_Obj_Activity.InvokeAxis2.Rule_Connect_SOAP.Action
Rule_Obj_Activity.InvokeHTTPConnector.Rule_Connect_HTTP.Action

- On the INDEX node, enable DEBUG switches:

com.pega.pegarules.integration.engine.internal.services.soap.SOAPService

From the DEBUG data, the ROOT CAUSE was determined to be a condition where the REMOTE node logs clearly showed the outgoing Search Request, but the RESPONSE was composed of what appeared to be HTML mixed with binary.  Also of note: the INDEX node logfile showed no data at all, indicating that the request never made it to the INDEX node.

From the body of evidence, it was determined that the REMOTE node search request never made it to the INDEX node.  Further, the response payload in the REMOTE node logfile indicated that some network device did reply to the search query.  Therefore, the conclusion was that a PROXY, a SECURITY DEVICE, or a LOAD BALANCER intercepted the Remote Node search request, and blocked the request from making it to the INDEX node.

Resolution

Advised admin to identify the network device, and tune the device to permit pass-through of the search requests from the REMOTE nodes to the INDEX node.  Admin identified a SiteMinder Security Appliance as the culprit, and completed security alterations to the device to permit pass through of search packets.