Table of Contents

Troubleshooting Pega Platform Search

+

This content applies to On-Premises and Pega Cloud Services environments.

Prerequisites
    Stream service
    Search node
    Required reading
Search node and cluster mismanaged
    Cannot index custom properties
    Search returns incomplete or empty results
    Cannot add multiple Search nodes from the landing page
    Indexing host nodes forming separate Elasticsearch clusters
    When node is restarted, users cannot search for old work objects, rules, or data
    Indexing stuck in progress
Java class conflicts
    NoClassDefFoundError and NoSuchMethodError exceptions
Search index lost
    No Search index backup
Search queries for Report Definitions malformed
    Wrong search query generated for a value with spaces passed to a filter parameter 
    Unexpected items in search results for some values passed to pySearchString parameter
    Missing items in search results cause partial values to be passed to pySearchString
    Missing descendant work objects in search results
 

This article is related to other content about Pega Platform Search. See the required reading that is a prerequisite to using the information in this article:

Prerequisites

Symptoms of Pega Platform Search issues can usually be traced to the following problems:

Search node and cluster mismanaged

Java class conflicts

Search index lost

Search queries for Report Definitions malformed

Understanding how these problems manifest themselves (symptoms and errors) and learning root cause analysis of the patterns (explanations) leads to solutions (sometimes a choice of suggested approaches).

Prerequisites

Make sure that you have completed the following prerequisites.

Stream service

Search node

Required reading

Stream service

The Stream service is required. Make sure that the Stream service is running on at least one node of the cluster. See the Related Content for articles about the Stream service.

Search node

A Search node type is required for running the queue processor FTSIncrementalIndexer. See the Related Content for articles about Queue Processors and Node Classification.

Required reading

Read the following articles before learning how to troubleshoot the additional issues for Search that are identified in this article. Required hotfixes are identified in the two Troubleshooting articles. The configuration procedure article pertains to Pega 8.1 and Elasticsearch client-server mode.

Troubleshooting Elasticsearch, Common Issues

Troubleshooting pyIndexDirectory null after nodes restart

Configuring client-server mode for Elasticsearch

Search node and cluster mismanaged

Recognize, understand, and solve issues related to the mismanagement of Search nodes in a clustered environment.

Cannot index custom properties

Search returns incomplete or empty results

Cannot add multiple Search nodes from the landing page

Indexing host nodes forming separate Elasticsearch clusters

When node is restarted, users cannot search for old work objects, rules, or data

Indexing stuck in progress

Cannot index custom properties

Symptom

Business data properties are not indexed.

Explanation (Root Cause)

By default, Elasticsearch will index only the px, py, pz Pega properties and it will not index the business data.

Solution

  1. Create a data transform pySearchModel.
  2. Create or update instances of Data-CustomProperties-Search, which is driven by Data-Admin-System-Settings instance indexing/useDataInstance configuration.
    If
    indexing/useDataInstance is set to true, Elasticsearch uses the Data-CustomProperties-Search rule; otherwise, it uses pySearchModel.
    After 
    indexing/useDataInstance has been set to true, even if all the data transforms have not been converted, the Elasticsearch index will look for instances of Data-CustomProperties-Search only. 
    You will also need to re-index if you add more or modify the existing Data-CustomProperties-Search instances in the the pySearchModel data transform.

Search returns incomplete or empty results

Solution

Ensure that the following conditions are met:

  • Indexing is finished and available.
  • The Dynamic System Setting indexing/distributed/search_enabled is set to true.
  • On the Search landing page, the option Enable search indexing is selected.
  • There are no broken items in FTSIncrementalIndexers or all records are scheduled. (Search runs through Queue Processors; therefore, the Stream service needs to be enabled. If you add a new Stream service, then you must re-index data manually.)

Cannot add multiple Search nodes from the landing page

Symptom

The new node is not added when the Save Settings action is submitted.

Explanation (Root Cause)

Index directory of the node is not specified.

Solution

  1. Add the following JVM argument to the Search node:
    -Dindex.directory=/index_directory
  2. Make sure that the temp folder is empty.
  3. Restart the JVM.
  4. Check the Search landing page: The node should be added automatically.

Indexing host nodes forming separate Elasticsearch clusters

Symptom

Search results differ between nodes.

Behavior

When all nodes are restarted at the same time, clustering misbehaves. The unexpected behavior is caused by the fact that none of the nodes join the specified Elasticsearch cluster. Instead, a random node outside of the designated Elasticsearch nodes elects itself as a master node and the remaining nodes cluster with that random node.

Explanation (Root Cause)

Delayed cluster formation

Solutions (Suggested Approaches)

  • Choose a different database execution plan to speed up node startup and make the node discoverable.
  • Disable the Min Master Node Auto-Detection setting and set the min_master_node count explicitly so that the current node waits for other nodes to join.
  • Start the indexing host nodes first. Check the logs to see if the nodes are available.

When node is restarted, users cannot search for old work objects, rules, or data

Symptom

Data indexed before restart is not available for search.

Behavior

Data indexed before restart is not available, but new work can be searched.

Explanation (Root Cause)

The Node ID was changed after the nodes were restarted.

If the Node ID changed, a new directory is created in -Dindex.directory and reindexing starts by default. If there is already another indexing host node, it duplicates the index.

Solution

If reindexing does not start by default, explicitly initiate reindexing.

Indexing stuck in progress

Symptoms

Indexing status is incomplete for a long time. There are broken items in the Queue Processor and failure is caused by the following error:

Batch Indexing request failed. Possible cause: Quorum not met for writes to succeed. Action: Remove any offline nodes as index nodes. 

Explanation (Root Cause)

The quorum is not met. The number of unreachable Search nodes exceeds a healthy limit.

See also Troubleshooting Elasticsearch, common issues, Best practices, Guideline 3.

Solution

Restart the unhealthy Search nodes.

Java class conflicts

Recognize, understand, and solve issues related to Java class conflicts.

NoClassDefFoundError and NoSuchMethodError exceptions

NoClassDefFoundError and NoSuchMethodError exceptions

Symptoms

Initialization of Elasticsearch fails with a message containing errors such as java.lang.NoClassDefFoundError.

Search is not working. Indices might be unavailable or corrupted. There are errors in the PegaRULES logs such as java.lang.NoSuchMethodError.

Explanation (Root Cause)

The Pega Platform relies on many internal and external JAR libraries. It might happen that some of the JAR libraries have been upgraded or changed, and new versions are not compatible with the rest of the Pega Platform.

Solution

  1. Use Log-PegaRULESMove to determine if there were some Import or Export changes in the Pega Platform that might have changed JAR libraries.
  2. Find problematic JAR libraries:
    • Is there some information about the library in the logs?
    • What JAR files have been changed by Import?
    • Do you see the same libraries with multiple versions?
    • Can all existing JAR files be found with the following query?
       select * from pr_engineclasses; 
  3. If you suspect that a JAR file is wrong (its version could be wrong), you can choose to take one of the following actions:
    • Remove it or update it with the Import procedure in Pega Platform.
    • Remove it from the table pr_engineclasses.

Search index lost

Recognize, understand, and solve issues related to a lost Search index.

No Search index backup

No Search index backup

A lost Search index cannot be recovered if you have failed to back up the index.

Symptom

The Search index has been lost.

Explanation (Root Cause)

No backup of the Search index was created.

Solution: Create a backup of the Search index

If you are using a Pega Platform release prior to Pega 8.1.1, install the required hotfix. Then complete the steps below.

Required hotfixes are identified in Troubleshooting Elasticsearch, Common Issues and in Troubleshooting 'pyindexdirectory' null after nodes restart.

If you are using Pega 8.1.1 or a later release, complete the following steps.

Prerequisite: Before you start this procedure, ensure that the REST interface is enabled on Elasticsearch.

  1. On all Search nodes, create repository as the top-level directory and ensure that it is writable for the Operating System user who runs the Pega Platform application server.
  2. Create a new Dynamic System setting:
    Short Description: indexing/distributed/path/repo
    Owning Ruleset:    Pega-SearchEngine
    Setting Purpose:   indexing/distributed/path/repo
  3. Restart all Pega Platform Search nodes.
  4. Create a snapshot by following Elasticsearch Reference [7.3], Modules, Snapshot and Restore:
    1. Create a repository in Elasticsearch, for example
      curl -d '{ "type": "fs", "settings": { "location": "/home/prpc/elastic-backup/fs" } }'
      -H "Content-Type: application/json" -X POST http://$HOST:$PORT/_snapshot/my_backup?pretty

      Ensure that the command returns the `acknowledged` message.
    2. Issue a snapshot command, for example
      curl -d '{ "indices": "rule" }' -H "Content-Type: application/json"
      -X POST http://$HOST:$PORT/_snapshot/my_backup/$SNAPSHOT_NAME?pretty&wait_for_completion=true
  5. Restore by close index(es), perform restore, and open index(es):
    # Close index ('rule' in this case).
    curl -X POST http://$HOST:$PORT/rule/_close
    # Restore
    curl -X POST http://$HOST:$PORT/_snapshot/my_backup/$SNAPSHOT_NAME/_restore?pretty
    # Re-open index
    curl -X POST http://$HOST:$PORT/rule/_open

Search queries for Report Definitions malformed

Recognize, understand, and solve issues related to the incorrect use of pxRetrieveSearchData in Report Definitions.

Wrong search query generated for a value with spaces passed to a filter parameter defined in a report definition

Unexpected items in search results for some values passed to pySearchString parameter

Missing items in search results cause partial values to be passed to pySearchString or a filter parameter in report definition

Missing descendant work objects in search results

Wrong search query generated for a value with spaces passed to a filter parameter defined in a report definition

Symptom

Queries are created with the OR operator for values with spaces passed to a filter parameter in a report definition used by pxRetrieveSearchData.  

Example: For the UK Postcode M1 5EA passed to the filter parameter pyCategory, the query is created with the following condition: (pyCategory:M1 OR pyCategory:5EA)

Explanation (Root Cause)

Each report definition defines its filters. Filters do not set the property pyUseTokenizer. It is equal to null for each filter. When pyUseTokenizer is null or true, then the filter value is analyzed by the Pega tokenizer. When value has more tokens (for example, M1 5EA has the tokens M1 and 5EA), then a query is created with the OR operator between the tokens (for example, pyCategory:M1 OR pyCategory:5EA).

Solution

Find the generated search query by enabling DEBUG as directed in Troubleshooting Elasticsearch, Common Issues, Best Practices, Guideline 9. The node on which the search request was made will show what the user was searching for, the generated Elasticsearch query, and the top results. It will also indicate if it was unable to reach the indexing host node. Developers are encouraged to use this approach to aid in determining the root cause of node-specific search problems.

Alternatively, follow these steps if you are using Pega 7.2.2 or a later release:

  1. Load a report definition into the clipboard. 
  2. Set the property pyUseTokenizer to false for filtered items in the report definition. 
  3. Use that virtual report definition in pxRetrieveSearchData by setting the property pyReportPageName
    A query is created with all the tokens in one string, for example,  pyCategory:"M1 5EA" and default_operator=AND is applied.   

Unexpected items in search results for some values passed to pySearchString parameter

Symptom

Unexpected items appear in search results for some values, for example,  -Monday is passed to the pySearchString parameter.

Explanation (Root Cause)

Elasticsearch query string syntax is used for a value passed to pySearchString in pyRetrieveSearchData

For example, the value -Monday is parsed by Elasticsearch query string syntax as 'Monday' must not be present.  

Solution

Use pySearchString only when you need query string syntax features.

For more information, see Elasticsearch Reference [5.6] Query DSL, Full text queries, Query String Query, Query String Syntax.

Missing items in search results cause partial values to be passed to pySearchString or a filter parameter in report definition

Symptom

For example, items with hyphenated string some-thing are missing in search results when the value thing  is passed to  pySearchString in pyRetrieveSearchData or to a filter parameter in the report definition.

Explanation (Root Cause)

The hyphen character (-) is defined as a special character in the Pega custom tokenizer for Elasticsearch because it is used in item keys by Pega. 

This customization allows searching by item keys (for example, search by Case ID).

Solution

  1. Use hyphens only for item keys
  2. Search using strings including all token characters defined in the Pega custom tokenizer.
    The Pega custom tokenizer has the following token characters:
    • all letters
    • all digits
    • special characters: hyphen (-), underscore (_), exclamation point (!), percent (%), at sign (@) and period (.).  

Missing descendant work objects in search results

Symptom

pxRetrieveSearchData does not return work objects from descendant classes of the class specified in pyReportClassName.

Explanation (Root Cause)

When a class in pyReportClassName is concrete and not a work pool, then the filter query will have a condition to get only work objects from that class. For example, consider the pxObjClass:App-Work-Example condition in the filter query for a class App-Work-Example.

Solution

The class should be concrete and a work pool. Then the filter query will have a condition to search all classes that start with that class name, for example, the pxObjClass:App-Work-Example*  condition in the filter query.


Related Content

Have a question? Get answers now.

Visit the Pega Support Community to ask questions, engage in discussions, and help others.