SA-18976

Summary

Pega search was not working as expected, and the search landing page reported that there were 170,000 items in the indexing queue. Users see empty results while searching for work items that were created recently.

Error Messages

Not Applicable

Steps to Reproduce

Not Applicable

Root Cause

The search index queue fell behind when there was downtime in the indexing node.  However, because the indexer underperforms when the queue is large, leaving the most recent items in queue unindexed.

Resolution

There are two steps to resolve this issue.  

1a. TRUNCATE pr_sys_workindexer_queue to delete the agent's queue.  The performance of this agent degrades noticably as the queue increases.  
1b. Rebuild the queue using the reindex process from the search landing page.  This process will take a long time - potentially several days, but searching will begin to come online as each item is reindexed.  This works in a different mechanism from the indexing queue and does not perform less well against large numbers of items to index.

2. To prevent the indexing queue from falling behind when an indexing node goes down, provision multiple indexing nodes (new feature in 7.1.7 and newer versions.)  This will both distribute the indexing load, and allow a second node to take over as the primary node if the primary node is offline.