When running archival jobs, use the dataarchival/dailyLimitPerPolicy setting to change the rate at which you archive eligible cases.

When running purge jobs, use the dataarchival/purgeQueryLimit to change the rate at which a subquery deletes records.

For more information about archive and purge settings, see Creating and configuring case archive settings and Configuring purge cycle settings.

1. Plan to run your archive and purge jobs at an initial faster rate to clear your backlog.
   1. Find out the number of cases in your initial backlog and the rate at which cases become eligible for archiving.
   2. Determine a period of low-system load, then schedule an archive and purge process that finishes during that time period.

      For example, your system might experience low system load every day for five hours or every weekend for 12 hours.

   3. Determine how high your rate of archiving and purge can go by performing an experiment:
      1. Start the experiment by setting a low rate in the dataarchival/dailyLimitPerPolicy and dataarchival/purgeQueryLimit dynamic system settings, respectively.

         A reasonable low value for these settings is 100. For more information about archive and purge settings, see Creating and configuring case archive settings and Configuring purge cycle settings.

      2. Run a process with the following jobs with the low value for the number of cases:
         • pyPegaArchiver
         • pyPegaIndexer
         • pyPegaPurger

           For more information, see Case archiving and purging overview.

      3. Monitor the progress of your case archiving process.  For more information about monitoring the progress of your case archiving process, see Monitoring the progress of your case archival process.
      4. Determine the performance impact and time that this low rate of archive and purge process takes.
      5. Run the archiving process again with an increased rate of archiving and purging.

         A faster rate of archiving impacts system resources more and takes longer to complete.

         Note: In Pega Platform version 8.4.5 and earlier, the pyPegaPurger job can take a significant amount of time that even a small value, such as 5000, for the setting dataarchival/purgeQueryLimit can cause timeout errors. If you encounter an error, use a smaller value for dataarchival/purgeQueryLimit and then schedule multiple runs of only the pyPegaPurger job until the pr_metadata is empty. This indicates that all cases processed by pyPegaCopier and pyPegaIndexer have been purged by pyPegaPurger.
      6. Continue to increase the rate to determine a maximum that completes the process within the low system load duration and within an acceptable system impact.
2. Clear your backlog by using the maximum rate found above. The maximum rate of archiving and purging determines the time frame for clearing your backlog.
3. Adjust the archiving and purging process after clearing your backlog.
   • If your current rate of archiving and purging meets or exceeds the rate at which cases become eligible for archiving, keep that archiving rate.
   • If your current rate of archiving is slower than the rate at which cases become eligible for archiving, plan to run your archival and purge jobs more frequently to archive faster.
   For example: Run your archive and purge jobs twice a day instead of once if your rate does not exceed the rate at which cases become eligible for archiving.