Changing the data flow failure threshold

In the Real-time processing and Batch processing tabs, you can view the number of errors that occurred during stream and non-stream data processing. By clicking the number of errors in the # Failed records column, you can open the data flow errors report and determine the cause of the error. When the number of errors reaches the data flow failure threshold, the data flow fails.

Note: By default, the threshold for real-time runs is set to 1000 errors and for batch runs the threshold is set to only one error. A real-time data flow can continue even if some errors occur, while a batch data flow cannot continue.

If you want to continue the run longer or complete the run for all the records and look into the reason of the failure later, you can increase the default threshold.

  1. In the Explorer panel, click Records > SysAdmin > Dynamic System Settings.
  2. Filter Dynamic System Settings instances in the Setting Purpose column.

    1. Click the Filter icon.

    2. In the Search Text field enter dataflow, and click Apply.

  3. Open the dataflow/realtime/failureThreshold instance to change the threshold for real-time data flows and click Save.
  4. Open the dataflow/batch/failureThreshold instance to change the threshold for batch data flows and click Save.