LinkedIn
Copied!

Table of Contents

Data flow run limits

Version:

Only available versions of this content are shown in the dropdown

A large number of data flow runs that are active at the same time can deplete your system resources. To ensure the efficient processing of data flows, you can configure dynamic system settings to limit the number of concurrent active data flow runs for a node type.

If the number of active runs reaches the limit, the system adds subsequent runs to a queue. When the Data Flow service has enough capacity, which means that there are fewer active runs for the node type than the configured limit, the oldest queued runs start automatically.

You can limit the number of active data flow runs by using the following dynamic system settings:

dataflow/max_active_runs
This setting defines the maximum number of active runs for node types that do not have their own limits configured.
dataflow/<node_type>/max_active_runs
A dynamic system setting in this format defines the maximum number of active runs for a particular node type. The <node_type> value is case-sensitive, for example, BackgroundProcessing.

The following table shows the max_active_runs dynamic system settings that are included with the system, with their default values. You can modify these settings or create additional settings for other node types. When the value is set to a number greater than zero, the specified limit is used. When the value is set to –1, an unlimited number of active runs is allowed.

If multiple node types are added to a node, the limits apply to each node type separately. For example, Batch and Search node types may be configured on a single node. If a limit of 10 active runs is applied to each node type through either the default or individual dynamic system settings, this means that 20 concurrent data flow runs may be active on the node.

max_active_runs dynamic system settings

Dynamic system setting Default limit
dataflow/max_active_runs 10
dataflow/Batch/max_active_runs 10
dataflow/BackgroundProcessing/max_active_runs –1
dataflow/RealTime/max_active_runs –1
dataflow/Search/max_active_runs –1
  • Limiting active data flow runs

    Ensure the optimal use of system resources by limiting the number of concurrent active data flow runs that are allowed for a node type, such as Batch or Search. If the limit is reached, the system queues subsequent runs and waits for active runs to stop or finish before queued runs can start.

Related Content

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.