Stay organized with collections
Save and categorize content based on your preferences.
A flow control mechanism in Cloud Data Fusion prevents you from submitting
too many requests, which can cause stuck or failed pipeline runs. This mechanism
applies to start (Launch) requests for the following pipelines and jobs in all
editions of Cloud Data Fusion, versions 6.6 and later:
Batch pipelines
Real-time pipelines
Replication jobs
Thresholds
To control the number of outstanding start requests in Cloud Data Fusion,
flow control relies on two thresholds:
Launch threshold
Limits the number of launch requests being made. If the
number of launch requests received by Cloud Data Fusion that are not yet in
Running state is greater than the launch threshold (number of launch requests
that aren't running yet > launch threshold), the request is rejected. The
number includes batch pipelines, real-time pipelines, and
replication jobs.
Runs threshold
Limits the number of outstanding launch requests in
the system. If the number of outstanding launch requests received by
Cloud Data Fusion that have not yet finished is greater than a specified
runs threshold (number of outstanding requests that aren't finished yet > runs
threshold), the request is rejected.
Rejected requests
If either threshold is met, every new start request gets rejected. For example,
a rejected batch pipeline has a Rejected status in the logs, and a dialog in
the UI describes why the launch was rejected.
Metrics
The number of launching and running pipelines along with the number of rejected
pipelines in the system can be accessed through the metrics service as follows.
Launching metric
Reports the number of launching requests in the system. That is, it reports the
number of launch requests that are in Provisioning or Starting states.
Metrics name: flowcontrol.launching.count
POST /v3/metrics/query?metric=system.flowcontrol.launching.count
Running metric
Reports the number of running pipelines and jobs in the system. The run
threshold is based on the sum of the Launching and Running metrics values.
Metrics name: flowcontrol.running.count
POST /v3/metrics/query?metric=system.flowcontrol.running.count
Rejected metric
Reports the number of rejected runs due to the flow control mechanism.
Metrics name: program.rejected.runs
POST /v3/metrics/query?metric=system.program.rejected.runs
What's next
For examples of providing tags and aggregating the Rejected metric for
different pipelines, see
CDAP Metrics Microservices.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-29 UTC."],[[["\u003cp\u003eCloud Data Fusion employs a flow control mechanism to prevent an excess of start requests, which could lead to pipeline issues.\u003c/p\u003e\n"],["\u003cp\u003eThe flow control mechanism uses a launch threshold that limits the number of launch requests not yet in the "Running" state and a runs threshold that limits the number of outstanding requests not yet finished.\u003c/p\u003e\n"],["\u003cp\u003eWhen either the launch or runs threshold is exceeded, new start requests will be rejected, and the rejection will be noted in the logs and displayed in the UI.\u003c/p\u003e\n"],["\u003cp\u003eMetrics are available to monitor the number of launching requests (\u003ccode\u003eflowcontrol.launching.count\u003c/code\u003e), running pipelines and jobs (\u003ccode\u003eflowcontrol.running.count\u003c/code\u003e), and rejected runs (\u003ccode\u003eprogram.rejected.runs\u003c/code\u003e).\u003c/p\u003e\n"]]],[],null,["# Cloud Data Fusion flow control overview\n\nA flow control mechanism in Cloud Data Fusion prevents you from submitting\ntoo many requests, which can cause stuck or failed pipeline runs. This mechanism\napplies to start (`Launch`) requests for the following pipelines and jobs in all\neditions of Cloud Data Fusion, versions 6.6 and later:\n\n- Batch pipelines\n- Real-time pipelines\n- Replication jobs\n\nThresholds\n----------\n\nTo control the number of outstanding start requests in Cloud Data Fusion,\nflow control relies on two thresholds:\n\nLaunch threshold\n: Limits the number of launch requests being made. If the\n number of launch requests received by Cloud Data Fusion that are not yet in\n `Running` state is greater than the launch threshold (number of launch requests\n that aren't running yet \\\u003e launch threshold), the request is rejected. The\n number includes batch pipelines, real-time pipelines, and\n replication jobs.\n\nRuns threshold\n: Limits the number of outstanding launch requests in\n the system. If the number of outstanding launch requests received by\n Cloud Data Fusion that have not yet finished is greater than a specified\n runs threshold (number of outstanding requests that aren't finished yet \\\u003e runs\n threshold), the request is rejected.\n\nRejected requests\n-----------------\n\nIf either threshold is met, every new start request gets rejected. For example,\na rejected batch pipeline has a `Rejected` status in the logs, and a dialog in\nthe UI describes why the launch was rejected.\n\nMetrics\n-------\n\nThe number of launching and running pipelines along with the number of rejected\npipelines in the system can be accessed through the metrics service as follows.\n\n### `Launching` metric\n\nReports the number of launching requests in the system. That is, it reports the\nnumber of launch requests that are in `Provisioning` or `Starting` states.\n\nMetrics name: `flowcontrol.launching.count` \n\n POST /v3/metrics/query?metric=system.flowcontrol.launching.count\n\n### `Running` metric\n\nReports the number of running pipelines and jobs in the system. The run\nthreshold is based on the sum of the `Launching` and `Running` metrics values.\n\nMetrics name: `flowcontrol.running.count` \n\n POST /v3/metrics/query?metric=system.flowcontrol.running.count\n\n### `Rejected` metric\n\nReports the number of rejected runs due to the flow control mechanism.\n\nMetrics name: `program.rejected.runs` \n\n POST /v3/metrics/query?metric=system.program.rejected.runs\n\nWhat's next\n-----------\n\n- For examples of providing tags and aggregating the `Rejected` metric for different pipelines, see [CDAP Metrics Microservices](https://cdap.atlassian.net/wiki/spaces/DOCS/pages/477692194/Metrics+Microservices#Querying-a-Metric)."]]
