Smart closure for integrated CDC pipelines
Integrated CDC pipelines run in triggered mode. Each update extracts changes from the source database, applies them to the destination tables, and then stops. Smart closure is the policy that determines how long an update runs. It gives each update time to process changes, stops the update after it catches up with the source, and limits how long an update can run.
Smart closure applies to integrated CDC pipelines (also called direct CDC), which run the change extractor and the applier together in a single triggered pipeline rather than as separate components. It does not apply to the standard gateway-based architecture, where the ingestion gateway runs continuously. See Create an integrated CDC pipeline for SQL Server.
When an update stops
Databricks configures and adjusts the minimum runtime, maximum runtime, and lag threshold based on operational experience.
Condition | What it means |
|---|---|
Minimum runtime | Each update runs for a minimum amount of time before it can stop because it has caught up with the source. |
Caught up with the source | After the minimum runtime, the update stops when it has applied the pending change backlog and is close to current with the source. The pipeline event log records the completion reason |
Reached the runtime limit | If the update doesn't catch up, it stops at the maximum runtime. The next update resumes where this one stopped. The pipeline event log records the completion reason |
Source schema change | A source schema change stops the current pipeline update. The pipeline then starts a new update that uses the new schema. |
How smart closure helps
- Lower cost when there's little change: After the minimum runtime, an update ends when it has caught up. This behavior allows changes to accumulate between updates and reduces the cost of continuously running compute.
- Bounded, predictable runtime: A large backlog can't make a single update run indefinitely. Each update is capped, and large workloads are spread across subsequent scheduled updates.
- Visibility into completion: Each update records why it ended, so you can tell whether it caught up with the source or stopped at the runtime limit.
Observe update completion
The completion reason appears in the message of the COMPLETED event in the pipeline event log. An update that caught up with the source completes with reason lag-converged, and an update that stopped at the runtime limit completes with reason max-runtime-cap-hit.
To find the completion reason, query the pipeline event log for the extractor's COMPLETED event. Replace <pipeline-id> with your pipeline ID:
SELECT timestamp, message
FROM event_log('<pipeline-id>')
WHERE message LIKE '%Direct Cdc Extraction has COMPLETED%'
ORDER BY timestamp DESC
The event message embeds the reason, for example Direct Cdc Extraction has COMPLETED (reason=lag-converged).
Schedule recurring updates
Because update duration varies with how much change data the source has, a large backlog might not finish in a single update. To ingest data on a recurring schedule, create a Lakeflow Jobs task that runs the pipeline. Schedule it frequently enough for subsequent updates to catch up. A starting point of 60 minutes works well for most workloads. After an update stops, the next update starts at the configured schedule or after a manual trigger, whichever occurs first. If a scheduled trigger fires while a previous update is still running, Databricks skips that update and uses the following scheduled run.