Skip to main content
Last updated on

Order pipeline flow execution with depends_on

Preview

This feature is in Public Preview.

To use depends_on, configure your pipeline to use the Lakeflow pipelines PREVIEW channel. See channel in Pipeline configurations.

By default, a pipeline schedules flows by their data dependencies: if one flow reads the table that another flow writes, the reader runs after the writer. Flow ordering lets a flow wait on another flow that it does not read from, by declaring the dependency explicitly with depends_on.

depends_on="other_flow" means that this flow starts only after other_flow completes successfully. It is a scheduling edge: it controls when a flow starts, not how it runs. Declaring depends_on does not change a flow's trigger, mode, or whether it is a one-time flow. You still declare those on the flow itself, for example with once=True.

When to use flow ordering​

The primary use case is draining historical data before switching to a live source while preserving streaming state, such as migrating a table from a batch backfill to a live Apache Kafka source.

Reading both sources at the same time does not work well: the bounded backfill holds back the watermark, which delays state eviction and windowed results. Draining the backfill first, and then starting the live stream, avoids this. Flow ordering sequences the two.

Flow ordering also supports other patterns:

  • Strictly ordering several backfills into the same table.
  • Phased pipelines, such as an initial load, then a catch-up, then a live feed.
  • Sequencing flows that write to different tables but must run in a set order.

Order flows with depends_on​

depends_on is available on the @dp.append_flow and @dp.update_flow decorators in the Python pipelines API. It accepts a single flow name or a list of flow names. With a list, every named flow must complete before the dependent flow starts.

The following example drains a one-time backfill into the events streaming table, and then starts a live Kafka stream into the same table only after the backfill finishes:

Python
from pyspark import pipelines as dp

dp.create_streaming_table(name="events")

# Drain the historical backfill first.
@dp.append_flow(target="events", once=True, name="events_backfill")
def events_backfill():
    return spark.read.table("historical_events")

# Start the live stream only after the backfill completes.
@dp.append_flow(target="events", name="events_live", depends_on="events_backfill")
def events_live():
    return (
        spark.readStream.format("kafka")
        .option("kafka.bootstrap.servers", "<server>:<port>")
        .option("subscribe", "events")
        .load()
    )

To wait on more than one predecessor, pass a list. The following example runs two backfills in parallel and starts the live stream only after both complete:

Python
@dp.append_flow(target="events", once=True, name="backfill_2024")
def backfill_2024():
    return spark.read.table("events_2024")

@dp.append_flow(target="events", once=True, name="backfill_2025")
def backfill_2025():
    return spark.read.table("events_2025")

@dp.append_flow(
    target="events",
    name="events_live",
    depends_on=["backfill_2024", "backfill_2025"],
)
def events_live():
    return spark.readStream.format("kafka").option("subscribe", "events").load()

To order backfills one after another instead of in parallel, chain depends_on across them:

Python
@dp.append_flow(target="events", once=True, name="backfill_2024")
def backfill_2024():
    return spark.read.table("events_2024")

@dp.append_flow(
    target="events", once=True, name="backfill_2025", depends_on="backfill_2024"
)
def backfill_2025():
    return spark.read.table("events_2025")

Requirements and behavior​

The following rules apply to flow ordering:

  • Flow ordering works only inside a pipeline. Without the pipeline scheduler, the ordering cannot be honored.
  • A predecessor must write to a table or sink, not a view. A flow that writes to a view never reaches a terminal state, so a flow ordered after it would never start. Streaming table targets and foreachBatch sinks are both valid predecessors.
  • Cross-destination ordering is allowed. A flow can depend on a flow that writes to a different table.
  • Unknown flow names and cycles are caught at validation, before the pipeline runs.

What can be a predecessor in triggered and continuous pipelines​

The kinds of flow that can act as a predecessor depend on the pipeline's execution mode:

  • Triggered pipelines: any flow can be a predecessor. Every flow in a triggered run reaches a terminal state, so the ordering applies within each run.
  • Continuous pipelines: a predecessor must be a one-time (once) flow that reaches a terminal state. A flow that runs continuously never terminates, so a flow ordered after it would never start, and the pipeline rejects it at validation.

Because a foreachBatch flow is always a streaming sink and cannot be a one-time flow, it can act as a predecessor only in a triggered pipeline. In a continuous pipeline, a foreachBatch flow can wait on a once predecessor, but it cannot itself be a predecessor.

Operational behavior​

Completion state for a once predecessor is durable, so it survives restarts and pipeline updates:

  • Restart: already-drained flows stay drained and are skipped. The pipeline resumes at the first flow that has not yet completed. After execution reaches the live flow, later restarts resume only the live flow.
  • Full refresh: clears the completion state for the chain and re-runs it from the beginning, in order.
  • Checkpoint reset for a single flow: resetting the live flow replays only the live flow. Upstream once backfills stay drained and are not re-run. This is the usual way to recover a live query without re-draining history.

Additional resources​

On this page