Skip to main content
Last updated on

How to use Lakeflow pipelines

This page explains how you use Lakeflow pipelines across the life of a data pipeline, from the first design decisions through running at scale, and the trade-offs behind each stage. Each section links to the articles that show you how.

This guide assumes familiarity with core data engineering concepts. If you are new to pipelines, start with Apache Spark Declarative Pipelines for what the product is and the declarative model behind it, then work through the Tutorial: Build an ETL pipeline using change data capture.

Pipeline lifecycle overview​

A pipeline moves through six stages:

  1. Plan and design: Decide what you are building and choose the tools, language, and compute that fit.
  2. Ingest data: Bring source data into the pipeline reliably and incrementally.
  3. Transform and model: Clean, validate, join, and shape data into tables consumers can trust.
  4. Operationalize: Put the pipeline under version control, test it, schedule it, and promote it across environments.
  5. Run in production: Monitor, alert, debug, backfill, secure, and track lineage as the pipeline runs unattended.
  6. Mature and scale: Confirm production readiness and keep the pipeline healthy as volume and team size grow.

The stages are not strictly sequential, but they map to the order in which questions come up. Because Lakeflow pipelines handle orchestration, checkpointing, retries, and incremental processing, your work at each stage is mostly a design decision rather than implementation.

Plan and design​

Your first decisions shape everything downstream. For how the declarative model compares with writing procedural steps yourself, see Procedural vs. declarative data processing in Databricks.

A few choices set your starting configuration:

  • A standalone dataset or a pipeline. A single materialized view or streaming table can be defined in SQL as a standalone dataset, and Databricks manages the refresh pipeline behind it. Author and operate a Lakeflow pipeline as a unit when you need Python authoring, sinks, or multi-stage orchestration. See Standalone pipelines vs. Lakeflow pipelines.
  • SQL or Python (or both). SQL suits transformations that are mostly filters, joins, and aggregations. Python suits custom logic, external libraries, or generating many similar tables programmatically. The choice is per file rather than pipeline-wide, so you can mix both and don't have to settle it up front.
  • Serverless or classic compute. Serverless is the recommended default and removes cluster configuration. Choose classic when you need specific instance types, custom cluster policies, or an init script. See Configure a serverless pipeline and Configure classic compute for pipelines.
  • Triggered or continuous execution. Start triggered, since it only consumes compute while it runs. Continuous mode keeps compute running to process new data with minimal delay, which is usually the largest cost factor, so reserve it for a proven latency requirement. See Triggered vs. continuous pipeline mode.

A pipeline infers its execution graph from the datasets your code references, so design work is largely naming and sequencing datasets. The core decision is which type each output should be: a Streaming tables for append-heavy incremental data, or a Materialized views for recomputed aggregates and joins. That choice drives cost and correctness, because incremental processing scales with the rate of new data while a full recompute scales with your entire history. For which type fits which job, see What are pipelines?.

Because pipeline code is ordinary Python and SQL, you can write, lint, and validate it in your own editor before deploying to a shared workspace.

In this stage​

Questions to think about while in this stage:

Ingest data​

The central design question is whether a source is append-only or changes in place. That determines how you model the target:

  • Append-only sources, such as files landing in cloud storage or events on a message bus, ingest into a streaming table, which checkpoints its progress so a restart neither reprocesses nor drops data. Auto Loader handles files, discovering new ones and inferring and evolving schema as they arrive. Message buses such as Apache Kafka, Azure Event Hubs, Amazon Kinesis, and Google Pub/Sub read directly into a streaming table. Deduplicate downstream, since a bus can deliver the same event more than once. For Azure Event Hubs specifically, see Use Azure Event Hubs as a pipeline data source.
  • Sources that update and delete rows, such as most databases and many software as a service (SaaS) systems, use change data capture (CDC). A full copy on every run is wasteful and grows slower as the source grows, so CDC reads only the rows that changed since the last run. The AUTO CDC API applies those changes without hand-written merge logic; see The AUTO CDC APIs: Simplify change data capture with pipelines. A flow applies CDC into a streaming table, and several flows can feed one table, which is how you fan multiple sources into a single target.

Checkpointing and retries are automatic, so a pipeline resumes from the last processed offset rather than reprocessing everything. Two safeguards are opt-in:

  • A rescued-data column captures records that don't match the expected schema.
  • Expectations apply the row-level action you define.

If a streaming checkpoint becomes invalid, prefer the cheapest recovery that preserves table data.

In this stage​

Questions to think about while in this stage:

Transform and model​

Transformation turns ingested data into clean tables that people and tools can trust. This is where the medallion pattern (bronze to silver to gold) takes concrete form.

Cleaning and validation come first. Expectations are a built-in Lakeflow pipeline feature: data quality constraints the pipeline evaluates on every row of every run, reporting pass and fail counts, so quality is continuous rather than a one-time gate. Decide what happens when a row fails (warn and keep it, drop it, or fail the update) and where the gate belongs. Gates usually sit at the bronze-to-silver boundary, so everything downstream can be trusted without re-checking.

Joining and aggregating shape the silver-to-gold step. A materialized view fits a batch-style join or aggregation over existing tables, because it keeps results consistent with its sources: it refreshes incrementally when the query and sources allow and otherwise recomputes in full, producing the same result either way. That makes it the right choice when correctness matters more than latency, since it recomputes joins when a dimension changes. See How do pipelines refresh?. Joining live streams raises unbounded state, so streaming joins and aggregations need a watermark to bound how long the pipeline waits for late-arriving data.

Two correctness ideas run through this stage:

  • Idempotency means a pipeline produces the same result however many times it runs over the same input. Lakeflow pipelines are idempotent for the pieces they manage, such as checkpointed reads and key-based AUTO CDC upserts; you keep your own logic idempotent by avoiding non-deterministic functions in recomputed views.
  • At-least-once versus exactly-once processing. Managed Delta-to-Delta tables commit each micro-batch's inputs and outputs together, giving you exactly-once by default. That stops at the edges, such as a custom sink, a non-Delta target, or an unverified custom source, where you treat the write as at-least-once and make it idempotent, for example by upserting on a key.

Slowly changing dimensions (SCDs) also live here: AUTO CDC implements SCD Type 1 and Type 2 directly, so you set a type rather than write history-tracking logic.

In this stage​

Questions to think about while in this stage:

Operationalize​

Operationalizing moves a pipeline from something that runs for you to something the team can build, test, and ship repeatably. A pipeline is source code plus configuration, so ordinary software-engineering practices apply.

Testing covers two things at once: your transformation logic and the ongoing quality of the data flowing through it. Expectations handle the data side continuously. For logic, factor transformations into plain functions and unit-test them outside the runtime, then validate the pipeline graph with a dry run before materializing anything. See Unit testing for pipelines.

Keep pipeline code in Git and package it for deployment so it can be reviewed, reverted, and deployed consistently across environments. The package is not an alternative to Lakeflow pipelines. It is the project and CI/CD wrapper around your pipeline, and your data logic stays declarative. Parameterize environment-specific values like catalog names and paths so the same code runs unmodified in each environment. See Use parameters with pipelines.

To run a pipeline on a schedule, wrap it in a Run pipelines in a workflow: Databricks recommends scheduling and orchestrating pipelines with jobs, which also let you coordinate the pipeline with other work, such as chaining a downstream report or several pipelines. Within a run, a pipeline orders and parallelizes its own datasets, so orchestration only coordinates tasks outside the pipeline.

In this stage​

Questions to think about while in this stage:

Run in production​

Once a pipeline runs unattended against real data, the work becomes knowing whether it is healthy and fixing it when it isn't.

Monitoring works at three levels of depth. The Jobs & Pipelines list gives an at-a-glance status for recent runs. The pipeline monitoring UI shows every table and flow color-coded by status, with row counts, data quality metrics, and backlog metrics for streaming tables. The event log underneath both is the source of truth for anything programmatic or historical. Configure failure notifications so you learn about a broken run before your stakeholders report it. For an overview of the monitoring surfaces, see Monitor pipelines.

Debug by working backward from the failure highlighted on the graph to the full error detail in the event log, then re-run only what failed. Retry behavior differs by trigger: manually triggered updates disable automatic retries so you see errors immediately, while scheduled updates retry recoverable failures. A production alert might therefore clear itself on retry where the same failure won't during interactive development. While developing, Genie Code can help diagnose and fix code-level errors as you iterate, though today it targets authoring pipelines rather than diagnosing production runs.

Model a backfill as its own explicit, one-time flow feeding the same target as your regular incremental flow. Keeping it separate records when and how history was loaded and keeps the steady-state logic simple.

Secure a pipeline by controlling who can operate it, running it as a dedicated service principal rather than a personal account, and keeping credentials in a secret scope rather than in source code. Lineage is automatic, captured down to the column level. A pipeline writes to an external system through a Sinks in Lakeflow pipelines, the edge where the at-least-once thinking above applies.

In this stage​

Questions to think about while in this stage:

Mature and scale​

A mature pipeline runs unattended and grows without a rewrite. Confirming readiness and planning how to scale define this stage.

Production readiness is a checklist across data quality, reliability, observability, deployment, cost, and governance. Treat every unchecked item as a known gap: does each dataset that can receive bad data have an expectation, is the pipeline scheduled rather than hand-started, are failure notifications configured, does it run as a service principal, is it deployed from version control across at least a dev and prod target. Data quality and notifications are the cheapest to add and the most likely to catch an undetected bad run.

Scale in response to concrete signals that pipeline health is degrading:

  • Update duration is trending up.
  • Autoscaling is repeatedly hitting its ceiling.
  • Cost is growing faster than the underlying business.
  • Materialized views are falling back to full recomputes.

Try compute-level levers first, such as moving to serverless or matching its performance mode to your latency needs. Beyond that, how you organize datasets across pipelines matters most:

  • A pipeline has a concurrency limit: it updates only a set number of datasets at the same time. Once a pipeline has more datasets than that limit, the extra updates wait in a queue, so the pipeline's total update time grows.
  • Group related datasets, and split unrelated ones. Group by domain, shared refresh cadence, and dependency; split at ownership, layer, and latency boundaries. Separating ingestion from transformation, for example, keeps a slow ingest from delaying everything downstream and keeps each pipeline small enough to stay under the concurrency limit.

Merging two small pipelines later is easier than splitting one large pipeline already in production. For how to group and split datasets, see Organize datasets across Lakeflow pipelines.

In this stage​

Questions to think about while in this stage:

Additional resources​

On this page