Choose an ETL approach on Databricks

Databricks supports several ways to build extract, transform, and load (ETL) pipelines. Databricks recommends Lakeflow pipelines, built on Apache Spark™ Declarative Pipelines (SDP), for most ETL workloads. If you already use dbt or Spark, those approaches work on Databricks too.

Available approaches

Each approach differs in how you define transformations and who manages dependencies, incremental processing, and orchestration. For background on the declarative and procedural models behind these approaches, see Procedural vs. declarative data processing in Databricks.

Approach	What it does	When to use it	Key constructs	Getting started
Lakeflow pipelines (SDP)	You declare the tables and transformations, and the pipeline engine manages dependencies, incremental processing, and orchestration.	New batch and incremental ETL workloads.	streaming tables, materialized views, flows, AUTO CDC, REPLACE WHERE	Lakeflow pipelines
dbt on Databricks	Runs your existing dbt models on Databricks compute.	Your team already maintains a dbt project.	dbt models, dbt tasks in Lakeflow Jobs	Tutorial: Create, run, and test dbt models locally
Spark notebooks and jobs	Runs Spark code that you write in notebooks or scripts and orchestrate with Lakeflow Jobs.	Complex transformation logic that you want to express directly in Spark.	Spark DataFrames, notebooks, Lakeflow Jobs	Lakeflow Jobs
Stored procedures	Encapsulates procedural SQL logic that runs on a SQL warehouse.	Procedural SQL that you want to package and reuse.	CREATE PROCEDURE, SQL scripting	CREATE PROCEDURE

Approach	What it does	When to use it	Key constructs	Getting started
Lakeflow pipelines (SDP)	You declare the tables and transformations, and the pipeline engine manages dependencies, incremental processing, and orchestration.	New batch and incremental ETL workloads.	streaming tables, materialized views, flows, AUTO CDC, REPLACE WHERE	Lakeflow pipelines
dbt on Databricks	Runs your existing dbt models on Databricks compute.	Your team already maintains a dbt project.	dbt models, dbt tasks in Lakeflow Jobs	Tutorial: Create, run, and test dbt models locally
Spark notebooks and jobs	Runs Spark code that you write in notebooks or scripts and orchestrate with Lakeflow Jobs.	Complex transformation logic that you want to express directly in Spark.	Spark DataFrames, notebooks, Lakeflow Jobs	Lakeflow Jobs
Stored procedures	Encapsulates procedural SQL logic that runs on a SQL warehouse.	Procedural SQL that you want to package and reuse.	CREATE PROCEDURE, SQL scripting	CREATE PROCEDURE

Additional resources

• Lakeflow pipelines
• Procedural vs. declarative data processing in Databricks
• When to use REPLACE WHERE vs. materialized views
• Data engineering with Databricks
• Data engineering best practices