Develop pipeline code in your local development environment

You can author Python pipeline source code in your preferred integrated development environment (IDE), run it locally for testing, then validate, deploy, and run updates in your Databricks workspace without leaving your local environment.

Lakeflow pipelines are a superset of Apache Spark™ Declarative Pipelines. Code that uses only Apache Spark Declarative Pipelines APIs runs both locally and on Databricks, but code that uses features unique to Lakeflow pipelines, such as AUTO CDC and expectations, runs only on Databricks. For the feature differences, see Lakeflow pipelines Python language reference.

For interactive development and testing in the Databricks workspace, use the Lakeflow Pipelines Editor. See Develop and debug ETL pipelines with the Lakeflow Pipelines Editor.

Write pipeline code with IDE support

Write pipeline code using the pyspark.pipelines module, imported as dp:

Python

from pyspark import pipelines as dp

Because the module is part of Apache Spark, your IDE provides syntax checking, autocomplete, and type checking as you write. Apache Spark Declarative Pipelines code typically runs without modification on Databricks. Running the same import command in a Lakeflow pipeline imports the Databricks version of pipelines. For the full Lakeflow pipelines Python reference, see Lakeflow pipelines Python language reference.

Run pipelines locally for testing

You can also run pipelines locally to develop and test your code before you run it on Databricks. Use the spark-pipelines command line interface to initialize, validate, and run a pipeline with local Apache Spark. See the Spark Declarative Pipelines Programming Guide in the Apache Spark documentation.

You cannot run or test functionality that is specific to Lakeflow pipelines locally. This includes expectations and AUTO CDC functions.

Run pipelines in Databricks from your local environment

Use the databricks pipelines command group to validate, deploy, and run pipeline updates in your workspace, directly from your terminal:

Bash

databricks pipelines init      # scaffold a pipeline project
databricks pipelines dry-run   # validate the pipeline graph without publishing data
databricks pipelines deploy    # deploy the project to your workspace
databricks pipelines run       # run an update

Pipeline updates run in your Databricks workspace, not on your local machine, using the compute configured for the pipeline. These commands are interoperable with Declarative Automation Bundles bundle commands, so you can start with a simple project and adopt bundle configuration and CI/CD practices as it grows. To install and configure the CLI, see Install or update the Databricks CLI. For the full command reference, see pipelines command group. For a step-by-step walkthrough, see Develop pipelines with Declarative Automation Bundles.

Sync pipeline code from your IDE to a workspace

The following table summarizes options for syncing pipeline source code between your local IDE and a Databricks workspace:

Tool or pattern	Details
Databricks CLI (pipelines command group)	Use the databricks pipelines commands to deploy and run a pipeline project from your local environment. See pipelines command group.
Declarative Automation Bundles	Use Declarative Automation Bundles to deploy pipeline assets ranging in complexity from a single source code file to configurations for multiple pipelines, jobs, and source code files. See Convert a pipeline into a bundle project.
Databricks extension for Visual Studio Code	Databricks provides an integration with Visual Studio Code that includes easy syncing between your local IDE and workspace files. This extension also provides tools for using Declarative Automation Bundles to deploy pipelines assets. See Databricks extension for Visual Studio Code.
Workspace files	You can use Databricks workspace files to upload your pipeline source code to your Databricks workspace and then import that code into a pipeline. See What are workspace files?.
Git folders	Git folders let you sync code between your local environment and Databricks workspace using a Git repository as the intermediary. See Databricks Git folders.