pipelines command group
Databricks CLI use is subject to the Databricks License and Databricks Privacy Notice, including any Usage Data provisions.
The pipelines command group within the Databricks CLI contains two sets of functionality. The first set allows you to manage a pipeline project and its workflow. The second set allows you to create, edit, delete, start, and view details about pipeline objects in Databricks.
For information about pipelines, see Spark Declarative Pipelines.
Manage pipeline projects
The following commands allow you to manage pipelines in projects. A pipelines project is the bundle that can contain one or more pipeline objects.
databricks pipelines apply-environment
Apply the latest environment to the pipeline.
Applies the current pipeline environment onto the pipeline compute. The environment applied can be used by subsequent dev-mode updates.
databricks pipelines apply-environment PIPELINE_ID [flags]
Arguments
PIPELINE_ID
The ID of the pipeline.
Options
Examples
The following example applies the environment to a pipeline:
databricks pipelines apply-environment abc123
databricks pipelines deploy
Deploy pipelines by uploading all files defined in the project to the target workspace, and creating or updating the pipelines defined in the workspace.
To run this command, a Declarative Automation Bundles databricks.yml file must be in the root of the project directory. For a tutorial that creates a pipeline project, then deploys and runs the pipeline, see Develop pipelines with Declarative Automation Bundles.
databricks pipelines deploy [flags]
Arguments
None
Options
--auto-approve
Skip interactive approvals that might be required for deployment
--fail-on-active-runs
Fail if there are running pipelines in the deployment
--force-lock
Force acquisition of deployment lock. This option disables the mechanism that prevents concurrent deployments from interacting with each other. It should only be used if the previous deployment crashed or was interrupted and left a stale lock file.
databricks pipelines destroy
Destroy a pipelines project.
databricks pipelines destroy [flags]
Arguments
None
Options
--auto-approve
Skip interactive approvals for deleting pipelines
--force-lock
Force acquisition of deployment lock. This option disables the mechanism that prevents concurrent deployments from interacting with each other. It should only be used if the previous deployment crashed or was interrupted and left a stale lock file.
databricks pipelines dry-run
Validates correctness of the pipeline's graph, identified by KEY. Doesn't materialize or publish any datasets.
databricks pipelines dry-run [flags] [KEY]
Arguments
KEY
The unique name of the pipeline to dry run, as defined in its YAML file. If there's only one pipeline in the project, KEY is optional and the pipeline is auto-selected.
Options
--no-wait
Don't wait for the run to complete
--restart
Restart the run if it's already running
databricks pipelines generate
Generate configuration for an existing Spark pipeline.
This command looks for a spark-pipeline.yml or *.spark-pipeline.yml file in the specified directory and generates a new *.pipeline.yml configuration file in the resources folder of the project that defines the pipeline. If multiple spark-pipeline.yml files exist, specify the full path to a specific *.spark-pipeline.yml file.
databricks pipelines generate [flags]
To generate configuration for an existing pipeline in the Databricks workspace, see databricks bundle generate pipeline and Generate configuration for an existing job or pipeline using the Databricks CLI.
Options
--existing-pipeline-dir
Path to the existing pipeline directory in src (e.g., src/my_pipeline).
--force
Overwrite existing pipeline configuration file.
Examples
The following example looks in the current directory and reads src/my_pipeline/spark-pipeline.yml, then creates a configuration file resources/my_pipeline.pipeline.yml that defines the pipeline:
databricks pipelines generate --existing-pipeline-dir src/my_pipeline
databricks pipelines history
Retrieve past runs for a pipeline identified by KEY.
databricks pipelines history [flags] [KEY]
Arguments
KEY
The unique name of the pipeline, as defined in its YAML file. If there's only one pipeline in the project, KEY is optional and the pipeline is auto-selected.
Options
--end-time string
Filter updates before this time (format: 2025-01-15T10:30:00Z)
--start-time string
Filter updates after this time (format: 2025-01-15T10:30:00Z)
databricks pipelines init
Initialize a new pipelines project.
For a tutorial that walks through creating, deploying, and running a pipeline project using the Databricks CLI, see Develop pipelines with Declarative Automation Bundles.
databricks pipelines init [flags]
Arguments
None
Options
--config-file string
JSON file containing key value pairs of input parameters required for template initialization
--output-dir string
Directory to write the initialized template to
databricks pipelines logs
Retrieve events for the pipeline identified by KEY. By default, this command shows the events of the pipeline's most recent update.
databricks pipelines logs [flags] [KEY]
Arguments
KEY
The unique name of the pipeline, as defined in its YAML file. If there's only one pipeline in the project, KEY is optional and the pipeline is auto-selected.
Options
--end-time string
Filter for events that are before this end time (format: 2025-01-15T10:30:00Z)
--event-type strings
Filter events by list of event types
--level strings
Filter events by list of log levels (INFO, WARN, ERROR, METRICS)
-n, --number int
Number of events to return
--start-time string
Filter for events that are after this start time (format: 2025-01-15T10:30:00Z)
--update-id string
Filter events by update ID. If not provided, uses the most recent update ID
Examples
databricks pipelines logs pipeline-name --update-id update-1 -n 10
databricks pipelines logs pipeline-name --level ERROR,METRICS --event-type update_progress --start-time 2025-01-15T10:30:00Z
databricks pipelines open
Open a pipeline in the browser, identified by KEY.
databricks pipelines open [flags] [KEY]
Arguments
KEY
The unique name of the pipeline to open, as defined in its YAML file. If there's only one pipeline in the project, KEY is optional and the pipeline is auto-selected.
Options
--force-pull
Skip local cache and load the state from the remote workspace
databricks pipelines run
Run the pipeline identified by KEY. Refreshes all tables in the pipeline unless otherwise specified.
To run this command, a Declarative Automation Bundles databricks.yml file must be in the root of the project directory and the pipeline must have already been deployed. For a tutorial that creates a pipeline project, then deploys and runs the pipeline, see Develop pipelines with Declarative Automation Bundles.
databricks pipelines run [flags] [KEY]
Arguments
KEY
The unique name of the pipeline to run, as defined in its YAML file. If there's only one pipeline in the project, KEY is optional and the pipeline is auto-selected.
Options
--full-refresh strings
List of tables to reset and recompute
--full-refresh-all
Perform a full graph reset and recompute
--no-wait
Don't wait for the run to complete
--refresh strings
List of tables to run
--restart
Restart the run if it's already running
databricks pipelines stop
Stop the pipeline if it's running, identified by KEY or PIPELINE_ID. If there is no active update for the pipeline, this request is a no-op.
databricks pipelines stop [KEY|PIPELINE_ID] [flags]
Arguments
KEY
The unique name of the pipeline to stop, as defined in its YAML file. If there's only one pipeline in the project, KEY is optional and the pipeline is auto-selected.
PIPELINE_ID
The UUID of the pipeline to stop.
Options
--no-wait
do not wait to reach IDLE state
--timeout duration
maximum amount of time to reach IDLE state (default 20m0s)
Manage pipeline objects
The following commands allow you to manage pipeline objects in Databricks. A pipeline object is a single pipeline within a project.
databricks pipelines clone
Create a new pipeline using Unity Catalog from a pipeline using Hive Metastore. This method returns the ID of the newly created clone. Additionally, this method starts an update for the newly created pipeline.
databricks pipelines clone PIPELINE_ID [flags]
Arguments
PIPELINE_ID
Source pipeline to clone from.
Options
--json JSON
The inline JSON string or the @path to the JSON file with the request body.
Examples
The following example clones a pipeline:
databricks pipelines clone abc123
databricks pipelines create
Create a new data processing pipeline based on the requested configuration. If successful, this command returns the ID of the new pipeline.
databricks pipelines create [flags]
Arguments
None
Options
--json JSON
The inline JSON string or the @path to the JSON file with the request body.
databricks pipelines delete
Delete a pipeline.
databricks pipelines delete PIPELINE_ID [flags]
Arguments
PIPELINE_ID
The pipeline to delete.
Options
databricks pipelines get
Get a pipeline.
databricks pipelines get PIPELINE_ID [flags]
Arguments
PIPELINE_ID
The pipeline to get.
Options
databricks pipelines get-update
Get an update from an active pipeline.
databricks pipelines get-update PIPELINE_ID UPDATE_ID [flags]
Arguments
PIPELINE_ID
The ID of the pipeline.
UPDATE_ID
The ID of the update.
Options
databricks pipelines list-pipeline-events
Retrieve events for a pipeline.
databricks pipelines list-pipeline-events PIPELINE_ID [flags]
Arguments
PIPELINE_ID
The pipeline to retrieve events for.
Options
--filter string
Criteria to select a subset of results, expressed using a SQL-like syntax.
--max-results int
Max number of entries to return in a single page.
--page-token string
Page token returned by previous call.
databricks pipelines list-pipelines
List pipelines defined in the Delta Live Tables system.
databricks pipelines list-pipelines [flags]
Arguments
None
Options
--filter string
Select a subset of results based on the specified criteria.
--max-results int
The maximum number of entries to return in a single page.
--page-token string
Page token returned by previous call.
databricks pipelines list-updates
List updates for an active pipeline.
databricks pipelines list-updates PIPELINE_ID [flags]
Arguments
PIPELINE_ID
The pipeline to return updates for.
Options
--max-results int
Max number of entries to return in a single page.
--page-token string
Page token returned by previous call.
--until-update-id string
If present, returns updates until and including this update_id.
databricks pipelines start-update
Start a new update for the pipeline. If there is already an active update for the pipeline, the request will fail and the active update will remain running.
databricks pipelines start-update PIPELINE_ID [flags]
Arguments
PIPELINE_ID
The pipeline to start an update for.
Options
--cause StartUpdateCause
Supported values: [API_CALL, JOB_TASK, RETRY_ON_FAILURE, SCHEMA_CHANGE, SERVICE_UPGRADE, USER_ACTION]
--full-refresh
If true, this update will reset all tables before running.
--json JSON
The inline JSON string or the @path to the JSON file with the request body.
--validate-only
If true, this update only validates the correctness of pipeline source code but does not materialize or publish any datasets.
databricks pipelines update
Update a pipeline with the supplied configuration.
databricks pipelines update PIPELINE_ID [flags]
Arguments
PIPELINE_ID
Unique identifier for this pipeline.
Options
--allow-duplicate-names
If false, deployment will fail if the name has changed and it conflicts with the name of another pipeline.
--budget-policy-id string
Budget policy of this pipeline.
--catalog string
A catalog in Unity Catalog to publish data from this pipeline to.
--channel string
Lakeflow pipelines release channel that specifies which version to use.
--continuous
Whether the pipeline is continuous or triggered.
--edition string
Pipeline product edition.
--expected-last-modified int
If present, the last-modified time of the pipeline settings before the edit.
--id string
Unique identifier for this pipeline.
--json JSON
The inline JSON string or the @path to the JSON file with the request body.
--name string
Friendly identifier for this pipeline.
--photon
Whether Photon is enabled for this pipeline.
--pipeline-id string
Unique identifier for this pipeline.
--schema string
The default schema (database) where tables are read from or published to.
--serverless
Whether serverless compute is enabled for this pipeline.
--storage string
DBFS root directory for storing checkpoints and tables.
--target string
Target schema (database) to add tables in this pipeline to.
databricks pipelines get-permission-levels
Get pipeline permission levels.
databricks pipelines get-permission-levels PIPELINE_ID [flags]
Arguments
PIPELINE_ID
The pipeline for which to get or manage permissions.
Options
databricks pipelines get-permissions
Get the permissions of a pipeline. Pipelines can inherit permissions from their root object.
databricks pipelines get-permissions PIPELINE_ID [flags]
Arguments
PIPELINE_ID
The pipeline for which to get or manage permissions.
Options
databricks pipelines set-permissions
Set pipeline permissions.
Sets permissions on an object, replacing existing permissions if they exist. Deletes all direct permissions if none are specified. Objects can inherit permissions from their root object.
databricks pipelines set-permissions PIPELINE_ID [flags]
Arguments
PIPELINE_ID
The pipeline for which to get or manage permissions.
Options
--json JSON
The inline JSON string or the @path to the JSON file with the request body.
databricks pipelines update-permissions
Update the permissions on a pipeline. Pipelines can inherit permissions from their root object.
databricks pipelines update-permissions PIPELINE_ID [flags]
Arguments
PIPELINE_ID
The pipeline for which to get or manage permissions.
Options
--json JSON
The inline JSON string or the @path to the JSON file with the request body.
Global flags
--debug
Whether to enable debug logging.
-h or --help
Display help for the Databricks CLI or the related command group or the related command.
--log-file string
A string representing the file to write output logs to. If this flag is not specified then the default is to write output logs to stderr.
--log-format format
The log format type, text or json. The default value is text.
--log-level string
A string representing the log format level. If not specified then the log format level is disabled.
-o, --output type
The command output type, text or json. The default value is text.
-p, --profile string
The name of the profile in the ~/.databrickscfg file to use to run the command. If this flag is not specified then if it exists, the profile named DEFAULT is used.
--progress-format format
The format to display progress logs: default, append, inplace, or json
-t, --target string
If applicable, the bundle target to use