Skip to main content

Operations and CI/CD

Changes made by hand in the workspace UI don't scale past a single contributor. When a second person ships changes, or staging must match production, you need repeatable deployments, code review, and a promotion path from development to production. On Databricks, you get these by defining your projects as code, keeping them in Git, and deploying them with continuous integration and continuous delivery (CI/CD).

Separate infrastructure from projects​

Manage Databricks DevOps in two layers, each with its own tool and lifecycle:

Layer

What it includes

Tool

Examples

Platform infrastructure

Resources outside a workspace, such as accounts, networks, metastores, workspaces, and cloud Identity and Access Management (IAM)

Terraform

Terraform examples

Databricks projects

Resources inside a workspace, such as jobs, pipelines, schemas, dashboards, apps, and registered models

Declarative Automation Bundles

Bundle configuration examples

Layer

What it includes

Tool

Examples

Platform infrastructure

Resources outside a workspace, such as accounts, networks, metastores, workspaces, and cloud Identity and Access Management (IAM)

Terraform

Terraform examples

Databricks projects

Resources inside a workspace, such as jobs, pipelines, schemas, dashboards, apps, and registered models

Declarative Automation Bundles

Bundle configuration examples

Keeping the layers separate means that a change to a pipeline doesn't touch your network configuration, and a platform team can manage workspaces without editing the projects that run in them. To create workspaces with Terraform, see Create development, staging, and production workspaces.

Define projects with Declarative Automation Bundles​

Declarative Automation Bundles, formerly known as Databricks Asset Bundles, are an infrastructure-as-code approach to managing your Databricks projects. A bundle is an end-to-end definition of a project. It includes the source files, such as notebooks and Python files, the YAML definitions of the Databricks resources that the project uses, and the tests that validate it. You deploy the bundle as a single unit to a target, such as development, staging, or production. See What are Declarative Automation Bundles?.

The following video demonstrates how to work with bundles:

The Databricks CLI includes the bundle commands, so a CI/CD runner, such as a GitHub Actions runner, needs only the Databricks CLI and workspace credentials to validate and deploy a bundle. For the resources that a bundle can define, see Declarative Automation Bundles resources.

The following repositories contain example bundles:

Repository

Contents

bundle-examples/knowledge_base

Bundles for individual resources and patterns, such as apps, dashboards, Genie Agents, jobs, pipelines, and Lakebase databases

databricks-dab-examples/flights

An end-to-end project on a flights dataset, in simple, advanced, and bundle template versions

databricks-dab-examples/knowledge-base

Examples that include an Azure DevOps CI/CD pipeline, a React and Lakebase app, metric views, and a uv-managed bundle

Repository

Contents

bundle-examples/knowledge_base

Bundles for individual resources and patterns, such as apps, dashboards, Genie Agents, jobs, pipelines, and Lakebase databases

databricks-dab-examples/flights

An end-to-end project on a flights dataset, in simple, advanced, and bundle template versions

databricks-dab-examples/knowledge-base

Examples that include an Azure DevOps CI/CD pipeline, a React and Lakebase app, metric views, and a uv-managed bundle

Develop bundles in Visual Studio Code​

If your team develops in an integrated development environment (IDE), you can work on bundles locally with the Databricks extension for Visual Studio Code. You can define, deploy, and run bundles for jobs and pipelines in a workspace without leaving the editor. The following video shows how to develop a bundle in Visual Studio Code and deploy it with GitHub Actions:

See Databricks IDE extension.

Deploy bundles with CI/CD​

A typical CI/CD workflow validates the bundle and deploys it to a target workspace. Add your own test steps between validation and deployment. The following GitHub Actions workflow deploys a bundle to a staging target when changes are pushed to the main branch, such as when a pull request merges. It authenticates as a service principal with workload identity federation. GitHub doesn't store a client secret or access token, only the service principal's ID. The workflow reads the bundle target, catalog, and schema from the variables of a GitHub environment named staging. Before you use it, create a federation policy for the service principal. See Enable workload identity federation for GitHub Actions.

YAML
name: Deploy to staging
on:
push:
branches: [main]

permissions:
# Required for workload identity federation.
id-token: write
contents: read

jobs:
deploy:
runs-on: ubuntu-latest
environment: staging
env:
DATABRICKS_AUTH_TYPE: github-oidc
DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}
DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }} # The service principal UUID.
DATABRICKS_BUNDLE_TARGET: ${{ vars.ENVIRONMENT }}
BUNDLE_VAR_catalog: ${{ vars.CATALOG }}
BUNDLE_VAR_schema: ${{ vars.SCHEMA }}
steps:
- uses: actions/checkout@v4

- name: Install Databricks CLI
uses: databricks/setup-cli@main

- name: Validate bundle
run: databricks bundle validate --output json

- name: Deploy bundle
run: databricks bundle deploy

Environment variables that start with BUNDLE_VAR_ set the values of bundle variables, so the same bundle deploys to each environment with different catalogs and schemas. See Substitutions and variables in Declarative Automation Bundles. For a complete example, see Run a CI/CD workflow with a bundle that runs a pipeline update. The same validate-then-deploy pattern applies to other CI/CD tools, such as Azure DevOps. See Continuous integration and delivery on Databricks using Azure DevOps.

Verify the deployment​

After the workflow runs, check the following:

  1. In GitHub Actions, open the workflow run, and confirm that the Validate bundle and Deploy bundle steps succeeded.
  2. In the target workspace, confirm that the resources that the bundle defines are present.

Migrate existing workspace assets to bundles​

If a job, pipeline, dashboard, Genie Agent, or app already exists in a workspace, you can generate its bundle configuration instead of writing it by hand. The following video covers creating a bundle, migrating workspace assets, and creating a base CI/CD pipeline:

Generate the configuration​

Start from an existing bundle project. If you don't have one, see What are Declarative Automation Bundles?. Run databricks bundle generate with the resource type and the ID of the existing resource. For a job or a pipeline, for example:

Bash
databricks bundle generate job --existing-job-id <job-id>
databricks bundle generate pipeline --existing-pipeline-id <pipeline-id>

The command writes the configuration into your bundle. It also supports dashboard, genie-space, and app. See databricks bundle generate.

Bind the configuration to the existing resource​

Add --bind to the generate command, or run databricks bundle deployment bind <key> <resource-id> --target <target> afterward. Binding links the bundle resource to the workspace resource, so that the next deploy updates it.

Validate and deploy to a non-production target​

Run databricks bundle validate, and then deploy to a development or staging target first. See Migrate existing resources to a bundle.

warning

If you deploy generated configuration without binding it, Declarative Automation Bundles creates a new resource instead of updating the existing one. Bind every resource before you deploy it.

warning

The DABs migrator that the video demonstrates isn't an officially supported Databricks tool. Validate generated bundles in a non-production workspace before you rely on them in CI/CD.

Organize repositories​

Keep your Databricks code and bundle configuration in a single Git repository. A single repository makes it easier for people and AI tools to share code and practices. If projects have separate deployment lifecycles, give each project its own bundle, and keep the bundles in the same repository. The exception is regulated industries that need separate repositories for confidentiality. See Single repository.

If confidentiality requires separate repositories, use one repository per project, with one bundle, one deployment boundary, and one owning team. Each repository's CI runs only for its own project, and a failed deploy has a clear owner.

Use trunk-based development: keep the main branch deployable, work in short-lived feature branches, and merge them back quickly.

For example, a marketing Customer 360 project and a finance revenue forecasting project each have their own bundle in the same repository. Each team deploys its project on its own schedule.

When to use each tool​

Use this table to decide which tool owns a change. Terraform manages the platform, and Declarative Automation Bundles manage the projects inside a workspace, including the workspace configuration that a project needs.

Situation

Use

Provision workspaces, networks, or cloud IAM

Terraform

Platform settings must match across accounts or regions

Terraform

Deploy Databricks projects, such as jobs, pipelines, notebooks, and schemas

Declarative Automation Bundles

Changes need review before they reach production

Declarative Automation Bundles

A one-off notebook or prototype with a single owner, and nothing downstream depends on it

Neither

Situation

Use

Provision workspaces, networks, or cloud IAM

Terraform

Platform settings must match across accounts or regions

Terraform

Deploy Databricks projects, such as jobs, pipelines, notebooks, and schemas

Declarative Automation Bundles

Changes need review before they reach production

Declarative Automation Bundles

A one-off notebook or prototype with a single owner, and nothing downstream depends on it

Neither

Next steps​

This is the last section of the Starter Journey. To revisit an earlier section, see Starter Journey: Set up Databricks for production.