Skip to main content
Last updated on

Convert a pipeline into a bundle project

You can convert an existing pipeline into a Declarative Automation Bundles project. Bundles enable you to define and manage your Databricks data processing configuration in a single, source-controlled YAML file that provides easier maintenance and enables automated deployment to target environments.

For a tutorial that uses databricks pipelines commands to create a pipelines project, then deploys and runs a pipeline, see Develop pipelines with Declarative Automation Bundles.

Conversion process overview​

Diagram showing the specific steps in converting an existing pipeline to a bundle

The steps you take to convert an existing pipeline to a bundle are:

  1. Make sure you have access to a previously configured pipeline you want to convert to a bundle.
  2. Create or prepare a folder (preferably in a source-controlled hierarchy) to store the bundle.
  3. Generate a configuration for the bundle from the existing pipeline, using the Databricks CLI.
  4. Review the generated bundle configuration to ensure it is complete.
  5. Link the bundle to the original pipeline.
  6. Deploy the pipeline to a target workspace using the bundle config.

Requirements​

Before you start, you must have:

Step 1: Set up a folder for your bundle project​

You must have access to a Git repository that is configured in Databricks as a Git folder. You will create your bundle project in this repository, which will apply source control and make it available to other collaborators through a Git folder in the corresponding Databricks workspace. (For more details on Git folders, see Databricks Git folders.)

  1. Go to the root of the cloned Git repository on your local machine.

  2. At an appropriate place in the folder hierarchy, create a folder specifically for your bundle project. For example:

    Bash
    mkdir -p ~/source/my-pipelines/ingestion/events/my-bundle

  3. Change your current working directory to this new folder. For example:

    Bash
    cd ~/source/my-pipelines/ingestion/events/my-bundle

  4. Initialize a new bundle by running:

    Bash
    databricks bundle init

    Answer the prompts. Once it completes, you will have a project configuration file named databricks.yml in the new home folder for your project. This file is required for deploying your pipeline from the command line. For more details on this configuration file, see Declarative Automation Bundles configuration.

Step 2: Generate the pipeline configuration​

From this new directory in your cloned Git repository's folder tree, run the Databricks CLI bundle generate command, providing the ID of your pipeline as <pipeline-id>:

Bash
databricks bundle generate pipeline --existing-pipeline-id <pipeline-id> --profile <profile-name>

When you run the generate command, it creates a bundle configuration file for your pipeline in the bundle's resources folder and downloads any referenced artifacts to the src folder. The --profile (or -p flag) is optional, but if you have a specific Databricks configuration profile (defined in your .databrickscfg file created when you installed the Databricks CLI) that you'd rather use instead of the default profile, provide it in this command. For information about Databricks configuration profiles, see Databricks configuration profiles.

tip

If you have an existing Spark Declarative Pipelines (SDP) project (it has a spark-pipeline.yml file), you can copy that pipeline project to the src folder of the bundle, then use the databricks pipelines generate command to generate bundle configuration for it. See databricks pipelines generate.

Step 3: Review the bundle project files​

When the bundle generate command completes, it will have created two new folders:

  • resources is the project subdirectory that contains project configuration files.
  • src is the project folder where source files, such as queries and notebooks, are stored.

The command also creates some additional files:

  • *.pipeline.yml under the resources subdirectory. This file contains the specific configuration and settings for your pipeline.
  • Source files such as SQL queries under the src subdirectory, copied from your existing pipeline.
├── databricks.yml                            # Project configuration file created with the bundle init command
├── resources/
│   └── {your-pipeline-name.pipeline}.yml     # Pipeline configuration
└── src/
    └── {source folders and files...}         # Your pipeline's declarative queries

Step 4: Bind the bundle pipeline to your existing pipeline​

You must link, or bind, the pipeline definition in the bundle to your existing pipeline in order to keep it up to date as you make changes. To do this, run the Databricks CLI bundle deployment bind command:

Bash
databricks bundle deployment bind <pipeline-name> <pipeline-ID> --profile <profile-name>

<pipeline-name> is the name of the pipeline. This name should be the same as the prefixed string value of the file name for the pipeline configuration in your new resources directory. For example, if you have a pipeline configuration file named ingestion_data_pipeline.pipeline.yml in your resources folder, then you must provide ingestion_data_pipeline as your pipeline name.

<pipeline-ID> is the ID for your pipeline. It is the same as the one you copied as part of the requirements for these instructions.

Step 5: Deploy your pipeline using your new bundle​

Now, deploy your pipeline bundle to your target workspace using the Databricks CLI bundle deploy command:

Bash
databricks bundle deploy --target <target-name> --profile <profile-name>

The --target flag is required and must be set to a string that matches a configured target workspace name, such as development or production.

If this command is successful, you now have your pipeline configuration in an external project that can be loaded into other workspaces and run, and easily shared with other Databricks users in your account.

Promote across environments with targets​

A bundle defines named deployment environments called targets in databricks.yml, each pointing at its own workspace, catalog, and variable values. Targets are how you promote the same pipeline through dev, staging, and production, deploying identical source code to each successive environment without editing it:

YAML
bundle:
  name: orders_pipeline

variables:
  catalog:
    description: Unity Catalog to write to
    default: dev_catalog

targets:
  dev:
    mode: development
    default: true
    variables:
      catalog: dev_catalog

  prod:
    mode: production
    variables:
      catalog: prod_catalog
    run_as:
      service_principal_name: '12345678-90ab-cdef-1234-567890abcdef'

The mode you set on each target changes its deployment behavior:

  • mode: development marks a target as a personal, scratch deployment. Resources get a [dev username] prefix and schedules are paused by default, so your work doesn't affect anyone else.
  • mode: production disables those safety defaults. Combined with run_as, it lets you run the pipeline as a service principal rather than an individual's account, so runs don't break when someone leaves the team or changes roles. Databricks recommends a service principal for staging and production. service_principal_name takes the service principal's application ID, not its display name. You can retrieve the application ID from the service principal's page in your workspace admin settings.

For the full set of mode behaviors, see Declarative Automation Bundles deployment modes and Specify a run identity for a Declarative Automation Bundles workflow.

To promote, deploy the same bundle to each target in turn, verifying at each stage:

Bash
databricks bundle validate --target prod
databricks bundle deploy --target prod
databricks bundle run orders_pipeline --target prod

Rather than hardcoding catalog names or source paths per environment inside your transformation code, pass the values in from the target so the same source runs unmodified everywhere. How you set them depends on your source language. Pipeline parameters apply to SQL source code only. For Python source code, use the pipeline configuration field and read the values with spark.conf.get():

YAML
resources:
  pipelines:
    orders_pipeline:
      name: orders-pipeline
      # For SQL source code. Reference as ${source_catalog}.
      parameters:
        source_catalog: ${var.catalog}
        source_schema: raw
      # For Python source code. Read with spark.conf.get("source_catalog").
      configuration:
        source_catalog: ${var.catalog}
        source_schema: raw

For more about parameterizing pipeline code, see Use parameters with pipelines.

Set up CI/CD​

Because a converted pipeline is defined entirely as a bundle (YAML plus source files in Git), setting up continuous integration and continuous delivery (CI/CD) for it means running the bundle commands from a CI system such as GitHub Actions or Azure DevOps. On each pull request, a good baseline runs:

  1. pytest against your unit-testable transformation functions. See Unit testing for pipelines.
  2. databricks bundle validate --target <env> to catch configuration errors.
  3. Optionally, a databricks bundle run in a scratch target to exercise expectations against sample data.

The following GitHub Actions workflow deploys to staging on merge to main, using OpenID Connect (OIDC) federation instead of a stored token:

YAML
# .github/workflows/deploy.yml
name: Deploy pipeline bundle

on:
  push:
    branches: [main]

permissions:
  id-token: write
  contents: read

jobs:
  deploy-staging:
    runs-on: ubuntu-latest
    environment: staging
    env:
      DATABRICKS_AUTH_TYPE: github-oidc
      DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}
      DATABRICKS_CLIENT_ID: ${{ vars.DATABRICKS_CLIENT_ID }} # Service principal application ID
    steps:
      - uses: actions/checkout@v4

      - name: Install Databricks CLI
        uses: databricks/setup-cli@main

      - name: Validate bundle
        run: databricks bundle validate --target staging

      - name: Deploy bundle
        run: databricks bundle deploy --target staging

Gate the production deploy behind a manual approval (for example, a second job that requires a GitHub Environment approval, or a separate stage in Azure DevOps) so a person explicitly approves each promotion. The production job runs databricks bundle deploy --target prod using a service principal scoped to the production workspace. For more, see CI/CD on Databricks.

Troubleshooting​

Issue

Solution

“databricks.yml not found” error when running bundle generate

Currently, the bundle generate command doesn't create the bundle configuration file (databricks.yml) automatically. You must create the file using databricks bundle init or manually.

Existing pipeline settings don't match the values in the generated pipeline YAML configuration

The pipeline ID does not appear in the bundle configuration YML file. If you notice any other missing settings, you can manually apply them.

Issue

Solution

“databricks.yml not found” error when running bundle generate

Currently, the bundle generate command doesn't create the bundle configuration file (databricks.yml) automatically. You must create the file using databricks bundle init or manually.

Existing pipeline settings don't match the values in the generated pipeline YAML configuration

The pipeline ID does not appear in the bundle configuration YML file. If you notice any other missing settings, you can manually apply them.

Tips for success​

  • Always use version control. If you aren't using Databricks Git folders, store your project subdirectories and files in a Git or other version-controlled repository or file system.
  • Test your pipeline in a non-production environment (such as a “development” or “test” environment) before deploying it to a production environment. It's easy to introduce a misconfiguration by accident.

Additional resources​

For more information about using bundles to define and manage data processing, see:

On this page