Migrate existing resources to a bundle

When building your bundle, you may want to include Databricks resources that already exist and are fully configured in the remote workspace. You can use the Databricks CLI bundle generate command to quickly autogenerate configuration in your bundle for existing apps, dashboards, jobs, and pipelines. See Generate a bundle configuration file. Configuration that you can copy and manually paste into bundle resource configuration files is available in the Databricks UI for some resources, such as jobs and pipelines.

After you have generated configuration for a resource in your bundle and deployed the bundle, use the bundle deployment bind command to bind a resource in your bundle to the corresponding resource in the workspace. See Bind a bundle resource.

This page provides simple examples that use the Databricks CLI or UI to generate or retrieve bundle resource configuration.

For details about resource definitions in bundles, see Databricks Asset Bundles resources.

Generate an existing job or pipeline configuration using the Databricks CLI

To programmatically generate bundle configuration for an existing job or pipeline:

1. Retrieve the ID of the existing job or pipeline from the Job details or Pipeline details side panel for the job or pipeline in the UI. Alternatively, use the Databricks CLI databricks jobs list or databricks pipelines list-pipelines command.

2. Run the bundle generate job or bundle generate pipeline Databricks CLI command, setting the job or pipeline ID:

   Bash

   databricks bundle generate job --existing-job-id 6565621249
   Bash

   databricks bundle generate pipeline --existing-pipeline-id 6565621249

   This command creates a bundle configuration file for the resource in the bundle's resources folder and downloads any referenced artifacts to the src folder.

You can also generate configuration for an existing dashboard. See Generate dashboard configuration.

Retrieve an existing job definition using the UI

To retrieve the YAML representation of an existing job definition from the Databricks workspace UI:

1. In your Databricks workspace's sidebar, click Workflows.

2. On the Jobs tab, click your job's Name link.

3. Next to the Run now button, click the kebab, and then click Edit as YAML.

4. Copy the YAML and add it to your bundle's databricks.yml file, or create a configuration file for your job in the resources directory of your bundle project and reference it from your databricks.yml file. See resources.

5. Download and add any Python files and notebooks that are referenced in the existing job to the bundle's project source. Typically bundle artifacts are located in the src directory in a bundle.

   tip

   You can export an existing notebook from a Databricks workspace into the .ipynb format by clicking File > Export > IPython Notebook from the Databricks notebook user interface.

   After you add your notebooks, Python files, and other artifacts to the bundle, change the references to these files in your job definition to their local location. For example, if you had a hello_job.job.yml configuration file in the resources folder of your bundle, and you downloaded a notebook named hello.ipynb to the src folder of your bundle, the contents of the hello_job.job.yml file would be the following:

   YAML

   resources:
     jobs:
       hello_job:
         name: hello_job
         tasks:
           - task_key: hello_task
             notebook_task:
               notebook_path: ../src/hello.ipynb

For more information about viewing jobs as code in the UI, see View jobs as code.

Retrieve an existing pipeline definition using the UI

To retrieve the YAML representation of an existing pipeline definition from the Databricks workspace UI:

1. In your Databricks workspace's sidebar, click Workflows.

2. On the DLT tab, click your pipeline's Name link.

3. Next to the Development button, click the kebab, and then click View settings YAML.

4. Copy the pipeline definition's YAML in the Pipeline settings YAML dialog to your local clipboard by clicking the copy icon.

5. Add the YAML that you copied to your bundle's databricks.yml file, or create a configuration file for your pipeline in the resources folder of your bundle project and reference it from your databricks.yml file. See resources.

6. Download and add any Python files and notebooks that are referenced to the bundle's project source. Typically bundle artifacts are located in the src directory in a bundle.

   tip

   You can export an existing notebook from a Databricks workspace into the .ipynb format by clicking File > Export > IPython Notebook from the Databricks notebook user interface.

   After you add your notebooks, Python files, and other artifacts to the bundle, make sure that your pipeline definition properly references them. For example, for a notebook named hello.ipynb that is in the src/ directory of the bundle:

   YAML

   resources:
     pipelines:
       hello_pipeline:
         name: hello_pipeline
         libraries:
           - notebook:
               path: ../src/hello.ipynb

Bind a resource to its remote counterpart

Typically after you have added a resource to your bundle, you will want to ensure the resource in your bundle and the existing resource in the workspace stay in sync. The bundle deployment bind command allows you to link them. If you bind a resource, the linked Databricks resource in the workspace is updated based on the configuration defined in the bundle on the next bundle deploy.

For more information about bundle deployment bind and details about resource support, see Bind a bundle resource.

The following command binds the resource hello_job to its remote counterpart in the workspace. It prompts with a confirmation to ensure that updates to the job configuration in the bundle should be applied to the corresponding remote job when the bundle is next deployed.

Bash

databricks bundle deployment bind hello_job 6565621249

To remove the link between a bundle resource and its counterpart in the workspace, use bundle deployment unbind. See Unbind a bundle resource.

Bash

databricks bundle deployment unbind 6565621249