Skip to main content

Organize jobs and pipelines in workspace folders

Beta

This feature is in Beta. To use it, a workspace admin must turn on Jobs & Pipelines in folders from the Previews page. See Manage Databricks previews.

With Jobs & Pipelines in folders, you can store jobs and pipelines in workspace folders, next to the notebooks and files they run. Use folders to find related assets in one place, share a whole project by granting permissions on its folder, and restore deleted jobs and pipelines from Trash.

What changes when a job or pipeline is in a folder​

A job or pipeline in a folder:

  • Appears in the workspace browser, where you can open, rename, move, and favorite it.
  • Inherits permissions from its folder, in addition to permissions granted on it directly.
  • Moves to Trash when you delete it from the workspace browser.
  • Still appears on the Jobs & Pipelines page.

Existing jobs and pipelines don't change. They stay outside folders, appear only on the Jobs & Pipelines page, and get permissions only from direct grants, until you move them into a folder.

Create a job or pipeline in a folder​

Create a job or pipeline in a folder from the UI, the REST API, or the Databricks CLI. Declarative Automation Bundles and Terraform also support parent_path.

Jobs and pipelines created in the UI go into your home folder (/Workspace/Users/<your-username>/) by default. If you start from a folder in the workspace browser, they go into that folder.

To create a job or an extract, transform, and load (ETL) pipeline in your home folder:

  1. In your workspace, click Workflows icon. Jobs & Pipelines in the sidebar.
  2. Click Create, then click Job or ETL pipeline.
  3. Configure the job or pipeline. See Configure and edit Lakeflow Jobs or Configure pipelines.

To create jobs and pipelines outside folders instead, click your username in the top bar and turn off Settings > Developer > Editor settings > General > Create Jobs and Pipelines in folders. Existing jobs and pipelines don't move.

To create a job or pipeline in a specific folder:

  1. Click Workspace in the sidebar and open the folder.
  2. Click Create in the upper-right corner, then click Job or ETL pipeline.
  3. Configure the job or pipeline.

Pipeline source code folder​

When you create a pipeline in the UI, Databricks also creates a folder for the pipeline's source code. The pipeline is created next to its source code folder, not inside it:

Text
/Workspace/Projects/orders/
├── orders_pipeline <- the pipeline (settings, schedule, runs, permissions)
└── orders_pipeline_source/ <- the source code folder that the editor opens
├── bronze.py
└── silver.sql

The pipeline and its source code folder are separate workspace objects with their own permissions. See Share jobs and pipelines through folders.

Move a job or pipeline into a folder​

note

To move a job or pipeline, you need CAN MANAGE on it.

You can move an existing job or pipeline into a folder to organize it alongside your other assets and inherit the folder's permissions. Existing UI and API workflows continue to work as before. The notebooks, files, and source code that the job or pipeline references don't move with it.

To move a job or pipeline that isn't in a folder:

  1. In your workspace, click Workflows icon. Jobs & Pipelines in the sidebar.
  2. Open the job or pipeline that you want to move.
  3. In the details panel, click Select folder next to Location.
  4. In the dialog, select the folder that you want to move the job or pipeline to, and click Move.

To move a job or pipeline that's already in a folder, drag it to another folder in the workspace browser, or use Move as you would for other workspace objects. See Move an object.

Share jobs and pipelines through folders​

A job or pipeline in a folder inherits permissions from that folder and its parent folders, in addition to any permissions granted on it directly. A user who has a permission on a folder has the matching permission level on every job and pipeline in it.

Jobs and pipelines in folders support the CAN EDIT permission. CAN EDIT lets a user view, run, edit the settings of, and rename a job or pipeline, but not change its run-as identity. For the complete list of abilities, see Job ACLs and Lakeflow pipelines ACLs.

Delete and restore a job or pipeline​

warning

A job or pipeline that you delete from the Jobs & Pipelines page, the Jobs API, the Pipelines API, or the Databricks CLI doesn't go to Trash and can't be restored.

To delete a job or pipeline and keep the option to restore it, delete it from the workspace browser. Right-click the job or pipeline, or click Kebab menu icon. next to it, and select Move to Trash.

When you move a job or pipeline to Trash:

  • Job: active runs are canceled and the schedule stops.
  • Pipeline: active updates stop. The pipeline's tables are kept.

Items in Trash are permanently deleted after 30 days. To restore a job or pipeline, open the Trash folder, click Kebab menu icon. next to it, and select Restore. See Restore an object.

Limitations​

  • Only standard jobs and ETL pipelines can be in folders.
  • Jobs and pipelines in folders aren't supported in Git folders. To keep job and pipeline definitions in source control, use Declarative Automation Bundles.
  • After you move a job or pipeline into a folder, you can't move it back outside folders.
  • Standard resource limits for saved jobs and pipelines apply. Jobs and pipelines in Trash count toward these limits. To free capacity, permanently delete items from Trash.