Organize jobs and pipelines in workspace folders
This feature is in Beta. To use it, a workspace admin must turn on Jobs & Pipelines in folders from the Previews page. See Manage Databricks previews.
With Jobs & Pipelines in folders, you can store jobs and pipelines in workspace folders, next to the notebooks and files they run. Use folders to find related assets in one place, share a whole project by granting permissions on its folder, and restore deleted jobs and pipelines from Trash.
What changes when a job or pipeline is in a folder
A job or pipeline in a folder:
- Appears in the workspace browser, where you can open, rename, move, and favorite it.
- Inherits permissions from its folder, in addition to permissions granted on it directly.
- Moves to Trash when you delete it from the workspace browser.
- Still appears on the Jobs & Pipelines page.
Existing jobs and pipelines don't change. They stay outside folders, appear only on the Jobs & Pipelines page, and get permissions only from direct grants, until you move them into a folder.
Create a job or pipeline in a folder
Create a job or pipeline in a folder from the UI, the REST API, or the Databricks CLI. Declarative Automation Bundles and Terraform also support parent_path.
- UI
- REST API
- CLI
Jobs and pipelines created in the UI go into your home folder (/Workspace/Users/<your-username>/) by default. If you start from a folder in the workspace browser, they go into that folder.
To create a job or an extract, transform, and load (ETL) pipeline in your home folder:
- In your workspace, click
Jobs & Pipelines in the sidebar.
- Click Create, then click Job or ETL pipeline.
- Configure the job or pipeline. See Configure and edit Lakeflow Jobs or Configure pipelines.
To create jobs and pipelines outside folders instead, click your username in the top bar and turn off Settings > Developer > Editor settings > General > Create Jobs and Pipelines in folders. Existing jobs and pipelines don't move.
To create a job or pipeline in a specific folder:
- Click Workspace in the sidebar and open the folder.
- Click Create in the upper-right corner, then click Job or ETL pipeline.
- Configure the job or pipeline.
Set parent_path to the workspace path of the destination folder in a request to create a job or create a pipeline. If you omit parent_path, the job or pipeline is created outside folders, as before. Use Jobs API 2.1 and above.
The following request creates a job in the /Workspace/Projects/orders folder:
POST /api/2.2/jobs/create
{
"name": "orders-etl",
"parent_path": "/Workspace/Projects/orders"
}
The following request creates a pipeline in the same folder:
POST /api/2.0/pipelines
{
"name": "orders_pipeline",
"parent_path": "/Workspace/Projects/orders",
"catalog": "main",
"schema": "orders",
"serverless": true,
"libraries": [
{
"notebook": {
"path": "/Workspace/Projects/orders/ingest"
}
}
]
}
Set parent_path to the workspace path of the destination folder in the JSON request for databricks jobs create or databricks pipelines create. If you omit parent_path, the job or pipeline is created outside folders, as before. Use Databricks CLI version 1.21.0 and above. See jobs command group and pipelines command group.
The following command creates a job in the /Workspace/Projects/orders folder:
databricks jobs create --json '{
"name": "orders-etl",
"parent_path": "/Workspace/Projects/orders"
}'
The following command creates a pipeline in the same folder:
databricks pipelines create --json '{
"name": "orders_pipeline",
"parent_path": "/Workspace/Projects/orders",
"catalog": "main",
"schema": "orders",
"serverless": true,
"libraries": [
{
"notebook": {
"path": "/Workspace/Projects/orders/ingest"
}
}
]
}'
Pipeline source code folder
When you create a pipeline in the UI, Databricks also creates a folder for the pipeline's source code. The pipeline is created next to its source code folder, not inside it:
/Workspace/Projects/orders/
├── orders_pipeline <- the pipeline (settings, schedule, runs, permissions)
└── orders_pipeline_source/ <- the source code folder that the editor opens
├── bronze.py
└── silver.sql
The pipeline and its source code folder are separate workspace objects with their own permissions. See Share jobs and pipelines through folders.
Move a job or pipeline into a folder
To move a job or pipeline, you need CAN MANAGE on it.
You can move an existing job or pipeline into a folder to organize it alongside your other assets and inherit the folder's permissions. Existing UI and API workflows continue to work as before. The notebooks, files, and source code that the job or pipeline references don't move with it.
- UI
- REST API
To move a job or pipeline that isn't in a folder:
- In your workspace, click
Jobs & Pipelines in the sidebar.
- Open the job or pipeline that you want to move.
- In the details panel, click Select folder next to Location.
- In the dialog, select the folder that you want to move the job or pipeline to, and click Move.
To move a job or pipeline that's already in a folder, drag it to another folder in the workspace browser, or use Move as you would for other workspace objects. See Move an object.
To move a job or pipeline, set parent_path to the workspace path of the destination folder:
- Job: Include
parent_pathinnew_settingsin a request to partially update a job or update all job settings. - Pipeline: Include
parent_pathin a request to edit a pipeline. This request replaces all pipeline settings, so include the existing settings too.
The following request moves a job to the /Workspace/Projects/orders folder:
POST /api/2.2/jobs/update
{
"job_id": 11223344,
"new_settings": {
"parent_path": "/Workspace/Projects/orders"
}
}
Share jobs and pipelines through folders
A job or pipeline in a folder inherits permissions from that folder and its parent folders, in addition to any permissions granted on it directly. A user who has a permission on a folder has the matching permission level on every job and pipeline in it.
- To share all jobs and pipelines in a project, grant permissions on the project folder. See Manage access control lists with folders.
- To share a single job or pipeline, grant permissions on it directly. See Control access to a job or Configure pipeline permissions.
- To share a pipeline together with its source code, grant permissions on the folder that contains both, or on the pipeline and its source code folder separately. See Pipeline source code folder.
Jobs and pipelines in folders support the CAN EDIT permission. CAN EDIT lets a user view, run, edit the settings of, and rename a job or pipeline, but not change its run-as identity. For the complete list of abilities, see Job ACLs and Lakeflow pipelines ACLs.
Delete and restore a job or pipeline
A job or pipeline that you delete from the Jobs & Pipelines page, the Jobs API, the Pipelines API, or the Databricks CLI doesn't go to Trash and can't be restored.
To delete a job or pipeline and keep the option to restore it, delete it from the workspace browser. Right-click the job or pipeline, or click next to it, and select Move to Trash.
When you move a job or pipeline to Trash:
- Job: active runs are canceled and the schedule stops.
- Pipeline: active updates stop. The pipeline's tables are kept.
Items in Trash are permanently deleted after 30 days. To restore a job or pipeline, open the Trash folder, click next to it, and select Restore. See Restore an object.
Limitations
- Only standard jobs and ETL pipelines can be in folders.
- Jobs and pipelines in folders aren't supported in Git folders. To keep job and pipeline definitions in source control, use Declarative Automation Bundles.
- After you move a job or pipeline into a folder, you can't move it back outside folders.
- Standard resource limits for saved jobs and pipelines apply. Jobs and pipelines in Trash count toward these limits. To free capacity, permanently delete items from Trash.