Orchestrate workloads with Lakeflow Jobs
Orchestrate your workloads with Lakeflow Jobs, the workflow automation for Databricks, so that you can run production workloads unattended and see what happened in each run. A job coordinates the tasks in a workflow, such as running a pipeline, a notebook, or a SQL query, in the order that you define. It runs them on a schedule or when an event occurs, such as new data arriving in cloud storage. Jobs can retry failed tasks, send notifications, and record every run.
The following video introduces orchestration with Lakeflow Jobs:
Jobs, tasks, and triggers
These three terms describe how a job organizes and starts its work:
- Job: The resource that coordinates, schedules, and runs your tasks. A job can include conditional logic, such as if-else branches and for each loops. A graph shows the tasks and their dependencies.
- Task: A unit of work in a job, such as a notebook, a pipeline, a Python script, or a SQL query. A task can depend on other tasks, so that it runs only after they finish. See Types of tasks.
- Trigger: The condition that starts a job run. A trigger can run a job on a schedule, when a table changes, when files arrive in cloud storage, or continuously. See Automate jobs with schedules and triggers.
Requirements
You need the following before you create the job:
- The pipeline from Build a pipeline with Genie Code.
- The CAN RUN, CAN MANAGE, or IS OWNER permission on the pipeline. See Manage identities, permissions, and privileges for pipelines.
- A notebook to run after the pipeline update, in a location that you can access. See Notebook task for jobs.
Create a job
The following steps create a job that runs your pipeline, and then runs a notebook after the pipeline update completes.
Add a pipeline task
Create the first task, which runs your pipeline:
- In your workspace, click
Jobs & Pipelines in the sidebar.
- Click Create, and then click Job.
- Replace the default job name with a name for your job.
- Configure the first task:
- Task name: Enter a name, such as
refresh-pipeline. - Type: Select Pipeline.
- Pipeline: Select your pipeline.
- Task name: Enter a name, such as
- Click Save task.
Add a notebook task
Next, create a notebook task that runs after the pipeline update completes:
- Click
Add task > Notebook.
- Configure the task:
- Task name: Enter a name, such as
report. - Source: Select Workspace.
- Path: Select your notebook.
- Depends on: Confirm that Depends on shows the pipeline task.
- Task name: Enter a name, such as
- Click Save task.
Add a schedule
Set a schedule so the job runs without a manual start:
- In the Job details pane, scroll down to the Schedules & Triggers section, and then click Add trigger.
- In Trigger type, select Scheduled, and then set the schedule.
- Click Save.
To run the job when new data arrives instead, select the File arrival or Table update trigger type.
Add notifications
Pick where notifications go, and which run events trigger them:
- In the Job details pane, in the Job notifications section, click Edit notifications.
- Click Add notification. In Destination, select Email address or a system destination. An administrator must configure system destinations before you can select one.
- Select the events to notify about, such as Failure.
- Click Save.
Run the job
Click to run the job immediately. On the Runs tab, click the run's start time to view the output and status of each task.
If your workspace is enabled for Unity Catalog and serverless compute for jobs, serverless compute is the default for all tasks that support it.
For a quickstart with sample data, see Create your first workflow with Lakeflow Jobs.
Choose the UI or Declarative Automation Bundles
Build in the UI when you're setting up your first job and want to see it run quickly, when you need a one-time or exploratory workflow, or when you want to prototype a task graph before you write it as code. Start here before you move to Declarative Automation Bundles for production.
Switch to Declarative Automation Bundles when you need to version control the job and deploy it across environments, or when you want to define the pipelines, schemas, and compute alongside the job.
Define jobs as code
For production workflows, define jobs in Declarative Automation Bundles together with the pipelines and other resources that they run, so that you version control every change and deploy the same job to development, staging, and production. See Operations and CI/CD.
Troubleshooting
Use this entry if a task can't find its pipeline or notebook.
The pipeline or notebook doesn't appear when you configure the task
Confirm that you have the CAN RUN, CAN MANAGE, or IS OWNER permission on the pipeline, and that you can open the notebook path in the workspace. See Manage identities, permissions, and privileges for pipelines and Notebook task for jobs.
Next steps
-
- Answer business questions with Genie Ontology
- Turn your silver tables into governed metrics and a Genie Agent for business users
-
- Lakeflow Jobs
- Learn about job configuration, task types, triggers, and monitoring