Skip to main content

Orchestrate workloads with Lakeflow Jobs

Orchestrate your workloads with Lakeflow Jobs, the workflow automation for Databricks, so that you can run production workloads unattended and see what happened in each run. A job coordinates the tasks in a workflow, such as running a pipeline, a notebook, or a SQL query, in the order that you define. It runs them on a schedule or when an event occurs, such as new data arriving in cloud storage. Jobs can retry failed tasks, send notifications, and record every run.

The following video introduces orchestration with Lakeflow Jobs:

Jobs, tasks, and triggers​

These three terms describe how a job organizes and starts its work:

  • Job: The resource that coordinates, schedules, and runs your tasks. A job can include conditional logic, such as if-else branches and for each loops. A graph shows the tasks and their dependencies.
  • Task: A unit of work in a job, such as a notebook, a pipeline, a Python script, or a SQL query. A task can depend on other tasks, so that it runs only after they finish. See Types of tasks.
  • Trigger: The condition that starts a job run. A trigger can run a job on a schedule, when a table changes, when files arrive in cloud storage, or continuously. See Automate jobs with schedules and triggers.

Requirements​

You need the following before you create the job:

Create a job​

The following steps create a job that runs your pipeline, and then runs a notebook after the pipeline update completes.

Add a pipeline task​

Create the first task, which runs your pipeline:

  1. In your workspace, click Workflows icon. Jobs & Pipelines in the sidebar.
  2. Click Create, and then click Job.
  3. Replace the default job name with a name for your job.
  4. Configure the first task:
    • Task name: Enter a name, such as refresh-pipeline.
    • Type: Select Pipeline.
    • Pipeline: Select your pipeline.
  5. Click Save task.

Add a notebook task​

Next, create a notebook task that runs after the pipeline update completes:

  1. Click Plus icon. Add task > Notebook.
  2. Configure the task:
    • Task name: Enter a name, such as report.
    • Source: Select Workspace.
    • Path: Select your notebook.
    • Depends on: Confirm that Depends on shows the pipeline task.
  3. Click Save task.

Add a schedule​

Set a schedule so the job runs without a manual start:

  1. In the Job details pane, scroll down to the Schedules & Triggers section, and then click Add trigger.
  2. In Trigger type, select Scheduled, and then set the schedule.
  3. Click Save.

To run the job when new data arrives instead, select the File arrival or Table update trigger type.

Add notifications​

Pick where notifications go, and which run events trigger them:

  1. In the Job details pane, in the Job notifications section, click Edit notifications.
  2. Click Add notification. In Destination, select Email address or a system destination. An administrator must configure system destinations before you can select one.
  3. Select the events to notify about, such as Failure.
  4. Click Save.

Run the job​

Click Run Now Button to run the job immediately. On the Runs tab, click the run's start time to view the output and status of each task.

If your workspace is enabled for Unity Catalog and serverless compute for jobs, serverless compute is the default for all tasks that support it.

For a quickstart with sample data, see Create your first workflow with Lakeflow Jobs.

Choose the UI or Declarative Automation Bundles​

Build in the UI when you're setting up your first job and want to see it run quickly, when you need a one-time or exploratory workflow, or when you want to prototype a task graph before you write it as code. Start here before you move to Declarative Automation Bundles for production.

Switch to Declarative Automation Bundles when you need to version control the job and deploy it across environments, or when you want to define the pipelines, schemas, and compute alongside the job.

Define jobs as code​

For production workflows, define jobs in Declarative Automation Bundles together with the pipelines and other resources that they run, so that you version control every change and deploy the same job to development, staging, and production. See Operations and CI/CD.

Troubleshooting​

Use this entry if a task can't find its pipeline or notebook.

The pipeline or notebook doesn't appear when you configure the task

Confirm that you have the CAN RUN, CAN MANAGE, or IS OWNER permission on the pipeline, and that you can open the notebook path in the workspace. See Manage identities, permissions, and privileges for pipelines and Notebook task for jobs.

Next steps​