Pipeline task for jobs

Lakeflow Jobs provides a procedural approach to defining relationships between tasks. Lakeflow pipelines provide a declarative approach to defining relationships between datasets and transformations. Schedule a pipeline to run as a task in a job, using the Jobs UI, the Lakeflow pipelines UI, or SQL.

A pipeline task runs its pipeline in one of two ways, depending on the job's schedule:

• In a triggered or scheduled job, the pipeline task starts a single update and stops when the update completes.
• In a continuous job, the pipeline task runs the pipeline continuously. The job's schedule determines the execution mode, so the pipeline runs continuously even if its own Pipeline mode setting is triggered. See Run a pipeline continuously with a continuous job.

To learn more about triggered and continuous pipelines, see Triggered vs. continuous pipeline mode.

Configure a pipeline task with the Jobs UI

Lakeflow pipelines manage all configurations for source code and compute in the pipeline definition.

To add a pipeline to a job, complete the following steps:

1. Create and name a new task and select pipeline for the Type.
2. In the Pipeline drop-down menu, select an existing pipeline.
3. You can optionally trigger a full refresh on the pipeline.
4. You can optionally set parameter overrides in the Parameters field. See Parameters.
5. (Optional) To configure retries, run-duration or streaming-backlog thresholds, or notifications, see Advanced task settings.

note

You can also create a new ingestion pipeline when creating a task by choosing New ingestion pipeline from the Add task pane or the tasks Type drop-down.

To edit, clone, disable, or delete this task, see Configure and edit tasks in Lakeflow Jobs.

Run a pipeline continuously with a continuous job

When a continuous job contains a pipeline task, the job runs the pipeline continuously. You don't need to set the pipeline's built-in Pipeline mode to continuous: the job's schedule determines the execution mode and takes precedence over the Pipeline mode setting.

This applies only to Lakeflow pipelines. Standalone materialized views and streaming tables always run in triggered mode.

A continuous pipeline wrapped in a continuous job can use serverless performance modes, such as Standard mode, which the pipeline's built-in continuous mode doesn't support.

Databricks recommends running continuous pipelines with a continuous job rather than the pipeline's built-in continuous setting. To avoid unexpected behavior, set the pipeline's Pipeline mode to triggered (the default) when you wrap it in a continuous job. For more information, see Run a continuous pipeline with a continuous job.

You can create the continuous job in the Jobs UI, directly from the pipeline page, or with Declarative Automation Bundles. See Schedule a pipeline with the pipeline UI and Define a continuous pipeline job in Declarative Automation Bundles.

Define a continuous pipeline job in Declarative Automation Bundles

The following Declarative Automation Bundles example defines a continuous job that runs a pipeline as a task. Setting continuous.pause_status to UNPAUSED runs the pipeline continuously, and performance_target: STANDARD runs the job in Standard performance mode.

YAML

# resources/continuous_job.yml
resources:
  jobs:
    continuous_pipeline_job:
      name: continuous_pipeline_job
      performance_target: STANDARD
      continuous:
        pause_status: UNPAUSED
      email_notifications:
        on_failure:
          - your_email@example.com
      tasks:
        - task_key: refresh_pipeline
          pipeline_task:
            pipeline_id: ${resources.pipelines.example_pipeline.id}

To migrate an existing continuous pipeline to a continuous job, remove the continuous field from the pipeline definition. The job then configures continuous execution.

For the equivalent Jobs API payload, see the jobs/create reference.

Database Table Sync pipeline

The Database Table Sync pipeline task is a pipeline task that runs the pipeline maintaining a Lakebase synced table. Use it to refresh a synced table on a schedule, or when the source Unity Catalog table changes, so that operational applications read current data from Lakebase Postgres.

In the task Type drop-down, Database Table Sync pipeline appears under Ingestion and Transformation. Selecting it configures a pipeline task. In the Pipeline field, choose the pipeline associated with the synced table you want to refresh.

To configure the task for your synced table, see Database Table Sync pipeline task.

Ingestion pipeline

The Ingestion pipeline task is a pipeline task that runs an ingestion pipeline. In the task Type drop-down, choosing Ingestion pipeline starts the Add data wizard, which creates a pipeline task for an ingestion pipeline.

The wizard's first page asks for the data source. The following pages depend on the source you select. For example, if you select MySQL, see Option 1: Databricks UI for the steps. For the available sources, see Managed connectors in Lakeflow Connect.

Parameters

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

You can access job or task parameters in your pipeline task using dynamic value references. You can override parameters by adding Key-Value pairs in the Parameters of your task configuration.

For details on how to access the parameter values from within your pipeline, see Access parameter values from a task.

Concurrency limits with pipeline tasks

A pipeline can only run one update at a time. Jobs that contain a pipeline task are subject to the following concurrency caps:

• A job with max_concurrent_runs > 1 that contains a pipeline task is capped to a single concurrent run. The job UI shows a notice when this cap is applied.
• A pipeline task wrapped in a for-each task is capped to one concurrent iteration, regardless of the loop's configured concurrency.

Plan around these caps when designing parameterized pipelines that you intend to run with many parameter combinations or in tight schedules.

Schedule a pipeline with the pipeline UI

Adding a schedule to a pipeline creates a job with a single pipeline task. For more advanced triggering options, see Configure a pipeline task with the Jobs UI.

Configure a pipeline task in a scheduled job using the pipeline UI by completing the following steps:

1. In your workspace, click Jobs & Pipelines in the sidebar.
2. Click on the pipeline Name. The pipeline UI appears.
3. Click Schedule.
   • If no schedule exists for the pipeline, the New schedule dialog appears.
   • If one or more schedules already exist, click Add schedule.
4. In the Trigger type drop-down list, select a trigger type:
   • Scheduled creates a time-based schedule. Select Advanced for more options, including cron syntax.
   • Continuous creates a continuous job that runs the pipeline continuously. See Run a pipeline continuously with a continuous job.
5. Enter a unique name for the job in the Job name field.
6. (Optional) To run the pipeline in Standard performance mode, clear the Performance optimized checkbox. See Select a performance mode.
7. (Optional) Under More options, configure one or more email addresses to receive alerts on pipeline start, success, or failure.
8. Click Create.

For a continuous schedule, Databricks starts the run automatically. To stop it, click Stop on the pipeline page or pause the schedule. Both actions also cancel the active update.

note

If the pipeline is included in one or more scheduled jobs, the Schedule button shows the number of existing schedules, for example, Schedule (5).

Additional resources

• Pipeline task: Define a pipeline task as code with Declarative Automation Bundles.
• Triggered vs. continuous pipeline mode: Learn about pipeline trigger modes.
• Configure task parameters: Pass parameters to the task.