Pipeline task for jobs

Lakeflow Jobs provide a procedural approach to defining relationships between tasks. Lakeflow Declarative Pipelines provide a declarative approach to defining relationships between datasets and transformations. This page describes how you can schedule triggered Lakeflow Declarative Pipelines to run as a task in a job, using the Jobs UI, the Lakeflow Declarative Pipelines UI, or SQL.

note

A triggered pipeline is a pipeline that does not run continuously, but must be triggered to start. A pipeline task can be the triggering mechanism for a triggered pipeline. Continuous pipelines do not need to be triggered, so triggering them through a task would be redundant. To learn more about triggered and continuous pipelines, see Triggered vs. continuous pipeline mode.

Configure a pipeline task with the Jobs UI

Lakeflow Declarative Pipelines manage all configurations for source code and compute in the pipeline definition.

To add a pipeline to a job, complete the following steps:

1. Create and name a new task and select pipeline for the Type.
2. In the Pipeline drop-down menu, select an existing pipeline. The pipeline must be a triggered pipeline. Continuous pipelines are not supported as a job task.
3. You can optionally trigger a full refresh on the pipeline.

note

You can also create a new ingestion pipeline when creating a task by choosing + New ingestion pipeline from the Add Task pane or the tasks Type drop-down.

Schedule a pipeline with the pipeline UI

Adding a schedule to a pipeline creates a job with a single pipeline task. You can only configure time-based schedule triggers using this UI. For more advanced triggering options, see Configure a pipeline task with the Jobs UI.

Configure a pipeline task in a scheduled job using the pipeline UI by completing the following steps:

1. In your workspace, click Jobs & Pipelines in the sidebar.
2. Click on the pipeline Name. The pipeline UI appears.
3. Click Schedule.
   • If no schedule exists for the pipeline, the New schedule dialog appears.
   • If one or more schedules already exist, click Add schedule.
4. Enter a unique name for the job in the Job name field.
5. (Optional) Update the schedule frequency.
   • Select Advanced for more verbose options including cron syntax.
6. (Optional) Under More options, configure one or more email addresses to receive alerts on pipeline start, success, or failure.
7. Click Create.

note

If the pipeline is included in one or more scheduled jobs, the Schedule button shows the number of existing schedules, for example, Schedule (5).