Automating jobs with schedules and triggers
In Lakeflow Jobs, it is possible to configure jobs to automatically trigger in any of the following situations:
- On a time-based schedule
- On the arrival of files to a Unity Catalog storage location
- Continuously
You can also trigger job runs manually or through external orchestration tools.
Job schedules and triggers
Trigger type | Behavior |
|---|---|
Scheduled | Triggers a job run based on a time-based schedule. See Run jobs on a schedule. |
Table update | Triggers are job run when source tables are updated. See Trigger jobs when source tables are updated. |
File arrival | Triggers a job run when new files arrive in a monitored Unity Catalog storage location. See Trigger jobs when new files arrive. |
Continuous | To keep the job always running, trigger another job run whenever a job run completes or fails. See Run jobs continuously. |
None (manual) | Runs are triggered manually with the Run now button or programmatically using other orchestration tools. See Trigger a single job run |
By default, only a single run of a job can be active at a time. However, it is possible to increase this limit in the Advanced settings. Runs are skipped when they exceed the configured max concurrency for a job. See Configure maximum concurrent runs.
Configure a trigger on a job
-
Open the job that you want to configure a trigger on.
-
In the Job details pane, scroll down to the Schedules & Triggers section, and then click Add trigger.
-
In Schedules & Triggers, select the type of trigger you want to configure: Scheduled, Table update, File arrival, or Continuous.
Based on the type of trigger, other options are available to configure, as well.
-
Click Save. After you save your trigger, your job starts only when a new file arrives in the configured location.
If one or more tasks in a job with multiple tasks are unsuccessful, you can re-run the subset of unsuccessful tasks. See Re-run failed and skipped tasks.
Pause and resume job triggers
You can pause and resume your jobs in the Job details pane for your job under Schedules & Triggers. The Pause and Resume buttons appear only for jobs that have a trigger configured.
To pause any active job trigger, click Pause. When you pause a trigger, any currently active runs continue, but the trigger no longer starts new runs.
To resume the trigger, click Resume. When you resume a trigger, the configured behavior resumes on the same schedule previously configured.
When creating or editing a trigger, you can also control these settings in the Schedules & Triggers dialog. Toggle between the Active and Paused to control the Trigger Status.
If a run is active when a continuous trigger is resumed, the job scheduler waits until that run is complete to trigger a new run.