Automating jobs with schedules and triggers

In Jobs, it is possible to configure jobs to automatically trigger in any of the following situations:

On a time-based schedule
On the arrival of files to a Unity Catalog storage location
Continuously

You can also trigger job runs manually or through external orchestration tools.

Job schedules and triggers

Trigger type	Behavior
Scheduled	Triggers a job run based on a time-based schedule. See Run jobs on a schedule.
File arrival	Triggers a job run when new files arrive in a monitored Unity Catalog storage location. See Trigger jobs when new files arrive.
Continuous	To keep the job always running, trigger another job run whenever a job run completes or fails. See Run jobs continuously.
None (manual)	Runs are triggered manually with the Run now button or programmatically using other orchestration tools. See Trigger a single job run

By default, only a single run of a job can be active at a time. However, it is possible to increase this limit in the Advanced settings. Runs are skipped when they exceed the configured max concurrency for a job. See Configure maximum concurrent runs.

Configure a trigger on a job

Open the job that you want to configure a trigger on.
In the Job details pane, scroll down to the Schedules & Triggers section, and then click Add trigger.
In Schedules & Triggers, select the type of trigger you want to configure: Scheduled, File arrival, or Continuous.
For a file arrival trigger, select File arrival and enter the path in Storage location. Optionally, you can define a minimum time between triggers, or a delay between file arrival and the trigger starting the job. By default, there is a 60-second minimum time between triggers.
Click Save. After you save your trigger, your job will start only when a new file arrives in the configured location.

Note

If one or more tasks in a job with multiple tasks are unsuccessful, you can re-run the subset of unsuccessful tasks. See Re-run failed and skipped tasks.

Pause and resume job triggers

You can pause and resume your jobs in the Job details pane for your job under Schedules & Triggers. The Pause and Resume buttons appear only for jobs that have a trigger configured.

To pause any active job trigger, click Pause. When you pause a trigger, any currently active runs continue, but the trigger no longer starts new runs.

To resume the trigger, click Resume. When you resume a trigger, the configured behavior resumes on the same schedule previously configured.

When creating or editing a trigger, you can also control these settings in the Schedules & Triggers dialog. Toggle between the Active and Paused to control the Trigger Status.

Note

If a run is active when a continuous trigger is resumed, the job scheduler waits until that run is complete to trigger a new run.