Automate jobs with schedules and triggers

Configure Lakeflow Jobs to trigger automatically in any of the following situations:

• On a time-based schedule
• On the update of source tables
• On the arrival of files to a Unity Catalog storage location
• On the creation or update of a Unity Catalog model
• Continuously

You can also trigger job runs manually or through external orchestration tools.

Job schedules and triggers

Trigger type	Behavior
Scheduled	Triggers a job run based on a time-based schedule. See Run jobs on a schedule.
Table update	Triggers a job run when source tables are updated. See Trigger jobs when source tables are updated.
File arrival	Triggers a job run when new files arrive in a monitored Unity Catalog storage location. See Trigger jobs when new files arrive.
Model update	Triggers a job run when a Unity Catalog model is created, a model version is ready, or a model alias is set (Beta). See Trigger jobs when models are updated.
Continuous	To keep the job always running, trigger another job run whenever a job run completes or fails. See Run jobs continuously.
None (manual)	Runs are triggered manually with the Run now button or programmatically using other orchestration tools. See Trigger a single job run

Trigger type	Behavior
Scheduled	Triggers a job run based on a time-based schedule. See Run jobs on a schedule.
Table update	Triggers a job run when source tables are updated. See Trigger jobs when source tables are updated.
File arrival	Triggers a job run when new files arrive in a monitored Unity Catalog storage location. See Trigger jobs when new files arrive.
Model update	Triggers a job run when a Unity Catalog model is created, a model version is ready, or a model alias is set (Beta). See Trigger jobs when models are updated.
Continuous	To keep the job always running, trigger another job run whenever a job run completes or fails. See Run jobs continuously.
None (manual)	Runs are triggered manually with the Run now button or programmatically using other orchestration tools. See Trigger a single job run

By default, only a single run of a job can be active at a time. However, it is possible to increase this limit in the Advanced settings. Runs are skipped when they exceed the configured max concurrency for a job. See Configure maximum concurrent runs.

Configure a trigger on a job

1. Open the job that you want to configure a trigger on.

2. In the Job details pane, scroll down to the Schedules & Triggers section, and then click Add trigger.

3. In Schedules & Triggers, select the type of trigger you want to configure: Scheduled, Table update, File arrival, Model update, or Continuous.

   Based on the type of trigger, other options are available to configure, as well.

4. Click Save. After you save, the job runs automatically based on the configured trigger.

note

If one or more tasks in a job with multiple tasks are unsuccessful, you can re-run the subset of unsuccessful tasks. See Re-run failed and skipped tasks.

Add multiple triggers to a job

Beta

Support for multiple triggers in Lakeflow Jobs is in Beta. Workspace admins can control access to the feature from the workspace Previews page using the Multiple Triggers setting. See Manage Databricks previews.

During the Multiple Triggers Beta, a job can have up to five triggers.

You can add multiple triggers of the same or different types to a job. When any active trigger evaluates to true, it starts a run. For example, you can use a table update trigger to start a job run when data is ready, and add an additional scheduled trigger as a backup. You can also use multiple file arrival triggers to monitor different storage locations.

Each trigger can be paused and resumed independently. A continuous trigger cannot be combined with any other trigger because the job already runs continuously.

To add multiple triggers, follow the steps in Configure a trigger on a job for each trigger. Click Add trigger in the Schedules & Triggers section to add another trigger.

To configure multiple triggers with Jobs API 2.1 or later, use the triggers array in JobSettings. Don't set triggers in the same request as the legacy schedule, trigger, or continuous fields. If you update triggers, it replaces the entire array. See the Jobs API and the Bundles job.triggers reference.

Manage existing triggers

To edit, pause, resume, or remove a trigger, use the Schedules & Triggers section of the Job details pane. Each trigger has independent controls. These controls appear only for jobs that have a trigger configured.

• To modify the trigger, click Edit trigger.
• To pause an active trigger, click Pause. Any currently active runs continue, but the trigger no longer starts new runs. To resume the trigger, click Resume. The configured behavior resumes on the same schedule previously configured.
• To remove the trigger from the job, click Delete.

When creating or editing a trigger, you can also toggle between Active and Paused in the Schedules & Triggers dialog to control the Trigger Status.

note

If a run is active when a continuous trigger is resumed, the job scheduler waits until that run is complete to trigger a new run.

Limitations

The following limits apply to job triggers:

• A workspace can have up to 12,000 active triggers across all jobs and trigger types. Paused triggers don't count toward this limit. This limit applies whether or not the Multiple Triggers preview is enabled.