Run jobs on a schedule or continuously
You can run your Databricks job periodically with the Scheduled trigger type or ensure there’s always an active run of the job with the Continuous trigger type.
You can use a schedule to automatically run your Databricks job at specified times and periods. You can define a schedule to run your job on minute, hourly, daily, weekly, or monthly periods and at specified times. You can also specify a time zone for your schedule, and you can pause a scheduled job at any time.
When you run your job with the continuous trigger, Databricks Jobs ensures there is always one active run of the job. A new run of the job starts after the previous run completes successfully or with a failed status, or if there is no instance of the job currently running.
Add a job schedule
To define a schedule for the job:
In the sidebar, click Workflows.
In the Name column on the Jobs tab, click the job name.
Click Add trigger in the Job details panel and select Scheduled in Trigger type.
Specify the period, starting time, and time zone. Optionally select the Show Cron Syntax checkbox to display and edit the schedule in Quartz Cron Syntax.
Click Save.
You can also schedule a notebook job directly in the notebook UI.
Note
Databricks enforces a minimum interval of 10 seconds between subsequent runs triggered by the schedule of a job regardless of the seconds configuration in the cron expression.
You can choose a time zone that observes daylight saving time or UTC. If you select a time zone that observes daylight saving time, an hourly job will be skipped or might appear to be delayed by an hour or two when daylight saving time begins or ends. To run at every hour (absolute time), choose UTC.
The job scheduler is not intended for low latency jobs. Due to network or cloud issues, job runs might occasionally be delayed up to several minutes. In these situations, scheduled jobs run immediately upon service availability.
Pause and resume a job schedule
To pause a job, click Pause in the Job details panel.
To resume a paused job schedule, click Resume.
Run a continuous job
In the sidebar, click Workflows.
In the Name column on the Jobs tab, click the job name.
Click Add trigger in the Job details panel, click Add trigger in the Job details panel, select Continuous in Trigger type, and click Save.
To stop a continuous job, click next to Run Now and click Stop.
Note
To prevent unnecessary resource usage and reduce cost, Databricks automatically pauses a continuous job if there are more than five consecutive failures within a 24-hour period.
There can be only one running instance of a continuous job.
There is a small delay between a run finishing and a new run starting. This delay should be less than 60 seconds.
You cannot use retry policies or task dependencies with a continuous job.
Selecting Run now on a continuous job that is paused triggers a new job run. If the job is unpaused, an exception is thrown.
To have your continuous job pick up a new job configuration, cancel the existing run and then a new run automatically starts. You can also click Restart run to restart the job run with the updated configuration.