Run jobs continuously

Use continuous mode to schedule workloads to run continuously. Databricks recommends using continuous mode for always-on streaming workloads.

Continuous mode replaces legacy recommendations for Structured Streaming workloads to configure jobs with an unlimited retry policy and a maximum of one concurrent run.

important

Serverless compute for jobs does not support continuous mode.

Configure job to run in continuous mode

To configure a job to run in continuous mode, do the following:

1. In your Databricks workspace's sidebar, click Jobs & Pipelines.
2. Optionally, select the Jobs and Owned by me filters.
3. Click your job's Name link.
4. Click Add trigger in the Job details panel, select Continuous in Trigger type, and click Save.

To stop a continuous job, click the Pause button. Click Resume to restart the job in continuous mode.

note

• There can be only one running instance of a continuous job.
• A delay exists between a run finishing and a new run starting. This delay should be less than 60 seconds.
• You cannot use task dependencies with a continuous job.
• You cannot use retry policies in a continuous job. Instead, continuous jobs use exponential backoff to manage job run failures. Failed tasks in a multi-task continuous job are not retried and will stay in the Failed state until the job is manually restarted or all other tasks terminate.
• Select Run now to trigger a new job run on a paused continuous job.
• To have your continuous job pick up a new configuration, cancel the existing run. A new run automatically starts. You can also click Restart run to restart the job run with the updated configuration.

How are failures handled for continuous jobs?

Failures at the job level are managed using an exponential backoff algorithm.

Consecutive failures at a job level are managed using exponential backoff, which allows continuous jobs to run without pausing and return to a healthy state when recoverable failures occur.

When a continuous job exceeds the allowable threshold for consecutive failures, the following describes how subsequent job runs are managed:

1. The job is restarted after a retry period set by the system.
2. If the next job run fails, the retry period is increased, and the job is restarted after this new retry period.
   1. For each subsequent job run failure, the retry period is increased up to a maximum retry period set by the system. After reaching the maximum retry period, the job continues to be retried using the maximum retry period. There is no limit on the number of retries for a continuous job.
   2. If the job run completes successfully and starts a new run, or if the run exceeds a threshold without failure, the job is considered healthy, and the backoff sequence resets.

You can restart a continuous job in the exponential backoff state in the Jobs UI or by passing the job ID to the POST /api/2.1/jobs/run-now request in the Jobs 2.1 API or the POST /api/2.0/jobs/run-now request in the Jobs 2.0 API.