Manage updates for continuous pipelines

A continuous pipeline runs without stopping, so Databricks cannot wait for it to finish before applying platform updates such as new runtime versions or infrastructure maintenance. To apply these updates, Databricks restarts the pipeline. A maintenance window lets you control when those restarts happen: you select a day of the week, a start hour, and a time zone, and Databricks starts maintenance only within that window.

The maintenance window is a setting on the continuous job that runs the pipeline. If you don't set one, maintenance can run at any time.

How Databricks applies updates

To update a continuous pipeline, Databricks starts a new pipeline update on a fresh cluster and shuts down the previous one. Because continuous pipelines track their progress in a streaming checkpoint, the new update resumes from where the previous one left off. Data processing pauses briefly during the swap, then continues without losing data.

Databricks starts maintenance only within a one-hour window that begins at the start hour on the day you select. The restart itself is usually short.

Select a day and start hour when a brief restart is acceptable for your workload.

What updates are included?

Maintenance for a continuous pipeline can include:

• Databricks Runtime version upgrades
• Security patches and updates
• Operating system updates
• Spark Declarative Pipelines on Lakeflow features and enhancements
• Serverless infrastructure maintenance, such as the periodic recycling of long-running serverless clusters

Both runtime upgrades (service upgrades) and serverless infrastructure maintenance respect the maintenance window you configure.

important

Serverless clusters have a maximum lifetime. If that limit is reached before your next maintenance window, Databricks restarts the pipeline to stay within the limit, even if the current time is outside your window. This ensures a long-running pipeline is eventually recycled.

How often are updates applied?

Databricks applies an update only when maintenance is needed. New runtime versions are released periodically, and serverless infrastructure maintenance occurs on a recurring cadence. How often an individual pipeline restarts depends on:

• The cadence of new Databricks Runtime releases that apply to your pipeline
• For serverless pipelines, the serverless cluster recycle schedule
• How often your maintenance window occurs

When your maintenance window arrives and no update is pending, the window passes with no restart.

Configure a maintenance window

You can set a maintenance window in the Databricks UI or with the Jobs REST API.

A maintenance window has three required parts:

Setting	Description
Day of the week	A single day, from Monday to Sunday.
Start hour	The hour the window starts, as an integer from 0 to 23, in 24-hour time.
Time zone	A Java time zone ID, such as America/New_York. Databricks evaluates the window in this time zone.

Setting	Description
Day of the week	A single day, from Monday to Sunday.
Start hour	The hour the window starts, as an integer from 0 to 23, in 24-hour time.
Time zone	A Java time zone ID, such as America/New_York. Databricks evaluates the window in this time zone.

note

Maintenance windows apply only to continuous jobs and pipelines. The setting is not available for jobs that run on a schedule or a different trigger type.

Configure a maintenance window in the UI

You can configure a maintenance window from either the Jobs UI or the Pipelines UI.

From the Jobs UI

1. In your Databricks workspace's sidebar, click Jobs & Pipelines.
2. Click the job's name.
3. In the Job details pane, open the Continuous trigger and expand Advanced to find Maintenance window.
4. Select a day of the week, a start hour, and a time zone.
5. Click Save.

From the Pipelines UI

1. In your Databricks workspace's sidebar, click Jobs & Pipelines.
2. Click the pipeline's name.
3. Open Schedule and expand Advanced settings to find Maintenance window.
4. Select a day of the week, a start hour, and a time zone.
5. Click Save.

Configure a maintenance window with the Jobs API

Set the maintenance_window field inside the continuous settings when you create or update a job with the Jobs REST API. The following example configures maintenance to start at 2 AM Eastern Time on Tuesdays:

JSON

{
  "continuous": {
    "pause_status": "UNPAUSED",
    "maintenance_window": {
      "start_hour": 2,
      "day_of_week": "TUESDAY",
      "timezone_id": "America/New_York"
    }
  }
}

All three fields (start_hour, day_of_week, and timezone_id) are required when you set a maintenance window. start_hour must be an integer from 0 to 23.

Configure maintenance notifications

Databricks can notify you when a maintenance restart starts and when it finishes. Maintenance notifications use the standard Lakeflow Jobs notification system, so you can deliver them to email addresses or to system destinations such as Slack, PagerDuty, or Microsoft Teams. See Add notifications on a job.

Two events are available on a continuous job:

• Maintenance start: sent when a maintenance restart begins. Includes the job, the cause (for example, a service upgrade), and the start time.
• Maintenance complete: sent when the restart finishes. Includes the cause, the completion time, the restart duration, and the pipeline's status after the restart.

Daylight saving transitions

Databricks evaluates the maintenance window in the time zone you configure, so the window follows that zone's daylight saving transitions. On most days, the window starts at your configured local hour. On the day clocks change, the start can shift by up to an hour. Select a time zone that matches your operating schedule.

Handle disruptions during updates

A maintenance restart pauses data processing while the new update takes over. Streaming sources keep accumulating data during the pause, and the new update resumes from the pipeline's checkpoint, so no data is lost.

If your pipeline has strict latency requirements, select a maintenance window during a period when a brief pause is acceptable. To confirm the pipeline returned to a healthy state after a restart, monitor the Maintenance complete notification.