メインコンテンツまでスキップ
に最終更新

Lakeflow Spark Declarative Pipelines release notes and the release upgrade process

This article explains the Lakeflow Spark Declarative Pipelines release process, how the Lakeflow Spark Declarative Pipelines runtime is managed, and provides links to release notes for each Lakeflow Spark Declarative Pipelines release.

Lakeflow Spark Declarative Pipelines runtime channels​

注記

To see the Databricks Runtime versions used with a Lakeflow Spark Declarative Pipelines release, see the release notes for that release.

Lakeflow Spark Declarative Pipelines clusters use runtimes based on Databricks Runtime release notes versions and compatibility. Databricks automatically upgrades the Lakeflow Spark Declarative Pipelines runtimes to support enhancements and upgrades to the platform. You can use the channel field in the Lakeflow Spark Declarative Pipelines settings to control the Lakeflow Spark Declarative Pipelines runtime version that runs your pipeline. The supported values are:

  • current to use the current runtime version.
  • preview to test your pipeline with upcoming changes to the runtime version.

By default, your pipelines run using the current runtime version. Databricks recommends using the current runtime for production workloads. To learn how to use the preview setting to test your pipelines with the next runtime version, see Automate testing of your pipelines with the next runtime version.

重要

Features marked as generally available or Public Preview are available in the current channel.

For more information about Lakeflow Spark Declarative Pipelines channels, see the channel field in the Lakeflow Spark Declarative Pipelines pipeline settings.

To understand how Lakeflow Spark Declarative Pipelines manages the upgrade process for each release, see How do Lakeflow Spark Declarative Pipelines upgrades work?.

How do I find the Databricks Runtime version for a pipeline update?​

You can query the Lakeflow Spark Declarative Pipelines event log to find the Databricks Runtime version for a pipeline update. See Runtime information.

Lakeflow Spark Declarative Pipelines release notes​

Lakeflow Spark Declarative Pipelines release notes are organized by year and week-of-year. Because Lakeflow Spark Declarative Pipelines is versionless, both workspace and runtime changes take place automatically. The following release notes provide an overview of changes and bug fixes in each release:

How do Lakeflow Spark Declarative Pipelines upgrades work?​

Lakeflow Spark Declarative Pipelines is considered to be a versionless product, which means that Databricks automatically upgrades the Lakeflow Spark Declarative Pipelines runtime to support enhancements and upgrades to the platform. Databricks recommends limiting external dependencies for Lakeflow Spark Declarative Pipelines.

Databricks proactively works to prevent automatic upgrades from introducing errors or issues to production Lakeflow Spark Declarative Pipelines. See Lakeflow Spark Declarative Pipelines upgrade process.

Especially for users that deploy Lakeflow Spark Declarative Pipelines with external dependencies, Databricks recommends proactively testing pipelines with preview channels. See Automate testing of your pipelines with the next runtime version.

Lakeflow Spark Declarative Pipelines upgrade process​

Databricks manages the Databricks Runtime used by Lakeflow Spark Declarative Pipelines compute resources. Lakeflow Spark Declarative Pipelines automatically upgrades the runtime in your Databricks workspaces and monitors the health of your pipelines after the upgrade.

If Lakeflow Spark Declarative Pipelines detects that a pipeline cannot start because of an upgrade, the runtime version for the pipeline reverts to the previous version that is known to be stable, and the following steps are triggered automatically:

  • The pipeline's Lakeflow Spark Declarative Pipelines runtime is pinned to the previous known-good version.
  • Databricks support is notified of the issue.
    • If the issue is related to a regression in the runtime, Databricks resolves the issue.
    • If the issue is caused by a custom library or package used by the pipeline, Databricks contacts you to resolve the issue.
  • When the issue is resolved, Databricks initiates the upgrade again.
重要

Lakeflow Spark Declarative Pipelines only reverts pipelines running in production mode with the channel set to current.

Automate testing of your pipelines with the next runtime version​

To ensure changes in the next Lakeflow Spark Declarative Pipelines runtime version do not impact your pipelines, use the Lakeflow Spark Declarative Pipelines channels feature:

  1. Create a staging pipeline and set the channel to preview.
  2. In the Lakeflow Spark Declarative Pipelines UI, create a schedule to run the pipeline weekly and enable alerts to receive an email notification for pipeline failures. Databricks recommends scheduling weekly test runs of pipelines, especially if you use custom pipeline dependencies.
  3. If you receive a notification of a failure and are unable to resolve it, open a support ticket with Databricks.

Pipeline dependencies​

Lakeflow Spark Declarative Pipelines supports external dependencies in your pipelines; for example, you can install any Python package using the %pip install command. Lakeflow Spark Declarative Pipelines also supports using global and cluster-scoped init scripts. However, these external dependencies, particularly init scripts, increase the risk of issues with runtime upgrades. To mitigate these risks, minimize using init scripts in your pipelines. If your processing requires init scripts, automate testing of your pipeline to detect problems early; see Automate testing of your pipelines with the next runtime version. If you use init scripts, Databricks recommends increasing your testing frequency.