Delta Live Tables clusters use a runtime based on Databricks Runtime. Databricks automatically upgrades the Delta Live Tables runtime to support enhancements and upgrades to the platform. As with any software upgrade, a Delta Live Tables runtime upgrade may result in errors or issues running your pipelines. This article describes best practices to test your pipeline with upcoming releases of the Delta Live Tables runtime, and Delta Live Tables features that enhance the stability of your pipelines.
channel field in the Delta Live Tables pipeline settings controls the Delta Live Tables runtime version that runs your pipeline. The supported values are:
previewto test your pipeline with upcoming changes to the runtime version.
currentto use the current runtime version.
By default, your pipelines run using the
current runtime version. Databricks recommends using the
current runtime for production workloads. To learn how to use the
preview setting to test your pipelines with the next runtime version, see Automate testing of your pipelines with the next runtime version.
Delta Live Tables automatically upgrades the runtime in your Databricks workspaces and monitors the health of your pipelines after the upgrade. If Delta Live Tables detects that a pipeline cannot start because of an upgrade, the runtime version for the pipeline reverts to the previous known-good version, and the following steps are triggered automatically:
Delta Live Tables reverts only pipelines running in production mode and with the channel set to
The pipeline’s Delta Live Tables runtime is pinned to the previous known-good version.
The Delta Live Tables UI shows a visual indicator that the pipeline is pinned to a previous version because of an upgrade failure.
Databricks support is notified of the issue. If the issue is related to a regression in the runtime, Databricks will resolve the issue. If the issue is caused by a custom library or package used by the pipeline, Databricks will contact you to resolve the issue.
When the issue is resolved, Databricks will initiate the upgrade again.
To ensure changes in the next Delta Live Tables runtime version do not impact your pipelines, use the Delta Live Tables channels feature:
Create a staging pipeline and set the channel to
In the Delta Live Tables UI, create a schedule to run the pipeline weekly and enable alerts to receive an email notification for pipeline failures.
If you receive a notification of a failure and are unable to resolve it, open a support ticket with Databricks.
Delta Live Tables supports external dependencies in your pipelines; for example, you can install any Python package using the
%pip install command. Delta Live Tables also supports using global and cluster-scoped init scripts. However, these external dependencies, particularly init scripts, increase the risk of issues with runtime upgrades. To mitigate these risks, minimize using init scripts in your pipelines. If your processing requires init scripts, automate testing of your pipeline to detect problems early; see Automate testing of your pipelines with the next runtime version. If you use init scripts, Databricks recommends increasing your testing frequency.