December 6 - 9, 2022
These features and improvements were released with the 2022.49 release of Delta Live Tables.
For more information about Delta Live Tables channels, see the
channel field in the Delta Live Tables settings. For information about setting the channel when you create or edit a pipeline, see Create a pipeline. For information about viewing the channel used by a pipeline, see Pipeline details.
CURRENT (default): Databricks Runtime 11.0.11
PREVIEW: Databricks Runtime 11.3.2
You can now import files from a Databricks Repo as Python modules. You can import files from the current repo path or a specified repo path using
sys.path.append(). See Import Python modules from a Databricks repo.
Enhanced Autoscaling is now generally available. With this release, Enhanced Autoscaling is the default autoscaling mode for new pipelines created in the UI. Databricks recommends updating pipelines that use the existing cluster autoscaling to use Enhanced Autoscaling. For more information about Enhanced Autoscaling, see Databricks Enhanced Autoscaling. For more information about configuring Enhanced Autoscaling when you create or edit a pipeline, see Create, run, and manage Delta Live Tables pipelines.
For pipelines run in continuous mode, the default retry limit is now 2 when retryable failures occur. Including the original attempt, all flows have three attempts (with exponential backoff) to run when non-fatal failures occur. Previously, the defaults were unbounded retries for batch flows, and five retries for streaming flows. See Delta Live Tables settings.
The configuration parameter
pipelines.maxFlowRetryAttemptsnow applies to the retry limit for all flows, including batch and streaming flows, and continuous or triggered pipelines. Previously,
pipelines.maxFlowRetryAttemptsapplied to all retry limits in triggered mode but only batch flows in continuous mode; retries for streaming pipelines in continuous mode were controlled by
pipelines.numStreamRetryAttempts. For existing pipelines,
pipelines.numStreamRetryAttemptscontinues to be supported for streaming flows in continuous mode. If both
pipelines.numStreamRetryAttemptstakes priority for streaming flows in continuous mode. Otherwise, the configured value for
pipelines.maxFlowRetryAttemptsis used. See Delta Live Tables settings.
This release fixes a bug causing display of incorrect DBU per hour calculations for Photon-enabled clusters.
This release fixes a bug causing Delta Live Tables maintenance tasks to fail in Unity Catalog enabled workspaces.
This release fixes a bug where stopping or restarting a continuous pipeline that ingests data with Auto Loader causes the pipeline to hang for several minutes.