DLT release 2022.40

September 28 - October 5, 2022

These features and improvements were released with the 2022.40 release of DLT.

Databricks Runtime versions used by this release

Channel:

Using a cluster label other than default or maintenance in your pipeline settings causes failures during pipeline creation or edit. For more information about cluster configuration for Lakeflow Declarative Pipelines, see Configure classic compute for Lakeflow Declarative Pipelines.

You can now open or run pipelines directly from your Lakeflow Declarative Pipelines notebooks. See Develop and debug ETL pipelines with a notebook in Lakeflow Declarative Pipelines.

This release includes a breaking change in the pipeline settings data structure returned by DLT API requests. This change returns the edition field as an upper case value. Previously, this value was returned as mixed case. You must update your clients that read the edition value in API responses to ensure the correct processing of the return value. However, the edition value in API requests is case insensitive; your clients that submit requests including this field do not require updating.

This release fixes a bug that causes the DLT UI to show a pipeline in a RUNNING state even after update failure.

This update fixes a bug in the APPLY_CHANGES interface when re-processing input data where the data contains duplicate DELETE operations with the same value in the SEQUENCE column. This scenario results in NULL values in the target table and removal of the DeleteVersion value. This is equivalent to marking the record as NOT-DELETED. With this change, the duplicate DELETE operations are handled idempotently and always mark the record as DELETED.

This release fixes a bug causing rows to be hidden during SCD type 2 processing when a previous historical record is deleted containing the same columns as the next historical record.

This update fixes a bug causing a view with an expect_or_fail expectation to remain in the RUNNING state even if the expectation fails. With this fix, the pipeline fails and transitions to the FAILED state.