DLT release 2022.37

September 14 - 22, 2022

These features and improvements were released with the 2022.37 release of DLT.

Databricks Runtime versions used by this release

Channel:

• CURRENT (default): Databricks Runtime 10.3.7
• PREVIEW: Databricks Runtime 11.0.5

New features and improvements in this release

• The start update API request now returns the request_id field in the response body. The request_id is a stable identifier for the original request starting the update. If an update is retried or restarted, the new update inherits the request_id.

JSON

{
  "update_id": "the ID of the update that was started",
  "request_id": "The ID of the request that started this update"
}

The new requests API request (GET /pipelines/{pipeline_id}/requests/{request_id}) returns the status of the pipeline update associated with request_id. The response includes information about the latest update.

JSON

{
  "status": "ACTIVE",
  "latest_update": {}
}

• Your Python code can now call spark.sql operations outside of dlt.table() or dlt.view() functions, as long as the operation is not reading from a materialized view or streaming table.

• Event log entries now contain the maturity property to indicate the stability of the event schema. Possible values are stable, evolving, and deprecated. For more information about the DLT event log, see What is the Lakeflow Declarative Pipelines event log?.

• The error message is improved when incompatible changes are made to source tables used by a streaming table.

• You can now select a cluster policy in the DLT UI when you create or edit a pipeline. Previously, setting the cluster policy for a pipeline required editing the pipeline's JSON settings.

• Faster pipeline startup. This release includes enhancements that speed up the SETTING_UP_TABLES step when a pipeline is starting.

Bug Fixes in this release

• This release fixes a bug that prevents enhanced autoscaling from scaling up when no idle cluster instances are available.