Release 2022.37

September 14 - 22, 2022

These features and improvements were released with the 2022.37 release of Delta Live Tables.

Channels

For more information about Delta Live Tables channels, see the channel field in the Delta Live Tables settings. For information about setting the channel when you create or edit a pipeline, see Create a pipeline. For information about viewing the channel used by a pipeline, see Pipeline details.

  • CURRENT (default): Databricks Runtime 10.3.7

  • PREVIEW: Databricks Runtime 11.0.5

Changes

  • The Start a pipeline update API request now returns the request_id field in the response body. The request_id is a stable identifier for the original request starting the update. If an update is retried or restarted, the new update inherits the request_id.

{
  "update_id": "the ID of the update that was started",
  "request_id": "The ID of the request that started this update"
}

The new requests API request (GET /pipelines/{pipeline_id}/requests/{request_id}) returns the status of the pipeline update associated with request_id. The response includes information about the latest update.

{
   "status": "ACTIVE",
   "latest_update": {
   }
 }

  • Your Python code can now call spark.sql operations outside of dlt.table() or dlt.view() functions, as long as the operation is not reading from a live table.

  • Event log entries now contain the maturity property to indicate the stability of the event schema. Possible values are stable, evolving, and deprecated. For more information about the Delta Live Tables event log, see Monitor pipelines with the Delta Live Tables event log.

  • The error message is improved when incompatible changes are made to source tables used by a streaming table.

  • You can now select a cluster policy in the Delta Live Tables UI when you create or edit a pipeline. Previously, setting the cluster policy for a pipeline required editing the pipeline’s JSON settings. See Create, run, and manage Delta Live Tables pipelines.

  • Faster pipeline startup. This release includes enhancements that speed up the SETTING_UP_TABLES step when a pipeline is starting.

Bug Fixes

  • This release fixes a bug that prevents Enhanced Autoscaling from scaling up when no idle cluster instances are available.