Release 2022.37
September 14 - 22, 2022
These features and improvements were released with the 2022.37 release of Delta Live Tables.
Channels
For more information about Delta Live Tables channels, see the channel
field in the Delta Live Tables settings. For information about setting the channel when you create or edit a pipeline, see Create a pipeline. For information about viewing the channel used by a pipeline, see Pipeline details.
CURRENT (default): Databricks Runtime 10.3.7
PREVIEW: Databricks Runtime 11.0.5
Changes
The Start a pipeline update API request now returns the
request_id
field in the response body. Therequest_id
is a stable identifier for the original request starting the update. If an update is retried or restarted, the new update inherits therequest_id
.
{
"update_id": "the ID of the update that was started",
"request_id": "The ID of the request that started this update"
}
The new requests
API request (GET /pipelines/{pipeline_id}/requests/{request_id}
) returns the status of the pipeline update associated with request_id
. The response includes information about the latest update.
{
"status": "ACTIVE",
"latest_update": {
}
}
Your Python code can now call
spark.sql
operations outside ofdlt.table()
ordlt.view()
functions, as long as the operation is not reading from a live table.
Event log entries now contain the
maturity
property to indicate the stability of the event schema. Possible values arestable
,evolving
, anddeprecated
. For more information about the Delta Live Tables event log, see Monitor pipelines with the Delta Live Tables event log.
The error message is improved when incompatible changes are made to source tables used by a streaming table.
You can now select a cluster policy in the Delta Live Tables UI when you create or edit a pipeline. Previously, setting the cluster policy for a pipeline required editing the pipeline’s JSON settings. See Create, run, and manage Delta Live Tables pipelines.
Faster pipeline startup. This release includes enhancements that speed up the
SETTING_UP_TABLES
step when a pipeline is starting.