Spark Declarative Pipelines on Lakeflow release 2025.30

July 23 - 30, 2025

These features and improvements were released with the 2025.30 release of Spark Declarative Pipelines on Lakeflow.

Databricks Runtime versions used by this release

Channel:

• CURRENT (default): Databricks Runtime 16.1
• PREVIEW: Databricks Runtime 16.4

note

Because Spark Declarative Pipelines on Lakeflow channel releases follow a rolling upgrade process, channel upgrades are deployed to different regions at different times. Your release, including Databricks Runtime versions, might not be updated until a week or more after the initial release date. To find the Databricks Runtime version for a pipeline, see Runtime information.

New features and improvements

• We’ve added the following new capabilities to AUTO CDC APIs in Spark Declarative Pipelines on Lakeflow.

  • Multiple flows per target: The create_auto_cdc_flow command now supports a name parameter, allowing multiple independent AUTO CDC flows to write into the same destination table. This enables scenarios like applying both incremental updates and correction streams to a single table.

  • One-time backfills: A new once parameter supports one-time backfill operations (e.g., initial hydrations). When once = true, the flow ingests a snapshot of data a single time and ignores future file arrivals.

  • SQL support: New CREATE FLOW SQL syntax allows you to use AUTO CDC INTO directly in SQL, with the same support for name and once.

• The Lakeflow Editor now supports on-demand cluster connect + cluster keep-alive triggered by user activity.