Pular para o conteúdo principal
Última atualização em

Lakeflow Spark Declarative Pipelines release notes 2026

The following Lakeflow Spark Declarative Pipelines features, improvements, and bug fixes were released in 2026.

nota

Because Lakeflow Spark Declarative Pipelines channel releases follow a rolling upgrade process, channel upgrades are deployed to different regions at different times. Your release, including Databricks Runtime versions, might not be updated until a week or more after the initial release date. To find the Databricks Runtime version for a pipeline, see Runtime information.

January 2026​

These features and improvements to Lakeflow Spark Declarative Pipelines were released between November 14, 2025 and January 13, 2026.

Databricks Runtime versions used by this release​

Channel:

  • CURRENT (default): Databricks Runtime 16.4
  • PREVIEW: Databricks Runtime 17.3

New features and improvements​

  • You can now store and manage data quality expectations directly in Unity Catalog tables, centralizing data quality rules with your data governance framework. This enables version-controlled, auditable quality rules that can be shared across multiple pipelines.

  • Continuous pipelines running longer than 7 days now restart gracefully with minimal downtime and an explicit update cause (INFRASTRUCTURE_MAINTENANCE), instead of restarting abruptly when the underlying compute needs to be refreshed.

  • Pipelines now support queued execution mode, where multiple update requests are automatically queued and executed sequentially instead of failing with conflicts. This simplifies operations for pipelines with frequent update triggers and eliminates the need for manual retry coordination.

  • You can now materialize multiple SCD Type 2 views from a single change data source, improving efficiency when creating multiple historical views of the same data. This eliminates the need to reprocess source data for each SCD Type 2 output.

  • Pipeline schedules and configuration can now be stored and read from Unity Catalog table properties, enabling centralized settings management through data governance. This allows you to manage pipeline behavior alongside your data definitions.

  • MANAGE permissions are now automatically propagated to materialized views and streaming tables in Unity Catalog, simplifying permission management for pipeline outputs. This ensures consistent access control without manual permission grants.

  • SCD Type 2 operations now automatically coalesce duplicate records with the same natural key, ensuring data consistency and preventing duplicate historical records in your slowly changing dimension tables.

  • Pipelines now have an option to automatically drop inactive tables that are no longer part of the pipeline definition. This helps maintain clean data warehouses and reduces storage costs from obsolete tables. See Use Unity Catalog with pipelines.

  • Pipeline definition, patch operations, and run-as identity changes are now included in the audit log, providing comprehensive tracking of configuration changes for compliance and security monitoring. See Pipeline event log.

Bug fixes​

No significant bug fixes were included in this release period. All changes were new features and improvements.