Databricks SQL Release notes 2024

The following outlines the improvements and updates in Databricks SQL from January through December 2024.

November 21, 2024

SQL warehouse system tables (Public Preview)

  • The system.compute.warehouses table records when SQL warehouses are created, edited, and deleted. You can use the table to monitor changes to warehouse settings, including the warehouse name, type, size, channel, tags, auto-stop, and autoscaling settings. See Warehouses system table reference.

Data discovery

  • The Lineage tab in Catalog Explorer has been redesigned with an improved UI for entity filtering.

November 13, 2024

Legacy dashboards:

  • Resolved an issue where templated tooltips were not displaying detailed content for dual-axis and multi-field axis charts.

November 6, 2024

Human-readable schedule support for Databricks SQL streaming tables and materialized views

Users can now start, create, and alter schedules for streaming tables and materialized views using human-readable syntax instad of CRON scheduling. See ALTER MATERIALIZED VIEW, ALTER STREAMING TABLE, CREATE MATERIALIZED VIEW, and CREATE STREAMING TABLE.

Streaming tables now support time travel queries

You can now use time travel to query previous table versions based on timestamp or table version (as recorded in the transaction log). You may need to refresh your streaming table before using time travel queries. See What is Delta Lake time travel?.

Time travel queries are not supported for materialized views.

October 31, 2024

User interface updates

New SQL editor (Public Preview)

  • You can now run the active SQL query using the keyboard shortcut Command (or Ctrl) + Shift + Enter.

  • The parameters input area now shows a scrollbar when the text extends outside of the display window.

  • Fixed an issue that prevented the query profile details page from opening fully.

  • You can now rename queries by typing the new name into the tab title.

  • The Schedule button is now disabled for queries that have never been saved before.

October 24, 2024

Release notes for AI/BI tools

The release notes for AI/BI dashboards and AI/BI Genie have moved to AI/BI release notes. Future releases and updates will be documented there.

October 17, 2024

Notification destinations are now generally available

You can create and configure notification destinations that workspace users can add to certain workflows, like alerts, Databricks jobs, and AI/BI dashboard schedules, to send emails or webhooks when an event runs. See Manage notification destinations.

October 10, 2024

Materialized views and streaming tables are now generally available on Databricks SQL

Streaming tables enable incremental ingestion from cloud storage and message queues, while materialized views are pre-computed views that are automatically and incrementally updated as new data arrives. See Use materialized views in Databricks SQL and Load data using streaming tables in Databricks SQL.

Query insights

October 3, 2024

User interface updates

The features listed in this section are independent of the SQL warehouse compute versions described above.

Catalog Explorer

  • AI-generated comments are now supported for catalogs, schemas, volumes, models, and functions and users can use the inline chat Assistant to help edit their comments.

SQL AI functions The vector_search() function is now available in Public Preview. See vector_search function

September 26, 2024

September 11, 2024

User interface updates

The features listed in this section are independent of the SQL Warehouse compute versions described above.

SQL editor

You can now use named parameter marker syntax in the SQL editor. Named parameter marker syntax can be used across the SQL editor, notebooks, and AI/BI dashboards. See Work with query parameters.

Queries and legacy dashboards

For SQL queries and legacy dashboards, deleted items no longer appear in the listing pages. Find deleted items in the workspace trash folder. Workspace admins can view deleted items in all users’ trash folders.

September 5, 2024

Changes in 2024.40

Databricks SQL version 2024.40 includes the following behavioral changes, new features, and improvements.

Behavioral changes

  • Change to the default schema binding mode for views

    Views now adapt to schema changes in the underlying query by using schema compensation with regular casting rules. This is a change from the previous default of BINDING mode, which raised errors when a safe cast could not be performed when referencing the view.

    See CREATE VIEW and cast function.

  • Disallow using the undocumented ! syntax instead of NOT outside boolean expressions

    With this release, the use of ! as a synonym for NOT outside of boolean expressions is no longer allowed. For example, statements such as the following: CREATE ... IF ! EXISTS, IS ! NULL, a ! NULL column or field property, ! IN and ! BETWEEN, must be replaced with: CREATE ... IF NOT EXISTS, IS NOT NULL, a NOT NULL column or field property, NOT IN and NOT BETWEEN.

    This change ensures consistency, aligns with the SQL standard, and makes your SQL more portable.

    The boolean prefix operator ! (for example, !is_mgr or !(true AND false)) is unaffected by this change.

  • Disallow undocumented column definition syntax in views

    Databricks supports CREATE VIEW with named columns and column comments. Previously, the specification of column types, NOT NULL constraints, or DEFAULT has been allowed. With this release, you can no longer use this syntax.

    This change ensures consistency, aligns with the SQL standard, and supports future enhancements.

  • Adding a CHECK constraint on an invalid column now returns the UNRESOLVED_COLUMN.WITH_SUGGESTION error class

    To provide more useful error messaging, in Databricks Runtime 15.3 and above, an ALTER TABLE ADD CONSTRAINT statement that includes a CHECK constraint referencing an invalid column name returns the UNRESOLVED_COLUMN.WITH_SUGGESTION error class. Previously, an INTERNAL_ERROR was returned.

New features and improvements

  • Enable UniForm Iceberg using ALTER TABLE

    You can now enable UniForm Iceberg on existing tables without rewriting data files. See Enable Iceberg reads on an existing table.

  • UTF-8 validation functions

    This release introduces the following functions for validating UTF-8 strings:

    • is_valid_utf8 verified whether a string is a valid UTF-8 string.

    • make_valid_utf8 converts a potentially invalid UTF-8 string to a valid UTF-8 string using substitution characters

    • validate_utf8 raises an error if the input is not a valid UTF-8 string.

    • try_validate_utf8 returns NULL if the input is not a valid UTF-8 string.

  • to_avro and from_avro functions

    The to_avro and from_avro functions allow conversion of SQL types to Avro binary data and back.

  • try_url_decode function

    This release introduces the try_url_decode function, which decodes a URL-encoded string. If the string is not in the correct format, the function returns NULL instead of raising an error.

  • Optionally allow the optimizer to rely on unenforced foreign key constraints

    To improve query performance, you can now specify the RELY keyword on FOREIGN KEY constraints when you CREATE or ALTER a table.

  • Support for dropping the check constraints table feature

    Selective overwrites using replaceWhere now run jobs that delete data and insert new data in parallel, improving query performance and cluster utilization.

  • Parallelized job runs for selective overwrites

    Selective overwrites using replaceWhere now run jobs that delete data and insert new data in parallel, improving query performance and cluster utilization.

  • Improved performance for change data feed with selective overwrites

    Selective overwrites using replaceWhere on tables with change data feed no longer write separate change data files for inserted data. These operations use a hidden _change_type column present in the underlying Parquet data files to record changes without write amplification.

  • Improved query latency for the COPY INTO command

    This release includes a change that improves the query latency for the COPY INTO command. This improvement is implemented by making the loading of state by the RocksDB state store asynchronous. With this change, you should see an improvement in start times for queries with large states, such as queries with a large number of already ingested files.

User interface updates

The features listed in this section are independent of the SQL Warehouse compute versions described above.

Visualizations

  • Tooltip totals now appear only for stacked charts.

August 22, 2024

Visualizations

For grouped and multi-field configurations, tooltips now show totals when you hover over chart elements.

August 15, 2024

Visualizations

Fixed an issue where row numbers in table visualizations didn’t update after changing the page size.

Data discovery

The ability to expand and collapse nested complex column types in Unity Catalog tables is now supported.

August 1, 2024

Visualizations:

  • Table sorting is now preserved when data changes due to filtering.

SQL Editor:

  • Increased readability by adding additional padding between the last line of a query and the result output.

July 25, 2024

Databricks REST API:

  • APIs for managing queries, alerts, data sources, and permissions have changed. The legacy version will continue to be supported for six months. This transition period is intended to give you sufficient time to migrate your applications and integrations to the new version before the older version is phased out. See Update to the latest Databricks SQL API version

July 18, 2024

User interface updates

Catalog explorer:

  • A new catalog configuration wizard is now available for setting up workspace bindings, catalog privileges, and metadata when creating a catalog.

SQL Warehouse monitoring:

  • CAN MONITOR permission is now generally available. It allows privileged users to monitor SQL warehouses, including the associated query history and query profiles. See Vector search endpoint ACLs.

July 11, 2024

Databricks SQL Version 2024.35 Available

Rollout Schedule: Preview rollout for 2024.35: Between July 9th and July 15th

Changes in 2024.35

Disable column mapping with drop feature

You can now use DROP FEATURE to disable column mapping on Delta tables and downgrade the table protocol. See Disable column mapping.

Variant type syntax and functions in Public Preview

Built-in Apache Spark support for working with semi-structured data as VARIANT type is now available in Spark DataFrames and SQL. See Query variant data.

Variant type support for Delta Lake in Public Preview

You can now use VARIANT to store semi-structured data in tables backed by Delta Lake. See Variant support in Delta Lake.

Support for different modes of schema evolution in views

CREATE VIEW and ALTER VIEW now allow you to set a schema binding mode, enhancing how views handle schema changes in underlying objects. This feature enables views to either tolerate or adapt to schema changes in the underlying objects. It addresses changes in the query schema resulting from modifications to object definitions.

Performance improvement for some window functions

This release includes a change that improves the performance of some Spark window functions, specifically functions that do not include an ORDER BY clause or a window_frame parameter. In these cases, the system can rewrite the query to run it using an aggregate function. This change allows the query to run faster by using partial aggregation and avoiding the overhead of running window functions. The Spark configuration parameter spark.databricks.optimizer.replaceWindowsWithAggregates.enabled controls this optimization and is set to true by default. To turn this optimization off, set spark.databricks.optimizer.replaceWindowsWithAggregates.enabled to false.

Support for the try_mod function added

This release adds support for the PySpark try_mod() function. This function supports the ANSI SQL-compatible calculation of the integer remainder by dividing two numeric values. If the divisor argument is 0, the try_mod() function returns null instead of throwing an exception. You can use the try_mod() function instead of mod or %, which throws an exception if the divisor argument is 0 and ANSI SQL is enabled.

User interface updates

SQL Editor:

The inline assistant is now available in the SQL editor. Click the assistant icon in the editor box to toggle the input. Type a question or comment in English, then press Enter (not Shift+Enter, which runs a query) to generate a response with a different view directly in the editor.

Platform:

An API for notification destinations is now available. You can now programmatically manage webhook and email destinations for your alerts and job run notifications. See Notification Destinations.

July 4, 2024

Databricks SQL Version 2024.30 Available

Rollout schedule: Preview rollout for 2024.30: Between July 9th and July 15th

Changes in 2024.30

Lakehouse Federation is generally available (GA)

Lakehouse Federation connectors across the following database types are now generally available (GA):

  • MySQL

  • PostgreSQL

  • Amazon Redshift

  • Snowflake

  • Microsoft SQL Server

  • Azure Synapse (SQL Data Warehouse)

  • Databricks

This release also introduces the following improvements:

  • Support for additional pushdowns (string, math, miscellaneous functions).

  • Improved pushdown success rate across different query shapes.

  • Additional pushdown debugging capabilities:

    • The EXPLAIN FORMATTED output displays the pushed-down query text.

    • The query profile UI displays the pushed-down query text, federated node identifiers, and JDBC query execution times (in verbose mode). See View system-generated federated queries.

DESCRIBE HISTORY now shows clustering columns for tables that use liquid clustering

When you run a DESCRIBE HISTORY query, the operationParameters column shows a clusterBy field by default for CREATE OR REPLACE and OPTIMIZE operations. For a Delta table that uses liquid clustering, the clusterBy field is populated with the table’s clustering columns. If the table does not use liquid clustering, the field is empty.

Support for primary and foreign keys is generally available

Support for primary and foreign keys in Databricks Runtime is generally available. The GA release includes the following changes to the privileges required to use primary and foreign keys:

  • To define a foreign key, you must have the SELECT privilege on the table with the primary key that the foreign key refers to. You do not need to own the table with the primary key, which was previously required.

  • Dropping a primary key using the CASCADE clause does not require privileges on the tables that define foreign keys that reference the primary key. Previously, you needed to own the referencing tables.

  • Dropping a table that includes constraints now requires the same privileges as dropping tables that do not include constraints.

To learn how to use primary and foreign keys with tables or views, see CONSTRAINT clause, ADD CONSTRAINT clause, and DROP CONSTRAINT clause.

Liquid clustering is GA

Support for liquid clustering is now generally available using Databricks Runtime 15.2 and above. See Use liquid clustering for Delta tables.

Type widening is in Public Preview

You can now enable type widening on tables backed by Delta Lake. Tables with type widening enabled allow changing the type of columns to a wider data type without rewriting underlying data files. See Type widening.

Schema evolution clause added to SQL merge syntax

You can now add the WITH SCHEMA EVOLUTION clause to a SQL merge statement to enable schema evolution for the operation. See Schema evolution syntax for merge.

Vacuum inventory support

You can now specify an inventory of files to consider when running the VACUUM command on a Delta table. See the OSS Delta docs.

Support for Zstandard compression functions

You can now use the zst_compress, zstd_decompress, and try_zstd_decompress functions to compress and decompress BINARY data.

Query plans in the SQL UI now correctly display PhotonWriteStage

When displayed in the SQL UI, write commands in query plans incorrectly showed PhotonWriteStage as an operator. With this release, the UI is updated to show PhotonWriteStage as a stage. This is a UI change only and does not affect how queries are run.

User interface updates

API support:

June 27, 2024

Row Filters and Column Masks in Databricks SQL materialized views and streaming tables are Public Preview

Row filters and column masks in Databricks SQL materialized views and streaming tables are Public Preview. The Public Preview release includes the following changes:

  • You can add row filters and column masks to a Databricks SQL materialized view or streaming table.

    • You can define Databricks SQL materialized views or streaming tables on tables that include row filters and column masks.

    See CREATE MATERIALIZED VIEW and CREATE STREAMING TABLE.

User interface updates

Visualizations:

  • Improved interactivity in displaying tooltips when hovering over pie, scatter, and heatmap charts with many data points.

Catalog Explorer:

  • A revamped Catalog Explorer UI makes it easier to discover and favorite recent Unity Catalog assets from the Quick Access view. The navigation experience has also been simplified, allowing you to explore compute, storage, credentials, connections, DBFS, and management details using the Settings icon Settings in the upper-left corner of the screen. Delta Sharing, Clean Rooms, and External Data now have dedicated pages.

June 6, 2024

Fix for Databricks SQL materialized views and streaming tables

The issue causing ALTER SCHEDULE queries on Databricks SQL materialized views and streaming tables to take effect only after the next REFRESH operation has been fixed. Now, ALTER SCHEDULE queries are applied immediately. See Schedule materialized view refreshes.

Materialized views and streaming tables in Databricks SQL are Public Preview

Materialized views and streaming tables in Databricks SQL are Public Preview and available to all customers. The public preview release includes the following changes:

User interface updates

Dashboards:

  • Account users can now download visualization data from published dashboards

  • Unpublished dashboards can now be published using the Draft/Publish dropdown near the top of a dashboard.

  • Fixed an issue where parameters named limit were not detected.

  • Dashboards now appear in the side navigation for AWS GovCloud.

Visualizations:

  • Improved box-plot rendering in dark mode.

Query insights:

  • For all notebooks attached to SQL warehouses, you can access the query profile by clicking See performance under the cell that contains the query. If the cell includes multiple queries, a link to the query profile for each is provided for each statement.

May 30, 2024

New permission level for SQL warehouses

Can monitor permission allows users to monitor SQL warehouses, including the associated query history and query profiles. The Can monitor permission is now in Public Preview. See Vector search endpoint ACLs.

User interface updates

Dashboard improvements:

  • Added a kebab menu to dashboard widgets for downloading images and data.

  • Users can migrate legacy dashboard query-backed parameters and dropdown list parameters to dashboards.

May 23, 2024

User interface updates

  • You can now select multiple items in the workspace to move or delete. When multiple objects are selected, an action bar appears and provides options to move or delete items. Additionally, you can select multiple items using your mouse and drag them to a new location. Existing permissions on objects still apply during bulk move and delete operations.

  • You can now mark Unity Catalog assets as favorites in the Catalog Explorer and Schema Browser. This includes catalogs, schemas, tables, models, volumes, and functions. Unity Catalog assets that you mark as favorites are easily accessible from the Databricks homepage.

Dashboard updates:

  • Dual-axis combo charts now correctly display bar legends on the right axis and line legends accordingly.

  • Dual-axis charts now correctly show labels on bars.

Visualizations updates:

  • The table editor’s conditional format labels for if and then now support dark mode.

  • The redundant open link icon has been removed from the table editor’s format tooltips.

  • The default font color’s label in the table editor now aligns automatically.

May 16, 2024

Rollout schedule

  • Preview rollout for 2024.25: Completed May 1st

  • Current rollout for 2024.25: Between May 14th and May 21st

Note

An upgrade to the panda Python library (version 2.0.3) caused a breaking change in Databricks SQL version 2024.20. Databricks did not roll out version 2024.20 to the current channel. Instead, the preview channel was upgraded to 2024.25 on May 1, 2025. The current channel rollout goes directly from 2024.15 to 2024.25.

Changes in 2024.25

Data governance

Credential passthrough and Hive metastore table access controls are deprecated.

Credential passthrough and Hive metastore table access controls are legacy data governance models. Upgrade to Unity Catalog to simplify the security and governance of your data by providing a central place to administer and audit data access across multiple workspaces in your account. See What is Unity Catalog?.

Support for credential passthrough and Hive metastore table access controls will be removed in an upcoming DBR version.

SQL language features

The * (star) clause is now supported in the WHERE clause.

You can now use the star (*) clause in the WHERE clause to reference all columns from the SELECT list. For example, SELECT * FROM VALUES(1, 2) AS T(a1, a2) WHERE 1 IN(T.*).

Support for Cloudflare R2 storage to avoid cross-region egress fees since 2024.15

You can now use Cloudflare R2 as cloud storage for data registered in Unity Catalog. Cloudflare R2 is intended primarily for Delta Sharing use cases in which you want to avoid the data egress fees charged by cloud providers when data crosses regions.

Cloudflare R2 storage supports all of the Databricks data and AI assets supported in AWS S3.

See Use Cloudflare R2 replicas or migrate storage to R2 and Create a storage credential for connecting to Cloudflare R2.

User interface updates

The features listed in this section are independent of the SQL Warehouse compute versions described above.

Data discovery updates: The Hive metastore to Unity Catalog update wizard supports upgrading Hive metastore managed tables using all-purpose compute or SQL warehouses. Updating more than 20 tables creates a new notebook that contains the SYNC AND ALTER TABLE commands that perform the conversion.

Dashboard updates:

  • Dual-axis functionality is now available for Area, Bar, Line, and Scatter chart types.

    • When you enable a dual-axis chart, the axis title and range is no longer copied to the secondary axis.

    • The last field identified in the visualization configuration is automatically relocated to the right-side y-axis.

May 9, 2024

SQL Editor fixes:

  • The admin setting Results table clipboard features now applies to the SQL editor’s New result table.

Dashboard improvements:

  • Query-based parameters allow authors to define a list of selectable values that viewers can use as parameters for other visualizations on a dashboard canvas. See Use query-based parameters.

  • Column order in files downloaded from a table widget is now preserved.

  • The table editor now includes hover tooltips that display the names of columns.

  • When switching from other visualization types to a histogram, information encoding is now better preserved.

Dashboard fixes:

  • Fixed an issue where a single grid height filter displayed an unnecessary overflow scrollbar.

  • Fixed an issue that caused incorrectly rendered visualizations on published dashboards where a referenced dataset column was deleted.

May 2, 2024

Serverless SQL Warehouse support expanded: Serverless SQL warehouses are now available in the following regions:

  • ca-central-1

  • ap-northeast-2

See Features with limited regional availability

Dashboard updates:

  • Queries and visualizations can now be copied to a new dashboard from SQL editor. You can still add visualizations to legacy dashboards from the SQL editor. See Edit, download, or add to a dashboard.

  • Dashboards will now maintain a 24-hour result cache to optimize initial loading times. See Dataset optimization and caching.

  • Bar charts with categorical X and quantitative Y are now sorted in Databricks Assistant responses.

  • Corrected migration issue with legacy histogram COUNT (*) to ensure accurate migration.

  • Implemented the ability to mix numeric types and date types in a filter widget.

  • When creating charts, Databricks Assistant now automatically suggests relevant columns as you type.

Visualization updates:

  • User-selected color for tables now persists across light and dark modes in legacy charts.

  • Data truncation logic has been improved to enhance performance in combo, pie, heatmap, and histogram charts.

  • A tick mark is now always displayed at the top of a quantitative axis for basic charts.

April 23, 2024

UI updates:

  • For all Share dialogs in the UI, the All Users group has been renamed to All Workspace Users. The new name more accurately reflects the scope of the group, which has always included users assigned to the workspace. No change is made to group membership as part of this rename.

Dashboard improvements:

  • When a dashboard’s SQL warehouse is starting, a dialog appears to explain the wait time.

  • Scroll position is preserved when switching between the Canvas and Data tabs.

  • Cloning a legacy dashboard to create a Lakeview dashboard now supports some parameter conversion. See Adjust legacy parameters.

  • Relative dates, such as Today, Tomorrow, and 7 days ago, are now supported for date and date time parameters.

  • Number range sliders can be added as filters on a dashboard.

  • Histograms can now display disaggregated data.

  • Scatter plots now support size encoding.

Dashboard fixes:

  • Temporal color encoding can now change color assignments correctly.

Visualization updates:

  • Custom tooltop formats now function correctly for multi-axis charts.

  • The New charts preview tag is removed when users have not changed the toggle in the past 14 days.

April 18, 2024

Lakeview dashboards are generally available

Lakeview dashboards are now the default dashboarding tool. They have been renamed as Dashboards in the UI. Databricks SQL dashboards are now called Legacy dashboards. The names of the related API tools have not changed.

  • Dashboard improvements:

    • Audit logs are available for Lakeview dashboards. See Dashboards events.

    • Data downloaded from dashboards respect applied parameters.

    • Databricks Assistant is enabled on the Data tab without adding datasets first.

    • Stacked bar charts with multiple Y fields can support sorting the X-axis based on the sum of Y-axis values.

    • Toggle between Linear and Log(Symmetric) scale functions in visualization axis menus.

    • The default size of the filter widget is now more compact.

    • The initial load time for the text-entry filter widget has been reduced.

    • Improved automatic chart conversions when migrating from legacy dashboards.

  • Dashboard fixes:

    • The restricted viewing settings warning does not show if the dashboard has been shared with others.

    • The error messages in the Data tab SQL editor are now dark mode enabled.

User interface updates

The features listed in this section are independent of the SQL warehouse compute versions described above.

Improvements:

  • The tooltips on stacked charts now display the stack value and percentage by default.

  • The tooltips for multi-axis charts now highlight the hovered item.

  • Table visualizations for Databricks SQL now adapt a new query result’s data type when edited in the SQL editor.

  • The Catalog Explorer’s Query History table shows a tree-like view for Query Source attribution. You can use this to see which entities have triggered the query statement to run.

April 11, 2024

User interface updates

The features listed in this section are independent of the SQL warehouse compute versions described above.

Improvements:

  • You can now group by percentage when creating visualizations in Databricks SQL and notebooks.

  • For new charts (in Public Preview), you can zoom in along a single axis by clicking and dragging in a straight line parallel to the axis.

  • The Unity Catalog shared cluster Allowlist UI is now generally available. You can access it on the Metastore details page in Catalog Explorer. See How to add items to the allowlist.

  • Forms to create and edit external locations now open as a full page. They include the option to include a storage credential.

Fixes:

  • Corrected an issue for Histogram charts where negative values were erroneously marked as positive.

April 4, 2024

User interface updates

The features listed in this section are independent of the SQL warehouse compute versions described above.

Improvements:

  • Improvements to Histogram charts on Lakeview dashboards.

    • Added support for labels.

    • Bin settings are now retained when switching between different fields.

  • The samples gallery on the dashboard listing page now creates Lakeview dashboards. See Tutorial: Use sample dashboards.

  • Right-clicking on the border of a widget on a Lakeview dashboard opens a context menu.

  • The left-side navigation bar is retained for workspace users viewing published Lakeview dashboards.

  • Filter selections are retained when navigating between published and draft Lakeview dashboards.

  • Column names can now be inserted into the SQL editor when editing a query from the Data tab in a draft Lakeview dashboard.

  • Replacing a Lakeview dashboard keeps the existing dashboard name and replaces the contents.

  • Switching visualizations between heat maps and other chart types now preserves the relevant fields better.

Fixes:

  • Bar charts with color encodings now correctly restrict adding multiple Y-axis fields.

  • Resolved an issue where the Download as PNG button was missing from some visualizations.

  • Corrected formatting for negative big integers previously missing thousands of separators.

  • Fixed incorrect hover line placement when hovering over labels on line charts.

March 28, 2024

Databricks SQL version 2024.15 available

Rollout Schedule

  • Preview rollout for 2024.15: Between February 14th and March 4th

  • Current rollout for 2024.15: Between March 25th and April 8th

Changes in 2024.15

Delta updates

  • Delta UniForm is now generally available: UniForm is now generally available and uses the IcebergCompatV2 table feature. You can now enable or upgrade UniForm on existing tables. See Read Delta tables with Iceberg clients.

  • Recompute data skipping statistics for Delta tables: You can now recompute statistics stored in the Delta log after changing columns used for data skipping. See Specify Delta statistics columns.

SQL language updates

  • Declare temporary variables in a SQL session: This release introduces the ability to declare temporary variables in a session that can be set and then referred to from in queries. See Variables.

  • Native XML file format support (Public Preview): Native XML file format support is now in Public Preview. XML file format support enables ingestion, querying, and parsing of XML data for batch processing or streaming. It can automatically infer and evolve schema and data types, supports SQL expressions like from_xml, and can generate XML documents. It doesn’t require external jars and works seamlessly with Auto Loader, read_files, COPY INTO, and Delta Live Tables. See Read and write XML files.

  • Cloud Fetch is now enabled by default: Cloud Fetch is enabled by default in AWS workspaces with bucket versioning enabled. If you have bucket versioning enabled, Databricks recommends setting a lifecycle policy to remove old versions of uploaded query results automatically. See Cloud Fetch in ODBC (ODBC) and Cloud Fetch in JDBC (JDBC).

Apache Spark SQL updates

Databricks SQL 2024.15 include Apache Spark 3.5.0. Additional bug fixes and improvements for SQL are listed on the Databricks Runtime 14.3 release note. See Apache Spark and look for the [SQL] tag for a complete list.

User interface updates

The features listed in this section are independent of the SQL warehouse compute versions described above.

  • A new overview tab in the entity page of Catalog Explorer shows important metadata like filesize, data source, owner, table schema, and comments.

  • Lakeview dashboard updates:

    • Lakeview dashboards now support parameters. Authors can add parameters to dataset queries from the Data tab and then set parameters on the canvas using single-value selectors and date-pickers. See Work with dashboard parameters.

    • Lakeview dashboards are now supported in the workspace permissions API. See PATCH /api/workspace/workspace/updatepermissions in the REST API reference.

    • Control widgets on the canvas have been renamed to Filter widgets.

    • Combo charts no longer allow disaggregated fields on the x-axis.

    • The Copy link button in the Share dialog now includes parameters stored in the URL.

    • Widgets on published dashboards no longer show borders when hovering.

    • Resolved an issue where the Databricks Assistant and Download PNG buttons overlap with chart visuals.

  • Switching visualizations between heatmaps and other chart types now better preserves the relevant fields.

  • Bar charts with color encodings now correctly restrict adding multiple Y-axis fields.

March 21, 2024

  • Unity Catalog model lineage is now in Public Preview.

The table view in Catalog Explorer now has an Overview tab to describe its primary metadata.

  • SQL warehouses for notebooks, now generally available, allow you to take advantage of fully managed, instant, and scalable compute for your SQL workloads in the rich, collaborative authoring environment of a notebook. See Use a notebook with a SQL warehouse.

The following fixes and improvements apply to Lakeview dashboards:

  • Expanded API support for Lakeview adds the ability to create, get, update, and trash dashboards. See Lakeview in the REST API reference.

  • Added a refresh button for the Catalog browser on the Data tab.

  • Lakeview dashboards now appear before Dashboards in the New menu in the workspace sidebar. On the dashboard listing page, the Lakeview dashboards tab appears to the left of the Dashboards tab.

  • The Databricks Assistant experience for Lakeview has been updated with an input box and suggestions to improve discoverability and help users understand the prompts they can pose.

  • Lakeview visualizations now support median aggregations.

  • Updated the color picker in the Lakeview dashboard visualization editor for a more streamlined user experience when creating tables.

  • Improved pie chart migration to exclusively support scenarios with angle or color definitions.

  • Fixed a bug preventing grouping by charts named count. Charts can now be grouped by fields named count.

  • For bar charts, group and stack layout controls are now hidden when not applicable to the user-selected configuration.

March 14, 2024

For Lakeview dashboards:

  • Histograms now support custom categorical colors.

    • Heatmaps now support quantitative scales.

    • Titles and descriptions are retained when switching between visualization types, including Combo charts.

    • You can now open the underlying dataset associated with a draft dashboard widget by right-clicking on it. The dataset opens in the Data tab.

  • New charts now apply aliases and custom colors for null values in numeric columns.

  • New charts now render tick marks to show the top of the y-axis.

March 7, 2024

  • When viewing a table in Catalog Explorer, the Create button includes an option to create a Lakeview dashboard rather than a Databricks SQL dashboard.

  • Histograms are now available for Lakeview dashboards. Histograms are commonly used to visualize the distribution of a numeric field.

  • When cloning a Databricks SQL dashboard to create a Lakeview dashboard, dataset conversion issues now show as errors in the new widget on the Lakeview dashboard.

  • Color gradients are available when a numerical field is used for a visualization on a Lakeview dashboard.

  • Color gradients are now exposed in the Lakeview dashboard visualization editor when a Color by field is specified.

  • The title and description associated with a visualization no longer appear editable if the viewer lacks editing privileges on a draft Lakeview dashboard.

  • Fixed an issue where tooltips in charts with over 100 series incorrectly showed all series. Now, only the focused series is shown.

  • Reduced typing latency in the SQL editor by 30% through performance optimizations.

  • When managing queries in the SQL editor, moving a query to trash automatically closes the tab.

  • Fixed an issue in the SQL editor where text was accidentally selected when adjusting the side panel width.

February 29, 2024

  • Serverless SQL warehouse support has been added in the following regions:

    • ap-south-1

    • ap-southeast-1

    • ap-northeast-1

    • sa-east-1

    • eu-west-3

    See Databricks clouds and regions for a complete list of supported regions.

  • The schema browser in Catalog Explorer now displays column primary and foreign key constraints.

  • The retention time shown in the Lineage tab in Catalog Explorer has been increased to one year.

  • Tooltips on new charts in notebooks are now always rendered inside the visualization boundary.

  • Learn how to programmatically manage Lakeview dashboards using the REST API. See Manage dashboards with Workspace APIs.

  • Lakeview dashboards now support histograms.

  • Improved sharing and publishing in Lakeview dashboards:

    • Improved share and publish dialogs, allowing safe and easy sharing to any account user.

    • Dashboards opened from the workspace browser show the published dashboard if it exists. Viewers can now also see details of the latest published version, including publisher, time, and credentials.

    • For editors, a new drop-down switcher in the Lakeview Dashboard UI allows you to quickly move between draft and published versions.

Feburary 22, 2024

  • Improvements to the Sample Data tab in the Catalog Explorer table view enable you to sort columns, Copy selected data to your clipboard, and view line numbers. It can now better display special values, like JSON objects, dates, numeric and null values.

  • Lakeview dashboards now support sending periodic PDF snapshots of the dashboard to workspace users and notification destinations. See Schedules and subscriptions.

  • The list of visualization options in the Lakeview dropdown picker is now sorted alphabetically.

  • When copying Databricks SQL dashboards to Lakeview dashboards, widgets that cannot be converted now show the visualization configuration picker instead of an error message.

February 15, 2024

  • The documentation for code-based query filters, such as SELECT action AS 'action::filter', has been removed. Databricks recommends updating queries to remove this pattern.

  • For Lakeview dashboards, pie charts now display equal-sized slices when no angle field is specified.

  • Lakeview now supports combo charts, which combine bar and line charts to show two different values on the same chart.

  • Heatmap charts, which use color intensity to show the magnitude of the correlation between two discrete variables, are now available in Lakeview.

February 8, 2024

  • You can now request access when opening a link to a Lakeview dashboard you do not have permissions on.

  • Lakeview dashboard filters now have explicit All and None options. Authors can choose to hide the All option in single select filters.

  • You can now set minimum and maximum values for axes on Lakeview dashboard charts.

February 1, 2024

Databricks SQL Version 2024.10 Available

Rollout Schedule

  • Preview rollout for 2024.10: Between Jan 30, 2024 and Feb 5, 2024

  • Current rollout for 2024.10: Between Feb 13, 2023 and Feb 20, 2024

Changes in 2024.10

  • Fixed corrupt file handling in DML commands: The DML commands DELETE, UPDATE, and MERGE INTO no longer respect the read options ignoreCorruptFiles and ignoreMissingFiles. When encountering an unreadable file in a table, these commands now fail even if these options are specified.

  • Row-level concurrency is Generally Available and on by default: Row-level concurrency reduces conflicts between concurrent write operations by detecting changes at the row-level. Row-level concurrency is only supported on tables without partitioning, which includes tables with liquid clustering. Row-level concurrency is enabled by default on Delta tables with deletion vectors enabled. See Write conflicts with row-level concurrency.

  • Shallow clone for Unity Catalog external tables (Public Preview): You can now use shallow clone with Unity Catalog external tables. See Shallow clone for Unity Catalog tables.

  • Faster multi-threaded statistics collection: Statistics collection is up to 10 times faster on small clusters when running CONVERT TO DELTA or cloning from Iceberg and Parquet tables. See Convert to Delta Lake and Incrementally clone Parquet and Iceberg tables to Delta Lake.

  • Pushdown filters in the DeltaSource on Delta files: For better utilization, partition filters on Delta tables streaming queries are now pushed down to Delta before rate limiting.

User interface updates

The features listed in this section are independent of the SQL Warehouse compute versions described above.

  • A menu option, Clone to Lakeview dashboard, has been added to the Databricks SQL dashboard UI. You can use this tool to create a new Lakeview dashboard that includes the same queries and visualizations in your existing Databricks SQL dashboards. See Clone a legacy dashboard to an AI/BI dashboard.

  • Bar charts in Lakeview dashboards support stacking bars to normalize to 100%.

  • Fixed a problem where zooming in on a published Lakeview dashboard resulted in focusing on incorrect zoom intervals.

January 24, 2024

  • The Lakeview dashboard canvas automatically adjusts widget placement to remove empty vertical white space between rows when possible.

  • Reduced whitespace between title and description text in Lakeview dashboard visualizations.

January 18, 2024

  • Fixed a rendering issue for visualizations where bar charts showing a single date on the x-axis resulted in a very thin bar. New chart visualizations render as expected.

  • The Lakeview dashboard listing page shows your dashboards by default. You can use filters on that page to access Lakeview dashboards owned by other workspace users.

January 11, 2024

  • Databricks SQL Queries and Dashboard APIs support changing the Run as role setting programatically.

January 4, 2024