Databricks SQL Release notes 2023

The following outlines the improvements and updates in Databricks SQL from January through December 2023.

December 21, 2023

  • The Lakeview Counter visualization type shows colors when comparing BigInt values in the main Value and Target fields.

  • The tooltips that appear when toggling column visibility on tables in Lakeview have been improved. They behave as expected and do not persist.

  • Users can now use Databricks Assistant to create visualizations in Lakeview. See Create visualizations with Databricks Assistant.

  • For new charts, heatmap-type charts respect the reverseY setting.

  • Fixed a rendering performance issue for notebooks with a large number of visualizations.

December 14, 2023

  • Fixed a bug where Lakeview dashboards were not appearing in the Lakeview listing page without a manual page refresh.

  • Use the escape key to cancel the creation of a Lakeview widget when placing it on the canvas.

  • Catalog Explorer now displays Vector Search indexes in the UI as part of the Mosaic AI Vector Search public preview.

December 7, 2023

User interface updates

  • Lakeview dashboards can be added to favorites for quick access.

  • Copy and paste keyboard shortcuts are supported while drafting a Lakeview dashboard. Also, the delete key removes selected widgets.

  • Enhanced Lakeview widget titles and descriptions to prevent clipping text during load.

  • Corrected visualization formatting issue where large integer values were mistakenly displayed as floats.

  • Fixed an issue with Databricks SQL dashboards where expanded chart views were sometimes showing blank charts.

  • Bar charts with quantitative fields on both X and Y axes render more legible data labels.

  • Fixed issue in the SQL Editor so that tables with the word stream in the title no longer conflict with the reserved keyword. These tables now appear as expected in the schema browser and are not error-highlighted.

  • The query history page now supports column resize and column selections.

  • The query history page supports two new columns: Query source and Query source type.

  • BI options, like Tableau and Power BI, are easier to find in Catalog Explorer on eligible pages.

November 30, 2023

User interface updates

The features listed in this section are independent of the SQL Warehouse compute versions described above.

  • Pie charts in Lakeview can now have customized color assignments.

  • Visualization transformations in Lakeview are now retained when switching between compatible field types.

  • Added title settings for Lakeview pie chart angle channels.

  • The Lakeview dataset dropdown is now searchable for easier navigation.

  • Lakeview supports full numerical display for values under 10,000, eliminating abbreviations.

  • Added capability to color-code categorical date fields in Lakeview.

  • Lakeview users can now highlight chart legends with their cursor to copy and paste the values.

  • Pie charts in Lakeview now feature a label toggle option.

  • Standardized a default blue color across all Lakeview visualizations.

  • Lakeview column icons in transformations now consistently match the transformation method used.

  • Controls in Lakeview’s edit panel now auto-wrap for enhanced readability.

  • Released an enhanced color editor for Lakeview visualizations.

  • The control for table font conditions in Lakeview are now wrapped to improve readabilty.

  • Improved dark mode compatibility for labels in new charts.

  • New charts now consistently prioritize label display inside bars.

  • Fixed a bug where some right-click menu actions weren’t working in the SQL Editor.

November 16, 2023

User interface updates

The features listed in this section are independent of the SQL Warehouse compute versions described above.

  • Lakeview widgets are now easier to resize due to a larger resize trigger zone.

  • Workspace admins can now change the owner of a Lakeview dashboard. From edit mode on a Lakeview dashboard:

    • Click Share

    • Click Gear icon

    • Click Assign new owner

  • Users can toggle labels on or off in Lakeview.

Visualizations:

  • New chart labels now strongly prefer being inside a bar when possible.

  • New chart labels now appear properly on stacked bars that are wide enough to show the whole label.

  • Label colors inside bar charts are now more consistent.

November 9, 2023

Changes in 2023.50:

Highlights:

SQL Language updates: The following builtin functions have been added: - `from_xml`: Parses an XML STRING into a STRUCT. - `schema_of_xml`: Derives a schema from an XML STRING. - `session_user`: Returns the logged-in user. - `try_reflect`: Returns NULL instead of the exception if a Java method fails.

The following builtin functions have been enhanced:

  • function invocation. Table arguments to functions support partitioning and ordering: You can now use PARTITION BY and ORDER BY clauses to control how table arguments are passed to a function.

  • `mode`: Support for an optional parameter forcing a deterministic result.

  • `to_char`: New support for DATE, TIMESTAMP, and BINARY.

  • `to_varchar`: New support for DATE, TIMESTAMP, and BINARY.

  • `array_insert()` is 1-based for negative indexes: The array_insert function is 1-based for both positive and negative indexes. It now inserts a new element at the end of input arrays for the index -1.

User interface updates

The features listed in this section are independent of the SQL warehouse compute versions described above.

Visualizations:

  • Tooltips for new charts show only hovered items for high-cardinality charts.

  • New charts automatically align dual-axis charts at zero.

  • Charts changed rotated label and axis text angle direction from -90 to 90 degrees.

  • Charts now use the label’s width to decide rotation.

Lakeview Dashboards:

  • Fix: Table rows in Lakeview dashboards no longer add vertical padding when only a small number of rows are returned.

  • Cloning a new Lakeview dashboard widget first attempts to place the clone to the right of the original, provided there is sufficient space on the canvas. If insufficient space is available, the clone is placed below the original.

  • You can now reassign the owner of a Lakeview dashboard via the Share dialog.

SQL Editor: Autocomplete is now less aggressive and dismisses automatically at the end of statements

November 2, 2023

Improvements:

The following are improvements to Lakeview dashboard lineage:

  • Added distinct icons for tables, views, and materialized views.

  • Added support for vertical scrolling in the event of many upstream data sources.

  • Improved error message when viewer lacks permissions on the upstream object.

  • Clarified messaging around sample data tables and HMS data.

  • Added key value tags to upstream data sources.

Fix:

  • Fixed an issue in new charts that prevented rendering after renaming a series with boolean values.

October 26, 2023

Changes in Databricks SQL version 2023.45

Highlights:

Query optimizations:

  • Removed the outer join if they are all distinct aggregate functions. SPARK-42583

  • Optimized the order of filtering predicates. SPARK-40045

SQL function updates:

User interface updates

Improvements:

  • The Lakeview SQL editor now supports keyboard shortcuts to run queries.

  • Filters in Lakeview now list quick relative date range selections for the last 7, 14, 28, 30, 60, and 90 days.

  • Periods in data field names no longer result in blank charts.

  • The legacy schema browser now works with Unity Catalog.

  • Implemented performance improvements to load only the active tab in the SQL Editor, decreasing initial load time by up to 20%.

  • AI-generated table comments in Catalog Explorer are now generally available.

Fixes:

  • Bars on temporal bar charts are now centered over the date tick mark.

  • Data label templates with aggregate expressions now use the proper numeric formatters.

  • Zooming on new charts now works with a scale set to categorical using temporal data.

New article:

  • Released a new article showing all of the Lakeview visualizations, including screenshots and notes showing how to recreate each the visualizations in each screenshot. See Dashboard visualizations types.

October 19, 2023

Improvements:

  • Drag and drop in schema browser is now available.

  • The Select Table modal in Lakeview dashboards makes it easier to select from all tables in a catalog or schema and now uses autocomplete search predictions as you type.

Fixes:

  • Legend selection tooltip instructions specify Mac users use cmd and Windows users use ctrl.

  • Enhanced visibility of truncated messages in Lakeview dashboards when rendered data exceeds limits.

  • Charts with truncated data display consistent colors as seen in the editor.

October 12, 2023

Improvements:

  • Text filter values containing special characters now filter correctly in Lakeview dashboards.

  • New charts support @@name data labels on scatter plots.

  • Customized percentage formats apply to grouped chart tooltips in new charts.

  • Workspace binding extension UI is GA.

Fixes:

  • Improved label rendering on new charts to prevent labels from spilling outside chart boundaries.

  • Increased the contrast of tick and grid lines on new charts for improved visibility.

  • Increased the axis label spacing to improve readability on new charts.

October 5, 2023

Improvements:

  • Accurate tooltips have been added for publish mode actions and date lineage in Lakeview dashboards.

  • Conditional formatting and link templates in Lakeview table visualizations now support hidden columns.

  • Optimized label positioning for wide-bar temporal charts to enhance clarity in new charts.

  • Counter visualization in Lakeview retains its transformations even after other fields are removed, ensuring consistency.

  • Hovering over a series in a chart now dims the surrounding series in the tooltip to improve readability in new charts.

  • New charts using percentage values now display tooltips with absolute values.

  • Added autocomplete support for creating volumes.

  • Closing a non-active tab no longer switches tabs.

  • Selected run clearly indicates when highlighting text.

Fixes:

  • Improved error message wording in Pivot Tables when the data is truncated.

  • Fixed a rendering error in Pivot Tables where colors were not showing when using BigInt data types.

  • When downloading PNGs in new charts with numerous legend items, removed the color symbol for overflow legend entries.

  • Lines in new charts will maintain a consistent thickness even at the topmost view boundary.

  • In Lakeview, if no dataset exists, the dataset picker in the Canvas is empty.

  • Delta Live Tables are properly detected by the SQL Parser and won’t show-up as invalid tables in Schema Browser.

  • Tooltips was added to sidebar.

September 28, 2023

Improvements:

  • Published Lakeview dashboards now have a refresh button.

  • Improved error messages for users who do not have access to a Lakeview dashboard.

  • Filter configuration in Lakeview dashboards now lists valid fields at the top of the selection list.

  • Downloading a chart as a PNG from a Lakeview dashboard now retains the title and description.

  • Delta tables history improvement in Catalog Explorer includes adding filters for date range, user, and operation type, as well as sortable columns, and inline links to associated Jobs and Notebooks.

  • Dark mode support added across legends, tooltips and table visualization.

Fix:

  • Filter selections are no longer cleared when refreshing a Lakeview dashboard.

September 21, 2023

Improvements:

  • Pivot Table rendering performance has been improved.

  • New DuBois pattern for lineage tabular views in the UI.

September 14, 2023

Improvement:

September 7, 2023

Databricks SQL version 2023.40 available

Rollout schedule

  • Preview rollout for 2023.40: Between Sep 5, 2023 and Sep 11, 2023

  • Current rollout for 2023.40: Between Sep 18, 2023 and Sep 25, 2023

Changes in 2023.40:

  • Tags are now available with Unity Catalog.

  • Databricks Runtime returns an error if a file is modified between query planning and invocation.

  • Databricks ODBC/JDBC driver support.

  • Enable time series column labeling.

  • New bitmap SQL functions.

  • Improved encryption functions.

  • Unity Catalog support for REFRESH FOREIGN.

  • INSERT BY NAME is now supported.

  • Share materialized views with Delta Sharing.

User interface updates

Improvements:

  • New charts are now available, featuring faster render performance, beautiful colors, and improved interactivity. See New chart visualizations in Databricks.

  • In the graph view of Query Profile, you can now view the Join type on any node containing a join in the query plan.

  • Data Explorer is renamed to Catalog Explorer to recognize the fact that you can use it to work with all securable objects in Unity Catalog, not just data objects. See What is Catalog Explorer?.

  • The Databricks SQL Statement Execution API is now GA with Databricks SQL Version 2023.35 and above. The API allows you to submit SQL statements for execution on a Databricks SQL warehouse, check the status and fetch results, or cancel a running SQL statement execution. See Statement Execution API.

August 31, 2023

New feature:

August 24, 2023

Improvement:

  • Autocomplete stops suggesting recommendations after you press the spacebar.

  • Schema Browser no longer sees live Delta Live Tables as broken tables.

August 16, 2023

Improvement:

  • The Catalog dropdown in the SQL editor now closes when you switch tabs. Previously, when you switched tabs, the dropdown would remain open.

August 10, 2023

Improvement:

  • Autocomplete now supports the new syntax for setting Unity Catalog tags. For information on commands, see SQL language reference.

August 3, 2023

Improvements:

  • The underlying Monaco Editor now uses version 37.1.

  • Autocomplete support for SHOW ARCHIVED FILES FOR Delta commands.

July 27, 2023

Improvements:

  • The SQL editor is now compatible with Windows newline characters, ensuring that query formatting works as expected in all cases.

  • You can open the query profile navigation from notebook results. For queries, run your query with SQL warehouse.

July 20, 2023

Databricks SQL version 2023.35 available

Rollout schedule

  • Preview rollout for 2023.35: Between Jul 18, 2023 and Jul 24, 2023

  • Current rollout for 2023.35: Between Jul 31, 2023 and Aug 8, 2023

Changes in 2023.35:

  • Enhanced reliability for VACUUM with shallow clone in Unity Catalog.

  • Support for Python UDFs in SQL.

  • Delta Lake UniForm for Iceberg is in Public Preview.

  • Delta Lake liquid clustering is in Public Preview.

  • Archival support for Delta Lake.

  • IDENTIFIER clause support.

  • Unity Catalog support for Python and Pandas User-Defined Functions (UDFs).

Improvement:

  • Table popularity in Catalog Explorer is available to all Unity Catalog users.

July 13, 2023

Improvement:

  • Unity Catalog users can now view additional table insights in Catalog Explorer such as frequently joined tables, frequent users of a given table, and frequently used notebooks and dashboards.

June 22, 2023

Public Preview:

  • Databricks SQL now supports large language models (LLMs) hosted on model serving endpoints. Call ai_query() to access your LLM. This function is only available in Public Preview on Databricks SQL Pro and Serverless. To participate in Public Preview, submit the AI Functions Public Preview enrollment form.

June 15, 2023

New feature:

  • SQL tasks in Jobs are now generally available. You can orchestrate Queries, Dashboards, and Alerts from the Jobs page. See SQL task for jobs.

  • A new schema browser is now in Public Preview, featuring an updated UX, a For You tab, and improved filters. The schema browser is available in Databricks SQL, Catalog Explorer, and notebooks. See Browse data.

June 8, 2023

DBSQL version 2023.30 available

Changes in 2023.30

  • New SQL built-in functions, such as array_prepend(array, elem), try_aes_decrypt(expr, key [, mode [, padding]]), and sql_keywords() .

  • You can now use shallow clone to create new Unity Catalog managed tables from existing Unity Catalog managed tables. See Shallow clone for Unity Catalog tables.

  • You can now use CLONE and CONVERT TO DELTA with Iceberg tables that have partitions defined on truncated columns of types int, long, and string. Truncated columns of type decimal are not supported.

  • START VERSION is now deprecated for ALTER SHARE.

User interface updates

Fix:

  • Counter visualization now supports formatting when using Big Integers.

June 1, 2023

Improvements:

  • Binary data will now render as a hex string when using the Arrow format.

  • In the SQL Statement API, the CSV format is now supported for the EXTERNAL_LINKS disposition. This allows clients to extract up to 100 GiB of data in CSV format with pre-signed URLs, whereas the INLINE limit for JSON is 16 MiB.

May 29, 2023

New feature:

May 25, 2023

Improvements:

  • You can now toggle the autocompletion result panel.

  • Disable the enter key so it no longer accepts autocomplete suggestions. Under DBSQL User Settings, click Editor Settings, then New Editor settings. Turn off *Enter key accepts autocomplete suggestions*.

Fixes:

  • Sorted table headers now have colors.

  • Chart lines now render correctly.

May 18, 2023

Improvement:

  • In the SQL Statement API, the EXTERNAL_LINKS disposition now supports the JSON_ARRAY format. You can extract up to 100 GiB of data in JSON format with pre-signed URLs. The INLINE limit for JSON is 16 MiB.

May 11, 2023

New feature:

  • Schema Browser is now generally available in Catalog Explorer.

Improvements:

  • On-hover table detail panel showing is less sensitive.

  • The escape key now closes the autocomplete panel.

  • View definitions now have syntax highlighting in the Catalog Explorer details tab.

Fixes:

  • Pivot tables now correctly render on Windows devices.

  • Completion suggestions now properly follows the case of the first keyword.

May 4, 2023

Databricks SQL Version 2023.26 Available

Rollout Schedule

  • Preview rollout for 2023.26: Between April 19, 2023 and April 25, 2023

  • Current rollout for 2023.26: Between May 3, 2023 and May 10, 2023

Changes in 2023.26

  • Photon returns an error if a file is modified between query planning and execution.

  • New features and extended support for Predictive I/O features. See Databricks Runtime 13.0 (EoS).

  • Use the Databricks connector to connect to another Databricks workspace.

  • CREATE TABLE LIKE feature for Delta tables.

  • New metadata column fields denoting file block start and length.

  • New H3 geospatial functions. See H3 geospatial functions.

  • New SQL built-in functions. See Databricks Runtime 13.0 (EoS).

User interface updates

Improvements:

  • Administrators can change warehouse owners using the user interface or the API. See Manage a SQL warehouse.

  • Catalog Explorer now displays account service principals in user lists for assets in Unity Catalog. For example, account service principals are visible when editing privileges or changing owners in Catalog Explorer.

  • Custom chart labels support the ability to reference any column within the dataset.

  • Dashboard filters now load column names, even when using queries that don’t have catalog or schema info.

April 27, 2023

Improvements:

  • The SQL editor now relies on the Monaco editor for a more reliable editing experience.

  • SQL History list page (Queries) now includes the Dubois Design System.

April 20, 2023

Improvements:

  • Introduces new pivot tables that allow you to aggregate more than 64k results.

  • Databricks SQL tables and visualizations now support BigInt, 38bit Decimals, and non UTF-8 characters. For numbers, the default setting is now user-defined digit precision.

  • Autocomplete now suggests frequent past joins for Unity Catalog tables, powered by Unity Catalog lineage data in Databricks Runtime 12.0 and above.

  • Cloud Fetch is enabled by default in AWS workspaces with bucket versioning enabled. If you have bucket versioning enabled, Databricks recommends setting a lifecycle policy to automatically remove old versions of uploaded query results. See Cloud Fetch in ODBC (ODBC) and Cloud Fetch in JDBC (JDBC).

New feature:

  • Return text generated by a selected large language model (LLM) given the prompt with ai_generate_text. This function is only available as public preview on Databricks SQL Pro and Serverless. To participate in the public preview, populate and submit the AI Functions Public Preview enrollment form.

April 13, 2023

New feature:

  • The TIMESTAMP_NTZ type represents values comprising of fields year, month, day, hour, minute, and second. All operations are performed regardless of time zone. See TIMESTAMP_NTZ type.

Improvements:

  • Users can now send formatted results within alerts by using the QUERY_RESULT_TABLE tag in a custom alerts message.

  • Users can now view the file data size for Unity Catalog tables in Catalog Explorer.

April 6, 2023

Databricks SQL Version 2023.20 Available

Rollout Schedule

  • Preview rollout for 2023.20: Between Mar 15, 2023 and Mar 23, 2023

  • Current rollout for 2023.20: Between Mar 27, 2023 and Apr 3, 2023

Changes in 2023.20

  • Delta Lake schema evolution supports specifying source columns in merge statements.

  • Remove all NULL elements from an array using array_compact.

  • To append elements to an array, use array_append.

  • To anonymize sensitive string values, use the mask function.

  • Common error conditions now return SQLSTATE.

  • Invoke table-valued generator functions in the regular FROM clause of a query.

  • Use the from_protobuf and to_protobuf functions to exchange data between binary and struct types. See Read and write protocol buffers.

  • Improved consistency for Delta commit behavior for empty transactions relating to update, delete, and merge commands.

  • Behavior change

    • The lateral column alias feature introduces behavior changes during name resolution. See Behavior changes.

User interface updates

Improvements:

  • Catalog Explorer now supports Hive metastore delete.

  • The message for the visualization editor’s initial state changed from Missing required columns to No Data.

April 3, 2023

New feature:

March 30, 2023

Improvements:

  • On the warehouse monitoring page, you can now view the query history for the selected time range along with your scaling charts. You can also view currently running and queued queries, active SQL sessions, the warehouse status, and the current cluster count. See Monitor a SQL warehouse.

  • Map clustering is now off by default in Marker maps.

  • Tooltips for visualization truncation and render limits have been added.

Fixes:

  • Charts now respect the original order when sorting is disabled for the axis values and the chart has the group by column.

March 23, 2023

Improvements:

  • Visualizations now support time binning by week.

  • Total now calculates all data beyond the 100 X 100 cells up to 64k results in notebooks pivot tables.

  • Users can now format cell values in the new notebooks pivot table.

  • File, query, and feature store lineage are available.

March 16, 2023

Improvement:

  • Support for GROUP BY ALL syntax.

March 9, 2023

New feature:

  • The Databricks SQL Statement Execution API is now available in Public Preview. Statement Execution provides endpoints that are running SQL statements on a Databricks SQL warehouse. It can also check the status, fetch results, and cancel a running SQL statement.

Improvement:

  • The SQL admin console has been combined with the general admin settings to create a unified experience for admin users. All SQL admin settings are now accessed from the admin console.

  • Alert destinations are now called notification destinations.

Fixes:

  • Tables no longer display two scrollbars.

  • Blank widget titles no longer get overwritten on dashboards.

March 2, 2023

Fix:

  • You can now use relative URLs in tables to access dashboards.

February 23, 2023

Improvement:

  • TINYINT is now supported in results tables in notebooks.

Fix:

  • Fixed a bug where scrolling on the create dashboard filter modal resulted in an error.

February 16, 2023

Improvements:

  • Data labels on charts now show more characters to avoid truncating descriptions.

  • Autocomplete now recognizes range() and Python UDF create functions.

  • Autocomplete now avoids initiating snippets on decimals and within code comments.

Fixes:

  • Users can now zoom in on maps.

  • In notebooks, colors are now correctly assigned to charts.

February 2, 2023

Improvements:

  • Support for DESCRIBE DETAILS in the editor.

  • Improved schema browser loading speed.

  • You can now view a list of possible columns on the side panel of a SELECT *.

January 26, 2023

Improvement:

  • Your query’s error messages now include links to the related documentation topic that describes the error.

January 19, 2023

Improvements:

  • You can now find a What’s New panel that highlights key enhancements. You can open and close this panel by clicking the gift icon.

  • Admins can now change ownership of SQL warehouses.

  • You can now filter across multiple visualizations in a dashboard by clicking Add > Filter and selecting the query and columns for filtering.

January 12, 2023

Improvements:

  • Visualization widget titles on dashboards are now formatted as Visualization name - Query name by default.

  • Added H3 Geospatial functions to the inline panel reference.

  • Added inline references for SQL syntax like CREATE TABLE and OVER.

Fixes:

  • Sorting and reverse toggles are now available when the X-axis scale is set to automatic.

  • Heat maps and pivot tables are now responsive for certain edge cases.

  • The Categorical Y-axis tick marks are now sorted by default.

  • Query drafts are no longer lost after doing a browser refresh.