Amazon
Web Services
Microsoft Azure
Google Cloud PlatformDatabricks SQL Release notes 2022
The following outlines the improvements and updates in Databricks SQL from January through December 2022.
December 8, 2022
Databricks SQL alerts now support alerts for aggregations of query result columns such as
SUM,COUNT, andAVG.The default visualization title is now ‘VisualizationName - QueryName’ when creating new widgets on dashboards.
November 17, 2022
Alerts:
Chatworks, Mattermost, and Google Hangouts Chat are no longer notification destinations.
Improvement:
The y-axis now displays numbers as percentages when checking percent values.
The workspace administrator setting to disable the upload data UI now applies to the new upload data UI. This setting applies to the Data Science & Engineering, Databricks Mosaic AI, and Databricks SQL personas.
Fixes:
Fixed an issue in Databricks SQL alerts where comparing against null values evaluated incorrectly.
Fixed an issue where scrollbars on pivot tables disappeared.
Fixed an issue where the schema browser couldn’t resize with overflowed tabs.
November 10, 2022
Improvement:
You can now create a dashboard filter that works across multiple queries at the same time. In Edit dashboard mode, choose Add, then Filter, then New Dashboard Filter.
Autocomplete now supports
CREATE MATERIALIZED VIEW.
Fix:
Fixed an issue where scrolling to the end of a set of dashboard paged results would send an error.
Fixed an issue where switching from a stacked bar chart to a line chart kept the stacking property.
Fixed duplicated fetch calls.
November 3, 2022
Improvement:
When requesting access in Databricks SQL, the default permission is now “can run”.
Fixes:
Fixed an issue where sorting by
created_atusing the Queries and dashboards API did not return the correct sort order.Fixed an issue where columns containing URLs with HTML formatting had overly wide column widths.
Fixed an issue where
WHEREkeyword wasn’t highlighted.
October 27, 2022
Improvements:
The row limit for downloading query results to Excel has been increased from 64,000 rows to 100,000 rows. CSV & TSV download limits remain unchanged (about 1 GB of data).
Autocomplete now supports
LISTsyntax, URLs, and credentials.Consolidated and modernized Fix-me suggestion panels.
A new warehouse type, Databricks SQL Pro, is introduced for Databricks SQL. This warehouse type enables a Databricks SQL Warehouse to use Jobs integration, query federation, geospatial features, and predictive IO.
Fixes:
Fixed an issue where the warning banner in the editor overlapped full-height visualizations.
Fixed an issue where table column width was not preserved when columns of the table were moved.
Fixed an issue where the link to the dashboard in pop-up notifications was broken if a visualization was added from the SQL editor.
October 20, 2022
Improvements:
You can now find the query progress bar in the footer and the editing a visualization action in the kebab menu.
Autocomplete now supports Delta time travel, and provides column autocomplete when defining a foreign key.
Fix:
Fixed an issue where adding multiple visualizations to a dashboard in quick succession would result in visualizations not appearing on the dashboard.
October 13, 2022
Improvements:
You can now remove reported error messages.
COMMENT ONinstruction is now supported in the editor.You can now use Cmd+P or Ctr+P (for PC) as a shortcut for Top search. Use Cmd+I or Ctr+I (for PC) for Add parameters.
October 11, 2022
Improvements:
The add data UI provides access to common data sources configurations and file upload UIs. See Upload files to Databricks.
You can uploading small files to Delta Lake using a UI. See Create or modify a table using file upload.
October 6, 2022
Improvements:
EXTERNALis now a reserved table property. CommandsCREATE TABLE ... TBLPROPERTIESandALTER TABLE ... SET TBLPROPERTIESfail ifEXTERNALis specified in the properties.The
strfmtinformat_string(strfmt, obj, ...)andprintf(strfmt, obj, ...)no longer supports the use of0$as the first argument. The first argument should be referenced by1$when using an argument index to indicate the position of the argument in the argument list.Pie chart segments now have a thin border to delineate different segments.
You can now use Cmd+I (for Mac) or Ctr+I (for PC) as a shortcut for Add parameter. Use Cmd+P or Ctr+P (for PC) as a shortcut for Global search.
A feedback button is available for good or bad query error messages.
Fix me suggestions are now available as Quick fix.
Fixes:
lpadandrpadfunctions now work correctly withBINARYstring inputs. The output oflpadandrpadforBINARYstring inputs is now aBINARYstring.Fixed an issue where manual alert refreshes were not working.
Rolled back changes to automatic counter sizing to fix formatting issues.
September 29, 2022
Improvements:
You can now request access to Databricks SQL Queries, alerts, and dashboards from owners of those assets.
You can now filter by query duration and statement type on the Query history page.
To use the query duration filter, enter an amount of time and choose a time unit. The history result returns queries that run longer than the time specified.
To use the statement type filter, choose a statement type from the dropdown. The history result returns queries containing that statement.
Fix:
Fixed an issue where not all supported HTML tags were working in custom alert templates. All tags are available as documented.
Fixed an issue where Visualization notification toasts. For example, double clicking to zoom out on a visualization was previously not showing.
Fixed an issue where swapping the axes on a chart was not reflected in the chart name.
September 22, 2022
Improvements:
Counter widgets of the same size will have the same font sizing when multiple counter widgets are displayed on a dashboard.
Updated combination charts so that when using dual axis, only the same chart type (e.g., line, bar) can be used on the same axis. Series aliases are also applied to the axis.
Added autocomplete support for surrogate keys and
LISToperations.
Fix:
Fixed issue where text parameters did not accept
Nullas a valid value.
September 15, 2022
Fix:
Fixed an issue where viewing query history from the SQL warehouses listing page did not work.
September 8, 2022
Improvement:
Introducing the new ‘Open Source Integrations’ card in DSE/SQL homepages that displays open source integration options such as Delta Live Tables and dbt core.
Fix:
Fixed an issue where parameter dropdown menus were blocked by the visualization tab.
September 1, 2022
Improvements:
Introducing a new simplified UI to add parameters and filters. Choose <button>+</button> and choose to add a filter or parameter.
The parentheses of SQL tokens, such as ‘OVER()’ now get autocompleted.
Fixes:
Fixed an issue where viewing the dashboard in full-screen ignored the color palette.
Fixed an issue where typing quickly and then using the Run shortcut ran the previous query text, instead of the newly typed query text.
Fixed issue where using the keyboard command, ctrl+enter to run queries would submit duplicate queries.
August 25, 2022
Fix:
Fixed an issue where dashboard filters were not updating when query parameters changed.
August 18, 2022
For Databricks SQL, Unity Catalog (Public Preview) is available in the preview channel. For more information, see What is Unity Catalog?.
Documentation: Alerts API documentation has been released.
Visualizations: Users can now set default values for date filters. Any time the filter is refreshed on a query or dashboard, the default value is applied.
Fixes:
Fixed an issue where apply changes did not work if a dashboard was still reloading.
Fixed an issue where columns were too narrow when a query returns no results.
August 11, 2022
Improvements:
Users can receive emails when their refreshes fail. To enable such notifications, navigate to the SQL settings tab of the admin console. Under Failure Emails, choose the type of object (Query, Dashboard, or Alert) for which you wish to receive failure notifications. Failure reports are sent hourly.
Visualizations
Introducing a new, modern color palette for visualizations and dashboards. To change a dashboard to the new color palette, go to your dashboard, click on <button>Edit</button> -> <button>Colors</button> -> <button>Import</button> and select the Databricks Color Palette. SQL Admins can also set the new color palette as the default option for a workspace by going to <button>Settings</button> -> <button>QL Admin Console</button> -> <button>Workspace Colors</button> -> <button>Import</button> and selecting the new palette.
Fixes:
Fixed an issue where previously selecting <button>Apply Changes</button> to apply a filter did not work if a query was already being executed.
August 4, 2022
Improvements:
On cloning a dashboard, there is now an option for whether or not queries should be cloned as well.
Tab content is synced across browser tabs. The state of your query will now be in sync across all browser tabs. This means that if you are working on query1 in browser tab 1 and then switch to browser tab 2, you’ll see query1 in the state you left it in while in the original browser tab.
Fix:
Labels for empty strings in pie chart now reflect that the string is empty rather than the index of the value.
July 28, 2022
Alerts
Custom alert email templates have been updated to disallow certain HTML tags that may pose a security risk. Disallowed HTML tags and attributes are automatically sanitized. For example, <button> is a disallowed HTML tag, so instead of rendering a button, the text “button” displays. See Alerts for the list of allowed HTML tags and attributes.
Users can now subscribe other users to alerts without needing to create a notification destination, which requires admin permissions.
Downloads: Users can now download up to approximately 1GB of results data from Databricks SQL in CSV and TSV format, up from 64,000 rows previously.
Visualizations
You can now edit visualizations directly on the dashboard. In edit mode, click on the kebab menu and select Edit visualization to begin editing the visualization.
When downloading results associated with a visualization leveraging aggregations, the downloaded results are also aggregated. The option to download is moving from bottom kebab to the kebab associated with the tab. The downloaded results are from the most recent execution of the query that created the visualization.
SQL editor: Results tables now display a message when data displayed by the in-browser table has been limited to 64,000 rows. TSV and CSV download will still be up to approximately 1GB of data.
Query filters:
Query filters have been updated to work dynamically on either client- or server-side to optimize performance. Previous query filters (now legacy) operated client-side only. Users can still use legacy filters with the
::syntax, if desired.The updated filters are simpler: Users click a +Add Filter button and select a column from a dropdown. Previously, users had to modify the query text directly.
Relevant values are highlighted to make it easier to see which selections within a filter will return results given other filter selections.
Query history: Query details in Query History now show the Query Source, which is the origin of the executed query.
July 21, 2022
Notifications on share: Users will now be notified by email whenever a dashboard, query, or alert is shared with them.
Enhanced SQL editor experience via the new embedded editor toolkit
Live syntax error highlighting (for example, wrong keyword, table does not exist, and suggestions for fixing the error)
In context help: on hover (for example, full table name, detailed Function panel) and inline execution error messages (for example, highlight row with error post execution)
Intelligent ranking of suggestions (for example, parameter autocompletion, ranking formula, less noisy matching)
July 14, 2022
You can now upload TSV files using the
Create TableUI in addition to CSV files.Databricks SQL now provides the option to notify users by email whenever a dashboard, query, or alert is shared with them.
Visualization tables now optionally include row numbers displayed next to results.
When you select a geographic region for the Chloropleth visualization, you now get inline hints for accepted values.
June 23, 2022
SQL endpoint name change: Databricks changed the name from SQL endpoint to SQL warehouse because it is more than just an API entry point for running SQL commands. A SQL warehouse is a computation resource for all your data warehousing needs, an integral part of the Databricks platform. Compute resources are infrastructure resources that provide processing capabilities in the cloud.
For Choropleth visualizations, the Key column and Target field selections in the visualization editor have been renamed to Geographic Column and Geographic Type. This renaming for understandability does not introduce any behavior changes to new or existing Choropleths.
The limit 1000 query option has moved from a checkbox in the SQL query editor to a checkbox in the run button.
Cached queries in Query History table are now marked with a Cache tag.
Manually refreshing a dashboard uses the dashboard’s warehouse (if available) instead of each individual query’s warehouse.
Refreshing an alert always uses the alert’s warehouse, regardless of the Run as Viewer/Owner setting.
June 9, 2022
When you hover in the endpoint selector, the full endpoint name is displayed as a tooltip.
When you filter in the SQL editor schema browser, the search term is now highlighted in the search results.
The Close All dialog box in the SQL editor now displays a list of unsaved queries.
To reopen the last closed tab in the SQL editor, use this new keyboard shortcut:
<Cmd> + <Shift> + <Option> + TYou can now add data labels to combination charts.
The list of visualization aggregations operations now includes variance and standard deviation.
May 26, 2022
Authoring improvements:
You can now bypass aggregations when you author visualizations. This is particularly useful when your query already includes an aggregation. For example, if your query is
SELECT AVG(price_per_sqft), isStudio, location GROUP BY location, isStudio, the chart editor previously required explicitly specifying another layer of aggregation.When you author dashboards, you now have the ability to:
Duplicate textbox widgets
Expand the size of the edit textbox panel
The default aggregation for the error column when you author visualizations is standard deviation.
Fixes:
Edit actions for visualizations are only available when the dashboard is in edit mode. Edit actions are no longer available as a view mode action.
When you create a new query, it opens in a tab to the immediate right of the tab in focus rather than at the end of the list.
The open query modal shows which query is already open and provides the option to switch focus to that query tab.
The Sankey & Sunburst charts no longer treat 0 as null.
May 19, 2022
Fixed issue: When you have the focus of the SQL editor open on a specific visualization tab and share the link to another user, the user will have the same focus in the SQL editor when they click the shared link.
Improvements:
Microsoft Teams is now a supported notification destination.
The Date Range, Date and Time Range, and Date and Time Range (with seconds) parameters now support the option to designate the starting day of the week, with Sunday as the default.
May 12, 2022
Visualizations now support time binning directly in the UI. You can now easily switch between yearly, monthly, daily, or hourly bins of your data by changing a dropdown value rather than adding and modifying a
date_trunc()function in the query text itself.Dashboards now have color consistency by default. If you have the same series across multiple charts, the series is always colored the same across all charts – without requiring any manual configuration.
May 3, 2022
When sharing a dashboard with a user or group, we now also provide the ability to share all upstream queries used by visualizations and parameters.
When you do not have permission to share one or more of the upstream queries, you will receive a warning message that not all queries could be shared.
The permissions granted when sharing a dashboard do not override, negate, or expand upon existing permissions on the upstream queries. For example, if a user or group has CAN RUN as Owner permissions on the shared dashboard but only has Run as Viewer permissions on an upstream query, the effective permissions on that upstream query will be Run as Viewer.
April 27, 2022
Your dashboard layout is now retained when exporting to PDF on demand and generating scheduled subscription emails.
March 17, 2022
Charts includes a new combination visualization option. This allows you to create charts that include both bars and lines.
March 10, 2022
Unity Catalog (Preview) allows you to manage governance and access to your data at the level of the account. You can manage metastores and data permissions centrally, and you can assign a metastore to multiple workspaces in your account. You can manage and interact with Unity Catalog data and objects using the Databricks SQL Catalog Explorer or the SQL editor, and you can use Unity Catalog data in dashboards and visualizations. See What is Unity Catalog?.
Note
Unity Catalog requires SQL endpoints to use version 2022.11, which is in the preview channel.
Delta Sharing (Preview) allows you to share read-only data with recipients outside your organization. Databricks SQL supports querying Delta Sharing data and using it in visualizations and dashboards.
Delta Sharing is subject to applicable terms that must be accepted by an account admin to enable the feature.
Each time a dashboard is refreshed manually or on a schedule, all queries in the dashboard and upstream, including those used by parameters, are refreshed. When an individual visualization is refreshed, all upstream queries, including those used by parameters, are refreshed.
March 3, 2022
The cohort visualization has been updated such that cohorts are interpolated from min and max values rather than 0 and 100. It’s now much easier to distinguish cohorts within the actual range of data available. Previously, if all numbers were close together, they used the same color. Now, numbers that are close together are more likely to use different colors because the cohort is divided from the max to min range to form each series.
It’s easier to see whether a dashboard subscription schedule is active or paused. When you click Subscribe, if the dashboard subscription schedule is paused, the message This schedule has been paused appears. When a dashboard subscription schedule is paused, you can subscribe or unsubscribe from the dashboard, but scheduled snapshots are not sent and the dashboard’s visualizations are not updated.
When you view Query History, you can now sort the list by duration. By default, queries are sorted by start time.
February 24, 2022
In Catalog Explorer, you can now view the permissions users or groups have on a table, view, schema, or catalog. Click the object, then click Permissions and use the new filter box.
February 17, 2022
Visualizations just became a little smarter! When a query results in one or two columns, a recommended visualization type is automatically selected.
You can now create histograms to visualize the frequency that each value occurs within a dataset and to understand whether a dataset has values that are clustered around a small number of ranges or are more spread out.
In both Query History and Query Profile, you can now expand to full width the query string and the error message of a failed query. This makes it easier to analyse query plans and to troubleshoot failed queries.
In bar, line, area, pie, and heatmap visualizations, you can now perform aggregation directly in the visualization configuration UI, without the need to modify the query itself. When leveraging these new capabilities, the aggregation is performed over the entire data set, rather than being limited to the first 64,000 rows. When editing a visualization created prior to this release, you will see a message that says
This visualization uses an old configuration. New visualizations support aggregating data directly within the editor.If you want to leverage the new capabilities, you must re-create the visualization. See Enable aggregation in a visualization.
February 10, 2022
You can now set a custom color palette for a dashboard. All visualizations that appear in that dashboard will use the specified palette. Setting a custom palette does not affect how a visualization appears in other dashboards or the SQL editor.
You can specify hex values for a palette or import colors from another palette, whether provided by Databricks or created by a workspace admin.
When a palette is applied to a dashboard, all visualizations displayed in that dashboard will use the selected color palette by default, even if you configure custom colors when you create the visualization. To override this behavior, see Customize colors for a visualization.
Workspace admins can now create a custom color palette using the admin console. After the custom color palette is created, it can be used in new and existing dashboards. To use a custom color palette for a dashboard or to customize it, you can edit dashboard settings.
When you add a visualization that uses parameters to a dashboard from the SQL
menu, the visualization now uses dashboard-level parameters by default. This matches the behavior when you add a widget using the Add Visualization button in a dashboard.When you view the query history and filter the list by a combination of parameters, the number of matching queries is now displayed.
In visualizations, an issue was fixed where the Y-axis range could not be adjusted to specific values.
February 3, 2022
The tabbed SQL editor is now enabled by default for all users. For more information or to disable the tabbed editor, see Edit multiple queries.
January 27, 2022
Improvements have been made to how you can view, share, and import a query’s profile. See Query profile.
The Details visualization now allows you to rename columns just like the Table visualization.
You can now close a tab in the SQL editor by middle-clicking it.
The following Keyboard shortcuts have been added to the tabbed SQL editor:
Close all tabs: Cmd+Option+Shift+A (macOS) / Ctrl+Option+Shift+A (Windows)
Close other tabs: Cmd+Option+Shift+W (macOS) / Ctrl+Option+Shift+W (Windows)
These keyboard shortcuts provide an alternative to right-clicking on a tab to access the same actions. To view all keyboard shortcuts, click the
Keyboard icon in the tabbed SQL editor.
January 20, 2022
The default formatting for integer and float data types in tables has been updated to not include commas. This means that by default, values like
10002343will no longer have commas. To format these types to display with commas, click Edit Visualization, expand the area for the column, and modify the format to include a comma.To better align with browser rendering limits, visualizations now display a maximum of 10,000 data points. For example, a scatterplot will display a maximum of 10,000 dots. If the number of data points has been limited, a warning is displayed.