September 2023
These features and Databricks platform improvements were released in September 2023.
Note
Releases are staged. Your Databricks workspace might not be updated until a week or more after the initial release date.
Databricks Runtime 14.1 (Beta)
September 28, 2023
Databricks Runtime 14.1 and Databricks Runtime 14.1 ML are now available as Beta releases.
See Databricks Runtime 14.1 (Beta) and Databricks Runtime 14.1 for Machine Learning (Beta).
Prevent runs of a job from being skipped when you reach concurrency limits
September 28, 2023
To prevent runs of a Databricks job from being skipped because of concurrency limits, you can now enable queueing on the job. When queueing is enabled, if a concurrency limit is reached, the job is placed in a queue until capacity is available to run the job. See What if my job cannot run because of concurrency limits?.
GPU model serving optimized for LLMs in Public Preview
September 28, 2023
Model serving on Databricks supports deployment of PyTorch and Tensorflow models using GPU compute resources. This functionality is in Public Preview in select regions and also applies to GPU deployment of models logged with mlflow.pyfunc, mlflow.pytorch, mlflow.tensorflow, and mlflow.transformers flavors. See GPU workload types.
This preview also includes support of GPU deployment optimized for serving large language models. See Model serving optimized for large language models
Submit the enrollment form to participate in the GPU model serving Public Preview.
Inference tables for model serving endpoints is Public Preview
September 26, 2023
Enable inference tables on your model serving endpoints to capture incoming requests and outgoing responses to those endpoints and log them as a Unity Catalog Delta table. In turn, you can use the data in this table to monitor, debug, and improve ML models. See Inference tables for monitoring model serving endpoints.
To enroll in the Public Preview, reach out to your Databricks representative.
Databricks ODBC driver 2.7.5
September 23, 2023
We have released version 2.7.5 of the Databricks ODBC driver (download). This release adds the following new features and enhancements:
OAuth 2.0 client authentication enhancement and OIDC endpoint discovery support.
Server-side encryption with customer provided keys.
Timedate function.
Improved error messaging for Cloud Fetch.
Security updates.
This release also resolves the following regression in 2.7.3:
The
SQLForeignKeysfunction returns an empty result if the caller passes aNULLvalue for the catalog of the primary table or the foreign table.
The release 2.7.5 now returns all the primary keys or foreign keys associated with the related table if the catalog of the other table is set NULL.
Running jobs as a service principal is GA
September 21, 2023
Roles on service principals and the job’s setting to run jobs using the service principal’s identity is now GA.
See Roles for managing service principals and Run a job as a service principal.
Databricks SDK for Python updated to version 0.9.0 (Beta)
September 20, 2023
Databricks SDK for Python version 0.9.0 changes a few methods and fields, adds about 10 fields, adds about 20 dataclasses, and introduces additional changes and removals. For details, see the changelog for version 0.9.0.
Databricks SDK for Python updated to version 0.7.1 (Beta)
September 20, 2023
Databricks SDK for Python version 0.7.1 includes a minor fix to improve file download performance. For details, see the changelog for version 0.7.1.
Databricks Connect for Databricks Runtime 14.0
September 20, 2023
Databricks Connect for Databricks Runtime 14.0 supports full Structured Streaming (including foreachBatch) and long running queries that reattach to the query after interruption.
Configure Tableau and PowerBI OAuth with SAML SSO
September 20, 2023
Previously, SSO using OIDC was required to use a custom identity provider (IdP) for Tableau and PowerBI OAuth login. You can now configure Tableau and PowerBI OAuth login with SSO using SAML. See Connect Power BI to Databricks and Connect to Tableau.
Databricks Connect V2 is Public Preview for Scala
September 19, 2023
Databricks Connect for Databricks Runtime 13.3 LTS now supports Scala in Public Preview, including DataFrames, Delta, Structured Streaming, and more.
For Python, query cancellation is now supported, and the DatabricksSession class can now be used in Databricks notebooks and jobs.
GPU model serving in Public Preview
September 19, 2023
Model serving on Databricks supports deployment of PyTorch and Tensorflow models using GPU compute resources. This functionality is in [Public Preview] in select regions and also applies to GPU deployment of models logged with mlflow.pyfunc, mlflow.pytorch, mlflow.tensorflow, and mlflow.transformers flavors. See GPU workload types.
Submit the enrollment form to participate in the GPU model serving Public Preview.
Lakeview dashboards in Public Preview
September 20, 2023
Lakeview dashboards allow you to share visualizations and datasets using a simplified and unified data governance model. See Create and share Lakeview dashboards.
On-demand feature computation now available in Unity Catalog (Public Preview)
September 14, 2023
Machine learning features can now be computed on-demand at inference time. Feature computation logic, models, and data are all governed by Unity Catalog. This enables models to compute features using inputs that are only available at inference time, such as a user’s current location. For more information, see Compute features on demand using Python user-defined functions.
Unified Login Public Preview for accounts created before June 21, 2023
September 14, 2023
Unified login allows you to manage one single sign-on (SSO) configuration in your account that is used for the account and Databricks workspace. When SSO is enabled on your account, you can choose to enabled unified login for all workspaces or for selected workspaces. Unified login workspaces use the account-level SSO configuration and all users must sign in to those workspaces using SSO. For more information, see Enable unified login.
Unified login is GA for accounts created after June 21, 2023. See Unified Login GA for new accounts
Databricks Runtime 14.0 is GA
September 14, 2023
Databricks Runtime 14.0 and Databricks Runtime 14.0 ML are now generally available.
See Databricks Runtime 14.0 and Databricks Runtime 14.0 for Machine Learning.
Account email notifications upgrade
September 14, 2023
Databricks no longer sends email notifications when users are added to a Databricks account. Users will continue to receive email notifications when they are granted access to a workspace.
Databricks CLI updated to version 0.205.0 (Public Preview)
September 13, 2023
The Databricks command-line interface (Databricks CLI) has been updated to version 0.205.0. This is the first release of the Databricks CLI to support Databricks asset bundles in Public Preview. For details, see the changelog for version 0.205.0.
Reserved location for /Workspace
September 13, 2023
The location /Workspace is now reserved. You can no longer define a directory named /Workspace relative to the root path in Databricks.
Databricks asset bundles (Public Preview)
September 12, 2023
Databricks asset bundles, now in Public Preview, enable end-to-end data, analytics, and ML projects to be expressed as a collection of source files. This makes it simpler to apply data engineering best practices such as source control, code review, testing, and CI/CD.
Databricks Terraform provider updated to version 1.25.0
September 12, 2023
Databricks Terraform provider version 1.25.0 includes the addition of IS_OWNER permission for Databricks SQL warehouses, minor fixes, documentation updates, and more. For details, see the changelog for version 1.25.0.
Databricks extension for Visual Studio Code updated to version 1.1.2
September 12, 2023
The Databricks extension for Visual Studio Code version 1.1.2 adds experimental support for %run and # MAGIC in Databricks notebooks, fixes path issues with multiple local Databricks CLI installations, and more. For details, see the changelog for version 1.1.2.
Partner Connect supports Snowplow
September 12, 2023
You can now use Partner Connect to connect your Databricks workspace to Snowplow. For more information, see Connect to Snowplow.
Databricks CLI updated to version 0.204.1 (Public Preview)
September 11, 2023
The Databricks command-line interface (Databricks CLI) has been updated to version 0.204.1. For details, see the changelog for version 0.204.1.
Filter sensitive data with row filters and column masks (Public Preview)
September 11, 2023
You can now apply row filters and column masks to table to prevent access to sensitive data by certain users. These filters and masks are implemented as SQL user-defined functions (UDFs). See Filter sensitive table data with row filters and column masks.
System tables now include Marketplace schema (Public Preview)
September 8, 2023
System tables now includes a marketplace schema that stores Marketplace provider data. You can query the marketplace.listing_access_events table to retrieve the consumer info for actions taken on your listings. For more information, see Marketplace system table reference.
Pricing system table is now available (Public Preview)
September 8, 2023
System tables now include a pricing table (system.billing.list_prices) where you can monitor the cost of a SKU over time. These logs can help you perform cost analysis and monitor pricing changes. See Pricing system table reference.
Databricks ODBC driver 2.7.3
September 7, 2023
We have released version 2.7.3 of the Databricks ODBC driver (download). This release includes the following features and fixes:
Vulnerability fixes by upgrading third-party dependencies.
SQL_ATTR_MAX_ROWSsupport that allows a client to limit the number of rows to fetch in a query to expedite fast loading (such as preview).Updated MacOS support. The driver now supports Apple Silicon/M1.
OAuth2 browser-based authentication support to facilitate single sign-on (SSO).
Important
If you use primary key and foreign key APIs, the SQLForeignKeys function returns an empty result if the caller passes a NULL value for the catalog of the primary table or the foreign table. This issue is known to developers.
Databricks CLI updated to version 0.204.0 (Public Preview)
September 6, 2023
The Databricks command-line interface (Databricks CLI) has been updated to version 0.204.0. This release includes the grouping of permission related commands, adds and changes to several commands, and more. For details, see the changelog for version 0.204.0.
Data Explorer is now Catalog Explorer
September 6, 2023
Data Explorer is renamed to Catalog Explorer to recognize the fact that you can use it to work with all securable objects in Unity Catalog, not just data objects. See Discover and manage data and other AI assets using Catalog Explorer.
Databricks SDK for Python updated to version 0.8.0 (Beta)
September 4, 2023
Databricks SDK for Python version 0.8.0 renames permissions APIs, adds a half-dozen fields, adds a half-dozen dataclasses, and introduces several changes and removals. For details, see the changelog for version 0.8.0.
Delegate allowlist privileges in Unity Catalog
September 1, 2023
You can now delegate privileges for managing items in the allowlist for Unity Catalog. See Allowlist libraries and init scripts on shared compute.