September 2023

These features and Databricks platform improvements were released in September 2023.

Note

Releases are staged. Your Databricks workspace might not be updated until a week or more after the initial release date.

Lakehouse Federation available on single user clusters

September 29, 2023

Lakehouse Federation is now available on single user clusters using Databricks Runtime 13.1 and above. Only the connection owner can run queries on federated catalogs. See What is Lakehouse Federation?.

Databricks Runtime 14.1 (Beta)

September 28, 2023

Databricks Runtime 14.1 and Databricks Runtime 14.1 ML are now available as Beta releases.

See Databricks Runtime 14.1 and Databricks Runtime 14.1 for Machine Learning.

GPU model serving optimized for LLMs in Public Preview

September 28, 2023

Model serving on Databricks supports deployment of PyTorch and TensorFlow models using GPU compute resources. This functionality is in Public Preview in select regions and also applies to GPU deployment of models logged with mlflow.pyfunc, mlflow.pytorch, mlflow.tensorflow, and mlflow.transformers flavors. See GPU workload types.

This preview also includes support of GPU deployment optimized for serving large language models. See Optimized large language model (LLM) serving

Prevent runs of a job from being skipped when you reach concurrency limits

September 27, 2023 - October 2, 2023

To prevent runs of a Databricks job from being skipped because of concurrency limits, you can now enable queueing on the job. When queueing is enabled, if a concurrency limit is reached, the job is placed in a queue until capacity is available to run the job. See Enable queueing of job runs.

Databricks Terraform provider updated to version 1.27.0

September 27, 2023

Databricks Terraform provider version 1.27.0 adds owner and force_destroy to databricks_metastore_data_access, includes several additional improvements and bug fixes, and more. For details, see the changelog for version 1.27.0.

Databricks CLI updated to version 0.206.0 (Public Preview)

September 27, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.206.0. This release enables target overrides for pipeline clusters in Databricks Asset Bundles, improves the bundle template initialization experience, and more. For details, see the changelog for version 0.206.0.

Databricks SDK for Go updated to version 0.21.0 (Beta)

September 27, 2023

Databricks SDK for Go version 0.21.0 introduces over a dozen breaking API changes, adds over a dozen APIs, and adds a dozen fields. For details, see the changelog for version 0.21.0.

Inference tables for model serving endpoints is Public Preview

September 26, 2023

Enable inference tables on your model serving endpoints to capture incoming requests and outgoing responses to those endpoints and log them as a Unity Catalog Delta table. In turn, you can use the data in this table to monitor, debug, and improve ML models. See Inference tables for monitoring and debugging models.

To enroll in the Public Preview, reach out to your Databricks account team.

Databricks ODBC driver 2.7.5

September 23, 2023

We have released version 2.7.5 of the Databricks ODBC driver (download). This release adds the following new features and enhancements:

  • OAuth 2.0 client authentication enhancement and OIDC endpoint discovery support.

  • Server-side encryption with customer provided keys.

  • Timedate function.

  • Improved error messaging for Cloud Fetch.

  • Security updates.

This release also resolves the following regression in 2.7.3:

  • The SQLForeignKeys function returns an empty result if the caller passes a NULL value for the catalog of the primary table or the foreign table.

The release 2.7.5 now returns all the primary keys or foreign keys associated with the related table if the catalog of the other table is set NULL.

Databricks extension for Visual Studio Code updated to version 1.1.3

September 21, 2023

The Databricks extension for Visual Studio Code version 1.1.3 adds a few minor fixes. For details, see the changelog for version 1.1.3.

Running jobs as a service principal is GA

September 21, 2023

Roles on service principals and the job’s setting to run jobs using the service principal’s identity is now GA.

See Roles for managing service principals and Configure identity for job runs.

Databricks CLI updated to version 0.205.2 (Public Preview)

September 21, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.205.2. This release contains a few minor updates. For details, see the changelog for version 0.205.2.

Databricks CLI updated to version 0.205.1 (Public Preview)

September 20, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.205.1. This release enables target overrides for job tasks in Databricks Asset Bundles, and more. For details, see the changelog for version 0.205.1.

Databricks SDK for Go updated to version 0.20.0 (Beta)

September 20, 2023

Databricks SDK for Go version 0.20.0 adds over a dozen APIs, adds a half-dozen fields, changes a few fields and methods, and removes a few APIs. For details, see the changelog for version 0.20.0.

Databricks SDK for Python updated to version 0.9.0 (Beta)

September 20, 2023

Databricks SDK for Python version 0.9.0 changes a few methods and fields, adds about 10 fields, adds about 20 dataclasses, and introduces additional changes and removals. For details, see the changelog for version 0.9.0.

Databricks SDK for Python updated to version 0.7.1 (Beta)

September 20, 2023

Databricks SDK for Python version 0.7.1 includes a minor fix to improve file download performance. For details, see the changelog for version 0.7.1.

Databricks Terraform provider updated to version 1.26.0

September 20, 2023

Databricks Terraform provider version 1.26.0 adds support for init scripts from Unity Catalog volumes, includes several additional improvements and bug fixes, and more. For details, see the changelog for version 1.26.0.

Databricks Connect for Databricks Runtime 14.0

September 20, 2023

Databricks Connect for Databricks Runtime 14.0 supports full Structured Streaming (including foreachBatch) and long running queries that reattach to the query after interruption.

Configure Tableau and Power BI OAuth with SAML SSO

September 20, 2023

Previously, SSO using OIDC was required to use a custom identity provider (IdP) for Tableau and Power BI OAuth login. You can now configure Tableau and Power BI OAuth login with SSO using SAML. See Connect Power BI to Databricks and Connect Tableau and Databricks.

Lakeview dashboards in Public Preview

September 20, 2023

Lakeview dashboards allow you to share visualizations and datasets using a simplified and unified data governance model. See Dashboards.

Fleet clusters now support Graviton instances

September 19, 2023

Graviton instances are now available to use with fleet clusters. See AWS Fleet instance types.

Databricks Connect V2 is Public Preview for Scala

September 19, 2023

Databricks Connect for Databricks Runtime 13.3 LTS now supports Scala in Public Preview, including DataFrames, Delta, Structured Streaming, and more.

For Python, query cancellation is now supported, and the DatabricksSession class can now be used in Databricks notebooks and jobs.

GPU model serving in Public Preview

September 19, 2023

Model serving on Databricks supports deployment of PyTorch and TensorFlow models using GPU compute resources. This functionality is in [Public Preview] in select regions and also applies to GPU deployment of models logged with mlflow.pyfunc, mlflow.pytorch, mlflow.tensorflow, and mlflow.transformers flavors. See GPU workload types.

On-demand feature computation now available in Unity Catalog (Public Preview)

September 14, 2023

Machine learning features can now be computed on-demand at inference time. Feature computation logic, models, and data are all governed by Unity Catalog. This enables models to compute features using inputs that are only available at inference time, such as a user’s current location. For more information, see Compute features on demand using Python user-defined functions.

Unified login for accounts created before June 21, 2023 (Public Preview)

September 14, 2023

Unified login allows you to manage one single sign-on (SSO) configuration in your account that is used for the account and Databricks workspace. When SSO is enabled on your account, you can choose to enable unified login for all workspaces or for selected workspaces. Unified login workspaces use the account-level SSO configuration and all users must sign in to those workspaces using SSO. For more information, see Enable unified login.

Unified login is GA for accounts created after June 21, 2023. See Unified Login GA for new accounts

Databricks Runtime 14.0 is GA

September 14, 2023

Databricks Runtime 14.0 and Databricks Runtime 14.0 ML are now generally available.

See Databricks Runtime 14.0 (EoS) and Databricks Runtime 14.0 for Machine Learning (EoS).

Account email notifications upgrade

September 14, 2023

Databricks no longer sends email notifications when users are added to a Databricks account. Users will continue to receive email notifications when they are granted access to a workspace.

Databricks Terraform provider updated to version 1.25.1

September 13, 2023

Databricks Terraform provider version 1.25.1 has been released with minor improvements and bug fixes. For details, see the changelog for version 1.25.1.

Databricks SDK for Go updated to version 0.19.2 (Beta)

September 13, 2023

Databricks SDK for Go version 0.19.2 contains one minor fix. For details, see the changelog for version 0.19.2.

Databricks CLI updated to version 0.205.0 (Public Preview)

September 13, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.205.0. This is the first release of the Databricks CLI to support Databricks Asset Bundles in Public Preview. For details, see the changelog for version 0.205.0.

Reserved location for /Workspace

September 13, 2023

The location /Workspace is now reserved. You can no longer define a directory named /Workspace relative to the root path in Databricks.

Databricks Asset Bundles (Public Preview)

September 12, 2023

Databricks Asset Bundles, now in Public Preview, enable end-to-end data, analytics, and ML projects to be expressed as a collection of source files. This makes it simpler to apply data engineering best practices such as source control, code review, testing, and CI/CD.

Databricks Terraform provider updated to version 1.25.0

September 12, 2023

Databricks Terraform provider version 1.25.0 includes the addition of IS_OWNER permission for Databricks SQL warehouses, minor fixes, documentation updates, and more. For details, see the changelog for version 1.25.0.

Databricks extension for Visual Studio Code updated to version 1.1.2

September 12, 2023

The Databricks extension for Visual Studio Code version 1.1.2 adds experimental support for %run and # MAGIC in Databricks notebooks, fixes path issues with multiple local Databricks CLI installations, and more. For details, see the changelog for version 1.1.2.

Partner Connect supports Snowplow

September 12, 2023

You can now use Partner Connect to connect your Databricks workspace to Snowplow. For more information, see Connect to Snowplow.

Databricks CLI updated to version 0.204.1 (Public Preview)

September 11, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.204.1. For details, see the changelog for version 0.204.1.

Filter sensitive data with row filters and column masks (Public Preview)

September 11, 2023

You can now apply row filters and column masks to tables to prevent access to sensitive data by certain users. These filters and masks are implemented as SQL user-defined functions (UDFs). See Filter sensitive table data using row filters and column masks.

System tables now include Marketplace schema (Public Preview)

September 8, 2023

System tables now includes a marketplace schema that stores Marketplace provider data. You can query the marketplace.listing_access_events table to retrieve the consumer info for actions taken on your listings. For more information, see Marketplace system tables reference.

Pricing system table is now available (Public Preview)

September 8, 2023

System tables now include a pricing table (system.billing.list_prices) where you can monitor the cost of a SKU over time. These logs can help you perform cost analysis and monitor pricing changes. See Pricing system table reference.

Databricks SDK for Go updated to version 0.19.1 (Beta)

September 7, 2023

Databricks SDK for Go version 0.19.1 continues a few minor updates and fixes. For details, see the changelog for version 0.19.1.

Databricks ODBC driver 2.7.3

September 7, 2023

We have released version 2.7.3 of the Databricks ODBC driver (download). This release includes the following features and fixes:

  • Vulnerability fixes by upgrading third-party dependencies.

  • SQL_ATTR_MAX_ROWS support that allows a client to limit the number of rows to fetch in a query to expedite fast loading (such as preview).

  • Updated MacOS support. The driver now supports Apple Silicon/M1.

  • OAuth2 browser-based authentication support to facilitate single sign-on (SSO).

Important

If you use primary key and foreign key APIs, the SQLForeignKeys function returns an empty result if the caller passes a NULL value for the catalog of the primary table or the foreign table. This issue is known to developers.

Databricks CLI updated to version 0.204.0 (Public Preview)

September 6, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.204.0. This release includes the grouping of permission related commands, adds and changes to several commands, and more. For details, see the changelog for version 0.204.0.

Data Explorer is now Catalog Explorer

September 6, 2023

Data Explorer is renamed to Catalog Explorer to recognize the fact that you can use it to work with all securable objects in Unity Catalog, not just data objects. See What is Catalog Explorer?.

Databricks SDK for Go updated to version 0.19.0 (Beta)

September 4, 2023

Databricks SDK for Go version 0.19.0 renames permissions APIs, adds 8 APIs, adds 5 fields, changes and removes a few fields, and changes a few methods.. For details, see the changelog for version 0.19.0.

Databricks SDK for Python updated to version 0.8.0 (Beta)

September 4, 2023

Databricks SDK for Python version 0.8.0 renames permissions APIs, adds a half-dozen fields, adds a half-dozen dataclasses, and introduces several changes and removals. For details, see the changelog for version 0.8.0.

Delegate allowlist privileges in Unity Catalog

September 1, 2023

You can now delegate privileges for managing items in the allowlist for Unity Catalog. See Allowlist libraries and init scripts on shared compute.

GitHub apps integration in Repos is GA

September 1, 2023

GitHub apps integration in Repos is GA. The feature allows for simpler Git credential management when you integrate with GitHub in Repos.

Important

GitHub apps are not supported on GitHub Enterprise Server.