March 2023

These features and Databricks platform improvements were released in March 2023.

Note

Releases are staged. Your Databricks account might not be updated until a week or more after the initial release date.

Databricks Terraform provider updated to version 1.14.0

March 31, 2023

Version 1.14.0 adds an example for automatically rotating an access token, a run_if condition and a file arrival trigger to the databricks_job resource, and more. For more details, see the changelog for version 1.14.0.

Databricks Runtime 7.3 LTS ML support ends

March 31, 2023

Support for Databricks Runtime 7.3 LTS for Machine Learning ended on March 31. See Databricks support lifecycles.

C7g Graviton 3 instances are now supported on Databricks

March 27, 2023

Databricks now supports selecting c7g instances during cluster creation on AWS. C7g instances do not support Photon runtimes.

Distributed training with TorchDistributor

March 27, 2023

With Databricks Runtime 13.0 ML and above, you can perform distributed training on PyTorch ML models using TorchDistributor. See Distributed training with TorchDistributor.

TorchDistributor is an open-source module in PySpark that helps users do distributed training with PyTorch on their Spark clusters, so it lets you launch PyTorch training jobs as Spark jobs.

Databricks Runtime 13.0 (Beta)

March 24, 2023

Databricks Runtime 13.0 and Databricks Runtime 13.0 ML are now available as Beta releases.

See Databricks Runtime 13.0 (EoS) and Databricks Runtime 13.0 for Machine Learning (EoS).

Improved file editor

March 24, 2023

The file editor has been updated to bring many of the notebook features to your file editing experience. Additionally, you now can directly execute Python, SQL, R, and Scala files from the file editor. For details, see Use the Databricks notebook and file editor.

Databricks no longer creates a serverless starter SQL warehouse

March 22, 2023

Databricks no longer creates a serverless starter SQL warehouse. To create one yourself, see Create a SQL warehouse.

In SQL Warehouses API, enabling serverless compute now must be explicit

March 22, 2023

There was a minor change in the SQL Warehouses API when creating or updating a SQL warehouse.

For the field enable_serverless_compute, previously the default was true if serverless SQL warehouses were enabled for the workspace. The default is now false for most workspaces. However, if this workspace used the SQL Warehouses API to create a warehouse between September 1, 2022 and April 30, 2023, and fits the requirements for serverless SQL warehouses, the default remains set to true.

To avoid ambiguity, especially for organizations with many workspaces, Databricks strongly recommends that you always explicitly set the enable_serverless_compute field.

Changes for workspace settings for serverless SQL warehouses

March 22, 2023

For accounts that have serverless compute enabled, workspaces are now enabled for serverless features by default. As part of this change, the workspace-level setting to enable or disable serverless SQL warehouses was removed from the workspace settings UI.

Changes for serverless compute settings for accounts and workspaces

March 22, 2023

Existing accounts that are enabled for serverless compute now have all workspaces enabled for all serverless compute features, such as serverless SQL warehouses and Model Serving. If your account needs updated terms of use, workspace admins are prompted in the Databricks SQL UI.

For newly-enabled workspaces to support serverless SQL warehouses, the workspace must meet the requirements and might require an update to its instance profile role to add a trust relationship. Also see Changes for workspace settings for serverless SQL warehouses.

Databricks SQL Serverless is GA

March 22, 2023

Databricks SQL Serverless is now generally available in supported regions. Serverless SQL warehouses provide instant compute, minimal management, and cost optimization for SQL queries. Create serverless SQL warehouses, or convert pro or classic SQL warehouses to serverless.

.ipynb (Jupyter) notebook support in Repos (preview)

March 22, 2023

Support for Jupyter notebooks (.ipynb files) is available in Repos. You can clone repositories with .ipynb notebooks, work in Databricks UI, and then commit and push as .ipynb notebooks. Metadata such as a notebook dashboard is preserved. Admins can control whether outputs can be committed or not.

You can also:

  • Create new .ipynb notebooks.

  • Convert notebooks to .ipynb file format.

  • View diffs as Code diff (code changes in cells) or Raw diff (code changes in JSON, including metadata).

See Allow committing .ipynb notebook output.

Support for reload4j

March 21, 2023

Reload4j 1.2.19 is replacing log4j in Databricks Runtime versions 10.4 and earlier.

Execute SQL cells in the notebook in parallel

March 15, 2023

You can now run SQL cells in Databricks notebooks in parallel while attached to an interactive cluster. See Execute SQL cells in parallel.

Create job tasks using Python code stored in a Git repo

March 14, 2023

You can now retrieve Python code from a Git provider when you add a Python task to a Databricks job, simplifying integration of your jobs with your CI/CD workflows. See Use Git with jobs.

Databricks Terraform provider updated to version 1.13.0

March 14, 2023

Version 1.13.0 adds a databricks_sql_alert resource, and more. For more details, see the changelog for version 1.13.0.

Databricks Terraform provider updated to version 1.12.0

March 9, 2023

Version 1.12.0 adds a databricks_model_serving resource, deprecates the schedule block within the databricks_sql_query resource, and more. For more details, see the changelog for version 1.12.0.

SQL admin console and workspace admin console combined

March 9, 2023

The SQL admin console has been combined with the general admin settings to create a unified experience for admin users. All SQL admin settings are now accessed from the admin console.

Model Serving is GA

March 7, 2023

Model Serving, formerly Serverless Real-Time Inference, is now generally available.

Model Serving provides a highly available and low-latency service for deploying models. The service automatically scales up or down to meet demand changes within the chosen concurrency range. See Model serving with Databricks.

Automatic feature lookup is GA

March 7, 2023

Automatic feature lookup with Model Serving is now generally available. For details, see Automatic feature lookup with Databricks Model Serving.

New Catalog Explorer availability

March 6, 2023

A new version of Catalog Explorer is now available in all workspaces that use a supported version of Databricks Runtime 7.3 or later.

View frequent queries and users of a table using the Insights tab

March 6, 2023

You can now use the Insights tab in Catalog Explorer to view the most frequent queries and users of any table registered in Unity Catalog. You must have the SELECT privilege on a table to see this data, and you only see queries that you have permission to view. See View frequent queries and users of a table.

View lineage information for your Databricks jobs

March 3, 2023

If Unity Catalog is enabled in your workspace, you can view lineage information for your jobs in the Databricks Jobs UI, including upstream tables the job reads from and downstream tables the job writes to. See View lineage information for a job.

Databricks Runtime 12.2 LTS and Databricks Runtime 12.2 LTS ML are GA

March 2, 2023

Databricks Runtime 12.2 LTS and Databricks Runtime 12.2 LTS ML are now generally available.

See Databricks Runtime 12.2 LTS and Databricks Runtime 12.2 LTS for Machine Learning.

Workspace files are now in Public Preview

March 6, 2023

You can now work with non-notebook files in Databricks. Workspaces files are enabled by default in all workspaces. See What are workspace files?.