March 2023
These features and Databricks platform improvements were released in March 2023.
Note
Releases are staged. Your Databricks account might not be updated until a week or more after the initial release date.
Databricks Terraform provider updated to version 1.14.0
March 31, 2023
Version 1.14.0 adds an example for automatically rotating an access token, a run_if
condition and a file arrival trigger to the databricks_job
resource, and more. For more details, see the changelog for version 1.14.0.
Databricks Runtime 7.3 LTS ML support ends
March 31, 2023
Support for Databricks Runtime 7.3 LTS for Machine Learning ended on March 31. See Databricks support lifecycles.
C7g Graviton 3 instances are now supported on Databricks
March 27, 2023
Databricks now supports selecting c7g instances during cluster creation on AWS. C7g instances do not support Photon runtimes.
Distributed training with TorchDistributor
March 27, 2023
With Databricks Runtime 13.0 ML and above, you can perform distributed training on PyTorch ML models using TorchDistributor. See Distributed training with TorchDistributor.
TorchDistributor is an open-source module in PySpark that helps users do distributed training with PyTorch on their Spark clusters, so it lets you launch PyTorch training jobs as Spark jobs.
Databricks Runtime 13.0 (Beta)
March 24, 2023
Databricks Runtime 13.0 and Databricks Runtime 13.0 ML are now available as Beta releases.
See Databricks Runtime 13.0 (EoS) and Databricks Runtime 13.0 for Machine Learning (EoS).
Improved file editor
March 24, 2023
The file editor has been updated to bring many of the notebook features to your file editing experience. Additionally, you now can directly execute Python, SQL, R, and Scala files from the file editor. For details, see Use the Databricks notebook and file editor.
Databricks no longer creates a serverless starter SQL warehouse
March 22, 2023
Databricks no longer creates a serverless starter SQL warehouse. To create one yourself, see Create a SQL warehouse.
In SQL Warehouses API, enabling serverless compute now must be explicit
March 22, 2023
There was a minor change in the SQL Warehouses API when creating or updating a SQL warehouse.
For the field enable_serverless_compute
, previously the default was true
if serverless SQL warehouses were enabled for the workspace. The default is now false
for most workspaces. However, if this workspace used the SQL Warehouses API to create a warehouse between September 1, 2022 and April 30, 2023, and fits the requirements for serverless SQL warehouses, the default remains set to true
.
To avoid ambiguity, especially for organizations with many workspaces, Databricks strongly recommends that you always explicitly set the enable_serverless_compute
field.
Changes for workspace settings for serverless SQL warehouses
March 22, 2023
For accounts that have serverless compute enabled, workspaces are now enabled for serverless features by default. As part of this change, the workspace-level setting to enable or disable serverless SQL warehouses was removed from the workspace settings UI.
Changes for serverless compute settings for accounts and workspaces
March 22, 2023
Existing accounts that are enabled for serverless compute now have all workspaces enabled for all serverless compute features, such as serverless SQL warehouses and Model Serving. If your account needs updated terms of use, workspace admins are prompted in the Databricks SQL UI.
For newly-enabled workspaces to support serverless SQL warehouses, the workspace must meet the requirements and might require an update to its instance profile role to add a trust relationship. Also see Changes for workspace settings for serverless SQL warehouses.
Databricks SQL Serverless is GA
March 22, 2023
Databricks SQL Serverless is now generally available in supported regions. Serverless SQL warehouses provide instant compute, minimal management, and cost optimization for SQL queries. Create serverless SQL warehouses, or convert pro or classic SQL warehouses to serverless.
.ipynb (Jupyter) notebook support in Repos (preview)
March 22, 2023
Support for Jupyter notebooks (.ipynb files) is available in Repos. You can clone repositories with .ipynb notebooks, work in Databricks UI, and then commit and push as .ipynb notebooks. Metadata such as a notebook dashboard is preserved. Admins can control whether outputs can be committed or not.
You can also:
Create new .ipynb notebooks.
Convert notebooks to .ipynb file format.
View diffs as Code diff (code changes in cells) or Raw diff (code changes in JSON, including metadata).
Support for reload4j
March 21, 2023
Reload4j 1.2.19 is replacing log4j in Databricks Runtime versions 10.4 and earlier.
Execute SQL cells in the notebook in parallel
March 15, 2023
You can now run SQL cells in Databricks notebooks in parallel while attached to an interactive cluster. See Execute SQL cells in parallel.
Create job tasks using Python code stored in a Git repo
March 14, 2023
You can now retrieve Python code from a Git provider when you add a Python task to a Databricks job, simplifying integration of your jobs with your CI/CD workflows. See Use Git with jobs.
Databricks Terraform provider updated to version 1.13.0
March 14, 2023
Version 1.13.0 adds a databricks_sql_alert
resource, and more. For more details, see the changelog for version 1.13.0.
Databricks Terraform provider updated to version 1.12.0
March 9, 2023
Version 1.12.0 adds a databricks_model_serving
resource, deprecates the schedule
block within the databricks_sql_query
resource, and more. For more details, see the changelog for version 1.12.0.
SQL admin console and workspace admin console combined
March 9, 2023
The SQL admin console has been combined with the general admin settings to create a unified experience for admin users. All SQL admin settings are now accessed from the admin console.
Model Serving is GA
March 7, 2023
Model Serving, formerly Serverless Real-Time Inference, is now generally available.
Model Serving provides a highly available and low-latency service for deploying models. The service automatically scales up or down to meet demand changes within the chosen concurrency range. See Model serving with Databricks.
Automatic feature lookup is GA
March 7, 2023
Automatic feature lookup with Model Serving is now generally available. For details, see Automatic feature lookup with Databricks Model Serving.
New Catalog Explorer availability
March 6, 2023
A new version of Catalog Explorer is now available in all workspaces that use a supported version of Databricks Runtime 7.3 or later.
View frequent queries and users of a table using the Insights tab
March 6, 2023
You can now use the Insights tab in Catalog Explorer to view the most frequent queries and users of any table registered in Unity Catalog. You must have the SELECT
privilege on a table to see this data, and you only see queries that you have permission to view. See View frequent queries and users of a table.
Exact match search is available in global search
March 3, 2023
You can now search for exact phrases using quotation marks in global search. See Search for workspace objects.
View lineage information for your Databricks jobs
March 3, 2023
If Unity Catalog is enabled in your workspace, you can view lineage information for your jobs in the Databricks Jobs UI, including upstream tables the job reads from and downstream tables the job writes to. See View lineage information for a job.
Databricks Runtime 12.2 LTS and Databricks Runtime 12.2 LTS ML are GA
March 2, 2023
Databricks Runtime 12.2 LTS and Databricks Runtime 12.2 LTS ML are now generally available.
See Databricks Runtime 12.2 LTS and Databricks Runtime 12.2 LTS for Machine Learning.
Workspace files are now in Public Preview
March 6, 2023
You can now work with non-notebook files in Databricks. Workspaces files are enabled by default in all workspaces. See What are workspace files?.