Amazon
Web Services
Microsoft Azure
Google Cloud PlatformJune 2021
These features and Databricks platform improvements were released in June 2021.
Note
Releases are staged. Your Databricks account may not be updated until a week or more after the initial release date.
Correction: Repos for Git is enabled by default in new and existing workspaces in some regions
June 30, 2021
In a release note for June 2, 2021, Databricks announced that Repos for Git is enabled by default in new workspaces in some regions, but omitted the following important detail: Repos for Git is also enabled in existing workspaces if Notebook Git Versioning is already enabled. The original release note has been updated.
Change to Feature Store permissions
June 28 - July 6, 2021: Version 3.49
Users with CAN_VIEW_METADATA permission on a feature table can now publish it to online stores.
Improved access to results in the MLflow runs table
June 28 - July 6, 2021: Version 3.49
Changes to the MLflow runs table provide quicker access to results.
The Metrics columns now appear before the Parameters columns, making it easier to identify the best run.
The Start Time column now shows the age of the run, not the start time. This reduces the width of the column and allows more information to be visible without horizontal scrolling. To display the start time, hover over the value in the column.
Better cost visibility for Delta Live Tables
June 28 - July 6, 2021: Version 3.49
When you create a new pipeline or view the Pipeline Details UI, you will now see the DBU per hour for the Delta Live Tables default cluster.
Enhanced data quality constraints for Delta Live Tables
June 28 - July 6, 2021: Version 3.49
You can now use the expect_all_or_fail, expect_all_or_drop, and expect_all Python APIs to define multiple data quality constraints on Delta Live Tables datasets. These APIs are useful when applying a library of common expectations to one or more datasets in a pipeline. For more information, see Manage data quality with Delta Live Tables.
API changes for updating and replacing IP address lists
June 28 - July 6, 2021: Version 3.49
A new PATCH operation for /ip-access-lists/<list-id> in the Databricks REST API 2.0 allows you to programmatically update the individual values in an existing IP address list. For more information, see Update an IP access list.
The existing PUT operation for /ip-access-lists/<list-id> in the Databricks REST API 2.0 allows you to programmatically replace the contents of an existing IP address list while maintaining the list’s original internal ID. For more information, see Replace an IP access list.
Databricks ODBC driver 2.6.17
June 22, 2021
We have released version 2.6.17 of the Databricks ODBC driver (download). This release adds support for Cloud Fetch, which is a mechanism for fetching data in parallel through cloud storage to bring the data faster to BI tools. This release also reduces latency by removing unnecessary roundtrips between the client and the Databricks Runtime. See How We Achieved High-bandwidth Connectivity With BI Tools.
Use an API to download usage data directly
June 16, 2021: Version 3.48
You can now use the Account API to download usage data in CSV format directly without the need to set up delivery to an S3 bucket. See Return billable usage logs to download the logs.
Databricks Runtime 7.5 series support ends
June 16, 2021
Support for Databricks Runtime 7.5, Databricks Runtime 7.5 for Machine Learning, and Databricks Runtime 7.5 for Genomics ended on June 16. See Databricks runtime support lifecycles.
Optimize performance and control costs by using different pools for the driver node and worker nodes
June 14-21, 2021: Version 3.48
You can now optimize your cluster by attaching it to a different pool for the driver node and worker nodes. For example, you can use a compute-optimized pool for the driver node and a storage-optimized pool for nodes. This also helps you control costs by assigning different tags to each instance pool. If you choose, you can use the same pool for drivers and worker nodes.
For more information, see Pool configuration reference.
Photon runtimes now support i3.xlarge instances (Public Preview)
June 14, 2021: Version 3.48
Photon is the new native vectorized engine on Databricks, directly compatible with Apache Spark APIs. To provision clusters with Photon you must use a Photon image. The Photon image is now supported on i3.xlarge instances, the default instance type when you create a cluster.
Registry-wide permissions for Model Registry
June 14-21, 2021: Version 3.48
You can now set default model permissions for all models in a workspace’s Model Registry. Permissions set at this level apply to all new and existing models registered in Model Registry in that workspace. For more information, see MLFlow model ACLs.
A user’s home directory is no longer protected when you delete a user using the SCIM API
June 14-21, 2021: Version 3.48
When you delete a user using the SCIM API, the user’s home directory is no longer protected, and a workspace administrator can delete it. This behavior matches what happens when you delete a user using the admin console.
Accelerate SQL workloads with Photon (Public Preview)
June 9, 2021: Version 3.47
Photon is the new native vectorized engine on Databricks, directly compatible with Apache Spark APIs. It is included as part of a new high-performance runtime designed to run your SQL workloads faster and reduce your total cost per workload.
To provision clusters with Photon you must use a Photon image. Photon images consume DBUs at a different rate than the same instance type running the non-Photon runtime. For more information about Photon instances and DBU consumption, see the Databricks pricing page.
Photon is available for Databricks Runtime 8.3 (unsupported) and above.
Databricks Runtime 8.3 and 8.3 ML are GA; 8.3 Photon is Public Preview
June 8, 2021
Databricks Runtime 8.3 and 8.3 ML are now generally available. 8.3 Photon is in Public Preview.
For information, see the full release notes at Databricks Runtime 8.3 (unsupported) and Databricks Runtime 8.3 for ML (unsupported).
Python and SQL table access control (GA)
June 8, 2021
Table access control lets you programmatically grant and revoke access to your data using the Databricks view-based access control model. Table access control is available in two versions: SQL-only and Python and SQL. Previously only SQL-only table access control was generally available. Now Python and SQL table access control is generally available. See Enable Hive metastore table access control on a cluster (legacy).
Jobs UI and API now show the owner of a job run
June 2-7, 2021: Version 3.47
The jobs UI and API now include the user that a job will run as, making it easier to understand, debug, and audit job permissions. This is a read-only value set to the job creator if job access control is disabled or the job’s is_owner permission if job access control is enabled. For more information about managing job permissions, see Control access to a job.
Protect sensitive Spark configuration properties and environment variables using secrets (Public Preview)
June 7, 2021
You can protect sensitive Spark configuration properties and environment variables using secrets. Retrieved secrets are redacted from notebook output and Spark driver and executor logs. See Secrets.
If you currently use plaintext passwords in a Spark configuration, Databricks recommends that you rotate your passwords and use secret paths in the Spark configuration.
Repos for Git is enabled by default in new and existing workspaces in some regions
June 2, 2021
This release note was updated on June 30, 2021.
For new workspaces and for existing workspaces with Notebook Git Versioning enabled, Repos for Git is now enabled by default in some regions. Default Repos enablement will be rolled out to more regions in the coming months. To learn how to enable or disable Repos yourself, and to verify the settings for your workspace, see Git integration with Databricks Git folders.
Redesigned Workspace Settings UI
June 2-7, 2021: Version 3.47
The new Workspace Settings tab in the Admin Console replaces the Workspace Storage, Access Control Lists, and Advanced tabs for streamlined workspace configuration.
Updates to ListTokens and ListAllTokens database queries expired tokens
June 2-7, 2021: Version 3.47
Database queries ListTokens or ListAllTokens no longer return expired tokens. Indefinite tokens (those with expiryTime set to -1) are returned.
icon.