June 2021

These features and Databricks platform improvements were released in June 2021.

Note

Releases are staged. Your Databricks account may not be updated until a week or more after the initial release date.

Correction: Repos for Git is enabled by default in new and existing workspaces in some regions

June 30, 2021

In a release note for June 2, 2021, Databricks announced that Repos for Git is enabled by default in new workspaces in some regions, but omitted the following important detail: Repos for Git is also enabled in existing workspaces if Notebook Git Versioning is already enabled. The original release note has been updated.

Change to Feature Store permissions

June 28 - July 6, 2021: Version 3.49

Users with CAN_VIEW_METADATA permission on a feature table can now publish it to online stores.

Improved access to results in the MLflow runs table

June 28 - July 6, 2021: Version 3.49

Changes to the MLflow runs table provide quicker access to results.

  • The Metrics columns now appear before the Parameters columns, making it easier to identify the best run.
  • The Start Time column now shows the age of the run, not the start time. This reduces the width of the column and allows more information to be visible without horizontal scrolling. To display the start time, hover over the value in the column.
Hover over start time

Better cost visibility for Delta Live Tables

June 28 - July 6, 2021: Version 3.49

When you create a new pipeline or view the Pipeline Details UI, you will now see the DBU per hour for the Delta Live Tables default cluster.

Enhanced data quality constraints for Delta Live Tables

June 28 - July 6, 2021: Version 3.49

You can now use the expect_all_or_fail, expect_all_or_drop, and expect_all Python APIs to define multiple data quality constraints on Delta Live Tables datasets. These APIs are useful when applying a library of common expectations to one or more datasets in a pipeline. For more information, see Define data quality constraints.

API changes for updating and replacing IP address lists

June 28 - July 6, 2021: Version 3.49

A new PATCH operation for /ip-access-lists/<list-id> in the Databricks REST API 2.0 allows you to programmatically update the individual values in an existing IP address list. For more information, see Update an IP access list.

The existing PUT operation for /ip-access-lists/<list-id> in the Databricks REST API 2.0 allows you to programmatically replace the contents of an existing IP address list while maintaining the list’s original internal ID. For more information, see Replace an IP access list.

Use an API to download usage data directly

June 16, 2021: Version 3.48

You can now use the Account API to download usage data in CSV format directly without the need to set up delivery to an S3 bucket. See Download billable usage logs using the Account API.

Databricks Runtime 7.5 series support ends

June 16, 2021

Support for Databricks Runtime 7.5, Databricks Runtime 7.5 for Machine Learning, and Databricks Runtime 7.5 for Genomics ended on June 16. See Databricks runtime support lifecycle.

Optimize performance and control costs by using different pools for the driver node and worker nodes

June 14-21, 2021: Version 3.48

You can now optimize your cluster by attaching it to a different pool for the driver node and worker nodes. For example, you can use a compute-optimized pool for the driver node and a storage-optimized pool for nodes. This also helps you control costs by assigning different tags to each instance pool. If you choose, you can use the same pool for drivers and worker nodes.

For more information, see Attach a cluster to one or more pools.

Photon runtimes now support i3.xlarge instances (Public Preview)

June 14, 2021: Version 3.48

Photon is the new native vectorized engine on Databricks, directly compatible with Apache Spark APIs. To provision clusters with Photon you must use a Photon image. The Photon image is now supported on i3.xlarge instances, the default instance type when you create a cluster.

Registry-wide permissions for Model Registry

June 14-21, 2021: Version 3.48

You can now set default model permissions for all models in a workspace’s Model Registry. Permissions set at this level apply to all new and existing models registered in Model Registry in that workspace. For more information, see Configure permissions for all MLflow Models in Model Registry.

A user’s home directory is no longer protected when you delete a user using the SCIM API

June 14-21, 2021: Version 3.48

When you delete a user using the SCIM API, the user’s home directory is no longer protected, and a workspace administrator can delete it. This behavior matches what happens when you delete a user using the admin console.

Accelerate SQL workloads with Photon (Public Preview)

June 9, 2021: Version 3.47

Photon is the new native vectorized engine on Databricks, directly compatible with Apache Spark APIs. It is included as part of a new high-performance runtime designed to run your SQL workloads faster and reduce your total cost per workload.

To provision clusters with Photon you must use a Photon image. Photon images consume DBUs at a different rate than the same instance type running the non-Photon runtime. For more information about Photon instances and DBU consumption, see the Databricks pricing page.

Photon is available for Databricks Runtime 8.3 and Databricks Runtime 8.3 Photon and above.

Databricks Runtime 8.3 (GA)

June 8, 2021

Databricks Runtime 8.3 and Databricks Runtime 8.3 ML are now generally available.

For information, see the full release notes at Databricks Runtime 8.3 and Databricks Runtime 8.3 Photon and Databricks Runtime 8.3 for Machine Learning.

Python and SQL table access control (GA)

June 8, 2021

Table access control lets you programmatically grant and revoke access to your data using the Databricks view-based access control model. Table access control is available in two versions: SQL-only and Python and SQL. Previously only SQL-only table access control was generally available. Now Python and SQL table access control is generally available. See Enable table access control for a cluster.

Jobs UI and API now show the owner of a job run

June 2-7, 2021: Version 3.47

The jobs UI and API now include the user that a job will run as, making it easier to understand, debug, and audit job permissions. This is a read-only value set to the job creator if job access control is disabled or the job’s is_owner permission if job access control is enabled. For more information about managing job permissions, see Jobs access control.

Protect sensitive Spark configuration properties and environment variables using secrets (Public Preview)

June 7, 2021

You can protect sensitive Spark configuration properties and environment variables using secrets. Retrieved secrets are redacted from notebook output and Spark driver and executor logs. See Secrets.

If you currently use plaintext passwords in a Spark configuration, Databricks recommends that you rotate your passwords and use secret paths in the Spark configuration.

Repos for Git is enabled by default in new and existing workspaces in some regions

June 2, 2021

This release note was updated on June 30, 2021.

For new workspaces and for existing workspaces with Notebook Git Versioning enabled, Repos for Git is now enabled by default in some regions. Default Repos enablement will be rolled out to more regions in the coming months. To learn how to enable or disable Repos yourself, and to verify the settings for your workspace, see Repos for Git integration.

Redesigned Workspace Settings UI

June 2-7, 2021: Version 3.47

The new Workspace Settings tab in the Admin Console replaces the Workspace Storage, Access Control Lists, and Advanced tabs for streamlined workspace configuration.

Updates to ListTokens and ListAllTokens database queries expired tokens

June 2-7, 2021: Version 3.47

Database queries ListTokens or ListAllTokens no longer return expired tokens. Indefinite tokens (those with expiryTime set to -1) are returned.

Confirmation now required when granting or revoking Admin permissions

June 2-7, 2021: Version 3.47

A confirmation dialog now appears when an Admin user grants or revokes Admin privileges in the web UI.

Changes to keyboard shortcuts in the web UI

June 2-7, 2021: Version 3.47

In a notebook, the way to open the list of keyboard shortcuts has changed. To open the list, click the Keyboard Icon icon.