February 2021

These features and Databricks platform improvements were released in February 2021.

Note

Releases are staged. Your Databricks account may not be updated until a week or more after the initial release date.

New Databricks Power BI connector (GA)

February 26, 2021

The new Power BI connector for Azure Databricks, released in public preview in September 2020, is now GA. It provides:

Simple connection configuration: the new Power BI Databricks connector is integrated into Power BI, and you configure it using a simple dialog with a couple of clicks.
Faster imports and optimized metadata calls, thanks to the new Databricks ODBC driver, which comes with significant performance improvements.
Access to Databricks data through Power BI respects Databricks table access control.

For more information, see Connect Power BI to Databricks.

Add and manage account admins using the SCIM API (E2 accounts, Public Preview)

February 25, 2021

If your account is on the E2 version of the platform, the SCIM API (Account admins) joins the account console as a way to add multiple account administrators and give them access to the account console and Account API, where they can manage your Databricks account-level configurations, including workspace creation, delegated AWS access credentials, AWS storage configurations, and optional customer-managed VPCs. See Account Groups API.

Added modification_time to the DBFS REST API get-status and list responses

February 23 - March 2, 2021: Version 3.40

A file object’s last modification time is now available in the DBFS REST API. The modification_time is now returned in the FileInfo response structure by both the get-status and list API endpoints. An example JSON response for a get-status call on a directory:

{"path": "/test-directory", "is_dir": true, "file_size": 0, "modification_time": 1610366751000}

Easily copy long experiment names in MLflow

February 16-23, 2021: Version 3.39

You can now copy the experiment name from the experiment page, making it easier to set the active MLflow experiment to which a notebook logs runs. See Get experiment ID and path to experiment.

Adjust memory size and number of cores for serving clusters

February 16-23, 2021: Version 3.39

You can now customize clusters used for MLflow model serving. See Feature store integration.

Web terminal is now GA

February 16-23, 2021: Version 3.39

Web terminal is now generally available. It provides a convenient and highly interactive way for you to run shell commands and use editors on the Spark driver node. See Run shell commands in Databricks web terminal.

Auto-AZ: automatic selection of availability zone (AZ) when you launch clusters, available on all deployment types

February 16-23, 2021: Version 3.39

For all workspaces, regardless of deployment type, you can now enable automatic selection of availability zone (AZ) when your users launch clusters (including job clusters). After you enable Auto-AZ using the Clusters API, Databricks chooses an AZ automatically based on available IPs in the workspace subnets, and retries in other AZs in case insufficient capacity errors are returned by AWS.

See Availability zones.

Delegate account management beyond the account owner (E2 accounts, Public Preview)

February 16-23, 2021: Version 3.39

If your account uses the new account console, you can add multiple account administrators and give them access to the account console to manage your Databricks account-level configurations, including workspace creation, delegated AWS access credentials, AWS storage configurations, and optional customer-managed VPCs. See Assign account admin roles to a user.

Separate account-level and workspace-level audit logging configurations help you monitor account and workspace activity more effectively

February 16-23, 2021: Version 3.39

All audit log delivery is now “account level” or “workspace level” based on settings you make in your audit log delivery configuration. Account-level configurations deliver logs for all workspaces in a Databricks account as well as for events with no workspace association. Workspace-level configurations deliver logs only for specific workspaces.

There are also two types of audit events: workspace-level events that apply to a specific workspace and account-level events that apply to your account as a whole.

See Audit log reference.

Databricks Runtime 7.2 series support ends

February 11, 2021

Support for Databricks Runtime 7.2, Databricks Runtime 7.2 for Machine Learning, and Databricks Runtime 7.2 for Genomics ended on February 11. See Databricks support lifecycles.

Databricks Runtime 7.6 GA

February 8, 2021

Databricks Runtime 7.6 and Databricks Runtime 7.6 ML are now generally available.

For information, see the full release notes at Databricks Runtime 7.6 (EoS) and Databricks Runtime 7.6 for Machine Learning (EoS).

Databricks Runtime 8.0 (Beta)

February 1, 2021

Databricks Runtime 8.0 and Databricks Runtime 8.0 ML are now available as Beta releases.

For information, see the full release notes at Databricks Runtime 8.0 (EoS) and Databricks Runtime 8.0 for ML (EoS).

Databricks Runtime for Genomics now deprecated

February 1, 2021

Databricks is no longer building new Databricks Runtime for Genomics releases and will remove support for Databricks Runtime for Genomics on September 24, 2022, when Databricks Runtime for Genomics 7.3 LTS support ends. At that point Databricks Runtime for Genomics will no longer be available for selection when you create a cluster.

For more information about the Databricks Runtime deprecation policy and schedule, see Supported Databricks Runtime LTS releases.

View more readable JSON in the MLflow run artifact display

February 4-11, 2021: Version 3.38

JSON output in the MLflow artifact panel is now in a more readable format.

Provide comments in the Model Registry using REST API

February 4-11, 2021: Version 3.38

You can now add comments in the Model Registry using the REST API.

Easily specify default cluster values in API calls

February 4-11, 2021: Version 3.38

Now you can use the applyDefaultPolicyValues field when you create and edit all-purpose clusters using the Databricks Clusters API. When true, it uses policy default values for missing cluster attributes.

Tune cluster worker configuration according to current worker allocation

February 4-11, 2021: Version 3.38

The cluster list displays the number of workers allocated to each running cluster. This number is now shown in the cluster detail page so you can easily compare current sizing of the cluster with the cluster configuration details and make configuration adjustments as needed.

Pass context specific information to a job’s task with task parameter variables

February 4-11, 2021: Version 3.38

You can now pass variables to tasks containing context specific information such as the job ID or execution start time. For more information see What is a dynamic value reference?.

Error messages from job failures no longer contain possibly sensitive information

February 4-11, 2021: Version 3.38

Job failure error messages no longer include stack traces from Java exceptions. The stack traces, included in the miscMessage field, potentially exposed sensitive information to users.

Download usage data from the new account console for E2 accounts

February 4, 2021

The account console for customers on the E2 platform now includes a button on the Usage tab to download a CSV file of account usage data.