April 2021

These features and Databricks platform improvements were released in April 2021.

Note

Releases are staged. Your Databricks account may not be updated until a week or more after the initial release date.

Databricks Runtime 8.2 (GA)

April 22, 2021

Databricks Runtime 8.2 and Databricks Runtime 8.2 ML are now generally available.

For information, see the full release notes at Databricks Runtime 8.2 (EoS) and Databricks Runtime 8.2 for ML (EoS).

AWS PrivateLink for Databricks workspaces (Public Preview)

April 20, 2021

You can now use AWS PrivateLink to provision secure private workspaces by creating VPC endpoints to both the front-end and back-end interfaces of the Databricks infrastructure. The front-end VPC endpoint ensures that your users connect to the Databricks web application, REST APIs and JDBC/ODBC interface over your private network and the AWS network backbone. The back-end VPC endpoints ensure that clusters in your own managed VPC connect to the secure cluster connectivity relay and REST APIs over AWS network backbone. The workspace must be on the E2 version of the Databricks platform.

For detailed setup instructions, see Enable private connectivity using AWS PrivateLink.

Update running workspaces with new credentials or network configurations

April 20, 2021

You can now update a running workspace with new configurations for credentials (AWS IAM cross-account roles) and network (Configure a customer-managed VPC) using either the account console or the Account API. The workspace must be on the E2 version of the Databricks platform.

Databricks can now send in-product messages and product tours directly to your workspace (Public Preview)

April 20-26, 2021: Version 3.44

Databricks users can now benefit from suggestions and product tours delivered as in-product messages, improving onboarding and engagement. This feature is enabled by default, but admins can disable it using the admin settings page.

Easier job management with the enhanced jobs user interface

April 20-26, 2021: Version 3.44

Databricks has redesigned the Schedule and orchestrate workflows user interface to make it easier to manage jobs. You can use the new job details page to perform all job related actions, including running, cloning, and deleting jobs.

Cluster policy changes are applied automatically to existing clusters at restart and edit

April 20-26, 2021: Version 3.44

It is now much easier to apply cluster policy changes to existing clusters. Any policy definition changes are applied to associated clusters at cluster restart and edit. Nonconforming clusters won’t be able to restart.

Track retries in your job tasks when task attempts fail

April 20-26, 2021: Version 3.44

You can now pass the task_retry_count parameter variable to a job task. The value of this variable is a count of the attempts to retry the task when the initial attempt fails. For more information see What is a dynamic value reference?.

Quickly view cluster details when you create a new cluster

April 20-26, 2021: Version 3.44

You will now see cluster information, including worker node and driver details, to the right of Create Cluster when you create a new all-purpose or job cluster.

Change to default channel for `conda.yaml` files in MLflow

April 20-26, 2021: Version 3.44

When you save a model to MLflow, you can specify a conda.yaml file specifying package dependencies. Previously, if you did not specify channels in the conda.yaml file, the model used the defaults channel. This has changed. If you do not specify a channel, the new default channel is conda-forge.

New free trial and pay-as-you-go customers are now on the E2 version of the platform

April 12, 2021

You can now sign up for the 14-day free trial on Databricks without having to provide credit card information up front, and all free trial customers now use the new, more user-friendly account console to set up their subscriptions, create workspaces, add account admins, manage billing, and view usage. Both free trial and pay-as-you-go customers now enjoy the full benefits of the E2 platform: multiple workspaces per account, enterprise-grade security, account management APIs, and more.

See Get started with Databricks.

Databricks Runtime 8.2 (Beta)

April 8, 2021

Databricks Runtime 8.2 and Databricks Runtime 8.2 ML are now available as Beta releases.

For information, see the full release notes at Databricks Runtime 8.2 (EoS) and Databricks Runtime 8.2 for ML (EoS).

User and group limits

April 5-12, 2021: Version 3.43

Each Databricks workspace is now limited to 10,000 users and 5,000 groups.

Easier monitoring of job run status

April 5-12, 2021: Version 3.43

The job run details page now automatically refreshes every 5 seconds to make it easier to monitor the progress of your jobs.

Better governance with enhanced audit logging

April 5-12, 2021: Version 3.43

The audit logs now capture canceled job run events, allowing you to better monitor and troubleshoot jobs running in your Databricks clusters. See Job events.

Global init scripts no longer run on model serving clusters

April 5-12, 2021: Version 3.43

Global init scripts are run on every cluster in a workspace and can be used to enforce consistent cluster configurations. This configuration is typically not optimal for model serving clusters, so global init scripts are no longer run on model serving clusters.

If you need to run init scripts on model serving clusters, contact your Databricks account team.

Databricks Runtime 6.4 series support ends

April 1, 2021

Support for Databricks Runtime 6.4, Databricks Runtime 6.4 for Machine Learning, and Databricks Runtime 6.4 for Genomics ended on April 1. See Databricks support lifecycles.

Databricks Runtime 6.4 Extended Support will be supported through the end of 2021. For more information, see Databricks Runtime 6.4 Extended Support (EoS).