April 2021

These features and Databricks platform improvements were released in April 2021.

Note

Releases are staged. Your Databricks account may not be updated until a week or more after the initial release date.

Databricks Runtime 8.2 (GA)

April 22, 2021

Databricks Runtime 8.2 and Databricks Runtime 8.2 ML are now generally available.

For information, see the full release notes at Databricks Runtime 8.2 and Databricks Runtime 8.2 for Machine Learning.

Update running workspaces with new credentials or network configurations

April 20, 2021

You can now update a running workspace with new configurations for credentials (AWS IAM cross-account roles) and network (Customer-managed VPC) using either the account console or the Account API. The workspace must be on the E2 version of the Databricks platform.

Databricks can now send in-product messages and product tours directly to your workspace (Public Preview)

April 20-26, 2021: Version 3.44

Databricks users can now benefit from suggestions and product tours delivered as in-product messages, improving onboarding and engagement. This feature is enabled by default, but admins can disable it using the admin console. See Manage in-product messaging.

Easier job management with the enhanced jobs user interface

April 20-26, 2021: Version 3.44

Databricks has redesigned the Jobs user interface to make it easier to manage jobs. You can use the new job details page to perform all job related actions, including running, cloning, and deleting jobs.

Cluster policy changes are applied automatically to existing clusters at restart and edit

April 20-26, 2021: Version 3.44

It is now much easier to apply cluster policy changes to existing clusters. Any policy definition changes are applied to associated clusters at cluster restart and edit. Nonconforming clusters won’t be able to restart.

Track retries in your job tasks when task attempts fail

April 20-26, 2021: Version 3.44

You can now pass the task_retry_count parameter variable to a job task. The value of this variable is a count of the attempts to retry the task when the initial attempt fails. For more information see Task parameter variables.

Quickly view cluster details when you create a new cluster

April 20-26, 2021: Version 3.44

You will now see cluster information, including worker node and driver details, to the right of Create Cluster when you create a new all-purpose or job cluster.

MLflow sidebar reflects the most recent experiment

April 20-26, 2021: Version 3.44

The MLflow Experiment Runs sidebar now displays runs from the experiment that the notebook most recently logged to. Previously the sidebar showed runs only from the notebook experiment.

For details, see Track machine learning training runs.

Change to default channel for conda.yaml files in MLflow

April 20-26, 2021: Version 3.44

When you save a model to MLflow, you can specify a conda.yaml file specifying package dependencies. Previously, if you did not specify channels in the conda.yaml file, the model used the defaults channel. This has changed. If you do not specify a channel, the new default channel is conda-forge.

New free trial and pay-as-you-go customers are now on the E2 version of the platform

April 12, 2021

You can now sign up for the 14-day free trial on Databricks without having to provide credit card information up front, and all free trial customers now use the new, more user-friendly account console to set up their subscriptions, create workspaces, add account admins, manage billing, and view usage. Both free trial and pay-as-you-go customers now enjoy the full benefits of the E2 platform: multiple workspaces per account, enterprise-grade security, account management APIs, and more.

See Set up your Databricks account and deploy a workspace.

Databricks Runtime 8.2 (Beta)

April 8, 2021

Databricks Runtime 8.2 and Databricks Runtime 8.2 ML are now available as Beta releases.

For information, see the full release notes at Databricks Runtime 8.2 and Databricks Runtime 8.2 for Machine Learning.

User and group limits

April 5-12, 2021: Version 3.43

Each Databricks workspace is now limited to 10,000 users and 5,000 groups.

Easier monitoring of job run status

April 5-12, 2021: Version 3.43

The job run details page now automatically refreshes every 5 seconds to make it easier to monitor the progress of your jobs.

Better governance with enhanced audit logging

April 5-12, 2021: Version 3.43

The audit logs now capture canceled job run events, allowing you to better monitor and troubleshoot jobs running in your Databricks clusters. See the jobs events in the Workspace-level audit log events table.

Global init scripts no longer run on model serving clusters

April 5-12, 2021: Version 3.43

Global init scripts are run on every cluster in a workspace and can be used to enforce consistent cluster configurations. This configuration is typically not optimal for model serving clusters, so global init scripts are no longer run on model serving clusters.

If you need to run init scripts on model serving clusters, contact your Databricks representative.

Databricks Runtime 6.4 series support ends

April 1, 2021

Support for Databricks Runtime 6.4, Databricks Runtime 6.4 for Machine Learning, and Databricks Runtime 6.4 for Genomics ended on April 1. See Databricks runtime support lifecycle.

Databricks Runtime 6.4 Extended Support will be supported through the end of 2021. For more information, see Databricks Runtime 6.4 Extended Support.