January 2022

These features and Databricks platform improvements were released in January 2022.

Note

Releases are staged. Your Databricks account may not be updated until a week or more after the initial release date.

MLflow Model Registry Webhooks on Databricks (Public Preview)

January 31, 2022

You can now use webhooks to listen for MLFlow Model Registry events and trigger actions based on them. For example, you can use webhooks to automate and integrate your machine learning pipeline with existing CI/CD tools and workflows, trigger CI builds when a new model version is created, or notify your team members through Slack when a model transition to production is requested. For details, see MLflow Model Registry Webhooks on Databricks.

Breaking change: cluster idempotency token cleared on cluster termination

January 26, 2022

Beginning February 14th, 2022, Databricks cluster management will be modified to clear idempotency tokens when a cluster is terminated. This change is required to enable improvements to cluster management uptime guarantees.

You must update any clients or scripts that rely on an idempotency token to persist after terminating a cluster. If you submit a create cluster request using an idempotency token not assigned to a non-terminated cluster in the workspace, a new cluster is created.

The following illustrates the behavior before this change:

createCluster(idempotencyToken = tokenA) // launches new cluster A
terminateCluster(cluster A) // idempotency token tokenA is still associated with the terminated cluster A
startCluster(cluster = A)

createCluster(idempotencyToken = tokenA) // returns the currently running cluster A

The following illustrates the new behavior:

createCluster(idempotencyToken = tokenA) // launches new cluster A
terminateCluster(cluster A) // idempotencyToken token tokenA is cleared from the cluster A configuration
startCluster(cluster A)

createCluster(idempotencyToken = tokenA) // launches new cluster B

Databricks Runtime 10.3 (Beta)

January 26, 2022

Databricks Runtime 10.3, 10.3 Photon, and 10.3 ML are now available as Beta releases.

See the full release notes at Databricks Runtime 10.3 (Unsupported) and Databricks Runtime 10.3 for Machine Learning (Unsupported).

View information on recent job runs

January 25, 2022

You can now view in real time currently running and recently completed job runs in the Job runs page in the Databricks jobs user interface. See View recent job runs.

Use Markdown in Databricks Repos file editor

January 25-31, 2022: Version 3.64

The Databricks Repos file editor can now render markdown. For details, see Work with files in the UI.

Improved cluster management for jobs that orchestrate multiple tasks

January 24, 2022

You can now reuse job clusters in your jobs that orchestrate multiple tasks. Cluster reuse allows you to reduce cluster resource usage by using a single cluster to run multiple tasks in a job run. See Create, run, and manage Databricks Jobs.

Add or rotate the customer-managed key for managed services on a running workspace

January 25, 2022

Adding a customer-managed key (CMK) for managed services enables encryption with your customer-managed key for notebook source, secrets, and Databricks SQL query history. Previously, you needed to add the key when you created the workspace. You can now add this key on a running workspace or rotate (update) the key. Note that Databricks does not directly encrypt the data with the customer-managed key (CMK). Databricks uses both the CMK and the Databricks managed key (DMK) that is unique to your workspace to encrypt the Data Encryption Key (DEK). Databricks uses the DEK to encrypt the workspace’s managed services persisted data. Adding a CMK does not re-encrypt data that does not use a CMK. Rotating the CMK does not change the DEK. See Add or update a customer-managed key on a running workspace.

Delta Sharing Private Preview adds functionality and new terms

January 20, 2022

Delta Sharing Private Preview includes new functionality that must be enabled by an account admin.

Databricks Runtime 8.3 and Databricks Runtime 8.4 series support ends

January 20, 2022

Support for Databricks Runtime 8.3, Databricks Runtime 8.3 for Machine Learning, Databricks Runtime 8.4, and Databricks Runtime 8.4 for Machine Learning ended on January 20. See Databricks runtime support lifecycle.

Databricks JDBC driver 2.6.22

January 18, 2022

We have released version 2.6.22 of the Databricks JDBC driver (download). This release upgrades the log4j library to version 2.17.1 and removes the slf4j-log4j12 dependency.

Support for G5 family of GPU-accelerated EC2 instances (Public Preview)

January 10-18, 2022: Version 3.63

Databricks now supports G5 instances for GPU clusters. G5 instances can be used for a wide range of graphics-intensive and machine learning use cases.

New Share button replaces Permissions icon in notebooks

January 10-18, 2022: Version 3.63

To change permissions on a notebook, use the new Share button share button in the notebook UI, or select Permissions from the drop-down menu. The Permissions icon in the notebook toolbar Permissions icon has been removed. For more information about setting notebook permissions, see Configure notebook, folder, and repos permissions.