January 2022

These features and Databricks platform improvements were released in January 2022.

Note

Releases are staged. Your Databricks account may not be updated until a week or more after the initial release date.

MLflow Model Registry Webhooks on Databricks (Public Preview)

January 31, 2022

You can now use webhooks to listen for MLflow Model Registry events and trigger actions based on them. For example, you can use webhooks to automate and integrate your machine learning pipeline with existing CI/CD tools and workflows, trigger CI builds when a new model version is created, or notify your team members through Slack when there is a request to transition a model to production.

Breaking change: cluster idempotency token cleared on cluster termination

January 26, 2022

Beginning February 14th, 2022, Databricks cluster management will be modified to clear idempotency tokens when a cluster is terminated. This change is required to enable improvements to cluster management uptime guarantees.

You must update any clients or scripts that rely on an idempotency token to persist after terminating a cluster. If you submit a create cluster request using an idempotency token not assigned to a non-terminated cluster in the workspace, a new cluster is created.

The following illustrates the behavior before this change:

createCluster(idempotencyToken = tokenA) // launches new cluster A
terminateCluster(cluster A) // idempotency token tokenA is still associated with the terminated cluster A
startCluster(cluster = A)

createCluster(idempotencyToken = tokenA) // returns the currently running cluster A

The following illustrates the new behavior:

createCluster(idempotencyToken = tokenA) // launches new cluster A
terminateCluster(cluster A) // idempotencyToken token tokenA is cleared from the cluster A configuration
startCluster(cluster A)

createCluster(idempotencyToken = tokenA) // launches new cluster B

Databricks Runtime 10.3 (Beta)

January 26, 2022

Databricks Runtime 10.3, 10.3 Photon, and 10.3 ML are now available as Beta releases.

See the full release notes at Databricks Runtime 10.3 (unsupported) and Databricks Runtime 10.3 for ML (unsupported).

View information on recent job runs

January 25, 2022

You can now view in real time currently running and recently completed job runs in the Job runs page in the Databricks jobs user interface. See View recent job runs.

Use Markdown in Databricks Repos file editor

January 25-31, 2022: Version 3.64

The Databricks Repos file editor can now render markdown. For details, see Git integration with Databricks Git folders.

Improved cluster management for jobs that orchestrate multiple tasks

January 24, 2022

You can now reuse job clusters in your jobs that orchestrate multiple tasks. Cluster reuse allows you to reduce cluster resource usage by using a single cluster to run multiple tasks in a job run. See Create and run Databricks Jobs.

Add or rotate the customer-managed key for managed services on a running workspace

January 25, 2022

Adding a customer-managed key (CMK) for managed services enables encryption with your customer-managed key for notebook source, secrets, and Databricks SQL query history. Previously, you needed to add the key when you created the workspace. You can now add this key on a running workspace or rotate (update) the key. Note that Databricks does not directly encrypt the data with the customer-managed key (CMK). Databricks uses both the CMK and the Databricks managed key (DMK) that is unique to your workspace to encrypt the Data Encryption Key (DEK). Databricks uses the DEK to encrypt the workspace’s managed services persisted data. Adding a CMK does not re-encrypt data that does not use a CMK. Rotating the CMK does not change the DEK. See Customer-managed keys for encryption.

Delta Sharing Private Preview adds functionality and new terms

January 20, 2022

Delta Sharing Private Preview includes new functionality that must be enabled by an account admin.

Address AWS GuardDuty alerts related to Databricks access to your S3 bucket

January 20, 2022

If you use AWS GuardDuty and you access data using AWS IAM instance profiles, GuardDuty may create alerts for default Databricks behavior. You can configure your Databricks deployment to address these alerts. See AWS GuardDuty alerts related to S3 commit service.

Databricks Runtime 8.3 and Databricks Runtime 8.4 series support ends

January 20, 2022

Support for Databricks Runtime 8.3, Databricks Runtime 8.3 for Machine Learning, Databricks Runtime 8.4, and Databricks Runtime 8.4 for Machine Learning ended on January 20. See Databricks runtime support lifecycles.

Databricks JDBC driver 2.6.22

January 18, 2022

We have released version 2.6.22 of the Databricks JDBC driver (download). This release upgrades the log4j library to version 2.17.1 and removes the slf4j-log4j12 dependency.

Support for G5 family of GPU-accelerated EC2 instances (Public Preview)

January 10-18, 2022: Version 3.63

Databricks now supports G5 instances for GPU clusters. G5 instances can be used for a wide range of graphics-intensive and machine learning use cases.

New Share button replaces Permissions icon in notebooks

January 10-18, 2022: Version 3.63

To change permissions on a notebook, use the new Share button in the notebook UI, or select Permissions from the drop-down menu. The Permissions icon in the notebook toolbar has been removed. For more information about setting notebook permissions, see Collaborate using Databricks notebooks.

New workspace language options

January 10, 2022

Workspaces are now available in the following languages:

• Deutsch

• Italiano

• 日本語

• 한국어