January 2021

These features and Databricks platform improvements were released in January 2021.

Note

Releases are staged. Your Databricks account may not be updated until a week or more after the initial release date.

This month saw the release of Databricks platform version 3.37. There was no release of versions 3.35 or 3.36.

Databricks Runtime 7.1 series support ends

January 21, 2021

Support for Databricks Runtime 7.1, Databricks Runtime 7.1 for Machine Learning, and Databricks Runtime 7.1 for Genomics ended on January 21. See Databricks support lifecycles.

Start clusters faster with Docker images preloaded into instance pools

January 20-28, 2021: Version 3.37

When you create an instance pool using the API, you can now specify preloaded Docker images. Pool clusters that use a preloaded Docker image start faster as they do not have to wait for the image to download. See Create instance pool.

Notebook find and replace now supports changing all occurrences of a match

January 20-28, 2021: Version 3.37

You can now choose to replace all matches when using the find and replace capability in a notebook. For details, see Find and replace text.

Single Node clusters (GA)

January 20-28, 2021: Version 3.37

A Single Node cluster is a cluster consisting of a Spark driver and no Spark workers. In contrast, Standard mode clusters require at least one Spark worker to run Spark jobs. Single Node clusters are helpful in the following situations:

  • Running single node machine learning workloads that need Spark to load and save data

  • Lightweight exploratory data analysis (EDA)

For details, see Single-node or multi-node compute.

Free form cluster policy type renamed to Unrestricted

January 20-28, 2021: Version 3.37

The Free form cluster policy type has been renamed to Unrestricted. For details, see Create and manage compute policies and Policies.

Cluster policy field not shown if a user only has access to one policy

January 20-28, 2021: Version 3.37

When you create a standard or jobs cluster, the Cluster Policy field does not appear if you have access to only one policy or if no policies have been defined.

G4 family of GPU-accelerated EC2 instances GA

January 20, 2021

G4 instances are now generally available. G4 instances are optimized for deploying machine learning models in production.

Databricks Runtime 7.0 series support ends

January 14, 2021

Support for Databricks Runtime 7.0, Databricks Runtime 7.0 for Machine Learning, and Databricks Runtime 7.0 for Genomics ended on January 14. See Databricks support lifecycles.

Billable usage and audit log S3 bucket policy and object ACL changes

January 14, 2021

Databricks now supports automatic usage and audit log delivery to an S3 bucket that is hosted in an AWS account that is different from the account of the IAM role that you specify for storage access. Databricks provides a template from which to create the S3 bucket policy required for cross-account log delivery.

In addition, the ownership of usage and audit logs delivered to your S3 bucket can now leverage the BucketOwnerFullControl Canned ACL instead of an IAM role, giving account owners and designees full ownership.

See Configure audit log delivery and Deliver and access billable usage logs.

E2 platform comes to the Asia Pacific region

January 4, 2021

We are excited to announce that the E2 version of the platform is now available in AWS Asia Pacific regions. You can now create multiple workspaces per account in:

  • ap-southeast-2 (Sydney

  • ap-south-1 (Mumbai)

  • ap-northeast-1 (Tokyo)

This is in addition to the existing region support for E2:

  • us-west-2 (Oregon)

  • us-west-1 (N. California)

  • us-east-1 (N. Virginia)

  • us-east-2 (Ohio)

  • ca-central-1 (Montreal)

  • eu-west-2 (Ireland)

  • eu-central-1 (Frankfurt)