November 2020

These features and Databricks platform improvements were released in November 2020.

Note

Releases are staged. Your Databricks account may not be updated until a week or more after the initial release date.

Databricks Runtime 6.6 series support ends

November 26, 2020

Support for Databricks Runtime 6.6, Databricks Runtime 6.6 for Machine Learning, and Databricks Runtime 6.6 for Genomics ended on November 26. See Databricks support lifecycles.

MLflow Model Registry GA

November 18 - December 1, 2020: Version 3.33

MLflow Model Registry is now GA. Several improvements have been made since Model Registry was released for Public Preview:

  • Audit logging for actions on model registry objects. Actions in Model Registry are now captured in audit logs. See the modelRegistry entry in the audit log reference for the logged actions and parameters.

  • Comments for model versions. You can now add comments to model versions, allowing you to use Model Registry for team discussions to help manage your model productionization pipeline.

  • Tags on models and model versions. You can create tags for models and model versions, and search for them using the API.

  • Improvements to the URL of the registered models page. The URL of this page now keeps its history, so you can navigate with the browser back and forward buttons as you make queries and view models from this page. You can also share the URL with colleagues who will see the same view.

Filter experiment runs based on whether a registered model is associated

November 18 - December 1, 2020: Version 3.33

When viewing runs for an experiment, you can now filter runs based on whether they created a model version or not. For more information, see Filter runs.

Cluster policies now use allowlist and blocklist as policy type names

November 18 - December 1, 2020: Version 3.33

Cluster policies now use “allowlist” and “blocklist” as policy types, replacing “whitelist” and “blacklist.” See Compute policy reference. Note that this was originally announced as a version 3.31 feature, which was incorrect.

Automatic retries when the creation of a job cluster fails

Important

This update was reverted following the release of version 3.33.

November 18 - December 1, 2020: Version 3.33

Databricks now automatically retries the creation of job clusters when specific recoverable errors occur. Job runs remain in RunLifeCycleState: PENDING until successful cluster launch. Each attempt has a different cluster_id and name. When cluster creation succeeds, the run transitions to RunLifeCycleState: RUNNING.

Databricks SQL (Public Preview)

November 18, 2020

Databricks is pleased to introduce Databricks SQL, an intuitive environment for running ad-hoc queries and creating dashboards on data stored in your data lake. Databricks SQL empowers your organization to operate a multi-cloud lakehouse architecture that provides data warehousing performance with data lake economics while providing a delightful SQL analytics user experience. Databricks SQL:

  • Integrates with the BI tools you use today, like Tableau and Microsoft Power BI, to query the most complete and recent data in your data lake.

  • Complements existing BI tools with a SQL-native interface that allows data analysts and data scientists to query data lake data directly within Databricks.

  • Enables you to share query insights through rich visualizations and drag-and-drop dashboards with automatic alerting for important data changes.

  • Uses SQL warehouses to bring reliability, quality, scale, security, and performance to your data lake, so you can run traditional analytics workloads using your most recent and complete data.

See the What is data warehousing on Databricks? for details.

Web terminal available on Databricks Community Edition

November 5, 2020: Version 3.32

Web terminal is now enabled by default on Databricks Community Edition. For more information, see Run shell commands in Databricks web terminal.

Single Node clusters now support Databricks Container Services

November 4-10, 2020: Version 3.32

You can now use Databricks Container Services on Single Node clusters. For more information, see Single-node or multi-node compute and Customize containers with Databricks Container Service.

Databricks Runtime 7.4 GA

November 3, 2020

Databricks Runtime 7.4, Databricks Runtime 7.4 ML, and Databricks Runtime 7.4 for Genomics are now generally available.

For information, see the full release notes at Databricks Runtime 7.4 (EoS) and Databricks Runtime 7.4 for ML (EoS).

Databricks JDBC driver update

November 3, 2020

A new version of the Databricks JDBC driver has been released. The new version contains a number of bug fixes, most notably, the driver now returns the correct number of modified rows from DML operations when it is provided by Databricks Runtime.

Databricks Connect 7.3 (Beta)

November 3, 2020

Databricks Connect 7.3 is now available as a Beta release.