July 2021

These features and Databricks platform improvements were released in July 2021.

Note

Releases are staged. Your Databricks account may not be updated until a week or more after the initial release date.

Manage MLflow experiment permissions with the Databricks REST API

July 29 - Aug 2, 2021: Version 3.51

You can now use the Databricks REST APIs’ Permissions API to manage which users can read, edit, or manage MLflow experiments. See Permissions API.

Databricks web interface is localized in Portuguese and French (Public Preview)

July 29, 2021

You can now use Databricks in French and Portuguese, and more languages are planned for the following quarters. Go to Settings in the upper-right menu in the web UI and click the Preferences tab to change your language.

Databricks Runtime 5.5 LTS for Machine Learning support ends, replaced by Extended Support version

July 27, 2021

Databricks Runtime 5.5 ML LTS support ended on July 27, 2021, but the new Databricks Runtime 5.5 ML Extended Support (EoS) will be supported through the end of 2021. Databricks Runtime 5.5 ML Extended Support uses Ubuntu 18.04.5 LTS instead of the deprecated Ubuntu 16.04.6 LTS distribution used in the original Databricks Runtime 5.5 ML LTS. Ubuntu 16.04.6 LTS support ceased on April 1, 2021. For more information about the differences between Databricks Runtime 5.5 Extended Support and Databricks Runtime 5.5 LTS, see Databricks Runtime 5.5 ML Extended Support (EoS).

Databricks Light 2.4 support ends September 5, replaced by Extended Support version

July 27, 2021

Support for the original Databricks Light 2.4 ends on September 5. 2021, but the new Databricks Light 2.4 Extended Support (EoS) will be supported through July 27, 2022. Databricks Light 2.4 Extended Support uses Ubuntu 18.04.5 LTS instead of the deprecated Ubuntu 16.04.6 LTS distribution used in the original Databricks Light 2.4. Ubuntu 16.04.6 LTS support ceased on April 1, 2021. Databricks recommends that you migrate your Light workloads to the extended support version as soon as you can.

Reduced permissions for cross-account IAM roles

July 26 - August 2, 2021: Version 3.51

The following permissions are no longer required for cross-account IAM roles that you use with Databricks, and you can remove them:

  • CreatePlacementGroup

  • DeletePlacementGroup

  • DescribePlacementGroup

Feature freshness information available in Databricks Feature Store UI (Public Preview)

July 26 - August 2, 2021: Version 3.51

The Databricks Feature Store UI now shows the last time that data was written to a feature table. It also shows whether a feature table is updated by a scheduled job, and if so, the schedule and status of the job.

Display up to 10,000 result rows

July 26 - August 2, 2021: Version 3.51

Previously a maximum of 1000 rows was displayed in a notebook. Now, if a table returns more than 1000 rows, you can choose to re-execute the query and display up to 10,000 rows. Requires Databricks Runtime 8.4 or above. For more information, see Download results.

Bulk import and export notebooks in a folder as source files

July 26 - August 2, 2021: Version 3.51

You can now export the source for all notebooks in a workspace folder as a ZIP archive. You can import the ZIP archive into another workspace or another folder in the same workspace to recreate all the exported notebooks there. A notebook’s source file is in the notebook’s default language. For example, a Python notebook’s source is a Python file.

Notebook source files do not include notebook command results and are appropriate for using in a CI/CD pipeline. Previously, you could export the source for only one notebook at a time, from within the notebook editor, or you could export all notebooks in a folder as HTML files or as a Databricks archive (with the extension .dbc), which would include notebook command results unless the notebook was cleared before exporting.

For more information about importing and exporting notebooks, see Manage notebooks.

Autocomplete in SQL notebooks now uses all-caps for SQL keywords

July 26 - August 2, 2021: Version 3.51

When you use TAB to autocomplete a SQL keyword in a notebook, the autocompleted result is now capitalized. For example, SELECT is shown, rather than select. This will not effect the query since SQL keywords are case-insensitive.

Reorderable and resizable widgets in notebooks

July 26 - August 2, 2021: Version 3.51

You can now change the order and sizes of Databricks widgets in a notebook by clicking edit icon in the notebook’s widgets bar. See Configure widget settings.

Databricks UI usability fixes

July 26 - August 2, 2021: Version 3.51

This release resolves an issue that prevented using the ESC key to close some dialogs in the Databricks UI, including:

  • The View JSON dialog in the jobs UI.

  • The Edit Schedule dialog in the jobs UI.

  • The Single Logout Configurations dialog in the admin UI.

  • The Create Notebook dialog.

Quickly define pipeline settings when you create a new Delta Live Tables pipeline

July 26 - August 2, 2021: Version 3.51

When you create a new pipeline, you can now define settings directly in the Create Pipeline dialog in the Delta Live Tables user-interface.

Databricks Runtime 8.4 and 8.4 ML are GA; 8.4 Photon is Public Preview

July 20, 2021

Databricks Runtime 8.4 and 8.4 ML are now generally available. 8.4 Photon is in Public Preview.

For information, see the full release notes at Databricks Runtime 8.4 (EoS) and Databricks Runtime 8.4 for ML (EoS).

Use Spark SQL with the Delta Live Tables Python API

July 12-19, 2021: Version 3.50

You can now use the following functions with the Delta Live Tables Python API:

  • The spark.table function to perform batch reads on datasets defined in a pipeline.

  • The spark.sql function to use a SQL query to define a dataset. SQL queries can use Delta Live Tables specific features, including references to internal datasets using the LIVE keyword and incremental reads using the STREAM keyword.

For more details, see the Python language reference.

Enhanced data processing and analysis with Databricks jobs (Public Preview)

July 12-19, 2021: Version 3.50

Databricks is pleased to announce an update to Databricks jobs that allows you to orchestrate and run a job with multiple tasks. You can use this to define a directed acyclic graph (DAG) of task dependencies, making it easier to build and monitor reliable data and machine learning workflows. This feature is in public preview and is disabled by default.

Important

Once you enable this feature, you cannot disable it.

See Schedule and orchestrate workflows for details.

Reduced cost for Delta Live Tables default clusters (Public Preview)

July 12-19, 2021: Version 3.50

To control costs, the following changes have been made to Delta Live Tables default clusters:

  • Change default instance type to c5.2xlarge.

  • Enable autoscaling by default with min workers equal to 1 and max workers equal to 5.

Sort pipelines by name in the Delta Live Tables UI (Public Preview)

July 12-19, 2021: Version 3.50

You can now click the Name column header in the Delta Live Tables user interface to sort pipelines by name in ascending order (A -> Z) or descending order (Z -> A).

Changes to Compute page

July 12-19, 2021: Version 3.50

The Libraries, Job Run, Spark UI, and Logs links from the Compute page have been removed. To see information about libraries, or to view the Spark UI and logs, use the corresponding tabs on the cluster detail page. To see information about a job run, click Go To Job Run on that page.

Databricks Runtime 5.5 LTS support ends, replaced by Databricks Runtime 5.5 Extended Support through the end of 2021

July 9, 2021

Support for Databricks Runtime 5.5 LTS and Databricks Runtime 5.5 LTS for Machine Learning ends on July 10. See Databricks support lifecycles.

Databricks Runtime 5.5 Extended Support will be supported through the end of 2021. Databricks Runtime 5.5 Extended Support uses Ubuntu 18.04.5 LTS instead of the deprecated Ubuntu 16.04.6 LTS distribution used in the original Databricks Runtime 5.5 LTS. Ubuntu 16.04.6 LTS support ceased on April 1, 2021. For more information about the differences between Databricks Runtime 5.5 Extended Support and Databricks Runtime 5.5 LTS, see Databricks Runtime 5.5 Extended Support (EoS).

Repos API (Public Preview)

July 6, 2021

You can now programmatically update a Databricks repo to the latest version of a specific Git branch or to a tag using the new Repos API. For details, see Repos API.

Databricks Runtime 8.4 (Beta)

July 1, 2021

Databricks Runtime 8.4 and Databricks Runtime 8.4 ML are now available as Beta releases.

For information, see the full release notes at Databricks Runtime 8.4 (EoS) and Databricks Runtime 8.4 for ML (EoS).