September 2021
These features and Databricks platform improvements were released in September 2021.
Note
Releases are staged. Your Databricks account may not be updated until a week or more after the initial release date.
Databricks Runtime 10.0 (Beta)
September 29, 2021
Databricks Runtime 10.0, 10.0 Photon, and 10.0 ML are now available as Beta releases.
See the full release notes at Databricks Runtime 10.0 (EoS) and Databricks Runtime 10.0 for ML (EoS).
Databricks ODBC driver 2.6.18
September 28, 2021
We have released version 2.6.18 of the Databricks ODBC driver (download). This release prepares for the Unity Catalog by incorporating three-level namespace support and enabling management across multiple catalogs.
Customer control of workspace login by Databricks staff
September 24, 2021
You can now control at the account level how and when Databricks staff can log into your workspace for customer support. See Workspace access for Databricks personnel
Databricks Runtime 9.1 LTS and 9.1 LTS ML are GA; 9.1 LTS Photon is Public Preview
September 23, 2021
Databricks Runtime 9.1 LTS and 9.1 LTS ML are now generally available. 9.1 LTS Photon is in Public Preview. These images will be supported through September 2023.
Note
LTS means this version is under long-term support. See Databricks Runtime LTS version lifecycle.
See the full release notes at Databricks Runtime 9.1 LTS and Databricks Runtime 9.1 LTS for Machine Learning.
Databricks Runtime 8.1 series support ends
September 22, 2021
Support for Databricks Runtime 8.1 and Databricks Runtime 8.1 for Machine Learning ended on September 22. See Databricks support lifecycles.
Databricks JDBC driver 2.6.19
September 22, 2021
We have released version 2.6.19 of the Databricks JDBC driver (download). This release adds support for Cloud Fetch, which is a mechanism for fetching data in parallel through cloud storage to bring the data faster to BI tools. This release also reduces latency by removing unnecessary roundtrips between the client and the Databricks Runtime. See How We Achieved High-bandwidth Connectivity With BI Tools.
Security and usability improvements when resetting passwords
September 20-27, 2021: Version 3.55
When an account has multiple workspaces, a user is no longer able to initiate a password-reset request for a workspace the user doesn’t have access to. Previously, the user could request a password reset, but when the user clicked the link in the email to reset the password, the operation failed. Now, a user without access to the workspace receives a message that a password-reset email is being sent, but the email is not actually sent. This behavior helps prevent brute-force attempts to discover whether a given email address has access to a given workspace, as well as brute-force attempts to reset a password.
Repos now supports .gitignore
September 20-27, 2021: Version 3.55
Databricks Repos now supports .gitignore
, allowing you to specify that certain files not be tracked by Git. If you add a file to your repo and do not want it to be tracked by Git, create a .gitignore
file or use one cloned from your remote repository and add the filename, including the extension.
.gitignore
works only for files that are not already tracked by Git. If you add a file that is already tracked by Git to a .gitignore
file, the file is still tracked by Git.
For more information about Databricks Repos, see Git integration for Databricks Git folders.
Enhanced jobs UI is now standard for all workspaces
September 20-27, 2021: Version 3.55
An enhanced jobs UI featuring a Job details side panel providing access to common jobs settings is now standard for all workspaces. The classic UI is no longer supported, and the Switch back to the Classic UI button is removed from all workspaces. See Schedule and orchestrate workflows for more information on using the new jobs UI.
Databricks now available in region ap-southeast-1
September 15, 2021
Databricks is now available in region ap-southeast-1
. AWS PrivateLink is not supported for this region. See Databricks clouds and regions.
PrivateLink supported in all availability zones within the supported regions
September 14, 2021
Support for PrivateLink now includes all availability zones in the regions that support PrivateLink.
Databricks Runtime 9.1 (Beta)
September 14, 2021
Databricks Runtime 9.1, 9.1 Photon, and 9.1 ML are now available as Beta releases.
See the full release notes at Databricks Runtime 9.1 LTS and Databricks Runtime 9.1 LTS for Machine Learning.
Streamlined management of settings for Databricks jobs
September 8, 2021
In the new Job details side panel in the jobs UI, you can now manage common settings for your Databricks jobs, such as the job schedule, cluster settings, maximum concurrent runs, and job alerts.
Databricks SQL Public Preview available in all workspaces
September 7, 2021
Databricks SQL, an intuitive environment for running ad-hoc queries and creating dashboards on data stored in your data lake, has started rolling out in Public Preview to all workspaces. For more information, see the Databricks SQL release notes.
Delete feature tables from Feature Store
September 7-13, 2021: Version 3.54
You can now delete a feature table from Databricks Feature Store.
Grant view pipeline permissions in the Delta Live Tables UI (Public Preview)
September 7-13, 2021: Version 3.54
You can now grant users the permission to view your pipelines with the Edit Permissions button in the Delta Live Tables UI.
Reduce cluster resource usage with Delta Live Tables (Public Preview)
September 7-13, 2021: Version 3.54
You can now create a cluster for your Delta Live Tables pipeline with a single worker node, providing more control over the cluster resources used by the pipeline. To create a cluster with a single worker node, set Min Workers to 0 and Max Workers to 1 when you create a new pipeline. ## Development optimizations for Delta Live Tables (Public Preview)
September 7-13, 2021: Version 3.54
New pipelines created in Delta Live Tables now run in development mode by default. Development mode optimizes pipeline execution by reusing clusters and disabling automatic retries. Previously, production mode was the default for new pipelines.
Use MLflow models in your Delta Live Tables pipelines (Public Preview)
September 7-13, 2021: Version 3.54
You can now fetch models from the MLflow Model Registry and use these models for inference in a Delta Live Tables pipeline. See the ML lifecycle management using MLflow for more information on using the MLflow Model Registry with Databricks.
Find Delta Live Tables pipelines by name (Public Preview)
September 7-13, 2021: Version 3.54
You can now filter the list of pipelines on the Pipelines page, making it easier to find specific pipelines. You can filter by pipeline name or a partial text match on one or more pipeline names.
PyTorch TorchScript and other third-party libraries are now supported in Databricks jobs
September 7-13, 2021: Version 3.54
An issue causing failures with some third-party libraries in Python tasks, such as PyTorch TorchScript, has been fixed.
Databricks Runtime 8.0 series support ends
September 2, 2021
Support for Databricks Runtime 8.0 and Databricks Runtime 8.0 for Machine Learning ended on September 2. See Databricks support lifecycles.