Amazon
Web Services
Microsoft Azure
Google Cloud PlatformAugust 2023
These features and Databricks platform improvements were released in August 2023.
Note
Releases are staged. Your Databricks workspace might not be updated until a week or more after the initial release date.
Databricks CLI updated to version 0.203.3 (Public Preview)
August 30, 2023
The Databricks command-line interface (Databricks CLI) has been updated to version 0.203.3. For details, see the changelog for version 0.203.3.
Databricks JDBC driver 2.6.34
August 30, 2023
We have released version 2.6.34 of the Databricks JDBC driver (download). This release addresses a number of vulnerability issues by upgrading third-party libraries. In addition, this release fixes the following issues:
Thread leakage when a query fails to connect to the server.
Missing comments in the
REMARKScolumn of the table metadata.When a resultset closure operation returns an error, the connector does not clean up the operation handle entries from the heartbeat thread.
The release also improves error message handling by honoring SQLState and error codes from the server, and checks the server protocol version correctly.
Databricks SDK for Go updated to version 0.18.0 (Beta)
August 29, 2023
Databricks SDK for Go version 0.18.0 introduces 17 breaking API changes, adds 30 APIs, and adds about a dozen fields. For details, see the changelog for version 0.18.0.
Databricks SDK for Python updated to version 0.7.0 (Beta)
August 29, 2023
Databricks SDK for Python version 0.7.0 introduces a breaking change for the files API as well as 17 additional breaking changes. This release also adds 14 fields and adds 27 dataclasses. For details, see the changelog for version 0.7.0.
Databricks Terraform provider updated to version 1.24.1
August 29, 2023
Databricks Terraform provider version 1.24.1 includes a fix for verification of workspace reachability. For details, see the changelog for version 1.24.1.
Init scripts on DBFS end of life extended to Dec 1, 2023
August 29, 2023
Databricks previously announced that init scripts on DBFS would reach end of life on Sept 1, 2023. The end of life date has been extended to Dec 1, 2023.
Databricks Runtime 14.0 (Beta)
August 29, 2023
Databricks Runtime 14.0 and Databricks Runtime 14.0 ML are now available as Beta releases.
See Databricks Runtime 14.0 (EoS) and Databricks Runtime 14.0 for Machine Learning (EoS).
Custom workspace tags
August 28, 2023
You can now specify custom workspace tags during workspace creation using the Account API. See Create a workspace using the Account API.
Allow additional ports from your classic compute plane to control plane by January 31, 2024
August 28, 2023
If you configured a security group or firewall on the VPC in your classic compute plane (where your compute resources for clusters run) that restricts the open ports for communication to the Databricks control plane, you must configure your classic compute plane network to allow egress on additional ports to the control plane. In the future, these endpoints will be required. For new and existing workspaces, you must open up the ports 8443 through 8451 by January 31, 2024.
If you enabled back-end PrivateLink connectivity on a workspace, see the article for enabling PrivateLink and the AWS article on security groups.
Otherwise, see the article for customer-managed VPC and the AWS article on security groups.
The fully qualified domain name (FQDN) of the Databricks workspace remains the same. Databricks will manage the required network configuration changes for serverless compute resources. Contact help@databricks.com with any questions about this change.
Databricks Terraform provider updated to version 1.24.0
August 25, 2023
Databricks Terraform provider version 1.24.0 adds account-level API support for Unity Catalog objects, adds acl_principal_id to several data sources, and more. For details, see the changelog for version 1.24.0.
Databricks Runtime for Genomics setting removed from the workspace admin settings page
August 25, 2023
Workspace admins can no longer enable or disable Databricks Runtime for Genomics from the workspace admin settings page or the Workspace configuration API. Databricks Genomics runtime is deprecated and support for Databricks Runtime for Genomics was removed on September 24, 2022.
The Databricks Genomics runtime has been deprecated. For open source equivalents, see repos for genomics-pipelines and Glow.
For more information about the Databricks Runtime deprecation policy and schedule, see Supported Databricks Runtime LTS releases.
Container Services setting removed from the workspace admin settings page
August 25, 2023
Workspace admins can no longer enable or disable Container Services from the workspace admin settings page. Container Services enables you to use customer containers on your clusters. It is enabled by default on all new workspaces. Container Services can still be enabled or disabled using the Workspace configuration API. For more information, see Enable Container Services.
Databricks CLI updated to version 0.203.2 (Public Preview)
August 24, 2023
The Databricks command-line interface (Databricks CLI) has been updated to version 0.203.2. For details, see the changelog for version 0.203.2.
Go to definition for functions and variables in Python notebooks
August 24, 2023
In Databricks Runtime 12.2 LTS and above, when a Python notebook is attached to a cluster, you can automatically navigate to the definition of a variable or function that is defined in that notebook. See Go to definition for details.
Databricks Runtime 13.3 LTS is GA
August 22, 2023
Databricks Runtime 13.3 LTS and Databricks Runtime 13.3 LTS ML are now generally available.
See Databricks Runtime 13.3 LTS and Databricks Runtime 13.3 LTS for Machine Learning.
Introducing tags with Unity Catalog
August 22, 2023
You can now apply tags to your securable objects in Unity Catalog. Tags help you organize securable objects and simplify search and discovery of your data assets. See Apply tags to Unity Catalog securable objects.
Volumes support for init scripts and JARs is in Public Preview
August 23, 2023
You can now use Unity Catalog volumes to store init scripts and JARs on compute with assigned or shared access modes running Databricks Runtime 13.3 and above. See Cluster-scoped libraries and Install libraries from a volume.
Easier Databricks Repos .ipynb file output commits
August 21, 2023
Databricks Repos has added features to make .ipynb file output commits easier to control and see:
Repos can create an commit_output config file for you.
Notifications in the UI let you know the status of output commits.
From the File menu, you can control whether noteook outputs are committed.
IPYNB notebook support in Databricks Repos is GA
August 21, 2023
IPYNB notebook file support in Databricks Repos is GA. For more information about Repos, see:
Databricks SDK for Go updated to version 0.17.0 (Beta)
August 18, 2023
Databricks SDK for Go version 0.17.0 adds over 30 APIs and renames about 10 APIs. For details, see the changelog for version 0.17.0.
Databricks SDK for Python updated to version 0.6.0 (Beta)
August 18, 2023
Databricks SDK for Python version 0.6.0 adds about 20 dataclasses, adds about 10 APIs, and includes about 20 renames. For details, see the changelog for version 0.6.0.
Databricks CLI updated to version 0.203.1 (Public Preview)
August 16, 2023
The Databricks command-line interface (Databricks CLI) has been updated to version 0.203.1. For details, see the changelog for version 0.203.1.
Unified schema browser is now GA
August 16, 2023
A unified way to navigate your data is now available for notebooks, SQL Editor, and Catalog Explorer, including the following new capabilities:
Condensed version of the tree.
Display of Unity Catalog object count at the schema level when filtering.
Right-click menu to view actions.
Databricks SDK for Go updated to version 0.16.0 (Beta)
August 14, 2023
Databricks SDK for Go version 0.16.0 adds about a dozen fields, removes about a half-dozen fields, and changes the signatures of about a half-dozen APIs. For details, see the changelog for version 0.16.0.
Partner Connect supports Census
August 14, 2023
You can now use Partner Connect to connect your Databricks workspace to Census. For more information, see Connect to Census.
Databricks SDK for Python updated to version 0.5.0 (Beta)
August 11, 2023
Databricks SDK for Python version 0.5.0 changes the signatures of a half-dozen APIs, adds a dozen fields, removes a half-dozen fields, and adds a few dataclasses. For details, see the changelog for version 0.5.0.
Programmatic write support for workspace files
August 11, 2023
Clusters using Databricks Runtime 11.2 and above support writing to workspace files. You can programmatically create, update, and delete files and directories within your workspace.
Access resources from serving endpoints with instance profiles (Public Preview)
August 10, 2023
Add an instance profile to your model serving endpoint to access any AWS resources from the model permissible by the instance profile. See Add an instance profile to a model serving endpoint.
To enroll your workspace in the Public Preview, reach out to your Databricks account team or submit the instance profile preview enrollment form.
Databricks CLI updated to version 0.203.0 (Public Preview)
August 10, 2023
The Databricks command-line interface (Databricks CLI) version 0.203.0 improves the auth login command, and more. For details, see the changelog for version 0.203.0.
Databricks Terraform provider updated to version 1.23.0
August 9, 2023
Databricks Terraform provider version 1.23.0 extends RunJobTask with additional supported fields in databricks_job, adds exporter support for databricks_model_serving, and more. For details, see the changelog for version 1.23.0.
Groups can now be renamed
August 8, 2023
You can now rename existing account groups and workspace-local groups. See Change the name of a group.
Databricks SDK for Go updated to version 0.15.0 (Beta)
August 7, 2023
Databricks SDK for Go version 0.15.0 refactors permissions-related APIs to make them more understandable and easier to use. For details, see the changelog for version 0.15.0.
Databricks SDK for Python updated to version 0.4.0 (Beta)
August 7, 2023
Databricks SDK for Python version 0.4.0 refactors permissions-related APIs to make them more understandable and easier to use. For details, see the changelog for version 0.4.0.
Third-party iFraming prevention configuration setting was removed
August 7, 2023
Databricks reduces the risk of third-party domains from iFraming Databricks by adding appropriate HTTP headers. Third-party iFraming prevention is still enabled by default but Databricks removed the workspace admin setting to disable it. If you previously disabled third-party iFraming for a workspace, the feature continues to be disabled for that workspace. To re-enable third-party iFraming, contact your Databricks account team.
Databricks extension for Visual Studio Code updated to version 1.1.1
August 4, 2023
The Databricks extension for Visual Studio Code version 1.1.1 provides several minor fixes, for instance correctly handling the HOME environment variable on Windows, and more. For details, see the changelog for version 1.1.1.
LangChain available in 13.1 and above
August 4, 2023
LangChain is available as an experimental MLflow flavor which allows LangChain customers to leverage the robust tools and experiment tracking capabilities of MLflow directly from the Databricks environment.
LangChain is a software framework designed to help create applications that utilize large language models (LLMs) and combine them with external data to bring more training context for LLMs.
Databricks Runtime for Machine Learning includes langchain in Databricks Runtime 13.1 ML and above.
Feature Engineering in Unity Catalog is Public Preview
August 3, 2023
With Feature Engineering in Unity Catalog, Unity Catalog becomes your feature store. You can use any Delta table with a primary key as a feature table for model training or inference. Unity Catalog provides feature discovery and governance.
Improved error handling for repeated continuous job failures
August 3, 2023
Continuous jobs exceeding a threshold for consecutive failures now use an exponential backoff scheme to efficiently retry job runs and allow jobs to recover from transient issues. Previously, continuous jobs exceeding the failure threshold were paused and not retried.
Run tasks conditionally in your Databricks jobs
August 2, 2023
You can use the new Run if condition to run a job task based on the status of the task’s dependencies, providing more flexibility when you implement a workflow with a Databricks job. For example, you can use the Run if condition to run a task when some or all of the task’s dependencies have failed. See Configure task dependencies.
Compliance security profile works with serverless SQL warehouses in some regions (Public Preview)
August 1, 2023
If your workspace has the compliance security profile enabled, you can now use serverless SQL warehouses to process regulated data under HIPAA, PCI-DSS, and FedRAMP Moderate. This is available as a Public Preview that applies only in some regions, starting with the us-east-1 region. See Which compute resources get enhanced security.
As part of this preview, the serverless SQL support for the compliance security profile will be rolled out incrementally to all customers in the us-east-1 region over the next few weeks. To prioritize the enrollment of your account in the preview, contact your Databricks account team.
Databricks Terraform provider updated to version 1.22.0
August 1, 2023
Databricks Terraform provider version 1.22.0 adds a databricks_access_control_rule_set resource, adds databricks_metastore and databricks_metastore data sources, and more. For details, see the changelog for version 1.22.0.
Lakehouse Monitoring is Public Preview
August 1, 2023
Databricks Lakehouse Monitoring integrates data and model quality monitoring into the Databricks platform. It provides out of the box quality metrics such as summary statistics, changes in distribution over time, and model performance including fairness and bias. Lakehouse Monitoring automatically generates a summary dashboard to help you visualize data quality over time. You can also set up alerts based on the data.
Databricks Runtime 13.3 LTS (Beta)
August 1, 2023
Databricks Runtime 13.3 LTS and Databricks Runtime 13.3 LTS ML are now available as Beta releases.
See Databricks Runtime 13.3 LTS and Databricks Runtime 13.3 LTS for Machine Learning.
New Git operations are generally available: Merge branches, rebase and pull with conflict resolution
August 1, 2023
The following Git operations are now generally available in Databricks Repos:
Merge and rebase: You can now Git merge or rebase two branches directly in Databricks. This simplifies the process of combining branches and provides the ability to resolve conflicts that arise during the operation.
Pull with conflict resolution: You can now use Databricks Repos to manage and resolve conflicts that happen when you pull from a remote Git repo using Databricks.
With these powerful new conflict resolution tools, collaborating on code using Databricks Repos becomes even more streamlined and efficient.