Skip to main content

August 2023

These features and Databricks platform improvements were released in August 2023.

note

Releases are staged. Your Databricks workspace might not be updated until a week or more after the initial release date.

August 31, 2023

Tables registered in Unity Catalog now appear in navigational search in the top bar. See Search for workspace objects.

Databricks CLI updated to version 0.203.3 (Public Preview)​

August 30, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.203.3. For details, see the changelog for version 0.203.3.

Databricks JDBC driver 2.6.34​

August 30, 2023

We have released version 2.6.34 of the Databricks JDBC driver (download). This release addresses a number of vulnerability issues by upgrading third-party libraries. In addition, this release fixes the following issues:

  • Thread leakage when a query fails to connect to the server.
  • Missing comments in the REMARKS column of the table metadata.
  • When a resultset closure operation returns an error, the connector does not clean up the operation handle entries from the heartbeat thread.

The release also improves error message handling by honoring SQLState and error codes from the server, and checks the server protocol version correctly.

Databricks SDK for Go updated to version 0.18.0 (Beta)​

August 29, 2023

Databricks SDK for Go version 0.18.0 introduces 17 breaking API changes, adds 30 APIs, and adds about a dozen fields. For details, see the changelog for version 0.18.0.

Databricks SDK for Python updated to version 0.7.0 (Beta)​

August 29, 2023

Databricks SDK for Python version 0.7.0 introduces a breaking change for the files API as well as 17 additional breaking changes. This release also adds 14 fields and adds 27 dataclasses. For details, see the changelog for version 0.7.0.

Databricks Terraform provider updated to version 1.24.1​

August 29, 2023

Databricks Terraform provider version 1.24.1 includes a fix for verification of workspace reachability. For details, see the changelog for version 1.24.1.

Init scripts on DBFS end of life extended to Dec 1, 2023​

August 29, 2023

Databricks previously announced that init scripts on DBFS would reach end of life on Sept 1, 2023. The end of life date has been extended to Dec 1, 2023.

Databricks Runtime 14.0 (Beta)​

August 29, 2023

Databricks Runtime 14.0 and Databricks Runtime 14.0 ML are now available as Beta releases.

See Databricks Runtime 14.0 (EoS) and Databricks Runtime 14.0 for Machine Learning (EoS).

Custom workspace tags​

August 28, 2023

You can now specify custom workspace tags during workspace creation using the Account API. See Create a workspace using the Account API.

Unified navigation experience is GA​

August 28, 2023

The unified Databricks navigation experience is now generally available. Users can no longer revert to the classic navigation experience. For more information, see Workspace UI.

Allow additional ports from your classic compute plane to control plane by January 31, 2024​

August 28, 2023

If you configured a security group or firewall on the VPC in your classic compute plane (where your compute resources for clusters run) that restricts the open ports for communication to the Databricks control plane, you must configure your classic compute plane network to allow egress on additional ports to the control plane. In the future, these endpoints will be required. For new and existing workspaces, you must open up the ports 8443 through 8451 by January 31, 2024.

The fully qualified domain name (FQDN) of the Databricks workspace remains the same. Databricks will manage the required network configuration changes for serverless compute resources. Contact help@databricks.com with any questions about this change.

Databricks Terraform provider updated to version 1.24.0​

August 25, 2023

Databricks Terraform provider version 1.24.0 adds account-level API support for Unity Catalog objects, adds acl_principal_id to several data sources, and more. For details, see the changelog for version 1.24.0.

Standard access mode (formerly shared access mode) clusters can now connect to public networks outside of Databricks VPC​

August 25, 2023

Standard access mode clusters using Databricks Runtime 12.0 and above can now connect to public networks outside of the Databricks VPC. Users are no longer limited to creating network connections in ports 80 and 443. See Compute access mode limitations for Unity Catalog.

Databricks Runtime for Genomics setting removed from the workspace admin settings page​

August 25, 2023

Workspace admins can no longer enable or disable Databricks Runtime for Genomics from the workspace admin settings page or the Workspace configuration API. Databricks Genomics runtime is deprecated and support for Databricks Runtime for Genomics was removed on September 24, 2022.

The Databricks Genomics runtime has been deprecated. For open source equivalents, see repos for genomics-pipelines and Glow.

For more information about the Databricks Runtime deprecation policy and schedule, see Supported Databricks Runtime LTS releases.

Container Services setting removed from the workspace admin settings page​

August 25, 2023

Workspace admins can no longer enable or disable Container Services from the workspace admin settings page. Container Services enables you to use customer containers on your clusters. It is enabled by default on all new workspaces. Container Services can still be enabled or disabled using the Workspace configuration API. For more information, see Enable Container Services.

Databricks CLI updated to version 0.203.2 (Public Preview)​

August 24, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.203.2. For details, see the changelog for version 0.203.2.

Go to definition for functions and variables in Python notebooks​

August 24, 2023

In Databricks Runtime 12.2 LTS and above, when a Python notebook is attached to a cluster, you can automatically navigate to the definition of a variable or function that is defined in that notebook. See Go to definition for details.

Databricks Runtime 13.3 LTS is GA​

August 22, 2023

Databricks Runtime 13.3 LTS and Databricks Runtime 13.3 LTS ML are now generally available.

See Databricks Runtime 13.3 LTS and Databricks Runtime 13.3 LTS for Machine Learning.

Introducing tags with Unity Catalog​

August 22, 2023

You can now apply tags to your securable objects in Unity Catalog. Tags help you organize securable objects and simplify search and discovery of your data assets. See Apply tags to Unity Catalog securable objects.

Allowlist for init scripts, JARs, and Maven coordinates on Unity Catalog shared clusters is in Public Preview​

August 23, 2023

You can now use the Unity Catalog allowlist to control installation of init scripts, JARs, and Maven coordinates on compute with shared access mode running Databricks Runtime 13.3 and above. See Allowlist libraries and init scripts on compute with standard access mode (formerly shared access mode).

Volumes support for init scripts and JARs is in Public Preview​

August 23, 2023

You can now use Unity Catalog volumes to store init scripts and JARs on compute with assigned or shared access modes running Databricks Runtime 13.3 and above. See Compute-scoped libraries and Install libraries from a volume.

Easier Databricks Repos .ipynb file output commits​

August 21, 2023

Databricks Repos has added features to make .ipynb file output commits easier to control and see:

  • Repos can create an commit_output config file for you.
  • Notifications in the UI let you know the status of output commits.
  • From the File menu, you can control whether noteook outputs are committed.

See Control .ipynb notebook output commits.

IPYNB notebook support in Databricks Repos is GA​

August 21, 2023

IPYNB notebook file support in Databricks Repos is GA. For more information about Repos, see:

Databricks SDK for Go updated to version 0.17.0 (Beta)​

August 18, 2023

Databricks SDK for Go version 0.17.0 adds over 30 APIs and renames about 10 APIs. For details, see the changelog for version 0.17.0.

Databricks SDK for Python updated to version 0.6.0 (Beta)​

August 18, 2023

Databricks SDK for Python version 0.6.0 adds about 20 dataclasses, adds about 10 APIs, and includes about 20 renames. For details, see the changelog for version 0.6.0.

Databricks CLI updated to version 0.203.1 (Public Preview)​

August 16, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.203.1. For details, see the changelog for version 0.203.1.

Unified schema browser is now GA​

August 16, 2023

A unified way to navigate your data is now available for notebooks, SQL Editor, and Catalog Explorer, including the following new capabilities:

  • Condensed version of the tree.
  • Display of Unity Catalog object count at the schema level when filtering.
  • Right-click menu to view actions.

Databricks SDK for Go updated to version 0.16.0 (Beta)​

August 14, 2023

Databricks SDK for Go version 0.16.0 adds about a dozen fields, removes about a half-dozen fields, and changes the signatures of about a half-dozen APIs. For details, see the changelog for version 0.16.0.

Partner Connect supports Census​

August 14, 2023

You can now use Partner Connect to connect your Databricks workspace to Census. For more information, see Connect to Census.

Databricks SDK for Python updated to version 0.5.0 (Beta)​

August 11, 2023

Databricks SDK for Python version 0.5.0 changes the signatures of a half-dozen APIs, adds a dozen fields, removes a half-dozen fields, and adds a few dataclasses. For details, see the changelog for version 0.5.0.

Programmatic write support for workspace files​

August 11, 2023

Clusters using Databricks Runtime 11.2 and above support writing to workspace files. You can programmatically create, update, and delete files and directories within your workspace.

Access resources from serving endpoints with instance profiles (Public Preview)​

August 10, 2023

Add an instance profile to your model serving endpoint to access any AWS resources from the model permissible by the instance profile. See Add an instance profile to a model serving endpoint.

To enroll your workspace in the Public Preview, reach out to your Databricks account team or submit the instance profile preview enrollment form.

Databricks CLI updated to version 0.203.0 (Public Preview)​

August 10, 2023

The Databricks command-line interface (Databricks CLI) version 0.203.0 improves the auth login command, and more. For details, see the changelog for version 0.203.0.

Databricks Terraform provider updated to version 1.23.0​

August 9, 2023

Databricks Terraform provider version 1.23.0 extends RunJobTask with additional supported fields in databricks_job, adds exporter support for databricks_model_serving, and more. For details, see the changelog for version 1.23.0.

Groups can now be renamed​

August 8, 2023

You can now rename existing account groups and workspace-local groups. See Change the name of a group.

Databricks SDK for Go updated to version 0.15.0 (Beta)​

August 7, 2023

Databricks SDK for Go version 0.15.0 refactors permissions-related APIs to make them more understandable and easier to use. For details, see the changelog for version 0.15.0.

Databricks SDK for Python updated to version 0.4.0 (Beta)​

August 7, 2023

Databricks SDK for Python version 0.4.0 refactors permissions-related APIs to make them more understandable and easier to use. For details, see the changelog for version 0.4.0.

Third-party iFraming prevention configuration setting was removed​

August 7, 2023

Databricks reduces the risk of third-party domains from iFraming Databricks by adding appropriate HTTP headers. Third-party iFraming prevention is still enabled by default but Databricks removed the workspace admin setting to disable it. If you previously disabled third-party iFraming for a workspace, the feature continues to be disabled for that workspace. To re-enable third-party iFraming, contact your Databricks account team.

Databricks extension for Visual Studio Code updated to version 1.1.1​

August 4, 2023

The Databricks extension for Visual Studio Code version 1.1.1 provides several minor fixes, for instance correctly handling the HOME environment variable on Windows, and more. For details, see the changelog for version 1.1.1.

LangChain available in 13.1 and above​

August 4, 2023

LangChain is available as an experimental MLflow flavor which allows LangChain customers to leverage the robust tools and experiment tracking capabilities of MLflow directly from the Databricks environment.

LangChain is a software framework designed to help create applications that utilize large language models (LLMs) and combine them with external data to bring more training context for LLMs.

Databricks Runtime for Machine Learning includes langchain in Databricks Runtime 13.1 ML and above.

Feature Engineering in Unity Catalog is Public Preview​

August 3, 2023

With Feature Engineering in Unity Catalog, Unity Catalog becomes your feature store. You can use any Delta table with a primary key as a feature table for model training or inference. Unity Catalog provides feature discovery and governance.

Improved error handling for repeated continuous job failures​

August 3, 2023

Continuous jobs exceeding a threshold for consecutive failures now use an exponential backoff scheme to efficiently retry job runs and allow jobs to recover from transient issues. Previously, continuous jobs exceeding the failure threshold were paused and not retried.

Share schemas using Delta Sharing (Public Preview)​

August 2, 2023

You can now use Delta Sharing to share an entire schema, giving the recipient access to all of the tables and views in the schema at the moment you share it, along with any tables and views that are added to the schema in the future. Adding schemas to a share using SQL commands requires a SQL warehouse or a cluster running Databricks Runtime 13.2 or above. Doing the same using Catalog Explorer has no compute requirements. See Add schemas to a share.

Run tasks conditionally in your Databricks jobs​

August 2, 2023

You can use the new Run if condition to run a job task based on the status of the task's dependencies, providing more flexibility when you implement a workflow with a Databricks job. For example, you can use the Run if condition to run a task when some or all of the task's dependencies have failed. See Configure task dependencies.

Compliance security profile works with serverless SQL warehouses in some regions (Public Preview)​

August 1, 2023

If your workspace has the compliance security profile enabled, you can now use serverless SQL warehouses to process regulated data under HIPAA, PCI-DSS, and FedRAMP Moderate. This is available as a Public Preview that applies only in some regions, starting with the us-east-1 region. See Classic and serverless compute support by region.

As part of this preview, the serverless SQL support for the compliance security profile will be rolled out incrementally to all customers in the us-east-1 region over the next few weeks. To prioritize the enrollment of your account in the preview, contact your Databricks account team.

Databricks Terraform provider updated to version 1.22.0​

August 1, 2023

Databricks Terraform provider version 1.22.0 adds a databricks_access_control_rule_set resource, adds databricks_metastore and databricks_metastore data sources, and more. For details, see the changelog for version 1.22.0.

Lakehouse Monitoring is Public Preview​

August 1, 2023

Databricks Lakehouse Monitoring integrates data and model quality monitoring into the Databricks platform. It provides out of the box quality metrics such as summary statistics, changes in distribution over time, and model performance including fairness and bias. Lakehouse Monitoring automatically generates a summary dashboard to help you visualize data quality over time. You can also set up alerts based on the data.

Databricks Runtime 13.3 LTS (Beta)​

August 1, 2023

Databricks Runtime 13.3 LTS and Databricks Runtime 13.3 LTS ML are now available as Beta releases.

See Databricks Runtime 13.3 LTS and Databricks Runtime 13.3 LTS for Machine Learning.

New Git operations are generally available: Merge branches, rebase and pull with conflict resolution​

August 1, 2023

The following Git operations are now generally available in Databricks Repos:

  • Merge and rebase: You can now Git merge or rebase two branches directly in Databricks. This simplifies the process of combining branches and provides the ability to resolve conflicts that arise during the operation.
  • Pull with conflict resolution: You can now use Databricks Repos to manage and resolve conflicts that happen when you pull from a remote Git repo using Databricks.

With these powerful new conflict resolution tools, collaborating on code using Databricks Repos becomes even more streamlined and efficient.

Last updated on