December 2022

These features and Databricks platform improvements were released in December 2022.

Note

Releases are staged. Your Databricks account might not be updated until a week or more after the initial release date.

Databricks SQL Driver for Go is Generally Available

December 21, 2022

Databricks now provides full support for the Databricks SQL Driver for Go. The new version number is 1.0.0.

Changes include:

  • Direct results are now enabled. If maxRows is omitted when initializing the connection, it defaults to 100,000.

  • Context cancellation and timeout is now supported. If the underlying context of a query is cancelled or timed-out, the query will be cancelled on the server.

  • Client-side logging functionality allows users to set the verbosity of the logger and set various identifiers to track the lifetime of queries, connections, and requests.

  • dbsql.NewConnector() added to help initialize database connection.

  • Setting the initial namespace and session parameters including timezone is now supported.

  • Comprehensive usage examples, bug fixes, and performance improvements added.

See the full changelog.

Prevent concurrent workspace updates

December 15, 2022

A new feature prevents data corruption resulting from concurrent updates to a workspace.

When an admin attempts to create, update, or delete a workspace through the account console or REST API, they receive an error message if there is already an ongoing operation on the same workspace. This prevents conflicting workspace updates and data corruption. The admin can wait and retry the operation after the other update finishes.

Databricks Terraform provider updated to version 1.7.0

December 14, 2022

Version 1.7.0 adds the new data source databricks_cluster_policy, enables exporting of the databricks_service_principal and databricks_sql_global_config resources, and more. For details, see the changelogs for version 1.7.0.

Databricks Runtime 12.0 and 12.0 ML are GA

December 14, 2022

Databricks Runtime 12.0 and 12.0 ML are now generally available.

See Databricks Runtime 12.0 (EoS) and Databricks Runtime 12.0 for Machine Learning (EoS).

Billable usage graphs can now aggregate by individual tags

December 6, 2022

The Databricks billable usage graphs in the account console can now aggregate usage by individual tags. The billable usage CSV reports downloaded from the same page also include default and custom tags.

Use SQL to specify schema- and catalog-level storage locations for Unity Catalog managed tables

December 6, 2022

You can now use the MANAGED LOCATION SQL command to specify a cloud storage location for managed tables at the catalog and schema levels. Requires Databricks Runtime 11.3 and above. See CREATE CATALOG and CREATE SCHEMA.

Capturing lineage data with Unity Catalog is now generally available

December 6, 2022

Databricks is pleased to announce the general availability of Unity Catalog support for capturing and viewing lineage data. See Capture and view data lineage using Unity Catalog.

Databricks ODBC driver 2.6.29

December 5, 2022

Version 2.6.29 of the Databricks ODBC driver (download) is now available. This release adds support for Unity catalog’s primary and foreign keys, thereby enabling BI tools that use the ODBC driver to discover relationships between tables during data modeling.

This release also resolves the following issue:

  • The driver previously included cloud fetch’s presigned URLs in its log file when EnableCurlDebugLogging is set. These presigned URLs included sensitive tokens. Driver log entries now exclude these tokens.

For full details, see release-notes.txt included with the driver.

Databricks JDBC driver 2.6.32

December 5, 2022

Version 2.6.32 of the Databricks JDBC driver (download and Maven) is now available. This release updates the Jackson JSON parser libraries for enhanced security:

  • jackson-annotations-2.13.4 (previously 2.13.2)

    • jackson-core-2.13.4 (previously 2.13.2)

    • jackson-databind-2.13.4.2 (previously 2.13.2.2)

This release also resolves the following issues:

  • When using cloud fetch, the driver now cleans up certain resources properly.

    • The driver previously included cloud fetch’s presigned URLs in its log file. These presigned URLs included sensitive tokens. These log entries now exclude these tokens.

    • The driver logs a SQL statement whenever its execution hits an exception. Statements such as COPY INTO might contain secrets (for example WITH CREDENTIAL). Driver log entries now exclude such credentials.

For full details, see release-notes.txt included with the driver.

Partner Connect supports connecting to AtScale

December 1-6, 2022

You can now easily create a connection between AtScale and your Databricks workspace using Partner Connect. For more information, see Connect to AtScale using Partner Connect.

Improved serverless SQL warehouse support for customer-managed keys

December 1, 2022

Serverless SQL warehouses now support adding customer-managed keys to your workspace for managed services and your workspace’s root S3 bucket.

Serverless SQL warehouses do not use customer-managed keys for EBS storage for compute nodes, which is an optional part of the customer-managed keys for workspace storage feature. Disks for serverless compute resources are short-lived and tied to the lifecycle of the serverless workload. For example, when serverless SQL warehouses are stopped or scaled down, the VMs and their storage are destroyed. See Serverless compute and customer-managed keys.