May 2024

These features and Databricks platform improvements were released in May 2024.

Note

Releases are staged. Your Databricks account might not be updated until a week or more after the initial release date.

New changes to Git folder UI

May 17, 2024

You may notice some changes to the user interface for Git folder interactions. We’ve added the following:

• When you share a Git folder, you will see a new alert in a banner that prompts you to Copy link to create Git folder. When you click the button, a URL is copied to your local clipboard, which you can send to another user. When that recipient user loads that URL in a browser, the user is taken to the workspace where they can create their own Git folder cloned from the same remote Git repository. When the recipient accesses the URL, they will see a Create Git folder dialog in the UI that is pre-populated with the values taken from your Git folder.

  

• Similarly, a new button, Create Git folder, appears on a new alert banner when you view a Git folder created by another user. Click this button to create your own Git folder for the same Git repository, based on the pre-populated values in the Create Git folder dialog.

  

Unified Login now supported with AWS PrivateLink (Private Preview)

May 15, 2024

Unified login allows you to manage one single sign-on (SSO) configuration in your account that is used for the account and Databricks workspace. Unified login is now supported with private connectivity using AWS PrivateLink between users and their Databricks workspaces. See Enable unified login and Enable AWS PrivateLink. You must contact your Databricks account team to request access to this preview.

Foundation Model Training (Public Preview)

May 13, 2024

Databricks now supports Foundation Model Training. With Foundation Model Training, you use your own data to customize a foundation model to optimize its performance for your specific application. By fine-tuning or continuing training of a foundation model, you can train your own model using significantly less data, time, and compute resources than training a model from scratch. The training data, checkpoints, and fine-tuned model all reside on the Databricks platform and are integrated with its governance and productivity tools.

For details, see Foundation Model Training.

Attribute tag values for Unity Catalog objects can now be 1000 characters long (Public Preview)

May 8, 2024

Attribute tag values in Unity Catalog can now be up to 1000 characters long. The character limit for tag keys remains 255. See Apply tags to Unity Catalog securable objects.

New Previews page

May 8, 2024

Enable and manage access to Databricks previews on the new Previews page. See Manage Databricks Previews.

New capabilities for Databricks Vector Search

May 8, 2024

New capabilities include the following:

• PrivateLink and IP access lists are now supported.

• Customer Managed Keys (CMK) are now supported on endpoints created on or after May 8, 2024. Vector Search support for CMK is in Public Preview.

• Improved audit logs and cost attribution tracking. See Audit log reference.

• You can now save generated embeddings as a Delta table. See Create a vector search index.

See Databricks Vector Search.

Credential passthrough and Hive metastore table access controls are deprecated

May 7, 2024

Credential passthrough and Hive metastore table access controls are deprecated on Databricks Runtime 15.0 and support will be removed in an upcoming DBR version.

Upgrade to Unity Catalog to simplify the security and governance of your data by providing a central place to administer and audit data access across multiple workspaces in your account. See What is Unity Catalog?.

Databricks JDBC driver 2.6.38

May 6, 2024

We have released version 2.6.38 of the Databricks JDBC driver (download). This release adds the following new features and enhancements:

• Native Parameterized Query support if the server uses SPARK_CLI_SERVICE_PROTOCOL_V8. The limit of the number of parameters in a query is 256 in the native query mode.

• Data ingestion using a Unity Catalog volume support. See more about Unity Catalog volumes in Connect to cloud object storage using Unity Catalog. To use this, set UseNativeQuery to 1.

• QueryProfile interface added to IHadoopStatement allows applications to retrieve a query’s query id. The query id can be be used to fetch the query’s metadata using Databricks REST APIs.

• Async operations for metadata Thrift calls if the server uses SPARK_CLI_SERVICE_PROTOCOL_V9. To use this feature, set EnableAsyncModeForMetadataOperation property to 1.

• JWT assertion support. The connector now supports JWT assertion OAuth using client credentials. To do this, set the UseJWTAssertion property to 1.

This release also resolves the following issues:

• Jackson libraries updates. The connector now uses the following libraries for the Jackson JSON parser: jackson-annotations 2.16.0 (previously 2.15.2), jackson-core 2.16.0 (previously 2.15.2), jackson-databind-2.16.0 (previously 2.15.2)

• The connector contains unshaded class files in META-INF directory.

Databricks Runtime 15.2 (Beta)

May 2, 2024

Databricks Runtime 15.2 and Databricks Runtime 15.2 ML are now available as Beta releases.

See Databricks Runtime 15.2 (Beta) and Databricks Runtime 15.2 for Machine Learning (Beta).

Notebooks now detect and auto-complete column names for Spark Connect DataFrames

May 1, 2024

Databricks notebooks now automatically detect and display the column names in Spark Connect DataFrames and allow you to use auto-complete to select columns.