June 2024

These features and Databricks platform improvements were released in June 2024.

Note

Releases are staged. Your Databricks account might not be updated until a week or more after the initial release date.

New improved catalog creation UI

June 18, 2024

The new catalog creation UI takes you stepwise through the entire process of creating and configuring a catalog, providing helpful prompts and separating required from optional configurations. You can use the same user interface to create standard, shared, and foreign catalogs.

Catalog creation stepper UI

See Create a catalog.

The billing system tables are enabled by default

June 18, 2024

The system.billing schema is now enabled by default in all Unity Catalog workspaces. See Billable usage system table reference.

Lakehouse Monitoring is GA

June 13, 2024

Databricks Lakehouse Monitoring is now generally available.

Volume sharing using Delta Sharing is now generally available

June 13, 2024

Delta Sharing support for sharing non-tabular data as a volume is now GA. You can share volumes between Databricks workspaces on different Unity Catalog metastores (including workspaces on different Databricks accounts and different clouds).

Volumes are Unity Catalog objects that represent a logical volume of storage in a cloud object storage location. They are intended primarily to provide governance over non-tabular data assets. Delta Sharing on Databricks provides a native integration with Unity Catalog that allows you to manage, govern, audit, and track usage of shared volumes data.

See Add volumes to a share.

Create budgets to monitor account spending (Public Preview)

June 13, 2024

Account admins can now create budgets to track spending in their Databricks account. Budgets can include customized filters to track spending based on workspace and custom tags. See Use budgets to monitor account spending.

Importable cost management dashboard added to account console (Public Preview)

June 13, 2024

Account admins can now import a customizable cost management dashboard from the account console into any Unity Catalog workspace in their account. There are two versions of the dashboard, one that monitors account-wide usage, and another for single-workspace usage.

See Import a usage dashboard.

Unified login is now generally available

June 12, 2024

Unified login allows you to manage one SSO configuration for your Databricks account and workspaces. When SSO is enabled on your account, you can choose to enable unified login for all workspaces or selected workspaces. Unified login workspaces use the account-level SSO configuration, and all users, including account and workspace admins, must sign in to Databricks using SSO. see Enable unified login. Unified login has been generally available for accounts created after June 21, 2023 since June 21, 2023.

Customer managed keys (CMK) for Databricks Vector Search are now GA

June 12, 2024

Customer Managed Keys (CMK) on vector search endpoints are now generally available.

Mosaic AI Agent Framework (Public Preview)

June 12, 2024

Mosaic AI Agent Framework is a set of tools on Databricks designed to help developers build, deploy, and evaluate production-quality Retrieval Augmented Generation (RAG) applications. Agent Framework enables you to create or optimize LLM applications on a single platform using your own data.

Agent Framework includes the following capabilities:

  • Create and log agents and chains.

  • Parameterize agents and chains to experiment and iterate quickly.

  • Built-in metrics to compare runs across retrieval of relevant documents, response accuracy, and cost and latency.

  • Custom LLM judges for retrieval and response accuracy.

  • Deploy AI applications with automated request and response logging.

  • Built-in review app to get user feedback.

  • Agent Evaluation to evaluate the quality, latency, and cost of generative AI applications.

  • MLflow Tracing to analyze and compare traces across code execution.

For details, see What is Mosaic AI Agent Framework?.

Predictive optimization is now GA

June 11, 2024

Predictive optimization for Unity Catalog managed tables is now generally available. Predictive optimization now runs OPTIMIZE on tables with liquid clustering enabled to apply clustering. See Predictive optimization for Delta Lake.

Route optimization is available for serving endpoints

June 11, 2024

You can now create route optimized serving endpoints for your model serving or feature serving workflows. See Configure route optimization on serving endpoints.

Column mapping is now GA

June 10, 2024

Delta Lake column mapping is now generally available. You can now use DROP FEATURE to disable column mapping and downgrade table protocol. See Rename and drop columns with Delta Lake column mapping.

Workflow system tables are now available (Public Preview)

June 10, 2024

The system.workflow schema contains four new tables you can use to monitor jobs in your account. See Jobs system table reference.

If you don’t have access to these system tables, ensure you have enabled the system.workflow schema in your account. See Enable system table schemas.

Function calling is Public Preview

June 10, 2024

Function calling is now available in Public Preview. This functionality is available using Foundation Model APIs pay-per-token models: DBRX Instruct and Meta-Llama-3-70B-Instruct. See Function calling on Databricks.

GTE is now supported in Model Serving

June 7, 2024

Databricks Model Serving now supports General Text Embedding (GTE), a text embedding model that can map text to an embedding vector. GTE is supported as part of Foundation Model APIs and is available in pay-per-token serving endpoint regions. See Use Foundation Model APIs.

Route optimized serving endpoints enforce network restrictions

June 6, 2024

Route optimized serving endpoint now enforces network restrictions such as IP access control lists or PrivateLink. See Configure route optimization on serving endpoints.

Databricks Assistant: Threads & queries experience enabled by default

June 6, 2024

Databricks Assistant (Public Preview) now tracks query threads and history throughout editor contexts in your session experience by default. You can manage enablement for all workspaces in an account or individual workspaces.

For more information, see What is Databricks Assistant?

New entry points for in-product help

June 5, 2024

You can now access the Contact Support option by clicking your user icon in the top bar of the workspace. See Get help.

Improved notebook Markdown editor

June 5, 2024

You can now see a live preview of your Markdown cells in notebooks and access a toolbar for common Markdown elements like headers, lists, and links. Spend more time communicating your ideas and less time remembering syntax.

Quickly access Catalog Explorer tables from notebooks

June 5, 2024

You can now easily access Catalog Explorer tables directly from notebooks. Hover over a table name in a notebook cell to get a link to its catalog explorer page to see details like columns, permissions, and more.

Databricks ODBC driver 2.8.2

June 4, 2024

Databricks ODBC Driver (download) version 2.8.2 is now available. This release adds the following new features and enhancements:

  • Customers can now use extended SQL PUT, GET, and REMOVE commands to ingest and manage large datasets in Unity Catalog volumes.

  • Default OAuth configuration values are now updated and compliant. The updated configuration options include the authorization endpoint, OAuth client ID, OAuth user-to-machine scope, and OAuth machine-to-machine scope. For more details, see the release notes.

  • Third-party libraries have been updated, including expat 2.6.2 (previously 2.6.0) and libCURL 8.7.1 (previously 8.6.0).

  • The driver now accepts a private signed certification authority (CA) if it is added to the Windows Trusted Root CA certificate store.

This release also fixes the following issues:

  • The client previously returned an error if a table comment’s length exceeded MaxCommentLen. The default behavior now truncates the table comment to conform to the limit specified in MaxCommentLen.

  • The client previously returned an error when issuing SQLCancel under certain circumstances.

The new Databricks Notebook UI is now generally available

June 4, 2024

The new Databricks Notebooks UI provides developers with a streamlined, data-focused authoring experience. Key features include:

  • Modern UX: Enjoy a streamlined coding experience with an updated UI and other features to enhance Notebook organization.

  • New results table: Perform no-code data exploration with search and filtering capabilities directly on result outputs.