December 2024

These features and Databricks platform improvements were released in December 2024.

Note

Releases are staged. Your Databricks account might not be updated until a week or more after the initial release date.

Databricks Runtime 16.1 is GA

December 20, 2024

Databricks Runtime 16.1 and Databricks Runtime 16.1 ML are now generally available.

See Databricks Runtime 16.1 and Databricks Runtime 16.1 for Machine Learning.

The default format for new notebooks is now IPYNB (Jupyter) format

December 20, 2024

The default format for new notebooks you create in your Databricks workspace is now IPYNB (.ipynb). Previously, the default format for notebooks was Source (.py, .sql, .scala, .r). To change the default format, use the Default file format for notebooks setting in the Developer pane of your workspace user settings. See Notebook formats.

Python code executor for AI agents (Public Preview)

December 19, 2024

You can now quickly give your AI agents the ability to run Python code. Databricks now offers a pre-built Unity Catalog function that can be used by an AI agent as a tool to expand their capabilities beyond language generation.

See Code interpreter AI agent tools.

databricks-agents SDK 0.13.0 release

December 18, 2024

Version 0.13.0 of the databricks-agents SDK has been released to PyPI, containing the following changes:

  • Honor the current active Databricks CLI profile and MLflow model registry URI when calling agents.deploy() and other databricks.agents APIs. In particular, you can now specify a combination of DATABRICKS_CONFIG_PROFILE=my-profile and MLFLOW_REGISTRY_URI=databricks-uc://my-profile before calling agents.deploy() to specify the Databricks CLI profile to use to deploy and access agents.

  • In mlflow.evaluate(), only run retrieval and guidelines metrics if retrieval and guideline context are present, respectively.

  • Add secret-based authentication to clients for mlflow.evaluate().

External groups are now labeled and immutable

December 18, 2024

External groups are groups that are created in Databricks from your identity provider. These groups are created using a SCIM provisioning connector and stay in sync with your identity provider. External groups are now explicitly labeled as External and can no longer be updated from the Databricks account console or workspace admin settings page by default. To update external group membership from the Databricks UI, an account admin can disable Immutable external groups in the account console preview page.

See Types of groups in Databricks.

Test clean rooms with collaborators within the same metastore

December 17, 2024

You can now test your clean room before full deployment by adding a collaborator from within the same metastore. See Create clean rooms.

Assign compute resources to groups (Public Preview)

December 17, 2024

The new Dedicated access mode (previously Single user) allows you to assign a dedicated all-purpose compute to a group or single user. See Assign compute resources to a group.

This Public Preview also gives your workspace access to the new simplified compute UI. See Use the simple form to manage compute.

A workspace admin must enable this preview. See Manage Databricks Previews.

Version 2.2 of the Jobs API is released

December 16, 2024

The Jobs API version is updated from 2.1 to 2.2. Updates in version 2.2 of the Jobs API include default queueing of new or updated jobs, and enhancing paging of job and job run responses that include fields with large numbers of values. To learn more about the updates in this version, see Updating from Jobs API 2.1 to 2.2. To see the full Jobs API 2.2 documentation, see Jobs (latest). Although Databricks recommends using version 2.2 of the Jobs API, you can still access versions 2.1 and 2.0. See Jobs (2.1) and Jobs API 2.0.

Unity Catalog MANAGE privilege (Public Preview)

December 14, 2024

You can now grant users the MANAGE privilege on Unity Catalog securable objects. The MANAGE privilege allows users to perform key actions on a Unity Catalog object, including:

  • Managing privileges

  • Dropping the object

  • Renaming the object

  • Transferring ownership

See MANAGE.

Meta Llama 3.3 is now available for provisioned throughput workloads

December 13, 2024

Meta Llama 3.3, a model architecture built and trained by Meta, is now available for serving on Foundation Model APIs provisioned throughput.

View streaming workload metrics for your job runs (Public Preview)

December 12, 2024

When you view job runs in the Databricks Jobs UI, you can now view metrics such as backlog seconds, backlog bytes, backlog records, and backlog files for sources supported by Spark Structured Streaming, including Apache Kafka, Amazon Kinesis, and Auto Loader. See View metrics for streaming tasks.

View streaming workload metrics for your Delta Live Tables pipeline updates (Public Preview)

December 12, 2024

When you view pipeline updates in the Delta Live Tables UI, you can now view metrics such as backlog seconds, backlog bytes, backlog records, and backlog files for each streaming flow in the pipeline. Streaming metrics are supported for Spark Structured Streaming sources, including Apache Kafka, Amazon Kinesis, and Auto Loader. See View streaming metrics.

Lakehouse Federation supports Oracle (Public Preview)

December 12, 2024

You can now run federated queries on data managed by Oracle. See Run federated queries on Oracle.

Unified login is now enabled on accounts without single sign-on

December 12, 2024

Unified login allows you to manage one single sign-on (SSO) configuration in your account to use for the account and Databricks workspaces. If you have not already configured account-level or workspace-level SSO in Databricks, unified login is now enabled for all of your workspaces and cannot be disabled. Unified login is already enabled by default on accounts created after June 21, 2023. See Configure SSO in Databricks.

Databricks Runtime 16.1 (Beta)

December 11, 2024

Databricks Runtime 16.1 and Databricks Runtime 16.1 ML are now available as Beta releases.

See Databricks Runtime 16.1 and Databricks Runtime 16.1 for Machine Learning

Manage serverless outbound network connections with serverless egress control

December 11th, 2024

Serverless egress control lets you restrict outbound access to specified internet destinations. See What is serverless egress control?.

Network access events system table is now available (Public Preview)

December 11, 2024

Databricks system tables now include a network access table. This table logs an event whenever internet access is denied from your account. To access the table, admins must have the access system schema enabled. See Network access events system table reference.

Monitor and revoke personal access tokens in your account (Public Preview)

December 11, 2024

Account admins can now view a token report to monitor and revoke personal access tokens (PATs) in the account console. Databricks recommends you use OAuth access tokens instead of PATs for greater security and convenience. To join this preview, contact your Databricks account team. See Monitor and revoke personal access tokens in the account.

Unity Catalog can federate to Hive metastores and AWS Glue

December 11, 2024

You can now use Unity Catalog to access and govern data that is registered in a Hive metastore. This includes externally-managed Hive metastores, AWS Glue, and legacy internal Databricks Hive Hive metastores.

See Hive metastore federation: enable Unity Catalog to govern tables registered in a Hive metastore.

Remove metastore-level storage to enforce catalog-level storage isolation

December 11, 2024

If you have metastore-level storage for managed tables and volumes (also known as the metastore storage root), but you want to enforce data storage isolation at the catalog or schema level, you can now remove that metastore-level storage without interrupting existing workloads. See Remove metastore-level storage.

Specify response formats for ai_query

December 17, 2024

ai_query now supports the responseFormat field for structured outputs. Use responseFormat in your ai_query requests to specify the response format you want the model you are querying to follow. See ai_query function.

Meta Llama 3.3 70B Instruct is now available on Model Serving

December 11, 2024

Mosaic AI Model Serving now supports Meta Llama 3.3 70B Instruct, a state-of-the-art large language model built and trained by Meta. Llama 3.3 70B Instruct is available as part of Foundation Model APIs pay-per-token. This availability also includes support on Function Calling.

Starting December 11, 2024, Meta-Llama-3.1-70B-Instruct replaces support for Meta-Llama-3-70B-Instruct in Foundation Model APIs pay-per-token endpoints.

bamboolib is now deprecated

December 10, 2024

bamboolib is now deprecated. Users can still access bamboolib to perform low-code data analysis within notebooks, but Databricks is no longer actively developing nor supporting this tool. For assistance with code generation, use the Databricks Assistant.

Streamline AI Agent Evaluation using synthetic evaluation sets

December 9, 2024

Evaluate your AI agent by generating a representative evaluation set from your documents. The synthetic generation API is tightly integrated with Agent Evaluation, allowing you to quickly evaluate and improve the quality of your agent’s responses without going through the costly process of human labeling. See Synthesize evaluation sets.

Improve Databricks-to-Databricks Delta Sharing table read performance with history sharing (Public Preview)

December 5, 2024

Improve performance for Databricks-to-Databricks table shares by enabling history sharing. See Improve table read performance with history sharing.

Maximum personal access token lifetime now 730 days (two years)

December 5, 2024

The default maximum lifetime for newly created Databricks-issued personal access tokens is now set to 730 days (two years). Previously, personal access tokens could be created with no expiration by default. With this update, users cannot generate new tokens with a lifetime exceeding 730 days, and tokens created without a specified lifetime are set to a 730-day duration. If you configured the maximum token lifetime for your workspace to less than 730 days, the configuration remains unchanged. See Monitor and revoke personal access tokens and Databricks personal access token authentication.

Mosaic AI Model Training - serverless forecasting (Public Preview)

December 5, 2024

Mosaic AI Model Training - forecasting improves upon the existing AutoML forecasting experience with managed serverless compute, Unity Catalog support, access to deep learning algorithms, and an upgraded interface. See Forecasting (serverless) with AutoML.

Add budget policies to model serving endpoints

December 4, 2024

Budget policies are now supported on model serving endpoints. See Manage model serving endpoints.