July 2024

These features and Databricks platform improvements were released in July 2024.

Note

Releases are staged. Your Databricks account might not be updated until a week or more after the initial release date.

Node timeline system table is now available (Public Preview)

July 23, 2024

The system.compute schema now includes a node_timeline table. This table logs minute-by-minute utilization metrics for the all-purpose and jobs compute resources run in your account. See Node timeline table schema.

Note

To access this table, an admin must enable the compute schema if you have not already. See Enable system table schemas.

Meta Llama 3.1 is now supported in Model Serving

July 23, 2024

Mosaic AI Model Serving has partnered with Meta to support Meta Llama 3.1, a model architecture built and trained by Meta. Llama 3.1 is supported as part of Foundation Model APIs. See [See Use Foundation Model APIs.]

  • Meta-Llama-3.1-405B-Instruct is available in pay-per-token serving endpoint regions.

  • Meta-Llama-3.1-70B-Instruct is available based on regional compute availability in pay-per-token serving endpoint regions.

  • Production usage of the full suite of Llama 3.1 models (8B, 70B, and 405B) is available in the US using provisioned throughput.

Starting July 23, 2024, Meta-Llama-3.1-70B-Instruct support is available based on regional compute availability, and replaces support for Meta-Llama-3-70B-Instruct in Foundation Model APIs pay-per-token functionality. If Meta-Llama-3.1-70B-Instruct is not available in your region, you can continue to use Meta-Llama-3-70B-Instruct.

Unity Catalog will soon drop support for storage credentials that use non-self-assuming IAM roles

July 22, 2024

Starting on September 20, 2024, Databricks will require that AWS IAM roles for new storage credentials be self-assuming. On January 20, 2025, Databricks will enforce this requirement on all existing storage credentials. Storage credentials that violate this requirement will cease to work, which might cause dependent workloads and jobs to fail. To learn more about this requirement and how to check and update your storage credentials, see Self-assuming role enforcement policy.

/ in workspace asset names is deprecated

July 17, 2024

To avoid ambiguity in path strings, the use of ‘/’ in the names of new workspace assets (such as notebooks, folders, and queries) has been deprecated. Existing assets with ‘/’ in their names are not affected, but renaming of existing assets follows the same rules as new assets.

Delta Sharing lets you share tables that use liquid clustering

July 16, 2024

Delta Sharing now lets you share tables that are enabled for liquid clustering, and recipients can run batch queries against them. Liquid clustering simplifies data layout decisions and optimizes query performance. See Use liquid clustering for Delta tables and Delta Lake feature support matrix.

Query history system table is now available (Public Preview)

July 16, 2024

Databricks system tables now includes a query history table. This table logs detailed records of each query run on a SQL warehouse in your account. To access the table, admins must enable the new query system schema. See Query history system table reference.

Vulnerability scan reports are now emailed to admins

July 16, 2024

Vulnerability scan reports are now emailed to workspace admins in workspaces that enable enhanced security monitoring. Previously, workspace admins had to request them from Databricks. See Enhanced security monitoring.

Partition metadata logging for Unity Catalog external tables

July 15, 2024

In Databricks Runtime 13.3 LTS and above, you can optionally enable partition metadata logging, which is a partition discovery strategy consistent with Hive metastore for external tables registered to Unity Catalog that use Parquet, ORC, CSV, or JSON. See Partition discovery for external tables.

Serverless compute for workflows is GA

July 15, 2024

Serverless compute for workflows is now generally available. Serverless compute for workflows allows you to run your Databricks job without configuring and deploying infrastructure. With serverless compute for workflows, Databricks efficiently manages the compute resources that run your job, including optimizing and scaling compute for your workloads. See Run your Databricks job with serverless compute for workflows.

Serverless compute for notebooks is GA

July 15, 2024

Serverless compute for notebooks is now generally available. Serverless compute for notebooks gives you on-demand access to scalable compute in notebooks, letting you immediately write and run your Python or SQL code. See Serverless compute for notebooks.

Databricks Connect for Python now supports serverless compute

July 15, 2024

Databricks Connect for Python now supports connecting to serverless compute. This feature is available in Public Preview. See Configure a connection to serverless compute.

Filter data outputs using natural language prompts

July 11, 2024

You can now use the Databricks Assistant to filter data outputs using natural language prompts. For instance, to filter the Titanic survivors data table, you could type “Show me only males over 70.” See Filter data with natural language prompts.

Plaintext secrets support for external models

July 11, 2024

You can now directly input API keys as plaintext strings to model serving endpoints that host external models. See Configure the provider for an endpoint.

Forecast time series data using ai_forecast()

July 11, 2024

AI Functions now supports ai_forecast(), a new Databricks SQL function for analysts and data scientists designed to extrapolate time series data into the future. See ai_forecast function.

Lakehouse Federation supports Salesforce Data Cloud (Public Preview)

July 10, 2024

You can now run federated queries on data managed by Salesforce Data Cloud. See Run federated queries on Salesforce Data Cloud.

Databricks Assistant system table now available (Public Preview)

July 10, 2024

Databricks Assistant events are now logged in a system table located at system.access.assistant_events. See Databricks Assistant system table reference.

Account SCIM API v2.1 (Public Preview)

July 10, 2024

The Account SCIM APIs are updated from v2.0 to v2.1 for speed and reliability. You can download a PDF of the Account SCIM v2.1 API reference.

Mosaic AI Model Training available to all us-west-2 customers (Public Preview)

July 10, 2024

Mosaic AI Model Training, formerly Foundation Model Training, is now available to all customers in the us-west-2 region. Customers no longer need to request access to use this feature in this region.

With Mosaic AI Model Training, you use your own data to customize a foundation model to optimize its performance for your specific application. By fine-tuning or continuing training of a foundation model, you can train your own model using significantly less data, time, and compute resources than training a model from scratch. See Introduction to Mosaic AI Model Training.

UK Cyber Essentials Plus compliance controls

July 10, 2024

UK Cyber Essentials Plus (UKCE+) controls provide enhancements that help you with cyber essentials compliance for your workspace. UKCE+ is a certification created by the UK government to simplify and standardize IT security practices for commercial organizations who interact with UK government data. See UK Cyber Essentials Plus compliance controls.

End of life for Databricks-managed passwords

July 10, 2024

Starting on July 10, 2024, you can no longer use Databricks-managed passwords to authenticate to the Databricks UI or APIs, known as basic authentication. If you do not have single sign-on configured, users now receive a unique code via email to log in. For automation, Databricks recommends using OAuth authentication. You can also authenticate with personal access tokens.

See End of life for Databricks-managed passwords.

Sign-in with one-time passcodes and external accounts

July 10, 2024

You can now allow users to sign in to Databricks using one-time passcodes or common external accounts, such as Google or Microsoft. See Sign-in with email or external accounts.

Resource quota increase for tables per Unity Catalog metastore

July 3, 2024

Your Unity Catalog metastore can now register up to one million tables. See Resource quotas.

Databricks Assistant can diagnose notebook errors automatically

July 2, 2024

Databricks Assistant can now run /fix in notebooks automatically when it detects an error message. Assistant uses generative AI to analyze your code and the error message to suggest a fix directly in your notebook. For more information, see Automatically diagnose errors.

OAuth in Databricks on AWS GovCloud

July 1, 2024

OAuth authentication is now available in Databricks on AWS GovCloud. See Use a service principal to authenticate with Databricks.