August 2024

These features and Databricks platform improvements were released in August 2024.

Note

Releases are staged. Your Databricks account might not be updated until a week or more after the initial release date.

Monitor Unity Catalog object usage against quotas using the new Resource Quotas APIs

August 30, 2024

The new Resource Quotas APIs enable you to monitor your usage of Unity Catalog securable objects against resource quotas. Soon, you’ll also be able to receive email notifications when you approach quota limits. See Monitor your usage of Unity Catalog resource quotas and the Resource Quotas API reference.

MPT 7B Instruct and MPT 30B Instruct model retirements

August 30, 2024

The MPT 7B Instruct and MPT 30B Instruct models are now retired. See Retired models for recommended replacement models.

AskSupport replaced by Databricks assistant

August 29, 2024

AskSupport, the Databricks Slack-based support channel, has been replaced with Databricks Assistant, which is available in your Databricks workspace. Use Databricks Assistant to search technical documentation, create tickets, and get context-aware support.

To use AI Assistant, AI Assistant must be enabled in your account console.

Lakeflow system tables are extend with additional columns

August 23, 2024

The tables in system.lakeflow schema are being extended with additional columns. The following changes have been made:

  • jobs is extended with description column.

  • job_run_timeline is extended with run_type, run_name, compute_ids, termination_code and job_parameters columns.

  • job_task_run_timeline is extended with job_run_id, parent_run_id and termination_code columns.

The schema change is non-breaking and won’t interrupt your existing workflows. The new columns will not be backfilled for already emitted rows. For more information, see Jobs system table reference.

Python code formatting error highlights

August 23, 2024

Python code in notebooks and file editors can highlight formatting errors and warnings like unexpected indentation, long line length, and more. See Python formatting highlighting.

Databricks on AWS GovCloud is now GA

August 22, 2024

AWS GovCloud gives United States government customers and their partners the flexibility to architect secure cloud solutions that comply with the FedRAMP High baseline and other compliance regimes. Databricks on AWS GovCloud provides the Databricks platform deployed in AWS GovCloud with compliance and security controls. See Databricks on AWS GovCloud.

Delta Sharing: More Delta Lake features now supported by the Python and Power BI connectors

August 21, 2024

Delta Sharing Python connector 1.1.0+ and Power BI v2.132.908.0+ now support:

  • Column mapping name mode

  • Deletion vectors

  • Uniform format

These Delta Lake features were already supported on Databricks Runtime 14.1+ and open-source Delta Sharing Apache Spark connector 3.1+.

See Delta Lake feature support matrix.

Delta Sharing adds support for TimestampNTZ

August 21, 2024

Delta Sharing adds support for TimestampNTZ on Databricks Runtime 14.1 and above and open-source Delta Sharing Apache Spark connector 3.3 and above.

See Delta Lake feature support matrix.

The Databricks Jobs For each task is GA

August 21, 2024

The For each task is now generally available. You can use the For each task to run another task in a loop, passing a different set of parameters to each iteration of the task. The For each task can iterate over any of the standard job tasks, such as a notebook, JAR, Python script, or SQL task. See Run a parameterized Databricks job task in a loop.

Databricks Runtime 15.4 LTS is GA

August 19, 2024

Databricks Runtime 15.4 LTS and Databricks Runtime 15.4 LTS ML are now generally available.

See Databricks Runtime 15.4 LTS and Databricks Runtime 15.4 LTS for Machine Learning.

Personalized notebook autocomplete

August 19, 2024

Notebook autocomplete now prioritizes suggestions based on your individual Unity Catalog metadata and usage, providing custom suggestion rankings for each user. See Personalized autocomplete

Configure your workspace’s default access mode for jobs compute

August 16, 2024

Workspace admins can now configure the default access mode for jobs compute in their workspace. This default access mode is applied to compute resources without a defined access mode. For more information, see Default access mode for jobs compute.

New slash commands for Databricks Assistant

August 14, 2024

Databricks Assistant has added the following slash commands as shortcuts for common tasks:

  • /findTables: Searches for relevant tables based on Unity Catalog metadata.

  • /findQueries: Searches for relevant queries based on Unity Catalog metadata.

  • /prettify: Formats code for readability.

  • /rename: Suggests updated names to notebook cells and other elements, depending on the context.

  • /settings: Adjusts your notebook settings directly from Assistant.

For more information, see Use slash commands for prompts.

Workspace search now supports volumes

August 14, 2024

Volumes are now included in search results. See Search for workspace objects.

Databricks JDBC driver 2.6.40

August 13, 2024

Databricks JDBC Driver version 2.6.40 is now available from JDBC driver download. This release removes redundant WARNING log messages to increase logging usability and security.

This release includes the following enhancements and new features:

  • OIDC discovery endpoint support. The driver can now set an OIDC discovery endpoint to fetch a token and retrieve an authorization endpoint.

  • Updated Arrow support. The driver now uses Apache Arrow version 14.0.2. Earlier versions of the driver used Apache Arrow version 9.0.0.

  • ProxyIgnoreList support. The driver now supports ProxyIgnoreList property when UseProxy is set to 1.

  • Refresh token support. The driver now supports an optional refresh token. It saves the access token and reuses it for new connections as long as it is valid. If the driver cannot renew the access token using the refresh token, it will sign in again.

  • Updated authentication support. The driver now supports browser-based (U2M) and client credentials (M2M) authentication on Google Cloud.

  • Added unified default OAuth options.

  • You can now configure the OAuth redirect port. To do this, set the OAuth2RedirectUrlPort property to your port.

For complete configuration information, see the Databricks JDBC Driver Guide installed with the driver download package.

Databricks personal access tokens revoked if unused after 90 days

August 13, 2024

Databricks now automatically revokes any personal access tokens (PATs) that have not been used in 90 or more days. For more details, see Monitor and revoke personal access tokens.

Serverless SQL warehouses with the compliance security profile is now GA

August 13, 2024

Support for serverless SQL warehouses with the compliance security profile is now GA. Support varies by region. See Serverless compute feature availability.

In regions where this feature is supported, workspaces enabled for the compliance security profile now use serverless SQL warehouses as their default warehouse type. See Which compute resources get enhanced security.

Clusters API now supports partial configuration updates

August 13, 2024

A new API call allows you to partially update a cluster configuration, requiring you to specify only the attributes you want to update. See Update cluster configuration (partial) in the REST API reference.

Wrap lines in notebook cells

August 12, 2024

You can now enable or disable line wrapping in notebook cells, allowing text to either wrap onto multiple lines or remain on a single line with horizontal scrolling. See Line wrapping.

Files can no longer have identical names in workspace folders

August 9, 2024

Databricks now prevents you from creating or renaming assets in workspace folders when an asset’s name matches another file’s name exactly, accounting for the asset’s file extension. For example, you can no longer create a file named test.py if there is already a notebook with a base name of test with an extension of .py in the same workspace folder.

For more details, see Naming assets in workspace folders.

Compute policy enforcement now available

August 8, 2024

Policy compliance enforcement enables workspace admins to update their workspace’s compute resources to comply with the latest version of a policy. This feature can be used in the UI or through the Cluster Policies API.

See Enforce policy compliance or Cluster Policies API.

Foundation Model APIs pay-per-token is now GA

August 7, 2024

Foundation Model APIs pay-per-token is now generally available. See Pay-per-token Foundation Model APIs.

Collaborate on data projects securely and privately using Databricks Clean Rooms (Public Preview)

August 6, 2024

Databricks Clean Rooms uses Delta Sharing and serverless compute to provide a secure and privacy-protecting environment where multiple parties can share sensitive enterprise data and collaborate without direct access to each other’s data.

With Clean Rooms, users from other Databricks accounts can collaborate to generate unique insights about shared projects, such as advertising campaigns, investment decisions, or research and development, without moving or exposing sensitive data. Run complex workloads in an ephemeral environment using any language supported by Databricks notebooks, including Python, which provides native support for machine learning workloads.

To try it out, reach out to your Databricks representative.

See What is Databricks Clean Rooms?.

Mosaic AI Vector Search is now HIPAA-compliant

August 6, 2024

Mosaic AI Vector Search is now HIPAA-compliant in all regions except us-west-2.

Format columns in notebook and query results tables

August 6, 2024

Customize your results tables to be more readable with column-formatting options like Currency, Percentage, URL, control over decimal places, and more. See Format columns.

Row filters and column masks are now GA, with improvements

August 6, 2024

The ability to apply row filters and column masks to tables is now generally available on Databricks Runtime 12.2 and above. Row filters and colum masks prevent access to sensitive data by specified users. These filters and masks are implemented as SQL user-defined functions (UDFs). GA brings support for the following functionality that was not available in the public preview:

  • Constant expressions in policy parameters (strings, numeric, intervals, booleans, nulls).

  • Materialized views and streaming tables (Public Preview).

  • MERGE statements.

  • Table sampling.

The separate release of support for fine-grained access control on single user compute also expands compute options for working on tables with row filters and column masks applied.

See Filter sensitive table data using row filters and column masks.

Canadian Centre for Cybersecurity (CCCS) Medium (Protected B) compliance controls

August 2, 2024

Canadian Centre for Cybersecurity (CCCS) Medium (Protected B) controls provide enhancements that help you with compliance for your workspace. CCCS Medium is a profile of security controls by the Government of Canada appropriate for Protected B, medium integrity, and medum availabilityi (PBMM) categorized workloads. See Canadian Centre for Cybersecurity (CCCS) Medium (Protected B) compliance controls.

Lakehouse Federation is generally available (GA)

August 1, 2024

In Databricks Runtime 15.2 and later and Databricks SQL version 2024.30 and later, Lakehouse Federation connectors across the following database types are generally available (GA):

  • MySQL

  • PostgreSQL

  • Amazon Redshift

  • Snowflake

  • Microsoft SQL Server

  • Azure Synapse (SQL Data Warehouse)

  • Databricks

This release also introduces the following improvements:

  • Support for additional pushdowns (string, math, and miscellaneous functions).

  • Improved pushdown success rate across different query shapes.

  • Additional pushdown debugging capabilities:

    • The EXPLAIN FORMATTED output displays the pushed-down query text.

    • The query profile UI displays the pushed-down query text, federated node identifiers, and JDBC query execution times (in verbose mode). See View system-generated federated queries.

Meta Llama 3.1 405B models supported in Foundation Model Fine-tuning

August 1, 2024

Meta Llama 3.1 405B models are now supported in Foundation Model Fine-tuning. See Supported models.

Create a new workspace with enhanced security and compliance features

August 1, 2024

You can now enable the compliance security profile, add compliance standards, and enable enhanced security monitoring during workspace creation. See Create a new workspace with enhanced security and compliance features.