Amazon
Web Services
Microsoft Azure
Google Cloud PlatformNovember 2024
These features and Databricks platform improvements were released in November 2024.
Note
Releases are staged. Your Databricks account might not be updated until a week or more after the initial release date.
Additional metadata for the table tooltip in notebooks
November 25, 2024
When you hover over the name of a Unity Catalog table in a notebook cell, the tooltip now includes the following metadata:
Owner
Last updated
Size
Description
A link to view the table in the Catalog Explorer side panel
Databricks Assistant can use Databricks-hosted models
November 22, 2024
Power Databricks Assistant with Databricks-hosted models (Public Preview), rather than partner-hosted models for AI-driven features. Models are managed by Databricks and hosted on the same infrastructure that powers Databricks Model Serving.
FedRAMP Moderate and PCI-DSS now support serverless compute in us-west-2
November 21, 2024
FedRAMP Moderate and PCI-DSS compliance controls now support serverless SQL warehouses, serverless compute for notebooks and workflows, and serverless Delta Live Tables pipelines in us-west-2. See Which compute resources get enhanced security and Compliance security profile compliance standards with serverless compute availability.
Emergency access for workspaces without unified login
November 21, 2024
To prevent lockouts, workspace admins can now log in to Databricks workspaces with unified login disabled and single sign-on enabled using Databricks credentials and MFA. See Emergency access to prevent lockouts.
Automatic cluster update and enhanced security and compliance UI are generally available
November 19, 2024
Automatic cluster update is now generally available. Automatic cluster update ensures that all the clusters in a workspace are periodically updated to the latest host OS image and security updates. The ability to enable enhanced security and compliance features from the account console is also now generally available. See Automatic cluster update and Configure enhanced security and compliance settings.
Expanded support for Python debugger in notebooks
November 18, 2024
The built-in Python debugger in Databricks notebooks is now supported on serverless compute and compute configured with shared access mode in Databricks Runtime 14.3 LTS or above. To learn more about the debugger, see Debug notebooks.
Improved web terminal experience
November 18, 2024
Databricks made the following improvements to the web terminal experience:
Support for configuration files: You can now set persistent configurations for your Databricks web terminal with .bashrc configuration files. See Configure your web terminal.
Command cycle history: You can now cycle through commands executed in previous sessions with the “Up” and “Down” arrow keys. This makes it easy to recall and reuse previous commands without retyping them.
Shared Cluster Support: Our improved web terminal is available on shared clusters in Databricks Runtime 15.1 and above.
To learn more about using the web terminal, see Run shell commands in Databricks web terminal.
Query history and query profile support for Delta Live Tables
November 15, 2024
Databricks now offers improved tools for monitoring query performance in Delta Live Tables pipelines. See Access query history for Delta Live Tables pipelines.
Mosaic AI Model Training rebrand
November 15, 2024
Mosaic AI Model Training has been rebranded to encompass existing features:
Foundation Model Fine-tuning, previously known as Mosaic AI Model Training, continues to offer advanced customization for foundation models.
AutoML features can now be accessed from the Experiments section.
Reuse SQL outputs as _sqldf in subsequent SQL cells
November 15, 2024
In Databricks Runtime 14.3 and above, running a SQL query in a Databricks notebook generates an implicit DataFrame (_sqldf) that you can use in downstream SQL cells. This adds to the support for using for using the _sqldf variable in Python cells introduced in Databricks Runtime 13.3. See Explore SQL cell results.
Warehouses system table is now available (Public Preview)
November 15, 2024
The system.compute.warehouses table records SQL warehouse configurations. Each row is a snapshot of a SQL warehouse’s properties at a specific point in time. A new snapshot is created when the properties change. See Warehouses system table reference.
Cross-platform view sharing (Public Preview)
November 14, 2024
Databricks recipients can now query shared views using any Databricks compute resource. Previously, if a recipient’s Databricks account was different from the provider’s account, recipients could only query a shared view using a serverless SQL warehouse. See Read shared views.
View sharing also now extends to open sharing connectors. See Read data shared using Delta Sharing open sharing (for recipients).
Clean Rooms public preview is ungated, with new features
November 13, 2024
Databricks Clean Rooms uses Delta Sharing and serverless compute to provide a secure and privacy-protecting environment where multiple parties can share sensitive enterprise data and collaborate without direct access to each other’s data. The Clean Rooms public preview is now open to all customers on Azure and AWS, and includes the following new features:
Output tables are temporary read-only tables generated by a notebook run and shared to the notebook runner’s Unity Catalog metastore.
Clean room notebook task types let you run Databricks notebooks in a clean room as part of a workflow. Along with support for task values and output tables, this is a powerful tool for running complex workflows on data shared in clean rooms.
Serverless egress control lets you restrict outbound access from central clean rooms to internet destinations that you specify.
Databricks Runtime 16.0 is GA
November 11, 2024
Databricks Runtime 16.0 and Databricks Runtime 16.0 ML are now generally available.
See Databricks Runtime 16.0 and Databricks Runtime 16.0 for Machine Learning.
Breaking change: Hosted RStudio is end-of-life
November 11, 2024
In Databricks Runtime 16.0 and above, Databricks-hosted RStudio Server is end-of-life and unavailable on any Databricks workspace. To learn more and see a list of alternatives to RStudio, see Hosted RStudio Server deprecation.
Predictive optimization enabled for all new Databricks accounts
November 11, 2024
On November 11, Databricks enabled predictive optimization as the default for all new Databricks accounts. Previously, it was disabled by default and could be enabled through your administrator account. When predictive optimization is enabled, Databricks automatically runs maintenance operations for Unity Catalog managed tables. For more information on predictive optimization, see Predictive optimization for Unity Catalog managed tables.
Updated and unified read-only code blocks
November 8, 2024
A new read-only code block is now available across the Catalog Explorer, alerts page, and Genie.
Increased Quick Fix recommendations
November 8, 2024
Databricks Assistant Quick Fix now identifies and suggests corrections for a broader range of common SQL and Python errors. With a 50% increase in error detection, Quick Fix can help you debug your code faster. See Quick Fix.
Foundation Model APIs pay-per-token available in all Model Serving supported regions
November 7, 2024
Foundation Model APIs pay-per-token workloads are now supported in all regions where Mosaic AI Model Serving is available. If your workspace is in a Model Serving region but not in a U.S. or EU region, your workspace must be enabled for cross-Geo data processing. When enabled, your pay-per-token workload is routed to the U.S. Databricks Geo.
Unity Catalog-governed access to external cloud services using service credentials (Public Preview)
November 7, 2024
Service credentials enable simple and secure authentication with your cloud tenant’s services using IAM roles and Unity Catalog. Service credentials are in public preview and are supported on Databricks Runtime 15.4 LTS and above. See Manage access to external cloud services using service credentials.
Databricks Runtime 14.2 series support ends
November 5, 2024
Support for Databricks Runtime 14.2 and Databricks Runtime 14.2 for Machine Learning ended on November 5. See Databricks support lifecycles.