February 2024

These features and Databricks platform improvements were released in February 2024.

Note

Releases are staged. Your Databricks account might not be updated until a week or more after the initial release date.

Use Delta Live Tables in Feature Engineering (Public Preview)

February 29, 2024

Tables that are published to Unity Catalog from Delta Live Tables pipelines and that include a primary key can now be used as feature tables for model training or inference. Unity Catalog provides feature discovery, governance, lineage, and cross-workspace access. For more information, see Create a feature table in Unity Catalog.

Restrict creating a personal access token for a service principal

February 29, 2024

Workspace admins can create personal access tokens on behalf of any service principal in their workspace. Account admins can now enforce that workspace admins have the Service Principal User role on a service principal in order to create a personal access token on behalf of the service principal. You can automate enabling this setting after workspace creation. See Restrict workspace admins.

Restrict changing a job owner and the run as setting

February 29, 2024

Workspace admins can change a job owner to any user or service principal in their workspace by default and the job run as setting to any user in their workspace or service principals that they have the Service Principal User role on.

Account admins can now restrict workspace admins to only change a job owner to themselves and the job run as setting to service principals that they have the Service Principal User role on or to themselves. You can automate enabling this setting after workspace creation. See Restrict workspace admins.

Automatic cluster update is enabled if the compliance security profile is enabled (GA)

February 27, 2024

Automatic cluster update is now generally available. If a workspace has the compliance security profile enabled, automatic cluster update is permanently enabled and the maintenance window is configurable. The previous 25-day limit for running clusters was replaced by your maintenance window. See Automatic cluster update.

Account admins can enable enhanced security and compliance features (Public Preview)

February 27, 2024

Account admins can now enable features of the Enhanced Security and Compliance add-on: the compliance security profile, enhanced security monitoring, and automatic cluster update on a workspace. You can also now add account-level default for new workspaces for the compliance security profile and enhanced security monitoring. For both workspace-level and account-level settings, use the account console UI.

If you previously enabled the compliance security profile, your account default for new workspaces is unchanged. You can now change your account’s defaults for new workspaces.

To use these features, contact your Databricks account representative to add the Enhanced Security and Compliance add-on. See the pricing page.

The automatic cluster update feature and the ability to enable Enhanced Security and Compliance features on your own are in Public Preview. However, the compliance security profile itself and enhanced security monitoring are generally available (GA).

Support for Cloudflare R2 storage to avoid cross-region egress fees (Public Preview)

February 26, 2024

You can now use Cloudflare R2 as cloud storage for data registered in Unity Catalog. Cloudflare R2 is intended primarily for Delta Sharing use cases in which you want to avoid the data egress fees charged by cloud providers when data crosses regions. R2 storage supports all of the Databricks data and AI assets supported in AWS S3, Azure Data Lake Storage Gen2, and Google Cloud Storage. Support for R2 requires Databricks Runtime 14.3 or SQL Warehouse 2024.15. See Use Cloudflare R2 replicas or migrate storage to R2 and Create a storage credential for connecting to Cloudflare R2.

Notebooks for monitoring and managing Delta Sharing egress costs are now available

February 26, 2024

Notebooks for monitoring and managing Delta Sharing egress costs are now available through Databricks Marketplace. In addition, new documentation provides recommendations for avoiding egress fees. See Monitor and manage Delta Sharing egress costs (for providers).

Add data UI supports XML file format

February 23, 2024

The add data UI now supports XML file upload and ingestion from cloud object storage. See Upload files to Databricks.

Support for cloud storage firewall from serverless compute (Public Preview)

February 23, 2024

You can now configure firewalls on your resources to allow access from Databricks serverless compute plane. Account admins can enforce that serverless compute resources use stable IP CIDR blocks to connect to your resources. To join this preview, contact your Databricks account team.

See Configure a firewall for serverless compute access.

Use AI Functions to invoke a generative AI model from Foundation Model APIs

February 22, 2024

You can now use AI Functions to invoke a state-of-the-art generative AI model from Databricks Foundation Model APIs to perform AI tasks like sentiment analysis and classification.

See AI functions using Databricks Foundation Model APIs for available functions.

Unity Catalog volumes are GA

February 22, 2024

Unity Catalog volumes are now generally available in Databricks Runtime 14.3 LTS and above. Databricks recommends using volumes for managing read and write access to non-tabular data in cloud object storage. See What are Unity Catalog volumes?.

Full-page AI-powered search

February 16-22, 2024

Databricks search now leverages DatabricksIQ to provide a more intelligent AI-powered search experience. The new full-page search experience gives you more space to see results, more metadata for your objects, and more filters to narrow down your results.

See Search for workspace objects.

Run SQL notebook jobs on a SQL warehouse

February 8, 2024

You can now schedule and execute notebook jobs using a SQL warehouse as the compute resource. For details, see Use a notebook with a SQL warehouse.

File arrival triggers in Databricks Jobs is GA

February 7, 2024

File arrival triggers is now generally available in all cloud providers. With this release, you can use file arrival triggers to run a Databricks job when new files arrive in a Unity Catalog volume in addition to the existing support for Unity Catalog external locations. See Trigger jobs when new files arrive.

Databricks Git server proxy is GA

February 6, 2024

You can use Databricks Git server proxy to configure secure access from Databricks Repos to your on-premises Git server. See Set up private Git connectivity for Databricks Git folders (Repos).

Databricks Git server proxy no longer requires CAN_ATTACH_TO permissions

February 6, 2024

The GA release of Databricks Git server proxy has removed the need to provide all users with CAN_ATTACH_TO permissions.

If you previously configured Databricks Git server proxy with CAN_ATTACH_TO privileges, see Set up private Git connectivity for Databricks Git folders (Repos).

Workspace file support for the dbt and SQL file tasks is GA

February 1, 2024

Support for using dbt projects and SQL files stored in Databricks workspace folders is now generally available, adding to the existing support for using files stored in a remote Git repository. See dbt task for jobs and SQL task for jobs.

Databricks Connect is GA for Scala

February 1, 2024

Beginning with Databricks Runtime 14.3 LTS, Databricks Connect is now generally available for Scala.

Create tables from files in volumes

February 1, 2024

You can now use the UI to create Unity Catalog managed tables from data files stored in Unity Catalog volumes. See Create a table from data in a volume.

Databricks Runtime 14.3 LTS is GA

February 1, 2024

Databricks Runtime 14.3 LTS and Databricks Runtime 14.3 LTS ML are now generally available.

See Databricks Runtime 14.3 LTS and Databricks Runtime 14.3 LTS for Machine Learning.

Delta Sharing supports tables that use deletion vectors (Public Preview)

February 1, 2024

Delta Sharing now supports the sharing of tables that use deletion vectors, which are a storage optimization feature that you can enable on Delta tables. Recipients can read tables that use deletion vectors using a SQL warehouse, a cluster running Databricks Runtime 14.1 or above, or compute that is running open source delta-sharing-spark 3.1 or above. Support is rolling out gradually over the coming month.

See Add tables with deletion vectors or column mapping to a share, Read tables with deletion vectors or column mapping enabled, and Read tables with deletion vectors or column mapping enabled.