May 2023

These features and Databricks platform improvements were released in May 2023.

Note

Releases are staged. Your Databricks account might not be updated until a week or more after the initial release date.

Create or modify table UI supports Avro, Parquet, and text file uploads

May 29, 2023

The Create or modify table UI now supports the following file types:

  • Avro (.avro)

  • Parquet (.parquet)

  • Text (.txt)

For more information, see Create or modify a table using file upload.

Use the add data UI to load data using Unity Catalog external locations (Public Preview)

May 29, 2023

You can now use the add data UI to load data from a cloud object storage path that’s defined as a Unity Catalog external location. For more information, see Load data using a Unity Catalog external location.

Run Databricks notebooks on SQL warehouses (Public Preview)

May 31, 2023

You can now run Databricks notebooks on SQL warehouses. SQL warehouses are optimized for SQL analytics and deliver better price performance for SQL execution compared to all-purpose compute. For more information, see Use a notebook with a SQL warehouse.

Prevent Enter key from accepting autocomplete suggestion

May 26, 2023

You can now prevent the Enter key from accepting autocomplete suggestions in the SQL editor. Select Disable enter to accept autocomplete from the kebab menu in the SQL editor.

Databricks Terraform provider updated to version 1.17.0

May 22, 2023

Databricks Terraform provider 1.17.0 removes support for releasing 32-bit binaries, adds more information on the impact of using a cluster policy in the databricks_cluster resource, and more. For more details, see the changelog for version 1.17.0.

Upload data UI supports new column data types

May 19, 2023

The upload data UI now supports the following column data types:

  • STRUCT

  • ARRAY

  • DECIMAL(P,S)

For more information, see Create or modify a table using file upload.

Configure your workspace to use IMDS v2 (GA)

May 18, 2023

AWS provides an API called the IMDS API to read instance metadata in your notebooks. AWS announced IMDS version 2 (IMDSv2), which includes security improvements and a session-oriented flow with requests protected by session authentication. You can configure your workspace to enforce the use of IMDS v2 with a workspace admin setting that is now generally available.

If your workspace was created after October 1, 2022, this admin setting is enabled by default.

M7g and R7g Graviton3 instances are now supported on Databricks

May 17, 2023

Databricks now supports selecting M7g and R7g instances during cluster creation. These instances do not support Photon runtimes.

Bind Unity Catalog catalogs to specific workspaces

May 17, 2023

You can now limit catalog access to specific workspaces in your account, also known as workspace-catalog binding. This feature is especially helpful if you use workspaces to isolate user data access, for example if you have separate production and development workspaces or a dedicated workspace for handling sensitive data.

For more information, see Limit catalog access to specific workspaces.

All users can connect to Fivetran using Partner Connect (Public Preview)

May 17, 2023

The workspace admin user role is no longer required to connect to Fivetran using Partner Connect. For more information, see Connect to Fivetran using Partner Connect.

Authentication using OAuth tokens for service principals (Public Preview)

May 17, 2023

You can now authenticate to Databricks REST APIs using OAuth tokens for service principals. A service principal is an identity that you create in Databricks for use with automated tools, jobs, and applications. Account admins can create a client secret for a service principal. You can then use the client secret with the client ID, also known as the service principal’s application ID, to request an OAuth token for the service principal. You can use the same OAuth token for both the account and workspaces, as long as the service principal has the correct access. For more information, see Authenticate access to Databricks with a service principal using OAuth (OAuth M2M).

Schedule automatic cluster update (Public Preview)

May 16, 2023

For workspaces with the Enterprise pricing tier and the Enhanced Security and Compliance add-on, admins can configure a monthly or twice-monthly schedule for automatic restart of compute resources if needed to get the latest images and security updates. By default, the automatic cluster update are scheduled on the first Sunday of every month at 1:00 AM UTC. Admins can change the day of the month, the time, or switch to a twice-monthly schedule. This feature helps admins and users plan potential down time to restart the compute resources. If there are no updates for images for compute resources, they are not restarted. To participate in this preview, contact your Databricks account team. See Automatic cluster update.

If your workspace is not part of this preview, the previous behavior still applies: if the workspace enables the compliance security profile, clusters restart after 25 days.

Databricks Terraform provider updated to version 1.16.1

May 16, 2023

Databricks Terraform provider 1.16.1 adds documentation for the databricks_service_principal_secret resource and Unity Catalog integration in the databricks_pipeline resource, updates documentation for the databricks_secret_scope resource, and more. For more details, see the changelog for version 1.16.1.

Databricks JDBC driver 2.6.33

May 15, 2023

We have released version 2.6.33 of the Databricks JDBC driver (download). This release adds Primary Key and Foreign Key APIs when connecting to a server that has Unity Catalog enabled.

Partner Connect supports Alation

May 15-22, 2023

You can now connect your Databricks workspace to Alation using Partner Connect. For more information, see Connect Databricks to Alation.

New default theme for editor

May 12, 2023

Based on customer feedback, the default color scheme in the editor was updated for better readability.

Databricks Terraform provider updated to version 1.16.0

May 12, 2023

Databricks Terraform provider 1.16.0 adds a databricks_workspace_file resource, adds a notification_settings block to the databricks_job resource, and more. For more details, see the changelog for version 1.16.0.

New region: Europe (Paris)

May 11, 2023

Databricks is now available in AWS region eu-west-3, Paris, France. See Databricks clouds and regions.

Compliance security profile now supports more EC2 instance types

May 11, 2023

Workspaces that enable the compliance security profile must use an approved set of AWS Nitro instance types for compute resources. The compliance security profile now supports more instance types. For the latest list, see Compliance security profile features and technical controls.

Databricks Runtime 13.1 (Beta)

May 10, 2023

Databricks Runtime 13.1 and Databricks Runtime 13.1 ML are now available as Beta releases.

See Databricks Runtime 13.1 (EoS) and Databricks Runtime 13.1 for Machine Learning (EoS).

Run file-based SQL queries in a Databricks workflow

May 9, 2023

You can now retrieve SQL queries from a Git provider when you add a SQL task to a Databricks job. See Use Git with jobs.

Databricks Terraform provider updated to version 1.15.0

May 5, 2023

Version 1.15.0 adds an experimental databricks_sql_table resource, adds support for running databricks_cluster init scripts from workspace files, adds missing permissions to databricks_grants, and more. For more details, see the changelog for version 1.15.0.

Account nicknames now available in the account console

May 5, 2023

Account admins can now set account nicknames for their Databricks accounts. After you give your account a nickname, the name is displayed at the top of the account console and next to the account ID. For more information, see Add an account nickname.

Share notebooks using Delta Sharing

May 4, 2023

You can now use Delta Sharing to share notebook files securely using the Databricks-to-Databricks sharing flow.

Sharing notebooks empowers users to collaborate across metastores and accounts, and enables providers to demonstrate use cases and visualizations for the data that they share.

See Add notebook files to a share (for providers) and Read shared notebooks (for recipients).

Deprecation of cluster-scoped init scripts on DBFS

May 2, 2023

Cluster-scoped init scripts on DBFS are deprecated. They should be stored as workspace files instead. See Store init scripts in workspace files.

New region: South America (São Paulo)

May 1, 2023

Databricks is now available in AWS region sa-east-1, São Paulo, Brazil. See Databricks clouds and regions.

Unified navigation (Public Preview)

May 1, 2023

The new unified navigation experience is in Public Preview. Unified navigation reduces clicks and context switches required to complete tasks across product areas. For more information, see Navigate the workspace.

AWS fleet instance types now available

May 1, 2023

When creating a cluster or pool, you can now use fleet instance types. Fleet instance types map to multiple comparable AWS instance types, allowing the cluster to use whichever instance type has the best spot capacity and best on-demand availability. For more information, see AWS Fleet instance types.