Databricks Clusters UI changes

A new Clusters user interface is available with the following changes:

For more details on configuring compute for Unity Catalog, see Create clusters and SQL warehouses that can access Unity Catalog.

What is cluster access mode?

A new Access mode dropdown has replaced the Security mode drop down. Access modes are standardized as follows:

Access mode dropdown

Visible to user

Unity Catalog support

Supported languages

Single user

Always

Yes

Python, SQL, Scala, R

Shared

Always (Premium plan required)

Yes

Python, SQL

No isolation shared

This option will be hidden if “Enforce User Isolation” is configured in the admin console.

No

Python, SQL, Scala, R

Custom

This option will only be shown for existing clusters without access modes. It will be hidden for all new clusters.

No

Python, SQL, Scala, R

Important

Access mode in the clusters API is not yet supported. There are no changes to the existing Clusters API.

How do you configure cluster mode in the new Databricks Cluster UI?

The Cluster mode dropdown is replaced by a radio button with two options: Multi node and Single node. When a user toggles between the two modes of the radio button, the form will reset.

How do you configure credential passthrough in the new Cluster UI?

Credential passthrough does not work with Unity Catalog.

In Advanced options, the Enable credential passthrough checkbox will be available if the access mode is single user or shared. Check the box to enable credential passthrough on the cluster and disable Unity Catalog on the cluster.

How does backward compatability work with these changes?

All existing interactive clusters and job cluster definitions will continue to work as is. The behavior remains the same (no change) when you start or restart an existing cluster, or when an existing job run is triggered.

Note

Legacy cluster definitions will appear in the cluster UI as having a Custom access mode. This is for backward compatability only and cannot be selected from the dropdown list.

For workspaces without Unity Catalog (that is, no Unity Catalog metastore is assigned), use the following table to migrate from your original cluster settings to the new access mode when creating new clusters.

Original cluster settings

New access mode

Standard

No isolation shared

Standard + credential passthrough checkbox checked

Single user + credential passthrough checkbox checked

High concurrency (HC)*

No isolation shared

HC + Table ACL checkbox checked

Shared

HC + Credential Passthrough checkbox checked

Shared + credential passthrough checkbox checked

* We no longer differentiate high concurrency clusters from standard clusters as they have the same feature support

For customers with backward compatibility concerns, the new clusters UI can be disabled by toggling off the New UI is enabled switch on the Preview label: Cluster preview

Limitations

The following security-related Spark configurations are not available in the new UI when creating a new cluster. If you need to fine-tune a cluster configuration (for example, for integrations such as Privacera or Immuta) you can define a custom cluster policy to set these configurations. Contact your Databricks representative for assistance.

  • spark.databricks.repl.allowedLanguages

  • spark.databricks.acl.dfAclsEnabled

  • spark.databricks.passthrough.enabled

  • spark.databricks.pyspark.enableProcessIsolation