Databricks client unified authentication

Databricks client unified authentication centralizes setting up and automating authentication to Databricks. It enables you to configure Databricks authentication once and then use that configuration across multiple Databricks tools and SDKs without further authentication configuration changes.

Participating tools and SDKs

Participating Databricks tools and SDKs include:

All participating tools and SDKs accept special environment variables and Databricks configuration profiles for authentication. The Databricks Terraform provider and the Databricks SDKs for Python, Java, and Go also accept direct configuration of authentication settings within code. For details, see Developer tools and guidance for the tool’s or SDK’s documentation.

Default authenticate methods for client unified authentication

Whenever a participating tool or SDK needs to authenticate with Databricks, the tool or SDK tries the following types of authentication in the following order by default. When the tool or SDK succeeds with the type of authentication that it tries, the tool or SDK stops trying to authenticate with the remaining authentication types. To force an SDK to authenticate with a specific authentication type, set the Config API’s Databricks authentication type field.

  1. Databricks personal access token authentication

  2. Use a service principal to authenticate with Databricks (OAuth M2M)

  3. OAuth user-to-machine (U2M) authentication

For each authentication type that the participating tool or SDK tries, the tool or SDK tries to find authentication credentials in the following locations, in the following order. When the tool or SDK succeeds in finding authentication credentials that can be used, the tool or SDK stops trying to find authentication credentials in the remaining locations.

  1. Credential-related Config API fields (for SDKs).

  2. Credential-related environment variables.

  3. Credential-related fields in the DEFAULT configuration profile within the .databrickscfg file. To set configuration profile fields, see (/dev-tools/auth/config-profiles.md).

To provide maximum portability for your code, Databricks recommends that you create a custom configuration profile within the .databrickscfg file, add the belowrequired fields below for your target Databricks authentication type to the custom configuration profile, and then set the DATABRICKS_CONFIG_PROFILE environment variable to the name of the custom configuration profile.

Environment variables and fields for client unified authentication

The following tables list the names and descriptions of the supported environment variables and fields for Databricks client unified authentication. In the following tables:

  • Environment variable, where applicable, is the name of the environment variable.

  • .databrickscfg field, where applicable, is the name of the field within a Databricks configuration profiles file or Databricks Terraform configuration. To set .databrickscfg fields, see Databricks configuration profiles.

  • Terraform field, where applicable, is the name of the field within a Databricks Terraform configuration. To set Databricks Terraform fields, see Authentication in the Databricks Terraform provider documentation.

  • Config field is the name of the field within the Config API for the specified SDK.

General host, token, and account ID environment variables and fields

Common name

Description

Environment variable

.databrickscfg field, Terraform field

Config field

Databricks host

(String) The Databricks host URL for either the Databricks workspace endpoint or the Databricks accounts endpoint.

DATABRICKS_HOST

host

host (Python), setHost (Java), Host (Go)

Databricks token

(String) The Databricks personal access token.

DATABRICKS_TOKEN

token

token (Python), setToken (Java), Token (Go)

Databricks account ID

(String) The Databricks account ID for the Databricks account endpoint. Only has effect when the Databricks host is also set to https://accounts.cloud.databricks.com.

DATABRICKS_ACCOUNT_ID

account_id

account_id (Python), setAccountID (Java), AccountID (Go)

AWS-specific environment variables and fields

Common name

Description

Environment variable

.databrickscfg field, Terraform field

Config field

Databricks username

(String) The Databricks user’s username.

DATABRICKS_USERNAME

username

username (Python), setUsername (Java), Username (Go)

Service principal client ID

(String) The Databricks service principal’s client ID.

DATABRICKS_CLIENT_ID

client_id

client_id (Python), setClientId (Java), ClientId (Go)

Service principal secret

(String) The Databricks service principal’s secret.

DATABRICKS_CLIENT_SECRET

client_secret

client_secret (Python), setClientSecret (Java), ClientSecret (Go)

.databrickscfg-specific environment variables and fields

Use these environment variables or fields to specify non-default settings for .databrickscfg. See also Databricks configuration profiles.

Common name

Description

Environment variable

Terraform field

Config field

.databrickscfg file path

(String) A non-default path to the .databrickscfg file.

DATABRICKS_CONFIG_FILE

config_file

config_file (Python), setConfigFile (Java), ConfigFile (Go)

.databrickscfg default profile

(String) The default named profile to use, other than DEFAULT.

DATABRICKS_CONFIG_PROFILE

profile

profile (Python), setProfile (Java), Profile (Go)

Authentication type field

Use this environment variable or field to force an SDK to use a specific type of Databricks authentication.

Common name

Description

Terraform field

Config field

Databricks authentication type

(String) When multiple authentication attributes are available in the environment, use the authentication type specified by this argument.

auth_type

auth_type (Python), setAuthType (Java), AuthType (Go)

Supported Databricks authentication type field values include: