Authorizing access to Databricks resources

This page discusses the basic approaches for making secured Databricks CLI or REST API calls using Databricks account credentials, such as user accounts or service principals.

Authorization for the Databricks CLI and APIs

To access a Databricks resource with the Databricks CLI or REST APIs, clients must authorize using a Databricks account. This account must have permissions to access the resource, which can be configured by your Databricks administrator or a user account with adminstrator privileges.

There are two types of accounts that you can use, depending on how you intend to access your Databricks resources:

• User account: Use this to interactively enter Databricks CLI commands or REST API calls.
• Service principal: Use this to automate Databricks CLI commands or REST API calls without human interaction.

Acquire an access token

Once you have decided on the Databricks account type, you must acquire an access token that represents the account's credentials. You provide this access token when accessing the account's resources in your scripts or code, or in interactive sessions.

Your account's credentials are represented by a secure access token, which you provide either directly or indirectly to the CLI command or API call.

To securely run a Databricks CLI command or API request that requires authorized access to an account or workspace, you must provide an access token based on valid Databricks account credentials.

The following table shows the authorization methods available to your Databricks account.

Account-level APIs and workspace-level APIs

To authenticate with Databricks REST APIs, you must understand the difference between account-level and workspace-level APIs.

A Databricks account can host multiple workspaces and is managed through the account console. A workspace contains resources like jobs and notebooks and is managed by workspace admins.

• Databricks account-level APIs are available only to account owners and account admins. These APIs are hosted on the account console URL.
• Databricks workspace-level APIs are available to workspace users and admins. These APIs are hosted at workspace URLs.

Databricks authorization methods

Because Databricks tools and SDKs work with one or more supported Databricks authorization methods, you can select the best authorization method for your use case. For details, see the tool or SDK documentation in Local development tools.

Method	Description	Use case
OAuth token federation (Recommended)	OAuth tokens from your identity provider for users or service principals.	Enables you to authenticate to Databricks without managing Databricks secrets.
OAuth for service principals (OAuth M2M)	Short-lived OAuth tokens for service principals.	Unattended authentication scenarios, such as fully automated and CI/CD workflows.
OAuth for users (OAuth U2M)	Short-lived OAuth tokens for users.	Attended authentication scenario, where you use your web browser to authenticate with Databricks in real time, when prompted.
Personal access tokens (PAT) (Not recommended)	Short-lived or long-lived tokens for users or service principals.	Only use this in cases where your target tool does not support OAuth.

note

Basic authentication using a Databricks username and password reached end of life on July 10, 2024. See End of life for Databricks-managed passwords.

What authorization option should I choose?

Databricks provides two options for authorization or authentication with an access token:

• OAuth 2.0-based access tokens
• Personal access tokens (PATs)

important

Databricks strongly recommends you use OAuth tokens over PATs for authorization as OAuth tokens are automatically refreshed by default and do not require the direct management of the access token, improving your security against token hijacking and unwanted access.

Because OAuth automatically creates and manages the access token, you don't need to directly provide a token string. Choose PATs only when you are integrating a third-party tool or service that is unsupported by Databricks unified authentication or has no OAuth support.

How do I use OAuth to authorize access to Databricks resources?

Databricks unified authentication uses a standard set of environment variables to store credential values. These variables let you run Databricks CLI commands or call APIs without repeatedly configuring authentication.

User account authorization

• With tools and SDKs that support unified authentication, Databricks OAuth automatically creates and manages access tokens.
• If a tool doesn't support unified authentication, manually generate an OAuth code verifier and challenge pair and use it in CLI commands or API requests. See Step 1: Generate a code verifier and challenge.

Service principal authorization

• Databricks OAuth requires the caller to provide client credentials and a token endpoint URL. Tools and SDKs that support unified authentication handle this for you.
• Credentials include a unique client ID and client secret.
• The client (the Databricks service principal running your code) must be assigned to workspaces. After assignment, Databricks provides the client ID and secret, which you set using environment variables.

These environment variables are:

Environment variable	Description
DATABRICKS_HOST	This environment variable is set to the URL of either your Databricks account console (http://accounts.cloud.databricks.com) or your Databricks workspace URL (https://{workspace-id}.cloud.databricks.com). Choose a host URL type based on the type of operations your code performs. Specifically, if you're using Databricks account-level CLI commands or REST API requests, set this variable to your Databricks account URL. If you're using Databricks workspace-level CLI commands or REST API requests, use your Databricks workspace URL.
DATABRICKS_ACCOUNT_ID	Used for Databricks account operations. This is your Databricks account ID. To get it, see Locate your account ID.
DATABRICKS_CLIENT_ID	(Service principal OAuth only) The client ID you were assigned when creating your service principal.
DATABRICKS_CLIENT_SECRET	(Service principal OAuth only) The client secret you generated when creating your service principal.

You can set these directly, or through the use of a Databricks configuration profile (.databrickscfg) on your client machine.

To use an OAuth access token, your Databricks workspace or account administrator must have granted your user account or service principal the CAN USE privilege for the account and workspace features your code will access.

For more details on configuring OAuth authorization for your client and to review cloud provider-specific authorization options, see Unified client authentication.

Authentication for third-party services and tools

If you are writing code which accesses third-party services, tools, or SDKs you must use the authentication and authorization mechanisms provided by the third-party. However, if you must grant a third-party tool, SDK, or service access to your Databricks account or workspace resources, Databricks provides the following support:

• Databricks Terraform Provider: This tool can access Databricks APIs from Terraform on your behalf, using your Databricks user account. For more details, see Provision a service principal by using Terraform.
• Git providers such as GitHub, GitLab, and Bitbucket can access Databricks APIs using a Databricks service principal. For more details, see Service principals for CI/CD.
• Jenkins can access Databricks APIs using a Databricks service principal. For more details, see CI/CD with Jenkins on Databricks.

Databricks configuration profiles

A Databricks configuration profile contains settings and other information that Databricks needs to authorize access. Databricks configuration profiles are stored in local client files for your tools, SDKs, scripts, and apps to use. The standard configuration profile file is named .databrickscfg.

For more information, see Databricks configuration profiles.