Skip to main content

Authorize service principal access to Databricks with OAuth

This article explains how to authorize access to Databricks resources from unattended processes, such as automated CLI commands or REST API calls made from scripts or applications.

Databricks uses OAuth 2.0 as the preferred protocol for service principal authorization and authentication outside of the UI. Unified client authentication automates token generation and refresh. When a service principal signs in and is granted consent, OAuth issues an access token for the CLI, SDK, or other tool to use on its behalf. Each access token is valid for one hour, after which a new token is automatically requested.

In this article, authorization refers to using OAuth to grant a service principal access to Databricks resources, while authentication refers to validating credentials through access tokens.

For more high-level details, see Authorizing access to Databricks resources.

Ways to authorize a service principal​

Databricks supports two ways to authorize a service principal:

  • Automatic (recommended): Use unified authentication with supported tools and SDKs, such as the Databricks Terraform SDK. This approach handles token generation and refresh automatically, and is ideal for automation or other unattended workloads.

  • Manual: Generate a code verifier and challenge, then exchange them for an OAuth token. Use this method if your tool or API doesn’t support unified authentication. You may need to build your own token refresh mechanism for your application. For details, see Manually generate OAuth M2M access tokens.

Prerequisites​

Before you configure OAuth, perform the following steps:

  1. Create a Databricks service principal. See Add service principals to your account.
  2. Go to the Configuration tab for the service principal and select the entitlements it should have for this workspace.
  3. Go to the Permissions tab and grant access to any Databricks users, service principals, and groups that you want to manage and use this service principal. See Who can manage and use service principals?.

Step 1: Create an OAuth secret​

To authorize access to your Databricks resources with OAuth, you must create an OAuth secret. The secret is used to generate OAuth access tokens for authentication. A service principal can have up to five OAuth secrets, and each secret can be valid for up to two years.

Account admins and workspace admins can create an OAuth secret for a service principal.

  1. In the service principal’s details page, open the Secrets tab.
  2. Under OAuth secrets, click Generate secret.
  3. Set the secret’s lifetime in days (maximum 730 days).
  4. Copy the displayed secret and client ID, then click Done. The secret is shown only once. The client ID is the same as the service principal’s application ID.

Account admins can also create an OAuth secret from the account console. From the User management tab, select the service principal, then go to the Credentials & secrets tab.

note

To enable the service principal to use clusters or SQL warehouses, you must give the service principal access to them. See Compute permissions or Manage a SQL warehouse.

Step 2: Use OAuth authorization​

To use OAuth authorization with the unified authentication tool, you must set the following associated environment variables, .databrickscfg fields, Terraform fields, or Config fields:

  • The Databricks host, specified as https://accounts.cloud.databricks.com for account operations or the target workspace URL, for example https://dbc-a1b2345c-d6e7.cloud.databricks.com for workspace operations.
  • The Databricks account ID, for Databricks account operations.
  • The service principal client ID.
  • The service principal secret.

To perform OAuth service principal authentication, integrate the following within your code, based on the participating tool or SDK:

To use environment variables for a specific Databricks authentication type with a tool or SDK, see Authorizing access to Databricks resources or the tool's or SDK's documentation. See also Environment variables and fields for unified authentication and the Authentication method priority.

For account-level operations, set the following environment variables:

  • DATABRICKS_HOST, set to the value of your Databricks account console URL, https://accounts.cloud.databricks.com.
  • DATABRICKS_ACCOUNT_ID
  • DATABRICKS_CLIENT_ID
  • DATABRICKS_CLIENT_SECRET

For workspace-level operations, set the following environment variables:

  • DATABRICKS_HOST, set to the value of your Databricks workspace URL, for example https://dbc-a1b2345c-d6e7.cloud.databricks.com.
  • DATABRICKS_CLIENT_ID
  • DATABRICKS_CLIENT_SECRET

Manually generate OAuth M2M access tokens​

This section is for tools or services that don't support Databricks unified authentication. If you need to manually generate, refresh, or use Databricks OAuth tokens for M2M authentication, follow these steps.

To generate an OAuth M2M access token, use the service principal’s client ID and OAuth secret. Each access token is valid for one hour. After it expires, request a new token. You can generate tokens at either the account or workspace level:

Generate an account-level access token​

Use an account-level token to call REST APIs for the account and any workspaces the service principal can access.

  1. Locate your account ID.

  2. Construct the token endpoint URL by replacing <account-id> in the following URL with your account ID.

    https://accounts.cloud.databricks.com/oidc/accounts/<my-account-id>/v1/token

  3. Use curl to request an OAuth access token. Replace:

    • <token-endpoint-URL> with the URL above.
    • <client-id> with the service principal’s client ID (application ID).
    • <client-secret> with the service principal’s OAuth secret.
    Bash
    export CLIENT_ID=<client-id>
    export CLIENT_SECRET=<client-secret>

      curl --request POST \
      --url <token-endpoint-URL> \
      --user "$CLIENT_ID:$CLIENT_SECRET" \
      --data 'grant_type=client_credentials&scope=all-apis'

    This generates a response similar to:

    JSON
    {
      "access_token": "eyJraWQiOiJkYTA4ZTVjZ…",
      "token_type": "Bearer",
      "expires_in": 3600
    }

    The all-apis scope requests an OAuth access token that allows the service principal to call any Databricks REST API it has permission to access.

  4. Copy the access_token value from the response.

Generate a workspace-level access token​

Use a workspace-level token only with REST APIs in that workspace.

  1. Construct the token endpoint URL by replacing <databricks-instance> with your <databricks-instance> with the Databricks workspace instance name, for example dbc-a1b2345c-d6e7.cloud.databricks.com:

    https://<databricks-instance>/oidc/v1/token

  2. Use curl to request an OAuth access token. Replace:

    • <token-endpoint-URL> with the URL above.
    • <client-id> with the service principal’s client ID (application ID).
    • <client-secret> with the service principal’s OAuth secret.
    Bash
    export CLIENT_ID=<client-id>
    export CLIENT_SECRET=<client-secret>

    curl --request POST \
    --url <token-endpoint-URL> \
    --user "$CLIENT_ID:$CLIENT_SECRET" \
    --data 'grant_type=client_credentials&scope=all-apis'

    This generates a response similar to:

    JSON
    {
      "access_token": "eyJraWQiOiJkYTA4ZTVjZ…",
      "token_type": "Bearer",
      "expires_in": 3600
    }

  3. Copy the access_token value from the response.

note

To generate a token for a serving endpoint, include the endpoint ID and action in your request. See Fetch an OAuth token manually.

Call a Databricks REST API​

Use the OAuth access token to call account-level or workspace-level REST APIs. To call account-level APIs, the service principal must be an account admin.

Include the token in the authorization header with Bearer authentication.

Example account-level REST API request​

This example lists all workspaces for an account. Replace:

  • <oauth-access-token> with the service principal’s OAuth access token.
  • <account-id> with your account ID.
Bash
export OAUTH_TOKEN=<oauth-access-token>

curl --request GET --header "Authorization: Bearer $OAUTH_TOKEN" \
'https://accounts.cloud.databricks.com/api/2.0/accounts/<account-id>/workspaces'

Example workspace-level REST API request​

This example lists all available clusters in a workspace. Replace:

  • <oauth-access-token> with the service principal’s OAuth access token.
  • <databricks-instance> with the Databricks workspace instance name, for example dbc-a1b2345c-d6e7.cloud.databricks.com.
Bash
export OAUTH_TOKEN=<oauth-access-token>

curl --request GET --header "Authorization: Bearer $OAUTH_TOKEN" \
'https://<workspace-URL>/api/2.0/clusters/list'

Troubleshoot OAuth M2M authentication​

Use these steps to fix the most common issues with Databricks OAuth M2M authentication for service principals.

Quick checks​

Start by verifying these common configuration issues that cause OAuth M2M authentication failures:

  • Credentials: DATABRICKS_CLIENT_ID is set to the service principal's application ID (client ID), and DATABRICKS_CLIENT_SECRET is set to the OAuth secret value, both with no extra spaces.
  • Host: DATABRICKS_HOST points to https://accounts.cloud.databricks.com for account operations or the target workspace URL, for example https://dbc-a1b2345c-d6e7.cloud.databricks.com for workspace operations. Do not include /api.
  • Assignment: The service principal is assigned to the target workspace.
  • Permissions: The service principal has the required permissions on the target resource.
  • Conflicts: No conflicting variables set such as DATABRICKS_TOKEN, DATABRICKS_USERNAME. Run env | grep DATABRICKS and unset conflicts.
  • Tools: Use unified authentication and current CLI or SDK versions.

401 Unauthorized​

Likely causes and fixes:

  • Bad client ID or secret: Re-copy DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET. Regenerate the secret if unsure.
  • Expired secret: Create a new secret if the current one is expired.
  • Wrong token issuer: For M2M, use the Databricks OAuth token endpoint, not your IdP or cloud token endpoint.
  • Host mismatch: If you authenticate for workspace APIs, DATABRICKS_HOST must be the workspace URL you call.

403 Forbidden​

Likely causes and fixes:

  • Missing resource permissions: Grant the service principal CAN USE or CAN MANAGE on clusters or SQL warehouses, and the required object-level permissions for notebooks, jobs, or data objects.
  • No workspace assignment: Assign the service principal to the workspace in the account console.
  • Admin API access: For admin-only APIs, assign the service principal to the workspace admin group or grant account admin permissions.

Configuration problems​

Symptoms include timeouts, “host not found,” “account not found,” or “workspace not found.”

Fixes:

  • Host rules: Use the account console URL for account APIs. Use the workspace URL for workspace APIs. Don't include the /api suffix.
  • Account ID: Supply DATABRICKS_ACCOUNT_ID only for account-level operations. Use the UUID from the account console.
  • Profile selection: If you use multiple profiles, pass --profile <name> or set DATABRICKS_CONFIG_PROFILE.

Connectivity​

If OAuth M2M authentication fails due to network issues, use these tests to verify that your environment can reach Databricks endpoints:

  • DNS: nslookup <your-host> (should return IP addresses for the hostname)
  • TLS and reachability: curl -I https://<your-host> (should return HTTP status 200, 401, or 403)
  • Corporate network: Confirm that proxy or firewall rules allow HTTPS to Databricks endpoints

Additional resources​

Last updated on