Authentication using OAuth tokens for service principals

Preview

To authenticate to Databricks REST APIs, you can use Databricks OAuth tokens for service principals. A service principal is an identity that you create in Databricks for use with automated tools, jobs, and applications. For more information, see Manage service principals.

Note

An account admin’s username and password can also be used to authenticate. However, Databricks strongly recommends that you use OAuth for service principals.

Requirements

You must be an account admin to create and manage OAuth tokens for service principals

Step 1: Create a service principal in your account

Account admins can create service principals directly in the account by using the account console or _.

Workspace admins can also create service principals using the workspace admin console (if the workspace is enabled for identity federation) or _. Service principals created in workspaces are automatically synced to the account. For more information, see How does Databricks sync identities between workspaces and the account?.

To add a service principal to the account using the account console:

As an account admin, log in to the account console.
Click User management.
On the Service principals tab, click Add service principal.
Enter a name for the service principal.
Click Add.

Step 2: Create an OAuth secret for a service principal

Before you can use OAuth to authenticate to Databricks REST APIs, you must first create an OAuth secret, which can be used to generate OAuth tokens. A service principal can have up to five OAuth secrets. To create an OAuth secret for a service principal using the account console:

As an account admin, log in to the account console.
Click User management.
On the Service principals tab, select a service principal.
Under OAuth secrets, click Generate secret.
Copy the displayed Secret and Client ID, and then click Done.

The secret will only be revealed once during creation. The client ID is the same as the service principal’s application ID.

To create an OAuth secret using the API see, the Account API.

Step 3: Grant the service principal access to call Databricks APIs

You can use the client ID and client secret against Databricks APIs in the account and in any workspaces the service principal has been assigned to. The service principal can only call APIs that it has access to call. The instructions differ depending on whether you are granting access to account-level APIs or workspace-level APIs.

Account-level APIs

Databricks account-level APIs (such the Account API) can be called only by account admins. These APIs are hosted on the account console URL: https://accounts.cloud.databricks.com/.

Your service principal must be an account admin in order to authenticate to account-level APIs using OAuth tokens. You can grant your service principal the account admin role, following Assign account admin rights to a service principal.

Workspace-level APIs

Databricks workspace-level APIs can be called only by workspace admins and workspace users. These APIs are hosted on the workspace URLs, such as https://my-workspace.cloud.databricks.com

Your service principal must be a workspace admin or a workspace user to authenticate workspace-level API requests, depending on the specific API. You can grant your service principal workspace access, following Add a service principal to a workspace.

You can optionally grant your service principal the workspace admin role using the instructions in Assign workspace admin rights to a service principal and Databricks access control permissions using the instructions in Access control.

Step 4: Create an OAuth access token for Databricks APIs

You can now use the client ID and client secret to request an OAuth token to authenticate to both account-level APIs and workspace-level APIs. The scope of the OAuth access tokens depends on the level that you create the token from. You can create a token at either the account-level or the workspace-level.

If you are calling account-level APIs and APIs within workspaces that the service principal has access to, you must create an OAuth access token at the account-level. If you are calling APIs within one workspace, Databricks recommends that you create an OAuth access token at the workspace-level.

Create an OAuth access token at the account-level

An OAuth token created from the account-level can be used to against Databricks APIs in the account and in any workspaces the service principal has been assigned to.

As an account admin, log in to the account console.
Click the down arrow next to your username in the upper right corner.
Copy your Account ID.
Construct the token endpoint URL by replacing <my-account-id> in the following URL with the account ID that you copied.
```
https://accounts.cloud.databricks.com/oidc/accounts/<my-account-id>/v1/token
```
Request an OAuth access token with the token endpoint URL, the Client ID, and the OAuth Secret you created. The all-apis scope requests an OAuth access token that can be used to access all Databricks APIs that the service principal has been granted access to.
- Replace <token-endpoint-URL> with the token endpoint URL from above.
- Replace <client-id> with the service principal’s client ID, which is also known as an application ID.
- Replace <client-secret> with the OAuth secret that you created.
```
export CLIENT_ID<client-id>
export CLIENT_SECRET=<client-secret>

curl --request POST \
--url <token-endpoint-URL> \
-u "$CLIENT_ID:$CLIENT_SECRET" \
--data 'grant_type=client_credentials&scope=all-apis'
```
This generates a response similar to:
```
{
  "access_token": "eyJraWQiOiJkYTA4ZTVjZ…",
  "token_type": "Bearer",
  "expires_in": 3600
}
```
Copy the access_token from the response.

The access token will expire in one hour. You must request a new OAuth access token after the expiration.

Create an OAuth access token at the workspace-level

An OAuth token created from the workspace-level can only access APIs in that workspace, even if the service principal is an account admin or is a member of other workspaces.

Construct the token endpoint URL by replacing <databricks-instance> with the workspace URL of your Databricks deployment:
```
https://<databricks-instance>/oidc/v1/token
```
Request an OAuth access token with the token endpoint URL, the Client ID, and the OAuth Secret you created. The all-apis scope requests an OAuth access token that can be used to access all Databricks APIs that the service principal has been granted access to within the workspace that you are requesting the token from.
- Replace <token-endpoint-URL> with the token endpoint URL from above.
- Replace <client-id> with the service principal’s client ID, which is also known as an application ID.
- Replace <client-secret> with the OAuth secret that you created.
```
export CLIENT_ID<client-id>
export CLIENT_SECRET=<client-secret>

curl --request POST \
--url <token-endpoint-URL> \
-u "$CLIENT_ID:$CLIENT_SECRET" \
--data 'grant_type=client_credentials&scope=all-apis'
```
This generates a response similar to:
```
{
  "access_token": "eyJraWQiOiJkYTA4ZTVjZ…",
  "token_type": "Bearer",
  "expires_in": 3600
}
```
Copy the access_token from the response.

The access token will expire in one hour. You must request a new OAuth access token after the expiration.

Step 5: Call a Databricks API

You can now use the OAuth token to authenticate to Databricks account-level APIs and workspace-level APIs.

You can include the token in the header using Bearer authentication. You can use this approach with curl or any client that you build.

Example account-level API request

To grant the service principal access to call account-level APIs, see Account-level APIs. This example uses Bearer authentication to get a list of all workspaces associated with an account.

Replace <oauth-access-token> with the service principal’s OAuth token that you copied in the previous step.
Replace <account-id> with your account ID.

export OAUTH_TOKEN=<oauth-access-token>

curl -X GET --header "Authorization: Bearer $OAUTH_TOKEN" \
  'https://accounts.cloud.databricks.com/api/2.0/accounts/<account-id>/workspaces'

Example workspace-level API request

To grant the service principal access to call workspace-level APIs, see Workspace-level APIs. This example uses Bearer authentication to list all available clusters in the specified workspace.

Replace <oauth-access-token> with the service principal’s OAuth token that you copied in the previous step.
Replace <workspace-URL> with your base workspace URL, which has the form similar to dbc-a1b2345c-d6e7.cloud.databricks.com.

export OAUTH_TOKEN=<oauth-access-token>

curl -X GET --header "Authorization: Bearer $OAUTH_TOKEN" \
`https://<workspace-URL>/api/2.0/clusters/list`

Delete an OAuth secret for a service principal

Account admins can delete OAuth secrets for service principals using the account console. Once a secret is deleted, you cannot use it to request new OAuth access tokens. However existing access tokens generated from the secret continue to work until the token’s expiration.

As an account admin, log in to the account console.
Click User management.
On the Service principals tab, select a service principal.
Under OAuth secrets, click on the trash can icon next to the secret that you want to revoke.

Use OAuth with Terraform

You can use the OAuth to authenticate to the Databricks Terrafrom provider to automate both account-level and workspace-level configurations. To authenticate to Terraform, set the service principal’s client ID (also known as the application ID) and client secret as environment variables.

For information on how to authenticate to terraform, see Databricks Terraform provider.

For a tutorial on creating a Databricks workspace with terraform and OAuth, see Create Databricks workspaces using Terraform.