Authentication using OAuth for service principals

Preview

To authenticate to Databricks REST APIs, you can use Databricks OAuth for service principals. A service principal is an identity that you create in Databricks for use with automated tools, jobs, and applications. For more information, see Manage service principals.

You can use OAuth for service principals against Databricks APIs in the account and workspaces. The service principal can only call APIs that it has access to call. The service principal must be an account admin to call account-level APIs and it must be a member of a workspace to call workspace-level APIs.

Requirements

You must be an account admin to manage OAuth credentials for service principals.

Step 1: Create a service principal

You can create a service principal directly in the account or from a workspace. Service principals created in workspaces are automatically added to the account. If you have identity federation enabled on your workspaces, Databricks recommends creating the service principal in the account and assigning it to workspaces. If you do not have identity federation enabled and you want to use the service principal at the workspace-level, you must create your service principal from a workspace.

To add a service principal to the account using the account console:

As an account admin, log in to the account console.
Click User management.
On the Service principals tab, click Add service principal.
Enter a name for the service principal.
Click Add.
Optionally, on the Roles tab, turn on Account admin to call account-level APIs.

You can now assign your service principal to identity federated workspaces.

In the account console sidebar, click Workspaces.
Click your workspace name.
On the Permissions tab, click Add permissions.
Search for and select the service principal, assign the permission level (workspace User or Admin), and click Save.

As a workspace admin, log in to the Databricks workspace.
Click your username in the top bar of the Databricks workspace and select Admin Settings.
On the Service principals tab, click Add service principal.
Click the drop-down arrow in the search box and then click + Add new service principal.
Enter a name for the service principal.
Click Add.

The service principal is added to both your workspace and the Databricks account. In the following step, you must use the account console to create an OAuth secret for your service principal.

Step 2: Create an OAuth secret for a service principal

Before you can use OAuth to authenticate to Databricks REST APIs, you must first create an OAuth secret, which can be used to generate OAuth tokens. A service principal can have up to five OAuth secrets. To create an OAuth secret for a service principal by using the account console:

As an account admin, log in to the account console.
Click User management.
On the Service principals tab, select your service principal.
Under OAuth secrets, click Generate secret.
Copy the displayed Secret and Client ID, and then click Done.

The secret will only be revealed once during creation. The client ID is the same as the service principal’s application ID.

Step 3: Create an OAuth access token for Databricks APIs

You can use the client ID and client secret to request an OAuth token to authenticate to both account-level APIs and workspace-level APIs. The access token will expire in one hour. You must request a new OAuth access token after the expiration. The scope of the OAuth access tokens depends on the level that you create the token from. You can create a token at either the account-level or the workspace-level.

If you are calling account-level APIs and APIs within workspaces that the service principal has access to, you can manually create an OAuth access token at the account-level. If you are calling APIs within one workspace, you can manually create an OAuth access token at the workspace-level.

Important

Instead of manually creating an OAuth access token, you can use your client ID and client secret in participating Databricks tools or SDKs that implement the Databricks client unified authentication standard, and skip the rest of this article. These tools and SDKs automatically generate and replace expired OAuth access tokens for Databricks service principals for you. For more information see OAuth machine-to-machine (M2M) authentication.

Create an OAuth access token at the account-level

An OAuth token created from the account-level can be used to against Databricks APIs in the account and in any workspaces the service principal has been assigned to.

As an account admin, log in to the account console.
Click the down arrow next to your username in the upper right corner.
Copy your Account ID.
Construct the token endpoint URL by replacing <my-account-id> in the following URL with the account ID that you copied.
```
https://accounts.cloud.databricks.com/oidc/accounts/<my-account-id>/v1/token
```
Request an OAuth access token with the token endpoint URL, the Client ID, and the OAuth Secret you created. The all-apis scope requests an OAuth access token that can be used to access all Databricks APIs that the service principal has been granted access to.
- Replace <token-endpoint-URL> with the token endpoint URL from above.
- Replace <client-id> with the service principal’s client ID, which is also known as an application ID.
- Replace <client-secret> with the OAuth secret that you created.
```
export CLIENT_ID=<client-id>
export CLIENT_SECRET=<client-secret>

curl --request POST \
--url <token-endpoint-URL> \
-u "$CLIENT_ID:$CLIENT_SECRET" \
--data 'grant_type=client_credentials&scope=all-apis'
```
This generates a response similar to:
```
{
  "access_token": "eyJraWQiOiJkYTA4ZTVjZ…",
  "token_type": "Bearer",
  "expires_in": 3600
}
```
Copy the access_token from the response.

The access token will expire in one hour. You must request a new OAuth access token after the expiration.

Create an OAuth access token at the workspace-level

An OAuth token created from the workspace-level can only access APIs in that workspace, even if the service principal is an account admin or is a member of other workspaces.

Construct the token endpoint URL by replacing <databricks-instance> with the workspace URL of your Databricks deployment:
```
https://<databricks-instance>/oidc/v1/token
```
Request an OAuth access token with the token endpoint URL, the Client ID, and the OAuth Secret you created. The all-apis scope requests an OAuth access token that can be used to access all Databricks APIs that the service principal has been granted access to within the workspace that you are requesting the token from.
- Replace <token-endpoint-URL> with the token endpoint URL from above.
- Replace <client-id> with the service principal’s client ID, which is also known as an application ID.
- Replace <client-secret> with the OAuth secret that you created.
```
export CLIENT_ID=<client-id>
export CLIENT_SECRET=<client-secret>

curl --request POST \
--url <token-endpoint-URL> \
-u "$CLIENT_ID:$CLIENT_SECRET" \
--data 'grant_type=client_credentials&scope=all-apis'
```
This generates a response similar to:
```
{
  "access_token": "eyJraWQiOiJkYTA4ZTVjZ…",
  "token_type": "Bearer",
  "expires_in": 3600
}
```
Copy the access_token from the response.

The access token will expire in one hour. You must request a new OAuth access token after the expiration.

Step 4: Call a Databricks API

Note

Instead of using curl in this step to call Databricks account-level APIs and workspace-level APIs, you can use your client ID and client secret with participating Databricks tools or SDKs that implement the Databricks client unified authentication standard to call these APIs, and skip the rest of this article. These tools and SDKs automatically generate and replace expired OAuth tokens for Databricks service principals for you. For more information see OAuth machine-to-machine (M2M) authentication.

You can now use the OAuth token to authenticate to Databricks account-level APIs and workspace-level APIs. The service principal must be an account admin to call account-level APIs.

You can include the token in the header using Bearer authentication. You can use this approach with curl or any client that you build.

Example account-level API request

This example uses Bearer authentication to get a list of all workspaces associated with an account.

Replace <oauth-access-token> with the service principal’s OAuth token that you copied in the previous step.
Replace <account-id> with your account ID.

export OAUTH_TOKEN=<oauth-access-token>

curl -X GET --header "Authorization: Bearer $OAUTH_TOKEN" \
  'https://accounts.cloud.databricks.com/api/2.0/accounts/<account-id>/workspaces'

Example workspace-level API request

This example uses Bearer authentication to list all available clusters in the specified workspace.

Replace <oauth-access-token> with the service principal’s OAuth token that you copied in the previous step.
Replace <workspace-URL> with your base workspace URL, which has the form similar to dbc-a1b2345c-d6e7.cloud.databricks.com.

export OAUTH_TOKEN=<oauth-access-token>

curl -X GET --header "Authorization: Bearer $OAUTH_TOKEN" \
`https://<workspace-URL>/api/2.0/clusters/list`

Delete an OAuth secret for a service principal

Account admins can delete OAuth secrets for service principals using the account console. Once a secret is deleted, you cannot use it to request new OAuth access tokens. However existing access tokens generated from the secret continue to work until the token’s expiration.

As an account admin, log in to the account console.
Click User management.
On the Service principals tab, select a service principal.
Under OAuth secrets, click on the trash can icon next to the secret that you want to revoke.

Use OAuth with Terraform

You can use the OAuth to authenticate to the Databricks Terraform provider to automate both account-level and workspace-level configurations. To authenticate to Terraform, set the service principal’s client ID (also known as the application ID) and client secret as environment variables.

For information on how to authenticate to terraform, see Databricks Terraform provider.

For a tutorial on creating a Databricks workspace with Terraform and OAuth, see Create Databricks workspaces using Terraform.