Databricks personal access token authentication

Databricks personal access tokens are one of the most well-supported types of credentials for resources and operations at the Databricks workspace level. Many storage mechanisms for credentials and related information, such as environment variables and Databricks configuration profiles, provide support for Databricks personal access tokens. Although users can have multiple personal access tokens in a Databricks workspace, each personal access token works for only a single Databricks workspace. The number of personal access tokens per user is limited to 600 per workspace.

Note

To automate Databricks account-level functionality, you cannot use Databricks personal access tokens. Instead, you must use either OAuth tokens for Databricks account admin users or service principals, or the username and password of Databricks account-level admins. For more information, see:

• Basic authentication (legacy)

• OAuth machine-to-machine (M2M) authentication

• OAuth user-to-machine (U2M) authentication

Databricks personal access tokens for workspace users

To create a Databricks personal access token for your Databricks workspace user, do the following:

1. In your Databricks workspace, click your Databricks username in the top bar, and then select Settings from the drop down.

2. Click Developer.

3. Next to Access tokens, click Manage.

4. Click Generate new token.

5. (Optional) Enter a comment that helps you to identify this token in the future, and change the token’s default lifetime of 90 days. To create a token with no lifetime (not recommended), leave the Lifetime (days) box empty (blank).

6. Click Generate.

7. Copy the displayed token to a secure location, and then click Done.

Note

Be sure to save the copied token in a secure location. Do not share your copied token with others. If you lose the copied token, you cannot regenerate that exact same token. Instead, you must repeat this procedure to create a new token. If you lose the copied token, or you believe that the token has been compromised, Databricks strongly recommends that you immediately delete that token from your workspace by clicking the trash can (Revoke) icon next to the token on the Access tokens page.

If you are not able to create or use tokens in your workspace, this might be because your workspace administrator has disabled tokens or has not given you permission to create or use tokens. See your workspace administrator or the following:

• Enable or disable personal access token authentication for the workspace

• Personal access token permissions

Databricks personal access tokens for service principals

A workspace admin can create a Databricks personal access tokens on behalf of a service principal, as follows:

Note

You cannot use the Databricks user interface to generate Databricks personal access tokens for Databricks service principals. This process uses Databricks CLI version 0.205 or above to generate an access token for a Databricks service principal. If you do not already have the Databricks CLI installed, see Install or update the Databricks CLI.

1. Set up authentication for the Databricks CLI, if you have not done so already. One way to set this up is by using Databricks personal access token authentication for your Databricks workspace user first. See Databricks personal access token authentication.

2. Get the application ID for the Databricks service principal, if you do not already have it available:

   1. If the admin console for your workspace is not already open, click your username in the workspace’s top bar and click Settings.

   2. Under Workspace admin, click Identity and access.

   3. Next to Service principals, click Manage.

   4. Click the name of the Databricks service principal to open its settings page. If the name is not visible, use Filter service principals to find it.

   5. On the Configurations tab, note the Application Id value.

3. Use the Databricks CLI to run the following command, which generates the access token for the Databricks service principal.

   In the following command, replace these placeholders:

   • Replace <application-id> with the application ID of the Databricks service principal.

   • Replace <lifetime-seconds> with the number of seconds that the access token is valid for. For example, 1 day is 86400 seconds.

   • Optionally, replace <comment> with any meaningful comment about the access token’s purpose. If the --comment option is not specified, then no comment is generated.

   • Optionally, replace <profile-name> with the name of a Databricks configuration profile that contains authentication information for your Databricks user and target workspace. For instance, see Databricks personal access token authentication. If the -p option is not specified, the Databricks CLI will attempt to find and use a configuration profile named DEFAULT.

   databricks token-management create-obo-token <application-id> --lifetime-seconds <lifetime-seconds> --comment <comment> -p <profile-name>
   

4. In the response, copy the value of token_value, which is the access token for your Databricks service principal.

   Be sure to save the copied token in a secure location. Do not share your copied token with others. If you lose the copied token, you cannot regenerate that exact same token. Instead, you must repeat this procedure to create a new token.

   If you are not able to create or use tokens in your workspace, this might be because your workspace administrator has disabled tokens or has not given you permission to create or use tokens. See your workspace administrator or the following:

   • Enable or disable personal access token authentication for the workspace

   • Personal access token permissions

A service principal can then use its own Databricks personal access token to create additional Databricks personal access tokens for itself, as follows:

Note

You cannot use the Databricks user interface to generate Databricks personal access tokens for Databricks service principals. This process uses Databricks CLI version 0.205 or above to generate an access token for a Databricks service principal. If you do not already have the Databricks CLI installed, see Install or update the Databricks CLI.

This procedure assumes that you have already generated the first Databricks personal access token for the Databricks service principal. You use this access token to set up the Databricks CLI to authenticate the Databricks service principal so that it can then generate additional access tokens for itself. See Databricks personal access token authentication.

1. Use the Databricks CLI to run the following command, which generates another access token for the Databricks service principal.

   In the following command, replace these placeholders:

   • Optionally, replace <comment> with any meaningful comment about the access token’s purpose. If the --comment option is not specified, then no comment is generated.

   • Optionally, replace <lifetime-seconds> with the number of seconds that the access token is valid for. For example, 1 day is 86400 seconds. If the --lifetime-seconds option is not specified, the access token is set to never expire (not recommended).

   • Optionally, replace <profile-name> with the name of a Databricks configuration profile that contains authentication information for the Databricks service principal and the target workspace. If the -p option is not specified, the Databricks CLI will attempt to find and use a configuration profile named DEFAULT.

   databricks tokens create --comment <comment> --lifetime-seconds <lifetime-seconds> -p <profile-name>
   

2. In the response, copy the value of token_value, which is the access token for the Databricks service principal.

   Be sure to save the copied token in a secure location. Do not share your copied token with others. If you lose the copied token, you cannot regenerate that exact same token. Instead, you must repeat this procedure to create a new token.

   If you are not able to create or use tokens in your workspace, this might be because your workspace administrator has disabled tokens or has not given you permission to create or use tokens. See your workspace administrator or the following:

   • Enable or disable personal access token authentication for the workspace

   • Personal access token permissions

Perform Databricks personal access token authentication

To configure Databricks personal access token authentication, you must set the following associated environment variables, .databrickscfg fields, Terraform fields, or Config fields:

• The Databricks host, specified as the target Databricks workspace URL, for example https://dbc-a1b2345c-d6e7.cloud.databricks.com.

• The Databricks personal access token, for the Databricks user account or Databricks service principal.

To perform Databricks personal access token authentication, integrate the following within your code, based on the participating tool or SDK:

To use environment variables for a specific Databricks authentication type with a tool or SDK, see Supported authentication types by Databricks tool or SDK or the tool’s or SDK’s documentation. See also Environment variables and fields for client unified authentication and the Default order of evaluation for client unified authentication methods and credentials.

Set the following environment variables:

• DATABRICKS_HOST, set to the Databricks workspace URL, for example https://dbc-a1b2345c-d6e7.cloud.databricks.com.

• DATABRICKS_TOKEN

Create or identify a Databricks configuration profile with the following fields in your .databrickscfg file. If you create the profile, replace the placeholders with the appropriate values. To use the profile with a tool or SDK, see Supported authentication types by Databricks tool or SDK or the tool’s or SDK’s documentation. See also Environment variables and fields for client unified authentication and the Default order of evaluation for client unified authentication methods and credentials.

Set the following values in your .databrickscfg file. In this case, the host is the Databricks workspace URL, for example https://dbc-a1b2345c-d6e7.cloud.databricks.com:

[<some-unique-configuration-profile-name>]
host  = <workspace-url>
token = <token>

Instead of manually setting the preceding values in your .databrickscfg file, you can use the Databricks CLI to set these values instead, as follows:

Note

The following procedure uses the Databricks CLI to create a Databricks configuration profile with the name DEFAULT. If you already have a DEFAULT configuration profile, this procedure overwrites your existing DEFAULT configuration profile.

To check whether you already have a DEFAULT configuration profile, and to view this profile’s settings if it exists, use the Databricks CLI to run the command databricks auth env --profile DEFAULT.

To create a configuration profile with a name other than DEFAULT, replace the DEFAULT part of --profile DEFAULT in the following databricks configure command with a different name for the configuration profile.

1. Use the Databricks CLI to create a Databricks configuration profile named DEFAULT that uses Databricks personal access token authentication. To do this, run the following command:

   databricks configure --profile DEFAULT
   

2. For the prompt Databricks Host, enter your Databricks workspace instance URL, for example https://dbc-a1b2345c-d6e7.cloud.databricks.com.

3. For the prompt Personal Access Token, enter the Databricks personal access token for your workspace.

For the Databricks CLI, run the databricks configure command. At the prompts, enter the following settings:

• The Databricks host, specified as the target Databricks workspace URL, for example https://dbc-a1b2345c-d6e7.cloud.databricks.com.

• The Databricks personal access token, for the Databricks user account or Databricks service principal.

For more details, see Databricks personal access token authentication.

Note

Databricks personal access token authentication is supported on the following Databricks Connect versions:

• For Python, Databricks Connect for Databricks Runtime 13.3 LTS and above.

• For Scala, Databricks Connect for Databricks Runtime 13.3 LTS and above.

For Databricks Connect, you can use the Databricks CLI to set the values in your .databrickscfg file, for Databricks workspace-level operations as specified in this article’s “Profile” section, as follows:

Note

The following procedure uses the Databricks CLI to create a Databricks configuration profile with the name DEFAULT. If you already have a DEFAULT configuration profile, this procedure overwrites your existing DEFAULT configuration profile.

To check whether you already have a DEFAULT configuration profile, and to view this profile’s settings if it exists, use the Databricks CLI to run the command databricks auth env --profile DEFAULT.

To create a configuration profile with a name other than DEFAULT, replace the DEFAULT part of --profile DEFAULT in the databricks configure command as shown in the following step with a different name for the configuration profile.

1. Use the Databricks CLI to create a Databricks configuration profile named DEFAULT that uses Databricks personal access token authentication. To do this, run the following command:

   databricks configure --configure-cluster --profile DEFAULT
   

2. For the prompt Databricks Host, enter your Databricks workspace instance URL, for example https://dbc-a1b2345c-d6e7.cloud.databricks.com.

3. For the prompt Personal Access Token, enter the Databricks personal access token for your workspace.

4. In the list of available clusters that appears, use your up arrow and down arrow keys to select the target Databricks cluster in your workspace, and then press Enter. You can also type any part of the cluster’s display name to filter the list of available clusters.

Other supported approaches for Databricks Connect include the following:

• Manually set the values in your .databrickscfg file for Databricks workspace-level operations as specified in this article’s “Profile” section. Also set the cluster_id environment variable in your profile to your workspace instance URL, for example https://dbc-a1b2345c-d6e7.cloud.databricks.com.

• Set the environment variables for Databricks workspace-level operations as specified in this article’s “Environment” section. Also set the DATABRICKS_CLUSTER_ID environment variable to your workspace instance URL, for example https://dbc-a1b2345c-d6e7.cloud.databricks.com.

Values in your .databrickscfg file always take precedence over environment variables.

To initialize the Databricks Connect client with these environment variables or values in your .databrickscfg file, see one of the following:

• For Python, see Configure connection properties for Python.

• For Scala, see Configure connection properties for Scala.

For the Databricks extension for Visual Studio Code, do the following:

1. Set the values in your .databrickscfg file for Databricks workspace-level operations as specified in this article’s “Profile” section.

2. In the Configuration pane of the Databricks extension for Visual Studio Code, click Configure Databricks.

3. In the Command Palette, for Databricks Host, enter your workspace URL, for example https://dbc-a1b2345c-d6e7.cloud.databricks.com, and then press Enter.

4. In the Command Palette, select your target profile’s name in the list for your URL.

For more details, see Authentication setup for the Databricks extension for Visual Studio Code.

For default authentication:

provider "databricks" {
  alias = "workspace"
}

For direct configuration (replace the retrieve placeholders with your own implementation to retrieve the values from the console or some other configuration store, such as HashiCorp Vault. See also Vault Provider). In this case, the host is the Databricks workspace URL, for example https://dbc-a1b2345c-d6e7.cloud.databricks.com:

provider "databricks" {
  alias = "workspace"
  host  = <retrieve-workspace-url>
  token = <retrieve-token>
}

For more information about authenticating with the Databricks Terraform provider, see Authentication.

For default authentication:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
# ...

For direct configuration (replace the retrieve placeholders with your own implementation to retrieve the values from the console or some other configuration store, such as AWS Systems Manager Parameter Store). In this case, the host is the Databricks workspace URL, for example https://dbc-a1b2345c-d6e7.cloud.databricks.com:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient(
  host  = retrieve_workspace_url(),
  token = retrieve_token()
)
# ...

For more information about authenticating with Databricks tools and SDKs that use Python and that implement Databricks client unified authentication, see:

• Set up the Databricks Connect client for Python

• Authentication setup for the Databricks extension for Visual Studio Code

• Authenticate the Databricks SDK for Python with your Databricks account or workspace

For default authentication:

import com.databricks.sdk.WorkspaceClient;
// ...
WorkspaceClient w = new WorkspaceClient();
// ...

For direct configuration (replace the retrieve placeholders with your own implementation to retrieve the values from the console or some other configuration store, such as AWS Systems Manager Parameter Store). In this case, the host is the Databricks workspace URL, for example https://dbc-a1b2345c-d6e7.cloud.databricks.com:

import com.databricks.sdk.WorkspaceClient;
import com.databricks.sdk.core.DatabricksConfig;
// ...
DatabricksConfig cfg = new DatabricksConfig()
  .setHost(retrieveWorkspaceUrl())
  .setToken(retrieveToken());
WorkspaceClient w = new WorkspaceClient(cfg);
// ...

For more information about authenticating with Databricks tools and SDKs that use Java and that implement Databricks client unified authentication, see:

• Set up the Databricks Connect client for Scala (the Databricks Connect client for Scala uses the included Databricks SDK for Java for authentication)

• Authenticate the Databricks SDK for Java with your Databricks account or workspace

For default authentication:

import (
  "github.com/databricks/databricks-sdk-go"
)
// ...
w := databricks.Must(databricks.NewWorkspaceClient())
// ...

For direct configuration (replace the retrieve placeholders with your own implementation to retrieve the values from the console or some other configuration store, such as AWS Systems Manager Parameter Store). In this case, the host is the Databricks workspace URL, for example https://dbc-a1b2345c-d6e7.cloud.databricks.com:

import (
  "github.com/databricks/databricks-sdk-go"
)
// ...
w := databricks.Must(databricks.NewWorkspaceClient(&databricks.Config{
  Host:  retrieveWorkspaceUrl(),
  Token: retrieveToken(),
}))
// ...

For more information about authenticating with Databricks tools and SDKs that use Go and that implement Databricks client unified authentication, see Authenticate the Databricks SDK for Go with your Databricks account or workspace.