Databricks CLI reference

Note

This information applies to Databricks CLI versions 0.200 and above, which are in Public Preview. To find your version of the Databricks CLI, run databricks -v.

To learn how to quickly get started with Databricks CLI versions 0.200 and above, see the Databricks CLI tutorial.

The Databricks command-line interface (also known as the Databricks CLI) utility, provided by Databricks, provides an easy-to-use interface to automate the Databricks platform from your terminal, command prompt, or automation scripts.

For information about all of the available commands for Databricks CLI versions 0.200 and above, see CLI command groups.

Databricks CLI versions 0.200 and above are separate from legacy Databricks CLI versions 0.17 and below. For information about legacy Databricks CLI versions 0.17 and below, see Databricks CLI (legacy). To migrate from Databricks CLI version 0.17 or below to Databricks CLI version 0.200 or above, see Databricks CLI migration.

Set up the CLI

The following sections describe how to install and update the Databricks CLI, and how to set up the CLI for Databricks authentication.

Install or update the CLI

This section describes how to install or update your development machine to run the Databricks CLI executable.

Install the CLI

You can install the Databricks CLI manually, by using Homebrew for macOS or Linux, or by using curl.

For this installation option, you manually download a .zip file and then manually extract the Databricks CLI executable from the downloaded .zip file.

  1. Download onto your local development machine the correct Databricks CLI .zip file, as listed in the Releases section of the databricks/cli repository in GitHub. The .zip file must match your development machine’s operating system and architecture:

    Filename

    Architecture

    databricks_cli_X.Y.Z_darwin_amd64.zip

    macOS, Intel 64-bit

    databricks_cli_X.Y.Z_darwin_arm64.zip

    macOS, Apple Silicon

    databricks_cli_X.Y.Z_linux_amd64.zip

    Linux, Intel 64-bit

    databricks_cli_X.Y.Z_linux_arm64.zip

    Linux, ARM 64-bit

    databricks_cli_X.Y.Z_windows_386.zip

    Windows, Intel 32-bit

    databricks_cli_X.Y.Z_windows_amd64.zip

    Windows, Intel 64-bit

    databricks_cli_X.Y.Z_windows_arm64.zip

    Windows, ARM 64-bit

    To get your machine’s architecture, see your operating system’s documentation.

    If you need to verify the integrity of one of these .zip files, Databricks provides a checksum file named databricks_cli_X.Y.Z_SHA256SUMS in the same list as the .zip files. To run a checksum verification, see your operating system’s documentation.

  2. Extract the contents of the downloaded .zip file. To extract the .zip file, see your operating system’s documentation.

  3. In the extracted content, a folder appears with the same name as the .zip file. Inside of this folder is the Databricks CLI executable. You can leave the Databricks CLI executable there, or you can copy or move it to another location.

  4. Confirm whether the Databricks CLI is installed correctly. To do this, view the Databricks CLI executable’s version by using the -v option or by running the version command:

    databricks -v
    
    # Or:
    databricks version
    

    If a version number of 0.200 or above is listed, it means that the Databricks CLI is installed correctly.

    Note

    If you run databricks but get an error such as command not found: databricks, or if you run databricks -v and a version number of 0.17 or below is listed, this means that your machine cannot find the correct version of the Databricks CLI executable. To fix this, either prepend the full path to the target Databricks CLI executable to databricks, or make sure that the full path to the target Databricks CLI executable is referenced in your operating system’s PATH instead. Note that if you have multiple installations of the Databricks CLI, you should adjust your PATH to have the Databricks CLI executable that you use most often listed first in your PATH. This will prevent you from constantly prepending the full path to your most-used Databricks CLI executable. To manage your PATH, see your operating system’s documentation.

For this installation option, you use Homebrew to automatically download and install the Databricks CLI executable.

  1. Check whether Homebrew is already installed by running the following command from a macOS Terminal or Linux shell prompt. If Homebrew is installed, the Homebrew version number is displayed:

    brew -v
    
  2. If Homebrew is not already installed, install it by running the following command:

    /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
    

    Then run the following command to verify the Homebrew installation, which displays the Homebrew version number:

    brew -v
    
  3. Use Homebrew to add the databricks/homebrew-tap repository in GitHub to your list of available Homebrew Tap repositories, by running the following command:

    brew tap databricks/tap
    
  4. Use Homebrew to instruct the databricks/homebrew-tap repository to download and install the Databricks CLI executable, by running the following command:

    brew install databricks
    
  5. Confirm whether the Databricks CLI is installed correctly. To do this, view the Databricks CLI executable’s version by using the -v option or by running the version command:

    databricks -v
    
    # Or:
    databricks version
    

    If a version number of 0.200 or above is listed, it means that the Databricks CLI is installed correctly.

    Note

    If you run databricks but get an error such as command not found: databricks, or if you run databricks -v and a version number of 0.17 or below is listed, this means that your machine cannot find the correct version of the Databricks CLI executable. To fix this, either prepend the full path to the target Databricks CLI executable to databricks, or make sure that the full path to the target Databricks CLI executable is referenced in your operating system’s PATH instead. Note that if you have multiple installations of the Databricks CLI, you should adjust your PATH to have the Databricks CLI executable that you use most often listed first in your PATH. This will prevent you from constantly prepending the full path to your most-used Databricks CLI executable. To manage your PATH, see your operating system’s documentation.

For this installation option, you use curl to automatically download and install the Databricks CLI executable.

  1. curl should already be installed. You can check whether curl is already installed by running the following command from a macOS Terminal, Linux shell prompt, or Windows Command Prompt (note that you must specify an uppercase -V here). If curl is installed, the curl version number is displayed:

    curl -V
    
  2. If curl is not already installed, install it by following the instructions on the curl Releases and Downloads page for your operating system and architecture. To get your machine’s architecture, see your operating system’s documentation.

    Then run the following command to verify the curl installation, which displays the curl version number (note that you must specify an uppercase -V here):

    curl -V
    
  3. Use curl to download and install the Databricks CLI executable by running the following command (note that you must specify the correct uppercase and lowercase characters for -fsSL):

    curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh
    

    This command downloads and installs the Databricks CLI executable in the path /usr/local/bin/databricks on macOS and Linux, and C:\Windows\databricks.exe on Windows.

    If for some reason the Databricks CLI is already installed, the following error appears: “Target path <path> already exists.” To fix this, you must manually delete the Databricks CLI executable from the preceding path, and then run the curl command again.

    Note

    To view the script’s contents before you run it, see the install.sh file in the databricks/setup-cli repository in GitHub.

  4. Confirm whether the Databricks CLI is installed correctly. To do this, view the Databricks CLI executable’s version by using the -v option or by running the version command:

    databricks -v
    
    # Or:
    databricks version
    

    If a version number of 0.200 or above is listed, it means that the Databricks CLI is installed correctly.

    Note

    If you run databricks but get an error such as command not found: databricks, or if you run databricks -v and a version number of 0.17 or below is listed, this means that your machine cannot find the correct version of the Databricks CLI executable. To fix this, either prepend the full path to the target Databricks CLI executable to databricks, or make sure that the full path to the target Databricks CLI executable is referenced in your operating system’s PATH instead. Note that if you have multiple installations of the Databricks CLI, you should adjust your PATH to have the Databricks CLI executable that you use most often listed first in your PATH. This will prevent you from constantly prepending the full path to your most-used Databricks CLI executable. To manage your PATH, see your operating system’s documentation.

Update the CLI

The same methods you can use to install the Databricks CLI are available when updating, including manual installation, Homebrew, and curl. Databricks recommends using the same method you used during installation while updating.

  1. Optionally, delete the Databricks CLI executable, the .zip file, and the .zip file’s extracted folder, from the previous manual installation procedure.

  2. Install the latest version of the Databricks CLI by following the instructions in the previous manual installation procedure.

  3. Confirm whether the Databricks CLI is updated correctly. To do this, view the Databricks CLI executable’s version by using the -v option or by running the version command:

    databricks -v
    
    # Or:
    databricks version
    

    If the expected updated version number displays, then the Databricks CLI is updated correctly.

    Note

    If you run databricks but get an error such as command not found: databricks, or if you run databricks -v and a version number of 0.17 or below is listed, this means that your machine cannot find the correct version of the Databricks CLI executable. To fix this, either prepend the full path to the target Databricks CLI executable to databricks, or make sure that the full path to the target Databricks CLI executable is referenced in your operating system’s PATH instead. Note that if you have multiple installations of the Databricks CLI, you should adjust your PATH to have the Databricks CLI executable that you use most often listed first in your PATH. This will prevent you from constantly prepending the full path to your most-used Databricks CLI executable. To manage your PATH, see your operating system’s documentation.

  1. Use Homebrew to download and install the latest version of the Databricks CLI executable by running the following command:

    brew upgrade databricks
    
  2. Confirm whether the Databricks CLI is updated correctly. To do this, view the Databricks CLI executable’s version by using the -v option or by running the version command:

    databricks -v
    
    # Or:
    databricks version
    

    If the expected updated version number displays, then the Databricks CLI is installed correctly.

    Note

    If you run databricks but get an error such as command not found: databricks, or if you run databricks -v and a version number of 0.17 or below is listed, this means that your machine cannot find the correct version of the Databricks CLI executable. To fix this, either prepend the full path to the target Databricks CLI executable to databricks, or make sure that the full path to the target Databricks CLI executable is referenced in your operating system’s PATH instead. Note that if you have multiple installations of the Databricks CLI, you should adjust your PATH to have the Databricks CLI executable that you use most often listed first in your PATH. This will prevent you from constantly prepending the full path to your most-used Databricks CLI executable. To manage your PATH, see your operating system’s documentation.

    For a list of Databricks CLI versions, see the Releases page in the databricks/cli repository in GitHub.

  1. Delete the existing installation of the Databricks CLI executable from the path /usr/local/bin/databricks on macOS and Linux, or C:\Windows\databricks.exe on Windows.

  2. Use curl to download and install the latest version of the Databricks CLI executable by running the following command (note that you must specify the correct uppercase and lowercase characters for -fsSL):

    curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh
    

    This command downloads and installs the Databricks CLI executable in the path /usr/local/bin/databricks on macOS and Linux, and C:\Windows\databricks.exe on Windows.

    If the following error appears, you must manually delete the Databricks CLI executable from the preceding path, and then run the curl command again: “Target path <path> already exists.”

    Note

    To view the script’s contents before you run it, see the install.sh file in the databricks/setup-cli repository in GitHub.

  3. Confirm whether the Databricks CLI is updated correctly. To do this, view the Databricks CLI executable’s version by using the -v option or by running the version command:

    databricks -v
    
    # Or:
    databricks version
    

    If the expected updated version number displays, then the Databricks CLI is installed correctly.

    Note

    If you run databricks but get an error such as command not found: databricks, or if you run databricks -v and a version number of 0.17 or below is listed, this means that your machine cannot find the correct version of the Databricks CLI executable. To fix this, either prepend the full path to the target Databricks CLI executable to databricks, or make sure that the full path to the target Databricks CLI executable is referenced in your operating system’s PATH instead. Note that if you have multiple installations of the Databricks CLI, you should adjust your PATH to have the Databricks CLI executable that you use most often listed first in your PATH. This will prevent you from constantly prepending the full path to your most-used Databricks CLI executable. To manage your PATH, see your operating system’s documentation.

    For a list of Databricks CLI versions, see the Releases page in the databricks/cli repository in GitHub.

Set up authentication

Before you can run Databricks CLI commands, you must set up authentication between the Databricks CLI and Databricks.

You must authenticate the Databricks CLI to the relevant resources at run time in order to run Databricks automation commands within a Databricks account or workspace. Depending on whether you want to call Databricks workspace-level commands, Databricks account-level commands, or both, you must authenticate to the Databricks workspace, account, or both. For a list of Databricks workspace-level and account-level CLI command groups, run the command databricks -h. For a list of Databricks workspace-level and account-level REST API operations that the Databricks CLI commands cover, see the Databricks REST API.

Note

The Databricks CLI implements the Databricks client unified authentication standard, a consolidated and consistent architecural and programmatic approach to authentication. This approach helps make setting up and automating authentication with Databricks more centralized and predictable. It enables you to configure Databricks authentication once and then use that configuration across multiple Databricks tools and SDKs without further authentication configuration changes. For more information about this standard, see Databricks client unified authentication.

The following sections provide information about authentication between the Databricks CLI and Databricks:

Databricks personal access token authentication

Databricks personal access token authentication uses a Databricks personal access token to authenticate the target Databricks entity, such as a Databricks user account or a Databricks service principal. See also Databricks personal access token authentication.

To create a Databricks personal access token, see Databricks personal access token authentication.

Note

You cannot use Databricks personal access token authentication for authenticating with a Databricks account, as Databricks account-level commands do not use Databricks personal access tokens for authentication. To authenticate with a Databricks account, consider using one of the following authentication types instead:

To configure and use Databricks personal access token authentication, do the following:

  1. Create or identify a Databricks configuration profile with the following fields in your .databrickscfg file. If you create the profile, replace the placeholders with the appropriate values.

    [<some-unique-configuration-profile-name>]
    host  = <workspace-url>
    token = <personal-access-token>
    

    Note

    By default, the Databricks CLI looks for this .databrickscfg file in your ~ (your user home) folder on Unix, Linux, or macOS, or your %USERPROFILE% (your user home) folder on Windows. If the .databrickscfg file does not exist in this default location, create this file manually. Do not forget the dot (.) at the beginning of the file name.

  2. Use the Databricks CLI’s --profile or -p option followed by the name of your configuration profile, as part of the Databricks CLI command call, for example databricks clusters list -p <configuration-profile-name>.

Tip

For Databricks CLI version 0.201.0 and above, you can press Tab after --profile or -p to display a list of existing available configuration profiles to choose from, instead of entering the configuration profile name manually.

For Databricks personal access token authentication only, you can use the Databricks CLI to create a configuration profile instead of manually creating one in the preceding steps. To do this, use the Databricks CLI to run the configure command as follows:

databricks configure --host <workspace-url> --profile <some-unique-configuration-profile-name>

For <workspace-url>, enter https:// followed by your instance name, for example https://<prefix>.cloud.databricks.com. To get your instance name, see Workspace instance names, URLs, and IDs.

The command prompts you to enter your Databricks personal access token that maps to the specified <workspace-url>:

✔ Databricks Token:

After you enter your Databricks personal access token, a corresponding configuration profile is added to your .databrickscfg file. If the Databricks CLI cannot find this file in its default location, it creates this file for you first and then adds this configuration profile to the new file. The default location for this file is in your ~ (your user home) folder on Unix, Linux, or macOS, or your %USERPROFILE% (your user home) folder on Windows.

Basic authentication

Basic authentication uses a Databricks username and password to authenticate the target Databricks user account. See also Basic authentication.

To configure and use basic authentication, do the following:

  1. Create or identify a Databricks configuration profile with the following fields in your .databrickscfg file. If you create the profile, replace the placeholders with the appropriate values.

    For account-level commands, set the following values in your .databrickscfg file:

    [<some-unique-configuration-profile-name>]
    host       = <account-console-url>
    account_id = <account-id>
    username   = <username>
    password   = <password>
    

    For workspace-level commands, set the following values in your .databrickscfg file:

    [<some-unique-configuration-profile-name>]
    host       = <workspace-url>
    username   = <username>
    password   = <password>
    

    Note

    By default, the Databricks CLI looks for this .databrickscfg file in your ~ (your user home) folder on Unix, Linux, or macOS, or your %USERPROFILE% (your user home) folder on Windows. If the .databrickscfg file does not exist in this default location, create this file manually. Do not forget the dot (.) at the beginning of the file name.

  2. Use the Databricks CLI’s --profile or -p option followed by the name of your configuration profile, as part of the Databricks CLI command call, for example databricks account groups list -p <configuration-profile-name> or databricks clusters list -p <configuration-profile-name>.

    Tip

    For Databricks CLI version 0.201.0 and above, you can press Tab after --profile or -p to display a list of existing available configuration profiles to choose from, instead of entering the configuration profile name manually.

OAuth machine-to-machine (M2M) authentication

Instead of authenticating with Databricks by using Databricks personal access token authentication, you can use OAuth authentication. OAuth provides tokens with faster expiration times than Databricks personal access tokens, and offers better server-side session invalidation and scoping. Because OAuth access tokens expire in less than an hour, this reduces the risk associated with accidentally checking tokens into source control. See also OAuth machine-to-machine (M2M) authentication.

To configure and use OAuth M2M authentication, do the following:

  1. Complete the OAuth M2M authentication setup instructions. See Steps 1–3 in Authentication using OAuth for service principals.

    Important

    You only need to complete Steps 1–3 in the preceding article’s instructions.

    Step 4 in that article covers manually creating OAuth access tokens; however, the Databricks CLI automatically creates and manages OAuth access tokens for your target Databricks service principal on your behalf. Step 5 in that article covers using curl to call the Databricks REST API, instead of using the Databricks CLI.

  2. Create or identify a Databricks configuration profile with the following fields in your .databrickscfg file. If you create the profile, replace the placeholders with the appropriate values.

    For account-level commands, set the following values in your .databrickscfg file:

    [<some-unique-configuration-profile-name>]
    host          = <account-console-url>
    account_id    = <account-id>
    client_id     = <service-principal-client-id>
    client_secret = <service-principal-oauth-secret>
    

    For workspace-level commands, set the following values in your .databrickscfg file:

    [<some-unique-configuration-profile-name>]
    host          = <workspace-url>
    client_id     = <service-principal-client-id>
    client_secret = <service-principal-oauth-secret>
    

    Note

    By default, the Databricks CLI looks for this .databrickscfg file in your ~ (your user home) folder on Unix, Linux, or macOS, or your %USERPROFILE% (your user home) folder on Windows. If the .databrickscfg file does not exist in this default location, create this file manually. Do not forget the dot (.) at the beginning of the file name.

  3. Use the Databricks CLI’s --profile or -p option followed by the name of your configuration profile, as part of the Databricks CLI command call, for example databricks account groups list -p <configuration-profile-name> or databricks clusters list -p <configuration-profile-name>.

    Tip

    For Databricks CLI version 0.201.0 and above, you can press Tab after --profile or -p to display a list of existing available configuration profiles to choose from, instead of entering the configuration profile name manually.

OAuth user-to-machine (U2M) authentication

Instead of authenticating with Databricks by using token authentication, you can use OAuth authentication. OAuth provides tokens with faster expiration times than Databricks personal access tokens, and offers better server-side session invalidation and scoping. Because OAuth access tokens expire in less than an hour, this reduces the risk associated with accidentally checking tokens into source control. See also OAuth user-to-machine (U2M) authentication.

To configure and use OAuth U2M authentication, do the following:

  1. Complete the OAuth U2M authentication setup instructions. See OAuth user-to-machine (U2M) authentication.

  2. For account-level commands, initiate OAuth token management locally by running the following command. Replace the placeholders with the appropriate values. When prompted, complete the on-screen instructions.

    databricks auth login --host <account-console-url> --account-id <account-id>
    

    Note

    For Databricks CLI versions 0.203.0 and above, if you have an existing Databricks configuration profile with the host and account_id fields already set, you can substitute --host <account-console-url> --account-id <account-id> with --profile <profile-name>.

    For Databricks CLI versions 0.201.0 and above, if you do not provide the --host option, you are prompted for it at the command line.

    For Databricks CLI versions 0.200.1 and above, after you run the auth login command, you are prompted to save the account console URL and account ID as a Databricks configuration profile. When prompted, enter the name of a new or existing profile in your .databrickscfg file. Any existing profile with the same name in your .databrickscfg file is overwritten.

  3. For workspace-level commands, initiate OAuth token management locally by running the following command. Replace the placeholder with the appropriate value. When prompted, complete the on-screen instructions.

    databricks auth login --host <workspace-url>
    

    Note

    For Databricks CLI versions 0.203.0 and above, if you have an existing Databricks configuration profile with the host field already set, you can substitute --host <workspace-url> with --profile <profile-name>.

    For Databricks CLI versions 0.201.0 and above, if you do not provide the --host option, you are prompted for it at the command line.

    For Databricks CLI versions 0.200.1 and above, after you run the auth login command, you are prompted to save the workspace URL as a Databricks configuration profile. When prompted, enter the name of a new or existing profile in your .databrickscfg file. Any existing profile with the same name in your .databrickscfg file is overwritten.

  4. Create or identify a Databricks configuration profile with the following fields in your .databrickscfg file. If you create the profile, replace the placeholders with the appropriate values.

    For account-level commands, set the following values in your .databrickscfg file:

    [<some-unique-configuration-profile-name>]
    host       = <account-console-url>
    account_id = <account-id>
    

    For workspace-level commands, set the following values in your .databrickscfg file:

    [<some-unique-configuration-profile-name>]
    host = <workspace-url>
    

    Note

    By default, the Databricks CLI looks for this .databrickscfg file in your ~ (your user home) folder on Unix, Linux, or macOS, or your %USERPROFILE% (your user home) folder on Windows. If the .databrickscfg file does not exist in this default location, create this file manually. Do not forget the dot (.) at the beginning of the file name.

  5. Use the Databricks CLI’s --profile or -p option followed by the name of your configuration profile, as part of the Databricks CLI command call, for example databricks account groups list -p <configuration-profile-name> or databricks clusters list -p <configuration-profile-name>.

    Tip

    For Databricks CLI version 0.201.0 and above, you can press Tab after --profile or -p to display a list of existing available configuration profiles to choose from, instead of entering the configuration profile name manually.

Get information about configuration profiles

Adding multiple configuration profiles to the .databrickscfg file enables you to quickly run commands across various workspaces by specifying the target configuration profile’s name in the command’s --profile or -p option, for those commands that support this option. If you do not specify the --profile or -p option in a command that supports this option, the command will use the DEFAULT configuration profile by default.

Tip

For Databricks CLI version 0.201.0 and above, you can press Tab after --profile or -p to display a list of existing available configuration profiles to choose from, instead of entering the configuration profile name manually.

For example, you could have a configuration profile named DEV that references a Databricks workspace that you use for development workloads and a separate configuration profile named PROD that references a different Databricks workspace that you use for production workloads.

By default, the Databricks CLI looks for the .databrickscfg file in your ~ (your user home) folder on Unix, Linux, or macOS, or your %USERPROFILE% (your user home) folder on Windows. You can change the default path of the .databrickscfg file by setting the environment variable DATABRICKS_CONFIG_FILE. To learn how to set environment variables, see your operating system’s documentation.

To get information about an existing configuration profile, run the auth env command:

databricks auth env -p <configuration-profile-name>

# Or:
databricks auth env --host <account-console-url>

# Or:
databricks auth env --host <workspace-url>

For example, here is the output for a profile that is configured with Databricks personal access token authentication:

{
  "env": {
    "DATABRICKS_AUTH_TYPE": "pat",
    "DATABRICKS_CONFIG_PROFILE": "<configuration-profile-name>",
    "DATABRICKS_HOST": "<workspace-url>",
    "DATABRICKS_TOKEN": "<token-value>"
  }
}

To get information about all available profiles, run the auth profiles command:

databricks auth profiles

Output (the ellipses represent omitted content, for brevity):

{
  "profiles": [
    {
      "name": "<configuration-profile-name>",
      "host": "<workspace-url>",
      "cloud": "<cloud-id>",
      "auth_type": "<auth-type>",
      "valid": true
    },
    {
      "...": "..."
    }
  ]
}

The output of the auth profiles command does not display any access tokens. To display an access token, run the preceding auth env command.

Important

The Databricks CLI does not work with a .netrc file. You can have a .netrc file in your environment for other purposes, but the Databricks CLI will not use that .netrc file.

Test your DEFAULT configuration profile setup

To check whether you set up authentication correctly, you can run a command such as the following, which lists the available Databricks Runtime versions for the Databricks workspace that is associated with your DEFAULT profile.

The following call assumes that you do not have any special environment variables set, which take precedence over the settings in your DEFAULT profile. For more information, see Authentication order of evaluation.

databricks clusters spark-versions

Test your configuration profiles

To check whether you set up any configuration profiles correctly, you can run a command such as the following with one of your workspace-level configuration profile names. This command lists the available Databricks Runtime versions for the Databricks workspace that is associated with the specified configuration profile, represented here by the placeholder <configuration-profile-name>:

databricks clusters spark-versions -p <configuration-profile-name>

Tip

For Databricks CLI version 0.201.0 and above, you can press Tab after --profile or -p to display a list of existing available configuration profiles to choose from, instead of entering the configuration profile name manually.

To list details for a specific profile, run the following command:

databricks auth env -p <configuration-profile-name>

To list details for all of your available profiles, run the following command:

databricks auth profiles

Authentication order of evaluation

Whenever the Databricks CLI needs to gather the settings that are required to attempt to authenticate with a Databricks workspace or account, it searches for these settings in the following locations, in the following order.

  1. For bundle commands, the values of fields within a project’s bundle setting files. (Bundle setting files do not support the direct inclusion of access credential values.)

  2. The values of environment variables, as listed within this article and in Environment variables and fields for client unified authentication.

  3. Configuration profile field values within the .databrickscfg file, as listed previously within this article.

Whenever the Databricks CLI finds the required settings that it needs, it stops searching in other locations. For example:

  • The Databricks CLI needs the value of a Databricks personal access token. A DATABRICKS_TOKEN environment variable is set, and the .databrickscfg file also contains multiple personal access tokens. In this example, the Databricks CLI uses the value of the DATABRICKS_TOKEN environment variable and does not search the .databrickscfg file.

  • The databricks bundle deploy -e development command needs the value of a Databricks personal access token. A DATABRICKS_TOKEN environment variable is not set, and the .databrickscfg file contains multiple personal access tokens. The project’s bundle settings file contains a development environment declaration that references through its profile field a configuration profile named DEV. In this example, the Databricks CLI searches the .databrickscfg file for a profile named DEV and uses the value of that profile’s token field.

  • The databricks bundle run -e development hello-job command needs the value of a Databricks personal access token. A DATABRICKS_TOKEN environment variable is not set, and the .databrickscfg file contains multiple personal access tokens. The project’s bundle settings file contains a development environment declaration that references through its host field a specific Databricks workspace URL. In this example, the Databricks CLI searches through the configuration profiles within the .databrickscfg file for a profile that contains a host field with a matching workspace URL. The Databricks CLI finds a matching host field and then uses that profile’s token field value.

Use the CLI

This section shows you how to list Databricks CLI command groups and commands, display Databricks CLI help, and work with Databricks CLI output.

List CLI command groups

You list the command groups by using the --help or -h option. For example:

databricks -h

List CLI commands

You list the commands for any command group by using the --help or -h option. For example, to list the clusters commands:

databricks clusters -h

Display CLI command help

You display the help for a command by using the --help or -h option. For example, to display the help for the clusters list command:

databricks clusters list -h

Use jq to parse CLI output

Some Databricks CLI commands output responses are formatted as JSON. In many cases, the Databricks CLI formats the JSON output so that it is easier to read. However, sometimes it can be useful to parse out parts of the JSON instead of listing the entire response. For example, to list just the display name of a Databricks cluster with the specified cluster ID, you can use the utility jq:

databricks clusters get 1234-567890-abcde123 | jq -r .cluster_name

Output:

My-11.3-LTS-Cluster

You can install jq for example on macOS by using Homebrew with brew install jq or on Windows by using Chocolatey with choco install jq. For more information on jq, see the jq Manual.

JSON string parameters

The format of string parameters is handled differently in JSON depending on your operating system:

You must enclose JSON string parameters in double quotes, and you must enclose the entire JSON payload in single quotes. Some examples:

'{"cluster_id": "1234-567890-abcde123"}'
'["20230323", "Amsterdam"]'

You must enclose JSON string parameters and the entire JSON payload in double quotes, and the double-quote characters inside the JSON payload must be preceded by \. Some examples:

"{\"cluster_id\": \"1234-567890-abcde123\"}"
"[\"20230323\", \"Amsterdam\"]"

Global flags

The following flags are available to all Databricks CLI commands. Note that some flags do not apply to some commands. For more information, see the command’s documentation.

Flag

Description

-h or --help

Display help for the Databricks CLI or the related command group or the related command.

-e or --environment string

A string representing the bundle environment to use if applicable for the related command.

--log-file

A string representing the to write output logs to. If this flag is not specified then the default is to write output logs to stderr.

--log-format

text to write output logs to text or json to write output logs to JSON. If this flag is not specified then output logs are written as text.

--log-level

A string representing the log format level. If not specified then the log format level is disabled.

-o or --output

text to write output as text or json to write output as JSON. If this flag is not specified then output is written as text.

-p or --profile

A string representing the named configuration profile to use within your .databrickscfg file. If this flag is not specified then the DEFAULT named profile is used if one exists. For Databricks CLI version 0.201.0 and above

you can press Tab after --profile or -p to display a list of existing available configuration profiles to choose from

instead of entering the configuration profile name manually.

--progress-format

The format for progress logs to display (default (the default) or append or inplace or json).

CLI command groups

The Databricks CLI includes the command groups listed in the following table.

Help for these command groups and their commands is included within the Databricks CLI:

  • To display help for a command group and a list of the command group’s commands, run databricks <command-group> -h.

  • To display help for a command, run databricks <command-group> <command-name> -h.

Most CLI command groups map directly to methods documented in the Databricks REST API reference. CLI command groups not documented in the REST API reference have their own separate reference articles, which are linked in the following table:

Command group

Area

account

Databricks account operations.

alerts

Databricks SQL alerts operations.

auth

Manage Databricks CLI authentication settings.

bundle

Databricks asset bundle operations.

catalogs

Unity Catalog catalog operations.

clean-rooms

Clean room operations.

cluster-policies

Cluster policy operations.

clusters

Cluster operations.

completion

Enable Databricks CLI autocompletion.

configure

Manage Databricks CLI configuration.

current-user

Get information about the current authenticated Databricks user or Databricks service principal.

dashboards

Dashboard operations.

data-sources

List data source connection information for available Databricks SQL warhouses.

experiments

MLflow experiment operations.

external-locations

Unity Catalog external location operations.

fs

Databricks File System (DBFS) operations.

functions

Unity Catalog user-defined function (UDF) operations.

git-credentials

Git credentials operations.

global-init-scripts

Global init scripts operations.

grants

Unity Catalog access grant operations.

groups

Databricks workspace group operations.

help

Display help for the Databricks CLI or the related command group or the related command.

instance-pools

Instance pool operations.

instance-profiles

Instance profile operations.

ip-access-lists

IP access list operations.

jobs

Databricks job operations.

libraries

Library operations.

metastores

Unity Catalog metastore operations.

model-registry

Model registry operations.

permissions

Databricks object and endpoint permission operations.

pipelines

Delta Live Tables pipeline operations.

policy-families

Cluster policy family operations.

providers

Delta Sharing provider operations.

queries

Databricks SQL query operations.

query-history

Databricks SQL query history operations.

recipient-activation

Delta Sharing receipient activation operations.

recipients

Delta Sharing recipient operations.

repos

Databricks Repos operations.

schemas

Unity Catalog schema operations.

secrets

Databricks secrets operations.

service-principals

Databricks service principal operations.

serving-endpoints

Model serving endpoint operations.

shares

Delta Sharing share operations.

storage-credentials

Unity Catalog storage credential operations.

sync

Perform one-way synchronization of file changes within a local filesystem directory to a directory within a remote Databricks workspace.

table-constraints

Table constraint operations.

tables

Unity Catalog table operations.

token-management

Databricks personal access token management operations.

tokens

Databricks personal access token operations.

users

Databricks user operations.

version

Displays the Databricks CLI version.

volumes

Unity Catalog volume operations.

warehouses

Databricks SQL warehouse operations.

workspace

Databricks workspace operations for notebooks and folders.

workspace-bindings

Operations for Databricks workspace bindings for Unity Catalog catalogs.

workspace-conf

Databricks workspace settings configuration operations.

api

Call any available Databricks REST API endpoint. You should use this command group only for advanced scenarios such as preview releases of specific Databricks REST APIs for which the Databricks CLI does not already wrap the target Databricks REST API within a related command.