Databricks CLI (versions 0.100 and higher)
Note
This information applies to Databricks CLI versions 0.100 and higher, which are in Private Preview. To try them, reach out to your Databricks contact. To find your version of the Databricks CLI, run databricks -v
.
The Databricks command-line interface (also known as the Databricks CLI) utility, provided by Databricks, provides an easy-to-use interface to automate the Databricks platform from your terminal, command prompt, or automation scripts.
To skip ahead to usage information about all of the available commands for Databricks CLI versions 0.100 and higher, see CLI command groups.
Note
This feature is in Private Preview. To try it, reach out to your Databricks contact.
Databricks CLI versions 0.100 and higher are separate from Databricks CLI versions 0.99 and lower.
Databricks does not guarantee that scripts and commands targeting Databricks CLI versions 0.99 and lower will run without modification for Databricks CLI versions 0.100 and higher.
Databricks does not guarantee that scripts and commands targeting Databricks CLI versions 0.100 and higher will run without modification for Databricks CLI versions 0.99 and lower.
Set up the CLI
The following sections describe how to set up the Databricks CLI.
Requirements
There are no special requirements before you install the Databricks CLI. Databricks provides the Databricks CLI as a standalone executable for Linux, macOS, and Windows operating systems.
Install or update the CLI
This section describes how to install or update your development machine to run the Databricks CLI executable.
Install the CLI
Download onto your local development machine the correct Databricks CLI
.zip
file, as provided by your Databricks contact. The.zip
file must match your development machine’s operating system and architecture:Filename
Architecture
databricks_cli_X.Y.Z_darwin_amd64.zip
macOS, Intel 64-bit
databricks_cli_X.Y.Z_darwin_arm64.zip
macOS, Apple Silicon
databricks_cli_X.Y.Z_linux_amd64.zip
Linux, Intel 64-bit
databricks_cli_X.Y.Z_linux_arm64.zip
Linux, ARM 64-bit
databricks_cli_X.Y.Z_windows_386.zip
Windows, Intel 32-bit
databricks_cli_X.Y.Z_windows_amd64.zip
Windows, Intel 64-bit
databricks_cli_X.Y.Z_windows_arm64.zip
Windows, ARM 64-bit
To get your machine’s architecture, see your operating system’s documentation.
Databricks also provides a checksum file named
databricks_cli_X.Y.Z_SHA256SUMS
if you need to verify the integrity of the.zip
file. To get this checksum file, reach out to your Databricks contact. To run a checksum verification, see your operating system’s documentation.Extract the contents of the downloaded
.zip
file. (To extract the.zip
file, see your operating system’s documentation.)In the extracted content, a folder appears with the same name as the
.zip
file. Inside of this folder is the Databricks CLI executable. You can leave the Databricks CLI executable there, or you can copy or move it to another location.
Update the CLI
Optionally, delete the Databricks CLI executable, the
.zip
file, and the.zip
file’s extracted folder, from the previous procedure.Install the latest version of the Databricks CLI by following the instructions in the previous procedure.
To view the Databricks CLI executable’s version, run the version
command or use the -v
option:
databricks -v
Tip
If you try to run databricks
but get an error such as command not found: databricks
, either specify the path to the Databricks CLI executable, or omit the path and make sure that the Databricks CLI executable is referenced in your operating system’s PATH
environment variable. To manage your PATH
environment variable, see your operating system’s documentation.
Set up authentication
Before you can run Databricks CLI commands, you must set up authentication between the Databricks CLI and Databricks.
The Databricks CLI implements the Databricks client unified authentication standard, a consolidated and consistent architecural and programmatic approach to authentication. This approach helps make setting up and automating authentication with Databricks more centralized and predictable. It enables you to configure Databricks authentication once and then use that configuration across multiple Databricks tools and SDKs without further authentication configuration changes. See Databricks client unified authentication.
You must authenticate the Databricks CLI to the relevant resources at run time in order to run Databricks automation commands within a Databricks account or workspace. Depending on whether you want to call Databricks workspace-level REST APIs, Databricks account-level REST APIs, or both, you must authenticate to the Databricks workspace, account, or both.
The following sections provide information about setting up authenticate between the Databricks CLI and Databricks:
Token authentication
Token authentication uses a Databricks personal access token to authenticate the target Databricks entity, such as a Databricks user account. See Token authentication.
To create a Databricks personal access token, see Databricks personal access tokens.
Note
You cannot use token authentication for authenticating with a Databricks account, as Databricks account-level operations do not use Databricks personal access tokens for authentication. To authenticate with a Databricks account, consider using one of the following authentication types instead:
To configure and use token authentication, do the following:
Create or identify a Databricks configuration profile with the following fields in your
.databrickscfg
file. If you create the profile, replace the placeholders with the appropriate values.Use the Databricks CLI’s
--profile
or-p
option followed by the name of your configuration profile, as part of the Databricks CLI command call, for exampledatabricks clusters list -p <configuration-profile-name>
.
[<some-unique-configuration-profile-name>]
host = <workspace-url>
token = <token>
Tip
For workspace-level operations, for token authentication only, you can use the Databricks CLI to create a configuration profile instead of manually creating one. To do this, use the Databricks CLI to run the configure
command as follows:
databricks configure --host <workspace-url> -t -p <some-unique-configuration-profile-name>
For <workspace-url>
, enter https://
followed by your instance name, for example https://<prefix>.cloud.databricks.com
. To get your instance name, see Workspace instance names, URLs, and IDs.
If you try to run databricks
but get an error such as command not found: databricks
, either specify the path to the Databricks CLI executable, or omit the path and make sure that the Databricks CLI executable is referenced in your operating system’s PATH
environment variable. To manage your PATH
environment variable, see your operating system’s documentation.
The command prompts you to enter your access token that maps to the specified <workspace-url>
:
✔ Databricks Token:
After you enter your access token, a corresponding configuration profile is added to your .databrickscfg
file.
Basic authentication
Basic authentication uses a Databricks username and password to authenticate the target Databricks user account. See Basic authentication.
To configure and use basic authentication, do the following:
Create or identify a Databricks configuration profile with the following fields in your
.databrickscfg
file. If you create the profile, replace the placeholders with the appropriate values.Use the Databricks CLI’s
--profile
or-p
option followed by the name of your configuration profile, as part of the Databricks CLI command call, for exampledatabricks clusters list -p <configuration-profile-name>
.
For account-level commands, set the following values in your .databrickscfg
file:
[<some-unique-configuration-profile-name>]
host = <account-console-url>
account_id = <account-id>
username = <username>
password = <password>
For workspace-level commands, set the following values in your .databrickscfg
file:
[<some-unique-configuration-profile-name>]
host = <workspace-url>
username = <username>
password = <password>
OAuth machine-to-machine (M2M) authentication
Instead of authenticating with Databricks by using token authentication, you can use OAuth authentication. OAuth provides tokens with faster expiration times than Databricks personal access tokens, and offers better server-side session invalidation and scoping. Because OAuth access tokens expire in less than an hour, this reduces the risk associated with accidentally checking tokens into source control. See OAuth machine-to-machine (M2M) authentication.
To configure and use OAuth M2M authentication, do the following:
Complete the OAuth M2M authentication setup instructions. See Steps 1–3 in Authentication using OAuth tokens for service principals.
Important
You only need to complete Steps 1–3 in the preceding article’s instructions.
Step 4 in that article covers manually creating OAuth access tokens; however, the Databricks CLI automatically creates and manages OAuth access tokens for your target Databricks service principal on your behalf. Step 5 in that article covers using
curl
to call the Databricks REST API, instead of using the Databricks CLI.Create or identify a Databricks configuration profile with the following fields in your
.databrickscfg
file. If you create the profile, replace the placeholders with the appropriate values.Use the Databricks CLI’s
--profile
or-p
option followed by the name of your configuration profile, as part of the Databricks CLI command call, for exampledatabricks clusters list -p <configuration-profile-name>
.
For account-level commands, set the following values in your .databrickscfg
file:
[<some-unique-configuration-profile-name>]
host = <account-console-url>
account_id = <account-id>
client_id = <service-principal-client-id>
client_secret = <service-principal-oauth-secret>
For workspace-level commands, set the following values in your .databrickscfg
file:
[<some-unique-configuration-profile-name>]
host = <workspace-url>
client_id = <service-principal-client-id>
client_secret = <service-principal-oauth-secret>
OAuth user-to-machine (U2M) authentication
Instead of authenticating with Databricks by using token authentication, you can use OAuth authentication. OAuth provides tokens with faster expiration times than Databricks personal access tokens, and offers better server-side session invalidation and scoping. Because OAuth access tokens expire in less than an hour, this reduces the risk associated with accidentally checking tokens into source control. See OAuth user-to-machine (U2M) authentication.
To configure and use OAuth U2M authentication, do the following:
Complete the OAuth U2M authentication setup instructions. See OAuth user-to-machine (U2M) authentication.
For account-level commands, initiate OAuth token management locally by running the following command. Replace the placeholders with the appropriate values. When prompted, complete the on-screen instructions.
databricks auth login --host <account-console-url> --account-id <account-id>
For workspace-level commands, initiate OAuth token management locally by running the following command. Replace the placeholder with the appropriate value. When prompted, complete the on-screen instructions.
databricks auth login --host <workspace-url>
Create or identify a Databricks configuration profile with the following fields in your
.databrickscfg
file. If you create the profile, replace the placeholders with the appropriate values.Use the Databricks CLI’s
--profile
or-p
option followed by the name of your configuration profile, as part of the Databricks CLI command call, for exampledatabricks clusters list -p <configuration-profile-name>
.
For account-level commands, set the following values in your .databrickscfg
file:
[<some-unique-configuration-profile-name>]
host = <account-console-url>
account_id = <account-id>
For workspace-level commands, set the following values in your .databrickscfg
file:
[<some-unique-configuration-profile-name>]
host = <workspace-url>
Get information about configuration profiles
Adding multiple connection profiles to the .databrickscfg
file enables you to quickly run commands across various workspaces by specifying the target connection profile’s name in the command’s --profile
or -p
option, for those commands that support this option. If you do not specify the --profile
or -p
option in a command that supports this option, the command will use the DEFAULT
connection profile by default.
For example, you could have a connection profile named DEV
that references a Databricks workspace that you use for development workloads and a separate connection profile named PROD
that references a different Databricks workspace that you use for production workloads.
You can change the default path of the .databrickscfg
file by setting the environment variable DATABRICKS_CONFIG_FILE
. To learn how to set environment variables, see your operating system’s documentation.
To get information about an existing configuration profile, run the auth env
command:
databricks auth env -p <profile-name>
# Or:
databricks auth env --host <workspace-url>
Tip
If you try to run databricks
but get an error such as command not found: databricks
, either specify the path to the Databricks CLI executable, or omit the path and make sure that the Databricks CLI executable is referenced in your operating system’s PATH
environment variable. To manage your PATH
environment variable, see your operating system’s documentation.
For example, here is the output for a profile that is configured with Databricks access token authentication:
{
"env": {
"DATABRICKS_AUTH_TYPE": "pat",
"DATABRICKS_CONFIG_PROFILE": "<profile-name>",
"DATABRICKS_HOST": "<workspace-url>",
"DATABRICKS_TOKEN": "<token-value>"
}
}
To get information about all available profiles, run the auth profiles
command:
databricks auth profiles
Output (the ellipses represent omitted content, for brevity):
{
"profiles": [
{
"name": "<profile-name>",
"host": "<workspace-url>",
"cloud": "<cloud-id>",
"auth_type": "<auth-type>",
"valid": true
},
{
"...": "..."
}
]
}
The output of the auth profiles
command does not display any access tokens. To display an access token, run the preceding auth env
command.
Important
The Databricks CLI does not work with a .netrc file. You can have a .netrc
file in your environment for other purposes, but the Databricks CLI will not use that .netrc
file.
Test your DEFAULT configuration profile setup
To check whether you set up authentication correctly, you can run a command such as the following, which lists the available Databricks Runtime versions for the Databricks workspace that is associated with your DEFAULT
profile.
The following call assumes that you do not have any special environment variables set, which take precedence over the settings in your DEFAULT
profile. For more information, see the api command group reference.
databricks clusters spark-versions
Tip
If you try to run databricks
but get an error such as command not found: databricks
, either specify the path to the Databricks CLI executable, or omit the path and make sure that the Databricks CLI executable is referenced in your operating system’s PATH
environment variable. To manage your PATH
environment variable, see your operating system’s documentation.
Test your connection profiles
To check whether you set up any connection profiles correctly, you can run a command such as the following with one of your connection profile names. This command lists the available Databricks Runtime versions for the Databricks workspace that is associated with the specified connection profile, represented here by the placeholder <profile-name>
:
databricks clusters spark-versions -p <profile-name>
To list details for a specific profile, run the following command:
databricks auth env -p <profile-name>
To list details for all of your available profiles, run the following command:
databricks auth profiles
Authentication order of evaluation
Whenever the Databricks CLI needs to gather the settings that are required to attempt to authenticate with a Databricks workspace or account, it searches for these settings in the following locations, in the following order.
For bundle commands, the values of fields within a project’s bundle setting files. (Bundle setting files do not support the direct inclusion of access credential values.)
The values of environment variables, as listed within this article and in Environment variables and fields for client unified authentication.
Configuration profile field values within the
.databrickscfg
file, as listed previously within this article.
Whenever the Databricks CLI finds the required settings that it needs, it stops searching in other locations. For example:
The Databricks CLI needs the value of a Databricks personal access token. A
DATABRICKS_TOKEN
environment variable is set, and the.databrickscfg
file also contains multiple personal access tokens. In this example, the Databricks CLI uses the value of theDATABRICKS_TOKEN
environment variable and does not search the.databrickscfg
file.The
databricks bundle deploy -e development
command needs the value of a Databricks personal access token. ADATABRICKS_TOKEN
environment variable is not set, and the.databrickscfg
file contains multiple personal access tokens. The project’s bundle settings file contains adevelopment
environment declaration that references through itsprofile
field a configuration profile namedDEV
. In this example, the Databricks CLI searches the.databrickscfg
file for a profile namedDEV
and uses the value of that profile’stoken
field.The
databricks bundle run -e development hello-job
command needs the value of a Databricks personal access token. ADATABRICKS_TOKEN
environment variable is not set, and the.databrickscfg
file contains multiple personal access tokens. The project’s bundle settings file contains adevelopment
environment declaration that references through itshost
field a specific Databricks workspace URL. In this example, the Databricks CLI searches through the configuration profiles within the.databrickscfg
file for a profile that contains ahost
field with a matching workspace URL. The Databricks CLI finds a matchinghost
field and then uses that profile’stoken
field value.
Use the CLI
This section shows you how to list Databricks CLI command groups and commands, display Databricks CLI help, and work with Databricks CLI output.
List CLI command groups
You list the command groups by using the --help
or -h
option. For example:
databricks -h
List CLI commands
You list the commands for any command group by using the --help
or -h
option. For example, to list the clusters
commands:
databricks clusters -h
Display CLI command help
You display the help for a command by using the --help
or -h
option. For example, to display the help for the clusters list
command:
databricks clusters list -h
Use jq
to parse CLI output
Some Databricks CLI commands output responses are formatted as JSON. In many cases, the Databricks CLI formats the JSON output so that it is easier to read. However, sometimes it can be useful to parse out parts of the JSON instead of listing the entire response. For example, to list just the display name of a Databricks cluster with the specified cluster ID, you can use the utility jq
:
databricks clusters get 1234-567890-abcde123 | jq -r .cluster_name
Output:
My-11.3-LTS-Cluster
You can install jq
for example on macOS by using Homebrew with brew install jq
or on Windows by using Chocolatey with choco install jq
. For more information on jq
, see the jq Manual.
JSON string parameters
The format of string parameters is handled differently in JSON depending on your operating system:
You must enclose JSON string parameters in double quotes, and you must enclose the entire JSON payload in single quotes. Some examples:
'{"cluster_id": "1234-567890-abcde123"}'
'["20230323", "Amsterdam"]'
You must enclose JSON string parameters and the entire JSON payload in double quotes, and the double-quote characters inside the JSON payload must be preceded by \
. Some examples:
"{\"cluster_id\": \"1234-567890-abcde123\"}"
"[\"20230323\", \"Amsterdam\"]"
Global flags
The following flags are available to all Databricks CLI commands. Note that some flags do not apply to some commands. For more information, see the command’s documentation.
Flag |
Description |
---|---|
|
Display help for the Databricks CLI or the related command group or the related command. |
|
A string representing the bundle environment to use if applicable for the related command. |
|
A string representing the to write output logs to. If this flag is not specified then the default is to write output logs to stderr. |
|
|
|
A string representing the log format level. If not specified then the log format level is disabled. |
|
|
|
A string representing the named configuration profile to use within your |
|
The format for progress logs to display ( |
CLI command groups
The Databricks CLI includes the following command groups.
Help for these command groups is included within the Databricks CLI. To display help for a command group, run databricks <command-group> -h
. To display help for a command, run databricks <command-group> <command-name> -h
.
Command group |
Area |
---|---|
|
Databricks account operations. |
|
Databricks SQL alerts operations. |
Manage Databricks CLI authentication settings. |
|
Databricks application bundle operations. |
|
|
Unity Catalog catalog operations. |
|
Cluster policy operations. |
|
Cluster operations. |
Enable Databricks CLI autocompletion. |
|
Manage Databricks CLI configuration. |
|
|
Get information about the current authenticated Databricks user or Databricks service principal. |
|
Dashboard operations. |
|
List data source connection information for available Databricks SQL warhouses. |
|
MLflow experiment operations. |
|
Unity Catalog external location operations. |
|
Unity Catalog user-defined function (UDF) operations. |
|
Git credentials operations. |
|
Global init scripts operations. |
|
Unity Catalog access grant operations. |
|
Databricks workspace group operations. |
|
Display help for the Databricks CLI or the related command group or the related command. |
|
Instance pool operations. |
|
Instance profile operations. |
|
IP access list operations. |
|
Databricks job operations. |
|
Library operations. |
|
Unity Catalog metastore operations. |
|
Model registry operations. |
|
Databricks object and endpoint permission operations. |
|
Delta Live Tables pipeline operations. |
|
Cluster policy family operations. |
|
Delta Sharing provider operations. |
|
Databricks SQL query operations. |
|
Databricks SQL query history operations. |
|
Delta Sharing receipient activation operations. |
|
Delta Sharing recipient operations. |
|
Databricks Repos operations. |
|
Unity Catalog schema operations. |
|
Databricks secrets operations. |
|
Databricks service principal operations. |
|
Model serving endpoint operations. |
|
Delta Sharing share operations. |
|
Unity Catalog storage credential operations. |
Perform one-way synchronization of file changes within a local filesystem directory to a directory within a remote Databricks workspace. |
|
|
Table constraint operations. |
|
Unity Catalog table operations. |
|
Databricks personal access token management operations. |
|
Databricks personal access token operations. |
|
Databricks user operations. |
|
Displays the Databricks CLI version. |
|
Databricks SQL warehouse operations. |
|
Databricks workspace operations for notebooks and folders. |
|
Databricks workspace settings configuration operations. |
Call any available Databricks REST API endpoint. You should use this command group only for advanced scenarios such as preview releases of specific Databricks REST APIs for which the Databricks CLI does not already wrap the target Databricks REST API within a related command. |