Amazon
Web Services
Microsoft Azure
Google Cloud PlatformRuns CLI (legacy)
Important
This documentation has been retired and might not be updated.
This information applies to legacy Databricks CLI versions 0.18 and below. Databricks recommends that you use newer Databricks CLI version 0.205 or above instead. See What is the Databricks CLI?. To find your version of the Databricks CLI, run databricks -v.
To migrate from Databricks CLI version 0.18 or below to Databricks CLI version 0.205 or above, see Databricks CLI migration.
You run Databricks job runs CLI subcommands by appending them to databricks runs and Databricks jobs CLI subcommands by appending them to databricks jobs. For Databricks jobs CLI subcommands, see the Jobs CLI (legacy). Together, these subcommands call the Jobs API and Jobs API 2.0.
Important
The Databricks job runs CLI supports calls to two versions of the Databricks Jobs REST API: versions 2.1 and 2.0. (Job runs functionality is part of the Jobs REST API.) Version 2.1 adds support for orchestration of jobs with multiple tasks; see Create and run Databricks Jobs and Updating from Jobs API 2.0 to 2.1. Databricks recommends that you call version 2.1, unless you have legacy scripts that rely on version 2.0 and cannot be migrated.
Unless otherwise specified, the programmatic behaviors that are described in this article apply equally to versions 2.1 and 2.0.
Note
If you receive a 500-level error when making job runs CLI requests, Databricks recommends retrying requests for up to 10 minutes (with a minimum 30-second interval between retries).
Requirements to call the Jobs REST API 2.1
To set up the Databricks job runs CLI (and jobs CLI) to call the Jobs REST API 2.1, do the following:
Update the CLI to version 0.16.0 or above.
Do one of the following:
Run the command
databricks jobs configure --version=2.1. This adds the settingjobs-api-version = 2.1to the file~/.databrickscfgon Unix, Linux, or macOS, or%USERPROFILE%\.databrickscfgon Windows. All job runs CLI (and jobs CLI) subcommands will call the Jobs REST API 2.1 by default.Manually add the setting
jobs-api-version = 2.1to the file~/.databrickscfgon Unix, Linux, or macOS, or%USERPROFILE%\.databrickscfgon Windows. All job runs CLI (and jobs CLI) subcommands will call the Jobs REST API 2.1 by default.Append the option
--version=2.1(for example,databricks runs list --version=2.1) to instruct the job runs CLI to call the Jobs REST API 2.1 for that call only.
If you take none of the preceding actions, the job runs CLI (and jobs CLI) will call the Jobs REST API 2.0 by default.
Requirements to call the Jobs REST API 2.0
To set up the Databricks job runs CLI (and jobs CLI) to call the Jobs REST API 2.0, do one of the following:
Use a version of the Databricks CLI below 0.16.0, or
Update the CLI to version 0.16.0 or above, and then do one of the following:
Run the command
databricks jobs configure --version=2.0. This adds the settingjobs-api-version = 2.0to the file~/.databrickscfgon Unix, Linux, or macOS, or%USERPROFILE%\.databrickscfgon Windows. All job runs CLI (and jobs CLI) subcommands will call the Jobs REST API 2.0 by default.Manually add the setting
jobs-api-version = 2.0to the file~/.databrickscfgon Unix, Linux, or macOS, or%USERPROFILE%\.databrickscfgon Windows. All job runs CLI (and jobs CLI) subcommands will call the Jobs REST API 2.0 by default.Append the option
--version=2.1(for example,databricks runs list --version=2.0) to instruct the job runs CLI to call the Jobs REST API 2.0 for that call only.
If you take none of the preceding actions, the job runs CLI (and jobs CLI) will call the Jobs REST API 2.0 by default.
Subcommands and general usage
databricks runs --help
Usage: databricks runs [OPTIONS] COMMAND [ARGS]...
Utility to interact with jobs runs.
Options:
-v, --version [VERSION]
--debug Debug mode. Shows full stack trace on error.
--profile TEXT CLI connection profile to use. The default profile is
"DEFAULT".
-h, --help Show this message and exit.
Commands:
cancel Cancels the specified run.
get Gets the metadata about a run in JSON form.
get-output Gets the output of a run.
list Lists job runs.
submit Submits a one-time run.
Cancel a run
To display usage documentation, run databricks runs cancel --help.
databricks runs cancel --run-id 119
{}
Get information about a run
To display usage documentation, run databricks runs get --help.
Jobs CLI 2.0 response example
{
"job_id": 239,
"run_id": 119,
"number_in_job": 1,
"original_attempt_run_id": 119,
"state": {
"life_cycle_state": "TERMINATED",
"result_state": "SUCCESS",
"state_message": ""
},
"task": {
"notebook_task": {
"notebook_path": "/Users/someone@example.com/notebooks/my-notebook.ipynb"
}
},
"cluster_spec": {
"new_cluster": {
"spark_version": "8.1.x-scala2.12",
"aws_attributes": {
"zone_id": "us-west-2c",
"availability": "SPOT_WITH_FALLBACK"
},
"node_type_id": "m5d.large",
"enable_elastic_disk": false,
"num_workers": 1
}
},
"cluster_instance": {
"cluster_id": "1234-567890-abcd123",
"spark_context_id": "1234567890123456789"
},
"start_time": 1618510327335,
"setup_duration": 191000,
"execution_duration": 41000,
"cleanup_duration": 2000,
"end_time": 1618510561615,
"trigger": "ONE_TIME",
"creator_user_name": "someone@example.com",
"run_name": "my-notebook-run",
"run_page_url": "https://dbc-a1b2345c-d6e7.cloud.databricks.com/?o=1234567890123456#job/239/run/1",
"run_type": "JOB_RUN",
"attempt_number": 0
}
Get the output of a run
To display usage documentation, run databricks runs get-output --help.
Note
When a notebook_task returns a value from a call to dbutils.notebook.exit(), Databricks limits the returned value to the first 5 MB of data. To return a larger result, you can store job results in a cloud storage service.
Jobs CLI 2.0 response example
{
"metadata": {
"job_id": 239,
"run_id": 119,
"number_in_job": 1,
"original_attempt_run_id": 119,
"state": {
"life_cycle_state": "TERMINATED",
"result_state": "SUCCESS",
"state_message": ""
},
"task": {
"notebook_task": {
"notebook_path": "/Users/someone@example.com/notebooks/my-notebook.ipynb"
}
},
"cluster_spec": {
"new_cluster": {
"spark_version": "8.1.x-scala2.12",
"aws_attributes": {
"zone_id": "us-west-2c",
"availability": "SPOT_WITH_FALLBACK"
},
"node_type_id": "m5d.large",
"enable_elastic_disk": false,
"num_workers": 1
}
},
"cluster_instance": {
"cluster_id": "1234-567890-abcd123",
"spark_context_id": "1234567890123456789"
},
"start_time": 1618510327335,
"setup_duration": 191000,
"execution_duration": 41000,
"cleanup_duration": 2000,
"end_time": 1618510561615,
"trigger": "ONE_TIME",
"creator_user_name": "someone@example.com",
"run_name": "my-notebook-run",
"run_page_url": "https://dbc-a1b2345c-d6e7.cloud.databricks.com/?o=1234567890123456#job/239/run/1",
"run_type": "JOB_RUN",
"attempt_number": 0
},
"notebook_output": {}
}
Get information about all runs
To display usage documentation, run databricks runs list --help.
Jobs CLI 2.0 response example
{
"runs": [
{
"job_id": 239,
"run_id": 119,
"number_in_job": 1,
"original_attempt_run_id": 119,
"state": {
"life_cycle_state": "TERMINATED",
"result_state": "SUCCESS",
"state_message": ""
},
"task": {
"notebook_task": {
"notebook_path": "/Users/someone@example.com/notebooks/my-notebook.ipynb"
}
},
"cluster_spec": {
"new_cluster": {
"spark_version": "8.1.x-scala2.12",
"aws_attributes": {
"zone_id": "us-west-2c",
"availability": "SPOT_WITH_FALLBACK"
},
"node_type_id": "m5d.large",
"enable_elastic_disk": false,
"num_workers": 1
}
},
"cluster_instance": {
"cluster_id": "1234-567890-abcd123",
"spark_context_id": "1234567890123456789"
},
"start_time": 1618510327335,
"setup_duration": 191000,
"execution_duration": 41000,
"cleanup_duration": 2000,
"end_time": 1618510561615,
"trigger": "ONE_TIME",
"creator_user_name": "someone@example.com",
"run_name": "my-notebook-run",
"run_page_url": "https://dbc-a1b2345c-d6e7.cloud.databricks.com/?o=1234567890123456#job/239/run/1",
"run_type": "JOB_RUN",
"attempt_number": 0
},
...
],
"has_more": false
}
If has_more returns true, information about additional runs is available. Use the --offset option to return information about runs relative to the most recent run. For example, to return information starting with the tenth most recent run, specify --offset 10.
Use the --limit option to return information about a fixed number of runs. For example, to return information for up to the next 5 runs, specify --limit 5. You can specify up to 1000 runs. If not specified, the default is 20.
Submit a one-time run
To display usage documentation, run databricks runs submit --help.
Jobs REST API 2.0 request and response example
submit-run.json:
{
"run_name": "my-spark-run",
"new_cluster": {
"spark_version": "7.3.x-scala2.12",
"node_type_id": "r3.xlarge",
"aws_attributes": {
"availability": "ON_DEMAND"
},
"num_workers": 10
},
"libraries": [
{
"jar": "dbfs:/my-jar.jar"
},
{
"maven": {
"coordinates": "org.jsoup:jsoup:1.7.2"
}
}
],
"spark_jar_task": {
"main_class_name": "com.databricks.ComputeModels"
}
}
{
"run_id": 123
}
