REST API 1.2
Important
This article’s content has been retired and might not be updated. See Command Execution in the Databricks REST API Reference.
The Databricks REST API allows you to programmatically access Databricks instead of going through the web UI.
This article covers REST API 1.2. The REST API latest version, as well as REST API 2.1 and 2.0, are also available.
See also the REST API Reference.
Important
Use the Clusters API 2.0 for managing clusters programmatically and the Libraries API 2.0 for managing libraries programmatically.
The 1.2 Create an execution context and Run a command APIs continue to be supported.
Important
To access Databricks REST APIs, you must authenticate.
REST API use cases
Start Apache Spark jobs triggered from your existing production systems or from Airflow.
Programmatically bring up a cluster of a certain size at a fixed time of day and then shut it down at night.
API categories
Execution context: create unique variable namespaces where Spark commands can be called.
Command execution: run commands within a specific execution context.
Details
This REST API runs over HTTPS.
For retrieving information, use HTTP GET.
For modifying state, use HTTP POST.
For file upload, use
multipart/form-data
. Otherwise useapplication/json
.The response content type is JSON.
Basic authentication is used to authenticate the user for every API call.
User credentials are base64 encoded and are in the HTTP header for every API call. For example,
Authorization: Basic YWRtaW46YWRtaW4=
. If you usecurl
, alternatively you can store user credentials in a.netrc
file.For more information about using the Databricks REST API, see the Use the Databricks REST API.
Get started
To try out the examples in this article, replace
<databricks-instance>
with the workspace URL of your Databricks deployment.The following examples use curl and a .netrc file. You can adapt these
curl
examples with an HTTP library in your programming language of choice.
API reference
Get the list of clusters
Method and path:
GET /api/1.2/clusters/list
Example
Request:
curl --netrc --request GET \
https://<databricks-instance>/api/1.2/clusters/list
Response:
[
{
"id": "1234-567890-span123",
"name": "MyCluster",
"status": "Terminated",
"driverIp": "",
"jdbcPort": 10000,
"numWorkers":0
},
{
"..."
}
]
Response schema
An array of objects, with each object representing information about a cluster as follows:
Field |
---|
id Type: The ID of the cluster. |
name Type: The name of the cluster. |
status Type: The status of the cluster. One of:
|
driverIp Type: The IP address of the driver. |
jdbcPort Type: The JDBC port number. |
numWorkers Type: The number of workers for the cluster. |
Get information about a cluster
Method and path:
GET /api/1.2/clusters/status
Example
Request:
curl --netrc --get \
https://<databricks-instance>/api/1.2/clusters/status \
--data clusterId=1234-567890-span123
Response:
{
"id": "1234-567890-span123",
"name": "MyCluster",
"status": "Terminated",
"driverIp": "",
"jdbcPort": 10000,
"numWorkers": 0
}
Response schema
An object that represents information about the cluster.
Field |
---|
id Type: The ID of the cluster. |
name Type: The name of the cluster. |
status Type: The status of the cluster. One of:
|
driverIp Type: The IP address of the driver. |
jdbcPort Type: The JDBC port number. |
numWorkers Type: The number of workers for the cluster. |
Restart a cluster
Method and path:
POST /api/1.2/clusters/restart
Create an execution context
Method and path:
POST /api/1.2/contexts/create
Example
Request:
curl --netrc --request POST \
https://<databricks-instance>/api/1.2/contexts/create \
--data clusterId=1234-567890-span123 \
--data language=sql
Response:
{
"id": "1234567890123456789"
}
Get information about an execution context
Method and path:
GET /api/1.2/contexts/status
Example
Request:
curl --netrc https://<databricks-instance>/api/1.2/contexts/status?clusterId=1234-567890-span123&contextId=1234567890123456789
Response:
{
"id": "1234567890123456789",
"status": "Running"
}
Delete an execution context
Method and path:
POST /api/1.2/contexts/destroy
Example
Request:
curl --netrc --request POST \
https://<databricks-instance>/api/1.2/contexts/destroy \
--data clusterId=1234-567890-span123 \
--data contextId=1234567890123456789
Response:
{
"id": "1234567890123456789"
}
Run a command
Method and path:
POST /api/1.2/commands/execute
Example
Request:
curl --netrc --request POST \
https://<databricks-instance>/api/1.2/commands/execute \
--header 'Content-Type: application/json' \
--data @execute-command.json
execute-command.json
:
{
"clusterId": "1234-567890-span123",
"contextId": "1234567890123456789",
"language": "python",
"command": "print('Hello, World!')"
}
Response:
{
"id": "1234ab56-7890-1cde-234f-5abcdef67890"
}
Request schema
Field |
---|
clusterId Type: The ID of the cluster to run the command on. |
contextId Type: The ID of the execution context to run the command within. |
language Type: The language of the command. |
command Type: The command string to run. Specify either command or commandFile. |
commandFile Type: The path to a file containing the command to run. Specify either commandFile or command. |
options Type: An optional map of values used
downstream. For example,
a |
Get information about a command
Method and path:
GET /api/1.2/commands/status
Example
Request:
curl --netrc --get \
https://<databricks-instance>/api/1.2/commands/status \
--data clusterId=1234-567890-span123 \
--data contextId=1234567890123456789 \
--data commandId=1234ab56-7890-1cde-234f-5abcdef67890
Response:
{
"id": "1234ab56-7890-1cde-234f-5abcdef67890",
"status": "Finished",
"results": {
"resultType": "text",
"data": "Hello, World!"
}
}
Request schema
Field |
---|
clusterId Type: The ID of the cluster to get the command information about. |
contextId Type: The ID of the execution context that is associated with the command. |
commandId Type: The ID of the command to get information about. |
Response schema
Field |
---|
id Type: The ID of the command. |
status Type: The status of the command. One of:
|
results Type: The results of the command.
For
For
For
For
For
|
Cancel a command
Method and path:
POST/api/1.2/commands/cancel
Example
Request:
curl --netrc --request POST \
https://<databricks-instance>/api/1.2/commands/cancel \
--data clusterId=1234-567890-span123 \
--data contextId=1234567890123456789 \
--data commandId=1234ab56-7890-1cde-234f-5abcdef67890
Response:
{
"id": "1234ab56-7890-1cde-234f-5abcdef67890"
}
Get the list of libraries for a cluster
Important
This operation is deprecated. Use the Cluster status operation in the Libraries API instead.
Method and path:
GET /api/1.2/libraries/list
Upload a library to a cluster
Important
This operation is deprecated. Use the Install operation in the Libraries API instead.
Method and path:
POST /api/1.2/libraries/upload
Additional examples
The following additional examples provide commands that you can use with curl
or adapt with an HTTP library in your programming language of choice.
Create an execution context
Create an execution context on a specified cluster for a given programming language:
curl --netrc --request POST \
https://<databricks-instance>/api/1.2/contexts/create \
--header 'Content-Type: application/json' \
--data '{ "language": "scala", "clusterId": "1234-567890-span123" }'
Get information about the execution context:
curl --netrc --get \
https://<databricks-instance>/api/1.2/contexts/status \
--data 'clusterId=1234-567890-span123&contextId=1234567890123456789'
Delete the execution context:
curl --netrc --request POST \
https://<databricks-instance>/api/1.2/contexts/destroy \
--header 'Content-Type: application/json' \
--data '{ "contextId": "1234567890123456789", "clusterId": "1234-567890-span123" }'
Run a command
Known limitations: command execution does not support %run
.
Run a command string:
curl --netrc --request POST \
https://<databricks-instance>/api/1.2/commands/execute \
--header 'Content-Type: application/json' \
--data '{ "language": "scala", "clusterId": "1234-567890-span123", "contextId": "1234567890123456789", "command": "sc.parallelize(1 to 10).collect" }'
Run a file:
curl --netrc --request POST \
https://<databricks-instance>/api/1.2/commands/execute \
--header 'Content-Type: multipart/form-data' \
--form language=python \
--form clusterId=1234-567890-span123 \
--form contextId=1234567890123456789 \
--form command=@myfile.py
Show the command’s status and result:
curl --netrc --get \
https://<databricks-instance>/api/1.2/commands/status \
--data 'clusterId=1234-567890-span123&contextId=1234567890123456789&commandId=1234ab56-7890-1cde-234f-5abcdef67890'
Cancel the command:
curl --netrc --request POST \
https://<databricks-instance>/api/1.2/commands/cancel \
--data 'clusterId=1234-567890-span123&contextId=1234567890123456789&commandId=1234ab56-7890-1cde-234f-5abcdef67890' \
Upload and run a Spark JAR
Upload a JAR
Use the REST API (latest) to upload a JAR and attach it to a cluster.
Run a JAR
Create an execution context.
curl --netrc --request POST \ https://<databricks-instance>/api/1.2/contexts/create \ --data "language=scala&clusterId=1234-567890-span123"
{ "id": "1234567890123456789" }
Execute a command that uses your JAR.
curl --netrc --request POST \ https://<databricks-instance>/api/1.2/commands/execute \ --data 'language=scala&clusterId=1234-567890-span123&contextId=1234567890123456789&command=println(com.databricks.apps.logs.chapter1.LogAnalyzer.processLogFile(sc,null,"dbfs:/somefile.log"))'
{ "id": "1234ab56-7890-1cde-234f-5abcdef67890" }
Check on the status of your command. It may not return immediately if you are running a lengthy Spark job.
curl --netrc 'https://<databricks-instance>/api/1.2/commands/status?clusterId=1234-567890-span123&contextId=1234567890123456789&commandId=1234ab56-7890-1cde-234f-5abcdef67890'
{ "id": "1234ab56-7890-1cde-234f-5abcdef67890", "results": { "data": "Content Size Avg: 1234, Min: 1234, Max: 1234", "resultType": "text" }, "status": "Finished" }
Allowed values for
resultType
include:error
image
images
table
text