Cluster API

The Cluster API allows you to create/edit/delete clusters via the API. For the cost information, please see the Databricks pricing page.


Create

Endpoint HTTP Method
2.0/clusters/create POST

Creates a new Spark cluster. This method will acquire new instances from the cloud provider if necessary. This method is asynchronous; the returned cluster_id can be used to poll the cluster status. When this method returns, the cluster will be in a PENDING state. The cluster will be usable once it enters a RUNNING state. Note: Databricks may not be able to acquire some of the requested nodes, due to cloud provider limitations (account limits, spot price, ...) or transient network issues. If Databricks acquires at least 85% of the requested on-demand nodes, cluster creation will succeed. Otherwise the cluster will terminate with an informative error message.

An example request:

{
  "cluster_name": "my-cluster",
  "spark_version": "2.0.x-scala2.10",
  "node_type_id": "r3.xlarge",
  "spark_conf": {
    "spark.speculation": true
  },
  "aws_attributes": {
    "availability": "SPOT",
    "zone_id": "us-west-2a"
  },
  "num_workers": 25
}

See below as an example for an autoscaling cluster. Note that this cluster will start with 2 nodes, the minimum.

{
  "cluster_name": "autoscaling-cluster",
  "spark_version": "2.0.x-scala2.10",
  "node_type_id": "r3.xlarge",
  "autoscale" : {
    "min_workers": 2,
    "max_workers": 50
  }
}

Request Structure

Field Name Type Description
num_workers OR autoscale INT32 OR AutoScale

If num_workers, number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in spark_info will gradually increase from 5 to 10 as the new nodes are provisioned.

If autoscale, parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

cluster_name STRING Cluster name requested by the user. This doesn’t have to be unique. If not specified at creation, the cluster name will be an empty string.
spark_version STRING The Spark version of the cluster, e.g. “1.4.x-ubuntu15.10”. This is an optional parameter at cluster creation. If not specified, a default version of Spark will be used. A list of available Spark versions and the default value can be retrieved by using the Spark Versions API call.
spark_conf An array of SparkConfPair

An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

Example Spark confs: {"spark.speculation": true, "spark.streaming.ui.retainedBatches": 5} or {"spark.driver.extraJavaOptions": "-verbose:gc -XX:+PrintGCDetails"}

aws_attributes AwsAttributes Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.
node_type_id STRING This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads If not specified at cluster creation, a default value will be used. A list of available node types and the default value can be retrieved by using the List Node Types API call.
driver_node_type_id STRING The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.
ssh_public_keys An array of STRING SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.
custom_tags An array of ClusterTag

Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

  • Tags are not supported on legacy node types such as compute-optimized and memory-optimized
  • Currently, Databricks allows at most 45 custom tags
  • Clusters can only reuse cloud resources if the resources’ tags are a subset of the cluster tags
cluster_log_conf ClusterLogConf The configuration for delivering spark logs to a long-term storage destination. Two kinds of destinations (dbfs and s3) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every 5 mins. The destination of driver logs is $destination/$clusterId/driver, while the destination of executor logs is $destination/$clusterId/executor.
spark_env_vars An array of SparkEnvPair

An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e., export X='Y') while launching the driver and workers.

In order to specify an additional set of SPARK_DAEMON_JAVA_OPTS, we recommend appending them to $SPARK_DAEMON_JAVA_OPTS as shown in the example below. This ensures that all default databricks managed environmental variables are included as well.

Example Spark environment variables: {"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"} or {"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}

autotermination_minutes INT32 Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.
enable_elastic_disk BOOL Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

Response Structure

Field Name Type Description
cluster_id STRING  

Start

Endpoint HTTP Method
2.0/clusters/start POST
Starts a terminated Spark cluster given its id. This works similar to createCluster except:
  • The previous cluster id and attributes are preserved.

  • The cluster starts with the last specified cluster size.
    • If the previous cluster was an autoscaling cluster, the current cluster starts with the minimum number of nodes.
  • If the cluster is not currently in a TERMINATED state, nothing will happen.

  • Clusters launched to run a job cannot be started.

An example request:

{
  "cluster_id": "1202-211320-brick1"
}

Request Structure

Field Name Type Description
cluster_id STRING The cluster to be started. This field is required.

Restart

Endpoint HTTP Method
2.0/clusters/restart POST

Restarts a Spark cluster given its id. If the cluster is not currently in a RUNNING state, nothing will happen.

An example request:

{
  "cluster_id": "1202-211320-brick1"
}

Request Structure

Field Name Type Description
cluster_id STRING The cluster to be started. This field is required.

Resize

Endpoint HTTP Method
2.0/clusters/resize POST

Resizes a cluster to have a desired number of workers. This will fail unless the cluster is in a RUNNING state.

An example request:

{
  "cluster_id": "1202-211320-brick1",
  "num_workers": 30
}

Request Structure

Field Name Type Description
num_workers OR autoscale INT32 OR AutoScale

If num_workers, number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in spark_info will gradually increase from 5 to 10 as the new nodes are provisioned.

If autoscale, parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

cluster_id STRING The cluster to be resized. This field is required.

Delete

Endpoint HTTP Method
2.0/clusters/delete POST

Removes a Spark cluster given its id. The cluster is removed asynchronously. Once the deletion has completed, the cluster will be in a TERMINATED state. If the cluster is already in a TERMINATING or TERMINATED state, nothing will happen.

An example request:

{
  "cluster_id": "1202-211320-brick1"
}

Request Structure

Field Name Type Description
cluster_id STRING The cluster to be deleted. This field is required.

Get

Endpoint HTTP Method
2.0/clusters/get GET

Retrieves the information for a cluster given its identifier. Clusters can be described while they are running, or up to 60 days after they are terminated.

An example request:

/clusters/get?cluster_id=1202-211320-brick1

Request Structure

Field Name Type Description
cluster_id STRING The cluster about which to retrieve information. This field is required.

Response Structure

Field Name Type Description
num_workers OR autoscale INT32 OR AutoScale

If num_workers, number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in spark_info will gradually increase from 5 to 10 as the new nodes are provisioned.

If autoscale, parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

cluster_id STRING Canonical identifier for the cluster. This id is retained during cluster restarts and resizes, while each new cluster has a globally unique id.
creator_user_name STRING Creator user name. The field won’t be included in the response if the user has already been deleted.
driver SparkNode Node on which the Spark driver resides. The driver node contains the Spark master and the Databricks application that manages the per-notebook Spark REPLs.
executors An array of SparkNode Nodes on which the Spark executors reside.
spark_context_id INT64 A canonical SparkContext identifier. This value does change when the Spark driver restarts. The pair (cluster_id, spark_context_id) is a globally unique identifier over all Spark contexts.
jdbc_port INT32 Port on which Spark JDBC server is listening, in the driver nod. No service will be listeningon on this port in executor nodes.
cluster_name STRING Cluster name requested by the user. This doesn’t have to be unique. If not specified at creation, the cluster name will be an empty string.
spark_version STRING The Spark version of the cluster, e.g. “1.4.x-ubuntu15.10”. This is an optional parameter at cluster creation. If not specified, a default version of Spark will be used. A list of available Spark versions and the default value can be retrieved by using the Spark Versions API call.
spark_conf An array of SparkConfPair

An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

Example Spark confs: {"spark.speculation": true, "spark.streaming.ui.retainedBatches": 5} or {"spark.driver.extraJavaOptions": "-verbose:gc -XX:+PrintGCDetails"}

aws_attributes AwsAttributes Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.
node_type_id STRING This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads If not specified at cluster creation, a default value will be used. A list of available node types and the default value can be retrieved by using the List Node Types API call.
driver_node_type_id STRING The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.
ssh_public_keys An array of STRING SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.
custom_tags An array of ClusterTag

Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

  • Tags are not supported on legacy node types such as compute-optimized and memory-optimized
  • Currently, Databricks allows at most 45 custom tags
  • Clusters can only reuse cloud resources if the resources’ tags are a subset of the cluster tags
cluster_log_conf ClusterLogConf The configuration for delivering spark logs to a long-term storage destination. Two kinds of destinations (dbfs and s3) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every 5 mins. The destination of driver logs is $destination/$clusterId/driver, while the destination of executor logs is $destination/$clusterId/executor.
spark_env_vars An array of SparkEnvPair

An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e., export X='Y') while launching the driver and workers.

In order to specify an additional set of SPARK_DAEMON_JAVA_OPTS, we recommend appending them to $SPARK_DAEMON_JAVA_OPTS as shown in the example below. This ensures that all default databricks managed environmental variables are included as well.

Example Spark environment variables: {"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"} or {"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}

autotermination_minutes INT32 Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.
enable_elastic_disk BOOL Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.
state ClusterState Current state of the cluster.
state_message STRING A message associated with the most recent state transition (e.g., the reason why the cluster entered a TERMINATED state).
start_time INT64 Time (in epoch milliseconds) when the cluster creation request was received (when the cluster entered a PENDING state).
terminated_time INT64 Time (in epoch milliseconds) when the cluster was terminated, if applicable.
last_state_loss_time INT64 Time when the cluster driver last lost its state (due to a restart or driver failure).
last_activity_time INT64 Time (in epoch milliseconds) when the cluster was last active. A cluster is active if there is at least one command that has not finished on the cluster. This field is available after the cluster has reached a RUNNING state. Updates to this field are made as best-effort attempts. Certain versions of Spark do not support reporting of cluster activity. Please refer to Automatic Termination for details.
cluster_memory_mb INT64 Total amount of cluster memory, in megabytes
cluster_cores FLOAT Number of CPU cores available for this cluster. Note that this can be fractional, e.g. 7.5 cores, since certain node types are configured to share cores between Spark nodes on the same instance.
default_tags An array of ClusterTag

Tags that are added by Databricks regardless of any custom_tags, including:

  • Vendor: Databricks
  • Creator: <username_of_creator>
  • ClusterName: <name_of_cluster>
  • ClusterId: <id_of_cluster>
  • Name: <Databricks internal use>
cluster_log_status LogSyncStatus Cluster log delivery status.
termination_reason TerminationReason Information about why the cluster was terminated. This field only appears when the cluster is in a TERMINATING or TERMINATED state.

List

Endpoint HTTP Method
2.0/clusters/list GET

Returns information about all currently active clusters, and up to 100 most recently terminated clusters in the past 7 days. For example, if there are 5 active clusters and 101 terminated clusters in the past 7 days, it returns the 5 active clusters and 100 terminated clusters. If there are 5 active clusters and 99 terminated clusters in the past 7 days, it returns 5 active clusters and all the 99 terminated clusters.

Response Structure

Field Name Type Description
clusters An array of ClusterInfo  

List Zones

Endpoint HTTP Method
2.0/clusters/list-zones GET

Returns a list of availability zones where clusters can be created in (ex: us-west-2a). These zones can be used to launch a cluster.

Response Structure

Field Name Type Description
zones An array of STRING The list of available zones (e.g., [‘us-west-2c’, ‘us-east-2’]).
default_zone STRING The availability zone if no zone_id is provided in the cluster creation request.

List Node Types

Endpoint HTTP Method
2.0/clusters/list-node-types GET

Returns a list of supported Spark node types. These node types can be used to launch a cluster.

Response Structure

Field Name Type Description
node_types An array of NodeType The list of available Spark node types.
default_node_type_id STRING The default node type. Cluster creation requests that do not explicitly specify a node type will use this node type.

Spark Versions

Endpoint HTTP Method
2.0/clusters/spark-versions GET

Returns the list of available Spark versions. These versions can be used to launch a cluster.

Response Structure

Field Name Type Description
versions An array of SparkVersion All the available Spark versions.
default_version_key STRING Cluster creation requests that do not explicitly specify a Spark version will use this spark version.

Data Structures

AutoScale

Field Name Type Description
min_workers INT32 The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.
max_workers INT32 The maximum number of workers to which the cluster can scale up when overloaded. Note that max_workers must be strictly greater than min_workers.

AwsAttributes

Attributes set during cluster creation which are related to Amazon Web Services.

Field Name Type Description
first_on_demand INT32 The first first_on_demand nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
availability AwsAvailability Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero, this availability type will be used for the entire cluster.
zone_id STRING Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like “us-west-2a”. The provided availability zone must be in the same region as the Databricks deployment. For example, “us-west-2a” is not a valid zone id if the Databricks deployment resides in the “us-east-1” region. This is an optional field at cluster creation, and if not specified, a default zone will be used. The list of available zones as well as the default value can be found by using the List Zones method.
instance_profile_arn STRING

Nodes for this cluster will only be placed on AWS instances with this instance profile. If ommitted, nodes will be placed on instances without an IAM instance profile. The instance profile must have previously been added to the Databricks environment by an account administrator.

This feature may only be available to certain customer plans.

spot_bid_price_percent INT32 The bid price for AWS spot instances, as a percentage of the corresponding instance type’s on-demand price. For example, if this field is set to 50, and the cluster needs a new r3.xlarge spot instance, then the bid price is half of the price of on-demand r3.xlarge instances. Similarly, if this field is set to 200, the bid price is twice the price of on-demand r3.xlarge instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.
ebs_volume_type EbsVolumeType The type of EBS volumes that will be launched with this cluster.
ebs_volume_count INT32

The number of volumes launched for each instance. Users can choose up to 10 volumes. This feature is only enabled for supported node types. Legacy node types cannot specify custom EBS volumes. For node types with no instance store, at least one EBS volume needs to be specified; otherwise, cluster creation will fail.

These EBS volumes will be mounted at /ebs0, /ebs1, and etc. Instance store volumes will be mounted at /local_disk0, /local_disk1, and etc.

If EBS volumes are attached, Databricks will configure Spark to use only the EBS volumes for scratch storage because heterogenously sized scratch devices can lead to inefficient disk utilization. If no EBS volumes are attached, Databricks will configure Spark to use instance store volumes.

Please note that if EBS volumes are specified, then the Spark configuration spark.local.dir will be overridden.

ebs_volume_size INT32 The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096. Custom EBS volumes cannot be specified for the legacy node types (memory-optimized and compute-optimized).

ClusterInfo

Describes all of the metadata about a single Spark cluster in Databricks.

Field Name Type Description
num_workers OR autoscale INT32 OR AutoScale

If num_workers, number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in spark_info will gradually increase from 5 to 10 as the new nodes are provisioned.

If autoscale, parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

cluster_id STRING Canonical identifier for the cluster. This id is retained during cluster restarts and resizes, while each new cluster has a globally unique id.
creator_user_name STRING Creator user name. The field won’t be included in the response if the user has already been deleted.
driver SparkNode Node on which the Spark driver resides. The driver node contains the Spark master and the Databricks application that manages the per-notebook Spark REPLs.
executors An array of SparkNode Nodes on which the Spark executors reside.
spark_context_id INT64 A canonical SparkContext identifier. This value does change when the Spark driver restarts. The pair (cluster_id, spark_context_id) is a globally unique identifier over all Spark contexts.
jdbc_port INT32 Port on which Spark JDBC server is listening, in the driver nod. No service will be listeningon on this port in executor nodes.
cluster_name STRING Cluster name requested by the user. This doesn’t have to be unique. If not specified at creation, the cluster name will be an empty string.
spark_version STRING The Spark version of the cluster, e.g. “1.4.x-ubuntu15.10”. This is an optional parameter at cluster creation. If not specified, a default version of Spark will be used. A list of available Spark versions and the default value can be retrieved by using the Spark Versions API call.
spark_conf An array of SparkConfPair

An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

Example Spark confs: {"spark.speculation": true, "spark.streaming.ui.retainedBatches": 5} or {"spark.driver.extraJavaOptions": "-verbose:gc -XX:+PrintGCDetails"}

aws_attributes AwsAttributes Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.
node_type_id STRING This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads If not specified at cluster creation, a default value will be used. A list of available node types and the default value can be retrieved by using the List Node Types API call.
driver_node_type_id STRING The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.
ssh_public_keys An array of STRING SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.
custom_tags An array of ClusterTag

Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

  • Tags are not supported on legacy node types such as compute-optimized and memory-optimized
  • Currently, Databricks allows at most 45 custom tags
  • Clusters can only reuse cloud resources if the resources’ tags are a subset of the cluster tags
cluster_log_conf ClusterLogConf The configuration for delivering spark logs to a long-term storage destination. Two kinds of destinations (dbfs and s3) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every 5 mins. The destination of driver logs is $destination/$clusterId/driver, while the destination of executor logs is $destination/$clusterId/executor.
spark_env_vars An array of SparkEnvPair

An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e., export X='Y') while launching the driver and workers.

In order to specify an additional set of SPARK_DAEMON_JAVA_OPTS, we recommend appending them to $SPARK_DAEMON_JAVA_OPTS as shown in the example below. This ensures that all default databricks managed environmental variables are included as well.

Example Spark environment variables: {"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"} or {"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}

autotermination_minutes INT32 Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.
enable_elastic_disk BOOL Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.
state ClusterState Current state of the cluster.
state_message STRING A message associated with the most recent state transition (e.g., the reason why the cluster entered a TERMINATED state).
start_time INT64 Time (in epoch milliseconds) when the cluster creation request was received (when the cluster entered a PENDING state).
terminated_time INT64 Time (in epoch milliseconds) when the cluster was terminated, if applicable.
last_state_loss_time INT64 Time when the cluster driver last lost its state (due to a restart or driver failure).
last_activity_time INT64 Time (in epoch milliseconds) when the cluster was last active. A cluster is active if there is at least one command that has not finished on the cluster. This field is available after the cluster has reached a RUNNING state. Updates to this field are made as best-effort attempts. Certain versions of Spark do not support reporting of cluster activity. Please refer to Automatic Termination for details.
cluster_memory_mb INT64 Total amount of cluster memory, in megabytes
cluster_cores FLOAT Number of CPU cores available for this cluster. Note that this can be fractional, e.g. 7.5 cores, since certain node types are configured to share cores between Spark nodes on the same instance.
default_tags An array of ClusterTag

Tags that are added by Databricks regardless of any custom_tags, including:

  • Vendor: Databricks
  • Creator: <username_of_creator>
  • ClusterName: <name_of_cluster>
  • ClusterId: <id_of_cluster>
  • Name: <Databricks internal use>
cluster_log_status LogSyncStatus Cluster log delivery status.
termination_reason TerminationReason Information about why the cluster was terminated. This field only appears when the cluster is in a TERMINATING or TERMINATED state.

ClusterLogConf

Cluster log delivery config

Field Name Type Description
dbfs OR s3 DbfsStorageInfo OR S3StorageInfo

If dbfs, destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }

If s3, destination and either region or endpoint should also be provided. e.g. { "s3": { "destination" : "s3://cluster_log_bucket/prefix", "region" : "us-west-2" } } Cluster iam role is used to access s3, please make sure the cluster iam role in instance_profile_arn has permission to write data to the s3 destination.

ClusterTag

Field Name Type Description
key STRING The key of the tag. The key length must be between 1 and 127 UTF-8 characters, inclusive. For a list of all restrictions, see the AWS docs here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-restrictions
value STRING The value of the tag. The value length must be less than or equal to 255 UTF-8 characters. For a list of all restrictions, see the AWS docs here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-restrictions

DbfsStorageInfo

Field Name Type Description
destination STRING dbfs destination, e.g. dbfs:/my/path

LogSyncStatus

The log delivery status

Field Name Type Description
last_attempted INT64 The timestamp of last attempt. If the last attempt fails, last_exception will contain the exception in the last attempt.
last_exception STRING The exception thrown in the last attempt, it would be null (omitted in the response) if there is no exception in last attempted.

NodeType

A description of a Spark node type including both the dimensions of the node and the instance type on which it will be hosted.

Field Name Type Description
node_type_id STRING Unique identifier for this node type. This field is required.
memory_mb INT32 Memory (in MB) available for this node type. This field is required.
num_cores FLOAT Number of CPU cores available for this node type. Note that this can be fractional, e.g., 2.5 cores, if the the number of cores on a machine instance is not divisible by the number of Spark nodes on that machine. This field is required.
description STRING A string description associated with this node type, e.g., “r3.xlarge”. This field is required.
instance_type_id STRING An identifier for the type of hardware that this node runs on, e.g., “r3.2xlarge” in AWS. This field is required.
is_deprecated BOOL Whether the node type is deprecated. Non-deprecated node types offer greater performance.

ParameterPair

Field Name Type Description
key TerminationParameter  
value STRING  

S3StorageInfo

Field Name Type Description
destination STRING S3 destination, e.g. s3://my-bucket/some-prefix Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs.
region STRING S3 region, e.g. us-west-2. Either region or endpoint needs to be set. If both are set, endpoint will be used.
endpoint STRING S3 endpoint, e.g. https://s3-us-west-2.amazonaws.com. Either region or endpoint needs to be set. If both are set, endpoint will be used.
enable_encryption BOOL (Optional) Flag to enable server side encryption, false by default.
encryption_type STRING (Optional) The encryption type, it could be sse-s3 or sse-kms. It will be used only when encryption is enabled and the default type is sse-s3.
kms_key STRING (Optional) Kms key which will be used if encryption is enabled and encryption type is set to sse-kms.
canned_acl STRING (Optional) Set canned access control list for the logs, e.g. bucket-owner-full-control. If canned_cal is set, please make sure the cluster iam role has s3:PutObjectAcl permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set bucket-owner-full-control to make bucket owner able to read the logs.

SparkConfPair

Spark configuration key-value pairs

Field Name Type Description
key STRING  
value STRING  

SparkEnvPair

Spark environment variable key-value pairs

Field Name Type Description
key STRING  
value STRING  

SparkNode

Describes a specific Spark driver or executor.

Field Name Type Description
private_ip STRING Private IP address (typically a 10.x.x.x address) of the Spark node. Note that this is different from the private IP address of the host instance.
public_dns STRING Public DNS address of this node. This address can be used to access the Spark JDBC server on the driver node. To communicate with the JDBC server, traffic must be manually authorized by adding security group rules to the “worker-unmanaged” security group via the AWS console.
node_id STRING Globally unique identifier for this node.
instance_id STRING Globally unique identifier for the host instance from the cloud provider.
start_timestamp INT64 The timestamp (in millisecond) when the Spark node is launched.
node_aws_attributes SparkNodeAwsAttributes Attributes specific to AWS for a Spark node.
host_private_ip STRING The private IP address of the host instance.

SparkNodeAwsAttributes

Attributes specific to AWS for a Spark node.

Field Name Type Description
is_spot BOOL Whether this node is on an Amazon spot instance.

SparkVersion

Field Name Type Description
key STRING Spark version key, for example “2.1.x-scala2.11”. This is the value which should be provided as the “spark_version” when creating a new cluster. Note that the exact Spark version may change over time for a “wildcard” version (i.e., “2.1.x-scala2.11” is a “wildcard” version) with minor bug fixes.
name STRING A descriptive name for this Spark version, for example “Spark 2.1”.

TerminationReason

Field Name Type Description
code TerminationCode status code indicating why the cluster was terminated
parameters An array of ParameterPair list of parameters that provide additional information about why the cluster was terminated

AwsAvailability

The set of AWS availability types supported when setting up nodes for a cluster.

SPOT Use spot instances.
ON_DEMAND Use on-demand instances.
SPOT_WITH_FALLBACK Preferably use spot instances, but fall back to on-demand instances if spot instances cannot be acquired (e.g., if AWS spot prices are too high).

ClusterState

The state of a Cluster. The current allowable state transitions are as follows:

  • PENDING -> RUNNING
  • PENDING -> TERMINATING
  • RUNNING -> RESIZING
  • RUNNING -> RESTARTING
  • RUNNING -> TERMINATING
  • RESTARTING -> RUNNING
  • RESTARTING -> TERMINATING
  • RESIZING -> RUNNING
  • RESIZING -> TERMINATING
  • TERMINATING -> TERMINATED
PENDING Indicates a cluster that is in progress of being created.
RUNNING Indicates a cluster that has been started and is ready for use.
RESTARTING Indicates that a cluster is in the process of restarting.
RESIZING Indicates that a cluster is in the process of adding or removing nodes.
TERMINATING Indicates that a cluster is in the process of being destroyed.
TERMINATED Indicates a cluster which has been successfully destroyed.
ERROR This state is not used anymore. It was used to indicate a cluster which failed to be created. Terminating and Terminated are used instead.
UNKNOWN Indicates a cluster which is an unknown state. A cluster should never be in this state.

EbsVolumeType

All EBS volume types that Databricks supports. See https://aws.amazon.com/ebs/details/ for details.

GENERAL_PURPOSE_SSD Provision extra storage using AWS gp2 EBS volumes.
THROUGHPUT_OPTIMIZED_HDD Provision extra storage using AWS st1 volumes.

TerminationCode

The status code indicating why the cluster was terminated

USER_REQUEST
A user terminated the cluster directly. Parameters should include a username field
that indicates the specific user who terminated the cluster.
JOB_FINISHED This cluster was launched by a Job, and terminated when the Job completed.
INACTIVITY This cluster was terminated since it was idle.
CLOUD_PROVIDER_SHUTDOWN The instance that hosted the spark driver was terminated by the cloud provider. In AWS, for example, AWS may retire instances and directly shut them down. Parameters should include an aws_instance_state_reason field indicating the AWS-provided reason why the instance was terminated.
COMMUNICATION_LOST Databricks may lose connection to services on the driver instance. One such case is when problems arise in cloud networking infrastructure, or when the instance itself becomes unhealty.
CLOUD_PROVIDER_LAUNCH_FAILURE Databricks may hit cloud provider failures when requesting instances to launch clusters. For example, AWS limits the number of running instances and EBS volumes. If you ask Databricks to launch a cluster that requires instances or EBS volumes that exceed your AWS limit, the cluster will fail with this status code. Parameters should include one of aws_api_error_code, aws_instance_state_reason, or aws_spot_request_status to indicate the AWS-provided reason why Databricks could not request the required instances for the cluster.
SPARK_STARTUP_FAILURE The Spark driver failed to start. Possible reasons may include incompatible libraries and initialization scripts that corrupted the Spark container.
INVALID_ARGUMENT Cannot launch the cluster because the user specified an invalid argument. For example, the use might specify an invalid spark version for the cluster.
UNEXPECTED_LAUNCH_FAILURE While launching this cluster, Databricks failed to complete critical setup steps, terminating the cluster.
INTERNAL_ERROR Databricks encountered an unexpected error which forced the running cluster to be terminated. Please contact Databricks support for additional details.
INSTANCE_UNREACHABLE Databricks was not able to access instances in order to start the cluster. This can be a transient networking issue. If the problem persists, this usually indicates a networking environment misconfiguration.
REQUEST_REJECTED Databricks cannot handle the request at this moment. Please try again later and contact Databricks if the problem persists.

TerminationParameter

Possible keys that provide additional information as to why a cluster was terminated.

username The username of the user who terminated the cluster.
aws_api_error_code The AWS provided error code describing why cluster nodes could not be provisioned. For example, InstanceLimitExceeded indicates that the limit of EC2 instances for a specific instance type has been exceeded. For reference, see: http://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html
aws_instance_state_reason The AWS provided state reason describing why the driver node was terminated. For example, Client.VolumeLimitExceeded indicates that the limit of EBS volumes or total EBS volume storage has been exceeded. For reference, see http://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_StateReason.html,
aws_spot_request_status Describes why a spot request could not be fulfilled. For example, price-too-low indicates that the bid price was below the current spot price. For reference, see: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-bid-status.html#spot-instance-bid-status-understand
aws_spot_request_fault_code Provides additional details when a spot request fails. For example InsufficientFreeAddressesInSubnet indicates the subnet does not have free IP addresses to accommodate the new instance. For reference, see http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-spot-instance-requests.html
aws_impaired_status_details The AWS provided status check which failed and induced a node loss. This status may correspond to a failed instance or system check. For reference, see http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-system-instance-status-check.html
aws_instance_status_event The AWS provided scheduled event (e.g. reboot) which induced a node loss. For reference, see http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-instances-status-check_sched.html
aws_error_message Provides human-readable context of various failures from AWS
databricks_error_message Additional context that may explain the reason for cluster termination.
inactivity_duration_min An idle cluster was shutdown after being inactive for this duration.
instance_id The id of the instance that was hosting the Spark driver.