Cluster

Get GA

GET /api/2.1/clusters/get

Retrieves the information for a cluster given its identifier. Clusters can be described while they are running, or up to 60 days after they are terminated.

API scopes: clusters

Parameters

cluster_idstringquery: The cluster about which to retrieve information.

Response

cluster_idstring: Canonical identifier for the cluster. This id is retained during cluster restarts and resizes, while each new cluster has a globally unique id.

creator_user_namestring: Creator user name. The field won't be included in the response if the user has already been deleted.

statestring: Current state of the cluster.
Values: PENDING, RUNNING, RESTARTING, RESIZING, TERMINATING, TERMINATED, ERROR, UNKNOWN

state_messagestring: A message associated with the most recent state transition (e.g., the reason why the cluster entered a TERMINATED state).

cluster_memory_mbint64: Total amount of cluster memory, in megabytes

cluster_coresfloat: Number of CPU cores available for this cluster. Note that this can be fractional, e.g. 7.5 cores, since certain node types are configured to share cores between Spark nodes on the same instance.

default_tagsobject

Tags that are added by <Databricks> regardless of any custom_tags, including:

Vendor: <Databricks>
Creator: <username_of_creator>
ClusterName: <name_of_cluster>
ClusterId: <id_of_cluster>
Name: <<Databricks> internal use>

cluster_log_statusobject

Cluster log delivery status.

Show child attributesHide child attributes

last_attemptedint64: The timestamp of last attempt. If the last attempt fails, last_exception will contain the exception in the last attempt.

last_exceptionstring: The exception thrown in the last attempt, it would be null (omitted in the response) if there is no exception in last attempted.

termination_reasonobject

Information about why the cluster was terminated. This field only appears when the cluster is in a TERMINATING or TERMINATED state.

Show child attributesHide child attributes

codestring: status code indicating why the cluster was terminated
Values: UNKNOWN, USER_REQUEST, JOB_FINISHED, INACTIVITY, CLOUD_PROVIDER_SHUTDOWN, COMMUNICATION_LOST, CLOUD_PROVIDER_LAUNCH_FAILURE, INIT_SCRIPT_FAILURE, SPARK_STARTUP_FAILURE, INVALID_ARGUMENT, UNEXPECTED_LAUNCH_FAILURE, INTERNAL_ERROR, INSTANCE_UNREACHABLE, REQUEST_REJECTED, TRIAL_EXPIRED, DRIVER_UNREACHABLE, SPARK_ERROR, DRIVER_UNRESPONSIVE, METASTORE_COMPONENT_UNHEALTHY, DBFS_COMPONENT_UNHEALTHY, EXECUTION_COMPONENT_UNHEALTHY, AZURE_RESOURCE_MANAGER_THROTTLING, AZURE_RESOURCE_PROVIDER_THROTTLING, NETWORK_CONFIGURATION_FAILURE, CONTAINER_LAUNCH_FAILURE, INSTANCE_POOL_CLUSTER_FAILURE, SKIPPED_SLOW_NODES, ATTACH_PROJECT_FAILURE, UPDATE_INSTANCE_PROFILE_FAILURE, DATABASE_CONNECTION_FAILURE, REQUEST_THROTTLED, SELF_BOOTSTRAP_FAILURE, GLOBAL_INIT_SCRIPT_FAILURE, SLOW_IMAGE_DOWNLOAD, INVALID_SPARK_IMAGE, NPIP_TUNNEL_TOKEN_FAILURE, HIVE_METASTORE_PROVISIONING_FAILURE, AZURE_INVALID_DEPLOYMENT_TEMPLATE, AZURE_UNEXPECTED_DEPLOYMENT_TEMPLATE_FAILURE, SUBNET_EXHAUSTED_FAILURE, BOOTSTRAP_TIMEOUT, STORAGE_DOWNLOAD_FAILURE, CONTROL_PLANE_REQUEST_FAILURE, BOOTSTRAP_TIMEOUT_CLOUD_PROVIDER_EXCEPTION, AWS_INSUFFICIENT_INSTANCE_CAPACITY_FAILURE, DOCKER_IMAGE_PULL_FAILURE, AZURE_VNET_CONFIGURATION_FAILURE, NPIP_TUNNEL_SETUP_FAILURE, AWS_AUTHORIZATION_FAILURE, NEPHOS_RESOURCE_MANAGEMENT, STS_CLIENT_SETUP_FAILURE, SECURITY_DAEMON_REGISTRATION_EXCEPTION, AWS_REQUEST_LIMIT_EXCEEDED, AWS_INSUFFICIENT_FREE_ADDRESSES_IN_SUBNET_FAILURE, AWS_UNSUPPORTED_FAILURE, AZURE_QUOTA_EXCEEDED_EXCEPTION, AZURE_OPERATION_NOT_ALLOWED_EXCEPTION, NFS_MOUNT_FAILURE, K8S_AUTOSCALING_FAILURE, K8S_DBR_CLUSTER_LAUNCH_TIMEOUT, SPARK_IMAGE_DOWNLOAD_FAILURE, AZURE_VM_EXTENSION_FAILURE, WORKSPACE_CANCELLED_ERROR, AWS_MAX_SPOT_INSTANCE_COUNT_EXCEEDED_FAILURE, TEMPORARILY_UNAVAILABLE, WORKER_SETUP_FAILURE, IP_EXHAUSTION_FAILURE, GCP_QUOTA_EXCEEDED, CLOUD_PROVIDER_RESOURCE_STOCKOUT, GCP_SERVICE_ACCOUNT_DELETED, AZURE_BYOK_KEY_PERMISSION_FAILURE, SPOT_INSTANCE_TERMINATION, AZURE_EPHEMERAL_DISK_FAILURE, ABUSE_DETECTED, IMAGE_PULL_PERMISSION_DENIED, WORKSPACE_CONFIGURATION_ERROR, SECRET_RESOLUTION_ERROR, UNSUPPORTED_INSTANCE_TYPE, CLOUD_PROVIDER_DISK_SETUP_FAILURE, SSH_BOOTSTRAP_FAILURE, AWS_INACCESSIBLE_KMS_KEY_FAILURE, INIT_CONTAINER_NOT_FINISHED, SPARK_IMAGE_DOWNLOAD_THROTTLED, SPARK_IMAGE_NOT_FOUND, CLUSTER_OPERATION_THROTTLED, CLUSTER_OPERATION_TIMEOUT, SERVERLESS_LONG_RUNNING_TERMINATED, AZURE_PACKED_DEPLOYMENT_PARTIAL_FAILURE, INVALID_WORKER_IMAGE_FAILURE, WORKSPACE_UPDATE, INVALID_AWS_PARAMETER, DRIVER_OUT_OF_DISK, DRIVER_OUT_OF_MEMORY, DRIVER_LAUNCH_TIMEOUT, DRIVER_UNEXPECTED_FAILURE, UNEXPECTED_POD_RECREATION, GCP_INACCESSIBLE_KMS_KEY_FAILURE, GCP_KMS_KEY_PERMISSION_DENIED, DRIVER_EVICTION, USER_INITIATED_VM_TERMINATION, GCP_IAM_TIMEOUT, AWS_RESOURCE_QUOTA_EXCEEDED, CLOUD_ACCOUNT_SETUP_FAILURE, AWS_INVALID_KEY_PAIR, DRIVER_POD_CREATION_FAILURE, MAINTENANCE_MODE, INTERNAL_CAPACITY_FAILURE, EXECUTOR_POD_UNSCHEDULED, STORAGE_DOWNLOAD_FAILURE_SLOW, STORAGE_DOWNLOAD_FAILURE_THROTTLED, DYNAMIC_SPARK_CONF_SIZE_EXCEEDED, AWS_INSTANCE_PROFILE_UPDATE_FAILURE, INSTANCE_POOL_NOT_FOUND, INSTANCE_POOL_MAX_CAPACITY_REACHED, AWS_INVALID_KMS_KEY_STATE, GCP_INSUFFICIENT_CAPACITY, GCP_API_RATE_QUOTA_EXCEEDED, GCP_RESOURCE_QUOTA_EXCEEDED, GCP_IP_SPACE_EXHAUSTED, GCP_SERVICE_ACCOUNT_ACCESS_DENIED, GCP_SERVICE_ACCOUNT_NOT_FOUND, GCP_FORBIDDEN, GCP_NOT_FOUND, RESOURCE_USAGE_BLOCKED, DATA_ACCESS_CONFIG_CHANGED, ACCESS_TOKEN_FAILURE, INVALID_INSTANCE_PLACEMENT_PROTOCOL, BUDGET_POLICY_RESOLUTION_FAILURE, IN_PENALTY_BOX, DISASTER_RECOVERY_REPLICATION, BOOTSTRAP_TIMEOUT_DUE_TO_MISCONFIG, INSTANCE_UNREACHABLE_DUE_TO_MISCONFIG, STORAGE_DOWNLOAD_FAILURE_DUE_TO_MISCONFIG, CONTROL_PLANE_REQUEST_FAILURE_DUE_TO_MISCONFIG, CLOUD_PROVIDER_LAUNCH_FAILURE_DUE_TO_MISCONFIG, GCP_SUBNET_NOT_READY, CLOUD_OPERATION_CANCELLED, CLOUD_PROVIDER_INSTANCE_NOT_LAUNCHED, GCP_TRUSTED_IMAGE_PROJECTS_VIOLATED, BUDGET_POLICY_LIMIT_ENFORCEMENT_ACTIVATED, EOS_SPARK_IMAGE, NO_MATCHED_K8S, LAZY_ALLOCATION_TIMEOUT, DRIVER_NODE_UNREACHABLE, SECRET_CREATION_FAILURE, POD_SCHEDULING_FAILURE, POD_ASSIGNMENT_FAILURE, ALLOCATION_TIMEOUT, ALLOCATION_TIMEOUT_NO_UNALLOCATED_CLUSTERS, ALLOCATION_TIMEOUT_NO_MATCHED_CLUSTERS, ALLOCATION_TIMEOUT_NO_READY_CLUSTERS, ALLOCATION_TIMEOUT_NO_WARMED_UP_CLUSTERS, ALLOCATION_TIMEOUT_NODE_DAEMON_NOT_READY, ALLOCATION_TIMEOUT_NO_HEALTHY_CLUSTERS, NETVISOR_SETUP_TIMEOUT, NO_MATCHED_K8S_TESTING_TAG, CLOUD_PROVIDER_RESOURCE_STOCKOUT_DUE_TO_MISCONFIG, GKE_BASED_CLUSTER_TERMINATION, ALLOCATION_TIMEOUT_NO_HEALTHY_AND_WARMED_UP_CLUSTERS, DOCKER_INVALID_OS_EXCEPTION, DOCKER_CONTAINER_CREATION_EXCEPTION, DOCKER_IMAGE_TOO_LARGE_FOR_INSTANCE_EXCEPTION, DNS_RESOLUTION_ERROR, GCP_DENIED_BY_ORG_POLICY, SECRET_PERMISSION_DENIED, NETWORK_CHECK_NIC_FAILURE, NETWORK_CHECK_DNS_SERVER_FAILURE, NETWORK_CHECK_STORAGE_FAILURE, NETWORK_CHECK_METADATA_ENDPOINT_FAILURE, NETWORK_CHECK_CONTROL_PLANE_FAILURE, NETWORK_CHECK_MULTIPLE_COMPONENTS_FAILURE, DRIVER_UNHEALTHY, USAGE_POLICY_ENTITLEMENT_DENIED, K8S_ACTIVE_POD_QUOTA_EXCEEDED, CLOUD_ACCOUNT_POD_QUOTA_EXCEEDED, NETWORK_CHECK_NIC_FAILURE_DUE_TO_MISCONFIG, NETWORK_CHECK_DNS_SERVER_FAILURE_DUE_TO_MISCONFIG, NETWORK_CHECK_STORAGE_FAILURE_DUE_TO_MISCONFIG, NETWORK_CHECK_METADATA_ENDPOINT_FAILURE_DUE_TO_MISCONFIG, NETWORK_CHECK_CONTROL_PLANE_FAILURE_DUE_TO_MISCONFIG, NETWORK_CHECK_MULTIPLE_COMPONENTS_FAILURE_DUE_TO_MISCONFIG, DBR_IMAGE_RESOLUTION_FAILURE, CONTROL_PLANE_CONNECTION_FAILURE, CONTROL_PLANE_CONNECTION_FAILURE_DUE_TO_MISCONFIG, RATE_LIMITED, MTLS_PORT_CONNECTIVITY_FAILURE, HIVEMETASTORE_CONNECTIVITY_FAILURE

typestring: type of the termination
Values: SUCCESS, CLIENT_ERROR, SERVICE_FAULT, CLOUD_FAILURE

parametersobject: list of parameters that provide additional information about why the cluster was terminated

specobject

The spec contains a snapshot of the latest user specified settings that were used to create/edit the cluster. Note: not included in the response of the ListClusters API.

Show child attributesHide child attributes

apply_policy_default_valuesboolean: When set to true, fixed and default values from the policy will be used for fields that are omitted. When set to false, only fixed values from the policy will be applied.
Default: false

cluster_namestring: Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.

spark_versionstring: The Spark version of the cluster, e.g. 3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

aws_attributesobject

Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 0

availabilitystring: Default: SPOT_WITH_FALLBACK
Values: SPOT, ON_DEMAND, SPOT_WITH_FALLBACK

zone_idstring

Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the <Databricks> deployment. For example, "us-west-2a" is not a valid zone id if the <Databricks> deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, the zone "auto" will be used. If the zone specified is "auto", will try to place cluster in a zone with high availability, and will retry placement in a different AZ if there is not enough capacity.

The list of available zones as well as the default value can be found by using the List Zones method.

instance_profile_arnstring

Nodes for this cluster will only be placed on AWS instances with this instance profile. If ommitted, nodes will be placed on instances without an IAM instance profile. The instance profile must have previously been added to the <Databricks> environment by an account administrator.

This feature may only be available to certain customer plans.

spot_bid_price_percentint32: The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new r3.xlarge spot instance, then the bid price is half of the price of on-demand r3.xlarge instances. Similarly, if this field is set to 200, the bid price is twice the price of on-demand r3.xlarge instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.
Default: 100

ebs_volume_typestring: The type of EBS volumes that will be launched with this cluster.
Values: GENERAL_PURPOSE_SSD, THROUGHPUT_OPTIMIZED_HDD

ebs_volume_countint32

The number of volumes launched for each instance. Users can choose up to 10 volumes. This feature is only enabled for supported node types. Legacy node types cannot specify custom EBS volumes. For node types with no instance store, at least one EBS volume needs to be specified; otherwise, cluster creation will fail.

These EBS volumes will be mounted at /ebs0, /ebs1, and etc. Instance store volumes will be mounted at /local_disk0, /local_disk1, and etc.

If EBS volumes are attached, <Databricks> will configure Spark to use only the EBS volumes for scratch storage because heterogenously sized scratch devices can lead to inefficient disk utilization. If no EBS volumes are attached, <Databricks> will configure Spark to use instance store volumes.

Please note that if EBS volumes are specified, then the Spark configuration spark.local.dir will be overridden.

Default: 0

ebs_volume_sizeint32: The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.

ebs_volume_iopsint32: If using gp3 volumes, what IOPS to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.

ebs_volume_throughputint32: If using gp3 volumes, what throughput to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.

azure_attributesobject

Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

log_analytics_infoobject

Defines values necessary to configure and run Azure Log Analytics agent

Show child attributesHide child attributes

log_analytics_workspace_idstring

log_analytics_primary_keystring

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 1

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero, this availability type will be used for the entire cluster.
Default: ON_DEMAND_AZURE
Values: SPOT_AZURE, ON_DEMAND_AZURE, SPOT_WITH_FALLBACK_AZURE

spot_bid_max_pricedouble: The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.
Default: -1

capacity_reservation_groupstring

The Azure capacity reservation group resource ID to use for launching VMs. When specified, VMs will be launched using the provided capacity reservation.

Capacity reservations can only be specified when the workspace uses injected vnet (i.e. customer defined vnet not managed by databricks). Ensure the databricks-login-prod Enterprise Application is granted the following four permissions:

Microsoft.Compute/capacityReservationGroups/read
Microsoft.Compute/capacityReservationGroups/deploy/action
Microsoft.Compute/capacityReservationGroups/capacityReservations/read
Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action

Format: /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}

gcp_attributesobject

Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

use_preemptible_executorsboolean: This field determines whether the spark executors will be scheduled to run on preemptible VMs (when set to true) versus standard compute engine VMs (when set to false; default). Note: Soon to be deprecated, use the 'availability' field instead.
Default: false

google_service_accountstring: If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.

boot_disk_sizeint32: Boot disk size in GB

availabilitystring: This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
Default: ON_DEMAND_GCP
Values: PREEMPTIBLE_GCP, ON_DEMAND_GCP, PREEMPTIBLE_WITH_FALLBACK_GCP

zone_idstring

Identifier for the availability zone in which the cluster resides. This can be one of the following:

"HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
"AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.

Default: HA

local_ssd_countint32: If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to GCP documentation for the supported number of local SSDs for each instance type.

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 1

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring

The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence.

worker_node_type_flexibilityobject

Flexible node type configuration for worker nodes.

Show child attributesHide child attributes

alternate_node_type_idsarray of string: A list of node type IDs to use as fallbacks when the primary node type is unavailable.

driver_node_type_flexibilityobject

Flexible node type configuration for the driver node.

Show child attributesHide child attributes

alternate_node_type_idsarray of string: A list of node type IDs to use as fallbacks when the primary node type is unavailable.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject

The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every 5 mins. The destination of driver logs is $destination/$clusterId/driver, while the destination of executor logs is $destination/$clusterId/executor.

Show child attributesHide child attributes

dbfsobjectRequired

destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }

Show child attributesHide child attributes

destinationstring: dbfs destination, e.g. dbfs:/my/path

s3objectRequired

destination and either the region or endpoint need to be provided. e.g. { "s3": { "destination" : "s3://cluster_log_bucket/prefix", "region" : "us-west-2" } } Cluster iam role is used to access s3, please make sure the cluster iam role in instance_profile_arn has permission to write data to the s3 destination.

Show child attributesHide child attributes

destinationstring: S3 destination, e.g. s3://my-bucket/some-prefix Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs.

regionstring: S3 region, e.g. us-west-2. Either region or endpoint needs to be set. If both are set, endpoint will be used.

endpointstring: S3 endpoint, e.g. https://s3-us-west-2.amazonaws.com. Either region or endpoint needs to be set. If both are set, endpoint will be used.

enable_encryptionboolean: (Optional) Flag to enable server side encryption, false by default.

encryption_typestring: (Optional) The encryption type, it could be sse-s3 or sse-kms. It will be used only when encryption is enabled and the default type is sse-s3.

kms_keystring: (Optional) Kms key which will be used if encryption is enabled and encryption type is set to sse-kms.

canned_aclstring: (Optional) Set canned access control list for the logs, e.g. bucket-owner-full-control. If canned_cal is set, please make sure the cluster iam role has s3:PutObjectAcl permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set bucket-owner-full-control to make bucket owner able to read the logs.

volumesobjectRequired

destination needs to be provided, e.g. { "volumes": { "destination": "/Volumes/catalog/schema/volume/cluster_log" } }

Show child attributesHide child attributes

destinationstring: UC Volumes destination, e.g. /Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh or dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh

spark_env_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e., export X='Y') while launching the driver and workers.

In order to specify an additional set of SPARK_DAEMON_JAVA_OPTS, we recommend appending them to $SPARK_DAEMON_JAVA_OPTS as shown in the example below. This ensures that all default databricks managed environmental variables are included as well.

Example Spark environment variables: {"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"} or {"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}

autotermination_minutesint32: Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.

enable_elastic_diskboolean: Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWS
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

init_scriptsarray of object

The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If cluster_log_conf is specified, init script logs are sent to <destination>/<cluster-ID>/init_scripts.

Show child attributesHide child attributes

dbfsobjectRequired

destination needs to be provided. e.g. { "dbfs": { "destination" : "dbfs:/home/cluster_log" } }

Show child attributesHide child attributes

destinationstring: dbfs destination, e.g. dbfs:/my/path

s3objectRequired

destination and either the region or endpoint need to be provided. e.g. { \"s3\": { \"destination\": \"s3://cluster_log_bucket/prefix\", \"region\": \"us-west-2\" } } Cluster iam role is used to access s3, please make sure the cluster iam role in instance_profile_arn has permission to write data to the s3 destination.

Show child attributesHide child attributes

destinationstring: S3 destination, e.g. s3://my-bucket/some-prefix Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs.

regionstring: S3 region, e.g. us-west-2. Either region or endpoint needs to be set. If both are set, endpoint will be used.

endpointstring: S3 endpoint, e.g. https://s3-us-west-2.amazonaws.com. Either region or endpoint needs to be set. If both are set, endpoint will be used.

enable_encryptionboolean: (Optional) Flag to enable server side encryption, false by default.

encryption_typestring: (Optional) The encryption type, it could be sse-s3 or sse-kms. It will be used only when encryption is enabled and the default type is sse-s3.

kms_keystring: (Optional) Kms key which will be used if encryption is enabled and encryption type is set to sse-kms.

canned_aclstring: (Optional) Set canned access control list for the logs, e.g. bucket-owner-full-control. If canned_cal is set, please make sure the cluster iam role has s3:PutObjectAcl permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set bucket-owner-full-control to make bucket owner able to read the logs.

fileobjectRequired

destination needs to be provided, e.g. { "file": { "destination": "file:/my/local/file.sh" } }

Show child attributesHide child attributes

destinationstring: local file destination, e.g. file:/my/local/file.sh

gcsobjectRequired

destination needs to be provided, e.g. { "gcs": { "destination": "gs://my-bucket/file.sh" } }

Show child attributesHide child attributes

destinationstring: GCS destination/URI, e.g. gs://my-bucket/some-prefix

abfssobjectRequired

destination needs to be provided, e.g. abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>

Show child attributesHide child attributes

destinationstring: abfss destination, e.g. abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>.

workspaceobjectRequired

destination needs to be provided, e.g. { "workspace": { "destination": "/cluster-init-scripts/setup-datadog.sh" } }

Show child attributesHide child attributes

destinationstring: wsfs destination, e.g. workspace:/cluster-init-scripts/setup-datadog.sh

volumesobjectRequired

destination needs to be provided. e.g. { \"volumes\" : { \"destination\" : \"/Volumes/my-init.sh\" } }

Show child attributesHide child attributes

destinationstring: UC Volumes destination, e.g. /Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh or dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh

docker_imageobject

Custom docker image BYOC

Show child attributesHide child attributes

urlstring: URL of the docker image.

basic_authobjectRequired

Basic auth with username and password

Show child attributesHide child attributes

usernamestring: Name of the user

passwordstring: Password of the user

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

single_user_namestring: Single user name if data_security_mode is SINGLE_USER

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable LUKS on cluster VMs' local disks

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

workload_typeobject

Show child attributesHide child attributes

clientsobject

defined what type of clients can use the cluster. E.g. Notebooks, Jobs

Show child attributesHide child attributes

notebooksboolean: With notebooks set, this cluster can be used for notebooks
Default: true

jobsboolean: With jobs set, the cluster can be used for jobs
Default: true

data_security_modestring: Values: NONE, SINGLE_USER, USER_ISOLATION, LEGACY_TABLE_ACL, LEGACY_PASSTHROUGH, LEGACY_SINGLE_USER, LEGACY_SINGLE_USER_STANDARD, DATA_SECURITY_MODE_STANDARD, DATA_SECURITY_MODE_DEDICATED, DATA_SECURITY_MODE_AUTO

runtime_enginestring

Determines the cluster's runtime engine, either standard or Photon.

This field is not compatible with legacy spark_version values that contain -photon-. Remove -photon- from the spark_version and set runtime_engine to PHOTON.

If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.

Values: NULL, STANDARD, PHOTON

kindstring: Values: COMPUTE_KIND_UNSPECIFIED, CLASSIC_PREVIEW

use_ml_runtimeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

effective_spark_version is determined by spark_version (DBR release), this field use_ml_runtime, and whether node_type_id is gpu node or not.

is_single_nodeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

When set to true, <Databricks> will automatically set single node related custom_tags, spark_conf, and num_workers

remote_disk_throughputint32: If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.

total_initial_remote_disk_sizeint32: If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.

dependency_modestringBeta: Controls dependency configuration for the cluster.
Values: DEPENDENCY_MODE_UNSPECIFIED, DEPENDENCY_MODE_ENVIRONMENTS, DEPENDENCY_MODE_CLUSTER_LIBRARIES, DEPENDENCY_MODE_AUTO

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in spark_info will gradually increase from 5 to 10 as the new nodes are provisioned.

autoscaleobjectRequired

Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

Show child attributesHide child attributes

min_workersint32: The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.

max_workersint32: The maximum number of workers to which the cluster can scale up when overloaded. Note that max_workers must be strictly greater than min_workers.

driverobject

Node on which the Spark driver resides. The driver node contains the Spark master and the <Databricks> application that manages the per-notebook Spark REPLs.

Show child attributesHide child attributes

private_ipstring: Private IP address (typically a 10.x.x.x address) of the Spark node. Note that this is different from the private IP address of the host instance.

public_dnsstring: Public DNS address of this node. This address can be used to access the Spark JDBC server on the driver node. To communicate with the JDBC server, traffic must be manually authorized by adding security group rules to the "worker-unmanaged" security group via the AWS console.

node_idstring: Globally unique identifier for this node.

instance_idstring: Globally unique identifier for the host instance from the cloud provider.

start_timestampint64: The timestamp (in millisecond) when the Spark node is launched.

node_aws_attributesobject

Attributes specific to AWS for a Spark node.

Show child attributesHide child attributes

is_spotboolean: Whether this node is on an Amazon spot instance.

host_private_ipstring: The private IP address of the host instance.

executorsarray of object

Nodes on which the Spark executors reside.

Show child attributesHide child attributes

private_ipstring: Private IP address (typically a 10.x.x.x address) of the Spark node. Note that this is different from the private IP address of the host instance.

public_dnsstring: Public DNS address of this node. This address can be used to access the Spark JDBC server on the driver node. To communicate with the JDBC server, traffic must be manually authorized by adding security group rules to the "worker-unmanaged" security group via the AWS console.

node_idstring: Globally unique identifier for this node.

instance_idstring: Globally unique identifier for the host instance from the cloud provider.

start_timestampint64: The timestamp (in millisecond) when the Spark node is launched.

node_aws_attributesobject

Attributes specific to AWS for a Spark node.

Show child attributesHide child attributes

is_spotboolean: Whether this node is on an Amazon spot instance.

host_private_ipstring: The private IP address of the host instance.

spark_context_idint64: A canonical SparkContext identifier. This value does change when the Spark driver restarts. The pair (cluster_id, spark_context_id) is a globally unique identifier over all Spark contexts.

jdbc_portint32: Port on which Spark JDBC server is listening, in the driver nod. No service will be listeningon on this port in executor nodes.

cluster_namestring: Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.

spark_versionstring: The Spark version of the cluster, e.g. 3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

aws_attributesobject

Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 0

availabilitystring: Default: SPOT_WITH_FALLBACK
Values: SPOT, ON_DEMAND, SPOT_WITH_FALLBACK

zone_idstring

The list of available zones as well as the default value can be found by using the List Zones method.

instance_profile_arnstring

This feature may only be available to certain customer plans.

spot_bid_price_percentint32: The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new r3.xlarge spot instance, then the bid price is half of the price of on-demand r3.xlarge instances. Similarly, if this field is set to 200, the bid price is twice the price of on-demand r3.xlarge instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.
Default: 100

ebs_volume_typestring: The type of EBS volumes that will be launched with this cluster.
Values: GENERAL_PURPOSE_SSD, THROUGHPUT_OPTIMIZED_HDD

ebs_volume_countint32

These EBS volumes will be mounted at /ebs0, /ebs1, and etc. Instance store volumes will be mounted at /local_disk0, /local_disk1, and etc.

Please note that if EBS volumes are specified, then the Spark configuration spark.local.dir will be overridden.

Default: 0

ebs_volume_sizeint32: The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.

ebs_volume_iopsint32: If using gp3 volumes, what IOPS to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.

ebs_volume_throughputint32: If using gp3 volumes, what throughput to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.

azure_attributesobject

Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

log_analytics_infoobject

Defines values necessary to configure and run Azure Log Analytics agent

Show child attributesHide child attributes

log_analytics_workspace_idstring

log_analytics_primary_keystring

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 1

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero, this availability type will be used for the entire cluster.
Default: ON_DEMAND_AZURE
Values: SPOT_AZURE, ON_DEMAND_AZURE, SPOT_WITH_FALLBACK_AZURE

spot_bid_max_pricedouble: The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.
Default: -1

capacity_reservation_groupstring

The Azure capacity reservation group resource ID to use for launching VMs. When specified, VMs will be launched using the provided capacity reservation.

Microsoft.Compute/capacityReservationGroups/read
Microsoft.Compute/capacityReservationGroups/deploy/action
Microsoft.Compute/capacityReservationGroups/capacityReservations/read
Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action

Format: /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}

gcp_attributesobject

Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

use_preemptible_executorsboolean: This field determines whether the spark executors will be scheduled to run on preemptible VMs (when set to true) versus standard compute engine VMs (when set to false; default). Note: Soon to be deprecated, use the 'availability' field instead.
Default: false

google_service_accountstring: If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.

boot_disk_sizeint32: Boot disk size in GB

availabilitystring: This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
Default: ON_DEMAND_GCP
Values: PREEMPTIBLE_GCP, ON_DEMAND_GCP, PREEMPTIBLE_WITH_FALLBACK_GCP

zone_idstring

Identifier for the availability zone in which the cluster resides. This can be one of the following:

"HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
"AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.

Default: HA

local_ssd_countint32: If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to GCP documentation for the supported number of local SSDs for each instance type.

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 1

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring

The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

worker_node_type_flexibilityobject

Flexible node type configuration for worker nodes.

Show child attributesHide child attributes

alternate_node_type_idsarray of string: A list of node type IDs to use as fallbacks when the primary node type is unavailable.

driver_node_type_flexibilityobject

Flexible node type configuration for the driver node.

Show child attributesHide child attributes

alternate_node_type_idsarray of string: A list of node type IDs to use as fallbacks when the primary node type is unavailable.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject

Show child attributesHide child attributes

dbfsobjectRequired

destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }

Show child attributesHide child attributes

destinationstring: dbfs destination, e.g. dbfs:/my/path

s3objectRequired

Show child attributesHide child attributes

destinationstring: S3 destination, e.g. s3://my-bucket/some-prefix Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs.

regionstring: S3 region, e.g. us-west-2. Either region or endpoint needs to be set. If both are set, endpoint will be used.

endpointstring: S3 endpoint, e.g. https://s3-us-west-2.amazonaws.com. Either region or endpoint needs to be set. If both are set, endpoint will be used.

enable_encryptionboolean: (Optional) Flag to enable server side encryption, false by default.

encryption_typestring: (Optional) The encryption type, it could be sse-s3 or sse-kms. It will be used only when encryption is enabled and the default type is sse-s3.

kms_keystring: (Optional) Kms key which will be used if encryption is enabled and encryption type is set to sse-kms.

canned_aclstring: (Optional) Set canned access control list for the logs, e.g. bucket-owner-full-control. If canned_cal is set, please make sure the cluster iam role has s3:PutObjectAcl permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set bucket-owner-full-control to make bucket owner able to read the logs.

volumesobjectRequired

destination needs to be provided, e.g. { "volumes": { "destination": "/Volumes/catalog/schema/volume/cluster_log" } }

Show child attributesHide child attributes

destinationstring: UC Volumes destination, e.g. /Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh or dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh

spark_env_varsobject

autotermination_minutesint32: Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.

enable_elastic_diskboolean: Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWS
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

init_scriptsarray of object

Show child attributesHide child attributes

dbfsobjectRequired

destination needs to be provided. e.g. { "dbfs": { "destination" : "dbfs:/home/cluster_log" } }

Show child attributesHide child attributes

destinationstring: dbfs destination, e.g. dbfs:/my/path

s3objectRequired

Show child attributesHide child attributes

destinationstring: S3 destination, e.g. s3://my-bucket/some-prefix Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs.

regionstring: S3 region, e.g. us-west-2. Either region or endpoint needs to be set. If both are set, endpoint will be used.

endpointstring: S3 endpoint, e.g. https://s3-us-west-2.amazonaws.com. Either region or endpoint needs to be set. If both are set, endpoint will be used.

enable_encryptionboolean: (Optional) Flag to enable server side encryption, false by default.

encryption_typestring: (Optional) The encryption type, it could be sse-s3 or sse-kms. It will be used only when encryption is enabled and the default type is sse-s3.

kms_keystring: (Optional) Kms key which will be used if encryption is enabled and encryption type is set to sse-kms.

canned_aclstring: (Optional) Set canned access control list for the logs, e.g. bucket-owner-full-control. If canned_cal is set, please make sure the cluster iam role has s3:PutObjectAcl permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set bucket-owner-full-control to make bucket owner able to read the logs.

fileobjectRequired

destination needs to be provided, e.g. { "file": { "destination": "file:/my/local/file.sh" } }

Show child attributesHide child attributes

destinationstring: local file destination, e.g. file:/my/local/file.sh

gcsobjectRequired

destination needs to be provided, e.g. { "gcs": { "destination": "gs://my-bucket/file.sh" } }

Show child attributesHide child attributes

destinationstring: GCS destination/URI, e.g. gs://my-bucket/some-prefix

abfssobjectRequired

destination needs to be provided, e.g. abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>

Show child attributesHide child attributes

destinationstring: abfss destination, e.g. abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>.

workspaceobjectRequired

destination needs to be provided, e.g. { "workspace": { "destination": "/cluster-init-scripts/setup-datadog.sh" } }

Show child attributesHide child attributes

destinationstring: wsfs destination, e.g. workspace:/cluster-init-scripts/setup-datadog.sh

volumesobjectRequired

destination needs to be provided. e.g. { \"volumes\" : { \"destination\" : \"/Volumes/my-init.sh\" } }

Show child attributesHide child attributes

destinationstring: UC Volumes destination, e.g. /Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh or dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh

docker_imageobject

Custom docker image BYOC

Show child attributesHide child attributes

urlstring: URL of the docker image.

basic_authobjectRequired

Basic auth with username and password

Show child attributesHide child attributes

usernamestring: Name of the user

passwordstring: Password of the user

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

single_user_namestring: Single user name if data_security_mode is SINGLE_USER

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable LUKS on cluster VMs' local disks

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

workload_typeobject

Show child attributesHide child attributes

clientsobject

defined what type of clients can use the cluster. E.g. Notebooks, Jobs

Show child attributesHide child attributes

notebooksboolean: With notebooks set, this cluster can be used for notebooks
Default: true

jobsboolean: With jobs set, the cluster can be used for jobs
Default: true

data_security_modestring: Values: NONE, SINGLE_USER, USER_ISOLATION, LEGACY_TABLE_ACL, LEGACY_PASSTHROUGH, LEGACY_SINGLE_USER, LEGACY_SINGLE_USER_STANDARD, DATA_SECURITY_MODE_STANDARD, DATA_SECURITY_MODE_DEDICATED, DATA_SECURITY_MODE_AUTO

runtime_enginestring

Determines the cluster's runtime engine, either standard or Photon.

This field is not compatible with legacy spark_version values that contain -photon-. Remove -photon- from the spark_version and set runtime_engine to PHOTON.

If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.

Values: NULL, STANDARD, PHOTON

kindstring: Values: COMPUTE_KIND_UNSPECIFIED, CLASSIC_PREVIEW

use_ml_runtimeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

effective_spark_version is determined by spark_version (DBR release), this field use_ml_runtime, and whether node_type_id is gpu node or not.

is_single_nodeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

When set to true, <Databricks> will automatically set single node related custom_tags, spark_conf, and num_workers

remote_disk_throughputint32: If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.

total_initial_remote_disk_sizeint32: If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.

dependency_modestringBeta: Controls dependency configuration for the cluster.
Values: DEPENDENCY_MODE_UNSPECIFIED, DEPENDENCY_MODE_ENVIRONMENTS, DEPENDENCY_MODE_CLUSTER_LIBRARIES, DEPENDENCY_MODE_AUTO

start_timeint64: Time (in epoch milliseconds) when the cluster creation request was received (when the cluster entered a PENDING state).

terminated_timeint64: Time (in epoch milliseconds) when the cluster was terminated, if applicable.

last_state_loss_timeint64: Time when the cluster driver last lost its state (due to a restart or driver failure).

last_restarted_timeint64: the timestamp that the cluster was started/restarted

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired

Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

Show child attributesHide child attributes

min_workersint32: The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.

max_workersint32: The maximum number of workers to which the cluster can scale up when overloaded. Note that max_workers must be strictly greater than min_workers.

View as Markdown

Response

{
  "autotermination_minutes": 120,
  "aws_attributes": {
    "availability": "SPOT_WITH_FALLBACK",
    "ebs_volume_count": 0,
    "first_on_demand": 1,
    "spot_bid_price_percent": 100,
    "zone_id": "us-west-2c"
  },
  "cluster_id": "1234-567890-reef123",
  "cluster_name": "my-cluster",
  "cluster_source": "UI",
  "creator_user_name": "someone@example.com",
  "default_tags": {
    "ClusterId": "1234-567890-reef123",
    "ClusterName": "my-cluster",
    "Creator": "someone@example.com",
    "Vendor": "Databricks"
  },
  "disk_spec": {
    "disk_count": 0
  },
  "driver_instance_source": {
    "node_type_id": "i3.xlarge"
  },
  "driver_node_type_id": "i3.xlarge",
  "enable_elastic_disk": false,
  "enable_local_disk_encryption": false,
  "init_scripts_safe_mode": false,
  "instance_source": {
    "node_type_id": "i3.xlarge"
  },
  "last_state_loss_time": 1619739324740,
  "node_type_id": "i3.xlarge",
  "num_workers": 30,
  "spark_context_id": 4020997813441461760,
  "spark_version": "13.3.x-scala2.12",
  "spec": {
    "spark_version": "13.3.x-scala2.12"
  },
  "start_time": 1618263108824,
  "state": "TERMINATED",
  "state_message": "Inactive cluster terminated (inactive for 120 minutes).",
  "terminated_time": 1619746525713,
  "termination_reason": {
    "code": "INACTIVITY",
    "parameters": {
      "inactivity_duration_min": "120"
    },
    "type": "SUCCESS"
  }
}

List GA

GET /api/2.1/clusters/list

Return information about all pinned and active clusters, and all clusters terminated within the last 30 days. Clusters terminated prior to this period are not included.

API scopes: clusters

Parameters

page_tokenstringquery: Use next_page_token or prev_page_token returned from the previous request to list the next or previous page of clusters respectively.
Example: CAEaFDAzMTUtMTczODQ1LTcwdWExbDhw

page_sizeint32query: Use this field to specify the maximum number of results to be returned by the server. The server may further constrain the maximum number of results returned in a single page.
Default: 20
Constraints: [ 1 .. 100 ]

Response

clustersarray of object

Show child attributesHide child attributes

cluster_idstring: Canonical identifier for the cluster. This id is retained during cluster restarts and resizes, while each new cluster has a globally unique id.

creator_user_namestring: Creator user name. The field won't be included in the response if the user has already been deleted.

statestring: Current state of the cluster.
Values: PENDING, RUNNING, RESTARTING, RESIZING, TERMINATING, TERMINATED, ERROR, UNKNOWN

state_messagestring: A message associated with the most recent state transition (e.g., the reason why the cluster entered a TERMINATED state).

cluster_memory_mbint64: Total amount of cluster memory, in megabytes

cluster_coresfloat: Number of CPU cores available for this cluster. Note that this can be fractional, e.g. 7.5 cores, since certain node types are configured to share cores between Spark nodes on the same instance.

default_tagsobject

Tags that are added by <Databricks> regardless of any custom_tags, including:

Vendor: <Databricks>
Creator: <username_of_creator>
ClusterName: <name_of_cluster>
ClusterId: <id_of_cluster>
Name: <<Databricks> internal use>

cluster_log_statusobject

Cluster log delivery status.

Show child attributesHide child attributes

last_attemptedint64: The timestamp of last attempt. If the last attempt fails, last_exception will contain the exception in the last attempt.

last_exceptionstring: The exception thrown in the last attempt, it would be null (omitted in the response) if there is no exception in last attempted.

termination_reasonobject

Information about why the cluster was terminated. This field only appears when the cluster is in a TERMINATING or TERMINATED state.

Show child attributesHide child attributes

codestring: status code indicating why the cluster was terminated
Values: UNKNOWN, USER_REQUEST, JOB_FINISHED, INACTIVITY, CLOUD_PROVIDER_SHUTDOWN, COMMUNICATION_LOST, CLOUD_PROVIDER_LAUNCH_FAILURE, INIT_SCRIPT_FAILURE, SPARK_STARTUP_FAILURE, INVALID_ARGUMENT, UNEXPECTED_LAUNCH_FAILURE, INTERNAL_ERROR, INSTANCE_UNREACHABLE, REQUEST_REJECTED, TRIAL_EXPIRED, DRIVER_UNREACHABLE, SPARK_ERROR, DRIVER_UNRESPONSIVE, METASTORE_COMPONENT_UNHEALTHY, DBFS_COMPONENT_UNHEALTHY, EXECUTION_COMPONENT_UNHEALTHY, AZURE_RESOURCE_MANAGER_THROTTLING, AZURE_RESOURCE_PROVIDER_THROTTLING, NETWORK_CONFIGURATION_FAILURE, CONTAINER_LAUNCH_FAILURE, INSTANCE_POOL_CLUSTER_FAILURE, SKIPPED_SLOW_NODES, ATTACH_PROJECT_FAILURE, UPDATE_INSTANCE_PROFILE_FAILURE, DATABASE_CONNECTION_FAILURE, REQUEST_THROTTLED, SELF_BOOTSTRAP_FAILURE, GLOBAL_INIT_SCRIPT_FAILURE, SLOW_IMAGE_DOWNLOAD, INVALID_SPARK_IMAGE, NPIP_TUNNEL_TOKEN_FAILURE, HIVE_METASTORE_PROVISIONING_FAILURE, AZURE_INVALID_DEPLOYMENT_TEMPLATE, AZURE_UNEXPECTED_DEPLOYMENT_TEMPLATE_FAILURE, SUBNET_EXHAUSTED_FAILURE, BOOTSTRAP_TIMEOUT, STORAGE_DOWNLOAD_FAILURE, CONTROL_PLANE_REQUEST_FAILURE, BOOTSTRAP_TIMEOUT_CLOUD_PROVIDER_EXCEPTION, AWS_INSUFFICIENT_INSTANCE_CAPACITY_FAILURE, DOCKER_IMAGE_PULL_FAILURE, AZURE_VNET_CONFIGURATION_FAILURE, NPIP_TUNNEL_SETUP_FAILURE, AWS_AUTHORIZATION_FAILURE, NEPHOS_RESOURCE_MANAGEMENT, STS_CLIENT_SETUP_FAILURE, SECURITY_DAEMON_REGISTRATION_EXCEPTION, AWS_REQUEST_LIMIT_EXCEEDED, AWS_INSUFFICIENT_FREE_ADDRESSES_IN_SUBNET_FAILURE, AWS_UNSUPPORTED_FAILURE, AZURE_QUOTA_EXCEEDED_EXCEPTION, AZURE_OPERATION_NOT_ALLOWED_EXCEPTION, NFS_MOUNT_FAILURE, K8S_AUTOSCALING_FAILURE, K8S_DBR_CLUSTER_LAUNCH_TIMEOUT, SPARK_IMAGE_DOWNLOAD_FAILURE, AZURE_VM_EXTENSION_FAILURE, WORKSPACE_CANCELLED_ERROR, AWS_MAX_SPOT_INSTANCE_COUNT_EXCEEDED_FAILURE, TEMPORARILY_UNAVAILABLE, WORKER_SETUP_FAILURE, IP_EXHAUSTION_FAILURE, GCP_QUOTA_EXCEEDED, CLOUD_PROVIDER_RESOURCE_STOCKOUT, GCP_SERVICE_ACCOUNT_DELETED, AZURE_BYOK_KEY_PERMISSION_FAILURE, SPOT_INSTANCE_TERMINATION, AZURE_EPHEMERAL_DISK_FAILURE, ABUSE_DETECTED, IMAGE_PULL_PERMISSION_DENIED, WORKSPACE_CONFIGURATION_ERROR, SECRET_RESOLUTION_ERROR, UNSUPPORTED_INSTANCE_TYPE, CLOUD_PROVIDER_DISK_SETUP_FAILURE, SSH_BOOTSTRAP_FAILURE, AWS_INACCESSIBLE_KMS_KEY_FAILURE, INIT_CONTAINER_NOT_FINISHED, SPARK_IMAGE_DOWNLOAD_THROTTLED, SPARK_IMAGE_NOT_FOUND, CLUSTER_OPERATION_THROTTLED, CLUSTER_OPERATION_TIMEOUT, SERVERLESS_LONG_RUNNING_TERMINATED, AZURE_PACKED_DEPLOYMENT_PARTIAL_FAILURE, INVALID_WORKER_IMAGE_FAILURE, WORKSPACE_UPDATE, INVALID_AWS_PARAMETER, DRIVER_OUT_OF_DISK, DRIVER_OUT_OF_MEMORY, DRIVER_LAUNCH_TIMEOUT, DRIVER_UNEXPECTED_FAILURE, UNEXPECTED_POD_RECREATION, GCP_INACCESSIBLE_KMS_KEY_FAILURE, GCP_KMS_KEY_PERMISSION_DENIED, DRIVER_EVICTION, USER_INITIATED_VM_TERMINATION, GCP_IAM_TIMEOUT, AWS_RESOURCE_QUOTA_EXCEEDED, CLOUD_ACCOUNT_SETUP_FAILURE, AWS_INVALID_KEY_PAIR, DRIVER_POD_CREATION_FAILURE, MAINTENANCE_MODE, INTERNAL_CAPACITY_FAILURE, EXECUTOR_POD_UNSCHEDULED, STORAGE_DOWNLOAD_FAILURE_SLOW, STORAGE_DOWNLOAD_FAILURE_THROTTLED, DYNAMIC_SPARK_CONF_SIZE_EXCEEDED, AWS_INSTANCE_PROFILE_UPDATE_FAILURE, INSTANCE_POOL_NOT_FOUND, INSTANCE_POOL_MAX_CAPACITY_REACHED, AWS_INVALID_KMS_KEY_STATE, GCP_INSUFFICIENT_CAPACITY, GCP_API_RATE_QUOTA_EXCEEDED, GCP_RESOURCE_QUOTA_EXCEEDED, GCP_IP_SPACE_EXHAUSTED, GCP_SERVICE_ACCOUNT_ACCESS_DENIED, GCP_SERVICE_ACCOUNT_NOT_FOUND, GCP_FORBIDDEN, GCP_NOT_FOUND, RESOURCE_USAGE_BLOCKED, DATA_ACCESS_CONFIG_CHANGED, ACCESS_TOKEN_FAILURE, INVALID_INSTANCE_PLACEMENT_PROTOCOL, BUDGET_POLICY_RESOLUTION_FAILURE, IN_PENALTY_BOX, DISASTER_RECOVERY_REPLICATION, BOOTSTRAP_TIMEOUT_DUE_TO_MISCONFIG, INSTANCE_UNREACHABLE_DUE_TO_MISCONFIG, STORAGE_DOWNLOAD_FAILURE_DUE_TO_MISCONFIG, CONTROL_PLANE_REQUEST_FAILURE_DUE_TO_MISCONFIG, CLOUD_PROVIDER_LAUNCH_FAILURE_DUE_TO_MISCONFIG, GCP_SUBNET_NOT_READY, CLOUD_OPERATION_CANCELLED, CLOUD_PROVIDER_INSTANCE_NOT_LAUNCHED, GCP_TRUSTED_IMAGE_PROJECTS_VIOLATED, BUDGET_POLICY_LIMIT_ENFORCEMENT_ACTIVATED, EOS_SPARK_IMAGE, NO_MATCHED_K8S, LAZY_ALLOCATION_TIMEOUT, DRIVER_NODE_UNREACHABLE, SECRET_CREATION_FAILURE, POD_SCHEDULING_FAILURE, POD_ASSIGNMENT_FAILURE, ALLOCATION_TIMEOUT, ALLOCATION_TIMEOUT_NO_UNALLOCATED_CLUSTERS, ALLOCATION_TIMEOUT_NO_MATCHED_CLUSTERS, ALLOCATION_TIMEOUT_NO_READY_CLUSTERS, ALLOCATION_TIMEOUT_NO_WARMED_UP_CLUSTERS, ALLOCATION_TIMEOUT_NODE_DAEMON_NOT_READY, ALLOCATION_TIMEOUT_NO_HEALTHY_CLUSTERS, NETVISOR_SETUP_TIMEOUT, NO_MATCHED_K8S_TESTING_TAG, CLOUD_PROVIDER_RESOURCE_STOCKOUT_DUE_TO_MISCONFIG, GKE_BASED_CLUSTER_TERMINATION, ALLOCATION_TIMEOUT_NO_HEALTHY_AND_WARMED_UP_CLUSTERS, DOCKER_INVALID_OS_EXCEPTION, DOCKER_CONTAINER_CREATION_EXCEPTION, DOCKER_IMAGE_TOO_LARGE_FOR_INSTANCE_EXCEPTION, DNS_RESOLUTION_ERROR, GCP_DENIED_BY_ORG_POLICY, SECRET_PERMISSION_DENIED, NETWORK_CHECK_NIC_FAILURE, NETWORK_CHECK_DNS_SERVER_FAILURE, NETWORK_CHECK_STORAGE_FAILURE, NETWORK_CHECK_METADATA_ENDPOINT_FAILURE, NETWORK_CHECK_CONTROL_PLANE_FAILURE, NETWORK_CHECK_MULTIPLE_COMPONENTS_FAILURE, DRIVER_UNHEALTHY, USAGE_POLICY_ENTITLEMENT_DENIED, K8S_ACTIVE_POD_QUOTA_EXCEEDED, CLOUD_ACCOUNT_POD_QUOTA_EXCEEDED, NETWORK_CHECK_NIC_FAILURE_DUE_TO_MISCONFIG, NETWORK_CHECK_DNS_SERVER_FAILURE_DUE_TO_MISCONFIG, NETWORK_CHECK_STORAGE_FAILURE_DUE_TO_MISCONFIG, NETWORK_CHECK_METADATA_ENDPOINT_FAILURE_DUE_TO_MISCONFIG, NETWORK_CHECK_CONTROL_PLANE_FAILURE_DUE_TO_MISCONFIG, NETWORK_CHECK_MULTIPLE_COMPONENTS_FAILURE_DUE_TO_MISCONFIG, DBR_IMAGE_RESOLUTION_FAILURE, CONTROL_PLANE_CONNECTION_FAILURE, CONTROL_PLANE_CONNECTION_FAILURE_DUE_TO_MISCONFIG, RATE_LIMITED, MTLS_PORT_CONNECTIVITY_FAILURE, HIVEMETASTORE_CONNECTIVITY_FAILURE

typestring: type of the termination
Values: SUCCESS, CLIENT_ERROR, SERVICE_FAULT, CLOUD_FAILURE

parametersobject: list of parameters that provide additional information about why the cluster was terminated

specobject

The spec contains a snapshot of the latest user specified settings that were used to create/edit the cluster. Note: not included in the response of the ListClusters API.

Show child attributesHide child attributes

apply_policy_default_valuesboolean: When set to true, fixed and default values from the policy will be used for fields that are omitted. When set to false, only fixed values from the policy will be applied.
Default: false

cluster_namestring: Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.

spark_versionstring: The Spark version of the cluster, e.g. 3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

aws_attributesobject

Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 0

availabilitystring: Default: SPOT_WITH_FALLBACK
Values: SPOT, ON_DEMAND, SPOT_WITH_FALLBACK

zone_idstring

The list of available zones as well as the default value can be found by using the List Zones method.

instance_profile_arnstring

This feature may only be available to certain customer plans.

spot_bid_price_percentint32: The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new r3.xlarge spot instance, then the bid price is half of the price of on-demand r3.xlarge instances. Similarly, if this field is set to 200, the bid price is twice the price of on-demand r3.xlarge instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.
Default: 100

ebs_volume_typestring: The type of EBS volumes that will be launched with this cluster.
Values: GENERAL_PURPOSE_SSD, THROUGHPUT_OPTIMIZED_HDD

ebs_volume_countint32

These EBS volumes will be mounted at /ebs0, /ebs1, and etc. Instance store volumes will be mounted at /local_disk0, /local_disk1, and etc.

Please note that if EBS volumes are specified, then the Spark configuration spark.local.dir will be overridden.

Default: 0

ebs_volume_sizeint32: The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.

ebs_volume_iopsint32: If using gp3 volumes, what IOPS to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.

ebs_volume_throughputint32: If using gp3 volumes, what throughput to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.

azure_attributesobject

Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

log_analytics_infoobject: Defines values necessary to configure and run Azure Log Analytics agent

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 1

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero, this availability type will be used for the entire cluster.
Default: ON_DEMAND_AZURE
Values: SPOT_AZURE, ON_DEMAND_AZURE, SPOT_WITH_FALLBACK_AZURE

spot_bid_max_pricedouble: The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.
Default: -1

capacity_reservation_groupstring

The Azure capacity reservation group resource ID to use for launching VMs. When specified, VMs will be launched using the provided capacity reservation.

Microsoft.Compute/capacityReservationGroups/read
Microsoft.Compute/capacityReservationGroups/deploy/action
Microsoft.Compute/capacityReservationGroups/capacityReservations/read
Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action

Format: /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}

gcp_attributesobject

Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

use_preemptible_executorsboolean: This field determines whether the spark executors will be scheduled to run on preemptible VMs (when set to true) versus standard compute engine VMs (when set to false; default). Note: Soon to be deprecated, use the 'availability' field instead.
Default: false

google_service_accountstring: If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.

boot_disk_sizeint32: Boot disk size in GB

availabilitystring: This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
Default: ON_DEMAND_GCP
Values: PREEMPTIBLE_GCP, ON_DEMAND_GCP, PREEMPTIBLE_WITH_FALLBACK_GCP

zone_idstring

Identifier for the availability zone in which the cluster resides. This can be one of the following:

"HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
"AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.

Default: HA

local_ssd_countint32: If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to GCP documentation for the supported number of local SSDs for each instance type.

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 1

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring

The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

worker_node_type_flexibilityobject

Flexible node type configuration for worker nodes.

Show child attributesHide child attributes

alternate_node_type_idsarray of string: A list of node type IDs to use as fallbacks when the primary node type is unavailable.

driver_node_type_flexibilityobject

Flexible node type configuration for the driver node.

Show child attributesHide child attributes

alternate_node_type_idsarray of string: A list of node type IDs to use as fallbacks when the primary node type is unavailable.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject

Show child attributesHide child attributes

dbfsobjectRequired: destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }

s3objectRequired: destination and either the region or endpoint need to be provided. e.g. { "s3": { "destination" : "s3://cluster_log_bucket/prefix", "region" : "us-west-2" } } Cluster iam role is used to access s3, please make sure the cluster iam role in instance_profile_arn has permission to write data to the s3 destination.

volumesobjectRequired: destination needs to be provided, e.g. { "volumes": { "destination": "/Volumes/catalog/schema/volume/cluster_log" } }

spark_env_varsobject

autotermination_minutesint32: Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.

enable_elastic_diskboolean: Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWS
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

init_scriptsarray of object

Show child attributesHide child attributes

dbfsobjectRequired: destination needs to be provided. e.g. { "dbfs": { "destination" : "dbfs:/home/cluster_log" } }

s3objectRequired: destination and either the region or endpoint need to be provided. e.g. { \"s3\": { \"destination\": \"s3://cluster_log_bucket/prefix\", \"region\": \"us-west-2\" } } Cluster iam role is used to access s3, please make sure the cluster iam role in instance_profile_arn has permission to write data to the s3 destination.

fileobjectRequired: destination needs to be provided, e.g. { "file": { "destination": "file:/my/local/file.sh" } }

gcsobjectRequired: destination needs to be provided, e.g. { "gcs": { "destination": "gs://my-bucket/file.sh" } }

abfssobjectRequired: destination needs to be provided, e.g. abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>

workspaceobjectRequired: destination needs to be provided, e.g. { "workspace": { "destination": "/cluster-init-scripts/setup-datadog.sh" } }

volumesobjectRequired: destination needs to be provided. e.g. { \"volumes\" : { \"destination\" : \"/Volumes/my-init.sh\" } }

docker_imageobject

Custom docker image BYOC

Show child attributesHide child attributes

urlstring: URL of the docker image.

basic_authobjectRequired: Basic auth with username and password

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

single_user_namestring: Single user name if data_security_mode is SINGLE_USER

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable LUKS on cluster VMs' local disks

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

workload_typeobject

Show child attributesHide child attributes

clientsobject: defined what type of clients can use the cluster. E.g. Notebooks, Jobs

data_security_modestring: Values: NONE, SINGLE_USER, USER_ISOLATION, LEGACY_TABLE_ACL, LEGACY_PASSTHROUGH, LEGACY_SINGLE_USER, LEGACY_SINGLE_USER_STANDARD, DATA_SECURITY_MODE_STANDARD, DATA_SECURITY_MODE_DEDICATED, DATA_SECURITY_MODE_AUTO

runtime_enginestring

Determines the cluster's runtime engine, either standard or Photon.

This field is not compatible with legacy spark_version values that contain -photon-. Remove -photon- from the spark_version and set runtime_engine to PHOTON.

If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.

Values: NULL, STANDARD, PHOTON

kindstring: Values: COMPUTE_KIND_UNSPECIFIED, CLASSIC_PREVIEW

use_ml_runtimeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

effective_spark_version is determined by spark_version (DBR release), this field use_ml_runtime, and whether node_type_id is gpu node or not.

is_single_nodeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

When set to true, <Databricks> will automatically set single node related custom_tags, spark_conf, and num_workers

remote_disk_throughputint32: If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.

total_initial_remote_disk_sizeint32: If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.

dependency_modestringBeta: Controls dependency configuration for the cluster.
Values: DEPENDENCY_MODE_UNSPECIFIED, DEPENDENCY_MODE_ENVIRONMENTS, DEPENDENCY_MODE_CLUSTER_LIBRARIES, DEPENDENCY_MODE_AUTO

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired

Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

Show child attributesHide child attributes

min_workersint32: The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.

max_workersint32: The maximum number of workers to which the cluster can scale up when overloaded. Note that max_workers must be strictly greater than min_workers.

driverobject

Node on which the Spark driver resides. The driver node contains the Spark master and the <Databricks> application that manages the per-notebook Spark REPLs.

Show child attributesHide child attributes

private_ipstring: Private IP address (typically a 10.x.x.x address) of the Spark node. Note that this is different from the private IP address of the host instance.

public_dnsstring: Public DNS address of this node. This address can be used to access the Spark JDBC server on the driver node. To communicate with the JDBC server, traffic must be manually authorized by adding security group rules to the "worker-unmanaged" security group via the AWS console.

node_idstring: Globally unique identifier for this node.

instance_idstring: Globally unique identifier for the host instance from the cloud provider.

start_timestampint64: The timestamp (in millisecond) when the Spark node is launched.

node_aws_attributesobject

Attributes specific to AWS for a Spark node.

Show child attributesHide child attributes

is_spotboolean: Whether this node is on an Amazon spot instance.

host_private_ipstring: The private IP address of the host instance.

executorsarray of object

Nodes on which the Spark executors reside.

Show child attributesHide child attributes

private_ipstring: Private IP address (typically a 10.x.x.x address) of the Spark node. Note that this is different from the private IP address of the host instance.

public_dnsstring: Public DNS address of this node. This address can be used to access the Spark JDBC server on the driver node. To communicate with the JDBC server, traffic must be manually authorized by adding security group rules to the "worker-unmanaged" security group via the AWS console.

node_idstring: Globally unique identifier for this node.

instance_idstring: Globally unique identifier for the host instance from the cloud provider.

start_timestampint64: The timestamp (in millisecond) when the Spark node is launched.

node_aws_attributesobject

Attributes specific to AWS for a Spark node.

Show child attributesHide child attributes

is_spotboolean: Whether this node is on an Amazon spot instance.

host_private_ipstring: The private IP address of the host instance.

spark_context_idint64: A canonical SparkContext identifier. This value does change when the Spark driver restarts. The pair (cluster_id, spark_context_id) is a globally unique identifier over all Spark contexts.

jdbc_portint32: Port on which Spark JDBC server is listening, in the driver nod. No service will be listeningon on this port in executor nodes.

cluster_namestring: Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.

spark_versionstring: The Spark version of the cluster, e.g. 3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

aws_attributesobject

Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 0

availabilitystring: Default: SPOT_WITH_FALLBACK
Values: SPOT, ON_DEMAND, SPOT_WITH_FALLBACK

zone_idstring

The list of available zones as well as the default value can be found by using the List Zones method.

instance_profile_arnstring

This feature may only be available to certain customer plans.

spot_bid_price_percentint32: The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new r3.xlarge spot instance, then the bid price is half of the price of on-demand r3.xlarge instances. Similarly, if this field is set to 200, the bid price is twice the price of on-demand r3.xlarge instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.
Default: 100

ebs_volume_typestring: The type of EBS volumes that will be launched with this cluster.
Values: GENERAL_PURPOSE_SSD, THROUGHPUT_OPTIMIZED_HDD

ebs_volume_countint32

These EBS volumes will be mounted at /ebs0, /ebs1, and etc. Instance store volumes will be mounted at /local_disk0, /local_disk1, and etc.

Please note that if EBS volumes are specified, then the Spark configuration spark.local.dir will be overridden.

Default: 0

ebs_volume_sizeint32: The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.

ebs_volume_iopsint32: If using gp3 volumes, what IOPS to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.

ebs_volume_throughputint32: If using gp3 volumes, what throughput to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.

azure_attributesobject

Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

log_analytics_infoobject

Defines values necessary to configure and run Azure Log Analytics agent

Show child attributesHide child attributes

log_analytics_workspace_idstring

log_analytics_primary_keystring

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 1

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero, this availability type will be used for the entire cluster.
Default: ON_DEMAND_AZURE
Values: SPOT_AZURE, ON_DEMAND_AZURE, SPOT_WITH_FALLBACK_AZURE

spot_bid_max_pricedouble: The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.
Default: -1

capacity_reservation_groupstring

The Azure capacity reservation group resource ID to use for launching VMs. When specified, VMs will be launched using the provided capacity reservation.

Microsoft.Compute/capacityReservationGroups/read
Microsoft.Compute/capacityReservationGroups/deploy/action
Microsoft.Compute/capacityReservationGroups/capacityReservations/read
Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action

Format: /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}

gcp_attributesobject

Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

use_preemptible_executorsboolean: This field determines whether the spark executors will be scheduled to run on preemptible VMs (when set to true) versus standard compute engine VMs (when set to false; default). Note: Soon to be deprecated, use the 'availability' field instead.
Default: false

google_service_accountstring: If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.

boot_disk_sizeint32: Boot disk size in GB

availabilitystring: This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
Default: ON_DEMAND_GCP
Values: PREEMPTIBLE_GCP, ON_DEMAND_GCP, PREEMPTIBLE_WITH_FALLBACK_GCP

zone_idstring

Identifier for the availability zone in which the cluster resides. This can be one of the following:

"HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
"AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.

Default: HA

local_ssd_countint32: If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to GCP documentation for the supported number of local SSDs for each instance type.

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 1

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring

The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

worker_node_type_flexibilityobject

Flexible node type configuration for worker nodes.

Show child attributesHide child attributes

alternate_node_type_idsarray of string: A list of node type IDs to use as fallbacks when the primary node type is unavailable.

driver_node_type_flexibilityobject

Flexible node type configuration for the driver node.

Show child attributesHide child attributes

alternate_node_type_idsarray of string: A list of node type IDs to use as fallbacks when the primary node type is unavailable.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject

Show child attributesHide child attributes

dbfsobjectRequired

destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }

Show child attributesHide child attributes

destinationstring: dbfs destination, e.g. dbfs:/my/path

s3objectRequired

Show child attributesHide child attributes

destinationstring: S3 destination, e.g. s3://my-bucket/some-prefix Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs.

regionstring: S3 region, e.g. us-west-2. Either region or endpoint needs to be set. If both are set, endpoint will be used.

endpointstring: S3 endpoint, e.g. https://s3-us-west-2.amazonaws.com. Either region or endpoint needs to be set. If both are set, endpoint will be used.

enable_encryptionboolean: (Optional) Flag to enable server side encryption, false by default.

encryption_typestring: (Optional) The encryption type, it could be sse-s3 or sse-kms. It will be used only when encryption is enabled and the default type is sse-s3.

kms_keystring: (Optional) Kms key which will be used if encryption is enabled and encryption type is set to sse-kms.

canned_aclstring: (Optional) Set canned access control list for the logs, e.g. bucket-owner-full-control. If canned_cal is set, please make sure the cluster iam role has s3:PutObjectAcl permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set bucket-owner-full-control to make bucket owner able to read the logs.

volumesobjectRequired

destination needs to be provided, e.g. { "volumes": { "destination": "/Volumes/catalog/schema/volume/cluster_log" } }

Show child attributesHide child attributes

destinationstring: UC Volumes destination, e.g. /Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh or dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh

spark_env_varsobject

autotermination_minutesint32: Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.

enable_elastic_diskboolean: Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWS
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

init_scriptsarray of object

Show child attributesHide child attributes

dbfsobjectRequired

destination needs to be provided. e.g. { "dbfs": { "destination" : "dbfs:/home/cluster_log" } }

Show child attributesHide child attributes

destinationstring: dbfs destination, e.g. dbfs:/my/path

s3objectRequired

Show child attributesHide child attributes

destinationstring: S3 destination, e.g. s3://my-bucket/some-prefix Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs.

regionstring: S3 region, e.g. us-west-2. Either region or endpoint needs to be set. If both are set, endpoint will be used.

endpointstring: S3 endpoint, e.g. https://s3-us-west-2.amazonaws.com. Either region or endpoint needs to be set. If both are set, endpoint will be used.

enable_encryptionboolean: (Optional) Flag to enable server side encryption, false by default.

encryption_typestring: (Optional) The encryption type, it could be sse-s3 or sse-kms. It will be used only when encryption is enabled and the default type is sse-s3.

kms_keystring: (Optional) Kms key which will be used if encryption is enabled and encryption type is set to sse-kms.

canned_aclstring: (Optional) Set canned access control list for the logs, e.g. bucket-owner-full-control. If canned_cal is set, please make sure the cluster iam role has s3:PutObjectAcl permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set bucket-owner-full-control to make bucket owner able to read the logs.

fileobjectRequired

destination needs to be provided, e.g. { "file": { "destination": "file:/my/local/file.sh" } }

Show child attributesHide child attributes

destinationstring: local file destination, e.g. file:/my/local/file.sh

gcsobjectRequired

destination needs to be provided, e.g. { "gcs": { "destination": "gs://my-bucket/file.sh" } }

Show child attributesHide child attributes

destinationstring: GCS destination/URI, e.g. gs://my-bucket/some-prefix

abfssobjectRequired

destination needs to be provided, e.g. abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>

Show child attributesHide child attributes

destinationstring: abfss destination, e.g. abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>.

workspaceobjectRequired

destination needs to be provided, e.g. { "workspace": { "destination": "/cluster-init-scripts/setup-datadog.sh" } }

Show child attributesHide child attributes

destinationstring: wsfs destination, e.g. workspace:/cluster-init-scripts/setup-datadog.sh

volumesobjectRequired

destination needs to be provided. e.g. { \"volumes\" : { \"destination\" : \"/Volumes/my-init.sh\" } }

Show child attributesHide child attributes

destinationstring: UC Volumes destination, e.g. /Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh or dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh

docker_imageobject

Custom docker image BYOC

Show child attributesHide child attributes

urlstring: URL of the docker image.

basic_authobjectRequired

Basic auth with username and password

Show child attributesHide child attributes

usernamestring: Name of the user

passwordstring: Password of the user

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

single_user_namestring: Single user name if data_security_mode is SINGLE_USER

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable LUKS on cluster VMs' local disks

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

workload_typeobject

Show child attributesHide child attributes

clientsobject

defined what type of clients can use the cluster. E.g. Notebooks, Jobs

Show child attributesHide child attributes

notebooksboolean: With notebooks set, this cluster can be used for notebooks
Default: true

jobsboolean: With jobs set, the cluster can be used for jobs
Default: true

data_security_modestring: Values: NONE, SINGLE_USER, USER_ISOLATION, LEGACY_TABLE_ACL, LEGACY_PASSTHROUGH, LEGACY_SINGLE_USER, LEGACY_SINGLE_USER_STANDARD, DATA_SECURITY_MODE_STANDARD, DATA_SECURITY_MODE_DEDICATED, DATA_SECURITY_MODE_AUTO

runtime_enginestring

Determines the cluster's runtime engine, either standard or Photon.

This field is not compatible with legacy spark_version values that contain -photon-. Remove -photon- from the spark_version and set runtime_engine to PHOTON.

If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.

Values: NULL, STANDARD, PHOTON

kindstring: Values: COMPUTE_KIND_UNSPECIFIED, CLASSIC_PREVIEW

use_ml_runtimeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

effective_spark_version is determined by spark_version (DBR release), this field use_ml_runtime, and whether node_type_id is gpu node or not.

is_single_nodeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

When set to true, <Databricks> will automatically set single node related custom_tags, spark_conf, and num_workers

remote_disk_throughputint32: If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.

total_initial_remote_disk_sizeint32: If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.

dependency_modestringBeta: Controls dependency configuration for the cluster.
Values: DEPENDENCY_MODE_UNSPECIFIED, DEPENDENCY_MODE_ENVIRONMENTS, DEPENDENCY_MODE_CLUSTER_LIBRARIES, DEPENDENCY_MODE_AUTO

start_timeint64: Time (in epoch milliseconds) when the cluster creation request was received (when the cluster entered a PENDING state).

terminated_timeint64: Time (in epoch milliseconds) when the cluster was terminated, if applicable.

last_state_loss_timeint64: Time when the cluster driver last lost its state (due to a restart or driver failure).

last_restarted_timeint64: the timestamp that the cluster was started/restarted

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired

Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

Show child attributesHide child attributes

min_workersint32: The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.

max_workersint32: The maximum number of workers to which the cluster can scale up when overloaded. Note that max_workers must be strictly greater than min_workers.

next_page_tokenstring: This field represents the pagination token to retrieve the next page of results. If the value is "", it means no further results for the request.
Example: CAEaFDAzMTUtMTczODQ1LTcwdWExbDhw

prev_page_tokenstring: This field represents the pagination token to retrieve the previous page of results. If the value is "", it means no further results for the request.
Example: CAEaFDA4MTItMTc0Mzk0LTE5c2w5djhh

View as Markdown

Response

{
  "clusters": [
    {
      "autotermination_minutes": 120,
      "aws_attributes": {
        "availability": "SPOT_WITH_FALLBACK",
        "ebs_volume_count": 0,
        "first_on_demand": 1,
        "spot_bid_price_percent": 100,
        "zone_id": "us-west-2c"
      },
      "cluster_id": "1234-567890-reef123",
      "cluster_name": "my-cluster",
      "cluster_source": "UI",
      "creator_user_name": "someone@example.com",
      "default_tags": {
        "ClusterId": "1234-567890-reef123",
        "ClusterName": "my-cluster",
        "Creator": "someone@example.com",
        "Vendor": "Databricks"
      },
      "disk_spec": {
        "disk_count": 0
      },
      "driver_instance_source": {
        "node_type_id": "i3.xlarge"
      },
      "driver_node_type_id": "i3.xlarge",
      "enable_elastic_disk": false,
      "enable_local_disk_encryption": false,
      "init_scripts_safe_mode": false,
      "instance_source": {
        "node_type_id": "i3.xlarge"
      },
      "last_state_loss_time": 1619739324740,
      "node_type_id": "i3.xlarge",
      "num_workers": 30,
      "spark_context_id": 4020997813441461760,
      "spark_version": "8.2.x-scala2.12",
      "start_time": 1618263108824,
      "state": "TERMINATED",
      "state_message": "Inactive cluster terminated (inactive for 120 minutes).",
      "terminated_time": 1619746525713,
      "termination_reason": {
        "code": "INACTIVITY",
        "parameters": {
          "inactivity_duration_min": "120"
        },
        "type": "SUCCESS"
      }
    }
  ],
  "next_page_token": "CAEaFDA4MTItMTc0Mzk0LTE5c2w5djhh",
  "prev_page_token": "CAEaFDAzMTUtMTczODQ1LTcwdWExbDhw"
}

Create GA

POST /api/2.1/clusters/create

Creates a new Spark cluster. This method will acquire new instances from the cloud provider if necessary. This method is asynchronous; the returned cluster_id can be used to poll the cluster status. When this method returns, the cluster will be in a PENDING state. The cluster will be usable once it enters a RUNNING state. Note: <Databricks> may not be able to acquire some of the requested nodes, due to cloud provider limitations (account limits, spot price, etc.) or transient network issues.

If <Databricks> acquires at least 85% of the requested on-demand nodes, cluster creation will succeed. Otherwise the cluster will terminate with an informative error message.

Rather than authoring the cluster's JSON definition from scratch, Databricks recommends filling out the create compute UI and then copying the generated JSON definition from the UI.

API scopes: clusters

Request body

apply_policy_default_valuesboolean: When set to true, fixed and default values from the policy will be used for fields that are omitted. When set to false, only fixed values from the policy will be applied.
Default: false

clone_fromobject

When specified, this clones libraries from a source cluster during the creation of a new cluster.

Show child attributesHide child attributes

source_cluster_idstring: The cluster that is being cloned.

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired

Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

Show child attributesHide child attributes

min_workersint32: The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.

max_workersint32: The maximum number of workers to which the cluster can scale up when overloaded. Note that max_workers must be strictly greater than min_workers.

cluster_namestring: Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.

spark_versionstring: The Spark version of the cluster, e.g. 3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

aws_attributesobject

Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 0

availabilitystring: Default: SPOT_WITH_FALLBACK
Values: SPOT, ON_DEMAND, SPOT_WITH_FALLBACK

zone_idstring

The list of available zones as well as the default value can be found by using the List Zones method.

instance_profile_arnstring

This feature may only be available to certain customer plans.

spot_bid_price_percentint32: The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new r3.xlarge spot instance, then the bid price is half of the price of on-demand r3.xlarge instances. Similarly, if this field is set to 200, the bid price is twice the price of on-demand r3.xlarge instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.
Default: 100

ebs_volume_typestring: The type of EBS volumes that will be launched with this cluster.
Values: GENERAL_PURPOSE_SSD, THROUGHPUT_OPTIMIZED_HDD

ebs_volume_countint32

These EBS volumes will be mounted at /ebs0, /ebs1, and etc. Instance store volumes will be mounted at /local_disk0, /local_disk1, and etc.

Please note that if EBS volumes are specified, then the Spark configuration spark.local.dir will be overridden.

Default: 0

ebs_volume_sizeint32: The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.

ebs_volume_iopsint32: If using gp3 volumes, what IOPS to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.

ebs_volume_throughputint32: If using gp3 volumes, what throughput to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.

azure_attributesobject

Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

log_analytics_infoobject

Defines values necessary to configure and run Azure Log Analytics agent

Show child attributesHide child attributes

log_analytics_workspace_idstring

log_analytics_primary_keystring

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 1

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero, this availability type will be used for the entire cluster.
Default: ON_DEMAND_AZURE
Values: SPOT_AZURE, ON_DEMAND_AZURE, SPOT_WITH_FALLBACK_AZURE

spot_bid_max_pricedouble: The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.
Default: -1

capacity_reservation_groupstring

The Azure capacity reservation group resource ID to use for launching VMs. When specified, VMs will be launched using the provided capacity reservation.

Microsoft.Compute/capacityReservationGroups/read
Microsoft.Compute/capacityReservationGroups/deploy/action
Microsoft.Compute/capacityReservationGroups/capacityReservations/read
Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action

Format: /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}

gcp_attributesobject

Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

use_preemptible_executorsboolean: This field determines whether the spark executors will be scheduled to run on preemptible VMs (when set to true) versus standard compute engine VMs (when set to false; default). Note: Soon to be deprecated, use the 'availability' field instead.
Default: false

google_service_accountstring: If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.

boot_disk_sizeint32: Boot disk size in GB

availabilitystring: This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
Default: ON_DEMAND_GCP
Values: PREEMPTIBLE_GCP, ON_DEMAND_GCP, PREEMPTIBLE_WITH_FALLBACK_GCP

zone_idstring

Identifier for the availability zone in which the cluster resides. This can be one of the following:

"HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
"AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.

Default: HA

local_ssd_countint32: If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to GCP documentation for the supported number of local SSDs for each instance type.

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 1

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring

The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

worker_node_type_flexibilityobject

Flexible node type configuration for worker nodes.

Show child attributesHide child attributes

alternate_node_type_idsarray of string: A list of node type IDs to use as fallbacks when the primary node type is unavailable.

driver_node_type_flexibilityobject

Flexible node type configuration for the driver node.

Show child attributesHide child attributes

alternate_node_type_idsarray of string: A list of node type IDs to use as fallbacks when the primary node type is unavailable.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject

Show child attributesHide child attributes

dbfsobjectRequired

destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }

Show child attributesHide child attributes

destinationstring: dbfs destination, e.g. dbfs:/my/path

s3objectRequired

Show child attributesHide child attributes

destinationstring: S3 destination, e.g. s3://my-bucket/some-prefix Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs.

regionstring: S3 region, e.g. us-west-2. Either region or endpoint needs to be set. If both are set, endpoint will be used.

endpointstring: S3 endpoint, e.g. https://s3-us-west-2.amazonaws.com. Either region or endpoint needs to be set. If both are set, endpoint will be used.

enable_encryptionboolean: (Optional) Flag to enable server side encryption, false by default.

encryption_typestring: (Optional) The encryption type, it could be sse-s3 or sse-kms. It will be used only when encryption is enabled and the default type is sse-s3.

kms_keystring: (Optional) Kms key which will be used if encryption is enabled and encryption type is set to sse-kms.

canned_aclstring: (Optional) Set canned access control list for the logs, e.g. bucket-owner-full-control. If canned_cal is set, please make sure the cluster iam role has s3:PutObjectAcl permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set bucket-owner-full-control to make bucket owner able to read the logs.

volumesobjectRequired

destination needs to be provided, e.g. { "volumes": { "destination": "/Volumes/catalog/schema/volume/cluster_log" } }

Show child attributesHide child attributes

destinationstring: UC Volumes destination, e.g. /Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh or dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh

spark_env_varsobject

autotermination_minutesint32: Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.

enable_elastic_diskboolean: Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWS
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

init_scriptsarray of object

Show child attributesHide child attributes

dbfsobjectRequired

destination needs to be provided. e.g. { "dbfs": { "destination" : "dbfs:/home/cluster_log" } }

Show child attributesHide child attributes

destinationstring: dbfs destination, e.g. dbfs:/my/path

s3objectRequired

Show child attributesHide child attributes

destinationstring: S3 destination, e.g. s3://my-bucket/some-prefix Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs.

regionstring: S3 region, e.g. us-west-2. Either region or endpoint needs to be set. If both are set, endpoint will be used.

endpointstring: S3 endpoint, e.g. https://s3-us-west-2.amazonaws.com. Either region or endpoint needs to be set. If both are set, endpoint will be used.

enable_encryptionboolean: (Optional) Flag to enable server side encryption, false by default.

encryption_typestring: (Optional) The encryption type, it could be sse-s3 or sse-kms. It will be used only when encryption is enabled and the default type is sse-s3.

kms_keystring: (Optional) Kms key which will be used if encryption is enabled and encryption type is set to sse-kms.

canned_aclstring: (Optional) Set canned access control list for the logs, e.g. bucket-owner-full-control. If canned_cal is set, please make sure the cluster iam role has s3:PutObjectAcl permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set bucket-owner-full-control to make bucket owner able to read the logs.

fileobjectRequired

destination needs to be provided, e.g. { "file": { "destination": "file:/my/local/file.sh" } }

Show child attributesHide child attributes

destinationstring: local file destination, e.g. file:/my/local/file.sh

gcsobjectRequired

destination needs to be provided, e.g. { "gcs": { "destination": "gs://my-bucket/file.sh" } }

Show child attributesHide child attributes

destinationstring: GCS destination/URI, e.g. gs://my-bucket/some-prefix

abfssobjectRequired

destination needs to be provided, e.g. abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>

Show child attributesHide child attributes

destinationstring: abfss destination, e.g. abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>.

workspaceobjectRequired

destination needs to be provided, e.g. { "workspace": { "destination": "/cluster-init-scripts/setup-datadog.sh" } }

Show child attributesHide child attributes

destinationstring: wsfs destination, e.g. workspace:/cluster-init-scripts/setup-datadog.sh

volumesobjectRequired

destination needs to be provided. e.g. { \"volumes\" : { \"destination\" : \"/Volumes/my-init.sh\" } }

Show child attributesHide child attributes

destinationstring: UC Volumes destination, e.g. /Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh or dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh

docker_imageobject

Custom docker image BYOC

Show child attributesHide child attributes

urlstring: URL of the docker image.

basic_authobjectRequired

Basic auth with username and password

Show child attributesHide child attributes

usernamestring: Name of the user

passwordstring: Password of the user

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

single_user_namestring: Single user name if data_security_mode is SINGLE_USER

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable LUKS on cluster VMs' local disks

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

workload_typeobject

Show child attributesHide child attributes

clientsobject

defined what type of clients can use the cluster. E.g. Notebooks, Jobs

Show child attributesHide child attributes

notebooksboolean: With notebooks set, this cluster can be used for notebooks
Default: true

jobsboolean: With jobs set, the cluster can be used for jobs
Default: true

data_security_modestring: Values: NONE, SINGLE_USER, USER_ISOLATION, LEGACY_TABLE_ACL, LEGACY_PASSTHROUGH, LEGACY_SINGLE_USER, LEGACY_SINGLE_USER_STANDARD, DATA_SECURITY_MODE_STANDARD, DATA_SECURITY_MODE_DEDICATED, DATA_SECURITY_MODE_AUTO

runtime_enginestring

Determines the cluster's runtime engine, either standard or Photon.

This field is not compatible with legacy spark_version values that contain -photon-. Remove -photon- from the spark_version and set runtime_engine to PHOTON.

If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.

Values: NULL, STANDARD, PHOTON

kindstring: Values: COMPUTE_KIND_UNSPECIFIED, CLASSIC_PREVIEW

use_ml_runtimeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

effective_spark_version is determined by spark_version (DBR release), this field use_ml_runtime, and whether node_type_id is gpu node or not.

is_single_nodeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

When set to true, <Databricks> will automatically set single node related custom_tags, spark_conf, and num_workers

remote_disk_throughputint32: If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.

total_initial_remote_disk_sizeint32: If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.

dependency_modestringBeta: Controls dependency configuration for the cluster.
Values: DEPENDENCY_MODE_UNSPECIFIED, DEPENDENCY_MODE_ENVIRONMENTS, DEPENDENCY_MODE_CLUSTER_LIBRARIES, DEPENDENCY_MODE_AUTO

Response

cluster_idstring: Example: 1234-567890-cited123

View as Markdown

Request

{
  "autoscale": {
    "max_workers": 50,
    "min_workers": 2
  },
  "aws_attributes": {
    "availability": "SPOT_WITH_FALLBACK",
    "ebs_volume_count": 0,
    "first_on_demand": 1,
    "spot_bid_price_percent": 100,
    "zone_id": "auto"
  },
  "cluster_name": "autoscaling-cluster",
  "node_type_id": "r3.xlarge",
  "spark_version": "14.3.x-scala2.12"
}

{
  "aws_attributes": {
    "availability": "SPOT",
    "first_on_demand": 1,
    "zone_id": "us-west-2a"
  },
  "cluster_name": "my-cluster",
  "node_type_id": "r3.xlarge",
  "num_workers": 25,
  "spark_conf": {
    "spark.speculation": true
  },
  "spark_version": "14.3.x-scala2.12"
}

{
  "aws_attributes": {
    "availability": "SPOT_WITH_FALLBACK",
    "ebs_volume_count": 0,
    "first_on_demand": 1,
    "spot_bid_price_percent": 100,
    "zone_id": "auto"
  },
  "cluster_name": "machine-learning-with-kind-cluster",
  "kind": "CLASSIC_PREVIEW",
  "node_type_id": "i3.xlarge",
  "num_workers": 3,
  "spark_version": "15.4.x-scala2.12",
  "use_ml_runtime": true
}

{
  "aws_attributes": {
    "availability": "SPOT_WITH_FALLBACK",
    "ebs_volume_count": 0,
    "first_on_demand": 1,
    "spot_bid_price_percent": 100,
    "zone_id": "auto"
  },
  "cluster_name": "single-node-cluster",
  "custom_tags": {
    "ResourceClass": "SingleNode"
  },
  "node_type_id": "i3.xlarge",
  "num_workers": 0,
  "spark_conf": {
    "spark.databricks.cluster.profile": "singleNode",
    "spark.master": "local[*, 4]"
  },
  "spark_version": "14.3.x-scala2.12"
}

{
  "aws_attributes": {
    "availability": "SPOT_WITH_FALLBACK",
    "ebs_volume_count": 0,
    "first_on_demand": 1,
    "spot_bid_price_percent": 100,
    "zone_id": "auto"
  },
  "cluster_name": "single-node-with-kind-cluster",
  "is_single_node": true,
  "kind": "CLASSIC_PREVIEW",
  "node_type_id": "i3.xlarge",
  "spark_version": "14.3.x-scala2.12"
}

Response

{
  "cluster_id": "1234-567890-cited123"
}

Update GA

POST /api/2.1/clusters/edit

Updates the configuration of a cluster to match the provided attributes and size. A cluster can be updated if it is in a RUNNING or TERMINATED state.

If a cluster is updated while in a RUNNING state, it will be restarted so that the new attributes can take effect.

If a cluster is updated while in a TERMINATED state, it will remain TERMINATED. The next time it is started using the clusters/start API, the new attributes will take effect. Any attempt to update a cluster in any other state will be rejected with an INVALID_STATE error code.

Clusters created by the Databricks Jobs service cannot be edited.

API scopes: clusters

Request body

cluster_idstring: ID of the cluster

apply_policy_default_valuesboolean: When set to true, fixed and default values from the policy will be used for fields that are omitted. When set to false, only fixed values from the policy will be applied.
Default: false

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired

Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

Show child attributesHide child attributes

min_workersint32: The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.

max_workersint32: The maximum number of workers to which the cluster can scale up when overloaded. Note that max_workers must be strictly greater than min_workers.

cluster_namestring: Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.

spark_versionstring: The Spark version of the cluster, e.g. 3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

aws_attributesobject

Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 0

availabilitystring: Default: SPOT_WITH_FALLBACK
Values: SPOT, ON_DEMAND, SPOT_WITH_FALLBACK

zone_idstring

The list of available zones as well as the default value can be found by using the List Zones method.

instance_profile_arnstring

This feature may only be available to certain customer plans.

spot_bid_price_percentint32: The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new r3.xlarge spot instance, then the bid price is half of the price of on-demand r3.xlarge instances. Similarly, if this field is set to 200, the bid price is twice the price of on-demand r3.xlarge instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.
Default: 100

ebs_volume_typestring: The type of EBS volumes that will be launched with this cluster.
Values: GENERAL_PURPOSE_SSD, THROUGHPUT_OPTIMIZED_HDD

ebs_volume_countint32

These EBS volumes will be mounted at /ebs0, /ebs1, and etc. Instance store volumes will be mounted at /local_disk0, /local_disk1, and etc.

Please note that if EBS volumes are specified, then the Spark configuration spark.local.dir will be overridden.

Default: 0

ebs_volume_sizeint32: The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.

ebs_volume_iopsint32: If using gp3 volumes, what IOPS to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.

ebs_volume_throughputint32: If using gp3 volumes, what throughput to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.

azure_attributesobject

Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

log_analytics_infoobject

Defines values necessary to configure and run Azure Log Analytics agent

Show child attributesHide child attributes

log_analytics_workspace_idstring

log_analytics_primary_keystring

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 1

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero, this availability type will be used for the entire cluster.
Default: ON_DEMAND_AZURE
Values: SPOT_AZURE, ON_DEMAND_AZURE, SPOT_WITH_FALLBACK_AZURE

spot_bid_max_pricedouble: The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.
Default: -1

capacity_reservation_groupstring

The Azure capacity reservation group resource ID to use for launching VMs. When specified, VMs will be launched using the provided capacity reservation.

Microsoft.Compute/capacityReservationGroups/read
Microsoft.Compute/capacityReservationGroups/deploy/action
Microsoft.Compute/capacityReservationGroups/capacityReservations/read
Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action

Format: /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}

gcp_attributesobject

Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

use_preemptible_executorsboolean: This field determines whether the spark executors will be scheduled to run on preemptible VMs (when set to true) versus standard compute engine VMs (when set to false; default). Note: Soon to be deprecated, use the 'availability' field instead.
Default: false

google_service_accountstring: If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.

boot_disk_sizeint32: Boot disk size in GB

availabilitystring: This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
Default: ON_DEMAND_GCP
Values: PREEMPTIBLE_GCP, ON_DEMAND_GCP, PREEMPTIBLE_WITH_FALLBACK_GCP

zone_idstring

Identifier for the availability zone in which the cluster resides. This can be one of the following:

"HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
"AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.

Default: HA

local_ssd_countint32: If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to GCP documentation for the supported number of local SSDs for each instance type.

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 1

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring

The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

worker_node_type_flexibilityobject

Flexible node type configuration for worker nodes.

Show child attributesHide child attributes

alternate_node_type_idsarray of string: A list of node type IDs to use as fallbacks when the primary node type is unavailable.

driver_node_type_flexibilityobject

Flexible node type configuration for the driver node.

Show child attributesHide child attributes

alternate_node_type_idsarray of string: A list of node type IDs to use as fallbacks when the primary node type is unavailable.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject

Show child attributesHide child attributes

dbfsobjectRequired

destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }

Show child attributesHide child attributes

destinationstring: dbfs destination, e.g. dbfs:/my/path

s3objectRequired

Show child attributesHide child attributes

destinationstring: S3 destination, e.g. s3://my-bucket/some-prefix Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs.

regionstring: S3 region, e.g. us-west-2. Either region or endpoint needs to be set. If both are set, endpoint will be used.

endpointstring: S3 endpoint, e.g. https://s3-us-west-2.amazonaws.com. Either region or endpoint needs to be set. If both are set, endpoint will be used.

enable_encryptionboolean: (Optional) Flag to enable server side encryption, false by default.

encryption_typestring: (Optional) The encryption type, it could be sse-s3 or sse-kms. It will be used only when encryption is enabled and the default type is sse-s3.

kms_keystring: (Optional) Kms key which will be used if encryption is enabled and encryption type is set to sse-kms.

canned_aclstring: (Optional) Set canned access control list for the logs, e.g. bucket-owner-full-control. If canned_cal is set, please make sure the cluster iam role has s3:PutObjectAcl permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set bucket-owner-full-control to make bucket owner able to read the logs.

volumesobjectRequired

destination needs to be provided, e.g. { "volumes": { "destination": "/Volumes/catalog/schema/volume/cluster_log" } }

Show child attributesHide child attributes

destinationstring: UC Volumes destination, e.g. /Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh or dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh

spark_env_varsobject

autotermination_minutesint32: Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.

enable_elastic_diskboolean: Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWS
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

init_scriptsarray of object

Show child attributesHide child attributes

dbfsobjectRequired

destination needs to be provided. e.g. { "dbfs": { "destination" : "dbfs:/home/cluster_log" } }

Show child attributesHide child attributes

destinationstring: dbfs destination, e.g. dbfs:/my/path

s3objectRequired

Show child attributesHide child attributes

destinationstring: S3 destination, e.g. s3://my-bucket/some-prefix Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs.

regionstring: S3 region, e.g. us-west-2. Either region or endpoint needs to be set. If both are set, endpoint will be used.

endpointstring: S3 endpoint, e.g. https://s3-us-west-2.amazonaws.com. Either region or endpoint needs to be set. If both are set, endpoint will be used.

enable_encryptionboolean: (Optional) Flag to enable server side encryption, false by default.

encryption_typestring: (Optional) The encryption type, it could be sse-s3 or sse-kms. It will be used only when encryption is enabled and the default type is sse-s3.

kms_keystring: (Optional) Kms key which will be used if encryption is enabled and encryption type is set to sse-kms.

canned_aclstring: (Optional) Set canned access control list for the logs, e.g. bucket-owner-full-control. If canned_cal is set, please make sure the cluster iam role has s3:PutObjectAcl permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set bucket-owner-full-control to make bucket owner able to read the logs.

fileobjectRequired

destination needs to be provided, e.g. { "file": { "destination": "file:/my/local/file.sh" } }

Show child attributesHide child attributes

destinationstring: local file destination, e.g. file:/my/local/file.sh

gcsobjectRequired

destination needs to be provided, e.g. { "gcs": { "destination": "gs://my-bucket/file.sh" } }

Show child attributesHide child attributes

destinationstring: GCS destination/URI, e.g. gs://my-bucket/some-prefix

abfssobjectRequired

destination needs to be provided, e.g. abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>

Show child attributesHide child attributes

destinationstring: abfss destination, e.g. abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>.

workspaceobjectRequired

destination needs to be provided, e.g. { "workspace": { "destination": "/cluster-init-scripts/setup-datadog.sh" } }

Show child attributesHide child attributes

destinationstring: wsfs destination, e.g. workspace:/cluster-init-scripts/setup-datadog.sh

volumesobjectRequired

destination needs to be provided. e.g. { \"volumes\" : { \"destination\" : \"/Volumes/my-init.sh\" } }

Show child attributesHide child attributes

destinationstring: UC Volumes destination, e.g. /Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh or dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh

docker_imageobject

Custom docker image BYOC

Show child attributesHide child attributes

urlstring: URL of the docker image.

basic_authobjectRequired

Basic auth with username and password

Show child attributesHide child attributes

usernamestring: Name of the user

passwordstring: Password of the user

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

single_user_namestring: Single user name if data_security_mode is SINGLE_USER

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable LUKS on cluster VMs' local disks

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

workload_typeobject

Show child attributesHide child attributes

clientsobject

defined what type of clients can use the cluster. E.g. Notebooks, Jobs

Show child attributesHide child attributes

notebooksboolean: With notebooks set, this cluster can be used for notebooks
Default: true

jobsboolean: With jobs set, the cluster can be used for jobs
Default: true

data_security_modestring: Values: NONE, SINGLE_USER, USER_ISOLATION, LEGACY_TABLE_ACL, LEGACY_PASSTHROUGH, LEGACY_SINGLE_USER, LEGACY_SINGLE_USER_STANDARD, DATA_SECURITY_MODE_STANDARD, DATA_SECURITY_MODE_DEDICATED, DATA_SECURITY_MODE_AUTO

runtime_enginestring

Determines the cluster's runtime engine, either standard or Photon.

This field is not compatible with legacy spark_version values that contain -photon-. Remove -photon- from the spark_version and set runtime_engine to PHOTON.

If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.

Values: NULL, STANDARD, PHOTON

kindstring: Values: COMPUTE_KIND_UNSPECIFIED, CLASSIC_PREVIEW

use_ml_runtimeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

effective_spark_version is determined by spark_version (DBR release), this field use_ml_runtime, and whether node_type_id is gpu node or not.

is_single_nodeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

When set to true, <Databricks> will automatically set single node related custom_tags, spark_conf, and num_workers

remote_disk_throughputint32: If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.

total_initial_remote_disk_sizeint32: If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.

dependency_modestringBeta: Controls dependency configuration for the cluster.
Values: DEPENDENCY_MODE_UNSPECIFIED, DEPENDENCY_MODE_ENVIRONMENTS, DEPENDENCY_MODE_CLUSTER_LIBRARIES, DEPENDENCY_MODE_AUTO

Delete GA

POST /api/2.1/clusters/permanent-delete

Permanently deletes a Spark cluster. This cluster is terminated and resources are asynchronously removed.

In addition, users will no longer see permanently deleted clusters in the cluster list, and API users can no longer perform any action on permanently deleted clusters.

API scopes: clusters

Request body

cluster_idstring: The cluster to be deleted.
Example: 1202-211320-brick1

Change Cluster Owner GA

POST /api/2.1/clusters/change-owner

Change the owner of the cluster. You must be an admin and the cluster must be terminated to perform this operation. The service principal application ID can be supplied as an argument to owner_username.

API scopes: clusters

Request body

cluster_idstring

owner_usernamestring: New owner of the cluster_id after this RPC.

Delete Cluster GA

POST /api/2.1/clusters/delete

Terminates the Spark cluster with the specified ID. The cluster is removed asynchronously. Once the termination has completed, the cluster will be in a TERMINATED state. If the cluster is already in a TERMINATING or TERMINATED state, nothing will happen.

API scopes: clusters

Request body

cluster_idstring: The cluster to be terminated.
Example: 1234-567890-frays123

List Available Zones GA

GET /api/2.1/clusters/list-zones

Returns a list of availability zones where clusters can be created in (For example, us-west-2a). These zones can be used to launch a cluster.

API scopes: clusters

Response

zonesarray of string: The list of available zones (e.g., ['us-west-2c', 'us-east-2']).

default_zonestring: The availability zone if no zone_id is provided in the cluster creation request.

Pin Cluster GA

POST /api/2.1/clusters/pin

Pinning a cluster ensures that the cluster will always be returned by the ListClusters API. Pinning a cluster that is already pinned will have no effect. This API can only be called by workspace admins.

API scopes: clusters

Request body

cluster_idstring: Example: 1234-567890-reef123

Restart Cluster GA

POST /api/2.1/clusters/restart

Restarts a Spark cluster with the supplied ID. If the cluster is not currently in a RUNNING state, nothing will happen.

API scopes: clusters

Request body

cluster_idstring: The cluster to be started.

restart_userstring

Resize Cluster GA

POST /api/2.1/clusters/resize

Resizes a cluster to have a desired number of workers. This will fail unless the cluster is in a RUNNING state.

API scopes: clusters

Request body

cluster_idstring: The cluster to be resized.

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired

Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

Show child attributesHide child attributes

min_workersint32: The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.

max_workersint32: The maximum number of workers to which the cluster can scale up when overloaded. Note that max_workers must be strictly greater than min_workers.

Start Cluster GA

POST /api/2.1/clusters/start

Starts a terminated Spark cluster with the supplied ID. This works similar to createCluster except:

The previous cluster id and attributes are preserved.
The cluster starts with the last specified cluster size.
- If the previous cluster was an autoscaling cluster, the current cluster starts with the minimum number of nodes.
If the cluster is not currently in a TERMINATED state, nothing will happen.
Clusters launched to run a job cannot be started.

API scopes: clusters

Request body

cluster_idstring: The cluster to be started.
Example: 1202-211320-brick1

Unpin Cluster GA

POST /api/2.1/clusters/unpin

Unpinning a cluster will allow the cluster to eventually be removed from the ListClusters API. Unpinning a cluster that is not pinned will have no effect. This API can only be called by workspace admins.

API scopes: clusters

Request body

cluster_idstring: Example: 1234-567890-reef123

Update Cluster GA

POST /api/2.1/clusters/update

Updates the configuration of a cluster to match the partial set of attributes and size. Denote which fields to update using the update_mask field in the request body. A cluster can be updated if it is in a RUNNING or TERMINATED state. If a cluster is updated while in a RUNNING state, it will be restarted so that the new attributes can take effect. If a cluster is updated while in a TERMINATED state, it will remain TERMINATED. The updated attributes will take effect the next time the cluster is started using the clusters/start API. Attempts to update a cluster in any other state will be rejected with an INVALID_STATE error code. Clusters created by the Databricks Jobs service cannot be updated.

API scopes: clusters

CloudsAWSAzure

Request body

cluster_idstring: ID of the cluster.

clusterobject

The cluster to be updated.

Show child attributesHide child attributes

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired

Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

Show child attributesHide child attributes

min_workersint32: The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.

max_workersint32: The maximum number of workers to which the cluster can scale up when overloaded. Note that max_workers must be strictly greater than min_workers.

cluster_namestring: Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.

spark_versionstring: The Spark version of the cluster, e.g. 3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

aws_attributesobject

Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 0

availabilitystring: Default: SPOT_WITH_FALLBACK
Values: SPOT, ON_DEMAND, SPOT_WITH_FALLBACK

zone_idstring

The list of available zones as well as the default value can be found by using the List Zones method.

instance_profile_arnstring

This feature may only be available to certain customer plans.

spot_bid_price_percentint32: The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new r3.xlarge spot instance, then the bid price is half of the price of on-demand r3.xlarge instances. Similarly, if this field is set to 200, the bid price is twice the price of on-demand r3.xlarge instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.
Default: 100

ebs_volume_typestring: The type of EBS volumes that will be launched with this cluster.
Values: GENERAL_PURPOSE_SSD, THROUGHPUT_OPTIMIZED_HDD

ebs_volume_countint32

These EBS volumes will be mounted at /ebs0, /ebs1, and etc. Instance store volumes will be mounted at /local_disk0, /local_disk1, and etc.

Please note that if EBS volumes are specified, then the Spark configuration spark.local.dir will be overridden.

Default: 0

ebs_volume_sizeint32: The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.

ebs_volume_iopsint32: If using gp3 volumes, what IOPS to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.

ebs_volume_throughputint32: If using gp3 volumes, what throughput to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.

azure_attributesobject

Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

log_analytics_infoobject

Defines values necessary to configure and run Azure Log Analytics agent

Show child attributesHide child attributes

log_analytics_workspace_idstring

log_analytics_primary_keystring

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 1

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero, this availability type will be used for the entire cluster.
Default: ON_DEMAND_AZURE
Values: SPOT_AZURE, ON_DEMAND_AZURE, SPOT_WITH_FALLBACK_AZURE

spot_bid_max_pricedouble: The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.
Default: -1

capacity_reservation_groupstring

The Azure capacity reservation group resource ID to use for launching VMs. When specified, VMs will be launched using the provided capacity reservation.

Microsoft.Compute/capacityReservationGroups/read
Microsoft.Compute/capacityReservationGroups/deploy/action
Microsoft.Compute/capacityReservationGroups/capacityReservations/read
Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action

Format: /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring

The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

worker_node_type_flexibilityobject

Flexible node type configuration for worker nodes.

Show child attributesHide child attributes

alternate_node_type_idsarray of string: A list of node type IDs to use as fallbacks when the primary node type is unavailable.

driver_node_type_flexibilityobject

Flexible node type configuration for the driver node.

Show child attributesHide child attributes

alternate_node_type_idsarray of string: A list of node type IDs to use as fallbacks when the primary node type is unavailable.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject

Show child attributesHide child attributes

dbfsobjectRequired

destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }

Show child attributesHide child attributes

destinationstring: dbfs destination, e.g. dbfs:/my/path

s3objectRequired

Show child attributesHide child attributes

destinationstring: S3 destination, e.g. s3://my-bucket/some-prefix Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs.

regionstring: S3 region, e.g. us-west-2. Either region or endpoint needs to be set. If both are set, endpoint will be used.

endpointstring: S3 endpoint, e.g. https://s3-us-west-2.amazonaws.com. Either region or endpoint needs to be set. If both are set, endpoint will be used.

enable_encryptionboolean: (Optional) Flag to enable server side encryption, false by default.

encryption_typestring: (Optional) The encryption type, it could be sse-s3 or sse-kms. It will be used only when encryption is enabled and the default type is sse-s3.

kms_keystring: (Optional) Kms key which will be used if encryption is enabled and encryption type is set to sse-kms.

canned_aclstring: (Optional) Set canned access control list for the logs, e.g. bucket-owner-full-control. If canned_cal is set, please make sure the cluster iam role has s3:PutObjectAcl permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set bucket-owner-full-control to make bucket owner able to read the logs.

volumesobjectRequired

destination needs to be provided, e.g. { "volumes": { "destination": "/Volumes/catalog/schema/volume/cluster_log" } }

Show child attributesHide child attributes

destinationstring: UC Volumes destination, e.g. /Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh or dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh

spark_env_varsobject

autotermination_minutesint32: Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.

enable_elastic_diskboolean: Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWS
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

init_scriptsarray of object

Show child attributesHide child attributes

dbfsobjectRequired

destination needs to be provided. e.g. { "dbfs": { "destination" : "dbfs:/home/cluster_log" } }

Show child attributesHide child attributes

destinationstring: dbfs destination, e.g. dbfs:/my/path

s3objectRequired

Show child attributesHide child attributes

destinationstring: S3 destination, e.g. s3://my-bucket/some-prefix Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs.

regionstring: S3 region, e.g. us-west-2. Either region or endpoint needs to be set. If both are set, endpoint will be used.

endpointstring: S3 endpoint, e.g. https://s3-us-west-2.amazonaws.com. Either region or endpoint needs to be set. If both are set, endpoint will be used.

enable_encryptionboolean: (Optional) Flag to enable server side encryption, false by default.

encryption_typestring: (Optional) The encryption type, it could be sse-s3 or sse-kms. It will be used only when encryption is enabled and the default type is sse-s3.

kms_keystring: (Optional) Kms key which will be used if encryption is enabled and encryption type is set to sse-kms.

canned_aclstring: (Optional) Set canned access control list for the logs, e.g. bucket-owner-full-control. If canned_cal is set, please make sure the cluster iam role has s3:PutObjectAcl permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set bucket-owner-full-control to make bucket owner able to read the logs.

fileobjectRequired

destination needs to be provided, e.g. { "file": { "destination": "file:/my/local/file.sh" } }

Show child attributesHide child attributes

destinationstring: local file destination, e.g. file:/my/local/file.sh

gcsobjectRequired

destination needs to be provided, e.g. { "gcs": { "destination": "gs://my-bucket/file.sh" } }

Show child attributesHide child attributes

destinationstring: GCS destination/URI, e.g. gs://my-bucket/some-prefix

abfssobjectRequired

destination needs to be provided, e.g. abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>

Show child attributesHide child attributes

destinationstring: abfss destination, e.g. abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>.

workspaceobjectRequired

destination needs to be provided, e.g. { "workspace": { "destination": "/cluster-init-scripts/setup-datadog.sh" } }

Show child attributesHide child attributes

destinationstring: wsfs destination, e.g. workspace:/cluster-init-scripts/setup-datadog.sh

volumesobjectRequired

destination needs to be provided. e.g. { \"volumes\" : { \"destination\" : \"/Volumes/my-init.sh\" } }

Show child attributesHide child attributes

destinationstring: UC Volumes destination, e.g. /Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh or dbfs:/Volumes/catalog/schema/vol1/init-scripts/setup-datadog.sh

docker_imageobject

Custom docker image BYOC

Show child attributesHide child attributes

urlstring: URL of the docker image.

basic_authobjectRequired

Basic auth with username and password

Show child attributesHide child attributes

usernamestring: Name of the user

passwordstring: Password of the user

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

single_user_namestring: Single user name if data_security_mode is SINGLE_USER

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable LUKS on cluster VMs' local disks

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

workload_typeobject

Show child attributesHide child attributes

clientsobject

defined what type of clients can use the cluster. E.g. Notebooks, Jobs

Show child attributesHide child attributes

notebooksboolean: With notebooks set, this cluster can be used for notebooks
Default: true

jobsboolean: With jobs set, the cluster can be used for jobs
Default: true

data_security_modestring: Values: NONE, SINGLE_USER, USER_ISOLATION, LEGACY_TABLE_ACL, LEGACY_PASSTHROUGH, LEGACY_SINGLE_USER, LEGACY_SINGLE_USER_STANDARD, DATA_SECURITY_MODE_STANDARD, DATA_SECURITY_MODE_DEDICATED, DATA_SECURITY_MODE_AUTO

runtime_enginestring

Determines the cluster's runtime engine, either standard or Photon.

This field is not compatible with legacy spark_version values that contain -photon-. Remove -photon- from the spark_version and set runtime_engine to PHOTON.

If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.

Values: NULL, STANDARD, PHOTON

kindstring: Values: COMPUTE_KIND_UNSPECIFIED, CLASSIC_PREVIEW

use_ml_runtimeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

effective_spark_version is determined by spark_version (DBR release), this field use_ml_runtime, and whether node_type_id is gpu node or not.

is_single_nodeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

When set to true, <Databricks> will automatically set single node related custom_tags, spark_conf, and num_workers

dependency_modestringBeta: Controls dependency configuration for the cluster.
Values: DEPENDENCY_MODE_UNSPECIFIED, DEPENDENCY_MODE_ENVIRONMENTS, DEPENDENCY_MODE_CLUSTER_LIBRARIES, DEPENDENCY_MODE_AUTO

update_maskstring: Used to specify which cluster attributes and size fields to update. See https://google.aip.dev/161 for more details.
Example: num_workers,spark_version,spark_conf

View as Markdown

Request

{
  "cluster_id": "1202-211320-brick1",
  "update_mask": "custom_tags"
}

{
  "cluster": {
    "num_workers": 10,
    "spark_conf": {
      "spark.databricks.cluster.profile": "singleNode",
      "spark.master": "[*, 4]"
    },
    "spark_version": "14.3.x-scala2.12"
  },
  "cluster_id": "1202-211320-brick1",
  "update_mask": "num_workers,spark_version,spark_conf"
}

Response

{}

Get GA​

Parameters​

Response​

List GA​

Parameters​

Response​

Create GA​

Request body​

Response​

Update GA​

Request body​

Delete GA​

Request body​

Change Cluster Owner GA​

Request body​

Delete Cluster GA​

Request body​

List Available Zones GA​

Response​

Pin Cluster GA​

Request body​

Restart Cluster GA​

Request body​

Resize Cluster GA​

Request body​

Start Cluster GA​

Request body​

Unpin Cluster GA​

Request body​

Update Cluster GA​

Request body​

Get GA

Parameters

Response

List GA

Parameters

Response

Create GA

Request body

Response

Update GA

Request body

Delete GA

Request body

Change Cluster Owner GA

Request body

Delete Cluster GA

Request body

List Available Zones GA

Response

Pin Cluster GA

Request body

Restart Cluster GA

Request body

Resize Cluster GA

Request body

Start Cluster GA

Request body

Unpin Cluster GA

Request body

Update Cluster GA

Request body