Clusters

View as Markdown

Use and configure compute for Databricks

List EventsPOST /api/2.1/clusters/events

List Events GA

POST /api/2.1/clusters/events

Retrieves a list of events about the activity of a cluster. This API is paginated. If there are more events to read, the response includes all the parameters necessary to request the next page of events.

API scopes: clusters

Request body

cluster_idstring

The ID of the cluster to retrieve events about.

Example: 1234-567890-reef123

start_timeint64

The start time in epoch milliseconds. If empty, returns events starting from the beginning of time.

Example: 1617238800000

end_timeint64

The end time in epoch milliseconds. If empty, returns events up to the current time.

Example: 1619485200000

orderstring

The order to list events in; either "ASC" or "DESC". Defaults to "DESC".

Default: DESC

Values: DESC, ASC

event_typesarray of string

An optional set of event types to filter on. If empty, all event types are returned.

Values: CREATING, STARTING, RESTARTING, TERMINATING, EDITED, RUNNING, RESIZING, NODES_LOST, UPSIZE_COMPLETED, INIT_SCRIPTS_STARTED, INIT_SCRIPTS_FINISHED, DID_NOT_EXPAND_DISK, EXPANDED_DISK, FAILED_TO_EXPAND_DISK, DRIVER_HEALTHY, DRIVER_NOT_RESPONDING, DRIVER_UNAVAILABLE, SPARK_EXCEPTION, METASTORE_DOWN, DBFS_DOWN, AUTOSCALING_STATS_REPORT, NODE_BLACKLISTED, PINNED, UNPINNED, NODE_EXCLUDED_DECOMMISSIONED, ADD_NODES_FAILED, AUTOSCALING_BACKOFF, AUTOMATIC_CLUSTER_UPDATE, AUTOSCALING_FAILED, CLUSTER_MIGRATED, DECOMMISSION_STARTED, DECOMMISSION_ENDED, DEFERRED_POLICY_ENFORCEMENT_SCHEDULED, DEFERRED_POLICY_ENFORCEMENT_FAILED, UC_VOLUME_MISCONFIGURED

offsetint64

Deprecated: use page_token in combination with page_size instead.

The offset in the result set. Defaults to 0 (no offset). When an offset is specified and the results are requested in descending order, the end_time field is required.

Example: 5

limitint64

Deprecated: use page_token in combination with page_size instead.

The maximum number of events to include in a page of events. Defaults to 50, and maximum allowed value is 500.

Default: 50

Constraints: <= 500

page_tokenstring

Use next_page_token or prev_page_token returned from the previous request to list the next or previous page of events respectively. If page_token is empty, the first page is returned.

page_sizeint32

The maximum number of events to include in a page of events. The server may further constrain the maximum number of results returned in a single page. If the page_size is empty or 0, the server will decide the number of results to be returned. The field has to be in the range [0,500]. If the value is outside the range, the server enforces 0 or 500.

Default: 50

Response

eventsarray of object

Show child attributesHide child attributes

cluster_idstring

timestampint64

The timestamp when the event occurred, stored as the number of milliseconds since the Unix epoch. If not provided, this will be assigned by the Timeline service.

typestring

Values: CREATING, STARTING, RESTARTING, TERMINATING, EDITED, RUNNING, RESIZING, NODES_LOST, UPSIZE_COMPLETED, INIT_SCRIPTS_STARTED, INIT_SCRIPTS_FINISHED, DID_NOT_EXPAND_DISK, EXPANDED_DISK, FAILED_TO_EXPAND_DISK, DRIVER_HEALTHY, DRIVER_NOT_RESPONDING, DRIVER_UNAVAILABLE, SPARK_EXCEPTION, METASTORE_DOWN, DBFS_DOWN, AUTOSCALING_STATS_REPORT, NODE_BLACKLISTED, PINNED, UNPINNED, NODE_EXCLUDED_DECOMMISSIONED, ADD_NODES_FAILED, AUTOSCALING_BACKOFF, AUTOMATIC_CLUSTER_UPDATE, AUTOSCALING_FAILED, CLUSTER_MIGRATED, DECOMMISSION_STARTED, DECOMMISSION_ENDED, DEFERRED_POLICY_ENFORCEMENT_SCHEDULED, DEFERRED_POLICY_ENFORCEMENT_FAILED, UC_VOLUME_MISCONFIGURED

detailsobject

Show child attributesHide child attributes

current_num_workersint32

The current number of nodes in the cluster.

target_num_workersint32

The targeted number of nodes in the cluster.

previous_attributesobject

The cluster attributes before a cluster was edited.

Show child attributesHide child attributes

cluster_namestring

Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.

spark_versionstring

The Spark version of the cluster, e.g. 3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

spark_confobject

An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

aws_attributesobject

Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

azure_attributesobject

Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

gcp_attributesobject

Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

node_type_idstring

This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring

The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence.

worker_node_type_flexibilityobject

Flexible node type configuration for worker nodes.

driver_node_type_flexibilityobject

Flexible node type configuration for the driver node.

ssh_public_keysarray of string

SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

• Currently, <Databricks> allows at most 45 custom tags

• Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject

The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every 5 mins. The destination of driver logs is $destination/$clusterId/driver, while the destination of executor logs is $destination/$clusterId/executor.

spark_env_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e., export X='Y') while launching the driver and workers.

In order to specify an additional set of SPARK_DAEMON_JAVA_OPTS, we recommend appending them to $SPARK_DAEMON_JAVA_OPTS as shown in the example below. This ensures that all default databricks managed environmental variables are included as well.

Example Spark environment variables: {"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"} or {"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}

autotermination_minutesint32

Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.

enable_elastic_diskboolean

Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.

AWS

Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

init_scriptsarray of object

The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If cluster_log_conf is specified, init script logs are sent to <destination>/<cluster-ID>/init_scripts.

docker_imageobject

Custom docker image BYOC

instance_pool_idstring

The optional ID of the instance pool to which the cluster belongs.

single_user_namestring

Single user name if data_security_mode is SINGLE_USER

policy_idstring

The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean

Whether to enable LUKS on cluster VMs' local disks

driver_instance_pool_idstring

The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

workload_typeobject

data_security_modestring

Values: NONE, SINGLE_USER, USER_ISOLATION, LEGACY_TABLE_ACL, LEGACY_PASSTHROUGH, LEGACY_SINGLE_USER, LEGACY_SINGLE_USER_STANDARD, DATA_SECURITY_MODE_STANDARD, DATA_SECURITY_MODE_DEDICATED, DATA_SECURITY_MODE_AUTO

runtime_enginestring

Determines the cluster's runtime engine, either standard or Photon.

This field is not compatible with legacy spark_version values that contain -photon-. Remove -photon- from the spark_version and set runtime_engine to PHOTON.

If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.

Values: NULL, STANDARD, PHOTON

kindstring

Values: COMPUTE_KIND_UNSPECIFIED, CLASSIC_PREVIEW

use_ml_runtimeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

effective_spark_version is determined by spark_version (DBR release), this field use_ml_runtime, and whether node_type_id is gpu node or not.

is_single_nodeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

When set to true, <Databricks> will automatically set single node related custom_tags, spark_conf, and num_workers

remote_disk_throughputint32

If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.

total_initial_remote_disk_sizeint32

If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.

dependency_modestringBeta

Controls dependency configuration for the cluster.

Values: DEPENDENCY_MODE_UNSPECIFIED, DEPENDENCY_MODE_ENVIRONMENTS, DEPENDENCY_MODE_CLUSTER_LIBRARIES, DEPENDENCY_MODE_AUTO

attributesobject

• For created clusters, the attributes of the cluster.
• For edited clusters, the new attributes of the cluster.

Show child attributesHide child attributes

cluster_namestring

Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.

spark_versionstring

The Spark version of the cluster, e.g. 3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

spark_confobject

An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

aws_attributesobject

Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

azure_attributesobject

Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

gcp_attributesobject

Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

node_type_idstring

This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring

The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence.

worker_node_type_flexibilityobject

Flexible node type configuration for worker nodes.

driver_node_type_flexibilityobject

Flexible node type configuration for the driver node.

ssh_public_keysarray of string

SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

• Currently, <Databricks> allows at most 45 custom tags

• Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject

The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every 5 mins. The destination of driver logs is $destination/$clusterId/driver, while the destination of executor logs is $destination/$clusterId/executor.

spark_env_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e., export X='Y') while launching the driver and workers.

In order to specify an additional set of SPARK_DAEMON_JAVA_OPTS, we recommend appending them to $SPARK_DAEMON_JAVA_OPTS as shown in the example below. This ensures that all default databricks managed environmental variables are included as well.

Example Spark environment variables: {"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"} or {"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}

autotermination_minutesint32

Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.

enable_elastic_diskboolean

Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.

AWS

Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

init_scriptsarray of object

The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If cluster_log_conf is specified, init script logs are sent to <destination>/<cluster-ID>/init_scripts.

docker_imageobject

Custom docker image BYOC

instance_pool_idstring

The optional ID of the instance pool to which the cluster belongs.

single_user_namestring

Single user name if data_security_mode is SINGLE_USER

policy_idstring

The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean

Whether to enable LUKS on cluster VMs' local disks

driver_instance_pool_idstring

The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

workload_typeobject

data_security_modestring

Values: NONE, SINGLE_USER, USER_ISOLATION, LEGACY_TABLE_ACL, LEGACY_PASSTHROUGH, LEGACY_SINGLE_USER, LEGACY_SINGLE_USER_STANDARD, DATA_SECURITY_MODE_STANDARD, DATA_SECURITY_MODE_DEDICATED, DATA_SECURITY_MODE_AUTO

runtime_enginestring

Determines the cluster's runtime engine, either standard or Photon.

This field is not compatible with legacy spark_version values that contain -photon-. Remove -photon- from the spark_version and set runtime_engine to PHOTON.

If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.

Values: NULL, STANDARD, PHOTON

kindstring

Values: COMPUTE_KIND_UNSPECIFIED, CLASSIC_PREVIEW

use_ml_runtimeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

effective_spark_version is determined by spark_version (DBR release), this field use_ml_runtime, and whether node_type_id is gpu node or not.

is_single_nodeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

When set to true, <Databricks> will automatically set single node related custom_tags, spark_conf, and num_workers

remote_disk_throughputint32

If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.

total_initial_remote_disk_sizeint32

If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.

dependency_modestringBeta

Controls dependency configuration for the cluster.

Values: DEPENDENCY_MODE_UNSPECIFIED, DEPENDENCY_MODE_ENVIRONMENTS, DEPENDENCY_MODE_CLUSTER_LIBRARIES, DEPENDENCY_MODE_AUTO

previous_cluster_sizeobject

The size of the cluster before an edit or resize.

Show child attributesHide child attributes

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in spark_info will gradually increase from 5 to 10 as the new nodes are provisioned.

autoscaleobjectRequired

Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

cluster_sizeobject

The actual cluster size that was set in the cluster creation or edit.

Show child attributesHide child attributes

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in spark_info will gradually increase from 5 to 10 as the new nodes are provisioned.

autoscaleobjectRequired

Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

causestring

The cause of a change in target size.

Values: AUTOSCALE, USER_REQUEST, AUTORECOVERY, REPLACE_BAD_NODES, AUTOSCALE_V2, DBR_AUTOSCALE

reasonobject

A termination reason:

• On a TERMINATED event, this is the reason of the termination.
• On a RESIZE_COMPLETE event, this indicates the reason that we failed to acquire some nodes.

Show child attributesHide child attributes

codestring

status code indicating why the cluster was terminated

Values: UNKNOWN, USER_REQUEST, JOB_FINISHED, INACTIVITY, CLOUD_PROVIDER_SHUTDOWN, COMMUNICATION_LOST, CLOUD_PROVIDER_LAUNCH_FAILURE, INIT_SCRIPT_FAILURE, SPARK_STARTUP_FAILURE, INVALID_ARGUMENT, UNEXPECTED_LAUNCH_FAILURE, INTERNAL_ERROR, INSTANCE_UNREACHABLE, REQUEST_REJECTED, TRIAL_EXPIRED, DRIVER_UNREACHABLE, SPARK_ERROR, DRIVER_UNRESPONSIVE, METASTORE_COMPONENT_UNHEALTHY, DBFS_COMPONENT_UNHEALTHY, EXECUTION_COMPONENT_UNHEALTHY, AZURE_RESOURCE_MANAGER_THROTTLING, AZURE_RESOURCE_PROVIDER_THROTTLING, NETWORK_CONFIGURATION_FAILURE, CONTAINER_LAUNCH_FAILURE, INSTANCE_POOL_CLUSTER_FAILURE, SKIPPED_SLOW_NODES, ATTACH_PROJECT_FAILURE, UPDATE_INSTANCE_PROFILE_FAILURE, DATABASE_CONNECTION_FAILURE, REQUEST_THROTTLED, SELF_BOOTSTRAP_FAILURE, GLOBAL_INIT_SCRIPT_FAILURE, SLOW_IMAGE_DOWNLOAD, INVALID_SPARK_IMAGE, NPIP_TUNNEL_TOKEN_FAILURE, HIVE_METASTORE_PROVISIONING_FAILURE, AZURE_INVALID_DEPLOYMENT_TEMPLATE, AZURE_UNEXPECTED_DEPLOYMENT_TEMPLATE_FAILURE, SUBNET_EXHAUSTED_FAILURE, BOOTSTRAP_TIMEOUT, STORAGE_DOWNLOAD_FAILURE, CONTROL_PLANE_REQUEST_FAILURE, BOOTSTRAP_TIMEOUT_CLOUD_PROVIDER_EXCEPTION, AWS_INSUFFICIENT_INSTANCE_CAPACITY_FAILURE, DOCKER_IMAGE_PULL_FAILURE, AZURE_VNET_CONFIGURATION_FAILURE, NPIP_TUNNEL_SETUP_FAILURE, AWS_AUTHORIZATION_FAILURE, NEPHOS_RESOURCE_MANAGEMENT, STS_CLIENT_SETUP_FAILURE, SECURITY_DAEMON_REGISTRATION_EXCEPTION, AWS_REQUEST_LIMIT_EXCEEDED, AWS_INSUFFICIENT_FREE_ADDRESSES_IN_SUBNET_FAILURE, AWS_UNSUPPORTED_FAILURE, AZURE_QUOTA_EXCEEDED_EXCEPTION, AZURE_OPERATION_NOT_ALLOWED_EXCEPTION, NFS_MOUNT_FAILURE, K8S_AUTOSCALING_FAILURE, K8S_DBR_CLUSTER_LAUNCH_TIMEOUT, SPARK_IMAGE_DOWNLOAD_FAILURE, AZURE_VM_EXTENSION_FAILURE, WORKSPACE_CANCELLED_ERROR, AWS_MAX_SPOT_INSTANCE_COUNT_EXCEEDED_FAILURE, TEMPORARILY_UNAVAILABLE, WORKER_SETUP_FAILURE, IP_EXHAUSTION_FAILURE, GCP_QUOTA_EXCEEDED, CLOUD_PROVIDER_RESOURCE_STOCKOUT, GCP_SERVICE_ACCOUNT_DELETED, AZURE_BYOK_KEY_PERMISSION_FAILURE, SPOT_INSTANCE_TERMINATION, AZURE_EPHEMERAL_DISK_FAILURE, ABUSE_DETECTED, IMAGE_PULL_PERMISSION_DENIED, WORKSPACE_CONFIGURATION_ERROR, SECRET_RESOLUTION_ERROR, UNSUPPORTED_INSTANCE_TYPE, CLOUD_PROVIDER_DISK_SETUP_FAILURE, SSH_BOOTSTRAP_FAILURE, AWS_INACCESSIBLE_KMS_KEY_FAILURE, INIT_CONTAINER_NOT_FINISHED, SPARK_IMAGE_DOWNLOAD_THROTTLED, SPARK_IMAGE_NOT_FOUND, CLUSTER_OPERATION_THROTTLED, CLUSTER_OPERATION_TIMEOUT, SERVERLESS_LONG_RUNNING_TERMINATED, AZURE_PACKED_DEPLOYMENT_PARTIAL_FAILURE, INVALID_WORKER_IMAGE_FAILURE, WORKSPACE_UPDATE, INVALID_AWS_PARAMETER, DRIVER_OUT_OF_DISK, DRIVER_OUT_OF_MEMORY, DRIVER_LAUNCH_TIMEOUT, DRIVER_UNEXPECTED_FAILURE, UNEXPECTED_POD_RECREATION, GCP_INACCESSIBLE_KMS_KEY_FAILURE, GCP_KMS_KEY_PERMISSION_DENIED, DRIVER_EVICTION, USER_INITIATED_VM_TERMINATION, GCP_IAM_TIMEOUT, AWS_RESOURCE_QUOTA_EXCEEDED, CLOUD_ACCOUNT_SETUP_FAILURE, AWS_INVALID_KEY_PAIR, DRIVER_POD_CREATION_FAILURE, MAINTENANCE_MODE, INTERNAL_CAPACITY_FAILURE, EXECUTOR_POD_UNSCHEDULED, STORAGE_DOWNLOAD_FAILURE_SLOW, STORAGE_DOWNLOAD_FAILURE_THROTTLED, DYNAMIC_SPARK_CONF_SIZE_EXCEEDED, AWS_INSTANCE_PROFILE_UPDATE_FAILURE, INSTANCE_POOL_NOT_FOUND, INSTANCE_POOL_MAX_CAPACITY_REACHED, AWS_INVALID_KMS_KEY_STATE, GCP_INSUFFICIENT_CAPACITY, GCP_API_RATE_QUOTA_EXCEEDED, GCP_RESOURCE_QUOTA_EXCEEDED, GCP_IP_SPACE_EXHAUSTED, GCP_SERVICE_ACCOUNT_ACCESS_DENIED, GCP_SERVICE_ACCOUNT_NOT_FOUND, GCP_FORBIDDEN, GCP_NOT_FOUND, RESOURCE_USAGE_BLOCKED, DATA_ACCESS_CONFIG_CHANGED, ACCESS_TOKEN_FAILURE, INVALID_INSTANCE_PLACEMENT_PROTOCOL, BUDGET_POLICY_RESOLUTION_FAILURE, IN_PENALTY_BOX, DISASTER_RECOVERY_REPLICATION, BOOTSTRAP_TIMEOUT_DUE_TO_MISCONFIG, INSTANCE_UNREACHABLE_DUE_TO_MISCONFIG, STORAGE_DOWNLOAD_FAILURE_DUE_TO_MISCONFIG, CONTROL_PLANE_REQUEST_FAILURE_DUE_TO_MISCONFIG, CLOUD_PROVIDER_LAUNCH_FAILURE_DUE_TO_MISCONFIG, GCP_SUBNET_NOT_READY, CLOUD_OPERATION_CANCELLED, CLOUD_PROVIDER_INSTANCE_NOT_LAUNCHED, GCP_TRUSTED_IMAGE_PROJECTS_VIOLATED, BUDGET_POLICY_LIMIT_ENFORCEMENT_ACTIVATED, EOS_SPARK_IMAGE, NO_MATCHED_K8S, LAZY_ALLOCATION_TIMEOUT, DRIVER_NODE_UNREACHABLE, SECRET_CREATION_FAILURE, POD_SCHEDULING_FAILURE, POD_ASSIGNMENT_FAILURE, ALLOCATION_TIMEOUT, ALLOCATION_TIMEOUT_NO_UNALLOCATED_CLUSTERS, ALLOCATION_TIMEOUT_NO_MATCHED_CLUSTERS, ALLOCATION_TIMEOUT_NO_READY_CLUSTERS, ALLOCATION_TIMEOUT_NO_WARMED_UP_CLUSTERS, ALLOCATION_TIMEOUT_NODE_DAEMON_NOT_READY, ALLOCATION_TIMEOUT_NO_HEALTHY_CLUSTERS, NETVISOR_SETUP_TIMEOUT, NO_MATCHED_K8S_TESTING_TAG, CLOUD_PROVIDER_RESOURCE_STOCKOUT_DUE_TO_MISCONFIG, GKE_BASED_CLUSTER_TERMINATION, ALLOCATION_TIMEOUT_NO_HEALTHY_AND_WARMED_UP_CLUSTERS, DOCKER_INVALID_OS_EXCEPTION, DOCKER_CONTAINER_CREATION_EXCEPTION, DOCKER_IMAGE_TOO_LARGE_FOR_INSTANCE_EXCEPTION, DNS_RESOLUTION_ERROR, GCP_DENIED_BY_ORG_POLICY, SECRET_PERMISSION_DENIED, NETWORK_CHECK_NIC_FAILURE, NETWORK_CHECK_DNS_SERVER_FAILURE, NETWORK_CHECK_STORAGE_FAILURE, NETWORK_CHECK_METADATA_ENDPOINT_FAILURE, NETWORK_CHECK_CONTROL_PLANE_FAILURE, NETWORK_CHECK_MULTIPLE_COMPONENTS_FAILURE, DRIVER_UNHEALTHY, USAGE_POLICY_ENTITLEMENT_DENIED, K8S_ACTIVE_POD_QUOTA_EXCEEDED, CLOUD_ACCOUNT_POD_QUOTA_EXCEEDED, NETWORK_CHECK_NIC_FAILURE_DUE_TO_MISCONFIG, NETWORK_CHECK_DNS_SERVER_FAILURE_DUE_TO_MISCONFIG, NETWORK_CHECK_STORAGE_FAILURE_DUE_TO_MISCONFIG, NETWORK_CHECK_METADATA_ENDPOINT_FAILURE_DUE_TO_MISCONFIG, NETWORK_CHECK_CONTROL_PLANE_FAILURE_DUE_TO_MISCONFIG, NETWORK_CHECK_MULTIPLE_COMPONENTS_FAILURE_DUE_TO_MISCONFIG, DBR_IMAGE_RESOLUTION_FAILURE, CONTROL_PLANE_CONNECTION_FAILURE, CONTROL_PLANE_CONNECTION_FAILURE_DUE_TO_MISCONFIG, RATE_LIMITED, MTLS_PORT_CONNECTIVITY_FAILURE, HIVEMETASTORE_CONNECTIVITY_FAILURE

typestring

type of the termination

Values: SUCCESS, CLIENT_ERROR, SERVICE_FAULT, CLOUD_FAILURE

parametersobject

list of parameters that provide additional information about why the cluster was terminated

userstring

The user that caused the event to occur. (Empty if it was done by the control plane.)

previous_disk_sizeint64

Previous disk size in bytes

disk_sizeint64

Current disk size in bytes

free_spaceint64

instance_idstring

Instance Id where the event originated from

did_not_expand_reasonstring

driver_state_messagestring

More details about the change in driver's state

job_run_namestring

Unique identifier of the specific job run associated with this cluster event

• For clusters created for jobs, this will be the same as the cluster name

init_scriptsobject

List of global and cluster init scripts associated with this cluster event.

Show child attributesHide child attributes

reported_for_nodestring

The private ip of the node we are reporting init script execution details for (we will select the execution details from only one node rather than reporting the execution details from every node to keep these event details small)

This should only be defined for the INIT_SCRIPTS_FINISHED event

globalarray of object

The global init scripts associated with this cluster event.

clusterarray of object

The cluster scoped init scripts associated with this cluster event.

enable_termination_for_node_blocklistedboolean

Whether or not a blocklisted node should be terminated. For ClusterEventType NODE_BLACKLISTED.

current_num_vcpusint32

The current number of vCPUs in the cluster.

target_num_vcpusint32

The targeted number of vCPUs in the cluster.

data_plane_event_detailsobject

Show child attributesHide child attributes

event_typestring

Values: NODE_BLACKLISTED, NODE_EXCLUDED_DECOMMISSIONED

timestampint64

host_idstring

executor_failuresint32

next_pageobject

Deprecated: use next_page_token or prev_page_token instead.

The parameters required to retrieve the next page of events. Omitted if there are no more events to read.

Show child attributesHide child attributes

cluster_idstring

The ID of the cluster to retrieve events about.

Example: 1234-567890-reef123

start_timeint64

The start time in epoch milliseconds. If empty, returns events starting from the beginning of time.

Example: 1617238800000

end_timeint64

The end time in epoch milliseconds. If empty, returns events up to the current time.

Example: 1619485200000

orderstring

The order to list events in; either "ASC" or "DESC". Defaults to "DESC".

Default: DESC

Values: DESC, ASC

event_typesarray of string

An optional set of event types to filter on. If empty, all event types are returned.

Values: CREATING, STARTING, RESTARTING, TERMINATING, EDITED, RUNNING, RESIZING, NODES_LOST, UPSIZE_COMPLETED, INIT_SCRIPTS_STARTED, INIT_SCRIPTS_FINISHED, DID_NOT_EXPAND_DISK, EXPANDED_DISK, FAILED_TO_EXPAND_DISK, DRIVER_HEALTHY, DRIVER_NOT_RESPONDING, DRIVER_UNAVAILABLE, SPARK_EXCEPTION, METASTORE_DOWN, DBFS_DOWN, AUTOSCALING_STATS_REPORT, NODE_BLACKLISTED, PINNED, UNPINNED, NODE_EXCLUDED_DECOMMISSIONED, ADD_NODES_FAILED, AUTOSCALING_BACKOFF, AUTOMATIC_CLUSTER_UPDATE, AUTOSCALING_FAILED, CLUSTER_MIGRATED, DECOMMISSION_STARTED, DECOMMISSION_ENDED, DEFERRED_POLICY_ENFORCEMENT_SCHEDULED, DEFERRED_POLICY_ENFORCEMENT_FAILED, UC_VOLUME_MISCONFIGURED

offsetint64

Deprecated: use page_token in combination with page_size instead.

The offset in the result set. Defaults to 0 (no offset). When an offset is specified and the results are requested in descending order, the end_time field is required.

Example: 5

limitint64

Deprecated: use page_token in combination with page_size instead.

The maximum number of events to include in a page of events. Defaults to 50, and maximum allowed value is 500.

Default: 50

Constraints: <= 500

page_tokenstring

Use next_page_token or prev_page_token returned from the previous request to list the next or previous page of events respectively. If page_token is empty, the first page is returned.

page_sizeint32

The maximum number of events to include in a page of events. The server may further constrain the maximum number of results returned in a single page. If the page_size is empty or 0, the server will decide the number of results to be returned. The field has to be in the range [0,500]. If the value is outside the range, the server enforces 0 or 500.

Default: 50

total_countint64

Deprecated: Returns 0 when request uses page_token. Will start returning zero when request uses offset/limit soon.

The total number of events filtered by the start_time, end_time, and event_types.

next_page_tokenstring

This field represents the pagination token to retrieve the next page of results. If the value is "", it means no further results for the request.

prev_page_tokenstring

This field represents the pagination token to retrieve the previous page of results. If the value is "", it means no further results for the request.

View as Markdown

Request

{
  "cluster_id": "1234-567890-reef123",
  "start_time": 1617238800000,
  "end_time": 1619485200000,
  "order": "string",
  "event_types": [
    "string"
  ],
  "offset": 5,
  "limit": 0,
  "page_token": "string",
  "page_size": 0
}

Response

{
  "events": [
    {
      "cluster_id": "1202-211320-brick1",
      "event_details": {
        "username": "admin"
      },
      "event_type": "RESTARTING",
      "timestamp": 1509572145487
    },
    {
      "cluster_id": "1202-211320-brick1",
      "event_details": {
        "termination_reason": {
          "code": "USER_REQUEST",
          "parameters": {
            "username": "admin"
          }
        }
      },
      "event_type": "TERMINATING",
      "timestamp": 1509505807923
    }
  ],
  "next_page": {
    "cluster_id": "1202-211320-brick1",
    "end_time": 1509572145487,
    "offset": 50,
    "order": "DESC"
  },
  "total_count": 303
}