Classic compute termination error codes

This article provides troubleshooting guidance for common cluster termination error codes. Use the error code from your cluster event log to find the relevant cause and recommended fix.

BOOTSTRAP_TIMEOUT_DUE_TO_MISCONFIG

The VM bootstrap process timed out due to network connectivity issues, slow artifact downloads, or issues with the cloud provider. The bootstrap timeout is 700 seconds.

Example error message

[id: InstanceId([REDACTED]), status: INSTANCE_INITIALIZING, ...] with threshold 700 seconds timed out after 703891 milliseconds. Instance bootstrap inferred timeout reason: UnknownReason

Troubleshooting steps

1. Check connectivity to Databricks artifact storage.
2. Verify connectivity to the Databricks control plane.
3. Check DNS resolution for Databricks endpoints.
4. Verify firewall and security group rules.
5. Test whether the issue is consistent or intermittent.

Recommended fix

Ensure network connectivity to Databricks storage and control plane. Configure service endpoints or VPC endpoints for better network performance. Review firewall, DNS, and routing configuration. Contact Databricks support if network configuration is verified, but timeouts persist.

CLOUD_PROVIDER_LAUNCH_FAILURE

The cloud provider failed to launch the requested VM instance. This is usually a cloud-provider-side issue.

Example error message

Reason: CLOUD_PROVIDER_LAUNCH_FAILURE (CLOUD_FAILURE). Parameters: databricks_error_message:The VM launch failed due to transient cloud provider error, please try again later. [details] VM_MIN_COUNT_NOT_REACHED|INTERNAL_ERROR: Requested minimum count of 1 VMs could not be created.|Internal error. Please try again or contact Google Support. (Code: 'REDACTED')(Spot)

Troubleshooting steps

1. Check the gcp_error_message in the error parameters for the specific cloud-provider failure.

1. Check the cloud provider status page for ongoing incidents in your region.
2. Review quota limits and subnet capacity if the error mentions these.

Recommended fix

Try again later, as most cloud provider launch failures are transient. If the issue still occurs, contact your cloud provider support with the specific detailed error from the details.

COMMUNICATION_LOST

The cluster was terminated because the control plane lost communication with the instance. This may be caused by unexpected instance state, instance termination, or network-level issues where the control plane cannot ping the instance for a prolonged period.

Example error message

Cluster '[REDACTED]' was terminated. Reason: COMMUNICATION_LOST (CLOUD_FAILURE). Parameters: instance_id:[REDACTED], databricks_error_message:Node health check failed.

Troubleshooting steps

1. Check the network configuration between the Databricks compute plane and the SCC relay endpoint. If there is a firewall or proxy between them, it might block the health check communication. Consult with your network administrator.
2. Check CPU and memory usage of the node on cluster metrics. If resources are exhausted, the instance may not respond to the health check. Consider using a bigger instance type.
3. Check with your cloud provider if the instance was terminated or impaired externally (for example, AWS instance retirement, Azure host maintenance).
4. Review Spark driver and executor logs for errors that may have caused the instance to become unresponsive (for example, OOM or long GC pauses).

Recommended fix

Review firewall and proxy settings with your network administrator. If the error was caused by the cloud provider terminating the instance, try again later. If it was caused by resource exhaustion, consider upgrading to a larger instance type. If the issue persists, contact Databricks support.

CONTROL_PLANE_REQUEST_FAILURE_DUE_TO_MISCONFIG

VMs cannot reach the Databricks control plane due to DNS resolution failures, firewall rules, or network misconfiguration.

Example error message

Network health check reported that instance is unable to reach Databricks Control Plane. Please check that instances have connectivity to the Databricks Control Plane. Instance bootstrap inferred timeout reason: NetworkHealthCheck_CP_Failed

Troubleshooting steps

1. Decode any Base64-encoded error messages in the cluster event log.
2. Check DNS settings in your network configuration.
3. Review firewall rules and network security settings.
4. Test control plane connectivity from a VM in the same network.
5. Verify custom DNS servers are functional and reachable.

Recommended fix

Verify DNS server configuration and reachability. Ensure firewall rules allow outbound traffic to the Databricks control plane.

Contact Databricks support if the network configuration appears correct, but the issue persists.

DOCKER_IMAGE_PULL_FAILURE

The cluster failed to download the Docker image from the container registry due to network, authentication, or configuration issues.

Example error message

Failed to pull docker image: authentication required

Troubleshooting steps

1. Verify the Docker image name and tag are correct in the cluster configuration.
2. Check network connectivity to the container registry from the workspace.
3. Test registry access from a VM in the same network.
4. Verify authentication credentials for private registries.
5. Review node daemon logs for detailed error messages.

Recommended fix

Correct the Docker image configuration and verify authentication credentials. Ensure network rules allow access to the container registry.

Contact Databricks support if the configuration appears correct, but the issue persists.

DOCKER_IMAGE_TOO_LARGE_FOR_INSTANCE_EXCEPTION

The Docker image size exceeds the available disk space on the selected instance type.

Example error message

Failed to launch container as the docker image is too large for the instance.

Troubleshooting steps

1. Check the Docker image size.
2. Review the instance type's disk capacity.
3. Identify unnecessary layers or files in the Docker image.
4. Check whether multiple large images are being used.

Recommended fix

Use an instance type with a larger disk capacity, optimize the Docker image by removing unnecessary files and layers, use multi-stage builds to reduce image size, or split functionality across multiple smaller images. Contact Databricks support for assistance with image optimization.

EOS_SPARK_IMAGE

The Databricks Runtime (DBR) version configured for the cluster has reached end of support (EOS).

Example error message

Spark image release__11.0.x-snapshot-cpu-ml-scala2.12__databricks-universe__head__[REDACTED]__format-2 does not exist with exit code 2

Troubleshooting steps

1. Check the DBR version in the cluster configuration.
2. Review the Databricks Runtime release notes for EOS dates.
3. Identify which DBR versions are currently supported.
4. Check whether notebooks or jobs have DBR version dependencies.

Recommended fix

Update the cluster configuration to use a supported Databricks Runtime version. Review compatibility requirements for libraries and code before deploying to production. Contact Databricks support if you need assistance with DBR migration.

GCP_INSUFFICIENT_CAPACITY

Google Cloud does not have sufficient capacity for the requested machine type in the selected zone.

Example error messages

The zone 'projects/[REDACTED]/zones/us-west1-b' does not have enough resources available to fulfill the request. (resource type: compute)

Requested minimum count of 1 VMs could not be created. | The zone 'projects/[REDACTED]/zones/us-west1-b' does not have enough resources available to fulfill the request. Try a different zone, or try again later.

Troubleshooting steps

1. Check the Google Cloud Service Health page for known capacity issues.
2. Review machine type availability in different zones.
3. Verify whether preemptible instances have different availability.
4. Check recent capacity trends for the machine type.

Recommended fix

Try launching in a different zone, use a different machine type with similar specifications, switch to standard instances if using preemptible, or schedule launches during off-peak hours. Contact Google Cloud support for information on capacity availability.

GCP_IP_SPACE_EXHAUSTED

The GCP subnet has run out of available IP addresses for VM allocation.

Example error message

IP space of 'projects/[REDACTED]/regions/us-west1/subnetworks/[REDACTED]' is exhausted. Insufficient free IP addresses in the IP range '[REDACTED]/23'. Consider expanding the current IP range or selecting an alternative IP range.

Troubleshooting steps

1. Check the subnet IP range and usage in the GCP Console.
2. Review the number of instances and other resources consuming IPs.
3. Check for IP address reservations.
4. Verify whether secondary IP ranges are configured.

Recommended fix

Expand the subnet IP range, create a new subnet with a larger IP range, and migrate the workspace, clean up unused resources, use fewer but larger instances, or configure secondary IP ranges. Contact Databricks support for assistance with workspace migration if needed.

GCP_NOT_FOUND

The requested GCP resource (network, subnet, service account, and so on) was not found.

Example error message

The resource 'projects/databricks-[REDACTED]' was not found

Troubleshooting steps

1. Verify the resource name or ID in the cluster configuration.
2. Check whether the resource exists in the GCP Console.
3. Verify the project ID is correct.
4. Check whether the resource was deleted.
5. Verify permissions to access the resource.

Recommended fix

Correct the resource identifier in the configuration, recreate the deleted resource, verify project and resource names, or check service account permissions. Contact Databricks support if the configuration appears correct.

GCP_RESOURCE_QUOTA_EXCEEDED

The cluster launch would exceed GCP project quota limits for CPUs, IP addresses, or disk resources.

Example error messages

Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded. Limit: 30000.0 in region us-central1.

Quota 'SSD_TOTAL_GB' exceeded. Limit: 400.0 in region us-east1.

Troubleshooting steps

1. Check quota usage in GCP Console > IAM & Admin > Quotas.
2. Identify which specific quota is exceeded (CPUs, IPs, disks).
3. Review resource usage across all regions.
4. Check for stuck or orphaned resources.

Recommended fix

Request a quota increase through the GCP Console, clean up unused resources to free quota, distribute workloads across multiple regions or projects, or use different machine types. Contact Google Cloud support for quota increase requests.

INSTANCE_POOL_MAX_CAPACITY_REACHED

The instance pool has reached its configured maximum capacity limit and cannot provide additional instances.

Example error message

Instance pool is full, please consider increasing the pool size

Troubleshooting steps

1. Check the instance pool configuration for the maximum capacity setting.
2. Review how many instances are currently in use from the pool.
3. Identify which clusters are using the pool.
4. Check whether there are idle instances that can be freed.

Recommended fix

Increase the instance pool maximum capacity, create additional instance pools to distribute load, terminate idle clusters using the pool, or configure clusters to use different pools. Review pool sizing based on concurrent workload requirements.

INSTANCE_UNREACHABLE_DUE_TO_MISCONFIG

Instances are unreachable due to network misconfiguration, firewall rules, or connectivity issues.

Example error message

Bootstrap completes in the VM but control plane failed to reach the node. Please review your network configuration or firewall settings to allow Databricks to reach the node.

Troubleshooting steps

1. Review firewall rules and network security settings for required inbound ports.
2. Test connectivity from the control plane to the instance network.
3. Check for asymmetric routing issues.
4. Review firewall logs for dropped connections.
5. Verify that instances have the correct security group assignments.

Recommended fix

Ensure security groups or NSGs allow required inbound traffic from the Databricks control plane. Verify that route tables enable bidirectional communication. Contact Databricks support for assistance with network connectivity troubleshooting.

INVALID_ARGUMENT

Invalid configuration parameters, missing secrets, incorrect permissions, or misconfigured cluster settings prevented the cluster from starting.

Example error message

com.databricks.backend.manager.secret.SecretPermissionDeniedException: User does not have permission with scope: [REDACTED] and key: [REDACTED]

Troubleshooting steps

1. Review the error message to identify the specific invalid parameter.
2. For secret errors, verify the secret scope and key exist using the Databricks Secrets API.
3. Check user or service principal permissions for accessing secrets.
4. Review the cluster configuration for syntax errors.
5. Check init scripts for configuration errors.

Recommended fix

Correct the invalid parameter based on the error message. For secrets, verify scope and key existence, check permissions, and ensure network connectivity to secret providers. Validate all cluster configuration against the documentation. Contact Databricks support if the configuration appears correct.

NETWORK_CHECK_CONTROL_PLANE_FAILURE

A pre-bootstrap network health check failed when attempting to reach the Databricks control plane.

Example error message

Instance failed network health check before bootstrapping with fatal error: X_NHC_CONTROL_PLANE_UNREACHABLE
1 failed component(s): control_plane
Retryable: true

Troubleshooting steps

1. Review cluster event logs for specific connection failure details.
2. Test control plane connectivity from a VM in the same network.
3. Check whether a firewall is intercepting or blocking traffic.

Recommended fix

Verify that security group or NSG rules allow outbound traffic to the Databricks control plane. If using UDR with a firewall, ensure Databricks service tags route to the internet. Contact Databricks support if network configuration is verified correct.

NETWORK_CONFIGURATION_FAILURE

A network configuration error is preventing proper VM or cluster network setup.

Troubleshooting steps

1. Review firewall and security group or NSG rules.
2. Check route tables and routing configuration.
3. Verify subnet configuration.
4. Check for IP address conflicts.
5. Verify DNS settings.

Recommended fix

Correct the network configuration based on the specific error. Ensure security group or NSG rules allow required traffic, verify that subnet CIDR ranges don't overlap, check that route tables are properly configured, and ensure DNS is functional. Contact Databricks support for network configuration review.

NPIP_TUNNEL_SETUP_FAILURE

The bootstrap script failed to set up the NPIP tunnel connection within the timeout. This occurs after the cloud provider launches the instance and the bootstrap script attempts to establish the SCC relay tunnel.

Example error message

Cluster '[REDACTED]' was terminated. Reason: NPIP_TUNNEL_SETUP_FAILURE (SERVICE_FAULT). Parameters: databricks_error_message:VM setup failed due to Ngrok setup timeout. [details] NPIP_TUNNEL_SETUP_FAILURE: Instance bootstrap failed command: WaitForNgrokTunnel Failure message: Timed out waiting for ngrok tunnel to be up(OnDemand), instance_id:[REDACTED]

Troubleshooting steps

1. Check the network configuration between the SCC relay and the Databricks compute plane subnets.
2. Review firewall and proxy settings that might block tunnel setup on port 443 or 6666.

Recommended fix

Ensure network connectivity from the compute plane to the SCC relay endpoint. Launch an instance in the Databricks compute plane VPC/VNet and check connectivity to the SCC relay:

nslookup <SCC relay fqdn>
nc -vz <SCC relay fqdn> 443

If there is a firewall or proxy, verify it allows traffic to the relay on the required ports. Consult the public network configuration docs and ensure you have the right egress rules set up and can connect to SCC endpoint from your VPC/VNet. If the issue occurs even though there is no problem in your network configuration, contact Databricks support.

REQUEST_THROTTLED

API requests to the cloud provider are being throttled due to rate limiting.

Example error message

TEMPORARILY_UNAVAILABLE: Too many requests from workspace [REDACTED]

Troubleshooting steps

1. Check whether multiple clusters are launching simultaneously.
2. Review API request rate limits for your account.
3. Identify whether other services are making concurrent API calls.
4. Check for automated systems making frequent requests.

Recommended fix

Reduce concurrent cluster launches, request an API rate limit increase from your cloud provider, implement exponential backoff in automation scripts, or stagger cluster launch times.

SPOT_INSTANCE_TERMINATION

Spot or preemptible instances were terminated by the cloud provider due to capacity needs or pricing changes.

Example error message

Server.SpotInstanceTermination: Spot instance termination

Troubleshooting steps

1. Check the cluster event logs for the termination timestamp.
2. Review spot pricing history in your region.
3. Identify whether terminations occur at specific times.
4. Check whether multiple instances were terminated simultaneously.

Recommended fix

Switch to on-demand instances for production workloads, implement job retry logic to handle interruptions, or use a mix of on-demand and spot instances. Spot instances are best for fault-tolerant workloads.

SPARK_STARTUP_FAILURE

The Spark driver failed to start within the configured timeout. This may occur when the driver daemon startup was not completed within the timeout (typically 200 seconds) on the cluster driver instance.

Example error messages

Cluster '[REDACTED]' was terminated. Reason: SPARK_STARTUP_FAILURE (SERVICE_FAULT). Parameters: databricks_error_message:Spark failed to start: DEADLINE_EXCEEDED.

Cluster '[REDACTED]' was terminated. Reason: SPARK_STARTUP_FAILURE (SERVICE_FAULT). Parameters: databricks_error_message:Spark failed to start: Timed out after 200 seconds.

Troubleshooting steps

1. Review Spark configuration for misconfigurations (for example, invalid metastore URI or conflicting settings).
2. Check your init scripts for potential errors that could delay or prevent driver startup.

Recommended fix

Remove custom Spark configs and init scripts to isolate the issue. Try a different instance type, as hardware slowness on smaller instances can cause driver startup timeouts. If the issue persists, contact Databricks support with the cluster ID and error details.

STORAGE_DOWNLOAD_FAILURE_SLOW

Downloading artifacts from Databricks storage is failing or too slow due to network connectivity, firewall, or DNS issues.

Example error message

Instance bootstrap failed command: Command_UpdateWorker
Failure message: Trying DNS probe for: https://[REDACTED].blob.core.windows.net/update/worker-artifacts/...

Troubleshooting steps

1. Check firewall rules for Databricks storage endpoints.
2. Verify DNS resolution for storage URLs.
3. Test download speed from a VM in the same network.
4. Review network bandwidth utilization.
5. Check for proxy or network inspection devices.
6. Verify routes to storage endpoints.

Recommended fix

Ensure firewall rules allow access to Databricks storage endpoints.

Review and optimize network inspection devices if present. Contact Databricks support if connectivity to storage endpoints is verified but downloads still fail.

WORKSPACE_CONFIGURATION_ERROR

Workspace-level misconfiguration is preventing cluster launch, including issues with IAM roles or service principal permissions.

Troubleshooting steps

1. Review recent changes to workspace configuration.
2. Check the cloud provider console for policy or permission changes.

Recommended fix

Review workspace service account permissions and project configuration.

Contact Databricks support if the workspace configuration appears correct or if the cross-account role setup needs verification.