Skip to main content
Last updated on

Losing spot instances

When the cloud provider reclaims preemptible VM instances, the executors on those instances terminate and all their active tasks fail. Spark must rerun those tasks, which slows the job and might cause stage or job failures.

Use this page when the compute's Event log shows that preemptible VM instances were lost or reclaimed. If the event log shows that the compute resized because of autoscaling, removed executors are expected and not an error. If the event log doesn't explain the lost executors, see Failing jobs or executors removed.

Reduce spot instance losses​

Frequent preemptible VM instance losses might mean that the instance type has a high reclaim rate. To reduce losses, try these approaches:

  • Choose instance types with lower preemption rates. Change to an instance type that is preempted less often.
  • Stop using preemptible VMs. Clear the Use preemptible instances checkbox under Advanced performance in the compute configuration. To control whether compute falls back to on-demand instances when preemptible VMs aren't available, set gcp_attributes.gcp_availability. See Preemptible instances.

Use on-demand instances for the driver and latency-sensitive workloads​

If the cloud provider reclaims the driver, the job fails immediately. Databricks recommends using an on-demand instance for the driver node so that the cloud provider can't reclaim it. If you use a preemptible VM instance pool for workers, select an on-demand pool as the Driver type. See Pool best practices.

An on-demand driver doesn't protect the workers. When the cloud provider reclaims a worker, its executors terminate and their active tasks fail. The job continues, but Spark must rerun those tasks, which slows the job and might cause stage or job failures. Use preemptible VM instances only for workloads that can tolerate delays. See Classic compute configuration best practices.

On this page