Skip to main content
Last updated on

Losing spot instances

When the cloud provider reclaims spot instances, the executors on those instances terminate and all their active tasks fail. Spark must rerun those tasks, which slows the job and might cause stage or job failures.

Use this page when the compute's Event log shows that spot instances were lost or reclaimed. If the event log shows that the compute resized because of autoscaling, removed executors are expected and not an error. If the event log doesn't explain the lost executors, see Failing jobs or executors removed.

Reduce spot instance losses​

Frequent spot instance losses might mean that the instance type has a high reclaim rate. To reduce losses, try these approaches:

  • Choose instance types with lower interruption rates. To see how often each instance type is interrupted, use the Spot Instance Advisor.
  • Use a fleet instance type. A fleet instance type resolves to the available instance type of the same size with the best spot capacity and price.
  • Stop using spot instances. Clear the Use spot instance checkbox in the compute configuration. See Spot instances.

Use on-demand instances for the driver and latency-sensitive workloads​

If the cloud provider reclaims the driver, the job fails immediately. Databricks recommends using an on-demand instance for the driver node so that the cloud provider can't reclaim it. If you use a spot instance pool for workers, select an on-demand pool as the Driver type. See Pool best practices.

An on-demand driver doesn't protect the workers. When the cloud provider reclaims a worker, its executors terminate and their active tasks fail. The job continues, but Spark must rerun those tasks, which slows the job and might cause stage or job failures. Use spot instances only for workloads that can tolerate delays. See Classic compute configuration best practices.

On this page