Skip to main content
Last updated on

Failing jobs or executors removed

Failed jobs and removed executors both cost time. When an executor is removed, Spark reruns its unfinished tasks. When a Spark job fails, the query or task that runs the job also fails. Use this page to find out why jobs fail or executors are removed, and to decide whether the behavior is expected or a problem to fix.

Use this page when the jobs timeline shows failed jobs or removed executors, for example:

Failing Jobs

If the timeline doesn't show failed jobs or removed executors, review the jobs timeline for other patterns and use the other pages in this guide to diagnose the issue. See Diagnose cost and performance issues using the Spark UI.

The most common reasons for executors being removed are:

Failing jobs​

If you see any failing jobs, click them to open their pages. Then scroll down to see the failed stage and a failure reason:

Failure Reason

You may get a generic error. Click the link in the description to see more details:

Failure Description

If you scroll down on this page, you see why each task failed. In this case a memory issue appears to be the cause:

Failed Tasks

Failing executors​

To find out why your executors are failing, first check the compute's Event log to see whether there's an explanation for the executor failures. For example, if you're using spot instances, the cloud provider may be reclaiming them.

Event Log

Look for events explaining the loss of executors. For example, you may see messages indicating that the cluster is resizing or spot instances are being lost.

If you don't see any information in the event log, go back to the Spark UI and click the Executors tab:

Executors tab

Here you can get the logs from the failed executors:

Example of failed executors

Next step​

If you've gotten this far, the likeliest explanation is a memory issue. The next step is to dig into memory issues. See Spark memory issues.

On this page