Skip to main content
Last updated on

Spark driver overloaded

The driver coordinates all work on a cluster. It interprets every command, plans each query, and schedules tasks on the executors. When the driver is overloaded, it can't hand out work fast enough, so the executors sit idle while you still pay for them. This usually shows up as long gaps in the jobs timeline and a slow or unresponsive cluster, even though the workers aren't busy.

Use this page after you verify that the driver is heavily loaded in the cluster metrics while the workers aren't. To check this, see Driver is overloaded. If the driver isn't heavily loaded, the gaps have a different cause. Review the other causes in Gaps between Spark jobs and use the other pages in this guide to diagnose the issue. See Diagnose cost and performance issues using the Spark UI.

Common causes​

The driver is usually overloaded for one of these reasons:

  • Too many concurrent operations. Too many streams, queries, or Spark jobs run on the cluster at the same time. For example, some workloads use threads to run many Spark jobs concurrently. This is the most common cause.
  • Non-Spark code. Code that doesn't use Spark, such as a Python loop that calls native Python functions, runs on the driver and keeps it busy while the workers sit idle. To check whether your code runs in Spark, see Execution of non-Spark code.
  • Many attached notebooks. The driver maintains state information for every notebook attached to the cluster. Many attached notebooks increase the load on the driver even when they aren't running commands.

Fix an overloaded driver​

To reduce the load on the driver, try these approaches:

  • Increase the driver size. The driver node type defaults to the same type as the worker nodes. Select a larger driver type in the Advanced performance section of the compute configuration. Databricks recommends starting by doubling the driver size and checking how that affects the job. See Driver type.
  • Reduce the concurrency. If too many operations run at the same time, run fewer streams, queries, or threaded Spark jobs on the cluster.
  • Spread the load over multiple clusters. Move some of the concurrent workloads to separate compute so that each driver coordinates less work.
  • Move non-Spark code. If the driver is busy running non-Spark code, a larger driver doesn't help the idle workers. Rewrite the code to use Spark, or run it on serverless compute, where you don't pay for idle time. To rewrite the code:
  • Detach unused notebooks. Detach notebooks you aren't using from the cluster to reduce the state the driver maintains.
On this page