Many small Spark jobs

If the jobs timeline shows many short jobs that run one after another, the workload might be running many operations on small data, less than 10 GB. Each operation takes only a few seconds, but the time and the fixed overhead per operation add up. When small jobs run one at a time, much of the cluster might sit idle, so you pay for compute you don't use.

Use this page when the jobs timeline shows mostly small jobs. If the timeline shows a different pattern, see the page for that pattern:

• Diagnosing a long job in Spark
• Gaps between Spark jobs

Run operations in parallel

The best way to speed up small jobs is to run multiple operations in parallel. Use one of these approaches:

• Use Lakeflow pipelines. Databricks recommends using Lakeflow pipelines, which run operations in parallel automatically.
• Use multi-task jobs. Separate the operations into multiple notebooks, and run them in parallel on the same cluster with multi-task jobs.
• Use a SQL warehouse. If all the queries are written in SQL, run them on a SQL warehouse. SQL warehouses are designed to run many queries in parallel.
• Use a For each task. Parameterize the notebook, and use a For each task to run it multiple times in parallel. Set Concurrency to control the level of parallelism. This approach works well with serverless compute.

Right-size the compute

If you can't parallelize the operations, small operations on small data don't benefit from many workers. Use a smaller cluster or single-node compute, or run the workload on serverless compute so you don't manage cluster size yourself.