Diagnose Lakeflow Jobs performance

When a job or task takes longer than expected, use the Jobs UI to understand where time is spent and what you can do to reduce it. This page covers the run breakdown, streaming task metrics, and query performance metrics for serverless jobs. To find and open job runs, see Monitor Lakeflow Jobs.

View the run breakdown by phase

The run breakdown shows where a job or task run spends time. When you view job run details, hover over the run's Duration to see the total duration broken down into the following phases:

• Queued
• Waiting for resources
• Library installation
• Running

The run breakdown is available for both job runs and task runs. For a job with multiple tasks, the job-level breakdown counts Waiting for resources only until the first task starts running. For the full breakdown of each task, view the run breakdown in the task run output.

Use the breakdown to identify which phase takes the most time, then use the following sections to understand the possible causes and what you can do to reduce that time.

note

Pipeline tasks have different phases than most tasks. They are: created, waiting for resources, initializing, setting up tables, and running. For guidance on reducing high initialization and table setup times, see Fix high initialization times in pipelines.

Queued phase

Your job or task is waiting to start because of concurrency limits.

Possible causes:

• The job reached its maximum concurrent runs limit because a previous run of the same job is still in progress.
• Other jobs running at the same time reached the workspace-level concurrent run limit.

What you can do:

• Increase the maximum concurrent runs setting for the job.
• Review your job schedules to reduce overlap. If jobs consistently compete for resources, consider staggering their start times. For more information, see the Databricks resource limits.

Waiting for resources phase

Your job or task is waiting for compute to become available before it can start running. What you can do to reduce the amount of wait time depends on your compute type.

Compute type	Possible causes	What you can do
Serverless	The serverless job uses standard performance mode, which has a startup latency of 4 to 6 minutes, depending on compute availability and whether you use Spark.	Switch to performance-optimized mode to reduce startup time.
Classic	Slow VM launch time. Quota exceeded or insufficient capacity delaying cluster launch.	To optimize startup times without managing cluster or compute configuration, consider migrating your job to serverless. If you experience slow VM launch times, configure instance pools with idle instance auto-termination to span the gap between your scheduled jobs. See also pool best practices. Group successive jobs as tasks within a single multi-task job. Tasks within a job share the same cluster for the duration of the run, which avoids repeated cluster startup. This can provide significant speedups compared to successive job runs or third-party orchestrators, which cannot reuse clusters between job tasks. To handle quota and capacity issues, enable flexible node types so that Databricks automatically falls back to compatible instance types when your preferred type is unavailable. If you consistently hit quota limits, you can also request a quota increase from your cloud provider.

Library installation phase

Your job is spending time installing libraries before it begins running.

Compute type	Possible causes	What you can do
Serverless	Databricks has not yet cached the environment and must create it. The job installs many packages. A custom PyPI repository might be slow to respond, especially when many tasks install packages in parallel.	Reduce the number of packages needed. Consider using a workspace base environment.
Classic	Library installation time is high because of the size and complexity of the environment defined on the cluster, especially when you use init scripts to install libraries at cluster startup.	Verify that the job uses only the packages needed for the run.

Running phase

A run can take longer than expected for many reasons. The following list is not exhaustive, but it covers some common causes and possible actions.

Possible causes:

• Data volume increased, so a longer runtime might be expected.
• Job or compute configuration changed.
• Code changes introduced slower queries or inefficient operations.

What you can do:

• Review recent changes using Databricks system tables, such as the jobs system tables and compute system tables.
• For serverless jobs, use query history to find which queries took the longest, and open their query profile to identify optimization opportunities. To find the relevant queries, use the link in the Compute section of each task, or open the job's Timeline view, which integrates with query profiles.
• For jobs running on classic compute, use the Spark UI to diagnose slow stages and tasks. Start with the event timeline to spot long-running jobs, idle gaps, or failures, then drill into individual stages and tasks for details.

For serverless jobs, you can get additional task metrics (Public Preview) and query performance metrics (beta).

View metrics for streaming tasks

Preview

Streaming observability for Lakeflow Jobs is in Public Preview.

The Lakeflow Jobs UI provides streaming observability metrics for streaming workloads on the Job run details page. These metrics include backlog seconds, backlog bytes, backlog records, and backlog files for sources supported by Spark Structured Streaming, including Apache Kafka, Amazon Kinesis, Auto Loader, Google Pub/Sub, and Delta tables. When you view the run details for a task, the metrics appear as charts in the right-hand pane. Each chart shows maximum values aggregated by minute, for up to the previous 48 hours.

Each streaming source supports only specific metrics. The UI displays only the metrics that a streaming source supports. The following table shows the metrics available for supported streaming sources:

source	backlog bytes	backlog records	backlog seconds	backlog files
Kafka	✓	✓
Kinesis	✓		✓
Delta	✓			✓
Auto Loader	✓			✓
Google Pub/Sub	✓	✓

You can also specify thresholds for each streaming metric and configure notifications if a stream exceeds a threshold during a task run. See Configure notifications for slow jobs.

To view streaming metrics for a task run that streams data from one of the supported Structured Streaming sources:

1. On the Job run details page, click the task for which you want to view metrics.
2. Click the Metrics tab in the Task run pane.
3. To open the graph for a metric, click next to the metric name.
4. To view the metrics for a specific stream, enter the stream ID in the Filter by stream_id text box. You can find the stream ID in the output for the job run.
5. To change the time period for the metric graphs, use the time drop-down menu.
6. To move through the streams if the run contains more than ten streams, click Next or Previous.

Streaming observability limitations

• Databricks updates metrics every minute unless a run has more than four streams. If a run has more than four streams, Databricks updates the metrics every five minutes.
• Databricks collects metrics only for the first fifty streams in each run.
• Databricks collects metrics at one-second intervals. The metrics might not be visible if your triggerInterval setting is less than one second.
• Most data sources collect streaming metrics by default. However, for others, you must enable this feature. If your data source isn't collecting streaming metrics, set the spark.sql.streaming.metricsEnabled flag to True.

View query performance metrics for serverless jobs

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

When you run a serverless job, Databricks surfaces selected Query Profile metrics and performance insights directly in the job run UI, so you can identify performance issues without opening a separate query profile for each query. Use these metrics to investigate why a run is slow, or to compare performance between two runs.

Before you can view these metrics:

• The Improved Lakeflow Performance Observability preview must be enabled for your workspace. Workspace admins can enable it from the Previews page.
• Your workspace must have access to Query performance insights. Without it, the lightbulb indicators do not appear, though the aggregated metrics (rows read, rows written, and total query count) still display.

Databricks displays the following metrics, aggregated from the queries in a serverless job run:

• Rows read and rows written per task run.
• Total number of queries per task run.
• A performance insights indicator (lightbulb) on a task when one or more queries in that task have performance insights.

These metrics appear in different places depending on how you're viewing the run:

Where it appears	What you see
Task run sidebar	Rows read and written, total query count, and an insights indicator for the task run.
DAG view	A lightbulb badge on a task node when any of the task's queries have performance insights.
Timeline view	A lightbulb next to the task name with a count of insights across the task's queries, plus a lightbulb on each query within the task that has insights.
List view	A lightbulb in the Insights column when any of the task's queries have performance insights. If you don't see this column, add it from the column picker.

note

The timeline view is now available for single-task jobs as part of this Beta. Previously, only multi-task jobs had a timeline view.

Click and hover behavior:

• In the Timeline view, hover over a task to see its aggregated metrics and any performance insights for the task's queries.
• In the DAG view or List view, click the lightbulb on a task to open the Timeline view with that task's queries expanded.
• In the Timeline view, click the query text of a query that has a lightbulb to open a pane with an overview of the performance insights for that query.

To investigate why a serverless job runs slower than expected:

1. Open the job run.
2. Switch to the Timeline view.
3. Identify tasks taking longer than expected based on the duration distribution.
4. Hover over a long-running task to see:
   • Rows read and rows written: check whether the task processed more data than usual.
   • Total number of queries: spot changes in workload shape.
   • The performance insights indicator: spot regressions or inefficient code changes.
5. If higher data volume explains the increased duration, the slowdown might be expected. Otherwise, expand the task in the timeline to see its individual queries. Queries with detected insights show a lightbulb next to them.
6. Click the query text of a query with a lightbulb to open a pane with the performance insights for that query.
7. Apply the recommended changes and re-run the job to confirm the issue is resolved.

For the full list of insights and what they mean, see Query performance insights. For deeper query execution details, see Query profile.

Comparing two runs

These same metrics make it simple to spot differences between a slow run and a previous fast run. Open both runs side by side and compare:

• Rows read and rows written to identify changes in data volume.
• Total number of queries to identify changes in workload shape.
• Performance insights to identify inefficiencies introduced since the previous run.

Query performance metrics limitations

• These metrics and insights apply to serverless Lakeflow Jobs only. Job runs on classic compute don't show this information.
• Databricks aggregates metrics across the first 100 queries in a job run. If a run has more queries, the totals reflect only the first 100.