Monitor job health (Private Preview)
This feature is in Private Preview. To request access, contact your Databricks account team.
Lakeflow Jobs health is a single, roll-up score that helps you identify jobs that need attention without checking multiple signals in different places. The health score combines run status, failure rate, user-defined warnings, automated warnings, and (optionally) data quality signals into one indicator on the Jobs & Pipelines list and the job page.
This article describes how to use job health to:
- Filter the Jobs & Pipelines list to surface degraded jobs.
- Read a job's health summary to understand why it is degraded.
- Prioritize jobs by their impact score.
- Trigger Genie Code to help debug the issue.
- Enable data quality monitoring so the health score reflects table-level completeness and freshness.
How job health works
The job health score reflects the most severe status across all health criteria. The statuses, from most to least severe, are:
- Unhealthy: At least one criterion is unhealthy.
- Warning: At least one criterion is in a warning state, and no criterion is unhealthy.
- Healthy: No criteria are flagged.
Job health is only shown for jobs with recent runs. If a job has not run in approximately the last two months, the health column is blank. Job health does not apply to pipelines, so the column is also blank for pipelines during this preview.
Health criteria
The following criteria are evaluated at the end of each run or, for data quality signals, asynchronously after the run completes:
Category | Criterion | Status when triggered | Description |
|---|---|---|---|
Job failures | Last run status | Unhealthy | The last run failed after all retries. |
Job failures | Failure rate | Warning | More than 50% of the last 25 runs failed. |
User-defined warning | Metric threshold | Warning | A user-configured threshold for run duration or streaming backlog is exceeded. |
Automated warning | Outdated DBR | Warning | The job runs on a Databricks Runtime version that is approaching end-of-life. |
Automated warning | Duration anomaly | Warning | Run duration is more than two standard deviations from the mean of the last 25 successful runs. Requires at least three successful runs. |
Data quality | Data quality | Unhealthy | Aggregates table-level completeness and freshness from anomaly detection in data quality monitoring. The worst available table-level indicator is shown. Data profiling signals are not included. |
Data quality is only evaluated when monitoring is enabled at the schema level. See Enable data quality monitoring for a job.
Identify problematic jobs
To find jobs that need attention, use the Health filter on the Jobs & Pipelines list.
- In the sidebar, click
Jobs & Pipelines. - Use the Health filter to select Unhealthy, Warning, or both.
The Health column appears to the left of the recent runs column. Jobs that are unhealthy or in a warning state are surfaced at the top of the filtered list, so you can prioritize what to investigate first.
Jobs that have not run recently and pipelines have a blank Health column and are excluded from the Health filter results.
Understand a job's health at a glance
Open a job from the Jobs & Pipelines list to see its health summary. The summary shows which criteria are flagged, how long the job has been in its current state, and a short explanation for each degraded criterion.
- If the job is Unhealthy or in Warning, the health section is expanded by default.
- If the job is Healthy, the section is collapsed. Expand it to see the criteria that were evaluated.
Each degraded criterion can be expanded to view more details and, where relevant, provides cross-links to other parts of the product (for example, the runs list for failure rate, or the data quality monitoring page for a flagged table) so you can drill into the underlying signal.
Triage with the impact score
Impact indicates the importance of a job, so you can prioritize among unhealthy jobs. It is based on the downstream query volume of the tables the job produces: the more those tables are queried, the higher the impact. Impact is aggregated from the table level, so a single high-impact table makes the whole job high impact.
Impact | When it applies |
|---|---|
High | At least one table in the job's lineage is high impact. |
Medium | No table is high impact, and at least one table is medium impact. |
Low | No table is high or medium impact, or no lineage data is available for the job. |
Unknown | The job has not run recently, so impact has not been determined yet. |
Impact score is available in the following regions: us-east-1, us-east-2, us-west-2, eu-west-1, eu-central-1, and ap-southeast-2.
Debug a problematic job with Genie Code
From the health summary, you can hand off to Genie Code to investigate the degraded job.
Click Diagnose to open a chat where you can ask Genie Code to investigate and apply fixes. It reads the health summary, produces an explanation of why the job is degraded, and lets you continue the conversation to dig deeper or apply targeted changes after confirmation.
Genie Code uses the health score as its entry point and then queries job-related and data quality-related information to root cause the issue. For more on the underlying capabilities, see Genie Code. This page only describes how to launch Genie Code from the health summary.
Enable data quality monitoring for a job
Data quality is only included in the health score if monitoring is enabled for the schemas the job writes to. Enable monitoring from the job page:
- Open the job from the Jobs & Pipelines list.
- In the sidebar, select Job Health Configuration.
- In the Data quality monitoring section, click Enable monitoring.
- Review the list of schemas the job writes to, ranked by the number of tables in each schema.
- Confirm the selection and click Enable monitoring.
After monitoring is enabled, data quality signals appear in the job's health summary once the first quality evaluation completes. The health status updates whenever new data quality results become available.
Data quality signals are not evaluated in real time. There can be a delay between when a job run completes and when its data quality results appear in the health summary. The scan frequency is determined by anomaly detection. For details, see Anomaly detection.
Data quality is configured at the schema level.
Limitations
- Pipelines are not supported and have a blank health status.
- Health configuration options (for example, excluding specific criteria) are not available. To submit feedback on noisy criteria, contact your Databricks account team.
- You cannot manually override a job's health status.
- The health score is not persisted in system tables.
- Health is evaluated at the end of each job run. If a scheduled run is missed (the job does not run), the health status does not update until the job runs again, even if data quality monitoring detects an anomaly in the meantime.