Data quality monitoring
This feature is in Beta.
This page describes what data quality monitoring is, what it monitors, and how to use it. Data quality monitoring was formerly called anomaly detection.
To provide feedback on data quality monitoring, email lakehouse-monitoring-feedback@databricks.com
.
What is data quality monitoring?
Using data quality monitoring, you can easily monitor the data quality of all of the tables in a schema. Databricks leverages data intelligence to automatically assess data quality, specifically evaluating the freshness and completeness of each table. Data owners have access to the logging table, which allows them can quickly identify and resolve anomalies across the entire metastore. Results at the catalog, schema, and table level are available in the Catalog Explorer or Governance Hub (preview).
Data quality monitoring does not modify any tables it monitors, nor does it add overhead to any jobs that populate these tables.
Requirements
- Unity Catalog enabled workspace.
- Existing users need serverless compute enabled. For instructions, see Connect to serverless compute.
- To enable data quality monitoring on a schema, you must have MANAGE SCHEMA or MANAGE CATALOG privileges on the catalog schema.
How does data quality monitoring work?
Databricks creates a background job that monitors tables for freshness and completeness. Databricks uses smart scanning to determine when to scan tables.
Freshness refers to how recently a table has been updated. Data quality monitoring analyzes the history of commits to a table and builds a per-table model to predict the time of the next commit. If a commit is unusually late, the table is marked as stale.
Event freshness, which is based on event time columns and ingestion latency, was available only to legacy data quality monitoring users. In the current version, event freshness is not supported.
Completeness refers to the number of rows expected to be written to the table in the last 24 hours. Data quality monitoring analyzes the historical row count, and based on this data, predicts a range of expected number of rows. If the number of rows committed over the last 24 hours is less than the lower bound of this range, a table is marked as incomplete.
Enable data quality monitoring on a schema
To enable data quality monitoring on a schema, navigate to the schema in Unity Catalog.
-
On the schema page, click the Details tab.
-
Click the Data Quality Monitoring toggle to enable it.
-
A scan is initiated. Databricks automatically scans each table at the same frequency it’s updated, providing up-to-date insights without requiring manual configuration for each table. On the initial scan for schemas enabled prior to September 24, 2025, Databricks ran the monitor on historical data ("backtesting") to check the quality of your tables as if data quality monitoring had been enabled on your schema two weeks ago.
-
When the scan is complete, detected quality issues are logged in the output logging table with insights populated in the UI. You can access the UI at any time by clicking See results next to the Data Quality Monitoring toggle.
Data quality monitoring UI
On October 7, 2025, Databricks released a new version of data quality monitoring. Schemas enabled for data quality monitoring on or after that date have a new results UI.
For information about the new UI, see Data quality monitoring incidents UI.
For information about the legacy UI, see Data quality monitoring dashboard (legacy).
Databricks recommends that you enable the new version for all of your existing schemas.
To enable the new version, use the Data Quality Monitoring toggle to disable the feature, and then toggle again to re-enable it.
Data quality monitoring incidents UI
After you enable data quality monitoring on a schema, you can click "See results" to open the results page. You can also access the incidents view from Catalog Explorer or the Governance Hub (preview).
The incidents UI contains catalog and schema dropdowns. When you select a catalog, the schema dropdown is populated with schemas in that catalog that have data quality monitoring enabled.
- If you have MANAGE or SELECT privileges on the catalog, you can view incidents at the catalog level.
- To view incidents for a specific schema, you must also have MANAGE or SELECT privileges on that schema. Selecting a schema then shows incidents for just that schema.
The UI has two main sections:
- Summary section: Shows overall data quality for the selected scope, including the percentage of healthy tables and the percentage of schemas/tables currently monitored.
- Incident section: Lists incidents across all monitored tables in the selected scope. You can filter for incidents on high-impact tables determined using downstream impact analysis.
Each incident record includes the following details:
- Reason: Whether the table is unhealthy due to freshness or completeness.
- Since: The timestamp when the first incident was detected.
- Impact: A qualitative measure of downstream impact (High, Medium, or Low), based on the number of downstream tables and queries affected.
- Root cause: Information about upstream jobs contributing to the issue (see Review logged results for details).
- Review: A link to the table quality page for more in-depth status and trend information.
Data quality monitoring dashboard (legacy)
The data quality monitoring dashboard was available only to legacy users. In the current version, use Data quality monitoring incidents UI.
The first data quality monitor run creates a dashboard to summarize results and trends derived from the logging table. The dashboard is automatically populated with insights for the scanned schema. A single dashboard is created per workspace at this path: /Shared/Databricks Quality Monitoring/Data Quality Monitoring
.
Quality overview
The Quality Overview tab shows a summary of the latest quality status of tables in your schema based on the most recent evaluation.
To get started, you must enter the logging table for the schema you want to analyze to populate the dashboard.
The top section of the dashboard shows an overview of the results of the scan.
Below the summary is a table listing quality incidents by impact. Any identified root causes are displayed in the root_cause_analysis
column.
Below the quality incident table is a table of identified static tables that have not been updated in a long time.
Table quality details
The Table Quality Details UI allows you to dive deeper into trends and analyze specific tables in your schema. You can access this view in several ways:
- From the Incidents UI (new experience), by clicking on the review link in the incidents list.
- From the Monitoring Dashboard (legacy Lakeview dashboard), by clicking on the table name in the Quality Overview tab.
- From the UC Table viewer, by visiting the Quality tab on the table page.
All options take you to the same Table Quality Details view for the selected table.
Given a table, the UI shows summaries from each quality check for the table, with graphs of predicted and observed values at each evaluation timestamp. The graphs plot results from the last 1 week of data.
If the table failed the quality checks, the UI also displays any upstream jobs that were identified as the root cause.
Set up alerts
To configure a Databricks SQL alert on the output results table, follow these steps in the Databricks alerts UI.
By default, only account admins can access the system table system.data_quality_monitoring.table_results
. If other users need to configure alerts, make sure to grant appropriate access.
-
Configure the query for the alert:
SQLWITH rounded_data AS (
SELECT
DATE_TRUNC('HOUR', event_time) AS evaluated_at,
CONCAT(catalog_name, '.', schema_name, '.', table_name) AS full_table_name,
status,
MAX(downstream_impact.num_queries_on_affected_tables) AS impacted_queries,
MAX(freshness.commit_freshness.predicted_value) AS commit_expected,
MAX(freshness.commit_freshness.last_value) AS commit_actual,
MAX(completeness.daily_row_count.min_predicted_value) AS completeness_expected,
MAX(completeness.daily_row_count.last_value) AS completeness_actual
FROM system.data_quality_monitoring.table_results
GROUP BY ALL
)
SELECT
evaluated_at,
full_table_name,
status,
commit_expected,
commit_actual,
completeness_expected,
completeness_actual,
impacted_queries
FROM rounded_data
WHERE
evaluated_at >= current_timestamp() - INTERVAL 6 HOURS
-- enter the minimum number of table violations before the alert is triggered
AND impacted_queries > :min_tables_affected
AND status = 'Unhealthy';noteFor legacy beta jobs, existing alert configuration should replace
system.data_quality_monitoring.table_results
with<catalog>.<schema>._quality_monitoring_summary
. -
Configure the alert condition:
-
Customize the email template:
Html<h4>The following tables are failing quality checks in the last hour</h4>
<table>
<tr>
<td>
<table>
<tr>
<th>Table</th>
<th>Expected Staleness</th>
<th>Actual Staleness</th>
<th>Expected Row Volume</th>
<th>Actual Row Volume</th>
<th>Impact (queries)</th>
</tr>
{{#QUERY_RESULT_ROWS}}
<tr>
<td>{{full_table_name}}</td>
<td>{{commit_expected}}</td>
<td>{{commit_actual}}</td>
<td>{{completeness_expected}}</td>
<td>{{completeness_actual}}</td>
<td>{{impacted_queries}}</td>
</tr>
{{/QUERY_RESULT_ROWS}}
</table>
</td>
</tr>
</table>
Now, you have an alert that triggers based on the downstream impact of the quality issue, helping you debug the table that triggered the alert.
Disable data quality monitoring
To disable data quality monitoring, click the Data Quality Monitoring toggle to disable it. Disabling data quality monitoring stops scanning tables in the schema for quality issues.
Limitation
Data quality monitoring does not support the following:
- Views or materialized views.
- The determination of completeness does not take into account metrics such as the fraction of nulls, zero values, or NaN.
Advanced
Review logged results
By default, data quality monitoring scan results are stored in the system.data_quality_monitoring.table_results
table. Only account admins can access this table, and they must grant access to others as needed. Data quality monitoring uses default storage to store the anomaly detection results. You are not billed for the storage.
The results table system.data_quality_monitoring.table_results
contains all results across the entire metastore and includes sample values from tables in each catalog. Use caution when granting access to this table.
Each row in the results table corresponds to a single table in the schema that was scanned.
The table has the following schema:
Column name | Contents (for | Data type | Description | Example data |
---|---|---|---|---|
| timestamp | Time when the row was generated. |
| |
| string | Name of the catalog. Used to identify the table. |
| |
| string | Name of the schema. Used to identify the table. |
| |
| string | Name of the table. Used to identify the table. |
| |
| string | Stable ID for the catalog. |
| |
| string | Stable ID for the schema. |
| |
| string | Stable ID for the table. |
| |
| string | Consolidated health status at the table level. "Unhealthy" if any check or group is unhealthy. |
| |
| struct | Freshness checks. | ||
| string | Overall freshness status. |
| |
| Commit freshness check results. | |||
| struct | Completeness check results. | ||
| string | Status of completeness check. |
| |
| Total number of rows in the table over time. | |||
| Number of rows added each day. | |||
| struct | Summary of downstream impact based on dependency graph. | ||
| int | Severity indicator (0 = none, 1 = low, 2 = medium, 3 = high, 4 = very high). | 2 | |
| int | Number of downstream tables affected. | 5 | |
| int | Number of queries run on affected downstream tables over the last 30 days. | 120 | |
| struct | Information about upstream jobs contributing to the issue. | ||
| Metadata for each upstream job. |
commit_freshness
array structure
The commit_freshness
struct contains the following:
Item name | Data type | Description | Example data |
---|---|---|---|
| string | Status of commit freshness check. |
|
| string | Error message encountered during check. |
|
| timestamp | Last commit timestamp. |
|
| timestamp | Predicted time by which the table should have been updated. |
|
total_row_count
and daily_row_count
array structure
The total_row_count
and daily_row_count
structs contain the following:
Item name | Data type | Description | Example data |
---|---|---|---|
| string | Status of the check. |
|
| string | Error message encountered during check. |
|
| int | Number of rows observed in the last 24 hours. |
|
| int | Minimum expected number of rows in the last 24 hours. |
|
| int | Maximum expected number of rows in the last 24 hours. |
|
upstream_jobs
array structure
The structure of the array shown in the upstream_jobs
column is shown in the following table:
Item name | Data type | Description | Example data |
---|---|---|---|
| string | Job ID. |
|
| string | Workspace ID. |
|
| string | Job display name. |
|
| string | Status of the most recent run. |
|
| string | URL of Databricks job run page. |
|
Downstream impact information
In the logged results table, the column downstream_impact
is a struct
with the following fields:
Field | Type | Description |
---|---|---|
| int | Integer value between 1 and 4 indicating the severity of the data quality issue. Higher values indicate greater disruption. |
| int | Number of downstream tables that might be affected by the identified issue. |
| int | Total number of queries that have referenced the affected and downstream tables in the past 30 days. |
Set parameters for freshness and completeness evaluation (legacy)
Starting from July 21, 2025, configuration of the job parameters will not be supported for new customers. If you need to configure the job settings, please contact Databricks.
To edit the parameters that control the job, such as how often the job runs or the name of the logged results table, you must edit the job parameters on the Tasks tab of the job page.
The following sections describe specific settings. For information about how to set task parameters, see Configure task parameters.
Schedule and notifications
To customize the schedule for the job, or to set up notifications, use the Schedules & Triggers settings on the jobs page. See Automating jobs with schedules and triggers.
Name of logging table
To change the name of the logging table, or save the table in a different schema, edit the job task parameter logging_table_name
and specify the desired name. To save the logging table in a different schema, specify the full 3-level name.
Customize freshness
and completeness
evaluations
All of the parameters in this section are optional. By default, data quality monitoring determines thresholds based on an analysis of the table's history.
These parameters are fields inside the task parameter metric_configs
. The format of metric_configs
is a JSON string with the following default values:
[
{
"disable_check": false,
"tables_to_skip": null,
"tables_to_scan": null,
"table_threshold_overrides": null,
"table_latency_threshold_overrides": null,
"static_table_threshold_override": null,
"event_timestamp_col_names": null,
"metric_type": "FreshnessConfig"
},
{
"disable_check": true,
"tables_to_skip": null,
"tables_to_scan": null,
"table_threshold_overrides": null,
"metric_type": "CompletenessConfig"
}
]
The following parameters can be used for both freshness
and completeness
evaluations.
Field name | Description | Example |
---|---|---|
| Only the specified tables are scanned. |
|
| The specified tables are skipped during the scan. |
|
| Scan is not run. Use this parameter if you want to disable only the |
|
The following parameters apply only to the freshness
evaluation:
Field name | Description | Example |
---|---|---|
| List of timestamp columns tables in your schema might have. If a table has one of these columns, it is marked |
|
| A dictionary consisting of table names and thresholds (in seconds) that specify the maximum interval since the last table update before marking a table as |
|
| A dictionary consisting of table names and latency thresholds (in seconds) that specify the maximum interval since the last timestamp in the table before marking a table as |
|
| Amount of time (in seconds) before a table is considered as a static table (that is, one that is no longer updated). |
|
The following parameter applies only to the completeness
evaluation:
Field name | Description | Example |
---|---|---|
| A dictionary consisting of table names and row volume thresholds (specified as integers). If the number of rows added to a table over the previous 24 hours is less than the specified threshold, the table is marked |
|