Skip to main content

Data quality monitoring

Beta

This feature is in Beta.

This page describes what data quality monitoring is, what it monitors, and how to use it. Data quality monitoring was formerly called anomaly detection.

To provide feedback on data quality monitoring, email lakehouse-monitoring-feedback@databricks.com.

What is data quality monitoring?

Using data quality monitoring, you can easily monitor the data quality of all of the tables in a schema. Databricks leverages data intelligence to automatically assess data quality, specifically evaluating the freshness and completeness of each table. Quality insights are populated in health indicators so consumers can understand health at a glance. Data owners have access to logging tables and dashboards so they can quickly identify, set alerts, and resolve anomalies across an entire schema.

Data quality monitoring does not modify any tables it monitors, nor does it add overhead to any jobs that populate these tables.

Requirements

  • Unity Catalog enabled workspace.
  • Serverless compute enabled. For instructions, see Connect to serverless compute.
  • To enable data quality monitoring on a schema, you must have MANAGE SCHEMA or MANAGE CATALOG privileges on the catalog schema.

How does data quality monitoring work?

Databricks monitors enabled tables for freshness and completeness.

Freshness refers to how recently a table has been updated. Data quality monitoring analyzes the history of commits to a table and builds a per-table model to predict the time of the next commit. If a commit is unusually late, the table is marked as stale. For time series tables, you can specify event time columns. Data quality monitoring then detects if the data's ingestion latency, defined as the difference between the commit time and the event time, is unusually high.

Completeness refers to the number of rows expected to be written to the table in the last 24 hours. Data quality monitoring analyzes the historical row count, and based on this data, predicts a range of expected number of rows. If the number of rows committed over the last 24 hours is less than the lower bound of this range, a table is marked as incomplete.

How often does a data quality monitor run?

By default, a data quality monitor runs every hour. Before scanning each table, the system checks if it's likely that the table has been updated since the last run. If a table isn't expected to have been updated yet, the scan will be skipped.

Enable data quality monitoring on a schema

To enable data quality monitoring on a schema, navigate to the schema in Unity Catalog.

  1. On the schema page, click the Details tab.

    Details tab for the schema page in Catalog Explorer.

  2. Click the Data Quality Monitoring toggle to enable it.

    Data quality monitor selector enabled.

  3. A Databricks job is initiated to scan the schema. Databricks automatically scans each table at the same frequency it’s updated, providing up-to-date insights without requiring manual configuration for each table. For the first job run, Databricks also runs the monitor on historical data ("backtesting") to check the quality of your tables as if data quality monitoring had been enabled on your schema two weeks ago. These results are logged to the logging table.

  4. When the job is complete, you will see the detected quality issues logged in the output logging table with insights populated in the Data Quality Dashboard. You can access the dashboard at any time by clicking See results next to the Data Quality Monitoring toggle.

Data quality monitoring dashboard

The first data quality monitor run creates a dashboard to summarize results and trends derived from the logging table. The job run displays a button that you can use to access the dashboard. The dashboard is automatically populated with insights for the scanned schema. A single dashboard is created per workspace at this path: /Shared/Databricks Quality Monitoring/Data Quality Monitoring.

Quality overview

The Quality Overview tab shows a summary of the latest quality status of tables in your schema based on the most recent evaluation.

To get started, you must enter the logging table for the schema you want to analyze to populate the dashboard.

The top section of the dashboard shows an overview of the results of the scan.

Data quality monitor schema summary in Quality Overview tab of the Dashboard.

Below the summary is a table listing quality incidents by impact. Any identified root causes are displayed in the root_cause_analysis column.

Quality incidents by impact in Quality Overview tab of the Dashboard.

Below the quality incident table is a table of identified static tables that have not been updated in a long time.

Table quality details

The Table Quality Details UI allows you to dive deeper into trends and analyze specific tables in your schema. You can get to the UI by either clicking on the table names in the Quality Overview dashboard (see the clickable links in the previous screenshot), or by visiting the Quality Tab in the UC Table viewer.

Given a table, the UI shows summaries from each quality check for the table, with graphs of predicted and observed values at each evaluation timestamp. The graphs plot results from the last 1 week of data.

Table Quality Details UI for data quality monitor.

If the table failed the quality checks, the UI also displays any upstream jobs that were identified as the root cause.

Table Quality Details UI root cause table.

View health indicators

Data quality monitoring provides data consumers with a quick visual confirmation of the data freshness of the tables they use.

On the schema page, in the Overview tab, tables that passed the most recent freshness scan are marked with a green dot. Tables that failed the scan are shown with an orange dot.

Schema overview page in Catalog Explorer showing tables with quality passed mark.

Click on the dot to see the time and status of the most recent scan.

Popup showing details about health status.

As a data owner, you can easily assess the overall health of your schema by sorting tables based on quality. Use the Sort menu at the upper-right of the table list to sort tables by quality.

On the table page, in the Overview tab, a Quality indicator shows the status of the table and lists any anomalies that were identified in the most recent scan.

Healthy quality indicator on table page in Catalog Explorer.

Set up alerts

To configure a Databricks SQL alert on the output results table, follow these steps in the Databricks alerts UI.

  1. Configure the query for the alert:

    SQL
    WITH rounded_data AS (
    SELECT
    DATE_TRUNC('HOUR', evaluated_at) AS evaluated_at,
    CONCAT(catalog, ".", schema, ".", table_name) as full_table_name,
    table_name,
    status,
    MAX(downstream_impact.num_queries_on_affected_tables) AS impacted_queries,
    MAX(CASE WHEN quality_check_type = 'Freshness' AND additional_debug_info.commit_staleness.expectation IS NOT NULL
    THEN additional_debug_info.commit_staleness.expectation END) AS commit_expected,
    MAX(CASE WHEN quality_check_type = 'Freshness' AND additional_debug_info.commit_staleness.actual_value IS NOT NULL
    THEN additional_debug_info.commit_staleness.actual_value END) AS commit_actual,
    MAX(CASE WHEN quality_check_type = 'Freshness' AND additional_debug_info.event_staleness.expectation IS NOT NULL
    THEN additional_debug_info.event_staleness.expectation END) AS event_expected,
    MAX(CASE WHEN quality_check_type = 'Freshness' AND additional_debug_info.event_staleness.actual_value IS NOT NULL
    THEN additional_debug_info.event_staleness.actual_value END) AS event_actual,
    MAX(CASE WHEN quality_check_type = 'Completeness' AND additional_debug_info.daily_row_count.expectation IS NOT NULL
    THEN additional_debug_info.daily_row_count.expectation END) AS completeness_expected,
    MAX(CASE WHEN quality_check_type = 'Completeness' AND additional_debug_info.daily_row_count.actual_value IS NOT NULL
    THEN additional_debug_info.daily_row_count.actual_value END) AS completeness_actual
    FROM <catalog>.<schema>._quality_monitoring_summary
    GROUP BY ALL
    )
    SELECT
    evaluated_at,
    full_table_name,
    status,
    commit_expected,
    commit_actual,
    event_expected,
    event_actual,
    completeness_expected,
    completeness_actual,
    impacted_queries,
    CONCAT("<link-to-dashboard>&f_table-quality-details~table-quality-details-logging-table-name=<catalog>.<schema>._quality_monitoring_summary&f_table-quality-details~9d146eba=", table_name) AS dash_link
    FROM rounded_data
    WHERE
    evaluated_at >= current_timestamp() - INTERVAL 6 HOUR AND
    -- enter the minimum number of table violations before the alert is triggered
    impacted_queries > <min-tables-affected> AND
    status = "Unhealthy"
  2. Configure the alert condition:

    Configure the trigger condition

  3. Customize the email template:

    Html
    <h4>The following tables are failing quality checks in the last hour</h4>

    <table>
    <tr>
    <td>
    <table>
    <tr>
    <th>Table</th>
    <th>Expected Staleness</th>
    <th>Actual Staleness</th>
    <th>Expected Row Volume</th>
    <th>Actual Row Volume</th>
    <th>Impact (queries)</th>
    </tr>
    {{#QUERY_RESULT_ROWS}}
    <tr>
    <td><a href="{{dash_link}}">{{full_table_name}}</a></td>
    <td>{{commit_expected}}</td>
    <td>{{commit_actual}}</td>
    <td>{{completeness_expected}}</td>
    <td>{{completeness_actual}}</td>
    <td>{{impacted_queries}}</td>
    </tr>
    {{/QUERY_RESULT_ROWS}}
    </table>
    </td>
    </tr>
    </table>

Now, you have an alert that triggers based on the downstream impact of the quality issue, and surfaces a link to the dashboard to help you debug the table that triggered the alert.

Disable data quality monitoring

To disable data quality monitoring, click the Data Quality Monitoring toggle to disable it. The data quality monitoring job will be deleted, and all data quality monitoring tables and information will be deleted.

Data quality monitoring selector disabled.

Limitation

Data quality monitoring does not support the following:

  • Views or materialized views.
  • The determination of completeness does not take into account metrics such as the fraction of nulls, zero values, or NaN.
  • Health indicators for completeness
  • "Backtesting" for event-based freshness or completeness

Advanced

Review logged results

By default, data quality monitoring scan results are stored in the system.data_quality_monitoring.table_results table. Only account admins can access this table, and they must grant access to others as needed.

Each row in the results table corresponds to a single table in the schema that was scanned.

The table has the following schema:

Column name

Contents (for struct data type)

Data type

Description

Example data

event_time

timestamp

Time when the row was generated.

2025-06-27T12:00:00

catalog_name

string

Name of the catalog. Used to identify the table.

main

schema_name

string

Name of the schema. Used to identify the table.

default

table_name

string

Name of the table. Used to identify the table.

events

catalog_id

string

Stable ID for the catalog.

3f1a7d6e-9c59-4b76-8c32-8d4c74e289fe

schema_id

string

Stable ID for the schema.

3f1a7d6e-9c59-4b76-8c32-8d4c74e289fe

table_id

string

Stable ID for the table.

3f1a7d6e-9c59-4b76-8c32-8d4c74e289fe

status

string

Consolidated health status at the table level. "Unhealthy" if any check or group is unhealthy.

HEALTHY, UNHEALTHY, UNKNOWN

reason

enum

Reason if the table is unhealthy.

COMPLETENESS_REASON or COMMIT_FRESHNESS_REASON

commit_freshness

struct

Commit freshness check results.

status

string

Status of commit freshness check.

UNHEALTHY

error_code

enum

Error message encountered during check.

FAILED_TO_FIT_MODEL

last_commit

timestamp

Last commit timestamp.

2025-06-27T11:30:00

predicted_commit_by

timestamp

Predicted time by which the table should have been updated.

2025-06-27T11:45:00

completeness

struct

Completeness check results.

status

string

Status of completeness check.

UNHEALTHY

error_code

enum

Error message encountered during check.

FAILED_TO_FIT_MODEL

last_daily_row_volume

int

Number of rows observed in the last 24 hours.

500

min_predicted_daily_row_volume

int

Minimum expected number of rows in the last 24 hours.

10

violation_type

enum

Upper or lower bound. UPPER_BOUND means the table received fewer rows than expected. LOWER_BOUND means the table received more rows than expected.

LOWER_BOUND

max_predicted_daily_row_volume

int

Maximum expected number of rows in the last 24 hours.

1000

downstream_impact

struct

Summary of downstream impact based on dependency graph.

impact_level

int

Severity indicator (0 = none, 1 = low, 2 = medium, 3 = high, 4 = very high).

2

num_downstream_tables

int

Number of downstream tables affected.

5

num_queries_on_affected_tables

int

Number of queries run on affected downstream tables over the last 30 days.

120

root_cause_analysis

struct

Information about upstream jobs contributing to the issue.

upstream_jobs

array

Metadata for each upstream job.

detailed_quality_url

string

Link to Table Quality tab in UI for more information.

https://.../quality

upstream_jobs array structure

The structure of the array shown in the upstream_jobs column is shown in the following table:

Item name

Data type

Description

Example data

job_id

string

Job ID.

12345

workspace_id

string

Workspace ID.

6051921418418893

job_name

string

Job display name.

daily_refresh

last_run_status

string

Status of the most recent run.

SUCCESS

run_page_url

string

URL of Databricks job run page.

https://.../runs/123

Debugging information

In the logged results table, the column additional_debug_info provides information in the following format:

Bash
[
<metric_name>:
actual_value: <value> ,
expectation: “actual value < [predicted_value]
is_violated: true/false,
error_code: <error_code>
from_backtesting: true/false
...
]

For example:

JSON
{
commit_staleness:
actual_value: "31 min"
expectation: "actual_value < 1 day"
is_violated: "false"
error_code: "None"
from_backtesting: "false"
}

Downstream impact information

In the logged results table, the column downstream_impact is a struct with the following fields:

Field

Type

Description

impact_level

int

Integer value between 1 and 4 indicating the severity of the data quality issue. Higher values indicate greater disruption.

num_downstream_tables

int

Number of downstream tables that might be affected by the identified issue.

num_queries_on_affected_tables

int

Total number of queries that have referenced the affected and downstream tables in the past 30 days.

Set parameters for freshness and completeness evaluation (legacy customers only)

Starting from July 21, 2025, configuration of the job parameters will not be supported for new customers. If you need to configure the job settings, please contact Databricks.

To edit the parameters that control the job, such as how often the job runs or the name of the logged results table, you must edit the job parameters on the Tasks tab of the job page.

Jobs page showing data quality monitoring job.

The following sections describe specific settings. For information about how to set task parameters, see Configure task parameters.

Schedule and notifications

To customize the schedule for the job, or to set up notifications, use the Schedules & Triggers settings on the jobs page. See Automating jobs with schedules and triggers.

Name of logging table

To change the name of the logging table, or save the table in a different schema, edit the job task parameter logging_table_name and specify the desired name. To save the logging table in a different schema, specify the full 3-level name.

Customize freshness and completeness evaluations

All of the parameters in this section are optional. By default, data quality monitoring determines thresholds based on an analysis of the table's history.

These parameters are fields inside the task parameter metric_configs. The format of metric_configs is a JSON string with the following default values:

JSON
[
{
"disable_check": false,
"tables_to_skip": null,
"tables_to_scan": null,
"table_threshold_overrides": null,
"table_latency_threshold_overrides": null,
"static_table_threshold_override": null,
"event_timestamp_col_names": null,
"metric_type": "FreshnessConfig"
},
{
"disable_check": true,
"tables_to_skip": null,
"tables_to_scan": null,
"table_threshold_overrides": null,
"metric_type": "CompletenessConfig"
}
]

The following parameters can be used for both freshness and completeness evaluations.

Field name

Description

Example

tables_to_scan

Only the specified tables are scanned.

["table_to_scan", "another_table_to_scan"]

tables_to_skip

The specified tables are skipped during the scan.

["table_to_skip"]

disable_check

Scan is not run. Use this parameter if you want to disable only the freshness scan or only the completeness scan.

true, false

The following parameters apply only to the freshness evaluation:

Field name

Description

Example

event_timestamp_col_names

List of timestamp columns tables in your schema might have. If a table has one of these columns, it is marked Unhealthy if the maximum value of this column is exceeded. Using this parameter might increase evaluation time and cost.

["timestamp", "date"]

table_threshold_overrides

A dictionary consisting of table names and thresholds (in seconds) that specify the maximum interval since the last table update before marking a table as Unhealthy.

{"table_0": 86400}

table_latency_threshold_overrides

A dictionary consisting of table names and latency thresholds (in seconds) that specify the maximum interval since the last timestamp in the table before marking a table as Unhealthy.

{"table_1": 3600}

static_table_threshold_override

Amount of time (in seconds) before a table is considered as a static table (that is, one that is no longer updated).

2592000

The following parameter applies only to the completeness evaluation:

Field name

Description

Example

table_threshold_overrides

A dictionary consisting of table names and row volume thresholds (specified as integers). If the number of rows added to a table over the previous 24 hours is less than the specified threshold, the table is marked Unhealthy.

{"table_0": 1000}