Alerts for anomaly detection

This page describes how to set up and view alerts for anomaly detection. For an introduction to anomaly detection, see Anomaly detection.

Alerts notify you when anomaly detection identifies a data quality issue, such as a stale table or an unexpected drop in row count. You can work with alerts in the following ways:

• Data Quality Monitoring UI (Beta): Create and view alerts directly from the Data Quality Monitoring UI without leaving the page. See Create an alert in the Data Quality Monitoring UI.
• Databricks SQL: Configure an alert by querying the anomaly detection output system table. This approach supports advanced filtering and custom notification templates. See Create an alert with Databricks SQL.

View alerts in the Data Quality Monitoring UI

Beta

This feature is in Beta, and is visible to all users by default. Workspace admins don't need to enable the feature from the Previews page.

The Alerts button in the upper-right corner of the Data Quality Monitoring UI provides a summary of your anomaly detection alerts without leaving the page.

To view your alerts:

1. Navigate to a schema in Catalog Explorer. Data quality monitoring must be enabled for this schema.

2. Click the Details tab. Next to Data Quality Monitoring, click View results. This opens the Data Quality Monitoring UI.

3. In the upper-right corner of the page, click Alerts (Beta). This opens a panel showing a summary of your configured alerts. You can also click the Alert to quickly create a new alert.

4. Click any alert in the list to open its detailed view.

   

Creating alerts

You can create alerts using the Data Quality Monitoring UI or with Databricks SQL.

Create an alert in the Data Quality Monitoring UI

Beta

This feature is in Beta, and is visible to all users by default. Workspace admins don't need to enable the feature from the Previews page.

important

By default, only account admins can access the system table system.data_quality_monitoring.table_results. If other users need to configure alerts, make sure to grant appropriate access.

To create an alert in the Data Quality Monitoring UI, follow these steps:

1. Navigate to a schema that you want to create an alert for in Catalog Explorer. Data quality monitoring must be enabled for this schema.

2. Click the Details tab and then click View results next to Data Quality Monitoring. This opens the Data Quality Monitoring UI for the specific catalog and schema that you selected.

   To create an alert for all schemas in a catalog, select All Schemas from the schema drop-down menu.

3. If you're creating your first alert, click Create alert (Beta) in the upper-right corner.

   

   If an alert already exists, this button appears as Alerts (Beta). Click the button to open a panel showing a summary of your configured alerts. Click Alert to create a new alert.

   

4. Configure your alert. The alert is prepopulated with an SQL query, which you can edit using the Edit button in the top right corner. Additionally, you can click the Schedule button to configure an automated schedule for evaluating the alert, or click Run now to evaluate the alert now.

Create an alert with Databricks SQL

important

By default, only account admins can access the system table system.data_quality_monitoring.table_results. If other users need to configure alerts, make sure to grant appropriate access.

To create an alert on the anomaly detection output results table using Databricks SQL, follow these steps:

1. Click Alerts in the sidebar and click Create alert.

2. Enter the following text in the query editor:

   SQL

   WITH rounded_data AS (
   SELECT
      DATE_TRUNC('HOUR', event_time) AS evaluated_at,
      CONCAT(catalog_name, '.', schema_name, '.', table_name) AS full_table_name,
      status,
      MAX(downstream_impact.num_queries_on_affected_tables) AS impacted_queries,
      MAX(freshness.commit_freshness.predicted_value) AS commit_expected,
      MAX(freshness.commit_freshness.last_value) AS commit_actual,
      MAX(completeness.daily_row_count.min_predicted_value) AS completeness_expected,
      MAX(completeness.daily_row_count.last_value) AS completeness_actual
   FROM system.data_quality_monitoring.table_results
   GROUP BY ALL
   )
   SELECT
      evaluated_at,
      full_table_name,
      status,
      commit_expected,
      commit_actual,
      completeness_expected,
      completeness_actual,
      impacted_queries
   FROM rounded_data
   WHERE
      evaluated_at >= current_timestamp() - INTERVAL 6 HOURS
      -- enter the minimum number of table violations before the alert is triggered
      AND impacted_queries > :min_tables_affected
      AND status = 'Unhealthy';

   note

   For legacy beta jobs, existing alert configuration should replace system.data_quality_monitoring.table_results with <catalog>.<schema>._quality_monitoring_summary.

3. Run the query.

4. Using the alert editor on the right side of the screen, configure the alert condition. For details, see Create an alert.

   

5. (Optional) To customize the email template, open the Advanced tab on the right side of the screen, and check Customize template to open the template editor.

   An example custom email template is shown below. For more information about custom templates, see Advanced settings.

   Html

   <h4>The following tables are failing quality checks in the last hour</h4>
   
   <table>
     <tr>
       <td>
         <table>
           <tr>
             <th>Table</th>
             <th>Expected Staleness</th>
             <th>Actual Staleness</th>
             <th>Expected Row Volume</th>
             <th>Actual Row Volume</th>
             <th>Impact (queries)</th>
           </tr>
           {{#QUERY_RESULT_ROWS}}
           <tr>
             <td>{{full_table_name}}</td>
             <td>{{commit_expected}}</td>
             <td>{{commit_actual}}</td>
             <td>{{completeness_expected}}</td>
             <td>{{completeness_actual}}</td>
             <td>{{impacted_queries}}</td>
           </tr>
           {{/QUERY_RESULT_ROWS}}
         </table>
       </td>
     </tr>
   </table>

Now, you have an alert that triggers based on the downstream impact of the quality issue, helping you debug the table that triggered the alert.