Skip to main content

Monitor

View as Markdown

A data quality monitor on a Unity Catalog object. A schema monitor runs anomaly detection on the tables in the schema and is configured through anomaly_detection_config. A table monitor profiles a single table and is configured through data_profiling_config.

Monitor object​

A data quality monitor on a Unity Catalog object. A schema monitor runs anomaly detection on the tables in the schema and is configured through anomaly_detection_config. A table monitor profiles a single table and is configured through data_profiling_config.

object_typestring

The type of the monitored object. Can be one of the following: schema or table.

Example: schema

object_idstring

The UUID of the request object. It is schema_id for schema, and table_id for table.

Find the schema_id from either:

  1. The schema_id of the Schemas resource.
  2. In Catalog Explorer > select the schema > go to the Details tab > the Schema ID field.

Find the table_id from either:

  1. The table_id of the Tables resource.
  2. In Catalog Explorer > select the table > go to the Details tab > the Table ID field.

Example: 8f1c2d3e-4a5b-4c6d-8e9f-0a1b2c3d4e5f

anomaly_detection_configobject

Anomaly detection settings. Required when object_type is schema. When reading a table monitor, this holds the settings of the parent schema's monitor if the schema has one.

Show child attributesHide child attributes
excluded_table_full_namesarray of stringBeta

List of fully qualified table names to exclude from anomaly detection. Format: catalog.schema.table_name.

Example: main.sales.staging_orders

notification_settingsobject

Who to alert when anomaly detection finds unhealthy tables in this schema. Applicable to schema monitors only. This is the same schema alert you manage from Manage alerts in the Data Quality Monitoring UI, so a change made in either place shows in both.

Set the recipients in on_unhealthy_table; on_failure is not supported here. Only destination_ids and destination_user_ids are supported.

On update, an update mask of *, anomaly_detection_config, or anomaly_detection_config.notification_settings replaces the alert's recipients with the ones in the request; omitting the field, or leaving both recipient lists empty, removes the alert. Other masks leave the alert unchanged.

Show child attributesHide child attributes
on_failureobject

Recipients to notify when a refresh of the monitor fails or times out. Only supported on table monitors, via data_profiling_config, and only email_addresses is accepted there.

Show child attributesHide child attributes
email_addressesarray of string

The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via data_profiling_config; rejected on schema monitors.

destination_idsarray of string

The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

destination_user_idsarray of int64

The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

on_unhealthy_tableobject

Recipients to notify when anomaly detection finds unhealthy tables in the schema. Each recipient gets one notification per important unhealthy table. Only supported on schema monitors, via anomaly_detection_config.

Show child attributesHide child attributes
email_addressesarray of string

The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via data_profiling_config; rejected on schema monitors.

destination_idsarray of string

The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

destination_user_idsarray of int64

The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

data_profiling_configobject

Data profiling settings. Required when object_type is table. Exactly one of inference_log, time_series or snapshot must be set.

Show child attributesHide child attributes
output_schema_idstring

The UUID of the Unity Catalog schema where the profile and drift metrics tables are created.

Example: 6d1a3c0e-7b2f-4e5a-9c8d-2f3b4a5c6d7e

assets_dirstringImmutable

The absolute workspace path of the directory where the monitor's assets, such as the dashboard, are stored. The UI and the Python APIs normally prepopulate a default user location.

Example: /Workspace/Users/user@example.com/databricks_quality_monitoring/main.sales.orders

inference_logobject

Profiles an inference log table, which stores the requests and predictions of a model. Metrics are computed per time window and include model quality metrics.

Show child attributesHide child attributes
problem_typestring

The type of problem the model solves. It determines which model quality metrics are computed.

Values:

  • INFERENCE_PROBLEM_TYPE_CLASSIFICATION
  • INFERENCE_PROBLEM_TYPE_REGRESSION
timestamp_columnstring

Column containing the timestamp of each inference request. Its type must be TIMESTAMP or convertible to a timestamp.

granularitiesarray of string

List of granularities to use when aggregating data into time windows based on their timestamp.

Values:

  • AGGREGATION_GRANULARITY_5_MINUTES
  • AGGREGATION_GRANULARITY_30_MINUTES
  • AGGREGATION_GRANULARITY_1_HOUR
  • AGGREGATION_GRANULARITY_1_DAY
  • AGGREGATION_GRANULARITY_1_WEEK
  • AGGREGATION_GRANULARITY_2_WEEKS
  • AGGREGATION_GRANULARITY_3_WEEKS
  • AGGREGATION_GRANULARITY_4_WEEKS
  • AGGREGATION_GRANULARITY_1_MONTH
  • AGGREGATION_GRANULARITY_1_YEAR
prediction_columnstring

Column containing the model's predicted values.

label_columnstring

Column containing the ground truth for the model's predictions.

model_id_columnstring

Column containing the ID of the model that produced each prediction. A slice is created automatically for each distinct model ID.

time_seriesobject

Profiles a time series table. Metrics are computed per time window, as defined by timestamp_column and granularities.

Show child attributesHide child attributes
timestamp_columnstring

Column containing the timestamp of each row. Its type must be TIMESTAMP or convertible to a timestamp.

granularitiesarray of string

List of granularities to use when aggregating data into time windows based on their timestamp.

Values:

  • AGGREGATION_GRANULARITY_5_MINUTES
  • AGGREGATION_GRANULARITY_30_MINUTES
  • AGGREGATION_GRANULARITY_1_HOUR
  • AGGREGATION_GRANULARITY_1_DAY
  • AGGREGATION_GRANULARITY_1_WEEK
  • AGGREGATION_GRANULARITY_2_WEEKS
  • AGGREGATION_GRANULARITY_3_WEEKS
  • AGGREGATION_GRANULARITY_4_WEEKS
  • AGGREGATION_GRANULARITY_1_MONTH
  • AGGREGATION_GRANULARITY_1_YEAR
snapshotobject

Profiles the full contents of the table. Metrics are computed over all rows at each refresh.

slicing_exprsarray of string

List of column expressions to slice data with for targeted analysis. The data is grouped by each expression independently, resulting in a separate slice for each predicate and its complements. For example slicing_exprs=["col_1", "col_2 > 10"] will generate the following slices: two slices for col_2 > 10 (True and False), and one slice per unique value in col_1. For high-cardinality columns, only the top 100 unique values by frequency will generate slices.

custom_metricsarray of object

Custom metrics to compute in addition to the built-in ones. See custom metrics.

Show child attributesHide child attributes
namestring

Name of the metric in the output tables.

definitionstring

Jinja template for a SQL expression that specifies how to compute the metric. See create metric definition.

input_columnsarray of string

A list of column names in the input table the metric should be computed for. Can use ":table" to indicate that the metric needs information from multiple columns.

output_data_typestring

Specifies the Spark SQL data type of the metric value, either as a plain type name or as a JSON-encoded StructField.

Example: double

typestring

The type of the custom metric. DATA_PROFILING_CUSTOM_METRIC_TYPE_AGGREGATE metrics depend only on the columns of the input table, DATA_PROFILING_CUSTOM_METRIC_TYPE_DERIVED metrics depend only on previously computed aggregate metrics, and DATA_PROFILING_CUSTOM_METRIC_TYPE_DRIFT metrics depend on previously computed aggregate or derived metrics.

Values:

  • DATA_PROFILING_CUSTOM_METRIC_TYPE_AGGREGATE
  • DATA_PROFILING_CUSTOM_METRIC_TYPE_DERIVED
  • DATA_PROFILING_CUSTOM_METRIC_TYPE_DRIFT
baseline_table_namestring

The fully qualified name of a baseline table. Format: catalog.schema.table_name. Drift metrics compare the monitored table against the baseline data, so the two tables must have the same schema.

Example: main.sales.orders_baseline

scheduleobject

The schedule on which the monitor refreshes its metrics. Refreshes can also be triggered on demand through the Refresh resource.

Show child attributesHide child attributes
quartz_cron_expressionstring

The expression that determines when to run the monitor. See examples.

Example: 0 0 12 * * ?

timezone_idstring

The Java timezone ID in which the cron expression is evaluated. See Java TimeZone for the supported IDs.

Example: America/Los_Angeles

pause_statusstringOutput only

Read only field that indicates whether the schedule is paused or not.

Values:

  • CRON_SCHEDULE_PAUSE_STATUS_UNPAUSED
  • CRON_SCHEDULE_PAUSE_STATUS_PAUSED
notification_settingsobject

Who to notify about this monitor's refreshes. Only on_failure with email_addresses is supported here.

Show child attributesHide child attributes
on_failureobject

Recipients to notify when a refresh of the monitor fails or times out. Only supported on table monitors, via data_profiling_config, and only email_addresses is accepted there.

Show child attributesHide child attributes
email_addressesarray of string

The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via data_profiling_config; rejected on schema monitors.

destination_idsarray of string

The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

destination_user_idsarray of int64

The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

on_unhealthy_tableobject

Recipients to notify when anomaly detection finds unhealthy tables in the schema. Each recipient gets one notification per important unhealthy table. Only supported on schema monitors, via anomaly_detection_config.

Show child attributesHide child attributes
email_addressesarray of string

The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via data_profiling_config; rejected on schema monitors.

destination_idsarray of string

The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

destination_user_idsarray of int64

The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

skip_builtin_dashboardbooleanInput only

Whether to skip creating a default dashboard summarizing data quality metrics.

warehouse_idstringInput only

The ID of the SQL warehouse used to create the dashboard. If not specified, the first running warehouse the caller can access is used.

Example: a7066a8ef796be84

monitored_table_namestringOutput only

The fully qualified name of the monitored table. Format: catalog.schema.table_name.

statusstringOutput only

The status of the monitor. DATA_PROFILING_STATUS_PENDING means the monitor's assets are being created or updated. DATA_PROFILING_STATUS_ACTIVE means the monitor is ready and can be refreshed. DATA_PROFILING_STATUS_FAILED and DATA_PROFILING_STATUS_ERROR mean the monitor is not operational, and latest_monitor_failure_message describes the cause when one was recorded. DATA_PROFILING_STATUS_DELETE_PENDING means the monitor is being deleted.

Values:

  • DATA_PROFILING_STATUS_ACTIVE
  • DATA_PROFILING_STATUS_PENDING
  • DATA_PROFILING_STATUS_DELETE_PENDING
  • DATA_PROFILING_STATUS_ERROR
  • DATA_PROFILING_STATUS_FAILED
latest_monitor_failure_messagestringOutput only

The latest error message for a monitor failure.

profile_metrics_table_namestringOutput only

Table that stores profile metrics data. Format: catalog.schema.table_name.

drift_metrics_table_namestringOutput only

Table that stores drift metrics data. Format: catalog.schema.table_name.

dashboard_idstringOutput only

The ID of the dashboard that visualizes the computed metrics. It is empty while the monitor is pending, when the dashboard was skipped with skip_builtin_dashboard, or when the caller is in a different workspace from the one where the monitor was created.

monitor_versionint64Output only

Represents the current monitor configuration version in use. The version will be represented in a numeric fashion (1,2,3...). The field has flexibility to take on negative values, which can indicate corrupted monitor_version numbers.

effective_warehouse_idstringOutput only

The warehouse for dashboard creation

Read a monitor GA​

GET /api/data-quality/v1/monitors/{object_type}/{object_id}

Read a data quality monitor on a Unity Catalog object.

For the table object_type, the caller must have either of the following sets of permissions:

  1. MANAGE and USE_CATALOG on the table's parent catalog.
  2. USE_CATALOG on the table's parent catalog, and MANAGE and USE_SCHEMA on the table's parent schema.
  3. USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and SELECT on the table.

For the schema object_type, the caller must have either of the following sets of permissions:

  1. MANAGE and USE_CATALOG on the schema's parent catalog.
  2. USE_CATALOG on the schema's parent catalog, and USE_SCHEMA on the schema.

The returned information includes configuration values on the entity and parent entity as well as information on assets created by the monitor. Some information (e.g. dashboard) may be filtered out if the caller is in a different workspace than where the monitor was created.

API scopes: dataquality

Parameters​

object_typestringRequiredpath

The type of the monitored object. Can be one of the following: schema or table.

object_idstringRequiredpath

The UUID of the request object. It is schema_id for schema, and table_id for table.

Find the schema_id from either:

  1. The schema_id of the Schemas resource.
  2. In Catalog Explorer > select the schema > go to the Details tab > the Schema ID field.

Find the table_id from either:

  1. The table_id of the Tables resource.
  2. In Catalog Explorer > select the table > go to the Details tab > the Table ID field.

Response​

Returns the Monitor object.

List monitors GA​

GET /api/data-quality/v1/monitors

(Unimplemented) List data quality monitors.

API scopes: dataquality

Parameters​

page_tokenstringquery

Pagination token from a previous response, used to fetch the next page.

page_sizeint32<= 100query

The maximum number of monitors to return in one page.

Default: 25

Response​

Returns a list of Monitor objects.

Create a monitor GA​

POST /api/data-quality/v1/monitors

Create a data quality monitor on a Unity Catalog object. The caller must provide either anomaly_detection_config for a schema monitor or data_profiling_config for a table monitor.

For the table object_type, the caller must have either of the following sets of permissions:

  1. MANAGE and USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and SELECT on the table
  2. USE_CATALOG on the table's parent catalog, MANAGE and USE_SCHEMA on the table's parent schema, and SELECT on the table.
  3. USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and MANAGE and SELECT on the table.

Workspace assets, such as the dashboard, will be created in the workspace where this call was made.

For the schema object_type, the caller must have either of the following sets of permissions:

  1. MANAGE and USE_CATALOG on the schema's parent catalog.
  2. USE_CATALOG on the schema's parent catalog, and MANAGE and USE_SCHEMA on the schema.

API scopes: dataquality

Request body​

The monitor to create.

object_typestringRequired

The type of the monitored object. Can be one of the following: schema or table.

Example: schema

object_idstringRequired

The UUID of the request object. It is schema_id for schema, and table_id for table.

Find the schema_id from either:

  1. The schema_id of the Schemas resource.
  2. In Catalog Explorer > select the schema > go to the Details tab > the Schema ID field.

Find the table_id from either:

  1. The table_id of the Tables resource.
  2. In Catalog Explorer > select the table > go to the Details tab > the Table ID field.

Example: 8f1c2d3e-4a5b-4c6d-8e9f-0a1b2c3d4e5f

anomaly_detection_configobject

Anomaly detection settings. Required when object_type is schema. When reading a table monitor, this holds the settings of the parent schema's monitor if the schema has one.

Show child attributesHide child attributes
excluded_table_full_namesarray of stringBeta

List of fully qualified table names to exclude from anomaly detection. Format: catalog.schema.table_name.

Example: main.sales.staging_orders

notification_settingsobject

Who to alert when anomaly detection finds unhealthy tables in this schema. Applicable to schema monitors only. This is the same schema alert you manage from Manage alerts in the Data Quality Monitoring UI, so a change made in either place shows in both.

Set the recipients in on_unhealthy_table; on_failure is not supported here. Only destination_ids and destination_user_ids are supported.

On update, an update mask of *, anomaly_detection_config, or anomaly_detection_config.notification_settings replaces the alert's recipients with the ones in the request; omitting the field, or leaving both recipient lists empty, removes the alert. Other masks leave the alert unchanged.

Show child attributesHide child attributes
on_failureobject

Recipients to notify when a refresh of the monitor fails or times out. Only supported on table monitors, via data_profiling_config, and only email_addresses is accepted there.

Show child attributesHide child attributes
email_addressesarray of string

The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via data_profiling_config; rejected on schema monitors.

destination_idsarray of string

The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

destination_user_idsarray of int64

The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

on_unhealthy_tableobject

Recipients to notify when anomaly detection finds unhealthy tables in the schema. Each recipient gets one notification per important unhealthy table. Only supported on schema monitors, via anomaly_detection_config.

Show child attributesHide child attributes
email_addressesarray of string

The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via data_profiling_config; rejected on schema monitors.

destination_idsarray of string

The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

destination_user_idsarray of int64

The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

data_profiling_configobject

Data profiling settings. Required when object_type is table. Exactly one of inference_log, time_series or snapshot must be set.

Show child attributesHide child attributes
output_schema_idstringRequired

The UUID of the Unity Catalog schema where the profile and drift metrics tables are created.

Example: 6d1a3c0e-7b2f-4e5a-9c8d-2f3b4a5c6d7e

assets_dirstringImmutable

The absolute workspace path of the directory where the monitor's assets, such as the dashboard, are stored. The UI and the Python APIs normally prepopulate a default user location.

Example: /Workspace/Users/user@example.com/databricks_quality_monitoring/main.sales.orders

inference_logobject

Profiles an inference log table, which stores the requests and predictions of a model. Metrics are computed per time window and include model quality metrics.

Show child attributesHide child attributes
problem_typestringRequired

The type of problem the model solves. It determines which model quality metrics are computed.

Values:

  • INFERENCE_PROBLEM_TYPE_CLASSIFICATION
  • INFERENCE_PROBLEM_TYPE_REGRESSION
timestamp_columnstringRequired

Column containing the timestamp of each inference request. Its type must be TIMESTAMP or convertible to a timestamp.

granularitiesarray of stringRequired

List of granularities to use when aggregating data into time windows based on their timestamp.

Values:

  • AGGREGATION_GRANULARITY_5_MINUTES
  • AGGREGATION_GRANULARITY_30_MINUTES
  • AGGREGATION_GRANULARITY_1_HOUR
  • AGGREGATION_GRANULARITY_1_DAY
  • AGGREGATION_GRANULARITY_1_WEEK
  • AGGREGATION_GRANULARITY_2_WEEKS
  • AGGREGATION_GRANULARITY_3_WEEKS
  • AGGREGATION_GRANULARITY_4_WEEKS
  • AGGREGATION_GRANULARITY_1_MONTH
  • AGGREGATION_GRANULARITY_1_YEAR
prediction_columnstringRequired

Column containing the model's predicted values.

label_columnstring

Column containing the ground truth for the model's predictions.

model_id_columnstringRequired

Column containing the ID of the model that produced each prediction. A slice is created automatically for each distinct model ID.

time_seriesobject

Profiles a time series table. Metrics are computed per time window, as defined by timestamp_column and granularities.

Show child attributesHide child attributes
timestamp_columnstringRequired

Column containing the timestamp of each row. Its type must be TIMESTAMP or convertible to a timestamp.

granularitiesarray of stringRequired

List of granularities to use when aggregating data into time windows based on their timestamp.

Values:

  • AGGREGATION_GRANULARITY_5_MINUTES
  • AGGREGATION_GRANULARITY_30_MINUTES
  • AGGREGATION_GRANULARITY_1_HOUR
  • AGGREGATION_GRANULARITY_1_DAY
  • AGGREGATION_GRANULARITY_1_WEEK
  • AGGREGATION_GRANULARITY_2_WEEKS
  • AGGREGATION_GRANULARITY_3_WEEKS
  • AGGREGATION_GRANULARITY_4_WEEKS
  • AGGREGATION_GRANULARITY_1_MONTH
  • AGGREGATION_GRANULARITY_1_YEAR
snapshotobject

Profiles the full contents of the table. Metrics are computed over all rows at each refresh.

slicing_exprsarray of string

List of column expressions to slice data with for targeted analysis. The data is grouped by each expression independently, resulting in a separate slice for each predicate and its complements. For example slicing_exprs=["col_1", "col_2 > 10"] will generate the following slices: two slices for col_2 > 10 (True and False), and one slice per unique value in col_1. For high-cardinality columns, only the top 100 unique values by frequency will generate slices.

custom_metricsarray of object

Custom metrics to compute in addition to the built-in ones. See custom metrics.

Show child attributesHide child attributes
namestringRequired

Name of the metric in the output tables.

definitionstringRequired

Jinja template for a SQL expression that specifies how to compute the metric. See create metric definition.

input_columnsarray of stringRequired

A list of column names in the input table the metric should be computed for. Can use ":table" to indicate that the metric needs information from multiple columns.

output_data_typestringRequired

Specifies the Spark SQL data type of the metric value, either as a plain type name or as a JSON-encoded StructField.

Example: double

typestringRequired

The type of the custom metric. DATA_PROFILING_CUSTOM_METRIC_TYPE_AGGREGATE metrics depend only on the columns of the input table, DATA_PROFILING_CUSTOM_METRIC_TYPE_DERIVED metrics depend only on previously computed aggregate metrics, and DATA_PROFILING_CUSTOM_METRIC_TYPE_DRIFT metrics depend on previously computed aggregate or derived metrics.

Values:

  • DATA_PROFILING_CUSTOM_METRIC_TYPE_AGGREGATE
  • DATA_PROFILING_CUSTOM_METRIC_TYPE_DERIVED
  • DATA_PROFILING_CUSTOM_METRIC_TYPE_DRIFT
baseline_table_namestring

The fully qualified name of a baseline table. Format: catalog.schema.table_name. Drift metrics compare the monitored table against the baseline data, so the two tables must have the same schema.

Example: main.sales.orders_baseline

scheduleobject

The schedule on which the monitor refreshes its metrics. Refreshes can also be triggered on demand through the Refresh resource.

Show child attributesHide child attributes
quartz_cron_expressionstringRequired

The expression that determines when to run the monitor. See examples.

Example: 0 0 12 * * ?

timezone_idstringRequired

The Java timezone ID in which the cron expression is evaluated. See Java TimeZone for the supported IDs.

Example: America/Los_Angeles

notification_settingsobject

Who to notify about this monitor's refreshes. Only on_failure with email_addresses is supported here.

Show child attributesHide child attributes
on_failureobject

Recipients to notify when a refresh of the monitor fails or times out. Only supported on table monitors, via data_profiling_config, and only email_addresses is accepted there.

Show child attributesHide child attributes
email_addressesarray of string

The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via data_profiling_config; rejected on schema monitors.

destination_idsarray of string

The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

destination_user_idsarray of int64

The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

on_unhealthy_tableobject

Recipients to notify when anomaly detection finds unhealthy tables in the schema. Each recipient gets one notification per important unhealthy table. Only supported on schema monitors, via anomaly_detection_config.

Show child attributesHide child attributes
email_addressesarray of string

The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via data_profiling_config; rejected on schema monitors.

destination_idsarray of string

The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

destination_user_idsarray of int64

The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

skip_builtin_dashboardbooleanInput only

Whether to skip creating a default dashboard summarizing data quality metrics.

warehouse_idstringInput only

The ID of the SQL warehouse used to create the dashboard. If not specified, the first running warehouse the caller can access is used.

Example: a7066a8ef796be84

Response​

Returns the Monitor object.

Update a monitor GA​

PATCH /api/data-quality/v1/monitors/{object_type}/{object_id}

Update a data quality monitor on a Unity Catalog object. Only the fields named in update_mask are changed.

For the table object_type, the caller must have either of the following sets of permissions:

  1. MANAGE and USE_CATALOG on the table's parent catalog.
  2. USE_CATALOG on the table's parent catalog, and MANAGE and USE_SCHEMA on the table's parent schema.
  3. USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and MANAGE on the table.

For the schema object_type, the caller must have either of the following sets of permissions:

  1. MANAGE and USE_CATALOG on the schema's parent catalog.
  2. USE_CATALOG on the schema's parent catalog, and MANAGE and USE_SCHEMA on the schema.

API scopes: dataquality

Parameters​

object_typestringRequiredpath

The type of the monitored object. Can be one of the following: schema or table.

object_idstringRequiredpath

The UUID of the request object. It is schema_id for schema, and table_id for table.

Find the schema_id from either:

  1. The schema_id of the Schemas resource.
  2. In Catalog Explorer > select the schema > go to the Details tab > the Schema ID field.

Find the table_id from either:

  1. The table_id of the Tables resource.
  2. In Catalog Explorer > select the table > go to the Details tab > the Table ID field.
update_maskstringRequiredquery

The field mask to specify which fields to update as a comma-separated list. Example value: data_profiling_config.custom_metrics,data_profiling_config.schedule.quartz_cron_expression

Request body​

The new values for the monitor. Only the fields named in update_mask are applied.

object_typestringRequired

The type of the monitored object. Can be one of the following: schema or table.

Example: schema

object_idstringRequired

The UUID of the request object. It is schema_id for schema, and table_id for table.

Find the schema_id from either:

  1. The schema_id of the Schemas resource.
  2. In Catalog Explorer > select the schema > go to the Details tab > the Schema ID field.

Find the table_id from either:

  1. The table_id of the Tables resource.
  2. In Catalog Explorer > select the table > go to the Details tab > the Table ID field.

Example: 8f1c2d3e-4a5b-4c6d-8e9f-0a1b2c3d4e5f

anomaly_detection_configobject

Anomaly detection settings. Required when object_type is schema. When reading a table monitor, this holds the settings of the parent schema's monitor if the schema has one.

Show child attributesHide child attributes
excluded_table_full_namesarray of stringBeta

List of fully qualified table names to exclude from anomaly detection. Format: catalog.schema.table_name.

Example: main.sales.staging_orders

notification_settingsobject

Who to alert when anomaly detection finds unhealthy tables in this schema. Applicable to schema monitors only. This is the same schema alert you manage from Manage alerts in the Data Quality Monitoring UI, so a change made in either place shows in both.

Set the recipients in on_unhealthy_table; on_failure is not supported here. Only destination_ids and destination_user_ids are supported.

On update, an update mask of *, anomaly_detection_config, or anomaly_detection_config.notification_settings replaces the alert's recipients with the ones in the request; omitting the field, or leaving both recipient lists empty, removes the alert. Other masks leave the alert unchanged.

Show child attributesHide child attributes
on_failureobject

Recipients to notify when a refresh of the monitor fails or times out. Only supported on table monitors, via data_profiling_config, and only email_addresses is accepted there.

Show child attributesHide child attributes
email_addressesarray of string

The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via data_profiling_config; rejected on schema monitors.

destination_idsarray of string

The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

destination_user_idsarray of int64

The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

on_unhealthy_tableobject

Recipients to notify when anomaly detection finds unhealthy tables in the schema. Each recipient gets one notification per important unhealthy table. Only supported on schema monitors, via anomaly_detection_config.

Show child attributesHide child attributes
email_addressesarray of string

The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via data_profiling_config; rejected on schema monitors.

destination_idsarray of string

The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

destination_user_idsarray of int64

The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

data_profiling_configobject

Data profiling settings. Required when object_type is table. Exactly one of inference_log, time_series or snapshot must be set.

Show child attributesHide child attributes
output_schema_idstringRequired

The UUID of the Unity Catalog schema where the profile and drift metrics tables are created.

Example: 6d1a3c0e-7b2f-4e5a-9c8d-2f3b4a5c6d7e

assets_dirstringImmutable

The absolute workspace path of the directory where the monitor's assets, such as the dashboard, are stored. The UI and the Python APIs normally prepopulate a default user location.

Example: /Workspace/Users/user@example.com/databricks_quality_monitoring/main.sales.orders

inference_logobject

Profiles an inference log table, which stores the requests and predictions of a model. Metrics are computed per time window and include model quality metrics.

Show child attributesHide child attributes
problem_typestringRequired

The type of problem the model solves. It determines which model quality metrics are computed.

Values:

  • INFERENCE_PROBLEM_TYPE_CLASSIFICATION
  • INFERENCE_PROBLEM_TYPE_REGRESSION
timestamp_columnstringRequired

Column containing the timestamp of each inference request. Its type must be TIMESTAMP or convertible to a timestamp.

granularitiesarray of stringRequired

List of granularities to use when aggregating data into time windows based on their timestamp.

Values:

  • AGGREGATION_GRANULARITY_5_MINUTES
  • AGGREGATION_GRANULARITY_30_MINUTES
  • AGGREGATION_GRANULARITY_1_HOUR
  • AGGREGATION_GRANULARITY_1_DAY
  • AGGREGATION_GRANULARITY_1_WEEK
  • AGGREGATION_GRANULARITY_2_WEEKS
  • AGGREGATION_GRANULARITY_3_WEEKS
  • AGGREGATION_GRANULARITY_4_WEEKS
  • AGGREGATION_GRANULARITY_1_MONTH
  • AGGREGATION_GRANULARITY_1_YEAR
prediction_columnstringRequired

Column containing the model's predicted values.

label_columnstring

Column containing the ground truth for the model's predictions.

model_id_columnstringRequired

Column containing the ID of the model that produced each prediction. A slice is created automatically for each distinct model ID.

time_seriesobject

Profiles a time series table. Metrics are computed per time window, as defined by timestamp_column and granularities.

Show child attributesHide child attributes
timestamp_columnstringRequired

Column containing the timestamp of each row. Its type must be TIMESTAMP or convertible to a timestamp.

granularitiesarray of stringRequired

List of granularities to use when aggregating data into time windows based on their timestamp.

Values:

  • AGGREGATION_GRANULARITY_5_MINUTES
  • AGGREGATION_GRANULARITY_30_MINUTES
  • AGGREGATION_GRANULARITY_1_HOUR
  • AGGREGATION_GRANULARITY_1_DAY
  • AGGREGATION_GRANULARITY_1_WEEK
  • AGGREGATION_GRANULARITY_2_WEEKS
  • AGGREGATION_GRANULARITY_3_WEEKS
  • AGGREGATION_GRANULARITY_4_WEEKS
  • AGGREGATION_GRANULARITY_1_MONTH
  • AGGREGATION_GRANULARITY_1_YEAR
snapshotobject

Profiles the full contents of the table. Metrics are computed over all rows at each refresh.

slicing_exprsarray of string

List of column expressions to slice data with for targeted analysis. The data is grouped by each expression independently, resulting in a separate slice for each predicate and its complements. For example slicing_exprs=["col_1", "col_2 > 10"] will generate the following slices: two slices for col_2 > 10 (True and False), and one slice per unique value in col_1. For high-cardinality columns, only the top 100 unique values by frequency will generate slices.

custom_metricsarray of object

Custom metrics to compute in addition to the built-in ones. See custom metrics.

Show child attributesHide child attributes
namestringRequired

Name of the metric in the output tables.

definitionstringRequired

Jinja template for a SQL expression that specifies how to compute the metric. See create metric definition.

input_columnsarray of stringRequired

A list of column names in the input table the metric should be computed for. Can use ":table" to indicate that the metric needs information from multiple columns.

output_data_typestringRequired

Specifies the Spark SQL data type of the metric value, either as a plain type name or as a JSON-encoded StructField.

Example: double

typestringRequired

The type of the custom metric. DATA_PROFILING_CUSTOM_METRIC_TYPE_AGGREGATE metrics depend only on the columns of the input table, DATA_PROFILING_CUSTOM_METRIC_TYPE_DERIVED metrics depend only on previously computed aggregate metrics, and DATA_PROFILING_CUSTOM_METRIC_TYPE_DRIFT metrics depend on previously computed aggregate or derived metrics.

Values:

  • DATA_PROFILING_CUSTOM_METRIC_TYPE_AGGREGATE
  • DATA_PROFILING_CUSTOM_METRIC_TYPE_DERIVED
  • DATA_PROFILING_CUSTOM_METRIC_TYPE_DRIFT
baseline_table_namestring

The fully qualified name of a baseline table. Format: catalog.schema.table_name. Drift metrics compare the monitored table against the baseline data, so the two tables must have the same schema.

Example: main.sales.orders_baseline

scheduleobject

The schedule on which the monitor refreshes its metrics. Refreshes can also be triggered on demand through the Refresh resource.

Show child attributesHide child attributes
quartz_cron_expressionstringRequired

The expression that determines when to run the monitor. See examples.

Example: 0 0 12 * * ?

timezone_idstringRequired

The Java timezone ID in which the cron expression is evaluated. See Java TimeZone for the supported IDs.

Example: America/Los_Angeles

notification_settingsobject

Who to notify about this monitor's refreshes. Only on_failure with email_addresses is supported here.

Show child attributesHide child attributes
on_failureobject

Recipients to notify when a refresh of the monitor fails or times out. Only supported on table monitors, via data_profiling_config, and only email_addresses is accepted there.

Show child attributesHide child attributes
email_addressesarray of string

The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via data_profiling_config; rejected on schema monitors.

destination_idsarray of string

The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

destination_user_idsarray of int64

The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

on_unhealthy_tableobject

Recipients to notify when anomaly detection finds unhealthy tables in the schema. Each recipient gets one notification per important unhealthy table. Only supported on schema monitors, via anomaly_detection_config.

Show child attributesHide child attributes
email_addressesarray of string

The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via data_profiling_config; rejected on schema monitors.

destination_idsarray of string

The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

destination_user_idsarray of int64

The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

skip_builtin_dashboardbooleanInput only

Whether to skip creating a default dashboard summarizing data quality metrics.

warehouse_idstringInput only

The ID of the SQL warehouse used to create the dashboard. If not specified, the first running warehouse the caller can access is used.

Example: a7066a8ef796be84

Response​

Returns the Monitor object.

Delete a monitor GA​

DELETE /api/data-quality/v1/monitors/{object_type}/{object_id}

Delete a data quality monitor on a Unity Catalog object.

For the table object_type, the caller must have either of the following sets of permissions:

  1. MANAGE and USE_CATALOG on the table's parent catalog.
  2. USE_CATALOG on the table's parent catalog, and MANAGE and USE_SCHEMA on the table's parent schema.
  3. USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and MANAGE on the table.

Note that the metric tables and dashboard will not be deleted as part of this call; those assets must be manually cleaned up (if desired).

For the schema object_type, the caller must have either of the following sets of permissions:

  1. MANAGE and USE_CATALOG on the schema's parent catalog.
  2. USE_CATALOG on the schema's parent catalog, and MANAGE and USE_SCHEMA on the schema.

API scopes: dataquality

Parameters​

object_typestringRequiredpath

The type of the monitored object. Can be one of the following: schema or table.

object_idstringRequiredpath

The UUID of the request object. It is schema_id for schema, and table_id for table.

Find the schema_id from either:

  1. The schema_id of the Schemas resource.
  2. In Catalog Explorer > select the schema > go to the Details tab > the Schema ID field.

Find the table_id from either:

  1. The table_id of the Tables resource.
  2. In Catalog Explorer > select the table > go to the Details tab > the Table ID field.