Monitor
A data quality monitor on a Unity Catalog object. A schema monitor runs anomaly detection on the tables in the
schema and is configured through anomaly_detection_config. A table monitor profiles a single table and is
configured through data_profiling_config.
Monitor object
A data quality monitor on a Unity Catalog object. A schema monitor runs anomaly detection on the tables in the
schema and is configured through anomaly_detection_config. A table monitor profiles a single table and is
configured through data_profiling_config.
- object_typestring
The type of the monitored object. Can be one of the following:
schemaortable.
- object_idstring
The UUID of the request object. It is
schema_idforschema, andtable_idfortable.Find the
schema_idfrom either:- The schema_id of the
Schemasresource. - In Catalog Explorer > select the
schema> go to theDetailstab > theSchema IDfield.
Find the
table_idfrom either:- The table_id of the
Tablesresource. - In Catalog Explorer > select the
table> go to theDetailstab > theTable IDfield.
- The schema_id of the
- anomaly_detection_configobject
Anomaly detection settings. Required when
object_typeisschema. When reading atablemonitor, this holds the settings of the parent schema's monitor if the schema has one.Show child attributesHide child attributes
- excluded_table_full_namesarray of stringBeta
List of fully qualified table names to exclude from anomaly detection. Format:
catalog.schema.table_name.
- notification_settingsobject
Who to alert when anomaly detection finds unhealthy tables in this schema. Applicable to
schemamonitors only. This is the same schema alert you manage from Manage alerts in the Data Quality Monitoring UI, so a change made in either place shows in both.Set the recipients in
on_unhealthy_table;on_failureis not supported here. Onlydestination_idsanddestination_user_idsare supported.On update, an update mask of
*,anomaly_detection_config, oranomaly_detection_config.notification_settingsreplaces the alert's recipients with the ones in the request; omitting the field, or leaving both recipient lists empty, removes the alert. Other masks leave the alert unchanged.Show child attributesHide child attributes
- on_failureobject
Recipients to notify when a refresh of the monitor fails or times out. Only supported on table monitors, via
data_profiling_config, and onlyemail_addressesis accepted there.Show child attributesHide child attributes
- email_addressesarray of string
The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via
data_profiling_config; rejected on schema monitors.
- destination_idsarray of string
The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- destination_user_idsarray of int64
The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- on_unhealthy_tableobject
Recipients to notify when anomaly detection finds unhealthy tables in the schema. Each recipient gets one notification per important unhealthy table. Only supported on schema monitors, via
anomaly_detection_config.Show child attributesHide child attributes
- email_addressesarray of string
The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via
data_profiling_config; rejected on schema monitors.
- destination_idsarray of string
The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- destination_user_idsarray of int64
The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- data_profiling_configobject
Data profiling settings. Required when
object_typeistable. Exactly one ofinference_log,time_seriesorsnapshotmust be set.Show child attributesHide child attributes
- output_schema_idstring
The UUID of the Unity Catalog schema where the profile and drift metrics tables are created.
- assets_dirstringImmutable
The absolute workspace path of the directory where the monitor's assets, such as the dashboard, are stored. The UI and the Python APIs normally prepopulate a default user location.
- inference_logobject
Profiles an inference log table, which stores the requests and predictions of a model. Metrics are computed per time window and include model quality metrics.
Show child attributesHide child attributes
- problem_typestring
The type of problem the model solves. It determines which model quality metrics are computed.
INFERENCE_PROBLEM_TYPE_CLASSIFICATIONINFERENCE_PROBLEM_TYPE_REGRESSION
- timestamp_columnstring
Column containing the timestamp of each inference request. Its type must be TIMESTAMP or convertible to a timestamp.
- granularitiesarray of string
List of granularities to use when aggregating data into time windows based on their timestamp.
AGGREGATION_GRANULARITY_5_MINUTESAGGREGATION_GRANULARITY_30_MINUTESAGGREGATION_GRANULARITY_1_HOURAGGREGATION_GRANULARITY_1_DAYAGGREGATION_GRANULARITY_1_WEEKAGGREGATION_GRANULARITY_2_WEEKSAGGREGATION_GRANULARITY_3_WEEKSAGGREGATION_GRANULARITY_4_WEEKSAGGREGATION_GRANULARITY_1_MONTHAGGREGATION_GRANULARITY_1_YEAR
- prediction_columnstring
Column containing the model's predicted values.
- label_columnstring
Column containing the ground truth for the model's predictions.
- model_id_columnstring
Column containing the ID of the model that produced each prediction. A slice is created automatically for each distinct model ID.
- time_seriesobject
Profiles a time series table. Metrics are computed per time window, as defined by
timestamp_columnandgranularities.Show child attributesHide child attributes
- timestamp_columnstring
Column containing the timestamp of each row. Its type must be TIMESTAMP or convertible to a timestamp.
- granularitiesarray of string
List of granularities to use when aggregating data into time windows based on their timestamp.
AGGREGATION_GRANULARITY_5_MINUTESAGGREGATION_GRANULARITY_30_MINUTESAGGREGATION_GRANULARITY_1_HOURAGGREGATION_GRANULARITY_1_DAYAGGREGATION_GRANULARITY_1_WEEKAGGREGATION_GRANULARITY_2_WEEKSAGGREGATION_GRANULARITY_3_WEEKSAGGREGATION_GRANULARITY_4_WEEKSAGGREGATION_GRANULARITY_1_MONTHAGGREGATION_GRANULARITY_1_YEAR
- snapshotobject
Profiles the full contents of the table. Metrics are computed over all rows at each refresh.
- slicing_exprsarray of string
List of column expressions to slice data with for targeted analysis. The data is grouped by each expression independently, resulting in a separate slice for each predicate and its complements. For example
slicing_exprs=["col_1", "col_2 > 10"]will generate the following slices: two slices forcol_2 > 10(True and False), and one slice per unique value incol_1. For high-cardinality columns, only the top 100 unique values by frequency will generate slices.
- custom_metricsarray of object
Custom metrics to compute in addition to the built-in ones. See custom metrics.
Show child attributesHide child attributes
- namestring
Name of the metric in the output tables.
- definitionstring
Jinja template for a SQL expression that specifies how to compute the metric. See create metric definition.
- input_columnsarray of string
A list of column names in the input table the metric should be computed for. Can use
":table"to indicate that the metric needs information from multiple columns.
- output_data_typestring
Specifies the Spark SQL data type of the metric value, either as a plain type name or as a JSON-encoded StructField.
- typestring
The type of the custom metric.
DATA_PROFILING_CUSTOM_METRIC_TYPE_AGGREGATEmetrics depend only on the columns of the input table,DATA_PROFILING_CUSTOM_METRIC_TYPE_DERIVEDmetrics depend only on previously computed aggregate metrics, andDATA_PROFILING_CUSTOM_METRIC_TYPE_DRIFTmetrics depend on previously computed aggregate or derived metrics.DATA_PROFILING_CUSTOM_METRIC_TYPE_AGGREGATEDATA_PROFILING_CUSTOM_METRIC_TYPE_DERIVEDDATA_PROFILING_CUSTOM_METRIC_TYPE_DRIFT
- baseline_table_namestring
The fully qualified name of a baseline table. Format:
catalog.schema.table_name. Drift metrics compare the monitored table against the baseline data, so the two tables must have the same schema.
- scheduleobject
The schedule on which the monitor refreshes its metrics. Refreshes can also be triggered on demand through the Refresh resource.
Show child attributesHide child attributes
- quartz_cron_expressionstring
The expression that determines when to run the monitor. See examples.
- timezone_idstring
The Java timezone ID in which the cron expression is evaluated. See Java TimeZone for the supported IDs.
- pause_statusstringOutput only
Read only field that indicates whether the schedule is paused or not.
CRON_SCHEDULE_PAUSE_STATUS_UNPAUSEDCRON_SCHEDULE_PAUSE_STATUS_PAUSED
- notification_settingsobject
Who to notify about this monitor's refreshes. Only
on_failurewithemail_addressesis supported here.Show child attributesHide child attributes
- on_failureobject
Recipients to notify when a refresh of the monitor fails or times out. Only supported on table monitors, via
data_profiling_config, and onlyemail_addressesis accepted there.Show child attributesHide child attributes
- email_addressesarray of string
The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via
data_profiling_config; rejected on schema monitors.
- destination_idsarray of string
The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- destination_user_idsarray of int64
The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- on_unhealthy_tableobject
Recipients to notify when anomaly detection finds unhealthy tables in the schema. Each recipient gets one notification per important unhealthy table. Only supported on schema monitors, via
anomaly_detection_config.Show child attributesHide child attributes
- email_addressesarray of string
The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via
data_profiling_config; rejected on schema monitors.
- destination_idsarray of string
The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- destination_user_idsarray of int64
The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- skip_builtin_dashboardbooleanInput only
Whether to skip creating a default dashboard summarizing data quality metrics.
- warehouse_idstringInput only
The ID of the SQL warehouse used to create the dashboard. If not specified, the first running warehouse the caller can access is used.
- monitored_table_namestringOutput only
The fully qualified name of the monitored table. Format:
catalog.schema.table_name.
- statusstringOutput only
The status of the monitor.
DATA_PROFILING_STATUS_PENDINGmeans the monitor's assets are being created or updated.DATA_PROFILING_STATUS_ACTIVEmeans the monitor is ready and can be refreshed.DATA_PROFILING_STATUS_FAILEDandDATA_PROFILING_STATUS_ERRORmean the monitor is not operational, andlatest_monitor_failure_messagedescribes the cause when one was recorded.DATA_PROFILING_STATUS_DELETE_PENDINGmeans the monitor is being deleted.DATA_PROFILING_STATUS_ACTIVEDATA_PROFILING_STATUS_PENDINGDATA_PROFILING_STATUS_DELETE_PENDINGDATA_PROFILING_STATUS_ERRORDATA_PROFILING_STATUS_FAILED
- latest_monitor_failure_messagestringOutput only
The latest error message for a monitor failure.
- profile_metrics_table_namestringOutput only
Table that stores profile metrics data. Format:
catalog.schema.table_name.
- drift_metrics_table_namestringOutput only
Table that stores drift metrics data. Format:
catalog.schema.table_name.
- dashboard_idstringOutput only
The ID of the dashboard that visualizes the computed metrics. It is empty while the monitor is pending, when the dashboard was skipped with
skip_builtin_dashboard, or when the caller is in a different workspace from the one where the monitor was created.
- monitor_versionint64Output only
Represents the current monitor configuration version in use. The version will be represented in a numeric fashion (1,2,3...). The field has flexibility to take on negative values, which can indicate corrupted monitor_version numbers.
- effective_warehouse_idstringOutput only
The warehouse for dashboard creation
Read a monitor GA
GET
Read a data quality monitor on a Unity Catalog object.
For the table object_type, the caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the table's parent catalog.
- USE_CATALOG on the table's parent catalog, and MANAGE and USE_SCHEMA on the table's parent schema.
- USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and SELECT on the table.
For the schema object_type, the caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the schema's parent catalog.
- USE_CATALOG on the schema's parent catalog, and USE_SCHEMA on the schema.
The returned information includes configuration values on the entity and parent entity as well as information on assets created by the monitor. Some information (e.g. dashboard) may be filtered out if the caller is in a different workspace than where the monitor was created.
API scopes: dataquality
Parameters
- object_typestringRequiredpath
The type of the monitored object. Can be one of the following:
schemaortable.
- object_idstringRequiredpath
The UUID of the request object. It is
schema_idforschema, andtable_idfortable.Find the
schema_idfrom either:- The schema_id of the
Schemasresource. - In Catalog Explorer > select the
schema> go to theDetailstab > theSchema IDfield.
Find the
table_idfrom either:- The table_id of the
Tablesresource. - In Catalog Explorer > select the
table> go to theDetailstab > theTable IDfield.
- The schema_id of the
Response
Returns the Monitor object.
List monitors GA
GET
(Unimplemented) List data quality monitors.
API scopes: dataquality
Parameters
- page_tokenstringquery
Pagination token from a previous response, used to fetch the next page.
- page_sizeint32<= 100query
The maximum number of monitors to return in one page.
Response
Returns a list of Monitor objects.
Create a monitor GA
POST
Create a data quality monitor on a Unity Catalog object. The caller must provide either anomaly_detection_config for a schema monitor or data_profiling_config for a table monitor.
For the table object_type, the caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and SELECT on the table
- USE_CATALOG on the table's parent catalog, MANAGE and USE_SCHEMA on the table's parent schema, and SELECT on the table.
- USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and MANAGE and SELECT on the table.
Workspace assets, such as the dashboard, will be created in the workspace where this call was made.
For the schema object_type, the caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the schema's parent catalog.
- USE_CATALOG on the schema's parent catalog, and MANAGE and USE_SCHEMA on the schema.
API scopes: dataquality
Request body
The monitor to create.
- object_typestringRequired
The type of the monitored object. Can be one of the following:
schemaortable.
- object_idstringRequired
The UUID of the request object. It is
schema_idforschema, andtable_idfortable.Find the
schema_idfrom either:- The schema_id of the
Schemasresource. - In Catalog Explorer > select the
schema> go to theDetailstab > theSchema IDfield.
Find the
table_idfrom either:- The table_id of the
Tablesresource. - In Catalog Explorer > select the
table> go to theDetailstab > theTable IDfield.
- The schema_id of the
- anomaly_detection_configobject
Anomaly detection settings. Required when
object_typeisschema. When reading atablemonitor, this holds the settings of the parent schema's monitor if the schema has one.Show child attributesHide child attributes
- excluded_table_full_namesarray of stringBeta
List of fully qualified table names to exclude from anomaly detection. Format:
catalog.schema.table_name.
- notification_settingsobject
Who to alert when anomaly detection finds unhealthy tables in this schema. Applicable to
schemamonitors only. This is the same schema alert you manage from Manage alerts in the Data Quality Monitoring UI, so a change made in either place shows in both.Set the recipients in
on_unhealthy_table;on_failureis not supported here. Onlydestination_idsanddestination_user_idsare supported.On update, an update mask of
*,anomaly_detection_config, oranomaly_detection_config.notification_settingsreplaces the alert's recipients with the ones in the request; omitting the field, or leaving both recipient lists empty, removes the alert. Other masks leave the alert unchanged.Show child attributesHide child attributes
- on_failureobject
Recipients to notify when a refresh of the monitor fails or times out. Only supported on table monitors, via
data_profiling_config, and onlyemail_addressesis accepted there.Show child attributesHide child attributes
- email_addressesarray of string
The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via
data_profiling_config; rejected on schema monitors.
- destination_idsarray of string
The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- destination_user_idsarray of int64
The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- on_unhealthy_tableobject
Recipients to notify when anomaly detection finds unhealthy tables in the schema. Each recipient gets one notification per important unhealthy table. Only supported on schema monitors, via
anomaly_detection_config.Show child attributesHide child attributes
- email_addressesarray of string
The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via
data_profiling_config; rejected on schema monitors.
- destination_idsarray of string
The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- destination_user_idsarray of int64
The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- data_profiling_configobject
Data profiling settings. Required when
object_typeistable. Exactly one ofinference_log,time_seriesorsnapshotmust be set.Show child attributesHide child attributes
- output_schema_idstringRequired
The UUID of the Unity Catalog schema where the profile and drift metrics tables are created.
- assets_dirstringImmutable
The absolute workspace path of the directory where the monitor's assets, such as the dashboard, are stored. The UI and the Python APIs normally prepopulate a default user location.
- inference_logobject
Profiles an inference log table, which stores the requests and predictions of a model. Metrics are computed per time window and include model quality metrics.
Show child attributesHide child attributes
- problem_typestringRequired
The type of problem the model solves. It determines which model quality metrics are computed.
INFERENCE_PROBLEM_TYPE_CLASSIFICATIONINFERENCE_PROBLEM_TYPE_REGRESSION
- timestamp_columnstringRequired
Column containing the timestamp of each inference request. Its type must be TIMESTAMP or convertible to a timestamp.
- granularitiesarray of stringRequired
List of granularities to use when aggregating data into time windows based on their timestamp.
AGGREGATION_GRANULARITY_5_MINUTESAGGREGATION_GRANULARITY_30_MINUTESAGGREGATION_GRANULARITY_1_HOURAGGREGATION_GRANULARITY_1_DAYAGGREGATION_GRANULARITY_1_WEEKAGGREGATION_GRANULARITY_2_WEEKSAGGREGATION_GRANULARITY_3_WEEKSAGGREGATION_GRANULARITY_4_WEEKSAGGREGATION_GRANULARITY_1_MONTHAGGREGATION_GRANULARITY_1_YEAR
- prediction_columnstringRequired
Column containing the model's predicted values.
- label_columnstring
Column containing the ground truth for the model's predictions.
- model_id_columnstringRequired
Column containing the ID of the model that produced each prediction. A slice is created automatically for each distinct model ID.
- time_seriesobject
Profiles a time series table. Metrics are computed per time window, as defined by
timestamp_columnandgranularities.Show child attributesHide child attributes
- timestamp_columnstringRequired
Column containing the timestamp of each row. Its type must be TIMESTAMP or convertible to a timestamp.
- granularitiesarray of stringRequired
List of granularities to use when aggregating data into time windows based on their timestamp.
AGGREGATION_GRANULARITY_5_MINUTESAGGREGATION_GRANULARITY_30_MINUTESAGGREGATION_GRANULARITY_1_HOURAGGREGATION_GRANULARITY_1_DAYAGGREGATION_GRANULARITY_1_WEEKAGGREGATION_GRANULARITY_2_WEEKSAGGREGATION_GRANULARITY_3_WEEKSAGGREGATION_GRANULARITY_4_WEEKSAGGREGATION_GRANULARITY_1_MONTHAGGREGATION_GRANULARITY_1_YEAR
- snapshotobject
Profiles the full contents of the table. Metrics are computed over all rows at each refresh.
- slicing_exprsarray of string
List of column expressions to slice data with for targeted analysis. The data is grouped by each expression independently, resulting in a separate slice for each predicate and its complements. For example
slicing_exprs=["col_1", "col_2 > 10"]will generate the following slices: two slices forcol_2 > 10(True and False), and one slice per unique value incol_1. For high-cardinality columns, only the top 100 unique values by frequency will generate slices.
- custom_metricsarray of object
Custom metrics to compute in addition to the built-in ones. See custom metrics.
Show child attributesHide child attributes
- namestringRequired
Name of the metric in the output tables.
- definitionstringRequired
Jinja template for a SQL expression that specifies how to compute the metric. See create metric definition.
- input_columnsarray of stringRequired
A list of column names in the input table the metric should be computed for. Can use
":table"to indicate that the metric needs information from multiple columns.
- output_data_typestringRequired
Specifies the Spark SQL data type of the metric value, either as a plain type name or as a JSON-encoded StructField.
- typestringRequired
The type of the custom metric.
DATA_PROFILING_CUSTOM_METRIC_TYPE_AGGREGATEmetrics depend only on the columns of the input table,DATA_PROFILING_CUSTOM_METRIC_TYPE_DERIVEDmetrics depend only on previously computed aggregate metrics, andDATA_PROFILING_CUSTOM_METRIC_TYPE_DRIFTmetrics depend on previously computed aggregate or derived metrics.DATA_PROFILING_CUSTOM_METRIC_TYPE_AGGREGATEDATA_PROFILING_CUSTOM_METRIC_TYPE_DERIVEDDATA_PROFILING_CUSTOM_METRIC_TYPE_DRIFT
- baseline_table_namestring
The fully qualified name of a baseline table. Format:
catalog.schema.table_name. Drift metrics compare the monitored table against the baseline data, so the two tables must have the same schema.
- scheduleobject
The schedule on which the monitor refreshes its metrics. Refreshes can also be triggered on demand through the Refresh resource.
Show child attributesHide child attributes
- quartz_cron_expressionstringRequired
The expression that determines when to run the monitor. See examples.
- timezone_idstringRequired
The Java timezone ID in which the cron expression is evaluated. See Java TimeZone for the supported IDs.
- notification_settingsobject
Who to notify about this monitor's refreshes. Only
on_failurewithemail_addressesis supported here.Show child attributesHide child attributes
- on_failureobject
Recipients to notify when a refresh of the monitor fails or times out. Only supported on table monitors, via
data_profiling_config, and onlyemail_addressesis accepted there.Show child attributesHide child attributes
- email_addressesarray of string
The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via
data_profiling_config; rejected on schema monitors.
- destination_idsarray of string
The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- destination_user_idsarray of int64
The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- on_unhealthy_tableobject
Recipients to notify when anomaly detection finds unhealthy tables in the schema. Each recipient gets one notification per important unhealthy table. Only supported on schema monitors, via
anomaly_detection_config.Show child attributesHide child attributes
- email_addressesarray of string
The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via
data_profiling_config; rejected on schema monitors.
- destination_idsarray of string
The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- destination_user_idsarray of int64
The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- skip_builtin_dashboardbooleanInput only
Whether to skip creating a default dashboard summarizing data quality metrics.
- warehouse_idstringInput only
The ID of the SQL warehouse used to create the dashboard. If not specified, the first running warehouse the caller can access is used.
Response
Returns the Monitor object.
Update a monitor GA
PATCH
Update a data quality monitor on a Unity Catalog object. Only the fields named in update_mask are changed.
For the table object_type, the caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the table's parent catalog.
- USE_CATALOG on the table's parent catalog, and MANAGE and USE_SCHEMA on the table's parent schema.
- USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and MANAGE on the table.
For the schema object_type, the caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the schema's parent catalog.
- USE_CATALOG on the schema's parent catalog, and MANAGE and USE_SCHEMA on the schema.
API scopes: dataquality
Parameters
- object_typestringRequiredpath
The type of the monitored object. Can be one of the following:
schemaortable.
- object_idstringRequiredpath
The UUID of the request object. It is
schema_idforschema, andtable_idfortable.Find the
schema_idfrom either:- The schema_id of the
Schemasresource. - In Catalog Explorer > select the
schema> go to theDetailstab > theSchema IDfield.
Find the
table_idfrom either:- The table_id of the
Tablesresource. - In Catalog Explorer > select the
table> go to theDetailstab > theTable IDfield.
- The schema_id of the
- update_maskstringRequiredquery
The field mask to specify which fields to update as a comma-separated list. Example value:
data_profiling_config.custom_metrics,data_profiling_config.schedule.quartz_cron_expression
Request body
The new values for the monitor. Only the fields named in update_mask are applied.
- object_typestringRequired
The type of the monitored object. Can be one of the following:
schemaortable.
- object_idstringRequired
The UUID of the request object. It is
schema_idforschema, andtable_idfortable.Find the
schema_idfrom either:- The schema_id of the
Schemasresource. - In Catalog Explorer > select the
schema> go to theDetailstab > theSchema IDfield.
Find the
table_idfrom either:- The table_id of the
Tablesresource. - In Catalog Explorer > select the
table> go to theDetailstab > theTable IDfield.
- The schema_id of the
- anomaly_detection_configobject
Anomaly detection settings. Required when
object_typeisschema. When reading atablemonitor, this holds the settings of the parent schema's monitor if the schema has one.Show child attributesHide child attributes
- excluded_table_full_namesarray of stringBeta
List of fully qualified table names to exclude from anomaly detection. Format:
catalog.schema.table_name.
- notification_settingsobject
Who to alert when anomaly detection finds unhealthy tables in this schema. Applicable to
schemamonitors only. This is the same schema alert you manage from Manage alerts in the Data Quality Monitoring UI, so a change made in either place shows in both.Set the recipients in
on_unhealthy_table;on_failureis not supported here. Onlydestination_idsanddestination_user_idsare supported.On update, an update mask of
*,anomaly_detection_config, oranomaly_detection_config.notification_settingsreplaces the alert's recipients with the ones in the request; omitting the field, or leaving both recipient lists empty, removes the alert. Other masks leave the alert unchanged.Show child attributesHide child attributes
- on_failureobject
Recipients to notify when a refresh of the monitor fails or times out. Only supported on table monitors, via
data_profiling_config, and onlyemail_addressesis accepted there.Show child attributesHide child attributes
- email_addressesarray of string
The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via
data_profiling_config; rejected on schema monitors.
- destination_idsarray of string
The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- destination_user_idsarray of int64
The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- on_unhealthy_tableobject
Recipients to notify when anomaly detection finds unhealthy tables in the schema. Each recipient gets one notification per important unhealthy table. Only supported on schema monitors, via
anomaly_detection_config.Show child attributesHide child attributes
- email_addressesarray of string
The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via
data_profiling_config; rejected on schema monitors.
- destination_idsarray of string
The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- destination_user_idsarray of int64
The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- data_profiling_configobject
Data profiling settings. Required when
object_typeistable. Exactly one ofinference_log,time_seriesorsnapshotmust be set.Show child attributesHide child attributes
- output_schema_idstringRequired
The UUID of the Unity Catalog schema where the profile and drift metrics tables are created.
- assets_dirstringImmutable
The absolute workspace path of the directory where the monitor's assets, such as the dashboard, are stored. The UI and the Python APIs normally prepopulate a default user location.
- inference_logobject
Profiles an inference log table, which stores the requests and predictions of a model. Metrics are computed per time window and include model quality metrics.
Show child attributesHide child attributes
- problem_typestringRequired
The type of problem the model solves. It determines which model quality metrics are computed.
INFERENCE_PROBLEM_TYPE_CLASSIFICATIONINFERENCE_PROBLEM_TYPE_REGRESSION
- timestamp_columnstringRequired
Column containing the timestamp of each inference request. Its type must be TIMESTAMP or convertible to a timestamp.
- granularitiesarray of stringRequired
List of granularities to use when aggregating data into time windows based on their timestamp.
AGGREGATION_GRANULARITY_5_MINUTESAGGREGATION_GRANULARITY_30_MINUTESAGGREGATION_GRANULARITY_1_HOURAGGREGATION_GRANULARITY_1_DAYAGGREGATION_GRANULARITY_1_WEEKAGGREGATION_GRANULARITY_2_WEEKSAGGREGATION_GRANULARITY_3_WEEKSAGGREGATION_GRANULARITY_4_WEEKSAGGREGATION_GRANULARITY_1_MONTHAGGREGATION_GRANULARITY_1_YEAR
- prediction_columnstringRequired
Column containing the model's predicted values.
- label_columnstring
Column containing the ground truth for the model's predictions.
- model_id_columnstringRequired
Column containing the ID of the model that produced each prediction. A slice is created automatically for each distinct model ID.
- time_seriesobject
Profiles a time series table. Metrics are computed per time window, as defined by
timestamp_columnandgranularities.Show child attributesHide child attributes
- timestamp_columnstringRequired
Column containing the timestamp of each row. Its type must be TIMESTAMP or convertible to a timestamp.
- granularitiesarray of stringRequired
List of granularities to use when aggregating data into time windows based on their timestamp.
AGGREGATION_GRANULARITY_5_MINUTESAGGREGATION_GRANULARITY_30_MINUTESAGGREGATION_GRANULARITY_1_HOURAGGREGATION_GRANULARITY_1_DAYAGGREGATION_GRANULARITY_1_WEEKAGGREGATION_GRANULARITY_2_WEEKSAGGREGATION_GRANULARITY_3_WEEKSAGGREGATION_GRANULARITY_4_WEEKSAGGREGATION_GRANULARITY_1_MONTHAGGREGATION_GRANULARITY_1_YEAR
- snapshotobject
Profiles the full contents of the table. Metrics are computed over all rows at each refresh.
- slicing_exprsarray of string
List of column expressions to slice data with for targeted analysis. The data is grouped by each expression independently, resulting in a separate slice for each predicate and its complements. For example
slicing_exprs=["col_1", "col_2 > 10"]will generate the following slices: two slices forcol_2 > 10(True and False), and one slice per unique value incol_1. For high-cardinality columns, only the top 100 unique values by frequency will generate slices.
- custom_metricsarray of object
Custom metrics to compute in addition to the built-in ones. See custom metrics.
Show child attributesHide child attributes
- namestringRequired
Name of the metric in the output tables.
- definitionstringRequired
Jinja template for a SQL expression that specifies how to compute the metric. See create metric definition.
- input_columnsarray of stringRequired
A list of column names in the input table the metric should be computed for. Can use
":table"to indicate that the metric needs information from multiple columns.
- output_data_typestringRequired
Specifies the Spark SQL data type of the metric value, either as a plain type name or as a JSON-encoded StructField.
- typestringRequired
The type of the custom metric.
DATA_PROFILING_CUSTOM_METRIC_TYPE_AGGREGATEmetrics depend only on the columns of the input table,DATA_PROFILING_CUSTOM_METRIC_TYPE_DERIVEDmetrics depend only on previously computed aggregate metrics, andDATA_PROFILING_CUSTOM_METRIC_TYPE_DRIFTmetrics depend on previously computed aggregate or derived metrics.DATA_PROFILING_CUSTOM_METRIC_TYPE_AGGREGATEDATA_PROFILING_CUSTOM_METRIC_TYPE_DERIVEDDATA_PROFILING_CUSTOM_METRIC_TYPE_DRIFT
- baseline_table_namestring
The fully qualified name of a baseline table. Format:
catalog.schema.table_name. Drift metrics compare the monitored table against the baseline data, so the two tables must have the same schema.
- scheduleobject
The schedule on which the monitor refreshes its metrics. Refreshes can also be triggered on demand through the Refresh resource.
Show child attributesHide child attributes
- quartz_cron_expressionstringRequired
The expression that determines when to run the monitor. See examples.
- timezone_idstringRequired
The Java timezone ID in which the cron expression is evaluated. See Java TimeZone for the supported IDs.
- notification_settingsobject
Who to notify about this monitor's refreshes. Only
on_failurewithemail_addressesis supported here.Show child attributesHide child attributes
- on_failureobject
Recipients to notify when a refresh of the monitor fails or times out. Only supported on table monitors, via
data_profiling_config, and onlyemail_addressesis accepted there.Show child attributesHide child attributes
- email_addressesarray of string
The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via
data_profiling_config; rejected on schema monitors.
- destination_idsarray of string
The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- destination_user_idsarray of int64
The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- on_unhealthy_tableobject
Recipients to notify when anomaly detection finds unhealthy tables in the schema. Each recipient gets one notification per important unhealthy table. Only supported on schema monitors, via
anomaly_detection_config.Show child attributesHide child attributes
- email_addressesarray of string
The list of email addresses to send the notification to. A maximum of 5 email addresses is supported. Only supported on table monitors, via
data_profiling_config; rejected on schema monitors.
- destination_idsarray of string
The notification-destination IDs (Slack / Teams / webhook / Asana / ServiceNow / AWS EventBridge) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- destination_user_idsarray of int64
The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via
anomaly_detection_config; rejected on table monitors.
- skip_builtin_dashboardbooleanInput only
Whether to skip creating a default dashboard summarizing data quality metrics.
- warehouse_idstringInput only
The ID of the SQL warehouse used to create the dashboard. If not specified, the first running warehouse the caller can access is used.
Response
Returns the Monitor object.
Delete a monitor GA
DELETE
Delete a data quality monitor on a Unity Catalog object.
For the table object_type, the caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the table's parent catalog.
- USE_CATALOG on the table's parent catalog, and MANAGE and USE_SCHEMA on the table's parent schema.
- USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and MANAGE on the table.
Note that the metric tables and dashboard will not be deleted as part of this call; those assets must be manually cleaned up (if desired).
For the schema object_type, the caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the schema's parent catalog.
- USE_CATALOG on the schema's parent catalog, and MANAGE and USE_SCHEMA on the schema.
API scopes: dataquality
Parameters
- object_typestringRequiredpath
The type of the monitored object. Can be one of the following:
schemaortable.
- object_idstringRequiredpath
The UUID of the request object. It is
schema_idforschema, andtable_idfortable.Find the
schema_idfrom either:- The schema_id of the
Schemasresource. - In Catalog Explorer > select the
schema> go to theDetailstab > theSchema IDfield.
Find the
table_idfrom either:- The table_id of the
Tablesresource. - In Catalog Explorer > select the
table> go to theDetailstab > theTable IDfield.
- The schema_id of the