Monitor

View as Markdown

Monitor for the data quality of unity catalog entities such as schema or table.

GetGET /api/data-quality/v1/monitors/{object_type}/{object_id}ListGET /api/data-quality/v1/monitorsCreatePOST /api/data-quality/v1/monitorsUpdatePATCH /api/data-quality/v1/monitors/{object_type}/{object_id}DeleteDELETE /api/data-quality/v1/monitors/{object_type}/{object_id}

Monitor object

Monitor for the data quality of unity catalog entities such as schema or table.

object_typestringPublic Preview

The type of the monitored object. Can be one of the following: schema or table.

Example: schema

object_idstringPublic Preview

The UUID of the request object. It is schema_id for schema, and table_id for table.

Find the schema_id from either:

1. The schema_id of the Schemas resource.
2. In Catalog Explorer > select the schema > go to the Details tab > the Schema ID field.

Find the table_id from either:

1. The table_id of the Tables resource.
2. In Catalog Explorer > select the table > go to the Details tab > the Table ID field.

anomaly_detection_configobjectPublic Preview

Anomaly Detection Configuration, applicable to schema object types.

Show child attributesHide child attributes

excluded_table_full_namesarray of stringBeta

List of fully qualified table names to exclude from anomaly detection.

data_profiling_configobjectPublic Preview

Data Profiling Configuration, applicable to table object types. Exactly one Analysis Configuration must be present.

Show child attributesHide child attributes

output_schema_idstringPublic Preview

ID of the schema where output tables are created.

assets_dirstringPublic Preview

Field for specifying the absolute path to a custom directory to store data-monitoring assets. Normally prepopulated to a default user location via UI and Python APIs.

inference_logobjectPublic Preview

Analysis Configuration for monitoring inference log tables.

Show child attributesHide child attributes

problem_typestringPublic Preview

Problem type the model aims to solve.

Values: INFERENCE_PROBLEM_TYPE_UNSPECIFIED, INFERENCE_PROBLEM_TYPE_CLASSIFICATION, INFERENCE_PROBLEM_TYPE_REGRESSION

timestamp_columnstringPublic Preview

Column for the timestamp.

granularitiesarray of stringPublic Preview

List of granularities to use when aggregating data into time windows based on their timestamp.

Values: AGGREGATION_GRANULARITY_UNSPECIFIED, AGGREGATION_GRANULARITY_5_MINUTES, AGGREGATION_GRANULARITY_30_MINUTES, AGGREGATION_GRANULARITY_1_HOUR, AGGREGATION_GRANULARITY_1_DAY, AGGREGATION_GRANULARITY_1_WEEK, AGGREGATION_GRANULARITY_2_WEEKS, AGGREGATION_GRANULARITY_3_WEEKS, AGGREGATION_GRANULARITY_4_WEEKS, AGGREGATION_GRANULARITY_1_MONTH, AGGREGATION_GRANULARITY_1_YEAR

prediction_columnstringPublic Preview

Column for the prediction.

label_columnstringPublic Preview

Column for the label.

model_id_columnstringPublic Preview

Column for the model identifier.

time_seriesobjectPublic Preview

Analysis Configuration for monitoring time series tables.

Show child attributesHide child attributes

timestamp_columnstringPublic Preview

Column for the timestamp.

granularitiesarray of stringPublic Preview

List of granularities to use when aggregating data into time windows based on their timestamp.

Values: AGGREGATION_GRANULARITY_UNSPECIFIED, AGGREGATION_GRANULARITY_5_MINUTES, AGGREGATION_GRANULARITY_30_MINUTES, AGGREGATION_GRANULARITY_1_HOUR, AGGREGATION_GRANULARITY_1_DAY, AGGREGATION_GRANULARITY_1_WEEK, AGGREGATION_GRANULARITY_2_WEEKS, AGGREGATION_GRANULARITY_3_WEEKS, AGGREGATION_GRANULARITY_4_WEEKS, AGGREGATION_GRANULARITY_1_MONTH, AGGREGATION_GRANULARITY_1_YEAR

snapshotobjectPublic Preview

Analysis Configuration for monitoring snapshot tables.

slicing_exprsarray of stringPublic Preview

List of column expressions to slice data with for targeted analysis. The data is grouped by each expression independently, resulting in a separate slice for each predicate and its complements. For example slicing_exprs=[“col_1”, “col_2 > 10”] will generate the following slices: two slices for col_2 > 10 (True and False), and one slice per unique value in col1. For high-cardinality columns, only the top 100 unique values by frequency will generate slices.

custom_metricsarray of objectPublic Preview

Custom metrics.

Show child attributesHide child attributes

namestringPublic Preview

Name of the metric in the output tables.

definitionstringPublic Preview

Jinja template for a SQL expression that specifies how to compute the metric. See create metric definition.

input_columnsarray of stringPublic Preview

A list of column names in the input table the metric should be computed for. Can use ":table" to indicate that the metric needs information from multiple columns.

output_data_typestringPublic Preview

The output type of the custom metric.

typestringPublic Preview

The type of the custom metric.

Values: DATA_PROFILING_CUSTOM_METRIC_TYPE_UNSPECIFIED, DATA_PROFILING_CUSTOM_METRIC_TYPE_AGGREGATE, DATA_PROFILING_CUSTOM_METRIC_TYPE_DERIVED, DATA_PROFILING_CUSTOM_METRIC_TYPE_DRIFT

baseline_table_namestringPublic Preview

Baseline table name. Baseline data is used to compute drift from the data in the monitored table_name. The baseline table and the monitored table shall have the same schema.

scheduleobjectPublic Preview

The cron schedule.

Show child attributesHide child attributes

quartz_cron_expressionstringPublic Preview

The expression that determines when to run the monitor. See examples.

timezone_idstringPublic Preview

A Java timezone id. The schedule for a job will be resolved with respect to this timezone. See Java TimeZone <http://docs.oracle.com/javase/7/docs/api/java/util/TimeZone.html>_ for details. The timezone id (e.g., America/Los_Angeles) in which to evaluate the quartz expression.

pause_statusstringPublic Preview

Read only field that indicates whether the schedule is paused or not.

Values: CRON_SCHEDULE_PAUSE_STATUS_UNSPECIFIED, CRON_SCHEDULE_PAUSE_STATUS_UNPAUSED, CRON_SCHEDULE_PAUSE_STATUS_PAUSED

notification_settingsobjectPublic Preview

Field for specifying notification settings.

Show child attributesHide child attributes

on_failureobjectPublic Preview

Destinations to send notifications on failure/timeout.

Show child attributesHide child attributes

email_addressesarray of stringPublic Preview

The list of email addresses to send the notification to. A maximum of 5 email addresses is supported.

skip_builtin_dashboardbooleanPublic Preview

Whether to skip creating a default dashboard summarizing data quality metrics.

warehouse_idstringPublic Preview

Optional argument to specify the warehouse for dashboard creation. If not specified, the first running warehouse will be used.

monitored_table_namestringPublic Preview

Unity Catalog table to monitor. Format: catalog.schema.table_name

statusstringPublic Preview

The data profiling monitor status.

Values: DATA_PROFILING_STATUS_UNSPECIFIED, DATA_PROFILING_STATUS_ACTIVE, DATA_PROFILING_STATUS_PENDING, DATA_PROFILING_STATUS_DELETE_PENDING, DATA_PROFILING_STATUS_ERROR, DATA_PROFILING_STATUS_FAILED

latest_monitor_failure_messagestringPublic Preview

The latest error message for a monitor failure.

profile_metrics_table_namestringPublic Preview

Table that stores profile metrics data. Format: catalog.schema.table_name.

drift_metrics_table_namestringPublic Preview

Table that stores drift metrics data. Format: catalog.schema.table_name.

dashboard_idstringPublic Preview

Id of dashboard that visualizes the computed metrics. This can be empty if the monitor is in PENDING state.

monitor_versionint64Public Preview

Represents the current monitor configuration version in use. The version will be represented in a numeric fashion (1,2,3...). The field has flexibility to take on negative values, which can indicate corrupted monitor_version numbers.

effective_warehouse_idstringPublic Preview

The warehouse for dashboard creation

View as Markdown

The Monitor object

{
  "object_type": "schema",
  "object_id": "string",
  "anomaly_detection_config": {
    "excluded_table_full_names": [
      "string"
    ]
  },
  "data_profiling_config": {
    "output_schema_id": "string",
    "assets_dir": "string",
    "inference_log": {},
    "time_series": {},
    "snapshot": {},
    "slicing_exprs": [
      "string"
    ],
    "custom_metrics": [
      {}
    ],
    "baseline_table_name": "string",
    "schedule": {},
    "notification_settings": {},
    "skip_builtin_dashboard": true,
    "warehouse_id": "string",
    "monitored_table_name": "string",
    "status": "string",
    "latest_monitor_failure_message": "string",
    "profile_metrics_table_name": "string",
    "drift_metrics_table_name": "string",
    "dashboard_id": "string",
    "monitor_version": 0,
    "effective_warehouse_id": "string"
  }
}

Get Public Preview

GET /api/data-quality/v1/monitors/{object_type}/{object_id}

Read a data quality monitor on a Unity Catalog object.

For the table object_type, the caller must have either of the following sets of permissions:

1. MANAGE and USE_CATALOG on the table's parent catalog.
2. USE_CATALOG on the table's parent catalog, and MANAGE and USE_SCHEMA on the table's parent schema.
3. USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and SELECT on the table.

For the schema object_type, the caller must have either of the following sets of permissions:

1. MANAGE and USE_CATALOG on the schema's parent catalog.
2. USE_CATALOG on the schema's parent catalog, and USE_SCHEMA on the schema.

The returned information includes configuration values on the entity and parent entity as well as information on assets created by the monitor. Some information (e.g. dashboard) may be filtered out if the caller is in a different workspace than where the monitor was created.

API scopes: dataquality

Parameters

object_typestringpath

The type of the monitored object. Can be one of the following: schema or table.

object_idstringpath

The UUID of the request object. It is schema_id for schema, and table_id for table.

Find the schema_id from either:

1. The schema_id of the Schemas resource.
2. In Catalog Explorer > select the schema > go to the Details tab > the Schema ID field.

Find the table_id from either:

1. The table_id of the Tables resource.
2. In Catalog Explorer > select the table > go to the Details tab > the Table ID field.

Response

Returns the Monitor object.

View as Markdown

Response

{
  "object_type": "schema",
  "object_id": "string",
  "anomaly_detection_config": {
    "excluded_table_full_names": [
      "string"
    ]
  },
  "data_profiling_config": {
    "output_schema_id": "string",
    "assets_dir": "string",
    "inference_log": {},
    "time_series": {},
    "snapshot": {},
    "slicing_exprs": [
      "string"
    ],
    "custom_metrics": [
      {}
    ],
    "baseline_table_name": "string",
    "schedule": {},
    "notification_settings": {},
    "skip_builtin_dashboard": true,
    "warehouse_id": "string",
    "monitored_table_name": "string",
    "status": "string",
    "latest_monitor_failure_message": "string",
    "profile_metrics_table_name": "string",
    "drift_metrics_table_name": "string",
    "dashboard_id": "string",
    "monitor_version": 0,
    "effective_warehouse_id": "string"
  }
}

List Public Preview

GET /api/data-quality/v1/monitors

(Unimplemented) List data quality monitors.

API scopes: dataquality

Parameters

page_tokenstringquery

page_sizeint32query

Default: 25

Constraints: <= 100

Response

Returns a list of Monitor objects.

View as Markdown

Response

{
  "monitors": [
    {
      "object_type": "schema",
      "object_id": "string",
      "anomaly_detection_config": {},
      "data_profiling_config": {}
    }
  ],
  "next_page_token": "string"
}

Create Public Preview

POST /api/data-quality/v1/monitors

Create a data quality monitor on a Unity Catalog object. The caller must provide either anomaly_detection_config for a schema monitor or data_profiling_config for a table monitor.

For the table object_type, the caller must have either of the following sets of permissions:

1. MANAGE and USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and SELECT on the table
2. USE_CATALOG on the table's parent catalog, MANAGE and USE_SCHEMA on the table's parent schema, and SELECT on the table.
3. USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and MANAGE and SELECT on the table.

Workspace assets, such as the dashboard, will be created in the workspace where this call was made.

For the schema object_type, the caller must have either of the following sets of permissions:

1. MANAGE and USE_CATALOG on the schema's parent catalog.
2. USE_CATALOG on the schema's parent catalog, and MANAGE and USE_SCHEMA on the schema.

API scopes: dataquality

Request body

monitorobject

The monitor to create.

Show child attributesHide child attributes

object_typestring

The type of the monitored object. Can be one of the following: schema or table.

Example: schema

object_idstring

The UUID of the request object. It is schema_id for schema, and table_id for table.

Find the schema_id from either:

1. The schema_id of the Schemas resource.
2. In Catalog Explorer > select the schema > go to the Details tab > the Schema ID field.

Find the table_id from either:

1. The table_id of the Tables resource.
2. In Catalog Explorer > select the table > go to the Details tab > the Table ID field.

anomaly_detection_configobject

Anomaly Detection Configuration, applicable to schema object types.

Show child attributesHide child attributes

excluded_table_full_namesarray of stringBeta

List of fully qualified table names to exclude from anomaly detection.

data_profiling_configobject

Data Profiling Configuration, applicable to table object types. Exactly one Analysis Configuration must be present.

Show child attributesHide child attributes

output_schema_idstring

ID of the schema where output tables are created.

assets_dirstring

Field for specifying the absolute path to a custom directory to store data-monitoring assets. Normally prepopulated to a default user location via UI and Python APIs.

inference_logobjectRequired

Analysis Configuration for monitoring inference log tables.

Show child attributesHide child attributes

problem_typestring

Problem type the model aims to solve.

Values: INFERENCE_PROBLEM_TYPE_UNSPECIFIED, INFERENCE_PROBLEM_TYPE_CLASSIFICATION, INFERENCE_PROBLEM_TYPE_REGRESSION

timestamp_columnstring

Column for the timestamp.

granularitiesarray of string

List of granularities to use when aggregating data into time windows based on their timestamp.

Values: AGGREGATION_GRANULARITY_UNSPECIFIED, AGGREGATION_GRANULARITY_5_MINUTES, AGGREGATION_GRANULARITY_30_MINUTES, AGGREGATION_GRANULARITY_1_HOUR, AGGREGATION_GRANULARITY_1_DAY, AGGREGATION_GRANULARITY_1_WEEK, AGGREGATION_GRANULARITY_2_WEEKS, AGGREGATION_GRANULARITY_3_WEEKS, AGGREGATION_GRANULARITY_4_WEEKS, AGGREGATION_GRANULARITY_1_MONTH, AGGREGATION_GRANULARITY_1_YEAR

prediction_columnstring

Column for the prediction.

label_columnstring

Column for the label.

model_id_columnstring

Column for the model identifier.

time_seriesobjectRequired

Analysis Configuration for monitoring time series tables.

Show child attributesHide child attributes

timestamp_columnstring

Column for the timestamp.

granularitiesarray of string

List of granularities to use when aggregating data into time windows based on their timestamp.

Values: AGGREGATION_GRANULARITY_UNSPECIFIED, AGGREGATION_GRANULARITY_5_MINUTES, AGGREGATION_GRANULARITY_30_MINUTES, AGGREGATION_GRANULARITY_1_HOUR, AGGREGATION_GRANULARITY_1_DAY, AGGREGATION_GRANULARITY_1_WEEK, AGGREGATION_GRANULARITY_2_WEEKS, AGGREGATION_GRANULARITY_3_WEEKS, AGGREGATION_GRANULARITY_4_WEEKS, AGGREGATION_GRANULARITY_1_MONTH, AGGREGATION_GRANULARITY_1_YEAR

snapshotobjectRequired

Analysis Configuration for monitoring snapshot tables.

slicing_exprsarray of string

List of column expressions to slice data with for targeted analysis. The data is grouped by each expression independently, resulting in a separate slice for each predicate and its complements. For example slicing_exprs=[“col_1”, “col_2 > 10”] will generate the following slices: two slices for col_2 > 10 (True and False), and one slice per unique value in col1. For high-cardinality columns, only the top 100 unique values by frequency will generate slices.

custom_metricsarray of object

Custom metrics.

Show child attributesHide child attributes

namestring

Name of the metric in the output tables.

definitionstring

Jinja template for a SQL expression that specifies how to compute the metric. See create metric definition.

input_columnsarray of string

A list of column names in the input table the metric should be computed for. Can use ":table" to indicate that the metric needs information from multiple columns.

output_data_typestring

The output type of the custom metric.

typestring

The type of the custom metric.

Values: DATA_PROFILING_CUSTOM_METRIC_TYPE_UNSPECIFIED, DATA_PROFILING_CUSTOM_METRIC_TYPE_AGGREGATE, DATA_PROFILING_CUSTOM_METRIC_TYPE_DERIVED, DATA_PROFILING_CUSTOM_METRIC_TYPE_DRIFT

baseline_table_namestring

Baseline table name. Baseline data is used to compute drift from the data in the monitored table_name. The baseline table and the monitored table shall have the same schema.

scheduleobject

The cron schedule.

Show child attributesHide child attributes

quartz_cron_expressionstring

The expression that determines when to run the monitor. See examples.

timezone_idstring

A Java timezone id. The schedule for a job will be resolved with respect to this timezone. See Java TimeZone <http://docs.oracle.com/javase/7/docs/api/java/util/TimeZone.html>_ for details. The timezone id (e.g., America/Los_Angeles) in which to evaluate the quartz expression.

pause_statusstring

Read only field that indicates whether the schedule is paused or not.

Values: CRON_SCHEDULE_PAUSE_STATUS_UNSPECIFIED, CRON_SCHEDULE_PAUSE_STATUS_UNPAUSED, CRON_SCHEDULE_PAUSE_STATUS_PAUSED

notification_settingsobject

Field for specifying notification settings.

Show child attributesHide child attributes

on_failureobject

Destinations to send notifications on failure/timeout.

skip_builtin_dashboardboolean

Whether to skip creating a default dashboard summarizing data quality metrics.

warehouse_idstring

Optional argument to specify the warehouse for dashboard creation. If not specified, the first running warehouse will be used.

monitored_table_namestring

Unity Catalog table to monitor. Format: catalog.schema.table_name

statusstring

The data profiling monitor status.

Values: DATA_PROFILING_STATUS_UNSPECIFIED, DATA_PROFILING_STATUS_ACTIVE, DATA_PROFILING_STATUS_PENDING, DATA_PROFILING_STATUS_DELETE_PENDING, DATA_PROFILING_STATUS_ERROR, DATA_PROFILING_STATUS_FAILED

latest_monitor_failure_messagestring

The latest error message for a monitor failure.

profile_metrics_table_namestring

Table that stores profile metrics data. Format: catalog.schema.table_name.

drift_metrics_table_namestring

Table that stores drift metrics data. Format: catalog.schema.table_name.

dashboard_idstring

Id of dashboard that visualizes the computed metrics. This can be empty if the monitor is in PENDING state.

monitor_versionint64

Represents the current monitor configuration version in use. The version will be represented in a numeric fashion (1,2,3...). The field has flexibility to take on negative values, which can indicate corrupted monitor_version numbers.

effective_warehouse_idstring

The warehouse for dashboard creation

Response

Returns the Monitor object.

View as Markdown

Request

{
  "monitor": {
    "object_type": "schema",
    "object_id": "string",
    "anomaly_detection_config": {},
    "data_profiling_config": {}
  }
}

Response

{
  "object_type": "schema",
  "object_id": "string",
  "anomaly_detection_config": {
    "excluded_table_full_names": [
      "string"
    ]
  },
  "data_profiling_config": {
    "output_schema_id": "string",
    "assets_dir": "string",
    "inference_log": {},
    "time_series": {},
    "snapshot": {},
    "slicing_exprs": [
      "string"
    ],
    "custom_metrics": [
      {}
    ],
    "baseline_table_name": "string",
    "schedule": {},
    "notification_settings": {},
    "skip_builtin_dashboard": true,
    "warehouse_id": "string",
    "monitored_table_name": "string",
    "status": "string",
    "latest_monitor_failure_message": "string",
    "profile_metrics_table_name": "string",
    "drift_metrics_table_name": "string",
    "dashboard_id": "string",
    "monitor_version": 0,
    "effective_warehouse_id": "string"
  }
}

Update Public Preview

PATCH /api/data-quality/v1/monitors/{object_type}/{object_id}

Update a data quality monitor on Unity Catalog object.

For the table object_type, the caller must have either of the following sets of permissions:

1. MANAGE and USE_CATALOG on the table's parent catalog.
2. USE_CATALOG on the table's parent catalog, and MANAGE and USE_SCHEMA on the table's parent schema.
3. USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and MANAGE on the table.

For the schema object_type, the caller must have either of the following sets of permissions:

1. MANAGE and USE_CATALOG on the schema's parent catalog.
2. USE_CATALOG on the schema's parent catalog, and MANAGE and USE_SCHEMA on the schema.

API scopes: dataquality

Parameters

object_typestringpath

The type of the monitored object. Can be one of the following: schema or table.

object_idstringpath

The UUID of the request object. It is schema_id for schema, and table_id for table.

Find the schema_id from either:

1. The schema_id of the Schemas resource.
2. In Catalog Explorer > select the schema > go to the Details tab > the Schema ID field.

Find the table_id from either:

1. The table_id of the Tables resource.
2. In Catalog Explorer > select the table > go to the Details tab > the Table ID field.

update_maskstringquery

The field mask to specify which fields to update as a comma-separated list. Example value: data_profiling_config.custom_metrics,data_profiling_config.schedule.quartz_cron_expression

Request body

monitorobject

The monitor to update.

Show child attributesHide child attributes

object_typestring

The type of the monitored object. Can be one of the following: schema or table.

Example: schema

object_idstring

The UUID of the request object. It is schema_id for schema, and table_id for table.

Find the schema_id from either:

1. The schema_id of the Schemas resource.
2. In Catalog Explorer > select the schema > go to the Details tab > the Schema ID field.

Find the table_id from either:

1. The table_id of the Tables resource.
2. In Catalog Explorer > select the table > go to the Details tab > the Table ID field.

anomaly_detection_configobject

Anomaly Detection Configuration, applicable to schema object types.

Show child attributesHide child attributes

excluded_table_full_namesarray of stringBeta

List of fully qualified table names to exclude from anomaly detection.

data_profiling_configobject

Data Profiling Configuration, applicable to table object types. Exactly one Analysis Configuration must be present.

Show child attributesHide child attributes

output_schema_idstring

ID of the schema where output tables are created.

assets_dirstring

Field for specifying the absolute path to a custom directory to store data-monitoring assets. Normally prepopulated to a default user location via UI and Python APIs.

inference_logobjectRequired

Analysis Configuration for monitoring inference log tables.

Show child attributesHide child attributes

problem_typestring

Problem type the model aims to solve.

Values: INFERENCE_PROBLEM_TYPE_UNSPECIFIED, INFERENCE_PROBLEM_TYPE_CLASSIFICATION, INFERENCE_PROBLEM_TYPE_REGRESSION

timestamp_columnstring

Column for the timestamp.

granularitiesarray of string

List of granularities to use when aggregating data into time windows based on their timestamp.

Values: AGGREGATION_GRANULARITY_UNSPECIFIED, AGGREGATION_GRANULARITY_5_MINUTES, AGGREGATION_GRANULARITY_30_MINUTES, AGGREGATION_GRANULARITY_1_HOUR, AGGREGATION_GRANULARITY_1_DAY, AGGREGATION_GRANULARITY_1_WEEK, AGGREGATION_GRANULARITY_2_WEEKS, AGGREGATION_GRANULARITY_3_WEEKS, AGGREGATION_GRANULARITY_4_WEEKS, AGGREGATION_GRANULARITY_1_MONTH, AGGREGATION_GRANULARITY_1_YEAR

prediction_columnstring

Column for the prediction.

label_columnstring

Column for the label.

model_id_columnstring

Column for the model identifier.

time_seriesobjectRequired

Analysis Configuration for monitoring time series tables.

Show child attributesHide child attributes

timestamp_columnstring

Column for the timestamp.

granularitiesarray of string

List of granularities to use when aggregating data into time windows based on their timestamp.

Values: AGGREGATION_GRANULARITY_UNSPECIFIED, AGGREGATION_GRANULARITY_5_MINUTES, AGGREGATION_GRANULARITY_30_MINUTES, AGGREGATION_GRANULARITY_1_HOUR, AGGREGATION_GRANULARITY_1_DAY, AGGREGATION_GRANULARITY_1_WEEK, AGGREGATION_GRANULARITY_2_WEEKS, AGGREGATION_GRANULARITY_3_WEEKS, AGGREGATION_GRANULARITY_4_WEEKS, AGGREGATION_GRANULARITY_1_MONTH, AGGREGATION_GRANULARITY_1_YEAR

snapshotobjectRequired

Analysis Configuration for monitoring snapshot tables.

slicing_exprsarray of string

List of column expressions to slice data with for targeted analysis. The data is grouped by each expression independently, resulting in a separate slice for each predicate and its complements. For example slicing_exprs=[“col_1”, “col_2 > 10”] will generate the following slices: two slices for col_2 > 10 (True and False), and one slice per unique value in col1. For high-cardinality columns, only the top 100 unique values by frequency will generate slices.

custom_metricsarray of object

Custom metrics.

Show child attributesHide child attributes

namestring

Name of the metric in the output tables.

definitionstring

Jinja template for a SQL expression that specifies how to compute the metric. See create metric definition.

input_columnsarray of string

A list of column names in the input table the metric should be computed for. Can use ":table" to indicate that the metric needs information from multiple columns.

output_data_typestring

The output type of the custom metric.

typestring

The type of the custom metric.

Values: DATA_PROFILING_CUSTOM_METRIC_TYPE_UNSPECIFIED, DATA_PROFILING_CUSTOM_METRIC_TYPE_AGGREGATE, DATA_PROFILING_CUSTOM_METRIC_TYPE_DERIVED, DATA_PROFILING_CUSTOM_METRIC_TYPE_DRIFT

baseline_table_namestring

Baseline table name. Baseline data is used to compute drift from the data in the monitored table_name. The baseline table and the monitored table shall have the same schema.

scheduleobject

The cron schedule.

Show child attributesHide child attributes

quartz_cron_expressionstring

The expression that determines when to run the monitor. See examples.

timezone_idstring

A Java timezone id. The schedule for a job will be resolved with respect to this timezone. See Java TimeZone <http://docs.oracle.com/javase/7/docs/api/java/util/TimeZone.html>_ for details. The timezone id (e.g., America/Los_Angeles) in which to evaluate the quartz expression.

pause_statusstring

Read only field that indicates whether the schedule is paused or not.

Values: CRON_SCHEDULE_PAUSE_STATUS_UNSPECIFIED, CRON_SCHEDULE_PAUSE_STATUS_UNPAUSED, CRON_SCHEDULE_PAUSE_STATUS_PAUSED

notification_settingsobject

Field for specifying notification settings.

Show child attributesHide child attributes

on_failureobject

Destinations to send notifications on failure/timeout.

skip_builtin_dashboardboolean

Whether to skip creating a default dashboard summarizing data quality metrics.

warehouse_idstring

Optional argument to specify the warehouse for dashboard creation. If not specified, the first running warehouse will be used.

monitored_table_namestring

Unity Catalog table to monitor. Format: catalog.schema.table_name

statusstring

The data profiling monitor status.

Values: DATA_PROFILING_STATUS_UNSPECIFIED, DATA_PROFILING_STATUS_ACTIVE, DATA_PROFILING_STATUS_PENDING, DATA_PROFILING_STATUS_DELETE_PENDING, DATA_PROFILING_STATUS_ERROR, DATA_PROFILING_STATUS_FAILED

latest_monitor_failure_messagestring

The latest error message for a monitor failure.

profile_metrics_table_namestring

Table that stores profile metrics data. Format: catalog.schema.table_name.

drift_metrics_table_namestring

Table that stores drift metrics data. Format: catalog.schema.table_name.

dashboard_idstring

Id of dashboard that visualizes the computed metrics. This can be empty if the monitor is in PENDING state.

monitor_versionint64

Represents the current monitor configuration version in use. The version will be represented in a numeric fashion (1,2,3...). The field has flexibility to take on negative values, which can indicate corrupted monitor_version numbers.

effective_warehouse_idstring

The warehouse for dashboard creation

Response

Returns the Monitor object.

View as Markdown

Request

{
  "monitor": {
    "object_type": "schema",
    "object_id": "string",
    "anomaly_detection_config": {},
    "data_profiling_config": {}
  }
}

Response

{
  "object_type": "schema",
  "object_id": "string",
  "anomaly_detection_config": {
    "excluded_table_full_names": [
      "string"
    ]
  },
  "data_profiling_config": {
    "output_schema_id": "string",
    "assets_dir": "string",
    "inference_log": {},
    "time_series": {},
    "snapshot": {},
    "slicing_exprs": [
      "string"
    ],
    "custom_metrics": [
      {}
    ],
    "baseline_table_name": "string",
    "schedule": {},
    "notification_settings": {},
    "skip_builtin_dashboard": true,
    "warehouse_id": "string",
    "monitored_table_name": "string",
    "status": "string",
    "latest_monitor_failure_message": "string",
    "profile_metrics_table_name": "string",
    "drift_metrics_table_name": "string",
    "dashboard_id": "string",
    "monitor_version": 0,
    "effective_warehouse_id": "string"
  }
}

Delete Public Preview

DELETE /api/data-quality/v1/monitors/{object_type}/{object_id}

Delete a data quality monitor on Unity Catalog object.

For the table object_type, the caller must have either of the following sets of permissions: MANAGE and USE_CATALOG on the table's parent catalog. USE_CATALOG on the table's parent catalog, and MANAGE and USE_SCHEMA on the table's parent schema. USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and MANAGE on the table.

Note that the metric tables and dashboard will not be deleted as part of this call; those assets must be manually cleaned up (if desired).

For the schema object_type, the caller must have either of the following sets of permissions:

1. MANAGE and USE_CATALOG on the schema's parent catalog.
2. USE_CATALOG on the schema's parent catalog, and MANAGE and USE_SCHEMA on the schema.

API scopes: dataquality

Parameters

object_typestringpath

The type of the monitored object. Can be one of the following: schema or table.

object_idstringpath

The UUID of the request object. It is schema_id for schema, and table_id for table.

Find the schema_id from either:

1. The schema_id of the Schemas resource.
2. In Catalog Explorer > select the schema > go to the Details tab > the Schema ID field.

Find the table_id from either:

1. The table_id of the Tables resource.
2. In Catalog Explorer > select the table > go to the Details tab > the Table ID field.

View as Markdown

Response

{}