data-quality command group
This information applies to Databricks CLI versions 0.205 and above. The Databricks CLI is in Public Preview.
Databricks CLI use is subject to the Databricks License and Databricks Privacy Notice, including any Usage Data provisions.
The data-quality command group within the Databricks CLI contains commands to manage the data quality of Unity Catalog objects.
databricks data-quality cancel-refresh
Cancel a data quality monitor refresh. Currently only supported for the table object_type. The call must be made in the same workspace as where the monitor was created.
The caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the table's parent catalog.
- USE_CATALOG on the table's parent catalog, and MANAGE and USE_SCHEMA on the table's parent schema.
- USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and MANAGE on the table.
databricks data-quality cancel-refresh OBJECT_TYPE OBJECT_ID REFRESH_ID [flags]
Arguments
OBJECT_TYPE
The type of the monitored object. Can be one of the following: schema or table.
OBJECT_ID
The UUID of the request object. It is schema_id for schema, and table_id for table.
Find the schema_id from either: (1) The schema_id of the Schemas resource. (2) In Catalog Explorer → select the schema → go to the Details tab → the Schema ID field.
Find the table_id from either: (1) The table_id of the Tables resource. (2) In Catalog Explorer → select the table → go to the Details tab → the Table ID field.
REFRESH_ID
Unique id of the refresh operation.
Options
Examples
The following example cancels a refresh operation:
databricks data-quality cancel-refresh table a1b2c3d4-e5f6-7890-a1b2-c3d4e5f67890 refresh-12345
databricks data-quality create-monitor
Create a data quality monitor on a Unity Catalog object. The caller must provide either anomaly_detection_config for a schema monitor or data_profiling_config for a table monitor.
For the table object_type, the caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and SELECT on the table.
- USE_CATALOG on the table's parent catalog, MANAGE and USE_SCHEMA on the table's parent schema, and SELECT on the table.
- USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and MANAGE and SELECT on the table.
Workspace assets, such as the dashboard, will be created in the workspace where this call was made.
For the schema object_type, the caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the schema's parent catalog.
- USE_CATALOG on the schema's parent catalog, and MANAGE and USE_SCHEMA on the schema.
databricks data-quality create-monitor OBJECT_TYPE OBJECT_ID [flags]
Arguments
OBJECT_TYPE
The type of the monitored object. Can be one of the following: schema or table.
OBJECT_ID
The UUID of the request object. It is schema_id for schema, and table_id for table.
Find the schema_id from either: (1) The schema_id of the Schemas resource. (2) In Catalog Explorer → select the schema → go to the Details tab → the Schema ID field.
Find the table_id from either: (1) The table_id of the Tables resource. (2) In Catalog Explorer → select the table → go to the Details tab → the Table ID field.
Options
--json JSON
The inline JSON string or the @path to the JSON file with the request body
Examples
The following example creates a data quality monitor for a table:
databricks data-quality create-monitor table a1b2c3d4-e5f6-7890-a1b2-c3d4e5f67890 --json '{"data_profiling_config": {"enabled": true}}'
The following example creates a monitor using a JSON file:
databricks data-quality create-monitor table a1b2c3d4-e5f6-7890-a1b2-c3d4e5f67890 --json @monitor-config.json
databricks data-quality create-refresh
Create a refresh. The call must be made in the same workspace as where the monitor was created.
The caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the table's parent catalog.
- USE_CATALOG on the table's parent catalog, and MANAGE and USE_SCHEMA on the table's parent schema.
- USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and MANAGE on the table.
databricks data-quality create-refresh OBJECT_TYPE OBJECT_ID [flags]
Arguments
OBJECT_TYPE
The type of the monitored object. Can be one of the following: schema or table.
OBJECT_ID
The UUID of the request object. It is schema_id for schema, and table_id for table.
Find the schema_id from either: (1) The schema_id of the Schemas resource. (2) In Catalog Explorer → select the schema → go to the Details tab → the Schema ID field.
Find the table_id from either: (1) The table_id of the Tables resource. (2) In Catalog Explorer → select the table → go to the Details tab → the Table ID field.
Options
--json JSON
The inline JSON string or the @path to the JSON file with the request body
Examples
The following example creates a refresh for a table monitor:
databricks data-quality create-refresh table a1b2c3d4-e5f6-7890-a1b2-c3d4e5f67890
The following example creates a refresh using JSON:
databricks data-quality create-refresh table a1b2c3d4-e5f6-7890-a1b2-c3d4e5f67890 --json '{}'
databricks data-quality delete-monitor
Delete a data quality monitor on Unity Catalog object.
For the table object_type, the caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the table's parent catalog.
- USE_CATALOG on the table's parent catalog, and MANAGE and USE_SCHEMA on the table's parent schema.
- USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and MANAGE on the table.
The metric tables and dashboard will not be deleted as part of this call; those assets must be manually cleaned up (if desired).
For the schema object_type, the caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the schema's parent catalog.
- USE_CATALOG on the schema's parent catalog, and MANAGE and USE_SCHEMA on the schema.
databricks data-quality delete-monitor OBJECT_TYPE OBJECT_ID [flags]
Arguments
OBJECT_TYPE
The type of the monitored object. Can be one of the following: schema or table.
OBJECT_ID
The UUID of the request object. It is schema_id for schema, and table_id for table.
Find the schema_id from either: (1) The schema_id of the Schemas resource. (2) In Catalog Explorer → select the schema → go to the Details tab → the Schema ID field.
Find the table_id from either: (1) The table_id of the Tables resource. (2) In Catalog Explorer → select the table → go to the Details tab → the Table ID field.
Options
Examples
The following example deletes a data quality monitor:
databricks data-quality delete-monitor table a1b2c3d4-e5f6-7890-a1b2-c3d4e5f67890
databricks data-quality get-monitor
Read a data quality monitor on a Unity Catalog object.
For the table object_type, the caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the table's parent catalog.
- USE_CATALOG on the table's parent catalog, and MANAGE and USE_SCHEMA on the table's parent schema.
- USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and SELECT on the table.
For the schema object_type, the caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the schema's parent catalog.
- USE_CATALOG on the schema's parent catalog, and USE_SCHEMA on the schema.
The returned information includes configuration values on the entity and parent entity as well as information on assets created by the monitor. Some information (e.g. dashboard) may be filtered out if the caller is in a different workspace than where the monitor was created.
databricks data-quality get-monitor OBJECT_TYPE OBJECT_ID [flags]
Arguments
OBJECT_TYPE
The type of the monitored object. Can be one of the following: schema or table.
OBJECT_ID
The UUID of the request object. It is schema_id for schema, and table_id for table.
Find the schema_id from either: (1) The schema_id of the Schemas resource. (2) In Catalog Explorer → select the schema → go to the Details tab → the Schema ID field.
Find the table_id from either: (1) The table_id of the Tables resource. (2) In Catalog Explorer → select the table → go to the Details tab → the Table ID field.
Options
Examples
The following example gets information about a data quality monitor:
databricks data-quality get-monitor table a1b2c3d4-e5f6-7890-a1b2-c3d4e5f67890
databricks data-quality get-refresh
Get data quality monitor refresh information. The call must be made in the same workspace as where the monitor was created.
For the table object_type, the caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the table's parent catalog.
- USE_CATALOG on the table's parent catalog, and MANAGE and USE_SCHEMA on the table's parent schema.
- USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and SELECT on the table.
For the schema object_type, the caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the schema's parent catalog.
- USE_CATALOG on the schema's parent catalog, and USE_SCHEMA on the schema.
databricks data-quality get-refresh OBJECT_TYPE OBJECT_ID REFRESH_ID [flags]
Arguments
OBJECT_TYPE
The type of the monitored object. Can be one of the following: schema or table.
OBJECT_ID
The UUID of the request object. It is schema_id for schema, and table_id for table.
Find the schema_id from either: (1) The schema_id of the Schemas resource. (2) In Catalog Explorer → select the schema → go to the Details tab → the Schema ID field.
Find the table_id from either: (1) The table_id of the Tables resource. (2) In Catalog Explorer → select the table → go to the Details tab → the Table ID field.
REFRESH_ID
Unique id of the refresh operation.
Options
Examples
The following example gets information about a refresh:
databricks data-quality get-refresh table a1b2c3d4-e5f6-7890-a1b2-c3d4e5f67890 refresh-12345
databricks data-quality list-refresh
List data quality monitor refreshes. The call must be made in the same workspace as where the monitor was created.
For the table object_type, the caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the table's parent catalog.
- USE_CATALOG on the table's parent catalog, and MANAGE and USE_SCHEMA on the table's parent schema.
- USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and SELECT on the table.
For the schema object_type, the caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the schema's parent catalog.
- USE_CATALOG on the schema's parent catalog, and USE_SCHEMA on the schema.
databricks data-quality list-refresh OBJECT_TYPE OBJECT_ID [flags]
Arguments
OBJECT_TYPE
The type of the monitored object. Can be one of the following: schema or table.
OBJECT_ID
The UUID of the request object. It is schema_id for schema, and table_id for table.
Find the schema_id from either: (1) The schema_id of the Schemas resource. (2) In Catalog Explorer → select the schema → go to the Details tab → the Schema ID field.
Find the table_id from either: (1) The table_id of the Tables resource. (2) In Catalog Explorer → select the table → go to the Details tab → the Table ID field.
Options
--page-size int
Maximum number of refreshes to return per page.
--page-token string
Token to retrieve the next page of results.
Examples
The following example lists all refreshes for a monitor:
databricks data-quality list-refresh table a1b2c3d4-e5f6-7890-a1b2-c3d4e5f67890
The following example lists refreshes with pagination:
databricks data-quality list-refresh table a1b2c3d4-e5f6-7890-a1b2-c3d4e5f67890 --page-size 10
databricks data-quality update-monitor
Update a data quality monitor on Unity Catalog object.
For the table object_type, the caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the table's parent catalog.
- USE_CATALOG on the table's parent catalog, and MANAGE and USE_SCHEMA on the table's parent schema.
- USE_CATALOG on the table's parent catalog, USE_SCHEMA on the table's parent schema, and MANAGE on the table.
For the schema object_type, the caller must have either of the following sets of permissions:
- MANAGE and USE_CATALOG on the schema's parent catalog.
- USE_CATALOG on the schema's parent catalog, and MANAGE and USE_SCHEMA on the schema.
databricks data-quality update-monitor OBJECT_TYPE OBJECT_ID UPDATE_MASK OBJECT_TYPE OBJECT_ID [flags]
Arguments
OBJECT_TYPE
The type of the monitored object. Can be one of the following: schema or table.
OBJECT_ID
The UUID of the request object. It is schema_id for schema, and table_id for table.
UPDATE_MASK
The field mask to specify which fields to update as a comma-separated list. Example value: data_profiling_config.custom_metrics,data_profiling_config.schedule.quartz_cron_expression.
Options
--json JSON
The inline JSON string or the @path to the JSON file with the request body
Examples
The following example updates a monitor's configuration:
databricks data-quality update-monitor table a1b2c3d4-e5f6-7890-a1b2-c3d4e5f67890 "data_profiling_config.schedule.quartz_cron_expression" table a1b2c3d4-e5f6-7890-a1b2-c3d4e5f67890 --json '{"data_profiling_config": {"schedule": {"quartz_cron_expression": "0 0 12 * * ?"}}}'
Global flags
--debug
Whether to enable debug logging.
-h or --help
Display help for the Databricks CLI or the related command group or the related command.
--log-file string
A string representing the file to write output logs to. If this flag is not specified then the default is to write output logs to stderr.
--log-format format
The log format type, text or json. The default value is text.
--log-level string
A string representing the log format level. If not specified then the log format level is disabled.
-o, --output type
The command output type, text or json. The default value is text.
-p, --profile string
The name of the profile in the ~/.databrickscfg file to use to run the command. If this flag is not specified then if it exists, the profile named DEFAULT is used.
--progress-format format
The format to display progress logs: default, append, inplace, or json
-t, --target string
If applicable, the bundle target to use