Monitor usage with system tables
Preview
This feature is in Public Preview.
This article explains the concept of system tables in Databricks and highlights resources you can use to get the most out of your system tables data.
What are system tables?
System tables are a Databricks-hosted analytical store of your account’s operational data. System tables can be used for historical observability across your account.
Which system tables are available?
Currently, Databricks hosts system tables for:
Audit logs: Located at
system.access.audit
.Billable usage logs: Located at
system.billing.usage
.Pricing table: Located at
system.billing.list_prices
.Table and column lineage: Both tables located under
system.access
.Marketplace listing access: Located at
system.marketplace.listing_access_events
.
Note
You may see other system tables in your account besides the ones listed above. Those tables are in Private Preview currently and are empty by default. If you are interested in using any of these tables, please reach out to your Databricks account team.
Enable system tables
Since system tables are governed by Unity Catalog, you need to have at least one Unity Catalog-enabled workspace in your account to enable and access system tables. System tables include data from all workspaces in your account but they can only be accessed from a Unity Catalog-enabled workspace.
System tables must be enabled by an account admin. You can enable system tables using the Unity Catalog REST API.
List available system schemas
Use the following curl command to list available system schemas:
curl -v -X GET -H "Authorization: Bearer <PAT Token>" "https://<workspace>.cloud.databricks.com/api/2.0/unity-catalog/metastores/<metastore-id>/systemschemas"
The following is an example output of the GET
command:
{"schemas":[{"schema":"access","state":"<AVAILABLE OR EnableCompleted>"},{"schema":"billing","state":"<AVAILABLE OR EnableCompleted>"},{"schema":"information_schema","state":"<AVAILABLE OR EnableCompleted>"}]}
state: AVAILABLE
: The system schema is available but has not yet been enabled.
state: EnableCompleted
: You have enabled the system schema and it is visible in Catalog Explorer.
Enable a system schema
Use the following curl command to enable a system schema:
curl -v -X PUT -H "Authorization: Bearer <PAT Token>" "https://<workspace.databricks.com/api/2.0/unity-catalog/metastores/<metastore-id>/systemschemas/<SCHEMA_NAME>"
If the system schema is enabled successfully, result code 200
is returned.
If you attempt to re-enable a system schema, the following is returned: "error_code":"SCHEMA_ALREADY_EXISTS","message":"Schema <schema-name> already exists"
.
Grant access to system tables
System tables access is governed by Unity Catalog. By default, no users have access to system tables. To grant access, metastore admins must grant USE
and SELECT
permissions on the select system schemas. See Manage privileges in Unity Catalog.
Note
System tables are read-only and cannot be modified.
Do system tables contain data for all workspaces in your account?
The audit log and lineage tables contain operational data for all workspaces in your account deployed within the same cloud region. The billing system table (system.billing.usage
) contains data for all workspaces in your account, no matter what region they are deployed in.
Even though system tables can only be accessed through a Unity Catalog workspace, the tables also include operational data for non-Unity Catalog workspaces in your account.
Where are the system tables located?
The system tables in your account are located in a catalog called system
, which is included in every Unity Catalog metastore. In the system
catalog you’ll see schemas such as access
and billing
that contain the system tables.
Note
During the system tables Public Preview, Databricks will retain all your system tables data.
Limitations
The system tables public preview currently has the following limitations:
No support for real-time monitoring. Data is updated throughout the day. If you don’t see a log for a recent event, check back later.
Known issues
The system schemas
system.operational_data
andsystem.lineage
might contain empty tables.The system schemas
system.access
andsystem.billing
are in Public Preview and available to all customers. Databricks recommends using Public Preview system schemas only.
If your workspace uses a customer-managed VPC, you might be denied access to the S3 bucket where the logs are stored. If so, you need to update your VPC endpoint policy to allow access to the S3 bucket where your region’s system tables data is stored. For a list of regional bucket names, see the System tables bucket column in Storage bucket addresses table.