Paginate Unity Catalog REST API operations
Pagination bounds response sizes, making requests more predictable and reducing failures caused by increasingly large responses as your metastore grows. Hence, unpaginated requests to the Unity Catalog REST API operations listed on this page will soon be unsupported. These operations will remain available in paginated form. For more information, see Unpaginated requests to some Unity Catalog REST API operations will soon be rejected.
This page describes how to find and update your clients that call these operations.
Operations that require pagination
The following operations use their paginated form when max_results is specified. Set max_results=0 to let the server choose the page size, or set a positive value to request a page size:
Operation | API reference |
|---|---|
List metastores | |
List external locations | |
List connections | |
List catalogs | |
Get workspace bindings | |
List schemas | |
List tables | |
List functions | |
Get permissions | |
Get effective permissions | |
List system schemas | |
List storage credentials |
To find clients that call these operations without pagination, see Find unpaginated callers in the audit log.
Other Unity Catalog list operations, such as List volumes and List table summaries, already return pages when you omit max_results. Operations that use a page_size parameter, such as List permissions, also return pages by default. Your client must still follow with page tokens for these operations to retrieve every result.
Use newer operations where available
For new integrations, Databricks recommends using the following newer operations. Existing clients can continue using the non-deprecated operations in paginated form. Get catalog workspace bindings is deprecated and requires migration. When switching operations, update your client to account for differences in URLs, request parameters, or response fields.
Older operation | Newer operation | What to do |
|---|---|---|
List storage credentials | Use List credentials for new integrations. Existing clients can continue to call List storage credentials with | |
Get catalog workspace bindings | Migrate. Get catalog workspace bindings is deprecated. Call Get workspace bindings with | |
Get permissions | Use List permissions for new integrations. Existing clients can continue to call Get permissions with | |
Get effective permissions | Use List effective permissions for new integrations. Existing clients can continue to call Get effective permissions with |
Use a current SDK or CLI version
The Databricks SDKs and Databricks CLI list commands paginate for you. Use the latest supported release of the Python, Java, or Go SDK, and the latest generally available release of the Databricks CLI.
Older SDK and CLI versions might not paginate for every Unity Catalog list operation. If your integration pins a version, verify that its list command or SDK iterator retrieves all pages, and upgrade if it returns only the first page.
Direct REST calls, including calls made with databricks api get, don't paginate automatically. Your client must follow with next_page_token as described in the next section.
Fetch all pages with a direct REST client
To retrieve a complete result set from an affected operation:
- On the first request, set
max_results=0and omitpage_token. Settingmax_results=0lets the server choose the page size. - Read
next_page_tokenfrom the response. On the next request, pass its value aspage_tokenand keep all other request parameters the same. - Repeat until a response omits
next_page_tokenor returns it as an empty value. Keep requesting pages even if a page contains no results.
The response field that contains the results varies by operation. Check the API reference for each operation.
The following example lists all tables in a schema. It reads the workspace URL and a personal access token from the DATABRICKS_HOST and DATABRICKS_TOKEN environment variables:
import os
import requests
def list_tables(catalog_name, schema_name):
url = f"{os.environ['DATABRICKS_HOST'].rstrip('/')}/api/2.1/unity-catalog/tables"
headers = {"Authorization": f"Bearer {os.environ['DATABRICKS_TOKEN']}"}
params = {
"catalog_name": catalog_name,
"schema_name": schema_name,
"max_results": 0, # Let the server choose the page size.
}
tables = []
while True:
response = requests.get(url, headers=headers, params=params, timeout=30)
response.raise_for_status()
body = response.json()
tables.extend(body.get("tables", []))
# Stop when the response no longer includes a page token.
page_token = body.get("next_page_token")
if not page_token:
return tables
params["page_token"] = page_token
For more REST API guidance, including how to handle HTTP 429 responses and omit response fields you don't need, see Best practices.
Find unpaginated callers in the audit log
To find clients that call the affected operations without pagination, query the audit log system table for requests that omit max_results. The following query checks the last 30 days and groups requests by operation and caller. To identify and contact the owner of each client, use the OAuth app name when available, the user agent, and the sample request ID.
WITH unpaginated_calls AS (
SELECT
action_name,
COALESCE(
NULLIF(request_params['workspace_id'], '0'),
NULLIF(CAST(workspace_id AS STRING), '0'),
'account-level or unknown'
) AS originating_workspace,
COALESCE(
user_identity.email,
user_identity.subject_name,
'unknown'
) AS caller,
identity_metadata.acting_resource_display_name AS oauth_app,
user_agent,
source_ip_address,
event_time,
request_id,
request_params
FROM system.access.audit
WHERE event_date >= date_sub(current_date(), 30)
AND service_name = 'unityCatalog'
AND action_name IN (
'listMetastores',
'listExternalLocations',
'listConnections',
'listCatalogs',
'GetWorkspaceBindings',
'listSchemas',
'listTables',
'listFunctions',
'getPermissions',
'getEffectivePermissions',
'listSystemSchemas',
'listStorageCredentials'
)
AND request_params['max_results'] IS NULL
)
SELECT
action_name,
originating_workspace,
caller,
oauth_app,
user_agent,
COUNT(*) AS unpaginated_requests,
MIN(event_time) AS first_seen,
MAX(event_time) AS last_seen,
MAX_BY(source_ip_address, event_time) AS sample_source_ip,
MAX_BY(request_id, event_time) AS sample_request_id
FROM unpaginated_calls
GROUP BY
action_name,
originating_workspace,
caller,
oauth_app,
user_agent
ORDER BY unpaginated_requests DESC;
If the query returns no rows, extend the time range to catch jobs that run infrequently. Most audit logs are available only in the region of the workspace, so run the query in each region that contains your workspaces. An empty result doesn't guarantee that all of your clients paginate.