Unity Catalog credential vending for external system access

Preview

This feature is in Public Preview.

This article describes how Unity Catalog credential vending functionality supports access to data in Databricks from external processing engines.

Credential vending supports external systems that connect to Unity Catalog using the Unity REST API and Iceberg REST catalog. See Read Databricks tables from Delta clients and Access Databricks data using external systems.

What is Unity Catalog credential vending?

Credential vending grants short-lived credentials using the Unity Catalog REST API. The granted credentials inherit the privileges of the Databricks principal used to configure the integration. These credentials grant external clients access to data registered in your Unity Catalog metastore.

Note

Unity Catalog credential vending supports read-only access to Unity Catalog data. Some clients support access to tables backed by Delta Lake, while others require you enable Iceberg reads (UniForm) on tables. See Enable Iceberg reads on Delta tables (UniForm).

In order to be granted a temporary credential, the Databricks principal (user, group, or service principal) that makes the request must have the EXTERNAL USE SCHEMA privilege on the schema that contains the table that they need to access from the external engine. The Unity Catalog metastore that contains the schema must also be enabled explicitly for external access. See Enable external data access to Unity Catalog.

Credentials include a short-lived access token string and cloud storage location URL that the external engine can use to access table data and metadata from the cloud storage location.

Requirements

  • You must configure external access on the metastore and grant EXTERNAL USE SCHEMA to the principal configuring the connection. See Enable external data access to Unity Catalog.

  • To access the Databricks workspace using Unity Catalog Open APIs or Iceberg REST APIs, the workspace URL must be accessible to the engine performing the request. This includes workspaces that use IP access lists or AWS Private Link.

  • To access the underlying cloud storage location for Unity Catalog-registered data objects, the storage URLs generated by the Unity Catalog temporary credentials API must be accessible to the engine performing the request. This means that the engine must be allowed on the firewall and network access control lists for the underlying cloud storage accounts.

Request a temporary credential for external data access

Support for credential vending varies by external client. Where supported, the client should automatically leverage vended credentials when a connection is configured.

This section provides an example of explicitly calling the credential vending API endpoint. Some external clients might require you to explicitly set configurations to access data and metadata in cloud object storage backing your Unity Catalog tables. You can use values returned by credential vending to configure access.

Note

You can retrieve a list of tables that support credential vending by invoking the ListTables API with the include_manifest_capabilities option enabled. Only tables marked HAS_DIRECT_EXTERNAL_ENGINE_READ_SUPPORT or HAS_DIRECT_EXTERNAL_ENGINE_WRITE_SUPPORT are eligible for reference in the temporary-table-credentials API. See GET /api/2.1/unity-catalog/tables.

The following curl example explicitly requests a temporary credential for external data access. This request must be completed by a sufficiently privileged workspace principal.

curl -X POST -H "Authentication: Bearer $OAUTH_TOKEN" \
https://<workspace-instance>/api/2.1/unity-catalog/temporary-table-credentials \
-d '{"table_id": "<string>", "operation_name": "<READ|READ_WRITE>"}'

For details, see POST /api/2.1/unity-catalog/temporary-table-credentials in the Databricks REST API reference.

Limitations

The following limitations exist:

  • Not all external clients support credential vending, and support might vary depending on underlying cloud object storage.

  • Only Unity Catalog managed tables and Unity Catalog external tables backed by Delta Lake are supported.

  • The following table types or tables with features enabled are not supported:

    • Tables with row filters or column masks.

    • Tables shared using Delta Sharing.

    • Lakehouse federated tables (foreign tables).

    • Views.

    • Materialized views.

    • Delta Live Tables streaming tables.

    • Online tables.

    • Vector Search indexes.