Skip to main content
Last updated on

External secrets in Unity Catalog

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

External secrets connect a Unity Catalog schema to an external secret manager. Instead of Databricks storing the secret values, the secrets stay in your cloud secret manager and appear in Unity Catalog as securable objects that you can govern and reference.

How external secrets work​

You configure the secret backend per schema. By default, Databricks stores and manages every secret in a schema, unless an external secret manager backs it. A schema cannot mix both.

When a schema is backed by an external secret manager, Databricks does the following:

  • Surfaces the secrets the connection can list. Every secret that the connection is allowed to list appears as a Unity Catalog secret in Catalog Explorer, dbutils, and the Unity Catalog REST API. Naming restrictions apply. See Naming restrictions.
  • Reads values on demand. Databricks does not store the secret value in Unity Catalog; each retrieval reads the current value directly from the external secret manager, which adds latency and can incur cloud provider costs. Retrieved values may be cached on the compute that requested them, as with other secrets.
  • Governs access with Unity Catalog privileges. READ SECRET and REFERENCE SECRET, and privilege inheritance, apply the same way they do for Databricks-managed secrets. Because external secrets are read-only, Unity Catalog rejects CREATE SECRET, WRITE SECRET, and delete operations. See Limitations. All access to the external secret manager goes through the connection's service credential, so cloud-side audit logs attribute access to that identity, while Unity Catalog audit logs still record the Databricks user.

Listing a schema's secrets triggers importing. It is not a continuous background sync. The import runs asynchronously, so Unity Catalog reflects a secret added to or removed from the external secret manager only after a subsequent list triggers an import, which can take up to a few minutes for stores with many secrets. Reading a specific secret imports or refreshes just that secret, and each read fetches the latest value. When you permanently delete a secret in the external secret manager, Databricks removes the corresponding Unity Catalog secret, including its metadata, permissions, and tags.

Naming restrictions​

External secret names must comply with Unity Catalog naming conventions to be imported:

  • Unity Catalog does not import names longer than 248 characters.
  • Unity Catalog secret names are case-insensitive. If two external secrets differ only by case, Unity Catalog uses only the first one imported and ignores the rest.
  • Unity Catalog substitutes characters it does not support: . becomes : and / becomes |. Databricks recommends avoiding these characters in external secret names.

Manage secret quotas​

Unity Catalog limits how many secrets a schema and metastore can hold. See Limitations. If a schema has more secrets than the quota allows, Databricks imports the oldest secrets first by creation date in the external secret manager, up to the quota, and does not import the rest. Expose only the secrets you require in Databricks so the schema stays within quota:

Add a tag filter to the connection so Unity Catalog surfaces only tagged secrets. See Filter imported secrets by tag.

Limitations​

In addition to the limitations that apply to all Unity Catalog secrets, externally backed secrets have the following limitations:

  • Read-only in Unity Catalog. You cannot create, update, rename, or delete secrets in an externally backed schema from Databricks. Manage those secrets in the external secret manager.
  • One backend per schema. All secrets in a schema use the same backend. A schema cannot mix Databricks-managed and externally backed secrets.
  • Switching the backend removes imported secrets. Changing a schema's backend or pointing it at a different connection removes the previously imported external secrets from the schema, including their Unity Catalog metadata, permissions, and tags.
  • Cross-cloud access is not supported. Back schemas with the secret manager native to the workspace's cloud.
  • Read latency and cost. Every read fetches the value from the external secret manager, which adds latency and can incur cloud provider costs.
  • Eventually consistent listing. Secrets added to or removed from the external secret manager can take a few minutes to appear in Unity Catalog, especially for large secret stores.
  • Tag filtering is AWS-only. The tag_key and tag_value options apply to AWS Secrets Manager connections.

Additional resources​

On this page