Skip to main content
Last updated on

Configure external secrets in Unity Catalog

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

This page shows how to connect a Unity Catalog schema to an external secret manager so its secret values remain in AWS Secrets Manager.

For how external secrets work and their limitations, see External secrets in Unity Catalog.

Before you begin​

  • Meet the requirements for Unity Catalog secrets.
  • The external secrets Beta must be enabled for your workspace.
  • The schema you want to back externally must not contain any Databricks-managed secrets. Delete existing secrets before you switch the backend.
  • You must have a Unity Catalog connection to your external secret manager and a Unity Catalog service credential that the connection uses to authenticate. To create these, you must have USE CONNECTION on an existing connection or CREATE CONNECTION on the metastore to create one, and access to a service credential, or CREATE CREDENTIAL on the metastore to create one.
  • AWS Secrets Manager backing is available only on Databricks on AWS, using an AWS_SECRETS_MANAGER connection. The secret manager must be in the region you configure on the connection.

Set up external secrets​

Create a service credential​

The connection authenticates to your external secret manager with a Unity Catalog service credential.

Create the service credential and its IAM role as described in Create service credentials, including the required trust policy. Attach the following permission policy to the role so it can read your secrets in AWS Secrets Manager. Replace <AWS-ACCOUNT-ID> and <AWS-IAM-ROLE-NAME> with the account ID and name of the role.

JSON
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadAllSecrets",
      "Effect": "Allow",
      "Action": ["secretsmanager:ListSecrets", "secretsmanager:DescribeSecret", "secretsmanager:GetSecretValue"],
      "Resource": "*"
    },
    {
      "Sid": "SelfAssume",
      "Effect": "Allow",
      "Action": ["sts:AssumeRole"],
      "Resource": ["arn:aws:iam::<AWS-ACCOUNT-ID>:role/<AWS-IAM-ROLE-NAME>"]
    }
  ]
}

Create a connection​

Create a Unity Catalog connection that references the service credential from the previous step and points to your external secret manager.

  1. In Databricks, open Catalog Explorer, click the + menu, and select Create a connection.
  2. Enter a Connection name and select the AWS Secrets Manager connection type.
  3. Select the service credential to authenticate with, and set the AWS region of your secret manager.
  4. Click Create.
note

Make sure the service credential's permissions and the connection's tag filter cover the same secrets. If the role can list a secret but cannot read its value, Unity Catalog imports the secret but reads of it fail.

To surface only a subset of your secrets in Unity Catalog, add a tag filter to the connection. See Filter imported secrets by tag.

Back a schema with the connection​

Configure the schema's secret backend in Catalog Explorer.

  1. In Databricks, open Catalog Explorer and go to the schema.

  2. In the schema's details, locate the external secrets manager setting and click Enable.

    The schema must not contain any Databricks-managed secrets. If it does, remove them first. Enable stays disabled until the schema has no secrets.

  3. Select the connection you created in the previous step, then confirm.

The schema is now backed by your external secret manager, and its secrets appear in Unity Catalog.

To point the schema at a different connection, use Edit. To return the schema to Databricks-managed storage, edit the schema and disable external secrets.

Setting or changing a schema's secret backend requires USE CATALOG on the parent catalog, ownership of the schema or MANAGE on it, and USE CONNECTION on the connection.

Read external secrets​

After the schema is backed externally, its secrets appear in Unity Catalog and you read them like any other Unity Catalog secret. Listing a schema triggers an import, so a newly added secret appears only after the next list. The read commands are the same on both clouds.

dbutils applies secret redaction and is the recommended way to read a value. Requires Databricks Runtime 17.3 LTS or above, or serverless environment version 4 or above.

Python
# List the secrets in the schema
all_secrets = dbutils.secrets.list(catalog="main", schema="default")

# Read a specific secret value from the external secret manager
my_secret = dbutils.secrets.get(catalog="main", schema="default", key="example_secret")

You can also browse and list external secrets in Catalog Explorer, the same way as Databricks-managed secrets. See Read a secret.

If a secret does not appear in Unity Catalog yet, you can still read it by name. Provide the name as it appears in the external secret manager, with the character substitutions from Naming restrictions applied.

For example, for an AWS secret named myOrg/mySecret.v3, read it with:

Python
dbutils.secrets.get(catalog="my_catalog", schema="my_schema", key="myOrg|mySecret:v3")

Filter imported secrets by tag​

To surface only some of the secrets that the connection can access, filter by an AWS resource tag. Set the tag options on the connection when you create or edit it:

  • tag_key: Surface only secrets that carry this tag key.
  • tag_value: Optional. When you set it together with tag_key, Unity Catalog surfaces only secrets whose tag matches the key and value. When you omit it, Unity Catalog surfaces all secrets with tag_key regardless of value.

Tag matching is case-sensitive and uses exact string equality.

The connection tag filter controls only which secrets Unity Catalog surfaces; to also restrict which secrets the credential can read, add a tag condition (secretsmanager:ResourceTag) to the IAM role's permission policy.

When you create or edit the AWS Secrets Manager connection, add the tag_key option, and optionally tag_value, as connection options.

Additional resources​

On this page