Skip to main content

Secrets in Unity Catalog

A Unity Catalog secret is a securable object that stores a sensitive credential, such as a password, token, or API key. Secrets use the three-level namespace (catalog.schema.secret) and are available in every workspace attached to the metastore.

You govern secrets with the same Unity Catalog privileges as your other data assets, down to the individual secret. Users can browse a secret without being able to read or use it, you can encrypt secret values with your own key, and audit logs record every read or change.

Unity Catalog secrets can be Databricks-managed or backed by AWS Secrets Manager. See AWS Secrets Manager secrets in Unity Catalog.

Requirements​

Create a secret​

Secrets live in schemas. Create secrets in a schema that only the principals who need them can access. To decide how to group secrets into schemas, see Group secrets by who needs them.

A schema holds Databricks-managed secrets or AWS Secrets Manager-backed secrets, not both, so you choose where values are stored when you set up the schema. For an AWS Secrets Manager-backed schema, see Create a secret in an AWS Secrets Manager-backed schema.

Create a Databricks-managed secret​

To create a Databricks-managed secret, you need USE CATALOG on the catalog, and either ownership of the schema or CREATE SECRET and USE SCHEMA on it.

To create a schema for an application's secrets, run the following:

SQL
CREATE SCHEMA IF NOT EXISTS main.orders_app;
  1. In your Databricks workspace, click Catalog to open Catalog Explorer.
  2. Go to the schema where you want to create the secret.
  3. Click Create > Secret.
  4. Enter a name and value. Optionally, add a comment and an expiration date. If a secret expires, Catalog Explorer shows a warning.
  5. Click Create.

Create a secret in an AWS Secrets Manager-backed schema​

An admin backs a schema with AWS Secrets Manager one time. After that, you create, update, and rotate its secrets in AWS Secrets Manager, and they appear in the schema in Unity Catalog. To back a schema, see AWS Secrets Manager secrets in Unity Catalog.

  1. Create the secret in AWS Secrets Manager, in the AWS region that the schema's connection uses. If the connection has a tag filter, add the matching tag.
  2. Read the secret by name. See Read a secret.

Grant access to secrets​

Grant the privilege that matches what each principal needs to do. Except with BROWSE, every principal also needs USE CATALOG and USE SCHEMA on the parent catalog and schema. A privilege granted on a catalog or schema applies to the secrets in it, including secrets created later.

To let a principal

Grant

Read the value in a notebook, job, or REST API call

READ SECRET

Discover specific secrets without seeing their values

REFERENCE SECRET on the secret or schema, when BROWSE on the whole catalog is too broad. It doesn't give read access or let the principal use the value.

Update the value

WRITE SECRET. Granting only this privilege does not give read access to the secret.

Create secrets in a schema

CREATE SECRET on the schema or catalog

Grant privileges on secrets to others

Ownership, or MANAGE on the secret, schema, or catalog

To let a principal

Grant

Read the value in a notebook, job, or REST API call

READ SECRET

Discover specific secrets without seeing their values

REFERENCE SECRET on the secret or schema, when BROWSE on the whole catalog is too broad. It doesn't give read access or let the principal use the value.

Update the value

WRITE SECRET. Granting only this privilege does not give read access to the secret.

Create secrets in a schema

CREATE SECRET on the schema or catalog

Grant privileges on secrets to others

Ownership, or MANAGE on the secret, schema, or catalog

To grant privileges, use Catalog Explorer: open the secret or schema, click the Permissions tab, and click Grant. To grant with SQL, run the following:

SQL
-- Let a job's service principal read every secret in a schema
GRANT USE CATALOG ON CATALOG main TO `etl-service-principal`;
GRANT USE SCHEMA, READ SECRET ON SCHEMA main.orders_app TO `etl-service-principal`;

-- Let the application's developers create secrets in its schema
GRANT USE SCHEMA, CREATE SECRET ON SCHEMA main.orders_app TO `orders-developers`;

See Manage privileges in Unity Catalog.

Read a secret​

To read a secret value, you need ownership of the secret, or READ SECRET on the secret or on its schema or catalog.

Databricks recommends dbutils, because it redacts the value in cell output and logs. The following example reads a database username and password to configure a JDBC read:

Python
username = dbutils.secrets.get(catalog="main", schema="orders_app", key="db_username")
password = dbutils.secrets.get(catalog="main", schema="orders_app", key="db_password")

df = (spark.read
.format("jdbc")
.option("url", "<jdbc-url>")
.option("dbtable", "<table-name>")
.option("user", username)
.option("password", password)
.load()
)

See Secrets utility (dbutils.secrets).

List, update, and delete secrets​

To update a value, you need WRITE SECRET on the secret. To change its comment or expiration date, or to delete it, you need ownership of the secret or MANAGE on it, its schema, or its catalog.

For an AWS Secrets Manager-backed schema, update and delete secrets in AWS Secrets Manager. Deletes carry over to Unity Catalog.

  1. In Catalog Explorer, go to the schema and click Secrets in the Overview pane to list its secrets.
  2. Click the secret you want to change.
  3. In the top-right corner, click the kebab menu (vertical dots) and select Edit or Delete.
  4. To edit, enter a new value or expiration date and click Confirm. To delete, enter the full name of the secret and click Delete.

Tag secrets​

Unity Catalog secrets support key-value tags, for example to classify secrets or record the owning team or cost center. See Apply tags to Unity Catalog securable objects.

To add or remove tags on a secret, you must own the secret or have all of the following privileges:

  • APPLY TAG on the secret
  • USE SCHEMA on the secret's parent schema
  • USE CATALOG on the secret's parent catalog

To assign a governed tag, you also need the ASSIGN permission on the governed tag. See Manage permissions on governed tags.

To add or remove a tag with SQL, run the following:

SQL
SET TAG ON SECRET main.orders_app.db_password cost_center = hr;

UNSET TAG ON SECRET main.orders_app.db_password cost_center;

See SET TAG and UNSET TAG.

To view the tags on a secret, use one of the following:

  1. In Catalog Explorer, go to the schema and click Secrets in the Overview pane.
  2. Click the secret. Its tags appear under Tags on the Overview tab.

Secret tags don't appear in the output of DESCRIBE SECRET, in the response of the API that gets a secret, in information schema tables, or in workspace search.

Audit secret access​

The system.access.audit system table records secret events. Reads appear with action_name = 'getSecret'. To list the last 7 days of secret events, run the following:

SQL
SELECT event_time, user_identity.email, action_name, request_params
FROM system.access.audit
WHERE service_name = 'unityCatalog'
AND action_name LIKE '%Secret%'
AND event_date >= current_date() - INTERVAL 7 DAYS
ORDER BY event_time DESC;

See Audit log system table reference.

Encrypt secret values with customer-managed keys​

Databricks encrypts secret values with Databricks-managed keys by default. To use your own key, configure customer-managed key (CMK) settings on the catalog that holds the secrets. Secret values are stored in Unity Catalog, not in the catalog's storage location, so this works for any catalog, including one with an external storage location. For requirements and setup, see Customer-managed keys for Unity Catalog.

External secret values stay in your cloud secret manager, which encrypts them with its own keys. CMK settings on the catalog don't apply to them.

Use secrets in UDFs​

A Unity Catalog Python or Scala UDF can use a secret value without giving callers access to the secret. Declare the secret in the UDF's SECRETS clause. The UDF reads it with the function owner's permissions:

  • The owner needs READ SECRET on the secret to create or replace the UDF. If the owner loses READ SECRET, the UDF fails.
  • Callers need only EXECUTE on the UDF and no privilege on the secret.

The UDF definition must set environment_version to 6 or above. On classic compute, use Databricks Runtime 19 and above in standard access mode. For examples, see Python UDFs and Scala UDFs.

Session-scoped UDFs, which use databricks.secrets.get() or com.databricks.Secrets.get(), read secrets with the caller's permissions, so every caller needs READ SECRET. See Python UDFs and Scala UDFs.

On dedicated access mode compute, you can't call these UDFs directly. You can call one from a Unity Catalog SQL function that you use as an attribute-based access control (ABAC) column mask, which Databricks enforces on serverless compute. See Unsupported compute versions.

warning

Use secret values only inside the UDF. Do not return them in UDF results. Redaction reduces accidental exposure in errors and logs, but it does not stop UDF code from returning a secret in query results.

Limitations​

  • Quotas: A schema can hold up to 1,000 secrets, and a metastore up to 10,000.
  • No global search or information schema: Secrets don't appear in global search or information schema tables. Use Catalog Explorer, dbutils, or the REST API.
  • OAuth API scope: The Unity Catalog secrets API requires the unity-catalog OAuth API scope. The secrets scope applies only to workspace secret scopes.
  • Init scripts: You can't use Unity Catalog secrets in global or cluster-scoped init scripts.
  • Remote development: Databricks Connect and compiled JAR run modes can't read Unity Catalog secrets, because dbutils.secrets requires a Databricks Runtime-backed notebook or job.

Additional resources​