Secret management
A Databricks secret stores a sensitive value, such as a password, API key, or hashing salt, so that notebooks, jobs, and functions can use it without the value appearing in code. You can manage secrets in two places:
- Workspace secret scopes: Named collections of secrets in one workspace. Permissions apply to the whole scope. See Workspace secret scopes.
- Secrets in Unity Catalog: Securable objects (
catalog.schema.secret) that are available in every workspace attached to the metastore. You can grant access to each secret. See Secrets in Unity Catalog.
Unity Catalog secrets can be Databricks-managed or backed by AWS Secrets Manager. See AWS Secrets Manager secrets in Unity Catalog.
Choose where to store a secret
The following table compares where each option stores secret values, which workspaces can use them, and how you control access.
| Workspace secret scope | Databricks-managed secret in Unity Catalog | AWS Secrets Manager-backed secret in Unity Catalog (Beta) |
|---|---|---|---|
Where the value is stored | Databricks, in the workspace | Databricks, in Unity Catalog | AWS Secrets Manager (Databricks stores only the secret name in Unity Catalog) |
Encryption with your own key | Yes, with a customer-managed key for managed services | Yes, with a customer-managed key on the catalog | - |
Available in | One workspace | Every workspace attached to the metastore | Every workspace attached to the metastore |
Access control | Secret scope permissions (no secret-level access control) | Unity Catalog privileges on the secret, schema, or catalog | Unity Catalog privileges on the secret, schema, or catalog |
To rotate | Update the value in Databricks | Update the value in Databricks | Rotate in AWS Secrets Manager. No change in Databricks. |
Where you can use secrets
Workspace secret scopes and Unity Catalog secrets are supported in different places. Before you choose one, check that it works where your code runs.
Surface | Workspace secret scope | Secret in Unity Catalog |
|---|---|---|
| Yes. See Read a secret. | Yes, on serverless environment version 4 and above or Databricks Runtime 17.3 LTS and above. See Read a secret. |
REST API and Databricks CLI | Yes. Values returned by the API aren't redacted. | Yes. Values returned by the API aren't redacted. |
Remote development, such as Databricks Connect | Yes, with | No |
Unity Catalog Python and Scala UDFs | No | Yes. Callers need only |
SQL warehouses | Yes, with the secret function | Only through a Unity Catalog Python or Scala UDF that declares the secret |
Spark configuration properties and environment variables on compute | Yes. See Use a secret in a Spark configuration property or environment variable. | No |
Model serving endpoint environment variables | Yes. See Add secrets-based environment variables. | No |
Default package repositories | Yes, in the | No |
Additional resources
-
- Secrets in Unity Catalog
- Create Unity Catalog secrets, grant access, and read them in notebooks, jobs, and UDFs.
-
- Best practices for Unity Catalog secrets
- Organize secrets by environment, application, or team, and let workloads use secrets without reading them.