Customer-managed keys for encryption
This feature requires the Enterprise tier.
This page provides an overview of customer-managed keys for encryption. Some services and data support adding a customer-managed key to help protect and control access to encrypted data. You can use the key management service in your cloud to maintain a customer-managed encryption key.
For instructions on how to configure keys, see Configure customer-managed keys for encryption.
Customer-managed key use cases
Databricks has two customer-managed key use cases that involve different types of data and locations:
- Managed services: Data in the Databricks control plane (notebooks, secrets, and Databricks SQL query data).
- Workspace storage: The two workspace storage buckets and the GCE Persistent Disk volumes of compute resources.
Customer-managed keys for managed services
Managed services data in the Databricks control plane is encrypted at rest. You can add a customer-managed key for managed services to help protect and control access to the following types of encrypted data:
- Notebook source in the Databricks control plane.
- Notebook results for notebooks run interactively (not as jobs) that are stored in the control plane. By default, larger results are also stored in your workspace root bucket. You can configure Databricks to store all interactive notebook results in your cloud account.
- Secrets stored in Databricks secrets.
- AI/BI dashboards.
- Databricks SQL queries and query history.
- Personal access tokens (PAT) or other credentials used to set up Git integration with Databricks Git folders.
- Vector Search indexes and metadata.
To configure customer-managed keys for managed services, see Configure customer-managed keys for encryption.
Only AI/BI dashboards created after November 1, 2024 are encrypted and compatible with customer-managed keys.
Customer-managed keys for workspace storage
You can add a customer-managed key for workspace storage to protect and control access to the following types of encrypted data:
- Your workspace storage buckets: If you add a workspace storage encryption key, Databricks encrypts the data on the two GCS buckets associated with the Google Cloud project that Databricks created when you created your workspace. These are known as the workspace storage buckets. One bucket contains DBFS root, which includes the FileStore area, MLflow Models, and Lakeflow Declarative Pipelines data in your DBFS root (not DBFS mounts). Another bucket contains workspace system data, which includes job results, Databricks SQL results, notebook revisions, and other workspace data.
- Your cluster's GCE persistent disks: Your workspace storage encryption key is used to encrypt the GCE persistent disks of Databricks Runtime cluster nodes and other compute resources in the classic compute plane.
Compare customer-managed keys use cases
The following table lists which customer-managed key features are used for which types of data.
Type of data | Location | Which customer-managed key feature to use |
|---|---|---|
AI/BI dashboards | Control plane | Managed services |
Notebook source and metadata | Control plane | Managed services |
Personal access tokens (PAT) or other credentials used for Git integration with Databricks Git folders | Control plane | Managed services |
Secrets stored by the secret manager APIs | Control plane | Managed services |
Databricks SQL queries and query history | Control plane | Managed services |
The remote GCE Persistent Disk volumes for Databricks Runtime cluster nodes and other compute resources. | Workspace storage | |
DBFS root in your workspace storage bucket. This also includes the FileStore area. | Workspace storage | |
Job results | Workspace storage buckets in your Google Cloud account | Workspace storage |
Databricks SQL query results | Workspace storage buckets in your Google Cloud account | Workspace storage |
Workspace storage buckets in your Google Cloud account | Workspace storage | |
If you use a DBFS path in your DBFS root, this is stored in the workspace storage buckets in your Google Cloud account. This does not apply to DBFS paths that represent mount points to other data sources. | Workspace storage | |
By default, when you run a notebook interactively (rather than as a job) results are stored in the control plane for performance with some large results stored in your workspace storage bucket in your Google Cloud account. You can choose to configure Databricks to store all interactive notebook results in your Google Cloud account. See Configure the storage location for interactive notebook results. | For partial results in the control plane, use a customer-managed key for managed services. For results in the two GCS buckets, which you can configure for all result storage, use a customer-managed key for workspace storage. See Configure the storage location for interactive notebook results. |
Serverless compute and customer-managed keys
Databricks SQL Serverless and serverless compute support:
- Keys for managed services like Databricks SQL queries, query history, notebook source and metadata, and vector search indexes and metadata.
- Workspace storage keys including root storage for Databricks SQL and notebook results.
Encryption for remote GCE Persistent Disk volumes does not apply to serverless compute because disks for serverless compute resources are short-lived and tied to the lifecycle of the serverless workload. When serverless compute resources are stopped or scaled down, the VMs and their storage are destroyed.
Model Serving
Resources for Model Serving, a serverless compute feature, are generally in two categories:
- Resources that you create for the model are stored in your workspace's root storage. This includes the model's artifacts and version metadata. Both the workspace model registry and MLflow use this storage. You can configure this storage to use customer-managed keys.
- Resources that Databricks creates directly on your behalf include the model image and ephemeral serverless compute storage. These are encrypted with Databricks-managed keys and do not support customer-managed keys.