Customer managed keys for notebooks

Preview

This feature is in Private Preview. It is not available on all Databricks deployment types and subscriptions. Contact your Databricks representative to request access.

Note

This feature requires that your account be on the E2 version of the Databricks platform or on a custom plan that has been enabled by Databricks for this feature.

Introduction

Security-conscious organizations have risk management processes that evaluate risks of public cloud use, SaaS applications, and third-party services. Reducing risk from third-party service providers helps build a strong case for using external services. Some regulated industries may require encryption of some types of data with keys that they manage. These considerations are especially important for sectors that regularly use personal data or other confidential information.

Workspace notebooks are primarily stored in the Databricks Control Plane in a database. The Databricks platform allows you to encrypt notebooks with your own key. The key must be provided at the time a workspace is created.

Note

This feature does not encrypt data stored outside of the control plane. For example, it does not encrypt data in your root S3 bucket.

Important

Workspace data plane VPCs can be in AWS regions ap-south-1, ap-southeast-2, ca-central-1, eu-west-1, eu-central-1, us-east-1, us-east-2, us-west-1, and us-west-2. However, you cannot use a VPC in us-west-1 if you want to use customer-managed keys to encrypt notebooks.

How it works

A customer-managed key encrypts the workspace’s notebooks in the control plane. Customers provide a secret revocable key called a customer-managed key (CMK), which is specified by its ID in the cloud service’s key management system. In AWS, customer keys are managed by AWS Key Management Service (KMS).

Additionally, Databricks creates a Databricks-managed key (DMK) for each workspace. The DMK is wrapped by the CMK to generate the combined encryption key, called the data encryption key (DEK). Databricks uses the DEK to encrypt the workspace’s notebook.

The DEK is cached in memory for several read/write operations and evicted from memory at a regular interval such that new requests require another request to your cloud service’s key management system. If you delete or revoke your key, reading or writing to notebooks fails at the end of the cache time interval.

You add the CMK to your Databricks workspace configuration during workspace creation.

Customer-managed keys work for notebooks

Adding a customer-managed key for notebooks

To add a customer-managed for notebooks, you must add the CMK when you create a workspace using the Account API.

To configure your CMK:

  1. Create or select a symmetric key in AWS KMS, following the instructions in Creating symmetric CMKs or Viewing keys.

  2. Copy these values. You will use them when you create the workspace:

    • Key ARN — Get the ARN from the console or the API (the Arn field in the JSON response).
    • Key alias — An alias specifies a display name for a the CMK in AWS KMS. Use an alias to identify a CMK in cryptographic operations. For more information, see the AWS documentation: AWS::KMS::Alias and Working with aliases.
  3. On the Key policy tab, switch to policy view and edit the key policy so that Databricks can use the key to perform encryption and decryption operations. Set it to the following:

    {
      "Version": "2012-10-17",
      "Id": "key-policy-databricks",
      "Statement": [
        {
          "Sid": "Allow Databricks to use KMS key for Notebooks",
          "Effect": "Allow",
          "Principal": {
            "AWS": "arn:aws:iam::414351767826:root"
          },
          "Action": [
            "kms:Encrypt",
            "kms:Decrypt"
          ],
          "Resource": "*"
        }
      ]
    }
    

    For more information, see Editing keys.

  4. To register the key, follow the instructions in Create a new workspace using the Account API, specifically Step 4: Configure customer-managed key for notebooks (optional).

    Important

    You must set up your keys during workspace creation. You cannot add these keys after you have created the workspace.