Skip to main content

Configure encryption for S3 with KMS

This article covers how to configure server-side encryption with a KMS key for reading files in s3a:// paths using Unity Catalog. It is provided for users who need cross-cloud access to external tables and volumes in encrypted S3 buckets.

Configure encryption for S3 using Unity Catalog​

You can configure server-side encryption to allow external tables and volumes in Unity Catalog to access data in S3. SSE encryption is not supported with external tables shared using Delta Sharing.

Step 1: Update your KMS key policy in AWS​

To protect data in S3, AWS supports server-side encryption (SSE) with Amazon S3 managed keys (SSE-S3) or AWS KMS keys (SSE-KMS). If you use an AWS S3 managed key, skip to step 2.

  1. In AWS, go to the KMS service.

  2. Click the key to which you want to add permission.

  3. In the Key Policy section, select Switch to policy view.

  4. Edit the key policy section that allows S3 to use the key, for example:

    JSON
    {
        "Sid": "Allow access through S3 for all principals in the account that are authorized to use S3",
        "Effect": "Allow",
        "Principal": {
            "AWS": "*"
        },
        "Action": [
            "kms:Encrypt",
            "kms:Decrypt",
            "kms:ReEncrypt*",
            "kms:GenerateDataKey*",
            "kms:DescribeKey"
        ],
        "Resource": "*",
        "Condition": {
            "StringEquals": {
                "kms:CallerAccount": "<AWS ACCOUNT ID>",
                "kms:ViaService": "s3.<REGION>.amazonaws.com"
            }
        }
    },

  5. Click Save changes.

Step 2: Configure access to S3 using Unity Catalog​

  1. Create a storage credential to connect to S3, using the instructions in Create a storage credential for connecting to Google Cloud Storage.

    Ensure that you create the IAM policy in the same account as the S3 bucket. If you are using SSE-KMS, include the following in the policy:

    JSON
      {
        "Action": [
            "kms:Decrypt",
            "kms:Encrypt",
            "kms:GenerateDataKey*"
        ],
        "Resource": [
            "arn:aws:kms:<KMS-KEY>"
        ],
        "Effect": "Allow"
    },

    See Step 1: Create an IAM role.

  2. Create an external location to connect to S3, using the instructions in Create an external location to connect cloud storage to Databricks.

  3. Configure server-side encryption on your external location, using the instructions in Configure an encryption algorithm on an external location (AWS S3 only).

Last updated on