Step 2: Create a bucket policy

This article outlines how to create a bucket policy that links back to an instance profile. This is the second step in securing access to S3 buckets in Databricks.

To complete these steps, you should already have configured an instance profile that grants access from EC2 to a named S3 bucket. See Step 1: Create an instance profile. In this step, you’ll configure permissions from the S3 bucket back to that instance profile.

Requirements for the bucket policy

At a minimum, the S3 policy must include the ListBucket and GetObject actions, which provide read-only access to a bucket. Delta Lake uses DeleteObject and PutObject permissions during regular operations. The permissions in the example policy below are the recommended defaults for clusters that read and write data.

Create the bucket policy

  1. Go to your S3 console. From the Buckets list, select the bucket for which you want to create a policy.

  2. Click Permissions.

  3. Under Bucket policy, click Edit.

  4. Paste in a policy. A sample cross-account bucket IAM policy could be the following, replacing <aws-account-id-databricks> with the AWS account ID where the Databricks environment is deployed, <iam-role-for-s3-access> with the instance profile role, and <s3-bucket-name> with the bucket name.

    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Example permissions",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<aws-account-id-databricks>:role/<iam-role-for-s3-access>"
      },
      "Action": [
        "s3:GetBucketLocation",
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::<s3-bucket-name>"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<aws-account-id-databricks>:role/<iam-role-for-s3-access>"
      },
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:PutObjectAcl"
      ],
      "Resource": "arn:aws:s3:::<s3-bucket-name>/*"
    }
  ]
}

  5. Click Save.

Next steps

After you’ve set up the bucket policy, you need to modify the policy attached to the deployment role. See Step 3: Add an S3 IAM role to the EC2 policy.