Step 2: Configure credentials for audit log delivery

This article describes how to set up IAM services for audit log delivery. To use different credentials for different workspaces, repeat the procedures in this article for each workspace or group of workspaces.

Note

To use different S3 bucket names, you need to create separate IAM roles.

Create the IAM role

Log into your AWS Console as a user with administrator privileges and go to the IAM service.
Click the Roles tab in the sidebar.
Click Create role.
1. In Select type of trusted entity, click AWS service.
2. Under Use Case, select EC2.
3. Click the Next button.
4. Click the Next button.
5. In the Role name field, enter a role name.
6. Click Create role. The list of roles displays.

Create the inline policy

In the list of roles, click the role you created.

Add an inline policy.

On the Permissions tab, click Add permissions then Create inline policy.
In the policy editor, click the JSON tab.

Copy this access policy and modify it. Replace the following values in the policy with your own configuration values:

<s3-bucket-name>: The bucket name of your AWS S3 bucket.
<s3-bucket-path-prefix>: (Optional) The path to the delivery location in the S3 bucket. If unspecified, the logs are delivered to the root of the bucket. This path must match the delivery_path_prefix argument when you call the log delivery API.

{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Effect":"Allow",
      "Action":[
        "s3:GetBucketLocation"
      ],
      "Resource":[
        "arn:aws:s3:::<s3-bucket-name>"
      ]
    },
    {
      "Effect":"Allow",
      "Action":[
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:PutObjectAcl",
        "s3:AbortMultipartUpload"
      ],
      "Resource":[
        "arn:aws:s3:::<s3-bucket-name>/<s3-bucket-path-prefix>/",
        "arn:aws:s3:::<s3-bucket-name>/<s3-bucket-path-prefix>/*"
      ]
    },
    {
      "Effect":"Allow",
      "Action":[
        "s3:ListBucket",
        "s3:ListMultipartUploadParts",
        "s3:ListBucketMultipartUploads"
      ],
      "Resource":"arn:aws:s3:::<s3-bucket-name>",
      "Condition":{
        "StringLike":{
          "s3:prefix":[
            "<s3-bucket-path-prefix>",
            "<s3-bucket-path-prefix>/*"
          ]
        }
      }
    }
  ]
}

You can customize the policy usage of the path prefix in the following ways:

If you do not want to use the bucket path prefix, remove <s3-bucket-path-prefix>/ (including the final slash) from the policy each time it appears.
If you want log delivery configurations for different workspaces that share the S3 bucket but use different path prefixes, you can include multiple path prefixes. There are two separate parts of the policy that reference <s3-bucket-path-prefix>. For each case, duplicate the two lines that reference the path prefix. For example:

{
  "Resource":[
    "arn:aws:s3:::<mybucketname>/field-team/",
    "arn:aws:s3:::<mybucketname>/field-team/*",
    "arn:aws:s3:::<mybucketname>/finance-team/",
    "arn:aws:s3:::<mybucketname>/finance-team/*"
  ]
}

Click Review policy.
In the Name field, enter a policy name.
Click Create policy.
If you use service control policies to deny certain actions at the AWS account level, ensure that sts:AssumeRole is whitelisted so Databricks can assume the cross-account role.

Create the trust policy

On the role summary page, click the Trust Relationships tab.

Paste this access policy into the editor, replacing <databricks-account-id> with your Databricks account ID. Note the policy is slightly different if you use Databricks on AWS GovCloud.

{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Effect":"Allow",
      "Principal":{
        "AWS":"arn:aws:iam::414351767826:role/SaasUsageDeliveryRole-prod-IAMRole-3PLHICCRR1TK"
      },
      "Action":"sts:AssumeRole",
      "Condition":{
        "StringEquals":{
          "sts:ExternalId":[
            "<databricks-account-id>"
          ]
        }
      }
    }
  ]
}

{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Effect":"Allow",
      "Principal":{
        "AWS":"arn:aws-us-gov:iam::044793339203:role/SaasUsageDeliveryRole-prod-aws-gov-IAMRole-L4QM0RCHYQ1G"
      },
      "Action":"sts:AssumeRole",
      "Condition":{
        "StringEquals":{
          "sts:ExternalId":[
            "<databricks-account-id>"
          ]
        }
      }
    }
  ]
}

In the role summary, copy the Role ARN. You need this value to call the create credential configuration API in the next step.

Call the create credential configuration API

To finish settings up your credentials, call the Create credential configuration API.

This request establishes cross-account trust and returns a reference ID you can use when creating a new workspace.

Replace <account-id> with your Databricks account ID.
Set credentials_name to a name that is unique within your account.
Set role_arn to the role ARN that you just created.

The response body includes a credentials_id field. Copy this field so you can use it to create the log delivery configuration in Step 4.

For example:

curl -X POST -n \
  'https://accounts.cloud.databricks.com/api/2.0/accounts/<databricks-account-id>/credentials' \
  -d '{
  "credentials_name": "databricks-credentials-v1",
  "aws_credentials": {
    "sts_role": {
      "role_arn": "arn:aws:iam::<aws-account-id>:role/my-company-example-role"
    }
  }
}'

Example response:

{
  "credentials_id": "<databricks-credentials-id>",
  "account_id": "<databricks-account-id>",
  "aws_credentials": {
    "sts_role": {
      "role_arn": "arn:aws:iam::<aws-account-id>:role/my-company-example-role",
      "external_id": "<databricks-account-id>"
    }
  },
  "credentials_name": "databricks-credentials-v1",
  "creation_time": 1579753556257
}

Again, copy the credentials_id field from the response for later use.

Next steps

If you need to set up cross-account delivery (your S3 bucket is in a different AWS account than the IAM role used for log delivery), see Step 3: Configure cross-account support (Optional).

If your S3 bucket is in the same AWS account as your IAM role used for log delivery, skip to the final step of calling the log delivery API. See Step 4: Call the log delivery API.