Create an instance profile

This article explains how to create an instance profile using the AWS console.

Databricks recommends deploying compute resources with instance profiles for secure access to data stored in S3. Instance profiles are just another name for AWS IAM roles associated with EC2 instances.

Administrators configure IAM roles in AWS, link them to the Databricks workspace, and grant access to privileged users to associate instance profiles with compute. All users that have access to compute resources with an instance profile attached to it gain the privileges granted by the instance profile.

Use the AWS console to create an instance profile

  1. In the AWS console, go to the IAM service.

  2. Click the Roles tab in the sidebar.

  3. Click Create role.

    1. Under Select trusted entity, select AWS service.

    2. Under Use case, select EC2.

    3. Click Next.

    4. On the Add permissions page, click Next.

    5. In the Role name field, type a role name.

    6. Click Create role. The list of roles displays.

  4. In the role list, click the role.

  5. Add an inline policy to the role. This policy grants access to the S3 bucket.

    1. In the Permissions tab, click Add permissions > Create inline policy.

    2. Click the JSON tab.

    3. Copy this policy and set <s3-bucket-name> to the name of your bucket.

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "s3:ListBucket"
            ],
           "Resource": [
              "arn:aws:s3:::<s3-bucket-name>"
            ]
          },
          {
            "Effect": "Allow",
            "Action": [
              "s3:PutObject",
              "s3:GetObject",
              "s3:DeleteObject",
              "s3:PutObjectAcl"
            ],
            "Resource": [
               "arn:aws:s3:::<s3-bucket-name>/*"
            ]
          }
        ]
      }
      
    4. Click Review policy.

    5. In the Name field, type a policy name.

    6. Click Create policy.

  6. In the role summary, copy the Role ARN.

    Instance profile ARN

Note

If you intend to enable encryption for the S3 bucket, you must add the IAM role as a Key User for the KMS key provided in the configuration. See Configure KMS encryption for s3a:// paths.

Enable the policy to work with serverless resources

This step ensures that your instance profile will work if you choose to configure Databricks SQL to use this instance profile. By configuring this policy, your instance profile can use serverless, pro, or classic SQL warehouses.

  1. In the role list, click your instance profile.

  2. Select the Trust Relationships tab.

  3. Click Edit Trust Policy.

  4. Within the existing Statement array, append the following JSON block to the end of the existing trust policy. Ensure that you don’t overwrite the existing policy.

     {
     "Effect": "Allow",
     "Principal": {
       "AWS": [
         "arn:aws:iam::790110701330:role/serverless-customer-resource-role"
       ]
     },
     "Action": "sts:AssumeRole",
     "Condition": {
       "StringEquals": {
           "sts:ExternalId": [
             "databricks-serverless-<YOUR_WORKSPACE_ID1>",
             "databricks-serverless-<YOUR_WORKSPACE_ID2>"
           ]
         }
       }
     }
    

    The only thing you need to change in the statement is the workspace IDs. Replace the YOUR_WORKSPACE-IDs with one or more Databricks workspace IDs for the workspaces that will use this role.

    Note

    To get your workspace ID, check the URL when you’re using your workspace. For example, in https://<databricks-instance>/?o=6280049833385130, the number after o= is the workspace ID.

    Do not edit the principal of the policy. The Principal.AWS field must continue to have the value arn:aws:iam::790110701330:role/serverless-customer-resource-role. This references a Serverless compute role managed by Databricks.

  5. Click Review policy.

  6. Click Save changes.

Next steps

After you create an instance profile, you need to create the S3 bucket policy. See Create a bucket policy for the target S3 bucket.