Configure S3 access with instance profiles

This article walks you through how to create an instance profile with read, write, update, and delete permissions on a single S3 bucket. You can grant privileges for multiple buckets using a single IAM role and instance profile. It is also possible to use instance profiles to grant only read and list permissions on S3.

Note

Databricks recommends using Unity Catalog external locations to connect to S3 instead of instance profiles. Unity Catalog simplifies security and governance of your data by providing a central place to administer and audit data access across multiple workspaces in your account. See What is Unity Catalog? and Connect to S3 with Unity Catalog.

Before you begin

This tutorial is designed for workspace administrators. You must have sufficient privileges in the AWS account containing your Databricks workspace, and be a Databricks workspace administrator.

This tutorial assumes the following existing permissions and assets:

  • Privileges to edit the IAM role used to deploy the Databricks workspace.

  • Privileges to create new IAM roles in AWS.

  • Privileges to edit permissions on an S3 bucket.

Step 1: Create an instance profile

In this step, you create a new IAM role and define an inline policy. Together, these settings define the instance profile deployed to EC2 instances. Here you can also add a trust relationship so the instance profile can work with serverless compute resources.

See Step 1: Create an instance profile.

Step 2: Create an S3 bucket policy

In this step, you add a trust relationship from the S3 bucket to the IAM role you created in Step 1.

Note

S3 buckets have universally unique names and do not require an account ID for universal identification. If you choose to link an S3 bucket to an IAM role and Databricks workspace in a different AWS account, you must specify the account ID when configuring your S3 bucket policy.

Make sure you copied the role ARN from Step 1. And ensure you’re creating the policy on the S3 bucket you specified in your IAM role.

See Step 2: Create a bucket policy.

Step 3: Modify the IAM role for the Databricks workspace

Databricks uses a role configured during workspace deployment to manage EC2 instances in your AWS account. To make an instance profile available in your Databricks workspace, you need to modify the policy attached to this role.

See Step 3: Add an S3 IAM role to the EC2 policy.

Step 4: Add the instance profile to the Databricks workspace

As a final step, add the role ARN from Step 1 into your workspace by using the Databricks workspace admin settings.

See Step 4: Add an instance profile to Databricks.

Manage instance profiles

You can manage instance profiles similar to other workspace assets using workspace ACLs.

See Manage instance profile access in Databricks

Deploy compute resources with an instance profile

Users with permissions to deploy clusters can deploy clusters with any of their assigned instance profiles. All users with access to the cluster gain the permissions as defined by the instance profile.

See Launch a compute resource with the instance profile.

SQL warehouses use a single instance profile for each workspace and then use table access control for fine-grained permissions.

See Hive metastore privileges and securable objects (legacy).

Note

Hive metastore table access control is a legacy data governance model. Databricks recommends that you use Unity Catalog instead for its simplicity and account-centered governance model. You can upgrade the tables managed by the Hive metastore to the Unity Catalog metastore.

Edit instance profile role ARN

For instance profiles that you’ve already created, you can later edit them but only to specify a different IAM role ARN. This step is required for Databricks SQL Serverless to work with an instance profile whose role name (the text after the last slash in the role ARN) and the instance profile name (the text after the last slash in the instance profile ARN) do not match. For related information, see Use serverless SQL warehouses.

  1. Go to the admin settings page.

  2. Click the Instance Profiles tab.

  3. Click the name of your instance profile that you want to edit.

  4. Click Edit. A dialog appears.

    Edit instance profile Role ARN

    Edit the IAM role ARN field and paste in the role ARN associated with your instance profile. As an admin, you can get this value from the AWS console.

  5. Click Save.