Permissions in cross-account IAM roles
This article lists permissions in the cross-account IAM role and the purpose of each role.
The permissions are different based on how you configure your VPC.
IAM permissions for Databricks-managed VPCs
Databricks requires the following list of IAM permissions to operate and manage clusters in an effective manner. This configuration applies only to workspaces that use the default (Databricks-managed) VPC. To create the AWS cross-account role policy for use with the default Databricks-managed VPC, see Create an IAM role for workspace deployment.
The following table lists Databricks IAM cross-account role permissions in the default configuration, the resources that they control, and the purpose for each permission.
AWS IAM permission | AWS resource | Purpose |
|---|---|---|
| Elastic IP address | Allocates an Elastic IP that is associated with the NAT Gateway used in secure cluster connectivity |
| Network Interface | Assigns a private IP to EC2 instance. |
| DHCP | Associates a set of DHCP options (or no DHCP options) with a VPC. |
| InstanceProfile | Associate an instance profile with a running EC2 instance. This allows a Databricks pool instance to be used by clusters with different instance profiles throughout its lifetime in the pool. |
| RouteTable | Associates a subnet with a route table. |
| InternetGateway | Attaches an Internet gateway to a VPC, enabling connectivity between the Internet and the VPC. This is currently required to connect to S3 buckets and update code for the workers and spark containers. |
| EBS Volume | Attaches volume for EBS auto-scaling. |
| SecurityGroup | Adds egress rules to the security groups if required. |
| SecurityGroup | Adds ingress rules to the security groups. |
| SpotInstance | Cancels spot instances. |
| Dhcp | Creates DHCP options. |
| EC2 Fleet | Creates an EC2 fleet (used with Databricks fleet clusters). |
| InternetGateway | Creates an Internet Gateway. |
| Launch Template | Creates a launch template (used with Databricks fleet clusters). |
| Launch Template | Creates a launch template version (used with Databricks fleet clusters). |
| NatGateway | Creates a NAT gateway. |
| Route | Creates routes during workspace setup. |
| RouteTable | Creates routes during workspace setup. |
| SecurityGroup | Creates security groups during initial setup |
| Subnet | Create subnets for the VPC during workspace setup. |
| Tags | Adds tags on Databricks resources. |
| EBS Volume | Creates volume. |
| VPC | Creates the Databricks-managed VPC. |
| VPCEndpoint | Creates VPC endpoints as part of configuring the VPC. |
| DHCPOptions | Deletes DHCPOptions |
| EC2 Fleet | Deletes an EC2 fleet (used with Databricks fleet clusters) |
| InternetGateway | Deletes Internet Gateway during workspace deletion. |
| Launch Template | Deletes a launch template and all its versions. |
| Launch Template | Deletes a version from a launch template (used with Databricks fleet clusters). |
| NatGateway | Deletes NAT gateway as needed to setup the secure cluster connectivity relay. |
| Route | Deletes routes. |
| RouteTable | Deletes route table. |
| SecurityGroup | Deletes security groups during workspace deletion. |
| Subnet | Deletes subnet. |
| Tags | Removes tags from cluster resources to allows Databricks pool instances to be reused by clusters with different tags. |
| EBS Volume | Deletes a volume for EBS auto-scaling. See this page. |
| VPC | Deletes the VPC when customers during workspace deletion. |
| VPCEndpoints | Deletes the VPC endpoints during workspace deletion. |
| AvailabilityZones | Gets a list of Availability Zones in a region so that Databricks can deploy resources in that zone. |
| EC2 Fleet | Lists events in an EC2 fleet (used with Databricks fleet clusters). |
| EC2 Fleet | Lists instances in an EC2 fleet (used with Databricks fleet clusters). |
| EC2 Fleet | Describes details of an EC2 fleet (used with Databricks fleet clusters). |
| InstanceProfile | Checks the current instance profile that is set on an EC2 instance so that the right profile is set on a Databricks pool instance before it’s reused by a cluster. |
| Instance | Confirms that Databricks AWS instances are healthy. |
| Instance | Confirms that Databricks AWS instances are healthy. |
| InternetGateway | Describes InternetGateway to confirm that Databricks AWS instances have a route to the internet. |
| Launch Template | Deletes a launch template (used with Databricks fleet clusters). |
| Launch Template | Describes details of launch template versions (used with Databricks fleet clusters). |
| NATGateway | Describes a NAT Gateway to confirm that Databricks AWS instances have a route to the internet in the secure cluster connectivity architecture. |
| PrefixList | Creates a prefix list ID to create an outbound security group rule that allows traffic from a VPC so that Databricks can access an AWS service through a gateway VPC endpoint. |
| Instance | Describes Reserved Instance pricing in support of AWS spot instance pricing. |
| RouteTable | Confirms that the route tables are set up correctly in the Databricks-managed VPC. |
| SecurityGroup | Confirms that AWS security groups are set up correctly. |
| Instance | Describes spot instances. |
| SpotInstance | Describes spot instances. |
| Subnet | Confirms that subnets are setup correctly in Databricks VPC. |
| Volume | Lists volumes. |
| VPC | Confirm that the workspace’s VPC was set up correctly. |
| InternetGateway | Detaches the Databricks created Internet Gateway during workspace deletion. |
| InstanceProfile | Disassociates an instance profile from an EC2 instance so that xDatabricks pool instances can be used by clusters with different instance profiles. |
| RouteTable | Detaches the Databricks created route table during workspace deletion. |
| Launch Templates | Gets config of a launch template (used with Databricks fleet clusters). |
| Availability Zones | Gets list of AZs with best spot capacity for given instance type(s). |
| EC2 Fleet | Modifies an EC2 fleet (used with Databricks fleet clusters). |
| Launch Templates | Modifies an existing launch template (used with Databricks fleet clusters). |
| VPCAttribute | Configures the Databricks-managed VPC. |
| Address | Detach the Databricks created address during workspace deletion. |
| InstanceProfile | Swaps one instance profile for another on an EC2 instance so that Databricks pool instances can be used by clusters with different instance profiles. |
| SpotInstance | Requests spot instances. |
| SecurityGroup | Updates Databricks-managed security groups if required. |
| SecurityGroup | Updates security groups. |
| Instance | Launches AWS instances to create Spark Clusters. Also leveraged during scaling up an existing Spark cluster. |
| Instance | Terminates Spark EC2 nodes during cluster scale down or to terminate a given Spark cluster. |
| ServiceLinkedRole | Sets up support for spot instances. |
| RolePolicy | Configures Databricks to use spot instances. |
IAM permissions for customer-managed VPC
If you use a customer-managed VPC, there’s a smaller set of permissions needed for the cross-account IAM role. This feature requires the Premium plan or above.
To create the AWS cross-account role policy for use with a customer-managed VPC, see Option 2: Customer-managed VPC with default restrictions policy.
The permissions can be scoped down further if needed. To create the AWS cross-account role policy for use with a customer-managed VPC with additional custom restrictions on resources, see Option 3: Customer-managed VPC with custom policy restrictions.
The following table lists Databricks IAM cross-account role permissions for a customer-managed VPC, the resources that they control, and the purpose for each permission.
AWS IAM permission | AWS resource | Purpose |
|---|---|---|
| InstanceProfile | Associates an instance profile with a running EC2 instance so that a Databricks pool instance can be used by clusters with different instance profiles throughout its lifetime in the pool. |
| Volume | Attaches a volume. |
| SecurityGroup | Add egress rules to the security groups if required. |
| SecurityGroup | Adds ingress rules to the security groups. |
| SpotInstance | Cancels spot instances. |
| EC2 Fleet | Creates an EC2 fleet (used with Databricks fleet clusters). |
| Launch Template | Creates a launch template (used with Databricks fleet clusters). |
| Tags | Adds tags on Databricks resources. |
| Volume | Creates a volume. |
| EC2 Fleet | Deletes an EC2 fleet (used with Databricks fleet clusters) |
| Launch Template | Deletes a launch template and all its versions. |
| Launch Template | Deletes a version from a launch template (used with Databricks fleet clusters). |
| Tags | Removes tags from cluster resources so that Databricks pool instances can be reused by clusters with different tags. |
| Volume | Deletes a volume. |
| AvailabilityZones | Gets a list of Availability Zones in a region so that Databricks can deploy the resources in that zone. |
| InstanceProfile | Checks the current instance profile set on an EC2 instance to confirm that the right profile is set on a Databricks pool instance before it’s reused by a cluster. |
| Instance | Confirms that Databricks AWS instances are healthy. |
| Instance | Confirm that Databricks AWS instances are healthy. |
| InternetGateway | Describes InternetGateway to confirm that Databricks AWS instances have a route to the internet. |
| EC2 Fleet | Lists events in an EC2 fleet (used with Databricks fleet clusters). |
| EC2 Fleet | Lists instances in an EC2 fleet (used with Databricks fleet clusters). |
| EC2 Fleet | Describes details of an EC2 fleet (used with Databricks fleet clusters). |
| Launch Template | Deletes a launch template (used with Databricks fleet clusters). |
| Launch Template | Describes details of launch template versions (used with Databricks fleet clusters). |
| NATGateway | Describes NATGateway to confirm that Databricks AWS instances have a route to the internet in the secure cluster connectivity architecture. |
| NetworkAcl | Confirms the correct Network ACL setup. |
| PrefixList | Gets a list of prefix list IDs to create an outbound security group rule that allows traffic from a VPC to access an AWS service through a gateway VPC endpoint. |
| Instance | Gets Reserved Instance pricing as the starting point for AWS spot instance pricing. |
| RouteTable | Confirms that route tables are set up correctly in the VPC. |
| SecurityGroup | Confirms that AWS security groups are set up correctly. |
| Instance | Describes spot instance. |
| SpotInstance | Describes spot instances. |
| Subnet | Confirms that subnets are setup correctly in the VPC. |
| Volume | List volumes. |
| VPC | Describes VPC attributes including but not limited to |
| VPC | Confirms that the Databricks workspace VPC was created. |
| Volume | Detaches an EBS volume from EC2 instances during cluster shutdown. |
| InstanceProfile | Disassociates an instance profile from an EC2 instance so that pool instances can be used by clusters with different instance profiles. |
| Launch Templates | Gets config of a launch template (used with Databricks fleet clusters). |
| EC2 Fleet | Modifies an EC2 fleet (used with Databricks fleet clusters). |
| Launch Templates | Modifies an existing launch template (used with Databricks fleet clusters). |
| InstanceProfile | Swaps one instance profile for another on an EC2 instance so that pool instances can be used by clusters with different instance profiles. |
| SpotInstance | Requests spot instances. |
| SecurityGroup | Updates Databricks-managed security groups if required |
| SecurityGroup | Updates security groups. |
| Instance | Launches AWS instances to create Spark Clusters. Also used to scale up an existing Spark cluster. |
| Instance | Terminates Spark EC2 nodes during cluster scale down or to terminate a Spark cluster. |
| ServiceLinkedRole | Sets up support for spot instances. |
| RolePolicy | Configures Databricks to use spot instances. |