Permissions in cross-account IAM roles
This article lists permissions in the cross-account IAM role and the purpose of each role.
The permissions are different based on how you configure your VPC.
IAM permissions for Databricks-managed VPCs
Databricks requires the following list of IAM permissions to operate and manage clusters in an effective manner. This configuration applies only to workspaces that use the default (Databricks-managed) VPC. To create the AWS cross-account role policy for use with the default Databricks-managed VPC, see Create an IAM role for workspace deployment.
The following table lists Databricks IAM cross-account role permissions in the default configuration, the resources that they control, and the purpose for each permission.
AWS IAM permission |
AWS resource |
Purpose |
---|---|---|
|
Elastic IP address |
Allocates an Elastic IP that is associated with the NAT Gateway used in secure cluster connectivity |
|
Network Interface |
Assigns a private IP to EC2 instance. |
|
DHCP |
Associates a set of DHCP options (or no DHCP options) with a VPC. |
|
InstanceProfile |
Associate an instance profile with a running EC2 instance. This allows a Databricks pool instance to be used by clusters with different instance profiles throughout its lifetime in the pool. |
|
RouteTable |
Associates a subnet with a route table. |
|
InternetGateway |
Attaches an Internet gateway to a VPC, enabling connectivity between the Internet and the VPC. This is currently required to connect to S3 buckets and update code for the workers and spark containers. |
|
EBS Volume |
Attaches volume for EBS auto-scaling. |
|
SecurityGroup |
Adds egress rules to the security groups if required. |
|
SecurityGroup |
Adds ingress rules to the security groups. |
|
SpotInstance |
Cancels spot instances. |
|
Dhcp |
Creates DHCP options. |
|
EC2 Fleet |
Creates an EC2 fleet (used with Databricks fleet clusters). |
|
InternetGateway |
Creates an Internet Gateway. |
|
Launch Template |
Creates a launch template (used with Databricks fleet clusters). |
|
Launch Template |
Creates a launch template version (used with Databricks fleet clusters). |
|
NatGateway |
Creates a NAT gateway. |
|
Route |
Creates routes during workspace setup. |
|
RouteTable |
Creates routes during workspace setup. |
|
SecurityGroup |
Creates security groups during initial setup |
|
Subnet |
Create subnets for the VPC during workspace setup. |
|
Tags |
Adds tags on Databricks resources. |
|
EBS Volume |
Creates volume. |
|
VPC |
Creates the Databricks-managed VPC. |
|
VPCEndpoint |
Creates VPC endpoints as part of configuring the VPC. |
|
DHCPOptions |
Deletes DHCPOptions |
|
EC2 Fleet |
Deletes an EC2 fleet (used with Databricks fleet clusters) |
|
InternetGateway |
Deletes Internet Gateway during workspace deletion. |
|
Launch Template |
Deletes a launch template and all its versions. |
|
Launch Template |
Deletes a version from a launch template (used with Databricks fleet clusters). |
|
NatGateway |
Deletes NAT gateway as needed to setup the secure cluster connectivity relay. |
|
Route |
Deletes routes. |
|
RouteTable |
Deletes route table. |
|
SecurityGroup |
Deletes security groups during workspace deletion. |
|
Subnet |
Deletes subnet. |
|
Tags |
Removes tags from cluster resources to allows Databricks pool instances to be reused by clusters with different tags. |
|
EBS Volume |
Deletes a volume for EBS auto-scaling. See this page. |
|
VPC |
Deletes the VPC when customers during workspace deletion. |
|
VPCEndpoints |
Deletes the VPC endpoints during workspace deletion. |
|
AvailabilityZones |
Gets a list of Availability Zones in a region so that Databricks can deploy resources in that zone. |
|
EC2 Fleet |
Lists events in an EC2 fleet (used with Databricks fleet clusters). |
|
EC2 Fleet |
Lists instances in an EC2 fleet (used with Databricks fleet clusters). |
|
EC2 Fleet |
Describes details of an EC2 fleet (used with Databricks fleet clusters). |
|
InstanceProfile |
Checks the current instance profile that is set on an EC2 instance so that the right profile is set on a Databricks pool instance before it’s reused by a cluster. |
|
Instance |
Confirms that Databricks AWS instances are healthy. |
|
Instance |
Confirms that Databricks AWS instances are healthy. |
|
InternetGateway |
Describes InternetGateway to confirm that Databricks AWS instances have a route to the internet. |
|
Launch Template |
Deletes a launch template (used with Databricks fleet clusters). |
|
Launch Template |
Describes details of launch template versions (used with Databricks fleet clusters). |
|
NATGateway |
Describes a NAT Gateway to confirm that Databricks AWS instances have a route to the internet in the secure cluster connectivity architecture. |
|
PrefixList |
Creates a prefix list ID to create an outbound security group rule that allows traffic from a VPC so that Databricks can access an AWS service through a gateway VPC endpoint. |
|
Instance |
Describes Reserved Instance pricing in support of AWS spot instance pricing. |
|
RouteTable |
Confirms that the route tables are set up correctly in the Databricks-managed VPC. |
|
SecurityGroup |
Confirms that AWS security groups are set up correctly. |
|
Instance |
Describes spot instances. |
|
SpotInstance |
Describes spot instances. |
|
Subnet |
Confirms that subnets are setup correctly in Databricks VPC. |
|
Volume |
Lists volumes. |
|
VPC |
Confirm that the workspace’s VPC was set up correctly. |
|
InternetGateway |
Detaches the Databricks created Internet Gateway during workspace deletion. |
|
InstanceProfile |
Disassociates an instance profile from an EC2 instance so that xDatabricks pool instances can be used by clusters with different instance profiles. |
|
RouteTable |
Detaches the Databricks created route table during workspace deletion. |
|
Launch Templates |
Gets config of a launch template (used with Databricks fleet clusters). |
|
Availability Zones |
Gets list of AZs with best spot capacity for given instance type(s). |
|
EC2 Fleet |
Modifies an EC2 fleet (used with Databricks fleet clusters). |
|
Launch Templates |
Modifies an existing launch template (used with Databricks fleet clusters). |
|
VPCAttribute |
Configures the Databricks-managed VPC. |
|
Address |
Detach the Databricks created address during workspace deletion. |
|
InstanceProfile |
Swaps one instance profile for another on an EC2 instance so that Databricks pool instances can be used by clusters with different instance profiles. |
|
SpotInstance |
Requests spot instances. |
|
SecurityGroup |
Updates Databricks-managed security groups if required. |
|
SecurityGroup |
Updates security groups. |
|
Instance |
Launches AWS instances to create Spark Clusters. Also leveraged during scaling up an existing Spark cluster. |
|
Instance |
Terminates Spark EC2 nodes during cluster scale down or to terminate a given Spark cluster. |
|
ServiceLinkedRole |
Sets up support for spot instances. |
|
RolePolicy |
Configures Databricks to use spot instances. |
IAM permissions for customer-managed VPC
If you use a customer-managed VPC, there’s a smaller set of permissions needed for the cross-account IAM role. This feature requires the Premium plan or above.
To create the AWS cross-account role policy for use with a customer-managed VPC, see Option 2: Customer-managed VPC with default restrictions policy.
The permissions can be scoped down further if needed. To create the AWS cross-account role policy for use with a customer-managed VPC with additional custom restrictions on resources, see Option 3: Customer-managed VPC with custom policy restrictions.
The following table lists Databricks IAM cross-account role permissions for a customer-managed VPC, the resources that they control, and the purpose for each permission.
AWS IAM permission |
AWS resource |
Purpose |
---|---|---|
|
InstanceProfile |
Associates an instance profile with a running EC2 instance so that a Databricks pool instance can be used by clusters with different instance profiles throughout its lifetime in the pool. |
|
Volume |
Attaches a volume. |
|
SecurityGroup |
Add egress rules to the security groups if required. |
|
SecurityGroup |
Adds ingress rules to the security groups. |
|
SpotInstance |
Cancels spot instances. |
|
EC2 Fleet |
Creates an EC2 fleet (used with Databricks fleet clusters). |
|
Launch Template |
Creates a launch template (used with Databricks fleet clusters). |
|
Tags |
Adds tags on Databricks resources. |
|
Volume |
Creates a volume. |
|
EC2 Fleet |
Deletes an EC2 fleet (used with Databricks fleet clusters) |
|
Launch Template |
Deletes a launch template and all its versions. |
|
Launch Template |
Deletes a version from a launch template (used with Databricks fleet clusters). |
|
Tags |
Removes tags from cluster resources so that Databricks pool instances can be reused by clusters with different tags. |
|
Volume |
Deletes a volume. |
|
AvailabilityZones |
Gets a list of Availability Zones in a region so that Databricks can deploy the resources in that zone. |
|
InstanceProfile |
Checks the current instance profile set on an EC2 instance to confirm that the right profile is set on a Databricks pool instance before it’s reused by a cluster. |
|
Instance |
Confirms that Databricks AWS instances are healthy. |
|
Instance |
Confirm that Databricks AWS instances are healthy. |
|
InternetGateway |
Describes InternetGateway to confirm that Databricks AWS instances have a route to the internet. |
|
EC2 Fleet |
Lists events in an EC2 fleet (used with Databricks fleet clusters). |
|
EC2 Fleet |
Lists instances in an EC2 fleet (used with Databricks fleet clusters). |
|
EC2 Fleet |
Describes details of an EC2 fleet (used with Databricks fleet clusters). |
|
Launch Template |
Deletes a launch template (used with Databricks fleet clusters). |
|
Launch Template |
Describes details of launch template versions (used with Databricks fleet clusters). |
|
NATGateway |
Describes NATGateway to confirm that Databricks AWS instances have a route to the internet in the secure cluster connectivity architecture. |
|
NetworkAcl |
Confirms the correct Network ACL setup. |
|
PrefixList |
Gets a list of prefix list IDs to create an outbound security group rule that allows traffic from a VPC to access an AWS service through a gateway VPC endpoint. |
|
Instance |
Gets Reserved Instance pricing as the starting point for AWS spot instance pricing. |
|
RouteTable |
Confirms that route tables are set up correctly in the VPC. |
|
SecurityGroup |
Confirms that AWS security groups are set up correctly. |
|
Instance |
Describes spot instance. |
|
SpotInstance |
Describes spot instances. |
|
Subnet |
Confirms that subnets are setup correctly in the VPC. |
|
Volume |
List volumes. |
|
VPC |
Describes VPC attributes including but not limited to |
|
VPC |
Confirms that the Databricks workspace VPC was created. |
|
Volume |
Detaches an EBS volume from EC2 instances during cluster shutdown. |
|
InstanceProfile |
Disassociates an instance profile from an EC2 instance so that pool instances can be used by clusters with different instance profiles. |
|
Launch Templates |
Gets config of a launch template (used with Databricks fleet clusters). |
|
EC2 Fleet |
Modifies an EC2 fleet (used with Databricks fleet clusters). |
|
Launch Templates |
Modifies an existing launch template (used with Databricks fleet clusters). |
|
InstanceProfile |
Swaps one instance profile for another on an EC2 instance so that pool instances can be used by clusters with different instance profiles. |
|
SpotInstance |
Requests spot instances. |
|
SecurityGroup |
Updates Databricks-managed security groups if required |
|
SecurityGroup |
Updates security groups. |
|
Instance |
Launches AWS instances to create Spark Clusters. Also used to scale up an existing Spark cluster. |
|
Instance |
Terminates Spark EC2 nodes during cluster scale down or to terminate a Spark cluster. |
|
ServiceLinkedRole |
Sets up support for spot instances. |
|
RolePolicy |
Configures Databricks to use spot instances. |