Attach a cluster to one or more pools

To reduce cluster start time, you can designate predefined pools of idle instances to create worker nodes and the driver node. This is also called attaching the cluster to the pools. The cluster is created using instances in the pools. If an pool does not have sufficient idle resources to create the requested driver node or worker nodes, the pool expands by allocating new instances from the instance provider. When the cluster is terminated, the instances it used are returned to the pool and can be reused by a different cluster.

You can attach a different pool for the driver node and worker nodes, or attach the same pool for both.

Important

You must use a pool for both the driver node and worker nodes, or for neither. Otherwise, an error occurs and your cluster isn’t created. This prevents a situation where the driver node has to wait for worker nodes to be created, or vice versa.

Requirements

  • You must have permission to attach to each pool; see Pool access control.
  • You must add the following permissions to the access policy associated with the cross-account IAM role or keys used to create your AWS account:
{
  "Action": [
    "ec2:AssociateIamInstanceProfile",
    "ec2:DescribeIamInstanceProfileAssociations",
    "ec2:DisassociateIamInstanceProfile",
    "ec2:ReplaceIamInstanceProfileAssociation",
    "ec2:DeleteTags"
  ]
}

For the complete list of permissions and instructions on how to update your existing cross-account IAM role or keys, see Configure your AWS account (cross-account IAM role) or contact a workspace administrator.

Important

Without these permissions, the cluster fails to start.

Configure the cluster

To attach a cluster to a pool using the cluster creation UI, select the pool from the Driver Type or Worker Type drop-down when you configure the cluster. Available pools are listed at the top of each drop-down list. You can use the same pool or different pools for the driver node and worker nodes.

Select pool

If you use the Clusters API, you must specify driver_instance_pool_id for the driver node and instance_pool_id for the worker nodes.

Inherited configuration

When you attach a cluster to a pool, the following configuration properties are inherited from the pool:

  • Cluster node type.
  • Custom Cluster Tags: You can add additional custom tags for the cluster, and both the cluster-level tags and those inherited from pools are applied. You cannot add a cluster-specific custom tag with the same key name as a custom tag inherited from a pool (that is, you cannot override a custom tag that is inherited from the pool).
  • AWS configurations: Apart from Configure clusters, these cannot be modified at the cluster level.