Launch a cluster with a fleet instance type

This article explains what the fleet instance type is and how to launch clusters using fleet instance types.

Note

If your workspace was created before May 2023, an account admin might need to update your workspace’s IAM role access policy to allow permissions for fleet instance types. See Update IAM permissions.

What are fleet instance types?

A fleet instance type is a variable instance type that automatically resolves to the best available instance type of the same size.

For example, if you select the fleet instance type m-fleet.xlarge, your node will resolve to whichever .xlarge, general purpose instance type has the best spot capacity and price at that moment. The instance type your cluster resolves to will always have the same memory and number of cores as the fleet instance type you chose.

Improved functionality

Fleet clusters maximize availability in the following ways:

  • Improved Auto-AZ with Spot Placement Score API: Fleet instance types use AWS’s Spot Placement Score API to choose the best and most likely to succeed availability zone for your cluster at startup time.

  • Auto-AZ available with instance pools: You can select auto as the availability zone for your pool if the pool uses a fleet instance type.

  • High availability zone (HA): Clusters with a fleet instance type for both driver and worker can have their zone_id set to HA. This allows the cluster’s driver and workers to be allocated from any mix of availability zones in the region, wherever capacity is best.

    Note

    Having a single cluster’s driver and workers spread out across multiple availability zones can incur extra inter-zone data egress fees.

Update IAM role permissions to use fleet instance types

If your workspace was created before May 2023, an account admin might need to update your workspace’s IAM role access policy to allow permissions for Fleet clusters.

The account admin should update the deployment IAM role access policy with the latest permissions as described in Create an access policy.

Create a cluster that uses a fleet instance type

To create a compute resourcing using AWS Fleet:

  1. Select or create a compute resource.

  2. Select one of the fleet instance types for the Worker type, Driver type, or both (Databricks recommends both).

  3. Click Create Cluster.

Limitations

  • The Max spot price setting under Advanced options has no effect when the worker node type is set to a fleet instance type. This is because there is no single on-demand instance to use as a reference point for the spot price.

  • Fleet clusters do not support Graviton or GPU instances.

  • A small percentage of older workspaces do not yet support fleet instance types. If this is the case for your workspace, you’ll see an error indicating this when attempting to create a cluster or instance pool using a fleet instance type. We’re working to bring support to these remaining workspaces.