Instance Pool

View as Markdown

GetGET /api/2.0/instance-pools/getListGET /api/2.0/instance-pools/listCreatePOST /api/2.0/instance-pools/createUpdatePOST /api/2.0/instance-pools/editDeletePOST /api/2.0/instance-pools/delete

InstancePoolAndStats object

statsobject

Usage statistics about the instance pool.

Show child attributesHide child attributes

used_countint32

Number of active instances in the pool that are part of a cluster.

idle_countint32

Number of active instances in the pool that are NOT part of a cluster.

pending_used_countint32

Number of pending instances in the pool that are part of a cluster.

pending_idle_countint32

Number of pending instances in the pool that are NOT part of a cluster.

statusobject

Status of failed pending instances in the pool.

Show child attributesHide child attributes

pending_instance_errorsarray of object

List of error messages for the failed pending instances. The pending_instance_errors follows FIFO with maximum length of the min_idle of the pool. The pending_instance_errors is emptied once the number of exiting available instances reaches the min_idle of the pool.

Show child attributesHide child attributes

instance_idstring

messagestring

instance_pool_idstring

Canonical unique identifier for the pool.

default_tagsobject

Tags that are added by <Databricks> regardless of any custom_tags, including:

• Vendor: <Databricks>

• InstancePoolCreator: <user_id_of_creator>

• InstancePoolName: <name_of_pool>

• InstancePoolId: <id_of_pool>

statestring

Current state of the instance pool.

Values: ACTIVE, STOPPED, DELETED

instance_pool_namestring

Pool name requested by the user. Pool name must be unique. Length must be between 1 and 100 characters.

Example: my-pool

min_idle_instancesint32

Minimum number of idle instances to keep in the instance pool

Example: 5

max_capacityint32

Maximum number of outstanding instances to keep in the pool, including both instances used by clusters and idle instances. Clusters that require further instance provisioning will fail during upsize requests.

Example: 200

aws_attributesobject

Attributes related to instance pools running on Amazon Web Services. If not specified at pool creation, a set of default values will be used.

Show child attributesHide child attributes

availabilitystring

Availability type used for the spot nodes.

Values: SPOT, ON_DEMAND, SPOT_WITH_FALLBACK

zone_idstring

Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the <Databricks> deployment. For example, "us-west-2a" is not a valid zone id if the <Databricks> deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, a default zone will be used. The list of available zones as well as the default value can be found by using the List Zones method.

spot_bid_price_percentint32

Calculates the bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new r3.xlarge spot instance, then the bid price is half of the price of on-demand r3.xlarge instances. Similarly, if this field is set to 200, the bid price is twice the price of on-demand r3.xlarge instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.

Default: 100

instance_profile_arnstringBeta

All AWS instances belonging to the instance pool will have this instance profile. If omitted, instances will initially be launched with the workspace's default instance profile. If defined, clusters that use the pool will inherit the instance profile, and must not specify their own instance profile on cluster creation or update. If the pool does not specify an instance profile, clusters using the pool may specify any instance profile. The instance profile must have previously been added to the <Databricks> environment by an account administrator.

This feature may only be available to certain customer plans.

node_type_idstring

This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

custom_tagsobject

Additional tags for pool resources. <Databricks> will tag all pool resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

• Currently, <Databricks> allows at most 45 custom tags

idle_instance_autotermination_minutesint32

Automatically terminates the extra instances in the pool cache after they are inactive for this time in minutes if min_idle_instances requirement is already met. If not set, the extra pool instances will be automatically terminated after a default timeout. If specified, the threshold must be between 0 and 10000 minutes. Users can also set this value to 0 to instantly remove idle instances from the cache if min cache size could still hold.

Example: 30

enable_elastic_diskboolean

Autoscaling Local Storage: when enabled, this instances in this pool will dynamically acquire additional disk space when its Spark workers are running low on disk space. In AWS, this feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

disk_specobject

Defines the specification of the disks that will be attached to all spark containers.

Show child attributesHide child attributes

disk_typeobject

The type of disks that will be launched with this cluster.

Show child attributesHide child attributes

ebs_volume_typestring

Values: GENERAL_PURPOSE_SSD, THROUGHPUT_OPTIMIZED_HDD

azure_disk_volume_typestring

Values: PREMIUM_LRS, STANDARD_LRS

disk_countint32

The number of disks launched for each instance:

• This feature is only enabled for supported node types.
• Users can choose up to the limit of the disks supported by the node type.
• For node types with no OS disk, at least one disk must be specified; otherwise, cluster creation will fail.

If disks are attached, <Databricks> will configure Spark to use only the disks for scratch storage, because heterogenously sized scratch devices can lead to inefficient disk utilization. If no disks are attached, <Databricks> will configure Spark to use instance store disks.

Note: If disks are specified, then the Spark configuration spark.local.dir will be overridden.

Disks will be mounted at:

• For AWS: /ebs0, /ebs1, and etc.
• For Azure: /remote_volume0, /remote_volume1, and etc.

Default: 0

disk_sizeint32

The size of each disk (in GiB) launched for each instance. Values must fall into the supported range for a particular instance type.

For AWS:

• General Purpose SSD: 100 - 4096 GiB
• Throughput Optimized HDD: 500 - 4096 GiB

For Azure:

• Premium LRS (SSD): 1 - 1023 GiB
• Standard LRS (HDD): 1- 1023 GiB

disk_iopsint32

disk_throughputint32

preloaded_docker_imagesarray of object

Custom Docker Image BYOC

Show child attributesHide child attributes

urlstring

URL of the docker image.

basic_authobject

Basic auth with username and password

Show child attributesHide child attributes

usernamestring

Name of the user

passwordstring

Password of the user

preloaded_spark_versionsarray of string

A list containing at most one preloaded Spark image version for the pool. Pool-backed clusters started with the preloaded Spark version will start faster. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

azure_attributesobject

Attributes related to instance pools running on Azure. If not specified at pool creation, a set of default values will be used.

Show child attributesHide child attributes

availabilitystring

Availability type used for the spot nodes.

Default: ON_DEMAND_AZURE

Values: SPOT_AZURE, ON_DEMAND_AZURE, SPOT_WITH_FALLBACK_AZURE

spot_bid_max_pricedouble

With variable pricing, you have option to set a max price, in US dollars (USD) For example, the value 2 would be a max price of $2.00 USD per hour. If you set the max price to be -1, the VM won't be evicted based on price. The price for the VM will be the current price for spot or the price for a standard VM, which ever is less, as long as there is capacity and quota available.

Default: -1

capacity_reservation_groupstringGA

The Azure capacity reservation group resource ID to use for launching VMs in this pool. When specified, VMs will be launched using the provided capacity reservation.

NOTE: Omitting this field will clear any existing configured capacity reservation group on the pool.

Capacity reservations can only be specified when the workspace uses injected vnet (i.e. customer defined vnet not managed by databricks). Ensure the databricks-login-prod Enterprise Application is granted the following four permissions:

1. Microsoft.Compute/capacityReservationGroups/read
2. Microsoft.Compute/capacityReservationGroups/deploy/action
3. Microsoft.Compute/capacityReservationGroups/capacityReservations/read
4. Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action

Format: /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}

gcp_attributesobject

Attributes related to instance pools running on Google Cloud Platform. If not specified at pool creation, a set of default values will be used.

Show child attributesHide child attributes

gcp_availabilitystring

Default: ON_DEMAND_GCP

Values: PREEMPTIBLE_GCP, ON_DEMAND_GCP, PREEMPTIBLE_WITH_FALLBACK_GCP

local_ssd_countint32

If provided, each node in the instance pool will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to GCP documentation for the supported number of local SSDs for each instance type.

zone_idstring

Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west1-a". The provided availability zone must be in the same region as the <Databricks> workspace. For example, "us-west1-a" is not a valid zone id if the <Databricks> workspace resides in the "us-east1" region. This is an optional field at instance pool creation, and if not specified, a default zone will be used.

This field can be one of the following:

• "HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region
• A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones (e.g. "us-west1-a").

If empty, <Databricks> picks an availability zone to schedule the cluster on.

node_type_flexibilityobject

Flexible node type configuration for the pool.

Show child attributesHide child attributes

alternate_node_type_idsarray of string

A list of node type IDs to use as fallbacks when the primary node type is unavailable.

remote_disk_throughputint32

If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED types.

total_initial_remote_disk_sizeint32

If set, what the total initial volume size (in GB) of the remote disks should be. Currently only supported for GCP HYPERDISK_BALANCED types.

View as Markdown

The InstancePoolAndStats object

{
  "stats": {
    "used_count": 0,
    "idle_count": 0,
    "pending_used_count": 0,
    "pending_idle_count": 0
  },
  "status": {
    "pending_instance_errors": [
      {}
    ]
  },
  "instance_pool_id": "string",
  "default_tags": {},
  "state": "string",
  "instance_pool_name": "my-pool",
  "min_idle_instances": 5,
  "max_capacity": 200,
  "aws_attributes": {
    "availability": "string",
    "zone_id": "string",
    "spot_bid_price_percent": 0,
    "instance_profile_arn": "string"
  },
  "node_type_id": "string",
  "custom_tags": {},
  "idle_instance_autotermination_minutes": 30,
  "enable_elastic_disk": true,
  "disk_spec": {
    "disk_type": {},
    "disk_count": 0,
    "disk_size": 0,
    "disk_iops": 0,
    "disk_throughput": 0
  },
  "preloaded_docker_images": [
    {
      "url": "string",
      "basic_auth": {}
    }
  ],
  "preloaded_spark_versions": [
    "string"
  ],
  "azure_attributes": {
    "availability": "string",
    "spot_bid_max_price": 0,
    "capacity_reservation_group": "string"
  },
  "gcp_attributes": {
    "gcp_availability": "string",
    "local_ssd_count": 0,
    "zone_id": "string"
  },
  "node_type_flexibility": {
    "alternate_node_type_ids": [
      "string"
    ]
  },
  "remote_disk_throughput": 0,
  "total_initial_remote_disk_size": 0
}

Get GA

GET /api/2.0/instance-pools/get

Retrieve the information for an instance pool based on its identifier.

API scopes: instance-pools

Parameters

instance_pool_idstringquery

The canonical unique identifier for the instance pool.

Example: 1234-567890-fetch12-pool-A3BcdEFg

Response

statsobject

Usage statistics about the instance pool.

Show child attributesHide child attributes

used_countint32

Number of active instances in the pool that are part of a cluster.

idle_countint32

Number of active instances in the pool that are NOT part of a cluster.

pending_used_countint32

Number of pending instances in the pool that are part of a cluster.

pending_idle_countint32

Number of pending instances in the pool that are NOT part of a cluster.

statusobject

Status of failed pending instances in the pool.

Show child attributesHide child attributes

pending_instance_errorsarray of object

List of error messages for the failed pending instances. The pending_instance_errors follows FIFO with maximum length of the min_idle of the pool. The pending_instance_errors is emptied once the number of exiting available instances reaches the min_idle of the pool.

Show child attributesHide child attributes

instance_idstring

messagestring

instance_pool_idstring

Canonical unique identifier for the pool.

default_tagsobject

Tags that are added by <Databricks> regardless of any custom_tags, including:

• Vendor: <Databricks>

• InstancePoolCreator: <user_id_of_creator>

• InstancePoolName: <name_of_pool>

• InstancePoolId: <id_of_pool>

statestring

Current state of the instance pool.

Values: ACTIVE, STOPPED, DELETED

instance_pool_namestring

Pool name requested by the user. Pool name must be unique. Length must be between 1 and 100 characters.

Example: my-pool

min_idle_instancesint32

Minimum number of idle instances to keep in the instance pool

Example: 5

max_capacityint32

Maximum number of outstanding instances to keep in the pool, including both instances used by clusters and idle instances. Clusters that require further instance provisioning will fail during upsize requests.

Example: 200

aws_attributesobject

Attributes related to instance pools running on Amazon Web Services. If not specified at pool creation, a set of default values will be used.

Show child attributesHide child attributes

availabilitystring

Availability type used for the spot nodes.

Values: SPOT, ON_DEMAND, SPOT_WITH_FALLBACK

zone_idstring

Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the <Databricks> deployment. For example, "us-west-2a" is not a valid zone id if the <Databricks> deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, a default zone will be used. The list of available zones as well as the default value can be found by using the List Zones method.

spot_bid_price_percentint32

Calculates the bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new r3.xlarge spot instance, then the bid price is half of the price of on-demand r3.xlarge instances. Similarly, if this field is set to 200, the bid price is twice the price of on-demand r3.xlarge instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.

Default: 100

instance_profile_arnstringBeta

All AWS instances belonging to the instance pool will have this instance profile. If omitted, instances will initially be launched with the workspace's default instance profile. If defined, clusters that use the pool will inherit the instance profile, and must not specify their own instance profile on cluster creation or update. If the pool does not specify an instance profile, clusters using the pool may specify any instance profile. The instance profile must have previously been added to the <Databricks> environment by an account administrator.

This feature may only be available to certain customer plans.

node_type_idstring

This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

custom_tagsobject

Additional tags for pool resources. <Databricks> will tag all pool resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

• Currently, <Databricks> allows at most 45 custom tags

idle_instance_autotermination_minutesint32

Automatically terminates the extra instances in the pool cache after they are inactive for this time in minutes if min_idle_instances requirement is already met. If not set, the extra pool instances will be automatically terminated after a default timeout. If specified, the threshold must be between 0 and 10000 minutes. Users can also set this value to 0 to instantly remove idle instances from the cache if min cache size could still hold.

Example: 30

enable_elastic_diskboolean

Autoscaling Local Storage: when enabled, this instances in this pool will dynamically acquire additional disk space when its Spark workers are running low on disk space. In AWS, this feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

disk_specobject

Defines the specification of the disks that will be attached to all spark containers.

Show child attributesHide child attributes

disk_typeobject

The type of disks that will be launched with this cluster.

Show child attributesHide child attributes

ebs_volume_typestringRequired

Values: GENERAL_PURPOSE_SSD, THROUGHPUT_OPTIMIZED_HDD

azure_disk_volume_typestringRequired

Values: PREMIUM_LRS, STANDARD_LRS

disk_countint32

The number of disks launched for each instance:

• This feature is only enabled for supported node types.
• Users can choose up to the limit of the disks supported by the node type.
• For node types with no OS disk, at least one disk must be specified; otherwise, cluster creation will fail.

If disks are attached, <Databricks> will configure Spark to use only the disks for scratch storage, because heterogenously sized scratch devices can lead to inefficient disk utilization. If no disks are attached, <Databricks> will configure Spark to use instance store disks.

Note: If disks are specified, then the Spark configuration spark.local.dir will be overridden.

Disks will be mounted at:

• For AWS: /ebs0, /ebs1, and etc.
• For Azure: /remote_volume0, /remote_volume1, and etc.

Default: 0

disk_sizeint32

The size of each disk (in GiB) launched for each instance. Values must fall into the supported range for a particular instance type.

For AWS:

• General Purpose SSD: 100 - 4096 GiB
• Throughput Optimized HDD: 500 - 4096 GiB

For Azure:

• Premium LRS (SSD): 1 - 1023 GiB
• Standard LRS (HDD): 1- 1023 GiB

disk_iopsint32

disk_throughputint32

preloaded_docker_imagesarray of object

Custom Docker Image BYOC

Show child attributesHide child attributes

urlstring

URL of the docker image.

basic_authobjectRequired

Basic auth with username and password

Show child attributesHide child attributes

usernamestring

Name of the user

passwordstring

Password of the user

preloaded_spark_versionsarray of string

A list containing at most one preloaded Spark image version for the pool. Pool-backed clusters started with the preloaded Spark version will start faster. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

azure_attributesobject

Attributes related to instance pools running on Azure. If not specified at pool creation, a set of default values will be used.

Show child attributesHide child attributes

availabilitystring

Availability type used for the spot nodes.

Default: ON_DEMAND_AZURE

Values: SPOT_AZURE, ON_DEMAND_AZURE, SPOT_WITH_FALLBACK_AZURE

spot_bid_max_pricedouble

With variable pricing, you have option to set a max price, in US dollars (USD) For example, the value 2 would be a max price of $2.00 USD per hour. If you set the max price to be -1, the VM won't be evicted based on price. The price for the VM will be the current price for spot or the price for a standard VM, which ever is less, as long as there is capacity and quota available.

Default: -1

capacity_reservation_groupstring

The Azure capacity reservation group resource ID to use for launching VMs in this pool. When specified, VMs will be launched using the provided capacity reservation.

NOTE: Omitting this field will clear any existing configured capacity reservation group on the pool.

Capacity reservations can only be specified when the workspace uses injected vnet (i.e. customer defined vnet not managed by databricks). Ensure the databricks-login-prod Enterprise Application is granted the following four permissions:

1. Microsoft.Compute/capacityReservationGroups/read
2. Microsoft.Compute/capacityReservationGroups/deploy/action
3. Microsoft.Compute/capacityReservationGroups/capacityReservations/read
4. Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action

Format: /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}

gcp_attributesobject

Attributes related to instance pools running on Google Cloud Platform. If not specified at pool creation, a set of default values will be used.

Show child attributesHide child attributes

gcp_availabilitystring

Default: ON_DEMAND_GCP

Values: PREEMPTIBLE_GCP, ON_DEMAND_GCP, PREEMPTIBLE_WITH_FALLBACK_GCP

local_ssd_countint32

If provided, each node in the instance pool will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to GCP documentation for the supported number of local SSDs for each instance type.

zone_idstring

Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west1-a". The provided availability zone must be in the same region as the <Databricks> workspace. For example, "us-west1-a" is not a valid zone id if the <Databricks> workspace resides in the "us-east1" region. This is an optional field at instance pool creation, and if not specified, a default zone will be used.

This field can be one of the following:

• "HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region
• A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones (e.g. "us-west1-a").

If empty, <Databricks> picks an availability zone to schedule the cluster on.

node_type_flexibilityobject

Flexible node type configuration for the pool.

Show child attributesHide child attributes

alternate_node_type_idsarray of string

A list of node type IDs to use as fallbacks when the primary node type is unavailable.

remote_disk_throughputint32

If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED types.

total_initial_remote_disk_sizeint32

If set, what the total initial volume size (in GB) of the remote disks should be. Currently only supported for GCP HYPERDISK_BALANCED types.

View as Markdown

Response

{
  "stats": {
    "used_count": 0,
    "idle_count": 0,
    "pending_used_count": 0,
    "pending_idle_count": 0
  },
  "status": {
    "pending_instance_errors": [
      {}
    ]
  },
  "instance_pool_id": "string",
  "default_tags": {},
  "state": "string",
  "instance_pool_name": "my-pool",
  "min_idle_instances": 5,
  "max_capacity": 200,
  "aws_attributes": {
    "availability": "string",
    "zone_id": "string",
    "spot_bid_price_percent": 0,
    "instance_profile_arn": "string"
  },
  "node_type_id": "string",
  "custom_tags": {},
  "idle_instance_autotermination_minutes": 30,
  "enable_elastic_disk": true,
  "disk_spec": {
    "disk_type": {},
    "disk_count": 0,
    "disk_size": 0,
    "disk_iops": 0,
    "disk_throughput": 0
  },
  "preloaded_docker_images": [
    {
      "url": "string",
      "basic_auth": {}
    }
  ],
  "preloaded_spark_versions": [
    "string"
  ],
  "azure_attributes": {
    "availability": "string",
    "spot_bid_max_price": 0,
    "capacity_reservation_group": "string"
  },
  "gcp_attributes": {
    "gcp_availability": "string",
    "local_ssd_count": 0,
    "zone_id": "string"
  },
  "node_type_flexibility": {
    "alternate_node_type_ids": [
      "string"
    ]
  },
  "remote_disk_throughput": 0,
  "total_initial_remote_disk_size": 0
}

List GA

GET /api/2.0/instance-pools/list

Gets a list of instance pools with their statistics.

API scopes: instance-pools

Response

Returns a list of InstancePoolAndStats objects.

View as Markdown

Response

{
  "instance_pools": [
    {
      "stats": {},
      "status": {},
      "instance_pool_id": "string",
      "default_tags": {},
      "state": "string",
      "instance_pool_name": "my-pool",
      "min_idle_instances": 5,
      "max_capacity": 200,
      "aws_attributes": {},
      "node_type_id": "string",
      "custom_tags": {},
      "idle_instance_autotermination_minutes": 30,
      "enable_elastic_disk": true,
      "disk_spec": {},
      "preloaded_docker_images": [
        {}
      ],
      "preloaded_spark_versions": [
        "string"
      ],
      "azure_attributes": {},
      "gcp_attributes": {},
      "node_type_flexibility": {},
      "remote_disk_throughput": 0,
      "total_initial_remote_disk_size": 0
    }
  ]
}

Create GA

POST /api/2.0/instance-pools/create

Creates a new instance pool using idle and ready-to-use cloud instances.

API scopes: instance-pools

Request body

instance_pool_namestring

Pool name requested by the user. Pool name must be unique. Length must be between 1 and 100 characters.

Example: my-pool

min_idle_instancesint32

Minimum number of idle instances to keep in the instance pool

Example: 5

max_capacityint32

Maximum number of outstanding instances to keep in the pool, including both instances used by clusters and idle instances. Clusters that require further instance provisioning will fail during upsize requests.

Example: 200

aws_attributesobject

Attributes related to instance pools running on Amazon Web Services. If not specified at pool creation, a set of default values will be used.

Show child attributesHide child attributes

availabilitystring

Availability type used for the spot nodes.

Values: SPOT, ON_DEMAND, SPOT_WITH_FALLBACK

zone_idstring

Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the <Databricks> deployment. For example, "us-west-2a" is not a valid zone id if the <Databricks> deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, a default zone will be used. The list of available zones as well as the default value can be found by using the List Zones method.

spot_bid_price_percentint32

Calculates the bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new r3.xlarge spot instance, then the bid price is half of the price of on-demand r3.xlarge instances. Similarly, if this field is set to 200, the bid price is twice the price of on-demand r3.xlarge instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.

Default: 100

instance_profile_arnstringBeta

All AWS instances belonging to the instance pool will have this instance profile. If omitted, instances will initially be launched with the workspace's default instance profile. If defined, clusters that use the pool will inherit the instance profile, and must not specify their own instance profile on cluster creation or update. If the pool does not specify an instance profile, clusters using the pool may specify any instance profile. The instance profile must have previously been added to the <Databricks> environment by an account administrator.

This feature may only be available to certain customer plans.

node_type_idstring

This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

custom_tagsobject

Additional tags for pool resources. <Databricks> will tag all pool resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

• Currently, <Databricks> allows at most 45 custom tags

idle_instance_autotermination_minutesint32

Automatically terminates the extra instances in the pool cache after they are inactive for this time in minutes if min_idle_instances requirement is already met. If not set, the extra pool instances will be automatically terminated after a default timeout. If specified, the threshold must be between 0 and 10000 minutes. Users can also set this value to 0 to instantly remove idle instances from the cache if min cache size could still hold.

Example: 30

enable_elastic_diskboolean

Autoscaling Local Storage: when enabled, this instances in this pool will dynamically acquire additional disk space when its Spark workers are running low on disk space. In AWS, this feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

disk_specobject

Defines the specification of the disks that will be attached to all spark containers.

Show child attributesHide child attributes

disk_typeobject

The type of disks that will be launched with this cluster.

Show child attributesHide child attributes

ebs_volume_typestringRequired

Values: GENERAL_PURPOSE_SSD, THROUGHPUT_OPTIMIZED_HDD

azure_disk_volume_typestringRequired

Values: PREMIUM_LRS, STANDARD_LRS

disk_countint32

The number of disks launched for each instance:

• This feature is only enabled for supported node types.
• Users can choose up to the limit of the disks supported by the node type.
• For node types with no OS disk, at least one disk must be specified; otherwise, cluster creation will fail.

If disks are attached, <Databricks> will configure Spark to use only the disks for scratch storage, because heterogenously sized scratch devices can lead to inefficient disk utilization. If no disks are attached, <Databricks> will configure Spark to use instance store disks.

Note: If disks are specified, then the Spark configuration spark.local.dir will be overridden.

Disks will be mounted at:

• For AWS: /ebs0, /ebs1, and etc.
• For Azure: /remote_volume0, /remote_volume1, and etc.

Default: 0

disk_sizeint32

The size of each disk (in GiB) launched for each instance. Values must fall into the supported range for a particular instance type.

For AWS:

• General Purpose SSD: 100 - 4096 GiB
• Throughput Optimized HDD: 500 - 4096 GiB

For Azure:

• Premium LRS (SSD): 1 - 1023 GiB
• Standard LRS (HDD): 1- 1023 GiB

disk_iopsint32

disk_throughputint32

preloaded_docker_imagesarray of object

Custom Docker Image BYOC

Show child attributesHide child attributes

urlstring

URL of the docker image.

basic_authobjectRequired

Basic auth with username and password

Show child attributesHide child attributes

usernamestring

Name of the user

passwordstring

Password of the user

preloaded_spark_versionsarray of string

A list containing at most one preloaded Spark image version for the pool. Pool-backed clusters started with the preloaded Spark version will start faster. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

azure_attributesobject

Attributes related to instance pools running on Azure. If not specified at pool creation, a set of default values will be used.

Show child attributesHide child attributes

availabilitystring

Availability type used for the spot nodes.

Default: ON_DEMAND_AZURE

Values: SPOT_AZURE, ON_DEMAND_AZURE, SPOT_WITH_FALLBACK_AZURE

spot_bid_max_pricedouble

With variable pricing, you have option to set a max price, in US dollars (USD) For example, the value 2 would be a max price of $2.00 USD per hour. If you set the max price to be -1, the VM won't be evicted based on price. The price for the VM will be the current price for spot or the price for a standard VM, which ever is less, as long as there is capacity and quota available.

Default: -1

capacity_reservation_groupstring

The Azure capacity reservation group resource ID to use for launching VMs in this pool. When specified, VMs will be launched using the provided capacity reservation.

NOTE: Omitting this field will clear any existing configured capacity reservation group on the pool.

Capacity reservations can only be specified when the workspace uses injected vnet (i.e. customer defined vnet not managed by databricks). Ensure the databricks-login-prod Enterprise Application is granted the following four permissions:

1. Microsoft.Compute/capacityReservationGroups/read
2. Microsoft.Compute/capacityReservationGroups/deploy/action
3. Microsoft.Compute/capacityReservationGroups/capacityReservations/read
4. Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action

Format: /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}

gcp_attributesobject

Attributes related to instance pools running on Google Cloud Platform. If not specified at pool creation, a set of default values will be used.

Show child attributesHide child attributes

gcp_availabilitystring

Default: ON_DEMAND_GCP

Values: PREEMPTIBLE_GCP, ON_DEMAND_GCP, PREEMPTIBLE_WITH_FALLBACK_GCP

local_ssd_countint32

If provided, each node in the instance pool will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to GCP documentation for the supported number of local SSDs for each instance type.

zone_idstring

Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west1-a". The provided availability zone must be in the same region as the <Databricks> workspace. For example, "us-west1-a" is not a valid zone id if the <Databricks> workspace resides in the "us-east1" region. This is an optional field at instance pool creation, and if not specified, a default zone will be used.

This field can be one of the following:

• "HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region
• A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones (e.g. "us-west1-a").

If empty, <Databricks> picks an availability zone to schedule the cluster on.

node_type_flexibilityobject

Flexible node type configuration for the pool.

Show child attributesHide child attributes

alternate_node_type_idsarray of string

A list of node type IDs to use as fallbacks when the primary node type is unavailable.

remote_disk_throughputint32

If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED types.

total_initial_remote_disk_sizeint32

If set, what the total initial volume size (in GB) of the remote disks should be. Currently only supported for GCP HYPERDISK_BALANCED types.

Response

instance_pool_idstring

The ID of the created instance pool.

Example: 1234-567890-fetch12-pool-A3BcdEFg

View as Markdown

Request

{
  "instance_pool_name": "my-pool",
  "min_idle_instances": 5,
  "max_capacity": 200,
  "aws_attributes": {
    "availability": "string",
    "zone_id": "string",
    "spot_bid_price_percent": 0,
    "instance_profile_arn": "string"
  },
  "node_type_id": "string",
  "custom_tags": {},
  "idle_instance_autotermination_minutes": 30,
  "enable_elastic_disk": true,
  "disk_spec": {
    "disk_type": {},
    "disk_count": 0,
    "disk_size": 0,
    "disk_iops": 0,
    "disk_throughput": 0
  },
  "preloaded_docker_images": [
    {
      "url": "string",
      "basic_auth": {}
    }
  ],
  "preloaded_spark_versions": [
    "string"
  ],
  "azure_attributes": {
    "availability": "string",
    "spot_bid_max_price": 0,
    "capacity_reservation_group": "string"
  },
  "gcp_attributes": {
    "gcp_availability": "string",
    "local_ssd_count": 0,
    "zone_id": "string"
  },
  "node_type_flexibility": {
    "alternate_node_type_ids": [
      "string"
    ]
  },
  "remote_disk_throughput": 0,
  "total_initial_remote_disk_size": 0
}

Response

{
  "instance_pool_id": "1234-567890-fetch12-pool-A3BcdEFg"
}

Update GA

POST /api/2.0/instance-pools/edit

Modifies the configuration of an existing instance pool.

API scopes: instance-pools

Request body

instance_pool_idstring

Instance pool ID

Example: 1234-567890-fetch12-pool-A3BcdEFg

instance_pool_namestring

Pool name requested by the user. Pool name must be unique. Length must be between 1 and 100 characters.

Example: my-pool

min_idle_instancesint32

Minimum number of idle instances to keep in the instance pool

Example: 5

max_capacityint32

Maximum number of outstanding instances to keep in the pool, including both instances used by clusters and idle instances. Clusters that require further instance provisioning will fail during upsize requests.

Example: 200

aws_attributesobject

Attributes related to instance pools running on Amazon Web Services. If not specified at pool creation, a set of default values will be used.

Show child attributesHide child attributes

availabilitystring

Availability type used for the spot nodes.

Values: SPOT, ON_DEMAND, SPOT_WITH_FALLBACK

zone_idstring

Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the <Databricks> deployment. For example, "us-west-2a" is not a valid zone id if the <Databricks> deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, a default zone will be used. The list of available zones as well as the default value can be found by using the List Zones method.

spot_bid_price_percentint32

Calculates the bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new r3.xlarge spot instance, then the bid price is half of the price of on-demand r3.xlarge instances. Similarly, if this field is set to 200, the bid price is twice the price of on-demand r3.xlarge instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.

Default: 100

instance_profile_arnstringBeta

All AWS instances belonging to the instance pool will have this instance profile. If omitted, instances will initially be launched with the workspace's default instance profile. If defined, clusters that use the pool will inherit the instance profile, and must not specify their own instance profile on cluster creation or update. If the pool does not specify an instance profile, clusters using the pool may specify any instance profile. The instance profile must have previously been added to the <Databricks> environment by an account administrator.

This feature may only be available to certain customer plans.

node_type_idstring

This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

custom_tagsobject

Additional tags for pool resources. <Databricks> will tag all pool resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

• Currently, <Databricks> allows at most 45 custom tags

idle_instance_autotermination_minutesint32

Automatically terminates the extra instances in the pool cache after they are inactive for this time in minutes if min_idle_instances requirement is already met. If not set, the extra pool instances will be automatically terminated after a default timeout. If specified, the threshold must be between 0 and 10000 minutes. Users can also set this value to 0 to instantly remove idle instances from the cache if min cache size could still hold.

Example: 30

enable_elastic_diskboolean

Autoscaling Local Storage: when enabled, this instances in this pool will dynamically acquire additional disk space when its Spark workers are running low on disk space. In AWS, this feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

disk_specobject

Defines the specification of the disks that will be attached to all spark containers.

Show child attributesHide child attributes

disk_typeobject

The type of disks that will be launched with this cluster.

Show child attributesHide child attributes

ebs_volume_typestringRequired

Values: GENERAL_PURPOSE_SSD, THROUGHPUT_OPTIMIZED_HDD

azure_disk_volume_typestringRequired

Values: PREMIUM_LRS, STANDARD_LRS

disk_countint32

The number of disks launched for each instance:

• This feature is only enabled for supported node types.
• Users can choose up to the limit of the disks supported by the node type.
• For node types with no OS disk, at least one disk must be specified; otherwise, cluster creation will fail.

If disks are attached, <Databricks> will configure Spark to use only the disks for scratch storage, because heterogenously sized scratch devices can lead to inefficient disk utilization. If no disks are attached, <Databricks> will configure Spark to use instance store disks.

Note: If disks are specified, then the Spark configuration spark.local.dir will be overridden.

Disks will be mounted at:

• For AWS: /ebs0, /ebs1, and etc.
• For Azure: /remote_volume0, /remote_volume1, and etc.

Default: 0

disk_sizeint32

The size of each disk (in GiB) launched for each instance. Values must fall into the supported range for a particular instance type.

For AWS:

• General Purpose SSD: 100 - 4096 GiB
• Throughput Optimized HDD: 500 - 4096 GiB

For Azure:

• Premium LRS (SSD): 1 - 1023 GiB
• Standard LRS (HDD): 1- 1023 GiB

disk_iopsint32

disk_throughputint32

preloaded_docker_imagesarray of object

Custom Docker Image BYOC

Show child attributesHide child attributes

urlstring

URL of the docker image.

basic_authobjectRequired

Basic auth with username and password

Show child attributesHide child attributes

usernamestring

Name of the user

passwordstring

Password of the user

preloaded_spark_versionsarray of string

A list containing at most one preloaded Spark image version for the pool. Pool-backed clusters started with the preloaded Spark version will start faster. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

azure_attributesobject

Attributes related to instance pools running on Azure. If not specified at pool creation, a set of default values will be used.

Show child attributesHide child attributes

availabilitystring

Availability type used for the spot nodes.

Default: ON_DEMAND_AZURE

Values: SPOT_AZURE, ON_DEMAND_AZURE, SPOT_WITH_FALLBACK_AZURE

spot_bid_max_pricedouble

With variable pricing, you have option to set a max price, in US dollars (USD) For example, the value 2 would be a max price of $2.00 USD per hour. If you set the max price to be -1, the VM won't be evicted based on price. The price for the VM will be the current price for spot or the price for a standard VM, which ever is less, as long as there is capacity and quota available.

Default: -1

capacity_reservation_groupstring

The Azure capacity reservation group resource ID to use for launching VMs in this pool. When specified, VMs will be launched using the provided capacity reservation.

NOTE: Omitting this field will clear any existing configured capacity reservation group on the pool.

Capacity reservations can only be specified when the workspace uses injected vnet (i.e. customer defined vnet not managed by databricks). Ensure the databricks-login-prod Enterprise Application is granted the following four permissions:

1. Microsoft.Compute/capacityReservationGroups/read
2. Microsoft.Compute/capacityReservationGroups/deploy/action
3. Microsoft.Compute/capacityReservationGroups/capacityReservations/read
4. Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action

Format: /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}

gcp_attributesobject

Attributes related to instance pools running on Google Cloud Platform. If not specified at pool creation, a set of default values will be used.

Show child attributesHide child attributes

gcp_availabilitystring

Default: ON_DEMAND_GCP

Values: PREEMPTIBLE_GCP, ON_DEMAND_GCP, PREEMPTIBLE_WITH_FALLBACK_GCP

local_ssd_countint32

If provided, each node in the instance pool will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to GCP documentation for the supported number of local SSDs for each instance type.

zone_idstring

Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west1-a". The provided availability zone must be in the same region as the <Databricks> workspace. For example, "us-west1-a" is not a valid zone id if the <Databricks> workspace resides in the "us-east1" region. This is an optional field at instance pool creation, and if not specified, a default zone will be used.

This field can be one of the following:

• "HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region
• A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones (e.g. "us-west1-a").

If empty, <Databricks> picks an availability zone to schedule the cluster on.

node_type_flexibilityobject

Flexible node type configuration for the pool.

Show child attributesHide child attributes

alternate_node_type_idsarray of string

A list of node type IDs to use as fallbacks when the primary node type is unavailable.

remote_disk_throughputint32

If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED types.

total_initial_remote_disk_sizeint32

If set, what the total initial volume size (in GB) of the remote disks should be. Currently only supported for GCP HYPERDISK_BALANCED types.

View as Markdown

Request

{
  "instance_pool_id": "1234-567890-fetch12-pool-A3BcdEFg",
  "instance_pool_name": "my-pool",
  "min_idle_instances": 5,
  "max_capacity": 200,
  "aws_attributes": {
    "availability": "string",
    "zone_id": "string",
    "spot_bid_price_percent": 0,
    "instance_profile_arn": "string"
  },
  "node_type_id": "string",
  "custom_tags": {},
  "idle_instance_autotermination_minutes": 30,
  "enable_elastic_disk": true,
  "disk_spec": {
    "disk_type": {},
    "disk_count": 0,
    "disk_size": 0,
    "disk_iops": 0,
    "disk_throughput": 0
  },
  "preloaded_docker_images": [
    {
      "url": "string",
      "basic_auth": {}
    }
  ],
  "preloaded_spark_versions": [
    "string"
  ],
  "azure_attributes": {
    "availability": "string",
    "spot_bid_max_price": 0,
    "capacity_reservation_group": "string"
  },
  "gcp_attributes": {
    "gcp_availability": "string",
    "local_ssd_count": 0,
    "zone_id": "string"
  },
  "node_type_flexibility": {
    "alternate_node_type_ids": [
      "string"
    ]
  },
  "remote_disk_throughput": 0,
  "total_initial_remote_disk_size": 0
}

Response

{}

Delete GA

POST /api/2.0/instance-pools/delete

Deletes the instance pool permanently. The idle instances in the pool are terminated asynchronously.

API scopes: instance-pools

Request body

instance_pool_idstring

The instance pool to be terminated.

Example: 1234-567890-fetch12-pool-A3BcdEFg

View as Markdown

Request

{
  "instance_pool_id": "1234-567890-fetch12-pool-A3BcdEFg"
}

Response

{}