Node Type

View as Markdown

A description of a Spark node type including both the dimensions of the node and the instance type on which it will be hosted.

ListGET /api/2.1/clusters/list-node-types

NodeType object

A description of a Spark node type including both the dimensions of the node and the instance type on which it will be hosted.

node_type_idstring

Unique identifier for this node type.

memory_mbint32

Memory (in MB) available for this node type.

num_coresfloat

Number of CPU cores available for this node type. Note that this can be fractional, e.g., 2.5 cores, if the number of cores on a machine instance is not divisible by the number of Spark nodes on that machine.

descriptionstring

A string description associated with this node type, e.g., "r3.xlarge".

instance_type_idstring

An identifier for the type of hardware that this node runs on, e.g., "r3.2xlarge" in AWS.

is_deprecatedboolean

Whether the node type is deprecated. Non-deprecated node types offer greater performance.

Default: false

categorystring

A descriptive category for this node type. Examples include "Memory Optimized" and "Compute Optimized".

support_ebs_volumesboolean

Whether this node type support EBS volumes. EBS volumes is disabled for node types that we could place multiple corresponding containers on the same hosting instance.

Default: true

support_cluster_tagsboolean

Whether this node type support cluster tags.

Default: true

num_gpusint32

Number of GPUs available for this node type.

node_instance_typeobject

The NodeInstanceType object corresponding to instance_type_id

Show child attributesHide child attributes

instance_type_idstring

Unique identifier across instance types

local_disksint32

Number of local disks that are present on this instance.

local_disk_size_gbint32

Size of the individual local disks attached to this instance (i.e. per local disk).

local_nvme_disk_size_gbint32

Size of the individual local nvme disks attached to this instance (i.e. per local disk).

local_nvme_disksint32

Number of local nvme disks that are present on this instance.

is_hiddenboolean

Whether this node is hidden from presentation in the UI.

support_port_forwardingboolean

Whether this node type supports port forwarding.

Default: true

display_orderint32

An optional hint at the display order of node types in the UI. Within a node type category, lowest numbers come first.

Default: 0

is_io_cache_enabledboolean

Whether this node comes with IO cache enabled by default.

Default: false

node_infoobject

A collection of node type info reported by the cloud provider

Show child attributesHide child attributes

statusarray of string

Status as reported by the cloud provider

Values: NOT_ENABLED_ON_SUBSCRIPTION, NOT_AVAILABLE_IN_REGION

photon_worker_capableboolean

Default: false

photon_driver_capableboolean

Default: false

is_encrypted_in_transitboolean

AWS specific, whether this instance supports encryption in transit, used for hipaa and pci workloads.

Default: false

is_gravitonboolean

Whether this is an Arm-based instance.

Default: false

View as Markdown

The NodeType object

{
  "node_type_id": "string",
  "memory_mb": 0,
  "num_cores": 0,
  "description": "string",
  "instance_type_id": "string",
  "is_deprecated": true,
  "category": "string",
  "support_ebs_volumes": true,
  "support_cluster_tags": true,
  "num_gpus": 0,
  "node_instance_type": {
    "instance_type_id": "string",
    "local_disks": 0,
    "local_disk_size_gb": 0,
    "local_nvme_disk_size_gb": 0,
    "local_nvme_disks": 0
  },
  "is_hidden": true,
  "support_port_forwarding": true,
  "display_order": 0,
  "is_io_cache_enabled": true,
  "node_info": {
    "status": [
      "string"
    ]
  },
  "photon_worker_capable": true,
  "photon_driver_capable": true,
  "is_encrypted_in_transit": true,
  "is_graviton": true
}

List GA

GET /api/2.1/clusters/list-node-types

Returns a list of supported Spark node types. These node types can be used to launch a cluster.

API scopes: clusters

Response

Returns a list of NodeType objects.

View as Markdown

Response

{
  "node_types": [
    {
      "category": "Memory Optimized",
      "description": "r4.xlarge",
      "display_order": 0,
      "instance_type_id": "r4.xlarge",
      "is_deprecated": false,
      "is_hidden": false,
      "is_io_cache_enabled": false,
      "memory_mb": 31232,
      "node_instance_type": {
        "instance_family": "EC2 r4 Family vCPUs",
        "instance_type_id": "r4.xlarge",
        "local_disk_size_gb": 0,
        "local_disks": 0,
        "swap_size": "10g"
      },
      "node_type_id": "r4.xlarge",
      "num_cores": 4,
      "num_gpus": 0,
      "support_cluster_tags": true,
      "support_ebs_volumes": true,
      "support_port_forwarding": true,
      "supports_elastic_disk": true
    }
  ]
}