describe-my-ec2(Python)

Import Notebook

Describe My EC2 Spark Instances with static and dynamic EC2 instance metadata

Get the instance id, region, and AZ of the Master/Driver

from urllib.request import urlopen
import json
instance = json.loads(urlopen(’http://169.254.169.254/latest/dynamic/instance-identity/document').read())
instance
Out[11]: 'http://169.254.169.254/latest/dynamic/instance-identity/document'

Get the public hostname of the Master/Driver

from urllib.request import urlopen
public_hostname = urlopen(’http://169.254.169.254/latest/meta-data/local-hostname').read()
public_hostname
Out[12]: 'http://169.254.169.254/latest/meta-data/public-hostname'

Get the public hostnames and IPs of all Workers

from urllib.request import urlopen
import json
 
# modify this to be the number of Workers you have running (in Clusters UI)
numWorkers = 10 
 
worker_instance_ids = sc.parallelize(range(numWorkers)).map(lambda x: 
  (urlopen('http://169.254.169.254/latest/meta-data/local-hostname').read(),urlopen('http://169.254.169.254/latest/meta-data/local-ipv4').read())
)
 
print (worker_instance_ids.distinct().collect())
[('ec2-54-191-168-248.us-west-2.compute.amazonaws.com', '54.191.168.248'), ('ec2-54-213-157-27.us-west-2.compute.amazonaws.com', '54.213.157.27')]