Proxy Traffic Through a NAT gateway

This guide illustrates how to configure your Databricks deployment in AWS so that specific traffic between EC2 instances and another IP address is proxied through a NAT gateway. You might want to do this to point your BI tools to a static IP address. For example, this can proxy all traffic to a Redshift cluster through the NAT gateway, so from the point of view of the Redshift cluster, all instances have a stable public IP address.

Note

For this to work with Redshift, the Redshift cluster must be publicly accessible with a stable IPv4 address. This can be achieved by launching the Redshift cluster with an elastic IP.

This topic walks through allocating a new public subnet in your Databricks VPC, adding a NAT gateway inside the subnet, and updating the default route table to make sure specific traffic goes through the NAT gateway.

Step 1: Create a subnet in Databricks VPC for the NAT gateway

  1. Contact Databricks to retrieve the ID of your Databricks VPC. You will use this Databricks VPC ID several times in the following steps.

  2. Log in to your AWS VPC console and select Subnet from the left panel.

  3. Type the Databricks VPC ID in the search box. You can see all subnets that Databricks created in the VPC.

    Create a subnet

    By default, Databricks creates one subnet per availability zone (AZ). You must choose an unused CIDR block from the same B-class range (i.e., a.b.0.0). In the example, there are 3 used CIDR blocks: 10.29.224.0/19, 10.29.192.0/19, 10.29.160.0/19. The address range 10.29.0.0 ~ 10.29.159.255 are all free. You can choose a new C-class subnet within that range, say 10.29.100.0/24.

    Note

    • Databricks allocates CIDR blocks starting from the top of the B-class range. When choosing the CIDR block for the gateway subnet, it is better to use a lower range to avoid future conflict in case AWS adds more availability zones to the region.
    • The size of the public subnet can be arbitrarily small — it just needs to accommodate a single host for the NAT gateway.
  4. Click Create Subnet and type in all necessary information.

    Create a subnet dialog
    • Name tag: any name, suggest to have a gateway in it.
    • VPC: choose Databricks VPC-ID.
    • Availability Zone: choose any one from the list
    • CIDR block: an unused CIDR block (for example, a.b.c.0/24)
  5. Click Yes, Create.

Step 2: Create a NAT gateway in the subnet

  1. Select NAT gateways from AWS VPC console’s left panel, and click Create NAT gateway.

    Create a NAT gateway
  2. Choose the subnet that you just created in Step 1 from the Subnet field, and choose any available IP from Elastic IP Allocation ID, and then click Create a NAT gateway. If there are no available elastic IPs, you may need to create a new elastic IP for the NAT gateway to use.

    NAT gateway created
  3. On the success page, click Edit Route Tables.

Step 3: Associate a route table with the gateway subnet

  1. Click Create route table, enter any name in the Name tag field.

  2. Select the Databricks VPC-ID from the VPC field.

  3. Click Yes, Create.

    Create a route table
  4. Select the newly-created route table from the list, go to the Routes tab, click Edit and Add another route.

  5. Enter 0.0.0.0/0 in Destination, and choose Databricks VPC’s internet gateway (starts with igw-xxxxxx) from Target, and click Save.

    Add a route
  6. Go to the Subnet Associations tab, edit and check the gateway subnet you created in Step 1, and click Save. This grants direct internet access for the gateway subnet, making it a “public subnet”.

    Subnet associations

Step 4: Configure routing to an external system

This step forwards all traffic between VPC instances in the VPC and an external system (for example, Redshift), ensuring that all their internet traffic goes through the NAT gateway.

  1. In the AWS VPC console’s left panel, select Route Tables and type the Databricks VPC-ID in the search box. The list shows all route tables used by Databricks VPC. One of them is the gateway route table you created in Step 3, the other (marked as Main) is the default route table for all other subnets in the VPC.

  2. Select the route table marked as Main from the list, go to the Routes tab, and click Edit.

  3. Find the Elastic IP address of the external system (for example, Redshift cluster). Here using 1.2.3.4.

  4. Add the Target for the external system 1.2.3.4/32 to use the NAT gateway you created in Step 2 (starting with nat-xxxxxxxxxx), and click Save.

    Configure routing

Now all traffic between all instances in the VPC to the IP address you specified above will go through the NAT gateway. From the point of view of the external system, all nodes have the same IP address as the NAT gateway’s elastic IP. All other traffic will use the VPC’s original routing rules.

Step 5: Add an S3 endpoint to the VPC

All external traffic that goes through the NAT gateway is subject to additional charges (see Amazon VPC pricing). To make sure that instances can access Databricks assets stored in S3 buckets faster and for free, you can add an S3 endpoint to the Databricks VPC.

Note

VPC endpoints do not support cross-region access. Accessing S3 buckets that reside in another region will still go through the NAT gateway.

  1. In the AWS VPC console’s left panel, select Endpoints and click Create Endpoint.

    ../../../_images/add-ep-1.png
  2. In the VPC field, select the Databricks VPC-ID, select the only available S3 service from the Service box, and click Next Step.

    ../../../_images/add-ep-2.png
  3. Check the main route table to associate the S3 endpoint with all private subnets and click Create Endpoint.

    ../../../_images/add-ep-3.png

Step 6: Verify subnet settings

Go to the Subnets page and the Route Table tab, type the Databricks VPC-ID in the search box, and verify that all settings are correct.

  • The gateway subnet should have a default routing using the VPC’s internet gateway (igw-xxxxxx):

    ../../../_images/verify-1.png
  • All other subnets should have a default routing using the NAT gateway (nat-xxxxxx).

  • If the S3 endpoint is configured correctly, a vpce-xxxx <target> should show up in the list.

  • The VPC peering connection to the Databricks control plane VPC (pcx-xxxxx) should remain untouched.

    ../../../_images/verify-2.png

Step 7: Create a cluster with NAT gateway proxy

After you confirm that the NAT gateway proxy is set up properly, log in to your Databricks workspace and create a new cluster. Wait for the cluster to become ready. Contact Databricks support if the cluster cannot start.

Step 8 (optional): Disable the public IP address on workers

With a NAT gateway proxy enabled, all outbound internet traffic from the Apache Spark cluster goes through the gateway. You can further increase security by removing the public-facing IP address on each worker instance.

Important

You can only disable the public IP address on newly created AWS instances.

  1. Contact Databricks support and inform them that you would like to disable public IP addresses.
  2. After Databricks confirms that the configuration change is complete, terminate all clusters.
  3. Wait one hour or manually terminate the existing instances in the AWS console. If you do not perform this step, Databricks might reuse existing instances that still have the previously assigned public IP addresses for any new clusters.
  4. Log in to your Databricks deployment and create a new cluster. Wait for the cluster to become ready.

If all configurations are correct, the cluster should come up as usual. You can verify that the cluster has no public IP address by going to the AWS EC2 dashboard and checking the configuration of worker instances:

../../../_images/verify-3.png