Configure a private network gateway

Private Preview

Private network gateway is in Private Preview. To request access, contact your Databricks account team.

This article describes how to set up and manage a private network gateway using the account REST API. For an overview of what a private network gateway is and how it works, see Private network gateway.

Before you begin

Before you create a private network gateway, make sure you have the following:

• An AWS Databricks account with at least one workspace that has serverless compute enabled.
• Databricks account admin privileges. Your account must also be on the Enterprise pricing tier.
• A dedicated VPC subnet in your target region and Availability Zone. A /28 is the practical minimum. The subnet must have available IP addresses and downstream connectivity to your intended resources. Databricks recommends creating a subnet dedicated to the gateway.
• Permission in your AWS account to create an IAM role that Databricks can assume.
• The ability to create a security group in the same VPC to control gateway egress.
• The IP address of a DNS resolver that is reachable from the gateway subnet. When you use Amazon-provided DNS, this is the VPC+2 address (for example, 10.0.0.2 if your VPC CIDR is 10.0.0.0/16), or the link-local address 169.254.169.253.

Private network gateway is available in a limited set of AWS regions and physical Availability Zones during Private Preview. Contact your Databricks account team to confirm the supported regions and Availability Zones for your deployment. The gateway and its subnet must be in the same region as the NCC.

Enable the preview

To request access to the private network gateway Private Preview, contact your Databricks account team and provide the following:

• Your Databricks account ID.
• The regions where you intend to use the gateway.

Set up a private network gateway

All private network gateway configuration uses the account REST API. There is no UI or Terraform support during Private Preview.

Create or select an NCC

If you already have an NCC in the region you want to use, skip this step.

1. Obtain an OAuth access token using a service principal that has the account admin role. An account admin creates the service principal in the account console beforehand. To request the token, run the following:

   Bash

   curl --location 'https://accounts.cloud.databricks.com/oidc/accounts/{account_id}/v1/token' \
     --header 'Content-Type: application/x-www-form-urlencoded' \
     --header 'Authorization: Basic <base64(client_id:client_secret)>' \
     --data-urlencode 'grant_type=client_credentials' \
     --data-urlencode 'scope=all-apis'

2. Create an NCC, run the following:

   Bash

   curl --request POST \
     'https://accounts.cloud.databricks.com/api/2.0/accounts/{account_id}/network-connectivity-configs' \
     --header 'Authorization: Bearer <access_token>' \
     --header 'Content-Type: application/json' \
     --data '{ "name": "my-ncc", "region": "us-east-1" }'

   Save the network_connectivity_config_id from the response.

Prepare your VPC networking

Provide the following in your AWS account:

• A subnet in the target Availability Zone, with available IP addresses. Private Preview supports exactly one subnet in one Availability Zone.
• A security group that controls what the gateway network interface can reach. Databricks recommends no inbound rules and outbound rules scoped to your destination ports.
• A DNS resolver IP address that is reachable from the gateway subnet. You can use the AWS VPC resolver at 169.254.169.253, the VPC CIDR base plus two (for example, 10.0.0.2 for a 10.0.0.0/16 VPC), a Route 53 Resolver endpoint, or your custom DNS server.

The subnet, security group, and DNS resolver must be in, or reachable from, the same VPC and region as your NCC and target workspaces.

Create the cross-account IAM role

1. Create an IAM role in your AWS account that Databricks assumes to create and manage the gateway network interface.

2. Set the trust policy to allow the Databricks role to assume this role, using your Databricks account ID as the external ID:

   JSON

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::414351767826:role/png-management-role"
         },
         "Action": "sts:AssumeRole",
         "Condition": {
           "StringEquals": {
             "sts:ExternalId": "<your_databricks_account_id>"
           }
         }
       }
     ]
   }

3. Attach the following permission policy. Replace <region>, <customer_account>, and <vpc_id> with your values:

   JSON

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "CreateNicInVpc",
         "Effect": "Allow",
         "Action": "ec2:CreateNetworkInterface",
         "Resource": [
           "arn:aws:ec2:<region>:<customer_account>:network-interface/*",
           "arn:aws:ec2:<region>:<customer_account>:subnet/*",
           "arn:aws:ec2:<region>:<customer_account>:security-group/*"
         ]
       },
       {
         "Sid": "ManageAndCleanupNic",
         "Effect": "Allow",
         "Action": ["ec2:CreateNetworkInterfacePermission", "ec2:DeleteNetworkInterface", "ec2:CreateTags"],
         "Resource": "arn:aws:ec2:<region>:<customer_account>:network-interface/*",
         "Condition": { "StringEquals": { "aws:RequestedRegion": "<region>" } }
       },
       {
         "Sid": "CreateGatewayInVpcOnly",
         "Effect": "Allow",
         "Action": "vpc-lattice:CreateResourceGateway",
         "Resource": "arn:aws:vpc-lattice:<region>:<customer_account>:resourcegateway/*",
         "Condition": {
           "StringEquals": {
             "aws:RequestedRegion": "<region>",
             "vpc-lattice:VpcId": "<vpc_id>",
             "aws:RequestTag/Vendor": "Databricks"
           }
         }
       },
       {
         "Sid": "CreateManagedGatewayResources",
         "Effect": "Allow",
         "Action": ["vpc-lattice:CreateResourceConfiguration", "vpc-lattice:TagResource"],
         "Resource": [
           "arn:aws:vpc-lattice:<region>:<customer_account>:resourcegateway/*",
           "arn:aws:vpc-lattice:<region>:<customer_account>:resourceconfiguration/*"
         ],
         "Condition": { "StringEquals": { "aws:RequestedRegion": "<region>", "aws:RequestTag/Vendor": "Databricks" } }
       },
       {
         "Sid": "ManageManagedGatewayResources",
         "Effect": "Allow",
         "Action": [
           "vpc-lattice:GetResourceGateway",
           "vpc-lattice:DeleteResourceGateway",
           "vpc-lattice:GetResourceConfiguration",
           "vpc-lattice:DeleteResourceConfiguration",
           "vpc-lattice:PutResourcePolicy",
           "vpc-lattice:DeleteResourcePolicy"
         ],
         "Resource": [
           "arn:aws:vpc-lattice:<region>:<customer_account>:resourcegateway/*",
           "arn:aws:vpc-lattice:<region>:<customer_account>:resourceconfiguration/*"
         ],
         "Condition": { "StringEquals": { "aws:RequestedRegion": "<region>", "aws:ResourceTag/Vendor": "Databricks" } }
       },
       {
         "Sid": "ListAndDescribeInRegion",
         "Effect": "Allow",
         "Action": [
           "vpc-lattice:GetResourcePolicy",
           "vpc-lattice:ListResourceGateways",
           "vpc-lattice:ListResourceConfigurations",
           "ec2:DescribeNetworkInterfaces",
           "ec2:DescribeVpcs",
           "ec2:DescribeSubnets",
           "ec2:DescribeSecurityGroups",
           "ec2:DescribeAvailabilityZones",
           "ec2:DescribeRouteTables"
         ],
         "Resource": "*",
         "Condition": { "StringEquals": { "aws:RequestedRegion": "<region>" } }
       },
       {
         "Sid": "VpcLatticeServiceLinkedRole",
         "Effect": "Allow",
         "Action": "iam:CreateServiceLinkedRole",
         "Resource": "arn:aws:iam::<customer_account>:role/aws-service-role/vpc-lattice.amazonaws.com/AWSServiceRoleForVpcLattice",
         "Condition": {
           "StringEquals": { "iam:AWSServiceName": "vpc-lattice.amazonaws.com" }
         }
       }
     ]
   }

   The Databricks AWS account is 414351767826. The external ID is your Databricks account ID. Some actions cannot be scoped to a specific VPC, so the tightest available scope is account and region. Grant the policy in full. All statements are required. This is a single forward-compatible policy for provisioning and managing the gateway, so you configure the role once and do not need to update it as the gateway evolves.

Create the private network gateway

To create the gateway, send a POST request to the private-network-gateways endpoint under your NCC. The request body accepts the following fields:

• gateway_name (required): A human-readable name for the gateway.
• aws_cloud_connection.gateway_subnets (required): The subnet where the gateway network interface is placed. Private Preview accepts exactly one subnet.
• aws_cloud_connection.cross_account_role.role_arn (required): The ARN of the IAM role you created in the previous step.
• aws_cloud_connection.security_group_ids (required): The security groups attached to the gateway network interface, which control egress.
• private_dns_resolvers (required): The DNS resolver IP addresses you identified when you prepared your VPC networking.
• traffic_mode (required): SPECIFIC_DESTINATIONS or ALL_TRAFFIC. See Traffic modes.
• destinations (required when traffic_mode is SPECIFIC_DESTINATIONS): The DNS names to route through the gateway.

To create a gateway in SPECIFIC_DESTINATIONS mode, run the following:

Bash

curl --request POST \
  'https://accounts.cloud.databricks.com/api/2.0/accounts/{account_id}/network-connectivity-configs/{network_connectivity_config_id}/private-network-gateways' \
  --header 'Authorization: Bearer <access_token>' \
  --header 'Content-Type: application/json' \
  --data '{
    "gateway_name": "my-aws-png",
    "traffic_mode": "SPECIFIC_DESTINATIONS",
    "aws_cloud_connection": {
      "gateway_subnets": [{ "subnet_id": "subnet-0abc123def456" }],
      "cross_account_role": {
        "role_arn": "arn:aws:iam::<customer_account>:role/databricks-png-role"
      },
      "security_group_ids": ["sg-customer-xyz"]
    },
    "destinations": [
      { "destination_type": "DNS_NAME", "value": "app.customer.internal" }
    ],
    "private_dns_resolvers": [
      { "resolver_type": "IP_ADDRESS", "value": "10.0.0.2" }
    ]
  }'

The gateway is created in the CREATING state. It moves to ESTABLISHED after Databricks creates the gateway network interface in your subnet, which usually takes under two minutes.

Confirm the gateway is established

To check the gateway state, send a GET request until state is ESTABLISHED:

Bash

curl --request GET \
  'https://accounts.cloud.databricks.com/api/2.0/accounts/{account_id}/network-connectivity-configs/{network_connectivity_config_id}/private-network-gateways/{gateway_id}' \
  --header 'Authorization: Bearer <access_token>'

Confirm that state is ESTABLISHED before you continue.

Attach the NCC to your workspaces

If your workspace is already attached to the NCC, skip this step. An NCC is a regional object and can attach only to workspaces in the same region.

In the account console, go to Workspaces, select the workspace, open Network Connectivity, and attach your NCC. Repeat for each workspace that should use the gateway.

You can also attach the NCC using the account REST API:

Bash

curl --request PATCH \
  'https://accounts.cloud.databricks.com/api/2.0/accounts/{account_id}/workspaces/{workspace_id}' \
  --header 'Authorization: Bearer <access_token>' \
  --header 'Content-Type: application/json' \
  --data '{ "network_connectivity_config_id": "{network_connectivity_config_id}" }'

DNS configuration

A gateway uses the resolver you specify in private_dns_resolvers to resolve hostnames for the destinations you configure. The resolver must be reachable from the gateway subnet.

If your private resources are registered in a Route 53 private hosted zone, associate that zone with the VPC that contains the gateway subnet so the hostnames resolve correctly. To use the built-in AWS VPC resolver, specify 169.254.169.253 or the VPC CIDR base plus two. If you have custom private zones with overlapping names, use your own resolver instead.

Manage a gateway

You can update, delete, and list gateways with the account REST API.

Update a gateway

You can update gateway_name, destinations, private_dns_resolvers, and traffic_mode in place with a PATCH request. The update_mask query parameter is required and specifies which fields to update. When you switch traffic_mode to ALL_TRAFFIC, clear destinations in the same request. When you switch to SPECIFIC_DESTINATIONS, include the destinations to route. To change gateway_subnets, cross_account_role, or security_group_ids, delete the gateway and create a new one.

To update the destinations, run the following:

Bash

curl --request PATCH \
  'https://accounts.cloud.databricks.com/api/2.0/accounts/{account_id}/network-connectivity-configs/{network_connectivity_config_id}/private-network-gateways/{gateway_id}?update_mask=destinations' \
  --header 'Authorization: Bearer <access_token>' \
  --header 'Content-Type: application/json' \
  --data '{ "destinations": [ { "destination_type": "DNS_NAME", "value": "app2.customer.internal" } ] }'

Delete a gateway

warning

Deleting a gateway cuts off connectivity to the private resources that depend on it. Confirm that no active workloads rely on the gateway before you delete it.

To delete the gateway, send a DELETE request to the gateway endpoint under your NCC:

Bash

curl --request DELETE \
  'https://accounts.cloud.databricks.com/api/2.0/accounts/{account_id}/network-connectivity-configs/{network_connectivity_config_id}/private-network-gateways/{gateway_id}' \
  --header 'Authorization: Bearer <access_token>'

List gateways

To list the private network gateways in an NCC, send a GET request to the gateway endpoint under your NCC:

Bash

curl --request GET \
  'https://accounts.cloud.databricks.com/api/2.0/accounts/{account_id}/network-connectivity-configs/{network_connectivity_config_id}/private-network-gateways' \
  --header 'Authorization: Bearer <access_token>'

Gateway states

A gateway reports one of the following states:

State	Description
CREATING	Databricks is creating the gateway network interface in your subnet.
ESTABLISHED	The gateway is ready to route traffic.
DELETING	The gateway is being removed.
FAILED	The gateway could not be provisioned. This state is terminal: delete the gateway and create a new one after fixing the underlying cause, such as an unsupported region or Availability Zone.

State	Description
CREATING	Databricks is creating the gateway network interface in your subnet.
ESTABLISHED	The gateway is ready to route traffic.
DELETING	The gateway is being removed.
FAILED	The gateway could not be provisioned. This state is terminal: delete the gateway and create a new one after fixing the underlying cause, such as an unsupported region or Availability Zone.