Configure a private network gateway
This feature is in Private Preview. To try it, reach out to your Databricks contact.
This article describes how to set up and manage a private network gateway. For an overview of what a private network gateway is and how it works, see Private network gateway.
Requirements
Before you create a private network gateway, make sure you have the following:
- An AWS Databricks account with at least one workspace that has serverless compute enabled.
- Databricks account admin privileges. Your account must also be on the Enterprise pricing tier.
- A dedicated VPC subnet in your target region and Availability Zone. A
/28is the practical minimum. The subnet must have available IP addresses and downstream connectivity to your intended resources. Databricks recommends creating a subnet dedicated to the gateway. - Permission in your AWS account to create an IAM role that Databricks can assume.
- The ability to create a security group in the same VPC to control gateway egress.
- DNS resolution configured in your VPC for the resources you want to access. The gateway uses Amazon-provided DNS.
Private network gateway is available in a limited set of AWS regions and physical Availability Zones during Private Preview. Contact your Databricks account team to confirm the supported regions and Availability Zones for your deployment. The gateway and its subnet must be in the same region as the NCC.
Enable the preview
To request access to the private network gateway Private Preview, contact your Databricks account team and provide the following:
- Your Databricks account ID.
- The regions where you intend to use the gateway.
Set up a private network gateway
All private network gateway configuration uses the account REST API. There is no UI or Terraform support during the preview.
The REST API steps below use /api/networking/v1/ for gateway operations. NCC creation and workspace attachment continue to use /api/2.0/accounts/....
Create or select an NCC
If you already have an NCC in the region you want to use, skip this step.
-
Obtain an OAuth access token using a service principal that has the account admin role. An account admin creates the service principal in the account console beforehand. To request the token, run the following:
Bashcurl --location 'https://accounts.cloud.databricks.com/oidc/accounts/{account_id}/v1/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--header 'Authorization: Basic <base64(client_id:client_secret)>' \
--data-urlencode 'grant_type=client_credentials' \
--data-urlencode 'scope=all-apis' -
Create an NCC, run the following:
Bashcurl --request POST \
'https://accounts.cloud.databricks.com/api/2.0/accounts/{account_id}/network-connectivity-configs' \
--header 'Authorization: Bearer <access_token>' \
--header 'Content-Type: application/json' \
--data '{ "name": "my-ncc", "region": "us-east-1" }'Save the
network_connectivity_config_idfrom the response.
Prepare your VPC networking
Provide the following in your AWS account:
- A subnet in the target Availability Zone, with available IP addresses. Private Preview supports exactly one subnet in one Availability Zone.
- A security group that controls what the gateway network interface can reach. Databricks recommends no inbound rules and outbound rules scoped to your destination ports.
- DNS resolution through Amazon-provided DNS. Associate any required Route 53 private hosted zones with the VPC that contains the gateway subnet.
The subnet and security group must be in the same VPC, in the region of your NCC and target workspaces.
Create the cross-account IAM role
-
Create an IAM role in your AWS account that Databricks assumes to create and manage the gateway network interface.
-
Set the trust policy to allow the Databricks role to assume this role, using your Databricks account ID as the external ID:
JSON{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::414351767826:role/png-management-role"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "<your_databricks_account_id>"
}
}
}
]
} -
Attach the following permission policy. Replace
<region>,<customer_account>, and<vpc_id>with your values:JSON{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "CreateNicInVpc",
"Effect": "Allow",
"Action": "ec2:CreateNetworkInterface",
"Resource": [
"arn:aws:ec2:<region>:<customer_account>:network-interface/*",
"arn:aws:ec2:<region>:<customer_account>:subnet/*",
"arn:aws:ec2:<region>:<customer_account>:security-group/*"
]
},
{
"Sid": "ManageAndCleanupNic",
"Effect": "Allow",
"Action": ["ec2:CreateNetworkInterfacePermission", "ec2:DeleteNetworkInterface", "ec2:CreateTags"],
"Resource": "arn:aws:ec2:<region>:<customer_account>:network-interface/*",
"Condition": { "StringEquals": { "aws:RequestedRegion": "<region>" } }
},
{
"Sid": "CreateGatewayInVpcOnly",
"Effect": "Allow",
"Action": "vpc-lattice:CreateResourceGateway",
"Resource": "arn:aws:vpc-lattice:<region>:<customer_account>:resourcegateway/*",
"Condition": {
"StringEquals": {
"aws:RequestedRegion": "<region>",
"vpc-lattice:VpcId": "<vpc_id>",
"aws:RequestTag/Vendor": "Databricks"
}
}
},
{
"Sid": "CreateManagedGatewayResources",
"Effect": "Allow",
"Action": ["vpc-lattice:CreateResourceConfiguration", "vpc-lattice:TagResource"],
"Resource": [
"arn:aws:vpc-lattice:<region>:<customer_account>:resourcegateway/*",
"arn:aws:vpc-lattice:<region>:<customer_account>:resourceconfiguration/*"
],
"Condition": { "StringEquals": { "aws:RequestedRegion": "<region>", "aws:RequestTag/Vendor": "Databricks" } }
},
{
"Sid": "ManageManagedGatewayResources",
"Effect": "Allow",
"Action": [
"vpc-lattice:GetResourceGateway",
"vpc-lattice:DeleteResourceGateway",
"vpc-lattice:GetResourceConfiguration",
"vpc-lattice:DeleteResourceConfiguration",
"vpc-lattice:PutResourcePolicy",
"vpc-lattice:DeleteResourcePolicy"
],
"Resource": [
"arn:aws:vpc-lattice:<region>:<customer_account>:resourcegateway/*",
"arn:aws:vpc-lattice:<region>:<customer_account>:resourceconfiguration/*"
],
"Condition": { "StringEquals": { "aws:RequestedRegion": "<region>", "aws:ResourceTag/Vendor": "Databricks" } }
},
{
"Sid": "ListAndDescribeInRegion",
"Effect": "Allow",
"Action": [
"vpc-lattice:GetResourcePolicy",
"vpc-lattice:ListResourceGateways",
"vpc-lattice:ListResourceConfigurations",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeVpcs",
"ec2:DescribeSubnets",
"ec2:DescribeSecurityGroups",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeRouteTables"
],
"Resource": "*",
"Condition": { "StringEquals": { "aws:RequestedRegion": "<region>" } }
},
{
"Sid": "VpcLatticeServiceLinkedRole",
"Effect": "Allow",
"Action": "iam:CreateServiceLinkedRole",
"Resource": "arn:aws:iam::<customer_account>:role/aws-service-role/vpc-lattice.amazonaws.com/AWSServiceRoleForVpcLattice",
"Condition": {
"StringEquals": { "iam:AWSServiceName": "vpc-lattice.amazonaws.com" }
}
}
]
}The Databricks AWS account is
414351767826. The external ID is your Databricks account ID. Some actions cannot be scoped to a specific VPC, so the tightest available scope is account and region. Grant the policy in full. All statements are required. This is a single forward-compatible policy for provisioning and managing the gateway, so you configure the role once and do not need to update it as the gateway evolves.
Create the private network gateway
To create the gateway, send a POST request to the private-network-gateways endpoint under your NCC. The request body accepts the following fields:
display_name(required): A human-readable name for the gateway.aws_cloud_connection.gateway_subnets(required): The subnet where the gateway network interface is placed. Private Preview accepts exactly one subnet.aws_cloud_connection.cross_account_role.role_arn(required): The ARN of the IAM role you created in the previous step.aws_cloud_connection.security_group_ids(required): The security groups attached to the gateway network interface, which control egress.private_dns_resolvers(optional): Omit this field or set it to[]to use Amazon-provided DNS. Do not specify the Amazon-provided resolver IP addresses.traffic_mode(required):SPECIFIC_DESTINATIONSorALL_TRAFFIC. See Traffic modes.destinations(required whentraffic_modeisSPECIFIC_DESTINATIONS): The DNS names to route through the gateway.
To make gateway creation idempotent, add the optional request_id query parameter to the POST URL, such as ?request_id={request_id}. Use a unique ID for each new gateway and reuse it when retrying the same request to return the original operation without creating another gateway.
To create a gateway in SPECIFIC_DESTINATIONS mode, run the following:
curl --request POST \
'https://accounts.cloud.databricks.com/api/networking/v1/accounts/{account_id}/network-connectivity-configs/{network_connectivity_config_id}/private-network-gateways' \
--header 'Authorization: Bearer <access_token>' \
--header 'Content-Type: application/json' \
--data '{
"display_name": "my-aws-png",
"traffic_mode": "SPECIFIC_DESTINATIONS",
"aws_cloud_connection": {
"gateway_subnets": [{ "subnet_id": "subnet-0abc123def456" }],
"cross_account_role": {
"role_arn": "arn:aws:iam::<customer_account>:role/databricks-png-role"
},
"security_group_ids": ["sg-customer-xyz"]
},
"destinations": [
{ "destination_type": "DNS_NAME", "value": "app.customer.internal" }
],
"private_dns_resolvers": []
}'
The response is a long-running operation. Save its name, which has the form accounts/{account_id}/network-connectivity-configs/{network_connectivity_config_id}/private-network-gateways/{gateway_id}/operations/{operation_id}. This operation name is different from the gateway's resource name.
For example, a pending operation includes the following fields:
{
"name": "accounts/{account_id}/network-connectivity-configs/{network_connectivity_config_id}/private-network-gateways/{gateway_id}/operations/{operation_id}",
"done": false
}
Confirm the gateway is established
To poll the create operation, replace {operation_name} with the complete value of the top-level name field in the create response. This value starts with accounts/ and ends with operations/{operation_id}. Append it to https://accounts.cloud.databricks.com/api/networking/v1/ as shown below:
curl --request GET \
'https://accounts.cloud.databricks.com/api/networking/v1/{operation_name}' \
--header 'Authorization: Bearer <access_token>'
Wait until done is true. If the operation contains error, resolve the provisioning issue. If it contains response, the gateway is established. A successful operation includes the following fields:
{
"name": "accounts/{account_id}/network-connectivity-configs/{network_connectivity_config_id}/private-network-gateways/{gateway_id}/operations/{operation_id}",
"done": true,
"response": {
"name": "accounts/{account_id}/network-connectivity-configs/{network_connectivity_config_id}/private-network-gateways/{gateway_id}",
"display_name": "my-aws-png",
"state": "ESTABLISHED"
}
}
Save response.name as {gateway_name}. To retrieve the gateway later, use its complete resource name:
curl --request GET \
'https://accounts.cloud.databricks.com/api/networking/v1/{gateway_name}' \
--header 'Authorization: Bearer <access_token>'
The GET response is the gateway resource itself, without the operation wrapper.
Attach the NCC to your workspaces
If your workspace is already attached to the NCC, skip this step. An NCC is a regional object and can attach only to workspaces in the same region.
In the account console, go to Workspaces, select the workspace, open Network Connectivity, and attach your NCC. Repeat for each workspace that should use the gateway.
You can also attach the NCC using the account REST API:
curl --request PATCH \
'https://accounts.cloud.databricks.com/api/2.0/accounts/{account_id}/workspaces/{workspace_id}' \
--header 'Authorization: Bearer <access_token>' \
--header 'Content-Type: application/json' \
--data '{ "network_connectivity_config_id": "{network_connectivity_config_id}" }'
DNS configuration
Omit private_dns_resolvers or set it to [] to use Amazon-provided DNS. The V1 API rejects explicitly supplied Amazon-provided resolver addresses, including 169.254.169.253 and the VPC CIDR base plus two.
If your private resources are registered in a Route 53 private hosted zone, associate that zone with the VPC that contains the gateway subnet so the hostnames resolve correctly. For custom DNS requirements, contact your Databricks account team.
Manage a gateway
You can update, delete, and list gateways with the account REST API.
Update a gateway
You can update display_name, destinations, private_dns_resolvers, and traffic_mode in place with a PATCH request. The update_mask query parameter is required and specifies which fields to update. When you switch traffic_mode to ALL_TRAFFIC, clear destinations in the same request. When you switch to SPECIFIC_DESTINATIONS, include the destinations to route. To change gateway_subnets, cross_account_role, or security_group_ids, delete the gateway and create a new one.
To switch to Amazon-provided DNS, include private_dns_resolvers in update_mask and set private_dns_resolvers to [].
To update the destinations, run the following:
curl --request PATCH \
'https://accounts.cloud.databricks.com/api/networking/v1/accounts/{account_id}/network-connectivity-configs/{network_connectivity_config_id}/private-network-gateways/{gateway_id}?update_mask=destinations' \
--header 'Authorization: Bearer <access_token>' \
--header 'Content-Type: application/json' \
--data '{ "destinations": [ { "destination_type": "DNS_NAME", "value": "app2.customer.internal" } ] }'
The update response is the gateway resource itself. It is not wrapped in a private_network_gateway field.
Delete a gateway
Deleting a gateway cuts off connectivity to the private resources that depend on it. Confirm that no active workloads rely on the gateway before you delete it.
To delete the gateway, send a DELETE request to the gateway endpoint under your NCC:
curl --request DELETE \
'https://accounts.cloud.databricks.com/api/networking/v1/accounts/{account_id}/network-connectivity-configs/{network_connectivity_config_id}/private-network-gateways/{gateway_id}' \
--header 'Authorization: Bearer <access_token>'
The delete response is empty. Subsequent get requests return NOT_FOUND, and the gateway is omitted from list responses while infrastructure cleanup continues. Delete does not return an operation to poll.
List gateways
To list the private network gateways in an NCC, send a GET request to the gateway endpoint under your NCC:
curl --request GET \
'https://accounts.cloud.databricks.com/api/networking/v1/accounts/{account_id}/network-connectivity-configs/{network_connectivity_config_id}/private-network-gateways' \
--header 'Authorization: Bearer <access_token>'
Read gateways from the private_network_gateways array. If the response includes next_page_token, pass that value as the page_token query parameter in the next list request.
Gateway states
A gateway reports one of the following states:
State | Description |
|---|---|
| Databricks is creating the gateway network interface in your subnet. |
| The gateway is ready to route traffic. |
| The gateway could not be provisioned. This state is terminal: delete the gateway and create a new one after fixing the underlying cause, such as an unsupported region or Availability Zone. |