Private network gateway
Private network gateway is in Private Preview. To request access, contact your Databricks account team.
Private network gateway is an enterprise feature. It is not billed during Private Preview, but Databricks will charge for it in the future.
A private network gateway connects Databricks serverless compute to one or more resources in your VPC through a single managed gateway.
After you provide one subnet, a cross-account IAM role, and a security group to Databricks, your serverless environment can reach the same resources your subnet can, including networks transitively connected to your VPC such as on-premises systems over AWS Direct Connect or a VPN.
A private network gateway complements Private Link rather than replacing it:
- Use a Private Link private endpoint rule for a direct, private connection to a specific cloud-managed resource, such as object storage.
- Use a private network gateway to reach resources in your VPC or a connected network, or to route serverless egress through your own firewall or a dedicated egress path.
To set up a private network gateway, see Configure a private network gateway.
Overview
A private network gateway addresses several serverless connectivity needs with a single setup:
- Broad connectivity without per-resource onboarding. Reach many resources in your VPC, or on-premises systems over a connected AWS Direct Connect or VPN, without configuring a separate private endpoint for each resource.
- Egress through your own security appliances. Route serverless egress through your own firewall or network security appliance, such as Palo Alto or your cloud provider's firewall service, for content inspection before traffic leaves your network.
- Stable, identifiable source IP addresses. Send serverless egress from your own set of IP addresses, so downstream systems can identify the traffic at the network layer. This is useful for secure allowlisting in multi-tenant environments.
You can also use a private network gateway to control how serverless compute reaches the internet:
- Firewall serverless internet access through your own security appliances.
- Reach the internet from your own dedicated set of IP addresses.
How it works
When you create a private network gateway, Databricks places a cross-account network interface in the subnet you provide in your VPC. Databricks assumes the cross-account IAM role you create to provision and manage that network interface. Serverless traffic to the destinations you configure flows through this interface into your VPC and onward to any network your VPC routes to.
Databricks evaluates outbound serverless traffic against your configured network paths in priority order, using the first path that matches each connection's destination:
Priority | Path | Applies to |
|---|---|---|
1 | Private endpoint rules (Private Link) | Traffic to a specific cloud-managed resource that has a private endpoint rule. |
2 | AWS: gateway endpoints for Amazon S3 and DynamoDB. Azure: service endpoints. | Traffic that matches those endpoints, which stays on the cloud provider's backbone and can't be overridden by the gateway. |
3 | Private network gateway | The destinations you configure in |
4 | Default Databricks serverless egress | All other outbound traffic. |
A private network gateway works with your existing serverless networking as follows:
- Reuses your existing NCC: A private network gateway reuses your existing network connectivity configuration (NCC) rather than introducing a new object model. An NCC is an account-level object that manages serverless networking. After you create a private network gateway inside an NCC and attach that NCC to your workspaces, the serverless products in those workspaces can use the gateway automatically.
- Egress controls allow gateway destinations automatically: When you configure a private network gateway in
SPECIFIC_DESTINATIONSmode, Databricks automatically allows the destinations you specify in your serverless egress control. You don't need to add them to your network policy separately. This mirrors how domains added as Private Link entries for a network load balancer are implicitly allowlisted. If your private network gateway is inALL_TRAFFICmode, however, you must explicitly add all the destinations you intend to connect to through the gateway to your egress network policy. For more information, see What is serverless egress control?. - Private endpoint rules take precedence: If a private endpoint rule exists for a specific resource, traffic to that resource always uses the private endpoint rather than the private network gateway, even in
ALL_TRAFFICmode.
A private network gateway connects to resources in your VPC and transitively connected networks. It does not connect to cloud-managed services that are reached through gateway VPC endpoints.
Traffic modes
A gateway routes traffic in one of two modes, set by traffic_mode when you create it.
SPECIFIC_DESTINATIONS. Routes only the DNS names you list indestinationsthrough the gateway. All other traffic follows existing routing rules. This mode is recommended for most use cases because it gives fine-grained control without affecting unrelated traffic.ALL_TRAFFIC. Routes all outbound traffic from serverless compute through the gateway, except traffic that matches a more specific route such as Private Link. Use this mode when you need all egress to pass through your own firewall or security appliance.
Destination suffix matching is supported for SPECIFIC_DESTINATIONS mode: for example, specifying mydb.contoso.com also matches sub.mydb.contoso.com.
Limitations
The following limitations apply during Private Preview:
- The gateway and the subnet it connects to must be in the same region as the NCC.
- Private Preview supports exactly one subnet in one Availability Zone.
- You configure the gateway only through the account REST API. There is no UI or Terraform support.
- A gateway connects to resources in your VPC and transitively connected networks. It does not connect to AWS-managed services, such as Amazon S3, that are reached through VPC gateway or interface endpoints.
- A gateway supports serverless Databricks Runtime products.
- An NCC supports at most two gateways. A gateway supports at most two DNS resolvers and at most 100 destinations.