Skip to main content
Last updated on

Configure inbound PrivateLink for performance-intensive services

This page shows how to configure PrivateLink for inbound connectivity to performance-intensive services on the Databricks platform. This private connection allows external clients and users to access services on the Databricks platform, such as Zerobus Ingest and Lakebase Autoscaling.

Benefits​

  • Enhanced security: Traffic between your network and Databricks services remains within the AWS network infrastructure.
  • Access to performance-intensive services: Private connections to services like Zerobus Ingest and Lakebase Autoscaling.
  • Compliance requirements: Meet regulatory requirements that mandate private network connectivity.
  • Cost efficiency: PrivateLink costs less than public connectivity options such as NAT gateways.

Databricks bills for certain networking costs associated with inbound PrivateLink connections to performance-intensive services. For more information, see Understand Databricks networking costs.

Requirements​

  • Your Databricks account must be on the Enterprise tier.
  • You must be a Databricks account admin to register VPC endpoints and manage private access settings.
  • You must have permissions in AWS to create VPC endpoints.
  • Your AWS region must support inbound PrivateLink. See PrivateLink VPC endpoint services for regional VPC endpoint services.

Step 1: Create VPC endpoints​

To create inbound VPC endpoints in the AWS Management Console:

  1. Go to the VPC endpoints section of the AWS Management Console.

  2. In the upper right, set the region to the same region as your transit VPC region.

  3. Click Create Endpoint.

  4. When naming the endpoint, Databricks recommends including the region and the purpose, such as databricks-us-west-2-inbound-pl-vpce.

  5. Under Service Category, select Endpoint services that use NLBs and GWLBs.

  6. In the service name field, paste the service name for your region. Use the table in PrivateLink VPC endpoint services to find the inbound PrivateLink service names (labeled Inbound private link for performance-intensive services in the table).

  7. Click Verify service and check that Service name verified appears in a green box. If you encounter an error that states "Service name could not be verified", verify that the regions of your VPC, subnets, and new VPC endpoint are correctly matched.

  8. In VPC, select your transit VPC.

  9. In Subnets, select a subnet. If you are deploying in ap-northeast-1, ap-northeast-2, us-east-1, or us-west-2, verify that you are using only the supported availability zones listed in Availability zone support.

  10. In the Security groups section, select the security group that you created for inbound connections.

  11. (Optional) Enable Private DNS names for the VPC endpoint.

    warning

    We recommend you do not enable Private DNS names in this step. Enabling private DNS immediately routes traffic over PrivateLink, but requests are rejected until you complete VPC endpoint registration and configuration in the following steps. Instead, configure DNS after completing setup, in Step 5.

  12. Click Create endpoint.

  13. Record the following information for later steps:

    • The VPC endpoint ID
    • The IPv4 address of the VPC endpoint in the subnet you selected
note

The security group that you select for inbound connections must allow the inbound traffic for the service you connect to over the VPC endpoint:

  • Lakebase Autoscaling: Allow inbound TCP on port 5432 (Postgres) from the security groups or CIDR ranges of the clients that connect. This covers psql, Postgres drivers and ORMs, and the Tables editor and SQL editor.
  • Zerobus Ingest: Allow inbound TCP on port 443.

You do not need to configure outbound rules. Security groups are stateful, so response traffic for an allowed inbound connection is permitted automatically.

Step 2: Register VPC endpoints​

After you create your VPC endpoints in the AWS Management Console, register them with Databricks. You can't update a network configuration after you create it. Follow these steps to register your VPC endpoints:

  1. Go to the Databricks account console.
  2. Click Security in the sidebar.
  3. Click Private endpoints from the vertical navigation.
  4. Click Register a private endpoint.
  5. Enter a descriptive name for your VPC endpoint registration.
    • A naming convention that includes the region and purpose is recommended, such as VPCE us-west-2 for Service-Direct.
  6. Select the appropriate region. The region must match both your workspace region and the region of the AWS VPC endpoint you're registering.
  7. Paste the AWS VPC endpoint ID in the AWS VPC endpoint ID field.
  8. Click Register new VPC endpoint.

If you use Classic compute and have multiple workspaces that share the same customer-managed VPC, you can share AWS VPC endpoints among them. For multiple Databricks accounts, you can register the AWS VPC endpoint in each account.

Step 3: Configure private access policy​

Once you've registered your Service Direct VPC endpoint, there are two ways to configure a private access policy for your workspace: with context-based ingress, or with private access settings. Using the context-based ingress approach lets you configure fine-grained access by combining identity, request type, and network source conditions. See Context-based ingress control. Private access settings are all-or-nothing, region-bound policies.

For traffic to be allowed, both the private access settings and context-based ingress must permit the endpoint. By default, context-based ingress is set to Allow access from all private endpoints, which defers the access decision to the workspace's private access setting. If you would like to configure context-based private access policies, make sure your workspace has a private access setting attached, with all registered private endpoints allowed (see more below). This will defer the access decision to your workspace's context-based ingress policy. You can reuse the same "allow all access" private access setting across any workspace in the same region.

Preview

Configuring private access to workspaces using context-based ingress (Method 1) is in Beta.

  1. In the account console, click Security in the sidebar.

  2. Click Context-based ingress & egress control in the sidebar.

  3. Under Workspace level policies, click New workspace policy.

  4. Under Ingress > Private Network Access, define your private access policy.

    1. By default, all registered endpoints are allowed: Allow access from all private endpoints. If this is acceptable, proceed to the next step. This default includes only the first 200 registered VPC endpoints; if your policy requires more, contact your account team for an increase.
    2. If you want to allowlist specific registered endpoints while denying all other endpoints, uncheck Allow access from all private endpoints and add an allow rule.
      1. Select the identities and workspace destinations you want to allow access to (by default all are allowed).
      2. Then select source type = Selected private endpoints, and select the Service Direct VPC endpoint(s) you registered. The region of this VPC endpoint must match the region of your workspace. A Service Direct endpoint can only serve workspaces in the same region.
      3. Click Confirm.
      4. You can also add Deny rules in your policy, which define exceptions to your allow rules.

  5. When you are done configuring your private access policy, you can also configure your public access policy in Ingress > Public Network Access. You can disable all public IP access by unchecking Allow access from all public IPs. Databricks recommends keeping public access enabled while testing your DNS configuration, then disabling it after DNS is finalized to enforce exclusive private connectivity.

  6. Test your ingress policy in Dry run mode first, before switching it to Enforced. Dry run mode access denials are logged in the system.access.inbound_network system table but will not block access.

  7. Make sure your Egress policy is correct.

  8. Attach your policy to your workspace.

  9. Save your network policy. Context-based ingress policy updates take under 10 minutes to take effect. You can use one network policy per workspace, or share a single policy across workspaces that use the same ingress and egress rules, so you define it once and attach it anywhere.

  10. Attach an "allow all access" private access setting to your workspace. Workspaces with no private access settings attached have all private access denied, regardless of network policy.

    1. In the account console, click Security in the sidebar.
    2. Click Private access settings in the sidebar.
    3. Click Add private access settings.
    4. Enter a name for your new private access settings object.
    5. Select a region that matches your workspace region.
    6. Configure the Public access enabled field to True.
    7. Select a Private Access Level: Account.
    8. Click Add private access settings.
    9. Now in your workspace, click Advanced configurations to view additional fields.
    10. In the PrivateLink dropdown, choose the name of the private access settings object that you created in the previous steps.
    11. Click Save.

To complete your PrivateLink configuration, configure custom DNS settings to route traffic through your VPC endpoints. For detailed instructions, see Configure DNS for AWS inbound Private Link.

Step 5: Verify network connectivity​

Test connectivity from your client to verify that the inbound PrivateLink is configured correctly.

Verify DNS resolution​

Confirm that DNS queries resolve to the private IP address of the VPC endpoint you created earlier:

Bash
dig <region>.service-direct.privatelink.cloud.databricks.com

Or use nslookup as an alternative:

Bash
nslookup <region>.service-direct.privatelink.cloud.databricks.com

Both commands should return the private IP address of your VPC endpoint.

Test basic connectivity​

Verify that you can connect to the PrivateLink endpoint:

Bash
nc -vz <region>.service-direct.privatelink.cloud.databricks.com 443

The 443 check only confirms that the VPC endpoint is reachable. It is not the data path for Lakebase Autoscaling, which uses Postgres on port 5432. To validate Lakebase Autoscaling connectivity, test the Postgres port against your Lakebase host: nc -vz <lakebase-host> 5432.

If the connection succeeds, you configured your inbound PrivateLink correctly. For product-specific connectivity testing (such as Zerobus Ingest or Lakebase Autoscaling), see the documentation for those services.

Limitations​

Be aware of the following limitations:

  • VPC endpoint limit: The default limit for VPC endpoints of any type is 110 per account. If you require a quota increase, contact your Databricks account team.
  • Availability zone restrictions: Some AWS regions support PrivateLink only in specific availability zones. See Availability zone support.
  • Classic compute restrictions: When accessing inbound PrivateLink from a standard classic compute resource, the system blocks traffic to IP addresses in your workspace CIDR except on ports 80, 443, and 53. This restriction applies only when you create the VPC endpoint inside your workspace VPC. To work around this limitation, create the VPC endpoint in a separate VPC and configure VPC peering between the two VPCs.

Availability zone support​

Some AWS regions support PrivateLink for performance-intensive services only in specific availability zones. If you are deploying in one of the following regions, ensure your VPC endpoint subnets are in the supported availability zones:

Region

Supported availability zones

ap-northeast-1

apne1-az1, apne1-az2, apne1-az4

ap-northeast-2

apne2-az1, apne2-az3

us-east-1

use1-az1, use1-az2, use1-az4

us-west-2

usw2-az1, usw2-az2, usw2-az3

Region

Supported availability zones

ap-northeast-1

apne1-az1, apne1-az2, apne1-az4

ap-northeast-2

apne2-az1, apne2-az3

us-east-1

use1-az1, use1-az2, use1-az4

us-west-2

usw2-az1, usw2-az2, usw2-az3

All other supported regions allow any availability zone.

Additional resources​

On this page