Skip to main content
Last updated on

Configure front-end PrivateLink for performance-intensive services

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

This page shows how to configure AWS PrivateLink for front-end connectivity to performance-intensive services on the Databricks platform. This private connection allows external clients and users to access services on the Databricks platform, such as Zerobus Ingest and Lakebase Autoscaling.

Benefits​

  • Enhanced security: Traffic between your network and Databricks services remains within the AWS network infrastructure.
  • Access to performance-intensive services: Private connections to services like Zerobus Ingest and Lakebase Autoscaling.
  • Compliance requirements: Meet regulatory requirements that mandate private network connectivity.
  • Cost efficiency: PrivateLink costs less than public connectivity options such as NAT gateways.
note

Databricks does not currently bill for networking costs associated with front-end PrivateLink connections to performance-intensive services. Charges may be introduced in the future.

Requirements​

  • Your Databricks account must be on the Enterprise tier.
  • You must be a Databricks account admin to register VPC endpoints and manage private access settings.
  • You must have permissions in AWS to create VPC endpoints.
  • Your AWS region must support front-end PrivateLink. See PrivateLink VPC endpoint services for regional VPC endpoint services.

Step 1: Create VPC endpoints​

To create front-end VPC endpoints in the AWS Management Console:

  1. Go to the VPC endpoints section of the AWS Management Console.

  2. In the upper right, set the region to the same region as your transit VPC region.

  3. Click Create Endpoint.

  4. When naming the endpoint, Databricks recommends including the region and service-direct, such as databricks-us-west-2-service-direct-vpce.

  5. Under Service Category, select Endpoint services that use NLBs and GWLBs.

  6. In the service name field, paste the service name for your region. Use the table in PrivateLink VPC endpoint services to find the front-end PrivateLink service names (labeled Service-Direct in the table).

  7. Click Verify service and check that Service name verified appears in a green box. If you encounter an error that states "Service name could not be verified", verify that the regions of your VPC, subnets, and new VPC endpoint are correctly matched.

  8. In VPC, select your transit VPC.

  9. In Subnets, select a subnet. If you are deploying in ap-northeast-1, ap-northeast-2, us-east-1, or us-west-2, verify that you are using only the supported availability zones listed in Availability zone support.

  10. In the Security groups section, select the security group that you created for front-end connections.

  11. (Optional) Enable Private DNS names for the VPC endpoint.

    warning

    We recommend you do no enable Private DNS names in this step. Enabling private DNS immediately routes traffic over PrivateLink, but requests are rejected until you complete VPC endpoint registration and configuration in the following steps. Instead, configure DNS after completing setup, in Step 5.

  12. Click Create endpoint.

  13. Record the following information for later steps:

    • The VPC endpoint ID
    • The IPv4 address of the VPC endpoint in the subnet you selected

Step 2: Register VPC endpoints​

After you create your VPC endpoints in the AWS Management Console, register them with Databricks. You can't update a network configuration after you create it. Follow these steps to register your VPC endpoints:

  1. Go to the Databricks account console.
  2. Click Cloud resources in the sidebar.
  3. Select Network.
  4. Click VPC endpoint registrations from the vertical navigation.
  5. Click Register a VPC endpoint.
  6. Enter a descriptive name for your VPC endpoint registration.
    • A naming convention that includes the region and purpose is recommended, such as VPCE us-west-2 for service-direct.
  7. Select the appropriate region. The region must match both your workspace region and the region of the AWS VPC endpoints that you're registering.
  8. Paste the AWS VPC endpoint ID in the AWS VPC endpoint ID field.
  9. Click Register new VPC endpoint.

If you have multiple workspaces that share the same customer-managed VPC, you can share AWS VPC endpoints among them. For multiple Databricks accounts, register the AWS VPC endpoint in each account.

Step 3: Create private access settings​

Private access settings describe your workspace's PrivateLink connectivity and are required to enable private connectivity. To use PrivateLink, you must attach a private access settings object when you create your workspace. Follow these steps to create your private access settings:

  1. As an account admin, log in to the account console.
  2. In the sidebar, click Private access settings.
  3. Click Add private access settings.
  4. Enter a name for your new private access settings object.
  5. Select a region that matches your workspace region.
  6. Configure the Public access enabled field:
    • False (default): The front-end connection is accessible exclusively through PrivateLink, blocking public internet access.
    • True: The front-end connection is accessible through both PrivateLink and the public internet.
  7. Select a Private Access Level:
    • Account: Limit connections to VPC endpoints registered in your Databricks account.
    • Endpoint: Limit connections to an explicit set of VPC endpoints. Include your front-end VPC endpoint registration.
  8. Click Add private access settings.

To complete this step, your workspace must already be using a customer-managed VPC and secure cluster connectivity.

  1. To create a workspace, see Create a classic workspace. Refer to that page for guidance on workspace fields such as workspace URL, region, Unity Catalog, credential configurations, and storage configurations. Don't click the Save button yet.

  2. Click Advanced configurations to view additional fields.

  3. In the PrivateLink dropdown, choose the name of the private access settings object that you created in the previous steps.

  4. Click Save.

  5. After you create or update a workspace, wait until it becomes available before you use or create compute.

    The workspace status remains RUNNING and the VPC change happens immediately. However, you can't use or create compute for another 20 minutes. If you try to create or use compute before this interval ends, the compute might fail to launch or cause other unexpected behavior.

To complete your PrivateLink configuration, configure custom DNS settings to route traffic through your VPC endpoints. For detailed instructions, see Configure DNS for AWS front-end PrivateLink.

Step 6: Verify network connectivity​

Test connectivity from your client to verify that the front-end PrivateLink is configured correctly.

Verify DNS resolution​

Confirm that DNS queries resolve to the private IP address of the VPC endpoint you created earlier:

Bash
dig <region>.service-direct.privatelink.cloud.databricks.com

Or use nslookup as an alternative:

Bash
nslookup <region>.service-direct.privatelink.cloud.databricks.com

Both commands should return the private IP address of your VPC endpoint.

Test basic connectivity​

Verify that you can connect to the PrivateLink endpoint:

Bash
nc -vz <region>.service-direct.privatelink.cloud.databricks.com 443

If the connection succeeds, you configured your front-end PrivateLink correctly.

After verifying connectivity, optionally set public_access_enabled to false in your Private Access Setting to enforce private-only access.

note

For product-specific connectivity testing (such as Zerobus Ingest or Lakebase Autoscaling), refer to the documentation for those specific services.

Limitations​

Be aware of the following limitations:

  • VPC endpoint limit: The default limit for VPC endpoints of any type is 110 per account. If you require a quota increase, contact your Databricks account team.
  • Availability zone restrictions: Some AWS regions support PrivateLink only in specific availability zones. See Availability zone support.
  • Classic compute restrictions: When accessing front-end PrivateLink from a standard classic compute resource, the system blocks traffic to IP addresses in your workspace CIDR except on ports 80, 443, and 53. This restriction applies only when you create the VPC endpoint inside your workspace VPC. To work around this limitation, create the VPC endpoint in a separate VPC and configure VPC peering between the two VPCs.

Availability zone support​

Some AWS regions support PrivateLink for performance-intensive services only in specific availability zones. If you are deploying in one of the following regions, ensure your VPC endpoint subnets are in the supported availability zones:

Region

Supported availability zones

ap-northeast-1

apne1-az1, apne1-az2, apne1-az4

ap-northeast-2

apne2-az1, apne2-az3

us-east-1

use1-az1, use1-az2, use1-az4, use1-az5, use1-az6

us-west-2

usw2-az1, usw2-az2, usw2-az3, usw2-az4

All other supported regions allow any availability zone.

Next steps​