Configure private connectivity to resources in your VPC

Preview

This feature is in Public Preview.

note

Effective October 7, 2024, Databricks began charging customers for networking costs incurred from serverless compute resources connecting to external resources. Serverless network billing is rolling out in phases, which might result in gradual billing changes. For more information on billing, see Understand Databricks serverless networking costs.

This page explains how to use the Databricks account console UI to configure PrivateLink connections from serverless compute to resources in your virtual private cloud (VPC) through a network load balancer (NLB).

Configuring private connectivity for serverless compute provides:

• A dedicated and private connection: Your private endpoint is exclusively tied to your Databricks account, ensuring access is restricted to authorized workspaces only.
• Enhanced data exfiltration mitigation: While Databricks Serverless with Unity Catalog provides built-in data exfiltration protection, PrivateLink adds an extra layer of network defense. By placing your resources in a private subnet and controlling access through VPC endpoint services and dedicated VPC endpoints, you reduce the risk of unauthorized data movement.

Requirements

• The workspace is on the Enterprise plan.
• You are the account admin of your Databricks account.
• You have at least one workspace using serverless compute.
• Each Databricks account can have up to 10 NCCs per region.
• Each region can have 30 private endpoints, distributed as needed across 1-10 NCCs.
• Each NCC can be attached to up to 50 workspaces.
• Each private endpoint rule for private connectivity to resources in your VPC supports up to 10 domain names.

Step 1: Create a network load balancer (NLB) in AWS

Create a Network Load Balancer in AWS that will serve as the frontend for your VNet resources. To create an NLB, follow the instructions in the AWS documentation. Configure the NLB to route traffic to your target resource using an internal scheme.

Step 2: Create a VPC endpoint service in your AWS account

You must create a VPC endpoint service to securely expose your NLB to your private endpoint. For instructions, refer to the AWS documentation Create a service powered by AWS PrivateLink..

note

If your VPC endpoint service is in us-east-1, provision the VPC endpoint in one or more of the following availability zones:

• use1-az1
• use1-az2
• use1-az4
• use1-az5
• use1-az6

Be sure to complete the following:

• Ensure the Acceptance required box is checked for Required acceptance for endpoints.
• Allowlist a Databricks serverless stable IAM role. Choose one of these two methods:
  • For simplified deployment:
    • After your VPC endpoint service is created, navigate to the Allow principals tab.
      • Add the Databricks Serverless stable IAM role to the allowlist. This enables Databricks to create a VPC endpoint that links to your service. Use the following format: arn:aws:iam::565502421330:role/private-connectivity-role-<region>.
    • Example: For us-east-1 region, allowlist arn:aws:iam::565502421330:role/private-connectivity-role-us-east-1.
  • For granular permission control:
    • After your VPC endpoint service is created, allow * in the Allow principals tab. This allows Databricks to create one VPC endpoint to link to your VPC endpoint service.
• Ensure that Enforce inbound rules on PrivateLink traffic is not selected.

note

If your target resource is an RDS multi-AZ deployment, RDS does not provide stable IPs. In this case, the RDS IP changes each time the failover occurs. To automate NLB target group IP address synchronization, see Access Amazon RDS across VPCs using AWS PrivateLink and Network Load Balancer.

Step 3: Create a Network Connectivity Configuration (NCC) object

You can skip this step if an NCC exists in the same region and AWS account that you intend to use.

1. In the account console, click Cloud resources.
2. Click Network.
3. Click Network Connectivity Configuration.
4. Click Add Network Connectivity Configuration.
5. Enter a name for the NCC.
6. Choose the region. This must match your workspace region.
7. Click Add.

Step 4: Create an AWS interface VPC endpoint

This step links your VPC endpoint service to your NCC. To create a private endpoint:

1. In the account console, click Cloud resources.
2. Click Network Connectivity Configurations.
3. Select the NCC object you created in Step 3.
4. In the Private endpoint rules tab, click Add private endpoint rule.
5. Enter the full AWS VPC endpoint service name. This service name establishes the connection to the private endpoint's destination resources.
6. Provide the fully qualified domain name(s) (FQDNs) of the destination resource. The domain name should correspond to the destination resource referenced in your connection URL.
   • Example: If your destination is an RDS instance, the FQDN typically follows the format: <RDS instance DB identifier>.<random hash>.<region>.rds.amazonaws.com.
7. Return to the Private endpoint rules tab of the NCC detail page.
8. Select your new VPC endpoint.
9. Copy the VPC endpoint ID.

note

Domains added as PrivateLink entries are implicitly allowlisted in network policies. When a domain is removed or the private endpoint is deleted, it may take up to 24 hours for network policy controls to fully enforce the change.

Step 5: Accept the VPC endpoint on your VPC endpoint service

After creating the VPC endpoint rule in Databricks, you must approve the connection request in the AWS. To approve the connection:

1. Open the Amazon VPC console.
2. In the navigation pane, choose Endpoint services.
3. Select the endpoint service.
4. From the Endpoint connections tab, filter for the endpoint ID returned in step 5 using the search box.
5. Choose Actions. Accept endpoint connection request.
6. When prompted for confirmation, choose Accept.

It can take a few minutes for the connection to fully establish.

Step 6: Confirm VPC endpoint status

Verify the VPC endpoint connection is successfully established. To confirm the connection:

1. Refresh the Network Connectivity Configuration page in the account console.
2. On the Private endpoint rules tab, confirm that the Status column for your new private endpoint is ESTABLISHED.

Step 7: Attach the NCC to one or more workspaces

This step associates your configured private connectivity with your workspaces. Skip this step if your workspace is already attached to the desired NCC. To attach the NCC to a workspace:

1. Navigate to Workspaces in the left-hand navigation.
2. Select an existing workspace.
3. Select Update Workspace.
4. Under Network Connectivity Configuration, select the dropdown and choose the NCC you’ve created.
5. Repeat for all workspaces you’d like this NCC to apply to.

note

NCCs are regional objects that can only be attached to workspaces in the same region.

What's next

• Manage private endpoint rules: Control network traffic to and from your VPC endpoints by defining specific rules that permit or deny connections. See Manage private endpoint rules.
• Configure a firewall for serverless compute access: Implement a firewall to restrict and secure inbound and outbound network connections for your serverless compute environments. See Configure a firewall for serverless compute access.
• Understand data transfer and connectivity costs: Data transfer and connectivity refer to moving data into and out of Databricks serverless environments. Networking charges for serverless products only apply to customers using Databricks serverless compute. See Understand Databricks serverless networking costs.