Configure private connectivity to Databricks

This page covers configuration steps for front-end private connectivity between users and their Databricks workspaces. To add additional security to your workspace's serverless implementation, you must have front-end private connectivity.

• For an overview of private connectivity at Databricks see Classic compute plane networking.
• To enable back-end private connectivity to Databricks, see Enable private connectivity using AWS PrivateLink.

Enable front-end PrivateLink for your workspace

Establish secure, private connections from your AWS VPCs or on-premises networks to Databricks services using front-end PrivateLink, which routes traffic through a VPC interface endpoint instead of the public internet.

With front-end PrivateLink you can:

• Configure private access: You can configure front-end PrivateLinkconnections to the Databricks web application, REST API, and Databricks Connect API.
• Easily enable private access: You can configure PrivateLink during the creation of a new workspace or enabled on an existing one.
• Enforce mandatory private connectivity: You must configure private connectivity from users to Databricks and from the control plane to the compute plane in order to mandate private connectivity for a workspace.

Requirements

• Your Databricks account is on the Enterprise pricing tier.
• Your Databricks workspace must use a customer-managed VPC. You can't convert an existing workspace from a Databricks-managed VPC to a customer-managed VPC.See Configure a customer-managed VPC.
• You must have all necessary AWS permissions to set up a Databricks workspace and to create new VPC endpoints for your workspace.
• To establish a front-end PrivateLink connection for accessing the workspace from your on-premises network, you must connect your on-premises network to an AWS VPC using Direct Connect or VPN.
• You must allow network traffic from all relevant address spaces within your local network to connect to your VPC endpoint using TCP port 443.

Step 1: Create VPC endpoints

A front-end endpoint originates from your transit VPC, typically serving as the source for user web application access. This is usually a separate VPC from the workspace’s compute plane VPC and can be connected to an on-premises network. If you have multiple Databricks accounts, you can share a front-end VPC endpoint across these accounts. Register the endpoint in each relevant Databricks account.

To create front-end VPC endpoints in the AWS Management Console:

1. Go to the VPC endpoints section of the AWS Management Console.
2. In the upper right, set the region to the same region as your transit VPC region.
3. Click Create Endpoint.
4. When naming the endpoint, we suggest you include the region and either the word workspace or frontend, such as databricks-us-west-2-workspace-vpce.
5. Under Service Category, select Endpoint services that use NLBs and GWLBs.
6. In the service name field, paste the service name. Use the table in PrivateLink VPC endpoint services to find the regional service names. Copy the one labeled Workspace (including REST API).
7. Click Verify service and ensure that the page displays Service name verified in a green box. If you encounter an error that states “Service name could not be verified”, verify that the regions of your VPC, subnets, and new VPC endpoint are correctly matched.
8. In VPC, select your transit VPC.
9. In Subnets, select a subnet.
10. In the Security groups section, select the security group you created for front-end connections.
11. Click Create endpoint.

Step 2: Register VPC endpoints

After you create your VPC endpoints in the AWS Management Console, register them with Databricks to establish VPC endpoint registrations. You cannot update a network configuration after it has been created. Follow these steps to register your VPC endpoints:

1. Navigate to the Databricks account console.

2. Click Cloud resources in the sidebar.

3. Select Network.

4. Click VPC endpoint registrations from the vertical navigation.

5. Click Register a VPC endpoint.

6. Enter a descriptive name for your VPC endpoint registration.

   • Use a naming convention that includes the region and purpose like VPCE us-west-2 for SCC.

7. Select the appropriate region. The region must match both your workspace region and the region of the AWS VPC endpoints you're registering.

8. Paste the AWS VPC endpoint ID in the AWS VPC endpoint ID field.

   

9. Click Register new VPC endpoint

If you have multiple workspaces that share the same customer-managed VPC, you can share AWS VPC endpoints among them. For multiple Databricks accounts, register the AWS VPC endpoint in each account.

Step 3: Create private access settings

Private access settings describe your workspace's PrivateLink connectivity and are required for enabling private connectivity. To use PrivateLink, you must attach a private access settings object when you create your workspace. You cannot add it to an existing workspace. Follow these steps create your private access settings:

1. In the account console, click Cloud Resources in the sidebar.

2. Select Network.

3. Click Private access settings in the sidebar.

4. Click Add private access settings.

   

5. Enter a name for your new private access settings object.

6. Select a region that matches your workspace region.

7. Configure the Public access enabled field:

   • False (default): The front-end connection is accessible exclusively through PrivateLink, blocking public internet access.
   • True: The front-end connection is accessible through both PrivateLink and the public internet.

8. Select a Private Access Level:

   • Account: Limit connections to VPC endpoints registered in your Databricks account.
   • Endpoint: Limit connections to an explicit set of VPC endpoints. Include your front-end VPC endpoint registration.

9. Click Add private access setting.

Step 4: Create your workspace with PrivateLink objects

To complete this step, your workspace must already be using a customer-managed VPC and secure cluster connectivity.

1. See Manually create a workspace (existing Databricks accounts) to create a workspace. Refer to that article for guidance on workspace fields such as workspace URL, region, Unity Catalog, credential configurations, and storage configurations. Do not click the Save button yet.

2. Click Advanced configurations to view additional fields.

3. In the Private Link dropdown, choose the name of the private access settings object that you created.

4. Click Save.

5. After you create or update a workspace, wait until it’s available before you use or create clusters.

   The workspace status stays at status RUNNING and the VPC change happens immediately. However, you can't use or create clusters for another 20 minutes. If you create or use clusters before this time interval elapses, clusters do not launch successfully, fail, or could cause other unexpected behavior.

Step 5: Configure internal DNS to redirect user requests to the web application

To route user requests to your front-end PrivateLink connection, you must update your private DNS settings. This process ensures that your workspace URL resolves to the private IP of your VPC endpoint.

Configure DNS mapping

1. Verify that your workspace includes PrivateLink.

2. Configure your internal DNS to map the workspace URL to your front-end VPC endpoint private IP.

3. Test the DNS resolution using the nslookup command:

   Bash

   nslookup my-workspace-name-here.cloud.databricks.com

Example response:

Non-authoritative answer:
my-workspace-name-here.cloud.databricks.com     canonical name = oregon.cloud.databricks.com.
oregon.cloud.databricks.com     canonical name = a89b3c627d423471389d6ada5c3311b4-f09b129745548506.elb.us-west-2.amazonaws.com.
Name:   a89b3c627d423471389d6ada5c3311b4-f09b129745548506.elb.us-west-2.amazonaws.com
Address: 44.234.192.47F

DNS mapping examples

For a workspace with a front-end VPC endpoint in AWS region us-east-1:

Default DNS mapping:

• myworkspace.cloud.databricks.com maps to nvirginia.privatelink.cloud.databricks.com. In this case nvirginia is the control plane instance short name in that region.
• nvirginia.privatelink.cloud.databricks.com maps to nvirginia.cloud.databricks.com.
• nvirginia.cloud.databricks.com maps to the AWS public IPs.

Required DNS mapping after your changes:

• myworkspace.cloud.databricks.com maps to nvirginia.privatelink.cloud.databricks.com.
• nvirginia.privatelink.cloud.databricks.com maps to the private IP of your VPC endpoint for front-end connectivity.

Configuration options

To map the workspace URL to the VPC endpoint private IP from your on-premises network, choose one of these options:

1. Configure conditional forwarding for the workspace URL to use AmazonDNS.
2. Create an A-record for the workspace URL in your on-premises DNS that maps to the VPC endpoint private IP.
3. Create a direct mapping where myworkspace.cloud.databricks.com points directly to the VPC endpoint private IP.

Required DNS records

Create the following records on your DNS server to enable access to all services:

Record type	Record name	Value
A	<deployment-name>.cloud.databricks.com	PrivateLink interface IP
CNAME	dbc-dp-<workspace-id>.cloud.databricks.com	<deployment-name>.cloud.databricks.com

Verification

After updating your DNS configuration:

1. Test the configuration by accessing the Databricks workspace web application from your transit VPC.
2. Test the REST API connectivity.
3. Create a VPC endpoint in the transit VPC if needed for testing.

If you encounter errors, verify that all required DNS records have been properly configured.

Step 6: Configure front-end PrivateLink with unified login

Preview

Unified login with front-end PrivateLink is in Private Preview. You must contact your Databricks account team to request access to this preview. For more information on unified login, see Enable unified login.

If your users have access to the public internet, this step is not required.

When a user logs in to a workspace where unified login is enabled, they are redirected to accounts.cloud.databricks.com in the authentication flow. To use unified login with a workspace where front-end PrivateLink is enabled and the user has no access to the public internet, you must follow these steps to configure your identity provider and internal DNS.

Authorize the PrivateLink redirect URI in your identity provider

1. As an account admin, log in to the account console.
2. In the sidebar, click Settings.
3. Click the Authentication tab.
4. Next to Authentication, click Manage.
5. Choose Single sign-on with my identity provider.
6. Click Continue.
7. Copy the value in the Databricks Redirect URL field.
8. Replace accounts with accounts-pl-auth to get the Databricks PrivateLink Redirect URI.
9. Go to your identity provider.
10. Add the Databricks PrivateLink Redirect URI as an additional redirect URL. If you configure SSO using SAML, also add the Databricks PrivateLink Redirect URI as an additional entity ID.

If you have both private link and non-private link workspaces in your account, do not remove the Databricks Redirect URL with account from your identity provider redirect URLs.

Set up a private hosted zone for your transit VPC

Perform the following configuration in your transit VPC to ensure that the DatabricksPrivateLink Redirect URI maps to the VPC endpoint private IP address for your workspace VPC endpoint.

1. From your transit VPC, Use the nslookup Unix command line tool to get the DNS resolution using your workspace URL. See the example in Step 5: Configure internal DNS to redirect user requests to the web application.
2. Copy the control plane instance URL of your PrivateLink workspace. The control plane instance URL is in the format <region>.privatelink.cloud.databricks.com.
3. In your transit VPC, create a private hosted zone with domain name privatelink.cloud.databricks.com.
4. Add a CNAME record that resolves accounts-pl-auth.privatelink.cloud.databricks.com to your control plane instance URL.
5. Test the configuration by accessing the Databricks PrivateLink Redirect URI from your transit VPC.