Skip to main content

Configure private connectivity to Databricks

This page covers configuration steps for front-end private connectivity between users and their Databricks workspaces. To add security to your workspace's serverless implementation, you must use front-end private connectivity.

Establish secure, private connections from your AWS VPCs or on-premises networks to Databricks services using front-end PrivateLink, which routes traffic through a VPC interface endpoint instead of the public internet.

With front-end PrivateLink, you can:

  • Configure private access: Configure front-end PrivateLink connections to the Databricks web application, REST API, and Databricks Connect API.
  • Enable private access: Configure PrivateLink when you create a new workspace or on an existing one.
  • Enforce private connectivity: Configure private connectivity from users to Databricks and from the control plane to the compute plane.

Requirements​

  • Your Databricks account is on the Enterprise plan.
  • Your Databricks workspace must use a customer-managed VPC. You can't convert an existing workspace from a Databricks-managed VPC to a customer-managed VPC. See Configure a customer-managed VPC.
  • You must have all necessary AWS permissions to set up a Databricks workspace and to create new VPC endpoints for your workspace.
  • To establish a front-end PrivateLink connection for accessing the workspace from your on-premises network, connect your on-premises network to an AWS VPC using Direct Connect or VPN.
  • Allow network traffic from all relevant address spaces within your local network to connect to your VPC endpoint using TCP port 443.

Step 1: Create VPC endpoints​

A front-end endpoint originates from your transit VPC, typically serving as the source for user web application access. This is usually a separate VPC from the workspace's compute plane VPC and can be connected to an on-premises network. If you have multiple Databricks accounts, you can share a front-end VPC endpoint across these accounts. Register the endpoint in each relevant Databricks account.

To create front-end VPC endpoints in the AWS Management Console:

  1. Go to the VPC endpoints section of the AWS Management Console.
  2. In the upper right, set the region to the same region as your transit VPC region.
  3. Click Create Endpoint.
  4. When naming the endpoint, Databricks recommends including the region and either workspace or front-end, such as databricks-us-west-2-workspace-vpce.
  5. Under Service Category, select Endpoint services that use NLBs and GWLBs.
  6. In the service name field, paste the service name. Use the table in PrivateLink VPC endpoint services to find the regional service names. Copy the one labeled Workspace (including REST API).
  7. Click Verify service and check that Service name verified appears in a green box. If you encounter an error that states "Service name could not be verified", verify that the regions of your VPC, subnets, and new VPC endpoint are correctly matched.
  8. In VPC, select your transit VPC.
  9. In Subnets, select a subnet.
  10. In the Security groups section, select the security group that you created for front-end connections.
  11. Click Create endpoint.

Step 2: Register VPC endpoints​

After you create your VPC endpoints in the AWS Management Console, register them with Databricks. You can't update a network configuration after it has been created. Follow these steps to register your VPC endpoints:

  1. Go to the Databricks account console.

  2. Click Cloud resources in the sidebar.

  3. Select Network.

  4. Click VPC endpoint registrations from the vertical navigation.

  5. Click Register a VPC endpoint.

  6. Enter a descriptive name for your VPC endpoint registration.

    • A naming convention that includes the region and purpose is recommended, such as VPCE us-west-2 for SCC.

  7. Select the appropriate region. The region must match both your workspace region and the region of the AWS VPC endpoints that you're registering.

  8. Paste the AWS VPC endpoint ID in the AWS VPC endpoint ID field.

    Register VPC Endpoint.

  9. Click Register new VPC endpoint.

If you have multiple workspaces that share the same customer-managed VPC, you can share AWS VPC endpoints among them. For multiple Databricks accounts, register the AWS VPC endpoint in each account.

Step 3: Create private access settings​

Private access settings describe your workspace's PrivateLink connectivity and are required to enable private connectivity. To use PrivateLink, you must attach a private access settings object when you create your workspace. Follow these steps to create your private access settings:

  1. In the account console, click Security in the sidebar.

  2. Click Private access settings in the sidebar.

  3. Click Add private access settings.

    Private access settings object.

  4. Enter a name for your new private access settings object.

  5. Select a region that matches your workspace region.

  6. Configure the Public access enabled field:

    • False (default): The front-end connection is accessible exclusively through PrivateLink, blocking public internet access.
    • True: The front-end connection is accessible through both PrivateLink and the public internet.

  7. Select a Private Access Level:

    • Account: Limit connections to VPC endpoints registered in your Databricks account.
    • Endpoint: Limit connections to an explicit set of VPC endpoints. Include your front-end VPC endpoint registration.

  8. Click Add private access settings.

To complete this step, your workspace must already be using a customer-managed VPC and secure cluster connectivity.

  1. See Manually create a workspace (existing Databricks accounts) to create a workspace. Refer to that page for guidance on workspace fields such as workspace URL, region, Unity Catalog, credential configurations, and storage configurations. Do not click the Save button yet.

  2. Click Advanced configurations to view additional fields.

  3. In the PrivateLink dropdown, choose the name of the private access settings object that you created in previous steps.

  4. Click Save.

  5. After you create or update a workspace, wait until it is available before you use or create clusters.

    The workspace status remains RUNNING and the VPC change happens immediately. However, you can't use or create clusters for another 20 minutes. If you create or use clusters before this interval ends, clusters might fail to launch or cause other unexpected behavior.

Step 5: Configure internal DNS to redirect user requests to the web application​

To route user requests to your front-end PrivateLink connection, update your private DNS settings. This process ensures that your workspace URL resolves to the private IP of your VPC endpoint.

Configure DNS mapping​

  1. Verify that your workspace includes PrivateLink.

  2. Configure your internal DNS to map the workspace URL to the private IP of your front-end VPC endpoint.

  3. Test the DNS resolution using the nslookup command:

    Bash
    nslookup my-workspace-name-here.cloud.databricks.com

Example response:

Non-authoritative answer:
my-workspace-name-here.cloud.databricks.com     canonical name = oregon.cloud.databricks.com.
oregon.cloud.databricks.com     canonical name = a89b3c627d423471389d6ada5c3311b4-f09b129745548506.elb.us-west-2.amazonaws.com.
Name:   a89b3c627d423471389d6ada5c3311b4-f09b129745548506.elb.us-west-2.amazonaws.com
Address: 44.234.192.47

DNS mapping examples​

For a workspace with a front-end VPC endpoint in AWS region us-east-1:

Default DNS mapping:

  • myworkspace.cloud.databricks.com maps to nvirginia.privatelink.cloud.databricks.com. In this case nvirginia is the control plane instance short name in that region.
  • nvirginia.privatelink.cloud.databricks.com maps to nvirginia.cloud.databricks.com.
  • nvirginia.cloud.databricks.com maps to the AWS public IPs.

Required DNS mapping after your changes:

  • myworkspace.cloud.databricks.com maps to nvirginia.privatelink.cloud.databricks.com.
  • nvirginia.privatelink.cloud.databricks.com maps to the private IP of your VPC endpoint for front-end connectivity.

Configuration options​

To map the workspace URL to the VPC endpoint private IP from your on-premises network, choose one of these options:

  1. Configure conditional forwarding for the workspace URL to use AmazonDNS.
  2. Create an A record for the workspace URL in your on-premises DNS that maps to the VPC endpoint private IP.
  3. Create a direct mapping where myworkspace.cloud.databricks.com points directly to the VPC endpoint private IP.

Required DNS records​

Create the following records on your DNS server to enable access to all services:

Record type

Record name

Value

A

<deployment-name>.cloud.databricks.com

PrivateLink interface IP

CNAME

dbc-dp-<workspace-id>.cloud.databricks.com

<deployment-name>.cloud.databricks.com

Verification​

After updating your DNS configuration:

  1. Test the configuration by accessing the Databricks workspace web application from your transit VPC.
  2. Test the REST API connectivity.
  3. Create a VPC endpoint in the transit VPC if needed for testing.

If you encounter errors, verify that all required DNS records have been properly configured.

Preview

Unified login with front-end PrivateLink is in Private Preview. You must contact your Databricks account team to request access to this preview. For more information on unified login, see Enable unified login.

If your users have access to the public internet, this step is not required.

When a user logs in to a workspace where unified login is enabled, the system redirects them to accounts.cloud.databricks.com in the authentication flow. To use unified login with a workspace where front-end PrivateLink is enabled and the user has no access to the public internet, follow these steps to configure your identity provider and internal DNS.

  1. As an account admin, log in to the account console.
  2. In the sidebar, click Settings.
  3. Click the Authentication tab.
  4. Next to Authentication, click Manage.
  5. Choose Single sign-on with my identity provider.
  6. Click Continue.
  7. Copy the value in the Databricks Redirect URL field.
  8. Replace accounts with accounts-pl-auth to get the Databricks PrivateLink Redirect URI.
  9. Go to your identity provider.
  10. Add the Databricks PrivateLink Redirect URI as an additional redirect URL. If you configure SSO using SAML, also add the Databricks PrivateLink Redirect URI as an additional entity ID.

If you have both PrivateLink and non-PrivateLink workspaces in your account, do not remove the Databricks Redirect URL with account from your identity provider redirect URLs.

Set up a private hosted zone for your transit VPC​

Perform the following configuration in your transit VPC to verify that the Databricks PrivateLink Redirect URI maps to the VPC endpoint private IP address for your workspace VPC endpoint.

  1. From your transit VPC, use the nslookup Unix command-line tool to get the DNS resolution using your workspace URL. See the example in Step 5: Configure internal DNS to redirect user requests to the web application.
  2. Copy the control plane instance URL of your PrivateLink workspace. The control plane instance URL is in the format <region>.privatelink.cloud.databricks.com.
  3. In your transit VPC, create a private hosted zone with the domain name privatelink.cloud.databricks.com.
  4. Add a CNAME record that resolves accounts-pl-auth.privatelink.cloud.databricks.com to your control plane instance URL.
  5. Test the configuration by accessing the Databricks PrivateLink Redirect URI from your transit VPC.

Next steps​

Last updated on