Skip to main content
Last updated on

Configure inbound Private Service Connect for performance-intensive services

Beta

This feature is in Beta. Account admins can manage access to this feature from the account console Previews page. See Manage account-level previews.

This page shows how to configure Private Service Connect for inbound connectivity to performance-intensive services on the Databricks platform. This private connection allows external clients and users to access services on the Databricks platform, such as Zerobus Ingest and Lakebase Autoscaling.

Databricks already supports Private Service Connect for connections to the workspace APIs and UI. This page covers Private Service Connect endpoints that target performance-intensive services instead. For workspace connectivity, see Configure Inbound Private Service Connect.

Benefits​

  • Enhanced security: Traffic between your network and Databricks services remains within the Google Cloud network infrastructure.
  • Access to performance-intensive services: Private connections to services like Zerobus Ingest and Lakebase Autoscaling.
  • Compliance requirements: Meet regulatory requirements that mandate private network connectivity.
  • Cost efficiency: Private Service Connect costs less than public connectivity options such as NAT gateways.

Requirements​

  • Your Databricks account must be on the Enterprise tier.
  • You must be a Databricks account admin to register VPC endpoints and manage ingress policies.
  • You must have permissions in Google Cloud to create Private Service Connect endpoints.

Regional service attachments reference​

To create a Private Service Connect endpoint for performance-intensive services, you need the service attachment URI for your region. The URI ends with the suffix regional-plproxy-psc-endpoint-all-ports, which distinguishes it from the workspace service attachment used by inbound Private Service Connect to the workspace APIs and UI.

To find the service attachment URI for your region, see Private Service Connect (PSC) attachment URIs and project numbers.

Configure a Private Service Connect endpoint​

Create a Private Service Connect endpoint in Google Cloud​

Create the endpoint in the Google Cloud project and VPC that your clients connect from:

  1. In the Google Cloud console, go to the Private Service Connect page.
  2. Click the Connected endpoints tab.
  3. Click + Connect endpoint.
  4. For Target, select Published service.
  5. In the Target service field, paste the service attachment URI for your region. See Regional service attachments reference.
  6. Enter a name for the endpoint. Databricks recommends a name that includes the region and purpose, such as us-central1-performance-intensive-psc.
  7. Select the VPC network and subnet that host the endpoint.
  8. Reserve an IP address for the endpoint.
  9. Click Create.
  10. Record the following values, which you need when you register the endpoint:
    • Your Google Cloud project ID
    • The Private Service Connect endpoint name
    • The private IP address that you reserved for the endpoint
note

After you create the endpoint, its connection state is Pending. This is expected. The connection is accepted when you register the endpoint with Databricks in the next step.

Performance-intensive services connect over TCP: Zerobus Ingest uses port 443, and Lakebase Autoscaling uses port 5432 (Postgres). Confirm that your firewall rules allow outbound TCP traffic on these ports from your clients to the endpoint IP address.

Register the endpoint with Databricks​

Register your Private Service Connect endpoint using the VPC Endpoint Configurations API. Send a POST request to /api/2.0/accounts/{accountId}/vpc-endpoints with the following fields:

  • endpoint_region: The Google Cloud region where the Private Service Connect endpoint exists.
  • project_id: The Google Cloud project where you created the endpoint.
  • psc_endpoint_name: The endpoint name in the Google Cloud console.
  • vpc_endpoint_name: A human-readable name for the registration.
JSON
{
  "gcp_vpc_endpoint_info": {
    "endpoint_region": "us-central1",
    "project_id": "my-prod-project",
    "psc_endpoint_name": "performance-intensive-psc"
  },
  "vpc_endpoint_name": "psc-vpce"
}

The response includes use_case set to SERVICE_DIRECT, the vpc_endpoint_id, and the psc_connection_id and service_attachment_id values in gcp_vpc_endpoint_info. Record the vpc_endpoint_id, which you need in the next step.

Create a private access settings object​

Create a private access settings object that allows workspace access using your new VPC endpoint. Send a POST request to /api/2.0/accounts/{accountId}/private-access-settings with the following fields:

  • private_access_settings_name: A name for the private access settings object.
  • region: The region that matches your workspace region.
  • private_access_level: Either ACCOUNT or ENDPOINT. ACCOUNT allows all VPC endpoints registered in your account. ENDPOINT limits access to the endpoints that you list.
  • allowed_vpc_endpoint_ids: The VPC endpoint IDs that can access the workspace. Set this field only when private_access_level is ENDPOINT.
  • public_access_enabled: Whether the workspace also accepts connections from the public internet.

Databricks recommends setting public_access_enabled to true while you test your DNS configuration, then changing it to false during a maintenance window to enforce exclusive private connectivity. When public_access_enabled is false, the workspace is reachable only through Private Service Connect.

To add the endpoint to an existing private access settings object instead, send a PUT request to the same endpoint.

Attach the private access settings object to a workspace​

Attach the private access settings object to each workspace that uses the endpoint. Send a PATCH request to /api/2.0/accounts/{accountId}/workspaces/{workspaceId} with the private_access_settings_id field set to the ID of your private access settings object.

Configure DNS to route traffic over Private Service Connect​

By default, performance-intensive services resolve using the public DNS name <product-instance>.<product>.<region>.gcp.databricks.com. For example, Zerobus Ingest uses <workspace-id>.zerobus.<region>.gcp.databricks.com.

Databricks recommends overriding DNS at the product level so that all traffic for the product flows toward your Private Service Connect endpoint:

  1. Create a private hosted zone for <product>.<region>.gcp.databricks.com and attach it to the VPC that hosts your Private Service Connect endpoint. A Google Cloud private zone resolves only for the networks that you attach it to, so the zone must be attached to that VPC for clients to use it.
  2. In that zone, add a wildcard A record that points to the private IP address of your Private Service Connect endpoint.
note

A wildcard record sends every hostname for that product and region to one endpoint IP address. If you use more than one Private Service Connect endpoint for the same product and region, add a record for each specific hostname instead of a wildcard, so that each product instance resolves to its intended endpoint.

Verify private connectivity​

Verify connectivity from a client in the VPC that hosts your Private Service Connect endpoint. The following example uses a Zerobus Ingest host.

First, confirm that the product host resolves to the private IP address of your Private Service Connect endpoint:

Bash
nslookup <workspace-id>.zerobus.<region>.gcp.databricks.com

The lookup must return the private IP address that you reserved in Create a Private Service Connect endpoint in Google Cloud. If it returns a public address instead, revisit your DNS configuration.

Then test the connection:

Bash
nc -vz <workspace-id>.zerobus.<region>.gcp.databricks.com 443

If the connection succeeds, your Private Service Connect endpoint for performance-intensive services is configured correctly. Testing the hostname rather than the IP address confirms that both DNS resolution and connectivity work over the private path. For product-specific connectivity testing, see the documentation for that service.

Limitations​

Be aware of the following limitations:

  • Supported services: On Google Cloud, inbound Private Service Connect for performance-intensive services supports Zerobus Ingest and Lakebase Autoscaling.
  • Regional availability: These services are available in a subset of Google Cloud regions. Confirm that the service you use is supported in your region before you configure the endpoint. See Ingestion availability.
  • VPC endpoint limit: The default limit for VPC endpoints of any type is 100 per account. If you require a quota increase, contact your Databricks account team.
  • Endpoints per region: Each account is limited to five VPC endpoints of type SERVICE_DIRECT per region.
  • Endpoints per project: Each customer Google Cloud project ID is limited to 10 Private Service Connect endpoints for performance-intensive services.

Troubleshooting​

If you encounter connectivity problems or want to share feedback on the Beta, contact your Databricks account team.

Additional resources​

On this page