Skip to main content
Last updated on

Configure inbound Private Service Connect for performance-intensive services

This page shows how to configure Private Service Connect for inbound connectivity to performance-intensive services on the Databricks platform. This private connection allows external clients and users to access services on the Databricks platform, such as Zerobus Ingest and Lakebase Autoscaling.

Databricks already supports Private Service Connect for connections to the workspace APIs and UI. This page covers Private Service Connect endpoints that target performance-intensive services instead. For workspace connectivity, see Configure inbound Private Service Connect for workspaces.

Benefits​

  • Enhanced security: Traffic between your network and Databricks services remains within the Google Cloud network infrastructure.
  • Access to performance-intensive services: Private connections to services like Zerobus Ingest and Lakebase Autoscaling.
  • Compliance requirements: Meet regulatory requirements that mandate private network connectivity.
  • Cost efficiency: Private Service Connect costs less than public connectivity options such as NAT gateways.

Requirements​

  • Your Databricks account must be on the Enterprise tier.
  • You must be a Databricks account admin to register VPC endpoints and manage ingress policies.
  • You must have permissions in Google Cloud to create Private Service Connect endpoints.

Regional service attachments reference​

To create a Private Service Connect endpoint for performance-intensive services, you need the service attachment URI for your region. The URI ends with the suffix regional-plproxy-psc-endpoint-all-ports, which distinguishes it from the General Access service attachment (plproxy-psc-endpoint-all-ports) used by inbound Private Service Connect to the workspace APIs and UI.

To find the service attachment URI for your region, see Private Service Connect (PSC) attachment URIs and project numbers.

Configure a Private Service Connect endpoint​

Create a Private Service Connect endpoint in Google Cloud​

Create the endpoint in the Google Cloud project and VPC that your clients connect from:

  1. In the Google Cloud console, go to the Private Service Connect page.
  2. Click the Connected endpoints tab.
  3. Click + Connect endpoint.
  4. For Target, select Published service.
  5. In the Target service field, paste the service attachment URI for your region. See Regional service attachments reference.
  6. Enter a name for the endpoint. Databricks recommends a name that includes the region and purpose, such as us-central1-performance-intensive-psc.
  7. Select the VPC network and subnet that host the endpoint.
  8. Reserve an IP address for the endpoint.
  9. Click Create.
  10. Record the following values, which you need when you register the endpoint:
    • Your Google Cloud project ID
    • The Private Service Connect endpoint name
    • The private IP address that you reserved for the endpoint
note

After you create the endpoint, its connection state is Pending. This is expected. The connection is accepted when you register the endpoint with Databricks in the next step.

Performance-intensive services connect over TCP: Zerobus Ingest uses port 443, and Lakebase Autoscaling uses port 5432 (Postgres wire) and 443 (Data API and other HTTPS-based clients). Confirm that your firewall rules allow outbound TCP traffic on these ports from your clients to the endpoint IP address.

Register the endpoint with Databricks​

Register your Private Service Connect endpoint using the Databricks account console:

  1. Go to the Databricks account console.
  2. Click Security in the sidebar.
  3. Click Private endpoints.
  4. Click Register a private endpoint.
  5. Fill in the required fields:
    • VPC endpoint name: A human-readable name for the registration. Databricks recommends a name that includes the region and purpose.
    • Region: The Google Cloud region where the Private Service Connect endpoint exists. The region must match your workspace region.
    • Google Cloud VPC network project ID: The Google Cloud project where you created the endpoint.
    • Google Cloud Private Service Connect (PSC) endpoint ID: The endpoint name in the Google Cloud console.

You can also register the endpoint using the VPC Endpoint Configurations API. Send a POST request to /api/2.0/accounts/{accountId}/vpc-endpoints with the following fields:

  • endpoint_region: The Google Cloud region where the Private Service Connect endpoint exists.
  • project_id: The Google Cloud project where you created the endpoint.
  • psc_endpoint_name: The endpoint name in the Google Cloud console.
  • vpc_endpoint_name: A human-readable name for the registration.
JSON
{
  "gcp_vpc_endpoint_info": {
    "endpoint_region": "us-central1",
    "project_id": "my-prod-project",
    "psc_endpoint_name": "performance-intensive-psc"
  },
  "vpc_endpoint_name": "psc-vpce"
}

The response includes use_case set to SERVICE_DIRECT, the vpc_endpoint_id, and the psc_connection_id and service_attachment_id values in gcp_vpc_endpoint_info.

Configure private access policy​

After you register your Private Service Connect endpoint, there are two ways to configure a private access policy for your workspace: with context-based ingress, or with private access settings. Using the context-based ingress approach lets you configure fine-grained access by combining identity, request type, and network source conditions. See Context-based ingress control. Private access settings are all-or-nothing, region-bound policies.

For traffic to be allowed, both the private access settings and context-based ingress must permit the endpoint. By default, context-based ingress is set to Allow access from all private endpoints, which defers the access decision to the workspace's private access settings. To use context-based private access policies, attach a private access settings object to your workspace with Public access enabled set to True and Private Access Level set to Account. This defers the access decision to your workspace's context-based ingress policy. You can reuse the same "allow all access" private access settings object across any workspace in the same region.

note

To configure private access to workspaces using context-based ingress (Method 1), an account admin must turn on Context-Based Ingress: Workspace Private Access Policies from the account console Previews page. See Manage account-level previews.

  1. In the account console, click Security in the sidebar.

  2. Click Context-based ingress & egress control in the sidebar.

  3. Under Workspace level policies, click New workspace policy.

  4. Under Ingress > Private Network Access, define your private access policy.

    1. By default, all registered endpoints are allowed: Allow access from all private endpoints. If this is acceptable, proceed to the next step. This default includes only the first 200 registered VPC endpoints. If your policy requires more, contact your account team for an increase.
    2. To allowlist specific registered endpoints while denying all other endpoints, uncheck Allow access from all private endpoints and add an allow rule.
      1. Select the identities and workspace destinations you want to allow access to. By default, all are allowed.
      2. For Source type, select Selected private endpoints, and select the Private Service Connect endpoints you registered. The region of the endpoint must match the region of your workspace.
      3. Click Confirm.
      4. Optionally, add Deny rules to your policy to define exceptions to your allow rules.

  5. Configure your public access policy in Ingress > Public Network Access. To disable all public IP access, uncheck Allow access from all public IPs. Databricks recommends keeping public access enabled while testing your DNS configuration, then disabling it after DNS is finalized to enforce exclusive private connectivity.

  6. Test your ingress policy in Dry run mode before switching it to Enforced. Dry run mode access denials are logged in the system.access.inbound_network system table but don't block access.

  7. Make sure your Egress policy is correct.

  8. Attach your policy to your workspace.

  9. Save your network policy. Context-based ingress policy updates typically take effect in under 10 minutes. You can use one network policy per workspace, or share a single policy across workspaces that use the same ingress and egress rules, so you define it one time and attach it anywhere.

  10. Attach an "allow all access" private access settings object to your workspace. Workspaces with no private access settings object attached have all private access denied, regardless of network policy. Follow the steps in Method 2, with Public access enabled set to True and Private Access Level set to Account. Leave Public access enabled set to True and use your context-based ingress policy to restrict access.

Configure DNS to route traffic over Private Service Connect​

By default, performance-intensive services resolve using the public DNS name <product-instance>.<product>.<region>.gcp.databricks.com. For example, Zerobus Ingest uses <workspace-id>.zerobus.<region>.gcp.databricks.com.

Databricks recommends overriding DNS at the product level so that all traffic for the product flows toward your Private Service Connect endpoint:

  1. Create a private hosted zone for <product>.<region>.gcp.databricks.com and attach it to the VPC that hosts your Private Service Connect endpoint. A Google Cloud private zone resolves only for the networks that you attach it to, so the zone must be attached to that VPC for clients to use it.
  2. In that zone, add a wildcard A record that points to the private IP address of your Private Service Connect endpoint.
note

A wildcard record sends every hostname for that product and region to one endpoint IP address. If you use more than one Private Service Connect endpoint for the same product and region, add a record for each specific hostname instead of a wildcard, so that each product instance resolves to its intended endpoint.

Verify private connectivity​

Verify connectivity from a client in the VPC that hosts your Private Service Connect endpoint. The following example uses a Zerobus Ingest host.

First, confirm that the product host resolves to the private IP address of your Private Service Connect endpoint:

Bash
nslookup <workspace-id>.zerobus.<region>.gcp.databricks.com

The lookup must return the private IP address that you reserved in Create a Private Service Connect endpoint in Google Cloud. If it returns a public address instead, revisit your DNS configuration.

Then test the connection:

Bash
nc -vz <workspace-id>.zerobus.<region>.gcp.databricks.com 443

If the connection succeeds, your Private Service Connect endpoint for performance-intensive services is configured correctly. Testing the hostname rather than the IP address confirms that both DNS resolution and connectivity work over the private path. For product-specific connectivity testing, see the documentation for that service.

Limitations​

Be aware of the following limitations:

  • Supported services: On Google Cloud, inbound Private Service Connect for performance-intensive services supports Zerobus Ingest and Lakebase Autoscaling.
  • Regional availability: These services are available in a subset of Google Cloud regions. Confirm that the service you use is supported in your region before you configure the endpoint. See Ingestion availability.
  • Endpoints per project and region: For each Databricks account, a customer Google Cloud project ID is limited to 10 Private Service Connect endpoints for performance-intensive services in each region. If you require a quota increase, contact your Databricks account team.

Troubleshooting​

If you encounter connectivity problems or want to share feedback, contact your Databricks account team.

Additional resources​

On this page