Skip to main content
Unlisted page
This page is unlisted. Search engines will not index it, and only users having a direct link can access it.
Last updated on

Configure private connectivity to Google APIs

Preview

This feature is in Beta. To join this Beta, contact your Databricks account team.

note

There are currently no networking charges for serverless features. In a later release, you might be charged. Databricks will provide advance notice for networking pricing changes.

See Understand Databricks networking costs.

Serverless compute can connect privately to Google APIs, such as Cloud Storage and BigQuery, using GCP Private Service Connect (PSC). You configure this connection in the Databricks account console.

Configuring private connectivity for serverless compute provides:

  • A dedicated and private connection: The private endpoint is dedicated to your Databricks account and accessible only from your authorized workspaces.
  • Enhanced data exfiltration mitigation: Serverless compute with Unity Catalog provides built-in data exfiltration protection. Private Service Connect adds a network-layer control: you can allowlist your dedicated PSC endpoint in your VPC Service Controls perimeter so that only traffic from that endpoint can reach your Google API resources.

How private connectivity to Google APIs works​

Serverless network connectivity is managed with network connectivity configurations (NCCs). An account admin creates an NCC in the account console, and a single NCC can be attached to one or more workspaces.

When you add a private endpoint rule that targets Google APIs, Databricks provisions a Private Service Connect endpoint from the serverless compute plane toward the Google APIs you specify. Databricks then returns a PSC endpoint URI for the rule. You add that URI to your VPC Service Controls perimeter as an ingress rule, then enable the rule so that serverless traffic to those Google APIs routes over the Google Cloud backbone instead of the public internet.

For more information on NCCs, see Serverless compute plane networking.

Requirements​

  • All workspaces attached to the NCC are on the Enterprise plan.
  • You are an account admin of your Databricks account.
  • You have at least one active workspace using serverless compute.
  • This feature is available in Google Cloud regions where serverless compute is enabled. See Features with limited regional availability. An NCC is a regional object, and you can attach it only to workspaces in the same region.
  • There is no Terraform automation support for the Google APIs private endpoint.

Each Databricks account has the following quotas on Google Cloud:

  • Each NCC can be attached to up to 50 workspaces.
  • Each account can have up to 10 NCCs per region.
  • Each region can have up to 10 private endpoints, distributed as needed across your NCCs.

Enable the preview​

  1. Contact your account team to request access to the preview. Your account team files a ticket for approval. Allowlisting can take up to seven days.
  2. After your account is allowlisted, go to the account console, select Previews in the sidebar, and enable Outbound (serverless) GCP Private Link to Google APIs.

Configure private connectivity​

Configure private connectivity to Google APIs using the account console.

Create a network connectivity configuration​

You can skip this step if you have an existing NCC in the same region that you want to use.

  1. As an account admin, go to the account console.
  2. In the sidebar, click Security.
  3. Click the Networking tab.
  4. Under Network Connectivity Configurations, click Add Network Connectivity Configuration.
  5. Enter a name for the NCC.
  6. Select the region. This must match your workspace region.
  7. Click Add.

Create a private endpoint rule toward Google APIs​

  1. Select the NCC that you created, and go to the Private endpoint rules section.
  2. Click Add private endpoint rule.
  3. Select Google API as the resource type.
  4. Choose the Google APIs to target:
    • Select specific Google APIs, such as Cloud Storage or BigQuery.
    • Or select All Google APIs to create a catch-all rule for all Google APIs.
    • Or select All VPC-SC compatible services to create a catch-all rule for all Google APIs that support VPC Service Controls. New services are discovered automatically.
  5. Click Add.

Get the PSC endpoint URI​

Wait a few minutes for the private endpoint to finish provisioning. The connection status transitions to ESTABLISHED. Refresh the page, then copy the PSC endpoint URI from the rule.

Allowlist the endpoint in your VPC Service Controls perimeter​

important

You perform this step in your own Google Cloud account, not in Databricks.

Add the PSC endpoint URI to your VPC Service Controls perimeter as an ingress rule. The Private Service Connect endpoint is outbound from Databricks serverless compute toward your Google API resources.

Enable the rule​

After the endpoint has access to your resources within your perimeter, enable the rule.

  1. In the Private endpoint rules section, find your rule and click the kebab menu Kebab menu icon..
  2. Click Update rule.
  3. Select Enable rule.
important

This step routes traffic for the configured Google APIs through Private Service Connect for any workspace attached to the NCC. Before you continue, verify that you have allowlisted the PSC endpoint URI in your VPC Service Controls perimeter.

Attach the NCC to one or more workspaces​

Skip this step if your workspace is already attached to the NCC. To attach the NCC to a workspace:

  1. Go to Workspaces in the sidebar.
  2. Select an existing workspace.
  3. Click Update Workspace.
  4. Under Network Connectivity Configuration, select the NCC you created.
  5. Repeat for all workspaces you want this NCC to apply to.

An NCC is a regional object, so you can attach it only to workspaces in the same region.

Verify connectivity​

The following example tests connectivity to a Cloud Storage bucket by registering it as an external location and running a query:

  1. Register your bucket as an external location. See external locations.

  2. Open the SQL editor and attach a serverless SQL warehouse.

  3. Run the following query:

    SQL
    CREATE TABLE {catalog}.{schema}.test_connectivity LOCATION 'gs://{your-gcs-bucket}/test_connectivity'

It can take a few minutes for the connection to fully establish.

note

If your network policy restricts external access, direct access to your bucket from a notebook or REPL can fail. Register the bucket as an external location to test connectivity, as shown in the preceding steps. See Manage network policies for serverless egress control.

Connection states​

After you create a private endpoint rule, you can view its connection status in the Private endpoint rules section of the account console:

  • CREATING: The private endpoint is being created.
  • PENDING: Pending approval on the resource.
  • ESTABLISHED: Established on the resource.
  • REJECTED: Rejected on the resource.
  • DISCONNECTED: Disconnected on the resource.
  • EXPIRED: The rule expired after being in the REJECTED, DISCONNECTED, or PENDING state for 14 days.

Most changes to private endpoint rules propagate to serverless compute within 10 minutes, but can take up to 24 hours to fully apply.

Troubleshooting and feedback​

For troubleshooting, contact your account team. Databricks collects feedback on this preview through your account team, so track any problems you encounter to help improve future iterations of the feature.

Next steps​

On this page