Configure a firewall for serverless compute access

Preview

This feature is in Public Preview. To join this preview, contact your Databricks account team.

This article describes how to configure a firewall for serverless compute using the Databricks account console UI. You can also use the Network Connectivity Configurations API. Firewall enablement is not supported for Amazon S3 or Amazon DynamoDB.

Note

Starting October 7, 2024, Databricks will charge customers for networking costs incurred from serverless compute resources connecting to external resources. Over the next few months, serverless network billing will roll out in phases, which might result in gradual billing changes. For more information on billing, see Databricks pricing.

Overview of firewall enablement for serverless compute

Serverless network connectivity is managed with network connectivity configurations (NCCs). Account admins create NCCs in the account console and an NCC can be attached to one or more workspaces

An NCC contains a list of IPs. When an NCC is attached to a workspace, serverless compute in that workspace uses one of those IP addresses to connect your resources. You can allow list those networks on your resource firewalls.

NCC firewall enablement is supported from serverless SQL warehouses, jobs, notebooks, Delta Live Tables pipelines, and model serving endpoints.

For more information on NCCs, see What is a network connectivity configuration (NCC)?.

Requirements

  • Your workspace must be on the Premium plan or above.

  • You must be a Databricks account admin.

  • Each NCC can be attached to up to 50 workspaces.

  • Each Databricks account can have up to 10 NCCs per supported region. For the list of supported regions, see Databricks clouds and regions.

  • Your target resource must be publicly accessible.

Step 1: Create a network connectivity configuration and copy the stable IPs

Databricks recommends sharing NCCs among workspaces in the same business unit and those sharing the same region.

  1. As an account admin, go to the account console.

  2. In the sidebar, click Cloud Resources.

  3. Click Network.

  4. Click Network Connectivity Configuration.

  5. Click Add Network Connectivity Configuration.

  6. Type a name for the NCC.

  7. Choose the region. This must match your workspace region.

  8. Click Add.

  9. Click the Default Rules tab.

  10. Under Stable IPs, click Copy all IPs and save the list of IPs.

Step 2: Attach an NCC to workspaces

You can attach an NCC to up to 50 workspaces in the same region as the NCC.

To use the API to attach an NCC to a workspace, see the Account Workspaces API.

  1. In the account console sidebar, click Workspaces.

  2. Click your workspace’s name.

  3. Click Update workspace.

  4. In the Network Connectivity Configuration field, select your NCC. If it’s not visible, confirm that you’ve selected the same region for both the workspace and the NCC.

  5. Click Update.

  6. Wait 10 minutes for the change to take effect.

  7. Restart any running serverless compute resources in the workspace.

Step 3: Update your resource access rules to allowlist the IPs

Add the stable IPs to your resource access rules. For more information, see AWS global condition context keys in the AWS documentation.

Creating a storage firewall also affects connectivity from classic compute plane resources to resources. You must also update your resource access rules to allowlist the IPs to connect to them from classic compute resources.

NCC firewall enablement is not supported for Amazon S3 or Amazon DynamoDB. When reading or writing to Amazon S3 buckets in the same region as your workspace, serverless compute resources use direct access to S3 using AWS gateway endpoints. This applies when serverless SQL compute reads and writes to your workspace storage bucket in your AWS account and to other S3 data sources in the same region.