Skip to main content
Last updated on

Networking recommendations for Lakehouse Federation

Lakehouse Federation requires a network path between your Databricks clusters or SQL warehouses and the external database system that you connect to. This page explains how to set up that path when either side has network access restrictions or a custom DNS server.

Consider the following when you set up network connectivity for Lakehouse Federation:

  • All network traffic for federated queries is directly between Databricks clusters (or SQL warehouses) and the external database system. Neither Unity Catalog nor the Databricks control plane is on the network path.
  • Connections that use OAuth have an additional requirement. When a connection authenticates with OAuth, the OAuth token exchange happens from the Databricks control plane, not the compute plane. The control plane must be able to reach the external system's OAuth endpoint. Allowlisting only the compute plane is not sufficient for these connections. For connector-specific requirements, see the OAuth endpoint requirements in Snowflake query federation limitations and Create a SQL Server connection.
  • Databricks compute (that is, clusters and SQL warehouses) always deploys in the cloud, but the external database system can be on-premises or hosted on any cloud provider, as long as there's a viable network path between your Databricks compute and the external database.
  • If you have inbound or outbound network restrictions on either Databricks compute or the external database system, refer to the following sections for general guidance to help you create a viable network path.

For more information on networking in Databricks workspaces, see Networking.

Database system and Databricks compute both accessible from internet​

The connection should work without any configuration.

Database system has network access restrictions​

If the external database system has inbound or outbound network access restrictions and the Databricks cluster or SQL warehouse is accessible from the internet, configure one of the following network solutions to connect from classic compute resources:

  • Stable egress IP on Databricks compute.

    From the classic compute plane, set up a stable IP address with a load balancer, NAT gateway, internet gateway, or equivalent, and connect it to the subnet where Databricks compute is deployed. This allows the compute resource to share a stable public IP address that can be allowlisted on the external database side.

From the serverless compute plane, Databricks publishes its outbound IP, which the external database system can allowlist. You must keep these IPs current by building automation. See Serverless compute firewall configuration.

From the classic compute plane, the external database system should allowlist the Databricks compute stable IP for both ingress and egress traffic.

  • Private Link (only when the external database is on the same cloud as Databricks compute)

    From the classic compute plane, configure a Private Link connection between the network where the database is deployed and the network where Databricks compute is deployed.

Databricks compute has network access restrictions​

If the external database system is accessible from the Internet and the Databricks compute has inbound or outbound network access restrictions (which is only possible if you are on a customer-managed network), then perform one of the following configurations:

  • Allowlist the hostname of the external database in the firewall rules of the subnet where Databricks compute is deployed.

    If you choose to allowlist the external database IP address rather than hostname, make sure that the external database has a stable IP address.

  • Private Link (only when the external database is on same cloud as Databricks compute)

    Configure a Private Link connection between the network where the database is deployed and the network where Databricks compute is deployed.

Databricks compute has a custom DNS server​

If the external database system is accessible from the Internet and the Databricks compute has a custom DNS server (which is only possible if you are on a customer-managed network), add the database system's hostname to your custom DNS server so that it can be resolved.

Connector-specific networking​

Some connectors have networking requirements beyond the general guidance on this page, such as required endpoints, connectivity checks, or private connectivity options. For these details, see the networking section of the relevant connection page:

On this page