Skip to main content
Last updated on

Query on-premises databases using Lakehouse Federation

Lakehouse Federation lets you run federated queries against external relational databases, including on-premises databases, using JDBC. It provides governed, read-only access to external data through Unity Catalog foreign catalogs, with automatic query pushdown and fine-grained access controls at the table level. Queries run both in Databricks and on the remote database.

Databricks compute querying on-premises databases over JDBC through Unity Catalog foreign catalogs.

What problem it solves​

Lakehouse Federation lets you query data in on-premises databases without complex and time-consuming extract, transform, and load (ETL) processing. It gives you access to on-premises databases when:

  • You can't ingest the data into Databricks.
  • You want to use existing on-premises compute capacity.
  • You want the benefits of Unity Catalog, including fine-grained access control.

When to use Lakehouse Federation​

Use Lakehouse Federation for the following:

  • Ad hoc reporting.
  • Proof-of-concept work.
  • Exploration of new ETL pipelines.
  • Support for workloads during an incremental migration.

Use Lakehouse Federation when you want to minimize data movement. If performance on high data volumes and low latency are priorities, use Lakeflow Connect instead.

Network requirements​

To use query federation with on-premises databases, establish a network path between your Databricks compute and your on-premises environment:

  • In a hybrid cloud setup, use VPC peering for classic compute, or a Private Link service for serverless compute, to reach your on-premises network.

For more information, see Networking recommendations for Lakehouse Federation.

Considerations​

Review the following before you use Lakehouse Federation with on-premises databases:

  • Query federation supports MySQL, PostgreSQL, Teradata, Oracle, Amazon Redshift, Salesforce Data 360, Snowflake, Microsoft SQL Server, Azure Synapse (SQL Data Warehouse), Google BigQuery, and Databricks. For the current list, see Supported data sources.
  • Classic compute must use Databricks Runtime 13.3 LTS and above in standard or dedicated access mode.
  • If a connection authenticates with OAuth, the Databricks control plane must be able to reach the OAuth endpoint of your on-premises system.

Additional resources​

On this page