Enable Google Cloud Lakehouse catalog federation

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

Google Cloud Lakehouse catalog federation lets Unity Catalog read Google Cloud Lakehouse Iceberg tables directly from cloud storage, which can provide better performance and lower cost than query federation.

With catalog federation, Unity Catalog directly accesses the Google Cloud Lakehouse Iceberg table in Google Cloud Storage (GCS), and the query runs entirely on Databricks compute. Foreign tables in a Google Cloud Lakehouse catalog are read-only.

Google Cloud Lakehouse federation uses a dedicated GOOGLE_CLOUD_LAKEHOUSE connection type. This is a separate connection type from BigQuery query federation.

When you create a foreign catalog, you specify the Google Cloud Lakehouse warehouse path. Unity Catalog discovers the namespaces and tables under that warehouse and maps them into the standard Unity Catalog three-level namespace:

Google Cloud Lakehouse object	Unity Catalog name
Warehouse	<foreign-catalog>
Namespace	<foreign-catalog>.<schema>
Table	<foreign-catalog>.<schema>.<table>

Google Cloud Lakehouse object	Unity Catalog name
Warehouse	<foreign-catalog>
Namespace	<foreign-catalog>.<schema>
Table	<foreign-catalog>.<schema>.<table>

For example, if you federate a warehouse with the namespace analytics containing the table orders, the table appears in Unity Catalog as <foreign-catalog>.analytics.orders.

Before you begin

Review the following requirements before you set up Google Cloud Lakehouse catalog federation.

Workspace requirements:

• Google Cloud Lakehouse federation is supported only on Databricks workspaces on Google Cloud.
• The workspace must be enabled for Unity Catalog. See Get started with Unity Catalog.
• Because this feature is in Beta, a workspace admin must enable it from the Previews page. See Manage Databricks previews.

Compute requirements:

• Databricks compute must use Databricks Runtime 19 or above.

  Databricks Runtime 19 is a unified runtime that receives features as dated updates rather than as new minor versions. See Databricks Runtime 19.

• SQL warehouses must be pro or serverless.

• Dedicated clusters (formerly single-user clusters) are not supported. See Limitations.

Networking requirements:

• If you enforce Virtual Private Cloud (VPC) Service Controls or firewall rules in Google Cloud, you must allowlist the Databricks egress IP ranges for both the control plane and the data plane. See Networking recommendations for Lakehouse Federation.

Permissions required:

• To create a connection, you must be a metastore admin or a user with the CREATE CONNECTION privilege on the Unity Catalog metastore attached to the workspace.
• To create a foreign catalog, you must have the CREATE CATALOG permission on the metastore. You must also own the connection or have the CREATE FOREIGN CATALOG privilege on it.
• To enter authorized paths for the foreign catalog, you must have the CREATE FOREIGN SECURABLE privilege on an external location that covers those paths. The owner of the external location has this privilege by default.

Each task-based section that follows specifies additional permission requirements.

Step 1: Set up Google Cloud resources

Complete the following steps in Google Cloud. No Databricks action is required in this step.

1. Create a Google Cloud Lakehouse warehouse.

   Create the warehouse and select or create a GCS bucket to back it. For example, a warehouse backed by the bucket my-warehouse has the warehouse path gs://my-warehouse.

2. Create Iceberg tables in the warehouse.

   Use any Iceberg-compatible engine, such as Spark, Flink, or BigQuery SQL. Verify that the tables are readable from BigQuery. For example:

   SQL

   SELECT * FROM `PROJECT_ID.NAMESPACE.TABLE`;

3. Create two least-privilege service accounts.

   Google Cloud Lakehouse identity and access management (IAM) roles use the biglake namespace, so the roles in the following table appear under BigLake in the Google Cloud console.

   Service account	Purpose	Required IAM roles
   Connection service account	Reads catalog metadata.	roles/biglake.viewer on the project.
   Storage service account	Reads table data in GCS.	roles/storage.objectViewer on the GCS bucket. Add roles/bigquery.dataViewer if the tables are BigQuery-managed.

   Service account	Purpose	Required IAM roles
   Connection service account	Reads catalog metadata.	roles/biglake.viewer on the project.
   Storage service account	Reads table data in GCS.	roles/storage.objectViewer on the GCS bucket. Add roles/bigquery.dataViewer if the tables are BigQuery-managed.

4. Generate a JSON key for the connection service account.

   In the Google Cloud console, generate and download a JSON key for the connection service account. You provide this key when you create the Unity Catalog connection in Step 2: Create a connection. Store the key securely.

   For instructions on creating service accounts, assigning IAM roles, and generating keys, see the Google Cloud IAM documentation.

Step 2: Create a connection

A connection specifies a path and credentials for accessing an external system. Create a connection by using Catalog Explorer or the CREATE CONNECTION SQL command in a Databricks notebook or the Databricks SQL query editor.

note

You can also use the Databricks REST API or the Databricks CLI to create a connection. See POST /api/2.1/unity-catalog/connections and Unity Catalog commands.

Permissions required: Metastore admin or user with the CREATE CONNECTION privilege.

Catalog Explorer

1. In your Databricks workspace, click Catalog.
2. At the top of the Catalog pane, click the Add icon and select Create a connection from the menu.
3. On the Connection basics page of the Set up connection wizard, enter a user-friendly Connection name.
4. Select a Connection type of Google Cloud Lakehouse, and then click Next.
5. On the Authentication page, enter the Google Cloud project ID and the Service account key JSON for the connection service account that you created in Step 1: Set up Google Cloud resources.
6. (Optional) Add a comment.
7. Click Create connection.

SQL

Run the following command in a notebook or the Databricks SQL query editor. Replace the placeholder values:

• <connection-name>: Name for the connection in Databricks.
• <gcp-project-id>: The Google Cloud project ID that contains the Google Cloud Lakehouse catalog.
• <secret-scope> and <secret-key>: The secret scope and key that store the connection service account's JSON key.

SQL

CREATE CONNECTION <connection-name> TYPE GOOGLE_CLOUD_LAKEHOUSE
OPTIONS (
  gcp_project_id '<gcp-project-id>',
  service_account_json secret('<secret-scope>','<secret-key>')
);

Databricks recommends that you use secrets instead of plaintext strings for sensitive values like the service account key. For information about setting up secrets, see Secret management.

Step 3: Create a storage credential and external location

Configure a storage credential and external location in Unity Catalog to govern access to the GCS buckets that hold your Google Cloud Lakehouse tables. External locations are Unity Catalog securable objects that associate storage credentials with cloud storage container paths.

You must create at least the following:

• One credential and external location covering the catalog metadata storage root. This location stores metadata for the Iceberg tables in the catalog, so the Databricks-managed service account must have read-write access. This can be any GCS bucket. It does not need to be related to the table buckets. You specify this location as the storage_root option when you create the foreign catalog.
• One or more credentials and external locations covering the table storage locations. These can be read-only.

To create a storage credential and external location, use Catalog Explorer or SQL. For example, to create an external location using SQL:

SQL

CREATE EXTERNAL LOCATION <name>
URL 'gs://<bucket-path>'
WITH (STORAGE CREDENTIAL <gcp-storage-credential>);

For detailed instructions, see Connect to a Google Cloud Storage (GCS) external location.

Step 4: Create a foreign catalog

A foreign catalog mirrors your Google Cloud Lakehouse catalog so that you can query and manage access to its tables using Databricks and Unity Catalog. To create a foreign catalog, use the connection that you created in Step 2: Create a connection.

Create a foreign catalog by using Catalog Explorer or the CREATE FOREIGN CATALOG SQL command in a Databricks notebook or the Databricks SQL query editor.

Permissions required: CREATE CATALOG permission on the metastore and either ownership of the connection or the CREATE FOREIGN CATALOG privilege on the connection.

Catalog Explorer

1. In your Databricks workspace, click Catalog to open Catalog Explorer.

2. At the top of the Catalog pane, click the Add icon and select Add a catalog from the menu.

3. Enter a Catalog name and select a catalog Type of Foreign.

4. Select the Connection that you created in Step 2: Create a connection from the drop-down menu.

5. For Warehouse, enter the Google Cloud Lakehouse warehouse path to federate. Use a gs:// path for a Cloud Storage warehouse or a bq:// path for a BigQuery warehouse.

6. For Authorized paths, enter the cloud storage paths that can be accessed through the catalog. Only tables under these paths can be queried through the foreign catalog. Paths must be covered by external locations. See What are authorized paths?.

   You can edit authorized paths after creating the catalog.

7. In the Storage location field, specify the writable GCS location that stores metadata for the Iceberg tables in this catalog.

8. Click Create catalog.

9. Assign workspace access, an owner, and privileges as prompted.

SQL

Run the following command in a notebook or the Databricks SQL query editor. Items in brackets are optional. Replace the placeholder values:

• <catalog-name>: Name for the catalog in Databricks.
• <connection-name>: The name of the connection that you created in Step 2: Create a connection.
• <warehouse-path>: The Google Cloud Lakehouse warehouse path. Use a gs:// path for a Cloud Storage warehouse or a bq:// path for a BigQuery warehouse.
• <path1>,<path2>: Cloud storage paths that can be accessed through the catalog. Paths must be covered by external locations. See What are authorized paths?.
• <storage-location>: A writable GCS location where catalog metadata is stored.

SQL

CREATE FOREIGN CATALOG [IF NOT EXISTS] <catalog-name>
USING CONNECTION <connection-name>
OPTIONS (
  warehouse '<warehouse-path>',
  authorized_paths '<path1>,<path2>',
  storage_root '<storage-location>'
);

Unity Catalog discovers all namespaces and tables under the catalog:

• Google Cloud Lakehouse namespaces become Unity Catalog schemas.
• Google Cloud Lakehouse Iceberg tables become Unity Catalog foreign tables.

Grant permissions and query the catalog

After you set up catalog federation, users must have the appropriate Unity Catalog permissions to access federated tables:

• All users need USE CATALOG and USE SCHEMA permissions on the catalog and schema respectively.
• To read from a federated table, users need the SELECT permission.

For more information about Unity Catalog privileges and how to grant them, see Manage privileges in Unity Catalog.

After you grant permissions, users can query the foreign tables from Databricks:

SQL

SELECT * FROM <catalog-name>.<schema>.<table>;

Attribute-based and fine-grained access control

Foreign tables support attribute-based access control (ABAC) and fine-grained access control (FGAC). Add tags using Catalog Explorer or the ALTER TABLE ... SET TAGS command. See Apply tags to Unity Catalog securable objects.

Limitations

Google Cloud Lakehouse catalog federation has the following limitations:

• Schema and table names follow standard Unity Catalog naming limitations. Databricks does not support names that contain a period (.), space ( ), or forward slash (/).
• Foreign tables in a Google Cloud Lakehouse catalog are read-only.
• Tables that contain a column with the TIME type are not supported.
• GCS buckets must be single-region. Multi-region and dual-region buckets are not supported.
• Dedicated clusters (formerly single-user clusters) are not supported.

See Limitations for Iceberg-related limitations.

Troubleshooting

The following section describes common errors and their resolutions.

Catalog not found or Failed to load catalog

Common causes:

• The warehouse option doesn't match a Google Cloud Lakehouse warehouse path in the Google Cloud project.
• The connection service account is missing the roles/biglake.viewer role on the project.

403 Forbidden when reading table data

Common causes:

• The storage service account is missing the roles/storage.objectViewer role on the GCS bucket.
• If the tables are BigQuery-managed, the storage service account is also missing the roles/bigquery.dataViewer role.

DBR version not supported

The compute is on a Databricks Runtime version below the supported minimum. Use Databricks Runtime 19 or above. See Before you begin.

Schemas appear but tables do not

Verify that an external location covers the storage path where the tables reside, and that the authorized_paths option covers all bucket paths that the tables actually use. See Step 3: Create a storage credential and external location.

Authentication failed when creating the connection

The service account key JSON is not valid or expired, or the service account is missing the roles/biglake.viewer role. Regenerate the key in the Google Cloud console and confirm the service account's roles. See Step 1: Set up Google Cloud resources.