Skip to main content

Run federated queries on Google BigQuery

This article describes how to set up Lakehouse Federation to run federated queries on BigQuery data that is not managed by Databricks. To learn more about Lakehouse Federation, see What is Lakehouse Federation?.

To connect to your BigQuery database using Lakehouse Federation, you must create the following in your Databricks Unity Catalog metastore:

  • A connection to your BigQuery database.
  • A foreign catalog that mirrors your BigQuery database in Unity Catalog so that you can use Unity Catalog query syntax and data governance tools to manage Databricks user access to the database.

Before you begin​

Workspace requirements:

  • Workspace enabled for Unity Catalog.

Compute requirements:

  • Network connectivity from your Databricks Runtime cluster or SQL warehouse to the target database systems. See Networking recommendations for Lakehouse Federation.
  • Databricks clusters must use Databricks Runtime 16.1 or above and standard or dedicated access mode (formerly shared and single user).
  • SQL warehouses must be Pro or Serverless.

Permissions required:

  • To create a connection, you must be a metastore admin or a user with the CREATE CONNECTION privilege on the Unity Catalog metastore attached to the workspace.
  • To create a foreign catalog, you must have the CREATE CATALOG permission on the metastore and be either the owner of the connection or have the CREATE FOREIGN CATALOG privilege on the connection.

Additional permission requirements are specified in each task-based section that follows.

Create a connection​

A connection specifies a path and credentials for accessing an external database system. To create a connection, you can use Catalog Explorer or the CREATE CONNECTION SQL command in a Databricks notebook or the Databricks SQL query editor.

note

You can also use the Databricks REST API or the Databricks CLI to create a connection. See POST /api/2.1/unity-catalog/connections and Unity Catalog commands.

Permissions required: Metastore admin or user with the CREATE CONNECTION privilege.

  1. In your Databricks workspace, click Catalog icon Catalog.

  2. At the top of the Catalog pane, click the Add or plus icon Add icon and select Add a connection from the menu.

    Alternatively, from the Quick access page, click the External data > button, go to the Connections tab, and click Create connection.

  3. On the Connection basics page of the Set up connection wizard, enter a user-friendly Connection name.

  4. Select a Connection type of Google BigQuery, and then click Next.

  5. On the Authentication page, enter the Google service account key json for your BigQuery instance.

    This is a raw JSON object that is used to specify the BigQuery project and provide authentication. You can generate this JSON object and download it from the service account details page in Google Cloud under 'KEYS'. The service account must have proper permissions granted in BigQuery, including BigQuery User and BigQuery Data Viewer. The following is an example.

    JSON
    {
      "type": "service_account",
      "project_id": "PROJECT_ID",
      "private_key_id": "KEY_ID",
      "private_key": "PRIVATE_KEY",
      "client_email": "SERVICE_ACCOUNT_EMAIL",
      "client_id": "CLIENT_ID",
      "auth_uri": "https://accounts.google.com/o/oauth2/auth",
      "token_uri": "https://accounts.google.com/o/oauth2/token",
      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/SERVICE_ACCOUNT_EMAIL",
      "universe_domain": "googleapis.com"
    }

  6. (Optional) Enter the Project ID for your BigQuery instance:

    This is a name for the BigQuery project used for billing for all queries run under this connection. Defaults to the project ID of your service account. The service account must have proper permissions granted for this project in BigQuery, including BigQuery User. Additional dataset used for storing temporary tables by BigQuery might be created in this project.

  7. (Optional) Add a comment.

  8. Click Create connection.

  9. On the Catalog basics page, enter a name for the foreign catalog. A foreign catalog mirrors a database in an external data system so that you can query and manage access to data in that database using Databricks and Unity Catalog.

  10. (Optional) Click Test connection to confirm that it works.

  11. Click Create catalog.

  12. On the Access page, select the workspaces in which users can access the catalog you created. You can select All workspaces have access, or click Assign to workspaces, select the workspaces, and then click Assign.

  13. Change the Owner who will be able to manage access to all objects in the catalog. Start typing a principal in the text box, and then click the principal in the returned results.

  14. Grant Privileges on the catalog. Click Grant:

    1. Specify the Principals who will have access to objects in the catalog. Start typing a principal in the text box, and then click the principal in the returned results.
    2. Select the Privilege presets to grant to each principal. All account users are granted BROWSE by default.
      • Select Data Reader from the drop-down menu to grant read privileges on objects in the catalog.
      • Select Data Editor from the drop-down menu to grant read and modify privileges on objects in the catalog.
      • Manually select the privileges to grant.
    3. Click Grant.

  15. Click Next.

  16. On the Metadata page, specify tags key-value pairs. For more information, see Apply tags to Unity Catalog securable objects.

  17. (Optional) Add a comment.

  18. Click Save.

Create a foreign catalog​

note

If you use the UI to create a connection to the data source, foreign catalog creation is included and you can skip this step.

A foreign catalog mirrors a database in an external data system so that you can query and manage access to data in that database using Databricks and Unity Catalog. To create a foreign catalog, use a connection to the data source that has already been defined.

To create a foreign catalog, you can use Catalog Explorer or CREATE FOREIGN CATALOG in a Databricks notebook or the Databricks SQL query editor. You can also use the Databricks REST API or the Databricks CLI to create a catalog. See POST /api/2.1/unity-catalog/catalogs or Unity Catalog commands.

Permissions required: CREATE CATALOG permission on the metastore and either ownership of the connection or the CREATE FOREIGN CATALOG privilege on the connection.

  1. In your Databricks workspace, click Catalog icon Catalog to open Catalog Explorer.

  2. At the top of the Catalog pane, click the Add or plus icon Add icon and select Add a catalog from the menu.

    Alternatively, from the Quick access page, click the Catalogs button, and then click the Create catalog button.

  3. (Optional) Enter the following catalog property:

    Data Project Id: A name for the BigQuery project containing data that will be mapped to this catalog. Defaults to the billing project ID set at the connection level.

  4. Follow the instructions for creating foreign catalogs in Create catalogs.

Supported pushdowns​

The following pushdowns are supported:

  • Filters
  • Projections
  • Limit
  • Functions: partial, only for filter expressions. (String functions, Mathematical functions, Data, Time and Timestamp functions, and other miscellaneous functions, such as Alias, Cast, SortOrder)
  • Aggregates
  • Sorting, when used with limit
  • Joins (Databricks Runtime 16.1 or above)

The following pushdowns are not supported:

  • Windows functions

Data type mappings​

The following table shows the BigQuery to Spark data type mapping.

BigQuery type

Spark type

bignumeric, numeric

DecimalType

int64

LongType

float64

DoubleType

array, geography, interval, json, string, struct

VarcharType

bytes

BinaryType

bool

BooleanType

date

DateType

datetime, time, timestamp

TimestampType/TimestampNTZType

When you read from BigQuery, BigQuery Timestamp is mapped to Spark TimestampType if preferTimestampNTZ = false (default). BigQuery Timestamp is mapped to TimestampNTZType if preferTimestampNTZ = true.

Last updated on