Ingest data from Gmail

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

Create a managed Gmail ingestion pipeline to load a mailbox's messages, labels, drafts, filters, and profile into Unity Catalog tables. You can author the pipeline in the data ingestion UI, with Declarative Automation Bundles, or through the Pipelines API. Each pipeline ingests the single mailbox set on the connection.

Requirements

• To create an ingestion pipeline, you must first meet the following requirements:

  • Your workspace must be enabled for Unity Catalog.

  • Serverless compute must be enabled for your workspace. See Serverless compute requirements.

  • To create a new connection, you must have CREATE CONNECTION privileges on the metastore. See Manage privileges in Unity Catalog.

    If the connector supports UI-based pipeline authoring, an admin can create the connection and the pipeline at the same time by completing the steps on this page. However, if the users who create pipelines use API-based pipeline authoring or are non-admin users, an admin must first create the connection in Catalog Explorer. See Connect to managed ingestion sources.

  • To use an existing connection, you must have USE CONNECTION privileges or ALL PRIVILEGES on the connection object.

  • You must have USE CATALOG privileges on the target catalog.

  • You must have USE SCHEMA and CREATE TABLE privileges on an existing schema or CREATE SCHEMA privileges on the target catalog.

• To ingest from Gmail, you must first complete the steps in Create a Gmail connection.

Create an ingestion pipeline

Each ingested table is written to a streaming table. The source schema is default. For the list of tables that you can ingest, see Supported tables.

Databricks UI

1. In the sidebar of the Databricks workspace, click Data Ingestion.
2. On the Add data page, under Databricks connectors, click Gmail.
3. On the Connection page of the ingestion wizard, select the connection that stores your Gmail access credentials. If you have the CREATE CONNECTION privilege on the metastore, you can click Create connection to create a new connection with the authentication details in Create a Gmail connection.
4. Click Next.
5. On the Ingestion setup page, enter a unique name for the pipeline.
6. Select a catalog and a schema to write event logs to. If you have USE CATALOG and CREATE SCHEMA privileges on the catalog, you can click Create schema in the drop-down menu to create a new schema.
7. Click Create pipeline and continue.
8. On the Source page, select the tables to ingest.
9. Click Save and continue.
10. On the Destination page, select a catalog and a schema to load data into. If you have USE CATALOG and CREATE SCHEMA privileges on the catalog, you can click Create schema in the drop-down menu to create a new schema.
11. Click Save and continue.
12. (Optional) On the Schedules and notifications page, click Create schedule. Set the frequency to refresh the destination tables.
13. (Optional) Click Add notification to set email notifications for pipeline operation success or failure, then click Save and run pipeline.

Declarative Automation Bundles

Use Declarative Automation Bundles to manage Gmail pipelines as code. Bundles can contain YAML definitions of jobs and tasks, are managed using the Databricks CLI, and can be shared and run in different target workspaces (such as development, staging, and production). For more information, see What are Declarative Automation Bundles?.

1. Create a bundle using the Databricks CLI:

   Bash

   databricks bundle init

2. Add two new resource files to the bundle:

   • A pipeline definition file (for example, resources/gmail_pipeline.yml). See pipeline.ingestion_definition and Examples.
   • A job definition file that controls the frequency of data ingestion (for example, resources/gmail_job.yml).

3. Deploy the pipeline using the Databricks CLI:

   Bash

   databricks bundle deploy

Databricks notebook

1. Modify the pipeline specification with your pipeline configuration details. See pipeline.ingestion_definition and Examples.
2. Run the notebook.

Mailbox selection

The mailbox to read from is set on the connection, not the pipeline. Configure it with the Mailbox Email (impersonate_email) field when you create the connection. See Create a Gmail connection. The service account impersonates this user through domain-wide delegation, and the connector reads that user's mailbox. If you don't set it, the connection authenticates as the service account itself, which has no mailbox to ingest. The connector stamps the mailbox value as a mailbox column on every row.

Each connection ingests a single mailbox. To ingest more than one mailbox, create a separate connection and pipeline for each mailbox.

Schedule the pipeline to run at least weekly

Databricks recommends scheduling the pipeline to run at least once every seven days. The messages and message_labels tables sync incrementally using Gmail's History API, and Gmail retains history for a limited window (typically about seven days).

warning

If the pipeline runs less frequently than Gmail's history window, the stored historyId cursor can expire. When it does, the next run performs a full refresh of messages and message_labels.

Examples

Use these examples to configure your pipeline.

Ingest a single source table

Declarative Automation Bundles

The following pipeline definition file ingests a single source table. The pipeline_gmail resource is the main pipeline, and objects defines an array of tables to ingest. This example ingests the messages table.

YAML

variables:
  dest_catalog:
    default: main
  dest_schema:
    default: ingest_destination_schema

# The main pipeline for gmail_dab
resources:
  pipelines:
    pipeline_gmail:
      name: gmail_pipeline
      catalog: ${var.dest_catalog}
      schema: ${var.dest_schema}
      ingestion_definition:
        connection_name: <gmail-connection>
        objects:
          # An array of objects to ingest from Gmail. This example ingests the messages table.
          - table:
              source_schema: default
              source_table: messages
              destination_catalog: ${var.dest_catalog}
              destination_schema: ${var.dest_schema}

Databricks notebook

The following pipeline specification ingests a single source table:

Python

pipeline_spec = """
{
  "name": "<pipeline-name>",
  "ingestion_definition": {
    "connection_name": "<gmail-connection>",
    "objects": [
      {
        "table": {
          "source_schema": "default",
          "source_table": "messages",
          "destination_catalog": "main",
          "destination_schema": "ingest_destination_schema"
        }
      }
    ]
  }
}
"""
create_pipeline(pipeline_spec)

Ingest multiple source tables

Declarative Automation Bundles

The following pipeline definition file ingests multiple source tables:

YAML

variables:
  dest_catalog:
    default: main
  dest_schema:
    default: ingest_destination_schema

# The main pipeline for gmail_dab
resources:
  pipelines:
    pipeline_gmail:
      name: gmail_pipeline
      catalog: ${var.dest_catalog}
      schema: ${var.dest_schema}
      ingestion_definition:
        connection_name: <gmail-connection>
        objects:
          # An array of objects to ingest from Gmail. This example ingests the messages and message_labels tables.
          - table:
              source_schema: default
              source_table: messages
              destination_catalog: ${var.dest_catalog}
              destination_schema: ${var.dest_schema}
          - table:
              source_schema: default
              source_table: message_labels
              destination_catalog: ${var.dest_catalog}
              destination_schema: ${var.dest_schema}

Databricks notebook

The following pipeline specification ingests multiple source tables:

Python

pipeline_spec = """
{
  "name": "<pipeline-name>",
  "ingestion_definition": {
    "connection_name": "<gmail-connection>",
    "objects": [
      {
        "table": {
          "source_schema": "default",
          "source_table": "messages",
          "destination_catalog": "main",
          "destination_schema": "ingest_destination_schema"
        }
      },
      {
        "table": {
          "source_schema": "default",
          "source_table": "message_labels",
          "destination_catalog": "main",
          "destination_schema": "ingest_destination_schema"
        }
      }
    ]
  }
}
"""
create_pipeline(pipeline_spec)

Declarative Automation Bundles job definition file

The following is an example job definition file to use with Declarative Automation Bundles. The job runs every day, exactly one day from the last run.

YAML

resources:
  jobs:
    gmail_dab_job:
      name: gmail_dab_job

      trigger:
        periodic:
          interval: 1
          unit: DAYS

      email_notifications:
        on_failure:
          - <email-address>

      tasks:
        - task_key: refresh_pipeline
          pipeline_task:
            pipeline_id: ${resources.pipelines.pipeline_gmail.id}

Next steps

Start, schedule, and set alerts on your pipeline. See Common pipeline maintenance tasks.

Additional resources

• Gmail connector FAQs
• Gmail connector limitations
• Gmail connector reference
• Troubleshoot Gmail ingestion