Gmail connector reference

This page contains reference material for the Gmail connector in Databricks Lakeflow Connect.

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

General connector behavior

• The connector is read-only. It talks only to https://gmail.googleapis.com and uses the https://www.googleapis.com/auth/gmail.readonly scope by default. It never modifies the source mailbox.
• Each connection ingests a single mailbox. The connector stamps the mailbox value as a mailbox column on every row. To ingest more than one mailbox, create a separate connection and pipeline for each mailbox.
• The source schema is default.
• The messages and message_labels tables sync incrementally using the Gmail History API. The profile, labels, labels_details, drafts, and filters tables are full-refresh only.
• Message attachments are contained within the messages table's payload column (payload.parts[].body.attachmentId). There is no separate attachments table.

Supported tables

The connector ingests the following tables from the default source schema.

Table	Primary key	Sync mode
profile	emailAddress	Full refresh
labels	mailbox, id	Full refresh
labels_details	mailbox, id	Full refresh
drafts	id	Full refresh
filters	id	Full refresh
messages	id	Incremental (Gmail History API, historyId)
message_labels	message_id	Incremental (Gmail History API, historyId)

Table	Primary key	Sync mode
profile	emailAddress	Full refresh
labels	mailbox, id	Full refresh
labels_details	mailbox, id	Full refresh
drafts	id	Full refresh
filters	id	Full refresh
messages	id	Incremental (Gmail History API, historyId)
message_labels	message_id	Incremental (Gmail History API, historyId)

Destination schema

The following sections describe the columns in each destination table.

profile

Column	Type
emailAddress	string (primary key)
messagesTotal	long
threadsTotal	long
historyId	string
mailbox	string

Column	Type
emailAddress	string (primary key)
messagesTotal	long
threadsTotal	long
historyId	string
mailbox	string

labels

Column	Type
mailbox	string (primary key)
id	string (primary key)
name	string
messageListVisibility	string
labelListVisibility	string
type	string
messagesTotal	long
messagesUnread	long
threadsTotal	long
threadsUnread	long
color	struct{textColor: string, backgroundColor: string}

Column	Type
mailbox	string (primary key)
id	string (primary key)
name	string
messageListVisibility	string
labelListVisibility	string
type	string
messagesTotal	long
messagesUnread	long
threadsTotal	long
threadsUnread	long
color	struct{textColor: string, backgroundColor: string}

labels_details

The labels_details table has the same columns as labels (mailbox, id, name, type, the visibility fields, the message and thread counts, and color). Each label is enriched with the response from the labels.get API.

drafts

Column	Type
id	string (primary key)
message	struct{id: string, threadId: string}
mailbox	string

Column	Type
id	string (primary key)
message	struct{id: string, threadId: string}
mailbox	string

filters

Column	Type
id	string (primary key)
criteria	struct{from: string, to: string, subject: string, query: string, negatedQuery: string, hasAttachment: boolean, excludeChats: boolean, size: long, sizeComparison: string}
action	struct{addLabelIds: array<string>, removeLabelIds: array<string>, forward: string}
mailbox	string

Column	Type
id	string (primary key)
criteria	struct{from: string, to: string, subject: string, query: string, negatedQuery: string, hasAttachment: boolean, excludeChats: boolean, size: long, sizeComparison: string}
action	struct{addLabelIds: array<string>, removeLabelIds: array<string>, forward: string}
mailbox	string

messages

Column	Type
id	string (primary key)
threadId	string
snippet	string
historyId	string
internalDate	string
payload	struct (see payload structure)
sizeEstimate	long
mailbox	string
_ingestion_timestamp	timestamp
_row_deleted	boolean
_row_truncated	boolean

Column	Type
id	string (primary key)
threadId	string
snippet	string
historyId	string
internalDate	string
payload	struct (see payload structure)
sizeEstimate	long
mailbox	string
_ingestion_timestamp	timestamp
_row_deleted	boolean
_row_truncated	boolean

payload structure

The payload column materializes the message MIME tree up to 8 levels of nesting. Each level has the following structure:

struct{
  partId: string,
  mimeType: string,
  filename: string,
  headers: array<struct{name: string, value: string}>,
  body: struct{attachmentId: string, size: long, data: string},
  parts: array<payload>
}

Attachments are contained within payload.parts[].body.attachmentId. Parts nested more than 8 levels deep aren't expanded into struct columns.

message_labels

Column	Type
message_id	string (primary key)
threadId	string
labelIds	array<string>
mailbox	string
_ingestion_timestamp	timestamp
_row_deleted	boolean
_row_truncated	boolean

Column	Type
message_id	string (primary key)
threadId	string
labelIds	array<string>
mailbox	string
_ingestion_timestamp	timestamp
_row_deleted	boolean
_row_truncated	boolean

Incremental sync

The messages and message_labels tables sync incrementally:

• The first run performs a full bootstrap crawl of the mailbox.
• Subsequent runs call users.history.list, keyed on the historyId cursor taken from the profile resource, to fetch only the changes since the previous run.
• Deletions are emitted as _row_deleted tombstones.
• If Gmail expires the stored historyId (the History API returns a 404 because the cursor is older than Gmail's retention window), the connector automatically falls back to a full refresh of the affected table.

important

Gmail retains history for a limited window, typically about seven days. Schedule the pipeline to run at least once every seven days so the stored historyId stays within that window. If the cursor expires, the next run performs a full refresh of messages and message_labels.

The messages and message_labels tables do not support SCD Type 2 history tracking; configuring SCD Type 2 for these tables causes pipeline validation to fail.

Rate limiting

When Gmail returns an HTTP 403 response, the connector reads the Retry-After header (with a minimum backoff of 1 second) and retries the request automatically.