Skip to main content
Last updated on

Gmail connector limitations

This page lists limitations and considerations for ingesting data from Gmail using Databricks Lakeflow Connect.

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

General SaaS connector limitations​

The limitations in this section apply to all SaaS connectors in Lakeflow Connect.

  • When you run a scheduled pipeline, alerts don't trigger immediately. Instead, they trigger when the next update runs.
  • When a source table is deleted, the destination table is not automatically deleted. You must delete the destination table manually. This behavior is not consistent with Spark Declarative Pipelines on Lakeflow behavior.
  • During source maintenance periods, Databricks might not be able to access your data.
  • If a source table name conflicts with an existing destination table name, the pipeline update fails.
  • Multi-destination pipeline support is API-only.
  • You can optionally rename a table that you ingest. If you rename a table in your pipeline, it becomes an API-only pipeline, and you can no longer edit the pipeline in the UI.
  • If you select a column after a pipeline has already started, the connector does not automatically backfill data for the new column. To ingest historical data, manually run a full refresh on the table.
  • Databricks can't ingest two or more tables with the same name in the same pipeline, even if they come from different source schemas.
  • The source system assumes that the cursor columns are monotonically increasing.
  • The connector ingests raw data without transformations. Use downstream Spark Declarative Pipelines on Lakeflow pipelines for transformations.

Connector-specific​

The limitations in this section are specific to the Gmail connector.

  • The profile, labels, labels_details, drafts, and filters tables are full-refresh only. They are re-ingested in full on every pipeline run and do not sync incrementally.
  • Only the messages and message_labels tables sync incrementally, using the Gmail History API keyed on a historyId cursor.
  • The messages and message_labels tables do not support SCD Type 2 history tracking; configuring SCD Type 2 for these tables causes pipeline validation to fail.
  • If Gmail expires the stored historyId (the History API returns a 404 because the cursor is older than Gmail's retention window), the connector automatically falls back to a full refresh of the affected table.
  • Gmail retains history for a limited window, typically about seven days. Databricks recommends scheduling the pipeline to run at least once every seven days. If the pipeline runs less frequently, the stored historyId can expire and force a full refresh of messages and message_labels.
  • Each connection ingests a single mailbox. To ingest more than one mailbox, create a separate connection and pipeline per mailbox. The mailbox value is stamped as a mailbox column on every row.
  • The messages payload MIME structure is materialized up to 8 levels of nesting. Parts nested more deeply are not expanded into struct columns.
  • The connector is read-only and requires the gmail.readonly scope. It does not modify the source mailbox.
On this page