Skip to main content
Last updated on

Microsoft SharePoint connector limitations

This page lists limitations and considerations for ingestion from Microsoft SharePoint using Databricks Lakeflow Connect.

Connector-specific limitations​

  • Unstructured (BINARYFILE) ingestion supports only SCD_TYPE_1 storage mode. Structured ingestion (CSV, JSON, XML, EXCEL, and other formats) supports only APPEND_ONLY storage mode. SCD type 2 is not supported. When configuring storage mode, set storage_mode in table_configuration. Setting the scd_type field throws an error.

  • During unstructured ingestion using binary file, each file's content is loaded into memory as a single record, so files larger than 100 MB can cause the update to fail (for example, with an out-of-memory error or by exceeding the 2 GB limit on binary columns in Delta). To prevent this, exclude large files using a row_filter on the length column in table_configuration. For example, "row_filter": "length <= 104857600" skips files larger than 100 MB.

  • The connector does not support individual file selection. It ingests all files in a configured folder, drive, subsite, or site.

  • The connector does not support ingesting file-level access control lists (ACLs) from SharePoint.

  • The connector does not support ingesting files linked to a different SharePoint document library.

  • The utils provided for downstream usage are limited to single-user clusters. However, single-user clusters cannot access streaming tables created by other users. Each downstream user must create their own ingestion pipeline. You can modify the utils to make them work on serverless and shared clusters, but this can affect performance. See File access examples.

  • Some fields (for example, quick_xor_hash, mime_type) are not supported for all file formats. File download and other metadata ingestion continue to work in these cases.

On this page