File connectors
Databricks Lakeflow Connect has connectors for ingesting files from cloud object storage and enterprise file storage services. Choose managed connectors for a configuration-based ingestion service, or standard connectors based on Spark and SQL APIs for a wider range of file sources and finer control.
Managed file connectors
Managed file connectors handle authentication for each source, incremental reads, and automated retries for you:
-
- Google Drive
- Ingest files from Google Drive into the Lakehouse using managed file source ingestion.
-
- Microsoft SharePoint
- Ingest files from Microsoft SharePoint sites using managed file source ingestion.
Standard file connectors
Standard file connectors use Spark and SQL APIs, giving you more control over how files are read and processed:
-
- Cloud object storage
- Ingest files from S3, ADLS, or GCS using Auto Loader or
COPY INTO.
-
- SFTP servers
- Ingest files from SFTP servers.
-
- Google Drive
- Ingest files from Google Drive using Spark reader APIs.
-
- SharePoint
- Ingest files from SharePoint using Spark reader APIs.
Connector components
A file source connector has the following components:
Component | Description |
|---|---|
Connection | A Unity Catalog securable object that stores authentication details for the file storage service. |
Ingestion pipeline | A pipeline that copies files from the source into the destination tables. The ingestion pipeline runs on serverless compute. |
Destination tables | The tables where the ingestion pipeline writes the data. These are streaming tables, which are Delta tables with extra support for incremental data processing. |