Skip to main content
Last updated on

Ingest data from OneDrive

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page by opting into the Lakeflow Connect for OneDrive For Business preview. See Manage Databricks previews.

This page shows how to create a managed Microsoft OneDrive ingestion pipeline using Lakeflow Connect. The pipeline ingests files from OneDrive into Delta tables and keeps them in sync with the source.

Before you begin​

  • To create the ingestion pipeline, you must first meet the following requirements:

    • Your workspace must be enabled for Unity Catalog.

    • Serverless compute must be enabled for your workspace. See Serverless compute requirements.

    • To create a new connection, you must have CREATE CONNECTION privileges on the metastore. See Manage privileges in Unity Catalog.

      If the connector supports UI-based pipeline authoring, an admin can create the connection and the pipeline at the same time by completing the steps on this page. However, if the users who create pipelines use API-based pipeline authoring or are non-admin users, an admin must first create the connection in Catalog Explorer. See Connect to managed ingestion sources.

    • To use an existing connection, you must have USE CONNECTION privileges or ALL PRIVILEGES on the connection object.

    • You must have USE CATALOG privileges on the target catalog.

    • You must have USE SCHEMA and CREATE TABLE privileges on an existing schema or CREATE SCHEMA privileges on the target catalog.

  • Use Databricks Runtime 19 or above. In an API or Declarative Automation Bundles pipeline definition, set channel to PREVIEW.

  • To use Declarative Automation Bundles, use Databricks CLI version 1.23.0 and above. Earlier versions drop onedrive_options from the pipeline definition.

  • To ingest from OneDrive, you must first configure a supported authentication method and create a connection. See Create a OneDrive connection.

Create an ingestion pipeline​

For a list of supported file formats, see file_ingestion_options parameters. For connector-specific limitations, see Microsoft OneDrive connector limitations.

Use Declarative Automation Bundles to manage OneDrive pipelines as code. Bundles can contain YAML definitions of jobs and tasks, are managed using the Databricks CLI, and can be shared and run in different target workspaces (such as development, staging, and production). See What are Declarative Automation Bundles?.

  1. Create a bundle using the Databricks CLI:

    Bash
    databricks bundle init

  2. Add two new resource files to the bundle:

  3. Deploy the pipeline using the Databricks CLI:

    Bash
    databricks bundle deploy

Deploying the bundle creates the pipeline and the job. The job runs the pipeline on the schedule in the job definition.

Choose a source URL​

Each object in a pipeline definition specifies its source in connector_options.onedrive_options.url. The URL must resolve to one of the following resources in a OneDrive for Business My files drive:

  • The drive root
  • A folder
  • A file
  • A valid share link

The URL must use a personal OneDrive host, such as <tenant>-my.sharepoint.com. The following URLs are not supported:

  • SharePoint site URLs (/sites/...)
  • Consumer OneDrive URLs, such as onedrive.live.com or 1drv.ms
  • Bare host URLs
  • Paths outside a personal drive

For URL examples and instructions to copy a URL from OneDrive, see Read files from OneDrive.

Examples​

Use these examples to configure your pipeline. Replace each placeholder with values for your environment.

Ingest files as unstructured data​

Ingest files from OneDrive as binary content. Each file becomes a row that contains the file's bytes and metadata. Use this approach for PDFs, Office documents, images, and other files you intend to process downstream, for example for RAG applications.

note

Available in Beta, the FILE type stores a governed reference to each file and its metadata instead of the file's bytes, which improves read performance. To ingest files as the FILE type instead of BINARYFILE, see Ingest files as the FILE type (Beta).

YAML
resources:
  pipelines:
    onedrive_binary_pipeline:
      name: onedrive_binary_pipeline
      catalog: main
      schema: ingest_destination_schema
      channel: PREVIEW
      ingestion_definition:
        connection_name: <onedrive-connection>
        objects:
          - table:
              destination_catalog: main
              destination_schema: ingest_destination_schema
              destination_table: drive_files
              connector_options:
                onedrive_options:
                  entity_type: FILE
                  url: https://mytenant-my.sharepoint.com/personal/user_mytenant_onmicrosoft_com/Documents
                  file_ingestion_options:
                    format: BINARYFILE
                    schema_evolution_mode: NONE

Ingest files as the FILE type (Beta)​

Beta

This feature is in Beta.

Ingest files as the FILE type to copy each source file into a Unity Catalog volume and store a governed reference to each file and its metadata in the destination table. The destination table exposes a file column that you can pass to functions such as ai_parse_document and to user-defined functions (UDFs).

To use the FILE type, configure the following in your pipeline:

  • Set the pipeline channel to PREVIEW.
  • Set file_ingestion_options.format to FILE.
  • Set the databricks.filespace-preview table property to declare the volume Unity Catalog uses as managed storage.

For FILE requirements and behavior, see Ingest files as the FILE type.

YAML
resources:
  pipelines:
    onedrive_file_pipeline:
      name: onedrive_file_pipeline
      catalog: <catalog>
      schema: <schema>
      channel: PREVIEW
      ingestion_definition:
        connection_name: <onedrive-connection>
        objects:
          - table:
              destination_catalog: <catalog>
              destination_schema: <schema>
              destination_table: <destination-table>
              table_configuration:
                table_properties:
                  databricks.filespace-preview: /Volumes/<catalog>/<schema>/<volume>/
              connector_options:
                onedrive_options:
                  entity_type: FILE
                  url: https://mytenant-my.sharepoint.com/personal/user_mytenant_onmicrosoft_com/Documents
                  file_ingestion_options:
                    format: FILE
                    schema_evolution_mode: NONE

Ingest structured files​

Ingest structured files (for example, JSON files) from a OneDrive folder. Each row in the source files becomes a row in the destination table.

YAML
resources:
  pipelines:
    onedrive_json_pipeline:
      name: onedrive_json_pipeline
      catalog: main
      schema: ingest_destination_schema
      channel: PREVIEW
      ingestion_definition:
        connection_name: <onedrive-connection>
        objects:
          - table:
              destination_catalog: main
              destination_schema: ingest_destination_schema
              destination_table: json_files
              connector_options:
                onedrive_options:
                  entity_type: FILE
                  url: https://mytenant-my.sharepoint.com/personal/user_mytenant_onmicrosoft_com/Documents/json_folder
                  file_ingestion_options:
                    format: JSON
                    schema_evolution_mode: NONE

Ingest file metadata only​

Ingest file metadata (name, size, timestamps, path) without downloading file contents. Use this approach when you need an inventory of files without the overhead of ingesting their content.

Set entity_type to FILE_METADATA. FILE_METADATA ingestion must use the BINARYFILE format. Structured and semi-structured formats are not supported for metadata-only ingestion.

YAML
resources:
  pipelines:
    onedrive_metadata_pipeline:
      name: onedrive_metadata_pipeline
      catalog: main
      schema: ingest_destination_schema
      channel: PREVIEW
      ingestion_definition:
        connection_name: <onedrive-connection>
        objects:
          - table:
              destination_catalog: main
              destination_schema: ingest_destination_schema
              destination_table: file_metadata
              connector_options:
                onedrive_options:
                  entity_type: FILE_METADATA
                  url: https://mytenant-my.sharepoint.com/personal/user_mytenant_onmicrosoft_com/Documents
                  file_ingestion_options:
                    format: BINARYFILE
                    schema_evolution_mode: NONE

Declarative Automation Bundles job definition file​

This example job definition file runs the pipeline every day. It refreshes the pipeline from Ingest files as unstructured data. To refresh a different pipeline, replace onedrive_binary_pipeline with that pipeline's resource key.

YAML
resources:
  jobs:
    onedrive_job:
      name: onedrive_job

      trigger:
        periodic:
          interval: 1
          unit: DAYS

      email_notifications:
        on_failure:
          - <email-address>

      tasks:
        - task_key: refresh_pipeline
          pipeline_task:
            pipeline_id: ${resources.pipelines.onedrive_binary_pipeline.id}

Configure file ingestion options​

The file_ingestion_options block controls how files are processed. All options are set inside the connector_options.onedrive_options.file_ingestion_options block in the pipeline definition. The following snippets use Declarative Automation Bundles YAML syntax. In a notebook or API pipeline definition, use the same keys in JSON. For all parameters, see file_ingestion_options parameters.

File filters​

Use file_filters to restrict which files are ingested from the source URL:

YAML
file_ingestion_options:
  format: CSV
  file_filters:
    - path_filter: invoices/*.csv
    - modified_after: 2026-01-01T00:00:00

Schema evolution and schema hints​

Set schema_evolution_mode to control how new columns in incoming files are handled. Override inferred column types using schema_hints.

Schema evolution applies only to CSV, JSON, XML, PARQUET, and AVRO. For other formats (BINARYFILE, FILE, EXCEL, and ORC), set schema_evolution_mode to NONE.

YAML
file_ingestion_options:
  format: CSV
  schema_evolution_mode: ADD_NEW_COLUMNS_WITH_TYPE_WIDENING
  schema_hints: order_id INT, amount DOUBLE, ts TIMESTAMP

For more information about schema evolution and hints, see Configure schema inference and evolution in Auto Loader.

Format-specific options​

Pass format-specific options using format_options:

YAML
file_ingestion_options:
  format: CSV
  format_options:
    header: 'true'
    sep: ','

Supported keys are the standard Auto Loader format options. See Spark data source options.

Common patterns​

For advanced pipeline configurations, see Common patterns for managed ingestion pipelines.

Next steps​

Start, schedule, and set alerts on your pipeline. See Common pipeline maintenance tasks.

Additional resources​

On this page