Microsoft OneDrive connector reference

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page by opting into the Lakeflow Connect for OneDrive For Business preview. See Manage Databricks previews.

This page contains reference material for the Microsoft OneDrive connector in Databricks Lakeflow Connect.

onedrive_options parameters

Set these options inside the connector_options.onedrive_options block of each table in your pipeline definition.

Parameter	Type	Required	Description
url	String	Yes	The URL of the OneDrive for Business My files drive, folder, file, or share link to ingest from. The URL must use a personal <tenant>-my.sharepoint.com host. See Read files from OneDrive for URL format details.
entity_type	String	No	The type of entity to ingest. Supported values: FILE (ingest file content and metadata) and FILE_METADATA (ingest metadata only, without downloading file contents). The default is FILE. Use FILE_METADATA only with the BINARYFILE format.
file_ingestion_options	Object	Yes	Controls file format and ingestion behavior. See file_ingestion_options parameters.

Parameter	Type	Required	Description
url	String	Yes	The URL of the OneDrive for Business My files drive, folder, file, or share link to ingest from. The URL must use a personal <tenant>-my.sharepoint.com host. See Read files from OneDrive for URL format details.
entity_type	String	No	The type of entity to ingest. Supported values: FILE (ingest file content and metadata) and FILE_METADATA (ingest metadata only, without downloading file contents). The default is FILE. Use FILE_METADATA only with the BINARYFILE format.
file_ingestion_options	Object	Yes	Controls file format and ingestion behavior. See file_ingestion_options parameters.

file_ingestion_options parameters

Set these options inside onedrive_options.file_ingestion_options.

Parameter	Type	Required	Description
format	String	Yes	The file format to ingest. Supported values: BINARYFILE, CSV, JSON, XML, EXCEL, PARQUET, AVRO, ORC, FILE (Beta). Use BINARYFILE or FILE for unstructured ingestion (PDFs, Office files, images). Use a structured format to parse file contents into rows. Structured formats apply only when entity_type is FILE. When entity_type is FILE_METADATA, use BINARYFILE.
file_filters	Array of objects	No	Filters that restrict which files are ingested. Each filter object can contain one of the following keys: path_filter (string): A glob pattern matched against file names or paths. Based on Spark path glob filter. modified_before (string): A timestamp in YYYY-MM-DDTHH:mm:ss format. Only files modified before this time are ingested. modified_after (string): A timestamp in YYYY-MM-DDTHH:mm:ss format. Only files modified after this time are ingested.
infer_column_types	Boolean	No	Whether to infer the exact data types for structured and semi-structured data. The default is true.
schema_evolution_mode	String	No	Controls how new columns in incoming files are handled. Modes match Auto Loader schema evolution modes. Supported values: ADD_NEW_COLUMNS_WITH_TYPE_WIDENING (default), ADD_NEW_COLUMNS, RESCUE, FAIL_ON_NEW_COLUMNS, NONE. Schema evolution applies only to CSV, JSON, XML, PARQUET, and AVRO. For other formats, set NONE. See How does Auto Loader schema evolution work?.
schema_hints	String	No	Overrides inferred column types. Specify as a comma-separated list of column_name TYPE pairs, for example order_id INT, amount DOUBLE. See Override schema inference with schema hints.
ignore_corrupt_files	Boolean	No	Whether to skip corrupt files during ingestion.
corrupt_record_column	String	No	The column that stores malformed records for formats that support a corrupt-record column.
rescued_data_column	String	No	The column that stores data that does not match the inferred or provided schema.
single_variant_column	String	No	The column that stores an entire input record as a single VARIANT value for formats that support this option.
reader_case_sensitive	Boolean	No	Whether column matching is case-sensitive when rescuing data.
format_options	Object	No	Format-specific parsing options. Keys are standard Auto Loader format option names. See Format options.

Parameter	Type	Required	Description
format	String	Yes	The file format to ingest. Supported values: BINARYFILE, CSV, JSON, XML, EXCEL, PARQUET, AVRO, ORC, FILE (Beta). Use BINARYFILE or FILE for unstructured ingestion (PDFs, Office files, images). Use a structured format to parse file contents into rows. Structured formats apply only when entity_type is FILE. When entity_type is FILE_METADATA, use BINARYFILE.
file_filters	Array of objects	No	Filters that restrict which files are ingested. Each filter object can contain one of the following keys: path_filter (string): A glob pattern matched against file names or paths. Based on Spark path glob filter. modified_before (string): A timestamp in YYYY-MM-DDTHH:mm:ss format. Only files modified before this time are ingested. modified_after (string): A timestamp in YYYY-MM-DDTHH:mm:ss format. Only files modified after this time are ingested.
infer_column_types	Boolean	No	Whether to infer the exact data types for structured and semi-structured data. The default is true.
schema_evolution_mode	String	No	Controls how new columns in incoming files are handled. Modes match Auto Loader schema evolution modes. Supported values: ADD_NEW_COLUMNS_WITH_TYPE_WIDENING (default), ADD_NEW_COLUMNS, RESCUE, FAIL_ON_NEW_COLUMNS, NONE. Schema evolution applies only to CSV, JSON, XML, PARQUET, and AVRO. For other formats, set NONE. See How does Auto Loader schema evolution work?.
schema_hints	String	No	Overrides inferred column types. Specify as a comma-separated list of column_name TYPE pairs, for example order_id INT, amount DOUBLE. See Override schema inference with schema hints.
ignore_corrupt_files	Boolean	No	Whether to skip corrupt files during ingestion.
corrupt_record_column	String	No	The column that stores malformed records for formats that support a corrupt-record column.
rescued_data_column	String	No	The column that stores data that does not match the inferred or provided schema.
single_variant_column	String	No	The column that stores an entire input record as a single VARIANT value for formats that support this option.
reader_case_sensitive	Boolean	No	Whether column matching is case-sensitive when rescuing data.
format_options	Object	No	Format-specific parsing options. Keys are standard Auto Loader format option names. See Format options.

table_configuration parameters

Set these options inside the table_configuration block of each table in your pipeline definition. table_configuration is a sibling of connector_options, not nested inside it.

Parameter	Type	Required	Description
storage_mode	String	No	The storage mode for the destination table. Supported values: SCD_TYPE_1 (default and only supported mode for BINARYFILE and FILE): Replaces the current row when a file changes and removes the current row when a file is deleted. APPEND_ONLY (default and only supported mode for structured formats): Appends new rows from new or updated files. Because these are the defaults and the only supported values, setting storage_mode explicitly is optional. Setting scd_type throws an error. Use storage_mode instead.

Parameter	Type	Required	Description
storage_mode	String	No	The storage mode for the destination table. Supported values: SCD_TYPE_1 (default and only supported mode for BINARYFILE and FILE): Replaces the current row when a file changes and removes the current row when a file is deleted. APPEND_ONLY (default and only supported mode for structured formats): Appends new rows from new or updated files. Because these are the defaults and the only supported values, setting storage_mode explicitly is optional. Setting scd_type throws an error. Use storage_mode instead.

Format options

The format_options block accepts standard Auto Loader format option keys for the selected format. Express each value as a string. For example:

JSON

"file_ingestion_options": {
  "format": "CSV",
  "format_options": {
    "header": "true",
    "sep": ","
  }
}

For format-specific option names and behavior, see Spark data source options.

Ingested data format

The schema of the destination table depends on the entity_type and format you configure.

FILE entity type with BINARYFILE format

When entity_type is FILE and format is BINARYFILE, each ingested file becomes one row with the following columns:

Field	Type	Description
path	string	The path of the file.
modificationTime	timestamp	The time the file was last modified.
length	bigint	The size of the file in bytes.
content	binary	The file content.
_file_id	string	The unique OneDrive identifier of the file.
_onedrive_metadata	struct	Contains OneDrive-specific metadata for the file. See _onedrive_metadata fields.
_file_metadata	struct	Standard file metadata added by Databricks during ingestion. See _file_metadata fields.

Field	Type	Description
path	string	The path of the file.
modificationTime	timestamp	The time the file was last modified.
length	bigint	The size of the file in bytes.
content	binary	The file content.
_file_id	string	The unique OneDrive identifier of the file.
_onedrive_metadata	struct	Contains OneDrive-specific metadata for the file. See _onedrive_metadata fields.
_file_metadata	struct	Standard file metadata added by Databricks during ingestion. See _file_metadata fields.

FILE entity type with FILE format

When entity_type is FILE and format is FILE, each ingested file becomes one row with the following columns:

Field	Type	Description
path	string	The path of the file.
size	bigint	The size of the file in bytes.
modification_time	timestamp	The time the file was last modified.
file	FILE	A managed reference to the copy of the source file in the configured Unity Catalog volume.
_file_id	string	The unique OneDrive identifier of the file.
_onedrive_metadata	struct	Contains OneDrive-specific metadata for the file. See _onedrive_metadata fields.
_file_metadata	struct	Standard file metadata added by Databricks during ingestion. See _file_metadata fields.

Field	Type	Description
path	string	The path of the file.
size	bigint	The size of the file in bytes.
modification_time	timestamp	The time the file was last modified.
file	FILE	A managed reference to the copy of the source file in the configured Unity Catalog volume.
_file_id	string	The unique OneDrive identifier of the file.
_onedrive_metadata	struct	Contains OneDrive-specific metadata for the file. See _onedrive_metadata fields.
_file_metadata	struct	Standard file metadata added by Databricks during ingestion. See _file_metadata fields.

FILE entity type with a structured format

When entity_type is FILE and format is a structured format (CSV, JSON, XML, EXCEL, PARQUET, AVRO, or ORC), the destination table schema matches the schema of the source files, plus the _onedrive_metadata and _file_metadata columns. Columns are inferred from the file contents, subject to the schema_evolution_mode and schema_hints settings.

FILE_METADATA entity type

When entity_type is FILE_METADATA, file content is not downloaded. The destination table contains only the metadata columns from the _onedrive_metadata and _file_metadata structs described above, plus _file_id.

_onedrive_metadata fields

The _onedrive_metadata struct contains the following fields. A field can be null when Microsoft Graph does not return a value.

Field	Type	Description
id	string	The unique OneDrive identifier of the file.
site_id	string	The identifier of the personal SharePoint site that owns the OneDrive drive.
drive_id	string	The OneDrive drive identifier.
drive_type	string	The type of OneDrive drive.
parent_id	string	The identifier of the parent folder.
parent_name	string	The name of the parent folder.
parent_path	string	The path of the parent folder.
web_url	string	A link to the file in OneDrive.
mime_type	string	The MIME type of the file.
created_by_email	string	The email address of the user who created the file.
created_by_name	string	The name of the user who created the file.
created_timestamp	timestamp	The time the file was created in OneDrive.
last_modified_by_email	string	The email address of the last user who modified the file.
last_modified_by_name	string	The name of the last user who modified the file.
etag	string	The entity tag for the file.
ctag	string	The content tag for the file.
description	string	The description of the file.
additional_metadata	variant	Additional metadata returned by Microsoft Graph that is not represented by another field.

Field	Type	Description
id	string	The unique OneDrive identifier of the file.
site_id	string	The identifier of the personal SharePoint site that owns the OneDrive drive.
drive_id	string	The OneDrive drive identifier.
drive_type	string	The type of OneDrive drive.
parent_id	string	The identifier of the parent folder.
parent_name	string	The name of the parent folder.
parent_path	string	The path of the parent folder.
web_url	string	A link to the file in OneDrive.
mime_type	string	The MIME type of the file.
created_by_email	string	The email address of the user who created the file.
created_by_name	string	The name of the user who created the file.
created_timestamp	timestamp	The time the file was created in OneDrive.
last_modified_by_email	string	The email address of the last user who modified the file.
last_modified_by_name	string	The name of the last user who modified the file.
etag	string	The entity tag for the file.
ctag	string	The content tag for the file.
description	string	The description of the file.
additional_metadata	variant	Additional metadata returned by Microsoft Graph that is not represented by another field.

_file_metadata fields

The _file_metadata struct contains standard file metadata added by Databricks during ingestion:

Field	Type	Description
file_path	string	The source path of the file.
file_name	string	The name of the file.
file_size	bigint	The size of the file in bytes.
file_block_start	bigint	The start byte offset of the file block.
file_block_length	bigint	The length of the file block in bytes.
file_modification_time	timestamp	The time the file was last modified.

Field	Type	Description
file_path	string	The source path of the file.
file_name	string	The name of the file.
file_size	bigint	The size of the file in bytes.
file_block_start	bigint	The start byte offset of the file block.
file_block_length	bigint	The length of the file block in bytes.
file_modification_time	timestamp	The time the file was last modified.