Google Drive connector reference

This page contains reference documentation for the Google Drive connector in Databricks Lakeflow Connect.

gdrive_options parameters

Set these options inside the connector_options.gdrive_options block of each table in your pipeline definition.

Parameter	Type	Required	Description
entity_type	String	Yes	The type of entity to ingest. Supported values: FILE: Ingest file content and metadata. FILE_METADATA: Ingest metadata only, without downloading file contents.
url	String	Yes	The URL of the Google Drive folder or shared drive to ingest from. Examples: https://drive.google.com/drive/folders/<folder_id> A shared drive URL
file_ingestion_options	Object	Yes	Controls file format and ingestion behavior. See file_ingestion_options parameters.

file_ingestion_options parameters

Set these options inside gdrive_options.file_ingestion_options.

Parameter	Type	Required	Description
format	String	Yes	The file format to ingest. Supported values: BINARYFILE, CSV, JSON, XML, EXCEL, PARQUET, AVRO, ORC. Use BINARYFILE for unstructured ingestion (PDFs, Office files, images). Use a structured format to parse file contents into rows.
file_filters	Array of objects	No	Filters that restrict which files are ingested. Each filter object can contain one of the following keys: path_filter (string): A glob pattern matched against file paths. Based on Spark path glob filter. modified_before (string): A timestamp in YYYY-MM-DDTHH:mm:ss format. Only files modified before this time are ingested. modified_after (string): A timestamp in YYYY-MM-DDTHH:mm:ss format. Only files modified after this time are ingested.
schema_evolution_mode	String	No	Controls how new columns in incoming files are handled. Modes match Auto Loader schema evolution modes. Supported values: ADD_NEW_COLUMNS_WITH_TYPE_WIDENING (default), ADD_NEW_COLUMNS, RESCUE, FAIL_ON_NEW_COLUMNS, NONE.
schema_hints	String	No	Overrides inferred column types. Specify as a comma-separated list of column_name TYPE pairs, for example order_id INT, amount DOUBLE. See Override schema inference with schema hints.
format_options	Object	No	Format-specific parsing options. Keys are standard Auto Loader format option names. See Format options.

table_configuration parameters

Set these options inside the table_configuration block of each table in your pipeline definition. table_configuration is a sibling of connector_options, not nested inside it.

Parameter	Type	Required	Description
storage_mode	String	No	The storage mode for the destination table. Supported values: SCD_TYPE_1 (default for BINARYFILE): Overwrites records when files change or are deleted. APPEND_ONLY (default for structured formats): Appends new rows from new or updated files. Because these are the defaults and the only supported values, setting storage_mode explicitly is optional. Do not use the scd_type field — it throws an error.

Format options

The format_options block accepts standard Auto Loader format option keys, organized below by file format. For details, see Auto Loader.

JSON

Key	Description
allowBackslashEscapingAnyCharacter	Allows backslashes to escape any character.
allowComments	Allows Java- and C++-style comments in JSON content.
allowNonNumericNumbers	Allows NaN and Infinity as valid float values.
allowNumericLeadingZeros	Allows leading zeros in integer values.
allowSingleQuotes	Allows single quotes as string delimiters in addition to double quotes.
allowUnquotedControlChars	Allows unquoted control characters in JSON strings.
allowUnquotedFieldNames	Allows unquoted field names.
badRecordsPath	Path to store corrupt or unparseable records instead of failing the pipeline.
charset / encoding	Character encoding of the file (for example, UTF-8, ISO-8859-1).
dateFormat	Pattern for parsing date strings (for example, yyyy-MM-dd).
dropFieldIfAllNull	Ignores columns where all values are null or empty during schema inference.
inferTimestamp	Infers TimestampType for strings that match a timestamp pattern.
lineSep	Line separator character or string.
locale	Locale for parsing dates and numbers (for example, en-US).
mode	Behavior for malformed records: PERMISSIVE (default), DROPMALFORMED, or FAILFAST.
multiLine	Parses records that span multiple lines.
prefersDecimal	Infers DecimalType instead of FloatType or DoubleType where possible.
primitivesAsString	Infers all primitive values as StringType.
readerCaseSensitive	Enables case-sensitive column name matching against the schema.
timestampFormat	Pattern for parsing timestamp strings (for example, yyyy-MM-dd'T'HH:mm:ss).
timeZone	Time zone for parsing timestamps (for example, UTC, America/New_York).

CSV

Supports all JSON options above, plus the following CSV-specific options:

Key	Description
charToEscapeQuoteEscaping	Escape character used before a quote character inside a quoted field.
comment	Character that marks a line as a comment; lines beginning with this character are skipped.
delimiter / sep	Column delimiter character (default: ,).
emptyValue	String to use for empty values when writing.
enforceSchema	Applies the declared schema to CSV data, ignoring header names.
escape	Escape character (default: \).
header	Whether the first row contains column names (default: false).
ignoreLeadingWhiteSpace	Trims leading whitespace from values.
ignoreTrailingWhiteSpace	Trims trailing whitespace from values.
maxCharsPerColumn	Maximum number of characters allowed per column value.
maxColumns	Maximum number of columns allowed in a record.
mergeSchema	Merges schema across multiple CSV files.
nanValue	String representation of NaN.
negativeInf	String representation of negative infinity.
nullValue	String that represents a null value.
parserCaseSensitive	Enables case-sensitive matching between header names and schema field names.
positiveInf	String representation of positive infinity.
preferDate	Infers DateType for date-like strings instead of TimestampType.
quote	Quote character used to enclose field values that contain the delimiter (default: ").
skipRows	Number of rows to skip at the beginning of the file before the header or data.
unescapedQuoteHandling	How to handle unescaped quote characters inside quoted fields.

XML

Key	Description
arrayElementName	Name of the XML element wrapping each array item when writing.
attributePrefix	Prefix added to XML attribute names to distinguish them from element names (default: _).
compression	Compression codec for reading (for example, gzip, bzip2).
declaration	XML declaration string to prepend when writing.
encoding	Character encoding of the XML file.
excludeAttribute	Excludes XML element attributes from parsing.
ignoreSurroundingSpaces	Ignores whitespace surrounding element values.
ignoreNamespace	Ignores XML namespace prefixes during parsing.
locale	Locale for parsing dates and numbers.
mode	Behavior for malformed records: PERMISSIVE, DROPMALFORMED, or FAILFAST.
nullValue	String that represents a null value.
rootTag	Root element tag name.
rowTag	XML element tag that identifies each row (required).
rowValidationXSDPath	Path to an XSD schema file for validating each row element.
samplingRatio	Fraction of rows sampled for schema inference (default: 1.0).
timestampFormat	Pattern for parsing timestamp strings.
timestampNTZFormat	Pattern for parsing timestamp-without-timezone strings.
timeZone	Time zone for parsing timestamps.
validateName	Validates that XML element names conform to the XML specification.
valueTag	Tag name used for text values in elements that also have attributes (default: _VALUE).

Parquet

Key	Description
datetimeRebaseMode	Handling for dates and timestamps written in Julian calendar format: EXCEPTION, CORRECTED, or LEGACY.
int96RebaseMode	Handling for INT96 timestamps written in Julian calendar format: EXCEPTION, CORRECTED, or LEGACY.
mergeSchema	Merges schema across multiple Parquet files.

Avro

Key	Description
avroSchema	Avro schema in JSON string format. Use to enforce a specific schema during reads.
datetimeRebaseMode	Handling for dates and timestamps written in Julian calendar format: EXCEPTION, CORRECTED, or LEGACY.
mergeSchema	Merges schema across multiple Avro files.

Ingested data format

The schema of the destination table depends on the entity_type and format you configure.

BINARYFILE entity type (FILE)

When entity_type is FILE and format is BINARYFILE, each ingested file becomes one row with the following columns:

Field	Type	Description
file_id	String	The Google Drive identifier of the file.
file_metadata	Struct	Contains generic file metadata: name (string): The name of the file, as it appears in Google Drive. size_in_bytes (bigint): The size of the file. created_timestamp (timestamp): The timestamp at which the file was created in Google Drive. last_modified_timestamp (timestamp): The timestamp at which the file was last modified in Google Drive. created_by_email (string): The email address of the user who created the file. May be null if not available. last_modified_by_email (string): The email address of the last user who modified the file. May be null if not available.
_file_metadata	Struct	Contains Google Drive-specific metadata for the file: drive_id (string): The Google Drive identifier of the shared drive. Null for files in My Drive. file_folder_path (string): The file path of the file in Google Drive. mime_type (string): The MIME type of the file. web_url (string): A link to the file in Google Drive.
content	Struct	Contains file content.
_metadata	Struct	Standard file metadata added by Databricks during ingestion. Contains source file information such as path and modification time.

Structured entity type (FILE with structured format)

When entity_type is FILE and format is a structured format (CSV, JSON, XML, EXCEL, PARQUET, AVRO, or ORC), the destination table schema matches the schema of the source files. Columns are inferred from the file contents, subject to the schema_evolution_mode and schema_hints settings.

FILE_METADATA entity type

When entity_type is FILE_METADATA, file content is not downloaded. The destination table contains only the metadata columns from the file_metadata and _file_metadata structs described above, plus file_id.