Work with unstructured data

Store, govern, and process unstructured files, such as documents, images, audio, and video, alongside structured data in Unity Catalog.

Get started

Try an end-to-end example to see how the pieces fit together.

• • FILE functions quickstart
  • Create a table with a FILE column and work with files using FILE functions.
• • Unstructured data tutorial
  • Store, query, process, and share unstructured files using Unity Catalog volumes.

Store and reference files

Choose how to store and reference files based on whether you access them by path or as values in a table.

• • Unity Catalog volumes
  • Govern storage for non-tabular files. Upload, organize, and access files in a volume by path.
• • FILE type
  • Reference files and their metadata as first-class values in a table, governed by access to the rows that reference them. For the type definition, see the FILE type reference.
• • Binary data source
  • Store the raw bytes of a file in a BINARY column.
• • Image data source
  • Read image files into a DataFrame.

Process files

After you store your files in Unity Catalog, extract content, run custom code, and build pipelines.

• • Parse documents
  • Extract structured content from documents with ai_parse_document and other AI functions.
• • Process files with UDFs
  • Read file metadata and bytes, and generate new files, using your own code in UDFs.
• • Ingest files as FILE references
  • Load files into tables as FILE references.
• • Build a file pipeline
  • Process files end to end in a pipeline.
• • Search files in a volume
  • Index the files in a volume so that users and agents can search them.

Govern files

Control who can access files and manage their lifecycle with Unity Catalog.

• • Volume access control
  • Grant privileges on volumes to control who can read and write files by path.
• • FILE MANAGED lifecycle
  • Let Databricks manage the storage and lifecycle of the files that a table references.
• • Row filters on FILE columns
  • Restrict access to files by applying row filters to the rows that reference them.