Work with unstructured data

Store, govern, and process unstructured files, such as documents, images, audio, and video, alongside structured data in Unity Catalog.

Store and reference files

Store and reference unstructured files in Unity Catalog in the following ways:

• Unity Catalog volumes govern storage for non-tabular files. You can upload, organize, and access files in a volume by path. See What are Unity Catalog volumes?.
• The FILE type references files and their metadata as first-class values in a table, governed by access to the rows that reference them. See FILE type and unstructured data and FILE type.
• The binary file data source stores the raw bytes of a file in a BINARY column. See Read binary files.
• The image data source reads image files into a DataFrame. See Read image files.

Process files

After storing your files in Unity Catalog, process them in the following ways:

• Extract structured content from documents with ai_parse_document function and other AI functions.
• Process files with your own code using UDFs. See Process files with UDFs.
• Ingest files into tables as FILE references. See Ingest files as the FILE type.
• Build a pipeline that processes files end to end. See Tutorial: Build a file-processing pipeline with the FILE type.

In this section

• Read binary files
• Read image files
• FILE type and unstructured data
• Process files with UDFs