FileType

Applies to: Databricks Runtime 18 LTS and above

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

FileType is the PySpark type for the SQL FILE type, a reference to an unstructured file and its metadata. Use it to declare FILE parameters and return types in Python user-defined functions (UDFs). In Python, a FILE value is a FileRef object, which holds the file's metadata and reads its bytes.

For the SQL type reference and conceptual overview, see FILE type and FILE type and unstructured data. For file-processing UDF examples, see Process files with UDFs.

note

The FILE type isn't supported on serverless notebooks. It is supported on notebooks attached to serverless Databricks SQL warehouses.

Import

Python

from pyspark.sql.types import FileType, FileRef

FileType

FileType is a subclass of pyspark.sql.types.DataType. Use it as the parameter or return type of a UDF, or as a field type in a schema.

FileType doesn't specify MANAGED or EXTERNAL. Those qualifiers apply only to a FILE table column, where the column determines whether a reference is stored as managed or external. A UDF passes and returns FILE references, and the target column decides how each reference is stored when you write the result to a table.

You can use FileType in the following ways, which also apply to SQL and Scala UDFs and to SQL stored procedures:

• As a top-level type.
• Nested inside a StructType or an ArrayType.
• As the value type of a MapType, but not as a MapType key.
• Inside a VariantType, but only for external files.

When a query returns a FILE column to Python, either inside a UDF or through DataFrame.collect(), each value is a FileRef.

FileRef

A FileRef is the Python value for a FILE. It holds the file's metadata and has methods to read the file's bytes and to create new references.

Attributes

Attribute	Type	Description
uri	str	The URI of the file. Always set.
offset	int	An offset into the file, in bytes.
size	int	The size of the file in bytes.
content_type	str	The MIME type of the file, when known.
checksum	str	An integrity token for the file's bytes, of the form <algorithm>:<digest>. For the recognized algorithms, see Checksums.

Attribute	Type	Description
uri	str	The URI of the file. Always set.
offset	int	An offset into the file, in bytes.
size	int	The size of the file in bytes.
content_type	str	The MIME type of the file, when known.
checksum	str	An integrity token for the file's bytes, of the form <algorithm>:<digest>. For the recognized algorithms, see Checksums.

Methods

Method	Description
as_local_file()	Returns a pathlib.Path to the file. Pass the result to any library that accepts a path. Available on Databricks compute (notebooks and UDF workers). Not available on a Databricks Connect client, such as an IDE or local application that runs your code outside Databricks compute.
open()	Opens the file for binary reading and returns a file object. The caller closes it. Has the same compute requirement as as_local_file().
from_bytes(content, path=None, content_type=None)	Class method. Uploads content (a bytes value) to the Unity Catalog volume path given by path and returns a FileRef. Fails if a file already exists at the target path. Supported only inside a UDF.
from_local_file(local_file, path=None, content_type=None)	Class method. Uploads a local file to a Unity Catalog volume and returns a FileRef. The path and content_type parameters behave as in from_bytes. Supported only inside a UDF.

Method	Description
as_local_file()	Returns a pathlib.Path to the file. Pass the result to any library that accepts a path. Available on Databricks compute (notebooks and UDF workers). Not available on a Databricks Connect client, such as an IDE or local application that runs your code outside Databricks compute.
open()	Opens the file for binary reading and returns a file object. The caller closes it. Has the same compute requirement as as_local_file().
from_bytes(content, path=None, content_type=None)	Class method. Uploads content (a bytes value) to the Unity Catalog volume path given by path and returns a FileRef. Fails if a file already exists at the target path. Supported only inside a UDF.
from_local_file(local_file, path=None, content_type=None)	Class method. Uploads a local file to a Unity Catalog volume and returns a FileRef. The path and content_type parameters behave as in from_bytes. Supported only inside a UDF.

Example

In the following code, a scalar UDF receives a FILE value as a FileRef, opens the image, and returns its dimensions:

Python

from pyspark.sql.functions import col, udf
from pyspark.sql.types import StringType
from PIL import Image

@udf(returnType=StringType())
def image_resolution(file):
    with Image.open(file.as_local_file()) as img:
        return f"{img.width}x{img.height}"

spark.read.table("images").select(image_resolution(col("photo"))).display()

For more file-processing UDF examples, including generating files with a UDTF, see Process files with UDFs. For general UDF authoring, see Python scalar user-defined functions (UDFs) and Python user-defined table functions (UDTFs).

Limitations

Using FileType in PySpark has the following limitations:

• PySpark doesn't support declaring FILE MANAGED or FILE EXTERNAL table columns or ingesting files in bulk. Use SQL for those operations. PySpark only supports FILE, as FileType, in UDFs and file reads.
• as_local_file() and open() require cluster-side access, so they aren't available on a Databricks Connect client. Call them in a UDF or on cluster compute instead.
• For from_bytes and from_local_file, Python UDFs infer content_type from the path extension, whereas Scala UDFs infer it from the file's magic bytes.
• Returning a FileRef from a UDF that writes to a FILE MANAGED column isn't supported.

Related resources

• FILE type
• FILE type and unstructured data
• Process files with UDFs
• Python scalar user-defined functions (UDFs)
• Python user-defined table functions (UDTFs)
• Ingest files as the FILE type