Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage.
Auto Loader provides a Structured Streaming source called
cloudFiles. Given an input directory path on the cloud file storage, the
cloudFiles source automatically processes new files as they arrive, with the option of also processing existing files in that directory.
For details on how to use Auto Loader, see:
- Load files from Azure Data Lake Storage Gen2 (ADLS Gen2) using Auto Loader
- Load files from AWS S3 using Auto Loader
- Load files from Google Cloud Storage (GCS) using Auto Loader
- Ingest CSV data with Auto Loader
- Ingest JSON data with Auto Loader
- Ingest image data with Auto Loader
- Schema inference and evolution in Auto Loader