Auto Loader

Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage.

Auto Loader provides a Structured Streaming source called cloudFiles. Given an input directory path on the cloud file storage, the cloudFiles source automatically processes new files as they arrive, with the option of also processing existing files in that directory.

For details on how to use Auto Loader, see: