Clean up processed files with Auto Loader
cloudFiles.cleanSource is available in Databricks Runtime 16.4 and above.
Use cloudFiles.cleanSource to move or delete files from the source directory after they are processed. Removing processed files reduces storage costs and shortens the duration of future listing operations.
Mode | Description |
|---|---|
| The files in the source directory are not moved or deleted. |
| The files in the source directory are moved to the path specified by |
| The files in the source directory are deleted after the retention duration (default 30 days) defined by |
Additional option | Default | Valid values | Description |
|---|---|---|---|
|
| A CalendarInterval string such as | Amount of time to wait before processed files become candidates for cleanup with clean source. Must be greater than 7 days for |
|
|
| This option is available in Databricks Runtime 19 and above. Clean source is by default a best-effort operation. If the stream completes processing files before clean source is done moving or deleting files, the clean source operation is terminated. Setting This only applies when the stream completes on its own (for example, an |
| None | A cloud storage or Unity Catalog volume path | Path to archive processed files to when The move location must:
Auto Loader must have write permissions to this directory. |
Considerations before enabling cloudFiles.cleanSource
- Databricks does not recommend using this option when multiple streams consume data from the same source directory. The fastest stream cleans up the files, so the slower streams never ingest them.
- Enabling this feature requires Auto Loader to maintain additional state in its checkpoint, which incurs performance overhead but enables improved observability through the
cloud_files_statetable-valued function. Seecloud_files_statetable-valued function. - Clean source uses the current setting to decide whether to
MOVEorDELETEa given file. For example, suppose that the setting wasMOVEwhen the file was originally processed but was changed toDELETEwhen the file became a candidate for cleanup 30 days later. In this case, clean source deletes the file. - Files are not guaranteed to be cleaned as soon as the
cloudFiles.cleanSource.retentionDurationexpires. To keep costs low, Auto Loader cleans up files concurrently with stream processing and terminates as soon as the stream processing is complete or is terminated. Files that were candidates for cleanup, but couldn't be cleaned during the stream processing are picked up the next time Auto Loader runs.
Notes about clean source
-
Clean source only runs if there is a batch of files to process. It is not a background process that runs independently of ingestion. If there are no new files to ingest in the source directory, clean source does not start for the current stream run. As a result, if a stream stops receiving new files, files that have already passed their retention duration are not cleaned up until a later stream run processes a new batch.
This batch requirement applies regardless of
cloudFiles.cleanSource.waitForCompletion. That option only keeps the stream alive long enough to finish an in-progress cleanup within a run. It does not start clean source when there is no batch to process. -
If a file is ingested on the Nth stream run, the
commit_timefor the file is set on the N+1 stream run.commit_timemust be set before clean source can determine whether a file is eligible for move or delete, so the earliest a file can become a candidate for cleanup is the N+2 stream run. -
Setting
commit_timeis necessary but not sufficient. A file is only cleaned once its retention duration has elapsed, measured from itscommit_time. For example, with the defaultcloudFiles.cleanSource.retentionDurationof 30 days, a file processed today is not eligible for cleanup until 30 days after itscommit_timeis set. This holds regardless of how many stream runs occur in between. Both conditions must be met before the file is moved or deleted.