Skip to main content
Last updated on

CF_CLEAN_SOURCE_UNAUTHORIZED_WRITE_PERMISSION error condition

SQLSTATE: 42501

Auto Loader cannot archive processed files because it does not have write permissions to the source directory or the move destination.

NEED_PERMISSIONS​

<reason>

To fix you can either:

  1. Grant write permissions to the source directory and move destination OR

  2. Set cleanSource to 'OFF'

You can also unblock your stream by setting the SQLConf spark.databricks.cloudFiles.cleanSource.disabledDueToAuthorizationErrors to 'true'.

USE_VOLUMES_FOR_MOVE​

Cannot create cloud access tokens scoped to both the source and move destination. Please define a Volume over the parent directory of your source data, and move the files within the volume to a separate directory.

You can also unblock your stream by setting the SQLConf spark.databricks.cloudFiles.cleanSource.disabledDueToAuthorizationErrors to 'true'.