Azure Data Lake Storage Gen2 FAQ

Frequently asked questions (FAQ)

Can I use the abfs scheme to access Azure Data Lake Storage Gen2?

Yes. However, Databricks recommends that you use the abfss scheme, which uses SSL encrypted access. You must use abfss with OAuth or Azure Active Directory-based authentication because of the requirement for secure transport of Azure AD tokens.

When I accessed an Azure Data Lake Storage Gen2 account with the hierarchical namespace enabled, I experienced a java.io.FileNotFoundException error, and the error message includes FilesystemNotFound.

If the error message includes the following information, it is because your command is trying to access a Blob storage container created through the Azure portal:

StatusCode=404
StatusDescription=The specified filesystem does not exist.
ErrorCode=FilesystemNotFound
ErrorMessage=The specified filesystem does not exist.

When a hierarchical namespace is enabled, you do not need to create containers through Azure portal. If you see this issue, delete the Blob container through Azure portal. After a few minutes, you will be able to access the container. Alternatively, you can change your abfss URI to use a different container, as long as this container is not created through Azure portal.

Known issues

See Known issues with Azure Data Lake Storage Gen2 in the Microsoft documentation.