Azure Data Lake Storage Gen2 FAQ
Frequently asked questions (FAQ)
Can I use the
abfs scheme to access Azure Data Lake Storage Gen2?
Yes. However, Databricks recommends that you use the
abfss scheme, which uses SSL encrypted access. You must use
abfss with OAuth or Azure Active Directory-based authentication because of the requirement for secure transport of Azure AD tokens.
When I accessed an Azure Data Lake Storage Gen2 account with the hierarchical namespace enabled, I experienced a
java.io.FileNotFoundException error, and the error message includes
If the error message includes the following information, it is because your command is trying to access a Blob storage container created through the Azure portal:
StatusCode=404 StatusDescription=The specified filesystem does not exist. ErrorCode=FilesystemNotFound ErrorMessage=The specified filesystem does not exist.
When a hierarchical namespace is enabled, you do not need to create containers through Azure portal. If you see this issue,
delete the Blob container through Azure portal. After a few minutes, you will be able to access the container. Alternatively, you can
abfss URI to use a different container, as long as this container is not created through Azure portal.
See Known issues with Azure Data Lake Storage Gen2 in the Microsoft documentation.