Connect to cloud object storage using Unity Catalog

This article gives an overview of the cloud storage connection configurations that are required to work with data using Unity Catalog.

Databricks recommends using Unity Catalog to manage access to all data stored in cloud object storage. Unity Catalog provides a suite of tools to configure secure connections to cloud object storage. These connections provide access to complete the following actions:

Ingest raw data into a lakehouse.
Create and read managed tables in secure cloud storage.
Register or create external tables containing tabular data.
Read and write unstructured data.

Warning

Do not give end users storage-level access to Unity Catalog managed tables or volumes. This compromises data security and governance.

Avoid granting users direct access to Amazon S3 or Cloudflare R2 buckets that are used as Unity Catalog managed storage. The only identity that should have access to data managed by Unity Catalog is the identity used by Unity Catalog. Ignoring this creates the following issues in your environment:

Access controls established in Unity Catalog can be circumvented by users who have direct access to S3 or R2 buckets.
Auditing, lineage, and other Unity Catalog monitoring features will not capture direct access.
The lifecycle of data is broken. That is, modifying, deleting, or evolving tables in Databricks will break the consumers that have direct access to storage, and writes outside of Databricks could result in data corruption.

Note

If your workspace was created before November 8, 2023, it might not be enabled for Unity Catalog. An account admin must enable Unity Catalog for your workspace. See Enable a workspace for Unity Catalog.

How does Unity Catalog connect object storage to Databricks?

Databricks on AWS supports both AWS S3 and Cloudflare R2 buckets (Public Preview) as cloud storage locations for data assets registered in Unity Catalog. R2 is intended primarily for uses cases in which you want to avoid data egress fees, such as Delta Sharing across clouds and regions. For more information, see Use Cloudflare R2 replicas or migrate storage to R2.

To manage access to the underlying cloud storage that holds tables and volumes, Unity Catalog uses the following object types:

A storage credential represents an authentication and authorization mechanism for accessing data stored on your cloud tenant, using an IAM role for S3 buckets or an R2 API token for Cloudflare R2 buckets. Each storage credential is subject to Unity Catalog access-control policies that control which users and groups can access the credential. If a user does not have access to a storage credential in Unity Catalog, the request fails and Unity Catalog does not attempt to authenticate to your cloud tenant on the user’s behalf. Permission to create storage credentials should only be granted to users who need to define external locations. See Create a storage credential for connecting to AWS S3 and Create a storage credential for connecting to Cloudflare R2.
An external location is an object that combines a cloud storage path with a storage credential that authorizes access to the cloud storage path. Each storage location is subject to Unity Catalog access-control policies that control which users and groups can access the credential. If a user does not have access to a storage location in Unity Catalog, the request fails and Unity Catalog does not attempt to authenticate to your cloud tenant on the user’s behalf. Permission to create and use external locations should only be granted to users who need to create external tables, external volumes, or managed storage locations. See Create an external location to connect cloud storage to Databricks.

External locations are used both for external data assets, like external tables and external volumes, and for managed data assets, like managed tables and managed volumes. For more information about the difference, see Tables and Volumes.

When an external location is used for storing managed tables and managed volumes, it is called a managed storage location. Managed storage locations can exist at the metastore, catalog, or schema level. Databricks recommends configuring managed storage locations at the catalog level. If you need more granular isolation, you can specify managed storage locations at the schema level. Workspaces that are enabled for Unity Catalog automatically have no metastore-level storage by default, but you can specify a managed storage location at the metastore level to provide default storage when no catalog-level storage is defined. Workspaces that are enabled for Unity Catalog manually receive a metastore-level managed storage location by default. See Specify a managed storage location in Unity Catalog and Unity Catalog best practices.

Volumes are the securable object that most Databricks users should use to interact directly with non-tabular data in cloud object storage. See Create and work with volumes.

Note

While Unity Catalog supports path-based access to external tables and external volumes using cloud storage URIs, Databricks recommends that users read and write all Unity Catalog tables using table names and access data in volumes using /Volumes paths.

Next steps

If you’re just getting started with Unity Catalog as an admin, see Set up and manage Unity Catalog.

If you’re a new user and your workspace is already enabled for Unity Catalog, see Tutorial: Create your first table and grant privileges in Unity Catalog.