Storage credentials

Unity Catalog and the built-in Databricks Hive metastore use default locations for managed tables. Unity Catalog introduces several new securable objects to grant privileges to data in cloud object storage.

Storage credential

A storage credential is a securable object representing an AWS IAM role.

Once a storage credential is created access to it can be granted to principals (users and groups).

A user or group with permission to use a storage credential can access any storage path covered by the storage credential by using WITH (CREDENTIAL = credential) in your SQL command.

For more fine-grained access control, combine a storage credential with an external location.

Storage credential names are unqualified and must be unique within the metastore.

Graphical Representation of relationships

The following diagram describes the relationship between:

  • storage credentials

  • external locations

  • external tables

  • storage paths

  • IAM entities

  • Azure service accounts

External location ER diagram

Examples

Using CLI create a storage credential my_storage_cred for an AWS IAM role.

databricks unity-catalog create-storage-credential --json '{"name": "my_storage_cred", "aws_iam_role": {"role_arn": "arn:aws:iam::123456789:role/us-west-2-my-account"}}'

The rest of the commands can be run within SQL.

-- Grant access to the storage credential
> GRANT READ FILES ON STORAGE CREDENTIAL my_aws_storage_cred TO ceo;

-- ceo can directly read from any storage path using my_aws_storage_cred
> SELECT count(1) FROM `delta`.`s3://depts/finance/forecast/somefile` WITH (CREDENTIAL my_aws_storage_cred);
  100
> SELECT count(1) FROM `delta`.`s3://depts/hr/employees` WITH (CREDENTIAL my_aws_storage_cred);
  2017

-- Create an external location on specific path to which `my_aws_storage_cred` has access
> CREATE EXTERNAL LOCATION finance_loc URL 's3://depts/finance'
     WITH (CREDENTIAL my_aws_storage_cred)
     COMMENT 'finance';