Create an external location

This article describes how to configure an external location in Unity Catalog on Databricks.

External locations associate Unity Catalog storage credentials with cloud object storage containers. External locations are used to define managed storage locations for catalogs and schemas, and to define locations for external tables and external volumes.

You can create an external location using Catalog Explorer, the Databricks CLI, SQL commands in a notebook or Databricks SQL query, or Terraform.

Note

When you define a volume, you can no longer access any paths that overlap the volume location using external locations in the Catalog Explorer or cloud URIs.

Requirements

  • To create external locations, you must have the CREATE EXTERNAL LOCATION privilege on both the metastore and the storage credential referenced in the external location. Metastore admins have CREATE EXTERNAL LOCATION on the metastore by default.

  • Each cloud storage path can be associated with only one external location. If you attempt to create a second external location that references the same path, the command fails.

Important

The name of the S3 bucket that you want users to read from and write to cannot use dot notation (for example, incorrect.bucket.name.notation). For more bucket naming guidance, see the AWS bucket naming rules.

Create an external location using SQL

To create an external location using SQL, run the following command in a notebook or the SQL query editor. Replace the placeholder values:

  • <location-name>: A name for the external location. If location_name includes special characters, such as hyphens (-), it must be surrounded by backticks ( ` ` ). See Names.

  • <bucket-path>: The path in your cloud tenant that this external location grants access to.

  • <storage-credential-name>: The name of the storage credential that contains the IAM role ARN that authorizes reading from and writing to the S3 bucket. If the storage credential name includes special characters, such as hyphens (-), it must be surrounded by backticks ( ` ` ).

CREATE EXTERNAL LOCATION [IF NOT EXISTS] `<location-name>`
URL 's3://<bucket-path>'
WITH ([STORAGE] CREDENTIAL `<storage-credential-name>`)
[COMMENT '<comment-string>'];

Create an external location using an AWS CloudFormation template

If you create an external location using the AWS CloudFormation template, Databricks configures the external location and creates a storage credential for you. You also have the option to create the external location manually, which requires that you first create an IAM role that gives access to the S3 bucket that is referenced by the external location and a storage credential that references that IAM role. For details, see Create a storage credential.

Permissions required: Metastore admin or user with the CREATE EXTERNAL LOCATION and CREATE STORAGE CREDENTIAL privileges.

To create the external location:

  1. Open a workspace that is attached to the metastore.

  2. Click Catalog icon Catalog to open Catalog Explorer.

  3. Click the + Add button and select Add an external location.

  4. On the Create a new external location dialog, select AWS Quickstart (Recommended) then click Next.

    The AWS Quickstart configures the external location and creates a storage credential for you. If you choose to use the Manual option, you must manually create an IAM role that gives access to the S3 bucket and create the storage credential in Databricks yourself.

  5. On the Create external location with Quickstart dialog, enter the path to the S3 bucket in the Bucket Name field.

  6. Click Generate new token to generate the personal access token that you will use to authenticate between Databricks and your AWS account.

  7. Copy the token and click Launch in Quickstart.

  8. In the AWS CloudFormation template that launches (labeled Quick create stack), paste the token into the Databricks Account Credentials field.

  9. Accept the terms at the bottom of the page (I acknowledge that AWS CloudFormation might create IAM resources with custom names).

  10. Click Create stack.

    It may take a few minutes for the CloudFormation template to finish creating the external location object in Databricks.

  11. Return to your Databricks workspace and click Catalog to open Catalog Explorer.

  12. In the left pane of Catalog Explorer, scroll down and click External Data > External Locations.

  13. Confirm that a new external location has been created.

    Automatically-generated external locations use the naming syntax db_s3_external_databricks-S3-ingest-<id>.

  14. Grant permission to use the external location.

    For anyone to use the external location, you must grant permissions:

    • To use the external location to add a managed storage location to metastore, catalog, or schema, grant the CREATE MANAGED LOCATION privilege.

    • To create external tables or volumes, grant CREATE EXTERNAL TABLE or CREATE EXTERNAL VOLUME.

    To use Catalog Explorer to grant permissions:

    1. Click the external location name to open the details pane.

    2. On the Permissions tab, click Grant.

    3. On the Grant on <external location> dialog, select users, groups, or service principals in Principals field, and select the privilege you want to grant.

    4. Click Grant.

Create an external location using Catalog Explorer

You can create an external location using Catalog Explorer.

If you do not want to use the AWS CloudFormation template to create the external location and storage credential that is referenced by the external location, you can use the procedure described here. You must create a storage credential in advance. See Create a storage credential.

To create the external location:

  1. Click Catalog to open Catalog Explorer.

  2. Click the + Add button and select Add an external location.

  3. On the Create a new external location dialog, click Manual, then Next.

    To learn about the AWS Quickstart option, see Create an external location using an AWS CloudFormation template.

  4. In the Create a new external location manually dialog, enter an External location name.

  5. Optionally copy the S3 bucket path from an existing mount point.

  6. If you aren’t copying from an existing mount point, use the URL field to enter the S3 bucket path that you want to use as the external location.

  7. Select the storage credential that grants access to the external location.

  8. (Optional) If you want users to have read-only access to the external location, click Advanced Options and select Read only. For more information, see Mark an external location as read-only.

  9. Click Create.

  10. Grant permission to use the external location.

    For anyone to use the external location you must grant permissions:

    • To use the external location to add a managed storage location to metastore, catalog, or schema, grant the CREATE MANAGED LOCATION privilege.

    • To create external tables or volumes, grant CREATE EXTERNAL TABLE or CREATE EXTERNAL VOLUME.

    To use Catalog Explorer to grant permissions:

    1. Click the external location name to open the details pane.

    2. On the Permissions tab, click Grant.

    3. On the Grant on <external location> dialog, select users, groups, or service principals in Principals field, and select the privilege you want to grant.

    4. Click Grant.

Next steps