Create an external location to connect cloud storage to SAP Databricks

This article describes how to configure an external location in Unity Catalog to connect cloud storage to SAP Databricks.

warning

To prevent data loss, SAP Databricks requires that external locations be read-only.

External locations associate Unity Catalog storage credentials with cloud object storage containers. External locations are used to define managed storage locations for catalogs and schemas, and to define locations for external tables and external volumes.

You can create an external location that references storage in an AWS S3 bucket.

Before you begin

Prerequisites:

• You must create the S3 bucket that you want to use as an external location before you create the external location object in SAP Databricks.

  • The AWS CloudFormation template supports only S3 buckets.
  • The name of an S3 bucket that you want users to read from and write to cannot use dot notation (for example, incorrect.bucket.name.notation). For more bucket naming guidance, see the AWS bucket naming rules.
  • Avoid using a path in S3 that is already defined as an external location in another Unity Catalog metastore. You can safely read data in a single external S3 location from more than one metastore, but concurrent writes to the same S3 location from multiple metastores can lead to consistency issues.

• If you don't use the AWS CloudFormation template to create the external location, you must first create a storage credential in SAP Databricks that gives access to the cloud storage location path. See the following AWS doc: Create a storage credential.

  If you use the AWS CloudFormation flow, that storage credential is created for you.

Permissions requirements:

• You must have the CREATE EXTERNAL LOCATION privilege on both the metastore and the storage credential referenced in the external location. Metastore admins have CREATE EXTERNAL LOCATION on the metastore by default.
• If you are using the AWS CloudFormation template, you must also have the CREATE STORAGE CREDENTIAL privilege on the metastore. Metastore admins have CREATE STORAGE CREDENTIAL on the metastore by default.

Create an external location for an S3 bucket using an AWS CloudFormation template

If you create an external location using the AWS CloudFormation template, Databricks configures the external location and creates a storage credential for you. You also have the option to create the external location manually, which requires that you first create an IAM role that gives access to the S3 bucket that is referenced by the external location and a storage credential that references that IAM role.

Permissions and prerequisites: see Before you begin.

To create the external location:

1. Log in to a workspace that is attached to the metastore.

2. Click Catalog to open Catalog Explorer.

3. On the Quick access page, click the External data > button, go to the External Locations tab, and click Create external location.

4. On the Create a new external location dialog, select AWS Quickstart (Recommended) then click Next.

   The AWS Quickstart configures the external location and creates a storage credential for you. If you choose to use the Manual option, you must manually create an IAM role that gives access to the S3 bucket and create the storage credential in Databricks yourself.

5. On the Create external location with Quickstart dialog, enter the path to the S3 bucket in the Bucket Name field.

6. Click Generate new token to generate the personal access token that you will use to authenticate between Databricks and your AWS account.

7. Copy the token and click Launch in Quickstart.

8. In the AWS CloudFormation template that launches (labeled Quick create stack), paste the token into the Databricks Account Credentials field.

9. Accept the terms at the bottom of the page (I acknowledge that AWS CloudFormation might create IAM resources with custom names).

10. Click Create stack.

    It may take a few minutes for the CloudFormation template to finish creating the external location object in Databricks.

11. Return to your Databricks workspace and click Catalog to open Catalog Explorer.

12. On the Quick access page, click the External data > button to go to the External Locations tab.

13. Confirm that a new external location has been created.

    Automatically-generated external locations use the naming syntax db_s3_external_databricks-S3-ingest-<id>.

14. (Optional) Bind the external location to specific workspaces.

    By default, any privileged user can use the external location on any workspace attached to the metastore. If you want to allow access only from specific workspaces, go to the Workspaces tab and assign workspaces. See (Optional) Assign an external location to specific workspaces.

15. Grant permission to use the external location.

    For anyone to use the external location, you must grant permissions:

    • To use the external location to add a managed storage location to metastore, catalog, or schema, grant the CREATE MANAGED LOCATION privilege.
    • To create external tables or volumes, grant CREATE EXTERNAL TABLE or CREATE EXTERNAL VOLUME.

    To use Catalog Explorer to grant permissions:

    1. Click the external location name to open the details pane.
    2. On the Permissions tab, click Grant.
    3. On the Grant on <external location> dialog, select users, groups, or service principals in Principals field, and select the privilege you want to grant.
    4. Click Grant.

(Optional) Assign an external location to specific workspaces

By default, an external location is accessible from all of the workspaces in the metastore. This means that if a user has been granted a privilege (such as READ FILES) on that external location, they can exercise that privilege from any workspace attached to the metastore. If you use workspaces to isolate user data access, you might want to allow access to an external location only from specific workspaces. This feature is known as workspace binding or external location isolation.

Typical use cases for binding an external location to specific workspaces include:

• Ensuring that data engineers who have the CREATE EXTERNAL TABLE privilege on an external location that contains production data can create external tables on that location only in a production workspace.
• Ensuring that data engineers who have the READ FILES privilege on an external location that contains sensitive data can only use specific workspaces to access that data.

important

Workspace bindings are referenced at the point when privileges against the external location are exercised. For example, if a user creates an external table by issuing the statement example from the myWorkspace workspace, the following workspace binding checks are performed in addition to regular user privilege checks:

• Is the external location covering path bound to myWorkspace?
• Is the catalog myCat bound to myWorkspace with access level Read & Write?

If the external location is subsequently unbound from myWorkspace, then the external table continues to function.

This feature also allows you to populate a catalog from a central workspace and make it available to other workspaces using catalog bindings, without also having to make the external location available in those other workspaces.

Bind an external location to one or more workspaces

To assign an external location to specific workspaces, you can use Catalog Explorer.

Permissions required: Metastore admin, external location owner, or MANAGE on the external location.

Metastore admins can see all external locations in a metastore using Catalog Explorer—and external location owners can see all external locations that they own in a metastore—regardless of whether the external location is assigned to the current workspace. External locations that are not assigned to the workspace appear grayed out.

1. Log in to a workspace that is linked to the metastore.

2. In the sidebar, click Catalog.

3. On the Quick access page, click the External data > button to go to the External Locations tab.

4. Select the external location and go to the Workspaces tab.

5. On the Workspaces tab, clear the All workspaces have access checkbox.

   If your external location is already bound to one or more workspaces, this checkbox is already cleared.

6. Click Assign to workspaces and enter or find the workspaces you want to assign.

To revoke access, go to the Workspaces tab, select the workspace, and click Revoke. To allow access from all workspaces, select the All workspaces have access checkbox.