Manage Unity Catalog metastores

This article shows how to update, delete, and manage the behavior of Unity Catalog metastores in your Databricks account.

To learn about Unity Catalog metastores and how to create them, see Create a Unity Catalog metastore.

Enable a metastore to be automatically assigned to new workspaces

To assign an existing Unity Catalog metastore automatically to new workspaces in that metastore’s region, an account admin can enable workspace auto-assignment for the metastore. If this setting is not selected, the admin who creates a workspace in the same region as the metastore must manually enable the workspace for Unity Catalog and select the metastore from a drop-down.

Before an account admin enables this option, they should be sure to understand the following impacts on new workspaces:

• A workspace catalog will be created, and all workspace users will have the privileges required to create objects in it. See Automatic enablement of Unity Catalog.

• Workspace admins will have the permissions required to create metastore-level securable objects, like catalogs and external locations. See Workspace admin privileges when workspaces are enabled for Unity Catalog automatically.

• If metastore-level storage is already enabled for the metastore, the workspace will be able to use that storage. See Specify a managed storage location in Unity Catalog.

• If a metastore admin is defined for the metastore, they will be able to manage access to all securable objects in all workspaces attached to the metastore. See Metastore admins.

• The Delta Sharing setting (enabled or disabled) for the metastore will apply to all workspaces attached to the metastore. See Set up Delta Sharing for your account (for providers).

To enable automatic assignment:

1. As an account admin, go to the Databricks account console.

2. Click Catalog.

3. Select your metastore.

4. On the Configuration tab, under Workspace assignment, select Automatically assign new workspaces in <region> to this metastore.

5. On the confirmation dialog, click Enable auto-assignment.

Add managed storage to an existing metastore

Metastore-level managed storage is optional, and it is not included for metastores that were created automatically. You might want to add metastore-level storage to your metastore if you prefer a data isolation model that stores data centrally for multiple workspaces. You need metastore-level storage if you want to share notebooks using Delta Sharing or if you are a Databricks partner who uses personal staging locations.

See also Specify a managed storage location in Unity Catalog.

Requirements

• You must have at least one workspace attached to the Unity Catalog metastore.

• Databricks permissions required:

  • To create an external location, you must be a metastore admin or user with the CREATE EXTERNAL LOCATION and CREATE STORAGE CREDENTIAL privileges.

  • To add the storage location to the metastore definition, you must be an account admin.

• AWS permissions required: the ability to create S3 buckets, IAM roles, IAM policies, and cross-account trust relationships.

Step 1: Create the storage location

Follow the instructions in Step 1 (Optional): Create an S3 bucket for metastore-level managed storage in AWS to create a dedicated S3 bucket in an AWS account in the same region as your metastore.

Step 2: Create an external location in Unity Catalog

In this step, you create an external location in Unity Catalog that represents the bucket that you just created.

1. Open a workspace that is attached to the metastore.

2. Click Catalog to open Catalog Explorer.

3. Click the + Add button and select Add an external location.

4. On the Create a new external location dialog, click AWS Quickstart (Recommended) and click Next.

   The AWS Quickstart configures the external location and creates a storage credential for you. If you choose to use the Manual option, you must manually create an IAM role that gives access to the S3 bucket and create the storage credential in Databricks yourself.

5. On the Create external location with Quickstart dialog, enter the path to the S3 bucket in the Bucket Name field.

6. Click Generate new token to generate the personal access token that you will use to authenticate between Databricks and your AWS account.

7. Copy the token and click Launch in Quickstart.

8. In the AWS CloudFormation template that launches (labeled Quick create stack), paste the token into the Databricks Account Credentials field.

9. Accept the terms at the bottom of the page (I acknowledge that AWS CloudFormation might create IAM resources with custom names).

10. Click Create stack.

    It may take a few minutes for the CloudFormation template to finish creating the external location object in Databricks.

11. Return to your Databricks workspace and go to the External locations pane in Catalog Explorer.

    In the left pane of Catalog Explorer, scroll down and click External Data > External Locations.

12. Confirm that a new external location has been created.

    Automatically-generated external locations use the naming syntax db_s3_external_databricks-S3-ingest-<id>.

13. Grant yourself the CREATE MANAGED STORAGE privilege on the external location.

    1. Click the external location name to open the details pane.

    2. On the Permissions tab, click Grant.

    3. On the Grant on <external location> dialog, select yourself in the Principals field and select CREATE MANAGED STORAGE.

    4. Click Grant.

Step 3: Add the storage location to the metastore

After you have created an external location that represents the metastore storage bucket, you can add it to the metastore.

1. As an account admin, log in to the account console.

2. Click Catalog.

3. Click the metastore name.

4. Confirm that you are the Metastore Admin.

   If you are not, click Edit and assign yourself as the metastore admin. You can unassign yourself when you are done with this procedure.

5. On the Configuration tab, next to S3 bucket path, click Set.

6. On the Set metastore root dialog, enter the S3 bucket path that you used to create the external location, and click Update.

   You cannot modify this path once you set it.

Add a metastore admin

Metastore admins are optional, but there are situations where you might want one for your metastore. See Assign a metastore admin.

Delete a metastore

If you are closing your Databricks account or have another reason to delete access to data managed by your Unity Catalog metastore, you can delete the metastore.

Warning

All objects managed by the metastore will become inaccessible using Databricks workspaces. This action cannot be undone.

Managed table data and metadata will be auto-deleted after 30 days. External table data in your cloud storage is not affected by metastore deletion.

To delete a metastore:

1. As a metastore admin, log in to the account console.

2. Click Catalog.

3. Click the metastore name.

4. On the Configuration tab, click the three-button menu at the far upper right and select Delete.

5. On the confirmation dialog, enter the name of the metastore and click Delete.