Manage Unity Catalog metastores

This article shows how to update, delete, and manage the behavior of Unity Catalog metastores in your Databricks account.

To learn about Unity Catalog metastores and how to create them, see Create a Unity Catalog metastore.

Enable a metastore to be automatically assigned to new workspaces

To assign an existing Unity Catalog metastore automatically to new workspaces in that metastore’s region, an account admin can enable workspace auto-assignment for the metastore. If this setting is not selected, the admin who creates a workspace in the same region as the metastore must manually enable the workspace for Unity Catalog and select the metastore from a drop-down.

Before an account admin enables this option, they should be sure to understand the following impacts on new workspaces:

To enable automatic assignment:

  1. As an account admin, go to the Databricks account console.

  2. Click Catalog icon Catalog.

  3. Select your metastore.

  4. On the Configuration tab, under Workspace assignment, select Automatically assign new workspaces in <region> to this metastore.

  5. On the confirmation dialog, click Enable auto-assignment.

Add managed storage to an existing metastore

Metastore-level managed storage is optional, and it is not included for metastores that were created automatically. You might want to add metastore-level storage to your metastore if you prefer a data isolation model that stores data centrally for multiple workspaces. You need metastore-level storage if you are a Databricks partner who uses personal staging locations.

See also Specify a managed storage location in Unity Catalog.

Requirements

  • You must have at least one workspace attached to the Unity Catalog metastore.

  • Databricks permissions required:

    • To create an external location, you must be a metastore admin or user with the CREATE EXTERNAL LOCATION and CREATE STORAGE CREDENTIAL privileges.

    • To add the storage location to the metastore definition, you must be an account admin.

  • AWS permissions required: the ability to create S3 buckets, IAM roles, IAM policies, and cross-account trust relationships.

Step 1: Create the storage location

Follow the instructions in Step 1 (Optional): Create an S3 bucket for metastore-level managed storage in AWS to create a dedicated S3 bucket in an AWS account in the same region as your metastore.

Step 2: Create an external location in Unity Catalog

In this step, you create an external location in Unity Catalog that represents the bucket that you just created.

  1. Open a workspace that is attached to the metastore.

  2. Click Catalog icon Catalog to open Catalog Explorer.

  3. Click the + Add button and select Add an external location.

  4. On the Create a new external location dialog, click AWS Quickstart (Recommended) and click Next.

    The AWS Quickstart configures the external location and creates a storage credential for you. If you choose to use the Manual option, you must manually create an IAM role that gives access to the S3 bucket and create the storage credential in Databricks yourself.

  5. On the Create external location with Quickstart dialog, enter the path to the S3 bucket in the Bucket Name field.

  6. Click Generate new token to generate the personal access token that you will use to authenticate between Databricks and your AWS account.

  7. Copy the token and click Launch in Quickstart.

  8. In the AWS CloudFormation template that launches (labeled Quick create stack), paste the token into the Databricks Account Credentials field.

  9. Accept the terms at the bottom of the page (I acknowledge that AWS CloudFormation might create IAM resources with custom names).

  10. Click Create stack.

    It may take a few minutes for the CloudFormation template to finish creating the external location object in Databricks.

  11. Return to your Databricks workspace and go to the External locations pane in Catalog Explorer.

    In the left pane of Catalog Explorer, scroll down and click External Data > External Locations.

  12. Confirm that a new external location has been created.

    Automatically-generated external locations use the naming syntax db_s3_external_databricks-S3-ingest-<id>.

  13. Grant yourself the CREATE MANAGED STORAGE privilege on the external location.

    1. Click the external location name to open the details pane.

    2. On the Permissions tab, click Grant.

    3. On the Grant on <external location> dialog, select yourself in the Principals field and select CREATE MANAGED STORAGE.

    4. Click Grant.

Step 3: Add the storage location to the metastore

After you have created an external location that represents the metastore storage bucket, you can add it to the metastore.

  1. As an account admin, log in to the account console.

  2. Click Catalog icon Catalog.

  3. Click the metastore name.

  4. Confirm that you are the Metastore Admin.

    If you are not, click Edit and assign yourself as the metastore admin. You can unassign yourself when you are done with this procedure.

  5. On the Configuration tab, next to S3 bucket path, click Set.

  6. On the Set metastore root dialog, enter the S3 bucket path that you used to create the external location, and click Update.

    You cannot modify this path once you set it, but you can remove it and add a new path if necessary.

Remove metastore-level storage

If you have metastore-level storage for managed tables and volumes (also known as the metastore storage root), but you want to enforce data storage isolation at the catalog or schema level, you can remove the metastore-level storage option for the metastore. When you do, the following happens:

  • Existing catalogs that have no storage root specified are given the metastore storage root’s cloud storage location as their catalog-level managed storage location. In other words, the metastore storage root is “pushed down” to these catalogs. Access to data in these catalogs continues to function without interruption.

  • Depending on how your metastore was created, there might not be an external location securable defined in Unity Catalog for the metastore storage root. In that case, a new external location and associated storage credential are created for it. The new external location is named prior_metastore_root_location by default.

  • Every time a user creates a catalog, they must provide a dedicated storage location that is registered in Unity Catalog as an external location.

Note

If you use Delta Sharing to share notebooks and you used the metastore root as shared notebook storage, you must do the following before you can remove the metastore root:

  1. Remove your notebook from the share.

  2. Re-add the notebook using a dedicated storage location.

See Add notebook files to a share.

To remove the metastore storage root:

  1. As an account admin, log in to the account console.

  2. Click Catalog icon Catalog.

  3. Click the metastore name.

  4. On the Configuration tab, under S3 bucket path, click the Remove button.

  5. On the confirmation dialog, click Remove.

Add a metastore admin

Metastore admins are optional, but there are situations where you might want one for your metastore. See Assign a metastore admin.

Delete a metastore

If you are closing your Databricks account or have another reason to delete access to data managed by your Unity Catalog metastore, you can delete the metastore.

Warning

All objects managed by the metastore will become inaccessible using Databricks workspaces. This action cannot be undone.

Managed table data and metadata will be auto-deleted after 30 days. External table data in your cloud storage is not affected by metastore deletion.

To delete a metastore:

  1. As a metastore admin, log in to the account console.

  2. Click Catalog icon Catalog.

  3. Click the metastore name.

  4. On the Configuration tab, click the three-button menu at the far upper right and select Delete.

  5. On the confirmation dialog, enter the name of the metastore and click Delete.