Skip to main content
Last updated on

Resolve storage path conflicts

A storage path conflict occurs when an external location overlaps with the default Unity Catalog storage path of any workspace in your Databricks account. This overlap interferes with the internal processing of certain Unity Catalog features. If your account has a storage path conflict, you must update your external location configuration.

Unity Catalog validates external locations against every storage configuration registered to your Databricks account, not just in the workspace where the location was created. As a result, a conflict can arise if multiple workspaces in the same account share the same bucket.

How the conflict happens​

A conflict occurs when you create or update an external location that overlaps with the default Unity Catalog storage path of any workspace in your account. This default path is configured using the storage configuration object during classic workspace deployment. See Create a storage configuration. For example:

  • Default workspace storage path used to create workspace: s3://<your-bucket>/<region>/
  • Overlapping external location (conflict): s3://<your-bucket>/

This overlap prevents Unity Catalog from using the workspace storage location for internal processing, and it blocks certain Unity Catalog functionality. Having an overlap also weakens data governance because internal workspace data could be exposed by the external location. Consequently, external location paths must not overlap with any workspace default Unity Catalog storage path in your account.

You can, however, create an external location on more specific paths under a workspace default storage path, like the DBFS root location. This is because the DBFS root location is a sibling of the workspace's internal storage path and does not create a conflict.

Identify your scenario​

Review your external locations in Catalog Explorer in the Databricks UI to see if your external location path includes or is broader than your workspace's default storage path.
Example:

  • External location: s3://<your-bucket>/
  • Workspace default storage path: s3://<your-bucket>/<region>/

Consider the following scenarios to help you determine how to proceed.

Scenario A: External location updates without moving data​

You define an external location at a broad path (such as s3://<customer-bucket>/) but access data in a more specific sibling folder, such as legacy Databricks File System data stored at s3://<customer-bucket>/<region>/<workspace-id>/.

Action: (Recommended) Update your existing external location to the more specific path. This change resolves the conflict while continuing to allow access to all the data. For example, you would update the path from s3://<your-bucket>/ to s3://<your-bucket>/<region>/<workspace-id>/. This also prevents accidental data leakage of workspace internal data.

Scenario B: Managed tables stored at the root bucket location​

You define an external location at the root of your bucket (for example, s3://<your-bucket>/) and create managed tables directly under it, such as s3://<your-bucket>/__unity_storage/catalogs/<catalog_id>/tables/<table_id>.

In this case, updating the path is not possible. You must migrate your managed tables to a new location.

important

The following steps are a suggested procedure, not a complete or guaranteed migration plan. They don't account for every object type (for example, volumes or registered models) or every workload pattern. Use your own judgment to plan and validate the migration, and work with your Databricks account team on any issues specific to your environment.

Action: Complete the following steps to resolve the conflict:

  1. Identify all affected managed tables. As a metastore admin, query system.information_schema.tables to get a complete list of managed tables across all catalogs:

    SQL
    SELECT table_catalog, table_schema, table_name
    FROM system.information_schema.tables
    WHERE table_type = 'MANAGED';

  2. Choose a new metastore root subdirectory. Select a path that is not a parent of any existing internal storage paths (paths containing __unity_storage). For example: s3://<your-bucket>/uc-root/.

  3. Update the metastore managed storage location. See Add managed storage to an existing metastore. New managed tables are stored at the new location after this update. Existing managed tables are not moved automatically.

  4. Back up privilege grants on each affected table before migrating. Run this as the table owner or a metastore admin so that all grants are visible:

    SQL
    SHOW GRANTS ON TABLE <catalog>.<schema>.<table_name>;

  5. Stop all workloads that read from or write to the affected tables, and make sure no workloads or admins modify grants on them for the duration of the migration. Writes or grant changes that occur after you clone the tables are not captured and can result in data loss.

  6. Recreate each affected table at the new managed storage location. Use DEEP CLONE to preserve Delta Lake table history:

    SQL
    -- Recommended: deep clone preserves Delta history
    CREATE TABLE <catalog>.<schema>.<table_name>
    DEEP CLONE <old_catalog>.<schema>.<table_name>;

  7. Reapply all grants to the recreated tables:

    SQL
    GRANT <privilege> ON TABLE <catalog>.<schema>.<table_name> TO <principal>;

  8. Update all downstream workloads (jobs, notebooks, dashboards) to reference the recreated tables. Use Data lineage to identify dependent workloads before cutting over. Resume the workloads only after they point to the recreated tables.

  9. Verify the recreated tables, then drop the old tables.

  10. Update the external location to a more specific non-conflicting path, as described in Scenario A.

On this page