Skip to main content

Resolve storage path conflicts

A storage path conflict occurs when an external location overlaps with the internal path Databricks uses for your workspace's default Unity Catalog storage. This overlap interferes with how Unity Catalog uses the workspace's default storage for internal processing in certain features and requires you to update your external location configuration.

How the conflict happens​

A conflict exists when you create an external location with a path that overlaps your workspace's default Unity Catalog storage path. The workspace default storage is specified at workspace creation time when you create the storage configuration object following the workspace creation guide. For example:

  • Default workspace storage path used to create workspace: s3://<your-bucket>/<region>/
  • Overlapping external location (conflict): s3://<your-bucket>/

This overlap prevents Unity Catalog from using the workspace default storage location for internal processing, and certain functionalities in Unity Catalog are blocked. This also weakens data governance because the external location exposes the workspace internal data inside the location. As a result, external location paths must not overlap with the workspace default Unity Catalog storage paths. Creation of an external location on more specific paths under workspace default storage like the DBFS root location continues to be allowed, because the DBFS root location is a sibling of the workspace's internal storage path and doesn't cause a conflict.

Identify your scenario​

Review your external locations in Catalog Explorer in the Databricks UI to see if your external location path includes or is broader than your workspace's default storage path.
Example:

  • External location: s3://<your-bucket>/
  • Workspace default storage path: s3://<your-bucket>/<region>/

Consider the following scenarios to help you determine how to proceed.

Scenario A: External location updates without moving data​

You define an external location at a broad path (such as s3://<customer-bucket>/) but access data in a more specific sibling folder, such as legacy Databricks Filesystem data stored at s3://<customer-bucket>/<region>/<workspace-id>/.

Action: (Recommended) Update your existing external location to the more specific path. This change resolves the conflict while continuing to allow access to all the data. For example, you would update the path from s3://<your-bucket>/ to s3://<your-bucket>/<region>/<workspace-id>/. This also prevents accidental data leakage of workspace internal data.

Scenario B: Managed tables stored at the root bucket location​

You define an external location at the root of your bucket (for example, s3://<your-bucket>/) and create managed tables directly under it, such as s3://<your-bucket>/__unity_storage/catalogs/<catalog_id>/tables/<table_id>.

In this case, updating the path is not possible and you must move the data.

Action: Open a support ticket with Databricks Support. The support team can help you migrate your managed data to a new location and restore your metastore to a non-conflicting state.

Last updated on