Skip to main content

Manage, fail over, and troubleshoot managed disaster recovery

This page describes how to operate managed disaster recovery (DR) after you create a failover group: monitor replication, fail over and fail back, keep infrastructure-as-code deployments pointed at the current primary, tear down managed DR, and resolve replication errors. To create a failover group, see Set up managed disaster recovery.

Monitor replication​

In the account console, click Resilience > Failover groups. The tab shows each failover group's current state, replication point, and whether it has active errors.

State

Meaning

CREATING

The failover group is being provisioned.

INITIAL_REPLICATION

The failover group does not have a replication point yet. Failover and failback are not available. See Initial replication.

ACTIVE

Replication is in steady state. Failover is available.

FAILING_OVER

A failover is in progress.

FAILOVER_FAILED, CREATION_FAILED, DELETION_FAILED

The operation did not complete. Select the failover group's name to see its status details. To see the errors that block replication, query the system.replication.states system table. See Replication system table reference.

State

Meaning

CREATING

The failover group is being provisioned.

INITIAL_REPLICATION

The failover group does not have a replication point yet. Failover and failback are not available. See Initial replication.

ACTIVE

Replication is in steady state. Failover is available.

FAILING_OVER

A failover is in progress.

FAILOVER_FAILED, CREATION_FAILED, DELETION_FAILED

The operation did not complete. Select the failover group's name to see its status details. To see the errors that block replication, query the system.replication.states system table. See Replication system table reference.

Select a failover group's name to open its detail page. Replication runs continuously, but the replication point shows the last time all in-scope resources were copied together. Individual resources might be more current, but not all data written after the replication point is guaranteed to exist in the secondary, and that data can be lost during failover.

When a replication point is shown, everything not covered by an error has been replicated up to that point. Asset types that managed DR does not support at all do not appear in the system.replication.states system table. See Limitations. To verify a specific asset, inspect it from a separate read-only monitor workspace in the secondary region. The secondary workspace can't run compute while managed DR is enabled. See Wait for initial replication.

To monitor historical recovery point objective (RPO) trends and see the errors that block replication, query the system.replication.states system table. See Replication system table reference. To resolve an error, see Troubleshoot replication errors.

Initial replication​

A failover group is in the INITIAL_REPLICATION state until it has its first replication point. This happens in the following cases:

  • After you create the failover group.
  • After a failover, when replication starts in the reverse direction.
  • After you add a catalog to the failover group's replication scope.

During initial replication, failover is not available, and the failover group reports its progress as the percentage of in-scope assets replicated so far. Errors that block the first replication point appear in the system.replication.states system table. When the first replication point exists, the failover group transitions to ACTIVE.

Fail over and fail back​

The same procedure covers planned failovers (DR tests, scheduled maintenance) and unplanned failovers (a regional outage). To fail back, repeat the procedure with the regions reversed.

When you trigger a failover, Databricks:

  • Points the stable URL, if attached, to the new primary region.
  • Reverses the direction of replication.
  • Pauses job schedules in the former primary.
  • Transitions the failover group through FAILING_OVER to INITIAL_REPLICATION. You can't fail back until the failover group returns to ACTIVE. See Initial replication.

To fail over:

  1. Notify your team that a failover is starting.

  2. For a planned failover only:

    1. In the primary workspace, terminate all running clusters and stop all SQL warehouses.
    2. Confirm that writes to the primary have stopped, then wait for replication to catch up. To check, open the failover group's detail page and confirm the replication point is within a few seconds of the time you stopped writes.
  3. In the account console, click Resilience > Failover groups, then click the failover group's name.

  4. Click Fail over.

  5. Select the new primary region and confirm. The failover completes in minutes.

  6. In the new primary, start the compute that was running before the failover. Replicated clusters and SQL warehouses arrive in the new primary in TERMINATED and STOPPED state, respectively.

  7. Resume the job schedules you need in the new primary.

Clients connected through the stable URL continue working after the failover. Repoint clients that still use the original workspace URL to either the stable URL or the new primary's workspace URL.

important

In an unplanned failover, data written to the primary after the last replication point might be lost. Confirm that any loss falls within your RPO target.

tip

Test failover regularly, such as every quarter, so your team is familiar with the procedure before an actual outage.

Deploy with Declarative Automation Bundles and Terraform​

Declarative Automation Bundles and the Databricks Terraform provider target a workspace either by its workspace host URL, or by a combination of the custom URL of the Databricks account and the workspace ID. To keep deploying to the current primary after a failover, set the host to your custom URL, which is the host portion of the stable URL, and set the workspace_id field to the stable workspace ID. Together they resolve to the current primary, so CI/CD pipelines need no configuration change after a failover.

  • New deployments: Use the custom URL and stable workspace ID from the first deployment.
  • Existing deployments: Import the state from your previous Terraform project into a new project configured with the custom URL and stable workspace ID, then remove the previous project. Do not repoint an existing project in place. The deployment no longer recognizes the resources it created against the original per-workspace URL, so a redeploy destroys and recreates them.
  • Bundles: Turn on Replicate workspace assets on the failover group. A bundle stores its deployment state in the workspace, and that state reaches the new primary only as part of workspace asset replication.
note

After a failover, the first redeploy recreates any resources that managed DR does not replicate, because they do not exist on the new primary. Replicated resources are left in place. See Limitations.

Stable workspace ID​

Each stable URL has a stable workspace ID that resolves to the current primary. Use it wherever a tool asks for a workspace ID, in the same way you would use a regular workspace ID.

To find the stable workspace ID, in the account console, click Resilience, then click the Stable URLs tab. The stable workspace ID is listed with each stable URL.

You can also list the stable URLs for your account with the Databricks CLI and read the stable_workspace_id field of the relevant stable URL:

Bash
databricks api get /api/disaster-recovery/v1/accounts/<account-id>/stable-urls

Tear down managed DR​

  1. In the account console, click Resilience > Failover groups, then click the failover group's name and delete it. You cannot turn off Mission Critical while a failover group is active on the workspace.
  2. To stop billing at the Mission Critical rate, turn off Mission Critical on each workspace from the Add-ons tab.

Troubleshoot replication errors​

When a resource cannot be replicated, the failover group surfaces an error class in the system.replication.states system table, along with a message identifying the affected resource. The following sections cover the most common error classes and how to resolve them. Replication recovers automatically after you correct the underlying problem.

DR_MISSING_DEPENDENCY

An asset references a dependency that does not exist in the secondary, so the asset cannot be replicated. The subclass identifies the missing dependency type and appears as DR_MISSING_DEPENDENCY.CATALOG, .SCHEMA, .TABLE, or .RESOURCE. The resolution is the same for all of them.

  1. Check whether the asset is also broken in the primary because of the missing dependency. If it is, fix or remove the asset in the primary.
  2. If the asset is valid in the primary, the dependency is either not in the replication scope of any failover group, or it is in scope of this or another failover group but failed to replicate. If the dependency is not in scope, edit the failover group's replication scope to also replicate it. If the dependency is already in scope, check system.replication.states for the error blocking its replication and resolve that error.

DR_INVALID_CONFIGURATION.MISSING_LOCATION_MAPPING

No storage mapping covers a source storage location, so managed DR cannot determine where in the secondary to place the asset. For external tables and volumes, a missing mapping means the same location URI is used on the primary and secondary. The storage_location in the message is the unmapped source path.

  1. In the account console, go to Resilience > Failover groups and edit the failover group.
  2. Under Storage mappings, add or widen a mapping so it covers the source location in the message. To cover child paths, map a parent path and add the /* suffix.
  3. Confirm that an external location in the secondary metastore covers the mapping's target path. If it doesn't, create it first.

For the mapping rules, see Plan your storage mappings.

DR_INVALID_CONFIGURATION.MISSING_EXTERNAL_LOCATION

A storage mapping resolved a replicated asset to a target path in the secondary, but no external location in the secondary metastore covers that path, so Unity Catalog has nowhere to place the asset's data. The storage_location in the message is the uncovered secondary (target) path.

This usually means one of two things: an external location that previously covered the path was removed or narrowed, or a newly replicated asset resolves to a secondary path that no external location covers. The second case happens, for example, when you create an external table in the primary under a storage path that none of your storage mappings cover. Managed DR then falls back to the table's original path, which no external location in the secondary metastore covers.

  1. Identify the uncovered secondary path from the message's storage_location.
  2. Decide which external location in the secondary metastore should cover that path: an existing external location that you extend, or a new one that you create.
  3. Either adjust the failover group's storage mappings so the path resolves under an external location that already exists, or create the external location (with its storage credential) and extend the mapping to point to it. See Configure storage.

DR_INTERNAL_ERROR

A system-side fault occurred during replication. No action is required. The system recovers automatically. Contact Databricks support if the problem does not resolve on its own.

DR_INVALID_CONFIGURATION.CROSS_CATALOG_VIEW_PERMISSION

Managed DR replicates the view along with its grants, but a view that references objects in other catalogs also needs its owner to have access to those referenced objects in the secondary, because the view runs with the owner's privileges. This error means the owner is missing that access on the secondary, so you must grant it on the referenced objects there.

  1. Find the objects the view references and the view's owner. Referenced objects appear as fully qualified catalog.schema.object names in the definition. Grants must go to the owner, which you can also read from the Owner field in Catalog Explorer.

    SQL
    SHOW CREATE TABLE <catalog>.<schema>.<view>;
  2. On the secondary, check the owner's current privileges on each referenced object. Reading a table requires USE CATALOG on its catalog, USE SCHEMA on its schema, and SELECT on the table.

    SQL
    SHOW GRANTS `<view_owner>` ON CATALOG <ref_catalog>;
    SHOW GRANTS `<view_owner>` ON SCHEMA <ref_catalog>.<ref_schema>;
    SHOW GRANTS `<view_owner>` ON TABLE <ref_catalog>.<ref_schema>.<ref_table>;
  3. Grant the view owner any missing privileges on each referenced object.

    SQL
    GRANT USE CATALOG ON CATALOG <ref_catalog> TO `<view_owner>`;
    GRANT USE SCHEMA ON SCHEMA <ref_catalog>.<ref_schema> TO `<view_owner>`;
    GRANT SELECT ON TABLE <ref_catalog>.<ref_schema>.<ref_table> TO `<view_owner>`;
  4. Confirm that every catalog the view references is included in a failover group's replication scope, so it also exists in the secondary.

See Manage privileges in Unity Catalog.

DR_INVALID_CONFIGURATION.NETWORK_UNAUTHORIZED_ACCESS

During cross-region table-data replication, the secondary workspace's serverless compute reads data from the source storage, and the storage denied the network connection. A storage firewall or network rule blocked it, or a required private endpoint is missing or unapproved.

Verify that your source and secondary storage allow Databricks serverless network access in both directions. See Configure network access.

DR_INVALID_CONFIGURATION.STORAGE_COPY_CREDENTIAL

Managed DR could not generate a storage credential to replicate data files. Unity Catalog can issue that credential only when, in each metastore, the storage credentials assigned to the primary and secondary external locations reference the same AWS Identity and Access Management (IAM) role. This error means they do not.

  1. Confirm that both the primary and secondary S3 buckets are registered as external locations in both metastores. If one is missing, create it.
  2. In each metastore, confirm that the storage credentials on both external locations reference the same IAM role. The two credentials can differ, as long as they point to one role.

See Configure storage.

DR_INVALID_CONFIGURATION.SERVERLESS_COMPUTE_PERMISSION

Managed DR uses serverless compute in the secondary workspace to copy data, and serverless compute is not permitted there. This usually means serverless is turned off for the account or workspace, or the workspace is not eligible.

  1. Confirm the secondary workspace is eligible. Serverless compute is available by default in Unity Catalog-enabled workspaces in a supported region. See Connect to serverless compute.
  2. Check for an account-wide opt-out. In the account console, go to Settings > Feature enablement and check whether the serverless toggle is present and off.
  3. Enable serverless for the scope you need. To enable every eligible workspace, an account admin turns the account-level serverless toggle on. To enable only the secondary workspace, leave the account-level toggle off and have a workspace admin enable serverless from the workspace's Previews.
  4. If no toggle is available, or serverless still does not run after you enable it, contact your Databricks account team.

DR_INVALID_CONFIGURATION.OVERLAPPING_EXTERNAL_LOCATIONS

When managed DR creates a replicated object at its mapped target path in the secondary, Unity Catalog rejects the path because it overlaps storage that already exists there, such as an existing external location, a leftover securable from an earlier or partial setup, a managed location, or the workspace's default (DBFS) storage.

  1. Identify what occupies the path.

    See Resolve storage path conflicts.

  2. If the conflicting object should not own the path, remove it. A common cause is a leftover external table or external volume from a previous setup. Drop it if it is no longer needed. If it is an external location that should not cover the path, remove or redefine it.

  3. Otherwise, repoint the failover group's storage mapping to a dedicated, non-overlapping target path. Prefer a specific subpath over a broad bucket root, and avoid the workspace's default (DBFS) storage.

DR_UNSUPPORTED_FEATURE

The asset uses a feature that managed DR cannot replicate. The subclass identifies the unsupported feature and appears, for example, as DR_UNSUPPORTED_FEATURE.ABAC_POLICY. There are two ways to resolve this error.

  1. Remove the unsupported feature from the asset in the primary workspace.
  2. If you cannot remove the feature, consider removing the asset from the failover group's replication scope.

DR_UNSUPPORTED_FEATURE.CROSS_CATALOG_POLICY_FUNCTION

A row filter, column mask, or attribute-based access control (ABAC) policy uses a function from a different catalog than the table or policy that uses it. Managed DR cannot replicate it, even when both catalogs are in the failover group. Granting privileges in the secondary does not resolve this error. The message names the affected asset and the function.

  1. In the primary workspace, create the function in the same catalog as the table, or as the securable the ABAC policy is attached to.

  2. Point the row filter, column mask, or policy at the new function:

    SQL
    -- Row filter
    ALTER TABLE <table_name> SET ROW FILTER <catalog>.<schema>.<function_name> ON (<column_name>, ...);

    -- Column mask
    ALTER TABLE <table_name> ALTER COLUMN <column_name> SET MASK <catalog>.<schema>.<function_name>;

    For ABAC policies, run CREATE OR REPLACE POLICY with the same name and securable, and specify every clause with the new function. See Edit a policy.

  3. If you cannot move the function, remove the asset from the failover group's replication scope.

Replication recovers on the next cycle after the asset uses a function from its own catalog.

DR_UNSUPPORTED_FEATURE.STORAGE_PRIVATE_ENDPOINT

The S3 bucket that holds your source or secondary data is behind Private Link, and replication over Private Link is not enabled for your account. The message names the affected table and bucket.

Contact your Databricks account team. See Configure network access.

DR_UNSUPPORTED_FEATURE.TABLE_ARCHIVED_FILES

The table has Delta Lake archival support enabled (delta.timeUntilArchived). Managed DR clones every data file, so archived objects must be restored to a hot storage tier and the table property must be unset on the primary before replication can continue. See Archival support in Databricks.

  1. Disable the cloud lifecycle archival policy on the table's storage location so additional files are not archived after you list them.

  2. List files that need restore:

    SQL
    SHOW ARCHIVED FILES FOR <table_name>;
  3. Restore those objects in your cloud provider, then wait until restore completes. See Restore archived files.

  4. Unset the delta.timeUntilArchived table property. This only updates metadata so Databricks no longer tracks archival on the table:

    SQL
    ALTER TABLE <table_name> UNSET TBLPROPERTIES ('delta.timeUntilArchived');
  5. Confirm a full read succeeds, for example SELECT COUNT(*) FROM <table_name>;. After that, replication can clone the table.

Unsetting the property does not restore objects. Restoring objects is a cloud provider operation that you must do manually.