Skip to main content
Last updated on

Automatic identity management readiness report

The automatic identity management readiness report helps account admins identify and resolve external ID and group membership divergences between Databricks and your identity provider (IdP). Because automatic identity management only functions within identity-federated workspaces, Databricks recommends enabling identity federation across all workspaces in the account.

For an overview of automatic identity management, see Automatic identity management. To migrate an existing account, see Migrate to automatic identity management with Microsoft Entra ID.

About divergences​

Automatic identity management requires an identity-federated workspace. It uses an externalId to link Databricks users and groups directly to your identity provider, specifically the object ID in Microsoft Entra ID or the user ID or group ID in Okta.

Missing or incorrect externalId values break metadata syncs and create duplicate identities. Because Databricks group memberships are editable, they can also drift out of sync with your identity provider and cause complications if SCIM provisioning is turned off.

note

The readiness report is only available to account admins.

Issues the report detects​

The report helps you discover identities provisioned in Databricks whose external IDs don't have a corresponding match in your identity provider. It also detects divergences between Databricks and identity provider group memberships. The following examples show issues that the report helps you detect and resolve.

Duplicate identities appear in the product

When two identities with the same name but different sources appear in the Databricks admin UIs, a misconfigured external ID is usually the cause. One account identity and one identity provider identity both appear in admin UIs and sharing dialogs.

To resolve this issue, fix the following error categories:

  • AIM_MISCONFIGURATION_CATEGORY_EXTERNAL_ID_NOT_IN_IDP
  • AIM_MISCONFIGURATION_CATEGORY_EXTERNAL_ID_MATCH_NAME_MISMATCH
  • AIM_MISCONFIGURATION_CATEGORY_NAME_MATCH_EXTERNAL_ID_MISMATCH

The group member count in the identity provider doesn't match the count in Databricks

The Databricks UIs show the member count from the identity provider. The count excludes members that exist in the Databricks group but not in the identity provider, even though those memberships still grant permissions.

To resolve this issue, fix the following error category:

  • AIM_MISCONFIGURATION_CATEGORY_GROUP_HAS_LOCAL_MEMBERS

Provisioning an identity provider group fails

When you import an identity provider group, you might get an error that the group already exists in Databricks. An existing account group usually reserves the name, because Databricks enforces unique group names.

To resolve this issue, fix the following error category:

  • AIM_MISCONFIGURATION_CATEGORY_NAME_MATCH_EXTERNAL_ID_MISMATCH

Run the readiness report​

  1. Log in to the Databricks account console as an account admin.
  2. Go to Security > Identity provider setup.
  3. Follow the automatic identity management setup steps for Microsoft Entra ID or Okta. See Configure Microsoft Entra ID for automatic identity management or Configure Okta for automatic identity management.
  4. Before you enable automatic identity management, start an identity provider misconfigurations scan.
  5. Wait for the scan to finish processing. This can take from a few minutes to several hours, depending on the size of your account. Databricks recommends enabling automatic identity management after you confirm that critical issues from the scan are resolved. You can also enable automatic identity management and view scan results afterward.
  6. When the scan is complete, open the scan results in the same section.

From this point on, you can view scan results and start a new scan in the Identity management section.

After a scan completes, you can request a new scan. To recheck a single principal, click Refresh on the principal. After you resolve the issue, the principal no longer appears in the misconfigurations list.

You can also download a CSV of the entire scan results or of a single category's results. See Interpret the report output for how to read the CSV.

Interpret the report output​

The CSV export contains the following columns:

  • principalId: The Databricks internal ID.
  • principalType: The type of principal for the principal ID. This can be a user, group, or service principal.
  • category: The error category for the principal ID. See Error categories and remediation for more information.
  • localPrincipalExternalId: The external ID stored in Databricks for the provisioned Databricks principal.
  • localPrincipalName: The username, application ID, or group name stored in Databricks for the provisioned Databricks principal.
  • idpPrincipalExternalId: The external ID stored in the identity provider for a given principal.
  • idpPrincipalName: The username, application ID, or group name stored in the identity provider for a given principal.
  • localMembersWithExternalId: Semicolon-separated internal IDs of group members that have an external ID but are not members in the identity provider group.
  • localMembersWithoutExternalId: Semicolon-separated internal IDs of group members that exist only in Databricks and have no external ID.
  • hasMoreLocalMembersWithExternalId: Whether there are more localMembersWithExternalId members than are provided in the CSV.
  • hasMoreLocalMembersWithoutExternalId: Whether there are more localMembersWithoutExternalId members than are provided in the CSV.
  • failureReason: The reason the principal's scan failed.

Error categories and remediation​

The report groups issues into the error categories below. Each category explains what the error means and how to resolve it. Fix each affected principal using the resolution for its category.

AIM_MISCONFIGURATION_CATEGORY_EXTERNAL_ID_NOT_IN_IDP

The provisioned identity has an external ID set, but it doesn't match any identity of the same type in the identity provider. The externalId is misconfigured.

To resolve:

  1. Update the externalId to a valid value, or remove it. If you set a new externalId, make sure no other identity uses it. See Update the external ID for a principal or Remove the external ID for a group.
  2. To determine which externalId to use, see the AIM_MISCONFIGURATION_CATEGORY_NAME_MATCH_EXTERNAL_ID_MISMATCH category.

If unresolved: If the externalId should link to an identity provider identity, you might see duplicate identities: one with an incorrect externalId and one from the identity provider.

AIM_MISCONFIGURATION_CATEGORY_EXTERNAL_ID_MATCH_NAME_MISMATCH

The Databricks identity has an external ID that maps to an identity with a different unique name in the identity provider.

To resolve:

  • Users and service principals: Update the Databricks username. File a support ticket to request the change.
  • Groups: Check whether any account groups reserve the group name (Databricks enforces unique group names). If so, rename the account group so the external group can claim the name.

If unresolved: When users log in, this frequently creates a second user with the same externalId but a different username. For groups, the external group often can't sync its name with its identity provider counterpart.

AIM_MISCONFIGURATION_CATEGORY_NAME_MATCH_EXTERNAL_ID_MISMATCH

The Databricks identity has a unique name match with an identity provider identity, but its externalId doesn't match.

To resolve, in most cases, update the Databricks externalId to match the identity provider identity. Confirm this is correct for your setup, because it can vary based on your identity provider and local data. Use the idpPrincipalExternalId field to find the externalId to use, then follow Update the external ID for a principal.

If unresolved: You might see duplicate identities: one with an incorrect or missing externalId and one from the identity provider. Provisioning an identity provider group with the same name might also fail, because an account group already uses the name (Databricks enforces unique group names).

AIM_MISCONFIGURATION_CATEGORY_GROUP_HAS_LOCAL_MEMBERS

The Databricks group has members, with or without an externalId, that have no corresponding membership in the identity provider.

To resolve, make the identity provider the source of truth: Databricks recommends removing locally added members from the group using SCIM. If a member belongs in the group, create the member in the identity provider and add it to the identity provider group. See the localMembersWithExternalId and localMembersWithoutExternalId fields in the output for the affected members.

If unresolved: Members inherit permissions from the identity provider group but don't appear in the identity provider, which makes auditing permissions difficult. Member counts in the UI reflect only identity provider members, so they exclude these members.

Update the external ID for a principal​

To update the externalId, use the Account SCIM API. For the full request format, see Account SCIM v2.1 API reference.

tip

Log the API calls that you make so that you can roll back changes if any problems come up during the process.

Send a PATCH request to the endpoint for the principal type:

Text
PATCH https://<account-console-url>/api/2.1/accounts/<account-id>/scim/v2/<Users|Groups|ServicePrincipals>/<databricks-id>
JSON
{
  "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
  "Operations": [
    {
      "op": "replace",
      "path": "externalId",
      "value": "<new-external-id>"
    }
  ]
}

Remove the external ID for a group​

To remove the externalId for a group, use the Account SCIM API. Set the value to an empty string.

note

Databricks currently supports this operation for groups only.

Text
PATCH https://<account-console-url>/api/2.1/accounts/<account-id>/scim/v2/Groups/<databricks-id>
JSON
{
  "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
  "Operations": [
    {
      "op": "replace",
      "path": "externalId",
      "value": ""
    }
  ]
}

Additional resources​

On this page