Migrate to automatic identity management with Okta

note

Automatic identity management requires the Premium plan or above or above.

This page describes how to migrate a Databricks account to automatic identity management with Okta.

For the steps to configure Okta, see Configure Okta for automatic identity management.

For an overview of automatic identity management, see Automatic identity management.

Prerequisites

Before enabling automatic identity management, confirm that your environment meets the following requirements:

• Single Okta tenant: Automatic identity management does not support cross-tenant identities. All users and groups must belong to a single Okta tenant. Environments with cross-tenant identities must remain on System for Cross-domain Identity Management (SCIM) provisioning.
• Identity federation enabled on at least one workspace: Automatic identity management requires at least one identity-federated workspace. Non-federated workspaces continue to function but do not use automatic identity management. To enable it, see Enable identity federation.
• Account admin role: Only account admins can enable automatic identity management.

Automatic identity management with Okta also requires that Okta single sign-on (SSO) is configured for your Databricks account using the same tenant. If SSO is not configured, or you need to use a different tenant, contact your Databricks account team. To configure Okta, see Configure Okta for automatic identity management.

What stays the same

When you enable automatic identity management, the following are preserved:

• SCIM-synced group memberships: Automatic identity management does not remove group memberships that were originally synced by SCIM. This is by design to avoid breaking jobs and permissions that depend on those memberships.
• Existing permissions: Workspace and Unity Catalog permissions continue to resolve against the same principals.
• Active sessions: Automatic identity management does not force user logouts or interrupt running sessions.
• Workspace-local groups: Automatic identity management only manages account-level identities. Workspace-local groups continue to work but are not synced from Okta.

Migrate without existing SCIM provisioning

Use this path if your Databricks account has no account-level SCIM provisioning. This is the simplest migration path.

1. Enable automatic identity management:

   Configure Okta and enable automatic identity management by following Configure Okta for automatic identity management. This creates an Okta app with the required roles and permissions and turns on automatic identity management. Allow 5 to 10 minutes for the change to propagate.

2. (Optional) Configure the account access denylist. Use the account access denylist to restrict specific Okta identities from accessing your Databricks account. See Deny identities access to your account.

3. (Optional) Notify workspace admins. Share the automatic identity management overview with workspace admins so they understand the expected behavior changes, including just-in-time provisioning.

After enabling automatic identity management, complete the validation steps.

Migrate from existing SCIM provisioning

Use this path if your Databricks account currently uses account-level SCIM provisioning. Databricks recommends enabling automatic identity management alongside your existing SCIM provisioning and running both in parallel. You can disable SCIM after validating that automatic identity management is working correctly.

What changes when you enable automatic identity management

Behavior	Before automatic identity management	After automatic identity management
Identity origin	SCIM pushes users and groups from Okta to the account SCIM endpoint on a schedule.	Databricks reads identities directly from Okta using the Okta Identity APIs. Provisioning is just-in-time (JIT) on first use or login.
Group renames in Okta	SCIM updates the display name on the next sync cycle.	Group names are not proactively re-synced. An account admin opening the group detail page in the account console triggers a refresh. You can also call the resolveByExternalId API to trigger a sync.
Group membership sync latency	Wait for the next SCIM cycle, typically 40 minutes.	Browser sign-in: up to 5 minutes. Non-browser (jobs, CLI, service principals): up to 40 minutes.

Behavior	Before automatic identity management	After automatic identity management
Identity origin	SCIM pushes users and groups from Okta to the account SCIM endpoint on a schedule.	Databricks reads identities directly from Okta using the Okta Identity APIs. Provisioning is just-in-time (JIT) on first use or login.
Group renames in Okta	SCIM updates the display name on the next sync cycle.	Group names are not proactively re-synced. An account admin opening the group detail page in the account console triggers a refresh. You can also call the resolveByExternalId API to trigger a sync.
Group membership sync latency	Wait for the next SCIM cycle, typically 40 minutes.	Browser sign-in: up to 5 minutes. Non-browser (jobs, CLI, service principals): up to 40 minutes.

Prepare for migration

Before you enable automatic identity management, configure the Okta connection. Follow Step 1: Set up an Okta application and grant the required role and permissions. Then, run Test connection from Step 2: Configure Okta in Databricks. Do not click Enable AIM until you complete the preparation steps below.

1. Run the Automatic identity management readiness report.

   The readiness report identifies external ID and group membership mismatches between Databricks and Okta.

   note

   Automatic identity management uses the Okta ID as the authoritative link for syncing identities. If a principal's externalId in Databricks does not match its Okta ID, identities can appear multiple times in the product and external ID-based sync can fail. Run the readiness report before enabling automatic identity management to identify and resolve these misconfigurations. For the full list of misconfigurations and their impact, see Error categories and remediation.

2. (Optional) Configure the account access denylist. See Deny identities access to your account.

3. (Optional) Notify workspace admins. Share the automatic identity management overview with workspace admins so they understand the expected behavior changes, including just-in-time provisioning.

Enable automatic identity management

Follow the instructions in Step 2: Configure Okta in Databricks to set the Okta org URL, Client ID, and Client Private key, run Test connection, and click Enable AIM. Allow 5 to 10 minutes for the change to propagate.

Automatic identity management and SCIM provisioning run in parallel while both are active. SCIM continues to manage identities that it originally provisioned. After you validate automatic identity management, you can disable SCIM. See Disable SCIM provisioning.

Validate automatic identity management

After enabling automatic identity management, verify that it is syncing identities correctly.

Test by role

Have a representative user from each role run the following tests.

Account admin

1. Provision a user from the account console:

   1. Go to Account console > User Management > Users and click Add user.
   2. Search for a user in Okta who has not been provisioned in Databricks.
   3. Select the user and click Add user.
   4. Confirm the user appears in Databricks with a status of Active.

2. Provision a group from the account console:

   1. Go to Account console > User Management > Groups and click Add group.
   2. Search for a group in Okta that has not been provisioned in Databricks.
   3. Confirm the group appears in Databricks. Group members are provisioned on a just-in-time basis when they log in.

3. Verify workspace access:

   1. Assign the group from the previous step to a workspace.
   2. Have a member of that group authenticate to the workspace.
   3. Confirm that authentication succeeds and that the user is provisioned in Databricks and added to the workspace.

Workspace admin

1. Provision a user from the workspace:

   1. Go to Settings > Identity and Access > Manage users and click Add user.
   2. Search for a user in Okta who has not been provisioned in Databricks.
   3. Confirm the user appears with a status of Active.

2. Provision a group from the workspace:

   1. Open a workspace asset such as a query or dashboard and click Share.
   2. Search for a group in Okta that has not been provisioned in Databricks and share the asset with that group.
   3. Confirm that the group is provisioned in Databricks.
   4. Have members of that group authenticate to the workspace and verify that they can access the shared asset.

Verify automatic identity management in audit logs

Query the system.access.audit table to confirm that automatic identity management is active. A non-zero count for events such as add, addPrincipalToGroup, createGroup, and updateUser confirms that identities are being synced.

SQL

SELECT action_name, count(*) AS n
FROM system.access.audit
WHERE request_params.endpoint = 'autoUserCreation'
  AND event_time > current_timestamp() - INTERVAL 2 DAYS
GROUP BY action_name
ORDER BY n DESC;

For more information on automatic identity management audit events, see Audit automatic identity management events.

Disable automatic identity management

For steps to disable automatic identity management, see Disable automatic identity management.

When you disable automatic identity management after migrating from SCIM:

• Identities created by automatic identity management remain in Databricks but are no longer synced with Okta.
• Group memberships that were synced by automatic identity management are no longer used to resolve permissions. Users who previously inherited permissions through those group memberships lose access.
• Users who authenticated to Databricks while automatic identity management was active can continue to access dashboards they previously viewed, but their group memberships are not refreshed from Okta.
• Users who have never accessed Databricks cannot access it after automatic identity management is disabled, even if they belong to an assigned group in Okta.

Databricks recommends setting up SCIM provisioning before disabling automatic identity management. See Configure SCIM provisioning for Okta.

Disable SCIM provisioning

After validating automatic identity management, you can optionally disable SCIM provisioning. Disable SCIM only after confirming that all of the following conditions are met in your environment:

• External ID alignment: Groups replicated from Okta using SCIM have externalId values in Databricks that match the corresponding Okta ID values. Run the Automatic identity management readiness report to identify any mismatches.
• No local membership modifications: Group memberships have been managed only through SCIM and have not been manually modified in Databricks. The readiness report can detect locally modified memberships.

If your environment does not meet all of these conditions, Databricks recommends continuing to run SCIM provisioning and automatic identity management in parallel.

Known limitations after disabling SCIM

Persistent SCIM-synced group memberships

Group memberships that were synced by SCIM persist after SCIM is disabled. Databricks recommends cleaning up direct group memberships from external groups before disabling SCIM.

Manually modified group memberships

If group memberships have been manually edited in Databricks, those memberships are not synchronized with Okta after you disable SCIM and may become stale over time. When automatic identity management is enabled, Okta is the source of truth for group memberships, so Databricks recommends removing any locally modified memberships. Run the Automatic identity management readiness report to identify these memberships before disabling SCIM.

Migration considerations

Group members are provisioned on a just-in-time basis

Unlike SCIM, automatic identity management does not proactively sync group memberships into Databricks. Group members are provisioned when they have activity in Databricks (for example, by logging in) or when an account admin or workspace admin explicitly adds them.

Deleted users are not immediately deactivated

When a user is deleted from Okta, they continue to appear as active in Databricks, but they can no longer log in.

Email address changes create new users

If a user's email address changes in Okta (for example, alice@example.com becomes alice-new@example.com), automatic identity management creates a new user in Databricks rather than updating the existing record. This is consistent with current SCIM behavior. For email address changes, contact Databricks Support.

SCIM API does not return members provisioned through automatic identity management

The SCIM GET /groups/{id} API returns only group members provisioned through the Databricks UI or SCIM. It does not return users who were provisioned through automatic identity management or users who exist in Okta but have not yet been provisioned in Databricks.

Unity Catalog permissions require provisioned identities

Unity Catalog permission-granting APIs cannot reference identities that have not yet been provisioned in Databricks. Before granting permissions to an identity, provision it using the resolveByExternalId API. After the identity is provisioned, you can grant permissions as usual.

Functions that reference a principal by name, such as is_account_group_member(), also require the principal to be provisioned first.

Workspace-local groups are not managed by automatic identity management

Automatic identity management does not manage workspace-local groups. Workspace-local groups and their assigned permissions continue to work, but membership updates must be handled outside of automatic identity management.

Databricks recommends migrating workspace-local groups to account-level groups so that automatic identity management can manage their memberships. See Migrate workspace-local groups to account groups.