Manage identities in Unity Catalog

Preview

Unity Catalog is in Public Preview. To participate in the preview, contact your Databricks representative.

This article describes how to manage users, groups, and service principals in Databricks accounts that use Unity Catalog.

What is the identity management model for Unity Catalog?

In Unity Catalog, you manage identities at the account level, not the workspace level. This section describes the identity types available in Databricks and explains how the account-centric identity model works in Unity Catalog.

Account-level identities

There are three types of account-level identities:

  • Users: Account-level users can be either account admins or account users.

    Account admins can manage account-level configurations like workspace creation, network and storage configuration, audit logging, billing, and identity management. They can also manage Unity Catalog metastores and data access.

    Account admins can assign users to any workspaces that are assigned to a Unity Catalog metastore.

  • Service principals: Service principals are service users for use with automated tools. An account admin can assign service principals to workspaces that are assigned to a Unity Catalog metastore, where they can be used to run jobs and applications.

  • Groups: Groups simplify identity management, making it easier to assign access to workspaces, data, and other securable objects.

How do account admins assign users to workspaces?

Unity Catalog uses a federated identity model, enabling you to manage all of your users, service principals, and groups at the account level and assign workspace access to those account-level identities.

You can add users, service principals, and groups to your account using the account console UI, your IdP’s SCIM provisioning connector, or the account-level SCIM API. These users can be account admins, who have the ability to perform account-level administrative tasks (like adding users), or regular users, who don’t.

To enable an account-level identity to work in Databricks, you need to assign them to one or more workspaces. You can assign an account-level user to a workspace as a workspace user or as a workspace admin. Workspace admins have the ability to perform many administrative functions for that workspace (like controlling access to cluster creation or shared notebooks).

Conversely, if you add a user or service principal directly to a workspace that is assigned to a Unity Catalog metastore, that user or service principal will automatically get synced to an account-level identity. Databricks does not recommend this upstream flow; it’s more straightforward to add your users at the account level and then assign them to workspaces. Workspace groups are not synced automatically as account-level groups. If you want to replicate a workspace-local group as an account-level group, you must do so manually.

Account-level identity diagram

In order to assign an account-level user to a workspace, that workspace must be enabled for Unity Catalog. In other words, it has to be assigned to a Unity Catalog metastore. You don’t have to assign all of your workspaces to Unity Catalog metastores. For those workspaces that aren’t assigned to a Unity Catalog metastore, you manage your workspace users and workspace admins entirely within the scope of the workspace (the legacy model). You can’t use the account console or APIs to assign account-level users to these workspaces.

What happens when I enable an existing workspace for Unity Catalog?

When you enable an existing workspace for Unity Catalog, all of the existing workspace users and admins are synced automatically to the account as users, and you can start managing them (updating, removing, assigning admin permissions, assigning them to other workspaces) at the account level. If the workspace user shares a username (email address) with an account-level user or admin that already exists, those users are merged.

How does the Unity Catalog identity management model differ from accounts that don’t use Unity Catalog?

If you don’t use Unity Catalog, you cannot use account-level interfaces to assign users, service principals, and groups to a workspace. You have to add and manage your workspace users and groups entirely from within the workspace.

How does SCIM provisioning work differently for workspace- and account-level identities?

If you use account-level identities to manage users in workspaces (which requires that the workspace be assigned to a Unity Catalog metastore), you should turn off any SCIM provisioning that you have set up to add, update, and delete users, service principals, or groups in those workspaces. You should instead set up SCIM provisioning at the account level. Workspace-level SCIM provisioning in this case may result in errors for some operations (for example, provisioning new groups).

If your workspace is not identity-federated, you can continue to use workspace-level SCIM provisioning.

How do you manage SSO for workspace- and account-level identities?

SSO for account-level and workspace-level identities must be managed separately.

You should configure SSO identically at the account level and the workspace level. Preferably, use OIDC for SSO configuration at the account level. SAML SSO is also supported.

Requirements

  • You must be a Databricks account admin.

  • Your Databricks account must be on the Premium plan.

  • You must have a Unity Catalog metastore.

  • To manage account-level identities, you must be an account-level admin.

  • To sync identities from your identity provider (IdP), you must also have adequate permissions on your IdP.

Manage account-level identities manually

A Unity Catalog metastore can be shared across multiple Databricks workspaces. So that Databricks has a consistent view of users and groups across all workspaces, you can now create users and groups as account-level identities. Follow these steps to create account-level identities.

Note

To manually add a user:

  1. Log in to the account console (requires the user to be an account admin).

  2. Click Users Icon Users and Groups.

  3. To add a user:

    1. Click Users.

    2. Click Add User.

    3. Enter a name and email address for the user.

    4. Click Send Invite.

    To log into a workspace, a user must also be added to the workspace. To create workspace-level users, see Manage workspace-level users.

  4. To add a group:

    1. Click Groups.

    2. Click Add Group.

    3. Enter a name for the group.

    4. Click Confirm.

    5. When prompted, add users to the group.

Sync account-level identities from your IdP

You can sync account-level users and groups from your identity provider (IdP) to Unity Catalog, rather than manually adding users and groups.

Requirements

You must have an appropriate level of access to configure your IdP.

Configure Databricks

  1. Log in to the Databricks account console.

  2. Click User Settings Icon Settings.

  3. Click User Provisioning.

  4. Click Enable user provisioning.

    Make a note of the SCIM token and the account SCIM URL. You will use these to configure your IdP.

Configure your IdP

Follow these steps to enable your IdP to sync users and groups to your Databricks account. This configuration is separate from any configurations you have created to sync users and groups to workspaces.

  1. Log in to your IdP.

  2. Follow the instructions from Provision users and groups using SCIM. Select the linked article for your IdP.

    • For the SAML provisioning URL, enter the SCIM url from the previous task.

    • For the provisioning API token, enter the SCIM token from the previous task.

Rotate the SCIM token

If the account-level SCIM token is compromised or if you have business requirements to rotate authentication tokens periodically, you can rotate the SCIM token.

  1. Log in to the Databricks account console.

  2. Click User Settings Icon Settings.

  3. Click User Provisioning.

  4. Click Regenerate token. Make a note of the new token. The previous token will continue to work for 24 hours.

  5. Within 24 hours, update your IdP’s configuration to use the new SCIM token.

    1. Log in to your IdP.

    2. Follow the instructions from Provision users and groups using SCIM. Select the linked article for your IdP.

      For the provisioning API token, enter the SCIM token you just entered.