Skip to main content
Last updated on

Create service credentials

This article describes how to create a service credential object in Unity Catalog that lets you govern access from Databricks to external cloud services like AWS Glue or AWS Secrets Manager. A service credential in Unity Catalog encapsulates a long-term cloud credential that grants access to such services.

Service credentials are not intended for governing access to cloud storage that is used as a Unity Catalog managed storage location or external storage location. For those use cases, use a storage credential. See Connect to cloud object storage using Unity Catalog.

note

Service credentials are the Unity Catalog alternative to instance profiles, with the advantage that access is not tied to a specific compute resource but instead to users, groups, or service principals.

To create a service credential for access to AWS services, you create an IAM role that authorizes access to the service and reference that IAM role in the service credential definition.

Requirements​

Before you create a service credential, you must meet the following requirements:

In Databricks:

  • Databricks workspace enabled for Unity Catalog.

  • CREATE SERVICE CREDENTIAL privilege on the Unity Catalog metastore attached to the workspace. Account admins and metastore admins have this privilege by default. If your workspace was enabled for Unity Catalog automatically, workspace admins also have this privilege.

In your AWS account:

  • An AWS service in the same region as the workspaces you want to access the data from.
  • The ability to create IAM roles.

Create a service credential that references an AWS IAM role​

This section describes how to:

  • Create an IAM role that meets Databricks requirements for accessing an AWS service.
  • How to create a service credential securable object in Unity Catalog that can be used to access the AWS service from Databricks.

Step 1: Create an IAM role​

In AWS, create an IAM role that gives access to the service that you want your users to access. This IAM role must be defined in the same account as the service.

tip

If you have already created an IAM role that provides this access, you can skip this step and go straight to Step 2: Give Databricks the IAM role details.

  1. Create an IAM role that will allow access to the service.

    Role creation is a two-step process. In this step you create the role, adding a temporary trust relationship policy and a placeholder external ID that you then modify after creating the service credential in Databricks.

    You must modify the trust policy after you create the role because your role must be self-assuming (that is, it must be configured to trust itself). The role must therefore exist before you add the self-assumption statement. For information about self-assuming roles, see this Amazon blog article. For information about the Databricks self-assuming role enforcement policy, see Self-assuming role enforcement policy.

    To create the policy, you must use a placeholder external ID.

    1. Create the IAM role with a Custom Trust Policy.

    2. In the Custom Trust Policy field, paste the following policy JSON.

      This policy establishes a cross-account trust relationship so that Unity Catalog can assume the role to access service on behalf of Databricks users. This is specified by the ARN in the Principal section. It is a static value that references a role created by Databricks. Note that the policy is slightly different if you use Databricks on AWS GovCloud or AWS GovCloud DOD.

      The policy sets the external ID to 0000 as a placeholder. You update this to the external ID of your service credential in a later step.

      JSON
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Principal": {
              "AWS": ["arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-14S5ZJVKOTYTL"]
            },
            "Action": "sts:AssumeRole",
            "Condition": {
              "StringEquals": {
                "sts:ExternalId": "0000"
              }
            }
          }
        ]
      }

    3. Skip the permissions policy configuration. You'll go back to add that in a later step.

    4. Save the IAM role.

  2. Create an IAM policy in the same account as the service.

    Here are two sample policies that you can use as guidelines. One sample policy is for a service credential that connects to AWS Glue. The other references AWS Secrets Manager. The actual actions and resources that you add depend on the service that you are connecting to and the level of access you need. See the AWS IAM documentation for your service.

    The sts:AssumeRole action is the same regardless of service.

    Replace the following values:

    • <AWS-ACCOUNT-ID>: The Account ID of your AWS account (not your Databricks account).
    • <AWS-IAM-ROLE-NAME>: The name of the AWS IAM role that you created in the previous step.
    JSON
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Action": ["secretsmanager:GetResourcePolicy", "secretsmanager:GetSecretValue"],
          "Resource": ["arn:aws:secretsmanager:us-west-2:111122223333:secret:aes128-1a2b3c"],
          "Effect": "Allow"
        },
        {
          "Action": ["sts:AssumeRole"],
          "Resource": ["arn:aws:iam::<AWS-ACCOUNT-ID>:role/<AWS-IAM-ROLE-NAME>"],
          "Effect": "Allow"
        }
      ]
    }

  3. Attach the IAM policy to the IAM role.

    In the Role's Permission tab, attach the IAM Policy you just created.

Step 2: Give Databricks the IAM role details​

Permissions required: The CREATE SERVICE CREDENTIAL privilege. The metastore admin and account admin roles both include this privilege. Workspace admins in workspaces that were enabled for Unity Catalog automatically also have this privilege.

  1. In Databricks, log in to a workspace that is linked to the metastore.

  2. Click Data icon. Catalog.

  3. Click Plug icon. Connect, then Credentials.

  4. Click Create credential.

  5. Select Service Credential.

  6. Enter a Credential name, the role's Amazon Resource Name (ARN) in IAM role (ARN), and an optional comment. Use the role that authorizes Unity Catalog to access the service on your cloud tenant.

  7. Click Create.

  8. In the role-configuration instructions, click Copy policy to copy the generated Trust policy JSON.

    The generated policy includes the Unity Catalog principal, your IAM role as a trusted principal, and the credential's External ID. Keep the instructions open while you update the IAM role in the next step.

    You can also view the external ID on the service credential details page. See View a service credential.

Step 3: Update the IAM role policy​

In AWS, apply the trust policy generated for your service credential. It allows Unity Catalog to assume the role and makes the role self-assuming.

  1. In the AWS IAM console, open your saved IAM role.
  2. Open Trust relationships and click Edit trust policy.
  3. Update the policy using the JSON copied in the previous step:
    • For a new, dedicated role, replace the temporary trust policy with the complete generated policy.
    • For a reused role, preserve any trust statements that other workloads independently require. Add the generated statement, replacing any temporary Unity Catalog statement. Retain the generated principals, self-assume entry, and external-ID condition together.
  4. Click Update policy.

Use the generated values for this credential. Leaving the placeholder external ID or copying the external ID from another credential prevents the intended trust relationship from working.

For a reused role, keep the generated statement as one entry in the policy's Statement array, alongside any statements that other workloads require. This example shows its structure. Replace the placeholders with the values from Copy policy:

JSON
{
  "Effect": "Allow",
  "Principal": {
    "AWS": ["<unity-catalog-role-arn>", "<your-iam-role-arn>"]
  },
  "Action": "sts:AssumeRole",
  "Condition": {
    "StringEquals": {
      "sts:ExternalId": "<service-credential-external-id>"
    }
  }
}

Step 4: Validate the service credential​

After you have made the changes to the IAM role trust policy in Step 3: Update the IAM role policy, verify that your IAM role is properly configured to be used as a service credential:

  1. In Databricks, log in to a workspace that is linked to the metastore.
  2. Click Data icon. Catalog.
  3. Click Plug icon. Connect, then click Credentials.
  4. Select the service credential that you want to validate.
  5. Click Validate Configuration button. Validation can take 1-2 minutes.
  6. If any of the checks fail, return to Step 3: Update the IAM role policy and review the IAM role's trust policy to configure them correctly.

Self-assuming role enforcement policy​

On June 30, 2023, AWS updated its IAM role trust policy to require that IAM roles explicitly self-trust for STS:AssumeRole calls. As a result, Databricks requires that AWS IAM roles for service credentials be self-assuming and will soon prohibit non-self-assuming service credentials. For details, see this community blog post.

Databricks will begin prohibiting the creation of service credentials with non-self-assuming AWS IAM roles on September 20, 2024. Existing service credentials with non-self-assuming IAM roles will continue to work, but you will not be able to create new service credentials using those roles.

On January 20, 2025, Databricks will begin blocking the usage of existing service credentials with non-self-assuming IAM roles. This can potentially break workloads and jobs that run using non-self-assuming credentials.

To check whether an AWS IAM role for a service credential is self-assuming, follow the instructions in Step 4: Validate the service credential. If the Self Assume Role check fails, revisit Step 3: Update the IAM role policy and reconfigure the IAM role's trust policy to trust itself.

(Optional) Assign a service credential to specific workspaces​

By default, a service credential is accessible from all of the workspaces in the metastore. This means that if a user has been granted a privilege on that service credential, they can exercise that privilege from any workspace attached to the metastore. If you use workspaces to isolate user data access, you may want to allow access to a service credential only from specific workspaces. This feature is known as workspace binding or service credential isolation.

A typical use case for binding a service credential to specific workspaces is the scenario in which a cloud admin configures a service credential using a production cloud account credential, and you want to ensure that Databricks users use this credential to access an external cloud service only in the production workspace.

For more information about workspace binding, see Assign a storage credential to specific workspaces and Workspace-catalog binding.

Bind a service credential to one or more workspaces​

To assign a service credential to specific workspaces, use Catalog Explorer.

Permissions required: Metastore admin or service credential owner.

note

In Catalog Explorer, metastore admins can see all service credentials in a metastore, and service credential owners can see all credentials they own, regardless of workspace bindings. Service credentials that are not assigned to the workspace appear grayed out.

  1. Log in to a workspace that is linked to the metastore.

  2. In the sidebar, click Data icon. Catalog.

  3. Click Plug icon. Connect, then Credentials.

  4. Select the service credential and go to the Workspaces tab.

  5. On the Workspaces tab, clear the All workspaces have access checkbox.

    If your service credential is already bound to one or more workspaces, this checkbox is already cleared.

  6. Click Assign to workspaces and enter or find the workspaces you want to assign.

To revoke access, go to the Workspaces tab, select the workspace, and click Revoke. To allow access from all workspaces, select the All workspaces have access checkbox.

Next steps​

Limitations​

The following limitations apply:

  • Databricks Runtime versions 16.1 and below support only Python.
  • SQL warehouses support service credentials for batch Unity Catalog Python UDFs only.
  • Some audit events for actions performed on service credentials will not appear in the system.access.audit table. Audit information about who created, deleted, updated, read, listed, or used a service credential will be available. See Audit log system table reference.
  • During the service credentials preview, INFORMATION_SCHEMA.STORAGE_CREDENTIALS (deprecated) displayed both storage credentials and service credentials, and INFORMATION_SCHEMA.STORAGE_CREDENTIAL_PRIVILEGES (deprecated) displayed privileges that applied both to storage credentials and service credentials. This is no longer the case. You should instead use INFORMATION_SCHEMA.CREDENTIALS and INFORMATION_SCHEMA.CREDENTIAL_PRIVILEGES for both storage and service credentials.
On this page