Enable workload identity federation for CircleCI

Preview

Databricks OAuth token federation is in Public Preview.

Databricks OAuth token federation, also known as OpenID Connect (OIDC), allows your automated workloads running outside of Databricks to securely access Databricks without the need for Databricks secrets. See Authenticate access to Databricks using OAuth token federation.

To enable workload identity federation for CircleCI:

1. Create a federation policy
2. Configure the CircleCI YAML

After you enable workload identity federation, the Databricks SDKs and the Databricks CLI automatically fetch workload identity tokens from CircleCI and exchange them for Databricks OAuth tokens.

Create a federation policy

First use the Databricks CLI to create a workload identity federation policy. For CircleCI, set the following values for the policy:

• issuer: https://oidc.circleci.com/org/<org_id>, where <org-id> is replaced with your organization ID
• audiences: your CircleCI organization ID
• subject: the CircleCI project ID
• subject_claim: oidc.circleci.com/project-id

For example, given an organization ID 1234 and a project ID 5678, with a Databricks service principal numeric ID of 5581763342009999, create a federation policy using the following Databricks CLI command:

Bash

databricks account service-principal-federation-policy create 5581763342009999 --json '{
  "oidc_policy": {
	"issuer": "https://oidc.circleci.com/org/1234",
	"audiences": [
  	  "1234"
	],
	"subject": "5678",
  "subject_claim": "oidc.circleci.com/project-id"
  }
}'

Configure the CircleCI YAML

Next, modify the CircleCI configuration file. In addition to the following variables, set DATABRICKS_OIDC_TOKEN_ENV in your CircleCI config.yml file to instruct the Databricks SDKs or CLI to search for the token in CIRCLE_OIDC_TOKEN_V2. (You can also use the older CIRCLE_OIDC_TOKEN environment variable.)

• DATABRICKS_AUTH_TYPE: env-oidc
• DATABRICKS_HOST: your Databricks workspace URL
• DATABRICKS_CLIENT_ID: the service principal (application) ID

YAML

version: 2.1

jobs:
  build:
    docker:
  	  - image: cimg/base:current
    environment:
      DATABRICKS_HOST: https://my-workspace.cloud.databricks.com/
      DATABRICKS_CLIENT_ID: a1b2c3d4-ee42-1eet-1337-f00b44r
      DATABRICKS_OIDC_TOKEN_ENV: CIRCLE_OIDC_TOKEN
      DATABRICKS_AUTH_TYPE: env-oidc
    steps:
      - checkout
      - run:
          name: Install Databricks CLI
          command: |
            curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sudo sh
            databricks --version
      - run:
          name: Run Databricks CLI commands
          command: databricks current-user me