Skip to main content

Enable workload identity federation for Azure DevOps Pipelines

Databricks OAuth token federation, also known as OpenID Connect (OIDC), allows your automated workloads running outside of Databricks to securely access Databricks without the need for Databricks secrets. See Authenticate access to Databricks using OAuth token federation.

To enable workload identity federation for Azure DevOps Pipelines:

  1. Create a federation policy
  2. Configure the YAML

After you enable workload identity federation, the Databricks SDKs and the Databricks CLI automatically fetch workload identity tokens from Azure DevOps Pipelines and exchange them for Databricks OAuth tokens.

Create a federation policy​

First, create a custom workload identity federation policy. For instructions, see Configure a service principal federation policy. For Azure DevOps, set the following values for the policy:

  • Issuer URL: https://vstoken.dev.azure.com/<org_id>, where <org-id> is the GUID of your Azure DevOps organization
  • Audiences: api://AzureADTokenExchange
  • Subject: p://<org-name>/<project-name>/<pipeline-name> where <org-name> is your Azure DevOps organization name, <project-name> is your Azure DevOps project name and <pipeline-name> is the name of your Azure DevOps pipeline

For example, the following Databricks CLI command creates a federation policy for an organization ID 7f1078d6-b20d-4a20-9d88-05a2f0d645a3 and a Databricks service principal numeric ID of 5581763342009999:

Bash
databricks account service-principal-federation-policy create 5581763342009999 --json '{
  "oidc_policy": {
	"issuer": "https://vstoken.dev.azure.com/7f1078d6-b20d-4a20-9d88-05a2f0d645a3",
	"audiences": [
  	    "api://AzureADTokenExchange"
	],
	"subject": "p://my-org/my-project/my-pipeline"
  }
}
'

Configure the YAML​

Next, modify the configuration file. In addition to setting the following workspace environment variables, use the Azure CLI to obtain a token in the job, and store it in DATABRICKS_OIDC_TOKEN.

  • DATABRICKS_AUTH_TYPE: env-oidc
  • DATABRICKS_HOST: your Databricks workspace URL
  • DATABRICKS_CLIENT_ID: the service principal (application) ID
YAML
trigger: none
pool: test # my self-hosted pool name

variables:
  DATABRICKS_HOST: https://my-workspace.cloud.databricks.com/
  DATABRICKS_AUTH_TYPE: env-oidc
  DATABRICKS_CLIENT_ID: a1b2c3d4-ee42-1eet-1337-f00b44r

steps:
  - script: |
      OIDC_REQUEST_URL="${SYSTEM_OIDCREQUESTURI}?api-version=7.1"
      export DATABRICKS_OIDC_TOKEN=$(curl -s -H "Content-Length: 0" -H "Content-Type: application/json" -H "Authorization: Bearer $(System.AccessToken)" -X POST $OIDC_REQUEST_URL | jq -r '.oidcToken')
      databricks current-user me
    displayName: 'Display Databricks current user information'
Last updated on