Service principals for CI/CD

This article describes how to use service principals for CI/CD with Databricks. A service principal is an identity created for use with automated tools and applications, including:

As a security best practice, Databricks recommends using a Databricks service principal and its token instead of your Databricks user or your Databricks personal access token for your workspace user to give CI/CD platforms access to Databricks resources. Some benefits to this approach include the following:

  • You can grant and restrict access to Databricks resources for a Databricks service principal independently of a user. For instance, this allows you to prohibit a Databricks service principal from acting as an admin in your Databricks workspace while still allowing other specific users in your workspace to continue to act as admins.

  • Users can safeguard their access tokens from being accessed by CI/CD platforms.

  • You can temporarily disable or permanently delete a Databricks service principal without impacting other users. For instance, this allows you to pause or remove access from a Databricks service principal that you suspect is being used in a malicious way.

  • If a user leaves your organization, you can remove that user without impacting any Databricks service principal.

To give a CI/CD platform access to your Databricks workspace, do the following:

  1. Create a Databricks service principal in your workspace.

  2. Generate a Databricks access token for a Databricks service principal.

  3. Give this Databricks access token to the CI/CD platform.

To complete Steps 1 and 2, see Manage service principals.

To complete Step 3, complete the instructions in this article.

Requirements

  • The Databricks access token for a Databricks service principal. To create a Databricks service principal and its Databricks access token, see Manage service principals.

  • An account with your Git provider.

Set up GitHub Actions

GitHub Actions must be able to access your Databricks workspace. If you want to use Databricks Git folders, your workspace must also be able to access GitHub.

To enable GitHub Actions to access your Databricks workspace, you must register the Databricks access token for your Databricks service principal with GitHub Actions.

If you also want to enable your Databricks workspace to access GitHub when you use Databricks Git folders, you must add the GitHub personal access token for a GitHub machine user to your workspace.

Register the Databricks access token for your Databricks service principal with GitHub Actions

This section describes how to enable GitHub Actions to access your Databricks workspace.

As a security best practice, Databricks recommends that you do not enter a Databricks access token directly into the body of a GitHub Actions file. You should register the Databricks access token with GitHub Actions by using GitHub encrypted secrets instead.

GitHub Actions, such as the ones that Databricks lists in Continuous integration and delivery using GitHub Actions, as well as the onpush.yml and onrelease.yml files as part of the Basic Python Template in dbx for GitHub Actions, rely on GitHub encrypted secrets such as:

  • DATABRICKS_HOST, which is the value https:// followed by your workspace instance name, for example dbc-a1b2345c-d6e7.cloud.databricks.com.

  • DATABRICKS_TOKEN, which is the value of the token_value value that you copied after you created the Databricks access token for the Databricks service principal.

For more information about which GitHub encrypted secrets are required for a GitHub Action, see Manage service principals and the documentation for that GitHub Action.

To add these GitHub encrypted secrets to your GitHub repository, see Creating encrypted secrets for a repository in the GitHub documentation. For other approaches to add these GitHub repository secrets, see Encrypted secrets in the GitHub documentation.

Add the GitHub personal access token for a GitHub machine user to your Databricks workspace

This section describes how to enable your Databricks workspace to access GitHub with Databricks Git folders. This is an optional task in CI/CD scenarios.

As a security best practice, Databricks recommends that you use GitHub machine users instead of GitHub personal accounts, for many of the same reasons that you should use a Databricks service principal instead of a Databricks user. To add the GitHub personal access token for a GitHub machine user to your Databricks workspace, do the following:

  1. Create a GitHub machine user, if you do not already have one available. A GitHub machine user is a GitHub personal account, separate from your own GitHub personal account, that you can use to automate activity on GitHub. Create a new separate GitHub account to use as a GitHub machine user, if you do not already have one available.

    Note

    When you create a new separate GitHub account as a GitHub machine user, you cannot associate it with the email address for your own GitHub personal account. Instead, see your organization’s email administrator about getting a separate email address that you can associate with this new separate GitHub account as a GitHub machine user.

    See your organization’s account administrator about managing the separate email address and its associated GitHub machine user and its GitHub personal access tokens within your organization.

  2. Give the GitHub machine user access to your GitHub repository. See Inviting a team or person in the GitHub documentation. To accept the invitation, you may first need to sign out of your GitHub personal account, and then sign back in as the GitHub machine user.

  3. Sign in to GitHub as the machine user, and then create a GitHub personal access token for that machine user. See Create a personal access token in the GitHub documentation. Be sure to give the GitHub personal access token repo access.

  4. Gather the Databricks access token for your Databricks service principal, your GitHub machine username, and then see Use a service principal with Databricks Git folders.

Set up GitLab CI/CD

GitLab CI/CD must be able to access your Databricks workspace. If you also want to use Databricks Git folders, your workspace must be able to access GitLab CI/CD.

To access your Databricks workspace, GitLab CI/CD .gitlab-ci.yml files, such as the one as part of the Basic Python Template in dbx, rely on custom CI/CD variables such as:

  • DATABRICKS_HOST, which is the value https:// followed by your workspace instance name, for example dbc-a1b2345c-d6e7.cloud.databricks.com.

  • DATABRICKS_TOKEN, which is the value of the token_value value that you copied after you created the Databricks access token for the Databricks service principal.

To add these custom variables to your GitLab CI/CD project, see Add a CI/CD variable to a project in the GitLab CI/CD documentation.

If your workspace uses Databricks Git folders, and you want to enable your workspace to access GitLab CI/CD, gather:

  • The Databricks access token for your Databricks service principal

  • Your GitLab CI/CD username

Then, see Use a service principal with Databricks Git folders.