Skip to main content
Last updated on

CI/CD for Feature Views with GitHub Actions

Preview

This feature is in Private Preview. To try it, reach out to your Databricks contact.

To set up CI/CD for Feature Views and materialization sets, use GitHub Actions with Declarative Automation Bundles. On each pull request, a workflow validates the bundle and plans the change without deploying it. Reviewers see the planned change before any materialization starts billing. When the pull request merges, a second workflow plans the merged configuration and deploys exactly that plan, optionally after an approver reviews it. Both workflows authenticate with workload identity federation instead of a secret stored in the repository.

This page builds on the bundle from Manage Feature Views using Declarative Automation Bundles. For GitHub Actions with jobs, pipelines, and other Databricks resources, see GitHub Actions.

Requirements​

The workflows on this page require the following:

  • A Databricks service principal that deploys, added to each workspace you deploy to, with these privileges:

    • USE CATALOG and USE SCHEMA on each catalog and schema that the bundle reads from or writes to.
    • CREATE FEATURE on the schema where the bundle creates Feature Views.
    • SELECT on each source table and on each Stream's ingestion table.
    • CREATE TABLE on each offline and online destination schema.
    • CAN USE on the Lakebase instance or project that backs the online store.

    Materializing a Feature View also requires MANAGE and READ FEATURE on that Feature View. You don't grant these. The Databricks service principal that deploys owns the Feature Views because it creates them. Owners can perform every action on the objects they own. For details, see Permissions and Ownership.

  • A Databricks service principal that plans, added to the same workspaces, with these read-only privileges:

    • USE CATALOG and USE SCHEMA on each catalog and schema that the bundle reads from or writes to.
    • READ FEATURE on the schema where the bundle creates Feature Views.
    • SELECT on each source table and on each Stream's ingestion table. Without this privilege, the plan can't see a Feature View's source and reports a recreate for the Feature View.
    • CAN READ on the prod target's root_path folder, which holds the prod bundle state. Without this privilege, the plan can't read the state and reports every resource as create, without an error. See Deploy to more than one workspace.

  • A bundle that declares features and materialization_sets resources and validates locally. See Manage Feature Views using Declarative Automation Bundles.

  • Admin access to the GitHub repository that contains the bundle, to add the repository variables and configure the prod environment. If the repository is private or internal, environments need GitHub Pro, GitHub Team, or GitHub Enterprise, and required reviewers on an environment need GitHub Enterprise.

  • An online store, and the source tables and Streams that your bundle reads from, available to each workspace you deploy to. A bundle references these by name and doesn't create them. See Create an online store and Create a stream.

Set up the workflows​

The following steps set up authentication, the two workflows, and a target for each workspace.

Authenticate the runner​

Each workflow job runs on a GitHub Actions runner, a virtual machine that GitHub hosts. With workload identity federation, the runner authenticates with Databricks using a short-lived OpenID Connect (OIDC) token instead of a credential stored in your repository.

  1. On each Databricks service principal, create a GitHub Actions federation policy for each of its subjects in the following table, and record both application IDs. GitHub issues each job a token with a sub (subject) claim that identifies the repository and what triggered the job. Databricks matches this claim exactly against a policy's subject, so each subject needs its own policy. See Enable workload identity federation for GitHub Actions.

    Job

    Subject

    Authenticates as

    The plan job on a pull request

    repo:<org>/<repo>:pull_request

    The Databricks service principal that plans

    The plan job on a push to main

    repo:<org>/<repo>:ref:refs/heads/main

    The Databricks service principal that plans

    The deploy job, which runs in the prod environment

    repo:<org>/<repo>:environment:prod

    The Databricks service principal that deploys

    Job

    Subject

    Authenticates as

    The plan job on a pull request

    repo:<org>/<repo>:pull_request

    The Databricks service principal that plans

    The plan job on a push to main

    repo:<org>/<repo>:ref:refs/heads/main

    The Databricks service principal that plans

    The deploy job, which runs in the prod environment

    repo:<org>/<repo>:environment:prod

    The Databricks service principal that deploys

    Repositories created, renamed, or transferred on or after July 15, 2026 use a subject format that includes numeric IDs. To find your repository's exact subject, see Create a federation policy.

    Only a job in the prod environment can authenticate as the Databricks service principal that deploys. When you add the deploy workflow, you limit that environment to the main branch, so that a pull request can't deploy.

    A pull request can change the workflow it runs. Anyone who can push a branch to the repository can therefore authenticate as the Databricks service principal that plans and use all of its read privileges. Grant that Databricks service principal only the privileges listed in Requirements.

  2. In your repository, go to Settings > Secrets and variables > Actions, select the Variables tab, and add the following repository variables:

    Variable

    Value

    DATABRICKS_HOST

    Your workspace instance URL, for example https://1234567890123456.7.gcp.databricks.com

    DATABRICKS_PLAN_CLIENT_ID

    The application ID of the Databricks service principal that plans

    DATABRICKS_CLIENT_ID

    The application ID of the Databricks service principal that deploys

    Variable

    Value

    DATABRICKS_HOST

    Your workspace instance URL, for example https://1234567890123456.7.gcp.databricks.com

    DATABRICKS_PLAN_CLIENT_ID

    The application ID of the Databricks service principal that plans

    DATABRICKS_CLIENT_ID

    The application ID of the Databricks service principal that deploys

    None of these values is sensitive, so all three are variables rather than secrets.

Add the pull request workflow​

The pull request workflow validates the bundle and plans the change without deploying it. Deploying a materialization set creates one or more pipelines and jobs that start materializing immediately, so deploying each pull request would start billing for changes that are still in review. For what materialization costs, see Feature Store cost management.

databricks bundle plan reads the bundle state and the deployed resources and changes nothing, so the plan job authenticates as the Databricks service principal that plans.

Add .github/workflows/features-pr.yml to your repository:

YAML
name: Plan feature changes

on:
  pull_request:
    branches: [main]

permissions:
  id-token: write # required for workload identity federation
  contents: read

jobs:
  plan:
    name: Plan
    runs-on: ubuntu-latest
    env:
      DATABRICKS_AUTH_TYPE: github-oidc
      DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}
      DATABRICKS_CLIENT_ID: ${{ vars.DATABRICKS_PLAN_CLIENT_ID }}
    steps:
      - uses: actions/checkout@v4

      - name: Install Databricks CLI
        uses: databricks/setup-cli@v1.24.0

      - name: Validate bundle
        run: databricks bundle validate --target prod

      - name: Plan
        run: databricks bundle plan --target prod --output json > plan.json

      - uses: actions/upload-artifact@v4
        with:
          name: feature-plan
          path: plan.json

The last step uploads plan.json as an artifact. The deploy workflow runs this same job and applies the plan it uploads.

A change to a Feature View or a materialization set either applies in place or replaces the resource. A replacement drops and recomputes the resource's materialized data. See Changing a Feature View and Changing a materialization set.

To limit what a deployment can change, use one or both of the following:

  • Check the plan in the workflow. --output json writes a machine-readable plan, so a script of your own can inspect it and exit non-zero. For example, a plan that replaces a materialization set reports recreate for it. This output is abridged to the fields a check reads:

    JSON
    {
      "plan_version": 2,
      "plan": {
        "resources.features.transaction_count": {
          "action": "skip"
        },
        "resources.materialization_sets.transactions": {
          "action": "recreate"
        }
      }
    }

    Run the script after the plan step:

    YAML
    - name: Check the plan
      run: ./.github/scripts/check-plan.sh plan.json

  • Set prevent_destroy on each resource that you want to protect. Any deployment that would replace or delete a protected resource then fails, wherever it runs. See Guard specific actions in the bundle configuration.

Guard specific actions in the bundle configuration​

Add a lifecycle block to a resource in your production target to block changes that would destroy it. Because the setting is part of the bundle configuration, it applies to every deployment, including ones run outside the workflow.

prevent_destroy is the only lifecycle setting that features and materialization_sets support. Setting it refuses both of the plan actions that destroy materialized data, recreate and delete:

YAML
targets:
  prod:
    mode: production
    resources:
      features:
        transaction_count:
          lifecycle:
            prevent_destroy: true
      materialization_sets:
        transactions:
          lifecycle:
            prevent_destroy: true

When a plan would replace or delete a guarded resource, databricks bundle deploy and databricks bundle destroy fail before making any change:

Text
Error: resources.materialization_sets.transactions has lifecycle.prevent_destroy set, but the plan calls for this resource to be recreated or destroyed. To avoid this error, disable lifecycle.prevent_destroy for resources.materialization_sets.transactions

The --auto-approve flag doesn't override this. To make an intended replacement, remove prevent_destroy in a separate pull request, so that reviewers approve the replacement on its own.

note

databricks bundle plan doesn't enforce prevent_destroy, so a pull request that would replace a guarded resource passes its plan job and fails only at deploy, after it merges. To catch the replacement in the pull request, check the plan.

Add the deploy workflow​

The deploy workflow plans the merged configuration, waits for approval if you require it, and then applies that plan.

Its plan job is the same as the one in the pull request workflow. Add .github/workflows/features-deploy.yml:

YAML
name: Deploy feature changes

on:
  push:
    branches: [main]

# Only one deployment reaches the workspace at a time.
concurrency: feature-store-prod

permissions:
  id-token: write
  contents: read

jobs:
  plan:
    name: Plan
    runs-on: ubuntu-latest
    env:
      DATABRICKS_AUTH_TYPE: github-oidc
      DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}
      DATABRICKS_CLIENT_ID: ${{ vars.DATABRICKS_PLAN_CLIENT_ID }}
    steps:
      - uses: actions/checkout@v4

      - name: Install Databricks CLI
        uses: databricks/setup-cli@v1.24.0

      - name: Validate bundle
        run: databricks bundle validate --target prod

      - name: Plan
        run: databricks bundle plan --target prod --output json > plan.json

      - uses: actions/upload-artifact@v4
        with:
          name: feature-plan
          path: plan.json

  deploy:
    name: Deploy
    needs: plan
    runs-on: ubuntu-latest
    environment: prod
    env:
      DATABRICKS_AUTH_TYPE: github-oidc
      DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}
      DATABRICKS_CLIENT_ID: ${{ vars.DATABRICKS_CLIENT_ID }}
    steps:
      - uses: actions/checkout@v4
      - name: Install Databricks CLI
        uses: databricks/setup-cli@v1.24.0

      - uses: actions/download-artifact@v4
        with:
          name: feature-plan

      - name: Deploy
        run: databricks bundle deploy --target prod --plan plan.json --auto-approve

In your repository, go to Settings > Environments, select prod, and limit Deployment branches and tags to main. A job from a pull request then can't run in the prod environment, so it can't authenticate as the Databricks service principal that deploys. To require an approver as well, under Deployment protection rules, select Required reviewers and add the approvers. The deploy job then waits, and the approver can download the feature-plan artifact from the workflow run to read the plan before allowing it to run.

--auto-approve skips the confirmation prompt that databricks bundle deploy shows before it replaces or deletes a resource, because a workflow can't answer it. If the prod environment requires reviewers, their approval takes its place.

--plan applies the plan in the file instead of computing a new one, so the deployment is exactly what the plan job reported. A plan file records the state it was built from, so if another deployment reaches the target while the approval is pending, the deploy fails because the plan no longer matches the state:

Text
Error: plan serial 4 does not match state serial 5; the state has been modified since the plan was created. Please run 'bundle plan' again

The deploy workflow builds its own plan rather than reusing the pull request's plan file, because the merge commit can differ from the pull request's head commit. For example, other pull requests might merge first. A plan built before the merge doesn't describe what deploying the merge commit does.

Report the materialization state​

databricks bundle deploy returns as soon as the materialization sets exist, before their first materialization runs finish. Optionally, add a step that reads each set back from the server to record what a deployment created. The report shows the generated table names and whether each materialized feature has run yet. Put the step in the deploy job of features-deploy.yml, after the Deploy step:

YAML
- name: Report materialization state
  run: |
    for SET_ID in $(databricks bundle summary --target prod --output json \
        | jq -r '.resources.materialization_sets[].id'); do
      databricks feature-engineering get-materialization-set "$SET_ID" --target prod --output json \
        | jq '.materialized_features[]
              | {feature_name, is_online, table_name, last_materialization_time}'
    done

For each materialized feature in a set, the report shows the generated table_name and a last_materialization_time that stays empty until its first materialization run finishes. For how materialized features are computed and how to view their status, see Materialize Feature Views.

note

Read the set from the API rather than from databricks bundle summary, which the step uses only to look up the set's ID. The summary reports the configuration that the bundle holds locally, where table_name and last_materialization_time are empty and every materialized feature appears offline.

Deploy to more than one workspace​

Give each workspace its own target, and use bundle variables for the values that differ, so that the same Feature View definitions deploy everywhere. In this example, developers deploy dev from their own machines to test a change before they open a pull request, and the workflows deploy only prod:

YAML
targets:
  dev:
    mode: development
    default: true
    workspace:
      host: <dev-workspace-url>
    variables:
      catalog: main
      schema: features_${workspace.current_user.short_name}
      online_store: <dev-online-store>
  prod:
    mode: production
    workspace:
      host: <prod-workspace-url>
      root_path: /Workspace/Users/<deploy-service-principal-application-id>/.bundle/${bundle.name}/${bundle.target}
    variables:
      catalog: main
      schema: features_prod
      online_store: <prod-online-store>

Development mode gives each developer their own copy of the bundle state. It adds a [dev <user-name>] prefix to the names of jobs and pipelines, and doesn't rename Feature Views. Two developers who deploy dev would otherwise write the same Feature Views, so the dev target puts each developer's Feature Views in their own schema, features_<user-name>. Each developer creates their schema before their first deployment.

The prod target's root_path is in the home folder of the Databricks service principal that deploys. After the first deployment creates the folder, give the Databricks service principal that plans CAN READ on it, so that the plan job can read the bundle state.

Because dev is the default target, a command without --target deploys to dev. Pass --target on every command in a workflow. When a target sets workspace.host, the CLI uses it instead of the DATABRICKS_HOST variable. Set the variable to the same host as the prod target, so that the two don't disagree. For variables, see Substitutions and variables in Declarative Automation Bundles. For what each mode changes, see Declarative Automation Bundles deployment modes.

important

Each target that you deploy materialization sets to bills for materialization on the sets' schedules, including development-mode targets. To stop the cost of a target you no longer need, run databricks bundle destroy --target <target>. For what materialization costs, see Feature Store cost management.

Limitations​

  • You can't add a materialized feature to an existing materialization set without replacing the set. If the materialization set has prevent_destroy: true, remove it in a separate pull request before you add the feature.
  • The Databricks service principal that deploys can't delete or replace a materialized feature that a user created, because only its creator can delete it. Deploy prod only from the workflow, from the start.

Additional resources​

On this page