Skip to main content
Last updated on

Manage Feature Views using Declarative Automation Bundles

Preview

This feature is in Private Preview. To try it, reach out to your Databricks contact.

Declarative Automation Bundles (formerly known as Databricks Asset Bundles) enable you to programmatically validate, deploy, and run Databricks resources such as Feature Views and the pipelines that materialize them. See What are Declarative Automation Bundles?.

This tutorial builds a bundle that declares four Feature Views and materializes them three ways: two batch features written to both an offline and an online store, a streaming feature written continuously to an online store, and a streaming feature with a sawtooth window, which materializes to an offline and an online store in batch mode and to the online store in streaming mode. You then change the configuration, redeploy, and tear everything down.

Declaring features in a bundle gives you the same definitions in every workspace you deploy to, reviewed as source alongside the rest of your project. It is the declarative alternative to calling create_feature and materialize_features from a notebook. See Feature Views.

Each step shows the configuration in both YAML and Python. Pick a language and the page follows your choice.

Requirements​

The streaming steps use real-time mode, so a Kinesis Stream must use an enhanced fan-out (EFO) consumer, with consumerMode set to efo in its extra_options. See Amazon Kinesis.

A source table and a Stream each have their own lifecycle and neither is a bundle resource. Create them first and reference them by name in the bundle, as Step 3 does for the table and Step 4 for the Stream.

Feature Store resources in a bundle​

A bundle declares two Feature Store resource types, and the difference between them decides what a deployment costs.

  • Feature View: the definition of a feature, covering its source, its aggregation, and its keys. It is lightweight metadata. See Feature Views, and for the configuration reference, feature.
  • Materialization set: a group of materialized features. A materialized feature is one Feature View computed into a table through pipelines or jobs, so this is the heavyweight half. Grouping materialized features in one set tells Databricks that they can be computed together, so it packs them into the fewest tables and Lakeflow pipelines that hold them. See Materialize Feature Views, and for the configuration reference, materialization_set and materialization_set.materialized_features.

The steps that follow declare batch features, a streaming feature, and a streaming feature with a sawtooth window. Declare only the ones your project needs. For how materialization works, see Materialize Feature Views.

Step 1: Set up authentication​

First, set up authentication between Databricks CLI on your development machine and your Databricks workspace. This page assumes that you want to use OAuth user-to-machine (U2M) authentication and a corresponding Databricks configuration profile named DEFAULT for authentication.

note

U2M authentication is appropriate for trying out these steps in real time. For fully automated workflows, Databricks recommends that you use OAuth machine-to-machine (M2M) authentication instead. See the M2M authentication setup instructions in Authorize Databricks service principal access to Databricks with OAuth.

  1. Use the Databricks CLI to initiate OAuth token management locally by running the following command for each target workspace.

    In the following command, replace <workspace-url> with your Databricks workspace instance URL, for example https://dbc-a1b2345c-d6e7.cloud.databricks.com.

    Bash
    databricks auth login --host <workspace-url>

  2. The Databricks CLI prompts you to save the information that you entered as a Databricks configuration profile. Press Enter to accept the suggested profile name, or enter the name of a new or existing profile. Any existing profile with the same name is overwritten with the information that you entered. You can use profiles to quickly switch your authentication context across multiple workspaces.

    To get a list of any existing profiles, in a separate terminal or command prompt, use the Databricks CLI to run the command databricks auth profiles. To view a specific profile's existing settings, run the command databricks auth env --profile <profile-name>.

  3. In your web browser, complete the on-screen instructions to log in to your Databricks workspace.

  4. To view a profile's current OAuth token value and the token's upcoming expiration timestamp, run one of the following commands:

    • databricks auth token --host <workspace-url>
    • databricks auth token -p <profile-name>
    • databricks auth token --host <workspace-url> -p <profile-name>

    If you have multiple profiles with the same --host value, you might need to specify the --host and -p options together to help the Databricks CLI find the correct matching OAuth token information.

Step 2: Create the bundle​

Create an empty directory for the bundle, with a databricks.yml file at its root and a resources directory beside it.

Each of the following steps adds one file under resources/ to declare bundle resources:

Text
databricks.yml
resources/
  batch_feature.yml
  streaming_feature.yml
  streaming_feature_sawtooth.yml

databricks.yml holds three things:

  • The bundle name.
  • The include entry, which decides which files the bundle loads its resources from.
  • The variables, which hold values that resources can reference, such as the catalog, schema, and source names.
YAML
bundle:
  name: feature-store-quickstart

include:
  - resources/*.yml

variables:
  catalog:
    description: The catalog that features are written to.
    default: main
  schema:
    description: The schema that features are written to.
    default: features_demo
  source_table:
    description: The three-level name of the source Delta table.
    default: main.features_demo.transactions
  stream:
    description: The three-level name of a Stream registered in Unity Catalog.
    default: main.features_demo.events
  online_store:
    description: The online store that serves the features.
    default: my_online_store

Declaring everything in databricks.yml instead also works. Omit include and collect each step's declarations under one resources mapping:

YAML
bundle:
  name: feature-store-quickstart

variables:
  # The variables above.

resources:
  features:
    # The Feature Views from the following steps.
  materialization_sets:
    # The materialization sets from the following steps.

Step 3: Add batch features​

These two Feature Views count transactions per user over a trailing seven days and a trailing 30 days, both recomputed daily, and materialize together to an offline and an online store.

A Feature View on its own is only a definition. The materialization set is what produces data from it.

Each resource carries two names. transaction_count is the resource key: a name you choose, unique within the bundle, which databricks bundle commands use to address the resource and which other configuration references. For instance, the two materialized features of transaction_count in the set transactions reference the Feature View's name as ${resources.features.transaction_count.full_name}, which tells the bundle that the set depends on the Feature View transaction_count.

Because of this dependency, transaction_count is created before the set transactions when both are created, and deleted after it when both are destroyed. A hardcoded name like main.features_demo.transaction_count_7d, written without ${}, omits that dependency and deploys in an unpredictable order.

full_name is the three-level name you choose for the Feature View to be registered under Unity Catalog, and the one you query later. The two names are independent.

A duration is a count of seconds with an s suffix, as in 604800s for seven days. See Specifying a duration for a window, source, or function.

Each entry under materialized_features is one materialized feature member: one Feature View materialized to one destination. Each of these Feature Views materializes to both an offline and an online destination, so each contributes two materialized feature members, and each member carries its own trigger. Both Feature Views write to the same offline and online destination, because a set has one offline destination and at most one online destination.

Create resources/batch_feature.yml:

YAML
resources:
  features:
    transaction_count:
      full_name: ${var.catalog}.${var.schema}.transaction_count_7d
      description: Transactions per user over a trailing seven days
      source:
        delta_table_source:
          full_name: ${var.source_table}
      function:
        aggregation_function:
          count_function:
            input: transaction_id
          time_window:
            sliding:
              window_duration: 604800s
              slide_duration: 86400s
      entities:
        - name: user_id
      timeseries_column:
        name: transaction_time

    transaction_count_30d:
      full_name: ${var.catalog}.${var.schema}.transaction_count_30d
      description: Transactions per user over a trailing 30 days
      source:
        delta_table_source:
          full_name: ${var.source_table}
      function:
        aggregation_function:
          count_function:
            input: transaction_id
          time_window:
            sliding:
              window_duration: 2592000s
              slide_duration: 86400s
      entities:
        - name: user_id
      timeseries_column:
        name: transaction_time

  materialization_sets:
    transactions:
      materialized_features:
        # Batch offline materialized feature of transaction_count.
        - feature_name: ${resources.features.transaction_count.full_name}
          offline_store_config:
            catalog_name: ${var.catalog}
            schema_name: ${var.schema}
            table_name_prefix: transactions_offline
          cron_schedule_trigger:
            mode: DERIVED
        # Batch online materialized feature of transaction_count.
        - feature_name: ${resources.features.transaction_count.full_name}
          online_store_config:
            online_store_name: ${var.online_store}
            catalog_name: ${var.catalog}
            schema_name: ${var.schema}
            table_name_prefix: transactions_online
          cron_schedule_trigger:
            mode: DERIVED
        # Batch offline materialized feature of transaction_count_30d.
        - feature_name: ${resources.features.transaction_count_30d.full_name}
          offline_store_config:
            catalog_name: ${var.catalog}
            schema_name: ${var.schema}
            table_name_prefix: transactions_offline
          cron_schedule_trigger:
            mode: DERIVED
        # Batch online materialized feature of transaction_count_30d.
        - feature_name: ${resources.features.transaction_count_30d.full_name}
          online_store_config:
            online_store_name: ${var.online_store}
            catalog_name: ${var.catalog}
            schema_name: ${var.schema}
            table_name_prefix: transactions_online
          cron_schedule_trigger:
            mode: DERIVED

Step 4: Add a streaming feature​

This Feature View counts events per key over a trailing 90 minutes and materializes continuously to the online store. It writes to one destination, so its set declares a single member.

A streaming materialized feature member carries streaming_mode and no cron_schedule_trigger. For the modes and what each one means, see materialization_set.materialized_features.streaming_mode.

This is a second materialization set rather than a materialized feature added to the set defined in Step 3, because a materialization set cannot mix Feature Views materialized in streaming mode with Feature Views materialized in batch. A streaming Feature View with a sawtooth window is the exception. Its batch and streaming materialized features belong in the same set, as in Step 5.

The Feature View reads a Stream by name. A Stream is created and versioned on its own, outside the bundle, in the same way as the Delta table the batch feature reads. To create one, see Create a stream.

A streaming Feature View addresses an input column as a path into the stream record rather than as a plain column name, so substitute the names your Stream's schema uses wherever the examples show value.*.

Create resources/streaming_feature.yml:

YAML
resources:
  features:
    event_count_90m:
      full_name: ${var.catalog}.${var.schema}.event_count_90m
      description: Events per key over a trailing 90 minutes
      source:
        stream_source:
          full_name: ${var.stream}
      function:
        aggregation_function:
          count_function:
            input: value.amount
          time_window:
            rolling:
              window_duration: 5400s
      entities:
        - name: value.key
      timeseries_column:
        name: value.event_timestamp

  materialization_sets:
    events_online:
      materialized_features:
        - feature_name: ${resources.features.event_count_90m.full_name}
          online_store_config:
            online_store_name: ${var.online_store}
            catalog_name: ${var.catalog}
            schema_name: ${var.schema}
            table_name_prefix: event_count_90m_online
          streaming_mode:
            mode: STREAMING_MODE_TYPE_RTM

Step 5: Add a streaming feature with a sawtooth window​

Materializing a streaming Feature View whose aggregation uses a sawtooth window produces three materialized features: an offline batch materialized feature, an online batch materialized feature, and an online streaming materialized feature. The server requires all three, and declaring a subset of the materialized features is rejected. The two batch materialized features must use DERIVED as the cron_schedule_trigger, so that Databricks chooses the schedule from the feature definition. The batch online materialized feature and the streaming online materialized feature must share the same online store configuration. A materialization set that does not meet these requirements is rejected on deploy. For the full list, see Rules a materialization set must satisfy.

Create resources/streaming_feature_sawtooth.yml:

YAML
resources:
  features:
    amount_sum:
      full_name: ${var.catalog}.${var.schema}.amount_sum_3d
      description: Sum of amount per key over a sawtooth window
      source:
        stream_source:
          full_name: ${var.stream}
      function:
        aggregation_function:
          sum:
            input: value.amount
          time_window:
            sawtooth:
              window_duration: 259200s
      entities:
        - name: value.key
      timeseries_column:
        name: value.event_timestamp

  materialization_sets:
    amounts:
      materialized_features:
        # Batch offline materialized feature.
        - feature_name: ${resources.features.amount_sum.full_name}
          offline_store_config:
            catalog_name: ${var.catalog}
            schema_name: ${var.schema}
            table_name_prefix: amount_sum_offline
          cron_schedule_trigger:
            mode: DERIVED
        # Batch online materialized feature.
        - feature_name: ${resources.features.amount_sum.full_name}
          online_store_config:
            online_store_name: ${var.online_store}
            catalog_name: ${var.catalog}
            schema_name: ${var.schema}
            table_name_prefix: amount_sum_online
          cron_schedule_trigger:
            mode: DERIVED
        # Streaming online materialized feature.
        - feature_name: ${resources.features.amount_sum.full_name}
          online_store_config:
            online_store_name: ${var.online_store}
            catalog_name: ${var.catalog}
            schema_name: ${var.schema}
            table_name_prefix: amount_sum_online
          streaming_mode:
            mode: STREAMING_MODE_TYPE_RTM

Step 6: Validate and plan​

Check the configuration, which reads databricks.yml and every resource file the bundle loads:

Bash
databricks bundle validate

validate is also the cheapest way to find out whether a field is accepted. It names any field the bundle does not recognize, with its path and the file and line it came from:

Text
Warning: unknown field: bogus_field
  at resources.features.transaction_count.function.aggregation_function.time_window.sliding
  in resources/batch_feature.yml:16:15

An unrecognized field is reported as a warning and its value is dropped, so read the output rather than only the exit code. Pass --strict to treat warnings as errors. For the fields each window, function, and source accepts, see feature.

Rules the server enforces, such as the rule that a Feature View with a sawtooth window must declare all three materialized features, are not checked here and surface on databricks bundle deploy.

Then review what the deployment does before it runs:

Bash
databricks bundle plan

On a first deployment, all seven resources (four Feature Views and three materialization sets) report create:

Text
create features.amount_sum
create features.event_count_90m
create features.transaction_count
create features.transaction_count_30d
create materialization_sets.amounts
create materialization_sets.events_online
create materialization_sets.transactions

Plan: 7 to add, 0 to change, 0 to delete, 0 unchanged

Run plan on every subsequent change too, to see what action is planned.

Step 7: Deploy​

Deploy the bundle to your workspace:

Bash
databricks bundle deploy

The deploy reports each resource as it creates it:

Text
Uploading bundle files to /Workspace/Users/someone@example.com/.bundle/feature-store-quickstart/default/files...
Created features.transaction_count_30d
Created features.transaction_count
Created features.amount_sum
Created features.event_count_90m
Created materialization_sets.transactions
Created materialization_sets.events_online
Created materialization_sets.amounts
Files: 4 uploaded, 0 deleted
Resources: 7 created, 0 changed, 0 deleted, 0 unchanged

Step 8: Check what is created​

databricks bundle summary -o json reports your bundle configuration with the resource identifiers, such as the materialization set ID, recorded at the last deploy. For the materialization set defined in Step 5:

JSON
{
  "id": "1234567890abcdef1234567890abcdef",
  "materialized_features": [
    {
      "cron_schedule_trigger": {
        "mode": "DERIVED"
      },
      "feature_name": "${resources.features.amount_sum.full_name}",
      "offline_store_config": {
        "catalog_name": "main",
        "schema_name": "features_demo",
        "table_name_prefix": "amount_sum_offline"
      }
    },
    {
      "cron_schedule_trigger": {
        "mode": "DERIVED"
      },
      "feature_name": "${resources.features.amount_sum.full_name}",
      "online_store_config": {
        "catalog_name": "main",
        "online_store_name": "my_online_store",
        "schema_name": "features_demo",
        "table_name_prefix": "amount_sum_online"
      }
    },
    {
      "feature_name": "${resources.features.amount_sum.full_name}",
      "online_store_config": {
        "catalog_name": "main",
        "online_store_name": "my_online_store",
        "schema_name": "features_demo",
        "table_name_prefix": "amount_sum_online"
      },
      "streaming_mode": {
        "mode": "STREAMING_MODE_TYPE_RTM"
      }
    }
  ]
}

To see what is stored on the server, read the set back with the Feature Engineering command, using the id from the databricks bundle summary output:

Bash
databricks feature-engineering get-materialization-set <materialization-set-id>

The response carries what the server stored for the same set, including the fields it generates:

JSON
{
  "materialization_set_id": "1234567890abcdef1234567890abcdef",
  "materialized_features": [
    {
      "cron_schedule_trigger": {
        "cron_expression": "0 0 2 * * ?",
        "mode": "DERIVED",
        "timezone_id": "UTC"
      },
      "feature_name": "main.features_demo.amount_sum_3d",
      "is_online": true,
      "job_id": 123456789012345,
      "materialization_set_id": "1234567890abcdef1234567890abcdef",
      "materialized_feature_id": "22222222222222222222222222222222",
      "online_store_config": {
        "catalog_name": "",
        "online_store_name": "my_online_store",
        "schema_name": "",
        "table_name_prefix": ""
      },
      "pipeline_schedule_state": "ACTIVE",
      "table_name": "main.features_demo.amount_sum_online_ceq88t"
    },
    {
      "feature_name": "main.features_demo.amount_sum_3d",
      "is_online": true,
      "job_id": 234567890123456,
      "materialization_set_id": "1234567890abcdef1234567890abcdef",
      "materialized_feature_id": "33333333333333333333333333333333",
      "online_store_config": {
        "catalog_name": "",
        "online_store_name": "my_online_store",
        "schema_name": "",
        "table_name_prefix": ""
      },
      "pipeline_id": "12345678-1234-1234-1234-123456789012",
      "pipeline_schedule_state": "ACTIVE",
      "streaming_mode": {
        "mode": "STREAMING_MODE_TYPE_RTM"
      },
      "table_name": "main.features_demo.amount_sum_online_qiohun"
    },
    {
      "cron_schedule_trigger": {
        "cron_expression": "0 0 2 * * ?",
        "mode": "DERIVED",
        "timezone_id": "UTC"
      },
      "feature_name": "main.features_demo.amount_sum_3d",
      "is_online": false,
      "job_id": 123456789012345,
      "materialization_set_id": "1234567890abcdef1234567890abcdef",
      "materialized_feature_id": "11111111111111111111111111111111",
      "pipeline_schedule_state": "ACTIVE",
      "table_name": "main.features_demo.amount_sum_offline_ceq88t"
    }
  ]
}

Step 9: Change the configuration​

The following examples change the Feature Views and materialization sets from earlier steps, and show what each change does on deployment. Unless an example names a different starting point, each one starts from the deployment in Step 7. Each one is an edit to the file that declares the resource, or to the resources mapping in databricks.yml if you declared everything in one file. In the Python tab, the edit goes in resources/__init__.py. For which changes are applied in place, which replace the resource, and which are rejected, see Changing a Feature View and Changing a materialization set.

Removing all of a Feature View's materialized features​

Removing every materialized feature of one Feature View from a materialization set is applied in place, so no feature data is recomputed. See Changing a materialization set.

The following takes the set transactions defined in Step 3 for Feature Views transaction_count and transaction_count_30d, and removes every materialized feature of Feature View transaction_count_30d. The materialized features of transaction_count in the same set are untouched:

Diff
 materialization_sets:
   transactions:
     materialized_features:
       # Batch offline materialized feature of transaction_count.
       - feature_name: ${resources.features.transaction_count.full_name}
         offline_store_config: # As in Step 3.
         cron_schedule_trigger:
           mode: DERIVED
       # Batch online materialized feature of transaction_count.
       - feature_name: ${resources.features.transaction_count.full_name}
         online_store_config: # As in Step 3.
         cron_schedule_trigger:
           mode: DERIVED
-      # Batch offline materialized feature of transaction_count_30d.
-      - feature_name: ${resources.features.transaction_count_30d.full_name}
-        offline_store_config: # As in Step 3.
-        cron_schedule_trigger:
-          mode: DERIVED
-      # Batch online materialized feature of transaction_count_30d.
-      - feature_name: ${resources.features.transaction_count_30d.full_name}
-        online_store_config: # As in Step 3.
-        cron_schedule_trigger:
-          mode: DERIVED

databricks bundle plan reports the set as an update:

Text
update materialization_sets.transactions

Plan: 0 to add, 1 to change, 0 to delete, 6 unchanged

Changing an immutable field of a Feature View​

When a materialization set depends on a Feature View, changing an immutable field of the Feature View fails on deploy. Replacing the Feature View deletes it first, and the delete is rejected while the Feature View has materialized features. For which fields are immutable, and for the recommended way to change one, see Changing a Feature View.

The following takes the Feature View transaction_count defined in Step 3 and widens its sliding window duration, which is an immutable field:

Diff
   features:
     transaction_count:
       # The other keys, as in Step 3.
       function:
         aggregation_function:
           count_function:
             input: transaction_id
           time_window:
             sliding:
-              window_duration: 604800s
+              window_duration: 1209600s
               slide_duration: 86400s

databricks bundle plan reports a recreate:

Text
recreate features.transaction_count

Plan: 1 to add, 0 to change, 1 to delete, 6 unchanged

databricks bundle deploy asks for confirmation, and its message says why the change cannot go through as written:

Text
This action will result in the deletion or recreation of the following Feature Views.
A Feature View that has materialized features in a materialization set cannot be deleted or
recreated. Remove its materialized features from the set in a separate deploy first, then delete
or recreate the Feature View. Deleting or recreating a Feature View that a FeatureSpec serves is
not blocked, and leaves the FeatureSpec stale:
  recreate resources.features.transaction_count

If you approve, the deploy fails, because the set transactions still holds the Feature View's two materialized features:

Text
Error: cannot recreate resources.features.transaction_count: deleting old id=main.features_demo.transaction_count_7d: Cannot delete feature 'main.features_demo.transaction_count_7d' because it has 2 materialized feature(s). Delete all materialized features first before deleting the feature. (400 BAD_REQUEST)

To make the change, deploy twice, as described in Changing a Feature View:

  1. Remove the Feature View's materialized features from the set, and deploy. The set is updated in place, as in Removing all of a Feature View's materialized features.

  2. Change the Feature View, add its materialized features back to the set, and deploy. The plan re-creates the Feature View for the changed window, and re-creates the set, because adding materialized features to a set replaces it:

    Text
    Warn: planning resources.materialization_sets.transactions: the member for feature "main.features_demo.transaction_count_7d" is not in the deployed set, and members cannot be added to an existing set, so the whole set will be recreated: every member is dropped and recomputed
    Warn: planning resources.materialization_sets.transactions: the member for feature "main.features_demo.transaction_count_7d" is not in the deployed set, and members cannot be added to an existing set, so the whole set will be recreated: every member is dropped and recomputed
    recreate features.transaction_count
    recreate materialization_sets.transactions

    Plan: 2 to add, 0 to change, 2 to delete, 5 unchanged

    After you approve, the deploy re-creates the Feature View and then the set:

    Text
    Recreated features.transaction_count
    Recreated materialization_sets.transactions
    Files: 3 uploaded, 0 deleted
    Resources: 2 created, 0 changed, 2 deleted, 5 unchanged

Changing an immutable field of a materialization set​

Changing a field of any materialized feature in a set replaces the whole materialization set, including the materialized features that the edit does not touch. See Changing a materialization set.

The following takes the set transactions defined in Step 3 and changes the table_name_prefix of its online destination, which is immutable. Both online materialized features carry the prefix, because the set has one online destination:

Diff
 materialization_sets:
   transactions:
     materialized_features:
       # Batch offline materialized feature of transaction_count.
       - feature_name: ${resources.features.transaction_count.full_name}
         offline_store_config: # As in Step 3.
         cron_schedule_trigger:
           mode: DERIVED
       # Batch online materialized feature of transaction_count.
       - feature_name: ${resources.features.transaction_count.full_name}
         online_store_config:
           online_store_name: ${var.online_store}
           catalog_name: ${var.catalog}
           schema_name: ${var.schema}
-          table_name_prefix: transactions_online
+          table_name_prefix: transactions_online_v2
         cron_schedule_trigger:
           mode: DERIVED
       # Batch offline materialized feature of transaction_count_30d.
       - feature_name: ${resources.features.transaction_count_30d.full_name}
         offline_store_config: # As in Step 3.
         cron_schedule_trigger:
           mode: DERIVED
       # Batch online materialized feature of transaction_count_30d.
       - feature_name: ${resources.features.transaction_count_30d.full_name}
         online_store_config:
           online_store_name: ${var.online_store}
           catalog_name: ${var.catalog}
           schema_name: ${var.schema}
-          table_name_prefix: transactions_online
+          table_name_prefix: transactions_online_v2
         cron_schedule_trigger:
           mode: DERIVED

databricks bundle plan reports a recreate of the set, so its two batch offline materialized features are dropped and recomputed as well:

Text
recreate materialization_sets.transactions

Plan: 1 to add, 0 to change, 1 to delete, 6 unchanged

Adding materialized features to an existing set​

Adding a materialized feature to a materialization set that is already deployed replaces the set, so every materialized feature in it is recomputed. See Changing a materialization set.

The following takes the set transactions after the removal of every materialized feature of Feature View transaction_count_30d in Removing all of a Feature View's materialized features, and adds the materialized features of Feature View transaction_count_30d back:

Diff
 materialization_sets:
   transactions:
     materialized_features:
       # Batch offline materialized feature of transaction_count.
       - feature_name: ${resources.features.transaction_count.full_name}
         offline_store_config: # As in Step 3.
         cron_schedule_trigger:
           mode: DERIVED
       # Batch online materialized feature of transaction_count.
       - feature_name: ${resources.features.transaction_count.full_name}
         online_store_config: # As in Step 3.
         cron_schedule_trigger:
           mode: DERIVED
+      # Batch offline materialized feature of transaction_count_30d.
+      - feature_name: ${resources.features.transaction_count_30d.full_name}
+        offline_store_config: # As in Step 3.
+        cron_schedule_trigger:
+          mode: DERIVED
+      # Batch online materialized feature of transaction_count_30d.
+      - feature_name: ${resources.features.transaction_count_30d.full_name}
+        online_store_config: # As in Step 3.
+        cron_schedule_trigger:
+          mode: DERIVED

databricks bundle plan warns about each added materialized feature and reports a recreate:

Text
Warn: planning resources.materialization_sets.transactions: the member for feature "main.features_demo.transaction_count_30d" is not in the deployed set, and members cannot be added to an existing set, so the whole set will be recreated: every member is dropped and recomputed
Warn: planning resources.materialization_sets.transactions: the member for feature "main.features_demo.transaction_count_30d" is not in the deployed set, and members cannot be added to an existing set, so the whole set will be recreated: every member is dropped and recomputed
recreate materialization_sets.transactions

Plan: 1 to add, 0 to change, 1 to delete, 6 unchanged

databricks bundle deploy asks for confirmation before it re-creates the set:

Text
This action will result in the deletion or recreation of the following materialization sets.
Every materialized feature in a recreated set is dropped and recomputed from scratch, including
the members whose configuration did not change:
  recreate resources.materialization_sets.transactions

Removing some of a Feature View's materialized features​

Removing some but not all of a Feature View's materialized features from a materialization set is rejected. See Changing a materialization set.

The following takes the set amounts defined in Step 5 for Feature View amount_sum, which has a sawtooth window, and removes only its batch offline materialized feature:

Diff
 materialization_sets:
   amounts:
     materialized_features:
-      # Batch offline materialized feature.
-      - feature_name: ${resources.features.amount_sum.full_name}
-        offline_store_config: # As in Step 5.
-        cron_schedule_trigger:
-          mode: DERIVED
       # Batch online materialized feature.
       - feature_name: ${resources.features.amount_sum.full_name}
         online_store_config: # As in Step 5.
         cron_schedule_trigger:
           mode: DERIVED
       # Streaming online materialized feature.
       - feature_name: ${resources.features.amount_sum.full_name}
         online_store_config: # As in Step 5.
         streaming_mode:
           mode: STREAMING_MODE_TYPE_RTM

The edit removes the batch offline materialized feature of amount_sum, and that deletion also deletes the other two materialized features of the same Feature View, which the edit does not ask for. The deployment is rejected:

Text
Error: cannot update resources.materialization_sets.amounts: updating id=1234567890abcdef1234567890abcdef: Deleting materialized features '11111111111111111111111111111111' of feature 'main.features_demo.amount_sum_3d' also deletes materialized features '22222222222222222222222222222222', '33333333333333333333333333333333', which the request does not delete. Delete the unrequested materialized features as well. (400 INVALID_PARAMETER_VALUE)

Step 10: Clean up​

Materialization pipelines keep running on their configured schedule until the set is deleted. Tear the bundle down when you are finished:

Bash
databricks bundle destroy

This deletes the materialization sets and the Feature Views declared in every file the bundle includes. The tables, pipelines, and jobs behind a materialized feature are removed by a background process after the last materialized feature sharing them is gone, so there is a short delay before the associated cost stops. Only the user who created a materialized feature can delete it, so run databricks bundle destroy as the same identity that deployed the bundle. See delete_materialized_feature().

For what materialization costs and how to attribute it, see Feature Store cost management.

Limitations​

  • A Stream is not a bundle resource. Create it outside the bundle and reference it by name.
  • A FeatureSpec is not a bundle resource. Create one with the Python API to serve features online. See Serve Feature Views.
  • Adding a materialized feature to an existing materialization set re-creates the set, so every materialized feature in it is recomputed.
  • A Feature View on a request_source or feature_view_source, or one using a custom_udf function, cannot be materialized. See Materialize Feature Views.

Next steps​

On this page