Skip to main content
Last updated on

Materialize and serve declarative features

Beta

This feature is Beta and is available in the following regions: us-east-1 and us-west-2.

After you have created your declarative feature definitions, which are stored in Unity Catalog, you can produce feature data from your source table using the feature definitions. This process is called materializing your features. Databricks creates and manages Lakeflow Spark Declarative Pipelines to populate tables in Unity Catalog for model training and batch scoring or online serving.

Requirements​

  • Features must be created with the declarative feature API and stored in Unity Catalog.
  • For version requirements, see Requirements.

API data structures​

OfflineStoreConfig​

Configuration for the offline store where materialized features will be written. When materialize_features is called, the feature store backend creates tables using this prefix. Each pipeline run materializes the latest feature values to the table according to materialization schedule.

Python
OfflineStoreConfig(
    catalog_name: str,        # Catalog name for the offline table where materialized features will be stored
    schema_name: str,         # Schema name for the offline table
    table_name_prefix: str    # Table name prefix for the offline table. The pipeline may create multiple tables with this prefix, each updated at different cadences
)
Python
from databricks.feature_engineering.entities import OfflineStoreConfig

offline_store = OfflineStoreConfig(
    catalog_name="main",
    schema_name="feature_store",
    table_name_prefix="customer_features"
)

OnlineStoreConfig​

Configuration for the online store, which stores features used by model serving. Materialization creates Delta tables with the catalog.schema.table_name_prefix, and streams the tables to the Online Feature Store with the same name.

Python
from databricks.feature_engineering.entities import OnlineStoreConfig

online_store = OnlineStoreConfig(
    catalog_name="main",
    schema_name="feature_store",
    table_name_prefix="customer_features_serving",
    online_store_name="customer_features_store"
)

MaterializedFeature​

Represents a declarative feature that has been materialized, that is, that has a precomputed representation available in Unity Catalog. There is a distinct materialized feature for the offline table and online table. Typically, users will not instantiate a MaterializedFeature directly.

API function calls​

materialize_features()​

Materializes a list of declarative features into either an offline Delta table or to an Online Feature Store.

Python
FeatureEngineeringClient.materialize_features(
    features: List[Feature],                                               # List of declarative features to materialize
    offline_config: OfflineStoreConfig,                                    # Offline store config if materializing offline
    online_config: Optional[OnlineStoreConfig] = None,                     # Online store config if materializing online
    pipeline_state: Union[MaterializedFeaturePipelineScheduleState, str],  # Materialization pipeline state - must be "ACTIVE"
    cron_schedule: Optional[str] = None,                                   # Materialization schedule, specified in quartz cron syntax. Currently must be provided.
) -> List[MaterializedFeature]:

The method returns a list of materialized features, which contain metadata such as cron schedule when feature values are updated and information about the Unity Catalog tables where the features are materialized.

If both an OnlineStoreConfig and an OfflineStoreConfig are provided, then two materialized features are returned per feature provided, one for each type of store.

Materialize to offline store​

Python
from databricks.feature_engineering import FeatureEngineeringClient
from databricks.feature_engineering.entities import OfflineStoreConfig

fe = FeatureEngineeringClient()

materialized = fe.materialize_features(
    features=features,
    offline_config=OfflineStoreConfig(
        catalog_name="main",
        schema_name="feature_store",
        table_name_prefix="customer_features"
    ),
    pipeline_state="ACTIVE",
    cron_schedule="0 0 * * * ?"  # Hourly
)

Materialize to online store​

note

To materialize features to an online store, you must also materialize to an offline store. Both offline_config and online_config are required.

Python
from databricks.feature_engineering import FeatureEngineeringClient
from databricks.feature_engineering.entities import OfflineStoreConfig, OnlineStoreConfig

fe = FeatureEngineeringClient()

materialized = fe.materialize_features(
    features=features,
    offline_config=OfflineStoreConfig(
        catalog_name="main",
        schema_name="feature_store",
        table_name_prefix="customer_features"
    ),
    online_config=OnlineStoreConfig(
        catalog_name="main",
        schema_name="feature_store",
        table_name_prefix="customer_features_serving",
        online_store_name="customer_features_store"
    ),
    pipeline_state="ACTIVE",
    cron_schedule="0 0 * * * ?"  # Hourly
)

list_materialized_features()​

Returns a list of all materialized features in the user's Unity Catalog metastore.

By default, a maximum of 100 features are returned. You can change this limit using the max_results parameter.

To filter the returned materialized features by a feature name, use the optional feature_name parameter.

Python
FeatureEngineeringClient.list_materialized_features(
    feature_name: Optional[str] = None,     # Optional feature name to filter by
    max_results: int = 100,                 # Maximum number of features to be returned
) -> List[MaterializedFeature]:

How to delete a materialized feature​

To delete a materialized feature, use list_materialized_features(). Check the table_name attribute, navigate to that table in Unity Catalog, and delete the table containing the feature. Use the Lineage tab to identify any associated pipelines and delete them as well. Finally, ensure that for online tables, the offline pipeline and table are also deleted.

In beta, deletion APIs are not supported. If needed, you can manually delete feature pipelines and feature tables via the Databricks UI.

Use online features in real-time applications​

important

Feature Serving endpoints are not supported for Declarative Feature Engineering. To serve features online, deploy a model serving endpoint using a model logged through Unity Catalog.

Models that are trained using features from Databricks automatically track lineage to the features they were trained on. When deployed as model serving endpoints, these models use Unity Catalog to look up features from online stores.

Deploy a model serving endpoint​

Use the MLflow Deployments SDK to create a model serving endpoint for a model registered in Unity Catalog:

Python
import mlflow.deployments

client = mlflow.deployments.get_deploy_client("databricks")

# Create a serving endpoint for a UC model
endpoint = client.create_endpoint(
    name="fraud-detection-endpoint",
    config={
        "served_entities": [
            {
                "entity_name": "main.ecommerce.fraud_model",
                "entity_version": "1",
                "workload_size": "Small",
                "scale_to_zero_enabled": True,
            }
        ]
    },
)

For more details on configuring model serving endpoints, see Create custom model serving endpoints.

Query the endpoint​

Python
import mlflow.deployments

client = mlflow.deployments.get_deploy_client("databricks")

response = client.predict(
    endpoint="fraud-detection-endpoint",
    inputs={
        "dataframe_records": [
            {"user_id": "user_123", "transaction_time": "2026-03-01T12:00:00"},
        ]
    },
)

Limitations​

  • Continuous features cannot be materialized. Due to their high fidelity of time correctness, continuous features for offline training or batch inference are generated on the fly for each data point.
  • Deleting and pausing a feature must be manually managed at the pipeline level.