Skip to main content

Purge feature values for entities

Beta

This feature is in Beta. To use it, a workspace admin must turn on Feature Views Advanced Functionality from the Previews page. See Manage Databricks previews.

Use a feature entity purge to remove the computed feature values of one or more Feature Views for selected entities. By default, a purge sets the selected feature values to NULL in matching rows of applicable offline and online materialization tables in Unity Catalog and Lakebase. To delete matching rows, set delete_rows=True. Run the purge in the workspace where the materializations you want to purge were created. It does not delete the Feature View definitions, materializations, or source data. Materializations created in other workspaces are not affected.

For example, you can purge a customer's materialized feature values after receiving a data deletion request. To also remove the rows that contain the customer's entity keys, set delete_rows=True.

warning

Before you submit a purge, you must remove the selected entities from the feature source data and prevent the source from receiving new data for those entities. A purge does not change the source or stop scheduled and continuous materialization. If matching source data remains or arrives later, a materialization recreates the values.

Requirements​

  • The features must be registered Feature Views. Each Feature View must have at least one materialization created in the workspace where you submit the purge. To purge materializations created in different workspaces, submit a separate request in each workspace. RequestSource, FeatureViewSource, and other nonmaterializable features are not supported.
  • Specify from 1 through 500 unique features by their three-part Unity Catalog names.
  • No feature can have a backfill in progress, or a purge in its logical deletion phase, when you submit the request. See Backfill Feature Views.
  • The entities table must be a Unity Catalog Delta table with at least one column. See Prepare the entities table.
  • Use Databricks Feature Engineering client version 0.18.0 and above.
  • For row deletion with delete_rows=True, use Databricks Feature Engineering client version 0.18.3 and above.
  • For delete_rows=True, include every feature that shares each affected materialized table. A request can span multiple tables, and the requirement applies independently to each table. Validation rejects incomplete coverage with INVALID_PARAMETER_VALUE and lists the affected tables and the columns that the request doesn't cover.

Permissions​

For production, Databricks recommends running purges as the same service principal that creates and materializes the Feature Views. This principal normally already has the required privileges. If privileges have changed, verify the following requirements before submitting a purge.

To submit and complete a purge, you must have the following privileges:

  • MANAGE on every Feature View in the request.
  • SELECT on the entities table.
  • SELECT and MODIFY on each offline, partial-aggregate, and online table that the purge updates.
  • Access to the Lakebase instance or project used by the online store, including permission to read and update its online tables, and to delete rows when delete_rows=True. See Grant project permissions.
  • For a schema-registry Stream without an embedded schema, permission to read its ingestion-table metadata, such as SELECT or READ_METADATA where available. Purge validation reads the schema to resolve entity-key types, not the ingestion-table rows.
  • USE CATALOG and USE SCHEMA on the parent catalog and schema of each resource listed above.

For full Unity Catalog privilege descriptions, see Unity Catalog privileges reference. For Feature View permissions, see Materialize Feature Views.

Prepare the entities table​

Create a Delta table that identifies the entities to purge. Each column must correspond to an entity key of every Feature View in the request. The column names and data types must match the Feature View entity keys. Don't include non-key columns.

The table can contain a subset of the entity keys that is a leading prefix of the Feature View's entities list. The purge matches all entities with the supplied key values, regardless of the remaining keys. Null key values are supported.

important

The entities-table columns must contain all entity keys or a leading prefix that matches the Feature View's entities list order. The order of columns in the entities table does not matter.

For existing online tables, the primary-key order can differ from the Feature View's entities list order. In that case, validation uses the online table's primary-key prefix instead and reports the required order if a partial-key request does not match.

Requested Feature Views may have different entity-key sets, as long as the entities-table columns are a subset of every Feature View's entity-key set.

The purge job resolves one Delta table version and uses that snapshot for every target. Don't modify the entities table after you submit the request. To purge entities added after the job resolves the version, submit another request.

Choose NULL values or row deletion​

By default, a purge sets the selected feature values to NULL and keeps the rows, including their entity keys and other columns. Use this mode to remove feature values while retaining the rows.

Set delete_rows=True to remove matching rows, including their entity keys. Use row deletion for a data deletion request that requires removing the entity keys as well as the feature values. Include every feature that shares each affected materialized table, as described in Requirements.

How cleanup works​

A purge has two phases:

  • Logical deletion (PENDING_PURGE and RUNNING_PURGE states): The purge deletes matching rows or sets feature values to NULL, then synchronizes the changes online. After this phase, the purge releases its lock on the features, so another purge can start during cleanup.
  • Cleanup (PENDING_CLEANUP and RUNNING_CLEANUP states): For offline tables, the purge waits at least seven days, then removes obsolete data files and transaction logs. The operation remains incomplete until cleanup finishes.

If an offline table's file or log retention is longer than seven days, the purge lowers it to seven days for the entire table, which also limits time travel on that table to seven days. Don't increase either retention setting above seven days during the operation, or cleanup fails.

The retention properties are delta.deletedFileRetentionDuration for data files and delta.logRetentionDuration for transaction logs. The purge does not restore their original values after cleanup.

For Lakebase, Databricks recommends a point-in-time restore (PITR) window of seven days or less. See Configure your restore window. Purges do not wait for Lakebase's automatic history cleanup or verify physical deletion of that history.

Submit a purge​

Call FeatureEngineeringClient.purge_feature_entities. The delete_rows argument defaults to False, so the following example sets feature values to NULL and retains the rows. The method returns a long-running operation without waiting for the purge to finish.

Python
from databricks.feature_engineering import FeatureEngineeringClient

fe = FeatureEngineeringClient()

operation = fe.purge_feature_entities(
features=[
"main.feature_store.customer_purchase_count_30d",
"main.feature_store.customer_purchase_total_30d",
],
entities_table="main.feature_store.entities_to_purge",
)

print(f"Purge operation: {operation.name()}")

To delete matching rows instead:

Python
operation = fe.purge_feature_entities(
features=[
"main.feature_store.customer_purchase_count_30d",
"main.feature_store.customer_purchase_total_30d",
],
entities_table="main.feature_store.entities_to_purge",
delete_rows=True,
)

Monitor operation states​

Use operation.done() to refresh the operation and check whether it has finished. Inspect its state with operation.metadata() or poll fe.get_operation(name=operation.name()) and read metadata["state"].

For offline purges with cleanup, the states describe these phases:

  • PENDING_PURGE: The logical deletion is waiting to start.
  • RUNNING_PURGE: The purge deletes matching rows or sets feature values to NULL and synchronizes the changes online.
  • PENDING_CLEANUP: The purge is waiting for the offline retention period to expire.
  • RUNNING_CLEANUP: The purge is cleaning up obsolete offline data files and transaction logs.

Online-only purges and runs without cleanup tasks can report PENDING or RUNNING instead. The operation finishes with SUCCEEDED, FAILED, or CANCELLED.

Check per-feature results​

Per-feature results are available after the operation finishes. In the submitting session, use operation.wait() after operation.done() returns True. To retrieve results in a later session, call fe.get_operation(name=...) with the operation name printed when you submitted the purge:

Python
from databricks.feature_engineering import FeatureEngineeringClient
from databricks.sdk.service.ml import PurgeFeatureEntitiesResponse

fe = FeatureEngineeringClient()
status = fe.get_operation(name="operations/<operation_id>")

if not status.done:
print(f"Purge is still in progress: {status.metadata['state']}")
elif status.error:
print(f"Purge operation error: {status.error}")
else:
response = PurgeFeatureEntitiesResponse.from_dict(status.response)
print(f"Purge state: {response.state}")

for result in response.results or []:
print(
result.feature,
result.offline_state,
result.online_state,
result.error,
)

The completed response contains one result per Feature View:

  • offline_state reports the outcome for offline materializations.
  • online_state reports the combined outcome for online materializations.
  • An outcome is SUCCEEDED, FAILED, or NOT_APPLICABLE.
  • An outcome is NOT_APPLICABLE when the Feature View has no materialization target for that store type. For example, an online-only Feature View has an offline outcome of NOT_APPLICABLE, and a Feature View without an online materialization has an online outcome of NOT_APPLICABLE.
  • error contains the Feature View-specific error when an applicable target fails.

Check every per-feature result. An operation can finish with different outcomes for different Feature Views or store types.

Also check response.state and response.error for an operation-level failure.

Recover from partial failure​

If a purge partially fails, inspect the per-feature errors and correct any access or configuration problems. Then submit a new purge request with a new request_id for the Feature Views with failed outcomes. For row deletion, include every feature sharing each affected materialized table, even if some of those features succeeded in the previous request. Keep the source data clean before retrying. If failures persist, contact Databricks support.

Retry a request safely​

Set request_id to a UUID4 value and retain it until the request is accepted. If a network or client failure makes the submission result ambiguous, retry the same request with the same ID. Databricks returns the existing operation instead of starting a second purge.

The feature set and entities table must match the original request. Reusing an ID with different request parameters is rejected. The client generates an ID when you omit one, but an explicit ID is required to make application-level retries idempotent. This is also useful when an outer orchestration layer might retry the operation.

Attribute purge compute costs​

Purge runs use serverless compute. With Databricks Feature Engineering client version 0.18.1 and above, use tags or budget_policy_id to attribute the compute cost:

Python
operation = fe.purge_feature_entities(
features=["main.feature_store.customer_purchase_count_30d"],
entities_table="main.feature_store.entities_to_purge",
tags={"team": "recommendations", "workflow": "privacy-deletion"},
budget_policy_id="00000000-0000-0000-0000-000000000000",
)

The tags apply to purge compute, not to the Unity Catalog Feature resources. A request supports at most 25 tags.