Authentication for working with online feature stores

This article describes how to configure authentication for publishing feature tables to online stores and looking up features from online stores.

Authentication for publishing feature tables to online stores

To publish feature tables to an online store, you must provide write authentication.

Databricks recommends that you provide write authentication through an instance profile attached to a Databricks cluster. Alternatively, you can store credentials in Databricks secrets, and then refer to them in a write_secret_prefix when publishing.

The instance profile or IAM user should have all of the following permissions:

  • dynamodb:DeleteItem

  • dynamodb:DeleteTable

  • dynamodb:PartiQLSelect

  • dynamodb:DescribeTable

  • dynamodb:PartiQLInsert

  • dynamodb:GetItem

  • dynamodb:CreateGlobalTable

  • dynamodb:BatchGetItem

  • dynamodb:UpdateTimeToLive

  • dynamodb:BatchWriteItem

  • dynamodb:ConditionCheckItem

  • dynamodb:PutItem

  • dynamodb:PartiQLUpdate

  • dynamodb:Scan

  • dynamodb:Query

  • dynamodb:UpdateItem

  • dynamodb:DescribeTimeToLive

  • dynamodb:CreateTable

  • dynamodb:UpdateGlobalTableSettings

  • dynamodb:UpdateTable

  • dynamodb:PartiQLDelete

  • dynamodb:DescribeTableReplicaAutoScaling

Provide write authentication through an instance profile attached to a Databricks cluster

On clusters running Databricks Runtime 10.5 ML and above, you can use the instance profile attached to the cluster for write authentication when publishing to DynamoDB online stores.

Note

Use these steps only for write authentication when publishing to DynamoDB online stores.

  1. Create an instance profile that has write permission to the online store.

  2. Attach the instance profile to a Databricks cluster by following these two steps in Secure access to S3 buckets using instance profiles:

    1. Add the instance profile to Databricks.

    2. Launch a cluster with the instance profile.

  3. Select the cluster with the attached instance profile to run the code to publish to the online store. You do not need to provide explicit secret credentials or write_secret_prefix to the online store spec.

Provide write credentials using Databricks secrets

Follow the instructions in the next section.

Authentication for looking up features from online stores with served MLflow models

To enable Databricks-hosted MLflow models to connect to online stores and look up feature values, you must provide read authentication. These credentials must be kept in Databricks secrets, and you must pass a read_secret_prefix when publishing. Follow these steps:

  1. Create two secret scopes that contain credentials for the online store: one for read-only access (shown here as <read_scope>) and one for read-write access (shown here as <write_scope>). Alternatively, you can reuse existing secret scopes.

    If you intend to use an instance profile for write authentication, you only need to create the <read_scope>.

  2. Pick a unique name for the target online store, shown here as <prefix>.

    For DynamoDB (requires Feature Store client v0.3.8 and above), create the following secrets:

    • Access key ID for the IAM user with read-only access to the target online store: databricks secrets put --scope <read_scope> --key <prefix>-access-key-id

    • Secret access key for the IAM user with read-only access to the target online store: databricks secrets put --scope <read_scope> --key <prefix>-secret-access-key

    • Access key ID for the IAM user with read-write access to the target online store: databricks secrets put --scope <write_scope> --key <prefix>-access-key-id

    • Secret access key for the IAM user with read-write access to the target online store: databricks secrets put --scope <write_scope> --key <prefix>-secret-access-key

    For SQL stores, create the following secrets:

    • User with read-only access to the target online store: databricks secrets put --scope <read_scope> --key <prefix>-user

    • Password for user with read-only access to the target online store: databricks secrets put --scope <read_scope> --key <prefix>-password

    • User with read-write access to the target online store: databricks secrets put --scope <write_scope> --key <prefix>-user

    • Password for user with read-write access to the target online store: databricks secrets put --scope <write_scope> --key <prefix>-password

Note

There is a limit on the number of secret scopes per workspace. To avoid hitting this limit, you can define and share a single secret scope for accessing all online stores.