Authentication for working with online stores

This article describes how to configure authentication for publishing feature tables to online stores and looking up features from online stores.

The table shows the authentication methods supported for each action:

Online store provider	Publish	Feature lookup in Legacy MLflow Model Serving	Feature lookup in Model Serving
Amazon DynamoDB (any version of Feature Engineering client, or Feature Store client v0.3.8 and above)	Instance profile attached to a Databricks cluster or Databricks secrets using write_secret_prefix in AmazonDynamoDBSpec	Databricks secrets using read_secret_prefix in AmazonDynamoDBSpec. Instance profiles are not supported for legacy feature lookup.	Instance profile attached to a Databricks Serving Endpoint or Databricks secrets using read_secret_prefix in AmazonDynamoDBSpec. .
Amazon Aurora (MySQL-compatible)	Databricks secrets using write_secret_prefix in AmazonRdsMySqlSpec.	Databricks secrets using read_secret_prefix in AmazonRdsMySqlSpec.	Not supported.
Amazon RDS MySQL	Databricks secrets using write_secret_prefix in AmazonRdsMySqlSpec.	Databricks secrets using read_secret_prefix in AmazonRdsMySqlSpec.	Not supported.

Authentication for publishing feature tables to online stores

To publish feature tables to an online store, you must provide write authentication.

Databricks recommends that you provide write authentication through an instance profile attached to a Databricks cluster. Alternatively, you can store credentials in Databricks secrets, and then refer to them in a write_secret_prefix when publishing.

The instance profile or IAM user should have all of the following permissions:

• dynamodb:DeleteItem

• dynamodb:DeleteTable

• dynamodb:PartiQLSelect

• dynamodb:DescribeTable

• dynamodb:PartiQLInsert

• dynamodb:GetItem

• dynamodb:CreateGlobalTable

• dynamodb:BatchGetItem

• dynamodb:UpdateTimeToLive

• dynamodb:BatchWriteItem

• dynamodb:ConditionCheckItem

• dynamodb:PutItem

• dynamodb:PartiQLUpdate

• dynamodb:Scan

• dynamodb:Query

• dynamodb:UpdateItem

• dynamodb:DescribeTimeToLive

• dynamodb:CreateTable

• dynamodb:UpdateGlobalTableSettings

• dynamodb:UpdateTable

• dynamodb:PartiQLDelete

• dynamodb:DescribeTableReplicaAutoScaling

Provide write authentication through an instance profile attached to a Databricks cluster

On clusters running Databricks Runtime 10.5 ML and above, you can use the instance profile attached to the cluster for write authentication when publishing to DynamoDB online stores.

Note

Use these steps only for write authentication when publishing to DynamoDB online stores.

1. Create an instance profile that has write permission to the online store.

2. Attach the instance profile to a Databricks cluster by following these two steps:

   1. Add the instance profile to Databricks.

   2. Launch a cluster with the instance profile.

3. Select the cluster with the attached instance profile to run the code to publish to the online store. You do not need to provide explicit secret credentials or write_secret_prefix to the online store spec.

Provide write credentials using Databricks secrets

Follow the instructions in Use Databricks secrets.

Authentication for looking up features from online stores with served MLflow models

To enable Databricks-hosted MLflow models to connect to online stores and look up feature values, you must provide read authentication.

Databricks recommends that you provide lookup authentication through an instance profile attached to a Databricks served model. Alternatively, you can store credentials in Databricks secrets, and then refer to them in a read_secret_prefix when publishing.

Provide lookup authentication through an instance profile configured to a served model

1. Create an instance profile that has write permission to the online store.

   1. Configure your Databricks serving endpoint to use instance profile. .. note:: When publishing your table, you do not have to specify a read_prefix, and any read_prefix specified is overridden with the instance profile.

Provide read credentials using Databricks secrets

Follow the instructions in Use Databricks secrets.

Use Databricks secrets for read and write authentication.

This section shows the steps to follow to set up authentication with Databricks secrets. For code examples illustrating how to use these secrets, see Publish features to an online store.

1. Create two secret scopes that contain credentials for the online store: one for read-only access (shown here as <read-scope>) and one for read-write access (shown here as <write-scope>). Alternatively, you can reuse existing secret scopes.

   If you intend to use an instance profile for write authentication (configured at Databricks cluster level), you do not need to include the <write-scope>. If you intend to use an instance profile for read authentication (configured at Databricks Serving endpoint level), you do not need to include the <read-scope>.

2. Pick a unique name for the target online store, shown here as <prefix>.

   For DynamoDB (works with any version of Feature Engineering client, and Feature Store client v0.3.8 and above), create the following secrets:

   • Access key ID for the IAM user with read-only access to the target online store: databricks secrets put-secret <read-scope> <prefix>-access-key-id

   • Secret access key for the IAM user with read-only access to the target online store: databricks secrets put-secret <read-scope> <prefix>-secret-access-key

   • Access key ID for the IAM user with read-write access to the target online store: databricks secrets put-secret <write-scope> <prefix>-access-key-id

   • Secret access key for the IAM user with read-write access to the target online store: databricks secrets put-secret <write-scope> <prefix>-secret-access-key

   For SQL stores, create the following secrets:

   • User with read-only access to the target online store: databricks secrets put-secret <read-scope> <prefix>-user

   • Password for user with read-only access to the target online store: databricks secrets put-secret <read-scope> <prefix>-password

   • User with read-write access to the target online store: databricks secrets put-secret <write-scope> <prefix>-user

   • Password for user with read-write access to the target online store: databricks secrets put-secret <write-scope> <prefix>-password

Note

There is a limit on the number of secret scopes per workspace. To avoid hitting this limit, you can define and share a single secret scope for accessing all online stores.