Authentication for working with online stores
This article describes how to configure authentication for publishing feature tables to online stores and looking up features from online stores.
The table shows the authentication methods supported for each action:
Online store provider |
Publish |
Feature lookup in Legacy MLflow Model Serving |
Feature lookup in Model Serving |
---|---|---|---|
Amazon DynamoDB (any version of Feature Engineering client, or Feature Store client v0.3.8 and above) |
Instance profile attached to a Databricks cluster or Databricks secrets using |
Databricks secrets using Instance profiles are not supported for legacy feature lookup. |
Instance profile attached to a Databricks Serving Endpoint or Databricks secrets using . |
Amazon Aurora (MySQL-compatible) |
Databricks secrets using |
Databricks secrets using |
Not supported. |
Amazon RDS MySQL |
Databricks secrets using |
Databricks secrets using |
Not supported. |
Authentication for publishing feature tables to online stores
To publish feature tables to an online store, you must provide write authentication.
Databricks recommends that you provide write authentication through an instance profile attached to a Databricks cluster. Alternatively, you can store credentials in Databricks secrets, and then refer to them in a write_secret_prefix
when publishing.
The instance profile or IAM user should have all of the following permissions:
dynamodb:DeleteItem
dynamodb:DeleteTable
dynamodb:PartiQLSelect
dynamodb:DescribeTable
dynamodb:PartiQLInsert
dynamodb:GetItem
dynamodb:CreateGlobalTable
dynamodb:BatchGetItem
dynamodb:UpdateTimeToLive
dynamodb:BatchWriteItem
dynamodb:ConditionCheckItem
dynamodb:PutItem
dynamodb:PartiQLUpdate
dynamodb:Scan
dynamodb:Query
dynamodb:UpdateItem
dynamodb:DescribeTimeToLive
dynamodb:CreateTable
dynamodb:UpdateGlobalTableSettings
dynamodb:UpdateTable
dynamodb:PartiQLDelete
dynamodb:DescribeTableReplicaAutoScaling
Provide write authentication through an instance profile attached to a Databricks cluster
On clusters running Databricks Runtime 10.5 ML and above, you can use the instance profile attached to the cluster for write authentication when publishing to DynamoDB online stores.
Note
Use these steps only for write authentication when publishing to DynamoDB online stores.
Create an instance profile that has write permission to the online store.
Attach the instance profile to a Databricks cluster by following these two steps:
Select the cluster with the attached instance profile to run the code to publish to the online store. You do not need to provide explicit secret credentials or
write_secret_prefix
to the online store spec.
Provide write credentials using Databricks secrets
Follow the instructions in Use Databricks secrets.
Authentication for looking up features from online stores with served MLflow models
To enable Databricks-hosted MLflow models to connect to online stores and look up feature values, you must provide read authentication.
Databricks recommends that you provide lookup authentication through an instance profile attached to a Databricks served model. Alternatively, you can store credentials in Databricks secrets, and then refer to them in a read_secret_prefix
when publishing.
Provide lookup authentication through an instance profile configured to a served model
Create an instance profile that has write permission to the online store.
Configure your Databricks serving endpoint to use instance profile. .. note:: When publishing your table, you do not have to specify a
read_prefix
, and anyread_prefix
specified is overridden with the instance profile.
Provide read credentials using Databricks secrets
Follow the instructions in Use Databricks secrets.
Use Databricks secrets for read and write authentication.
This section shows the steps to follow to set up authentication with Databricks secrets. For code examples illustrating how to use these secrets, see Publish features to an online store.
Create two secret scopes that contain credentials for the online store: one for read-only access (shown here as
<read-scope>
) and one for read-write access (shown here as<write-scope>
). Alternatively, you can reuse existing secret scopes.If you intend to use an instance profile for write authentication (configured at Databricks cluster level), you do not need to include the
<write-scope>
. If you intend to use an instance profile for read authentication (configured at Databricks Serving endpoint level), you do not need to include the<read-scope>
.Pick a unique name for the target online store, shown here as
<prefix>
.For DynamoDB (works with any version of Feature Engineering client, and Feature Store client v0.3.8 and above), create the following secrets:
Access key ID for the IAM user with read-only access to the target online store:
databricks secrets put-secret <read-scope> <prefix>-access-key-id
Secret access key for the IAM user with read-only access to the target online store:
databricks secrets put-secret <read-scope> <prefix>-secret-access-key
Access key ID for the IAM user with read-write access to the target online store:
databricks secrets put-secret <write-scope> <prefix>-access-key-id
Secret access key for the IAM user with read-write access to the target online store:
databricks secrets put-secret <write-scope> <prefix>-secret-access-key
For SQL stores, create the following secrets:
User with read-only access to the target online store:
databricks secrets put-secret <read-scope> <prefix>-user
Password for user with read-only access to the target online store:
databricks secrets put-secret <read-scope> <prefix>-password
User with read-write access to the target online store:
databricks secrets put-secret <write-scope> <prefix>-user
Password for user with read-write access to the target online store:
databricks secrets put-secret <write-scope> <prefix>-password
Note
There is a limit on the number of secret scopes per workspace. To avoid hitting this limit, you can define and share a single secret scope for accessing all online stores.