Configure access to resources from model serving endpoints
Important
This is an experimental feature and the API definition may change.
This article describes how to configure access to external and private resources from model serving endpoints using Databricks secrets.
What are secrets-based environment variables?
With secrets-based environment variables, you can securely store credentials in a Databricks secrets scope and reference those secrets in model serving. This allows credentials to be fetched at serving time from model serving endpoints.
For example, you can pass credentials to call OpenAI and other external model endpoints or access external data storage locations directly from model serving.
Databricks recommends this feature for deploying OpenAI and LangChain MLflow model flavors to serving. It is also applicable to other SaaS models requiring credentials with the understanding that the access pattern is based on using environment variables and API keys and tokens.
Requirements
This functionality currently is only supported via the Databricks REST API.
To use this feature, you must store credentials like your API key or other tokens as a Databricks secret.
The endpoint creator must have Read access to the Databricks secrets being referenced in the configs.
Create a secret scope
During model serving, the secrets are retrieved from Databricks secrets by the secret scope and key. These get assigned to the secret environment variable names that can be used inside the model.
First, create a secret scope for this purpose. The following are CLI commands:
databricks secrets create-scope --scope my_secret_scope
You can then add your secret to a desired secret scope and key as shown below:
databricks secrets put --scope my_secret_scope --key my_secret_key
The secret information and the name of the environment variable can then be passed to your endpoint configuration during endpoint creation or as an update to the configuration of an existing endpoint.
Add secret scopes to endpoint configuration
During model serving endpoint creation and configuration updates, you are able to provide a list of secret environment variable specifications for each served model inside the API request using environment_vars field. The following example assigns the value from the secret created in the provided code to the environment variable OPENAI_API_KEY.
The following is an example for creating a serving endpoint. See Create and manage model serving endpoints
POST /api/2.0/serving-endpoints
{
"name": "endpoint-name",
"config":{
"served_models": [{
"model_name": "model-name",
"model_version": "1",
"workload_size": "Small",
"scale_to_zero_enabled": "true",
"environment_vars":{
"OPENAI_API_KEY": "{{secrets/my_secret_scope/my_secret_key}}"
}
}]
}
}
You can also update a serving endpoint, as in the following example:
PUT /api/2.0/serving-endpoints/{name}/config
{
"served_models": [{
"model_name": "model-name",
"model_version": "2",
"workload_size": "Small",
"scale_to_zero_enabled": "true",
"environment_vars":{
"OPENAI_API_KEY": "{{secrets/my_secret_scope/my_secret_key}}"
}
}]
}
After the endpoint is created or updated, model serving automatically fetches the secret key from the Databricks secrets scope and populates the environment variable for your model inference code to use.
Notebook example
See the following notebook for an example of how to configure access to external and private resources from model serving endpoints with secret-based environment variables.