Configure AI Gateway on model serving endpoints
This feature is in Preview.
In this article, you learn how to configure Mosaic AI Gateway on a model serving endpoint.
Requirements
- A Databricks workspace in one of the following regions:
- An external models supported region
- A Foundation Model APIs supported region for provisioned throughput or pay-per-token.
- A model serving endpoint. You can use one of the preconfigured pay-per-token endpoints on your workspace.
Configure AI Gateway using the UI
In the AI Gateway section of the endpoint creation page, you can individually configure AI Gateway features.
The following table summarizes how to configure AI Gateway during endpoint creation using the Serving UI.
Configure AI Guardrails in the UI
The following table shows how to configure supported guardrails.
Guardrail | How to enable | Details |
|---|---|---|
Safety | Select Safety to enable safeguards to prevent your model from interacting with unsafe and harmful content. | |
Personally identifiable information (PII) detection | Select PII detection to detect PII data such as names, addresses, credit card numbers. | |
Valid topics | You can type topics directly into this field. If you have multiple entries, be sure to press enter after each topic. Alternatively, you can upload a | A maximum of 50 valid topics can be specified. Each topic cannot exceed 100 characters |
Invalid keywords | You can type topics directly into this field. If you have multiple entries, be sure to press enter after each topic. Alternatively, you can upload a | A maximum of 50 invalid keywords can be specified. Each keyword cannot exceed 100 characters. |