Priority pay-per-token for Foundation Model APIs
This page describes priority pay-per-token for Foundation Model APIs pay-per-token, including how it behaves and how to send priority requests.
What is priority pay-per-token?
Priority pay-per-token, also referred to as priority mode, is a pay-per-token capability for latency-sensitive, real-time applications. When you send a request with the service_tier parameter set to "priority", Databricks admits the request ahead of standard best-effort pay-per-token traffic on the same model. This keeps availability more consistent during periods of high traffic.
Priority pay-per-token is opt-in per request, so you can send priority and standard requests to the same model. It requires no capacity commitment and is billed at a higher per-token rate than standard pay-per-token requests.
Databricks recommends priority mode when:
- You need more consistent performance and availability than standard pay-per-token, but aren't ready to commit to dedicated capacity.
- Your production applications require higher availability.
Supported models
The following pay-per-token models support priority mode. To send a priority request, use the model's endpoint name with the service_tier parameter set to "priority".
Provider | Model | Endpoint name | Notes |
|---|---|---|---|
OpenAI |
| ||
OpenAI |
| ||
OpenAI |
| ||
OpenAI |
| ||
OpenAI |
| ||
OpenAI |
| ||
OpenAI |
| ||
OpenAI |
| ||
OpenAI |
| ||
OpenAI |
| ||
OpenAI |
| ||
| Available on the global endpoint only. Requires cross geography routing. | ||
| Available on the global endpoint only. Requires cross geography routing. | ||
| Available on the global endpoint only. Requires cross geography routing. | ||
| Available on the global endpoint only. Requires cross geography routing. |
How priority pay-per-token behaves
Consider the following behavior before you use priority pay-per-token:
- Consistent performance and availability Priority pay-per-token is designed to keep availability consistent under load. It does not guarantee a specific time-to-first-token or end-to-end latency target. Priority requests target a higher availability than standard pay-per-token requests. Databricks defines availability at a tier as the number of successful requests divided by the total number of admitted requests at that tier.
- Best-effort capacity. Priority mode does not reserve capacity and there is no capacity commitment. For guaranteed capacity, use provisioned throughput.
- Fallback to standard pay-per-token. If priority capacity is fully subscribed, requests are served at standard pay-per-token availability and billed at standard pay-per-token rates.
- Per-token premium. Priority requests are billed at a higher per-token rate than standard pay-per-token requests.
Priority pay-per-token compared to provisioned throughput
Priority pay-per-token and provisioned throughput both target production workloads, but they make different trade-offs:
Consideration | Priority pay-per-token | Provisioned throughput |
|---|---|---|
Capacity | Best-effort, shared. | Dedicated, reserved capacity. |
Commitment | None. Opt in per request. | Requires a provisioned endpoint. |
Availability | More consistent than standard pay-per-token under load. | Predictable, based on reserved capacity. |
Billing | Per-token, at a premium over standard pay-per-token. | Based on provisioned model units. |
Send a priority request
To use priority mode, set the service_tier parameter to "priority" on a per-request basis. The following example uses the OpenAI client:
from databricks_openai import DatabricksOpenAI
client = DatabricksOpenAI()
response = client.chat.completions.create(
model="databricks-model-name",
messages=[
{
"role": "user",
"content": "What is a mixture of experts model?",
}
],
max_tokens=256,
service_tier="priority",
)
See Foundation model REST API reference for parameter syntax and Use foundation models for more query options.
Capacity limits
Each cluster supports a maximum total number of tokens per minute across all tenants. Databricks sets a per-tenant limit during onboarding so that a single tenant cannot consume all cluster capacity. If your workload requires more capacity than the per-tenant limit allows, request provisioned throughput.
For more Foundation Model APIs limits, see Foundation Model APIs limits and quotas.