Supported foundation models on Mosaic AI Model Serving
This article describes the foundation models you can serve using Mosaic AI Model Serving.
Foundation models are large, pre-trained neural networks that are trained on both large and broad ranges of data. These models are designed to learn general patterns in language, images, or other data types, and can be fine-tuned for specific tasks with additional training. Your use of certain foundation models is subject to the model developer's license and acceptable use policy. See Applicable model developer licenses and terms.
Model Serving offers flexible options for hosting and querying foundation models based on your needs:
- AI Functions optimized models: A subset of Databricks-hosted models are optimized for AI Functions. You can apply AI to your data and run batch inference production workloads at scale using these functions and their supported models.
- Pay-per-token: Ideal for experimentation and quick exploration. This option allows you to query pre-configured endpoints in your Databricks workspace without upfront infrastructure commitments.
- Provisioned throughput: Recommended for production use cases requiring performance guarantees. This option enables the deployment of fine-tuned foundation models with optimized serving endpoints.
- External models: This option enables access to foundation models hosted outside of Databricks, such as those provided by OpenAI or Anthropic. These models can be centrally managed within Databricks for streamlined governance.
Foundation models hosted on Databricks
Databricks hosts state-of-the-art open foundation models, like Meta Llama. These models are made available using Foundation Model APIs.
The following table summarizes which Databricks-hosted model and model families are supported in each region based on the Model Serving feature.
Region | AI Functions optimized models | Foundation Model APIs pay-per-token | Foundation Model APIs provisioned throughput |
|---|---|---|---|
| Not supported | Not supported | Not supported |
| The following models are supported: | The following models are supported: | The following model families are supported:
|
| The following models are supported: | The following models are supported: | The following model families are supported:
|
| Not supported | Not supported | Not supported |
| Not supported | Not supported | Not supported |
| The following models are supported: | The following models are supported: | The following model families are supported:
|
| The following models are supported: | The following models are supported: | The following model families are supported:
|
| Not supported | Not supported | Not supported |
| The following models are supported: | The following models are supported: | The following model families are supported:
|
| The following models are supported: | The following models are supported: | The following model families are supported:
|
| The following models are supported: | The following models are supported: | The following model families are supported:
|
| The following models are supported: | The following models are supported: | The following model families are supported:
|
| Not supported | Not supported | Not supported |
* This model is supported based on GPU availability and requires cross geography routing to be enabled.
Access foundation models hosted outside of Databricks
Foundation models created by LLM providers, such as OpenAI and Anthropic, are also accessible on Databricks using External models. These models are hosted outside of Databricks and you can create an endpoint to query them. These endpoints can be centrally governed from Databricks, which streamlines the use and management of various LLM providers within your organization.
The following table presents a non-exhaustive list of supported models and corresponding endpoint types. You can use the listed model associations to help you configure your an endpoint for any newly released model types as they become available with a given provider. Customers are responsible for ensuring compliance with applicable model licenses.
With the rapid development of LLMs, there is no guarantee that this list is up to date at all times. New model versions from the same provider are typically supported even if they are not on the list.
Model provider | llm/v1/completions | llm/v1/chat | llm/v1/embeddings |
|---|---|---|---|
OpenAI** |
|
|
|
Azure OpenAI** |
|
|
|
Anthropic |
|
| |
Cohere** |
|
|
|
Mosaic AI Model Serving | Databricks serving endpoint | Databricks serving endpoint | Databricks serving endpoint |
Amazon Bedrock | Anthropic:
Cohere:
AI21 Labs:
| Anthropic:
Cohere:
Amazon:
| Amazon:
Cohere:
|
AI21 Labs |
| ||
Google Cloud Vertex AI | text-bison |
|
|
** Model provider supports fine-tuned completion and chat models. To query a fine-tuned model, populate the name field of the external model configuration with the name of your fine-tuned model.
† Model provider supports custom completion models.
Create foundation model serving endpoints
To query and use foundation models in your AI applications, you must first create a model serving endpoint. Model Serving uses a unified API and UI for creating and updating foundation model serving endpoints.
- To create an endpoint that serves fine-tuned variants of foundation models made available using Foundation Model APIs provisioned throughput, see Create your provisioned throughput endpoint using the REST API.
- For creating serving endpoints that access foundation models made available using the External models offering, see Create an external model serving endpoint.
Query foundation model serving endpoints
After you create your serving endpoint you are able to query your foundation model. Model Serving uses a unified OpenAI-compatible API and SDK for querying foundation models. This unified experience simplifies how you experiment with and customize foundation models for production across supported clouds and providers.