Skip to main content

Query a chat model

In this article, you learn how to write query requests for foundation models that are optimized for chat tasks and send them to your model serving endpoint.

The examples in this article apply to querying foundation models that are made available using either:

Requirements

Query examples

The examples in this section show how to query the Meta Llama 3.3 70B Instruct model made available by the Foundation Model APIs pay-per-token endpoint, databricks-meta-llama-3-3-70b-instruct, using the different client options.

For a batch inference example, see Perform batch LLM inference using AI Functions.

To use the OpenAI client, specify the model serving endpoint name as the model input.

Python

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
openai_client = w.serving_endpoints.get_open_ai_client()

response = openai_client.chat.completions.create(
model="databricks-meta-llama-3-3-70b-instruct",
messages=[
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "What is a mixture of experts model?",
}
],
max_tokens=256
)

To query foundation models outside of your workspace, you must use the OpenAI client directly. You also need your Databricks workspace instance to connect the OpenAI client to Databricks. The following example assumes you have a Databricks API token and openai installed on your compute.

Python

import os
import openai
from openai import OpenAI

client = OpenAI(
api_key="dapi-your-databricks-token",
base_url="https://example.staging.cloud.databricks.com/serving-endpoints"
)

response = client.chat.completions.create(
model="databricks-meta-llama-3-3-70b-instruct",
messages=[
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "What is a mixture of experts model?",
}
],
max_tokens=256
)

As an example, the following is the expected request format for a chat model when using the REST API. For external models, you can include additional parameters that are valid for a given provider and endpoint configuration. See Additional query parameters.

Bash
{
"messages": [
{
"role": "user",
"content": "What is a mixture of experts model?"
}
],
"max_tokens": 100,
"temperature": 0.1
}

The following is an expected response format for a request made using the REST API:

JSON
{
"model": "databricks-meta-llama-3-3-70b-instruct",
"choices": [
{
"message": {},
"index": 0,
"finish_reason": null
}
],
"usage": {
"prompt_tokens": 7,
"completion_tokens": 74,
"total_tokens": 81
},
"object": "chat.completion",
"id": null,
"created": 1698824353
}

Supported models

See Foundation model types for supported chat models.

Additional resources