# Query

Launch stage: GA

`POST /serving-endpoints/{name}/invocations`

Query a serving endpoint

API scopes: model-serving, model-serving-inference

## Path parameters

- `name` (string, optional)
  The name of the serving endpoint. This field is required and is provided via the path parameter.

## Request body

- `prompt` (object, optional, Public Preview)
  The prompt string (or array of strings) field used ONLY for __completions external & foundation model__
   serving endpoints and should only be used with other completions query fields.
- `input` (object, optional, Public Preview)
  The input string (or array of strings) field used ONLY for __embeddings external & foundation model__
   serving endpoints and is the only field (along with extra_params if needed) used by embeddings queries.
- `messages` (array of object, optional, Public Preview)
  The messages field used ONLY for __chat external & foundation model__ serving endpoints.
   This is an array of ChatMessage objects and should only be used with other chat query fields.
  - `role` (string, optional, Public Preview)
    The role of the message. One of [system, user, assistant].
    Possible values: `CHAT_MESSAGE_ROLE_UNSPECIFIED`, `SYSTEM`, `USER`, `ASSISTANT`
  - `content` (string, optional, Public Preview)
    The content of the message.
    Example: `What is MLflow?`
- `temperature` (double, optional, Public Preview)
  The temperature field used ONLY for __completions__ and __chat external & foundation model__ serving
   endpoints. This is a float between 0.0 and 2.0 with a default of 1.0 and should only be used with other
   chat/completions query fields.
  Example: `0.5`
- `stop` (array of string, optional, Public Preview)
  The stop sequences field used ONLY for __completions__ and __chat external & foundation model__ serving
   endpoints. This is a list of strings and should only be used with other chat/completions query fields.
- `max_tokens` (int32, optional, Public Preview)
  The max tokens field used ONLY for __completions__ and __chat external & foundation model__ serving
   endpoints. This is an integer and should only be used with other chat/completions query fields.
  Example: `100`
- `n` (int32, optional, Public Preview)
  The n (number of candidates) field used ONLY for __completions__ and __chat external & foundation model__
   serving endpoints. This is an integer between 1 and 5 with a default of 1 and should only be used with
   other chat/completions query fields.
  Example: `5`
- `stream` (boolean, optional, Public Preview)
  The stream field used ONLY for __completions__ and __chat external & foundation model__ serving endpoints.
   This is a boolean defaulting to false and should only be used with other chat/completions query fields.
  Example: `true`
- `extra_params` (object, optional, Public Preview)
  The extra parameters field used ONLY for __completions, chat,__ and __embeddings external & foundation
   model__ serving endpoints. This is a map of strings and should only be used with other external/foundation
   model query fields.
- `dataframe_records` (array of object, optional)
  Pandas Dataframe input in the records orientation.
- `dataframe_split` (object, optional)
  Pandas Dataframe input in the split orientation.
  - `index` (array of int32, optional)
    Index array for the dataframe
  - `columns` (array of object, optional)
    Columns array for the dataframe
  - `data` (array of object, optional)
    Data array for the dataframe
- `instances` (array of object, optional)
  Tensor-based input in row format.
- `inputs` (object, optional)
  Tensor-based input in columnar format.
- `client_request_id` (string, optional)
  Optional user-provided request identifier that will be recorded in the inference table and the usage tracking table.
- `usage_context` (object, optional)
  Optional user-provided context that will be recorded in the usage tracking table.

## Returns

- `choices` (array of object, optional, Public Preview)
  The list of choices returned by the __chat or completions external/foundation model__ serving endpoint.
  - `text` (string, optional, Public Preview)
    The text response from the __completions__ endpoint.
    Example: `MLflow is an open source platform for managing the end-to-end machine learning lifecycle.`
  - `message` (object, optional, Public Preview)
    The message response from the __chat__ endpoint.
    - `role` (string, optional, Public Preview)
      The role of the message. One of [system, user, assistant].
      Possible values: `CHAT_MESSAGE_ROLE_UNSPECIFIED`, `SYSTEM`, `USER`, `ASSISTANT`
    - `content` (string, optional, Public Preview)
      The content of the message.
      Example: `What is MLflow?`
  - `index` (int32, optional, Public Preview)
    The index of the choice in the __chat or completions__ response.
    Example: `0`
  - `finishReason` (string, optional, Public Preview)
    The finish reason returned by the endpoint.
    Example: `stop`
  - `logprobs` (int32, optional, Public Preview)
    The logprobs returned only by the __completions__ endpoint.
    Example: `5`
- `data` (array of object, optional, Public Preview)
  The list of the embeddings returned by the __embeddings external/foundation model__ serving endpoint.
  - `embedding` (array of double, optional, Public Preview)
    The embedding vector
  - `index` (int32, optional, Public Preview)
    The index of the embedding in the response.
    Example: `0`
  - `object` (string, optional, Public Preview)
    This will always be 'embedding'.
    Possible values: `EMBEDDINGS_V1_RESPONSE_EMBEDDING_ELEMENT_OBJECT_UNSPECIFIED`, `EMBEDDING`
- `model` (string, optional, Public Preview)
  The name of the __external/foundation model__ used for querying. This is the name of the model that was
   specified in the endpoint config.
  Example: `gpt-4`
- `usage` (object, optional, Public Preview)
  The usage object that may be returned by the __external/foundation model__ serving endpoint. This
   contains information about the number of tokens used in the prompt and response.
  - `prompt_tokens` (int32, optional, Public Preview)
    The number of tokens in the prompt.
    Example: `5`
  - `completion_tokens` (int32, optional, Public Preview)
    The number of tokens in the chat/completions response.
    Example: `5`
  - `total_tokens` (int32, optional, Public Preview)
    The total number of tokens in the prompt and response.
    Example: `10`
- `id` (string, optional, Public Preview)
  The ID of the query that may be returned by a __completions or chat external/foundation model__ serving endpoint.
  Example: `88fd3f75a0d24b0380ddc40484d7a31b`
- `created` (int64, optional, Public Preview)
  The timestamp in seconds when the query was created in Unix time returned by a __completions or chat external/foundation model__ serving endpoint.
  Example: `1699617587`
- `object` (string, optional, Public Preview)
  The type of object returned by the __external/foundation model__ serving endpoint, one of
   [text_completion, chat.completion, list (of embeddings)].
  Possible values: `QUERY_ENDPOINT_RESPONSE_OBJECT_UNSPECIFIED`, `TEXT_COMPLETION`, `CHAT_COMPLETION`, `LIST`
- `predictions` (array of object, optional)
  The predictions returned by the serving endpoint.
- `outputs` (array of object, optional)
  The outputs of the feature serving endpoint.

## Request

### Chat External/Foundation Model Endpoint

```json
{
  "max_tokens": 100,
  "messages": [
    {
      "content": "What is mlflow?",
      "role": "user"
    }
  ],
  "temperature": 0.1
}
```

### Chat External/Foundation Model Endpoint with chat_template_kwargs

```json
{
  "chat_template_kwargs": {
    "clear_thinking": false
  },
  "max_tokens": 100,
  "messages": [
    {
      "content": "What is mlflow?",
      "role": "user"
    }
  ],
  "temperature": 0.1
}
```

### Completions External/Foundation Model Endpoint

```json
{
  "extra_params": {
    "top_p": 0.9
  },
  "max_tokens": 100,
  "n": 1,
  "prompt": "What is mlflow?",
  "stop": [
    "Human:"
  ],
  "stream": false,
  "temperature": 0.1
}
```

### Dataframe input in split orientation

```json
{
  "dataframe_split": {
    "columns": [
      "sepal length (cm)",
      "sepal width (cm)",
      "petal length (cm)",
      "petal width (cm)"
    ],
    "data": [
      [
        5.1,
        3.5,
        1.4,
        0.2
      ],
      [
        4.9,
        3,
        1.4,
        0.2
      ]
    ],
    "index": [
      0,
      1
    ]
  }
}
```

### Embeddings External/Foundation Model Endpoint

```json
{
  "input": [
    "embedding text"
  ]
}
```

### Feature Serving Endpoint

```json
{
  "dataframe_records": [
    {
      "destination_id": 10,
      "user_latitude": 37,
      "user_longitude": 122.4
    }
  ]
}
```

### Tensor input in columnar format

```json
{
  "inputs": {
    "t1": [
      "a",
      "b"
    ],
    "t2": [
      [
        1,
        2,
        3,
        4,
        5
      ],
      [
        6,
        7,
        8,
        9,
        10
      ]
    ],
    "t3": [
      [
        [
          1,
          2
        ],
        [
          3,
          4
        ],
        [
          5,
          6
        ]
      ],
      [
        [
          7,
          8
        ],
        [
          9,
          10
        ],
        [
          11,
          12
        ]
      ]
    ]
  }
}
```

### Tensor input in row format

```json
{
  "instances": [
    {
      "t1": "a",
      "t2": [
        1,
        2,
        3,
        4,
        5
      ],
      "t3": [
        [
          1,
          2
        ],
        [
          3,
          4
        ],
        [
          5,
          6
        ]
      ]
    },
    {
      "t1": "b",
      "t2": [
        6,
        7,
        8,
        9,
        10
      ],
      "t3": [
        [
          7,
          8
        ],
        [
          9,
          10
        ],
        [
          11,
          12
        ]
      ]
    }
  ]
}
```

## Response

### Chat External/Foundation Model Endpoint

```json
{
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "message": {
        "content": "MLflow is an open-source platform for managing the end-to-end machine learning (ML) lifecycle. It helps data scientists and ML engineers to manage and track experiments, reproduce and share results, and deploy ML models. MLflow was created by LinkedIn and is now a part of the Linux Foundation's AI and Machine Learning projects.\n\nMLflow provides a set of tools and services that enable data scientists and ML engineers to manage the entire",
        "role": "assistant"
      }
    }
  ],
  "created": 1698824353,
  "id": null,
  "model": "llama2-70b-chat",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 74,
    "prompt_tokens": 7,
    "total_tokens": 81
  }
}
```

### Completions External/Foundation Model Endpoint

```json
{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "logprobs": null,
      "text": "MLflow is an open-source platform for managing the end-to-end machine learning lifecycle. It provides tools for tracking experiments, managing and deploying models, and collaborating on projects. MLflow also supports various machine learning frameworks and languages, making it easier to work with different tools and environments. It is designed to help data scientists and machine learning engineers streamline their workflows and improve the reproducibility and scalability of their models."
    }
  ],
  "created": 1698809382,
  "id": "cmpl-8FwDGc22M13XMnRuessZ15dG622BH",
  "model": "gpt-3.5-turbo-instruct",
  "object": "text_completion",
  "usage": {
    "completion_tokens": 83,
    "prompt_tokens": 5,
    "total_tokens": 88
  }
}
```

### Embeddings External/Foundation Model Endpoint

```json
{
  "data": [
    {
      "embedding": [
        -0.024203148,
        0.012129078,
        -0.025440808
      ],
      "index": 0,
      "object": "embedding"
    }
  ],
  "model": "text-embedding-ada-002-v2",
  "object": "list",
  "usage": {
    "prompt_tokens": 2,
    "total_tokens": 2
  }
}
```

### Feature Serving Endpoint

```json
{
  "outputs": [
    {
      "destination_id": 10,
      "distance": 8629.506027865182,
      "latitude": 47.60621,
      "longitude": -122.33207,
      "name": "Seattle, Washington",
      "user_latitude": 37,
      "user_longitude": 122.4
    }
  ]
}
```

### Predictions Response Example

```json
{
  "predictions": [
    0.15,
    0.83
  ]
}
```