Model Serving Query

View as Markdown

Use real-time inference for machine learning

Query GA

POST /serving-endpoints/{name}/invocations

Query a serving endpoint

API scopes: model-serving, model-serving-inference

Parameters

namestringpath: The name of the serving endpoint. This field is required and is provided via the path parameter.

Request body

promptobjectPublic Preview: The prompt string (or array of strings) field used ONLY for completions external & foundation model serving endpoints and should only be used with other completions query fields.

inputobjectPublic Preview: The input string (or array of strings) field used ONLY for embeddings external & foundation model serving endpoints and is the only field (along with extra_params if needed) used by embeddings queries.

messagesarray of objectPublic Preview

The messages field used ONLY for chat external & foundation model serving endpoints. This is an array of ChatMessage objects and should only be used with other chat query fields.

Show child attributesHide child attributes

rolestringPublic Preview: The role of the message. One of [system, user, assistant].
Values: CHAT_MESSAGE_ROLE_UNSPECIFIED, SYSTEM, USER, ASSISTANT

contentstringPublic Preview: The content of the message.
Example: What is MLflow?

temperaturedoublePublic Preview: The temperature field used ONLY for completions and chat external & foundation model serving endpoints. This is a float between 0.0 and 2.0 with a default of 1.0 and should only be used with other chat/completions query fields.
Example: 0.5

stoparray of stringPublic Preview: The stop sequences field used ONLY for completions and chat external & foundation model serving endpoints. This is a list of strings and should only be used with other chat/completions query fields.

max_tokensint32Public Preview: The max tokens field used ONLY for completions and chat external & foundation model serving endpoints. This is an integer and should only be used with other chat/completions query fields.
Example: 100

nint32Public Preview: The n (number of candidates) field used ONLY for completions and chat external & foundation model serving endpoints. This is an integer between 1 and 5 with a default of 1 and should only be used with other chat/completions query fields.
Example: 5

streambooleanPublic Preview: The stream field used ONLY for completions and chat external & foundation model serving endpoints. This is a boolean defaulting to false and should only be used with other chat/completions query fields.
Example: true

extra_paramsobjectPublic Preview: The extra parameters field used ONLY for completions, chat, and embeddings external & foundation model serving endpoints. This is a map of strings and should only be used with other external/foundation model query fields.

dataframe_recordsarray of object: Pandas Dataframe input in the records orientation.

dataframe_splitobject

Pandas Dataframe input in the split orientation.

Show child attributesHide child attributes

indexarray of int32: Index array for the dataframe

columnsarray of object: Columns array for the dataframe

dataarray of object: Data array for the dataframe

instancesarray of object: Tensor-based input in row format.

inputsobject: Tensor-based input in columnar format.

client_request_idstring: Optional user-provided request identifier that will be recorded in the inference table and the usage tracking table.

usage_contextobject: Optional user-provided context that will be recorded in the usage tracking table.

Response

choicesarray of objectPublic Preview

The list of choices returned by the chat or completions external/foundation model serving endpoint.

Show child attributesHide child attributes

textstringPublic Preview: The text response from the completions endpoint.
Example: MLflow is an open source platform for managing the end-to-end machine learning lifecycle.

messageobjectPublic Preview

The message response from the chat endpoint.

Show child attributesHide child attributes

rolestringPublic Preview: The role of the message. One of [system, user, assistant].
Values: CHAT_MESSAGE_ROLE_UNSPECIFIED, SYSTEM, USER, ASSISTANT

contentstringPublic Preview: The content of the message.
Example: What is MLflow?

indexint32Public Preview: The index of the choice in the chat or completions response.
Example: 0

finishReasonstringPublic Preview: The finish reason returned by the endpoint.
Example: stop

logprobsint32Public Preview: The logprobs returned only by the completions endpoint.
Example: 5

dataarray of objectPublic Preview

The list of the embeddings returned by the embeddings external/foundation model serving endpoint.

Show child attributesHide child attributes

embeddingarray of doublePublic Preview: The embedding vector

indexint32Public Preview: The index of the embedding in the response.
Example: 0

objectstringPublic Preview: This will always be 'embedding'.
Values: EMBEDDINGS_V1_RESPONSE_EMBEDDING_ELEMENT_OBJECT_UNSPECIFIED, EMBEDDING

modelstringPublic Preview: The name of the external/foundation model used for querying. This is the name of the model that was specified in the endpoint config.
Example: gpt-4

usageobjectPublic Preview

The usage object that may be returned by the external/foundation model serving endpoint. This contains information about the number of tokens used in the prompt and response.

Show child attributesHide child attributes

prompt_tokensint32Public Preview: The number of tokens in the prompt.
Example: 5

completion_tokensint32Public Preview: The number of tokens in the chat/completions response.
Example: 5

total_tokensint32Public Preview: The total number of tokens in the prompt and response.
Example: 10

idstringPublic Preview: The ID of the query that may be returned by a completions or chat external/foundation model serving endpoint.
Example: 88fd3f75a0d24b0380ddc40484d7a31b

createdint64Public Preview: The timestamp in seconds when the query was created in Unix time returned by a completions or chat external/foundation model serving endpoint.
Example: 1699617587

objectstringPublic Preview: The type of object returned by the external/foundation model serving endpoint, one of [text_completion, chat.completion, list (of embeddings)].
Values: QUERY_ENDPOINT_RESPONSE_OBJECT_UNSPECIFIED, TEXT_COMPLETION, CHAT_COMPLETION, LIST

predictionsarray of object: The predictions returned by the serving endpoint.

outputsarray of object: The outputs of the feature serving endpoint.

View as Markdown

Request

{
  "max_tokens": 100,
  "messages": [
    {
      "content": "What is mlflow?",
      "role": "user"
    }
  ],
  "temperature": 0.1
}

{
  "dataframe_split": {
    "columns": [
      "sepal length (cm)",
      "sepal width (cm)",
      "petal length (cm)",
      "petal width (cm)"
    ],
    "data": [
      [
        5.1,
        3.5,
        1.4,
        0.2
      ],
      [
        4.9,
        3,
        1.4,
        0.2
      ]
    ],
    "index": [
      0,
      1
    ]
  }
}

{
  "inputs": {
    "t1": [
      "a",
      "b"
    ],
    "t2": [
      [
        1,
        2,
        3,
        4,
        5
      ],
      [
        6,
        7,
        8,
        9,
        10
      ]
    ],
    "t3": [
      [
        [
          1,
          2
        ],
        [
          3,
          4
        ],
        [
          5,
          6
        ]
      ],
      [
        [
          7,
          8
        ],
        [
          9,
          10
        ],
        [
          11,
          12
        ]
      ]
    ]
  }
}

{
  "instances": [
    {
      "t1": "a",
      "t2": [
        1,
        2,
        3,
        4,
        5
      ],
      "t3": [
        [
          1,
          2
        ],
        [
          3,
          4
        ],
        [
          5,
          6
        ]
      ]
    },
    {
      "t1": "b",
      "t2": [
        6,
        7,
        8,
        9,
        10
      ],
      "t3": [
        [
          7,
          8
        ],
        [
          9,
          10
        ],
        [
          11,
          12
        ]
      ]
    }
  ]
}

Response

{
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "message": {
        "content": "MLflow is an open-source platform for managing the end-to-end machine learning (ML) lifecycle. It helps data scientists and ML engineers to manage and track experiments, reproduce and share results, and deploy ML models. MLflow was created by LinkedIn and is now a part of the Linux Foundation's AI and Machine Learning projects.\n\nMLflow provides a set of tools and services that enable data scientists and ML engineers to manage the entire",
        "role": "assistant"
      }
    }
  ],
  "created": 1698824353,
  "id": null,
  "model": "llama2-70b-chat",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 74,
    "prompt_tokens": 7,
    "total_tokens": 81
  }
}

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "logprobs": null,
      "text": "MLflow is an open-source platform for managing the end-to-end machine learning lifecycle. It provides tools for tracking experiments, managing and deploying models, and collaborating on projects. MLflow also supports various machine learning frameworks and languages, making it easier to work with different tools and environments. It is designed to help data scientists and machine learning engineers streamline their workflows and improve the reproducibility and scalability of their models."
    }
  ],
  "created": 1698809382,
  "id": "cmpl-8FwDGc22M13XMnRuessZ15dG622BH",
  "model": "gpt-3.5-turbo-instruct",
  "object": "text_completion",
  "usage": {
    "completion_tokens": 83,
    "prompt_tokens": 5,
    "total_tokens": 88
  }
}

{
  "data": [
    {
      "embedding": [
        -0.024203148,
        0.012129078,
        -0.025440808
      ],
      "index": 0,
      "object": "embedding"
    }
  ],
  "model": "text-embedding-ada-002-v2",
  "object": "list",
  "usage": {
    "prompt_tokens": 2,
    "total_tokens": 2
  }
}

{
  "outputs": [
    {
      "destination_id": 10,
      "distance": 8629.506027865182,
      "latitude": 47.60621,
      "longitude": -122.33207,
      "name": "Seattle, Washington",
      "user_latitude": 37,
      "user_longitude": 122.4
    }
  ]
}

Query GA​

Parameters​

Request body​

Response​

Query GA

Parameters

Request body

Response