Agent Evaluation input schema

Preview

This article explains the input schema required by Agent Evaluation to assess your application’s quality, cost, and latency.

During development, evaluation takes place offline, and an evaluation set is a required input to Agent Evaluation.
When an application is in production, all inputs to Agent Evaluation come from your inference tables or production logs.

The input schema is identical for both online and offline evaluations.

For general information about evaluation sets, see Evaluation sets.

Evaluation input schema

The following table shows Agent Evaluation’s input schema.

Column	Data type	Description	Application passed as input argument	Previously generated outputs provided
request_id	string	Unique identifier of request.	Optional	Optional
request	See Schema for request.	Input to the application to evaluate, user’s question or query. For example, “What is RAG?”.	Required	Required
expected_retrieved_context	array	Array of objects containing the expected retrieved context for the request (if the application includes a retrieval step). Array schema	Optional	Optional
expected_response	string	Ground-truth (correct) answer for the input request. See expected_response guidelines.	Optional	Optional
response	string	Response generated by the application being evaluated.	Generated by Agent Evaluation	Optional. If not provided then derived from the Trace. Either `response` or `trace` is required.
retrieved_context	array	Retrieval results generated by the retriever in the application being evaluated. If multiple retrieval steps are in the application, this is the retrieval results from the last step (chronologically in the trace). Array schema	Generated by Agent Evaluation	Optional. If not provided then derived from the provided trace.
trace	JSON string of MLflow Trace	MLflow Trace of the application’s execution on the corresponding request.	Generated by Agent Evaluation	Optional. Either `response` or `trace` is required.

`expected_response` guidelines

The ground truth expected_response should include only the minimal set of facts that is required for a correct response. If you copy a response from another source, be sure to edit the response to remove any text that is not required for an answer to be considered correct.

Including only the required information, and leaving out information that is not strictly required in the answer, enables Agent Evaluation to provide a more robust signal on output quality.

Schema for request

The request schema can be one of the following:

A plain string. This format supports single-turn conversations only.
A messages field that follows the OpenAI chat completion schema and can encode the full conversation.
A query string field for the most recent request and an optional history field that encodes previous turns of the conversation.

For multi-turn chat applications, use the second or third option above.

The following example shows all three options in the same request column of the evaluation dataset:

import pandas as pd

data = {
  "request": [

      # Plain string
      "What is the difference between reduceByKey and groupByKey in Spark?",

      # Using the `messages` field for a single- or multi-turn chat
      {
          "messages": [
              {
                  "role": "user",
                  "content": "How can you minimize data shuffling in Spark?"
              }
          ]
      },

      # Using the query and history fields for a single- or multi-turn chat
      {
          "query": "Explain broadcast variables in Spark. How do they enhance performance?",
          "history": [
              {
                  "role": "user",
                  "content": "What are broadcast variables?"
              },
              {
                  "role": "assistant",
                  "content": "Broadcast variables allow the programmer to keep a read-only variable cached on each machine."
              }
          ]
      }
  ],

  "expected_response": [
    "expected response for first question",
    "expected response for second question",
    "expected response for third question"
  ]
}

eval_dataset = pd.DataFrame(data)

Schema for arrays in evaluation input

The schema of the arrays expected_retrieved_context and retrieved_context is shown in the following table:

Column	Data type	Description	Application passed as input argument	Previously generated outputs provided
content	string	Contents of the retrieved context. String in any format, such as HTML, plain text, or Markdown.	Optional	Optional
doc_uri	string	Unique identifier (URI) of the parent document where the chunk came from.	Required	Required

Computed metrics

The columns in the following table indicate the data included in the input, and ✓ indicates that the metric is supported when that data is provided.

For details about what these metrics measure, see How quality, cost, and latency are assessed by Agent Evaluation.

Calculated metrics	`request`	`request` and `expected_response`	`request`, `expected_response`, and `expected_retrieved_context`
`response/llm_judged/relevance_to_query/rating`	✓	✓	✓
`response/llm_judged/safety/rating`	✓	✓	✓
`response/llm_judged/groundedness/rating`	✓	✓	✓
`retrieval/llm_judged/chunk_relevance_precision`	✓	✓	✓
`agent/total_token_count`	✓	✓	✓
`agent/input_token_count`	✓	✓	✓
`agent/output_token_count`	✓	✓	✓
`response/llm_judged/correctness/rating`		✓	✓
`retrieval/llm_judged/context_sufficiency/rating`		✓	✓
`retrieval/ground_truth/document_recall`			✓