Core concepts for agent observability
Agent observability and quality is built around one repeating loop: find an issue in your agent, measure it systematically, fix it, confirm the fix, and monitor so it doesn't recur. Every concept on this page is a piece of that loop.
This page explains each piece and how they relate.
The quality loop
Each section below maps to a stage in this loop.
Traces and spans
A trace is a structured record of one complete execution of your agent — the top-level inputs and outputs, every intermediate step (LLM calls, retriever lookups, tool invocations), latency, and any errors. Traces are the raw material that feeds every other stage in the loop.
Internally, a trace is a tree of spans. Each span represents one operation: it records what went in, what came out, how long it took, and any metadata. The root span represents the full agent request; child spans represent sub-operations that occurred during that request.
Traces are stored in Unity Catalog as Delta tables (recommended). That gives you governed access, SQL queryability, and no per-experiment storage cap — the same governance model as any other UC table. Once stored, you can explore traces in the UI, query them with SQL, or ask questions in natural language using Genie Code.
Related: Instrument your agent · View traces in the UI · Search and query traces
Sessions
A session is a tag that groups related traces from one multi-turn conversation: set mlflow.trace.session on each trace. Session grouping lets you evaluate conversation quality end-to-end, not just turn by turn.
Assessments: feedback and expectations
An assessment is a quality measurement attached to a trace. There are two kinds:
Type | Who adds it | What it captures |
|---|---|---|
Feedback | Scorers, end users, domain experts | A quality judgment: pass/fail, numeric score, or free-text comment. Examples: a |
Expectations | Domain experts | The correct output for a given input — ground truth. Examples: the expected answer to a question; the required facts a response must contain. |
Feedback is the primary signal in routine quality work; you don't need expectations to get value from MLflow. Expectations are optional but enable the most precise scoring: judges like Correctness compare the agent's response against a known-good answer.
Related: Dev annotations · Expert feedback · Collect user feedback
Scorers
A scorer is a function that evaluates a trace and returns a feedback assessment. Every scorer follows the same contract: receive a trace → extract relevant fields → evaluate → return Feedback. The same scorer works in offline evaluation and in live production monitoring — you write quality logic once and it applies everywhere.
Types of scorers
Type | How it evaluates | Use when |
|---|---|---|
Built-in LLM judges | Pre-built LLM-powered evaluators for common dimensions: correctness, relevance, safety, groundedness, guideline adherence, and more. | You want immediate quality coverage without writing custom code. |
Custom LLM judges | An LLM judge with a custom evaluation prompt and scoring rubric you define. | Built-in judges don't cover your domain-specific criteria, or you need fine-grained scoring (numeric grades, custom categories). |
Code-based scorers | A deterministic Python function — exact match, format validation, latency checks, business rules. | You need precise, reproducible logic that doesn't require a language model. |
Judges vs. scorers
Judges (for example, mlflow.genai.judges.is_correct) evaluate text based on specific criteria — but they don't know how to read a trace. Scorers are the adapters: they extract the relevant fields from a trace (request, response, retrieved context, …) and pass them to a judge or to custom logic. When you use a built-in judge directly in scorers=[Correctness()], MLflow wraps it in a scorer automatically.
Related: Scorers overview · Built-in judges · Custom LLM judges · Code-based scorers
Evaluation datasets and runs
Evaluation datasets
An evaluation dataset is a versioned, curated collection of test cases. Each record has:
inputs: what to send to the agentexpectations(optional): the correct output, used by scorers that need ground truth
You build datasets by selecting representative traces from production or development, writing cases from scratch, or importing from external sources. Datasets are versioned so you can track how your test suite grows over time.
Related: Build an evaluation dataset
Evaluation runs
An evaluation run is the result of calling mlflow.genai.evaluate(). Give it a dataset and a list of scorers; it:
- Runs your agent on every input in the dataset, capturing traces.
- Applies each scorer to every trace, producing feedback assessments.
- Stores aggregated pass rates and metrics alongside the individual traces.
Use evaluation runs to answer: did this change improve quality? and did it regress anything else? Compare runs side by side to track progress across iterations. Evaluation runs are a special type of MLflow Run and are queryable programmatically.
Related: Run an evaluation · How evaluation works
Production monitoring
Production monitoring schedules scorers to run automatically on live agent traffic. Attach a scorer to your deployed agent and the monitoring service scores incoming traces on a rolling basis, writing feedback assessments back to the same trace.
The feedback format is identical to offline evaluation, so quality trends are directly comparable between dev and prod. Production monitoring closes the loop: it surfaces new failures in live traffic, which you then curate into your evaluation dataset to fix in the next iteration.
Related: Production monitoring overview · Start and configure monitoring
How it all fits together
Each concept serves a specific stage of the quality loop:
Stage | Key concept(s) |
|---|---|
Trace your agent | Traces, spans |
Find an issue | Traces, sessions, UI and SQL querying, natural-language analysis via Genie Code |
Collect feedback and curate a dataset | Assessments (feedback + expectations), evaluation datasets |
Write a scorer, fix the agent, and evaluate the fix | Scorers, evaluation runs |
Monitor production | Production monitoring, scheduled scorers |
The loop connects them: production monitoring identifies new failure cases → you curate them into a dataset → you write or tune a scorer → you evaluate to confirm the fix → the improved agent goes to production → monitoring watches for the next issue.
Next steps
- Instrument your agent — add tracing to your agent in a few lines
- Observe and find issues — explore and analyze captured traces
- Build an evaluation dataset — curate traces into structured test cases
- Run an evaluation — score your agent against a dataset
- Set up production monitoring — detect quality regressions in live traffic