What is agent observability and quality?

Using a coding agent?

If you use a coding agent such as Claude Code, Cursor, VS Code, or OpenCode, install our skills so it can work with MLflow effectively:

Bash

uvx mlflow agent setup

This sets up the coding-agent tab you'll see on tasks throughout these docs. See Set up a coding agent and Set up the MLflow MCP server.

Agent observability and quality is the practice of measuring, debugging, and improving your AI agent from your first trace through production. On Databricks, you do this with MLflow. MLflow on Databricks is MLflow — the same open source product, just managed and integrated with the Databricks platform, so your traces, evaluations, and prompts live in Unity Catalog alongside the rest of your data.

Start with traces

Agent tracing captures a complete record of every agent execution. A trace includes the initial input and final output, plus every intermediate step — such as model calls, tool calls, and retrieval — with the inputs, outputs, latency, token usage, and cost for each step.

Tracing is the foundation for agent observability and quality. It lets you analyze how people use your agent and where it spends time and money. More importantly, it gives you the evidence required to improve an agent. With only an input and final response, you cannot tell which intermediate decision caused a failure. With traces, you can inspect every step, diagnose the problem, collect feedback in context, and turn representative failures into evaluation and golden datasets.

You use traces to:

• Analyze agent usage, latency, token consumption, and cost.
• Debug failures across model calls, retrieval, and tool use.
• Collect human feedback in the context of the execution being reviewed.
• Build evaluation and golden datasets from representative production failures.
• Run evaluations and monitor quality in production.

The quality loop

Improving an agent isn't a one-time step — it's a loop you run continuously. Each pass fixes one issue and surfaces the next.

1. Trace your agent. Instrument your agent so every execution is captured as a trace — the evidence every later step depends on.
2. Find an issue. Spot a failure in your development traces, or catch a regression in production.
3. Collect feedback and curate a dataset. Review the failing traces with developers, domain experts, or users. Record feedback on the trace so it stays connected to the execution being reviewed, then turn representative failures and known-good examples into an evaluation or golden dataset.
4. Write a scorer. Define a scorer (an LLM judge or code-based check) that catches the issue automatically.
5. Fix the agent. Iterate on your prompt or agent to address the root cause.
6. Evaluate the fix. Re-run evaluation against your dataset to confirm the fix — and that you didn't break anything else.
7. Monitor production. Run your scorers against live traffic so the issue doesn't recur — which surfaces the next one to fix.

The journey

Start at the top if you're new; jump to a stage if you already know where you are.

• • Instrument your agent
  • Add tracing to capture every input, step, and output. Traces are stored in Unity Catalog (recommended) — governed, SQL-queryable, and the data foundation for everything else.
• • Observe & find issues
  • Explore traces in the UI and ask questions in natural language with Genie Code — no SQL required. Cluster traces to surface recurring failure modes and find issues faster.
• • Collect feedback & build datasets
  • Gather developer and expert feedback, and turn failing traces into evaluation datasets.
• • Evaluate & improve
  • Run evaluations with built-in and custom scorers, then iterate on your prompt and agent to fix what you found.
• • Monitor in production
  • Score live traffic continuously to catch regressions and detect new issues before your users do.

Next step

Ready to start the loop? Instrument your agent to capture your first traces.