Monitor in production

Coding agent quickstart

Using a coding agent? Paste this prompt:

Read the documentation at https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/monitor-in-production and set up production quality monitoring for my agent.

Optionally, install mlflow/skills for a deeper integration with MLflow.

Production monitoring continuously runs MLflow scorers on a sample of your live production traces and attaches the results as feedback, so quality problems surface automatically after deployment. It closes the loop — the same scorers you built during evaluation now watch production.

Quick overview: score your production traces

Register one built-in scorer and start it with a sampling configuration:

Python

from mlflow.genai.scorers import Safety, ScorerSamplingConfig

# Register a built-in scorer against your experiment, then start monitoring
safety = Safety().register(name="safety")
safety = safety.start(sampling_config=ScorerSamplingConfig(sample_rate=1.0))

Then view the results:

1. Navigate to the MLflow experiment.
2. Open the Traces tab to see the scorer's feedback attached to sampled traces.
3. Use the monitoring dashboards to track quality trends over time.

If your traces are stored in Unity Catalog, you must first configure a SQL warehouse for monitoring to work. See Configure a SQL warehouse for Unity Catalog traces.

The quick overview scored production traces with one built-in scorer. The subsections below cover registering multiple and custom scorers, controlling sampling and cost, and operating monitoring at scale.

Next steps

• Set up production monitoring — Set up production monitoring.
• Manage production scorers — Manage production scorers.
• Agent observability recipes — Recipes for agents.