Skip to main content
Last updated on

Set up production monitoring

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

Production monitoring lets you automatically run MLflow 3 scorers on traces from your agents to continuously assess quality. You schedule scorers against an MLflow experiment, and the monitoring service evaluates a configurable sample of incoming traces. Results are attached as feedback to each evaluated trace.

Production monitoring includes the following:

  • Automated quality assessment using built-in or custom scorers, including multi-turn judges for evaluating entire conversations.
  • Configurable sampling rates so you can control the tradeoff between coverage and computational cost.
  • Use the same scorers in development and production to ensure consistent evaluation.
  • Continuous quality assessment with monitoring running in the background.
note

MLflow 3 production monitoring is compatible with traces logged from MLflow 2.

Prerequisites​

Before setting up production monitoring, ensure you have:

  • MLflow experiment: An MLflow experiment where traces are being logged. If no experiment is specified, the active experiment is used.
  • Instrumented production application: Your agent must log traces using MLflow Tracing. See the Production Tracing guide.
  • Defined scorers: Tested scorers that work with your application's trace format. If you used your production app as the predict_fn in mlflow.genai.evaluate() during development, your scorers are likely already compatible.
  • Serverless budget policy: If your workspace does not allow the default serverless budget policy, set a policy on the MLflow experiment before registering scorers. See Configure a serverless budget policy for an MLflow experiment.

Configure a SQL warehouse for Unity Catalog traces​

If your traces are stored in Unity Catalog, the monitoring job runs scorer queries against the Unity Catalog tables through a SQL warehouse. Configure a warehouse ID on the experiment before you register scorers, or monitoring jobs fail with an error indicating that the mlflow.monitoring.sqlWarehouseId tag is missing.

Set the SQL warehouse ID using set_databricks_monitoring_sql_warehouse_id(). This helper stores the ID in the mlflow.monitoring.sqlWarehouseId experiment tag, which is where the monitoring job reads it from:

Python
from mlflow.tracing import set_databricks_monitoring_sql_warehouse_id

# Set the SQL warehouse ID for monitoring
set_databricks_monitoring_sql_warehouse_id(
    sql_warehouse_id="<SQL_WAREHOUSE_ID>",
    experiment_id="<EXPERIMENT_ID>"  # Optional, uses active experiment if not specified
)
note

Setting the MLFLOW_TRACING_SQL_WAREHOUSE_ID environment variable in your notebook or application is not a substitute. It applies only to the process where you set it. The monitoring job runs separately and reads the warehouse ID from the experiment tag. Use set_databricks_monitoring_sql_warehouse_id() so the warehouse ID persists on the experiment.

Configuring monitoring for Unity Catalog traces requires the following workspace-level permissions:

  • CAN USE on the SQL warehouse.
  • CAN EDIT on the MLflow experiment.
  • Permission on the monitoring job (automatically granted when you register the first scorer).

The monitoring job runs under the identity of the user who first registered a scorer on the experiment. This user's permissions determine what the monitoring job can access.

Get started​

Monitoring begins the moment you start a scorer. Register a scorer with your experiment, then start it with a sampling configuration using the .register() and .start() pattern:

Python
from mlflow.genai.scorers import Safety, ScorerSamplingConfig

# Register and start a built-in judge
safety_judge = Safety().register(name="safety")
safety_judge = safety_judge.start(sampling_config=ScorerSamplingConfig(sample_rate=0.7))

This same two-step pattern works for any scorer type: built-in judges, custom judges, code-based scorers, and multi-turn judges.

For the scorer and judge types available, see Scorers and judges.

note

At any given time, at most 20 scorers can be associated with an experiment for continuous quality monitoring.

View results​

After scheduling scorers, allow 15-20 minutes for initial processing. Then:

  1. Navigate to your MLflow experiment.
  2. Open the Traces tab to see assessments attached to traces.
  3. Use the monitoring dashboards to track quality trends.

For multi-turn judges, assessments are attached to the first trace in each session. See How assessments are stored for details.

Best practices​

Sampling strategy​

  • For critical scorers such as safety and security checks, use sample_rate=1.0.

  • For expensive scorers, such as complex LLM judges, use lower sample rates (0.05-0.2).

  • For iterative improvement during development, use moderate rates (0.3-0.5).

  • Balance coverage with cost, as shown in the following examples:

    Python
    # High-priority scorers: higher sampling
    safety_judge = Safety().register(name="safety")
    safety_judge = safety_judge.start(sampling_config=ScorerSamplingConfig(sample_rate=1.0))  # 100% coverage for critical safety

    # Expensive scorers: lower sampling
    complex_scorer = ComplexCustomScorer().register(name="complex_analysis")
    complex_scorer = complex_scorer.start(sampling_config=ScorerSamplingConfig(sample_rate=0.05))  # 5% for expensive operations

Filter traces​

Use the filter_string parameter in ScorerSamplingConfig to control which traces a scorer evaluates. This uses the same filter syntax as mlflow.search_traces().

Python
from mlflow.genai.scorers import Safety, ScorerSamplingConfig

# Only evaluate traces that completed successfully
safety_judge = Safety().register(name="safety")
safety_judge = safety_judge.start(
    sampling_config=ScorerSamplingConfig(
        sample_rate=1.0,
        filter_string="attributes.status = 'OK'"
    ),
)

You can combine multiple conditions:

Python
import time

# Evaluate successful traces from the last 24 hours
one_day_ago = int((time.time() - 86400) * 1000)
safety_judge = safety_judge.start(
    sampling_config=ScorerSamplingConfig(
        sample_rate=0.5,
        filter_string=f"attributes.status = 'OK' AND attributes.timestamp_ms > {one_day_ago}"
    ),
)

Custom scorer design​

Keep custom scorers self-contained, as shown in the following example:

Python
@scorer
def well_designed_scorer(inputs, outputs):
    # All imports inside the function
    import re
    import json

    # Handle missing data gracefully
    response = outputs.get("response", "")
    if not response:
        return 0.0

    # Return consistent types
    return float(len(response) > 100)

Troubleshooting​

Scorers not running​

If scorers aren't executing, check the following:

  1. Check experiment: Ensure that traces are logged to the experiment, not to individual runs.
  2. Sampling rate: With low sample rates, it might take time to see results.
  3. Verify filter string: Ensure your filter_string matches actual traces.

Serialization issues​

Custom scorers for production monitoring are serialized so they can be executed remotely by the monitoring service. This imposes several constraints:

  • Notebook requirement: Custom @scorer functions must be defined and registered from a Databricks notebook. The serialization mechanism relies on the notebook environment.
  • Self-contained functions: All imports must be inline within the function body. References to external variables, modules, or objects defined outside the function are not captured during serialization.
  • No class-based scorers: Only @scorer decorator-based scorers can be registered. Class-based Scorer subclasses cannot be serialized for remote execution.
  • No type hints requiring imports: Type hints in the function signature that require import statements (for example, List from typing) cause serialization failures.

When you create a custom scorer, include imports in the function definition.

Python
# Avoid external dependencies
import external_library  # Outside function

@scorer
def bad_scorer(outputs):
    return external_library.process(outputs)

# Include imports in the function definition
@scorer
def good_scorer(outputs):
    import json  # Inside function
    return len(json.dumps(outputs))

# Avoid using type hints in scorer function signature that requires imports
from typing import List

@scorer
def scorer_with_bad_types(outputs: List[str]):
    return False

# Class-based scorers are not supported for production monitoring
class MyScorer(Scorer):
    name: str = "my_scorer"
    def __call__(self, outputs):
        return len(outputs) > 10

Next step: Manage production scorers

Additional resources​

Reference guides​

On this page