Manage production scorers

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

After you set up production monitoring, you can manage your scorers throughout their lifecycle. This page covers how to list, update, stop, restart, and delete scorers.

For the full API parameter reference, see Manage production scorers.

Scorer lifecycle

Scorer lifecycles are centered around MLflow experiments. Scorers are immutable: each lifecycle operation returns a new scorer instance rather than modifying the original.

State	Description	API
Unregistered	Scorer function is defined but not known to the server.
Registered	Scorer is registered to the active MLflow experiment.	`.register()`
Active	Scorer is running with a sample rate > 0.	`.start()`
Stopped	Scorer is registered but not running (sample rate = 0).	`.stop()`
Deleted	The scorer has been removed from the server and is no longer associated with the experiment.	`delete_scorer()`

State	Description	API
Unregistered	Scorer function is defined but not known to the server.
Registered	Scorer is registered to the active MLflow experiment.	`.register()`
Active	Scorer is running with a sample rate > 0.	`.start()`
Stopped	Scorer is registered but not running (sample rate = 0).	`.stop()`
Deleted	The scorer has been removed from the server and is no longer associated with the experiment.	`delete_scorer()`

Lifecycle example

The following example demonstrates a scorer moving through all lifecycle states:

Python
from mlflow.genai.scorers import Safety, scorer, ScorerSamplingConfig, delete_scorer

# Register → Start → Update → Stop → Delete
safety_judge = Safety().register(name="safety_check")
safety_judge = safety_judge.start(
    sampling_config=ScorerSamplingConfig(sample_rate=1.0),
)
safety_judge = safety_judge.update(
    sampling_config=ScorerSamplingConfig(sample_rate=0.8),
)
safety_judge = safety_judge.stop()
delete_scorer(name="safety_check")

Manage scorers

The following APIs are available to manage scorers.

API	Description	Example
`list_scorers()`	List all registered scorers for the current experiment.	List scorers
`get_scorer()`	Retrieve a registered scorer by name.	`Scorer.update()`
`Scorer.update()`	Modify the sampling configuration of an active scorer. This is an immutable operation.	`Scorer.update()`
`backfill_scorer()`	Retroactively apply new or updated metrics to historical traces.	Backfill historical traces with scorers
`delete_scorer()`	Delete a registered scorer by name.	Stop and delete scorers

API	Description	Example
`list_scorers()`	List all registered scorers for the current experiment.	List scorers
`get_scorer()`	Retrieve a registered scorer by name.	`Scorer.update()`
`Scorer.update()`	Modify the sampling configuration of an active scorer. This is an immutable operation.	`Scorer.update()`
`backfill_scorer()`	Retroactively apply new or updated metrics to historical traces.	Backfill historical traces with scorers
`delete_scorer()`	Delete a registered scorer by name.	Stop and delete scorers

List scorers

To view all registered scorers for your experiment:

Python
from mlflow.genai.scorers import list_scorers

# List all registered scorers
scorers = list_scorers()
for scorer in scorers:
    print(f"Name: {scorer.name}")
    print(f"Sample rate: {scorer.sample_rate}")
    print(f"Filter: {scorer.filter_string}")
    print("---")

Get and update a scorer

Use get_scorer() to retrieve a scorer by name, then update() to modify its configuration. Because scorers are immutable, update() returns a new instance.

Python
from mlflow.genai.scorers import get_scorer, ScorerSamplingConfig

# Get existing scorer and update its configuration (immutable operation)
safety_judge = get_scorer(name="safety_monitor")
updated_judge = safety_judge.update(sampling_config=ScorerSamplingConfig(sample_rate=0.8))

# The original scorer remains unchanged; update() returns a new scorer instance
print(f"Original sample rate: {safety_judge.sample_rate}")  # Original rate
print(f"Updated sample rate: {updated_judge.sample_rate}")   # New rate

Stop and delete scorers

Stopping a scorer sets its sample rate to 0 but keeps it registered. Deleting a scorer removes it from the server entirely.

Python
from mlflow.genai.scorers import get_scorer, delete_scorer, ScorerSamplingConfig

# Get existing scorer
databricks_scorer = get_scorer(name="databricks_mentions")

# Stop monitoring (sets sample_rate to 0, keeps scorer registered)
stopped_scorer = databricks_scorer.stop()
print(f"Sample rate after stop: {stopped_scorer.sample_rate}")  # 0

# Restart monitoring from a stopped scorer
restarted_scorer = stopped_scorer.start(sampling_config=ScorerSamplingConfig(sample_rate=0.5))

# Or remove scorer entirely from the server
delete_scorer(name=databricks_scorer.name)

Immutable updates

Scorers, including LLM Judges, are immutable objects. When you update a scorer, an updated copy is created rather than modifying the original. This immutability helps ensure that scorers meant for production are not accidentally modified.

Python
from mlflow.genai.scorers import Safety, ScorerSamplingConfig

original_judge = Safety().register(name="safety")
original_judge = original_judge.start(
   sampling_config=ScorerSamplingConfig(sample_rate=0.3),
)

# Update returns new instance
updated_judge = original_judge.update(
    sampling_config=ScorerSamplingConfig(sample_rate=0.8),
)

# Original remains unchanged
print(f"Original: {original_judge.sample_rate}")  # 0.3
print(f"Updated: {updated_judge.sample_rate}")    # 0.8

Best practices

Check the scorer state before operations using sample_rate.
Use the immutable pattern. Assign the results of .start(), .update(), .stop() to variables.
Understand the difference between .stop() (preserves registration) and delete_scorer() (removes entirely).

Scorer lifecycle API reference

Scorer instance methods

`Scorer.register()`

API Reference: Scorer.register

Python
@scorer
def custom_scorer(outputs):
    return len(str(outputs.get("response", "")))

# Register the custom scorer
my_scorer = custom_scorer.register(name="response_length")

Parameters:

name (str): Unique name for the scorer within the experiment. Defaults to the existing name of the scorer.

Returns: New Scorer instance with server registration

`Scorer.start()`

API Reference: Scorer.start

Begin online evaluation with the specified sampling configuration.

Python
from mlflow.genai.scorers import ScorerSamplingConfig

# Start monitoring with sampling
active_scorer = registered_scorer.start(
    sampling_config=ScorerSamplingConfig(
        sample_rate=0.5,
        filter_string="trace.status = 'OK'"
    ),
)

Parameters:

name (str): Name of the scorer. If not provided, defaults to the current name of the scorer.
sampling_config (ScorerSamplingConfig): Trace sampling configuration
- sample_rate (float): Fraction of traces to evaluate (0.0-1.0). Default: 1.0
- filter_string (str, optional): MLflow-compatible filter for trace selection

Returns: New Scorer instance in active state

`Scorer.update()`

API Reference: Scorer.update

Modify the sampling configuration of an active scorer. This is an immutable operation.

Python
# Update sampling rate (returns new scorer instance)
updated_scorer = active_scorer.update(
    sampling_config=ScorerSamplingConfig(
        sample_rate=0.8,
    ),
)

# Original scorer remains unchanged
print(f"Original: {active_scorer.sample_rate}")  # 0.5
print(f"Updated: {updated_scorer.sample_rate}")   # 0.8

Parameters:

name (str): Name of the scorer. If not provided, defaults to the current name of the scorer.
sampling_config (ScorerSamplingConfig): Trace sampling configuration
- sample_rate (float): Fraction of traces to evaluate (0.0-1.0). Default: 1.0
- filter_string (str, optional): MLflow-compatible filter for trace selection

Returns: New Scorer instance with updated configuration

`Scorer.stop()`

API Reference: Scorer.stop

Stop online evaluation by setting sample rate to 0. Keeps the scorer registered.

Python
# Stop monitoring but keep scorer registered
stopped_scorer = active_scorer.stop()
print(f"Sample rate: {stopped_scorer.sample_rate}")  # 0

Parameters:

name (str): Name of the scorer. If not provided, defaults to the current name of the scorer.

Returns: New Scorer instance with sample_rate=0

Scorer registry functions

`mlflow.genai.scorers.get_scorer()`

API Reference: get_scorer

Retrieve a registered scorer by name.

Python
from mlflow.genai.scorers import get_scorer

# Get existing scorer by name
existing_scorer = get_scorer(name="safety_monitor")
print(f"Current sample rate: {existing_scorer.sample_rate}")

Parameters:

name (str): Name of the registered scorer

Returns: Scorer instance

`mlflow.genai.scorers.list_scorers()`

API Reference: list_scorers

List all registered scorers for the current experiment.

Python
from mlflow.genai.scorers import list_scorers

# List all registered scorers
all_scorers = list_scorers()
for scorer in all_scorers:
    print(f"Name: {scorer._server_name}")
    print(f"Sample rate: {scorer.sample_rate}")
    print(f"Filter: {scorer.filter_string}")

Returns: List of Scorer instances

`mlflow.genai.scorers.delete_scorer()`

API Reference: delete_scorer

Delete a registered scorer by name.

Python
from mlflow.genai.scorers import delete_scorer

# Delete existing scorer by name
delete_scorer(name="safety_monitor")

Parameters:

name (str): Name of the registered scorer

Returns: None

Scorer properties

`Scorer.sample_rate`

Current sampling rate (0.0-1.0). Returns 0 for stopped scorers.

Python
print(f"Sampling {scorer.sample_rate * 100}% of traces")

`Scorer.filter_string`

Current trace filter string for MLflow trace selection.

Python
print(f"Filter: {scorer.filter_string}")

Configuration classes

`ScorerSamplingConfig`

API Reference: ScorerSamplingConfig

Data class that holds sampling configuration for a scorer.

Python
from mlflow.genai.scorers import ScorerSamplingConfig

config = ScorerSamplingConfig(
    sample_rate=0.5,
    filter_string="trace.status = 'OK'"
)

Attributes:

sample_rate (float, optional): Sampling rate between 0.0 and 1.0
filter_string (str, optional): MLflow trace filter

Metric backfill

`backfill_scorers()`

Python
from databricks.agents.scorers import backfill_scorers, BackfillScorerConfig

job_id = backfill_scorers(
    experiment_id="your-experiment-id",
    scorers=[
        BackfillScorerConfig(scorer=safety_scorer, sample_rate=0.8),
        BackfillScorerConfig(scorer=response_length, sample_rate=0.9)
    ],
    start_time=datetime(2024, 1, 1),
    end_time=datetime(2024, 1, 31)
)

Parameters:

All parameters are keyword-only.

experiment_id (str, optional): The ID of the experiment to backfill. If not provided, uses the current experiment context
scorers (Union[List[BackfillScorerConfig], List[str]], required): List of BackfillScorerConfig objects with custom sample rates (if sample_rate is not provided in BackfillScorerConfig, defaults to the registered scorer's sample rate), OR list of scorer names (strings) to use current sample rates from the experiment's scheduled scorers. Cannot be empty.
start_time (datetime, optional): Start time for backfill evaluation. If you omit this but pass end_time, the backfill starts one day before end_time
end_time (datetime, optional): End time for backfill evaluation. If you omit this but pass start_time, the backfill runs through the current time

important

If you omit both start_time and end_time, the backfill covers only the last seven days, not the full trace history. Omitting only one bound applies a different default, as described above. To evaluate older traces, pass an explicit start_time.

Returns: Job ID of the created backfill job for status tracking (str)

Next step: Agent observability recipes

Scorer lifecycle​

Lifecycle example​

Manage scorers​

List scorers​

Get and update a scorer​

Stop and delete scorers​

Immutable updates​

Best practices​

Scorer lifecycle API reference​

Scorer instance methods​

Scorer.register()​

Scorer.start()​

Scorer.update()​

Scorer.stop()​

Scorer registry functions​

mlflow.genai.scorers.get_scorer()​

mlflow.genai.scorers.list_scorers()​

mlflow.genai.scorers.delete_scorer()​

Scorer properties​

Scorer.sample_rate​

Scorer.filter_string​

Configuration classes​

ScorerSamplingConfig​

Metric backfill​

backfill_scorers()​

Scorer lifecycle

Lifecycle example

Manage scorers

List scorers

Get and update a scorer

Stop and delete scorers

Immutable updates

Best practices

Scorer lifecycle API reference

Scorer instance methods

`Scorer.register()`

`Scorer.start()`

`Scorer.update()`

`Scorer.stop()`

Scorer registry functions

`mlflow.genai.scorers.get_scorer()`

`mlflow.genai.scorers.list_scorers()`

`mlflow.genai.scorers.delete_scorer()`

Scorer properties

`Scorer.sample_rate`

`Scorer.filter_string`

Configuration classes

`ScorerSamplingConfig`

Metric backfill

`backfill_scorers()`