Skip to main content
Last updated on

Backfill historical traces with scorers

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

You can retroactively apply new or updated scorers to historical traces. This is useful when you add a new scorer and want to evaluate past production data, or when you update an existing scorer and want to re-evaluate previous traces with the new configuration.

important

If you omit both start_time and end_time, the backfill covers only the last seven days. To evaluate older traces, pass an explicit start_time. See Backfill the full trace history.

Prerequisites​

  • Scorers must be registered and started before they can be used for backfill.
  • You need the scorer names or BackfillScorerConfig objects to specify which scorers to apply.

Backfill recent data​

To backfill only recent traces, specify a start_time relative to the current date:

Python
from datetime import datetime, timedelta

# Backfill last week's data with higher sample rates
one_week_ago = datetime.now() - timedelta(days=7)

job_id = backfill_scorers(
    scorers=[
        BackfillScorerConfig(scorer=safety_judge, sample_rate=0.8),
        BackfillScorerConfig(scorer=response_length, sample_rate=0.9)
    ],
    start_time=one_week_ago
)

Backfill with custom sample rates and time range​

To apply scorers with different sample rates than their current configuration, or to limit the backfill to a specific time range, use BackfillScorerConfig:

Python
from databricks.agents.scorers import backfill_scorers, BackfillScorerConfig
from datetime import datetime
from mlflow.genai.scorers import Safety, scorer, ScorerSamplingConfig

safety_judge = Safety()
safety_judge = safety_judge.register(name="safety_check")
safety_judge = safety_judge.start(sampling_config=ScorerSamplingConfig(sample_rate=0.5))

@scorer(aggregations=["mean", "min", "max"])
def response_length(outputs):
    """Measure response length in characters"""
    return len(outputs)

response_length = response_length.register(name="response_length")
response_length = response_length.start(sampling_config=ScorerSamplingConfig(sample_rate=0.5))

# Define custom sample rates for backfill
custom_scorers = [
    BackfillScorerConfig(scorer=safety_judge, sample_rate=0.8),
    BackfillScorerConfig(scorer=response_length, sample_rate=0.9)
]

job_id = backfill_scorers(
    experiment_id=YOUR_EXPERIMENT_ID,
    scorers=custom_scorers,
    start_time=datetime(2024, 6, 1),
    end_time=datetime(2024, 6, 30)
)

Backfill using current sample rates​

To apply registered scorers to historical traces using their current sample rate configuration:

Python
from databricks.agents.scorers import backfill_scorers
from mlflow.genai.scorers import Safety, scorer, ScorerSamplingConfig

safety_judge = Safety()
safety_judge = safety_judge.register(name="safety_check")
safety_judge = safety_judge.start(sampling_config=ScorerSamplingConfig(sample_rate=0.5))

@scorer(aggregations=["mean", "min", "max"])
def response_length(outputs):
    """Measure response length in characters"""
    return len(outputs)

response_length = response_length.register(name="response_length")
response_length = response_length.start(sampling_config=ScorerSamplingConfig(sample_rate=0.5))

# Use existing sample rates for specified scorers
# Without start_time or end_time, this covers only the last seven days
job_id = backfill_scorers(
    scorers=["safety_check", "response_length"]
)

Backfill the full trace history​

Because omitting both bounds limits the backfill to the last seven days, request a longer range explicitly. Pass a start_time that predates your oldest trace and omit end_time, which runs the backfill through the current time:

Python
from datetime import datetime

# Backfill every trace logged since the start of 2024
job_id = backfill_scorers(
    scorers=["safety_check", "response_length"],
    start_time=datetime(2024, 1, 1)
)

start_time must be in the past, and if you pass both bounds, start_time must be earlier than end_time. A start_time earlier than your oldest retained trace is accepted, and the backfill evaluates whichever traces still exist in the experiment.

Best practices​

  • Start small. Begin with smaller time ranges to estimate job duration and resource usage.
  • Use appropriate sample rates. Consider the cost and time implications of using high sample rates on large historical datasets.

Troubleshooting​

"Scheduled scorer 'X' not found in experiment"

  • Ensure the scorer name matches a registered scorer in your experiment.
  • Check available scorers using the list_scorers() method.

Additional resources​

On this page