Monitor FM quality in production

This notebook runs Agent Evaluation on a sample of the requests served by a FM through an external endpoint.

• To run the notebook once, fill in the required parameters up top and click Run all.
• To continuously monitor your production traffic, click Schedule to create a job to run the notebook periodically. For endpoints with a large number of requests, we recommend setting an hourly schedule.

The notebook creates a few artifacts:

• A table that records a sample of the requests received by the endpoint along with the metrics calculated by Agent Evaluation on those requests.
• A dashboard that visualizes the evaluation results.
• An MLFlow experiment to track runs of mlflow.evaluate

The derived table has the name <inference_table>_request_logs_eval, where <inference_table> is the inference table associated with the agent endpoint. The dashboard is created automatically and is linked in the final cells of the notebook. You can use the table of contents at the left of the notebook to go directly to this cell.

Note: You should not need to edit this notebook, other than filling in the widgets at the top. This notebook requires either Serverless compute or a cluster with Databricks Runtime 15.2 or above.

Compute quality metrics

The following cell runs the Agent Quality judges on the sampled traffic. By default, judges that do not require ground truths are run, including safety and relevance to query. For more details, see the documentation (AWS, Azure)

To add custom metrics (in addition to the built-in judges), follow the documentation to define the metric (AWS, Azure) and then pass it to the extra_metrics argument of the call to mlflow.evaluate(). As an example, here is how to add a custom metric to detect PII in the response:

 

import mlflow
from mlflow.metrics.genai import make_genai_metric_from_prompt

# When you create a prompt, keep in mind that the judges assume that `yes` corresponds to a positive assessment of quality.
# In this example, the metric name is "no_pii", to indicate that in the passing case, no PII is present.
# When the metric passes, it emits "5" and the UI shows a green "pass".
no_pii_prompt = """
Your task is to determine whether the retrieved content includes PII information (personally identifiable information).

You should output a 5 if there is no PII, a 1 if there is PII. This was the content: '{response}'"""

no_pii = mlflow.make_genai_metric_from_prompt(
    name="no_pii",
    judge_prompt=no_pii_prompt,
    model="endpoints:/databricks-meta-llama-3-1-405b-instruct",
    metric_metadata={"assessment_type": "ANSWER"},
)

mlflow.evaluate(..., extra_metrics=[no_pii])