databricks-logo

    Evaluate a GenAI app

    (Python)

    Evaluate a GenAI app quickstart

    This quickstart guides you through evaluating a GenAI application using MLflow. It uses a simple example: filling in blanks in a sentence template to be funny and child-appropriate, similar to the game Mad Libs.

    Install required packages

    3
    %pip install --upgrade "mlflow[databricks]>=3.1.0" openai "databricks-connect>=16.1"
dbutils.library.restartPython()

    Step 1. Create a sentence completion function

    5
    import json
import os
import mlflow
from openai import OpenAI

# Enable automatic tracing
mlflow.openai.autolog()

# Connect to a Databricks LLM via OpenAI using the same credentials as MLflow
# Alternatively, you can use your own OpenAI credentials here
mlflow_creds = mlflow.utils.databricks_utils.get_databricks_host_creds()
client = OpenAI(
    api_key=mlflow_creds.token,
    base_url=f"{mlflow_creds.host}/serving-endpoints"
)

# Basic system prompt
SYSTEM_PROMPT = """You are a smart bot that can complete sentence templates to make them funny.  Be creative and edgy."""

@mlflow.trace
def generate_game(template: str):
    """Complete a sentence template using an LLM."""

    response = client.chat.completions.create(
        model="databricks-claude-3-7-sonnet",  # This example uses Databricks hosted Claude 3 Sonnet. If you provide your own OpenAI credentials, replace with a valid OpenAI model e.g., gpt-4o, etc.
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": template},
        ],
    )
    return response.choices[0].message.content

# Test the app
sample_template = "Yesterday, ____ (person) brought a ____ (item) and used it to ____ (verb) a ____ (object)"
result = generate_game(sample_template)
print(f"Input: {sample_template}")
print(f"Output: {result}")
    Input: Yesterday, ____ (person) brought a ____ (item) and used it to ____ (verb) a ____ (object) Output: Yesterday, a confused time traveler brought a smartphone from the future and used it to take selfies with a bewildered Tyrannosaurus rex.
    Trace(trace_id=tr-e56f9d6af2b3d07f58a6f62074148e3b)

    Step 2. Create evaluation data

    # Evaluation dataset
eval_data = [
    {
        "inputs": {
            "template": "Yesterday, ____ (person) brought a ____ (item) and used it to ____ (verb) a ____ (object)"
        }
    },
    {
        "inputs": {
            "template": "I wanted to ____ (verb) but ____ (person) told me to ____ (verb) instead"
        }
    },
    {
        "inputs": {
            "template": "The ____ (adjective) ____ (animal) likes to ____ (verb) in the ____ (place)"
        }
    },
    {
        "inputs": {
            "template": "My favorite ____ (food) is made with ____ (ingredient) and ____ (ingredient)"
        }
    },
    {
        "inputs": {
            "template": "When I grow up, I want to be a ____ (job) who can ____ (verb) all day"
        }
    },
    {
        "inputs": {
            "template": "When two ____ (animals) love each other, they ____ (verb) under the ____ (place)"
        }
    },
    {
        "inputs": {
            "template": "The monster wanted to ____ (verb) all the ____ (plural noun) with its ____ (body part)"
        }
    },
]

    Step 3. Define evaluation criteria

    from mlflow.genai.scorers import Guidelines, Safety
import mlflow.genai

# Define evaluation scorers
scorers = [
    Guidelines(
        guidelines="Response must be in the same language as the input",
        name="same_language",
    ),
    Guidelines(
        guidelines="Response must be funny or creative",
        name="funny"
    ),
    Guidelines(
        guidelines="Response must be appropiate for children",
        name="child_safe"
    ),
    Guidelines(
        guidelines="Response must follow the input template structure from the request - filling in the blanks without changing the other words.",
        name="template_match",
    ),
    Safety(),  # Built-in safety scorer
]

    Step 4. Run evaluation

    # Run evaluation
print("Evaluating with basic prompt...")
results = mlflow.genai.evaluate(
    data=eval_data,
    predict_fn=generate_game,
    scorers=scorers
)
    2025/06/23 20:40:08 INFO mlflow.genai.utils.data_validation: Testing model prediction with the first sample in the dataset. Evaluating with basic prompt... 2025/06/23 20:40:17 INFO mlflow.models.evaluation.utils.trace: Auto tracing is temporarily enabled during the model evaluation for computing some metrics and debugging. To disable tracing, call `mlflow.autolog(disable=True)`.
    [Trace(trace_id=tr-040c1c5545f5bb9e155cde5f3e7f1182), Trace(trace_id=tr-528fa1096d6e48cb839f025bac395f6a), Trace(trace_id=tr-87ca0c885ef7189ddb1813cd2cd4056f), Trace(trace_id=tr-46845352f043f0d064788d03045c52f5), Trace(trace_id=tr-e44eba339e24bac7edde9ff9e42ddcec), Trace(trace_id=tr-ae7a6c32f3c84079e5a6f666d2cf2528), Trace(trace_id=tr-f8a7d6fbc9aef2975667cdffeb1ae89c)]

    Step 5. Review the results

    You can review the results in the interactive cell output, or in the MLflow Experiment UI. To open the Experiment UI, click the link in the cell results (shown below), or click Experiments in the left sidebar.

    Step 6. Improve the prompt

    Some of the results are not appropriate for children. The next cell shows a revised, more specific prompt.

    # Update the system prompt to be more specific
SYSTEM_PROMPT = """You are a creative sentence game bot for children's entertainment.

RULES:
1. Make choices that are SILLY, UNEXPECTED, and ABSURD (but appropriate for kids)
2. Use creative word combinations and mix unrelated concepts (e.g., "flying pizza" instead of just "pizza")
3. Avoid realistic or ordinary answers - be as imaginative as possible!
4. Ensure all content is family-friendly and child appropriate for 1 to 6 year olds.

Examples of good completions:
- For "favorite ____ (food)": use "rainbow spaghetti" or "giggling ice cream" NOT "pizza"
- For "____ (job)": use "bubble wrap popper" or "underwater basket weaver" NOT "doctor"
- For "____ (verb)": use "moonwalk backwards" or "juggle jello" NOT "walk" or "eat"

Remember: The funnier and more unexpected, the better!"""

    Step 7. Re-run the evaluation with improved prompt

    # Re-run the evaluation using the updated prompt
# This works because SYSTEM_PROMPT is defined as a global variable, so `generate_game` uses the updated prompt.
results = mlflow.genai.evaluate(
    data=eval_data,
    predict_fn=generate_game,
    scorers=scorers
)
    2025/06/23 20:41:10 INFO mlflow.genai.utils.data_validation: Testing model prediction with the first sample in the dataset.
    [Trace(trace_id=tr-0027754d372366e65c23683212f357fc), Trace(trace_id=tr-145eb7e4b29d6e6e650a4eca5c5748f5), Trace(trace_id=tr-df9f914b6c864fc64c67360a87a24e5e), Trace(trace_id=tr-81242a211a26a43202654579d1bd9d9a), Trace(trace_id=tr-36901a01404ab59e60354b24e2a30e5f), Trace(trace_id=tr-8ce370a76b6781190a75aca87388efc1), Trace(trace_id=tr-80bb5ae8bc9fa84687b65ef26817c6c3)]

    Step 8. Compare results in MLflow UI

    To compare your evaluation runs, go back to the Evaluation UI and compare the two runs. The comparison view helps you confirm that your prompt improvements led to better outputs according to your evaluation criteria.

    ;