Skip to main content
Last updated on

Detect and monitor issues across MLflow traces

Beta

MLflow Issue Detection is in Beta. A workspace admin must enable MLflow Experiment Issues Tab from the Previews page. See Manage Databricks previews.

MLflow Issue Detection uses Genie Code to analyze traces in an MLflow experiment and find recurring quality and operational problems. It inspects inputs, outputs, spans, tool calls, timing, errors, assessments, and metadata, then groups related evidence into issues that you can investigate and fix, such as:

  • A customer-support agent invents a 30-day refund policy when answering a return request.
  • A research agent repeats the same web search five times, slowing the response and increasing tool and token costs.
  • A travel agent omits the required departure_date argument, causing the booking tool to fail.
  • An extraction agent returns prose instead of the JSON schema required by the calling application.

MLflow supports a continuous improvement cycle from discovery through production monitoring. You can review the traces behind an issue, fix and evaluate your application, and then track the issue rate after deployment.

Issue detection lifecycle: detect recurring patterns, review trace evidence, fix and evaluate the agent, and monitor the issue rate in production.

 

Start issue detection​

Open an MLflow experiment, select the Issues tab, and click Run issue detection. Genie Code starts analyzing traces in the experiment. The analysis typically takes 5 to 30 minutes, depending on the number and size of the traces. You can leave the page and return later to review the results.

tip

Use Auto-approve mode in Genie Code to prevent the analysis from pausing for tool approval.

The Issues tab with the Run issue detection button.

Alternatively, you can start the same workflow from the Traces tab by clicking Detect Issues.

Review issues​

After the analysis finishes, the Issues tab lists detected problems and shows supporting evidence for the selected issue. Each piece of evidence links to an affected source trace.

The Issues tab with detected issues, supporting evidence for a selected issue, and the Enable Monitor button.

If a finding is not applicable, update the status to Not an issue using the dropdown in the upper-right corner. MLflow learns from this feedback to avoid surfacing similar issues in later detection runs.

Fix issues with coding agents​

Each issue includes a fix prompt with the context and representative traces that a coding agent needs to investigate the problem. Copy the prompt from the Fix this issue section and paste it into a coding agent running in your application repository. The agent uses the issue evidence to inspect the traces and code, propose a focused change, and validate the fix.

Before you use the prompt:

  • Install the MLflow skills in your coding agent.
  • Make sure the coding agent has access to your Databricks workspace.

Review the proposed change before deployment. Then evaluate the updated agent against representative cases to confirm the fix and check for regressions.

Monitor a fix in production​

A fix that works in development can still fail on production traffic. An issue monitor shows whether the issue rate decreases after deployment and whether the problem returns.

Select the issue and click Enable Monitor in the Monitor section. MLflow creates a scorer for the issue and backfills it across recent traces to establish a baseline. MLflow then runs the scorer online on a sample of new traces to track the issue rate in production.

The Monitor chart showing flagged and unflagged traces and the issue rate over the last seven days.

After you deploy the fix, use the chart to compare the issue rate before and after deployment. When the issue no longer appears in production, mark it as Resolved.

Schedule detection​

Schedule issue detection to analyze new traces regularly.

Click Schedule detection on the Issues tab to choose how often MLflow scans for issues. You can limit analysis to a subset of traces by filtering by a metadata field or tag value, such as environment = production.

The Schedule issue detection dialog with cadence and trace filter settings.

Customize detection criteria​

Click Gear icon. on the Issues page to configure issue detection. You can tell MLflow which findings matter to your application, turn built-in focus areas on or off, and add custom categories for application-specific failures. For example, you can prioritize user frustration and ignore latency problems.

Access issues using the REST API​

Use the Issues REST API to integrate issue triage with external workflows. The API uses issue resource names in the form experiments/<experiment-id>/issues/<issue-id>.

Search issues​

Search supports title text, status, severity, category, tag, sorting, and pagination. The following request returns open high-severity issues with their full descriptions:

Bash
curl --request POST \
  "https://<workspace-host>/api/2.0/mlflow/experiments/<experiment-id>/issues:search" \
  --header "Authorization: Bearer <databricks-token>" \
  --header "Content-Type: application/json" \
  --data '{
    "parent": "experiments/<experiment-id>",
    "statuses": ["PENDING"],
    "severities": ["HIGH"],
    "sort_key": "LAST_SEEN_TIME",
    "sort_order": "DESCENDING",
    "page_size": 100,
    "view": "FULL"
  }'

Update an issue​

To update an issue, include an update mask. For example, the following request marks an issue as resolved:

Bash
curl --request PATCH \
  "https://<workspace-host>/api/2.0/mlflow/experiments/<experiment-id>/issues/<issue-id>?update_mask=status" \
  --header "Authorization: Bearer <databricks-token>" \
  --header "Content-Type: application/json" \
  --data '{"status": "RESOLVED"}'

Cost attribution​

Manual and scheduled issue detection runs use Genie Code. For pricing information, see Genie pricing.

Additional resources​

On this page