Agent observability and quality FAQ
Q: What's the difference between managed MLflow on Databricks and open-source MLflow?
Managed MLflow on Databricks is the same open-source MLflow — tracing, evaluation, LLM judges, scorers, the prompt registry, and human feedback are all open-source APIs, so your data and code stay portable. Databricks runs and governs it for you and adds platform capabilities on top; it doesn't replace or fork the open-source project.
Capability | Open-source MLflow | Managed MLflow on Databricks |
|---|---|---|
Data model, APIs, and SDK | Yes | Yes — identical, so your data and code stay portable |
Tracing, autolog, and the trace UI | Yes | Yes |
Evaluation: | Yes | Yes |
Prompt registry | Yes | Yes |
Human feedback and labeling | Yes | Yes, plus the hosted Review App — a Chat UI and expert-labeling UI for reviewers without workspace access |
Hosting and storage | Self-hosted: you run the tracking server and choose the backend | Fully managed; traces and experiments live in Unity Catalog Delta tables — governed, SQL-queryable, no per-experiment limit |
Governance, security, and audit | Your own setup | Workspace and Unity Catalog permissions, on-behalf-of-user auth, and audit logs in system tables |
AI assistant and natural-language analysis | AI Assistant (beta) — bring your own model provider or coding agent (for example, Claude Code) | Managed Genie Code, AI/BI dashboards, and Databricks SQL over your traces — no model setup |
Production monitoring | Run scorers ad hoc yourself | Managed service that continuously scores a sample of live traffic |
CI/CD and MLOps | Core APIs and the model registry | Declarative Automation Bundles, MLOps Stacks, and deployment jobs |
Your data is always yours: the data model and APIs are fully open source, so you can export and use your MLflow data anywhere. For more, see the Managed MLflow product page.
Q: What is the latency overhead introduced by Tracing?
Traces are written asynchronously to minimize performance impact. However, tracing still adds minimal latency, particularly when the trace size is large. MLflow recommends testing your application to understand tracing latency impacts before deploying to production.
The following table provides rough estimates for latency impact by trace size:
Trace size per request | Impact to response speed latency (ms) |
|---|---|
~10 KB | ~ 1 ms |
~ 1 MB | 50 ~ 100 ms |
10 MB | 150 ms ~ |
Q: What are the rate limits and quotas for MLflow Tracing in Databricks?
When using MLflow Tracing within a Databricks workspace quotas and rate limits apply to ensure service stability and fair usage. See Resource limits.
Q: I cannot open my trace in the MLflow UI. What should I do?
There are multiple possible reasons why a trace may not be viewable in the MLflow UI.
-
The trace is not completed yet: If the trace is still being collected, MLflow cannot display spans in the UI. Ensure that all spans are properly ended with either "OK" or "ERROR" status.
-
The browser cache is outdated: When you upgrade MLflow to a new version, the browser cache may contain outdated data and prevent the UI from displaying traces correctly. Clear your browser cache (Shift+F5) and refresh the page.
Q: I logged metrics and see "traces: 0". What's wrong?
mlflow.log_metric(), mlflow.log_param(), and mlflow.start_run() create MLflow Runs, a different concept from MLflow Traces. Runs appear in the Runs tab of an experiment. Traces appear in the Traces tab. They are independent. Logging a metric does not create a trace, and tracing does not require a run context.
To generate a trace, use automatic tracing (e.g. mlflow.openai.autolog(), mlflow.langchain.autolog()) or manual tracing (e.g. @mlflow.trace, mlflow.start_span()). The Traces tab stays empty until one of these is called and the instrumented code runs.
What you called | Where it appears |
|---|---|
| Runs tab |
| Traces tab |
| Traces tab |
| Traces tab |
Q: I can't find a specific trace by ID in the trace list. What should I do?
By default, the trace list returns the most recent 1,000 traces. If an older trace falls outside this window, it won't appear in search results, even if you match the trace ID.
To find an older trace, narrow the time range filter to create a smaller window of results. Once the trace falls within the 1,000 most recent entries for that specific period, the ID search will pick it up. If you know the experiment ID and trace ID, you can also navigate directly: <workspace-url>/ml/experiments/<experiment-id>/traces/<trace-id>.
Experiments not in Unity Catalog are also capped at 100,000 traces total. For scalable, governed, SQL-queryable storage with no trace limit, Databricks recommends storing traces in Unity Catalog. To move existing experiment traces, migrate to traces in Unity Catalog.
Q: The model execution gets stuck and my trace is "in progress" forever.
Sometimes a model or an agent gets stuck in a long-running operation or an infinite loop, causing the trace to be stuck in the "in progress" state.
To prevent this, you can set a timeout for the trace using the MLFLOW_TRACE_TIMEOUT_SECONDS environment variable. If the trace exceeds the timeout, MLflow will automatically halt the trace with ERROR status and export it to the backend, so that you can analyze the spans to identify the issue. By default, the timeout is not set.
The timeout only applies to MLflow trace. The main program, model, or agent, will continue to run even if the trace is halted.
For example, the following code sets the timeout to 5 seconds and simulates how MLflow handles a long-running operation:
import mlflow
import os
import time
# Set the timeout to 5 seconds for demonstration purposes
os.environ["MLFLOW_TRACE_TIMEOUT_SECONDS"] = "5"
# Simulate a long-running operation
@mlflow.trace
def long_running():
for _ in range(10):
child()
@mlflow.trace
def child():
time.sleep(1)
long_running()
MLflow monitors the trace execution time and expiration in a background thread. By default, this check is performed every second and resource consumption is negligible. If you want to adjust the interval, you can set the MLFLOW_TRACE_TIMEOUT_CHECK_INTERVAL_SECONDS environment variable.
Q: My trace is split into multiple traces when doing multi-threading. How can I combine them into a single trace?
As MLflow Tracing depends on Python ContextVar, each thread has its own trace context by default, but it is possible to generate a single trace for multi-threaded applications with a few additional steps. See Multi-threading section for more information.
Q: How do I temporarily disable tracing?
To disable tracing, mlflow.tracing.disable API will cease the collection of trace data from within MLflow and will not log
any data to the MLflow Tracking service regarding traces.
To enable tracing (if it had been temporarily disabled), mlflow.tracing.enable API will re-enable tracing functionality for instrumented models
that are invoked.
Q: My trace search results are too big for mlflow.search_traces(). How do I search traces at scale?
The MLflow API provides pagination through the MlflowClient.search_traces() method. However, for use cases not requiring pagination, mlflow.search_traces() is recommended since it provides more functionality and convenient defaults.
For large-scale trace analysis in production, it is generally best to use production monitoring to log traces to Delta tables in Unity Catalog. See Tracing overview for production tracing guidance.
Q: Can I enable or disable tracing without modifying my code?
Yes. Set the MLFLOW_TRACING_ENABLED environment variable:
# Disable tracing
export MLFLOW_TRACING_ENABLED=false
# Enable tracing (if previously disabled)
export MLFLOW_TRACING_ENABLED=true
This allows you to toggle tracing on or off at runtime without code changes, useful for testing or selectively disabling tracing in certain environments.
Q: Does MLflow Tracing work with async and await code?
Yes. The @mlflow.trace decorator works seamlessly with async functions:
import mlflow
@mlflow.trace
async def my_async_function(query: str):
result = await some_async_operation(query)
return result
# Call it
await my_async_function("hello")
Q: Can I view traces directly in Jupyter notebooks?
Yes. In MLflow 2.20 and later, the trace UI automatically renders within Jupyter notebooks when:
- Your code generates traces (via autolog or
@mlflow.trace). - You call
mlflow.search_traces()to display traces as a DataFrame.
The notebook UI shows the same trace details and exploration tools as the Databricks workspace UI.
Q: Is MLflow Tracing compatible with other observability tools?
Yes. MLflow Tracing is built on OpenTelemetry (OTel) standards, so it's vendor-agnostic. You can export traces to other observability systems (e.g., Jaeger, Datadog, New Relic) via OTLP (OpenTelemetry Protocol) export. See OTel export for configuration details.
Q: What is the maximum size for a trace?
Traces can be large; the latency impact depends on size. MLflow recommends testing your application to understand the impact:
Trace size per request | Latency impact |
|---|---|
~10 KB | ~1 ms |
~1 MB | 50–100 ms |
~10 MB | 150+ ms |
To optimize trace size in production, use custom span processors to filter or mask sensitive data before export.
Q: Why are my traces split into multiple traces when using multi-threading?
In the Traces tab you'll see the work show up as several separate traces rather than one, because each thread gets its own trace context. To keep everything in a single trace, propagate the context to your worker threads:
import contextvars
from concurrent.futures import ThreadPoolExecutor
import mlflow
@mlflow.trace
def main_task():
ctx = contextvars.copy_context()
with ThreadPoolExecutor() as executor:
# Pass the copied context to worker threads
executor.submit(ctx.run, worker_task)
def worker_task():
# This runs in the copied context, preserving the trace
print("I'm in the same trace as the parent")
main_task()
Alternatively, use mlflow.tracing.set_destination() with context_local=False to log all spans from your app to a single experiment and trace.