Programmatic access to traces
Search, read, and analyze traces programmatically. Use mlflow.search_traces() to query traces stored in Unity Catalog tables, the MLflow tracking server, or inference tables — and, for traces in Unity Catalog, query the Delta tables directly with SQL. Once you have a trace, read its object model — metadata, spans, assessments, and token usage — to inspect what happened. You can select subsets of traces to analyze or to create evaluation datasets.
mlflow.search_traces() API
def mlflow.search_traces(
experiment_ids: list[str] | None = None,
filter_string: str | None = None,
max_results: int | None = None,
order_by: list[str] | None = None,
extract_fields: list[str] | None = None,
run_id: str | None = None,
return_type: Literal['pandas', 'list'] | None = None,
model_id: str | None = None,
sql_warehouse_id: str | None = None,
include_spans: bool = True,
locations: list[str] | None = None,
) -> pandas.DataFrame | list[Trace]
mlflow.search_traces() lets you filter and select data along a few dimensions:
- Filter by a query string
- Filter by locations: experiment, run, model, or Unity Catalog schema
- Limit data: max results, include or exclude spans
- Adjust return value format: data format, data order
search_traces() returns either a pandas DataFrame or a list of Trace objects, which can then be analyzed further or reshaped into evaluation datasets. See the schema details of these return types.
See the mlflow.search_traces() API docs for full details.
Databricks-managed MLflow and OSS (open source software) MLflow share most search query syntax but have a few field-level differences. See Differences from OSS MLflow for details.
mlflow.search_traces() parameters
Category |
| Description | Example |
|---|---|---|---|
Filter by query string |
| See Search query syntax for supported filters and comparators. |
|
Filter by locations |
| This argument can be list of experiment IDs or Unity Catalog |
|
| MLflow run ID |
| |
| MLflow model ID |
| |
Limit data |
| Max number of traces (rows) to return |
|
| Include or exclude spans from the results. Spans include trace details and can make result sizes much larger. |
| |
Return value format |
| See the syntax and supported keys. |
|
| This function can return either a pandas DataFrame or a list of |
| |
Deprecated |
| Use | |
| Select fields in the returned DataFrame or trace objects instead. | ||
| Use the |
Search query syntax
The filter_string argument uses a SQL-like query language to filter traces. String values must be wrapped in single quotes (for example, trace.status = 'OK'), and numeric values must not be quoted (for example, trace.execution_time_ms > 1000). Combine conditions with AND. The OR operator is not supported.
Supported filters and comparators
The following fields and comparators are supported on Databricks-managed MLflow.
Filters marked (UC only) are supported only for MLflow traces stored in Unity Catalog. See Store OpenTelemetry traces in Unity Catalog.
Field type | Fields | Comparators | Example |
|---|---|---|---|
Trace status |
|
|
|
Trace timestamps |
|
|
|
Trace IDs |
|
|
|
String fields |
|
|
|
Request and response content (UC only) |
|
|
|
Token count (UC only) |
|
|
|
Linked prompts |
|
|
|
Span name, type, status, and service name (UC only) |
|
|
|
OTel span attributes (UC only) |
|
|
|
Tags |
|
For MLflow traces stored in an experiment (not in Unity Catalog), only |
|
Metadata |
|
For MLflow traces stored in an experiment (not in Unity Catalog), only |
|
Feedback (UC only) |
|
|
|
Expectations (UC only) |
|
|
|
Differences from OSS MLflow
The search query syntax on Databricks-managed MLflow closely tracks OSS MLflow, with the following differences:
Field | Databricks-managed MLflow | OSS MLflow | Notes |
|---|---|---|---|
| Supported (UC only) | Not supported | Use these fields to filter on serialized request and response content. |
| Supported (UC only) | Not supported | Filter traces by total token count. |
| Supported (UC only) | Not supported | Filter traces by OpenTelemetry span attributes. |
| Not supported | Supported (SQLAlchemy store only) | OSS exposes |
| Not supported | Supported (mapped to linked prompts tag) | On Databricks, use the top-level |
| Not supported | Supported | On Databricks, use |
| Not supported | Supported | Filter traces linked to a specific issue ID. |
Query trace tables with SQL
When traces are stored in Unity Catalog, you can query them with Databricks SQL in addition to the SDK. The MLflow service stores span data in OpenTelemetry-compliant tables and automatically creates Databricks SQL views that transform that data into the MLflow format. To set up Unity Catalog trace storage, see Store OpenTelemetry traces in Unity Catalog.
Databricks recommends querying the views (or using the SDK) rather than the underlying OpenTelemetry tables, whose schemas can change over time. For large trace volumes, view performance can degrade: create a materialized view over the views and update it incrementally, or use the SDK for best performance on recent data.
{table_prefix}_trace_unified
A unified view of all trace data, grouped by trace ID. Each row holds the raw span data plus the trace metadata (MLflow tags, metadata, and assessments). Top-level columns:
trace_id: STRING
client_request_id: STRING
request_time: TIMESTAMP
state: STRING
execution_duration_ms: DOUBLE
request: STRING
response: STRING
trace_metadata: VARIANT
tags: MAP<STRING, STRING>
spans: LIST<STRUCT> # per-span records: name, kind, timing, attributes, status, events, links
assessments: LIST<STRUCT> # feedback and expectation records with source, value, rationale, metadata
The trace_metadata column and span attributes fields are VARIANT. Read them with colon-path syntax and cast to the type you need, rather than with map key lookup:
SELECT spans[0].attributes:`mlflow.spanInputs`::STRING FROM my_catalog.my_schema.my_prefix_trace_unified
{table_prefix}_trace_metadata
Contains only the MLflow tags, metadata, and assessments grouped by trace ID. It is more performant than the unified view when you need MLflow annotation data but not span data. Columns: trace_id, client_request_id, tags, trace_metadata, and assessments (same structure as in the unified view).
Annotation data formats
MLflow annotation entities (metadata, tags, assessments, and run links) are also stored in the {table_prefix}_otel_annotations table, one row per entity with a typed annotation_type (METADATA, TAG, FEEDBACK, EXPECTATION, or RUN_LINK). The table is append-only with soft deletes, so de-duplicate on retrieval: take the latest row per annotation_id (order by updated_at descending) and drop rows where deleted_at is set. The value and metadata columns are VARIANT (JSON). For assessments, user-supplied metadata sits alongside internal MLflow keys (prefixed mlflow.); ignore the internal keys when reading user metadata.
Analyze query performance
To diagnose slow queries, inspect query profiles in the SQL warehouse query history: open the SQL warehouses page, select your warehouse, and click Query history. Filter for queries with MLflow as the source, then open a query to view its profile and check:
- Scheduling time: High scheduling time means queries are queued behind heavy warehouse load. Switch to a different warehouse in the MLflow UI, or configure a different warehouse in your client.
- Overall query performance: For consistently slow queries, use a larger SQL warehouse, tighten the bounds on
trace.timestamp_ms, and remove other filter predicates where possible.
Read trace data
An MLflow Trace has two components:
For the full trace object model and schema, see Trace data model reference.
Basic metadata properties
# Primary identifiers
print(f"Trace ID: {trace.info.trace_id}")
print(f"Client Request ID: {trace.info.client_request_id}")
# Status information
print(f"State: {trace.info.state}") # OK, ERROR, IN_PROGRESS
# Request/response previews (truncated)
print(f"Request preview: {trace.info.request_preview}")
print(f"Response preview: {trace.info.response_preview}")
Storage location and experiment
location = trace.info.trace_location
print(f"Location type: {location.type}")
# Stored in Unity Catalog (recommended)
if location.uc_table_prefix:
print(f"UC location: {location.uc_table_prefix.full_table_prefix}")
# Stored in an MLflow experiment
if location.mlflow_experiment:
print(f"Experiment ID: {trace.info.experiment_id}")
# Stored in a Databricks inference table
if location.inference_table:
print(f"Table: {location.inference_table.full_table_name}")
The experiment is the UI entry point regardless of backend. Use trace.info.experiment_id to open the trace in the MLflow UI even when it is stored in Unity Catalog.
Request and response previews
The request_preview and response_preview properties provide truncated summaries of the full request and response data, so you can understand what happened without loading the complete payloads.
request_preview = trace.info.request_preview
response_preview = trace.info.response_preview
# Full request/response data (see below)
full_request = trace.data.request
full_response = trace.data.response
Time-related properties
# Timestamps (milliseconds since epoch)
print(f"Start time (ms): {trace.info.request_time}")
print(f"Timestamp (ms): {trace.info.timestamp_ms}") # Alias for request_time
# Duration
print(f"Execution duration (ms): {trace.info.execution_duration}")
# Convert to human-readable format
import datetime
start_time = datetime.datetime.fromtimestamp(trace.info.request_time / 1000)
print(f"Started at: {start_time}")
Tags and metadata
# Tags (mutable, can be updated after creation)
for key, value in trace.info.tags.items():
print(f" {key}: {value}")
print(f"Environment: {trace.info.tags.get('environment')}")
# Trace metadata (immutable, set at creation)
for key, value in trace.info.trace_metadata.items():
print(f" {key}: {value}")
Token usage information
MLflow Tracing can track token usage of LLM calls, using token counts returned by LLM provider APIs.
# Get aggregated token usage (if available)
token_usage = trace.info.token_usage
if token_usage:
print(f"Input tokens: {token_usage.get('input_tokens')}")
print(f"Output tokens: {token_usage.get('output_tokens')}")
print(f"Total tokens: {token_usage.get('total_tokens')}")
How you track token usage depends on the LLM provider:
Scenario | How to track token usage |
|---|---|
Use the OpenAI client to verify that MLflow Tracing automatically tracks token usage. | |
LLM providers with native MLflow Tracing support | See the provider's integration page under MLflow Tracing Integrations to determine if native token tracking is supported. |
Providers without native MLflow Tracing support | Manually log token usage using |
Monitor multiple endpoints across your AI platform. | Use AI Gateway usage tracking for logging token usage to system tables across serving endpoints. |
Assessments
Find assessments with search_assessments():
# Get all assessments
all_assessments = trace.search_assessments()
# Search by name
helpfulness = trace.search_assessments(name="helpfulness")
if helpfulness:
assessment = helpfulness[0]
print(f"Helpfulness: {assessment.value}")
print(f"Source: {assessment.source.source_type} - {assessment.source.source_id}")
print(f"Rationale: {assessment.rationale}")
# Search by type
feedback_only = trace.search_assessments(type="feedback")
expectations_only = trace.search_assessments(type="expectation")
# Search by span ID
span_assessments = trace.search_assessments(span_id=retriever_span.span_id)
# Include overridden assessments
all_including_invalid = trace.search_assessments(all=True)
Access assessment details:
for assessment in trace.info.assessments:
print(f"Assessment: {assessment.name}")
print(f" Type: {type(assessment).__name__}")
print(f" Value: {assessment.value}")
print(f" Source: {assessment.source.source_type.value}")
if assessment.rationale:
print(f" Rationale: {assessment.rationale}")
if assessment.metadata:
print(f" Metadata: {assessment.metadata}")
if assessment.error:
print(f" Error: {assessment.error}")
Work with spans
Spans are the building blocks of traces, representing individual operations or units of work. The Span class represents immutable, completed spans retrieved from traces.
Access span properties
# Access all spans from a trace
spans = trace.data.spans
print(f"Total spans: {len(spans)}")
span = spans[0]
# Basic properties
print(f"Span ID: {span.span_id}")
print(f"Name: {span.name}")
print(f"Type: {span.span_type}")
print(f"Parent ID: {span.parent_id}") # None for root spans
# Timing (nanoseconds)
duration_ms = (span.end_time_ns - span.start_time_ns) / 1_000_000
print(f"Duration: {duration_ms:.2f}ms")
# Status
print(f"Status code: {span.status.status_code}")
# Inputs and outputs
print(f"Inputs: {span.inputs}")
print(f"Outputs: {span.outputs}")
Find specific spans
Use search_spans() to find spans matching specific criteria:
import re
from mlflow.entities import SpanType
# Search by exact name
retriever_spans = trace.search_spans(name="retrieve_documents")
# Search by regex pattern
tool_spans = trace.search_spans(name=re.compile(r".*_tool$"))
# Search by span type
chat_spans = trace.search_spans(span_type=SpanType.CHAT_MODEL)
llm_spans = trace.search_spans(span_type="CHAT_MODEL") # String also works
# Search by span ID
specific_span = trace.search_spans(span_id=retriever_spans[0].span_id)
# Combine criteria
tool_fact_check = trace.search_spans(
name="fact_check_tool",
span_type=SpanType.TOOL,
)
Span attributes
from mlflow.tracing.constant import SpanAttributeKey
chat_span = trace.search_spans(span_type=SpanType.CHAT_MODEL)[0]
# Get all attributes
for key, value in chat_span.attributes.items():
print(f" {key}: {value}")
# Get a specific attribute
specific_attr = chat_span.get_attribute("custom_attribute")
# Access chat-specific attributes using SpanAttributeKey
messages = chat_span.get_attribute(SpanAttributeKey.CHAT_MESSAGES)
tools = chat_span.get_attribute(SpanAttributeKey.CHAT_TOOLS)
# Access per-span token usage
input_tokens = chat_span.get_attribute("llm.token_usage.input_tokens")
output_tokens = chat_span.get_attribute("llm.token_usage.output_tokens")
Request and response data
import json
# Get root span request/response
request_json = trace.data.request
response_json = trace.data.response
# Parse JSON strings
if request_json:
request_data = json.loads(request_json)
if response_json:
response_data = json.loads(response_json)
Best practices
Keyword arguments
Always use keyword (named) arguments with mlflow.search_traces(). It allows positional arguments, but the function arguments are evolving.
Good practice: mlflow.search_traces(filter_string="trace.status = 'OK'")
Bad practice: mlflow.search_traces([], "trace.status = 'OK'")
filter_string gotchas
When searching using the filter_string argument to mlflow.search_traces(), remember to:
- Use prefixes:
trace.,tag., ormetadata. - Use backticks if tag or attribute names have dots:
tag.`mlflow.traceName` - Use single quotes only:
'value'not"value" - Use Unix timestamp (milliseconds) for time:
1749006880539not dates - Use AND only: No OR support
See Search query syntax for the full list of supported fields and operators.
SQL warehouse integration
A Databricks SQL warehouse is required to read traces stored in Unity Catalog experiments. Set MLFLOW_TRACING_SQL_WAREHOUSE_ID before calling mlflow.search_traces() or mlflow.get_trace() on a Unity Catalog-backed experiment. Without this env var set, the read fails with SQL warehouse ID is required for accessing traces in UC tables. See Store traces in Unity Catalog for setup.
For large non-Unity Catalog datasets such as inference tables, a SQL warehouse is optional and improves query performance.
import os
os.environ['MLFLOW_TRACING_SQL_WAREHOUSE_ID'] = 'fa92bea7022e81fb'
# Required for UC-backed experiments. Improves performance for large non-UC datasets.
traces = mlflow.search_traces(
filter_string="trace.status = 'OK'",
locations=['my_catalog.my_schema'],
)
Pagination
mlflow.search_traces() returns results in memory, which works well for smaller result sets. To handle large result sets, use MlflowClient.search_traces() since it supports pagination.
Additional resources
- Find issues across traces - Let MLflow detect issues across your traces.
- Trace data model reference - The full trace object model: spans, span types, and lifecycle.
- Enrich traces: tags, context, and feedback - Enrich traces with tags, metadata, and context for richer search.
- Build evaluation datasets - Convert queried traces into test datasets.
Next step: Collect feedback and build datasets