Skip to main content

Export MLflow traces to OpenTelemetry

MLflow tracing is built on OpenTelemetry (OTel). Because every trace MLflow captures is a set of OTel spans, you can export those spans outbound to any OpenTelemetry-compatible observability platform — such as Datadog, Grafana, or Splunk — over the standard OTLP protocol. You can send traces exclusively to an external collector, or use dual export to send them to both Databricks MLflow and your existing observability stack at the same time.


Export MLflow traces to an external OTel collector

MLflow can export traces outbound to any OpenTelemetry-compatible observability platform. Three modes are supported.

OTel-only export

To send traces exclusively to an external OTel collector (bypassing Databricks MLflow storage), set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT before starting any trace:

Python
import os
import mlflow

os.environ["OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"] = "http://localhost:4317/v1/traces"
os.environ["OTEL_SERVICE_NAME"] = "<your-service-name>"

# Trace is exported ONLY to the OTel collector at http://localhost:4317/v1/traces
with mlflow.start_span(name="foo") as span:
span.set_inputs({"a": 1})
span.set_outputs({"b": 2})

Dual export (MLflow + OTel)

To export traces to both Databricks MLflow and an external OTel collector simultaneously, set MLFLOW_ENABLE_DUAL_EXPORT:

Python
import os
import mlflow

os.environ["MLFLOW_ENABLE_DUAL_EXPORT"] = "true"
os.environ["OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"] = "http://localhost:4317/v1/traces"
os.environ["OTEL_SERVICE_NAME"] = "my-ml-service"

mlflow.set_tracking_uri("databricks")

# Traces are exported to BOTH MLflow and the OTel collector
with mlflow.start_span(name="dual_export_example") as span:
span.set_inputs({"model": "gpt-4", "prompt": "Hello world"})
span.set_outputs({"response": "Generated response"})

Metrics export

MLflow exports OTel metrics when a metrics endpoint is configured. For the complete list of exported metrics, see the MLflow documentation.

Python
import os

os.environ["OTEL_METRICS_EXPORTER"] = "otlp"
os.environ["OTEL_EXPORTER_OTLP_METRICS_ENDPOINT"] = "http://localhost:4317"
# Optional: configure export interval in milliseconds
os.environ["OTEL_METRIC_EXPORT_INTERVAL"] = "60000"

Supported platforms

MLflow uses the standard OTLP exporter and supports all OTLP exporter configurations. The following platforms have published OTel setup guides:

To use HTTP protocol instead of the default gRPC or to set custom headers:

Bash
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://localhost:4317/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="api_key=12345"

Additional resources

Next step: Observe and find issues