Track machine learning training runs

The MLflow tracking component lets you log source properties, parameters, metrics, tags, and artifacts related to training a machine learning model. MLflow tracking is based on two concepts, experiments and runs:

  • An MLflow experiment is the primary unit of organization and access control for MLflow runs; all MLflow runs belong to an experiment. Experiments let you visualize, search for, and compare runs, as well as download run artifacts and metadata for analysis in other tools.
  • An MLflow run corresponds to a single execution of model code. Each run records the following information:
    • Source: Name of the notebook that launched the run or the project name and entry point for the run.
    • Version: Notebook revision if run from a notebook or Git commit hash if run from an MLflow Project.
    • Start & end time: Start and end time of the run.
    • Parameters: Model parameters saved as key-value pairs. Both keys and values are strings.
    • Metrics: Model evaluation metrics saved as key-value pairs. The value is numeric. Each metric can be updated throughout the course of the run (for example, to track how your model’s loss function is converging), and MLflow records and lets you visualize the metric’s history.
    • Tags: Run metadata saved as key-value pairs. You can update tags during and after a run completes. Both keys and values are strings.
    • Artifacts: Output files in any format. For example, you can record images, models (for example, a pickled scikit-learn model), and data files (for example, a Parquet file) as an artifact.

Use the MLflow Tracking API to log parameters, metrics, tags, and artifacts from a model run. The Tracking API communicates with an MLflow tracking server. When you use Databricks, a Databricks-hosted tracking server logs the data. The hosted MLflow tracking server has Python, Java, and R APIs.

For information about controlling access to experiments, see MLflow Experiment permissions.

Note

MLflow is installed on Databricks Runtime ML clusters. To use MLflow on a Databricks Runtime cluster, you must install the PyPI library mlflow[extras]. See Install a library on a cluster.

Where MLflow runs are logged

All MLflow runs are logged to the active experiment, which can be set using any of the following ways:

If no active experiment is set, runs are logged to the notebook experiment.

Experiments

There are two types of experiments: workspace and notebook.

  • You can create a workspace experiment from the Workspace UI or the MLflow API. Workspace experiments are not associated with any notebook, and any notebook can log a run to these experiments by using the experiment ID or the experiment name.
  • A notebook experiment is associated with a specific notebook. Databricks automatically creates a notebook experiment if there is no active experiment when you start a run using mlflow.start_run().

To learn how to control access to experiments, see MLflow Experiment permissions.

Workspace experiments

This section describes how to create a workspace experiment using the Databricks UI. You can also use the MLflow API.

For instructions on logging runs to workspace experiments, see Log runs to a notebook or workspace experiment.

Create workspace experiment

  1. Click the Workspace button Workspace Icon or the Home button Home Icon in the sidebar.

  2. Go to the folder in which you want to create the experiment.

  3. Do one of the following:

    • Next to any folder, click the Menu Dropdown on the right side of the text and select Create > MLflow Experiment.

      Create experiment
    • In the Workspace or a user folder, click Down Caret and select Create > MLflow Experiment.

  4. In the Create MLflow Experiment dialog, enter a name for the experiment and an optional artifact location. If you do not specify an artifact location, artifacts are stored in dbfs:/databricks/mlflow-tracking/<experiment-id>.

    Databricks supports DBFS, S3, and Azure Blob storage artifact locations.

    To store artifacts in S3, specify a URI of the form s3://<bucket>/<path>. MLflow obtains credentials to access S3 from your clusters’s instance profile. Artifacts stored in S3 cannot be viewed in the MLflow UI; you must download them using an object storage client.

    To store artifacts in Azure Blob storage, specify a URI of the form wasbs://<container>@<storage-account>.blob.core.windows.net/<path>. Artifacts stored in Azure Blob storage cannot be viewed in the MLflow UI; you must download them using a blob storage client.

  5. Click Create. An empty experiment displays.

    Empty experiment

View workspace experiment

  1. Click the Workspace button Workspace Icon or the Home button Home Icon in the sidebar.
  2. Go to the folder containing the experiment.
  3. Click the experiment name.

Delete workspace experiment

  1. Click the Workspace button Workspace Icon or the Home button Home Icon in the sidebar.
  2. Go to the folder containing the experiment.
  3. Click the Menu Dropdown at the right side of the experiment and select Move to Trash.

Notebook experiments

When you use the mlflow.start_run() command in a notebook, the run logs metrics and parameters to the active experiment. If no experiment is active, Databricks creates a notebook experiment. A notebook experiment shares the same name and ID as its corresponding notebook. The notebook ID is the numerical identifier at the end of a Notebook URL and ID.

For instructions on logging runs to notebook experiments, see Log runs to a notebook or workspace experiment.

Note

If you delete a notebook experiment using the API (for example, MlflowClient.tracking.delete_experiment() in Python), the notebook itself is moved into the Trash folder.

View notebook experiment

To view a notebook experiment and its associated runs, click the Experiment icon Experiment in the notebook toolbar:

Notebook toolbar

From the Experiment Runs sidebar, you can view the run parameters and metrics:

View run parameters and metrics

You can also view the version of the notebook that created the run. Click the Notebook icon NotebookVersion in the box for that Experiment Run. The version of the notebook associated with that run appears in the main window with a highlight bar showing the date and time of the run.

Click the External Link icon in the Experiment Runs context bar to view the experiment:

View experiment

In the Experiment Runs sidebar, click the External Link icon next to the date Run Date

to view a run:

View run

Delete notebook experiment

Notebook experiments are part of the notebook and cannot be deleted separately. If you delete the notebook, the notebook experiment is deleted. If you delete a notebook experiment using the API (for example, MlflowClient.tracking.delete_experiment() in Python), the notebook is also deleted.

Runs

All MLflow runs are logged to the active experiment. If you have not explicitly set an experiment as the active experiment, runs are logged to the notebook experiment.

Log runs to a notebook or workspace experiment

This notebook shows examples of how to log runs to a notebook experiment and to a workspace experiment. Only MLflow runs initiated within a notebook can be logged to the notebook experiment. MLflow runs launched from any notebook or from the APIs can be logged to a workspace experiment. For information about viewing logged runs, see View notebook experiment and View workspace experiment.

Log MLflow runs notebook

Open notebook in new tab

You can use MLflow Python, Java or Scala, and R APIs to start runs and record run data. The following sections provide details. For example notebooks, see the Quick start.

  1. Install the PyPI library mlflow[extras] to a cluster, where the extra dependencies are:

    • scikit-learn when Python version >= ‘3.5’
    • scikit-learn == 0.20 when Python version < ‘3.5’
    • boto3 >= 1.7.12
    • mleap >= 0.8.1
    • azure-storage
    • google-cloud-storage
  2. Import MLflow library:

    import mlflow
    
  3. Start an MLflow run:

    with mlflow.start_run() as run:
    
  4. Log parameters, metrics, and artifacts:

    # Log a parameter (key-value pair)
    mlflow.log_param("param1", 5)
    # Log a metric; metrics can be updated throughout the run
    mlflow.log_metric("foo", 2, step=1)
    mlflow.log_metric("foo", 4, step=2)
    mlflow.log_metric("foo", 6, step=3)
    # Log an artifact (output file)
    with open("output.txt", "w") as f:
        f.write("Hello world!")
    mlflow.log_artifact("output.txt")
    
  1. Install the PyPI library mlflow and the Maven library org.mlflow:mlflow-client:1.0.0 to a cluster.

  2. Import MLflow and file libraries:

    import org.mlflow.tracking.ActiveRun
    import org.mlflow.tracking.MlflowContext
    import java.io.{File,PrintWriter}
    
  3. Create MLflow context:

    val mlflowContext = new MlflowContext()
    
  4. Create an experiment.

    val experimentName = "/Shared/QuickStart"
    val mlflowContext = new MlflowContext()
    val client = mlflowContext.getClient()
    val experimentOpt = client.getExperimentByName(experimentName);
    if (!experimentOpt.isPresent()) {
     client.createExperiment(experimentName)
    }
    mlflowContext.setExperimentName(experimentName)
    
  5. Log parameters, metrics, and file:

    import java.nio.file.Paths
    val run = mlflowContext.startRun("run")
    // Log a parameter (key-value pair)
    run.logParam("param1", "5")
    // Log a metric; metrics can be updated throughout the run
    run.logMetric("foo", 2.0, 1)
    run.logMetric("foo", 4.0, 2)
    run.logMetric("foo", 6.0, 3)
     new PrintWriter("/tmp/output.txt") { write("Hello, world!") ; close }
     run.logArtifact(Paths.get("/tmp/output.txt"))
    
  6. Close the run:

    run.endRun()
    
  1. Install the CRAN library mlflow to a cluster.

  2. Import and install MLflow libraries:

    library(mlflow)
    install_mlflow()
    
  3. Create a new run:

    run <- mlflow_start_run()
    
  4. Log parameters, metrics, and file:

    # Log a parameter (key-value pair)
    mlflow_log_param("param1", 5)
    # Log a metric; metrics can be updated throughout the run
    mlflow_log_metric("foo", 2, step = 1)
    mlflow_log_metric("foo", 4, step = 2)
    mlflow_log_metric("foo", 6, step = 3)
    # Log an artifact (output file)
    write("Hello world!", file = "output.txt")
    mlflow_log_artifact("output.txt")
    
  5. Close the run:

    mlflow_end_run()
    

View and manage runs in experiments

Within an experiment you can perform many operations on its contained runs.

Filter runs

To filter runs by a parameter or metric name, type the parameter or metric name in the Filter [Params|Metric] field and press Enter.

To filter runs that match an expression containing parameter and metric values:

  1. In the Search Runs field, specify an expression. For example: metrics.r2 > 0.3.

    Filter runs
  2. Click Search.

Download runs

  1. Select one or more runs.
  2. Click Download CSV. A CSV file containing the following fields downloads: Run ID,Name,Source Type,Source Name,User,Status,<parameter1>,<parameter2>,...,<metric1>,<metric2>,....

Display run details

Click the date link of a run. The run details screen displays. The fields in the detail page depend on whether you ran from a notebook or a Git project.

Notebook

If the run was launched locally in a Databricks notebook or job, it looks like:

Notebook run source

The link in the Source field opens the specific notebook version used in the run.

Notebook version
Git project

If the run was launched remotely from a Git project, it looks like:

Git project run source

The link in the Source field opens the master branch of the Git project used in the run. The link in the Git Commit field opens the specific version of the project used in the run.

Compare runs

  1. Select two or more runs.

  2. Click Compare. Either select a metric name to display a graph of the metric or select parameters and metrics from the X-axis and Y-axis drop-down lists to generate a scatter plot.

    Choose runs to compare

    The Comparing <N> Runs screen displays:

    Scatter plot

Delete runs

  1. Select the checkbox at the far left of one or more runs.

  2. Click Delete.

    Delete runs
  3. If the run is a parent run, decide whether you also want to delete child runs. This is the default.

  4. Click Delete to confirm.

After you delete a run you can still display it by selecting Deleted in the State field.

Access the MLflow tracking server from outside Databricks

You can also write to and read from the tracking server from outside Databricks, for example using the MLflow CLI.

Analyze MLflow runs using DataFrames

You can access MLflow run data programmatically using the following two DataFrame APIs:

This example demonstrates how to use the MLflow Python client to build a dashboard that visualizes changes in evaluation metrics over time, tracks the number of runs started by a specific user, and measures the total number of runs across all users:

Examples

The following notebooks demonstrate how to train several types of models and track the training data in MLflow and how to store tracking data in Delta Lake.