Track machine learning training runs

The MLflow tracking component lets you log source properties, parameters, metrics, tags, and artifacts related to training a machine learning model. MLflow tracking is based on two concepts, experiments and runs:

  • An MLflow experiment is the primary unit of organization and access control for MLflow runs; all MLflow runs belong to an experiment. Experiments let you visualize, search for, and compare runs, as well as download run artifacts and metadata for analysis in other tools.
  • An MLflow run corresponds to a single execution of model code. Each run records the following information:
    • Source: Name of the notebook that launched the run or the project name and entry point for the run.
    • Version: Notebook revision if run from a notebook or Git commit hash if run from an MLflow Project.
    • Start & end time: Start and end time of the run.
    • Parameters: Model parameters saved as key-value pairs. Both keys and values are strings.
    • Metrics: Model evaluation metrics saved as key-value pairs. The value is numeric. Each metric can be updated throughout the course of the run (for example, to track how your model’s loss function is converging), and MLflow records and lets you visualize the metric’s history.
    • Tags: Run metadata saved as key-value pairs. You can update tags during and after a run completes. Both keys and values are strings.
    • Artifacts: Output files in any format. For example, you can record images, models (for example, a pickled scikit-learn model), and data files (for example, a Parquet file) as an artifact.

Use the MLflow Tracking API to log parameters, metrics, tags, and artifacts from a model run. The Tracking API communicates with an MLflow tracking server. When you use Databricks, a Databricks-hosted tracking server logs the data. The hosted MLflow tracking server has Python, Java, and R APIs.

For information about controlling access to experiments, see MLflow Experiment permissions.

Note

MLflow is installed on Databricks Runtime ML clusters. To use MLflow on a Databricks Runtime cluster, you must install the mlflow library. For instructions on installing a library onto a cluster, see Install a library on a cluster. The specific packages to install for MLflow are:

  • For Python, select Library Source PyPI and enter mlflow in the Package field.
  • For R, select Library Source CRAN and enter mlflow in the Package field.
  • For Scala, install these two packages:
    • Select Library Source Maven and enter org.mlflow:mlflow-client:1.11.0 in the Coordinates field.
    • Select Library Source PyPI and enter mlflow in the Package field.

Where MLflow runs are logged

All MLflow runs are logged to the active experiment, which can be set using any of the following ways:

If no active experiment is set, runs are logged to the notebook experiment.

Experiments

There are two types of experiments: workspace and notebook.

  • You can create a workspace experiment from the Workspace UI or the MLflow API. Workspace experiments are not associated with any notebook, and any notebook can log a run to these experiments by using the experiment ID or the experiment name.
  • A notebook experiment is associated with a specific notebook. Databricks automatically creates a notebook experiment if there is no active experiment when you start a run using mlflow.start_run().

To learn how to control access to experiments, see MLflow Experiment permissions.

Workspace experiments

This section describes how to create a workspace experiment using the Databricks UI. You can also use the MLflow API.

For instructions on logging runs to workspace experiments, see Log runs to a notebook or workspace experiment.

Create workspace experiment

  1. Click the Workspace button Workspace Icon or the Home button Home Icon in the sidebar.

  2. Go to the folder in which you want to create the experiment.

  3. Do one of the following:

    • Next to any folder, click the Menu Dropdown on the right side of the text and select Create > MLflow Experiment.

      Create experiment
    • In the Workspace or a user folder, click Down Caret and select Create > MLflow Experiment.

  4. In the Create MLflow Experiment dialog, enter a name for the experiment and an optional artifact location. If you do not specify an artifact location, artifacts are stored in dbfs:/databricks/mlflow-tracking/<experiment-id>.

    Databricks supports DBFS, S3, and Azure Blob storage artifact locations.

    To store artifacts in S3, specify a URI of the form s3://<bucket>/<path>. MLflow obtains credentials to access S3 from your clusters’s instance profile. Artifacts stored in S3 cannot be viewed in the MLflow UI; you must download them using an object storage client.

    To store artifacts in Azure Blob storage, specify a URI of the form wasbs://<container>@<storage-account>.blob.core.windows.net/<path>. Artifacts stored in Azure Blob storage cannot be viewed in the MLflow UI; you must download them using a blob storage client.

  5. Click Create. An empty experiment displays.

View workspace experiment

  1. Click the Workspace button Workspace Icon or the Home button Home Icon in the sidebar.
  2. Go to the folder containing the experiment.
  3. Click the experiment name.

Delete workspace experiment

  1. Click the Workspace button Workspace Icon or the Home button Home Icon in the sidebar.
  2. Go to the folder containing the experiment.
  3. Click the Menu Dropdown at the right side of the experiment and select Move to Trash.

Notebook experiments

When you use the mlflow.start_run() command in a notebook, the run logs metrics and parameters to the active experiment. If no experiment is active, Databricks creates a notebook experiment. A notebook experiment shares the same name and ID as its corresponding notebook. The notebook ID is the numerical identifier at the end of a Notebook URL and ID.

For instructions on logging runs to notebook experiments, see Log runs to a notebook or workspace experiment.

Note

If you delete a notebook experiment using the API (for example, MlflowClient.tracking.delete_experiment() in Python), the notebook itself is moved into the Trash folder.

View notebook experiment

To view a notebook experiment and its associated runs, click the Experiment icon Experiment icon in the notebook toolbar:

Notebook toolbar

The Experiment Runs sidebar displays, showing run parameters and metrics.

View run parameters and metrics

To view the experiment, click the External Link icon at the far right, next to Experiment Runs. The Experiment page displays.

View experiment

To view a run, click on its Start Time in the Experiment table. You can also go directly to a run by clicking the External Link icon next to the date and time of the run in the Experiment Runs sidebar.

View run

To view the version of the notebook that created a run, do one of the following:

  • In the Experiment Runs sidebar, click the Notebook icon Notebook Version Icon in the box for that Experiment Run.
  • In the Run page, click the link next to Source.

The version of the notebook associated with the run appears in the main window with a highlight bar showing the date and time of the run.

Delete notebook experiment

Notebook experiments are part of the notebook and cannot be deleted separately. If you delete the notebook, the notebook experiment is deleted. If you delete a notebook experiment using the API (for example, MlflowClient.tracking.delete_experiment() in Python), the notebook is also deleted.

Runs

All MLflow runs are logged to the active experiment. If you have not explicitly set an experiment as the active experiment, runs are logged to the notebook experiment.

Log runs to a notebook or workspace experiment

This notebook shows examples of how to log runs to a notebook experiment and to a workspace experiment. Only MLflow runs initiated within a notebook can be logged to the notebook experiment. MLflow runs launched from any notebook or from the APIs can be logged to a workspace experiment. For information about viewing logged runs, see View notebook experiment and View workspace experiment.

Log MLflow runs notebook

Open notebook in new tab

You can use MLflow Python, Java or Scala, and R APIs to start runs and record run data. For details, see the MLflow quick start notebooks.

View and manage runs in experiments

Within an experiment you can perform many operations on its contained runs.

Filter runs

You can search for runs based on parameter or metric values. To search for runs that match an expression containing parameter and metric values, enter a query in the Search Runs field and click Search. Some query syntax examples are:

metrics.r2 > 0.3

params.elasticNetParam = 0.5

params.elasticNetParam = 0.5 AND metrics.avg_areaUnderROC > 0.3

You can also filter runs based on their state (Active or Deleted) and based on whether a model version is associated with the run. To do this, click Filter to the right of the Search box. The State and Linked Models drop-down menus appear below the Search Runs field. Make your selections from the drop-down menus.

Filter runs

Download runs

  1. Select one or more runs.
  2. Click Download CSV. A CSV file containing the following fields downloads: Run ID,Name,Source Type,Source Name,User,Status,<parameter1>,<parameter2>,...,<metric1>,<metric2>,....

Display run details

Click the date link of a run. The run details screen displays. This screen shows the parameters used for the run, the metrics resulting from the run, and any tags or notes. You also access artifacts saved from a run in this screen.

To view the specific version of the notebook or Git project used for a run:

  • If the run was launched locally in a Databricks notebook or job, click the link in the Source field to open the specific notebook version used in the run.
  • If the run was launched remotely from a Git project, click the link in the Git Commit field to open the specific version of the project used in the run. The link in the Source field opens the master branch of the Git project used in the run.

If you logged a model from a run, the model appears in the Artifacts section of this page. To display code snippets illustrating how to load and use the model to make predictions on Spark and pandas DataFrames, click the model name.

Compare runs

  1. In the experiment, select two or more runs by clicking in the checkbox to the left of the run.

  2. Click Compare. The Comparing <N> Runs screen displays.

  3. Do one of the following:

    • Select a metric name to display a graph of the metric.

    • Select parameters and metrics from the X-axis and Y-axis drop-down lists to generate a scatter plot.

      Scatter plot

Delete runs

  1. In the experiment, select one or more runs by clicking in the checkbox to the left of the run.
  2. Click Delete.
  3. If the run is a parent run, decide whether you also want to delete descendant runs. This option is selected by default.
  4. Click Delete to confirm or Cancel to cancel. Deleted runs are saved for 30 days. To display deleted runs, select Deleted in the State field.

Access the MLflow tracking server from outside Databricks

You can also write to and read from the tracking server from outside Databricks, for example using the MLflow CLI.

Analyze MLflow runs using DataFrames

You can access MLflow run data programmatically using the following two DataFrame APIs:

This example demonstrates how to use the MLflow Python client to build a dashboard that visualizes changes in evaluation metrics over time, tracks the number of runs started by a specific user, and measures the total number of runs across all users:

Examples

The following notebooks demonstrate how to train several types of models and track the training data in MLflow and how to store tracking data in Delta Lake.