Track machine learning training runs

The MLflow tracking component lets you log source properties, parameters, metrics, tags, and artifacts related to training a machine learning model. To get started with MLflow, try one of the MLflow quickstart tutorials.

MLflow tracking is based on two concepts, experiments and runs:

  • An MLflow experiment is the primary unit of organization and access control for MLflow runs; all MLflow runs belong to an experiment. Experiments let you visualize, search for, and compare runs, as well as download run artifacts and metadata for analysis in other tools.
  • An MLflow run corresponds to a single execution of model code. Each run records the following information:
    • Source: Name of the notebook that launched the run or the project name and entry point for the run.
    • Version: Notebook revision if run from a notebook or Git commit hash if run from an MLflow Project.
    • Start & end time: Start and end time of the run.
    • Parameters: Model parameters saved as key-value pairs. Both keys and values are strings.
    • Metrics: Model evaluation metrics saved as key-value pairs. The value is numeric. Each metric can be updated throughout the course of the run (for example, to track how your model’s loss function is converging), and MLflow records and lets you visualize the metric’s history.
    • Tags: Run metadata saved as key-value pairs. You can update tags during and after a run completes. Both keys and values are strings.
    • Artifacts: Output files in any format. For example, you can record images, models (for example, a pickled scikit-learn model), and data files (for example, a Parquet file) as an artifact.

Use the MLflow Tracking API to log parameters, metrics, tags, and artifacts from a model run. The Tracking API communicates with an MLflow tracking server. When you use Databricks, a Databricks-hosted tracking server logs the data. The hosted MLflow tracking server has Python, Java, and R APIs.

For information about controlling access to experiments, see MLflow Experiment permissions.

Note

MLflow is installed on Databricks Runtime ML clusters. To use MLflow on a Databricks Runtime cluster, you must install the mlflow library. For instructions on installing a library onto a cluster, see Install a library on a cluster. The specific packages to install for MLflow are:

  • For Python, select Library Source PyPI and enter mlflow in the Package field.
  • For R, select Library Source CRAN and enter mlflow in the Package field.
  • For Scala, install these two packages:
    • Select Library Source Maven and enter org.mlflow:mlflow-client:1.11.0 in the Coordinates field.
    • Select Library Source PyPI and enter mlflow in the Package field.

Where MLflow runs are logged

All MLflow runs are logged to the active experiment, which can be set using any of the following ways:

If no active experiment is set, runs are logged to the notebook experiment.

Experiments

There are two types of experiments: workspace and notebook.

  • You can create a workspace experiment from the Databricks Machine Learning UI or the MLflow API. Workspace experiments are not associated with any notebook, and any notebook can log a run to these experiments by using the experiment ID or the experiment name.
  • A notebook experiment is associated with a specific notebook. Databricks automatically creates a notebook experiment if there is no active experiment when you start a run using mlflow.start_run().

To learn how to control access to experiments, see MLflow Experiment permissions.

Get experiment name

To copy the experiment name, click the Copy Icon icon at the top of the experiment page. You can use this name in the MLflow command set_experiment to set the active MLflow experiment.

Experiment name icon

You can also copy the experiment name from the experiment sidebar in a notebook.

Workspace experiments

This section describes how to create a workspace experiment using the Databricks UI. You can also use the MLflow API.

For instructions on logging runs to workspace experiments, see Log runs to a notebook or workspace experiment.

Create workspace experiment

  1. Click Workspace Icon Workspace in the sidebar.

  2. Go to the folder in which you want to create the experiment.

  3. Do one of the following:

    • Next to any folder, click the Menu Dropdown on the right side of the text and select Create > MLflow Experiment.

      Create experiment

    • In the workspace or a user folder, click Down Caret and select Create > MLflow Experiment.
  4. In the Create MLflow Experiment dialog, enter a name for the experiment and an optional artifact location. If you do not specify an artifact location, artifacts are stored in dbfs:/databricks/mlflow-tracking/<experiment-id>.

    Databricks supports DBFS, S3, and Azure Blob storage artifact locations.

    To store artifacts in S3, specify a URI of the form s3://<bucket>/<path>. MLflow obtains credentials to access S3 from your clusters’s instance profile. Artifacts stored in S3 do not appear in the MLflow UI; you must download them using an object storage client.

    Note

    The maximum size for an MLflow artifact uploaded to DBFS on AWS is 5GB.

    To store artifacts in Azure Blob storage, specify a URI of the form wasbs://<container>@<storage-account>.blob.core.windows.net/<path>. Artifacts stored in Azure Blob storage do not appear in the MLflow UI; you must download them using a blob storage client.

    Note

    When you store an artifact in a location other than DBFS, the artifact does not appear in the MLflow UI. Models stored in locations other than DBFS cannot be registered in Model Registry.

  5. Click Create. An empty experiment displays.

View workspace experiment

  1. Click Workspace Icon Workspace in the sidebar.
  2. Go to the folder containing the experiment.
  3. Click the experiment name.

Delete workspace experiment

  1. Click Workspace Icon Workspace in the sidebar.
  2. Go to the folder containing the experiment.
  3. Click the Menu Dropdown at the right side of the experiment and select Move to Trash.

Notebook experiments

When you use the mlflow.start_run() command in a notebook, the run logs metrics and parameters to the active experiment. If no experiment is active, Databricks creates a notebook experiment. A notebook experiment shares the same name and ID as its corresponding notebook. The notebook ID is the numerical identifier at the end of a Notebook URL and ID.

For instructions on logging runs to notebook experiments, see Log runs to a notebook or workspace experiment.

Note

If you delete a notebook experiment using the API (for example, MlflowClient.tracking.delete_experiment() in Python), the notebook itself is moved into the Trash folder.

View notebook experiment

The Experiment icon Experiment icon in the notebook toolbar gives you access to runs created by the notebook.

Notebook toolbar

Click it to display the Experiment Runs sidebar. At the top of the sidebar is the name of the experiment that the notebook most recently logged runs to (either a notebook experiment or a workspace experiment). The sidebar shows a summary of each run associated with that experiment, including run parameters and metrics.

View run parameters and metrics

From the sidebar, you can navigate to the experiment page or directly to a run.

  • To view the experiment, click the External Link icon at the far right, next to Experiment Runs.

    View experiment

    The experiment page lists all runs associated with the experiment. From the table, you can open the run page for any run associated with the experiment by clicking on its Start Time. The Source column gives you access to the notebook version that created the run. You can also search and filter runs by metrics or parameter settings.

  • To view a run, click the External Link icon next to the date and time of the run in the Experiment Runs sidebar. The runs page displays. For details about this page, see Display run details.

    To display Notes, Parameters, Metrics, or Tags for this run, click right-pointing arrow to the left of the label.

    View run

To view the version of the notebook that created a run, you can also do one of the following:

  • In the Experiment Runs sidebar, click the Notebook icon Notebook Version Icon in the box for that Experiment Run.
  • In the Run page, click the link next to Source.

The version of the notebook associated with the run appears in the main window with a highlight bar showing the date and time of the run.

Delete notebook experiment

Notebook experiments are part of the notebook and cannot be deleted separately. If you delete the notebook, the notebook experiment is deleted. If you delete a notebook experiment using the API (for example, MlflowClient.tracking.delete_experiment() in Python), the notebook is also deleted.

Runs

All MLflow runs are logged to the active experiment. If you have not explicitly set an experiment as the active experiment, runs are logged to the notebook experiment.

Log runs to a notebook or workspace experiment

This notebook shows examples of how to log runs to a notebook experiment and to a workspace experiment. Only MLflow runs initiated within a notebook can be logged to the notebook experiment. MLflow runs launched from any notebook or from the APIs can be logged to a workspace experiment. For information about viewing logged runs, see View notebook experiment and View workspace experiment.

Log MLflow runs notebook

Open notebook in new tab

You can use MLflow Python, Java or Scala, and R APIs to start runs and record run data. For details, see the MLflow quickstart notebooks.

View and manage runs in experiments

Within an experiment you can perform many operations on its contained runs.

Filter runs

You can search for runs based on parameter or metric values. To search for runs that match an expression containing parameter and metric values, enter a query in the search field and click Search. Some query syntax examples are:

metrics.r2 > 0.3

params.elasticNetParam = 0.5

params.elasticNetParam = 0.5 AND metrics.avg_areaUnderROC > 0.3

You can also filter runs based on their state (Active or Deleted) and based on whether a model version is associated with the run. To do this, click Filter to the right of the Search box. The State and Linked Models drop-down menus appear. Make your selections from the drop-down menus.

Filter runs

Download runs

  1. Select one or more runs.

  2. Click Download CSV. A CSV file containing the following fields downloads:

    Run ID,Name,Source Type,Source Name,User,Status,<parameter1>,<parameter2>,...,<metric1>,<metric2>,...
    

Display run details

Click the date link of a run. The run details screen displays. This screen shows the parameters used for the run, the metrics resulting from the run, and any tags or notes. To display Notes, Parameters, Metrics, or Tags for this run, click right-pointing arrow to the left of the label.

You also access artifacts saved from a run in this screen.

View the notebook or Git project used for a run

To view the specific version of the notebook or Git project used for a run:

  • If the run was launched locally in a Databricks notebook or job, click the link in the Source field to open the specific notebook version used in the run.
  • If the run was launched remotely from a Git project, click the link in the Git Commit field to open the specific version of the project used in the run. The link in the Source field opens the main branch of the Git project used in the run.
Reproduce the software environment of a run

You can reproduce the exact software environment for the run by clicking Reproduce Run. The following dialog appears:

Reproduce run dialog

With the default settings, when you click Confirm:

  • The notebook is cloned to the location shown in the dialog.
  • If the original cluster still exists, the cloned notebook is attached to the original cluster and the cluster is started.
  • If the original cluster no longer exists, a new cluster with the same configuration, including any installed libraries, is created and started. The notebook is attached to the new cluster.

You can select a different location for the cloned notebook and inspect the cluster configuration and installed libraries:

  • To select a different folder to save the cloned notebook, click Edit Folder.
  • To see the cluster spec, click View Spec. To clone only the notebook and not the cluster, uncheck this option.
  • To see the libraries installed on the original cluster, click View Libraries. If you don’t care about installing the same libraries as on the original cluster, uncheck this option.
View code snippets for prediction

If you logged a model from a run, the model appears in the Artifacts section of this page. To display code snippets illustrating how to load and use the model to make predictions on Spark and pandas DataFrames, click the model name.

predict code snippets

Compare runs

  1. In the experiment, select two or more runs by clicking in the checkbox to the left of the run.

  2. Click Compare. The Comparing <N> Runs screen displays.

  3. Do one of the following:

    • Select a metric name to display a graph of the metric.

    • Select parameters and metrics from the X-axis and Y-axis drop-down lists to generate a scatter plot.

      Scatter plot

Delete runs

  1. In the experiment, select one or more runs by clicking in the checkbox to the left of the run.
  2. Click Delete.
  3. If the run is a parent run, decide whether you also want to delete descendant runs. This option is selected by default.
  4. Click Delete to confirm or Cancel to cancel. Deleted runs are saved for 30 days. To display deleted runs, select Deleted in the State field.

Access the MLflow tracking server from outside Databricks

You can also write to and read from the tracking server from outside Databricks, for example using the MLflow CLI.

Analyze MLflow runs using DataFrames

You can access MLflow run data programmatically using the following two DataFrame APIs:

This example demonstrates how to use the MLflow Python client to build a dashboard that visualizes changes in evaluation metrics over time, tracks the number of runs started by a specific user, and measures the total number of runs across all users:

Examples

The following notebooks demonstrate how to train several types of models and track the training data in MLflow and how to store tracking data in Delta Lake.