%md # Model Selection with MLflow & mlflow-apps

This example notebook demonstrates how to use [mlflow-apps](https://github.com/mlflow/mlflow-apps), a collection of pluggable applications written with [MLflow](https://mlflow.org/), to simplify ML model training & selection.

When tackling ML problems, it's often useful to experiment with a variety of different frameworks, comparing results to choose the best model for the job. MLflow makes this easy through a set of components accessible via CLIs & Python APIs:

* [Projects](https://mlflow.org/docs/latest/projects.html): running ML code out of the box, installing dependencies in an isolated conda environment

* [Tracking](https://mlflow.org/docs/latest/tracking.html): logging parameters & metrics across different runs, enabling cross-framework model comparison

* [Models](https://mlflow.org/docs/latest/models.html): performing inference using models trained in any framework

In terms of the above, mlflow-apps is a collection of runnable MLflow projects that use the tracking & models APIs to simplify model training & comparison.

In this notebook, we'll use mlflow-apps to fit [TensorFlow](https://github.com/mlflow/mlflow-apps/tree/master/apps/dnn-regression), [scikit-learn](https://github.com/mlflow/mlflow-apps/tree/master/apps/linear-regression), and [XGBoost](https://github.com/mlflow/mlflow-apps/tree/master/apps/gbt-regression) models on ggplot2's [diamonds dataset](https://ggplot2.tidyverse.org/reference/diamonds.html), running each model training app through a single MLflow API call. We'll also cover data preprocessing and using MLflow to predict with the trained models.

%md ## Prerequisites

To run this notebook, attach the following to your cluster as [PyPi Libraries](https://docs.databricks.com/user-guide/libraries.html).

We'll use these dependencies to preprocess our data & load our fitted models back for inference.

* `mlflow>=0.4.2`

* `pyarrow`

* `pandas==0.23.2`

* `xgboost==0.72`

* `tensorflow==1.8`

%md ## Analyzing the diamonds dataset

**Data**: The dataset contains properties of ~50,000 round-cut diamonds, including properties such as cut and price.

**Goal**: Predict the price of a diamond based on its physical properties. Having accurate prices lets sellers and customers determine accurate prices while dealing in diamonds.

**Approach**: We'll run projects from the mlflow-apps repository to train and test different models on the dataset. We can easily train and compare different models in a plug-and-play fashion using MLflow's APIs.

%md ## Load and understand the data

Note: The preprocessing logic below largely resembles existing logic in mlflow-apps (see `mlflow-apps/utils.py`)

We begin by loading our data, which is stored in [Comma-Separated Value (CSV) format](https://en.wikipedia.org/wiki/Comma-separated_values).

We know from ggplot2 that the columns of the dataset have the following meanings.

**Feature columns**:

* [carat](https://en.wikipedia.org/wiki/Carat_(mass%29): weight of the diamond `(0.2-5.01)`

* [cut](https://en.wikipedia.org/wiki/Diamond_cut): quality of the cut `(Fair, Good, Very Good, Premium, Ideal)`

* [color](https://en.wikipedia.org/wiki/Diamond_color): diamond color, from `J (worst)` to `D (best)`

* [clarity](https://en.wikipedia.org/wiki/Diamond_clarity): a measurement of how clear the diamond is `(I1 (worst), SI2, SI1, VS2, VS1, VVS2, VVS1, IF (best))`

* x: length in mm in x dimension `(0-10.74)`

* y: width in mm in y dimension `(0-58.9)`

* z: depth in mm in z dimension `(0-31.8)`

* depth: total depth percentage = z / mean(x, y) = 2 * z / (x + y) `(43-79)`

* table: width of top of diamond relative to widest point `(43-95)`

**Label column**:

* price: price in US dollars `($326-$18,823)`

The data is organized by increasing price - you can see the cheapest diamond on the first row at $326.

%md ## Preprocess data

So what do we need to do to get our data ready for use with `mlflow-apps`?

*Recall our goal*: We want to learn to predict the price of each diamond.  We refer to the price as our target "label".

*Features*: What can we use as features (info describing each row) to predict the `price` label?  We can use the rest of the columns!

Let's print the schema of our dataset to see the type of each column.

%md The DataFrame has categorical string-valued columns, namely the `cut`, `color`, and `clarity` columns. Apps in `mlflow-apps`, like the XGBoost GBT and sklearn Elastic Net apps, need numeric inputs, so let's index the categorical columns in our dataset.

%md #### Split data into training and test sets

We now split our dataset into separate training and test sets.  We can train and tune our model as much as we like on the training set, so long as we do not examine the test set - this helps us avoid problems such as [data dredging](https://en.wikipedia.org/wiki/Data_dredging). After we have a good model (based on the training set), we can validate it on the held-out test set to know with high confidence our well our model will make predictions on future (unseen) data.

%md #### Saving the data for use with mlflow-apps

Our model-training apps expect data stored in [parquet](https://en.wikipedia.org/wiki/Apache_Parquet) format, so we save our processed dataset into the following files:

* `train_diamonds.parquet`: training data for our models

* `test_diamonds.parquet`: test data for our models

In addition, to showcase using MLflow Python APIs to load back models and perform inference with them, we will save two dataframes:

* `diamond_predict`: features of 20 diamonds from `test_diamonds.parquet`

* `diamond_prices`: prices (labels) of the 20 diamonds in `diamond_predict`

%md ## Model Training with `mlflow-apps`

Now that we understand our data and have prepared it as a DataFrame with numeric values, let's learn how MLflow and mlflow-apps can be used to easily train models and make predictions on your data.

%md ##### Linear Regression

We start by training a linear regression (ElasticNet) model with `mlflow-apps`, launching training through a single `mlflow.projects.run` call.

`mlflow.projects.run()` runs a project at a specified `uri`, which can be a local directory or (in the case of mlflow-apps) a Git repository. Each app within mlflow-apps is an MLflow project in a subdirectory of the repository. Thus, the `uri` we specify is mlflow-apps' HTTPS clone URL, followed by a `#` and the path to the linear regression app within mlflow-apps.

We also pass a `parameters` dictionary containing parameter key-value pairs to be passed to the app. These key-value pairs are described by `MLproject` configuration files for each app. You can see the `MLproject` for the `linear-regression` app [here](https://github.com/mlflow/mlflow-apps/blob/master/apps/linear-regression/MLproject).

For more info, see the [Projects documentation](https://mlflow.org/docs/latest/projects.html) and the `mlflow.projects.run()` [API docs](https://www.mlflow.org/docs/latest/python_api/mlflow.projects.html#mlflow.projects.run).

%md `linear_run` exposes information about the model training run, including a run ID that can be used to load the model again for inference. The linear regression app logs metrics like the training and test R2 scores during training, which we can inspect through the `print_run_metrics` helper defined below.

%md For more detailed insight into our model's behavior, we can use MLflow's model APIs to load our model back and examine its predictions on test data:

%md #### Adjusting parameters

Now, let's tune our model to improve its performance. When training our linear regression model, we used default values for many of the parameters to our linear regression app - see the full set of parameters [here](https://github.com/mlflow/mlflow-apps/tree/master/apps/linear-regression). Let's try adjusting the `alpha` and `l1-ratio` parameters to get a more accurate model.

%md #### Changing Models

It seems like changing the parameters of the linear regression model didn't do much to increase the accuracy compared the first time. Perhaps this isn't the right model for the specific problem, and we should consider a different model. We can easily train a different model on the same data using `mlflow-apps` - let's try out the `gbt-regression` app.

%md #### Changing Models Pt. 2

We can see that the GBT model performed quite well - our R2 score improved from ~0.82 to ~0.98!

Let's see if we can get any better results from the final app in `mlflow-apps`, `dnn-regressor`. `dnn-regressor` trains and saves a Tensorflow Deep Neural Net Regressor based on our input data.

%md As we can see, the `dnn-regression` app produced worse results than the `gbt-regression` app. It looks like the GBT model is the best out of the ones we've tried!

%md #### Conclusion

mlflow-apps makes it simple to run pre-packaged ML code for model training through `mlflow.projects.run()` API calls. The apps log common metrics to simplify cross-framework model selection. Using the MLflow model APIs, you can load back models trained via the apps and apply them to data for inference.