%md # Model Selection with MLflow & mlflow-apps

This example notebook demonstrates how to use [mlflow-apps](https://github.com/mlflow/mlflow-apps), a collection of pluggable applications written with [MLflow](https://mlflow.org/), to simplify ML model training & selection.

When tackling ML problems, it's often useful to experiment with a variety of different frameworks, comparing results to choose the best model for the job. MLflow makes this easy through a set of components accessible via CLIs & Python APIs:

* [Projects](https://mlflow.org/docs/latest/projects.html): running ML code out of the box, installing dependencies in an isolated conda environment

* [Tracking](https://mlflow.org/docs/latest/tracking.html): logging parameters & metrics across different runs, enabling cross-framework model comparison

* [Models](https://mlflow.org/docs/latest/models.html): performing inference using models trained in any framework

In terms of the above, mlflow-apps is a collection of runnable MLflow projects that use the tracking & models APIs to simplify model training & comparison.

In this notebook, we'll use mlflow-apps to fit [TensorFlow](https://github.com/mlflow/mlflow-apps/tree/master/apps/dnn-regression), [scikit-learn](https://github.com/mlflow/mlflow-apps/tree/master/apps/linear-regression), and [XGBoost](https://github.com/mlflow/mlflow-apps/tree/master/apps/gbt-regression) models on ggplot2's [diamonds dataset](https://ggplot2.tidyverse.org/reference/diamonds.html), running each model training app through a single MLflow API call. We'll also cover data preprocessing and using MLflow to predict with the trained models.

%md ## Prerequisites

To run this notebook, attach the following to your cluster as [PyPi Libraries](https://docs.databricks.com/user-guide/libraries.html).

We'll use these dependencies to preprocess our data & load our fitted models back for inference.

* `mlflow>=0.4.2`

* `pyarrow`

* `pandas==0.23.2`

* `xgboost==0.72`

* `tensorflow==1.8`

%md ## Analyzing the diamonds dataset

**Data**: The dataset contains properties of ~50,000 round-cut diamonds, including properties such as cut and price.

**Goal**: Predict the price of a diamond based on its physical properties. Having accurate prices lets sellers and customers determine accurate prices while dealing in diamonds.

**Approach**: We'll run projects from the mlflow-apps repository to train and test different models on the dataset. We can easily train and compare different models in a plug-and-play fashion using MLflow's APIs.

%md ## Load and understand the data

Note: The preprocessing logic below largely resembles existing logic in mlflow-apps (see `mlflow-apps/utils.py`)

We begin by loading our data, which is stored in [Comma-Separated Value (CSV) format](https://en.wikipedia.org/wiki/Comma-separated_values).

We know from ggplot2 that the columns of the dataset have the following meanings.

**Feature columns**:

* [carat](https://en.wikipedia.org/wiki/Carat_(mass%29): weight of the diamond `(0.2-5.01)`

* [cut](https://en.wikipedia.org/wiki/Diamond_cut): quality of the cut `(Fair, Good, Very Good, Premium, Ideal)`

* [color](https://en.wikipedia.org/wiki/Diamond_color): diamond color, from `J (worst)` to `D (best)`

* [clarity](https://en.wikipedia.org/wiki/Diamond_clarity): a measurement of how clear the diamond is `(I1 (worst), SI2, SI1, VS2, VS1, VVS2, VVS1, IF (best))`

* x: length in mm in x dimension `(0-10.74)`

* y: width in mm in y dimension `(0-58.9)`

* z: depth in mm in z dimension `(0-31.8)`

* depth: total depth percentage = z / mean(x, y) = 2 * z / (x + y) `(43-79)`

* table: width of top of diamond relative to widest point `(43-95)`

**Label column**:

* price: price in US dollars `($326-$18,823)`

The data is organized by increasing price - you can see the cheapest diamond on the first row at $326.

%md ## Preprocess data

So what do we need to do to get our data ready for use with `mlflow-apps`?

*Recall our goal*: We want to learn to predict the price of each diamond.  We refer to the price as our target "label".

*Features*: What can we use as features (info describing each row) to predict the `price` label?  We can use the rest of the columns!

Let's print the schema of our dataset to see the type of each column.

%md The DataFrame has categorical string-valued columns, namely the `cut`, `color`, and `clarity` columns. Apps in `mlflow-apps`, like the XGBoost GBT and sklearn Elastic Net apps, need numeric inputs, so let's index the categorical columns in our dataset.

%md #### Split data into training and test sets

We now split our dataset into separate training and test sets.  We can train and tune our model as much as we like on the training set, so long as we do not examine the test set - this helps us avoid problems such as [data dredging](https://en.wikipedia.org/wiki/Data_dredging). After we have a good model (based on the training set), we can validate it on the held-out test set to know with high confidence our well our model will make predictions on future (unseen) data.

%md #### Saving the data for use with mlflow-apps

Our model-training apps expect data stored in [parquet](https://en.wikipedia.org/wiki/Apache_Parquet) format, so we save our processed dataset into the following files:

* `train_diamonds.parquet`: training data for our models

* `test_diamonds.parquet`: test data for our models

In addition, to showcase using MLflow Python APIs to load back models and perform inference with them, we will save two dataframes:

* `diamond_predict`: features of 20 diamonds from `test_diamonds.parquet`

* `diamond_prices`: prices (labels) of the 20 diamonds in `diamond_predict`

%md ## Model Training with `mlflow-apps`

Now that we understand our data and have prepared it as a DataFrame with numeric values, let's learn how MLflow and mlflow-apps can be used to easily train models and make predictions on your data.