%md # Build a linear regression model to predict a city's median home price from its population

%md ## Load and preprocess the data

This notebook uses an example dataset of housing prices in different cities. The dataset is included with Databricks. The goal is to build a simple model that predicts the median house price in a city from its population. 

%md

Some of the column names contain spaces, which are not compatible with PySpark. Rename the columns to replace spaces with underscores and shorten the names.

%md ## Visualize the data

To get a feel for the data, you can plot it using matplotlib.

%md ## Prepare the data for use with PySpark

PySpark machine learning algorithms require a single features column and a label column. Each row in the features column contains a vector of data points corresponding to the features used for prediction. The labels column contains the output label for each data point. 

Use the VectorAssembler method to create the features column from the input data. 

%md ## Build the linear regression model

**Goal**

* Predict y = 2015 median housing price

* Using feature x = 2014 population estimate

**References**

* [MLlib LinearRegression user guide](http://spark.apache.org/docs/latest/ml-classification-regression.html#linear-regression)

* [PySpark LinearRegression API](http://spark.apache.org/docs/latest/api/python/reference/api/pyspark.ml.regression.LinearRegression.html#linearregression)

%md

First, import the LinearRegression class and instantiate a linear regression model.

%md This notebook creates and compares two models, which use different values for the regularization parameter.

%md ## Make predictions

Use the `transform()` method on the model to generate predictions. 

The following code takes the first model (modelA) and creates a new table (predictionsA) containing both the label (original sales price) and the prediction (predicted sales price) based on the features (population).

%md ## Evaluate the model

You can evaluate the model's performance by calculating the root mean square error between the value predicted by the model (in the prediction column) and the actual value (in the label column).

Use the PySpark RegressionEvaluator. In this example modelA performed slightly better. 

%md ## Plot residuals versus fitted values

Residual analysis is an important step in evaluating a model's performance. Ideally, the model's residuals -- the difference between the predicted value and the actual value -- should be small and symmetric around 0. In this case there are some very large residuals and poor symmetry around 0. This is not surprising; housing prices depend on many more variables than a city's population. 

%md ## Linear regression plots

To plot the predicted values and actual values vs population, create a pandas DataFrame with columns for the population, price, and prediction from each model. The Koalas package allows you to use the pandas DataFrame API to access data in Apache Spark. 

%md ## Convert to pandas DataFrame and display the result

%md ## Create a scatterplot of the data and the predicted values from the two models

This code uses the matplotlib library imported in Cmd 11 and also imports the seaborn package.