Regression with XGBoost and MLlib pipelines

This notebook uses a bike-sharing dataset to illustrate MLlib pipelines and the XGBoost machine learning algorithm. The challenge is to predict the number of bicycle rentals per hour based on the features available in the dataset such as day of the week, weather, season, and so on. Demand prediction is a common problem across businesses; good predictions allow a business or service to optimize inventory and to match supply and demand to make customers happy and maximize profitability.

For more information about the PySpark ML XgboostRegressor estimator used in this notebook, see XgboostRegressor.

Requirements

Databricks Runtime 12.2 LTS ML or below. sparkdl.xgboost is removed in Databricks Runtime 13.0 ML and above.

Load the dataset

The dataset is from the UCI Machine Learning Repository and is provided with Databricks Runtime. The dataset includes information about bicycle rentals from the Capital bikeshare system in 2011 and 2012.

Load the data using the CSV datasource for Spark, which creates a Spark DataFrame.

Data description

The following columns are included in the dataset:

Index column:

• instant: record index

Feature columns:

• dteday: date
• season: season (1:spring, 2:summer, 3:fall, 4:winter)
• yr: year (0:2011, 1:2012)
• mnth: month (1 to 12)
• hr: hour (0 to 23)
• holiday: 1 if holiday, 0 otherwise
• weekday: day of the week (0 to 6)
• workingday: 0 if weekend or holiday, 1 otherwise
• weathersit: (1:clear, 2:mist or clouds, 3:light rain or snow, 4:heavy rain or snow)
• temp: normalized temperature in Celsius
• atemp: normalized feeling temperature in Celsius
• hum: normalized humidity
• windspeed: normalized wind speed

Label columns:

• casual: count of casual users
• registered: count of registered users
• cnt: count of total rental bikes including both casual and registered

Call display() on a DataFrame to see a sample of the data. The first row shows that 16 people rented bikes between midnight and 1am on January 1, 2011.

Preprocess data

This dataset is well prepared for machine learning algorithms. The numeric input columns (temp, atemp, hum, and windspeed) are normalized, categorial values (season, yr, mnth, hr, holiday, weekday, workingday, weathersit) are converted to indices, and all of the columns except for the date (dteday) are numeric.

The goal is to predict the count of bike rentals (the cnt column). Reviewing the dataset, you can see that some columns contain duplicate information. For example, the cnt column equals the sum of the casual and registered columns. You should remove the casual and registered columns from the dataset. The index column instant is also not useful as a predictor.

You can also delete the column dteday, as this information is already included in the other date-related columns yr, mnth, and weekday.

Print the dataset schema to see the type of each column.

Split data into training and test sets

Randomly split data into training and test sets. By doing this, you can train and tune the model using only the training subset, and then evaluate the model's performance on the test set to get a sense of how the model will perform on new data.

Visualize the data

You can plot the data to explore it visually. The following plot shows the number of bicycle rentals during each hour of the day. As you might expect, rentals are low during the night, and peak at commute hours.

To create plots, call display() on a DataFrame in Databricks and click the plot icon below the table.

To create the plot shown, run the command in the following cell. The results appear in a table. From the drop-down menu below the table, select "Line". Click Plot Options.... In the dialog, drag hr to the Keys field, and drag cnt to the Values field. Also in the Keys field, click the "x" next to <id> to remove it. In the Aggregation drop down, select "AVG".

Train the machine learning pipeline

Now that you have reviewed the data and prepared it as a DataFrame with numeric values, you're ready to train a model to predict future bike sharing rentals.

Most MLlib algorithms require a single input column containing a vector of features and a single target column. The DataFrame currently has one column for each feature. MLlib provides functions to help you prepare the dataset in the required format.

MLlib pipelines combine multiple steps into a single workflow, making it easier to iterate as you develop the model.

In this example, you create a pipeline using the following functions:

• VectorAssembler: Assembles the feature columns into a feature vector.
• VectorIndexer: Identifies columns that should be treated as categorical. This is done heuristically, identifying any column with a small number of distinct values as categorical. In this example, the following columns are considered categorical: yr (2 values), season (4 values), holiday (2 values), workingday (2 values), and weathersit (4 values).
• XgboostRegressor: Uses the XgboostRegressor estimator to learn how to predict rental counts from the feature vectors.
• CrossValidator: The XGBoost regression algorithm has several hyperparameters. This notebook illustrates how to use hyperparameter tuning in Spark. This capability automatically tests a grid of hyperparameters and chooses the best resulting model.

For more information:
VectorAssembler
VectorIndexer

The first step is to create the VectorAssembler and VectorIndexer steps.

Next, define the model. To use distributed training (requires Databricks Runtime for Machine Learning 9.1 LTS ML or above), set num_workers to the number of available workers in the cluster. To use the all Spark task slots, set num_workers=sc.defaultParallelism.

The third step is to wrap the model you just defined in a CrossValidator stage. CrossValidator calls the XgboostRegressor estimator with different hyperparameter settings. It trains multiple models and selects the best one, based on minimizing a specified metric. In this example, the metric is root mean squared error (RMSE).

Create the pipeline.

Train the pipeline.

Now that you have set up the workflow, you can train the pipeline with a single call.
When you call fit(), the pipeline runs feature processing, model tuning, and training and returns a fitted pipeline with the best model it found. This step takes several minutes.

Make predictions and evaluate results

The final step is to use the fitted model to make predictions on the test dataset and evaluate the model's performance. The model's performance on the test dataset provides an approximation of how it is likely to perform on new data. For example, if you had weather predictions for the next week, you could predict bike rentals expected during the next week.

Computing evaluation metrics is important for understanding the quality of predictions, as well as for comparing models and tuning parameters.

The transform() method of the pipeline model applies the full pipeline to the input dataset. The pipeline applies the feature processing steps to the dataset and then uses the fitted Xgboost Regressor model to make predictions. The pipeline returns a DataFrame with a new column predictions.

A common way to evaluate the performance of a regression model is the calculate the root mean squared error (RMSE). The value is not very informative on its own, but you can use it to compare different models. CrossValidator determines the best model by selecting the one that minimizes RMSE.

You can also plot the results, as you did the original dataset. In this case, the hourly count of rentals shows a similar shape.

Save and reload the model