SFO Survey Machine Learning Example

Each year, San Francisco Airport (SFO) conducts a customer satisfaction survey to find out what they are doing well and where they can improve. The survey gauges satisfaction with SFO facilities, services, and amenities. SFO compares results to previous surveys to discover elements of the guest experience that are not satisfactory.

The 2013 SFO Survey Results consists of customer responses to survey questions and an overall satisfaction rating with the airport. We investigated whether we could use machine learning to predict a customer's overall response given their responses to the individual questions. That in and of itself is not very useful because the customer has already provided an overall rating as well as individual ratings for various aspects of the airport such as parking, food quality and restroom cleanliness. However, we didn't stop at prediction instead we asked the question:

What factors drove the customer to give the overall rating?

Here is an outline of our data flow:

• Load data: Load the data as a DataFrame
• Understand the data: Compute statistics and create visualizations to get a better understanding of the data to see if we can use basic statistics to answer the question above.
• Create Model On the training dataset:
  • Evaluate the model: Now look at the test dataset. Compare the initial model with the tuned model to see the benefit of tuning parameters.
  • Feature Importance: Determine the importance of each of the individual ratings in determining the overall rating by the customer

This dataset is available as a public dataset from https://catalog.data.gov/dataset/2013-sfo-customer-survey-d3541.

Load the Data

Use display() to confirm the data has been loaded.

Understand the Data

Gain a basic understanding of the data by looking at the schema.

As you can see above there are many questions in the survey including what airline the customer flew on, where do they live, etc. For the purposes of answering the above, focus on the Q7A, Q7B, Q7C .. Q7O questions since they directly related to customer satisfaction, which is what you want to measure. If you drill down on those variables you get the following:

Column Name|Data Type|Description ---|---|---|--- Q7B_FOOD|INTEGER|Restaurants Q7C_SHOPS|INTEGER|Retail shops and concessions Q7D_SIGNS|INTEGER|Signs and Directions inside SFO Q7E_WALK|INTEGER|Escalators / elevators / moving walkways Q7F_SCREENS|INTEGER|Information on screens and monitors Q7G_INFOARR|INTEGER|Information booth near arrivals area Q7H_INFODEP|INTEGER|Information booth near departure areas Q7I_WIFI|INTEGER|Airport WiFi Q7J_ROAD|INTEGER|Signs and directions on SFO airport roadways Q7K_PARK|INTEGER|Airport parking facilities Q7L_AIRTRAIN|INTEGER|AirTrain Q7M_LTPARK|INTEGER|Long term parking lot shuttle Q7N_RENTAL|INTEGER|Airport rental car center Q7O_WHOLE|INTEGER|SFO Airport as a whole

The possible values for the above are:

0 = no answer, 1 = Unacceptable, 2 = Below Average, 3 = Average, 4 = Good, 5 = Outstanding, 6 = Not visited or not applicable

Select only the fields you are interested in.

Let's get some basic statistics such as looking at the average of each column.

Let's start with the overall rating.

The overall rating is only 3.87, so slightly above average. Let's get the averages of the constituent ratings:

Looking at the bar chart above - the overall average rating of the airport is actually lower than all of the individual ratings. This appears clearer using the bar chart visualization below.

So basic statistics can't seem to answer the question: What factors drove the customer to give the overall rating?

So let's try to use a predictive algorithm to see if these individual ratings can be used to predict an overall rating.

Create Model

Here use a decision tree algorithm to create a predictive model.

First need to treat responses of 0 = No Answer and 6 = Not Visited or Not Applicable as missing values. One of the ways you can do this is a technique called mean impute which is when we use the mean of the column as a replacement for the missing value. You can use a replace function to set all values of 0 or 6 to the average rating of 3. You also need a label column of type double so do that as well.

Define the machine learning pipeline.

Now call fit to build the model.

Now call display to view the tree.

Call transform to view the predictions.

Evaluate the model

Validate the model using root mean squared error (RMSE).

Save the model.

Feature Importance

Now let's look at the feature importances. The variable is shown below.

Feature importance is a measure of information gain. It is scaled from 0.0 to 1.0. As an example, feature 1 in the example above is rated as 0.0826 or 8.26% of the total importance for all the features.

Let's map the features to their proper names to make them easier to read.

Let's convert this to a DataFrame so you can view it and save it so other users can rely on this information.

As you can see below, the 3 most important features are:

1. Signs
2. Screens
3. Food

This is useful information for the airport management. It means that people want to first know where they are going. Second, they check the airport screens and monitors so they can find their gate and be on time for their flight. Third, they like to have good quality food.

This is especially interesting considering that taking the average of these feature variables told us nothing about the importance of the variables in determining the overall rating by the survey responder.

These 3 features combine to make up 65% of the overall rating.

You can also show this graphically.

So if you run SFO, artwork and shopping are nice-to-haves but signs, monitors, and food are what keep airport customers happy!