Model assertions - Tabular example(Python)

Import Notebook
%pip install model_assertions
Python interpreter will be restarted. Processing /root/.cache/pip/wheels/28/2f/b6/cc7d7573c85985ccbc1a63ca6b1518350677255221b659728d/model_assertions-0.0.3-py3-none-any.whl Installing collected packages: model-assertions Successfully installed model-assertions-0.0.3 Python interpreter will be restarted. 
# Imports
import functools
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import seaborn as sns
import sklearn
import sklearn.datasets
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
 
from model_assertions.checker import Checker
from model_assertions.per_row import PerRowAssertion
# We'll be using the Boston housing prices dataset for this example
# The goal is to predict the price of a house from features
 
dataset = sklearn.datasets.load_boston()
print(dataset.DESCR)
df = pd.DataFrame(dataset.data, columns=dataset.feature_names)
df['PRICE'] = dataset.target
.. _boston_dataset: Boston house prices dataset --------------------------- **Data Set Characteristics:** :Number of Instances: 506 :Number of Attributes: 13 numeric/categorical predictive. Median Value (attribute 14) is usually the target. :Attribute Information (in order): - CRIM per capita crime rate by town - ZN proportion of residential land zoned for lots over 25,000 sq.ft. - INDUS proportion of non-retail business acres per town - CHAS Charles River dummy variable (= 1 if tract bounds river; 0 otherwise) - NOX nitric oxides concentration (parts per 10 million) - RM average number of rooms per dwelling - AGE proportion of owner-occupied units built prior to 1940 - DIS weighted distances to five Boston employment centres - RAD index of accessibility to radial highways - TAX full-value property-tax rate per $10,000 - PTRATIO pupil-teacher ratio by town - B 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town - LSTAT % lower status of the population - MEDV Median value of owner-occupied homes in $1000's :Missing Attribute Values: None :Creator: Harrison, D. and Rubinfeld, D.L. This is a copy of UCI ML housing dataset. https://archive.ics.uci.edu/ml/machine-learning-databases/housing/ This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University. The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic prices and the demand for clean air', J. Environ. Economics & Management, vol.5, 81-102, 1978. Used in Belsley, Kuh & Welsch, 'Regression diagnostics ...', Wiley, 1980. N.B. Various transformations are used in the table on pages 244-261 of the latter. The Boston house-price data has been used in many machine learning papers that address regression problems. .. topic:: References - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261. - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann. 
# We'll train a standard linear model and check its train/test performance
 
 
# Splitting into train, test
X = df.drop('PRICE', axis=1)
y = df['PRICE']
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, test_size=0.2, random_state=42)
 
# Fitting
reg_all = LinearRegression()
reg_all.fit(X_train, y_train)
 
# Train performance
y_train_predict = reg_all.predict(X_train)
rmse = (np.sqrt(mean_squared_error(y_train, y_train_predict)))
r2 = round(reg_all.score(X_train, y_train), 2)
print('The model performance for training set')
print('--------------------------------------')
print(f'RMSE is {rmse:.2f}')
print(f'R2 score is {r2:.2f}')
print('\n')
 
 
# Test performance
y_pred = reg_all.predict(X_test)
rmse = (np.sqrt(mean_squared_error(y_test, y_pred)))
r2 = round(reg_all.score(X_test, y_test),2)
print('The model performance for training set')
print('--------------------------------------')
print(f'Root Mean Squared Error: {rmse:.2f}')
print(f'R^2: {r2:.2f}')
print('\n')
The model performance for training set -------------------------------------- RMSE is 4.65 R2 score is 0.75 The model performance for training set -------------------------------------- Root Mean Squared Error: 4.93 R^2: 0.67 
# Here, we'll define a simple assertion that says the predicted price of a house should be positive
 
def pred_fn(df, model=None):
    X = df.values
    y_pred = model.predict(X)
    return pd.DataFrame(y_pred, columns=['Price'])
 
def output_pos(inp, out):
    return out[0] <= 0
 
predictor = functools.partial(pred_fn, model=reg_all)
checker = Checker(name='Housing price checker', verbose=False)
output_pos_assertion = PerRowAssertion(output_pos)
checker.register_assertion(output_pos_assertion.get_assertion(), 'Output positive')
predictor = checker.wrap(predictor)
 
outs = predictor(X_test)
checker.retrieve_errors()
Out[4]: 
# We can plot the predicted prices to see that two examples have negative prices
 
plt.rcParams.update({'font.size': 15})
plt.scatter(y_test, y_pred)
neg_inds = np.where(y_pred <= 0)[0]
plt.scatter(y_test.values[neg_inds], y_pred[neg_inds], c='red')
plt.xlabel('Actual House Prices ($1000)')
plt.ylabel('Predicted House Prices ($1000)')
plt.title('Actual Prices vs Predicted prices')
plt.show()