Predicting Insurance Cost with Demographic Data¶
By Shane Chambers & Guillermo Veron
insurance.head()
Data used in this project can be found here. This is a Medical Cost Personal Datasets dataset designed for machine learning exploration from Kaggle.
This dataset contains ~1400 observations, each with 7 dimensions.
Goal:¶
Predict the cost of insurance for an individual (Regression)
Search for some underlying structure in the data (Unsupervised clustering)
Approach:¶
Regression: Fit LR, KNN, and random forest models using both normalized and unnormalized data
Clustering: K-means clustering
Relationships Between Variables¶
Now we can explore the relationships between variables using seaborn's pairplot:
import seaborn as sns
sns.pairplot(insurance, plot_kws= {'alpha' : .2})
We begin to see some interesting trends, namely three distinct groups that appear in the charges vs. age plot. We can try to investigate this by coloring the plots by some of our categorical variables:
Pairplot Colored by Smoking Status:¶
sns.pairplot(insurance, hue = 'smoker', plot_kws= {'alpha' : .2})
Here we begin to see the age vs. charges groups separate, and we see a new trend in the relationship between charges and bmi.
Predict Charges With Regression: Methods¶
- Encode data, train-test-split (both normalized and unnormalized)
- KNN (hyper-parameter tuning of # of neighbors, K)
- Linear Regression (no hyper-parameter tuning)
- Random Forest (hyper-parameter tuning of n_estimators and max depth)
Results: KNN Prediction of charges: Normalized¶
# Visualize and label the point of optimal nearest neighbors
fig = plt.figure(figsize=(15, 10))
sns.lineplot(x = model_opt['num_of_neighbors'],
y = model_opt['MSE'])
plt.text(knn_min_X_coord + 3, knn_min_Y_coord, 'Optimal Number of Neighbors: ' + str(knn_min_X_coord))
# Get mean error
knn_norm_me = np.sqrt(mean_squared_error(y_test, y_pred_nn))
# Visualize
fig = plt.figure(figsize=(15, 10))
sns.scatterplot(X_test[..., 0], y_test)
sns.scatterplot(X_test[..., 0], y_pred_nn)
plt.text(.4, 50000, 'ME: ' + str(round(knn_norm_me)))
As we can see, our model does a good job predicting charges in the bottom 'group', however has trouble deciding the y variable in the two groups above it with higher charges. Now, let's try this same approach but with the unnormalized data:
Results: KNN Prediction of charges: Unnormalized¶
# Write a for loop to iterate through all possible # of neighbors and stor MSE for each iteration
num_of_neighbors_unnorm = []
knn_MSE_unnorm = []
for i in range(1, X_unnorm_test.shape[0]):
nnr = KNeighborsRegressor(n_neighbors=i)
nnr.fit(X_unnorm_train, y_unnorm_train)
knn_preds = nnr.predict(X_unnorm_test)
MSE = mean_squared_error(y_unnorm_test, knn_preds)
num_of_neighbors_unnorm.append(i)
knn_MSE_unnorm.append(MSE)
# Store the # of neighbors and corresponding MSE as a pandas dataframe
model_opt_unnorm = pd.DataFrame({
'num_of_neighbors' : num_of_neighbors_unnorm,
'MSE' : knn_MSE_unnorm
})
# Find the # of neighbors corresponding with the lowest MSE
knn_min_X_coord_unnorm = model_opt_unnorm[model_opt_unnorm['MSE'] == model_opt_unnorm['MSE'].min()].iloc[0]['num_of_neighbors']
# Find the lowest MSE for plotting purposes
knn_min_Y_coord_unnorm = model_opt_unnorm[model_opt_unnorm['MSE'] == model_opt_unnorm['MSE'].min()]['MSE']
# Instantiate, fit, predict
nnr = KNeighborsRegressor(n_neighbors=int(knn_min_X_coord_unnorm))
nnr.fit(X_unnorm_train, y_unnorm_train)
y_pred_nn_unnorm = nnr.predict(X_unnorm_test)
# Get mean error
knn_unnorm_me = np.sqrt(mean_squared_error(y_unnorm_test, y_pred_nn_unnorm))
# visualize
plt.close()
fig = plt.figure(figsize=(15, 10))
sns.scatterplot(X_unnorm_test['age'], y_unnorm_test)
sns.scatterplot(X_unnorm_test['age'], y_pred_nn_unnorm)
plt.text(20, 50000, 'ME: ' + str(round(knn_unnorm_me)))
Here we see that normalization helps the model, with a lower ME than using the unnormalized data. The model still however has trouble predicting higher charges, so we will investigate whether multiple linear regression can outperform KNN in terms of predictive power.
Results: Linear Regression Prediction of charges: Normalized¶
from sklearn.linear_model import LinearRegression
y_model = LinearRegression(fit_intercept=False).fit(X_train, y_train).predict(X_test)
LR_norm_me = np.sqrt(mean_squared_error(y_test, y_model))
fig = plt.figure(figsize=(15, 10))
sns.scatterplot(X_test[..., 0], y_test)
sns.scatterplot(X_test[..., 0], y_model)
plt.text(.4, 50000, 'ME: ' + str(round(LR_norm_me)))
So far, this is the best performing model in therms of ME, however the model still struggles to predict those with higher charges. Below we can investigate what happens when we train a linear regression model on unnormalized data:
Results: Linear Regression Prediction of charges: Unormalized¶
y_unnorm_model = LinearRegression(fit_intercept=False).fit(X_unnorm_train, y_unnorm_train).predict(X_unnorm_test)
x_ax = 'age'
LR_unnorm_me = np.sqrt(mean_squared_error(y_unnorm_test, y_unnorm_model))
fig = plt.figure(figsize=(15, 10))
sns.scatterplot(X_unnorm_test[x_ax], y_unnorm_test)
sns.scatterplot(X_unnorm_test[x_ax], y_unnorm_model)
plt.text(20, 50000, 'ME: ' + str(round(LR_unnorm_me)))
Interestingly, this model is the best performing one thus far with an ME of 6109. Now, let's try a random forrest approach to see if this model can outperform the previous attempts.
Results: Random Forest Prediction of charges: Normalized¶
from sklearn.model_selection import KFold, GridSearchCV
from sklearn.ensemble import RandomForestRegressor
kf = KFold(n_splits=5, shuffle=True, random_state=1000)
# Create dt instance
rf = RandomForestRegressor()
# Create grid search instance
gscv = GridSearchCV(
rf,
{"max_depth": range(1, 16), "n_estimators": range(2, 16)},
cv=kf,
n_jobs=-1
)
gscv.fit(X_train, y_train)
# Get cross-validation data
cv_df = pd.DataFrame(gscv.cv_results_)
piv_df = cv_df.pivot(index="param_max_depth",
columns="param_n_estimators",
values="mean_test_score").round(3)
piv_df.style.background_gradient("nipy_spectral", axis=None)
Here we can programatically extract the optimal n_estimators and max_depth from our hyperparameter tuning corresponding with the grid above:
rf = RandomForestRegressor(max_depth = opt_max_depth_norm, n_estimators = opt_n_est_norm)
# Fit
rf.fit(X_train, y_train)
# Predict
y_predicted = rf.predict(X_test)
# Test/Visualize
RF_norm_me = np.sqrt(mean_squared_error(y_test, y_predicted))
fig = plt.figure(figsize=(15, 10))
sns.scatterplot(X_test[..., 0], y_test)
sns.scatterplot(X_test[..., 0], y_predicted)
plt.text(.4, 50000, 'ME: ' + str(round(RF_norm_me)))
Here we have a new lowest ME, ande we can see that the model is able to begin to predict the charges variable as the number gets higher,a task previous models were unable to handle.
Finally, we can attempt to repeat this same workflow with random Forest modeling with the unnormalized data and compare models
Results: Random Forest Prediction of charges: Unnormalized¶
kf = KFold(n_splits=5, shuffle=True, random_state=1000)
# Create dt instance
rf = RandomForestRegressor()
# Create grid search instance
gscv = GridSearchCV(
rf,
{"max_depth": range(1, 16), "n_estimators": range(2, 16)},
cv=kf,
n_jobs=-1
)
gscv.fit(X_unnorm_train, y_unnorm_train)
# Get cross-validation data
cv_df = pd.DataFrame(gscv.cv_results_)
piv_df = cv_df.pivot(index="param_max_depth",
columns="param_n_estimators",
values="mean_test_score").round(3)
piv_df.style.background_gradient("nipy_spectral", axis=None)
rf = RandomForestRegressor(max_depth = opt_max_depth_unnorm, n_estimators = opt_n_est_unnorm)
# Fit
rf.fit(X_unnorm_train, y_unnorm_train)
# Predict
y_predicted_unnorm = rf.predict(X_unnorm_test)
# Test/Visualize
RF_unnorm_me = np.sqrt(mean_squared_error(y_unnorm_test, y_predicted_unnorm))
fig = plt.figure(figsize=(15, 10))
sns.scatterplot(X_unnorm_test[x_ax], y_unnorm_test)
sns.scatterplot(X_unnorm_test[x_ax], y_predicted_unnorm)
plt.text(20, 50000, 'ME: ' + str(round(RF_unnorm_me)))
This model performs slightly better than the model using normalized data, however this iteration is clearly able to predict charges based on those two unidentified 'groups' we have been referring to.
Summary of Models¶
The table below summarizes the mean error of each model created:
model_summary = pd.DataFrame({'Model' : ['KNN', 'KNN', 'Linear Regression', 'Linear Regression', 'Random Forest', 'Random Forest'],
'Normalized' : ['Y', 'N', 'Y', 'N', 'Y', 'N'],
'Score' : [knn_norm_me, knn_unnorm_me, LR_norm_me, LR_unnorm_me, RF_norm_me, RF_unnorm_me]})
model_summary
Clustering with unsupervised learning¶
We've found an interesting relationship between age and charges, which our models have struggled to handle. This may mean that there are groups within these data not listed in the variables given. To explore this, we will use unsupervised learning to determine the optimal number of clusters to see if there is any hidden structure in our data.
from sklearn.cluster import KMeans
insur_clustering = pd.get_dummies(insurance.copy())
insur_norm_clustering = normalize(insur_clustering)
def plot_elbow(dataset, max_clusters):
"""Plot elbow curve for k-means."""
inertias = []
for i in range(1, max_clusters + 1):
kmeans = KMeans(n_clusters=i, random_state=768797)
kmeans.fit(dataset)
inertias.append(kmeans.inertia_)
plt.plot(range(1, max_clusters + 1), inertias)
plt.title("Elbow Plot")
plt.xlabel("K")
plt.ylabel("SSD")
plot_elbow(insur_norm_clustering, 10)
cluster_data = insurance.copy()
# Instantiate
model = KMeans(
n_clusters=2,
random_state=1000,
)
# fit and predict
cluster_labels = model.fit_predict(pd.get_dummies(insurance))
#Make the cluster group its own column in the pandas df
cluster_data['group'] = cluster_labels
cluster_data['group'] = cluster_data['group'].replace([0, 1], ['g1', 'g2'])
#Re-make the pairplot, coloring by these clusters
sns.pairplot(cluster_data, hue = 'group', plot_kws= {'alpha' : .2})
We don't see any convincing grouping here, so we can try with k = 3 below:
cluster_data = insurance.copy()
# Instantiate
model = KMeans(
n_clusters=3,
random_state=1000,
)
# fit and predict
cluster_labels = model.fit_predict(pd.get_dummies(cluster_data))
#Make the cluster group its own column in the pandas df
cluster_data['group'] = cluster_labels
cluster_data['group'] = cluster_data['group'].replace([0, 1, 2], ['g1', 'g2', 'g3'])
#Re-make the pairplot, coloring by these clusters
sns.pairplot(cluster_data, hue = 'group', plot_kws= {'alpha' : .2})
It has become clear that the groups are being defined simply based on splitting the charges variable into 3 groups, perhaps because the scale of this variable is so large. Moving forward, we should repeat this clustering with normalized variables.
Conclusion/Discussion¶
We were successfully able to predict the charges variable. This Would be useful information for insurance companies to tailor their charges based on an individuals demographic data
The K-means clustering seemed to be dominated by the charges variable, possibly because of it's scale. For better results, this could be repeated with scaled data, or possibly another clustering algorithm such as DBSCAN.