Linear Discriminant Analysis with Pokemon Stats (20pts)
Discussion
LDA could be a possibly useful method to examine your final project data and predict classes (you need to determine what classes you might have). Here is a very short lab with a few useful examples.
Linear Discriminant Analysis is a popular technique for performing dimensionality reduction on a dataset. The new variables are chosen (and the data reprojected) in a way that maximizes the linear separability of a certain set of classes in the underlying data.
So for example, an ideal application of a two-component LDA reduction will look like this:
An LDA transform is useful as a preprocessing step when modeling classes because it transforms the space in such a way that algorithms which then go and draw those boundaries, like support vector machines, perform much better on the transformed data than on the original projections.
However, it is also useful as an EDA technique. In this application, LDA can be compared to PCA. LDA, meanwhile, is based on categorical labels, and creates new variables that maximize the linear distinguishability of the underlying dataset.
As an EDA technique it tells us a lot about the complexity of our problem, and tells us which classes are most easily distinguishable and why. As a preprocessing technique it improves model performance for any useful model we apply to the dataset afterwards, but particularly for linear ones, like SVMs.
Application
In this notebook we will try out using LDA for exploring the Pokemon dataset. Our goal is to predict the type of Pokemon based only on its stat totals. Let's see what happens.
Data munging (15pts)
1. Import pokemon data and display the first 3 rows
2. Check how many Pokemon have a type2 value:
In order to avoid stat munging due to Pokemon with combined types, we are going to focus only on Pokemon with a single type (e.g. no dual types allowed).
3. Standardize data before applying LDA. X is created for you below:
df = pokemon[pokemon['type2'].isnull()].loc[
:, ['sp_attack', 'sp_defense', 'attack', 'defense', 'speed', 'hp', 'type1']
]
X = df.iloc[:, :-1].values
# your code
y = df.iloc[:, -1].values
4. Apply LDA
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
lda = LinearDiscriminantAnalysis(n_components=3)
lda.fit(X_scale, y)
Assessing class-wise variable importance using LDA coefficients
PCA provides a components_ attribute to the fitted reducer, which allows us to directly access the vector components. LDA does not provide this attribute. This is because in LDA, the methodology for transforming a vector is a bit more complicated than a simple w.T * x reprojection.
An LDA instead provides a coef_ attribute, which is analogous, albeit more mathematically complicated. The magnitudes of the components in the coef_ tell us how heavily each of the features loads towards the separability of that class.
If a particular class has a particularly high-magnitude coefficient (direction, positive or negative, notwithstanding) then that variable signals that class very well. That variable will factor very heavily into the LDA preprojection. A low-magnitude coefficient, meanwhile, corresponds with a weak signal, and hence will be mostly rubbed out in the reprojection.
If a class has mostly low-magnitude coefficients, that means that it is not easily linearly separable! That class is relatively close to the mean of the dataset or (in the weaker cases) relatively close to a subset of other classes in the dataset.
5. The heatmap that follows demonstrates what this maps out to:
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
import seaborn as sns
import numpy as np
fig, ax = plt.subplots(1, 1, figsize=(12, 10))
sns.heatmap(pd.DataFrame(lda.coef_,
columns=df.columns[:-1],
index=[lda.classes_]),
ax=ax, cmap='RdBu', annot=True)
plt.suptitle('LDA Feature Coefficients')
pass
In this heatmap we see classes which are probably easier to separate, given their large variable coefficients, as well as classes which are probably much harder.
A good example of a separable class is the rock type. Rock type Pokemon load very strongly on all of Special Attack, Special Defense, Speed, and HP, implying that a combination of these stats makes up the Rock archetype. Other highly separable classes are the ghost type and the fighting type, which both sport some high-magnitude coefficients.
ice and water are two classes that have very low class separability. They both have coefficients that are mostly close to 0.
6. We can summarize this heatmap by looking at the absolute coefficient totals for each of the classes.
pd.Series(np.abs(lda.coef_).sum(axis=1), index=lda.classes_).sort_values().plot.bar(
figsize=(12, 6), title="LDA Class Coefficient Sums"
)
Again, we see that rock and ghost is much more
separable than water and ice.
The y-values in both the heatmap and the bar plot can be treated as indicial. Higher is better, but the numbers themselves are not particularly interpretable.
To see what this difference translates to and to understand how well we perform overall, we need to move on to applying our LDA.
Assessing linear classifier performance by applying the LDA projection
7. To start with, as with any dimensionality reduction technique, it is important to note that each additional component used by the model adds less and less "gain" to the reconstructions. For example, here are the top three explained variances of the LDA decomposition:
lda.explained_variance_ratio_
8. Recall that PCA picks values which maximize these values directly. LDA picks values that maximize the differences between classes.
X_hat = lda.fit_transform(X, y)
import matplotlib as mpl
colors = mpl.cm.get_cmap(name='tab20').colors
categories = pd.Categorical(pd.Series(y)).categories
ret = pd.DataFrame(
{'C1': X_hat[:, 0], 'C2': X_hat[:, 1], 'Type': pd.Categorical(pd.Series(y))}
)
fig, ax = plt.subplots(1, figsize=(12, 6))
for col, cat in zip(colors, categories):
(ret
.query('Type == @cat')
.plot.scatter(x='C1', y='C2', color=col, label=cat, ax=ax,
s=100, edgecolor='black', linewidth=1,
title='Two-Component LDA Decomposition')
.legend(bbox_to_anchor=(1.2, 1))
)
This is a convoluted mess! What do we learn? We've learned that different classes of Pokemon are not very linearly distinguishable by stats alone.
In summary this plot tells us that classifying Pokemon using stats alone is a non-linear problem. This in turn tells us that given the current set of features, predicting Pokemon type is a very hard classification problem. Most problems that are not linearly separable are very hard
9. Reflection (5pts)
Think of how to apply LDA to NCAA data. Are there any classes you can define/create?