## Lecture 22: Principal Components Analysis – Data 100, Fall 2020¶

Notebook originally by Josh Hug (Fall 2019)

Edits by Anthony D. Joseph and Suraj Rampure (Fall 2020)

In [1]:
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np


We often create visualizations in order to facilitate exploratory data analysis. For example, we might create scatterplots to explore the relationship between pairs of variables in a dataset.

The dataset below gives the "percentage body fat, age, weight, height, and ten body circumference measurements" for 252 men.

http://jse.amstat.org/v4n1/datasets.johnson.html

For simplicity, we read in only 8 of the provided attributes, yielding the given dataframe.

In [2]:
#http://jse.amstat.org/datasets/fat.txt
df3 = pd.read_fwf("data/fat.dat.txt", colspecs = [(17, 21), (23, 29), (35, 37),
(39, 45), (48, 53), (73, 77),
(80, 85), (88, 93)], header=None, names = ["% fat", "density", "age", "weight", "height", "neck", "chest", "abdomen"])

Out[2]:
% fat density age weight height neck chest abdomen
0 12.3 1.0708 23 154.25 67.75 36.2 93.1 85.2
1 6.1 1.0853 22 173.25 72.25 38.5 93.6 83.0
2 25.3 1.0414 22 154.00 66.25 34.0 95.8 87.9
3 10.4 1.0751 26 184.75 72.25 37.4 101.8 86.4
4 28.7 1.0340 24 184.25 71.25 34.4 97.3 100.0

We see that percentage fat and density in g/cm^3 are almost completely redundant.

In [3]:
sns.scatterplot(data = df3, x = "% fat", y = "density");


By contrast, while there is a strong correlation between neck and chest measurements, the resulting data is very 2 dimensional.

In [4]:
sns.scatterplot(data = df3, x = "neck", y = "chest");


Age and height show a small correlation as peolpe seem to get slightly smaller with greater age in this dataset.

In [5]:
sns.scatterplot(data = df3, x = "age", y = "height");


We note that there is one outlier where a person is slightly less than 29.5 inches tall. While there are a extraordinarily small number of adult males who are less than three feet tall, reflection on the rest of the data from this observation suggest that this was simply an error.

In [6]:
df3.query("height < 40")

Out[6]:
% fat density age weight height neck chest abdomen
41 32.9 1.025 44 205.0 29.5 36.6 106.0 104.3
In [7]:
df3 = df3.drop(41)

In [8]:
sns.scatterplot(data = df3, x = "age", y = "height");


We can try to visualize more than 2 attributes at once, but the relationships displayed in e.g. the color and dot size space are much harder for human readers to see. For example, above we saw that density and % fat are almost entirely redundant, but this relationship is impossible to see when comparing the colors and dot sizes.

In [9]:
sns.scatterplot(data = df3, x = "neck", y = "chest", hue="density", size = "% fat");


Seaborn gives us the ability to create a matrix of all possible pairs of variables. This is can be useful, though even with only 8 variables it's still difficult to fully digest.

In [10]:
sns.pairplot(df3);


We should note that despite the very strong relationship between % fat and density, the numerical rank of the data matrix is still 8. For the rank to be 7, we'd need the data to be almost exactly on a line. We'll talk about techniques to reduce the dimensionality over the course of this lecture and the next.

In [11]:
np.linalg.matrix_rank(df3)

Out[11]:
8

## House of Representatives Voting Data¶

Next, let's consider voting data from the house of representatives in the U.S. during the month of September 2019. In this example, our goal will be to try to find clusters of representatives who vote in similar ways. For example, we might expect to find that Democrats and Republicans vote similarly to other members of their party.

In [12]:
from pathlib import Path
from ds100_utils import fetch_and_cache
from datetime import datetime
from IPython.display import display

import yaml

plt.rcParams['figure.figsize'] = (4, 4)
plt.rcParams['figure.dpi'] = 150
sns.set()

In [13]:
base_url = 'https://github.com/unitedstates/congress-legislators/raw/master/'
legislators_path = 'legislators-current.yaml'
f = fetch_and_cache(base_url + legislators_path, legislators_path)

def to_date(s):
return datetime.strptime(s, '%Y-%m-%d')

legs = pd.DataFrame(
columns=['leg_id', 'first', 'last', 'gender', 'state', 'chamber', 'party', 'birthday'],
data=[[x['id']['bioguide'],
x['name']['first'],
x['name']['last'],
x['bio']['gender'],
x['terms'][-1]['state'],
x['terms'][-1]['type'],
x['terms'][-1]['party'],
to_date(x['bio']['birthday'])] for x in legislators_data])


Using cached version that was downloaded (UTC): Tue Nov 17 07:50:34 2020

/opt/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:4: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
after removing the cwd from sys.path.

Out[13]:
leg_id first last gender state chamber party birthday
0 B000944 Sherrod Brown M OH sen Democrat 1952-11-09
1 C000127 Maria Cantwell F WA sen Democrat 1958-10-13
2 C000141 Benjamin Cardin M MD sen Democrat 1943-10-05
In [14]:
# February 2019 House of Representatives roll call votes

Out[14]:
chamber session roll call member vote
0 House 1 555 A000374 Not Voting
1 House 1 555 A000370 Yes
2 House 1 555 A000055 No
3 House 1 555 A000371 Yes
4 House 1 555 A000372 No
In [15]:
votes.merge(legs, left_on='member', right_on='leg_id').sample(5)

Out[15]:
chamber_x session roll call member vote leg_id first last gender state chamber_y party birthday
8638 House 1 551 K000391 No K000391 Raja Krishnamoorthi M IL rep Democrat 1973-07-19
13298 House 1 548 R000486 Yes R000486 Lucille Roybal-Allard F CA rep Democrat 1941-06-12
13176 House 1 547 R000616 Yes R000616 Harley Rouda M CA rep Democrat 1961-12-10
13018 House 1 541 R000395 Yes R000395 Harold Rogers M KY rep Republican 1937-12-31
5325 House 1 543 F000469 No F000469 Russ Fulcher M ID rep Republican 1973-07-19
In [16]:
def was_yes(s):
if s.iloc[0] == 'Yes':
return 1
else:
return 0

In [17]:
vote_pivot = votes.pivot_table(index='member',
columns='roll call',
values='vote',
aggfunc=was_yes,
fill_value=0)
print(vote_pivot.shape)

(441, 41)

Out[17]:
roll call 515 516 517 518 519 520 521 522 523 524 ... 546 547 548 549 550 551 552 553 554 555
member
A000055 1 0 0 0 1 1 0 1 1 1 ... 0 0 1 0 0 1 0 0 1 0
A000367 0 0 0 0 0 0 0 0 0 0 ... 0 1 1 1 1 0 1 1 0 1
A000369 1 1 0 0 1 1 0 1 1 1 ... 0 0 1 0 0 1 0 0 1 0
A000370 1 1 1 1 1 0 1 0 0 0 ... 1 1 1 1 1 0 1 1 1 1
A000371 1 1 1 1 1 0 1 0 0 0 ... 1 1 1 1 1 0 1 1 1 1

5 rows × 41 columns

In [18]:
vote_pivot.shape

Out[18]:
(441, 41)

This data has 441 observations (members of the House of Representatives including the 6 non-voting representatives) and 41 dimensions (votes). While politics is quite polarized, none of these columns are linearly dependent as we note below.

In [19]:
np.linalg.matrix_rank(vote_pivot)

Out[19]:
41

Suppose we want to find clusters of similar voting behavior. We might try by reducing our data to only two dimensions and looking to see if we can identify clear patterns. Let's start by looking at what votes were most controversial.

In [20]:
np.var(vote_pivot, axis=0).sort_values(ascending = False)

Out[20]:
roll call
555    0.249988
540    0.249896
530    0.249896
542    0.249783
537    0.249783
533    0.249711
534    0.249711
536    0.249711
543    0.249711
550    0.249628
552    0.249536
549    0.249536
546    0.249536
518    0.249433
517    0.249320
547    0.249320
545    0.249063
553    0.248765
525    0.248425
551    0.248240
531    0.247397
524    0.246389
526    0.246111
521    0.246111
529    0.244898
528    0.244230
527    0.243150
520    0.242378
523    0.241144
522    0.231796
539    0.231796
516    0.221461
538    0.216679
544    0.198066
515    0.112546
541    0.095218
554    0.078743
532    0.071153
535    0.051460
519    0.047398
548    0.043295
dtype: float64

We see that roll call 548 had very little variance. According to http://clerk.house.gov/evs/2019/roll548.xml, this bill was referring to the 2019 Whistleblower Complaint about President Trump and Ukraine. The full text of the house resolution for this roll call can be found at https://www.congress.gov/bill/116th-congress/house-resolution/576/text:

(1) the whistleblower complaint received on August 12, 2019, by the Inspector General of the Intelligence Community shall be transmitted immediately to the Select Committee on Intelligence of the Senate and the Permanent Select Committee on Intelligence of the House of Representatives; and

(2) the Select Committee on Intelligence of the Senate and the Permanent Select Committee on Intelligence of the House of Representatives should be allowed to evaluate the complaint in a deliberate and bipartisan manner consistent with applicable statutes and processes in order to safeguard classified and sensitive information.

We see that 421 congresspeople voted for this resolution, and 12 did not vote for this resolution. 2 members answered "present" but did not vote no, and 10 did not vote at all. Clearly, a scatterplot involving this particular dimension isn't going to be useful.

In [21]:
vote_pivot['548'].value_counts()

Out[21]:
1    421
0     20
Name: 548, dtype: int64

By contrast, we saw high variance for most of the other roll call votes. Most them had variances near 0.25, which is the maximum possible for a variable which can take on values 0 or 1. Let's consider the two highest variance variables, shown below:

In [22]:
vote_pivot['555'].value_counts()

Out[22]:
1    222
0    219
Name: 555, dtype: int64
In [23]:
vote_pivot['530'].value_counts()

Out[23]:
1    225
0    216
Name: 530, dtype: int64

Let's use these as our two dimensions for our scatterplot and see what happens.

In [24]:
sns.scatterplot(x='530', y='555', data=vote_pivot);


By adding some random noise, we can get rid of the overplotting.

In [25]:
vote_pivot_jittered = vote_pivot.copy()
vote_pivot_jittered.loc[:, '515':'555'] += np.random.