Lecture 19 – Data 100, Summer 2021

by Suraj Rampure

adapted from Josh Hug, Joseph Gonzalez

Motivating Logistic Regression

In this lecture, we will look at data from the 2017-18 NBA season.

We are eventually going to want to perform binary classification, which is where we predict a 1 or 0. A reasonable thing to want to do given this data is to predict whether or not a team wins. Right now, the WL column consists of "W" and "L".

Let's fix that, so that wins are encoded as 1 and losses are encoded as 0.

There is a row for each team and each game in this dataset. It contains the FG_PCT (field goal percentage) for each team per game.

Let's try and get the field goal percentage difference between two teams in a single game. We will then try and use this value to predict whether or not a team wins, given their field goal percentage difference.

This data cleaning and EDA is not the point of this lecture, but you may want to come back to this and try and understand it.

Let's start by looking at a sns.jointplot of FG_PCT_DIFF and WON.

A reasonable thing to do here might be to model the probability of winning, given FG_PCT_DIFF.

We already know how to use ordinary least squares, right? Why not use it here?

We'll also jitter the data, to get a better picture of what it looks like. But the line of best fit that's being drawn is on top of the original, non-jittered data.

The green line drawn is a valid model. It is the line that minimizes MSE for this set of $x$ (FG_PCT_DIFF) and $y$ (WON) data.

But there are some issues:

We need a better model. Let's try and replicate the graph of averages from Lecture 12, on Simple Linear Regression. Recall, we

We will do the same thing here, albeit with slightly different code. Here, we will formally partition the $x$-axis into 20 bins.

We now know which "bin" each game belongs to. We can plot the average WON for each bin.

It seems like our red graph of averages does a much better job at matching the data than our simple linear regression line.

What is this graph of averages plotting? Since the $y$ axis is only 0s and 1s, and we took the mean of the $y$-values in each bin for a given $x$, the graph of average is plotting the proportion of times a team won, given their FG_PCT_DIFF. Remember, WON = 1 each time a team won.

Logistic regression aims to model the probability of an observation belonging to class 1, given some set of features.

What is this mystery sigma function, and why does sigma(x * 30) match our graph of averages so well? Well... we're getting there.

For now, consider these questions:

What are:

  1. $P(Y = 1 | X = 0.0283)$?
  2. $P(Y = 0 | X = 0.0283)$?
  3. $\frac{P(Y = 1 | X = 0.0283)}{P(Y = 0 | X = 0.0283)}$? In other words, how many wins are there for each loss?

The odds of an event are defined as the probability that it happens divided by the probability that it doesn't happen.

If some event happens with probability $p$, then $\text{odds}(p) = \frac{p}{1-p}$.

If we plot the odds of these probabilities, they look exponential:

But if we take the log of these odds:

We noticed that the log-odds grows linearly with $x$.

In the lecture slides, we formalize what this means, and how this allows us to arrive at the sigma function above.

The Logistic Function

In the slides, we show that our model is

$$P(Y = 1 | x) = \sigma(x^T \theta)$$

where $$\sigma(t) = \frac{1}{1 + e^{-t}}$$

Let's explore the shape of the logistic function, $\sigma$.

First, the vanilla curve $\sigma(x)$:

Now, we look at $\sigma(\theta_1 x)$, for several values of $\theta_1$:

Let's explore the shape of $\sigma(\theta_0 + \theta_1x)$, for different values of $\theta_0, \theta_1$. There's quite a bit going on here, so let's use plotly.

Logistic Regression with Squared Loss

We've chosen a model. It's now time to choose a loss function. Why not squared loss?

So squared loss worked just fine here. But that won't always be the case! Consider this manufacturered example.

Let's plot the loss surface for this toy data using squared loss with the model $\hat{y} = \sigma(\theta x)$, where $\theta$ and $x$ are both scalars.

This loss surface is not convex! Depending on where we start our optimization search, we'll end up with different results. Let's explore with scipy.optimize.minimize.

Not only is it not convex, leading to the weird issues above, but squared loss just isn't well-suited for a probability task. Since $\hat{y_i}$ is between 0 and 1, and $y_i$ is either 0 or 1, the squared loss for a single point $(y_i - \hat{y_i})^2$ is bounded between 0 and 1.

What this means in practice: even if our prediction is terrible, the squared loss is never that large.

Motivating Cross-Entropy Loss

Let's look at a new loss, called the log loss, for when our true observation is 1.

We can see that this penalizes wrong predictions far more than squared loss does.

How to read this plot: Suppose the observation we're trying to predict is actually in class 1. If our model gives an 80% chance of being in class 1, the loss is relatively small (around 0.25).

If we give only a 40% of being in class 1, the loss is larger (around 1).

If we give only a 5% chance of being in class 1, the loss is 3.

And if we give a 0% chance of being in class 1, the loss is infinite.

What about when the true observation is 0?

Much of the formal derivation is in the slides. But the equation for cross-entropy loss for a single observation is

$$\textrm{loss} = -y \log(\hat{y}) - (1-y)\log(1-\hat{y})$$

For us, since $\hat{y} = \sigma(x^T \theta)$, the expression for average cross-entropy loss is

$$R(\theta) = -\frac{1}{n} \sum_{i = 1}^n \big(y_i \log (\sigma(\mathbb{X}_i^T \theta)) + (1 - y_i) \log (1 - \sigma(\mathbb{X}_i^T \theta))\big)$$

Let's look at the loss surface for average cross-entropy loss, on our toy data from before.

We see the resulting optimal $\hat{\theta}$ is slightly different than the one that minimized MSE:

And lastly, we can determine the $\hat{\theta}$ that minimizes mean cross-entropy loss for our NBA dataset from earlier:

Again, this is different than the $\hat{\theta}$ that minimizes mean squared error for the NBA dataset:

Predicting Probabilities

We can manually call scipy.optimize.minimize to determine the model parameters that minimize average cross-entropy loss, as we did above. We can then predict probabilities.

Once again, scikit-learn can do this for us.

The lm.LogisticRegression model is what we want to use here. In order to recreate our specific model, there are a few parameters we need to set:

We can see that the optimal theta (here there's just one, because our model only has one feature) found via scikit-learn is the same that we found manually before. (Small deviations due to numerical precision issues.)

scikit-learn has a built-in .predict_proba method that allows us to get the predicted probabilities under our model.

This is saying that if FG_PCT_DIFF = 0.1, that is, if you shoot 10% better than your opponent, there is a 95.5% chance you will win.

We can also apply this to our entire training set at once.

These values are the same as we computed manually above, as well!

Making Classifications

scikit-learn also has an in-built .predict method. Let's see what it does:

How did it come up with these 1s and 0s?