import numpy as np
import pandas as pd

import plotly.express as px
import plotly.graph_objects as go
from plotly.subplots import make_subplots

A Brief Introduction to Deep Learning

In this notebook, we provide a very quick (shallow?) introduction to neural networks and deep learning. We review the basic challenge of binary classification and linear decision functions and then show how features can be composed to express more complex decision surfaces. We then build a basic neural network to learn the feature functions and ultimately build more complex models for image classification.

Quick Review of Logistic Regression

We start by reviewing logistic regression. We construct a linearly separable data set and show how a logistic regression model fits this data.

n = 50
np.random.seed(42)
x = np.random.randn(4*n, 2) + 3*np.tile([[1,1], [-1,1], [-1,-1], [1,-1]],(n, 1))
y = x[:,0]>0
data = pd.DataFrame(np.hstack([x,y[:,np.newaxis]]), columns=["X1", "X2", "Y"]).sample(frac=1)
pos_ind = data["Y"]==1.0
pos_scatter = go.Scatter(x=data.loc[pos_ind,"X1"], y=data.loc[pos_ind,"X2"],
                         mode="markers", marker_symbol="cross", name="Pos")
neg_scatter = go.Scatter(x=data.loc[~pos_ind,"X1"], y=data.loc[~pos_ind,"X2"], 
                         mode="markers", name="Neg")
go.Figure([pos_scatter, neg_scatter])

We fit a logistic regression model.

from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
model.fit(data[["X1", "X2"]], data['Y'])

LogisticRegression()

The following block of code generates the prediction surface.

def plot_predictions(predict_fn):
    u = np.linspace(-10, 10, 100)
    (x0, x1) = np.meshgrid(u,u)
    X = np.vstack([x0.flatten(), x1.flatten()]).T
    Y_hat = predict_fn(X)
    return go.Contour(x=X[:,0], y=X[:,1], z=Y_hat)

go.Figure([pos_scatter, neg_scatter, 
           plot_predictions(lambda X: model.predict_proba(X)[:,1])])

Notice that in the above plot we assign near zero probability of being a "plus" to the region on the left and a near one probability to the region on the right. Also notice that in the middle there is a transition region as the probability being a "plus" goes from zero to one.

Non-linearly Separable Data

We can modify the above data slightly to construct a data set that is no longer linearly separable. Can you find a decision line that would separate this data into "plus" and "circle" regions?

n = 50
np.random.seed(42)
x = np.random.randn(4*n, 2) + 3*np.tile([[1,1], [-1,1], [-1,-1], [1,-1]],(n, 1))
y = np.logical_xor(x[:,0]>0, x[:,1]>0)
data = pd.DataFrame(np.hstack([x,y[:,np.newaxis]]), columns=["X1", "X2", "Y"]).sample(frac=1)
pos_ind = data["Y"]==1.0
pos_scatter = go.Scatter(x=data.loc[pos_ind,"X1"], y=data.loc[pos_ind,"X2"],
                         mode="markers", marker_symbol="cross", name="Pos")
neg_scatter = go.Scatter(x=data.loc[~pos_ind,"X1"], y=data.loc[~pos_ind,"X2"], 
                         mode="markers", name="Neg")
go.Figure([pos_scatter, neg_scatter])

When we fit a logistic regression classifier to this data we no longer get an effective model.

model = LogisticRegression()
model.fit(data[["X1", "X2"]], data['Y'])

LogisticRegression()

go.Figure([pos_scatter, neg_scatter, 
           plot_predictions(lambda X: model.predict_proba(X)[:,1])])