Lecture 17 – Gradient Descent

by Josh Hug (Fall 2019) and Joseph Gonzalez (Spring 2020)

• slides
• video playlist
• code
• code HTML

Important: This lecture is taken from both Fall 2019 and Spring 2020.

• In order to follow the lecture, you should be familiar with the ideas from Discussion 1 Problem 2 (Calculus).
• The reference to Homework 6 Problem 7 in 17.2 should be a reference to Homework 5 Problem 3.
• In Homework 7, you will get more practice with learning rates and gradient descent.
• There is an updated version of the Loss Game mentioned in 17.3.

	Video	Quick Check
17.1 Gradient descent in one dimension. Convexity.		17.1
17.2 Various methods of optimizing loss functions in one dimension.		17.2
17.3 Gradient descent in multiple dimensions. Interpretation of gradients.		17.3
17.4 Stochastic gradient descent (SGD). Comparison between gradient descent and SGD.		17.4