ZestCalc

Explainer

Gradient Descent

Training a model means walking downhill on its loss. See why the learning rate has a hard limit set by the largest eigenvalue, why a narrow bowl makes the walk zig-zag, what momentum adds, and why standardizing features speeds training up.

Training is walking downhill

A model's loss measures how wrong it is on the training data, as a function of its weights ww. Training means finding the weights where the loss is smallest. Gradient descent gets there one step at a time: compute the gradient ∇f(w)\nabla f(w), which points uphill, and step the other way.

w←w−η ∇f(w)w \leftarrow w - \eta\,\nabla f(w)

The step size η\eta is the learning rate. For least squares the loss is exactly a bowl. Near its minimum w∗w^{*} it reads

f(w)=12(w−w∗)TH (w−w∗)+constant,f(w) = \tfrac{1}{2}(w - w^{*})^{T} H\,(w - w^{*}) + \text{constant},

where H=XTXH = X^{T}X, the same matrix as in the normal equations of the Linear Regression Calculator. Most models aren't bowls everywhere, but close to a minimum almost every loss looks like one, so this picture explains most of what goes right and wrong in training.

0.4
4
0.5

Step 19 of 19. Loss below 0.001. Reached the bottom in 19 steps.

Condition number κ = 8. Plain gradient descent is stable for η below 2/λ_max = 0.5.

Each step multiplies the distance to the bottom by 1 − ηλ₁ = −0.6 along the first axis and by 1 − ηλ₂ = 0.8 along the second.

Contour lines of the loss, closer together where the bowl is steeper. Each dot is one gradient step from the hollow circle toward the bottom at the center. Tap the plot to start somewhere else.

The eigenvalues set the shape

The eigenvectors of HH are the bowl's axes, and each eigenvalue λi\lambda_i is how steeply the bowl curves along its axis. The Eigenvalue and Eigenvector Calculator finds them for any matrix. Along axis ii, one step multiplies the distance to the bottom by

1−η λi.1 - \eta\,\lambda_i .
  • If 0<ηλi<10 < \eta\lambda_i < 1, the step falls short and the distance shrinks smoothly.
  • If 1<ηλi<21 < \eta\lambda_i < 2, the step overshoots to the other side of the valley, but by less than before. The path zig-zags inward.
  • If ηλi>2\eta\lambda_i > 2, the overshoot is bigger than the distance it started with, and the loss blows up.

So the learning rate has a hard ceiling, η<2/λmax⁡\eta < 2/\lambda_{\max}, set by the steepest direction. Push the rate slider past it and watch the dots fly off.

Narrow bowls zig-zag

The ceiling comes from the steepest axis, but the shallowest axis decides how long the walk takes: there each step only shrinks the distance by 1−ηλmin⁡1 - \eta\lambda_{\min}. The ratio

κ=λmax⁡λmin⁡\kappa = \frac{\lambda_{\max}}{\lambda_{\min}}

is the condition number. With κ\kappa near 1 the bowl is round, the gradient points straight at the bottom, and a few steps do it. With a large κ\kappa the bowl is a long, narrow valley. A rate safe for the steep walls either bounces between them or creeps down them, and either way it barely moves along the floor. Set λ1=5\lambda_1 = 5, λ2=0.25\lambda_2 = 0.25, and η=0.3\eta = 0.3 to see it, then turn on momentum.

Momentum

Momentum remembers the last step and keeps part of it, like a ball rolling with inertia:

v←βv−η ∇f(w),w←w+v.v \leftarrow \beta v - \eta\,\nabla f(w), \qquad w \leftarrow w + v .

The bounces across the valley point in opposite directions, so they partly cancel. The steady drift along the floor points the same way every step, so it adds up. In a narrow bowl that gets to the bottom in far fewer steps. Momentum also raises the rate ceiling to 2(1+β)/λmax⁡2(1 + \beta)/\lambda_{\max}, though a ball with too much speed overshoots and circles before it settles.

Why standardizing features helps

When one feature is measured on a much larger scale than another, say floor area in square meters next to the number of bedrooms, its column of XX is much longer, and H=XTXH = X^{T}X gets a much larger eigenvalue in that direction. The bowl stretches and training crawls.

Standardizing, subtracting each feature's mean and dividing by its standard deviation, puts every feature on the same scale. For features that aren't strongly correlated, that makes the bowl nearly round.

κ = 9. At the best single learning rate, η = 0.2, the loss falls below 1% of where it started in 11 steps.

The same two features before and after standardizing. The first feature is spread three times as wide as the second, so its eigenvalue is 9 times larger. Each run uses the best single learning rate for its bowl.

Standardizing fixes different scales but not correlation: two features that move together still leave a tilted, narrow valley. The directions of that valley are exactly what the Covariance Matrix Calculator measures. And for a model as small as linear regression you can skip the walk entirely and solve the normal equations directly. Gradient descent earns its keep on models with millions of weights, where forming and solving HH is out of reach.