Training is walking downhill
A model's loss measures how wrong it is on the training data, as a function of its weights . Training means finding the weights where the loss is smallest. Gradient descent gets there one step at a time: compute the gradient , which points uphill, and step the other way.
The step size is the learning rate. For least squares the loss is exactly a bowl. Near its minimum it reads
where , the same matrix as in the normal equations of the Linear Regression Calculator. Most models aren't bowls everywhere, but close to a minimum almost every loss looks like one, so this picture explains most of what goes right and wrong in training.
Step 19 of 19. Loss below 0.001. Reached the bottom in 19 steps.
Condition number κ = 8. Plain gradient descent is stable for η below 2/λ_max = 0.5.
Each step multiplies the distance to the bottom by 1 − ηλ₁ = −0.6 along the first axis and by 1 − ηλ₂ = 0.8 along the second.
The eigenvalues set the shape
The eigenvectors of are the bowl's axes, and each eigenvalue is how steeply the bowl curves along its axis. The Eigenvalue and Eigenvector Calculator finds them for any matrix. Along axis , one step multiplies the distance to the bottom by
- If , the step falls short and the distance shrinks smoothly.
- If , the step overshoots to the other side of the valley, but by less than before. The path zig-zags inward.
- If , the overshoot is bigger than the distance it started with, and the loss blows up.
So the learning rate has a hard ceiling, , set by the steepest direction. Push the rate slider past it and watch the dots fly off.
Narrow bowls zig-zag
The ceiling comes from the steepest axis, but the shallowest axis decides how long the walk takes: there each step only shrinks the distance by . The ratio
is the condition number. With near 1 the bowl is round, the gradient points straight at the bottom, and a few steps do it. With a large the bowl is a long, narrow valley. A rate safe for the steep walls either bounces between them or creeps down them, and either way it barely moves along the floor. Set , , and to see it, then turn on momentum.
Momentum
Momentum remembers the last step and keeps part of it, like a ball rolling with inertia:
The bounces across the valley point in opposite directions, so they partly cancel. The steady drift along the floor points the same way every step, so it adds up. In a narrow bowl that gets to the bottom in far fewer steps. Momentum also raises the rate ceiling to , though a ball with too much speed overshoots and circles before it settles.
Why standardizing features helps
When one feature is measured on a much larger scale than another, say floor area in square meters next to the number of bedrooms, its column of is much longer, and gets a much larger eigenvalue in that direction. The bowl stretches and training crawls.
Standardizing, subtracting each feature's mean and dividing by its standard deviation, puts every feature on the same scale. For features that aren't strongly correlated, that makes the bowl nearly round.
κ = 9. At the best single learning rate, η = 0.2, the loss falls below 1% of where it started in 11 steps.
Standardizing fixes different scales but not correlation: two features that move together still leave a tilted, narrow valley. The directions of that valley are exactly what the Covariance Matrix Calculator measures. And for a model as small as linear regression you can skip the walk entirely and solve the normal equations directly. Gradient descent earns its keep on models with millions of weights, where forming and solving is out of reach.