A gradient for every weight
To train a network, gradient descent moves every weight a little downhill on the loss . For that it needs for every weight : how fast the loss changes when that one weight moves.
You could measure each one by nudging it and running the network again. That's slow, since a model with a million weights would need a million extra runs per training step. Backpropagation gets every gradient in one pass backwards through the network, at about the cost of one more forward pass. It's the chain rule from calculus, organized so that each step is a matrix product.
Measuring by nudging is still the best way to see what a gradient means, and it's how people check that a backpropagation implementation is right.
First layer W₁, one row per hidden unit
Output layer W₂
Nudging from 1 to 1.01 moves the loss from 2 to 2.06045.
Change ÷ 0.01 = 6.045. Backpropagation gives 6.
They differ slightly because the loss curves. A smaller nudge would get closer.
Forward, then backward
Here's a full forward and backward pass with every number shown. The network has the three moves from Neural Network Layers, with a squared error at the end:
The backward pass runs the same layers in reverse order. At each layer, it takes the gradient with respect to the layer's output and produces two things: the gradient for the layer's weights, and the gradient to hand back to the layer before.
Forward
Start with the input x = (1, 2). The network should output y = 1.
Two rules and a gate
Every step of the backward pass used one of three facts. Write for the gradient of the loss with respect to a layer's output .
Back through the matrix. The gradient with respect to the layer's input is
Entry carried forward into , so it carries backward into . Using every connection in reverse is exactly multiplying by the transpose, which the Matrix Transpose Calculator shows. The shapes check out: is , so is , one entry per input.
The weights. The gradient for the weights is an outer product:
Weight multiplies on its way into , so its gradient is : the error at its output times the value at its input. A product is , the same shape as , as it has to be for the update .
Through the bend. ReLU passes through where it's positive and outputs 0 elsewhere. Backwards it's a gate: the gradient goes through unchanged for units that were on and stops at units that were off. A unit that stays off for every input never learns, which is why it's called a dead unit.
Other bends use the same idea with their own slope. For , the gradient is multiplied by , which is never more than . Stacking many of those shrinks the gradient layer by layer, one reason ReLU became the default.
A batch is one matrix product
Training uses a batch of examples at a time. Put the inputs in the rows of and each example's in the rows of . The batch gradient sums every example's outer product, and that sum is a single matrix product, the kind the Matrix Multiplication Calculator computes:
Libraries that store the weights as and compute , as on Data as Vectors and Matrices, write the same thing transposed: .
X, one example per row
Δ₁, one example per row
All four at once: Δ₁ᵀX, a (3 × 4)(4 × 2) product. It equals the four outer products added up.
The result, rows h₁ to h₃
Here the batch loss is a sum. Most libraries take the mean, which divides every gradient by the batch size and changes nothing else.
Each layer's backward pass is two matrix products, for the weights and for the layer before, where the forward pass needed one. That's why a training step costs about three times as much as a prediction, and it's all matrix multiplication, the operation graphics cards are built for. Frameworks such as PyTorch and JAX record the forward pass and apply these rules automatically, but the rules themselves are the ones above.