ZestCalc

Explainer

Backpropagation in Matrix Form

Training needs the gradient of the loss for every weight. Backpropagation gets all of them in one backward pass, and in matrix form it comes down to two rules: the error flows back through Wᵀ, and each weight's gradient is an outer product.

A gradient for every weight

To train a network, gradient descent moves every weight a little downhill on the loss LL. For that it needs ∂L/∂w\partial L / \partial w for every weight ww: how fast the loss changes when that one weight moves.

You could measure each one by nudging it and running the network again. That's slow, since a model with a million weights would need a million extra runs per training step. Backpropagation gets every gradient in one pass backwards through the network, at about the cost of one more forward pass. It's the chain rule from calculus, organized so that each step is a matrix product.

Measuring by nudging is still the best way to see what a gradient means, and it's how people check that a backpropagation implementation is right.

First layer W₁, one row per hidden unit

Output layer W₂

Nudging (W1)32(W_1)_{32} from 1 to 1.01 moves the loss from 2 to 2.06045.

Change ÷ 0.01 = 6.045. Backpropagation gives 6.

They differ slightly because the loss curves. A smaller nudge would get closer.

The weights of the network traced below, with the input x = (1, 2) and the target 1. Choose any weight to nudge it up by 0.01 and see how much the loss moves.

Forward, then backward

Here's a full forward and backward pass with every number shown. The network has the three moves from Neural Network Layers, with a squared error at the end:

z=W1x+b1,h=max⁡(0, z),y^=W2h+b2,L=12(y^−y)2z = W_1 x + b_1, \quad h = \max(0,\, z), \quad \hat{y} = W_2 h + b_2, \quad L = \tfrac{1}{2}(\hat{y} - y)^2

The backward pass runs the same layers in reverse order. At each layer, it takes the gradient with respect to the layer's output and produces two things: the gradient for the layer's weights, and the gradient to hand back to the layer before.

x₁1x₂2h₁h₂h₃ŷL

Forward

x=[12],y=1x = \begin{bmatrix} 1 \\ 2 \end{bmatrix}, \quad y = 1

Start with the input x = (1, 2). The network should output y = 1.

Step 0 of 10
Two inputs, three hidden units with ReLU, and one output. In forward steps, a node shows its value. In backward steps, an amber node shows the gradient of the loss with respect to it. The dashed unit is off for this input.

Two rules and a gate

Every step of the backward pass used one of three facts. Write δ\delta for the gradient of the loss with respect to a layer's output z=Wx+bz = Wx + b.

Back through the matrix. The gradient with respect to the layer's input is

∂L∂x=WTδ\frac{\partial L}{\partial x} = W^{T} \delta

Entry WijW_{ij} carried xjx_j forward into ziz_i, so it carries δi\delta_i backward into xjx_j. Using every connection in reverse is exactly multiplying by the transpose, which the Matrix Transpose Calculator shows. The shapes check out: WW is k×dk \times d, so WTδW^{T}\delta is (d×k)(k×1)=d×1(d \times k)(k \times 1) = d \times 1, one entry per input.

The weights. The gradient for the weights is an outer product:

∂L∂W=δ xT,∂L∂b=δ\frac{\partial L}{\partial W} = \delta\, x^{T}, \qquad \frac{\partial L}{\partial b} = \delta

Weight WijW_{ij} multiplies xjx_j on its way into ziz_i, so its gradient is δixj\delta_i x_j: the error at its output times the value at its input. A (k×1)(1×d)(k \times 1)(1 \times d) product is k×dk \times d, the same shape as WW, as it has to be for the update W−η ∂L/∂WW - \eta\, \partial L / \partial W.

Through the bend. ReLU passes zz through where it's positive and outputs 0 elsewhere. Backwards it's a gate: the gradient goes through unchanged for units that were on and stops at units that were off. A unit that stays off for every input never learns, which is why it's called a dead unit.

Other bends use the same idea with their own slope. For σ\sigma, the gradient is multiplied by σ(z)(1−σ(z))\sigma(z)(1 - \sigma(z)), which is never more than 14\tfrac{1}{4}. Stacking many of those shrinks the gradient layer by layer, one reason ReLU became the default.

A batch is one matrix product

Training uses a batch of examples at a time. Put the inputs in the rows of XX and each example's δ\delta in the rows of Δ\Delta. The batch gradient sums every example's outer product, and that sum is a single matrix product, the kind the Matrix Multiplication Calculator computes:

∂L∂W=∑nδ(n)x(n)T=ΔTX\frac{\partial L}{\partial W} = \sum_{n} \delta^{(n)} x^{(n)T} = \Delta^{T} X

Libraries that store the weights as d×kd \times k and compute Y=XWY = XW, as on Data as Vectors and Matrices, write the same thing transposed: XTΔX^{T}\Delta.

X, one example per row

1222011−1

Δ₁, one example per row

0−1300.125−0.375001.125100

All four at once: Δ₁ᵀX, a (3 × 4)(4 × 2) product. It equals the four outer products added up.

The result, rows h₁ to h₃

1−1−0.75−1.752.256.375
Δ1TX=[1−1−0.75−1.752.256.375]\Delta_1^{T}X = \begin{bmatrix} 1 & -1 \\ -0.75 & -1.75 \\ 2.25 & 6.375 \end{bmatrix}
Four training examples, one per row of X, and each one's error δ₁ at the hidden layer, one per row of Δ₁. The batch loss is the sum of the four losses, so the gradient is the sum of the four outer products.

Here the batch loss is a sum. Most libraries take the mean, which divides every gradient by the batch size and changes nothing else.

Each layer's backward pass is two matrix products, ΔTX\Delta^{T}X for the weights and ΔW\Delta W for the layer before, where the forward pass needed one. That's why a training step costs about three times as much as a prediction, and it's all matrix multiplication, the operation graphics cards are built for. Frameworks such as PyTorch and JAX record the forward pass and apply these rules automatically, but the rules themselves are the ones above.