ZestCalc

Explainer

Neural Network Layers

A layer of a neural network is a matrix multiplication, a shift, and a bend. Watch a plane go through each part, see two layers untangle data that no straight line can split, and see why the bend is the part that matters.

A neuron is a dot product and a threshold

The smallest piece of a neural network is one neuron. It takes an input vector xx, multiplies it by a weight vector ww, adds a bias bb, and squashes the result:

z=w⋅x+b,σ(z)=11+e−zz = w \cdot x + b, \qquad \sigma(z) = \frac{1}{1 + e^{-z}}

The first part is the dot product that the Dot Product and Angle Calculator works through. The squashing function σ\sigma turns any zz into a number between 0 and 1, read as how sure the neuron is that xx belongs to class 1. It's exactly 12\tfrac{1}{2} where z=0z = 0.

So the neuron splits the plane along the line w⋅x+b=0w \cdot x + b = 0. On one side z>0z > 0 and it answers class 1, and on the other it answers class 0. The weights turn the line, since ww is always perpendicular to it, and the bias slides it without turning it. A single neuron like this is logistic regression.

−3−2−10123−3−2−10123x₁x₂
Class 1Class 0
1
0
0

17 of 20 points are on their own side of the line.

Chosen point, class 0:

z=(1)(0.26)+(0)(−1.43)+(0)=0.26z = (1)(0.26) + (0)(-1.43) + (0) = 0.26

σ(z) = 0.560.56, so the neuron answers class 1, which is wrong.

Twenty points of two kinds. The neuron computes z = w₁x₁ + w₂x₂ + b and answers class 1 where z > 0, the shaded side of the line z = 0. The arrow is the weight vector w, which always points across the line toward the shaded side.

The starting line, x1=0x_1 = 0, gets three points wrong. Raising w2w_2 to 1 turns it enough to fix all of them. Doubling w1w_1, w2w_2, and bb together doesn't move the line at all, since z=0z = 0 in the same places. It makes the neuron more confident instead, pushing σ(z)\sigma(z) closer to 0 or 1.

A layer is many neurons at once

A layer runs several neurons on the same input. Stack their weight vectors as the rows of a matrix WW and their biases into a vector bb, and the whole layer is one formula:

h=max⁡(0, Wx+b)h = \max(0,\, Wx + b)

That's three moves:

  1. Multiply by WW. A linear map that turns, stretches, and shears the plane. The Linear Transformation Visualizer lets you drag one around.
  2. Add bb. A shift.
  3. Bend. Apply a nonlinear function to each entry. Most networks use ReLU⁡(z)=max⁡(0,z)\operatorname{ReLU}(z) = \max(0, z), which keeps positive values and turns negative ones into 0. The S-shaped σ\sigma is usually kept for the final answer, where a probability is wanted.

Real layers have hundreds or thousands of neurons. This one has two, so its output is another point in the plane, and you can watch every move.

W=[−212.52.5],b=[3−1]W = \begin{bmatrix} -2 & 1 \\ 2.5 & 2.5 \end{bmatrix}, \quad b = \begin{bmatrix} 3 \\ -1 \end{bmatrix}
−1−0.500.511.52−1−0.500.511.52x₁x₂
Class 1Class 0
xx

The input. Each point is a vector x = (x₁, x₂), and the two half circles interleave, so no straight line splits them.

Step 0 of 3
The first layer of the network below, applied to two interleaved half circles. The faint grid starts as the square grid of the input plane and goes through the same steps as the points. The view refits at each step, so watch the axis numbers.

Why the bend matters

Take away the bend and stack two layers:

W2(W1x+b1)+b2=(W2W1) x+(W2b1+b2)W_2(W_1 x + b_1) + b_2 = (W_2 W_1)\,x + (W_2 b_1 + b_2)

That's one matrix and one shift. Composing linear maps gives a linear map, which is why the Matrix Multiplication Calculator can multiply W2W1W_2 W_1 ahead of time. However many layers you stack, a network without bends is a single layer in disguise, and a single neuron on top of it can only draw a straight boundary.

The bend breaks this. ReLU is linear on each side of 0 but not across it, so each one adds a crease to the map, and later layers fold creases on top of creases. That's enough to pull apart two classes that no straight line can separate. Step through the network below, then switch the bends off.

−1−0.500.511.52−1−0.500.511.52x₁x₂
Class 1Class 0
xx

The input: two half circles that no straight line can split.

Step 0 of 6
The whole network: two layers of two units, then one output neuron. Its weights were found by gradient descent on these 40 points and rounded to halves so they're easy to read.

At the last step, the output neuron's straight line in the final plane traces out a bent boundary in the input plane. That's how a network carves curved regions out of nothing but matrices, shifts, and max⁡(0,z)\max(0, z).

Where the weights come from

Nobody chose the numbers in W1W_1, b1b_1, W2W_2, b2b_2, and the output weights by hand. Training starts from random weights and moves every one of them a little downhill on a loss that measures how wrong the answers are, again and again. That's gradient descent. The hard part is finding the downhill direction for every weight at once, and that turns out to be more matrix multiplication, run backwards through the layers: backpropagation.

At scale nothing new happens. An image model's first layer might have a WW with 512 rows and 784 columns, and a batch of inputs goes through as one matrix product, as on Data as Vectors and Matrices. The three moves stay the same.