A neuron is a dot product and a threshold
The smallest piece of a neural network is one neuron. It takes an input vector , multiplies it by a weight vector , adds a bias , and squashes the result:
The first part is the dot product that the Dot Product and Angle Calculator works through. The squashing function turns any into a number between 0 and 1, read as how sure the neuron is that belongs to class 1. It's exactly where .
So the neuron splits the plane along the line . On one side and it answers class 1, and on the other it answers class 0. The weights turn the line, since is always perpendicular to it, and the bias slides it without turning it. A single neuron like this is logistic regression.
17 of 20 points are on their own side of the line.
Chosen point, class 0:
σ(z) = , so the neuron answers class 1, which is wrong.
The starting line, , gets three points wrong. Raising to 1 turns it enough to fix all of them. Doubling , , and together doesn't move the line at all, since in the same places. It makes the neuron more confident instead, pushing closer to 0 or 1.
A layer is many neurons at once
A layer runs several neurons on the same input. Stack their weight vectors as the rows of a matrix and their biases into a vector , and the whole layer is one formula:
That's three moves:
- Multiply by . A linear map that turns, stretches, and shears the plane. The Linear Transformation Visualizer lets you drag one around.
- Add . A shift.
- Bend. Apply a nonlinear function to each entry. Most networks use , which keeps positive values and turns negative ones into 0. The S-shaped is usually kept for the final answer, where a probability is wanted.
Real layers have hundreds or thousands of neurons. This one has two, so its output is another point in the plane, and you can watch every move.
The input. Each point is a vector x = (x₁, x₂), and the two half circles interleave, so no straight line splits them.
Why the bend matters
Take away the bend and stack two layers:
That's one matrix and one shift. Composing linear maps gives a linear map, which is why the Matrix Multiplication Calculator can multiply ahead of time. However many layers you stack, a network without bends is a single layer in disguise, and a single neuron on top of it can only draw a straight boundary.
The bend breaks this. ReLU is linear on each side of 0 but not across it, so each one adds a crease to the map, and later layers fold creases on top of creases. That's enough to pull apart two classes that no straight line can separate. Step through the network below, then switch the bends off.
The input: two half circles that no straight line can split.
At the last step, the output neuron's straight line in the final plane traces out a bent boundary in the input plane. That's how a network carves curved regions out of nothing but matrices, shifts, and .
Where the weights come from
Nobody chose the numbers in , , , , and the output weights by hand. Training starts from random weights and moves every one of them a little downhill on a loss that measures how wrong the answers are, again and again. That's gradient descent. The hard part is finding the downhill direction for every weight at once, and that turns out to be more matrix multiplication, run backwards through the layers: backpropagation.
At scale nothing new happens. An image model's first layer might have a with 512 rows and 784 columns, and a batch of inputs goes through as one matrix product, as on Data as Vectors and Matrices. The three moves stay the same.