How a neural network actually learns

The language around deep learning borrows heavily from biology, and the borrowing does more harm than good. Nothing fires. Nothing is inspired. A trained network is a very large collection of numbers arranged so that pushing data through them produces a useful answer, and training is the process of nudging those numbers until it does.

That is not a deflation. What follows is genuinely remarkable — it is just mechanical rather than mystical, and the mechanics are worth understanding if you are going to deploy one of these things somewhere consequential.

The forward pass

Data enters at the left. Each unit in the next layer takes every value from the previous layer, multiplies each by its own weight, adds them up, adds a bias, and passes the result through a non-linear function. That non-linearity is the whole reason depth is worth anything: without it, stacking a hundred layers of multiplication and addition collapses algebraically into a single layer, and you have an expensive way to draw a straight line.

Repeat through every layer and you arrive at an output. On the first pass, with weights initialised to small random values, that output is noise.

Being wrong, precisely

The output is compared against the correct answer using a loss function — a single number that says how wrong this prediction was. Squared error for a continuous quantity, cross-entropy for a classification. The choice matters more than it looks, because the loss function is the only definition of "good" the system will ever have. It will optimise exactly what you measured, including the parts you did not mean.

Backpropagation

Now the useful part. For a network with billions of weights, you need to know how the loss would change if you nudged each individual weight — a partial derivative per weight. Computing those one at a time would be hopeless.

Backpropagation computes all of them in a single backward sweep. It is the chain rule from calculus applied layer by layer: the gradient at the output is propagated backwards, and at each layer the local derivative is combined with what came from above. The cost of the backward pass is roughly the cost of the forward pass, regardless of how many weights there are. That single fact is why deep learning is practical at all.

Gradient descent

With a gradient for every weight, each one moves a small step in the direction that reduces the loss. How small is the learning rate, and it is the most consequential number in the whole setup. Too large and training diverges. Too small and it crawls, or settles into the first mediocre solution it finds.

In practice this is not done over the entire dataset at once. Batches of examples are used, which makes each step noisy — and the noise turns out to help, knocking the optimiser out of narrow valleys that would not have generalised anyway.

• One epoch is one pass over the training data. Training runs for many.
• Overfitting is the network memorising the training set instead of the pattern in it. The tell is training loss still falling while validation loss rises.
• Regularisation — dropout, weight decay, early stopping — exists to make memorising harder than generalising.

What the layers end up doing

Nobody assigns roles to layers. But in a trained vision network the early layers reliably respond to edges and simple gradients, the middle layers to textures and parts, and the late layers to whole objects. That hierarchy is not designed. It emerges because it is an efficient way to compress the problem, and gradient descent finds efficient compressions.

This is also why transfer learning works. The early layers of a network trained on millions of general images have learned something close to a universal visual vocabulary, and that vocabulary transfers to a task with far too little data to learn it from scratch.

Why any of this matters in deployment

A network does not know anything. It has a set of weights that produced low loss on the distribution it was trained on. Feed it data from a different distribution — a new city, a different camera, winter instead of summer — and the guarantee simply does not apply. The model will still produce a confident answer, because producing an answer is all it does.

A model is a compression of its training data. Ask it about something that was not in there and it will answer anyway.

That is the argument for narrow, well-understood models over general ones in civic infrastructure. Not because they are more sophisticated, but because you can state what distribution they were trained on and notice when reality has left it.