Convolutions, receptive fields, and why vision models see in layers
A convolutional network learns one small filter and applies it everywhere. That single constraint is what makes vision tractable — and it is still the right tool for a great many problems.
Connect every pixel of a modest 224 by 224 colour image to every unit of a modest hidden layer and you have around seventy-seven million weights in the first layer alone. It will overfit, it will be slow, and it will have learned nothing transferable: a filter that detects an edge in the top-left corner would have to be learned again, independently, for every other position.
Convolution fixes this with one assumption. A feature worth detecting in one part of an image is worth detecting everywhere, so learn the detector once and slide it across.
Two properties fall out for free
Parameter sharing: nine weights instead of one set per position, which is where the efficiency comes from. And translation equivariance: move the object in the input and its response moves correspondingly in the feature map, rather than the network simply failing to recognise it.
The receptive field
A unit in the first convolutional layer sees a three-by-three patch. A unit in the second layer sees a three-by-three patch of first-layer units, each of which saw three-by-three — so it depends on a five-by-five region of the original image. Stack more layers and the receptive field grows steadily; add pooling or stride and it grows much faster.
Where transformers took over, and where they did not
Vision transformers discard the convolutional prior entirely, cut the image into patches and let attention decide what relates to what. Given enough data they win, because the convolutional assumption is a constraint and constraints cost you something once you have data enough to learn the structure directly.
Given ordinary amounts of data, they usually lose. The convolutional prior is correct for images, and a correct prior is worth an enormous quantity of examples. For a civic dataset of tens of thousands of photographs rather than hundreds of millions, a convolutional backbone remains the sensible default.
- Stride and pooling trade spatial resolution for receptive field and speed — the reason segmentation architectures add paths that restore the resolution afterwards.
- Dilated convolutions widen the receptive field without adding parameters or losing resolution, by spacing the kernel out.
- Depthwise separable convolutions factor the operation into two cheaper ones, which is what makes on-device vision practical.
What it means for deployment
A convolutional model is comparatively legible. You can visualise which filters activated and where, and localise a decision to a region of the image. For a system whose classifications trigger municipal work orders, being able to point at the part of the photograph responsible is not a nicety.
Being able to show why is worth a few points of accuracy in anything that has to be defended to a person.