Quantization: how a model gets four times smaller and barely notices
Neural network weights are stored at far higher precision than they need. Dropping from 16 bits to 4 is mostly free — until, quite suddenly, it is not.
A model trained in 16-bit floating point stores every weight to roughly three decimal digits of precision. Almost none of that precision is doing anything. The weights were found by a noisy stochastic process and the network was regularised specifically to be insensitive to small perturbations.
Quantization exploits that slack: represent each weight with fewer bits, and reclaim the memory and bandwidth.
Why memory is the bottleneck, not maths
Generating one token requires reading every weight the model uses from memory. Modern accelerators can perform arithmetic far faster than they can fetch operands, so inference for a single request is bound by memory bandwidth rather than compute.
The curve, and the cliff
Eight bits is essentially free. Four bits costs a little, and modern methods recover most of it. Below four, quality degrades fast, and it degrades unevenly — reasoning and long-context behaviour break down well before fluency does, which makes the damage easy to miss in casual testing.
Outliers are the whole difficulty
Quantization maps a range of real values onto a small set of levels, so the range determines the resolution. In transformers a tiny number of activation channels carry values orders of magnitude larger than the rest, and those outliers stretch the range until every ordinary value collapses onto the same few levels.
- Per-channel scaling gives each channel its own range instead of one range for the whole tensor.
- Weight-only quantization leaves activations at full precision, sidestepping the worst outliers while still winning the memory back.
- GPTQ and AWQ use a small calibration set to decide which weights matter most and protect those, rather than rounding everything uniformly.
Quantization-aware training
Everything above is post-training: take a finished model and compress it. The alternative is to simulate the rounding during training so the optimiser can compensate. It produces better results at aggressive bit widths and costs a training run, which is why it is reserved for models that will be deployed at scale.
Post-training quantization is what you do to someone else's model. Quantization-aware training is what you do to your own.
How to know it worked
Perplexity barely moves under quantization and is therefore close to useless as a check. Evaluate on the task you actually care about, include long inputs, and compare against the unquantized model on the same inputs. A four-bit model that chats well and fails at structured extraction is a common and expensive surprise.