All articles
Deep learningResearchScaling

Scaling laws: why model size stopped being the interesting number

Loss falls as a predictable power law in compute, parameters and data. Working out which of the three to spend on changed the industry more than any architecture did.

8 min read

One of the more surprising empirical results in machine learning is that model quality improves in a way you can extrapolate. Plot loss against compute on log axes and you get a straight line over many orders of magnitude — a power law, holding across architectures and modalities.

Straight lines on log axes are how you plan a budget. They are the reason a lab will commit to a training run months before knowing what the model can do.

training compute (log)quality (1 − loss, log)power lawwhat people expecteda straight line on log axes
Returns diminish, but they diminish smoothly and predictably rather than flattening into a plateau. Each fixed multiple of compute buys a fixed reduction in loss.

The correction that reshaped the field

Early scaling work was read as saying: make the model bigger. Laboratories duly built enormous models trained on comparatively little data.

A later analysis showed that was badly unbalanced. For a fixed compute budget, parameters and training tokens should grow at roughly the same rate — and the large models of that era were substantially undertrained. A smaller model fed far more data outperformed models several times its size.

huge, undertrained1 rel.balanced1.6 rel.small, overtrained1.3 rel.same compute budget, different splits between size and data
Holding compute fixed, how you divide it between parameters and tokens changes the result more than most architectural choices do.

Then inference changed the objective again

Compute-optimal training minimises loss for a fixed training budget. It says nothing about serving. A model that is trained once and then answers billions of requests should be smaller and trained for far longer than the training-optimal point, because every parameter is a permanent tax on every request.

Where the extrapolation stops

  • The laws predict loss, and loss is not capability. Specific abilities appear at thresholds the smooth curve gives no warning of.
  • High-quality text is finite. Scaling data indefinitely runs into supply, which is why data curation and synthetic data became research priorities rather than plumbing.
  • Post-training now moves benchmark numbers more than a further order of magnitude of pretraining compute does.

The number that replaced parameter count

Parameter count survives as marketing because it is a single big number. It has stopped being informative: sparsity decouples it from compute, training duration decouples it from quality, and distillation decouples it from capability.

Ask what it cost to train, how many tokens it saw, and how much of it runs per request. Those three answer what parameter count only gestures at.