All articles
LLMsArchitectureEfficiency

Mixture of experts: bigger models that cost less to run

A sparse model holds a trillion parameters and uses a small fraction of them per token. That trade is what made the current generation of frontier models affordable to serve.

7 min read

For most of deep learning's history, every parameter participated in every prediction. Double the parameters and you doubled the cost of each forward pass. That relationship put a hard ceiling on model size long before it hit a ceiling on capability.

Mixture of experts breaks the link. The parameter count and the per-token compute become separate numbers, and only the second one shows up in your serving bill.

tokenroutertop-2expert 1expert 2 · activeexpert 3expert 4expert 5expert 6 · activeexpert 7expert 8every expertis stored2 of 8 run
A small router network scores each expert for each token and sends the token to the highest-scoring few. The other experts sit idle for that token.

How the routing works

Certain feed-forward layers are replaced by a bank of parallel copies — the experts. A small gating network produces a score per expert for each token, and only the top one or two run. Their outputs are combined, weighted by the gate's scores.

The router is trained by gradient descent along with everything else. Nobody assigns topics to experts, and inspecting a trained model rarely finds tidy specialisms like "the French expert". What emerges is subtler — often syntactic or positional rather than semantic.

The arithmetic that makes it worth doing

dense8×top-2 of 82×top-1 of 81×relative cost per token, holding total parameters fixed
With eight experts, routing to two costs a quarter of what a dense layer of the same total size would.

The catch is that every expert must be resident in memory even though most are idle. A sparse model is cheap in compute and expensive in VRAM, which changes what hardware it fits on rather than removing the constraint.

Load balancing, the failure mode nobody expects

Left alone, routers collapse. An expert that is slightly better early attracts more tokens, gets more gradient, becomes better still, and within a few thousand steps a handful of experts handle everything while the rest are dead weight.

What it costs you in practice

  • Batching is harder, because tokens in the same batch route to different experts and the hardware would rather do one big matrix multiply.
  • Expert parallelism spreads experts across devices, which turns routing into network traffic and makes interconnect bandwidth the bottleneck.
  • Fine-tuning is less predictable: updating a sparse model can shift routing decisions, changing which experts see which data.

Why it matters beyond the frontier labs

Sparsity is the clearest example of a broader pattern: capability per parameter has stopped being the interesting metric, and capability per unit of serving cost has taken over. That shift favours architectures that decide what work to skip.

The useful question is no longer how large the model is. It is how much of it runs when you ask it something.