RLHF, and what alignment actually tunes
A base model predicts text. Everything you recognise as an assistant — the helpfulness, the refusals, the tone — was added afterwards by training on human preference.
A freshly pretrained language model is not an assistant. Ask it a question and it may continue with three more questions, because a list of questions is a perfectly plausible continuation of a question in the text it was trained on. It has capability and no manners.
Turning that into something usable takes two further stages, and it is worth being precise about what each one changes.
Stage one: supervised fine-tuning
Humans write ideal responses to a set of prompts, and the model is trained on those pairs exactly as it was pretrained — predict the next token. This is cheap and gets you most of the way to something that answers rather than continues.
Its limit is that it can only teach by example. Demonstrating a good answer says nothing about which of two acceptable answers is better, and writing demonstrations does not scale.
Stage two: learning from comparisons
People are unreliable at writing the best possible answer and reliable at picking the better of two. Preference tuning is built on that asymmetry. The model produces several responses, a human ranks them, and those rankings train a separate reward model that predicts which response a human would prefer.
The language model is then optimised to score well under that reward model, with a penalty for drifting too far from where it started. That penalty matters: without it, the policy discovers degenerate text that the reward model loves and humans find unreadable.
What it does and does not change
This is the most misunderstood part. Preference tuning does not install knowledge and it does not remove capability. The weights that could produce a harmful output are still there; the model has learned that such outputs score badly.
Reward hacking
The reward model is a proxy, and optimisers exploit proxies. Length bias is the standard example: raters mildly prefer thorough answers, the reward model learns that longer is better, and the policy discovers that padding raises its score. The result is a model that hedges, repeats the question back, and produces a bulleted summary nobody asked for.
- DPO skips the separate reward model and optimises the policy on preference pairs directly — simpler, cheaper, and now the default for most teams.
- Constitutional methods replace some human labels with a model critiquing its own output against written principles, which scales but inherits the critic's blind spots.
- Whose preferences are being encoded is a real question, and it is answered by whoever recruited the raters.
For anyone deploying one
Alignment is a statistical tendency, not a guarantee, and it was tuned against a distribution of prompts that may not resemble yours. If a failure would be consequential, the safeguard belongs in your system — validation, human review, a constrained output format — and not in a hope about the model's disposition.
Refusal is a behaviour the model learned. It is not a property of the weights.