How diffusion models make an image from noise
Image generators do not paint. They start from static and repeatedly subtract what a network predicts is noise, until what remains is a picture.
The intuitive model of an image generator is a painter: it decides what to draw and then draws it. That is not what happens, and the actual mechanism is more interesting.
Training begins by destroying things. Take a real image, add a small amount of Gaussian noise, then add a little more, and keep going for perhaps a thousand steps until nothing is left but static. This is the forward process, and it requires no learning — it is just arithmetic.
The network is trained to invert one step of it. Given a noisy image and a number saying how noisy it is, predict the noise that was added. That is the whole training objective.
Why this beats predicting the image directly
You could train a network to jump straight from noise to a finished picture. Models that tried this produced blurry averages, because when many outputs are plausible, the loss-minimising answer is the mean of all of them — and the mean of many faces is a smudge.
Diffusion sidesteps the problem by breaking one impossible prediction into a thousand easy ones. Removing a little noise from something that is nearly an image is a well-posed problem with a nearly unique answer. Chaining a thousand such steps traverses the distance that a single step could not.
Where the prompt enters
Text conditioning is bolted onto the same loop. The prompt is encoded into a vector by a separate text model, and that vector is fed into the denoiser at every step through cross-attention. The network is not choosing what to draw; it is being told, at each step, which direction of denoising counts as correct.
Latent space, and why it made this practical
Running a thousand denoising steps at full resolution is ruinous. Latent diffusion first compresses the image roughly eightfold in each dimension with an autoencoder, runs the entire loop in that smaller space, and decodes once at the end.
Guidance, and the dial everyone turns
Classifier-free guidance runs the denoiser twice per step — once with the prompt, once without — and extrapolates away from the unconditioned prediction. Turn the guidance scale up and outputs follow the prompt more literally while becoming more saturated and less varied. Turn it down and you get diversity and drift.
- Fewer steps is faster and blurrier; distilled samplers now reach acceptable quality in a handful of steps rather than a thousand.
- The seed fixes the starting noise, which is why the same prompt with the same seed reproduces the same image exactly.
- Negative prompts work by substituting them for the unconditioned branch of guidance — you are pushing away from them, not filtering them out.
What this implies about the output
A diffusion model has no representation of the scene it is producing. There is no object list, no geometry, no lighting model. There is a denoiser that has learned what plausible images look like at every noise level.
It is not drawing a hand. It is removing noise in a way that tends to leave hand-shaped regions — which is exactly why hands were hard for so long.