Retrieval-augmented generation, and why embeddings are the hard part
RAG turns a recall problem into a reading problem. Most of the engineering is not in the model — it is in deciding what to put in front of it.
A language model's knowledge is frozen at training time and stored diffusely across billions of weights. It cannot cite, cannot be updated without retraining, and cannot tell you whether it actually knows something or is producing a fluent guess.
Retrieval-augmented generation avoids all of that by not relying on the model's memory. Find the relevant documents first, put them in the prompt, and ask the model to answer from them. Recall becomes reading, and reading is something these models are extremely good at.
Embeddings, briefly
An embedding model maps a passage of text to a vector of a few hundred to a few thousand numbers, trained so that passages with similar meaning land near each other. "How do I report a broken streetlight" and "streetlight outage reporting process" contain almost no words in common and should end up close together. Keyword search cannot do that; embeddings can.
Chunking is where most systems fail
Documents must be split before they are embedded, and the split determines everything downstream. Chunk too large and a single vector has to represent several unrelated ideas, blurring it toward the average of all of them. Chunk too small and each vector loses the context that made it meaningful.
Why pure vector search is not enough
Semantic similarity fails on exactly the queries where precision matters most: identifiers, part numbers, statute references, names. A vector search for ordinance 12-4-B will happily return ordinance 12-4-C, because they are semantically almost identical and differ by one character that the embedding barely registers.
- Hybrid search combines vector similarity with keyword scoring, so exact matches are not lost to semantic smoothing.
- A reranker — a slower cross-encoder that reads the question and each candidate together — reorders the top fifty into a better top five. This is usually the single highest-return addition.
- Metadata filters applied before search beat filtering afterwards, because they stop irrelevant regions of the space from competing at all.
The context window is not free
Long-context models tempt you to skip retrieval and paste everything in. Two things argue against it. Attention cost grows quadratically with sequence length, so it is expensive. And retrieval accuracy within a long context degrades in the middle — models attend well to the beginning and the end of what they are given and less well to the centre.
What good looks like
A RAG system worth deploying answers only from retrieved material, cites which passage each claim came from, and says it does not know when retrieval comes back empty. That last behaviour has to be engineered deliberately — the default is to answer anyway.
The model is the easy part. The corpus, the chunking and the reranker are the product.