RAG Engineering in Production

The answer is wrong long before the model sees it

A RAG system that returns a confident, well-written, wrong answer looks like a generation problem. It almost never is. By the time the model writes anything, every consequential decision has already been made: which retrieval strategy ran, how the documents were split, whether the embedding model understands your domain, whether anything reranked the candidates, and what actually got assembled into the prompt.

The model is the last component in the chain and the only one most teams instrument. That is why RAG problems are so hard to debug — the failure is visible at the end and caused at the beginning, and the standard pipeline raises no flag in between.

This series works through those upstream decisions one at a time. Each part names a specific, recurring cost — the retrieval tax, chunking debt, the precision gap, the evals blind spot, the staleness gap, the context assembly gap — because a failure with a name is a failure you can budget for and argue about, and an unnamed one just gets absorbed as "RAG is hard."

What makes these production problems

Every one of these failures is invisible at demo scale. Fifty curated documents and a handful of test queries will not surface a chunking strategy that severs claims from their qualifiers, an embedding model that has never seen your jargon, or an index that has been quietly wrong for eighteen months. They surface at production scale, on real queries, usually via a user who noticed before you did.

They also compound. Bad chunks degrade retrieval, which makes reranking look useless, which hides the fact that your evals only measure answer quality and never retrieval quality. Fixing one layer in isolation often changes nothing measurable, which is exactly why teams conclude that RAG tuning is guesswork. It is not — but you have to see the stack as a stack.