Designing Production RAG Systems That Don't Fall Over

Most RAG demos work beautifully on ten documents and fall over on ten thousand. The gap between "it works on my laptop" and "it works in production" is almost entirely about the parts nobody puts in the demo: chunking strategy, retrieval quality measurement, and what happens when the model gets a context window full of irrelevant text.

The chunking problem is a data modeling problem

Treating chunking as a generic text-splitting step is the first mistake. Different document types need different chunking strategies:

• Structured docs (API references, specs) chunk best along their own headings — a chunk should be a complete, self-contained unit of meaning.
• Conversational or narrative text benefits from overlapping windows, since meaning depends on surrounding context.
• Tables and code should almost never be split mid-structure. A table cut in half is worse than no table at all.

A single fixed chunk_size across an entire corpus is a sign the system hasn't been tuned against real retrieval failures yet.

Hybrid search beats pure vector search

Pure embedding similarity search is good at "these are semantically related" and bad at "this is the exact string the user is looking for." Combining vector search with keyword (BM25) search and re-ranking the merged results consistently outperforms either approach alone, especially for technical or precise queries — product names, error codes, version numbers.

Grounded citations aren't optional

If the system can't point to the exact passage that produced an answer, you can't debug retrieval failures and users can't verify the answer. Every production RAG system should return the source chunks alongside the generated answer, not just the final text.

What actually breaks in production

The failures that show up after launch are rarely about the language model. They're about:

1. Stale indexes — content changes, but nobody re-embeds it.
2. Silent retrieval degradation — nothing errors, the top-k results are just quietly getting worse as the corpus grows.
3. No evaluation set — without a fixed set of query/answer pairs to test against, every change is a guess.

Building the evaluation harness before scaling the corpus is the single highest-leverage thing I've found in this work. It turns "does this change help?" from a vibe into a number.