Skip to content

LLM engineering

Why your RAG demo works and your RAG product doesn’t

A retrieval demo on twenty documents proves almost nothing about the same system on twenty thousand. The four things that break in between.

6 min readByteWeave Studio

Retrieval-augmented generation demos beautifully. Point it at a folder of documents, ask three questions you already know the answers to, and it will answer all three. The demo is genuinely not a trick — the system really did retrieve and really did answer.

It is just that a corpus of twenty documents and a corpus of twenty thousand are different problems, and almost everything that separates them is invisible at demo scale.

At small scale, retrieval cannot fail

With twenty documents, nearly any embedding model puts the right chunk in the top five, because there are only a few hundred chunks and the competition is weak. Retrieval quality is not being tested. What you are seeing is the language model summarising a passage you handed it.

Scale the corpus and the same query now competes against thousands of near-duplicates: last year’s version of the policy, the draft, the regional variant, the slide deck that quotes it. Getting the right chunk into the context window becomes the actual engineering problem, and it was never exercised in the demo.

Chunking decides more than the model does

Most teams reach for a fixed token window with an overlap because that is what the tutorial did. It works acceptably on prose and badly on everything else. A table split across two chunks loses its header row, so the numbers arrive without column names. A clause split mid-sentence loses the condition it depended on.

Chunking on document structure rather than character count fixes more retrieval problems than swapping embedding models does. Keep a table whole. Keep a section with its heading. Carry the document title and section path into every chunk as a prefix, so a fragment retrieved on its own still says what it belongs to.

Nobody asks the questions you tested

Demo questions are well-formed and use the vocabulary of the documents. Real users type three words, use the internal abbreviation rather than the official term, ask two things at once, and refer to "the new one" without saying which.

This is where hybrid retrieval earns its cost. Dense embeddings handle paraphrase; keyword search handles the exact part number, the policy code and the internal acronym that has no semantic neighbours. Running both and merging results is unglamorous and reliably beats either alone.

The failure mode is confident, not absent

When retrieval misses at demo scale you notice, because there is nothing to answer from. At production scale retrieval almost always returns something, and a language model handed a plausible but wrong passage will write a fluent, well-structured, incorrect answer. Users cannot tell the difference, which is worse than an error message.

The fix is not a better prompt. It is a relevance threshold below which the system declines to answer, citations that point at the retrieved passage so a user can check, and an evaluation set that includes questions the corpus genuinely cannot answer. If the system never says "I don’t have that", it is not being honest, it is guessing.

  • RAG
  • LLM
  • Retrieval
  • Evaluation

More reading

Have a problem
worth solving?

Tell us what you're building. We'll help you figure out what's possible — and say so if we're not the right people for it.

Start a conversationSee the work