Production RAG: What Actually Breaks When Real Customers Start Asking

You can build a working Retrieval-Augmented Generation demo in an afternoon. Embed some documents, drop them into a vector store, retrieve the top five chunks by cosine similarity, paste them into a prompt. It answers questions. It looks like magic in a screen recording.

Then you put it in front of real customers, and the gap opens up.

Someone asks a question in the third language you support. Someone asks about a policy that exists in two contradictory versions on your own website. Someone asks something your content genuinely does not cover, and the system answers anyway — fluently, confidently, wrongly. Twenty people ask at once while a re-crawl is running. A PDF that was scanned rather than typed turns into retrieval noise that quietly poisons every nearby answer.

None of this shows up in a demo, because a demo uses clean documents, one language, one user, and questions the builder already knows the answers to. Everything expensive about RAG lives in the distance between those two situations.

This guide is about that distance. It assumes you already know what RAG is — if you do not, start with our explainer on what a RAG chatbot is and how it works, then come back. What follows is the layer above: the engineering that decides whether a retrieval system survives contact with real users.

Every time context windows get bigger, someone declares RAG obsolete. It has not happened, and the reasons are structural rather than temporary.

Usable context is smaller than advertised context. A model that accepts 200,000 tokens does not reason equally well across all 200,000. The effect is well documented — the 2023 "Lost in the Middle" paper by Liu et al. showed accuracy sagging for information buried in the middle of long inputs, and every subsequent generation of long-context models has shipped with some version of the same caveat. Quality degrades before the hard limit does. Meanwhile a mid-sized company's actual knowledge base is not 200 pages; it is tens of thousands.

Fine-tuning changes behavior, not knowledge. This is the most expensive misconception in the field. Fine-tuning is excellent for teaching a model a format, a tone, or a reasoning pattern. It is a poor and unreliable way to teach it facts, and it degrades badly when the facts change — which they do, every week, in pricing, policies, and inventory.

Cost scales the wrong way. Stuffing an entire corpus into every request means paying for the whole corpus on every question. Retrieval means paying for the relevant few thousand tokens. At any real message volume, that difference is the entire margin.

Auditability is a requirement, not a feature. In finance, healthcare, and law, an answer without a traceable source is not usable. Retrieval produces that trace for free, because the system already knows which document each passage came from.

So the interesting question is no longer whether to retrieve. It is why so many retrieval systems are bad at it.

The single most common cause of bad answers is not the model, the embeddings, or the vector database. It is the corpus.

We have seen a customer knowledge base where roughly half the documents were near-duplicates of the other half — the same policy re-published with small formatting differences across a marketing site, a help center, and an archived PDF. Retrieval dutifully returned five chunks that were five copies of the same paragraph, so the model saw one narrow slice of evidence instead of five perspectives, and the top-k budget was spent on redundancy. Fixing it was unglamorous, manual data-pipeline work. It also improved answer quality more than any retrieval tuning we did that month.

The recurring offenders:

• Scanned PDFs and OCR damage. Text that reads fine to a human eye can be structurally shredded — column order scrambled, tables flattened into word soup. Embeddings of shredded text retrieve unpredictably.
• Boilerplate and consent banners. Crawl a website naively and every page carries the same cookie notice, nav menu, and footer. Now every chunk shares a large identical prefix, and semantic similarity between unrelated pages goes up. This is why our crawler strips consent and chrome before anything is indexed.
• Auth walls and thin pages. A login page or an empty category page has text, so a naive pipeline ingests it. It contributes nothing and dilutes everything.
• Contradictions nobody has noticed. Two pages state different refund windows. Retrieval finds both. The model picks one. Whichever it picks, someone is going to be told the wrong thing.

The practical rule: a quality gate belongs before indexing, in exactly one place in your codebase, applied by every path that can add content. When ingestion quality rules live in three places, they drift, and the answer quality difference between your onboarding flow and your re-crawl becomes a mystery nobody can reproduce. For the hands-on version of this, see our guide to building a clean chatbot knowledge base.

A single dense vector search is the retrieval equivalent of a first draft. Production retrieval is a pipeline with four or five distinct stages, and each one fixes a failure mode the others cannot.

Query expansion. Real user queries are short, misspelled, and full of internal shorthand. Expanding a query with synonyms and spelled-out acronyms before embedding it measurably improves recall — especially for the two- and three-word questions that dominate real chat traffic.

Hybrid search: dense plus sparse. Dense embeddings capture meaning but miss exact tokens: SKUs, error codes, model numbers, surnames. Keyword search (BM25 over a Postgres tsvector, in our case) nails those and misses paraphrase. Running both and fusing the results with Reciprocal Rank Fusion is the production default, not an optimization.

One detail that matters more than it should: keyword search is language-specific. Postgres stems "prices" to "price" only if you tell it the text is English. Turkish, German, Spanish, French, Italian, and Portuguese each need their own configuration, and languages without a built-in stemmer — Japanese, Korean, Chinese — need a deliberate fallback rather than an accidental one. A multilingual RAG system that stems every query as English silently loses recall in every other language. If you serve more than one market, read our multilingual chatbot guide alongside this section.

Reranking. Fusion gives you twenty or thirty plausible candidates. A cross-encoder reranker scores each one against the actual query and routinely promotes the genuinely correct passage from rank 15 into the top 3. If your system "keeps citing a page that is almost right," a missing rerank stage is the first place to look.

Caching, carefully. Identical questions should not re-run the whole pipeline. But cache the retrieval, keyed by query and top-k — not the generated answer — or you will serve a stale reply after the underlying document has been updated.

Every serious RAG system scores its retrieval before generating, and uses that score to instruct the model how much to trust the context: answer directly, answer with a caveat, or decline and hand off. This is the most important hallucination defense there is.

It is also the setting most likely to be wrong, because the defaults are usually invented rather than measured.

Here is a mistake worth learning from — it was ours. Our "high confidence" threshold was set at 0.82 cosine similarity, a number that sounds appropriately strict. Then we checked it against live traffic and found that fewer than 2% of real replies ever crossed it. Dense semantic matches rarely exceed roughly 0.80 even when the retrieved passage is obviously, exactly correct. So the model was being told "this context is only partial" on nearly every question, and it hedged — accurate answers wrapped in unnecessary uncertainty. The retrieval was fine. The ruler was wrong.

The fix was not to loosen everything. We pulled the sample of real conversations in the ambiguous band and read them: at 0.64 to 0.68 the answers were specific and correct, quoting exact plan limits and exact policy rules. But genuine gaps — questions about an integration we do not support — scored in the same range and correctly hedged. So we lowered the high bar to a level real matches can reach, kept the ambiguous middle band as "partial," and left the bottom "do not fabricate" guard exactly where it was.

The transferable lessons: calibrate thresholds against your own distribution, not intuition; read the actual conversations in the band you are tuning, because aggregate scores hide the difference between a good answer and a well-scored miss; and make every threshold an environment variable so a bad calibration is a one-minute rollback rather than a deploy.

Chunking looks like a formatting decision. It is actually a retrieval decision, and it is where a surprising share of "the bot cannot find something that is definitely in our docs" complaints originate.

Fixed-length splitting at, say, every 500 tokens will cut a table in half, separate a heading from the paragraph it introduces, and split a numbered procedure across two chunks so that neither one is usable on its own. Structure-aware splitting — respect headings and paragraph boundaries first, then merge upward toward a target size, then split long segments at sentence boundaries — costs a day to implement and pays for itself immediately. Our own pipeline targets roughly 800 tokens per chunk with 200 tokens of overlap, which is a reasonable starting point for prose-heavy business content.

The subtler problem is that a chunk, once isolated, loses the context that made it meaningful. "Standard delivery takes 3 to 5 business days" is a useless retrieval target if the chunk does not say whose delivery, which region, or which product line. Retrieved alone, it can be applied by the model to the wrong question entirely.

The fix is to enrich each chunk with its own provenance before embedding it — document title, section heading, source — so the embedded text carries the context a human reader would have had from the page around it. Anthropic popularized a version of this as "contextual retrieval" and reported a substantial drop in retrieval failures. It does not require an LLM pass to be worthwhile: a deterministic prefix built from the document's own metadata captures much of the benefit at zero marginal cost, which is what we run in production.

One warning from experience: if you change your chunking or contextualization strategy, you own a migration. Every existing chunk was embedded under the old scheme. Plan for a targeted, per-document re-index — never a blind mass re-embed of a production corpus.

Retrieval quality is what teams argue about. Reliability is what actually takes systems down.

Ingestion is a long-running, multi-stage, partially external process: fetch, extract, chunk, embed (a paid API call that can rate-limit), write, mark complete. Anything that runs for minutes across a network boundary will be interrupted, and the interruptions are not rare.

Our most instructive outage-shaped bug was silent. A user started a site crawl, then navigated away. The serverless function handling their request was killed mid-flight — after the chunks and embeddings had been written, but before the document was flagged as processed. No exception. No error in monitoring. Just a document that had been fully indexed but was permanently displayed as "indexing," and a user who re-crawled the same site four times in nineteen hours trying to make a badge change, then left.

That class of bug taught us three invariants worth stealing:

• Order your writes so the user-facing truth is set first. Flip the processed flag immediately after the content is durably written; do statistics, cache invalidation, and other bookkeeping afterward, each one bounded and best-effort. Never let an optional step sit between real work and the flag that records it.
• Make every job handler idempotent, then requeue aggressively. If a worker dies mid-run, the next sweep should be able to re-run the whole job safely. "Delete-then-insert" handlers make retries free.
• Add a self-healing sweep. A periodic job that looks for documents stuck in an unfinished state and re-enqueues them turns a permanent user-visible failure into a delay of a few minutes. This is the single highest-value piece of reliability work in a RAG pipeline, and almost nobody builds it before they have been burned.

Under concurrent load, add the boring infrastructure too: a real queue instead of fire-and-forget promises, connection-pool limits that account for embedding calls holding connections open, and per-tenant isolation so one customer's 900-page crawl cannot starve everyone else's live chat.

You cannot eyeball RAG quality. Every team believes it can, and every team is wrong, because the failure mode of a bad RAG system is a fluent, plausible, well-formatted answer that happens to be false. It reads exactly like a good one.

The minimum viable evaluation setup is smaller than people fear:

A golden set. Thirty to a hundred real questions with known-correct answers, drawn from your actual support history rather than invented. Re-run it after every change to retrieval, chunking, prompts, or model version. This is a regression test, and it should fail loudly.

Separate retrieval scoring from answer scoring. When quality drops, you need to know whether the right passage was not retrieved or was retrieved and then ignored. These have completely different fixes, and a single end-to-end score cannot distinguish them.

Monitor the confidence distribution over time. A shift in the histogram of retrieval scores is an early warning — usually that someone added a large batch of low-quality content, or that traffic has moved to topics your corpus does not cover.

Treat "I do not know" as your most valuable telemetry. Every low-confidence answer and every escalation is a labeled content gap. The teams whose assistants get visibly better over months are, almost universally, the ones who read that list weekly and write the missing page. Our chatbot analytics guide covers the metrics worth watching.

And measure deflection honestly. A conversation is not resolved because the customer stopped typing; it is resolved because they did not email you an hour later. If your platform cannot connect those two events, you are optimizing a number that flatters you.

Two RAG systems with identical architecture can succeed and fail in the same industry. The difference is usually not in the pipeline.

Domain language is real work. Healthcare abbreviations, financial instrument names, legal citation formats, and SKU conventions all break generic retrieval in specific ways. A finance customer asks about "the 3-year" and means a specific product; a generic embedding thinks about time. This is fixed with synonym dictionaries, metadata filters, and content written for the machine as well as the human — not with a bigger model.

Scope discipline beats capability. The fastest way to destroy trust in an assistant is to let it answer questions outside what it actually knows. An explicit boundary — this agent answers about our products, our policies, and our documentation, and hands off everything else — reduces embarrassing answers more than any retrieval improvement. It is also the fix we shipped after watching real assistants drift into generic, unhelpful territory.

Adoption is a launch requirement. An assistant nobody told the support team about gets undermined by the support team. Someone has to own the content gaps, read the escalations, and decide what the assistant is allowed to promise.

Enterprise trust has a checklist. Role-based access so retrieval respects who is asking, audit trails that show which source produced which answer, clear data residency, retention controls, and an unambiguous statement that customer conversations are not used to train public models. These are not features you add after a security review; they are the reason the review either passes or does not. Our chatbot security and privacy guide goes through what to verify in a vendor.

If you are building this in-house, the honest scope is not "a vector database and a prompt." It is a content quality gate, a multi-stage retrieval pipeline, calibrated confidence thresholds, a job queue with idempotent handlers and a self-healing sweep, an evaluation harness, and an analytics loop — plus the ongoing operational work of all of it. That is worth doing when retrieval quality is your product. It is a poor use of a small team's year when retrieval is a means to answering customer questions.

If you are buying, the checklist from this article becomes your diligence list. Ask a vendor: Do you use hybrid retrieval or dense only? Is there a reranking stage? How is the "I do not know" threshold set, and can I change it? What happens to a document if ingestion is interrupted halfway? Can I see which source produced a given answer? What happens to keyword search in my language? Which questions did the assistant fail to answer last week? A vendor who cannot answer those has built a demo.

This is the layer Chatloom exists to own. The pipeline described throughout this article — structure-aware chunking with contextual enrichment, query expansion, hybrid dense-plus-sparse retrieval with language-aware stemming, RRF fusion, cross-encoder reranking, calibrated confidence with a genuine "I do not know," idempotent ingestion jobs with a self-healing sweep, and a dashboard that shows you exactly which questions went unanswered — is what runs behind every agent on the platform, in ten languages, without an ML team on your side of the table.

Try it on your own content, today. Create a free account, paste your website URL, and watch the whole loop run — crawl, quality gate, chunking, embedding, hybrid retrieval — in about the time it took you to read this article. Ask it the five questions your customers actually ask. If the answers are grounded and it admits what it does not know, you have your evaluation. The free plan needs no card.

Want the detail first? See how our RAG engine works, or read the practical companion piece on training an assistant on your own data.