Knowledge / Software & AI

Software & AI · Architecture & retrieval · Written and maintained by Haink’s AI adoption team · Updated August 2026 · 9 min read

How to Build a Production-Grade RAG System

A RAG demo takes an afternoon; a RAG system that is accurate, fast and trustworthy in production takes engineering. The difference is almost entirely in retrieval quality, evaluation and guardrails — not in the choice of language model. Teams that struggle with RAG are usually struggling with chunking and retrieval, then blaming the model.

The core pipeline

The retrieval ceiling

The single most useful fact about a RAG system is that its answer quality cannot exceed its retrieval recall. If the chunk containing the answer was never fetched, no model in the world produces the right answer from what it was given — it either declines or invents. The generator can only lose quality the retriever has already secured.

Which makes end-to-end quality a product of two numbers, and gives it the same compounding shape as everything else in this field:

Read the rows rather than the columns. Moving generation fidelity from 90% to 98% — a model upgrade, a prompt rewrite, a week of work — buys about eight points. Moving recall from 70% to 95% buys twenty-five. The model is almost never the binding constraint, and teams spend their effort there because retrieval quality is harder to see: a bad answer looks like a model failure, and nobody checks whether the right chunk was even in the context.

Three decisions that are expensive to reverse

Most of a RAG system can be changed in an afternoon. Three things cannot, because changing them means rebuilding the index, and that is the difference between a tuning session and a migration.

1. Vector type: dense, sparse or hybrid

Dense vectors capture semantic similarity and are what most people mean by “vector search”. Sparse retrieval — BM25, or learned variants such as SPLADE — matches exact terms, which is what you need for part numbers, case references, drug names, error codes and any vocabulary the embedding model never saw. Hybrid runs both and merges the results.

Hybrid wins on most real corpora, and the cost is honest: two indexes to maintain rather than one, and a fusion step to tune. Corpora heavy in identifiers — legal, medical, engineering, anything with part numbers — are where dense-only retrieval fails most visibly, because a model that understands the meaning of a query has no way to know that A-4471-B matters more than the words around it.

2. Chunking strategy — and the overlap you pay for

Fixed-size chunking splits by token count, typically something like 512 tokens with 50 tokens of overlap. It is predictable and it splits ideas in half. Structure-aware chunking respects headings, tables and section boundaries — better retrieval, variable chunk sizes, more implementation work. Hierarchical chunking keeps parent-child links so a small precise chunk can be expanded to its surrounding context at generation time; it is the best of both and the most complex.

Overlap is where a decision that looks like a detail turns into a bill. Every token of overlap is re-embedded and re-stored, so the multiplier on your vector count is simply C / (C − O):

That matters twice over: once in storage and query latency, and again every time you re-embed the corpus.

3. Embedding model and dimensionality

The embedding model is a long-term commitment, because changing it invalidates every vector you have stored. Dimensionality is the visible trade-off — 1536-dimension embeddings against 3072-dimension ones roughly double storage and slow search as the index grows, for a precision gain that is real but rarely proportional.

The reassuring part is that re-embedding is cheap: at typical embedding prices, re-indexing a hundred million tokens costs a few thousand dollars at most, and often far less. The expensive part is the migration — running two indexes in parallel, comparing retrieval quality on the same evaluation set, and shifting traffic once the new one demonstrably wins. Design for that from day one, with a versioning layer and the ability to hold two indexes at once, and a model swap becomes a scheduled change rather than a rebuild.

Where RAG systems go wrong

Five failure modes account for most of it. Each is worth reading as a root cause rather than a symptom, because the symptom in every case is the same — the answer was wrong — and the symptom tells you nothing about which one you have.

A sixth is worth naming separately because it is organisational rather than technical: retrieval that ignores permissions. A RAG system indexes documents, and an index does not inherit the access control of the source by default. Metadata filtering at query time, driven by the requesting user’s permissions, has to be designed in — retrofitting it after the index exists is how a knowledge assistant becomes an information leak.

Evaluation is the real work

You cannot improve what you don't measure. A production RAG system needs an evaluation set of representative questions with expected behavior, plus automated scoring for two things: retrieval relevance (did we fetch the right chunks?) and answer quality (was the response correct, grounded and complete?). This is what lets you tune chunking, retrieval and prompts with evidence instead of intuition — and catch regressions before users do. Most of the engineering effort in a good RAG project goes here.

Measure the two halves separately, because an aggregate score hides which one is failing:

• Retrieval — recall@k, the share of questions where the correct chunk appears in the top k results. This is the ceiling from the section above, and it is the number to move first.
• Retrieval — precision and rank, how much of what came back was useful and how high the best chunk sat. This is what a re-ranker improves.
• Generation — groundedness, whether every claim in the answer traces to a retrieved chunk rather than to the model's prior.
• Generation — completeness, whether the answer used everything relevant it was given.
• Abstention behaviour, whether the system declines when nothing relevant was retrieved. Measured on questions with no answer in the corpus, which most evaluation sets forget to include.

Fifty to a hundred representative questions with expected behaviour is enough to start, and building that set is the highest-return day of work in the project. It also has to include questions the corpus cannot answer — otherwise you never measure the failure mode that damages trust fastest, which is a confident answer to a question the system had no basis for.

Techniques that move the needle

• Structure-aware chunking that respects headings, tables and semantic boundaries instead of fixed character counts.
• Hybrid retrieval combining dense vectors with keyword/BM25 search for both meaning and exact matches.
• Re-ranking with a cross-encoder to promote the best candidates into the model's context window.
• Query rewriting to expand or clarify the user's question before retrieval.
• Metadata filtering so retrieval respects permissions, recency and document type.
• Citations back to source so users (and you) can verify every answer.

Deployment and privacy

RAG can run entirely on your infrastructure: open-weight models served on your own GPUs, with the vector store — pgvector, Qdrant, Weaviate or Milvus — and the documents themselves inside your network. This matters when the knowledge base holds anything you would not paste into a third-party service.

Whether it should run there is a separate question with an arithmetic answer, and the answer is usually no on cost alone: the break-even against a hosted API sits near 340 tokens per second sustained around the clock. Self-hosting is bought for residency, change control, rate limits or an air gap — the full calculation is in on-premises versus cloud LLM deployment. Where the answer is yes, sizing the GPUs to measured latency and throughput targets is part of the design, and quoting that hardware alongside the software is how Haink delivers private RAG under one contract.

A production readiness checklist

1. Retrieval is hybrid and tuned to your corpus, with a re-ranker.
2. Chunking respects document structure.
3. An evaluation set scores retrieval and answer quality, run on every change.
4. Answers cite sources and decline gracefully when nothing relevant is found.
5. Guardrails cover prompt injection and out-of-scope queries.
6. Monitoring tracks quality and latency in production.

Related Resources

• LLM Applications & RAG
• Software & AI Development Services
• Private LLM for Pharma — a worked private RAG deployment with sizing anchors

Frequently Asked Questions

What are the components of a RAG system?

Ingestion, chunking, embeddings, a vector store, retrieval (ideally hybrid vector + keyword), re-ranking, generation with citations, and evaluation plus guardrails. Most of the quality comes from retrieval and evaluation, not the model itself.

What is the biggest limit on RAG answer quality?

Retrieval recall. If the chunk containing the answer was never fetched, no model produces the right answer from what it was given — it declines or it invents. End-to-end quality is roughly recall multiplied by generation fidelity, so at 80% recall one question in five is unanswerable whatever model sits downstream. Moving recall from 70% to 95% buys about twenty-five points of end-to-end quality; moving generation fidelity from 90% to 98% buys about eight. The model is almost never the binding constraint.

What chunk size and overlap should I use?

There is no universal answer, but there is a cost you should know before choosing. Every token of overlap is re-embedded and re-stored, so the multiplier on your vector count is the chunk size divided by chunk size minus overlap: 512 tokens with 50 overlap costs 11% extra, 512 with 128 costs 33%, and 256 with 50 costs 24%. Halving the chunk size at constant overlap costs more than doubling the overlap at constant size. Start with fixed-size chunking, measure recall, and move to structure-aware or hierarchical chunking when the evaluation set says the splits are hurting you.

Which is expensive to change later in a RAG system?

Three things, because changing them means rebuilding the index: the vector type (dense, sparse or hybrid), the chunking strategy, and the embedding model. Re-embedding itself is cheap — a hundred million tokens costs a few thousand dollars at most — but the migration is not, because it means running two indexes in parallel and comparing retrieval quality before shifting traffic. Design for dual indexes from day one and a model swap becomes a scheduled change rather than a rebuild.

Why does my RAG demo work but production RAG fails?

Demos hide weak retrieval. In production, poor chunking, vector-only retrieval, no re-ranking and no evaluation cause inaccurate answers. Investing in retrieval quality and an evaluation set is what makes RAG reliable.

What is hybrid retrieval in RAG?

Combining dense vector search (semantic meaning) with keyword/BM25 search (exact terms, names, codes). Hybrid retrieval is more accurate than vector-only search for most real corpora.

How do you prevent hallucinations in RAG?

Ground answers in retrieved context with citations, validate outputs, measure accuracy with an evaluation set, add a re-ranker so the model gets good context, and make the system decline when nothing relevant is retrieved.

Can a RAG system run on-premises?

Yes. Open-weight models on your own GPUs, with the vector store and documents inside your network, let RAG run fully private for sensitive knowledge bases.