9 RAG Architectures You Should Actually Know in 2026
Every few months someone declares RAG dead because context windows got bigger. Then they ship a chatbot over 40,000 internal PDFs and rediscover why retrieval exists.
The more useful observation is this: many RAG failures begin at retrieval, not generation. The model may be working exactly as expected. It simply answered using the four irrelevant chunks it was given. So the interesting question is not only "which LLM" but also "which retrieval architecture," and there are more options than the tutorial version suggests.
Here are nine worth knowing, roughly in order of how much machinery they add.
1. Naive RAG
The baseline everyone starts with.
Chunk your documents at a fixed size with some overlap, embed each chunk, dump the vectors in a store. At query time, embed the question, pull the top k nearest neighbors, paste them into the prompt, generate.
Where it breaks:
- Fixed-size chunking cuts arguments in half. The claim is in chunk 12, the evidence is in chunk 13, and you retrieved chunk 12.
- Embedding similarity is not relevance. "How do I cancel my subscription" and "Why you should never cancel your subscription" are close in vector space and opposite in usefulness.
- Top k is fixed. Some questions need one passage, some need twelve, and this pipeline gives both the same five.
- There is no signal for retrieval failure. If nothing relevant exists, you still get five chunks and a confident answer.
Use it when: you need a baseline, the corpus is small, and the stakes are low. Every other architecture on this list is a patch on one of the four problems above, so it helps to know which one you actually have.
Research basis: Lewis et al. introduced the original RAG formulation (arXiv:2005.11401), while Karpukhin et al. presented Dense Passage Retrieval for open domain question answering (arXiv:2004.04906).
2. Retrieve and Rerank
The single highest-return upgrade over naive RAG, and it is not close.
Split retrieval into two stages with different cost profiles. Stage one uses a bi-encoder: query and documents are embedded separately, document vectors are precomputed, and approximate nearest neighbor search over millions of them takes milliseconds. Optimize this stage for recall and pull 50 to 100 candidates.
Stage two uses a cross-encoder. It takes the query and one document together in a single forward pass and scores relevance directly, with full attention across both. This is far more accurate and far more expensive, which is exactly why you only run it on the shortlist.
ColBERT-style late interaction sits between the two. It keeps per-token embeddings and scores with MaxSim, so it captures more than a single vector without paying full cross-encoder cost.
Use it when: your retrieval recall is decent but the top 3 results are noisy. Which is most of the time.
Cost: added latency that depends on the reranker, candidate count, hardware, and serving setup. Measure it on your own pipeline instead of assuming a fixed number.
Research basis: BERT passage reranking demonstrated the value of a stronger second stage ranker (arXiv:1901.04085). ColBERT introduced late interaction between token level query and document representations (arXiv:2004.12832).
3. Hybrid Search
Dense embeddings are good at meaning and bad at exact strings. Ask for error code E4021, part number RX-88B, or a function called parse_manifest, and semantic similarity will confidently hand you something adjacent and wrong.
BM25 has handled this since the 1990s. Run it alongside dense retrieval and merge the two result lists.
The clean way to merge is Reciprocal Rank Fusion:
score(d) = Σ 1 / (k + rank_i(d))with k usually set to 60. RRF uses ranks instead of scores, so you skip the miserable problem of normalizing a cosine similarity against a BM25 score that has no fixed range.
Use it when: your domain has identifiers, code, jargon, legal citations, drug names, or any vocabulary where the exact token matters. In practice: most enterprise corpora.
Research basis: BM25 remains a core sparse retrieval method (Robertson and Zaragoza). Reciprocal Rank Fusion was introduced as a simple way to combine ranked result lists (Cormack et al.).
4. Query Transformation
The user's question and the passage that answers it often do not live in the same neighborhood of embedding space. Questions are short and interrogative. Answers are long and declarative. So rewrite the query before you search.
Four variants worth knowing:
Multi-query (RAG-Fusion). Have the LLM write three to five paraphrases of the question, retrieve for each, fuse with RRF. Cheap insurance against one bad phrasing.
HyDE. Ask the LLM to hallucinate a plausible answer document, embed that, and use it as the search vector. Answer-to-answer similarity beats question-to-answer similarity surprisingly often. The catch is real: on domains the model knows nothing about, the fake document drifts and drags retrieval with it.
Decomposition. Break a multi-hop question into sub-questions, retrieve for each, then compose. "Which of our EU vendors failed the 2024 audit and also handles PII" is two lookups pretending to be one.
Step-back prompting. Ask a more general question first to pull in background concepts, then retrieve for the specific one. Helps when the specific question is too narrow to match anything.
Cost: one or more extra LLM calls before you have retrieved anything. On latency-sensitive products this is the layer people cut first.
Research basis: The main methods discussed here come from work on hypothetical document embeddings (arXiv:2212.10496), query rewriting (arXiv:2305.14283), RAG Fusion (arXiv:2402.03367), and step back prompting (arXiv:2310.06117).
5. Hierarchical and Structured Indexing
The chunking dilemma: small chunks retrieve precisely and lack context, large chunks carry context and retrieve poorly. Instead of picking a compromise size, decouple what you search from what you return.
Small-to-big / parent document retrieval. Index small chunks. When one hits, return the parent section it belongs to. Precision on the search side, context on the generation side.
Sentence window. Embed individual sentences, return the sentence plus its neighbors.
RAPTOR. The most interesting one here. Cluster your chunk embeddings, summarize each cluster with an LLM, embed the summaries, then cluster and summarize those. You end up with a tree where leaves are raw text and higher nodes are progressively more abstract summaries. Retrieval searches the whole tree at once, so a question like "what is this report arguing overall" hits a summary node while "what was Q3 revenue in Malaysia" hits a leaf. One index, both granularities.
Use it when: documents are long and questions vary between detail lookup and whole-document synthesis.
Research basis: RAPTOR introduced recursive clustering and summarization to create a tree organized retrieval index (arXiv:2401.18059).
6. Self-Correcting and Adaptive RAG
Everything above retrieves once and hopes. This family adds a feedback loop.
Self-RAG trains the model to emit reflection tokens. One decides whether retrieval is needed at all for this query. Others critique retrieved passages for relevance, check whether the generated claim is actually supported by them, and rate overall usefulness. The model can decline to retrieve, and it can throw out its own draft.
CRAG (Corrective RAG) puts a lightweight evaluator in front of generation. It scores retrieval confidence and branches three ways: confident (refine the passages, stripping irrelevant strips), wrong (discard everything and fall back to web search), ambiguous (do both). The refinement step matters more than it sounds. Passing a chunk that is 80% noise costs you accuracy even when the right sentence is in there.
FLARE retrieves during generation. It drafts the next sentence, checks token confidence, and if the model is unsure, it uses that tentative sentence as a query and retrieves before committing. Good for long-form output where the information needs shift as you write.
Adaptive-RAG trains a small classifier to route by query complexity: answer directly, retrieve once, or run a multi-step loop. You stop paying multi-hop latency on "what are your business hours."
Use it when: hallucination is expensive and your corpus has coverage gaps. Cost: latency, complexity, and a real dependency on evaluation. Add a correction loop without measuring it and you have just added a second thing that can be wrong.
Research basis: These four approaches are described in the Self RAG (arXiv:2310.11511), Corrective RAG (arXiv:2401.15884), FLARE (arXiv:2305.06983), and Adaptive RAG (arXiv:2403.14403) papers.
7. GraphRAG
Vector search cannot answer "what are the main themes across these 3,000 documents." No single chunk contains the answer, so no retrieval over chunks will find it.
GraphRAG builds a different index. An LLM extracts entities and relationships from every chunk into a knowledge graph. Community detection (Leiden) partitions that graph into clusters at multiple levels, and the LLM writes a summary for each community. Query time splits in two:
- Local search walks the neighborhood around specific entities. Good for multi-hop questions where the connection matters more than any single document.
- Global search map-reduces over community summaries. This is the sensemaking mode that plain RAG structurally cannot do.
The honest cost: indexing is expensive. You are running LLM calls over your entire corpus, and updates mean partial reindexing. For a static research corpus that is fine. For a wiki that changes hourly it is painful, which is why lazier variants that defer summarization exist.
Use it when: entities and their relationships are the point. Investigations, literature synthesis, compliance mapping, fraud networks.
Research basis: Microsoft Research proposed a Graph RAG approach that builds an entity graph, detects communities, and prepares community summaries for local and global search (arXiv:2404.16130, reference implementation).
8. Agentic RAG
Give an LLM agent retrieval as a tool and let it run the loop itself: reason about what it needs, retrieve, judge whether the results answer the question, reformulate, retrieve again, stop when satisfied.
Two common shapes. A router picks the right source per query: vector index, SQL database, web, internal API. A loop iterates until a stopping condition. Multi-agent versions assign a sub-agent per source and have a coordinator merge.
The strength is genuine. Some questions cannot be turned into a good query up front, because the right second query depends on what the first one returned.
The weakness is also genuine. Latency stacks, cost stacks, behavior is non-deterministic, and debugging means reading traces of a model arguing with itself. Errors compound: a bad step three poisons steps four through seven. Cap iterations, log every tool call, and do not use this where a router alone would have done.
Research basis: ReAct established a widely used pattern for combining model reasoning with actions and observations (arXiv:2210.03629). A later survey organizes agentic RAG patterns, components, and applications (arXiv:2501.09136).
9. Multimodal RAG
A large fraction of real enterprise knowledge lives in charts, sheets of figures, scanned forms, and slide decks. Text extraction throws away layout, and layout is often the meaning.
Two approaches:
Transcribe first. Run a vision model over pages, caption the figures, serialize the structured data, then do normal text RAG. Simple, works with your existing stack, lossy in ways you will not notice until a user complains about a number in a chart.
Retrieve visually. ColPali-style systems embed page screenshots directly with a vision language model using late interaction, skipping the parsing pipeline entirely. You retrieve page images and hand them to a VLM at generation time. Fewer moving parts than a PDF parsing chain, and it does not silently mangle a two-column layout.
Use it when: your corpus is PDFs where the visual structure carries information. Financial filings, engineering drawings, medical forms, decks.
Research basis: ColPali retrieves visually rich document pages using vision language model embeddings and late interaction (arXiv:2407.01449). M3DocRAG extends multimodal retrieval and question answering across text, charts, and figures (arXiv:2411.04952).
The Counterpoint: Long Context and Cache-Augmented Generation
Worth taking seriously. If your whole corpus fits in the context window, you can preload it, cache the KV state, and skip retrieval. That is Cache-Augmented Generation, and for a single product manual or a stable policy set it is simpler and often better than anything above.
It stops working when:
- The corpus exceeds the window, which most real ones do.
- Content changes often, invalidating the cache.
- You are paying per token on every request.
- Attention degrades. Models ace needle-in-a-haystack tests and get noticeably worse at reasoning over many scattered facts in the same long context.
There is also a point people forget: retrieval is a permissions boundary. Filtering documents at retrieval time is how you keep the finance team's files out of an intern's answer. A stuffed context window has no access control.
Research basis: Cache Augmented Generation has been proposed as an alternative when a bounded knowledge collection can be placed in long context and reused through cached model state (arXiv:2412.15605).
Picking One
| Symptom | Try |
|---|---|
| Top results are noisy | Reranking |
| Misses exact IDs, codes, names | Hybrid search |
| Right doc exists but never retrieved | Query transformation |
| Answers lack surrounding context | Hierarchical / parent-document |
| Confident answers with no support | Self-RAG or CRAG |
| Multi-hop, entity-heavy questions | GraphRAG (local) |
| "What are the themes across everything" | GraphRAG (global) |
| Query is unknowable in advance | Agentic RAG |
| Charts, structured data, scanned pages | Multimodal RAG |
| Small, stable corpus | Long context / CAG |
What I Would Actually Do First
Build the evaluation set before you touch the architecture. Fifty real questions with known correct source documents will teach you more about your own pipeline than any amount of reading. Measure recall@k on retrieval separately from answer quality. If recall@20 is 0.6, no reranker and no agent loop will save you, because the right document was never in the candidate pool.
Then fix chunking. It is the least interesting part of the system and routinely the largest source of error. Respect document structure, keep structured data intact, and add a little overlap.
Then hybrid search plus a reranker. That combination gets most teams most of the way, and it adds two components instead of ten.
Everything past that is earned. Each layer buys accuracy and charges latency, cost, and a new failure mode. Stacking all nine gives you a system nobody can debug at 2am.
The architecture is not the hard part. Knowing which one your failure mode calls for is.
References and Further Reading
- Lewis et al. Retrieval Augmented Generation for Knowledge Intensive NLP Tasks. arxiv.org/abs/2005.11401
- Karpukhin et al. Dense Passage Retrieval for Open Domain Question Answering. arxiv.org/abs/2004.04906
- Nogueira and Cho. Passage Re Ranking with BERT. arxiv.org/abs/1901.04085
- Khattab and Zaharia. ColBERT. arxiv.org/abs/2004.12832
- Robertson and Zaragoza. The Probabilistic Relevance Framework, BM25 and Beyond. nowpublishers.com/article/Details/INR-019
- Cormack, Clarke, and Buettcher. Reciprocal Rank Fusion. dl.acm.org/doi/10.1145/1571941.1572114
- Gao et al. Precise Zero Shot Dense Retrieval without Relevance Labels. arxiv.org/abs/2212.10496
- Rackauckas. RAG Fusion. arxiv.org/abs/2402.03367
- Sarthi et al. RAPTOR. arxiv.org/abs/2401.18059
- Asai et al. Self RAG. arxiv.org/abs/2310.11511
- Yan et al. Corrective Retrieval Augmented Generation. arxiv.org/abs/2401.15884
- Jiang et al. Active Retrieval Augmented Generation. arxiv.org/abs/2305.06983
- Jeong et al. Adaptive RAG. arxiv.org/abs/2403.14403
- Edge et al. From Local to Global, A Graph RAG Approach to Query Focused Summarization. arxiv.org/abs/2404.16130
- Yao et al. ReAct. arxiv.org/abs/2210.03629
- Faysse et al. ColPali. arxiv.org/abs/2407.01449
- Chan et al. Cache Augmented Generation. arxiv.org/abs/2412.15605
- Es et al. RAGAS, Automated Evaluation of Retrieval Augmented Generation. arxiv.org/abs/2309.15217
Want to share your own experience? Every member can write here: reach out and we'll help you publish your first post.