What is RAG? Retrieval-augmented generation explained

Retrieval-augmented generation, or RAG, is a way of building an AI system so that it finds relevant material first and then writes its answer using that material. Instead of relying only on what a language model absorbed during training, the system searches a collection of documents, such as a policy library, a product manual or a set of research papers, and places the best matches in front of the model alongside the question.[1]

That design sits behind many products that promise to “answer questions about your documents” or “cite their sources”. It also explains a familiar disappointment. Retrieval can supply the right passage and the system can still answer wrongly. Or retrieval can miss the passage entirely, and the model can fill the gap with something plausible. Knowing which part failed is the difference between fixing a RAG system and simply distrusting it.

This explainer builds on our guide to what a large language model is. It covers where the idea came from, how a typical pipeline works, how RAG compares with fine-tuning and long context windows, how it fails, how to evaluate it and when to choose something else.

Where RAG came from

The term comes from a 2020 paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, by Patrick Lewis and 11 co-authors, accepted at NeurIPS 2020. The paper started from a specific problem. Pre-trained language models store a good deal of factual knowledge in their parameters, but the authors noted that “their ability to access and precisely manipulate knowledge is still limited”. Showing where an answer came from, and updating what a model knows, were described as open research problems.[1]

Their proposal was to give the model a second kind of memory.

“models which combine pre-trained parametric and non-parametric memory for language generation”

— Lewis and colleagues, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020).[1]

Parametric memory is knowledge held in a model’s learned weights. Non-parametric memory is an external store the system can search, which is not itself a set of trained parameters. In the paper, that store was a December 2018 Wikipedia dump split into disjoint 100-word chunks, about 21 million passages in all. Each passage was converted into a numerical vector and indexed for fast similarity search.[1]

The retriever was initialised from Dense Passage Retrieval, published the same year. That work reported that a learned dense retriever outperformed a strong keyword-based BM25 system on top-20 passage retrieval across the open-domain question-answering datasets it tested.[2]

One experiment in the RAG paper captures the practical appeal. The researchers built an index from a 2016 Wikipedia dump and compared it with their 2018 index, asking “Who is {position}?” about 82 world leaders who had changed between those dates. With a matching index, the model answered 70% (2016) and 68% (2018) correctly; with mismatched indexes, accuracy fell to 12% and 4%. The authors concluded that the model’s world knowledge could be updated by replacing its non-parametric memory, rather than by retraining.[1]

Two caveats keep that history in proportion. First, the 2020 system fine-tuned its query encoder and generator together for each task. Many RAG products today instead connect a search step to a general-purpose model without joint training, so the name covers a looser family of designs than the original. A 2023 survey describes that family’s progression from “naive” to “advanced” and “modular” RAG.[3] Second, those accuracy figures describe one template-based test in 2020. They are evidence for the mechanism, not a performance claim about any current product.

How a RAG pipeline works

Most RAG systems have two phases. Indexing happens ahead of time: documents are prepared and stored so they can be searched. Answering happens each time someone asks a question: the system retrieves, selects and generates.

AI Chronicle

Chunking decides what can be found

Documents are usually split into smaller pieces before indexing. Microsoft’s Azure AI Search documentation explains that chunking is required when documents exceed a model’s input limit, “but it’s also beneficial if content is poorly represented as a single vector”. A long wiki page covering many subtopics is its example.[4]

Common approaches split text at a fixed size with some overlap, at sentence or heading boundaries, or by meaning. As a starting point, that Azure page (last updated July 2026) suggests 512-token chunks with 25% overlap and says the right overlap depends on the content. Anthropic’s 2024 description of a typical pipeline puts chunks at “usually no more than a few hundred tokens”.[4][5] Treat these as defaults to test, not rules.

Chunking has a cost: a chunk can lose the context that made it meaningful. Anthropic’s example is a passage reading “The company’s revenue grew by 3% over the previous quarter”, which names neither the company nor the quarter.[5] Common remedies attach the document title, section heading, date or version to each chunk. In one of three case studies reported by Barnett and colleagues, adding the file name and chunk number to the retrieved context helped the model extract the required information.[6]

Embeddings turn text into searchable meaning

An embedding model converts each chunk into a vector, a long list of numbers, arranged so that passages with related meaning end up close together. The question is converted the same way, and the system looks for the nearest chunks. This is often called semantic search. A question about “time off after having a baby” can match a section headed “Parental leave” even though the two share almost no words. That example is illustrative rather than a test result.

The vectors are typically stored in a vector database or a search index that supports similarity search. The same store is where access rules belong. OWASP’s guidance on RAG security recommends “permission-aware vector and embedding stores”, so that a search cannot surface material the person asking is not allowed to see.[7]

Retrieval: keyword, semantic or both

Semantic search is not always the better tool. Anthropic notes that embedding models “can miss crucial exact matches”, using a support query for an error code as its example. Keyword ranking functions such as BM25 look for the precise string. Many systems therefore run both searches and merge the results, an approach usually called hybrid retrieval.[5]

The retriever returns a short list of candidates, often described as the top k. Everything downstream depends on that list. If the passage containing the answer is not in it, later steps cannot recover it.

Reranking takes a second, slower look

First-stage retrieval is built for speed across a large collection. A reranker examines a smaller set of candidates more carefully, scoring each one against the question. Nogueira and Cho’s 2019 work, which re-implemented BERT for passage re-ranking, topped the MS MARCO passage-retrieval leaderboard at the time.[8]

Anthropic’s published setup retrieved 150 candidates, reranked them and passed the top 20 to the model. In its own tests across several datasets, combining reranking with its contextual retrieval method reduced the top-20 retrieval failure rate from 5.7% to 1.9%. That is a vendor-reported result on the company’s chosen data, and the post notes that reranking adds latency and cost.[5]

Generation with citations

Finally, the selected chunks are inserted into the model’s prompt with instructions: answer from these sources, cite them and say so when they do not contain the answer. Citations are the feature readers notice most, because they make checking possible.

A citation is a claim, however, not proof. The ALCE benchmark, built to evaluate how language models cite sources, reported in 2023 that on one long-form question dataset “even the best models lack complete citation support 50% of the time”.[9] That finding concerns the systems tested then. The habit it teaches still applies: open the cited passage and confirm it says what the answer claims. Our guide to verifying AI-generated research sets out that check step by step.

RAG vs. fine-tuning vs. long context

RAG is one of three common ways to get specific knowledge into an AI system’s answers. They change different things.

Fine-tuning continues training a model on task-specific data, which changes its parameters.[10] It is well suited to teaching a format, style or task behaviour. It is a weaker way to add facts. In one comparison of knowledge-injection methods, RAG consistently outperformed unsupervised fine-tuning on the knowledge-intensive tasks tested, and the authors found that “LLMs struggle to learn new factual information through unsupervised fine-tuning”.[11]

Long context means placing the whole document, or the whole collection, in the prompt. Anthropic’s September 2024 guidance suggested that a knowledge base smaller than 200,000 tokens, about 500 pages, could simply be included in full, with no RAG step.[5] That threshold was tied to its models at the time and should be rechecked for any current system. A 2024 study comparing the two approaches found that long-context models, when sufficiently resourced, outperformed RAG on average across its benchmarks. RAG’s much lower cost remained an advantage, and the authors proposed routing each query to whichever approach suited it.[12]

A larger context window is not the same as reliable use of that context. The Lost in the Middle study found that the models it tested often performed best when relevant information sat at the beginning or end of the input, and worse when it sat in the middle.[13] Our context windows explainer separates capacity from effective use.

Editorial comparison based on the sources cited in this section; not a measured ranking.

These options combine. A team might fine-tune a model to follow its house answer format, use RAG to supply current policy text, and use a long context window to read a whole retrieved document rather than isolated chunks.

AI Chronicle

A worked example (hypothetical): internal policy Q&A

This example is fictional. The company, documents, question, policy wording and acceptable answer were written by AI Chronicle to illustrate the pipeline. No product was tested and no output was captured.

A 200-person company wants staff to ask questions about its internal policies. Its knowledge base holds three documents: Travel and Expenses Policy v4 (effective 1 March 2026), the superseded Travel and Expenses Policy v3 (still sitting in the shared drive), and Remote Work Guidelines.

An employee asks: “Can I expense a hotel upgrade if I’m travelling to meet a client?”

1. Indexing. The team splits each document at its section headings rather than at a fixed length, so each policy rule stays in one chunk. Every chunk carries metadata: document title, version, effective date, status (current or superseded), section number and audience (all staff, or finance only).

2. Retrieval. Hybrid search returns eight candidates. The strongest matches are v4 §5.2 “Accommodation”, v3 §4.1 “Accommodation” and v4 §5.6 “Client entertainment”. The keyword search catches “hotel”; the semantic search catches “client trip” phrasing.

3. Filtering and reranking. A metadata filter removes superseded documents unless the question asks about history. The reranker then places v4 §5.2 first. Its fictional text reads:

5.2 Accommodation. Book a standard room within the nightly cap for
the destination city. Room upgrades are not reimbursable unless a
budget holder approves them in writing before travel.

4. Generation. The prompt tells the model to answer only from the supplied sections, cite the section and say plainly when the sections do not cover the question. An acceptable answer would say: not by default; under the Travel and Expenses Policy v4, §5.2, an upgrade is reimbursable only with a budget holder’s written approval before travel. That is an editorial acceptance criterion, not a captured model response.

5. Where it could go wrong. Suppose the old v3 text said upgrades were allowed on long trips, and the filter was never built. The superseded chunk could rank first, and the answer would be wrong, confident and cited. Now suppose the employee asks whether a hotel gym day pass is claimable, and neither policy mentions it. The right behaviour is to say the policy does not cover it and point to the finance team, not to infer a rule from nearby text. And if a finance-only chunk listing approval thresholds is in the same index, permission filtering has to stop it reaching general staff.

6. A small test set. Before launch, the team writes questions whose correct source it already knows:

Each row tests a different stage: retrieval, filtering, abstention, permissions and document selection. When a row fails, the team can see which stage to fix.

How RAG systems fail

Barnett and colleagues' experience report on RAG systems in research, education and biomedical settings named seven failure points.[6] Grouped by where they happen, they make a useful diagnostic list:

• Before the model sees anything: the answer is not in the documents at all (missing content); it is there but did not rank highly enough (missed the top-ranked documents); or it was retrieved but dropped when results were consolidated (not in context).
• After the model sees it: the answer is in the context but the model fails to extract it, often because of noise or contradictions (not extracted); it ignores a requested format (wrong format); it answers too generally or too narrowly (incorrect specificity); or it leaves out information that was available (incomplete).

The report’s first key takeaway is worth keeping in view:

“validation of a RAG system is only feasible during operation”

— Barnett and colleagues, Seven Failure Points When Engineering a Retrieval Augmented Generation System (2024).[6]

Retrieval also introduces security risks that a model without retrieval does not have. OWASP describes indirect prompt injection, where instructions hidden in external content, such as a document in a RAG repository, alter the model’s behaviour. Its guidance states that RAG and fine-tuning “do not fully mitigate prompt injection vulnerabilities”.[14] OWASP also lists poisoned knowledge-base content and leaks between users who share one vector database.[7] A RAG system is only as trustworthy as the documents it is allowed to retrieve.

Real deployments show the limits of marketing claims. A preregistered 2024 study of AI legal research tools found that RAG-based products from LexisNexis and Thomson Reuters hallucinated between 17% and 33% of the time on the study’s questions. That was less than a general-purpose chatbot, GPT-4, in the same evaluation, but far from the “hallucination-free” claims the authors examined.[15] Our AI hallucinations explainer covers why unsupported answers occur.

How to evaluate a RAG system

The most useful habit is to score retrieval and generation separately. The Ragas framework sets out the dimensions: whether retrieval identifies “relevant and focused context passages”, whether the model uses them faithfully, and the quality of the generated answer.[16] OWASP’s prompt-injection guidance refers to a similar “RAG Triad”: context relevance, groundedness and question/answer relevance.[14]

In practice, that becomes four questions for every test case:

1. Retrieval quality. Did the passage containing the answer appear in the top k results? This is recall at k; Anthropic’s tests measured how often relevant material was missing from the top 20.[5]
2. Groundedness. Is every claim in the answer supported by a cited passage? Check claims one at a time, not the answer as a whole.
3. Answer relevance. Does the answer address the question that was asked, at the right level of detail?
4. Abstention. When the documents do not contain the answer, does the system say so?

Build the test set from real questions, with the location of each correct answer recorded. Include unanswerable questions, superseded documents and permission traps, as in the worked example above. Re-run it whenever the documents, chunking method, embedding model, reranker or language model changes, since any of them can shift results.

Automated scoring helps teams iterate quickly. Ragas, for instance, was designed to evaluate without human-written reference answers.[16] An automated judge is itself a model, though, so spot-check its verdicts against human review. Our guide to what AI benchmarks actually measure explains why one aggregate score rarely tells the whole story.

When not to use RAG

RAG adds moving parts: an index to maintain, retrieval to tune and a new security surface. It is the wrong choice in several common situations.

• The material is small and stable. If the relevant documents fit comfortably in a context window and rarely change, placing them in the prompt is simpler and avoids retrieval misses.
• The problem is behaviour, not knowledge. If the model knows enough but answers in the wrong format or tone, better instructions or fine-tuning address that directly.
• The answer lives in a live system. An order status, an account balance or today’s stock level should come from querying the system of record through a tool, not from retrieving text that describes it. Our AI agents explainer covers how models use tools.
• The question needs the whole collection. “What themes run through all of last year’s support tickets?” cannot be answered from the top 20 chunks. Barnett and colleagues saw incomplete answers to questions spanning several documents and suggested asking such questions separately.[6]
• The documents are unreliable. Retrieval amplifies whatever it finds. A contradictory, outdated or unowned document set produces confident answers grounded in the wrong text.
• Nobody can review high-stakes answers. The legal-research study shows that RAG lowers some risks without removing them.[15]

Questions readers commonly ask

Is RAG the same as an AI that searches the web?

Web search is one possible retrieval source. RAG is the broader pattern of retrieving material and generating from it. The collection might be the public web, a company’s private documents or a single product manual. The source changes what the answers can be trusted for.

Does RAG stop hallucinations?

No. It can reduce unsupported answers when retrieval works and the model stays close to the sources, but published evaluations still find errors in RAG-based tools.[15] Check important claims against the cited passages.

Do I need a vector database to build RAG?

Not necessarily. Keyword search such as BM25 can act as the retriever, and many systems combine keyword and embedding search.[5] Choose the retrieval method that finds the right passages in your own test set.

Does RAG train the model on my documents?

No. In a typical RAG system, documents are retrieved and placed in the prompt at question time; the model’s weights do not change. How a provider stores or uses that data is a separate question, governed by its own terms and settings.

Will larger context windows make RAG unnecessary?

Not for every use. Long context can beat RAG on quality when cost is no object, but RAG remains cheaper per query, and it supports updating, citations and permission filtering at the passage level.[12] Uneven use of very long inputs is another reason to test rather than assume.[13]

The next step

You now have the core questions for any RAG system: What can it retrieve? Did it find the right passage? Does the answer follow from that passage?

Put them to work in our tutorial on how to build a research agent with source verification. It turns retrieval, citation and checking into a working process you can test.

Sources and evidence note

This explainer draws on original research papers, vendor technical documentation and OWASP security guidance. All pages were accessed on 15 October 2026. Older papers explain mechanisms or report findings under their stated conditions; they are not rankings of current products. Vendor-reported results are labelled as such. The policy documents, question, acceptance criteria and test set in the worked example are original illustrative material. No hands-on product test is claimed.

Related reading

• Glean and the workplace AI search problem
• Claude vs. ChatGPT for long-document work