Retrieval-Augmented Generation (RAG)
Definition: Retrieval-augmented generation (RAG) is a method where a language model first retrieves relevant documents from an index or the web, then writes its answer using those documents, usually citing them.
Also called: RAG, retrieval augmented generation
Updated
Where it comes from
The term comes from a 2020 paper by Lewis and colleagues at Facebook AI Research, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. The idea is to pair a model’s trained knowledge with a searchable store of documents, so answers can draw on fresher, checkable sources instead of memory alone.
How AI search uses it
ChatGPT search, Perplexity, Claude with web search and Google’s AI features all follow the same broad pattern:
- Turn the question into one or more searches.
- Retrieve candidate pages from an index.
- Pick the most relevant passages.
- Generate an answer from those passages and cite the sources.
The practical consequence for a site owner is blunt. If a page cannot be retrieved, because its crawler is blocked, the content only appears after JavaScript runs or the page is not indexed, the model cannot use it, however good it is.
What helps retrieval
- Crawlable, server-rendered HTML. Research shows most AI crawlers do not run JavaScript (Vercel).
- Sections that name their subject instead of relying on “it” from three paragraphs earlier.
- Headings that describe what the section answers.
- Dates and sources next to facts that change.
Our view: “chunk-friendly” writing is simply clear writing. Google says there is no need to break content into tiny pieces for AI. The AI crawlers guide lists which bot feeds which assistant’s retrieval index.