How Does RAG Work? Retrieval-Augmented Generation, Step by Step

RAG — retrieval-augmented generation — works in two halves. Before any question is asked, a collection of documents is split into small chunks, each chunk is converted into a numerical representation called an embedding, and those embeddings are stored in a searchable index; when a question arrives, it is embedded the same way, the index returns the chunks whose meanings sit closest to the question, and those passages are placed into the model’s prompt so the answer is generated from them. That is the entire trick, and it explains something that otherwise looks like magic: an AI system can quote your website accurately even though the model behind it was never trained on your website.

Why RAG exists

A large language model on its own knows only what it learned during training, and that knowledge is frozen at a cutoff date, blurred by compression, and impossible to check — the model cannot show you where a claim came from. RAG addresses all three limits. It gives the model access to information that is current, private, or specialized; it grounds the answer in specific passages the system actually retrieved; and, in systems that display citations, it lets a reader inspect the sources. Most AI answer products that cite web sources are running some version of this pattern at answer time: retrieve first, generate second.

Understanding RAG matters for anyone who publishes information, because visibility inside a RAG system is decided by the pipeline below — not by the model’s opinion of you, and not by traditional search ranking alone.

Part one: building the index (before any question)

Collecting documents. The process starts with a body of source material: web pages a crawler has fetched, a company’s internal documents, a help center, a product catalog. What never enters this collection can never be retrieved — the first gate in the whole system is simply being in the corpus.

Chunking. Documents are rarely indexed whole. They are split into chunks — passages of a few hundred words, give or take — because retrieval works at passage level: a question about opening hours needs the paragraph about opening hours, not a 4,000-word page. Chunking follows rules (fixed lengths, heading boundaries, overlapping windows so sentences are not severed mid-thought), and those rules quietly shape everything downstream. A fact split across a chunk boundary — the business name in one chunk, its prices in the next — may never be retrieved as a complete statement.

Embeddings. Each chunk is passed through an embedding model, which converts its text into a long list of numbers (a vector) that encodes its meaning: passages about similar things end up with numerically similar vectors, even when they share few words. “How much does it cost” and “what are your prices” land close together; “cost” in the sense of emotional toll lands elsewhere.

The index. The vectors are stored in a vector index (often a vector database) built for one job: given a new vector, find its nearest neighbors extremely fast. The original text of each chunk is stored alongside its vector so it can be handed to the model later.

Part two: answering a question

The question is embedded. Your question goes through the same embedding process, producing a vector in the same space as the chunks.

Similarity search. The index compares the question vector against the chunk vectors and returns the closest matches — typically a top set of passages ranked by similarity score. Note what this ranking rewards: semantic closeness to the question’s wording and meaning, not truth, not authority, and not how recently the page was updated, unless the system’s designers added separate signals for those.

Reranking (common practice). Many production systems add a second pass: a reranking model re-scores the retrieved passages for actual relevance to the question and reorders or trims the list before anything reaches the generator. This is widely used because raw similarity search is fast but crude; reranking is slower and more accurate. Not every system does it, and the details are rarely published — treat any specific claim about a named platform’s reranking as unverified unless the platform itself documents it.

Prompt assembly and generation. The surviving passages are inserted into the prompt alongside the question and instructions, and the language model generates an answer conditioned on that material. In systems with citations, the answer’s references point back to these passages. The model is, in effect, writing an answer with an open book in front of it — which is why the quality of what was retrieved sets a ceiling on the quality of the answer.

A worked example

Imagine a plumbing company’s website is in the corpus. Its services page was chunked, and one chunk reads: “Emergency callouts: evenings and weekends across Suffolk County. Standard callout fee $95, waived with any completed repair.” A homeowner asks an AI assistant, “What does emergency plumbing cost around Suffolk County on a weekend?” The question embeds near that chunk — “emergency,” “cost,” “weekend,” the place name — the chunk ranks in the top results, survives reranking, and lands in the prompt. The answer states the $95 fee and cites the page. The model never “knew” this business. The pipeline carried one well-formed passage from a website to an answer at exactly the moment it matched a question.

Where each step fails — and what it looks like

The document never enters the corpus. The page was not crawled, was blocked, or sits outside the system’s sources. In the answer: the business simply does not exist.
Bad chunking. A page that buries its facts — prices in one paragraph, conditions three pages later — produces chunks that are each incomplete. In the answer: the fee is quoted without the condition, or the passage is skipped because it makes no sense alone.
Embedding mismatch. The chunk is written in vocabulary far from how customers ask (“remuneration schedule” versus “prices”). Similarity search is meaning-based but not magic; obscure or evasive wording lowers the match. In the answer: a competitor’s plainer page is retrieved instead.
Retrieval misses or crowds out. Only the top handful of passages proceed. If ten similar businesses match, most are cut at this stage regardless of quality. In the answer: a short list that feels like “the” answer.
Contradictory passages. The index contains both your current page and an outdated directory copy with your old hours. Both are retrieved. In the answer: the model may pick either, blend them, or state the wrong one confidently — the conflict is inherited from the sources.
Generation drift. The model paraphrases loosely, merges two passages, or adds a plausible detail no passage contained. In the answer: a citation is present, but the sentence it supposedly supports is not quite what the source says.

Every one of these failures is a property of the pipeline, not of anyone’s intent — and most of them are influenced by how the source material was written and structured.

Misconceptions

“RAG means the model learned my website.” No. Retrieval leaves the model itself unchanged. Your content sits in an index, not in the model’s training — which is also why it can be updated or removed in ways trained knowledge cannot.

“RAG answers are accurate because they’re sourced.” Retrieval improves grounding; it does not verify truth. A wrong passage, retrieved faithfully, produces a wrong answer with a real citation attached.

“Being indexed means being cited.” Between index and answer stand similarity ranking, a top-k cutoff, often reranking, and a generator that may ignore a passage it was given. Presence in the corpus is the entry ticket, not the outcome.

“Citations prove the model used that page the way I’d expect.” A citation shows a passage was in the retrieved set (in systems that cite honestly); it does not show that every claim in the answer came from it.

What this means if you publish information

Write pages that chunk well: one complete idea per passage, facts stated fully in a single place rather than assembled across a page, headings that mark what each section is about, and plain vocabulary that matches the words your customers actually use in questions. The companies that appear in AI answers are disproportionately the ones whose information survives this pipeline intact — complete, self-contained, and retrievable.

How Does RAG Work?