How AI Systems Retrieve Information

By /

By Reputation.blog Editorial
Published: October 31, 2026 · Last reviewed: October 31, 2026

The short answer

When an AI system answers a question using the web, it does not read the internet live, page by page, the way a person browses. It follows a pipeline that was mostly built before the question was ever asked:

  1. Crawlers discover and download pages in advance.
  2. Those pages are processed and stored in an index — increasingly, in more than one kind of index.
  3. When a question arrives, the system usually rewrites or expands it into one or more searches (a technique Google calls query fan-out).
  4. A retriever pulls a pool of candidate documents or passages from the index.
  5. That pool is narrowed down to a small working set.
  6. The working set is placed into the language model’s context — grounding — and the model writes an answer conditioned on that material, often with citations.

Every stage is a filter. A page can fail at any one of them, and a page that fails early is invisible later: a page that was never crawled cannot be retrieved, and a page that is retrieved but not selected never reaches the model. This guide walks through each stage, explains the underlying technology in plain language, and shows where the major platforms — Google, ChatGPT, Perplexity, Microsoft Copilot, and Claude — genuinely differ. Where a mechanism is documented by the platform itself, we say so. Where it is not published, we say that too, rather than filling the gap with speculation.

This guide is the technical companion to How AI Search Works in our Knowledge Center. What happens after retrieval — how sources are chosen for citation and attributed — is covered in Understanding AI Citations.

First, a distinction: memory versus retrieval

A large language model carries two very different kinds of “knowledge”:

  • Parametric memory — what the model absorbed during training, compressed into its weights. It is broad but frozen at a training cut-off, has no sources attached, and cannot be updated without retraining.
  • Retrieved information — documents fetched from an external store (usually a web index) at the moment a question is asked, and handed to the model as context.

The architecture that combines the two was formalised in 2020 by Lewis and colleagues at Facebook AI Research in the paper that popularised the term retrieval-augmented generation (RAG). In their design, a retriever searches a dense vector index of documents, and the generator conditions its output on the retrieved passages alongside its parametric memory. The retrieval step is what provides provenance — the answer can point at documents — and freshness: the external index can change daily, while the model’s weights cannot.

Everything below is about the retrieval half of that combination: how the external store is built, how the right passages are found inside it, and how they reach the model.

Stage 1: Crawling and discovery — deciding what exists

Retrieval starts long before any question, with crawling: automated programs (bots, spiders) that move across the web following links, downloading pages, and reporting back what they find.

Google’s documentation describes the pattern every large search index follows. There is no central registry of web pages, so the crawler must discover URLs — from pages already known, from links extracted from those pages, and from sitemaps submitted by site owners. Once a URL is discovered, the crawler may fetch it: Google’s crawler, Googlebot, renders pages with a Chrome-based renderer so that content loaded by JavaScript is visible, and it paces itself to avoid overloading sites. Crucially, Google also documents what crawling is not: not every discovered page is crawled (a site’s robots.txt file may disallow it, or the page may sit behind a login), and Google does not guarantee that a page will be crawled, indexed, or served at all.

For AI retrieval specifically, the single most consequential change of the last few years is that the major AI platforms now run separate crawlers for separate purposes, and they document them openly:

  • OpenAI documents three agents. OAI-SearchBot crawls for ChatGPT’s search features — sites that opt out of it will not be shown in ChatGPT search answers. GPTBot crawls content that may be used for training foundation models, a separate decision. ChatGPT-User is not a crawler at all: it fetches a page live when a user’s request triggers it. OpenAI states explicitly that allowing GPTBot while blocking OAI-SearchBot (or the reverse) is a coherent, supported choice — training access and search visibility are independent switches.
  • Perplexity documents PerplexityBot, which builds the index used to surface and link websites in its search results and is, per Perplexity, not used to crawl content for training foundation models; and Perplexity-User, which fetches pages live in response to a user’s question and, because the fetch is user-initiated, generally ignores robots.txt.
  • Anthropic documents ClaudeBot (training data collection), Claude-SearchBot (which navigates the web to improve search results and index content for search), and Claude-User (live retrieval at a user’s direction).
  • Microsoft Copilot has no web crawler of its own in this picture: Microsoft’s documentation describes Copilot sending generated search queries to the Bing Search service, so the crawling and indexing layer underneath Copilot is Bing’s.
  • Google uses the same Googlebot pipeline for AI features as for classic search: its AI-features documentation states that, to be eligible to appear as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet — and that there are no additional technical requirements.

That last point deserves emphasis because it contradicts a great deal of current marketing. For the largest AI answer surface on the web, eligibility is the classic search eligibility: crawlable, indexed, snippet-eligible. The AI-specific crawlers of OpenAI, Perplexity, and Anthropic add new gates for their own platforms, but each platform’s gate is its own. Blocking GPTBot does nothing to your Google AI Overview eligibility; blocking OAI-SearchBot does nothing to your Perplexity visibility. Retrieval begins with a per-platform question: has this platform’s crawler been allowed to see the page?

Stage 2: Indexing — turning pages into something searchable

A downloaded page is not yet retrievable in any useful sense. It has to be analysed and stored in structures designed for fast lookup. Indexing is that processing step, and modern retrieval systems typically build more than one kind of index over the same content.

The keyword (lexical) index

The classical structure is the inverted index: for every term, a list of the documents (or passages) containing it, with positions and statistics. It is what makes it possible to find every page containing “quinceañera band” without scanning the web. Google’s documentation describes its indexing stage in these terms: analysing textual content and key tags and attributes, detecting duplicates and selecting a canonical page to represent a cluster of near-identical pages, and collecting signals about the page — its language, the country it is local to, its usability — that are stored in the index alongside the content.

Two consequences matter for everything downstream:

  • Deduplication happens at indexing, not at answer time. If three URLs carry substantially the same content, the index may pick one canonical representative. The other versions do not get a separate chance to be retrieved.
  • Indexing is not guaranteed. Google lists the common failure modes plainly: low-quality content, robots meta rules that disallow indexing, and site designs that make indexing difficult. A crawled page can simply fail to enter the index — and what is not in the index does not exist as far as retrieval is concerned.

Chunking: why retrieval works on passages, not pages

Retrieval systems rarely treat a whole web page as one unit. A 4,000-word page may cover six topics; matching the whole page against a narrow question blurs all six. Standard practice in retrieval engineering — visible throughout the RAG research literature, where the original system indexed Wikipedia as discrete passages — is to split documents into smaller segments (passages or “chunks”) and index those. The retriever can then return the one passage that answers the question rather than the whole document.

The chunking strategy of each commercial platform — how large the segments are, how they overlap, what metadata travels with them — is not published by Google, OpenAI, Perplexity, Microsoft, or Anthropic. That it happens in some form is a safe statement about retrieval engineering generally; any specific claim about a given platform’s chunk sizes is speculation.

The vector (semantic) index

Alongside the keyword index, modern systems build a second representation using embeddings.

An embedding is a list of numbers — a vector, typically hundreds or thousands of dimensions long — produced by a neural network trained so that texts with similar meanings end up with similar vectors. Google’s machine-learning documentation defines embeddings as dense vector representations that capture semantic relationships, in contrast to sparse one-hot encodings where every item is equally unrelated to every other. In an embedding space, a passage about “how much a live band costs for a wedding” sits close to a passage about “wedding band prices” even if the two share almost no words, and far from a passage about “wedding bands” in the jewellery sense, which sits near rings instead.

Indexing with embeddings means computing a vector for every passage in advance and storing the vectors in a structure that can answer one question very quickly: given this query vector, which stored vectors are nearest to it? Comparing a query against billions of stored vectors one by one is too slow, so vector indexes use approximate nearest-neighbour (ANN) search — algorithms, such as the HNSW graph method published by Malkov and Yashunin, that find nearly all of the true nearest neighbours in a small fraction of the time by navigating a graph of precomputed connections between vectors, in exchange for occasionally missing one. This trade-off — speed against perfect recall — is a designed property of large-scale semantic retrieval, not a bug: at web scale, exhaustive comparison is not an option.

Whether a given consumer AI platform retrieves primarily by keywords, primarily by vectors, or by a hybrid of both is, again, not published in detail. Hybrid retrieval is the standard architecture in the open retrieval literature and in enterprise RAG systems, and it is reasonable to describe it as the industry norm. It is not reasonable to state which weights Google or OpenAI assign to lexical versus semantic matching, because they have not said.

Stage 3: The question arrives — deciding to retrieve, and what to search for

Not every prompt triggers retrieval. Answer systems first make a judgement — sometimes a model decision, sometimes a rules-based one — about whether external information would improve the answer. Questions about recent events, prices, schedules, named businesses, and anything past the training cut-off clearly benefit; a request to reword a paragraph usually does not. ChatGPT’s help documentation describes exactly this behaviour from the user’s side: ChatGPT may search the web automatically when a question would benefit from current information, and users can also invoke search manually. Microsoft’s documentation describes the same gate for Copilot: Copilot parses the prompt and identifies the terms where web search would improve the response.

Once retrieval is triggered, the system rarely searches for the user’s raw words. It rewrites the question into search queries, and often into several:

  • Query rewriting. OpenAI’s documentation gives the clearest public description: ChatGPT search “typically rewrites your query into one or more targeted queries” that it sends to search providers, and may follow up with additional, more specific queries after reviewing initial results. Microsoft documents the equivalent for Copilot: the generated query “consists of a few words informed by the user’s prompt” rather than the whole prompt. In other words, the text that actually meets the index is a short, search-shaped query derived from the conversation — not the conversation itself.
  • Query fan-out. Google’s Search Central documentation names its version explicitly: AI Overviews and AI Mode “may use a ‘query fan-out’ technique — issuing multiple related searches across subtopics and data sources — to develop a response,” and while the response is being generated, its models identify further supporting pages. A single question about, say, planning a corporate event can fan out into searches about venues, catering, entertainment, and capacity, each pulling its own candidate set.

The practical significance is hard to overstate: your page is not retrieved by the user’s question; it is retrieved by the system’s derived queries, which you never see. A page competes not against a prompt but against a handful of short searches issued around it.

Stage 4: Candidate retrieval — building the pool

Each derived query is now matched against the index, and a pool of candidate documents or passages comes back. Two matching mechanisms do the work, corresponding to the two indexes:

  • Lexical matching finds passages containing the query’s terms (and close variants), weighted by how distinctive those terms are — a term appearing in few documents counts for more than a common one. This is precise for names, model numbers, prices, and exact phrases, and brittle when the searcher and the page use different words for the same thing.
  • Semantic matching converts the query into an embedding and finds the passages whose vectors are nearest. This forgives vocabulary differences — “how much does a live band cost” can match a page titled “Pricing” that never uses the word “cost” — but it can also blur fine distinctions that exact terms would have preserved.

Candidate pools are deliberately larger than the final answer needs. Retrieval at this stage optimises for recall: it is better to pull in marginally relevant passages and filter later than to miss the one passage that answers the question. How large the pool is on any given platform — dozens of passages, hundreds — is not published.

Stage 5: Narrowing the pool — reranking and selection

Between the candidate pool and the model sits a narrowing step. Passages are re-scored more carefully than the fast first pass allows — cross-checking the passage text against the actual question rather than against index-level signals — and most of the pool is discarded. Only a small working set survives to be placed into the model’s context.

This stage is standard, well-described practice in retrieval engineering and in the RAG literature. But it is important to be precise about the boundary of public knowledge: no major consumer platform publishes its reranking criteria, the number of passages it passes to the model, or the weights it assigns to relevance, freshness, authority, or any other factor. Google documents that its serving systems weigh “hundreds of factors” for classic search results. Anyone who presents a specific ranking formula for AI retrieval — percentages, thresholds, factor lists — is presenting inference or invention, not documentation.

What can be said from observation is that the surviving set is small: AI answers typically cite a handful of sources, and the context window, while much larger than it once was, is still a finite budget that the question, the conversation, and the retrieved material all compete for. A passage that is relevant but near-duplicative of a stronger passage has little reason to be kept. This is one mechanism behind a pattern anyone can observe: being retrieved and being used are different events, and only the second one is visible in an answer.

Stage 6: Grounding and context assembly — the answer is written

The surviving passages are now assembled into the model’s context, typically with source information attached, and the model generates its answer conditioned on that material. This is grounding: the model’s claims are anchored to supplied text rather than drawn only from parametric memory.

Google’s developer documentation for Gemini grounding describes the mechanics with unusual openness, because developers build on it: the model analyses the prompt, decides whether search would improve the answer, generates one or more search queries and executes them, processes the results, and returns a response whose text carries inline annotations mapping segments of the answer to the source URLs that support them. The consumer surfaces differ in presentation, but the sequence — analyse, search, synthesise, attribute — is the documented shape of grounded generation.

Two properties of grounding matter for anyone trying to understand a particular answer:

  • The model can only use what it was given. If the retrieved set contains an error, the answer can inherit it; if the retrieved set omits the correct source, no amount of model capability conjures it. Many “AI got it wrong” stories are, on inspection, retrieval stories: the wrong passages won the pool.
  • Citations are attached to the material that was supplied. The mechanics of how platforms decide which claims get citations — and which sources get shown — are the subject of Understanding AI Citations. Retrieval is what makes a page eligible for that decision.

The second path: live fetching at question time

Alongside the index pipeline, every major platform also documents a fetch-at-question-time agent: ChatGPT-User, Perplexity-User, and Claude-User. When a user asks about a specific page, pastes a URL, or asks a question that prompts the system to open a particular result, the platform may fetch that page in the moment, outside the pre-built index.

This has a consequence site owners often miss: because these fetches are user-initiated, Perplexity and OpenAI both note that their user agents may not observe robots.txt restrictions the way their crawlers do. A robots.txt rule is a strong control over indexing; it is a weaker control over live fetching. The two paths — indexed retrieval and live fetch — should be considered separately when deciding what to allow.

Platform differences at a glance

Google AI Overviews / AI ModeChatGPT searchPerplexityMicrosoft CopilotClaude
Retrieval sourceGoogle Search indexOwn search index (OAI-SearchBot) plus partner search providersOwn index (PerplexityBot) plus live fetchingBing Search serviceClaude-SearchBot index plus live fetching
Eligibility gate (documented)Indexed and snippet-eligible in Google Search; no additional technical requirementsAllow OAI-SearchBot; host/CDN must allow its published IPsAllow PerplexityBotBe indexed by Bing (Copilot sends queries to Bing)Allow Claude-SearchBot for search indexing
Query handling (documented)Query fan-out: multiple related searches across subtopicsRewrites the prompt into one or more targeted queries sent to providers; may issue follow-upsNot published in equivalent detailGenerates a short query of a few words from the promptNot published in equivalent detail
Selection/ranking criteriaNot published (hundreds of factors documented for classic search serving)“Multiple factors”; placement not guaranteed (per OpenAI help docs)Not publishedNot publishedNot published

The table’s empty cells are the honest part. Retrieval infrastructure is documented remarkably well; retrieval ranking is documented almost nowhere.

Why some pages are retrieved and others are ignored

Working backwards through the pipeline, a page misses an AI answer for one of a small number of mechanical reasons:

  1. It was never discovered. No links point to it, it is absent from sitemaps, and the crawler never learned the URL exists.
  2. It was blocked. A robots.txt rule, a firewall or CDN rule that rejects the crawler’s IP ranges, a login wall, or persistent server errors kept the platform’s crawler out. With per-platform crawlers, this failure is per-platform: a site can be fully visible to Google and completely absent from ChatGPT search because OAI-SearchBot is disallowed.
  3. It was crawled but not indexed. Duplicate clusters were canonicalised to another version, quality or design issues blocked indexing, or a noindex directive — sometimes inherited from a staging environment — told the index to stay away.
  4. It is indexed but does not match the derived queries. The page’s vocabulary and meaning do not line up with the short searches the system generated from the user’s question. This is where most well-built, genuinely relevant pages silently lose: they answer the topic, but not in the terms or the granularity the retrieval queries use.
  5. It matched but did not survive narrowing. Retrieved into the pool, then dropped in favour of passages that matched more precisely, covered the point more directly, or duplicated a stronger source.
  6. It survived but was not used or cited. The model grounded its answer in other passages from the working set. Retrieval is necessary for citation; it is nowhere near sufficient.

Note what is not on this list: there is no documented stage at which a page can pay, register, or submit its way into retrieval, and no platform documents a special file, tag, or schema type that grants retrieval. Google’s documentation states the opposite for its AI features — the eligibility requirements are the ordinary ones for search.

A worked example

To make the pipeline concrete, follow one question through it. This is an illustrative scenario, not a measurement: “What should we check before booking a live band for a company holiday party on Long Island?”

  • The question is conversational and multi-part — exactly the shape that triggers fan-out. Plausible derived queries include “book live band corporate event questions to ask”, “holiday party band Long Island”, “live band booking contract deposit”, and “band for company party pricing”. (These are invented examples of what fan-out produces; the platforms do not show their derived queries, with one exception — Microsoft documents that Copilot Chat users can see the queries it ran.)
  • Each derived query pulls its own candidate pool. A band’s dedicated page about corporate and holiday events matches two of the four queries semantically and lexically. Its homepage — generic language about “great music for any occasion” — matches none of them well. A directory listing with thin content but the exact words “holiday party band Long Island” matches one query lexically. A venue’s blog post about event planning matches the first query without mentioning any band.
  • The pools merge and narrow. The band’s events page survives on the strength of specific, on-point passages: what the booking includes, how far in advance to book, what the band needs on the day. The directory listing is retrieved but its passage adds nothing beyond a name, and it is dropped or kept only as a mention.
  • The model writes an answer about checking availability, contracts, insurance, and set-up requirements, grounding those claims in the passages it was given — the band’s page among them, cited because its passage supported a specific claim.

Same band, same website: the page built around the actual question gets retrieved and used; the generic page does not. Nothing in that outcome required anyone to “optimise for AI” as a separate discipline — it required the page to exist, be crawlable and indexed, and contain retrievable passages that match the way the question decomposes. That is the whole mechanism, and it is why retrieval rewards specificity.

What this means for businesses

Stated carefully, without the guarantees this publication does not make:

  • Retrieval is a chain of gates, and the early gates are binary. Crawl access for each platform’s search bot, indexation, and canonical hygiene decide whether your pages are in the pool at all. These are checkable, mechanical, and entirely within a site owner’s control — server logs show which bots visit, and each platform publishes its user agents and IP ranges precisely so owners can verify access.
  • You are matched against derived queries, not against your customers’ words. Pages that answer specific, real questions — with the answer stated plainly in the text, in passages that stand on their own — are structurally easier to retrieve than pages that gesture at a topic. This is not a trick; it is what the matching machinery does with any text.
  • Retrieval differs by platform because the indexes differ. A business visible in Google AI answers may be absent from ChatGPT or Perplexity simply because a different crawler met a different robots.txt rule or a different CDN configuration. Checking visibility platform by platform, starting with crawl access, is more informative than any aggregate “AI visibility” score.
  • Nothing here can be promised. Indexing is not guaranteed by any platform; retrieval is not promised to the indexed; citation is not promised to the retrieved. What can honestly be done is to remove the mechanical failures and publish information that is clear, specific, and consistent enough to be retrieved and used. Whether and when that produces visibility is the platforms’ decision, made query by query.

How a business can tell where it stands in this chain — crawl access, indexation, retrieval, citation — is a measurement question as much as a technical one, and it is the subject of the research side of this publication, organised through the Knowledge Center.

Common misconceptions

“AI systems browse the web live for every answer.”
Mostly no. The dominant path is retrieval from a pre-built index, exactly as with classic search; live fetching exists but is a secondary, user-triggered path. An answer produced with no search step at all comes from parametric memory and involves no retrieval.

“Blocking the AI training crawler removes you from AI search.”
No — and the platforms document the separation. GPTBot (training) and OAI-SearchBot (search) are independent controls at OpenAI; ClaudeBot and Claude-SearchBot are independent at Anthropic; PerplexityBot is, per Perplexity, not a training crawler at all. Blocking the wrong bot is a common way businesses accidentally remove themselves from AI answers while believing they only opted out of training.

“There is a special AI markup or file that gets you retrieved.”
No platform documents one. Google states that AI Overviews and AI Mode have no technical requirements beyond ordinary search eligibility. Structured data still matters for classic search features and for machine-readability generally, but claims that a particular schema type or llms.txt-style file is required, or is a documented ranking input for these platforms’ retrieval, are not supported by the platforms’ own documentation.

“If we’re indexed, we’re in the running for every answer.”
Indexing puts a page in the store; it does not match it to any particular question. Retrieval is query-shaped: the derived queries decide which corner of the index is even consulted.

“Ranking #1 in classic search means the AI answer uses your page.”
The candidate pools overlap, but the narrowing stage answers to the generated answer’s needs, not to the classic results page. Highly ranked pages are regularly absent from AI answers, and lower-ranked pages are regularly used, because a passage — not a page — is the unit being selected.

“The systems all work the same way.”
They share an architecture family (retrieve, narrow, ground, generate) and differ in almost every implementation detail that matters to a site owner: whose index, which crawlers, how queries are derived, and what is disclosed. Treating them as one system produces advice that is wrong for each of them in a different way.

Frequently asked questions

What is the difference between indexing and retrieval?
Indexing is the advance work: crawled pages are analysed and stored in structures built for fast lookup. Retrieval is the question-time work: finding, in those structures, the passages relevant to a specific (derived) query. A page must be indexed before it can be retrieved, and being indexed guarantees nothing about retrieval.

What is query fan-out?
Google’s term for issuing multiple related searches from one user question — across subtopics and data sources — and combining the results when generating the answer. OpenAI and Microsoft document functionally similar behaviour (rewriting a prompt into one or more targeted queries) without using the term.

Do AI systems use embeddings to find pages?
Embeddings and vector similarity are established, foundational retrieval technology, and dense retrieval is the retrieval mechanism in the RAG architecture the field is built on. The exact role of vector search inside each consumer platform’s production pipeline is not published; hybrid lexical-plus-semantic retrieval is the industry-standard architecture, but platform-specific claims should be treated as inference.

Can a page be retrieved but not cited?
Yes — routinely. Retrieval puts a passage in the candidate pool; narrowing and the model’s choices decide what is used and cited. Most retrieved passages in any given answer are not cited, because the pool is deliberately larger than the answer needs.

How can I check whether AI crawlers can access my site?
Each platform publishes its user agents and IP ranges (OpenAI, Perplexity, and Anthropic all maintain such pages, linked in the references). Compare those against your robots.txt rules, your CDN/firewall configuration, and your server logs. A bot that never appears in logs, on a site it should be crawling, is a finding worth investigating.

What remains unknown

Honesty about the boundary is part of this guide’s job. The following are not published by the platforms, and any source claiming them should be read accordingly: the reranking criteria and factor weights used to narrow candidate pools; candidate pool sizes and the number of passages passed to the model; each platform’s chunking strategy; the precise blend of lexical and semantic matching in production retrieval; and the triggers that decide whether an AI answer appears for a given query at all. These are proprietary mechanisms. They may be inferred, observed indirectly, and studied — several of those studies are worth reading — but they cannot currently be stated as fact.

References

Platform documentation

Technical and academic sources

Reputation.blog Editorial. This guide describes publicly documented platform behaviour and established retrieval technology as of its review date. Platform mechanisms change; the reference list records the documentation this account is based on.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top