How Do AI Systems Retrieve Information From Websites?

AI systems get information from websites the unglamorous way: automated programs called crawlers visit pages and copy them, the copies are processed into readable text and stored in indexes, and at answer time the relevant text is retrieved and handed to a model. The important detail is that “AI crawlers” are not one thing — the major operators publicly document three distinct kinds, one that collects material for training models, one that builds the search index used for live answers, and one that fetches a page on demand because a user asked about it, and each kind can be allowed or blocked separately.

The three jobs a crawler can be doing

Training crawlers gather web content that may be used to train future models. OpenAI documents GPTBot for this purpose; Anthropic documents ClaudeBot. What these collect shapes what models learn during training — knowledge with a cutoff date, not live answers.

Search crawlers build and refresh the index that AI assistants search when they answer questions with current information. OpenAI documents OAI-SearchBot for this; Perplexity documents PerplexityBot, which indexes pages to display and cite in its answers. If your pages are absent from these indexes, you are absent from the answers built on them — regardless of what the model learned in training.

User-triggered crawlers fetch a specific page in real time because a person asked the assistant to look at it — pasting a link, asking about a particular site. OpenAI documents ChatGPT-User for these fetches, and Perplexity documents Perplexity-User similarly. These do not crawl the web on their own schedule; they retrieve single pages on demand.

Google’s arrangement differs in form: Googlebot crawls for Search and AI features built on Search’s index, while Google-Extended is not a crawler at all but a robots.txt control token that lets a site opt its content out of certain Gemini training uses without leaving Google Search. The general pattern holds across operators: training, search, and on-demand fetching are separable, documented, and controlled separately.

From a web page to retrievable text

A crawler’s visit produces a copy of the page, but the raw page is not what gets indexed. Systems render or parse the HTML, strip navigation, scripts, and layout machinery, and extract the readable text — headings, paragraphs, lists, tables. That extracted text is what is indexed, chunked, and eventually retrieved.

Two practical consequences follow. First, rendering is limited and varies: some crawlers process pages much as a browser would; many work from the HTML largely as delivered, and content that only appears after complex JavaScript runs may be captured late, partially, or not at all. Operators do not publish complete rendering specifications for their AI crawlers, so the working assumption is the conservative one: text present in the page’s HTML is retrievable; text conjured by scripts is a gamble.

Second, extraction reads text, not design. Information presented as an image — a menu photographed and uploaded, a price list saved as a graphic, opening hours in a banner — contains no text for the extractor to take. To the retrieval pipeline, it is not hard to read. It is not there.

At answer time

When a question needs web information, the assistant’s system searches its index (the retrieval step described in our RAG explainer), pulls the best-matching passages, and gives them to the model, which composes the answer and, in products that show sources, cites the pages the passages came from. Your website participates in this only through what the crawler could copy and the extractor could read, possibly weeks before the question was asked. Nothing about your live site is consulted in that moment unless a user-triggered fetch is involved.

The site’s say in the matter: robots.txt and crawler controls

Website owners communicate with crawlers through robots.txt, a small public file naming which bots may access which parts of the site. Because the AI operators document their bots separately, a site can make separate decisions: allow the search crawler so its pages can appear in AI answers, while disallowing the training crawler; or block a bot from sensitive sections while leaving public pages open. The documented operators state that their crawlers honor robots.txt directives.

The decisions have real trade-offs, and they are frequently made by accident:

Blocking a training crawler (GPTBot, ClaudeBot) withholds future training use. It does not remove you from live AI answers, which are served by the search index.
Blocking a search crawler (OAI-SearchBot, PerplexityBot) does the opposite and more: your pages leave the index those assistants retrieve from, so the business becomes invisible in their answers while remaining perfectly visible in ordinary search results.
Blocking everything is a legitimate choice for some sites, but it should be a choice — made by someone who knows which of the three jobs each blocked bot was doing.

Beyond robots.txt, controls are limited. There is no universal registry, no single “AI opt-out” switch, and no guarantee that every crawler on the internet identifies itself honestly — the documented behavior in this article describes the major operators who publish their practices.

What this means for a business website

State your core facts — name, services, location, hours, prices where you publish them — as plain text in the page, not inside images or graphics.
Keep important content in the delivered HTML rather than dependent on scripts to appear.
Maintain one clear, crawlable page for each thing you want found, with the complete facts on it.
Check your robots.txt (yoursite.com/robots.txt — it is public) and confirm that any AI-related blocks in it are there on purpose, and that you know which job each blocked bot performs.
Keep the facts on your site in agreement with your profiles and listings, because the index holds copies: an outdated page keeps being retrieved until it is re-crawled and replaced.

A business does not need to do anything exotic to be retrievable. It needs its information published in the form the pipeline actually reads: text, in the open, structured so a machine copying the page takes the facts with it.

Misconceptions

“The AI reads my website the way a customer does.” No customer experience is involved at any stage: a crawler copies, an extractor strips, an index stores text, a model reads passages. Anything that exists only in the visual experience — images, layout emphasis, design polish — does not reach the model.

“We’re in Google, so we’re in the AI answers.” The indexes overlap in input but are operated by different companies for different products. Google’s index does not supply ChatGPT’s or Perplexity’s answers, and each operator’s search crawler must reach your site on its own.

“Blocking AI bots protects our content, with no downside.” Blocking the training crawler and blocking the search crawler are different acts with opposite visibility effects. An undifferentiated block-all is how businesses disappear from AI answers without ever deciding to.

“If the text is somewhere on the page, the AI can find it.” Text behind scripts that never render for the crawler, inside images, or inside files the crawler does not process is, for retrieval purposes, unpublished.