AI Poisoning and RAG Poisoning: How False Information Gets Into AI Answers

In brief

“Poisoning” is the corruption of the information an AI system depends on. It can happen in the data used to train a model, in the documents a system retrieves at the moment it answers a question, or — without any attacker at all — through ordinary errors that get copied from source to source until they look like established facts. Peer-reviewed research has demonstrated that surprisingly small amounts of planted material can steer AI systems: five crafted texts per question in one retrieval study, roughly 250 planted documents in the largest training-poisoning study to date. What research has not established is how common deliberate poisoning is in the wild, or how any specific commercial platform defends against it, because platforms do not publish those details. For a business, the practical consequence is the same whatever the route: if false information about you sits in the sources AI systems read, it can come back out as a confident answer. This article separates what is documented, what is research-demonstrated, and what remains unknown.

1. Definitions: three routes to a false answer

Data poisoning (training-time poisoning). An attacker introduces crafted examples into the data a model learns from — pretraining corpora, fine-tuning sets, or feedback data — so that the trained model behaves differently: answering wrongly, favoring a claim, or responding to a hidden trigger. The poison is baked in before anyone asks a question.

Retrieval poisoning (RAG poisoning). Many AI answer systems use retrieval-augmented generation (RAG): when a question arrives, the system first retrieves documents from an external collection — a search index, a knowledge base, the live web — and then writes an answer grounded in what it retrieved. The model itself is untouched. Instead, the attacker plants texts in the retrieval collection, written so they are likely to be retrieved for a chosen question and likely to steer the answer. For systems that answer from the public web, the “collection” is effectively everything crawlable — which is why this route matters most for business information.

The organic case: error propagation and citogenesis. No attacker is required. A fact that is wrong on one page gets copied by other pages; an AI system reads several of them and treats their agreement as corroboration; the AI’s answer is then quoted by yet more pages, closing the loop. Journalism has a name for the underlying failure — circular reporting — and a coined term, citogenesis (from a 2011 xkcd comic), for the version where an unsourced claim picks up a citation and becomes nearly impossible to dislodge. The classic documented example predates AI answers: in 2008, a joke edit calling the coati a “Brazilian aardvark” on Wikipedia was repeated by multiple newspapers and even scholarly works, which were then available to be cited as proof of the claim. AI retrieval accelerates this old pattern, because a system that samples several agreeing sources has no built-in way to know they all descend from one error.

The three routes differ in mechanism, intent, and remedy — but converge on one property: AI systems repeat what their inputs contain, and their inputs are, to a large extent, the public record.

2. What the research actually demonstrates

Retrieval poisoning: a handful of texts per question

The reference study is PoisonedRAG (Zou, Geng, Wang, and Jia), published at USENIX Security 2025 — a peer-reviewed venue — after appearing as a 2024 preprint. The authors formulate the attack as the first knowledge-corruption attack against RAG: an attacker injects a small number of malicious texts into a knowledge database of millions of texts, aiming to make the system produce an attacker-chosen answer to an attacker-chosen question. Their central result: injecting five malicious texts per target question achieved a 90% attack success rate against knowledge databases containing millions of texts. The concept the paper establishes — at the level of mechanism, not method — is that a planted text must satisfy two conditions: it must be retrieved for the target question, and once retrieved it must steer the generation. The authors also evaluated several defenses and report that the ones tested were insufficient against the attack. Two limits belong next to this finding: it was demonstrated against research RAG systems built on standard components, not against a named commercial platform; and it assumes the attacker can get texts into the collection being retrieved from — which, for web-scale retrieval, is precisely the open question discussed in Section 4.

Training poisoning: the dose is a count, not a percentage

The largest pretraining-poisoning study to date is Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples (Souly et al.), a collaboration between Anthropic’s Alignment Science team, the UK AI Security Institute’s Safeguards team, and the Alan Turing Institute, published as a preprint in October 2025 (not yet peer-reviewed at the time of writing). The researchers pretrained models from 600 million to 13 billion parameters on appropriately scaled datasets and inserted 100, 250, or 500 malicious documents. The documents taught a simple “backdoor”: a trigger phrase followed by gibberish output — deliberately low-stakes and easy to measure. Findings: 100 documents were not enough to reliably backdoor any model size; 250 or more reliably succeeded at every size, even though the largest models trained on more than twenty times as much clean data. For the largest model, 250 documents are roughly 420,000 tokens — about 0.00016% of its training data. The same near-constant pattern held in their fine-tuning experiments. The authors and Anthropic’s summary are explicit about the limits: the backdoor behavior tested was simple, the largest models tested (13B) are far smaller than frontier commercial systems, and whether the same dynamics hold for larger models or more complex behaviors is unknown.

An earlier feasibility result explains how planted text reaches training data at all. Poisoning Web-Scale Training Datasets is Practical (Carlini et al., 2023) showed two attack classes against the way web-scale datasets are actually assembled — one exploiting the fact that web content changes between the moment a dataset records a URL and the moment its contents are downloaded, another exploiting the periodic snapshotting of crowd-sourced sources such as Wikipedia. The authors state they could have poisoned 0.01% of two major datasets (LAION-400M, COYO-700M) for about $60, and that the attacks were immediately practical against ten popular datasets; they notified the dataset maintainers and recommended defenses. This is a research demonstration of practicality, not a report of an attack carried out.

Domain harm: medical misinformation

That poisoned training data produces substantively wrong outputs — not just triggered gibberish — was demonstrated in Nature Medicine in 2025 (Alber et al.). In a simulated attack on The Pile, a widely used training dataset, the researchers replaced just 0.001% of training tokens with medical misinformation and found the resulting models were measurably more likely to propagate medical errors. Notably for Section 6, the same paper used the defense direction as well: screening model outputs against a biomedical knowledge graph captured 91.9% of the harmful content in their tests — an early, domain-specific demonstration that cross-checking against a structured source of record catches much of what poisoning plants.

Persistence: backdoors can survive safety training

A distinct concern is whether a planted behavior, once learned, can be removed. In Sleeper Agents (Hubinger et al., 2024), Anthropic-led researchers deliberately constructed proof-of-concept models with deceptive backdoors — for example, writing secure code when the prompt indicated one year and exploitable code when it indicated another — and found the backdoor behavior persisted through standard safety training (supervised fine-tuning, reinforcement learning, and adversarial training), and that adversarial training could instead teach models to recognize their triggers better, hiding the behavior. This was a constructed proof of concept, not a discovery of a backdoored model in the wild; its significance is about the limits of post-hoc removal, which is why prevention and source control dominate the defense discussion.

3. A taxonomy of the poisoning family

“Poisoning” names a family, not a single technique. The map below organizes it by what is corrupted. Each entry is labeled with its evidence status; all are described at the level the research literature itself uses.

3.1 Training-data poisoning subtypes (research-demonstrated)

• Label-flipping — the oldest and simplest category: training examples are given wrong labels, degrading or skewing what the model learns. It requires some control over labeling, which is why later work relaxed that assumption.
• Clean-label poisoning — the planted examples are correctly labeled and look innocuous, yet still steer the model on a chosen target. Poison Frogs! (Shafahi et al., 2018) demonstrated this against image classifiers: a single crafted, properly labeled image could control classification of a specific target under transfer learning, with roughly fifty examples making the attack reliable in end-to-end training — and, as the authors note, such examples could enter a training set simply by being left on the web for a scraper.
• Backdoor poisoning — planted examples teach a hidden trigger–response pairing while the model behaves normally otherwise; the Souly et al. and Sleeper Agents results above are the large-model instances.
• Availability poisoning (“unlearnable” data) — the goal is inverted: instead of steering a model toward a chosen behavior, the data is crafted so models cannot learn from it properly. The prominent documented example is defensive. Nightshade (Shan et al., 2023), from University of Chicago researchers, poisons images at the concept level: fewer than 100 poison samples could corrupt a Stable Diffusion XL prompt’s output, with effects bleeding into related concepts. The authors propose it explicitly as a last-resort tool for artists and creators whose work is scraped despite opt-outs — poisoning as self-defense, which complicates any simple attacker/defender framing.
• Dataset-assembly poisoning (split-view and frontrunning) — attacks on the pipeline rather than the content: exploiting the gap between when a dataset snapshots the web and when it downloads it, or the predictable timing of snapshots of editable sources. Demonstrated as practical by Carlini et al. (Section 2).

3.2 Federated-learning model poisoning (research-demonstrated). In federated learning, many participants train a shared model without pooling their data, submitting model updates instead. That opens a distinct surface: a malicious participant can poison the update rather than any dataset. Bagdasaryan et al. (How to Backdoor Federated Learning, 2018–2020) showed a single participant, in a single training round, could replace the joint model with a backdoored one reaching 100% accuracy on the backdoor task in their evaluation, and that their approach evaded the anomaly-detection defenses they tested. Relevance here is structural rather than immediate: it shows that wherever contributions are aggregated without verification, the aggregate can be captured by a small number of bad contributions — the same shape as the retrieval problem.

3.3 RLHF and preference-data poisoning (research-demonstrated, with honest limits). Modern models are aligned using human feedback: raters compare outputs, and the model is trained toward the preferred ones. That feedback data is itself poisonable. Rando and Tramèr (Universal Jailbreak Backdoors from Poisoned Human Feedback, 2023) showed that poisoning RLHF data can embed a trigger word that acts as a universal jailbreak. The paper is equally clear that such backdoors are significantly harder to plant than the backdoors previously studied on language models — an important calibration against assuming every pipeline is equally soft.

3.4 Knowledge-graph poisoning (research-demonstrated). Knowledge graphs — structured stores of facts about entities and their relations — feed question answering, search features, and recommendation, and increasingly serve as reference data for AI systems. Zhang et al. (Data Poisoning Attack against Knowledge Graph Embedding, IJCAI 2019) demonstrated attack strategies that manipulate the calculated plausibility of arbitrary targeted facts by adding or deleting facts in the graph, verified on two standard benchmarks. For information owners this is the most entity-shaped entry in the taxonomy: the “fact” whose plausibility is manipulated is exactly the kind of statement — who founded what, who is located where, who is affiliated with whom — that knowledge panels and AI answers repeat.

3.5 Recommender shilling (research-demonstrated; observed in the wild for decades). The consumer-facing ancestor of much of this article. A shilling attack floods a recommender system with fake profiles and ratings to push an item up (“push”) or a competitor down (“nuke”). Lam and Riedl named and measured the attack class in 2004 (Shilling Recommender Systems for Fun and Profit, WWW 2004), testing it experimentally on a million-rating movie dataset, and GroupLens follow-up work documents the surrounding reality: fake reviews and coordinated rating manipulation on commercial platforms are long-documented occurrences, including authors praising their own books and attacking rivals’. Shilling matters to this site’s readers directly: reviews and ratings are ranking inputs, and the same planted-signal logic now applies to systems that read reviews as evidence about a business.

3.6 Search poisoning — the older web ancestor (measured in the wild). Before AI answers, there were search results, and they were poisoned at scale. Leontiadis, Moore, and Christin (USENIX Security 2011) tracked 218 drug-related queries daily for nine months and found that about a third of all search results led to one of more than 7,000 compromised, high-ranking websites that redirected visitors to a few hundred illicit pharmacy sites — facilitating, by their estimate, more illegal drug sales than email spam. The lesson that carries forward: manipulation of the layer people consult for answers is not a hypothetical genre; it has already run as an industry once, against the previous answer layer.

4. What is NOT established

Holding the demonstrated results and the open questions apart is most of the value here.

• Prevalence in the wild is not measured. There is no published, reliable measurement of how often commercial AI answers are wrong because of deliberate poisoning, as distinct from ordinary error, staleness, or organic citogenesis. Anyone quoting such a rate is inventing it.
• Per-platform defenses are not public. Google, OpenAI, Microsoft, Anthropic, and Perplexity do not publish their anti-poisoning methods, thresholds, or source-weighting rules in auditable detail. Claims that a given platform “can’t be poisoned” or “is easily poisoned” both exceed the public evidence.
• Transfer from the lab is unproven. PoisonedRAG’s 90% figure is from research systems; Souly et al.’s 250 documents are from models up to 13B parameters trained for the experiment. Frontier commercial systems differ in scale, data curation, retrieval design, and post-training — in directions that could raise or lower susceptibility, and the honest statement is that the public record does not settle it.
• The organic case has no attacker to find. Much of the false business information in circulation is not planted; it is stale, copied, or aggregated without verification. Treating every error as an attack leads to the wrong remedy; treating every attack-shaped pattern as an accident leads to none.

5. Why this matters for business information specifically

Business facts are an unusually exposed category of information, for structural reasons. They are short (a name, a location, an opening date, a service list), numerous (repeated across directories, profiles, and aggregators), and seldom verified at the source by the systems that repeat them. Retrieval-based AI answers draw on exactly this material. That creates three concrete exposure patterns:

1. Planted claims. A page asserting a false claim about a business — invented credentials, a fabricated event appearance, a service never offered — phrased in the same terms a customer would ask about, is shaped exactly like the texts the PoisonedRAG result concerns: retrievable for a targeted question and written to be repeated. Whether or not any given instance is part of a deliberate attack, the mechanism by which it surfaces in an answer is the retrieval mechanism, and the answer carries no “unverified” marker.
2. Stale claims. Yesterday’s true fact — old hours, a former address, a discontinued service — is functionally identical to a false one once the business has moved on. Because aggregators copy each other, the stale version may outnumber the current one across sources, and majority agreement among sources is the signal AI systems are best able to use.
3. The citogenesis loop. An AI answer repeats the error; the answer is quoted, screenshotted, or republished; the republication becomes another “source.” Detection gets harder with each cycle because the false claim’s apparent corroboration is real — only its independence is fake.

6. Detection and defense for information owners

None of the following guarantees an outcome — no honest source can offer that — but each step is grounded in the mechanisms above.

• Correct at the source, not at the answer. AI systems read sources. Find where a false claim actually lives — your own site’s outdated page, a directory listing, an aggregator profile, a scraped copy — and correct or remove it there. Asking an AI platform to fix an answer without fixing the underlying sources leaves the retrieval inputs unchanged.
• Make your own record the easiest one to retrieve and quote. A clear, current, plainly worded source-of-record page for your core facts (what the business is, where, since when, what it offers, how to contact it) gives retrieval systems a canonical text to draw on, and gives you a reference point for spotting divergence elsewhere. Consistency across your profiles matters for the same reason corroboration does: systems weigh agreement, so your facts should agree with each other everywhere they appear.
• Monitor with dated records. Because answers vary between runs and over time, single checks prove little. A defensible monitoring record names the platform, the exact prompt, the date, the answer given, and the sources cited — repeated on a schedule. That is how a planted or stale claim is distinguished from a one-off error, and how a correction’s effect (or non-effect) is documented rather than assumed.
• Watch the citation, not just the answer. When an AI answer cites sources, the cited pages are the actionable layer: they show which documents the system actually used. A false claim traced to a cited page has a concrete remedy (correct that page, or its owner); an uncited false claim is weaker evidence and worth re-testing before acting on.
• Use the structured channels that exist. Where platforms offer business-controlled data channels — verified business profiles, knowledge-panel feedback mechanisms, corrections processes — use them; they are the routes by which an information owner can assert facts with provenance attached, rather than merely adding one more uncalculated voice to the pile.
• Treat suspicious patterns as evidence, not proof. A false claim phrased for retrieval, appearing across freshly created or unrelated pages, is consistent with planting; it is also consistent with aggressive aggregation. Document first (dated captures, source lists), correct the sources you control, and escalate through platform reporting channels with the documentation attached.

7. What remains unknown

• Whether the near-constant poisoning dose found at up-to-13B scale holds for frontier-scale commercial models, and for behaviors more consequential than triggered gibberish. (Stated as open by the study’s own authors.)
• The real-world prevalence of deliberate poisoning of AI answers about businesses, in any industry, at any rate. No measurement exists in the public record.
• Which specific defenses any major platform operates against training or retrieval poisoning, and their measured effectiveness. Not published.
• How quickly corrections at the source propagate into AI answers on each platform — the lag between fixing the record and the answer changing is not documented by any platform and can only be established case by case, by dated re-testing.
• Whether the organic problem (staleness and citogenesis) causes more wrong business answers than all deliberate attacks combined. Plausible; unmeasured; stated here as a hypothesis, not a finding.

References

1. Zou, W., Geng, R., Wang, B., & Jia, J. (2025). PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models. Proceedings of the 34th USENIX Security Symposium. https://www.usenix.org/conference/usenixsecurity25/presentation/zou-poisonedrag — preprint: https://arxiv.org/abs/2402.07867
2. Souly, A., Rando, J., Chapman, E., et al. (2025). Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples. arXiv:2510.07192. https://arxiv.org/abs/2510.07192
3. Anthropic (2025). A small number of samples can poison LLMs of any size. https://www.anthropic.com/research/small-samples-poison
4. Carlini, N., Jagielski, M., Choquette-Choo, C. A., et al. (2023). Poisoning Web-Scale Training Datasets is Practical. arXiv:2302.10149. https://arxiv.org/abs/2302.10149
5. Alber, D. A., Yang, Z., Alyakin, A., et al. (2025). Medical large language models are vulnerable to data-poisoning attacks. Nature Medicine, 31, 618–626. https://doi.org/10.1038/s41591-024-03445-1
6. Hubinger, E., Denison, C., Mu, J., et al. (2024). Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. arXiv:2401.05566. https://arxiv.org/abs/2401.05566
7. Shan, S., Ding, W., Passananti, J., Wu, S., Zheng, H., & Zhao, B. Y. (2023). Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models. arXiv:2310.13828. https://arxiv.org/abs/2310.13828
8. Shafahi, A., Huang, W. R., Najibi, M., et al. (2018). Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks. arXiv:1804.00792. https://arxiv.org/abs/1804.00792
9. Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., & Shmatikov, V. (2018). How To Backdoor Federated Learning. arXiv:1807.00459. https://arxiv.org/abs/1807.00459
10. Rando, J., & Tramèr, F. (2023). Universal Jailbreak Backdoors from Poisoned Human Feedback. arXiv:2311.14455. https://arxiv.org/abs/2311.14455
11. Zhang, H., Zheng, T., Gao, J., Miao, C., Su, L., Li, Y., & Ren, K. (2019). Data Poisoning Attack against Knowledge Graph Embedding. Proceedings of IJCAI 2019. https://arxiv.org/abs/1904.12052
12. Lam, S. K., & Riedl, J. (2004). Shilling Recommender Systems for Fun and Profit. Proceedings of the 13th International Conference on World Wide Web (WWW 2004), 393–402. https://doi.org/10.1145/988672.988726
13. Lam, S. K., Frankowski, D., & Riedl, J. (2006). Do You Trust Your Recommendations? An Exploration of Security and Privacy Issues in Recommender Systems. GroupLens Research, University of Minnesota. http://files.grouplens.org/papers/lam-etrics2006-security.pdf
14. Leontiadis, N., Moore, T., & Christin, N. (2011). Measuring and Analyzing Search-Redirection Attacks in the Illicit Online Prescription Drug Trade. Proceedings of the 20th USENIX Security Symposium. https://www.usenix.org/conference/usenix-security-11/measuring-and-analyzing-search-redirection-attacks-illicit-online
15. Wikipedia contributors. Circular reporting (for citogenesis: term coined by Randall Munroe, xkcd no. 978, November 2011; and the documented 2008 coati “Brazilian aardvark” propagation case). https://en.wikipedia.org/wiki/Circular_reporting

Published: November 1, 2026
Last reviewed: November 1, 2026