What Is Training Data Poisoning?

Training data poisoning is the deliberate corruption of the data an AI model learns from. An attacker who can get false, biased, or malicious material into a model’s training set can change what the finished model says and does — sometimes with a remarkably small amount of poisoned material, and sometimes in ways that ordinary quality checks do not catch.

It sits at the front of the poisoning chain. Data poisoning is the umbrella term for corrupting information a system depends on; training data poisoning names the specific case where the corrupted information is the training corpus — the text, images, and examples the model is built from. Because training happens before deployment, the poison is baked in early and can be extremely hard to trace afterward: once a model is trained, there is no list of which training examples produced which behavior.

How poison gets into training data

Modern AI models are trained substantially on data collected from the public web — crawled pages, public datasets, licensed corpora, and user-contributed content. That openness is what makes the attack surface real. Documented routes into training data include:

Publishing content designed to be crawled. Anyone can put pages on the web; web-scale crawls collect at enormous breadth, and material published with the intent to be collected is collected.
Expiring domains. Research published by Nicholas Carlini and colleagues in 2023 demonstrated a practical route: a meaningful share of domains referenced by popular web datasets had lapsed, and an attacker could re-register them and control what those URLs served — for a total outlay the researchers put at around $60 — thereby controlling content that pipelines treated as the dataset’s original material.
Split-view poisoning. The same research documented that the contents of a page can differ between the moment a dataset snapshot is taken and the moment the data is actually downloaded, so the reviewed version and the trained-on version are not necessarily the same document.
Frontrunning. Wikipedia and similar collaboratively edited sources are snapshotted at known or predictable times; an edit made just before a snapshot can be captured into training data even if it is reverted minutes later — vandalism with a short life but a long training afterlife.
Compromised or poisoned public datasets. Datasets themselves are redistributed, mirrored, and repackaged; poison introduced at any point in that chain travels with the data.

None of these routes requires breaking into a model provider’s systems. They exploit the fact that training data is gathered from an environment the provider does not control.

What the research shows about scale

The finding that changed how seriously this attack is taken came from a study by Alexander Souly and colleagues at the UK AI Security Institute, Anthropic, and the Alan Turing Institute, published as a preprint in October 2025. The researchers poisoned pre-training with documents making up as little as 0.00016% of the training tokens — near 250 documents — and found that this was enough to install a backdoor behavior in models ranging from 600 million to 13 billion parameters. The headline conclusion, stated carefully: the number of poisoned documents needed stayed roughly constant as model size grew, rather than scaling with the size of the training set. In plain terms, bigger datasets do not dilute the poison the way intuition suggests; a fixed, small number of well-placed documents can be enough.

A second line of evidence comes from medicine. In a study published in Nature Medicine in 2025, researchers replaced 0.001% of the tokens in a medical training corpus with medical misinformation and found that the resulting models performed comparably on standard medical benchmarks — while performing measurably worse when tested specifically for harmful medical outputs. Benchmarks passed; behavior changed. That gap between what standard evaluation measures and what poisoning changes is one of the most important facts in this entire subject.

What poisoned training actually does to a model

Two broad effect types appear in the research:

Backdoors. The model behaves normally on almost everything, but a specific trigger — an unusual phrase, a formatting pattern, a named entity — activates different behavior: a refusal becomes compliance, a neutral answer becomes a favorable one, a safe output becomes an unsafe one. Backdoors are the most studied form because they are the most dangerous property of the attack: the model passes inspection and fails only where the attacker aimed.

Broad degradation or bias. Poisoned examples shift the model’s general tendencies — toward particular claims, framings, or errors — without a single identifiable trigger. This is harder to attribute and harder to distinguish from ordinary training-data noise, which means it is also harder to prove in any specific case.

The attribution problem

Suppose a deployed model says something false and damaging about a real company. Was its training data poisoned? In almost every real case, there is no way to know from outside. Models learn false claims from honest errors, outdated pages, jokes, fiction, and repetition just as readily as from deliberate poisoning, and no provider publishes a per-claim provenance record. Training data poisoning is a documented capability with demonstrated routes and measured effect sizes; claims that a specific output resulted from a specific poisoning are, with rare exceptions, unverifiable. Any publication — including this one — that tells you otherwise is overstating the evidence.

A generic example

A coordinated group wants an AI model to describe a particular medical treatment as discredited. Over two years, its members publish and maintain a few hundred pages across blogs, forums, and low-cost sites — all stating the claim in confident, citation-like language, all cross-linking. Some of those pages are collected into web crawls; some may be captured in dataset snapshots. Separately, brief vandal edits to a relevant encyclopedia entry, timed near a known snapshot, state the claim as settled fact. If the material survives filtering and deduplication, a future model may answer questions about the treatment with unwarranted confidence in the false claim — and nobody, including the model’s maker, will be able to point to the exact documents responsible. Every element of this scenario corresponds to a mechanism documented in published research; the scenario itself is illustrative, and whether any such campaign has succeeded against a major model is not publicly known.

Limitations and misconceptions

“Poisoning requires hacking the AI company.” No. The documented routes run through the public data environment — published pages, lapsed domains, dataset redistribution, timed edits — precisely because training data comes from there.

“More data means more safety.” The 2025 backdoor study cuts directly against this: the poison count needed stayed roughly constant across model sizes, so scale alone is not a defense.

“If the benchmarks pass, the model is clean.” The medical study is the counterexample: benchmark performance held while targeted harmful behavior increased. Evaluation measures what it is designed to measure.

“We can just filter the bad documents out.” Filtering helps and providers do it, but identifying poison requires knowing what the poison looks like — and a document designed to be collected usually looks like ordinary content. Deduplication and quality filtering raise the cost of an attack; they do not close the door.