Contamination and leakage

Leaked benchmark tasks inflate a score; n-gram overlap finds verbatim copies and misses paraphrases, canary strings say only that something leaked, and a time split shows the score dropping at the training cutoff.

Loading the animation…

Concept

A benchmark measures what a model can do with problems it has not seen. If its tasks, or their solutions, were in the training data, part of the score is memory, and the benchmark is measuring something else. Public benchmarks leak: their tasks sit on the open web, in repositories and discussions that later become training data. SWE-bench's tasks come from public GitHub issues, so the fix of every task is in the repository's own history.

The animation simulates it. 400 synthetic tasks each have a creation date (a quarter, 0 to 7) and a text; the model's training data stops after quarter 3. At a leak rate $L$ , each task from before the cutoff is in the training data with probability $L$ , as a verbatim copy or (40% of the time) a paraphrase; a leaked task is answered from memory with probability 0.7.

What it does to the score. The agent's clean score is 48.3%. At a 30% leak rate, 62 tasks have leaked and it scores 53.8%; at 60%, 60.8%. Nothing about the agent changed.

Three ways to look for it, each seeing part.

n-gram overlap. Look for the task's text in the training data, as GPT-3's authors did with 13-grams. Every verbatim leak is found: at 30%, 41 of 41. Paraphrases are not: 0 of 21, because a paraphrase that changes a quarter of the words keeps only 10.0% of the 8-grams. Dropping the flagged tasks brings the score to 50.7%, still 2.4 points above the clean score.
Canary strings. BIG-bench put a unique string in its files and asked that they be kept out of training data; a model that can complete the string has seen them. Here the canary comes back at every leak rate above zero. It says the benchmark leaked; it does not say which tasks, or by how much.
Time split. Tasks written after the training data was collected cannot have leaked. Compare scores before and after the cutoff. Clean, the two sides differ by 3.6 points ( $p$ = 0.47, noise); at a 30% leak rate by 14.3 points ( $p$ = 0.004). This is the idea behind LiveCodeBench, which keeps adding problems released after each model's cutoff and scores models only on those. It needs dated tasks and enough of them after the cutoff (chapter 4 again), and it assumes later tasks are no harder; at a 10% leak rate the gap, 7.0 points, is not yet distinguishable from noise ( $p$ = 0.16).

The safest defence is not detection but design: hold-out tasks that are never published, tasks generated fresh for each evaluation, and scores reported with the date of the tasks and of the model's training data.

Maths

Inflation. If a fraction $f$ of the benchmark leaked and a leaked task is recalled with probability $m$ , a task with pass probability $\mu$ now passes with probability $\mu + (1 - \mu)m$ , so the expected score rises by $m\,f\,\overline{(1 - \mu)}_{\text{leaked}}$ : most for hard tasks, which is where a benchmark is supposed to discriminate.

Why n-grams miss paraphrases. If each token of a copy is changed independently with probability $r$ , a particular $n$ -gram survives intact with probability $(1 - r)^n$ ; with $r$ = 0.25 and $n$ = 8, 0.100. A detector that needs half of a task's $n$ -grams to match will almost never fire. Shorter $n$ -grams survive more often but also match by chance more often; the threshold trades one error for the other.

The time split is the two-proportion test of chapter 4 between tasks before and after the cutoff. Under no leakage and no change in difficulty the difference has mean zero; the test's power to see an inflation $\delta$ on $n$ tasks a side is chapter 4's power formula, so small leaks on small benchmarks go unseen.

Code

The scan, cut from the vendored engine (contamination.ts):

const grams = ngrams(t.text, n);
let hit = 0;
for (const g of grams) if (corpus.has(g)) hit += 1;
const overlap = hit / grams.length;
const p = lk ? t.mu + (1.0 - t.mu) * cfg.memorise : t.mu;