The benchmarks

SWE-bench, τ-bench, GAIA, WebArena, OSWorld and Terminal-Bench from their papers: what each measures, how it grades, and its known problems; then one SWE-bench-style task, issue to patch to hidden tests.

Loading the animation…

Concept

The animation is one task in the style of SWE-bench, the benchmark most coding agents are quoted on. The agent gets a repository at a base commit and an issue written by a person. It reads, edits and runs things, then stops; its patch is the difference between the repository it leaves and the base commit. Only then are the hidden tests added and run. SWE-bench's dataset names them: FAIL_TO_PASS tests were written with the real fix and fail on the base commit, so they check the issue is fixed; PASS_TO_PASS tests passed before, so they check nothing else broke. The task is resolved only if every one of both kinds passes.

The three patches show what the grader can and cannot see. The right fix passes. A patch that breaks sub() while fixing add() fails a PASS_TO_PASS test, even though the agent said it had fixed the arithmetic. The overfit patch, return a * b - 1, gives $\text{add}(2, 3) = 5$ and nothing else right: the thorough hidden tests catch it, but with the weak tests (only the example from the issue) it is scored as resolved. A grader is only as good as its tests, and the agent's own report counts for nothing.

The benchmarks, from their papers

Benchmark	What it measures	Size	How it is graded	Known issues
SWE-bench (Jimenez et al., 2023)	resolving real GitHub issues in a full repository	2,294 tasks from 12 Python repositories	hidden unit tests (FAIL_TO_PASS, PASS_TO_PASS)	SWE-Bench+: of successful patches it checked, 32.67% had the solution in the issue or its comments and 31.08% passed weak tests; over 94% of issues predate the models' training cut-offs
SWE-bench Verified	the same, on a cleaner subset	500 tasks, human-validated	as SWE-bench	Prathifkumar et al. (2025): models located the files to edit far more often on Verified than on newer benchmarks, which suggests they have seen its tasks
τ-bench (Yao et al., 2024)	a tool-using agent serving a simulated user under a domain policy (retail, airline)	two domains of tasks	the final database state against the annotated goal; introduced pass^k	the user is simulated by a language model; the repository says its tasks are outdated and points to τ²-bench and its successor
GAIA (Mialon et al., 2023)	a general assistant: reasoning, browsing, files, tools	466 questions	each has one short answer	humans 92% against 15% for GPT-4 with plugins at release; answers to 300 are held back for the leaderboard, so the rest are public (our inference: those can leak)
WebArena (Zhou et al., 2023)	web tasks on self-hosted sites: shopping, a forum, code hosting, content management	812 tasks (repository)	functional correctness of the outcome	the sites must be reset to their initial state after each evaluation; at release the best agent scored 14.41% against 78.24% for people
OSWorld (Xie et al., 2024)	computer use in real operating systems and apps	369 tasks	a custom execution-based script per task	community-reported task issues led to OSWorld-Verified (2025); tasks that need an account or a proxy fail if those are not set up
Terminal-Bench 2.0 (Merrill et al., 2026)	hard, realistic work in a terminal	89 tasks	tests per task, each with a human-written solution	the harness moves scores: StateM reports one model rising from 83.1% to 92.1% on version 2.1 by changing only the harness; reward hacking (passing the checks while missing the point) is measured separately

Counts and results are each paper's own, at its release; benchmarks are revised, so check the current version before comparing numbers. Every arXiv entry above was checked at export.arxiv.org on 2026-10-13.

Three things recur down the last column, and the later chapters take each in turn: graders that pass wrong work (chapter 6), the harness being part of the result (chapter 8), and contamination, tasks the model may have seen in training (chapter 9).

Maths

Grading turns each task into one bit, $x_t \in \{0, 1\}$ , and the score is their mean, $\hat p = \frac{1}{T}\sum_t x_t$ . Every grader error moves that bit. If a fraction $f_+$ of failed tasks is wrongly passed (weak tests) and a fraction $f_-$ of solved ones is wrongly failed (flaky tests, a broken environment), the expected score is $p(1 - f_-) + (1-p)f_+$ instead of $p$ . A biased grader is not noise that more tasks average away: chapter 4's intervals would tighten around the wrong number.

Code

Running one hidden test alone, cut from the vendored engine (swe.ts):

const blocks = src.split("\n\n\n");
const keep = [blocks[0]!, ...blocks.slice(1).filter((b) => b.startsWith(`def ${name}(`))];