About this site

Agent Evals Explained is about measuring agents: what an eval is, the benchmarks agents are quoted on (described from their papers), pass@k against pass^k, the uncertainty in a score and how many tasks it takes to shrink it, comparing two agents, graders and LLM judges, grading whole trajectories, the harness as part of the result, contamination, and cost. Every chapter is built around an animation. It is the fifth of a family of agent sites, after Agent Harnesses Explained, Agent Protocols Explained, Agent Context Explained and Agent Orchestration Explained, next to the LLM-systems sites (the Transformer Decoder Explainer, LLM Inference Explained, Inference Trade-offs Explained and others, all in the header's menu).

A synthetic benchmark

No real agent is run. The engine draws a population of 4,000 tasks, each with a difficulty and a discrimination, under the two-parameter logistic model of item response theory (applied to language-model benchmarks by tinyBenchmarks); an agent is a skill; a run passes with probability 1 / (1 + exp(−a(θ − b))). Runs on one task can be correlated by a chosen amount. A benchmark samples tasks from the population, so its score estimates the population's, the view Miller (2024) takes. Because the truth is known, every interval can be checked against it. Chapter 2's task is SWE-bench's grading rule run on a three-function toy repository by the engine's harness with a scripted model. Chapter 6's judge is a Bradley–Terry choice with three bias terms; chapter 7 grades the engine's own traces, including three recorded once from Qwen2.5-1.5B-Instruct on a CPU and replayed token for token; chapter 8 runs a seeded stochastic policy (not a language model) through the engine's real harness under sixteen configurations, and chapter 10 prices those runs with the harness's token accounting and a dated price table; chapter 9 leaks synthetic tasks whose texts are random token ids.

The engine, and how it is tested

The engine is Agent_Loop_Sim's evals module (engine 1.11.1): a Python reference with a TypeScript port, vendored here at commit 46b445c with every file's SHA-256 recorded. In the engine's CI:

• the estimators are tested against scipy and statsmodels: the binomial and beta functions, Wald, Wilson and Clopper–Pearson intervals, cluster-robust standard errors (statsmodels' OLS), McNemar's test (exact and corrected), the two-proportion test and the paired and Welch t statistics; pass@k and pass^k against exact combinatorics; the power formulas against simulation; Cohen's κ against statsmodels, the judge correction's standard error against finite differences and its coverage by simulation, and the Pareto front against its definition;
• the special functions underneath (exp, ln Γ, the incomplete beta function, the normal distribution and its quantile) are written out from published algorithms (fdlibm, Numerical Recipes, Marsaglia, Wichura's AS 241), so the browser computes every value bit for bit as Python does: the TS port reproduces every fixture exactly, with no tolerance.

• In this site's CI, the Python reference installed from the vendored commit regenerates this site's fixtures (everything each chapter animates), and the unit tests require the vendored TS engine to reproduce them exactly.
• Every number in the prose is computed by the engine when the page is built, never typed in. The recorded run is on the data page.
• Every code block in a chapter is cut from the vendored engine, and every equation compiles; end-to-end tests play, step, scrub and reset every animation in light and dark mode at desktop and phone widths, and check its captions at key steps against the Python reference's frames.

What is illustrative

• The task population (difficulty spread 1.3, discrimination 0.7 to 1.9), every agent's skill, every within-task correlation, the number of tasks, runs, reruns and bootstrap resamples. Nothing is fitted to a real benchmark or model.
• Chapter 2's repository, issue, patches and agent. The benchmark table quotes each paper at its release; benchmarks are revised, so check the current version before comparing.
• Chapters 6 to 10: the judge's biases and noise, the pass/fail judge's sensitivity and specificity, the trajectory score's weights, the task suite, both models' skills and the pairing of those skills with two list prices, the malformed-output rates, and every leak, paraphrase and memorisation rate. Token counts and costs are what the engine's harness accounts for those simulated runs.
• The statistics are not illustrative: they are the standard estimators, computed exactly under the stated assumptions.

Accessibility

Every animation has keyboard controls (Space plays or pauses, the arrow keys step), a live caption read out by screen readers, and starts paused when the reader prefers reduced motion. Colours come from Okabe and Ito's colour-blind-safe palette, and nothing is told by colour alone: a failed run, and an interval that misses, are also hatched.

Source

The site's source is on GitHub (MIT). For the papers behind the chapters, see Papers 07: Evaluation and Benchmarks.