/learn
How to measure an agent
One chapter per idea, each opening with an animation. Every frame is computed by the Agent_Loop_Sim engine's evals module, identically in Python and in your browser. Toggle layers (Concept / Maths / Code) inside any chapter to choose how deep to go. For the papers behind it, see Papers 07: Evaluation and Benchmarks.
Tasks, runs, a grader and a score: runs accumulate on a small benchmark and the uncertainty band shrinks, but more slowly than it seems once you count tasks, not runs, as the unit.
SWE-bench, τ-bench, GAIA, WebArena, OSWorld and Terminal-Bench from their papers: what each measures, how it grades, and its known problems; then one SWE-bench-style task, issue to patch to hidden tests.
Some one of k tries passes, or all k do: the unbiased estimators derived, and the two curves pulling apart as k grows. Why reliability, not best-of-k, is what an agent's user feels.
The benchmark rerun again and again: Wald, Wilson, exact and bootstrap intervals and which ones cover the truth; why a few tenths of a point on a 500-task benchmark is a tie; and how many tasks you need.
Two agents on the same tasks: the paired difference and McNemar's test against the unpaired comparison, the correlation that pairing exploits, and how seeds and repeated runs add or remove variance.
- 06Graders
Unit tests, LLM judges and people: a simulated judge with position, length and self-preference biases, the swap test and what it cannot fix, Cohen's κ, and correcting a lenient pass/fail judge with a labelled calibration set.
Grade the path as well as the result: wasted steps, unsafe actions run or stopped, verification and false claims, on scripted runs and on traces recorded from a small open-weights model.
One simulated model under sixteen harness configurations (tools, call format, turn budget, model), run through the engine's harness: the same model 30 points apart, measured with paired comparisons.
Leaked benchmark tasks inflate a score; n-gram overlap finds verbatim copies and misses paraphrases, canary strings say only that something leaked, and a time split shows the score dropping at the training cutoff.
Success against dollars per task: the Pareto front of the harness configurations, dollars per solved task, why more retries cost more per success when attempts are correlated, and why the front is itself an estimate.