What an eval is

Tasks, runs, a grader and a score: runs accumulate on a small benchmark and the uncertainty band shrinks, but more slowly than it seems once you count tasks, not runs, as the unit.

Loading the animation…

Concept

An eval has four parts. A set of tasks (a benchmark); an agent that attempts each one, perhaps several times (runs); a grader that turns each attempt into pass or fail (a test suite, a checker, a judge); and a score, usually the fraction of runs that pass. The score is a measurement, and like any measurement it comes with an error bar. This site is about that error bar: where it comes from, how big it is, and how to make it smaller.

The animation grades 120 runs one at a time: 40 tasks drawn from a synthetic population, each run 3 times. After the first run of every task the score is 67.5%, and the 95% band that treats every run as independent (Wilson's interval, filled) is 27.9% wide. Two more rounds of runs triple the data and that band narrows to 16.5%.

The band that treats tasks as the unit (dashed) hardly moves: 29.4% after one run of each task, 26.7% after three. The reason is that the runs on one task are not independent: a task that is easy for this agent passes again and again, and a hard one keeps failing. Running the same 40 tasks three times tells you a lot about those 40 tasks and little more about the next 40. The uncertainty that matters, how the agent would do on tasks like these, is dominated by which tasks you happened to pick.

The dotted line is the population's true score, 62.1%; these 40 tasks happen to be easier (their own true score is 71.2%). The dashed band covers the population's score; the filled band, by the third round, claims a precision the data do not have. Set the within-task correlation $\rho$ to 0 and the two bands agree more closely, because then the only thing tying runs together is task difficulty; set it to 0.9 and extra runs add almost nothing.

Maths

With $N$ runs of which $\hat p N$ pass, the binomial standard error $\sqrt{\hat p(1-\hat p)/N}$ assumes the runs are independent draws. When each of $T$ tasks is run $m_t$ times with $S_t$ passes, the runs cluster by task. The cluster-robust (CR1) standard error treats each task's total as one observation: its residual $S_t - m_t\hat p$ is squared and summed, with the small-sample factor $T/(T-1)$ . The engine's tests check it against statsmodels' OLS with cluster-robust covariance.

The ratio of the two variances is the design effect. For clusters of $m$ runs with intra-class correlation $\rho_{\text{ICC}}$ it is Kish's $1 + (m-1)\rho_{\text{ICC}}$ ; here, after three runs of each task, it is 2.53, so the 120 runs carry about as much information as 2.53 times fewer independent ones. The ICC here is more than the generator's $\rho$ , because task difficulty also makes runs on one task alike. Miller (2024) makes the same recommendation for language-model evals: when questions come in groups, or are asked more than once, compute clustered standard errors.

The runs come from an item-response model: task $t$ has difficulty $b_t$ and discrimination $a_t$ , the agent has skill $\theta$ , and a run passes with probability $\mu_t = 1/(1 + e^{-a_t(\theta - b_t)})$ (chapter 3 says more).

Code

The cluster-robust standard error, cut from the vendored engine (stats.ts):

for (let i = 0; i < t; i++) {
  const d = successes[i]! - runs[i]! * p;
  acc = acc + d * d;
}
const se = t > 1 ? Math.sqrt((t / (t - 1)) * acc) / n : naive;