Uncertainty

The benchmark rerun again and again: Wald, Wilson, exact and bootstrap intervals and which ones cover the truth; why a few tenths of a point on a 500-task benchmark is a tie; and how many tasks you need.

Loading the animation…

Concept

A score is one draw from a distribution of scores you might have got. The animation makes the distribution visible: it reruns a benchmark 30 times and draws each rerun's 95% interval against the truth. "95%" is a promise about exactly this picture: over many reruns, 95% of the intervals should cover the true score. On the default 200-task benchmark, with new tasks and new runs each time, Wilson's intervals covered it in 87% of these 30 reruns, and on average they are 13.4% wide from end to end. 30 reruns are few, so the count itself wobbles by several points either way; the chart below gives the coverage exactly.

What is "the truth"? It depends on what a rerun redraws. If each rerun draws new tasks from the same population, the truth is the population's score, and task sampling is part of the error. If it keeps the same tasks and only reruns the agent, the truth is those tasks' own score and the intervals here are conservative: they still allow for task sampling that is not happening. The first question is usually the one that matters, "how would it do on tasks like these?", and it is the one a benchmark score is quoted as answering.

Why "68.2% against 67.8%" is a tie. Two agents score 341 and 339 out of 500 tasks: 68.2% and 67.8%. Their Wilson intervals are 64.0% to 72.1% and 63.6% to 71.7%, almost the same interval. The two-proportion test gives $z$ = 0.136 and $p$ = 0.892. On 500 tasks the smallest difference a comparison of separately run agents detects with 80% power is 8.3 points; telling these two apart would take 213,489 tasks per agent. Leaderboards that rank by the first decimal are ranking noise.

Loading the chart…

The chart shows what the promise is worth, computed exactly (a sum over every possible score) rather than by simulation. The textbook Wald interval, $\hat p \pm z\sqrt{\hat p(1-\hat p)/n}$ , collapses near 0% and 100%: on 20 tasks its coverage falls as low as 33.2%. Near the edges is where agent scores often sit (a hard benchmark at release, an easy one near saturation). Wilson's interval stays near 95% (lowest 92.7% on 20 tasks), and the exact (Clopper–Pearson) interval never drops below 95% (lowest 95.9%), at the price of being wider. The bootstrap resamples the tasks and takes percentiles; for a single pass rate it behaves much like Wald, and its strength is in statistics with no formula.

Loading the calculator…

How many tasks you need follows from the width you can live with. For $\pm 3$ points at 95% near a 50% score, 1,068 tasks; for $\pm 1$ point, 9,604. To tell an agent at 65% from one at 60% (a 5-point difference) at 80% power: 1,471 tasks each if they run on separate tasks, 626 if they share them (chapter 5 explains why sharing helps).

Maths

With $k$ passes on $n$ tasks, $\hat p = k/n$ . Wald inverts the normal approximation around $\hat p$ ; its standard error is zero at $\hat p \in \{0, 1\}$ , which is why it fails there. Wilson (1927) inverts the score test instead: the interval is every $p$ for which $|\hat p - p| \le z\sqrt{p(1-p)/n}$ , and solving that quadratic gives the formula above. Clopper–Pearson (1934) inverts two exact binomial tests: its ends are beta quantiles, $B(\alpha/2;\,k,\,n-k+1)$ and $B(1-\alpha/2;\,k+1,\,n-k)$ , which the engine finds by bisection on the regularised incomplete beta function. The percentile bootstrap (Efron 1979) resamples the $n$ tasks with replacement $B$ times and takes the 2.5th and 97.5th percentiles of the resampled means.

The coverage of a closed-form interval at true rate $p$ is exactly $\sum_{k=0}^{n} \binom{n}{k}p^k(1-p)^{n-k}\,\mathbf{1}[p \in \text{CI}(k)]$ , a step function of $p$ (the jagged lines). For sample size, a margin $h$ needs $n = z^2 p(1-p)/h^2$ , and detecting a difference $\delta = p_1 - p_2$ with power $1-\beta$ between independently sampled agents needs, per agent (Fleiss),

$n = \frac{\big(z_{1-\alpha/2}\sqrt{2\bar p(1-\bar p)} + z_{1-\beta}\sqrt{p_1(1-p_1) + p_2(1-p_2)}\big)^2}{\delta^2}.$

Code

Wilson's interval, cut from the vendored engine (stats.ts):

const half = z * Math.sqrt((p * (1.0 - p)) / n + z2 / (4.0 * n * n));
return clip((mid - half) / den, (mid + half) / den);