Comparing two agents

Two agents on the same tasks: the paired difference and McNemar's test against the unpaired comparison, the correlation that pairing exploits, and how seeds and repeated runs add or remove variance.

Loading the animation…

Concept

Most evals end in a comparison: is the new agent better than the old one? The cheapest way to get a sharper answer is not more tasks but a better design: run both agents on the same tasks and compare them task by task.

The reason is that much of the noise in a score is which tasks you drew (chapter 1). When both agents face the same tasks, that noise is shared: an easy draw flatters both, a hard one hurts both, and in the difference it cancels. In the animation, agent A (skill 0.45) and agent B (skill 0.1) run on the same 300 tasks, one run each. A's per-task scores and B's correlate at 0.39. After all 300 tasks, A leads by 7.3 points (the true gap on these tasks is 7.2 points). The paired 95% interval is 1.2% to 13.5% and excludes zero; the unpaired one, the interval you would report if the two had been run on separate task sets, is -0.6% to 15.2% and does not. Same data, a standard error 0.78 times the unpaired one, a different conclusion.

McNemar's test sees the same thing from the tiles. A task both agents pass, or both fail, says nothing about which is better; only the discordant tasks do, where exactly one passes. With one run each, A alone passed 56 tasks and B alone 34. If the agents were equally good, each discordant task would be a coin flip, and a split this uneven or more comes up with probability 0.0263.

Seeds and temperature add a second source of noise: the same agent on the same task does not always do the same thing. The bars under the chart split the variance of the difference into the part from which tasks were drawn and the part from the runs. Unpaired, the tasks account for 32% of that variance; paired, for 0.4%. Pairing removes almost all of the first part; more runs per task shrink the second. Switch to five runs per task and the paired standard error falls from 3.1 points to 2.5 points. Each run costs as much as a new task, so whether to buy runs or tasks depends on which part dominates.

Maths

Write $a_t$ and $b_t$ for the agents' scores on task $t$ (a pass, or a mean over runs) and $d_t = a_t - b_t$ . The difference in mean score is $\hat\Delta = \bar d$ , with standard error $s_d/\sqrt{T}$ . Expanding $s_d^2 = s_A^2 + s_B^2 - 2\,\widehat{\operatorname{cov}}(A, B)$ shows what pairing buys: the covariance, which the unpaired standard error $\sqrt{(s_A^2 + s_B^2)/T}$ ignores. The engine checks both against scipy's paired and Welch $t$ statistics.

For pass/fail outcomes the paired table has four cells; under "no difference" each of the $n_{10} + n_{01}$ discordant tasks goes either way with probability $\tfrac12$ , so the exact McNemar $p$ -value is a two-sided binomial test, $2\,P(X \le \min(n_{10}, n_{01}))$ with $X \sim \text{Binomial}(n_{10}+n_{01}, \tfrac12)$ , and the continuity-corrected $\chi^2$ above approximates it. Planning a paired study needs the discordance $\psi = p_{10} + p_{01}$ and the difference $\delta = p_{10} - p_{01}$ (Connor 1987):

$n = \frac{\big(z_{1-\alpha/2}\sqrt{\psi} + z_{1-\beta}\sqrt{\psi - \delta^2}\big)^2}{\delta^2}.$

With $m$ runs per task and within-task correlation $\rho$ , one task's mean has variance $\mu(1-\mu)\,\frac{1 + (m-1)\rho}{m}$ : more runs help only as far as $\rho$ allows, and not at all when $\rho = 1$ .

Code

The paired and unpaired standard errors, cut from the vendored engine (stats.ts):

const seP = Math.sqrt(vd / n);
const seU = Math.sqrt(va / n + vb / n);
const cov = (va + vb - vd) / 2.0;