pass@k and pass^k

Some one of k tries passes, or all k do: the unbiased estimators derived, and the two curves pulling apart as k grows. Why reliability, not best-of-k, is what an agent's user feels.

Loading the animation…

Concept

Agents are not deterministic: run one twice on the same task and it may pass once and fail once. So "how good is it?" splits into two questions with different answers.

pass@k: if the agent gets $k$ tries, does some one of them pass? This is the right question when something can pick the good attempt for you: hidden tests you are allowed to run, a verifier, a person reviewing $k$ drafts. It was popularised for code by Chen et al. (2021) with HumanEval.
pass^k ("pass hat k"): do all $k$ tries pass? This is what a user feels who runs the agent again and again and needs it to work every time. τ-bench introduced it for agents, and found agents that solved fewer than half of its tasks on one try were far less consistent over several: its retail pass^8 was under 25%.

At $k = 1$ they are the same number. Then they pull apart. In the animation (30 tasks, 10 samples each), pass@1 is 45.7%; by $k = 10$ , pass@10 has climbed to 80.0% and pass^10 has fallen to 16.7% (the generator's true values for these tasks: 77.3% and 10.4%). The same agent is "80% at ten tries" and "17% reliable over ten tries". Which one you report should depend on how the agent will be used, and a headline pass@k for an agent that has to work every time is the wrong number.

The tasks in the grid are sorted by how many of their ten samples pass. pass^k is carried by the tasks that always pass; pass@k by every task that ever passes. Raise the within-task correlation and runs on a task become more alike: pass@k grows more slowly (a task that fails tends to fail again) and pass^k decays more slowly.

Maths

Take $n$ samples on a task, of which $c$ pass, and pick $k$ of them at random without replacement. All $k$ fail with probability $\binom{n-c}{k}/\binom{n}{k}$ , and all $k$ pass with probability $\binom{c}{k}/\binom{n}{k}$ . If every sample passes independently with probability $\mu$ , then $c \sim \text{Binomial}(n, \mu)$ and the expectations over $c$ are exactly $(1-\mu)^k$ and $\mu^k$ : both estimators are unbiased for any $k \le n$ , and use all $n$ samples rather than only $k$ . Chen et al. compute the first as a product, which never forms the huge binomial coefficients:

$\frac{\binom{n-c}{k}}{\binom{n}{k}} = \prod_{i=n-c+1}^{n}\left(1 - \frac{k}{i}\right).$

The tempting shortcut, $1 - (1 - c/n)^k$ , is biased low: $x \mapsto (1-x)^k$ is convex, so by Jensen's inequality $\mathbb{E}[(1 - \hat\mu)^k] \ge (1-\mu)^k$ . The engine checks this on one task with $\mu$ = 0.3, $n$ = 10 and $k$ = 5 over 4,000 repeats: the unbiased estimator averages 0.8323 against a true 0.8319, the plug-in 0.7596. In the chart, the dashed plug-in curve sits under pass@k for every $k > 1$ .

When runs on a task are correlated, the per-task truth is a mixture. In the generator, every run copies the task's shared outcome $Z_t$ with probability $s = \sqrt\rho$ , so given $Z_t$ the runs are independent and

$\text{pass}^k_t = \mu_t\,\big(s + (1-s)\mu_t\big)^k + (1-\mu_t)\,\big((1-s)\mu_t\big)^k .$

Code

The unbiased pass@k, cut from the vendored engine (stats.ts):

if (n - c < k) return 1.0;
let prod = 1.0;
for (let i = n - c + 1; i <= n; i++) prod = prod * (1.0 - k / i);
return 1.0 - prod;