The harness is part of the result

One simulated model under sixteen harness configurations (tools, call format, turn budget, model), run through the engine's harness: the same model 30 points apart, measured with paired comparisons.

Loading the animation…

Concept

A leaderboard row names a model. The number beside it was produced by a model inside a harness: the tools it was given, the format its calls were written in, how many turns it was allowed, what happened when it wrote a broken call. Change the harness and the number changes, sometimes by more than the gap between models.

The animation measures that with the engine's own harness. The "model" is a seeded stochastic policy, not a language model, so that every difference has a known cause: it reads the file, then edits the buggy line, getting it right with the item-response probability of chapter 1 (attempts on one task are correlated, so a model that misreads a task tends to keep misreading it). With a shell, it runs the tests after each edit and tries again until they pass or the turns run out; without one, it stops after its edit. Its calls come out malformed now and then, more often as ReAct text (12%) than as native tool calls (2%), and the harness feeds the error back. Each configuration runs all 12 tasks 8 times, 96 runs in all.

The same model, 30.2 points apart. The smaller model resolves 43.8% of the runs in its worst scaffold and 74.0% in its best (read and edit only, against a shell, native calls and 16 turns). The larger model moves by 22.9 points. Giving the smaller model a shell and 16 turns puts it at 74.0%, ahead of the larger model without a shell, at 64.6%.

Is that a real difference? Because every configuration sees the same draws for the same task and run (common random numbers), the runs pair up, and chapter 5's tools apply. Shell and 16 turns against read-and-edit, for the smaller model: 28 runs resolved only with the shell, 0 only without, McNemar $p$ below one in a million. The smaller model with a shell against the larger without: 16 against 7, $p$ = 0.093, a paired 95% interval of -0.3% to 19.0%. Suggestive, not settled: 96 runs on 12 tasks are not enough, which is chapter 4's lesson again.

Turns are a knob. For the larger model with a shell, 16 turns instead of 4 resolve 17 more runs and lose none. A paper that reports one budget and a paper that reports another are not measuring the same thing.

The format costs turns, not (here) success. ReAct text breaks more often (0.78 malformed calls a run against 0.14 at 8 turns), but the harness's error feedback lets the model try again, so the resolved rates differ by one run of 96. The breakage shows up as model calls and dollars instead; a harness that stopped at the first malformed call would turn it into failures.

So report the harness with the score: tools, call format, turn and token budgets, retries, and how malformed output is handled. Comparing two models fairly means running both in the same harness, and preferably more than one.

Maths

Let $\mu$ be the chance one attempt at a task is right and $s = \sqrt{\rho}$ . Each attempt copies the run's shared outcome $Z \sim \text{Bernoulli}(\mu)$ with probability $s$ and otherwise draws its own, so two attempts correlate at exactly $\rho$ (chapter 1's construction). Given $Z = 1$ an attempt is right with probability $s + (1 - s)\mu$ ; given $Z = 0$ , with probability $(1 - s)\mu$ ; attempts are independent given $Z$ . Summing over $Z$ gives the equation above for $k$ attempts. With $\rho = 0$ it is $1 - (1 - \mu)^k$ , which reaches 1; with $\rho > 0$ the runs where the model has misread the task ( $Z = 0$ ) improve only at rate $(1 - s)\mu$ , so more turns help less and less. That is the flattening in the bars as the budget goes from 4 to 8 to 16.

With a shell, a budget of $B$ turns allows about $(B - 2)/2$ attempts: one turn to read, then an edit and a test per attempt, and one to answer. Without a shell, $k = 1$ whatever the budget.

Code

How the simulated model decides each edit, cut from the vendored engine (scaffold.ts):

const c = this.rEdit.random();
const own = this.rEdit.random();
const pick = this.rEdit.random();
const ok = c < this.s ? this.z : own < this.mu ? 1 : 0;