Cost-aware evaluation

Success against dollars per task: the Pareto front of the harness configurations, dollars per solved task, why more retries cost more per success when attempts are correlated, and why the front is itself an estimate.

Loading the animation…

Concept

A success rate without a cost is half a result. An agent that resolves a few more tasks by running ten times as long, or by calling a model that costs five times as much per token, may be the wrong choice; and two agents that score the same may differ several-fold in what each solved task costs. A cost-aware evaluation puts every configuration on a plane, cost per task against success rate, and asks which ones are worth considering at all.

The animation does that for chapter 8's sixteen configurations: two models (priced at 1 and 5 dollars per million input and output tokens, and at 3 and 15), with or without a shell, native calls or ReAct text, 4 to 16 turns. Every cost is the harness's own accounting of the tokens each run used. The points arrive cheapest first, and the line is the Pareto front: the configurations that nothing else beats on both cost and success. 11 of the sixteen are on it.

Off the front. The larger model without a shell resolves 64.6% of runs at $0.0052 a task. The smaller model with a shell, ReAct calls and 8 turns resolves 64.6% for $0.0049: as many, for less. No budget makes the first a sensible choice.

Cheapest per solved task is not the answer either. The smaller model with no shell has the lowest cost per solved task, $0.0039, because it gives up after one edit; it also resolves only 44.8%. The highest success, 86.5%, costs $0.0181 per solved task. Which point to pick depends on what an unsolved task costs you: a person's time to do it by hand, a customer who leaves. Put that price on the chart and the choice follows.

More turns cost more per solved task here. For the smaller model with a shell, going from 8 to 16 turns lifts the rate from 65.6% to 74.0%, and the cost per solved task from $0.0080 to $0.0133. The runs that are still failing at 8 turns are mostly the ones where the model has misread the task, and they spend the extra turns failing (chapter 8's maths), on prompts that grow with every turn.

The front is an estimate. Each point has an interval (the vertical lines), and on 96 runs neighbouring points overlap: some configurations are on the front because of the draw. Pareto fronts from small benchmarks should be read with the intervals, not as a ranking.

Maths

Dollars per solved task. With mean cost $\bar c$ per task and success rate $\hat p$ , the cost per solved task is $\bar c / \hat p$ : total spend over tasks solved.

Retrying until the tests pass. Suppose each attempt costs $c$ and succeeds independently with probability $p$ , and a run tries up to $k$ times. A run costs $c\,(1 - (1 - p)^k)/p$ on average and succeeds with probability $1 - (1 - p)^k$ , so the cost per solved task is exactly $c/p$ , whatever $k$ is: with independent attempts, more retries buy more successes at a constant price. Correlation breaks this. When attempts on a task are correlated (chapter 8, $\rho > 0$ ), the runs left unsolved are the hard-stuck ones, extra attempts on them rarely succeed, and the cost per solved task rises with $k$ , as it does in the simulation. Growing prompts (every turn resends the conversation) push the same way.

The front. A configuration $i$ is dominated when some $j$ has $c_j \le c_i$ and $p_j \ge p_i$ , with one inequality strict; the front is everything not dominated. The engine checks its front against that definition on random point sets. Sorting by cost, the front is the sequence of points each of which succeeds more than every cheaper point.

Code

The Pareto front, cut from the vendored engine (stats.ts):

for (let j = 0; j < points.length; j++) {
  const [c2, s2] = points[j] as [number, number];
  if (j !== i && c2 <= c && s2 >= s && (c2 < c || s2 > s)) {
    beaten = true;
    break;
  }
}