Everything the chapters quote, computed by the vendored engine as this page was built; the engine records the same run in evals_results.md and evals2_results.md. Every task, skill, bias, leak and correlation is synthetic; token counts and costs are the harness's own accounting.
Chapter 1: runs accumulating
| after | runs | score | Wilson 95% | tasks as unit 95% |
|---|
| run 1 of every task | 40 | 67.5% | 52.0% to 79.9% | 52.8% to 82.2% |
| run 2 of every task | 80 | 67.5% | 56.6% to 76.8% | 54.1% to 80.9% |
| run 3 of every task | 120 | 67.5% | 58.7% to 75.2% | 54.2% to 80.8% |
Chapter 2: one SWE-bench-style task
| patch | hidden tests | FAIL_TO_PASS | PASS_TO_PASS | verdict |
|---|
| fix | thorough | pass | pass | resolved |
| fix | weak | pass | pass | resolved |
| overfit | thorough | fail | pass | not resolved |
| overfit | weak | pass | pass | resolved |
| collateral | thorough | pass | fail | not resolved |
| collateral | weak | pass | fail | not resolved |
Chapter 3: pass@k and pass^k
| k | pass@k | true | plug-in | pass^k | true |
|---|
| 1 | 45.7% | 44.4% | 45.7% | 45.7% | 44.4% |
| 2 | 56.8% | 56.0% | 55.7% | 34.5% | 32.7% |
| 3 | 62.8% | 61.7% | 61.0% | 29.3% | 26.8% |
| 4 | 67.0% | 65.6% | 64.5% | 25.8% | 22.8% |
| 5 | 70.5% | 68.6% | 67.2% | 23.1% | 19.6% |
| 6 | 73.3% | 71.0% | 69.2% | 21.0% | 17.1% |
| 7 | 75.7% | 73.0% | 70.9% | 19.4% | 15.0% |
| 8 | 77.6% | 74.6% | 72.3% | 18.2% | 13.2% |
| 9 | 79.0% | 76.1% | 73.4% | 17.3% | 11.7% |
| 10 | 80.0% | 77.3% | 74.3% | 16.7% | 10.4% |
Chapter 4: coverage over 30 reruns
| tasks | reruns draw | truth | Wald | Wilson | exact | bootstrap |
|---|
| 50 | new tasks and runs | 58.1% | 93% | 93% | 97% | 97% |
| 50 | new runs only | 55.6% | 97% | 97% | 100% | 93% |
| 200 | new tasks and runs | 58.1% | 87% | 87% | 90% | 87% |
| 200 | new runs only | 57.1% | 97% | 97% | 97% | 97% |
| 500 | new tasks and runs | 58.1% | 93% | 93% | 93% | 93% |
| 500 | new runs only | 57.9% | 100% | 100% | 100% | 100% |
Exact coverage of a nominal 95% interval, lowest over p:
| n | Wald | Wilson | exact |
|---|
| 20 | 33.2% | 92.7% | 95.9% |
| 100 | 86.6% | 93.6% | 95.6% |
The tie: 68.2% against 67.8% on 500 tasks, p = 0.892; tasks per agent to detect it at 80% power: 213,489.
| difference | tasks per agent, unpaired | shared tasks, paired |
|---|
| 10.0 points | 356 | 155 |
| 5.0 points | 1,471 | 626 |
| 3.0 points | 4,129 | 1,742 |
| 1.0 points | 37,513 | 15,696 |
Chapter 5: two agents on the same tasks
| runs per task | A − B | paired 95% | unpaired 95% | McNemar p |
|---|
| 1 | 7.3 points | 1.2% to 13.5% | -0.6% to 15.2% | 0.0263 |
| 3 | 4.6 points | -0.9% to 10.0% | -2.4% to 11.5% | 0.9219 |
| 5 | 5.4 points | 0.4% to 10.4% | -1.4% to 12.2% | 0.2513 |
True difference on these tasks: 7.2 points.
Chapter 6: a simulated judge
| biases | accuracy, one order | κ, one order | first shown wins | orders agree | κ, swapped | right where decided |
|---|
| fair | 84.5% | 0.72 | 49.8% | 88.5% | 0.77 | 92.1% |
| position | 76.0% | 0.56 | 67.8% | 64.5% | 0.55 | 95.3% |
| verbosity | 80.0% | 0.64 | 48.3% | 86.5% | 0.64 | 86.1% |
| self | 80.5% | 0.65 | 49.8% | 88.5% | 0.67 | 86.4% |
| all | 76.0% | 0.56 | 64.5% | 71.0% | 0.49 | 86.6% |
Pass/fail grading: unit tests 40.7%, the judge 58.3%, corrected 41.6% (95% 30.9% to 52.3%).
Chapter 7: grading trajectories
| trajectory | outcome | self-report | trajectory | wasted | unsafe run / stopped | score |
|---|
| Scripted: fix, test, done | pass | pass | pass | 0 | 0 / 0 | 1.00 |
| Scripted: one malformed call, then the fix | pass | pass | pass | 1 | 0 / 0 | 0.90 |
| Scripted: the fix on a flaky shell | pass | pass | pass | 0 | 0 / 0 | 1.00 |
| Scripted: the fix, then rm -rf build | pass | pass | fail | 0 | 1 / 0 | 0.50 |
| Scripted: the fix, then installs and deletes, in a sandbox | pass | pass | fail | 0 | 1 / 2 | 0.50 |
| Recorded (Qwen2.5-1.5B): native tool calls | fail | fail | fail | 0 | 0 / 0 | 0.80 |
| Recorded (Qwen2.5-1.5B): ReAct text | fail | pass | fail | 2 | 0 / 0 | 0.60 |
| Scripted: search, calculate, answer | pass | pass | pass | 0 | 0 / 0 | 1.00 |
| Recorded (Qwen2.5-1.5B): search and answer | fail | pass | fail | 0 | 0 / 0 | 0.80 |
Chapter 8: one model, many harnesses
| model | tools | calls | turns | resolved | Wilson 95% | model calls | $ a run |
|---|
| small | read/edit | native | 4 | 44.8% | 35.2% to 54.7% | 3.07 | $0.0017 |
| small | read/edit | react | 4 | 43.8% | 34.3% to 53.7% | 3.26 | $0.0018 |
| small | with a shell | native | 4 | 53.1% | 43.2% to 62.8% | 4.00 | $0.0029 |
| small | with a shell | native | 8 | 65.6% | 55.7% to 74.4% | 6.11 | $0.0052 |
| small | with a shell | native | 16 | 74.0% | 64.4% to 81.7% | 8.68 | $0.0098 |
| small | with a shell | react | 4 | 46.9% | 37.2% to 56.8% | 4.00 | $0.0026 |
| small | with a shell | react | 8 | 64.6% | 54.6% to 73.4% | 6.31 | $0.0049 |
| small | with a shell | react | 16 | 72.9% | 63.3% to 80.8% | 9.07 | $0.0093 |
| large | read/edit | native | 4 | 64.6% | 54.6% to 73.4% | 3.07 | $0.0052 |
| large | read/edit | react | 4 | 63.5% | 53.6% to 72.5% | 3.26 | $0.0054 |
| large | with a shell | native | 4 | 68.8% | 58.9% to 77.1% | 4.00 | $0.0084 |
| large | with a shell | native | 8 | 79.2% | 70.0% to 86.1% | 5.43 | $0.0133 |
| large | with a shell | native | 16 | 86.5% | 78.2% to 91.9% | 7.03 | $0.0156 |
| large | with a shell | react | 4 | 65.6% | 55.7% to 74.4% | 4.00 | $0.0077 |
| large | with a shell | react | 8 | 77.1% | 67.7% to 84.4% | 5.67 | $0.0126 |
| large | with a shell | react | 16 | 84.4% | 75.8% to 90.3% | 7.48 | $0.0160 |
Chapter 9: contamination
| leak rate | leaked | flagged (verbatim / paraphrase) | score | clean | flagged dropped | before − after cutoff |
|---|
| 0% | 0 | 0 of 0 / 0 of 0 | 48.3% | 48.3% | 48.3% | 3.6 points (p = 0.470) |
| 10% | 23 | 15 of 15 / 0 of 8 | 50.0% | 48.3% | 49.4% | 7.0 points (p = 0.161) |
| 30% | 62 | 41 of 41 / 0 of 21 | 53.8% | 48.3% | 50.7% | 14.3 points (p = 0.004) |
| 60% | 131 | 80 of 80 / 0 of 51 | 60.8% | 48.3% | 55.3% | 27.9 points (p = 0.000) |
Chapter 10: cost
| configuration | resolved | $ a task | $ per solved | front |
|---|
| small · edit · native · 4 | 44.8% | $0.0017 | $0.0039 | yes |
| small · edit · react · 4 | 43.8% | $0.0018 | $0.0041 | |
| small · test · native · 4 | 53.1% | $0.0029 | $0.0054 | yes |
| small · test · native · 8 | 65.6% | $0.0052 | $0.0080 | yes |
| small · test · native · 16 | 74.0% | $0.0098 | $0.0133 | yes |
| small · test · react · 4 | 46.9% | $0.0026 | $0.0055 | yes |
| small · test · react · 8 | 64.6% | $0.0049 | $0.0076 | yes |
| small · test · react · 16 | 72.9% | $0.0093 | $0.0128 | yes |
| large · edit · native · 4 | 64.6% | $0.0052 | $0.0080 | |
| large · edit · react · 4 | 63.5% | $0.0054 | $0.0085 | |
| large · test · native · 4 | 68.8% | $0.0084 | $0.0123 | yes |
| large · test · native · 8 | 79.2% | $0.0133 | $0.0168 | yes |
| large · test · native · 16 | 86.5% | $0.0156 | $0.0181 | yes |
| large · test · react · 4 | 65.6% | $0.0077 | $0.0118 | |
| large · test · react · 8 | 77.1% | $0.0126 | $0.0163 | yes |
| large · test · react · 16 | 84.4% | $0.0160 | $0.0189 | |