Is the agent good, and how sure are you?

What you'll learn

An agent's score is a measurement: a sample of tasks, a few noisy runs on each, and a grader that can be fooled. This site takes the measurement apart. The benchmarks agents are quoted on, from their papers; pass@k against pass^k, best of k against reliable every time; intervals and which of them keep their promise; why two scores a few tenths apart are a tie; how comparing two agents on the same tasks gives a sharper answer for free; then the grader itself (LLM judges and their biases, grading the whole trajectory), the harness the score was measured in, leaked tasks, and what each point of success costs.

The animations are driven by Agent_Loop_Sim's evals module, identically in Python and in your browser: a synthetic benchmark under an item-response model, and the standard estimators, each tested against scipy and statsmodels. On a 500-task benchmark, two agents run on separate tasks must differ by about 8.3 points before a test at 80% power reliably tells them apart; and the same simulated model scores from 44% to 74% depending only on its harness. No language model runs anywhere.