Graders

Unit tests, LLM judges and people: a simulated judge with position, length and self-preference biases, the swap test and what it cannot fix, Cohen's κ, and correcting a lenient pass/fail judge with a labelled calibration set.

Loading the animation…

Concept

Every score in the earlier chapters came from a grader, and so far the grader was perfect: hidden unit tests that say exactly whether the patch works. Agents often do work no test can check (a summary, a plan, a conversation with a customer), and then the grader is a person or, more and more often, another model: an LLM judge. A judge is cheap and fast, but it is an instrument with its own errors, and its errors go straight into the score.

Zheng et al. measured three of those errors in GPT-4 as a judge. Position bias: it prefers whichever answer it reads first (or second). Verbosity bias: it prefers the longer answer. Self-preference: it prefers answers written by its own model family. The animation simulates a judge that has a chosen mix of the three, on 200 pairs of responses whose true quality is known (and called a tie when the two are within 0.3 of each other, as a careful human might).

The swap test. Show every pair twice, once in each order. A fair judge's two verdicts disagree only through its own noise: here in 11.5% of pairs. Give it a position bias of 0.8 and the share of verdicts that go to whichever answer came first rises from 49.8% to 67.8%, and the two orders disagree on 35.5% of pairs. The standard fix keeps a verdict only when both orders agree and calls the rest a tie. That removes the position effect exactly: of the verdicts it keeps, 95.3% are right, against 76.0% for one order. It cannot put back what the bias destroyed, though: it keeps only 64.5% of the pairs, and Cohen's κ against the truth stays at 0.55 (one order: 0.56; the fair judge: 0.72).

What swapping cannot fix. Length and family do not move when the order does. With a length bias of 0.6, the judge gets 81.4% of the 86 pairs where the longer answer is the worse one right (the fair judge: 96.5%), and both orders agree on 86.5% of pairs, so the swap test does not even see the bias. With self-preference, it gets 67.6% of the pairs where its own family wrote the worse answer right. These need other fixes: length-controlled comparisons, a judge from another family, and above all a check against people.

κ, not accuracy. Two graders that both say "A" most of the time agree a lot by chance. Cohen's κ subtracts the agreement their label frequencies would give by chance ( $p_e$ ) from the agreement observed ( $p_o$ ) and rescales: 1 is perfect, 0 is chance. It is the number to quote when you validate a judge against human labels.

Loading the chart…

Pass/fail judges and the correction. The second chart is the most common use: a judge reads each transcript and says pass or fail. If it passes a true pass with probability Se (sensitivity, 0.95 here) and a true fail with probability 1 − Sp (Sp, specificity, 0.7), it reports $\pi\,\text{Se} + (1-\pi)(1-\text{Sp})$ for an agent whose true rate is $\pi$ : a lenient judge flatters weak agents most. On 300 tasks, an agent the unit tests put at 40.7% gets 58.3% from the judge (κ = 0.62). Label 200 tasks both ways, estimate Se and Sp from them, and invert the formula: the corrected estimate is 41.6%, 95% interval 30.9% to 52.3%. The interval is wider than a perfect grader's would be, because the calibration set's own error is in it.

Maths

κ. For a table of counts $n_{ij}$ (truth $i$ , judge $j$ ) with total $n$ : $p_o = \sum_i n_{ii}/n$ and $p_e = \sum_i (n_{i\cdot}/n)(n_{\cdot i}/n)$ , the agreement expected if the two labelled independently with their own frequencies. Then $\kappa = (p_o - p_e)/(1 - p_e)$ (Cohen 1960). The engine checks it against statsmodels' cohens_kappa.

The swap test removes a position term exactly. In the model above, swapping the order flips the sign of every difference except $\beta_{\text{pos}}$ , so $P_{AB}(\text{A}) - P_{BA}(\text{A})$ is zero for a fair judge and grows with $\beta_{\text{pos}}$ : averaged over pairs, the share of verdicts won by the first-shown answer is $\tfrac12\big[P_{AB}(\text{first}) + P_{BA}(\text{first})\big]$ , which the tests compare with the simulation. The length and family terms are the same in both orders, so the swap cannot see them.

Correcting a pass/fail judge (Rogan and Gladen 1978). From $p_{\text{obs}} = \pi\,\text{Se} + (1 - \pi)(1 - \text{Sp})$ ,

$\hat\pi = \frac{p_{\text{obs}} + \widehat{\text{Sp}} - 1}{\widehat{\text{Se}} + \widehat{\text{Sp}} - 1},$

and when Se and Sp are estimated on a calibration set the delta method gives

$\operatorname{Var}(\hat\pi) \approx \frac{\operatorname{Var}(p_{\text{obs}}) + \hat\pi^2 \operatorname{Var}(\widehat{\text{Se}}) + (1 - \hat\pi)^2 \operatorname{Var}(\widehat{\text{Sp}})}{(\widehat{\text{Se}} + \widehat{\text{Sp}} - 1)^2}.$

Here the standard error is 0.0543, against 0.0412 if Se and Sp were known exactly. When $\text{Se} + \text{Sp} - 1$ is small (a judge little better than a coin), the correction blows up: no amount of arithmetic recovers a grader that does not discriminate.

Code

Cohen's κ, cut from the vendored engine (stats.ts):

const po = diag / n;
let pe = 0.0;
for (let i = 0; i < k; i++) {
  let row = 0.0;
  let col = 0.0;
  for (let j = 0; j < k; j++) {
    row = row + table[i]![j]!;
    col = col + table[j]![i]!;
  }
  pe = pe + (row / n) * (col / n);
}
const kappa = pe < 1.0 ? (po - pe) / (1.0 - pe) : 1.0;

The swap rule (judge.ts):

const v1 = u1 < pAb ? "A" : "B";
const v2 = u2 < pBa ? "B" : "A";