Skills Benchmark Report 003: The Referee Exam Was Ranking Judges Backwards

This report is about the referee, and the exam that decides which referee is allowed to work. I am building a machine that judges whether an AI marketing skill actually improves the writing it produces. That machine needs a judge model, and before any judge model grades a single post it has to pass a qualification exam. The old exam tested consistency. I showed a judge the same pair of posts several times, changing only details that should not matter, such as the order of the scoring criteria or the labels on the posts, and a judge that kept changing its answer was refused the seat. This cycle proved that exam was rewarding the wrong property, and it has been retired and replaced.

The old exam tested consistency. I showed a judge the same pair of posts several times, changing only details that should not matter, such as the order of the scoring criteria or the labels on the posts. A judge that kept changing its answer was refused the seat.

Consistency is not correctness. I proved that with two questions where I already knew the right answer.

The first question is a trick. I showed the judge the same post twice, the identical words in both slots. There is one honest answer, and it is a tie.

The second question is easy. I showed it a post written for the brief against a post written for a completely different brief. A person picks the right one instantly.

The judge that scored highest on the old exam answered the trick question perfectly, 72 times out of 72. Then it got the easy question right about half the time. Half is what a coin does.

The judge that scored lowest never once said tie. It named whichever post came first, on all 72 identical pairs. On the easy question it did better than the top-scoring judge.

So the exam was ranking judges backwards. It rewarded a judge that says the same thing about everything, and it punished a judge that at least notices a difference. Neither number touched the only question that matters, which is whether a referee can tell good work from bad.

The control ran under a design written and committed before the first call, which fixes the selection rule, both thresholds, the seat list, and a prediction that could turn out wrong. Two conditions were built entirely from records already committed to the repository, so nothing new was generated for this. The null condition pairs a post with itself, the identical bytes in both slots, where the only honest answer is a tie. The wide gap pairs a post written for the brief against a post written for a completely different brief, where the right answer is known in advance. Three judge seats, eighteen pairings per condition, four prompt variants each, for 432 blind calls in total.

Two conditions, both built from records already committed to the repository. Nothing new was written for this. The null condition pairs a post with itself, identical bytes. The wide gap pairs a post written for the brief against one written for a different brief.

Eighteen pairings per condition, drawn without replacement under a pinned seed, sorted before drawing so the result does not depend on the order files appear on disk. Three judge seats. Four prompt variants each. That is 432 blind calls, every one executed on a loopback-only local runtime with the model identities pinned before the first call.

Zero unreadable responses and zero determinism failures. In the null condition two of the four variants assemble a byte-identical prompt, because swapping two identical posts changes nothing. I used that as a determinism probe at temperature zero, and the runtime agreed with itself every time.

My prediction, recorded so it could fail: the null condition would not produce ties, and the wide gap would separate cleanly. Both halves were wrong. That is what a prediction is for.

Every number below traces to a committed artifact, and all 432 calls returned readable responses. The results split the three seats into two failure modes that the retired exam could not tell apart. Both gemma4 seats returned a tie on all 72 identical pairs, which means they carry no preference at all for whichever post appeared first. Those same seats then answered the known-answer question at roughly the rate a coin would. The mistral-nemo seat never returned a tie, naming the first post on all 72 identical pairs, and it still scored highest of the three on the known-answer question. The exam had ranked these three in exactly the reverse order.

Both gemma4 seats are perfectly honest and completely blind. They have no preference for whichever post came first, which is remarkable, because eight earlier audits looked like they were measuring exactly that. Then they perform at chance on the widest difference this corpus can produce. A stable instrument that measures nothing is still an instrument that measures nothing.

The mistral-nemo seat is the mirror image. It never ties. Shown two identical posts it named the first one every single time, which is position dependence in its purest recordable form. Show it the correct post in the second slot and it finds the correct post 22.2 percent of the time.

Sixty-one percent is the best of the three, on a question a person answers instantly. All three are close to useless.

Three failures are worth recording from this cycle, and the first one is mine. I shipped a scoring metric that was confounded by the design of the prompt variants, published a number based on it, and then had to disclose the error in a new section rather than quietly correct the old one. A completed run of 144 judge calls was then lost at its final write because two separate lists of approved directories disagreed with each other. And an earlier broken measurement turned out to have inverted a finding this project had already written into four documents. Each failure produced a control, and each control was proven against the exact defect before it was trusted.

I shipped a confounded metric and then had to disclose it

The four prompt variants are not balanced across presentation order. Three of them show the correct post first and one shows it second. My first pass counted correct answers across all four without separating them.

A judge that always picks whichever post came first therefore scores 75 percent on that metric, not 50. I wrote 50 in the commit message that published the run. That is arithmetic error, and it inflated the one seat with a position preference from a real 61.1 percent to an apparent 80.6.

The record is append-only, so the correction lives in a new section of the pre-registration rather than replacing the text that was wrong. Nothing above it was edited.

The lesson is now a rule. Before registering a metric, compute what it returns for a judge that is not judging: one that always ties, one that always picks the first item, and one that answers at random. Register those numbers as reference points. The 75 percent floor would have been obvious on paper.

A completed run died at its last line and lost everything

The first attempt finished 144 calls on the first seat, roughly 35 minutes of work, then hit its write and stopped with a filesystem denial.

Two separate lists of approved record directories exist in this repository. One is used by the wrapper that creates them. The other is used by the guard that permits writes into them. Nothing checked that the two agreed, and the new directory was in one and not the other.

Chasing that surfaced a larger one. The tournament lane's directories had the same gap and had never been added to the guard, so that lane could never have written a record either. Both are fixed, and a check now fails the build if the two lists ever disagree again.

The verification gap is the part worth keeping. My dry run proved the selection and the prompt assembly and deliberately wrote nothing, so it never exercised the write. I verified the half of the path I had built and not the half I had assumed.

A defective measurement inverted a published finding

An earlier seat recorded zero percent because the model degenerated into a repetition loop and produced unreadable output. Next to its reasoning-mode sibling at 22.2 percent, that read as strong evidence that reasoning mode helps.

Re-run with the instrument fixed, the same model scores 38.9 percent in control mode against 22.2 in reasoning mode. Control is roughly double, which is the opposite conclusion. The claim had already reached four documents and each had to be corrected by hand.

A broken measurement does active harm. It pulls every comparison it sits in toward a false answer, and states that answer with the same confidence as a real one, because nothing in the record separates the two.

The consistency exam is retired and a two-part exam replaces it, with both parts required. A seat must be honest, meaning it returns a tie on at least 90 percent of calls where the two posts are identical. A seat must also discriminate, meaning it reaches at least 80 percent accuracy on questions whose answer is already known, with presentation order weighted equally so that always picking the first post earns nothing. Both thresholds are fixed and will not move now that data exists under them. The reference points for a judge that is not judging were computed before the run and are asserted in the test suite, which is the property the retired exam lacked.

Honest means it does not invent a difference that is not there. A seat must return a tie on at least 90 percent of calls when both posts are identical.

Discriminating means it finds a difference that is there. A seat must reach at least 80 percent accuracy on the answer key, with presentation order weighted equally.

Eighty and not ninety, and the reason is on the record. Chance is 50 percent. The answer-key question is trivial for a person. A referee that cannot reach 80 percent there cannot rank two competitors writing to the same brief, which is strictly harder. Ninety was the old exam's number and importing it would have been superstition.

Both thresholds are fixed and will not move now that data exists. Every seat measured to date fails the second one, at 51.9, 56.5, and 61.1 percent.

The reference points are computed and asserted in the test suite rather than claimed in prose. A judge that always ties scores zero on accuracy. A judge that always picks the first post scores exactly 50, and so does one that always picks the second. No degenerate strategy reaches either bar, which is the property the old exam lacked.

Ten seats are now registered, largest first. Two of them are the biggest models on the machine and neither has ever judged a single pairing. They were excluded by a rule protecting against a judge favouring its own family's writing. In the answer key both posts come from the same generator, so there is no rival family to favour, and the rule does not reach this test. It still protects grading, unchanged.

Ten judge seats are queued against the new exam and the run is executing now, largest models first. Results publish here whichever way they come out, including the outcome where every seat fails, because a refusal to grade carries the same prominence in this benchmark as a grade. Two of those seats are the largest models available on the machine and neither has ever judged a single pairing, so any claim that local models cannot do this job was premature. If they all fail, two honest paths remain: a frontier judge behind an execution lane that does not exist yet, or a rubric built from each skill's own published claims that needs no judge model at all.

If they all fail, two paths remain and both are honest. The first is a frontier judge, which this repository cannot call today because the live lane does not exist yet and would have to be built and approved. Published work on panels of small judges and on order-swapped pairwise evaluation reports the same position bias in larger models, so I would test rather than assume.

The second path needs no judge model at all. Each skill in this benchmark already ships a set of claims quoted verbatim from its own instructions, turned into mechanical checks. A skill that promises a diagram in every post can be tested for a diagram in every post. Models fine-tuned for evaluation, such as Prometheus-2, are a third option I have not tried and would pre-flight before registering.

No judge is seated. No skill here is called best. The grades from the first graded cycle stand exactly as recorded, alongside the audit that says they do not measure writing quality.