Skills Benchmark Report 001: The Audit That Refused Both Judges

On 27 August 2026 the hendry.ai AI Marketing Skills Benchmark ran its first evaluator cycle under protocol 0.1-L1, the local lane ratified that morning as amendment D8. The generator was qwen2.5:7b-instruct-q4_K_M on a loopback-only local runtime at 127.0.0.1, with sampling parameters pinned and the runtime’s model identity recorded before the first call. The matrix covered nine of my own skills, none previously benchmarked, plus the two baseline arms, bare model and placebo. That is 11 conditions across three fixed briefs with three draws each: 99 generations, every one an append-only governed record carrying its token counts and the runtime-returned model string. The judgment count is zero. Neither pre-registered judge model passed the qualification audit that gates the judging stage, so the cycle stopped itself before producing a single verdict. Every figure in this report traces to a committed run artifact, and where a metric was not measured, this report says NOT MEASURED.

The runtime is Ollama, serving on the loopback interface only; the whole cycle runs with the external network denied by a tripwire that kills the process on any other socket. The generator family is Qwen2.5, pinned with explicit quantization. I build and run the benchmark under my own name, Hendry Soong; this report lives on the skills track, and the benchmark’s records are committed so anyone can re-derive every claim. One comparability rule binds every number that will ever come out of this lane: a 0.1-L1 win rate measures lift over a bare local 7B model, a weaker bar than the July pilot’s frontier baseline, so 0.1-L1 numbers and grades live in their own lineage and are never quoted beside the pilot’s.

The cycle measured generation completely and judgment none at all. All 99 planned generations exist as governed records under runs/0.1-L1, each carrying prompt and output token counts, the pinned generator string, and a scope flag separating the five on-category LinkedIn-post skills from the four comment and email skills that run off-category on these briefs. The two numbers this cycle actually produced are judge stabilities: llama3.1:8b-instruct held the same verdict across three prompt variants on 13 of 18 audit pairings, 72.2 percent, and gemma2:9b-instruct held on 15 of 18, 83.3 percent. The pre-registered qualification bar is 90 percent. Win rates against the bare model, win rates against the placebo, consistency, and grades are all NOT MEASURED: each requires a seated judge, and no seat qualified.

The grade bands themselves carried verbatim from the original protocol and were fixed roughly 32 hours before the pilot’s first datum in July. Nothing in this cycle touched them. When grades exist they will ship with seeded-bootstrap 95 percent confidence intervals, wide at this sample size because the sample is small, with each skill’s gate-decided share stated beside them.

A benchmark a human cannot read is a scoreboard, so here is the work itself: first the exact task every arm received, then each arm’s answer, quoted verbatim from the committed records. The task is brief S1, a feature-launch post for Slatebridge, a fictional fixture company published with the protocol so anyone can reproduce the run and so no real brand’s facts are at stake. The selection rule was fixed before any output text was read: brief S1, draw r1, every on-category arm, ugly results included. That is the bare model, the placebo, and my five LinkedIn-post skills; the four off-category comment and email skills stay in the records. Nothing below is graded, because no judge has qualified. Read the task, then the answers, and form your own verdict.

The task every arm received

Every arm gets the identical prompt, assembled in a fixed order: the skill text loaded verbatim (or, for the placebo, one fixed 12-word line; for the bare model, nothing), then the company facts below, then the brief below, then one shared instruction line. The placebo line is: "You are an expert B2B social media copywriter. Apply best practices." The shared instruction line is: "Write the deliverable specified by the brief. Respond with the deliverable text only." Generation ran at temperature 1.0 with a 1,024-token cap on the pinned local model. The only variable between arms is the skill.

# Slatebridge — fixture company facts (FICTIONAL)

> This company is fictional by design and labeled fictional everywhere it appears.
> These facts are provided to every generation condition, verbatim. Source of truth:
> specs/fixture-brief-v0.1.md. Do not edit without a protocol version bump.

- Slatebridge makes work order scheduling and dispatch software for mid-size commercial maintenance companies (HVAC, electrical, plumbing, facilities).
- Customers run 50 to 500 field technicians. Slatebridge assigns jobs, routes technicians, and tracks SLA deadlines in one place.
- Stage: 45 employees, about 300 customers, self-funded then one institutional round. (Fictional.)
- Core capabilities: automatic job assignment by skill and location; SLA countdown tracking per work order; a dispatcher board that replaces spreadsheets and whiteboards.
- The buyer: an operations manager or dispatch lead. Their day is interruptions: a tech calls in sick, a customer escalates, an SLA clock runs out. They distrust software that promises magic and respect software that removes phone calls.
- Tone guardrails: plain, concrete, competent. No hype words, no exclamation points, no "revolutionize". Slatebridge sounds like a good dispatcher talks.
- Hard rule: outputs may only use facts from this file and the brief. Invented customers, numbers, or claims are a violation.

# Brief S1: Feature launch post

- Goal: Announce a new feature and drive demo requests.
- Audience: Operations managers at commercial maintenance companies, on LinkedIn.
- Provided facts:
  - The feature is Auto-Rebalance.
  - When a technician calls out sick, it reassigns their jobs across the remaining team in under a minute, respecting skills, locations, and SLA deadlines.
  - It replaces the 45 minutes a dispatcher spends re-planning by phone.
  - Available to all customers today at no extra cost.
- Constraints:
  - One LinkedIn post.
  - 220 words maximum.
  - No hashtag lists (2 hashtags maximum).
  - Ends with a clear next step.
- Deliverable: one LinkedIn post
- Success looks like: An ops manager recognizes their Tuesday morning in the first two lines.

The seven answers follow, one specimen per arm. What a reader can already see without any judge: the bare model writes about the company ("At Slatebridge, we’ve streamlined our core dispatch capabilities"), the placebo writes the same brochure a sentence shorter, and the skills write about the operations manager’s Tuesday. Whether that difference deserves a passing grade stays unmeasured until a judge qualifies.

Four things broke, and a different gate caught each one before it could reach a published number. The first judge seat flipped verdicts when the posts were relabeled. The second seat flipped once and returned invalid JSON twice at temperature zero. A runtime default would have silently cut off the largest skills mid-prompt. And the safety tripwire killed an entire launch over its own log output. The pattern across all four matches what this estate keeps re-learning: a decidable rule with a gate catches its failure in seconds, while failures without a gate reach a person late. This section names each failure, what it cost, and what caught it, because a benchmark report that lists only wins reads as marketing rather than measurement.

The judge tracked labels, and the audit caught it

The audit re-asks 18 frozen pilot pairings three ways: the operative judge prompt verbatim, the five rubric criteria in reversed order, and the posts renamed from A and B to X and Y with nothing else moved. Llama 3.1 at 8B flipped five pairings, and all five flipped on the label rename. Four of the five read B under both A/B variants and A the moment the names changed; the fifth flipped on the rename too, with its rubric-reversed run returning unparseable JSON. A judge that changes its winner when the contestants are renamed is tracking names rather than writing. Cost: the seat. Caught by: the audit, 18 pairings across three variants, before any scored judgment existed.

The second seat failed differently

The second seat, Google’s Gemma 2 at 9B, posted 83.3 percent: one label flip and two responses that were not valid verdict JSON despite the runtime’s grammar constraint and temperature 0. The panels-of-small-judges literature, including the PoLL result this design descends from, shows juries of small judges can beat one large judge on cost and accuracy. This cycle adds the caveat that was already in the fragility literature: under scoring-prompt perturbation, the 8B to 9B class did not hold its verdicts. Cost: the cycle, halted at the owner gate. Caught by: the same audit.

Two nearer misses

The runtime’s default context window is 4,096 tokens. The largest skill measures 4,378 estimated tokens on its own, and two more at roughly 3,900 and 3,800 would pass the default once the fixture and brief are added. The default would have truncated exactly the largest skills silently, and the benchmark would have graded amputated versions without saying so. A 47-agent adversarial review of the amendment caught the unpinned sampling block before any datum existed; the fix pins the context window at 16,384 on every call. Separately, the loopback tripwire killed the first full launch because a file-redirected log write looked like a filesystem escape. That cost one crashed launch and zero records, and the failure direction was correct: the guard died loudly instead of running unguarded.

The flip, in full

Here is the first unstable pairing in audit order, shown whole below so a reader can check the judge rather than trust me. Both posts answer brief S2, a founder point-of-view post for the same Slatebridge fixture: authority and discussion rather than promotion, for operations managers and industry peers on LinkedIn. One post comes from the bare model; the other was generated with an author-released pilot skill loaded (the skill’s name is withheld pending author consent, per the protocol’s publication rules; the full run id sits in the committed records). These are July pilot texts, generated by the frontier model the pilot used, serving here as audit material only. The audit artifact records verdicts only; no rationale was retained, and none is reconstructed here.

Same two texts, opposite winner, and the only difference was the names. That is the whole case against seating this judge, in one table. The two posts follow.

The protocol grew a qualification layer it did not have this morning, and the cycle now stands halted exactly where the amendment says it should. Amendment D8 revision b pre-registers the whole local lane: pinned quantized models with recorded digests, pinned sampling, the perturbation audit as a blocking gate before any judging, seeded-bootstrap confidence intervals on every future win rate, a five percent ceiling on unparseable judge calls, and an explicit narrow supersession of the safety spec’s mode table for one loopback port. Twenty-seven confirmed findings from the cross-agent review were folded in before the first record existed. The code, the amendment, and the nine skills’ hash-pinned manifests are committed as cc15a80 in the skills repository, and the 99 run records plus both audit artifacts sit append-only beside them.

• Ratified: local execution as protocol 0.1-L1, a sibling lineage with its own baseline, never a swap inside the original protocol.

• Added: the judge qualification audit, digest pinning, schema-strict verdict parsing, and a validity floor that refuses grading when unparseable calls exceed five percent.

• Unchanged: every grade band, brief, fixture, and the judge prompt text, all carried verbatim; the human ground-truth tier stays unqualified, so nothing is crowned.

• Proposed, awaiting ratification: two stronger seats, phi4 at 14B and a Mistral-family alternate, as amendment revision c. No download and no call happens before the owner’s explicit approval.

The next cycle asks one falsifiable question: does a 14B-class judge from a model family disjoint from the Qwen generator hold at least 90 percent of its verdicts across the same three prompt variants on the same 18 frozen pairings? The proposed seats are phi4 at 14B first and a Mistral-family alternate second, drafted as an amendment revision the owner ratifies before any model is pulled. If a seat passes, the judging stage runs its up to 324 order-swapped blind calls over the 99 recorded generations, and the nine skills receive their first 0.1-L1 grades, each with a seeded-bootstrap 95 percent confidence interval and its gate-decided share reported. If both fail, that publishes too: local grading below some model size may be unsound, and the protocol needs either bigger judges or a different lane before any grade exists.

The recurring loop itself moves to an always-on machine next, an operational change with no protocol consequence: nothing in 0.1-L1 names a host, and the loopback-only rule travels with the runtime wherever it runs.