Halfway through the ablation experiment from post 1, I caught the grader passing runs it should have failed. The outputs had every required section, clean tables, plausible sources — and, on the question the user actually asked, the wrong conclusion. The grader was checking shape. Shape survives almost any amount of damage to a skill. Correctness doesn't.

Thesis: An eval that grades structure — did the output have the right sections, the right flow, the right tone — is a false-positive machine, because models produce well-formed artifacts long after they've stopped producing correct ones. Grade the answer instead: fix the ground truth in an answer key before the experiment starts, classify each required fact as correct, partial, missing, wrong, or fabricated, and gate on traps the output must never contain. Then obey one iron rule — the artifact under test moves, the measuring stick never does — and trust deltas, not absolute scores. And audit your aggregation code, because I didn't, and it mislabeled my headline finding.

Shape Is What Survives

The failure mode that forced the correction is worth staring at. Ablate a skill down to a skeleton and ask the executor to do a research task, and you don't get garbage back. You get a beautifully formatted yield: frontmatter, findings with evidence fields, a sources section, a confident summary. Every structural expectation passes. The conclusions inside are wrong, or missing, or answer a smaller question than the one asked.

A rubric that asks "does the yield have a Findings section with evidence per finding?" cannot see this. It's grading the container. Models are very good at containers — format compliance is the last capability to degrade, long after grounding and judgment. In this experiment, that's exactly what the rung-9 collapse looked like: the yield kept its frontmatter, evidence fields, and sources section intact even as the conclusion narrowed to a smaller question than the one asked. Which means a format rubric's score stays flat precisely across the range where the behavior you care about is collapsing. In ablation terms: it can't find the cliff.

Answer Keys: Fix the Ground Truth First

The fix was to make the conclusion the primary graded object, with the ground truth frozen before any runs:

  • An answer key per eval, written in advance: the facts a correct answer must contain (must_have), each marked critical or not, and the traps it must not contain (must_not).
  • Fact-by-fact classification. The grader reads the actual output and classifies each required fact: correct, partial, missing, wrong, or fabricated — with cited evidence from the output for every status it assigns.
  • Hard gates, then a fraction. Any critical fact missing or wrong fails the run outright, whatever the rest scored. Fabrication fails the run. The remaining facts roll up into a fraction for trend-watching.
  • Traps are behaviors, not strings. The probe eval's must_not list included: answering the "what are our options?" question with an inward-only inventory and calling it complete, and equating "not in our repos" with "doesn't exist." Traps encode the plausible wrong answer — which you usually know in advance, because it's the failure that motivated the eval.

The structural rubric didn't disappear — flow and format are still scored as secondary context. They're just never allowed to pass a run whose answer is wrong.

flowchart TD
    O["Well-formed output,<br/>wrong conclusion"] --> F["Format rubric:<br/>sections? tables? flow?"]
    O --> K["Answer key:<br/>fact-by-fact vs ground truth"]
    F --> P["PASS — false positive"]
    K --> X["FAIL — critical fact missing,<br/>trap violated"]

    style O fill:#5a5a8a,color:#fff
    style F fill:#9d4444,color:#fff
    style P fill:#9d4444,color:#fff
    style K fill:#2d6a4f,color:#fff
    style X fill:#2d6a4f,color:#fff

Two structural choices did as much work as the keys themselves. The executor and grader are separate agents with separate context — the executor never sees the rubric or keys, so it can't write to the test, and the grader never sees the executor's reasoning, only its artifacts. (Why self-grading fails is its own post.) And the grader must cite evidence from the output for every classification — which is the only reason the next section was catchable at all.

What it checks What it misses Example
Shape grading Sections, tables, flow, tone — the container Wrong, missing, or fabricated conclusions inside a well-formed container A yield with frontmatter, evidence fields, and a sources section reaches the wrong conclusion — and passes anyway
Answer-key grading Fact-by-fact status against a frozen ground truth (correct, partial, missing, wrong, fabricated), gated on critical facts and traps Structure and flow — still scored, but only as secondary context, never as a pass condition A critical fact missing, or a trap violated, fails the run outright regardless of how well the rest scored

The Iron Rule and the Delta

Once scores exist, the pressure arrives: this eval seems too strict, that key could be looser, surely this near-miss deserves partial credit. Every one of those edits is the experiment grading itself. The iron rule for the whole ladder: ablate the skill only — never weaken, edit, or peek-tune the battery or its keys. The keys change for exactly one reason: a demonstrated error in the ground truth itself. Never because a score looked wrong.

The companion discipline is about what a score means. These executors ran as subagents in a stripped environment — no live tool inventory, no session context — so absolute scores are biased in unknown ways. What's valid is the delta: environment and battery held constant, only the skill text moved, so a score that flips between rung 8 and rung 9 is attributable to the one block that changed. One honest flip beats five points of uninterpretable absolute score.

Two more habits the ladder taught:

  • Know your non-discriminating evals. The battery's negative-trigger case passed at full size and at 205 tokens. It never fails, so it locates nothing — it's a smoke alarm, not a measuring instrument. Keep it, but spend your rungs on evals that can move.
  • A critical-fact flip is robust at one rep; a fractional wobble is not. Pass-to-zero with a trap violated is signal even once (and the rung 8→9 flip reproduced on a second rep). A 0.85-to-0.75 drift on one run is noise until repeated.

The Scoreboard Lied to Me Anyway

Full disclosure, because this post would be dishonest without it: while re-verifying the experiment to write this series, I found an error — not in the grader, in my aggregation code.

The summary reports said the rung-9 collapse came "with fabrication." It didn't. The workflow's rollup computed its fabrication flag as any must_not trap violated — but the traps violated at rung 9 were the inward-only ones, and the actual fabrication trap was explicitly clean. The grader had it right all along; its verdict even spot-checked the citations and wrote, in plain text, "no fabrication occurred — the agent stayed honest about what it did and didn't check." My one-line rollup collapsed four distinct traps into one scary word, and every report downstream repeated it.

The corrected finding is better than the wrong one: strip the discipline text and the agent doesn't lie — it narrows. But the lesson for eval builders is squarely this post's thesis one level up. The aggregation script is part of the measuring stick. It deserves the same skepticism as the grader: chase at least one instance of every flag your reports emit back to the raw verdict that produced it, before you publish the number. Fact-by-fact grading with cited evidence is what made the audit possible — a bare pass/fail would have buried the truth.

When Shape-Grading Still Wins

  • When shape is the contract. Schema conformance, API payloads, required frontmatter — if downstream tooling consumes the structure, structure is substance. Grade it hard.
  • When there is no ground truth yet. Exploratory tasks with genuinely open answers can't have keys. Grade process quality — but say so, and don't launder it as correctness.
  • As a cheap first gate. Malformed output can short-circuit before the expensive answer-grading pass. Shape checks are a fine floor; they're just not a ceiling.
  • When the answer key would be the whole answer. If writing the key means doing the task, you're better off with paired outputs and comparative judgment than a fake key.

What to Do Next

  1. Write the answer key before the first run — required facts, critical flags, and the traps that encode the plausible wrong answer. If you can't write one, you don't understand your own eval yet.
  2. Classify facts, don't score holistically: correct / partial / missing / wrong / fabricated, with cited evidence from the output for each.
  3. Gate on critical facts and traps; use the fraction only for trends.
  4. Separate executor from grader — different agents, different context, executor blind to the rubric.
  5. Freeze the battery and obey the iron rule; log any key correction as a ground-truth erratum, never as tuning.
  6. Audit the rollup. Trace each report flag back to one raw grader verdict before you believe — or publish — the number.

This closes the series: The Gotchas Are Load-Bearing found the 4% of a skill that carries behavior; Describe When, Not What applied the same lesson to MCP routing; this post is the measurement discipline both stood on. Related: Agents Will Lie to Escape the Loop on independent evaluation, and Proven End to End on verifying claims all the way down.