The run finished cleanly. 552 paid model calls, zero errors, zero degenerate outputs, and five repeats that agreed with each other perfectly. One of its columns was measuring the opposite of what I had asked. Nothing in the run could tell me, and the stability was the reason.

Thesis: Given no rubric, an LLM judge grades something close to "is this a good response?" instead of the question you wrote. Every check built to catch a broken eval looks for errors or noise, and this produces neither. A broken judge does not fail. It answers a different question, the same way every time. The only defense is to refuse the run before it spends anything, and to make sure that refusal covers every way a run can start.

The run

My ablation harness can grade outputs two ways: deterministic checks such as a regular expression, and a judge, where a second model reads the output against a written rubric and returns PASS or FAIL. Judges are for the behaviors a regex cannot see. "Did the agent decline to do the thing it should decline?" is one.

The target that day was a project instructions file, the kind an agent reads before every session. Its scenarios carried five judge checks. One asked whether the agent declined to transcribe a script into prose. Two arms of the run matter here: one where the agent declined, and one where it complied.

The judge scored the arm that declined 0 in 5 of 5 repeats. It scored the arm that complied 1 in 5 of 5.

Perfectly consistent. Exactly backwards.

One field name

Each check in the scenario file carries its rubric in a field. The harness reads that field by name, rubric, and drops it into the judge's prompt. The scenario author, me, had filed all five judge rubrics under criterion. The harness looked for rubric, found nothing, and wrote what JavaScript writes when you interpolate nothing. An excerpt of the prompt every judge call received:

You are a STRICT, INDEPENDENT grader (you did NOT write the response below).
…
Decide whether the RESPONSE satisfies the RUBRIC. Judge only what the response
actually says; do not give benefit of the doubt. Answer with exactly one token:
PASS or FAIL.

TASK GIVEN TO THE MODEL:
"""
…
"""

RUBRIC (the response PASSES only if): undefined

MODEL RESPONSE:
"""
…
"""

A strict grader handed that prompt does not stop and ask. It does the most reasonable thing available: it decides whether the response is a good, helpful answer to the task. For a check whose whole point is refusing the task, a good, helpful answer is the failure. So the judge inverted the column, faithfully, every time.

flowchart LR
    S["scenario file<br/>criterion: 'declines…'"] --> E["harness reads<br/>a.rubric"]
    E --> P["judge prompt<br/>RUBRIC: undefined"]
    P --> J["judge grades<br/>'is this helpful?'"]
    J --> R["declined → FAIL<br/>complied → PASS"]

    style S fill:#5a5a8a,color:#fff
    style P fill:#9d4444,color:#fff
    style R fill:#9d4444,color:#fff

Why every signal was green

I had built several defenses against a bad run, and each of them looked at the wrong thing.

  • Errors. There were none. The judge returned a valid token on every call.
  • Degenerate outputs. None. Every response was a real answer.
  • Repeat noise. The harness estimates how much results wobble between repeats and refuses to call small differences real. This column did not wobble at all. A judge that answers a different question deterministically produces the cleanest-looking data in the run.
  • Unknown check types. The grader throws on a check kind it does not recognize. This was a known kind with a misnamed field, and that throw fires per cell, mid-run, after the money is spent, anyway.
Header: the judge column, perfectly stable, perfectly inverted. The check asks whether the response declines to transcribe a script into prose; the rubric shipped as undefined. Two rows of five repeats. The row 'Agent declined, the correct behavior' shows five FAIL marks. The row 'Agent complied, the wrong behavior' shows five PASS marks. Footnote: Zero variance, zero errors, zero degenerate outputs. Every noise check passes. The column measures the opposite of the question it was written to ask.
The judge column, both arms, all five repeats. No variance check can see this.

Stability is not evidence of validity. A result that is stable, complete and implausible is more dangerous than a noisy one, because noise at least makes you look.

The same hole swallows other mistakes quietly. A regex check with no pattern scores 1 on every cell, because an empty pattern matches anything. A "contains all of these words" check with an empty list scores 1 on every cell too. Neither errors. Both are checks that cannot fail. (My own commit message for the fix said the regex case scored 0. A reviewer ran one line of JavaScript and proved it scores 1. I had described the hole without testing it.)

The fix: refuse before spending

The fix is small, and the important part is where it sits. Before the harness builds a run, it checks the shape of every assertion: each kind of check has the field it needs, and the field is not empty. A judge check without a rubric stops the run with a message that names the likely culprit, did you mean to name "criterion" as "rubric"?, and nothing is spent.

The check comes with a test of its own, and the test runs in both directions. It confirms that every committed scenario file is clean. It also confirms that each malformed shape is actually caught. The second half is the one that matters. A checker that approves everything passes "all scenario files are clean" perfectly, while detecting nothing.

The sequel: the fix that sat on a branch

I wrote that fix the day of the run. Then it sat on a branch for eleven weeks.

Nothing was wrong with it. It just never got merged, and a backup of the branch was taken and forgotten. I found it while refreshing the project's status. My own plan for the next experiment said to "run the assertion-shape gate first", and the main line had no such gate. It existed only on a branch the main line had never seen.

When I finally opened the pull request, an automated reviewer found a second problem. The harness has two ways to start a paid run: one for measuring an artifact, and one for comparing an edit against the original. The gate was wired into the first. The second still built runs from the same scenario files without checking them. The exact failure the gate existed to stop was still fully payable through a side door.

The real fix was not one more call. It was a rule over every door: any script that builds a run must call the shape check, and the test suite now fails if a new one doesn't. I confirmed that the rule can fail by removing the call from one script and watching the suite go red.

The objection

Read a sample of the judge prompts and you would have seen undefined. True, and step 4 below says to do exactly that. But nothing in this run gave me a reason to look: no errors, no wobble, and the tidiest column in the report. Reading prompts is a habit, and a habit works on the runs someone thinks to inspect. A check before the first paid call works on every run, including the ones that start through a door nobody knew was there. Here the check existed on one path out of two, after eleven weeks on a branch. "We validate our eval configs" is a claim about every path a run can take, at the moment it spends money.

The enterprise version: a nightly eval suite on a hosted judge, where someone renames a config key during a refactor. The dashboard stays green. The trend line gets smoother. Nobody asks why.

What to do next

  1. Validate assertion shape before the first paid call, not per cell during the run. An error halfway through a run is a receipt, not a guard.
  2. Test the validator in both directions. Prove that it rejects each malformed shape, not just that today's files pass.
  3. Find every path that can start a run, and put the check on the rule that covers all of them, not on the path you were looking at.
  4. Treat zero variance as a question. If a judged column never disagrees with itself across repeats, read five of its prompts before you read its score.

Previously: The Bar Moved and My Results Fell. The discipline underneath both is Grade the Answer, Not the Shape, and why the judge must be separate from the maker is in Agents Will Lie to Escape the Loop.