The Bar Moved and My Results Fell

I had a correction written for the first post in this series. One of my own gotchas, it said, was making the agent worse. Before publishing it I made my measuring tool stricter, and re-checked everything it had ever told me. The correction did not survive. Neither did the result I had been using to explain the whole method.

Thesis: A stored eval result is only as good as the strictest instrument you have that could have rejected it. When I re-derived every result my ablation harness had produced under its current rules, six of the fourteen passes it had certified no longer cleared the bar, including both cases I led with. A result is not a fact you stored. It is a claim your current instrument has to be able to reject.

---

From one experiment to an instrument

Post 1 was a hand-run ablation. I took a 27,865-token skill, removed pieces against a frozen eval, and found the one rule the behavior hinged on. It worked because I watched every cell myself.

Posts 1–3 found the load-bearing text. These posts ask whether the instrument that found it could have said otherwise.

Watching every cell does not scale. So I built Cornerstone, a harness that does the same thing mechanically: split a skill, a memory file or a context doc into numbered units, rebuild it once per unit with that unit removed, run the same tasks against every version with a deliberately weaker model, and attribute each change in behavior to the unit that was missing. It also does two things I could not do by hand. It moves a unit to an appendix instead of deleting it, to separate what a line says from where it sits. And it substitutes a frozen paraphrase, to separate meaning from wording.

The part that matters for this post is the gate. Before the harness publishes a table saying which units matter, it asks whether the run could have produced a negative result. If it could not, it says so, and the table does not count.

The correction I did not publish

The harness's most interesting early finding looked like a harmful line. In a real skill of mine, on a run of two repeats, removing one bullet from the Gotchas list seemed to raise the score on a behavior the skill exists to produce: grounding the agent's stated reason in the thread that actually triggered the work. The bullet sat directly after another bullet that seemed, on the same two repeats, to protect that same behavior. Two adjacent lines that appeared to pull opposite ways, in a file I had read many times.

That was going to be the correction to post 1: the load-bearing/inert split was too simple, because some text is actively harmful, and here was a real example two bullets below the post's own headline rule.

Then I made the gate stricter

Two rules were added to the gate after those early runs. One is arithmetic. The other is a policy choice, and I will say which.

Repeats set the smallest effect you can see. With R repeats, a pass rate can only land on a multiple of 1/R. At two repeats the smallest possible change is 0.50, which is larger than the threshold the harness uses to call anything important (0.34). At two repeats the gate cannot reject. Every single flipped sample looks like a finding.

Dead checks dilute a run. An assertion that scores 1.0 with the skill and without it measures something the model already does. One that scores 0.0 in both measures something it cannot do. If more than a quarter of a run's assertions are dead in either direction, the run as a whole is not telling you much about the skill. The quarter is a policy, not a law: it was a half until I tightened it to match the bar the project's goals already stated.

Neither rule was aimed at any particular result. Then I re-derived every stored pass from its own graded data under the current rules. No new model calls: the inputs were already on disk.

Six of the fourteen certified passes flip. Eight still hold, three of them only just: at three repeats the gate clears its own arithmetic by less than a hundredth. The six that flip include the two I had built the story around.

What the headline case still says

The case I used to explain the method was a diagram skill. Removing one 12-token bullet, "Use subgraphs to group related elements", seemed to collapse the agent's habit of grouping large diagrams, and moving the same bullet to an appendix left it intact.

Those counts are still true. Across three diagram requests, nine outputs per version:

What does not survive is everything I built on top of it. Ten of that run's fourteen checks passed with the skill and without it: valid syntax, the right diagram type. The model already does those. The run mostly measured the model, so today's gate declines to certify it as a statement about the skill. Only twelve of the skill's twenty-six content units were tested one at a time, so my write-up's "exactly one is load-bearing" was never something the run could say. Its "~98% of the skill's tokens are dead weight" divided by the measured units, not the skill.

What I can say is this: in a run my current rules do not certify, one behavior moved with one bullet, on nine samples per version. It is an observation worth a proper run. It is not a finding.

The harmful gotcha was one coin flip

The harmful bullet was measured at two repeats. With the skill intact, the behavior passed once in two tries. With the bullet removed, it passed twice in two. That difference, 0.50, is exactly the smallest change two repeats can produce. One sample went the other way.

The two lines credited with protecting that behavior, its neighbor and a "Done when" paragraph earlier in the file, were the same arithmetic in reverse: one in two, then zero in two. The story about adjacent lines fighting each other was three single flips. I had even written the limitation into the draft. I had not let it change the headline.

So I re-ran it: the same skill text, the same four tasks, the same ten versions, five repeats instead of two. Today's gate certifies that run, with one caveat in its record: the executors were forbidden tools, yet the workflow logged nine tool uses across 200 of them, not traced to any cell. With the skill intact, the behavior passed 4 times in 5. With the "harmful" bullet removed, 5 in 5. One sample in five, well inside that run's own noise bound. The run classes the bullet as having no measurable effect, and the neighbor I had credited with protecting the same behavior lands in the same place. Nothing was fighting. The correction I almost published was two coin flips that happened to point at each other.

How much guidance does nothing?

I have quoted one number from this harness more than any other: across 57 measured units, 35 changed nothing when removed. My summary was that most guidance is inert.

Re-counted under today's rules, that number splits in two. Across every stored pass, 35 of 59 units read no-effect. Across only the passes that still hold, it is 10 of 26. But the passes that hold are mostly fixtures I built to contain load-bearing lines, to prove the harness can find them, so they lean the other way by design. The real skills, where "most of it does nothing" came from, are mostly in the six of the original fourteen passes that flipped. (The one real skill I have since re-run at five repeats holds, with 2 of its 7 measured units no-effect.)

So I do not know the share. I know it depends on which runs you are willing to count, and that the runs that made the claim are the ones the gate now declines.

What stands

Not everything moved. Three things from post 1 are untouched, because they never depended on these runs.

• Copy count. In post 1, sixteen of sixteen cells with at least one reachable copy of the rule passed, and one of seven with none did. That pools 23 cells across two environments; it was never a single-flip result.
• Confident narrowing. The failure mode, an agent that shrinks the question to fit what it can see, is a reading of transcripts, not a rate.
• Form is interchangeable, with the caveat post 1 already gave: each form variant ran once. By the rule above that is thin, and I hold it as a strong observation, not a measurement. The harness later produced results where wording mattered, but those came from paraphrases I wrote to delete the exact token the check looked for. They show the probe works. They do not overturn post 1.

Post 1's own headline, deleted 96% and the score didn't move, gets the same bar: the 1.00-to-0.00 critical-fact flip at the bottom of its ladder held two of two, reproduced on a second rep as post 3 records. Post 1 gives no rep count for the rungs above it, and any rung that read "no change" on a single rep is as thin as the results this post retires.

And one thing in the harness held up better than the results did: the eight passes that survive include a planted harmful line, a disclaimer the agent is told to add, that the harness found at five repeats and found again on a second independent run. The instrument can detect harm. I just have not caught any in the wild yet.

The objection

Changing the bar after the results are in is moving the goalposts. It would be, if the bar had moved toward a conclusion. Here it moved against mine, and it applies the same way to every run. Half of it is arithmetic I could have stated on day one: two repeats cannot reject. The other half is a policy I chose, and you can disagree with it. Under the old dead-check limit of a half, two of the six flips would still hold. The diagram case would not: 71% of its checks are dead under either limit. The alternative is keeping results that a stricter instrument rejects because they were stored under a looser one. That is the habit this series is about.

The enterprise version is common. An eval dashboard full of historical green results, computed by a scorer that has since been fixed. Nobody re-runs the old ones, because they passed. The launch review that approved the agent still cites them, and nobody can say which of them the fixed scorer would pass.

What to do next

1. Re-derive, do not re-quote. When your eval rules change, recompute every result you still cite from its stored inputs. Keep the inputs so you can.
2. Re-run before you retract. A correction is a claim too. Mine would have replaced one unreliable result with another.
3. Count dead checks before reading scores. A check that passes with and without the thing you are testing is measuring the model, not your text.
4. Put the limitation where the claim is. I wrote "two repeats" into my draft and still led with the finding. A caveat in the next paragraph does not stop the headline from travelling.

---

Previously in this series: The Gotchas Are Load-Bearing, the manual ablation this harness automates. Related: Proven End to End on claims that outrun their evidence, and Grade the Answer, Not the Shape.