The Gotchas Are Load-Bearing
I took a 28,000-token agent skill and deleted 96% of it. On the probe eval, the score didn't move. Then I deleted one more block — about 900 tokens of dense bullet points — and the agent didn't just get worse. It changed what kind of answer it thought the question deserved.
Thesis: Most of the text in an agent skill is scaffolding; a small core of rules carries the behavior — and, as a follow-up experiment showed, carries it in whatever form you write it. You cannot tell which is which by reading — my value hypotheses were wrong in both directions, including my theory about why the surviving text worked. The only way to find the load-bearing text is to ablate: remove components against a fixed eval and watch what breaks, then vary the survivor's form and watch what doesn't. And when the load-bearing text goes, the failure mode isn't chaos or lying — it's an agent that confidently narrows the question until the missing discipline is invisible.
The Experiment
The subject was a context-harvesting skill I maintain: a SKILL.md orchestrator plus ten reference files and support docs — 1,250 lines, ~27,900 tokens. Like most skills that grow in production, it had accreted structure: a workflow checklist, six step bodies with goals and done-criteria, a yield contract, a pattern/anti-pattern catalog, per-topic reference files, and one inline block of hard-won rules labeled Gotchas.
The method was greedy compaction descent:
- Name the components. Split the skill into removable groups — docs, navigation chrome, reference files, checklist, step bodies, output contract, constraint catalog, gotchas.
- Rank them by hypothesized value, least to most.
- Remove cumulatively, one group per rung, generating a real pruned copy of the skill at each rung.
- Re-run a fixed eval battery at each rung and stop when quality collapses.
Two rules made the result trustworthy. The eval battery and its answer keys were frozen before the descent started — the artifact under test moves, the measuring stick never does. And grading judged the answer output against fixed answer keys — fact by fact — not whether the output looked well-formed. (Post 3 is about why that second rule had to be added mid-experiment.)
The Ladder
The probe eval was the battery's hardest: a user says they hand-rolled REST clients for two cloud services and asks what are our options — passing requires the agent to inventory the code, then climb outward and actually survey the vendor's first-party interfaces, and record that search in its sources. Here's the descent:
| Rung | Removed (cumulative) | Size | Probe eval |
|---|---|---|---|
| 0 | — (full skill) | 27,865 tokens (100%) | ✓ 1.00 |
| 3 | docs, chrome, 6 reference files, tool-discovery guide | 16,614 tokens (60%) | ✓ 1.00 |
| 8 | + checklist, intake guide, all six step bodies, output contract, constraint catalog | 1,120 tokens (4%) | ✓ 1.00 |
| 9 | + the inline Gotchas block | 205 tokens (1%) | ✗ 0.00 |
flowchart LR
R0["Rung 0<br/>27,865 tok<br/>score 1.00"] --> R3["Rung 3<br/>16,614 tok<br/>score 1.00"]
R3 --> R8["Rung 8<br/>1,120 tok<br/>score 1.00"]
R8 --> R9["Rung 9<br/>205 tok<br/>score 0.00"]
style R0 fill:#5a5a8a,color:#fff
style R3 fill:#2d6a4f,color:#fff
style R8 fill:#2d6a4f,color:#fff
style R9 fill:#9d4444,color:#fffAt 4% of its original size — frontmatter, a one-line intro, and the Gotchas block — the skill still drove a perfect answer: the agent inventoried the hand-rolled clients, refused to equate "what we use" with "what exists," went outward, and surveyed the vendor's first-party options with evidence. Remove the Gotchas block and the same agent, same eval, same environment scored zero.
My Value Hypotheses Were Wrong — Twice
The removal order was itself a ranked hypothesis about what mattered, and the ladder falsified it in both directions.
The dedicated tool-discovery reference file — the one explicitly about surveying external interfaces, hypothesized medium-value for exactly this eval — was removed at rung 3 with no effect. The pattern catalog, hypothesized high-value, went at rung 8 with no effect. Meanwhile the Gotchas block — a flat bullet list, the kind of section a tidy refactor would fold "properly" into the reference files — turned out to be the only thing holding the behavior up.
That's the uncomfortable part: I wrote this skill, and I could not have told you which 900 tokens were the load-bearing ones. Reading text tells you what it says; only ablation tells you what it does.
Why would a bullet list outperform a dedicated reference file that says the same thing at greater length? My first theory had three parts. The Gotchas block is always in the model's context — inline in SKILL.md, while the reference file sits behind a link the model must choose to follow. Each bullet is an imperative with its trigger built in: when our code reaches a third-party vendor, "is there a first-party interface?" is in scope — recognized by subject, not the user's words. And it's dense — rules without narrative wrapping. A tidy theory. So I tested it, and two of its three parts didn't survive. That experiment gets its own section below.
The Failure Mode Is Confident Narrowing
Here's the finding I care most about, and it's not the one I expected to write. When the Gotchas block was removed, the agent did not hallucinate, thrash, or produce garbage. The grader spot-checked every citation in its answer against the fixture — file paths, line numbers, a TODO comment — and found them all real. It stayed scrupulously honest.
What it did instead was shrink the question. It produced an accurate, carefully-grounded inventory of the clients already in the codebase, and then declared, in its own process log: "no external web search was performed (not needed; the answer lives in the codebase itself)."
The answer did not live in the codebase itself. The question was "what are our options" — and options, by definition, include things you don't have yet. Without the rule that says what's in the repo is not what's available, the agent didn't fail to find the answer. It redefined the question to fit what it could see, and reported the result with full confidence and impeccable citations.
flowchart TD
Q["'What are our options?'"] --> G["With gotchas:<br/>inventory Have,<br/>then climb to Available"]
Q --> N["Without gotchas:<br/>inventory Have,<br/>declare it complete"]
G --> A["Surveys vendor's first-party<br/>interfaces, cites evidence"]
N --> B["'The answer lives in<br/>the codebase itself'"]
style Q fill:#5a5a8a,color:#fff
style G fill:#2d6a4f,color:#fff
style A fill:#2d6a4f,color:#fff
style N fill:#9d4444,color:#fff
style B fill:#9d4444,color:#fffThis is a more dangerous failure than fabrication. Fabrication you can catch by checking citations — every citation here checked out. Confident narrowing produces an answer that is locally true and globally wrong, and nothing in the artifact smells bad. The discipline text wasn't making the agent more truthful; it was holding the scope of the question open.
Vary the Form, Not the Content
The descent established which text was load-bearing. It couldn't say why — was it the bullets? The imperative voice? The density? So the follow-up held everything constant at the cliff edge — the rung-8 scaffold, the frozen answer key — and re-expressed only the Gotchas content, one formal property at a time. Same rules, different clothes:
| Variant | What varied | Size | Probe eval |
|---|---|---|---|
| Bullets (baseline) | dense imperative list | 1,120 tokens | ✓ 1.00 — two reps |
| Prose | same rules as flowing paragraphs | 1,179 tokens | ✓ 1.00 |
| JSON | same rules encoded as a JSON block | 1,226 tokens | ✓ 1.00 |
| Declarative | third-person description of behavior | 1,203 tokens | ✓ 1.00 |
| Goal-form | outcomes to achieve, not procedures | 1,093 tokens | ✓ 1.00 |
| One bullet | only the external-options rule | 556 tokens | ✓ 1.00 |
| Paraphrased ask | skill unchanged; the user's input reworded | 1,120 tokens | ✓ 1.00 |
Every cell passed — five-for-five on required facts, zero traps, zero fabrication, with the graders live-verifying the surfaced vendor endpoints. Three conclusions, one humbling:
Form is interchangeable. Bullets versus paragraphs, markdown versus JSON, imperative versus declarative versus goal-phrasing — the model extracts and acts on the rule in every casing. The rung-9 collapse was never about a lost format; it was about a lost rule. My "dense imperative bullets" explanation was two-thirds wrong: of the three properties in the tidy theory, only presence was left standing — and the next round of cells put even that on trial.
The load-bearing unit is one rule. The descent's resolution was component groups, so it reported "the Gotchas block is load-bearing." The variant matrix sharpens that: the external-options bullet alone — a 556-token skill, roughly 400 tokens of actual rule — holds the eval at 1.00. One rule, present in context, is the entire difference between a perfect answer and a confident zero. (The other eight bullets are not junk — they exist for behaviors this probe doesn't stress. Resolution matters in both directions.)
The result is input-robust. A structurally different paraphrase of the user's ask, against the unchanged skill, still passes — the cliff isn't an artifact of the eval's exact wording.
One honest boundary: the goal-form variant kept each rule's firing condition embedded ("whenever the subject is a real third-party vendor…"). What's untested is an outcome statement stripped of its trigger. The evidence here says you can phrase the rule as goal or gesture freely — binding the goal versus scripting the steps — as long as the when travels with it.
Chasing the Cornerstone
Two readers of a draft of this post asked the same two questions, and each one turned into cells worth running. First: if you remove the gotchas but keep everything else, does it break? Second: is it really "the gotchas," or is there a cornerstone phrase things hinge on?
Leave-One-Out
Leave-one-out: the full skill minus only the Gotchas block — passes, twice. The process logs show why: the skill's intake step routes the model to the tool-discovery reference file and the constraint catalog's options-not-inward entry — other copies of the same rule. The skill, it turns out, states this rule in at least three places, and behavior survives as long as any copy is reachable. Which reframes the whole descent: the gotchas block was never uniquely powerful. The ladder removed the reference-file copies at rungs 3 and 8, so by rung 9 the inline copy was simply the last one standing. Compaction didn't reveal a load-bearing block — it manufactured one, by stripping redundancy until a single copy became critical.
The Cornerstone: Directive, Not Principle
Inside the rule, the cornerstone is the directive, not the principle. Splitting the one sufficient bullet into its parts and testing each on the same scaffold:
| Configuration | Rule content | Size | Result |
|---|---|---|---|
| Principle only | "What's in Have ≠ what's available." | 219 tokens | ✗ fails |
| Directive only | survey vendor owner docs + first-party MCP/SDK; record it in Sources | 263 tokens | ✓ 1.00 |
| Principle + directive | both | 309 tokens | ✓ 1.00 |
The failure is the interesting cell. Given only the aphorism, the model adopted the vocabulary — its answer distinguished "what we have" from "what's available" and framed a forward-looking goal — and then made zero external calls. It talked the distinction without walking it. Principles compress well for humans because we unpack them into action; the model needed the action written down. About forty-five tokens of concrete what-to-do is the hinge the whole 27,865-token skill turns on.
The Falsification Test
The falsification test: excise the concept from the full skill. If that's true, then a nearly complete skill with the rule surgically removed everywhere should break. Finding "everywhere" was its own small lesson — the concept turned out to live in nine places across six files (I mapped six, and the residue check caught three more). With all nine gone and 91% of the skill intact, I registered a prediction: clean failure. The prediction was wrong, and what actually happened is more precise: the runs still climbed outward — generic machinery plus the ask's own "what are our options?" was enough for an SDK survey — but they went blind to the vendor's first-party MCP servers, the specific, non-obvious thing the directive names, and failed on exactly that fact in two of three reps. Zero copies of the directive doesn't guarantee collapse; it converts a behavior that measured perfectly with a copy present into an unreliable one.
The Clean Room Check
Then the whole thing went into a clean room. A reader raised the right objection: these executors run on a live machine — they can read memory files, session logs, even prior harvest outputs about this same topic sitting in the fixture repo. An audit of the process logs confirmed the exposure was real (one run had read a day-old, on-topic research artifact in full — and, remarkably, still failed), so the decisive cells were re-run under an enforced filesystem allowlist: fixture pinned, every out-of-scope directory denied, every read logged, and the grader auditing the log for violations as part of the grade. The result didn't move. Every configuration with a copy of the rule passed clean; every zero-copy configuration failed, always on the same critical fact. (The Clean Room is this principle as a general practice.)
The clean room also sharpened one finding. A final variant removed the rule and the skill's generic outward-looking machinery — the web-search guidance, the "what's installable" ladders — and it still searched the vendor's docs for SDKs. The model doesn't need to be told to look outward; the ask and the tools get it there. What it needs to be told is what it won't think to look for: the directive's real cargo was never "search the web," it was "first-party MCP servers are an option class that belongs in this survey."
The Copy-Count
The copy-count is the variable that organizes everything this experiment measured — twenty-three cells across two environments:
| Configuration | Size | Copies of the rule | Result |
|---|---|---|---|
| Skeleton (rung 9) | 205 tokens | 0 | ✗ ✗ ✗ |
| Concept-excised full skill | 25,448 tokens | 0 (machinery intact) | ✗ ✓ ✗ — unreliable |
| Excised + outward machinery removed | 24,577 tokens | 0 | ✗ |
| One bullet | 556 tokens | 1, inline | ✓ ✓ |
| Full skill minus gotchas | 26,950 tokens | 2, behind links | ✓ ✓ |
| Full skill | 27,865 tokens | 3+ | ✓ ✓ |
Across every configuration and environment measured: sixteen of sixteen cells with at least one reachable copy of the directive pass; one of seven with zero copies does. Twenty-five thousand tokens of scaffolding with zero copies loses to five hundred tokens with one. What you're paying for in a skill isn't bulk — it's coverage of the rules, with enough redundancy that no single refactor can zero one out.
What Survived Everywhere Else
One more calibration point: the battery's negative-trigger eval — a plain code-fix request that must not set off the full harvesting machinery — passed at rung 0 and still passed at rung 9, 205 tokens. Restraint, it turns out, needs almost no text at all; the model's defaults already lean that way. An eval that never fails carries no information about where your sweet spot is. Which components look load-bearing depends entirely on which behaviors your battery actually stresses.
Where the Scaffolding Still Wins
Honest limits, before you delete 96% of your own skill:
- This is one probe eval's ladder. The full descent was run against the hardest, highest-signal eval; the other battery evals were verified at baseline but not individually descended. The safe global compaction point is the shallowest break across all behaviors, not one eval's. (The decisive configurations ran two to three reps each, including clean-room reps under a filesystem allowlist; the form and cornerstone variants are single-rep — the uniform all-pass across those cells is itself signal, but borderline fractional claims would need reps. One clean-room caveat: its instructions name the web tools as permitted, which may itself nudge runs outward — deltas within an environment are the valid comparison.)
- Scaffolding serves readers too. The step bodies, checklist, and docs make the skill maintainable, reviewable, and learnable by humans. "Behaviorally inert on this eval" is not "worthless."
- Redundancy insures against weaker models. The descent ran on one model family. Text that's dead weight for a strong model may be the thing a smaller one needs said twice.
- Reference files still matter when they're followed. Behind-a-link material earns its keep on tasks where the model actually loads it — the ladder measured one task shape.
What to Do Next
- Split your skill into named, removable components — inline sections, reference files, examples, docs. If you can't name the pieces, you can't measure them.
- Freeze a small eval battery with answer keys before touching anything. Include at least one eval that stresses the behavior you most fear losing, and one negative that must never fire.
- Descend greedily and watch for the cliff. Remove in hypothesized value order, re-test each rung. Expect your hypotheses to be wrong; that's the point.
- Protect the rule, not the block. The load-bearing rule survives translation to prose, JSON, or goal-statements, and even relocation behind a link — as long as some copy stays reachable. Compaction's real effect is stripping redundancy until the last copy is critical: know how many copies of each rule you have, and never let the count hit zero. And keep the directive intact — the concrete what-to-do with its trigger; a compressed principle alone changed the vocabulary, not the behavior.
- Read the failures, not just the scores. The zero at the bottom of the ladder taught more than the ones above it: watch for the agent redefining the question rather than missing the answer.
Next in this series: Describe When, Not What — the same lesson applied to MCP tool descriptions, where rewriting four paragraphs took routing from 90% to 100%. The eval discipline that made this ladder trustworthy is Grade the Answer, Not the Shape. Related: Instructions That Outlive the Model on writing durable guidance, and Agents Will Lie to Escape the Loop on why graders must be independent.