What Ablation Cannot See

Every time my ablation harness measured a skill's description, the one or two lines at the top that say what the skill is for, it found that removing them changed nothing. Seven runs out of seven. Then the guidance tool did what it was built to do with a no-effect line: it suggested cutting it. Follow that advice and you delete the one part of a skill that decides whether it ever runs.

Thesis: A skill has two surfaces with opposite jobs. The body tells the agent what to do once the skill is loaded. The description decides whether it gets loaded at all. Ablation, as I built it, hands the agent the skill and then grades the work, so it can only ever measure the body. An eval that loads the skill for you cannot tell you whether anyone would have loaded it. The fix is not a better body eval. It is a different question. I have not built the instrument that asks it yet, but on paper it is cheaper and more trustworthy than the one I have been asking.

---

Seven for seven

Cornerstone splits an artifact into units and removes them one at a time. For a skill, the description field in the frontmatter is one of those units, and it is ablatable on purpose, because I wanted to know whether descriptions do anything.

Across every confirm run that contained one (the full pass, with each version repeated, rather than a quick screen):

All seven read no-effect. Four of them are in runs that still clear today's validity gate, the check that a run could have produced a negative result at all. Among those holding runs, it is the only kind of unit that read no-effect every time it appeared.

The guidance tool, given a no-effect unit, emitted a cut directive. To its credit it hedged, calling the unit a "candidate to drop" and warning that "the instrument does not separate prior-covered from unexercised, so widen coverage before trusting it." That is good advice for most units. For this one it does not help, because no amount of coverage would change the answer. On two real skills, following the tool's output would have deleted the description.

I have since changed it. A description that reads no-effect now lands in its own bucket, unmeasured, with the instruction "do not cut on this result". The measurement underneath is unchanged. Only the advice is.

What the rig actually does

Here is the core of the prompt every executor sees, in every variant, for every task, with the output rules that wrap it at both ends left out:

…
Follow this guidance document (a "skill") when completing the request:


…the variant, with or without its description…


User request:
…
…

The skill is already there. The agent did not choose it from a list. It was told to follow it. Whatever the description says about when to use the skill is irrelevant once the agent has been instructed to use it, so removing the description costs nothing, every time, by construction.

flowchart TB
    subgraph RIG["What the rig measures"]
        G["skill injected<br/>with or without description"] --> W["agent does the task"] --> S["grade the work"]
    end
    subgraph LIVE["What a description does in real use"]
        C["many candidate skills,<br/>descriptions only"] --> P{"agent picks one,<br/>or none"} --> L["only then is the body loaded"]
    end

    style G fill:#5a5a8a,color:#fff
    style P fill:#9d4444,color:#fff

The harness was never wrong about what it measured. It measured whether the description helps the agent do the work once the skill is loaded, and the answer is no. It just was not the question anybody needs answered about a description.

The other half of the measurement already exists

Describe When, Not What, post 2 of this series, measured exactly the half this rig cannot. I rewrote the descriptions of an MCP server's tools to say when to use each one instead of what it does, and routing on a 20-case battery moved from 90% to 100%. A tool description is not documentation; it is a routing surface. The agent reads it in competition with every other tool in context and decides from that text alone.

Put the two posts side by side and the conclusion is uncomfortable. One instrument says descriptions decide what gets used. The other says they do nothing. Both are right, about different questions. An author who only ran the second one, and trusted it, would delete their skills' trigger surfaces and watch them stop firing, with every eval still green.

Two surfaces, opposite economics

A skill's body is loaded only when the skill fires, so a long body costs nothing on the requests where it is not used. The description is resident in every request, because the agent needs it to decide. When I measured the skill surface across about 10,000 of my own session transcripts, 128 skills were on disk, their names and descriptions together came to about 10,500 tokens on disk, and most of them fired rarely or never: 80 had ever been invoked, and 35 of those exactly once.

So the part of a skill that is paid for on every single request is the part my instrument could not measure. The part it measures well is the part that is nearly free.

What would measure it

The question for a description is a selection question. Give the agent a request and a set of candidates, and see which one it reaches for. Remove or weaken one candidate's description and see whether the choice changes.

I have not built that instrument yet. On paper it beats the body tier, the ablation of what a loaded skill says, in three ways I did not expect when I started.

• The outcome is a label, not a judgment. Which skill was invoked is recorded in the tool-call log. Grading it is a lookup, so no judge model is needed, and none of last post's judge failure modes apply.
• It is cheaper. No judge calls means one model call per cell, instead of one plus one for every judged check.
• It can measure the direction nobody measures. Recall asks whether the skill fires when it should. Precision asks whether it fires when it should not. Trigger lists that tell the agent to fire even when the user never names the skill make recall look great and are never checked for precision, and over-firing is paid on every request.

Without the precision half, a selection pass would recommend ever-louder descriptions. That is the same mistake in the other direction.

The objection

No-effect is no-effect. If removing it changes nothing, it does nothing. Only on the question the rig asked. "Removing this changed nothing" always means "on these tasks, set up this way". When the setup makes the unit's job impossible to perform, the zero is a property of the setup.

The enterprise version is a platform team cleaning up a shared library of agent skills. They score every skill's text against an eval suite, find that descriptions never move the score, and standardize them down to one line each to save tokens. Usage of the long tail quietly drops, and nothing in the eval suite notices, because the eval suite loads every skill by hand.

What to do next

1. Never cut a description on the strength of a task eval. If your eval loads the skill, it cannot measure what the description does.
2. Measure selection separately: a request, a candidate set, and the tool-call log as the grade. It is deterministic and cheap.
3. Test precision, not just recall. Include requests the skill should not fire on. Otherwise you are tuning for loudness.
4. Ask what your instrument structurally cannot see before you read its zeros. Then write it down next to the result, where the next reader will find it.

---

Previously: The Run Where Every Signal Was Green. The measurement this post depends on is Describe When, Not What; the experiment that started the series is The Gotchas Are Load-Bearing.