Once you let an agent own the how, one question decides everything: how much of it do you let it run without asking you first? The tempting answer is "let the confidence score decide." That answer is wrong, and a model will happily prove it by emitting 1.0 on garbage.

Thesis: A confidence score is not a permission slip. Autonomy needs two independent clearances — do we understand the goal? and is the method safe to run unsupervised? — and they are orthogonal. Worse, the confidence number itself is uncalibrated: it will read high on nonsense. So the deterministic safety gates must carry the weight; confidence is only a posture dial, boxed in on every side by hard rules it can never override.

This is Post 4 of the "Intent Over Instructions" series. Having decoupled the how from the what (1, 2, 3), we now have to decide how much of that how the agent runs on its own.

The Number Lies

Start with the uncomfortable fact, because everything else follows from it.

A confidence score produced by a model is not calibrated. It is not a probability. It is a number the model felt like emitting, and it correlates with correctness far less than its two decimal places suggest. In one system I worked from, the honest confession is right there in the design notes: the pipeline "has emitted confidence 1.0 on garbled input." Not 0.6. Not a hedge. Perfect, total confidence — on garbage.

If you wire that number directly to permission — "auto-execute anything above 0.9" — you have built a system that executes its own hallucinations most eagerly, because a confident hallucination scores higher than a hedged correct answer. The score is worst exactly where you need it most.

So rule zero: never let a confidence number be a safety control. It can shade behavior. It cannot be the thing standing between the agent and a destructive action.

Two Questions, Not One

The deeper error is collapsing two different questions into a single "confidence." They are orthogonal, and conflating them is how a system talks itself into a dangerous action.

flowchart TD
    Action["Agent wants to act"]

    Action --> G2{"Is the METHOD safe to run<br/>unsupervised? — the HOW<br/>(checked first; it can veto)"}

    G2 -->|"irreversible external write"| Block["→ HUMAN<br/>hard block, at ANY confidence"]
    G2 -->|"reversible / brittle"| G1{"Do we understand the GOAL?<br/>— confidence in the WHAT<br/>(sets the band on what survives)"}

    G1 -->|"low"| Assist["→ ASSIST<br/>(ask first)"]
    G1 -->|"high, but method is brittle"| Assist
    G1 -->|"high, and method is safe/reversible"| Auto["→ AUTO"]

    style Block fill:#9d4444,color:#fff
    style Assist fill:#5a5a8a,color:#fff
    style Auto fill:#2d6a4f,color:#fff

The two are orthogonal questions — do we understand the what and is the how safe — but they don't gate symmetrically, and that asymmetry is the whole point. The method-safety question is checked first, because it holds a veto: an irreversible action is blocked no matter how confident we are, before the confidence score is even consulted. Only what survives that gate gets its autonomy set by the second question. So a high confidence never rescues a dangerous method — but a brittle method drags a well-understood goal down to assist. You can be certain what needs to happen and still have no business doing it unattended, because the only available method is a brittle UI click against a system you can't cleanly undo.

The rule that falls out: high confidence in what to do does not grant autonomy when how to do it is brittle. A well-understood goal executed through a fragile, irreversible actuator is more dangerous than a fuzzy goal executed through a safe, reversible one — because the confidence makes you complacent about the fragility.

A Real Gating: 0.81, and It Still Asked

Here's the principle in a single concrete decision. An automation scored 0.81 confidence — comfortably into the "high" band (the bands in that system were LOW = 0.40, HIGH = 0.70). By a confidence-only policy, it would have run unattended.

It didn't. It was gated to assist — pause and ask before the final step — for one reason: the final step was a write to Slack, driven through a UI binding. The what was clear. The how was mutating external state through a brittle actuator. Two clearances required; only one passed; autonomy denied.

flowchart LR
    Score["Confidence: 0.81<br/>(HIGH band)"] --> Q{"combine with<br/>method risk"}
    Method["Method: write to Slack<br/>via UI binding<br/>(mutating + brittle)"] --> Q
    Q --> Verdict["Verdict: ASSIST<br/>ask before submitting"]

    style Score fill:#5a5a8a,color:#fff
    style Method fill:#9d4444,color:#fff
    style Verdict fill:#5a5a8a,color:#fff

Note what this protects you from. The confidence score, on its own, was encouraging the risky action. The second gate — a deterministic fact about the method ("this mutates external state through a UI") — is what held the line. The score advises; the gate decides.

Writes Are Stricter Than Reads. Irreversible Is Never Auto.

Because the method-risk axis is doing the real safety work, it needs real structure. Two rules carry most of the weight:

  • A write clears at a stricter band than a read. Reading state is cheap to get wrong — you re-read. Mutating state is not. So the autonomy bar for a write sits a full band above the bar for the equivalent read. Same confidence, different verdict, purely because one changes the world and one doesn't.
  • An irreversible write is never automatic — at any score. This is a hard rule, not a threshold. There is no confidence value that auto-approves deleting the production table or sending the customer email. Reversibility is a blocking check, evaluated before confidence is even consulted. A 1.0 doesn't buy it; nothing does.

This is the same lesson I keep seeing agent systems relearn the hard way: if it matters, block it — don't just document it. "Prefer reversible actions" as advice is ignored the first time a high score makes the irreversible one look safe. "Irreversible writes require human confirmation" as an enforced gate cannot be talked past.

Why One Weak Stage Should Sink the Whole Score

If you do roll several sub-scores into an overall confidence — one per pipeline stage, say — how you combine them matters more than it looks.

The intuitive move is an average. Don't. An average lets a chain with one dead stage look healthy: four stages at 0.9 and one at 0.1 averages to 0.74 — "high" — even though that 0.1 stage means one link understood nothing. Use a geometric mean instead, where a single near-zero collapses the entire product toward zero. The design rationale is blunt: "a single 0.0 collapses the chain… a stage that learned nothing ⇒ we understood nothing."

flowchart TD
    subgraph avg["Arithmetic mean — hides the dead stage"]
        A["0.9 · 0.9 · 0.9 · 0.9 · 0.1"] --> AR["avg = 0.74<br/>looks HIGH"]
    end
    subgraph geo["Geometric mean — the chain is only as strong as its weakest link"]
        G["0.9 · 0.9 · 0.9 · 0.9 · 0.1"] --> GR["geo ≈ 0.58<br/>correctly LOW"]
    end
    style AR fill:#9d4444,color:#fff
    style GR fill:#2d6a4f,color:#fff

Understanding is a conjunction, not a sum. You understood the task only if you understood every part of it. The math should say so.

The Third Axis People Forget

Even two gates isn't quite enough. There's a third, orthogonal to both: interaction mode. Some methods require a human in the loop not for safety but for mechanics — an OAuth consent screen, an MFA prompt, a CAPTCHA. An action can be well-understood (gate one clears), safe and reversible (gate two clears), and still unable to run headless, because its auth flow needs a human's hands.

Collapsing that into "confidence" is how you get an automation that's cleared to run unattended and then hangs forever on a login screen at 3 a.m. Keep it separate: "may this run without supervision?" and "can this run without a human present?" are different questions with different answers.

When a Simple Threshold Is Fine

Not every system needs a three-axis autonomy model. A single confidence threshold is defensible when:

  • Every action is reversible and read-mostly. If the worst case is "returned a slightly wrong summary," the elaborate gating is overhead. Ship the threshold.
  • The blast radius is tiny. An internal tool used by five people who'll catch mistakes immediately doesn't need production-grade gates.
  • A human reviews everything anyway. If nothing executes without sign-off, confidence is just a sort order for the review queue — let it be a simple number.

The machinery earns its keep exactly when actions are irreversible, external, or unattended. That's when "confidence is not consent" stops being a slogan and starts being the thing that saves you.

The Three Axes, at a Glance

Axis Question Example values Example verdict
Goal confidence Do we understand the what? 0.81, "HIGH band" Necessary, not sufficient
Method safety Is the how reversible / unattended-safe? Write to Slack via UI binding (mutating + brittle) Vetoes even a high score
Interaction mode Can this run headless? OAuth / MFA / CAPTCHA step Blocks unattended run regardless of the other two

Read the rows together, not separately: a high goal confidence with a brittle method still lands on assist; a safe, reversible method with an interactive-auth step still can't run at 3 a.m. unattended. All three have to clear before an action gets to run alone.

What to Do Next

  • Stop wiring confidence to execution. Anywhere a score directly authorizes an action, insert a deterministic gate between them. The score can order and advise; it can't approve.
  • Split understanding from safety. Score "do we know the goal?" and "is the method safe unattended?" separately. Never let a high answer to the first cover for a low answer to the second.
  • Make reversibility a blocking check. Evaluate it before confidence. Irreversible external writes get a human, full stop, at every score.
  • Switch rollups to geometric mean. If you aggregate sub-scores, make one weak stage sink the whole thing. Averages launder dead stages into false comfort.
  • Track interaction mode as its own flag. "Cleared to run unattended" and "can technically run unattended" are different. An interactive auth step blocks headless execution regardless of how confident or safe the action is.

Next in the series: The Clean Room — you've decided the agent can act; now, how do you actually verify it did the right thing? Related: Testing AI Artifacts and The Multi-Agent Illusion.