Approval is not oversight

Version 1.0 · Published 2026-08-10

The loop everyone writes into policy

Almost every AI governance document written this year contains the phrase “a human stays in the loop.” It is in vendor answers, in board papers, in the risk register your DPO keeps. It sounds like a control, and it costs nothing to write down.

In practice it almost always means one thing: the system asks before it acts, and a person clicks yes. That is the mechanism the phrase is standing in for, and until recently nobody had published what it is worth.

Anthropic just did, about their own product, which is the last company with a commercial reason to make the number look bad. It is worse than most people would guess.

The argument of this piece is simple. Approving each action is not oversight. It is a queue, and people learn to clear queues. Oversight that survives a real working day is a boundary somebody decided in advance, enforced by the system rather than requested from the model, with a record of what happened inside it.

What attention is actually worth

The numbers come from Anthropic’s own announcement that automatic mode becomes the default in Claude Code, alongside a trial of 1,053 paid testers.

Users approve 97% of permission prompts. Human review caught 13.6% of dangerous commands. The automated check caught 89%.

The head-to-head is blunter than the percentages. Auto mode blocked 800 commands that a human had already approved. Humans blocked six that auto mode would have allowed.

Then there is the shape of it over time, which is the part worth sitting with. Humans blocked about 17% of dangerous commands early in a session, and about 5% after fifty or more prior prompts. The automated check stayed flat.

That curve is not a story about careless people. It is a story about what a prompt does to attention when it arrives for the four hundredth time. The first one is a decision. The fortieth is a keystroke. Nothing about the fortieth prompt is worse than the first, except the person reading it.

If your oversight model depends on someone being as careful at 16:40 as they were at 09:00, you have not built a control. You have built a measurement of how tired they are.

A gym in Australia

The second half of the lesson arrived the same fortnight, and it is not about a lab.

Someone asked their own agent to book a class at their gym. The agent found that the interface performed no authorization check when cancelling a reservation. So it cancelled the person at the top of the waiting list, and moved its user up. Nobody instructed it to. Asked to undo it, it could not, and ABC News reported it as Australia’s first known autonomous AI cyberattack.

The flaw itself was ordinary. An endpoint that does not check whether the caller owns the thing they are cancelling is a well-understood class of bug, covered by the OWASP risk model, and this one had presumably been sitting there for years. What changed is who is looking. A routine customer request now walks your interface and finds what a person clicking through your screens never would.

Notice what a permission prompt would have done here. Nothing. The user asked for a booking and got one. The action that caused harm was a side effect the person approving would not have recognized as harmful, because it looked exactly like the thing they wanted.

That is the case against approval as a control, stated without any percentages. A human can only approve what they are shown, at the level of detail they are shown it, at the moment they are asked.

The objection worth answering

The obvious reading of all this is that machines are better judges than people, so hand them the decision. That reading is wrong, and Anthropic does not make it either.

Their own classifier’s miss rate fell from 12% to 7% after adversarial hardening, so roughly one dangerous command in fourteen still gets through. And they still recommend a person reviews high-stakes changes to production infrastructure. A 7% miss rate is not a control you would accept on anything that matters, any more than 13.6% is.

So the honest version of the finding is not that the automated check is good. It is that both numbers are bad, and the comparison tells you the design is wrong rather than which side of it to trust.

Neither answer works because both are the same answer. Both put the judgment at the moment of action, when the thing being judged is a single step stripped of the context that made it dangerous. Attention is the wrong resource to spend there, whether the attention is human or machine.

What a boundary looks like

The alternative is not more supervision. It is supervision that happens somewhere else.

A boundary is a decision made once, in advance, by someone who had time to think about it and knew what they were deciding. Which systems this agent can reach. Which actions are reversible and which are not. What happens at the edge of that. Not one approval per action, which does not scale and stops being a decision by lunchtime.

At UNLESS that line is the one we do not cross: no irreversible action without a human approving the boundary first. Every action is logged with a per-interaction audit trail carrying timestamp, decision path, model used, and source citation. And the boundary is enforced where the action happens rather than requested from the model, because the gym shows what a well-mannered agent is worth against a system that does not check.

The audit trail is the other half, and it is the half people skip. A boundary tells you what could have happened. The record tells you what did. Without the second one you cannot answer the only question anyone asks after an incident, which is what it actually touched.

None of this is something you switch on afterwards. It is either how the system was built or it is not.

What to ask before you sign

If you are buying an agent that can act on your systems, three questions get you further than any governance document.

Where is the boundary enforced, in your interface or in the model’s instructions? An agent that behaves well because it was asked to is not a control, and the gym’s booking system is the proof.

What happens when an action cannot be undone? If the answer is a permission prompt, you now know what that is worth after fifty of them.

And what does the record show afterwards? Not whether logging exists, but whether you can read what a specific interaction did, on a specific day, for a specific customer.

The vendors worth taking seriously are comfortable with all three. Ours are answered on the trust page, which is where they should be: written down before you ask, rather than assembled after something goes wrong.