“Human in the loop” isn’t a safeguard. It’s a person who stops looking.

Here is the design everybody arrives at, and it sounds responsible.

The agent handles customer messages. When somebody wants a refund, the agent works out whether it is due and prepares it. A person gets a notification and taps approve. Money only moves when a human says so. Human in the loop. Signed off.

Now run it for three months. The agent is good, so nearly every refund it prepares is correct. The person approving them has now tapped approve four hundred times and been right four hundred times. What are they doing on tap four hundred and one?

They are not checking. They cannot be, and it is not a character flaw. The reliability of the thing they are watching is what stopped them watching, which is one of the oldest findings in this field.¹ You did not build an approval gate. You built a delay, and then you trained a person to remove it.

An approval gate’s protective power decays in proportion to how well the thing it guards is working. The better your agent gets, the less your safeguard is worth.

The failure was in the design, not in the person

When something does go wrong here, the account written afterwards will say somebody approved without checking. That account is true and it is useless, because it describes the last event rather than the cause.

Charles Perrow’s argument about tightly coupled systems is that serious failures in them are not deviations. They are what the design does under ordinary conditions, given enough time.²James Reason’s version is that every defence has holes in it, and trouble arrives when the holes in several layers line up at once.³

A single approval button is one layer with one very large hole in it, and the hole grows every time the system works correctly.

Where the safety actually comes from

Move the question. Stop asking “who approves this” and start asking “what is the worst thing this can do before anybody notices”. That number is your real safeguard, and unlike attention it does not decay.

Five things, in order of how much they are worth:

1. A cap it cannot exceed. Per action and per day, in rupees, enforced in the code rather than in the prompt. An instruction in a prompt is a request. A limit in the code is a limit. If the cap is two thousand rupees per refund and twenty thousand a day, the worst possible day is a known number you can decide to accept.
2. An undo path that a normal person can walk. Before launch, not after an incident. If reversing an action needs a developer and a database, you do not have an undo path, you have a hope.
3. A log that records the decision, not just the outcome. What the agent was told, what it saw, what it chose, and why. A log that says “refund issued” tells you nothing you did not already know from your bank statement.
4. A rate limit, separate from the cap. Twenty refunds in four minutes is a signal regardless of whether each one is under the cap. Most of the ways this goes badly are volume, not size.
5. An approval gate — last, and only for the exceptions. A gate that fires four hundred times is a reflex. A gate that fires twice a month, on the cases above the cap, is a decision, and the person making it will actually read the screen.

Notice the inversion. The approval gate is worth the least and it is the one every proposal leads with, because it is the one that is easy to explain in a meeting.

The question that settles it

Before an agent touches money, answer this in one sentence and write the number down:

If this agent went wrong at three in the morning and nobody noticed until nine, what is the largest amount of money that could have moved, and can we get it back?

If you cannot answer, the design is not finished. If the answer is a number you can live with and the money is recoverable, you can probably stop worrying about the approval button, which is the opposite of where most of these conversations start.

What we have not done

There is a version of this paper with an attack log in it — give an agent real refund authority in a sealed environment, spend a week trying to break it, publish what got through. That is the paper worth reading and we have not run it.

So this is a design argument, and you should weigh it as one. When we do run the exercise, the results get published including the attacks that worked and the ones we did not think of until afterwards. An attack log that reports only successful defences is marketing.