The Absent Human in the Loop

The tool I am building at work lets an investigator ask for linked police, dispatch and service records in plain language, and it writes the database query for them. On early feedback from the people using it we added a step in the middle: before the query runs it is shown to the investigator, who can read it, change it, or wave it through. That step is the human in the loop. I have rarely seen anybody use it.

My first assumption was that people skip it because they are busy. Having looked at the data though, I think that nothing about a generated query announces itself as worth a second look. It is syntactically valid. It returns rows. The rows look reasonable.

A human in the loop is the assumption nearly every rulebook for high-stakes AI rests on, and its shape is the same everywhere. If a system’s mistakes would matter, somebody qualified has to be positioned to see what it produced and step in. The American federal rule for government use, a 2025 White House budget office memorandum known as M-25-21, asks for human oversight, intervention and accountability suitable to the case. Europe’s AI Act and the main American standard for managing this kind of risk arrive at the same safeguard by other routes. Every one of them requires the reviewer to be there. Not one requires anybody to establish that the reviewer catches anything, so what the record finally shows is that a safeguard was installed and not whether it worked.

I could not measure that on the tool at work. But I had 2,341 scored answers sitting on disk from an experiment I ran earlier this year, which had asked how often a retrieval system of this kind reads a regulation correctly. So I put a different question to the same answers- when it was wrong, was there anything a reviewer could have gone on? A wrong figure is handed over as fact 13.8% of the time, and 86% of those wrong figures land within a factor of two of the correct one. Nothing marks them out. To catch a single one you would have to open the regulation and read the provision yourself, which is what the system was put in place to help with!

What was measured

The setup, briefly. Take 1,796 numbered provisions out of real regulation- the federal code, the New York City zoning rules, the Dallas development code and two others- and blank out the figure in each one. The question then reads something like no less than ___ feet, and the machine’s job is to find the right provision and fill in the blank. A keyword search ranks candidate passages out of the whole collection (BM25, the standard method, nothing clever), and three cheap, widely used language models are asked to supply the number. Because the right answer is a string that provably appears in the source text, scoring is a matter of exact match rather than anybody’s judgement, which is unusually clean as these things go.

The group that matters for oversight is narrower than the overall error rate. Of 2,341 answers, 323 are a wrong figure handed over as an answer. A further 171 are cases where the machine declined to answer because the governing passage genuinely had not been found, which is it behaving correctly and not something a reviewer needs to catch. Lumping those in with the failures would push the headline from 13.8% to 23%, and I did exactly that on my first pass through this data before I noticed.

Of the 323, a hundred and eight are cases where the right passage was sitting in front of the machine and it read a different figure out of it. The other two hundred and fifteen are numbers produced when the governing passage never arrived at all.

A number that is wrong by four hundred per cent will sometimes catch an experienced eye. A setback of forty feet where the code says thirty will not, and neither will a threshold of 65 where the answer is 60. Fig. 1 is why I do not think a reviewer’s attention can be rationed by looking at what comes out. There is no tail of obvious nonsense to skim off the top, and the wrong answers do not separate from the right ones on any feature a reviewer can see without doing the underlying work again.

The part I did not expect

The sharper result is what happens when the machine has nothing to read at all. In 419 of these cases the governing passage was never found. Abstaining was available, and on 171 occasions the model took it. On the other 248, which is 59.2%, the machine produced a figure regardless. Thirty-three of those were right by luck. The rest were confabulated: a number with no support anywhere in what the model had been given, delivered in exactly the same flat, assured tone as every correct answer.

That willingness to answer blind was consistent enough across all three vendors to look structural rather than like the quirk of one company: 53.9%, 53.9% and 70.1%. The most reticent of the three still answers, with nothing whatever to go on, more than half the time.

I want to be careful about one thing here, because it is the sort of claim that is easy to make and wrong. It is tempting to say these systems fail to flag their own errors, and this data cannot show that. The categories it uses are defined partly by whether the machine declined to answer, so “the wrong answers came with no warning” is true by the way the boxes were drawn rather than by anything I measured. What can be measured is the choice above- whether a machine handed nothing useful will say so. Mostly it does not.

A measurement that did not work

I wanted a stronger result than I got. If a wrong figure still appeared somewhere in the passage the machine cited, then the cheapest checking habit there is- confirm the number appears in the source- would pass it, which would have been a neat demonstration that spot-checking fails. It does not survive the obvious objection. Single-digit figures turn up in the passage 71.3% of the time against 15.1% for two-digit ones, and that gap is what coincidence looks like, since any regulatory passage contains a “1” and a “5” for reasons of its own. It is in the analysis script and I make no claim from it.

What this does and does not support

The limits are real. These are three small, cheap models filling in a blank in regulatory text, with keyword retrieval and queries lifted almost word for word from the source, which flatters recall badly. None of this measures frontier models, and none of it measures anything running in production, mine included.

What does carry is the shape of the thing. A reviewer needs something to act on, and in this data there is nothing: no confidence score, no figure that looks out of range, no sign that retrieval came back empty. The only route to catching an error is to redo the search and the reading by hand, and a reviewer who genuinely does that every time is not overseeing the machine so much as replacing it.

The opposite failure is much better documented. Prescribing software fires drug-interaction warnings so freely that clinicians dismiss the large majority of them, and the name for what follows is alert fatigue. What I am describing is its mirror image: there the warning is constant and gets tuned out, here there is no warning at all and the checking stops for want of anything to check. Both end with a reviewer who is not really reviewing, and the silent version is the harder one to notice, because a system that never cries wolf looks well behaved. What I would guess happens over the years a system is in use is that the checking gets faster and the catch rate falls, and that neither change is remarked on, because nobody is recording either.

Nobody has a reason to find out

So why does the number not exist? Not because it is hard to get. The measurements are ordinary- how often a reviewer rejects or amends an answer, how long they spend on one, and now and then slipping a known-wrong answer into the queue to see whether it comes back. What is missing is anybody who benefits from producing them.

A company selling one of these systems competes on accuracy, speed and price. An oversight number can only hurt it. Good, it proves little, because the buyer already assumed the reviewer works. Bad, it has handed a procurement officer a reason to buy elsewhere. The sensible thing is to instrument the model and leave the review alone.

The buyer is in the same position, and that is the part I find harder to argue away. A company that measures its catch rate and finds it low has acquired a problem: under the very rules that required the safeguard in the first place, it must now fix the system or stop using it. A company that never measures keeps a system that appears to be working and a compliance file with nothing missing from it. The incentive runs the wrong way on both sides of the transaction, and it runs that way for honest people with no intention of hiding anything.

That is the ordinary case for making something a requirement. The information is cheap, everyone gains from its existing, and nobody gains from being the one to produce it. The rules already oblige companies to put a person in the loop. Obliging them to report whether that person catches anything is a smaller step than the one already taken, and it is the only one that would tell us whether the first was worth anything.

Which brings me back to the step nobody presses. I built it, then spent a year not measuring whether it did anything, and nothing in my job required me to. I am the buyer in the paragraph above. That is the argument, and I would rather it had come from somewhere other than my own example.