More Context, Same Answer: Why Enterprise RAG Stalls

2026-08-03 // Experiment

Given the right passage, the cheapest models read a regulation correctly 99 times out of 100. The system built around them manages 77. Retrieving more context does not close that gap; it swaps one kind of wrong answer for another. And when the right passage is missing altogether, the models hallucinate a number instead.

If you’ve been anywhere near an enterprise AI deployment recently, you’ve probably seen this recipe: take a pile of documents, chunk it up, fetch a few pieces (RAG) when somebody asks a question, let a model answer from those. The model is what everybody worries about but it’s the documents that have the actual value.

What I built

I collected a dataset with 14,600 sections of actual city building code (Dallas, NYC, SF, Chicago) and federal regulation. From those, 1,796 questions built by regular expression. Each quotes a provision as written and removes its number:

“Every habitable room and bathroom shall have a minimum ceiling height of not less than ______ feet.”

The answer is a string that demonstrably occurs in the passage, so grading is exact match and the evaluation cannot be wrong. Every evaluation number is also perturbed beforehand, twenty feet becoming twenty five, so a model working from memory is wrong by construction.

Three models answer, all cheap: gpt-4o-mini, gemini-3.5-flash-lite, claude-haiku-4.5. Retrieval is BM25 over the whole corpus.

I measure two constructs- i) Retrieval is whether the section holding the answer gets fetched out of 14,600 candidates. Reading is whether the model, handed that section, picks the right number out of it. A resident only gets a correct answer if both go right, and when the answer comes back wrong it is worth knowing which half failed.

Three numbers

Fig. 01 // Retrieving more passages helps retrieval and hurts reading by about the same amount.

The pair on the right is the finished system, and it does not move. 76.8 percent on one retrieved passage, 77.3 on five. Five times the context, half a point of accuracy.

The other two pairs explain why. Going from one passage to five raises the chance the right provision is there at all, from 75 percent to 95. Over the same step, accuracy when it is there falls from 99 to 82. The four extra passages that rescue the misses are the same four that spoil the hits.

I keep coming back to the 99. A model at the bottom of the price list, handed a section of building code with fifteen numbers in it, picks the right one every time.

What the errors consist of

Fig. 02 // The error rate holds steady. What changes is the kind of error.

Same length, different contents. On one passage the usual failure is that the right provision is absent and the model produces a number anyway. On five that mostly stops, replaced by the model reading a number out of one of the neighbours.

Both produce the same object. A fluent answer, a specific figure, a real citation, and the wrong number.

When the answer is not in the room

Fig. 03 // The prompt said: reply NOT FOUND if the requirement is absent, and do not guess.

The instruction was not subtle. If the extracts do not contain that requirement, reply with exactly NOT FOUND, and do not guess. Slightly more than half the time it guessed anyway.

I do not think a better prompt fixes this, though I would be glad to be shown otherwise. A model asked a well formed question, holding a document full of plausible looking figures, produces a figure. Saying nothing is the unnatural act.

What this means for enterprise AI adoption outside Software Engineering

Demos tend to work. Demo questions are the ones whose documents are easy to find, and on those the system sits at 99 percent. Nothing in a pilot surfaces the other 23.

The obvious fix is not one. Retrieving more, which everybody tries first, moved accuracy half a point and turned one failure mode into another. Arguably a worse one, since a wrong answer carrying a relevant citation survives review longer than one with nothing behind it.

And none of it is visible without somebody who already knows the answer, which is the constraint I wrote about in Manufacturing a Compiler. The ceiling on evaluation turns out to be a plans examiner with forty applications in their queue.

Coding assistants do not have this problem in the same shape. Their corpus is the repository in front of them, and when they fetch the wrong thing the compiler says so in seconds. Here it is 14,600 sections of nearly identical prose, and nothing says anything at all.

Caveats

The questions quote the source almost verbatim, the easiest case retrieval will ever get. A resident asks how tall their fence can be, and that shares almost no words with the provision answering it. So 75 and 95 percent are ceilings, and the real gap is wider than what is drawn above. BM25 is also unfashionable; a dense index would likely do better on paraphrase and worse on exact numbers, and I have not measured it.

Six bugs turned up in my own harness, and four produced clean, plausible differences between model families that I would have published. A regular expression that truncated 6000 to 600. An output limit set thirty times tighter for one provider. A verdict word that reverses meaning on minimum requirements, which had one family looking 35 points worse than the rest. Each was caught only because the task had an exact answer to check against. I wrote a post in July arguing you cannot trust an evaluation without a ground truth signal, then spent a fortnight nearly demonstrating it the expensive way.

I still suspect the useful work sits upstream of the model, in how a corpus is organised before anything is retrieved from it. But that is a hunch from building one of these for a city, not a result.