Manufacturing a Compiler

A while ago I watched a zoning agent I had helped build answer a question about a single family residential permit (by far the most common kind we deal with at the City of Dallas). The answer was fluent, well organised, and cited a section of the code that looked exactly like the section it ought to have cited. Something about it felt wrong. I had no idea whether it actually was. There was nothing to run. My subject matter expert contact in the Planning and Development department was out on vacation, and I was not going to be the most urgent thing in their week when they got back. So working out whether the agent had been wrong at all was going to take a fortnight.

If that had been a bug in a function I would have run the tests and known in four seconds. That gap- four seconds against a fortnight- is most of what people mean when they say AI works better for code, though it is rarely put that way.

What the compiler actually gives you

The usual explanation for why AI took to software engineering faster than to anything else is the compiler- a deterministic ground truth bolted into the workflow, so that a probabilistic model gets told immediately and for free when it is wrong. I have made that argument myself, in Coding Was Easy Mode. I still think it is right. But stated that way it is too vague to carry anywhere else, and the vagueness is why the lesson does not transfer.

The compiler is not valuable because it is accurate. A senior engineer reading the diff is more accurate than a compiler. It is valuable because it holds five properties at the same time:

It is free, in that each invocation costs essentially nothing. It is instant, seconds rather than days. It is total, running on every output rather than a sample. It is independent, in that it did not write the code and therefore does not share the author’s blind spots. And it is deterministic, so asking twice gets the same answer.

Software engineering got all five, for free, in 1957, and has never had to price them. Every domain without a compiler has to manufacture a substitute. The thing I did not understand for most of the first year is that you never get all five back.

Nothing gives you all five

Read down the columns rather than across the rows. Schema validation is the only substitute that keeps all five properties, and it cannot tell you whether anything is true. The moment a check can speak to truth, it starts costing money, or time, or independence, or repeatability, and usually more than one of those.

There is no cheap truth signal. Every architecture in the rest of this post is a consequence of that one sentence.

The two properties nobody budgets for

Cost and latency get discussed constantly. Independence and determinism almost never do, and they are the two that quietly decide whether your evaluation means anything.

Independence first. A compiler did not write the code it is checking. An LLM judge drawn from the same family as the generator did, more or less- same training distribution, same inductive biases, same gaps. It will therefore be most forgiving exactly where you most need it to be strict, because the generator’s characteristic errors look reasonable to it. This is measured, not speculative. Panickssery, Bowman and Feng showed in 2024 that models have non-trivial accuracy at recognising their own generations, and that self-recognition correlates linearly with how much they favour those generations (paper). The more recent result is worse for anyone hoping to solve it by adding judges. Guneet Kohli’s Nine Judges, Two Effective Votes (Apple, May 2026) put nine frontier judges from seven model families on the same items and found they supplied about 2.18 independent votes between them, with roughly three quarters of the panel’s nominal independence lost to shared mistakes. Aggregation did not rescue it- even with oracle access to the correct answers, established aggregation methods closed at most 11% of the accuracy gap (paper). The bottleneck is the correlation, not the algorithm.

Determinism is the one I underestimated. A compiler returns the same verdict twice. A judge does not, and neither does a panel of them. That means the number your evaluation harness reports moves on its own, and when it moves by two points after a prompt change you cannot say whether you improved anything. You have not merely lost the ability to measure; you have lost the ability to iterate, which is worse, because iteration is the thing that was supposed to get you to a good system in the first place. Nobody budgets for this. It shows up as a team six months in with a dashboard full of numbers and no shared belief about whether last month was better than this one.

You cannot verify an answer. You can verify a claim.

So far this is a list of things that do not work, which is the easy half. The way out, and I am fairly confident about this one because it is the only thing that has consistently worked for me, is to stop trying to check the output and start changing its shape until parts of it fall into the cheap column.

“Is this a good answer about my permit” is not a checkable question. But an answer is made of smaller assertions, and several of those are:

Does every factual assertion in this answer trace to a passage that was actually retrieved? That is a mechanical comparison. Is the ordinance it cites a real ordinance? That is a lookup against a table you already have. Is the phone number a real phone number, and does it appear anywhere in the knowledge base? Lookup. Is the deadline it states the deadline in the source document, or a different number that sounds like one? Comparison.

None of these tell you the answer is good. Together they eliminate a surprising fraction of the failures that actually reach a resident, because the failures that reach residents are mostly not subtle disagreements about interpretation. They are confident inventions of a specific, checkable thing.

Which is why I have come to think typed output contracts are badly sold. Pydantic schemas and structured output get pitched on developer ergonomics, as though the point were tidier code. The point is that an unstructured paragraph is a single opaque unit that only a human can assess, and a structured object is a set of fields, most of which a machine can assess for free. Structure is not neatness. It is the precondition for verification, and it is the one move that converts an expensive check into a cheap one rather than merely choosing between expensive checks.

Verification is a budget, and it should be spent by consequence

Here I am stating a position rather than reporting an observation, because I have not seen it done anywhere and I cannot point you to evidence that it works.

Almost every system I have seen verifies uniformly. The same grounding check, the same judge, the same sampling rate for review, applied to every query that comes in. That is wrong in both directions at once. It overspends enormously on somebody asking when the library closes, where the cost of being wrong is a wasted trip, and it underspends catastrophically on somebody asking whether they qualify for an exemption, where the cost of being wrong is a person not filing something they were entitled to file. Uniform verification is a way of pretending those are the same query.

The alternative is to treat verification as a budget allocated by cost-of-error rather than a gate every response passes through identically. Cheap structural checks on everything, grounding checks on anything factual, judges reserved for the middle band, and human expert review spent only where a wrong answer has a consequence somebody would want to appeal.

The obvious objection is that this just moves the problem. Deciding how much a given query is worth verifying is itself a judgement about stakes, and judging stakes is exactly the kind of unverifiable call that started all this. The router inherits the problem it was built to solve. I do not think that fully escapes, but it softens more than I expected, because stakes correlate with things you can actually detect in the output. Does the answer contain a deadline, a dollar figure, an eligibility determination, an instruction to file something? Those are checkable properties even when “how much does this matter” is not. You are not measuring consequence. You are measuring proxies for it, and then deliberately erring toward over-verification when the proxies fire, which is affordable precisely because they fire on a minority of traffic.

The ceiling is set by an org chart

All of which brings me to the thing that took longest to accept.

Look back at the table. Human expert review is the only signal in it that catches nearly everything. Every other row is an attempt to approximate that row more cheaply. So the quality ceiling of the whole system is set by how much expert attention you can get, and expert attention is not a technical resource.

The expert is a plans examiner, an assistant city attorney, a records supervisor. They know whether the answer is right. They have a full-time job that is not this. They do not report to you, their manager did not agree to this, and the work of grading a bot’s output shows up nowhere in anything they are evaluated on. In the best case somebody senior has asked them to help, which converts it into a favour, and favours are a currency you spend down rather than a budget you draw on.

This has a few consequences that are not obvious from inside an engineering team.

The first is that eval quality is rate-limited by something you cannot buy. You can add GPUs, buy a better model, pay for more tokens. You cannot buy the person who knows whether a variance request was handled correctly. There is exactly one of them and they are busy.

The second is that most of what looks like technical architecture is actually an attempt to economise on that one scarce input. Grounding checks, judges, structured outputs, sampling strategies- read them again with this in mind and they are all answers to the question “how do I get one more unit of confidence for one less minute of expert time”. That reframing changed how I prioritised. The best return was never a better judge. It was making expert review sessions shorter and less frequent by pre-filtering aggressively with the cheap checks, so that when I did get forty minutes, all forty were spent on cases where the cheap checks had already failed to reach a verdict.

The third is that the review has to be designed for the reviewer rather than for the harness. An expert handed a spreadsheet of 200 outputs and a 1-5 scale will produce noise, and will not come back. An expert handed twelve cases, each with the retrieved sources beside the answer and one specific question about the part the system was unsure of, will engage with it, because that resembles work they recognise. The bottleneck being social rather than computational means the interface to it is a social interface, and it should be designed with the same care as anything else.

I have started to think this is the real content of the phrase “human in the loop”, which usually gets deployed to mean nothing more than a person being nearby when something goes wrong. The loop has a throughput, and the throughput is measured in the attention of people who did not sign up for it.

The thing nobody priced

Most people now expect AI to be slower and costlier to deploy outside software than in it, and most of the estimates I have seen attribute that to compute, or integration work, or regulators. I think the larger share of it sits in the error signal, and it goes unnoticed because the thing being replaced was invisible. Nobody ever put a price on the compiler. It came free with the toolchain and has been sitting there since before most of us started working.

What you are buying back, in a domain without a compiler, is something free, instant, total, independent and deterministic. You will not get all five. You may not get three. And the last one on the list, the one that catches nearly everything, turns out to be a person in another department who has forty applications in their queue.

I am still not sure what a good version of that looks like at scale. I am fairly sure it is not a better judge.