Building reliable systems out of an unreliable ingredient

Every serious conversation about deploying AI in a regulated or operationally complex business hits the same wall within the first ten minutes: "But what about when it's wrong?"

It's the right instinct. Large language models will produce confident, articulate, completely wrong answers. They'll invent citations, misapply rules, and present fabrications with the same tone they use for facts. Anyone who's spent time with these models knows this. The buyer who raises it is doing their job.

But the question "what about when it's wrong?" contains a hidden assumption that makes it unanswerable. It assumes that the system's reliability is a property of the model. It isn't. The model is an ingredient. Reliability is a property of the system you build around it.

The ingredient analogy

Consider concrete. It's enormously strong in compression and nearly useless in tension. If you try to build a bridge out of concrete alone, it will crack and fail. But nobody builds bridges out of concrete alone. You build them out of reinforced concrete concrete for compression, steel for tension, engineered together so each material handles what it's good at.

Nobody calls concrete "unreliable." They call it a material with known properties, and they design around those properties.

Large language models are the concrete. They're enormously capable at synthesis, pattern recognition, language understanding, and generating structured outputs from unstructured inputs. They're unreliable at factual precision, logical consistency, and knowing when they don't know something. Those aren't bugs that will be fixed in the next model release. They're properties of the material.

The engineering question isn't "is the model reliable?" It's "can you build a reliable system using this ingredient?" The answer is yes but it requires designing the system, not just deploying the model.

Where the work actually is

The gap between a demo and a production system is almost entirely a reliability gap, and it breaks down into three categories of work that most teams underestimate.

Separating judgment from process. Every workflow contains decisions that genuinely require human judgment and decisions that are deterministic but happen to be done by humans because nobody has modeled the rules. An AI system that tries to handle both with the same approach will fail at the judgment calls and be overbuilt for the deterministic ones.

When we redesigned insurance underwriting workflows at Fidelity Life, the first step wasn't building anything it was separating which decisions in the application pipeline genuinely required underwriting judgment from which ones were deterministic and merely being done by hand. The deterministic path became an automated workflow. The judgment path kept humans in control but gave them better instrumentation. Application cycles went from two to four days down to twenty to thirty minutes not because the AI was smarter, but because the system was designed around the actual boundary between human judgment and mechanical process.

That separation is the first design question in any serious AI deployment, and it's a design question before it's an engineering one.

Defining what "good" means before you build. This is where most projects go wrong. They build the system, then try to evaluate it, then discover they can't agree on what a correct output looks like. The evaluation conversation should happen first and it's harder than anyone expects.

You can't write an evaluation for "is this response relevant?" until you've decided what relevant means for this document type, this user, this edge case, this regulatory constraint. What counts as correct for a billing dispute versus an outage? How complete does an answer need to be before a nurse can act on it? Which failures are annoying and which are catastrophic?

That's requirements elicitation wearing an engineering name. Building evaluations forces the conversation nobody has had yet, and it produces two things at once: a system you can measure, and a specification you didn't have. We call this Evaluation-Driven Design, and it threads through every engagement from the initial diagnostic through to the production system.

Instrumenting the boundary. Once you've separated judgment from process and defined what "good" means, the system needs continuous instrumentation at the boundary between what the AI handles and what humans handle. Not a test suite that runs once an instrument panel that shows you, in production, whether the system is performing against the criteria you defined.

When something regresses and it will, because models update, data drifts, and edge cases surface you need to know exactly what dimension failed, on what kind of input, and whether it's an annoying failure or a catastrophic one. That's the difference between a system you can maintain and a system you have to rebuild every time something goes wrong.

The trust-calibration problem

There's a deeper design problem that the reliability conversation usually misses, and it's the one that determines whether people actually use the system you build.

When we designed clinical data review tools for a federal health system, the central question wasn't "can the AI find the patterns?" It could. The question was: how much should the system assert versus how much should it show? Too much assertion and clinicians stop trusting it one confident wrong answer and the tool goes unused. Too little assertion and it's just a slightly fancier dashboard it doesn't save anyone time.

The right answer is different for every domain, every user role, and every type of decision. It depends on the cost of a wrong answer, the user's ability to verify, and the system's track record on that particular kind of input. Calibrating that interface where the system shows its confidence, where it shows its reasoning, where it flags uncertainty, where it just handles the task is design work, not engineering work. And it's where most AI deployments either earn trust or lose it.

Issue detection and data exploration effort dropped fifty to eighty percent not because the system was more capable than the clinicians, but because it was calibrated to surface patterns without overstepping the professional judgment it was designed to support.

What this means in practice

If you're evaluating whether AI can work in your operation, here's the framework that separates the projects that ship from the ones that stall:

Start with the separation question. Which decisions in your workflow genuinely require human judgment? Which ones are deterministic but done by hand? The boundary between those two categories is where the AI system lives and defining that boundary is the single highest-value diagnostic you can do.

Build the evaluation before you build the system. For every AI-assisted decision point, define what "good" looks like before you write the first line of code. What dimensions matter? What's the threshold for each one? What triggers human review? This isn't a test plan it's the requirements document for the system you haven't built yet.

Design the trust interface. How will people interact with what the AI produces? Where does the system assert and where does it show? How does it communicate confidence? How does it handle the cases where it doesn't know? This is where the system earns or loses adoption and it's almost always underinvested.

Instrument continuously. Evaluations aren't a pass/fail gate. They're an instrument panel. Build them into the system from the first prototype, not as a quality check at the end. When something regresses, you should know what broke, why, and whether it matters without waiting for a user to report it.

The buyer's objection "what about when it's wrong?" is the right starting point. The answer isn't "the model is getting better." The answer is: we build the system so that when the model is wrong, the system catches it, routes it to a human, and tells you exactly what happened. That's engineering, not hope.

The models are the ingredient. The reliability is in the design.