Production AI Is an Engineering Discipline, Not a Demo

Over 25 years in enterprise tech, and more intensely in the last couple of years building AI products and advising teams on agentic systems, I've watched the same pattern repeat itself so many times that I can predict it almost word for word. Someone gets pressure from the top to "do something with AI." The conversation starts with the wrong question: which model should we use, GPT or Claude? A model gets picked, a few features get bolted on, it gets tested against a clean, predictable dataset, it demos beautifully, leadership signs off, and it ships.

Then, a few weeks later, someone asks why the thing is behaving nothing as it did in the demo. Nobody has a good answer. Money and credibility quietly drain away, and the "AI initiative" joins the long list of pilots that never actually scaled.

I think of this as vibe coding applied to AI systems, optimizing for a good demo instead of a measurable, traceable, accountable system. A demo only has to survive a curated set of inputs, once, in front of a friendly audience. Production has to survive everything, indefinitely, with nobody watching every interaction. Those are different engineering problems, and treating them as the same thing is where most AI projects go to die.

Here's the framework I actually use, five pillars I think about before I write a line of agent code, built from the projects where I got this wrong before I got it right.

The three gaps that explain almost every failure

When I trace back why an AI system fell apart in production, it's almost always one of three things.

The observability gap. If I can't trace every decision a system makes, I have no way to operate it responsibly. Something will eventually go wrong, and "I don't know what happened" is not an answer I can give a client, a regulator, or my own team.

The evaluation gap. Teams, including, early on, me, talk about "accuracy" and "groundedness" as if they're self-evident, without ever pinning down the one number that actually matters to the business, let alone building a system to track it continuously.

The governance gap. Nobody has decided what happens when the AI fails. Who's accountable? Who do you call at 3 a.m.? Who owns the data the AI is reasoning over? Without answers to these questions before launch, you don't have a production system; you have exposure.

The five pillars below are how I close those three gaps in practice.

1. Evaluation: write the spec before you write the code

I treat evaluation as the actual specification for what I'm building, and I do it before I've decided on a model, a framework, or a single feature. That means writing down, in numbers, what success looks like: what accuracy is good enough for this specific use case, what false-positive rate is tolerable, what the deflection or automation target is. "It seems pretty good" isn't a spec.

A concrete example from my own work: when I was building the relevance-detection layer for a social media monitoring tool, I didn't start by picking a classification approach. I started by sitting down and writing out forty or so real examples of posts that should and shouldn't trigger an alert, including the genuinely ambiguous ones, sarcasm, indirect mentions, and competitor comparisons. That set became the spec. Only after I had it did I design the three-tier detection architecture, because now I had something concrete to test each tier against instead of eyeballing outputs and saying "that looks right."

I split evaluation into three layers, and I treat them as separate engineering problems:

Deterministic checks: the cheap, solved stuff. Regex validation, format checks, classic NER for PII. There's no excuse for skipping these; they catch a surprising fraction of failures for almost no cost.

Semantic checks: this is where LLM-as-judge comes in: a second model scoring the primary model's output for groundedness, safety, and relevance against your golden set.

Behavioral checks: the layer I see skipped most often, including by me in earlier projects. Is the agent calling the right tool? Is it looping? Is it making three calls when one would do? When I was designing the agent plugin architecture, this is exactly where the real cost showed up: an agent could give a perfectly correct answer while quietly making redundant tool calls behind the scenes. Invisible in a quick test. Expensive the moment you're running it at any real volume.

2. Observability: you can't govern what you can't see

Every decision an agent makes, what it classified the intent as, what data it pulled, what it retrieved from a knowledge base, how it reasoned, what guardrail it checked, needs to be traced and timestamped. I don't treat this as optional instrumentation I'll add later. In the EU AI Act compliance work, this was non-negotiable: you genuinely cannot make a credible compliance case for an AI system if you can't reconstruct, after the fact, exactly what it did and why.

The practical example I keep coming back to is from designing the Phase B observability specs. I built out parallel spec sets, one using Langfuse, one as a native build, specifically because I wanted to compare what "good tracing" looks like across two different philosophies before committing. The exercise made it obvious that tracing isn't a logging afterthought; it's an architectural decision that shapes how every agent in the system is built, because you're designing not just for "what answer did it give" but "what was the full decision path."

3. Data foundation: where I spend most of my actual time

This is the pillar that quietly consumes more of my time than any other, often more than half of a project. The reason is structural: data has always been built for humans, and humans are forgiving. If a report has a stale number, someone notices and fixes it. Agents don't forgive bad data; they'll confidently hand back a wrong answer pulled from something outdated, and nobody finds out until it's already caused a problem.

I split this into two categories in every project I scope:

Question data: whatever the agent needs to actually answer correctly: source documents, APIs, retrieval indexes.

Tracking data: the observability exhaust itself, which needs its own schema and storage plan, especially once you're running more than a handful of agents.

A real example: I ended up building a deterministic context-management tool because of exactly this problem. As I worked with longer-running agent sessions, I kept hitting situations where context windows filled up with irrelevant or stale history, and the agent's answers degraded in ways that were hard to diagnose unless you actually inspected what was sitting in context. Building that tool taught me that data hygiene for AI isn't just about the source-of-truth documents; it's about actively managing what the model is allowed to see at every step, because more context is not the same as better context.

4. Multi-agent orchestration: complexity doesn't scale linearly

One agent is manageable. Five agents are a fundamentally different problem: coordination, waiting on each other, retries, and failure handling, all of which compound. I think in terms of three patterns:

Orchestrator-worker: a central orchestrator delegates to specialized agents, and every request flows through one place. When something breaks, I know exactly where to look.

Choreography: agents are autonomous, reacting to events on a shared bus, with no central coordinator. Lower latency because work happens in parallel, but you trade away the simplicity of one place to debug.

Human-in-the-loop: a human is pulled in whenever an agent's confidence drops below a defined threshold.

When I was designing a multi-module go-to-market automation pipeline, this distinction mattered immediately. The strategy pipeline needed something closer to orchestrator-worker, a database-backed pipeline where each module's output feeds the next in a controlled, auditable sequence, because the cost of a wrong sequence (say, generating outreach copy before the ICP module had actually finished) was high enough that I wanted central control over the handoffs rather than letting modules react independently to a shared event stream. That's the kind of decision I now make explicitly, up front, instead of discovering it the hard way after agents start stepping on each other.

5. Governance: accountability, not just access control

This isn't generic data governance; it's specific to how the AI system behaves once it's live. I think of it in four parts: audit trails for every action and request, pre-validation and PII detection on inputs and outputs, model change management, and prompt versioning treated as real change control.

On model change management: I never take a provider's benchmark numbers at face value when a new model version comes out. I've seen "upgraded" models perform worse on my own evaluation set than the version they replaced, simply because the public benchmark doesn't reflect my domain or my data. The only way I trust a model swap is by re-running it against my own golden set first.

On prompt versioning: this is something I built directly into the 14-document AI governance framework I've been developing: prompts get treated like code, with commit messages that explain why something changed and what failure it was meant to fix, not just "updated prompt." Mapping that discipline against frameworks like the EU AI Act and NIST AI RMF made it clear that "we changed the prompt" is not an acceptable answer to an auditor; "we changed the prompt because the agent was misclassifying refund requests in this specific scenario, and here's the test case that proves the fix" is.

A few practices I now treat as permanent, not one-time

Your evaluation dataset is a living system, not a deliverable you finish. It starts small: a few dozen or a couple of hundred real cases is a reasonable starting point, and it should keep growing as production surfaces new failure modes. I give it an owner and categorize it by failure type, so patterns become visible over time instead of getting buried in a pile of one-off test cases.

Behavioral evaluations get expensive as the test set grows, because every change means re-running an increasingly large suite. My practical fix: run a small subset on every change, and reserve the full suite for merges to the main branch.

And I build the incident playbook before I need it, not after. The shape is simple: detect (via the evaluation dashboard), diagnose (via tracing), contain (roll back the prompt version, deflect to a human, or apply a fault-tolerance pattern), fix (using the test case library), and close the loop by adding the failure as a new permanent test case. Most teams build the first two steps and stop. The ones whose AI systems actually hold up in production build all five.

The takeaway

None of this is exotic. Deterministic checks, test datasets, tracing, and change management, these are practices engineering teams have used for decades. What's different with AI is that the systems are non-deterministic, data-hungry, and unforgiving of the small data quality issues humans routinely shrug off. The projects of mine that have actually survived contact with production are the ones where I built the boring infrastructure, evaluation, tracing, data foundations, orchestration discipline, governance, before I let myself get attached to a model.

Execution beats ideas. But for AI specifically, execution means doing the unglamorous engineering work first.