Prompts Are Code. Treat Them Like It.

Prompt versioning, model routing, and why your agentic system's inference bill is lying to you.

There's a pattern I keep seeing in teams building agentic systems. The architecture is solid. The agents work. The critic-falsifier loop catches real problems. Everything looks good in the demo.

Then someone changes a prompt — a single system prompt, one paragraph reworded — and three agents downstream start behaving differently. No error. No schema violation. Just subtly worse outputs that take a week to notice and another week to trace back to the prompt change.

Or the inference bill comes in, and it's four times what anyone expected. Not because the agents are doing unnecessary work, but because nobody designed for the cost implications of an autonomous loop that retries, evaluates, and retries again.

Prompt management and cost governance. The two operational problems that every agentic team hits eventually, and almost nobody designs for upfront.

A prompt change is not a text change

This is the thing I need teams to understand before anything else: changing a prompt is not like editing a config file. It's closer to changing a function signature in a codebase where nothing is statically typed. The compiler won't catch it. The tests might not catch it. But something downstream will behave differently, and you might not know for days.

In a single-agent system, this is manageable. You change the prompt, you test the output, you move on. In a multi-agent system with a critic and a falsifier in the loop, a prompt change in one agent can shift the entire dynamic. The agent's output changes slightly. The critic, which was calibrated to the previous output pattern, starts flagging things it used to pass. The retry count goes up. The cost goes up. The latency goes up. And the final output might actually be worse than before, even though each step is technically working correctly.

This is why I treat prompts as code. Not metaphorically. Literally.

How I version prompts

I've used two approaches, depending on the project. Both work. Both solve the same core problem: knowing exactly which prompt was running during any given agent execution.

The first approach: Langfuse as a prompt registry. Langfuse has a built-in prompt management system that handles versioning, retrieval, and tracking. Each prompt lives in Langfuse with a name and version number. The agent pulls its prompt at runtime by name and version. Every run logs which prompt version was used, so when I'm debugging a failure three days later, I can trace it back to the exact text the agent was working with.

The second approach: markdown files in Git with a resolver pattern. Each prompt is a separate markdown file, version-controlled in Git like any other code artifact. A resolver function takes the prompt name and version, finds the right file, and returns the content. This gives you Git's full history — diffs, blame, branches, pull requests. When a prompt changes, it goes through the same review process as a code change. Because it is one.

In both approaches, the critical piece is the same: every agent run records the prompt name and version in its trace, tied to the correlation ID. When something breaks, I don't have to guess which prompt was active. I look at the run, I see the version, I pull the diff from the last known good version. That's where the investigation starts.

You need evals. Per prompt. Per model.

Here's where it gets harder.

A prompt doesn't exist in isolation. It exists in the context of a specific model. The same prompt sent to Claude produces different results from when sent to GPT-5. Not drastically different, the general direction is usually the same, but the nuances change. Formatting, verbosity, how the model handles ambiguity, how it structures its reasoning, how it responds to constraints in the system prompt. These differences are subtle enough to miss in a quick test and significant enough to break a downstream agent that expects a specific output pattern.

This means prompt changes need to be validated per model. Not "I tested it on Claude, and it works, ship it." Tested on every model that the prompt might hit in production. Because if you're doing model routing, and you should be, which I'll get to — the same prompt might be served by different models depending on the task complexity or cost constraints.

The way I think about it: every prompt needs its own evaluation suite. Not a massive test harness — a focused set of test cases that cover the expected outputs, edge cases, and integration points with downstream agents. Run the evals before the prompt change goes live. Run them against every model the prompt might hit. If the evals pass, ship it. If they don't, the prompt change isn't ready, no matter how small the text diff looks.

This sounds heavy. It's not. Most prompt evals are ten to twenty test cases. The investment is small compared to the cost of debugging a production issue that traces back to a prompt change nobody validated.

The cost problem nobody plans for

Let me walk through what actually happens cost-wise in an agentic system.

A task comes in. The orchestrator routes it to an agent. The agent makes an LLM call — that's one. The agent calls a tool, gets results, and processes them; that might be another LLM call. The result goes to the critic — another call. The critic says, "This is incomplete." The feedback goes back to the agent. The agent retries with the critic's feedback — another call. New result goes to the critic — another call. The critic passes it. The result goes to the falsifier — another call. The falsifier probes an edge case — another call. The result passes.

That's seven or eight LLM calls for a single task. And that's a clean run. If the critic rejects twice, or the falsifier finds a real problem and the loop cycles back, you're looking at twelve to fifteen calls for one task.

Now multiply that across hundreds of tasks a day.

The cost isn't linear with the number of agents. It's driven by the loop dynamics — how often the critic rejects, how many retry cycles happen, how strict the falsification criteria are. Two teams can have the same number of agents and wildly different costs depending on how their verification loops behave.

And here's the thing that catches most teams off guard: you can't predict this from the architecture diagram. The cost depends on runtime behaviour — model responses, critic thresholds, data quality, task complexity. You only discover the real cost profile in production, which means you need to be tracking it from day one, not estimating it from a spreadsheet.

How I actually control cost

The instinct most teams have is to set hard cost limits immediately. Cap each agent at X tokens, cap each run at Y dollars, done. That instinct is wrong, or at least premature.

Here's why: you can't set meaningful limits before you understand the baseline. A cost cap without production data is a guess. Set it too low, and you're killing legitimate runs that need the extra retry to get to a correct result. Set it too high and it's not a cap, it's decoration.

So I run it uncapped first. Deliberately. I let the system operate in production; I track every run, how many LLM calls went through, why they went through, which models were used, and whether those were the right models for the work. This observation phase is where the real cost insights come from. You start seeing patterns: an agent making calls that could be skipped, a critic that's too aggressive and forcing unnecessary retries, a task type that consistently burns three times the tokens of everything else.

That's when you optimise. Not by adding caps, but by eliminating unnecessary work.

First pass: cut the calls that shouldn't be LLM calls at all. Not everything needs to go through a model. Schema validation, format checks, basic data completeness, these are deterministic operations. Use Pydantic. Use code. Use conditionals. Every check you can move from "ask the LLM" to "run a validation function" is a call you never pay for again. I've seen agents where 30% of the LLM calls were doing work that a ten-line Python function could handle faster and for free.

Second pass: tune the prompts that are driving retries. When a specific agent consistently triggers critic rejections, the problem usually isn't the agent's capability — it's the prompt. The output is ambiguous, or partially complete, or formatted in a way the critic interprets as a failure. Tightening the prompt, making the expected output more explicit, and adding constraints that guide the model toward the right structure, this reduces retry rates without reducing quality.

Third pass: cache aggressively. Semantic caching — where similar inputs return cached outputs instead of making a fresh LLM call- cuts costs significantly, especially for agents that handle recurring task patterns. If thirty percent of your tasks are variations of the same query, you're paying for the same reasoning thirty times. A good caching layer catches the duplicates and near-duplicates before they hit the model.

Then set the caps. Once I've optimised and have a stable baseline, what a clean run actually costs, what a complex run with retries costs, I set hard limits at baseline plus a reasonable buffer. Now the cap is meaningful. If a run exceeds it, something genuinely unusual is happening, and the cap becomes an alert, not an arbitrary cutoff.

The order matters: observe, optimise, then cap. Most teams try to do it backwards.

Route the task to the right model, not the most expensive one

This is the single highest-leverage cost decision in an agentic system: not every agent needs a frontier model.

I think about model selection the way I think about staffing a project. You don't put your most senior architect on every task. Complex reasoning and nuanced decisions- that's where you want the strongest model. Execution tasks with clear instructions and structured outputs, a mid-tier model handles that fine. Quick classification, routing, simple extraction- a small, fast model does the job at a fraction of the cost.

In practice, this means I route different agents to different models based on what they're doing:

Complex reasoning and ambiguous tasks go to the most capable model available. This is where you need the model to think deeply, handle edge cases, and produce nuanced outputs. Cutting costs here is false economy — a cheaper model that gets it wrong means more retries, more critic rejections, and ultimately higher total cost.

Execution and structured output go to a mid-tier model. The task is well-defined, the expected output is clear, and the schema validation catches any issues. A strong mid-tier model handles this reliably, and the cost difference is significant.

Quick tasks — classification, routing, simple extraction — go to the smallest model that can do the job. These calls happen frequently, they're not complex, and the cost savings compound fast.

The important thing is that this isn't a global setting — it's per agent, per task type. The orchestrator knows which model each agent should use. And this is where prompt evals per model become essential: if your classification agent's prompt was written and tested on a frontier model, you can't just point it at a smaller model and assume it works. Validate first, route second.

Track cost per run, not per month

Most teams track LLM costs at the monthly aggregate level. That's useful for budgeting. It's useless for cost governance.

What you need is cost per correlation ID — how much did this specific run cost, broken down by agent, by model, by retry count. Because cost problems in agentic systems aren't uniform. They're concentrated. One edge case that triggers five retry cycles in the critic loop costs more than fifty clean runs combined.

When you track cost per run, you start seeing patterns. A specific task type that consistently triggers retries. A specific agent whose prompt produces outputs the critic keeps rejecting. A falsification check that's too strict and forces unnecessary loops. These aren't cost problems — they're architecture problems that show up in the cost data.

The tooling I use — Langfuse handles per-run cost tracking well, and the correlation ID ties it back to the agent trace. But even without specialised tooling, you can build basic cost tracking into your orchestrator: log the model, token count, and estimated cost for every LLM call, keyed to the correlation ID. When costs spike, you pull the expensive runs, trace the loop behaviour, and fix the root cause.

The balance

There's a tension in all of this that I want to name explicitly.

Rigorous verification — critics, falsifiers, retry loops — makes your system more reliable. It also makes it more expensive. Every quality check is another LLM call. Every retry is another set of tokens.

The temptation is to cut quality checks to save costs. That's the wrong move. The right move is to make the quality checks smarter. Tighter critic prompts that reject for specific reasons, not vague "this could be better" feedback. Falsification criteria that focus on the edge cases that actually matter for the business workflow, not theoretical completeness. Retry limits that are strict enough to prevent runaway loops but generous enough to let the self-healing mechanism work.

And underneath all of it: route the right model to the right task. The single biggest cost lever isn't fewer quality checks — it's not using a frontier model for work that a mid-tier model handles just fine.

Prompt versioning gives you traceability. Evals give you confidence. Model routing gives you cost control. Cost tracking per run gives you visibility. Together, they turn your agentic system from a black box with a growing bill into something you can actually operate.

This is the third in a series on building and operating agentic systems in production. Previously: What "Agentic" Actually Means. Next: Your Agent Broke at 2 AM. Now What? — debugging, observability, and why standard monitoring tools weren't built for agents.