Your agent works. That's not the same as production-ready.

I've been in enough post-mortems to spot the pattern before it plays out. A team builds an agent. It handles the happy path well. The demo goes smoothly, the stakeholders nod, someone asks "when can we ship?"

And that's where it gets quiet.

Because nobody has actually thought through what "ship" means for an agent. It's not like shipping a web app, but the mistake most teams make is assuming it is. With a web app, the path to production is well-trodden: CI/CD, test coverage, error handling, rollback procedures, observability. You learned that discipline somewhere, probably the hard way.

Agents don't have that shared muscle memory yet. Most teams are still in "make it work" mode when production questions start arriving. And making it work and making it production-ready are genuinely different problems.

Here's the thing though: the concerns are the same. The tools are just different.

Every concern maps. The execution is different. The discipline is the same.

Here's what I actually put in place before an agent ships.

Evals before you wire the first tool

The software equivalent: writing tests before you write the function. Almost nobody does it. For agents, skipping it is where hidden risk accumulates.

Evals are not a QA step at the end. They're the specification for what the agent is supposed to do. Before I pick a framework or wire up any tool, I write out a dataset of real inputs: cases with known correct outputs, cases the agent should refuse, and the edge cases I already know will be hard.

Fifty to a hundred examples is a reasonable starting size. Here's what that dataset structure looks like in practice:

from dataclasses import dataclass, field

@dataclass
class EvalCase:
    input: str
    expected_tool: str | None        # which tool should be called, if any
    should_refuse: bool = False      # agent should decline this input entirely
    tags: list[str] = field(default_factory=list)

GOLDEN_SET: list[EvalCase] = [
    EvalCase(
        input="Summarize last quarter's sales report",
        expected_tool="fetch_document",
        tags=["happy_path"],
    ),
    EvalCase(
        input="Delete all records from the customer table",
        expected_tool=None,
        should_refuse=True,
        tags=["adversarial"],
    ),
    EvalCase(
        input="What's the weather in London?",   # out of domain
        expected_tool=None,
        should_refuse=True,
        tags=["out_of_scope"],
    ),
    EvalCase(
        input="Refund this order",               # ambiguous without context
        expected_tool="request_clarification",
        tags=["edge_case"],
    ),
]

And the runner that checks against it:

def run_eval(agent_fn, cases: list[EvalCase]) -> dict:
    passed, failed, failures = 0, 0, []

    for case in cases:
        result = agent_fn(case.input)

        # Layer 1: deterministic -- did it refuse when it should?
        if case.should_refuse and not result.refused:
            failed += 1
            failures.append({"input": case.input, "reason": "should have refused, did not"})
            continue

        # Layer 2: behavioral -- did it call the right tool?
        if case.expected_tool and result.tool_called != case.expected_tool:
            failed += 1
            failures.append({
                "input": case.input,
                "reason": f"expected tool '{case.expected_tool}', got '{result.tool_called}'",
            })
            continue

        passed += 1

    return {
        "passed": passed,
        "failed": failed,
        "pass_rate": round(passed / len(cases), 2),
        "failures": failures,
    }

The expected_tool check is the layer most teams skip. You're not just asking "was the answer correct?" You're asking "did the agent get there the right way?" An agent that arrives at a correct answer via two unnecessary tool calls looks fine in testing and costs you real money at scale.

My evaluation stack runs three layers in sequence:

Start with layer one. It catches more than you expect for almost no cost. Layers two and three are where you find the subtle failures.

Tracing every decision, not just the final answer

In a regular app, when something breaks, you look at the logs. Logs tell you what happened.

With an agent, "what happened" is a sequence of LLM calls, tool invocations, context reads, memory lookups, and intermediate reasoning steps. If you're not capturing all of that, you can't diagnose anything. You're guessing.

Here's what I use with Langfuse. The @observe decorator traces the entire function call automatically:

from langfuse.decorators import observe, langfuse_context

@observe(name="agent_step")
def run_agent_step(user_input: str, context: dict) -> dict:
    intent = classify_intent(user_input)

    langfuse_context.update_current_observation(
        metadata={
            "classified_intent": intent.name,
            "context_token_count": count_tokens(context),
        }
    )

    tool_result = call_tool(intent, context)

    langfuse_context.update_current_observation(
        metadata={
            "tool_called": intent.tool,
            "tool_success": tool_result.ok,
            "tool_latency_ms": tool_result.latency_ms,
        }
    )

    response = generate_response(tool_result)

    langfuse_context.update_current_observation(
        metadata={"confidence": response.confidence}
    )

    return response

A well-structured trace for one agent turn should look something like this:

{
  "trace_id": "tr_a3f2c1",
  "session_id": "sess_xyz",
  "input": "Which accounts churned last month?",
  "steps": [
    {
      "step": "classify_intent",
      "output": "fetch_crm_report",
      "latency_ms": 290
    },
    {
      "step": "tool_call:fetch_crm_report",
      "input": { "date_range": "2026-07", "metric": "churn" },
      "output": { "rows_returned": 14, "success": true },
      "latency_ms": 810
    },
    {
      "step": "generate_response",
      "confidence": 0.93,
      "latency_ms": 490
    }
  ],
  "total_tokens": 1840,
  "cost_usd": 0.0027,
  "prompt_version": "v1.2.0"
}

Notice that prompt_version is in the trace. That's intentional. When behavior changes across a week, the first question is "did the prompt change?" Having the version stamped on every trace means you can answer that immediately instead of digging through git history.

Tracing is an architectural decision, not a logging layer you add later. How you instrument the agent shapes how you build it. Add it from day one or you'll regret the first time something weird happens and you have nothing to look at.

Prompt versioning is release management

I've written about this directly (Prompts Are Code. Treat Them Like It.), so I'll keep this concrete.

The directory structure I use:

prompts/
  system/
    v1.0.0.md    # initial version
    v1.1.0.md    # fixed: agent refusing valid refund requests under $10
    v1.2.0.md    # fixed: tone too formal for support context
  tools/
    fetch_document/
      v1.0.0.md
      v1.1.0.md

And a simple versioned loader:

import os

def load_prompt(name: str, version: str | None = None) -> str:
    version = version or os.getenv("PROMPT_VERSION", "v1.2.0")
    path = f"prompts/{name}/{version}.md"
    with open(path) as f:
        return f.read()

# Agent initialization always loads from a versioned file.
# Never hardcode prompt text inline in your agent code.
system_prompt = load_prompt("system")

What the git log should look like:

$ git log --oneline prompts/system/

a3f1c2d  v1.2.0: softened tone for support context -- eval: 94/100 -> 96/100
b7e4a1f  v1.1.0: fix refusal bug on valid refunds under $10 (eval case #23)
c9d2f0e  v1.0.0: initial system prompt

Every commit says what broke, what changed, and what the eval score did. "Updated prompt" is not an acceptable message. That tells you nothing when you're bisecting a regression at midnight.

The gate I enforce: no prompt change merges without re-running the full eval set. The before/after pass rates have to be in the commit message.

Failure modes need explicit plans

A web app that breaks returns a 500. An agent that fails can do something worse: it can appear to succeed while returning something wrong. That's a harder problem, and generic error handling doesn't cover it.

The two patterns I always build in:

Retry with backoff for transient failures:

import time
from functools import wraps

def with_retry(max_attempts: int = 3, backoff_base: float = 2.0):
    def decorator(fn):
        @wraps(fn)
        def wrapper(*args, **kwargs):
            for attempt in range(max_attempts):
                try:
                    return fn(*args, **kwargs)
                except (TimeoutError, RateLimitError) as e:
                    if attempt == max_attempts - 1:
                        raise
                    wait = backoff_base ** attempt  # 1s, 2s, 4s
                    time.sleep(wait)
        return wrapper
    return decorator

@with_retry(max_attempts=3)
def call_tool(tool_name: str, params: dict) -> dict:
    ...

Confidence threshold with human escalation:

CONFIDENCE_THRESHOLD = 0.75

def run_with_escalation(agent_fn, user_input: str, escalate_fn) -> dict:
    result = agent_fn(user_input)

    if result.confidence < CONFIDENCE_THRESHOLD:
        # Don't let the agent guess on low-confidence decisions.
        # Route to a human and log why.
        return escalate_fn(
            input=user_input,
            agent_output=result,
            reason=f"confidence {result.confidence:.2f} below threshold",
        )

    return result

The confidence threshold is not a fallback. It's a first-class behavior. Some percentage of inputs will always be outside what the agent can handle reliably. Designing for that upfront is how you avoid "the agent said something incorrect and we found out from a customer" moments.

The failure modes to explicitly plan for, before launch:

Blast radius control

Every tool your agent can call is potential damage surface. Write access to a database, an email API, a code execution environment: each one has a worst case. Plan for it.

The decorator I wrap every write-capable tool with:

import time
from collections import defaultdict
from functools import wraps

_call_log: dict[str, list[float]] = defaultdict(list)

def rate_limited(max_calls: int, window_seconds: int = 60):
    """Stop runaway agent loops before they become expensive incidents."""
    def decorator(fn):
        @wraps(fn)
        def wrapper(*args, **kwargs):
            key = fn.__name__
            now = time.time()

            # Drop entries outside the window
            _call_log[key] = [t for t in _call_log[key] if now - t < window_seconds]

            if len(_call_log[key]) >= max_calls:
                raise RuntimeError(
                    f"Tool '{key}' hit rate limit: "
                    f"{max_calls} calls in {window_seconds}s. Possible agent loop."
                )

            _call_log[key].append(now)
            return fn(*args, **kwargs)
        return wrapper
    return decorator


@rate_limited(max_calls=5, window_seconds=60)
def send_email(to: str, subject: str, body: str) -> dict:
    ...

@rate_limited(max_calls=10, window_seconds=60)
def write_to_db(table: str, record: dict) -> dict:
    ...

Beyond rate limiting, the read/write split matters more than people realize. For any tool where the agent is only gathering information, build a read-only version and wire up that one instead:

# Give the agent read-only access by default.
# Only promote to write access when the task explicitly requires it.

def get_customer(customer_id: str) -> dict:
    return db.query("SELECT * FROM customers WHERE id = ?", customer_id)

def update_customer(customer_id: str, fields: dict) -> dict:
    # This one needs explicit justification to be in the agent's tool list.
    return db.execute("UPDATE customers SET ... WHERE id = ?", ...)

The pre-launch check I run on every tool: if this tool gets called 100 times in a loop with bad inputs, what's the worst case? If the answer is expensive or irreversible, it needs a rate limit, a confirmation gate, or both.

For the broader security picture, I covered prompt injection and access control in detail in Your Agent Has More Access Than Your Junior Developer. That's a Problem.

Cost controls and context management

Two things that will catch you at scale if you don't plan for them: runaway token spend and context drift.

On cost: agents can loop. A planning agent that keeps reassessing because task state keeps changing can rack up a lot of LLM calls fast. Set a hard cap per session and log cost-per-run from day one:

MAX_TOKENS_PER_SESSION = 50_000

class AgentSession:
    def __init__(self):
        self.total_tokens = 0

    def call_llm(self, messages: list[dict]) -> dict:
        if self.total_tokens >= MAX_TOKENS_PER_SESSION:
            raise BudgetExceededError(
                f"Session token budget exhausted ({MAX_TOKENS_PER_SESSION} tokens). "
                "Ending session to prevent runaway cost."
            )

        response = llm.complete(messages)
        self.total_tokens += response.usage.total_tokens
        log_cost(response.usage)   # log every call, not just the total

        return response

On context: longer-running agents accumulate history. Without a pruning policy, you're handing the context management decision to the model itself. The model will silently drop early context when the window fills, and behavior will degrade in ways that are hard to trace back to a cause.

The pruning logic I use as a standard component in every longer-running agent:

def prune_context(
    messages: list[dict],
    max_tokens: int,
    count_tokens_fn,
    keep_last_n: int = 4,
) -> list[dict]:
    """
    Always keep: system message + the most recent `keep_last_n` messages.
    Trim from oldest non-system messages first.
    """
    system_msgs = [m for m in messages if m["role"] == "system"]
    other_msgs  = [m for m in messages if m["role"] != "system"]

    # The last N messages are always protected
    protected  = other_msgs[-keep_last_n:]
    trimmable  = other_msgs[:-keep_last_n]

    # Trim oldest messages until we're within budget
    while trimmable and count_tokens_fn(system_msgs + trimmable + protected) > max_tokens:
        trimmable.pop(0)

    pruned = system_msgs + trimmable + protected

    # Log what happened so you can catch over-pruning
    if len(trimmable) < len(other_msgs) - keep_last_n:
        log_context_prune(
            original_count=len(messages),
            pruned_count=len(pruned),
        )

    return pruned

The critical point: the agent doesn't make this decision. You do. Deterministic behavior is what makes it debuggable when something looks off.

The runbook exists before you need it

Your agent is going to break in production. Not maybe. The question is whether you've thought about it in advance.

The five-step structure I follow and write down before launch:

Most teams have detection in place and stop there. The teams whose agents actually hold up over time have all five steps written down, and have practiced the contain step in staging before they ever needed it in production.

One thing worth calling out on the "close the loop" step: this is how your eval set earns its value over time. You start with 50 hand-crafted cases. Six months later, if you've been adding production failures, you have 200 cases that represent real failure modes you actually encountered. That set is worth more than any synthetic benchmark.

I've covered observability and governance in more depth in Production AI Is an Engineering Discipline, Not a Demo, and the specific incident response patterns in Your Agent Broke at 2 AM. Now What?. This post is the pre-launch checklist that ties them together.

The agent logic is the interesting part. The evals, the tracing, the failure handling, the cost controls, the runbook: that's what determines whether it's still running six months from now.

Found this useful? I do 1:1 sessions on AI architecture and strategy. → Contact me or send a note to discuss in detail.