Infrastructure Patterns for Production AI Agents

Most teams start building agents the same way they build any other web feature: wrap the model in a route handler, parse the request, and wait for the response.

This is fine for a demo. It's also the first thing to break in production. Agents are long-running, stateful, and non-deterministic. They call tools, wait for external APIs, branch into subtasks, hit rate limits, and crash halfway through multi-step sequences. If your agent is tied to a single HTTP request, you've coupled your application's reliability to the wall-clock time of a non-deterministic loop. Moving past the demo means breaking that coupling. This post walks through the three core patterns teams use to turn fragile agent scripts into scalable, resilient production systems.

At the core of an agent is a loop: receive a goal, decide what to do next, call a model or tool, observe the result, update state, repeat until done. That loop has a few properties that make it hostile to naive web architectures. Agents are long-running. A normal API request should finish quickly. Agents often don't, somtimes taking hours or even days to complete. Agents are stateful. A run isn't one function call. It's a goal, a plan, tool calls and outputs, retries, errors, decisions, and a final output. If the process crashes, you need to know what already happened. If not, you either lose progress or rerun work blindly. Neither is great, especially for long-running agents. Agents are non-deterministic. Traditional workflow code says:

Agent code says:

The model chooses the next action at runtime, which makes recovery, replay, and debugging much harder. A run may take three steps or thirty. It may fan out across several tools, wait for a human, produce large intermediate artifacts, or stop early. Infrastructure has to set boundaries around that unpredictability with timeouts, budgets, checkpoints, approvals, and explicit termination conditions. Taken together, these properties mean agents need more than model calls wrapped in application code. They need architecture that decouples runs from requests, and infrastructure that preserves progress, records decisions, controls side effects, and manages artifacts separately from model context.

The first production pattern is to stop running the whole agent inside the web request. A request is the wrong lifetime for an agent run. It should create a durable run record, enqueue the work, and return immediately. That gives the application a simple boundary:

The API enqueues the job. The queue stores the job durably. The worker executes the job. The database records progress.

The client gets a run ID immediately. The agent runs somewhere else. The user checks status, subscribes to updates, or receives a callback when the run completes. A queue is a durable buffer between the thing that creates work and the thing that performs it. This is the default starting point when runs are longer than a normal request, tasks are mostly independent, and you need worker scaling, retries, or burst absorption. The trap: a queue knows a job exists, but it doesn't know the logical process that job belongs to. One background job is fine. Twenty dependent steps, three retries, two branches, and a human approval pause is not something a queue manages for you. You'll build that logic yourself. As we'll see later, that's where workflow engines come in. Reliability: Safe Retries and Partial Failure Moving work to a queue solves the lifetime problem. It does not solve for durability of execution. Most production queues are at-least-once: a job may run more than once. That's a deliberate tradeoff to avoid losing work, and your code has to survive it. Once an agent can call tools, write records, send messages, or provisions resources, retry behavior becomes part of the application’s correctness model. Two disciplines matter most. Idempotency answers what happens when a single step runs twice. Compensation answers what happens when a run stops partway through a sequence of steps. Retries make the first a requirement. Permanent failures make the second necessary for production. Agents that touch remote services needs both.

Bad worker code assumes no crash between steps:

Better code creates an idempotency boundary:

For agents, every side-effecting tool call needs the same treatment: check for a completed record before calling the tool, upsert a "running" record, call the tool, mark it complete.

Retry is not a recovery strategy unless the retried operation is safe.

Consider an agent that completes three of five steps:

it charges a card, provisions a resource, and sends a confirmation email.

Then step four then fails permanently. None of the first three steps can be rolled back with a database transaction, since a charge is not undone by deleting a database row; the money has already moved. Restarting the run from step one would recharge the card and resend the email, recreating the exact duplicate-side-effect problem idempotency was meant to prevent. Distributed systems have a standard answer to this: the saga pattern. For every side-effecting action an agent can take, define a compensating action that reverses the effect. A charge is reversed with a refund. An email is reversed with a correction message. When a run fails permanently, the orchestrator walks the completed steps in reverse order and runs their compensations.

Compensations must be idempotent in the same way forward actions are. Compensations can also fail on their own; a refund API can be unavailable just as easily as a charge API. A compensation chain needs a bounded retry, a dead-letter path for compensations that will not complete, and a manual escape hatch. Without these, a failed run can end up half-undone instead of half-done, which is not an improvement. Compensation is not always the right response to a partial failure. If four of five parallel sub-orders succeeded and one failed, completing the four and dropping the fifth is often the better outcome. This avoids unwinding work that succeeded. Compensation is worth building when partial success is unacceptable, not as a default response to any failure. Idempotency covers a step that runs twice. Compensation covers a run that stops halfway through. Both are required whether the run is executed by a simple worker or by a workflow engine. A queue distributes work, but it does not remember the shape of a run. It knows that a job exists. It does not know that step three depends on step two, that a human approval is pending, that two branches need to join before synthesis, or that compensation should run if the final step fails.

Once the hard part is no longer where should this job run? but what should happen next, given what already happened?, you have moved from queueing into orchestration. A workflow engine provides that orchestration layer. It stores the history of a run: which steps started, completed, failed, retried, timed out, or waited for external input. In a workflow, a crash doesn't mean starting over. Every workflow system divides the code into the same two roles: a coordinator that decides what happens next, and steps that do the actual work — model calls, tool executions, record writes. The names vary from engine to engine, but the division of responsibility is the concept that matters: decisions live in one layer, effects live in the other.

The coordinator has one hard rule: given the same history, it must always make the same decisions. This is because of how recovery works. After a crash, the engine rebuilds a run by re-executing the decision logic from the top, substituting recorded results for work that already completed. Replay only lands in the same place if the code decides the same way every time. So anything that could answer differently on a second pass — a model call, the current time, a random number, a call to an external service — belongs in a step, where the result is recorded the first time it runs and read back from history ever after. A stray clock read inside the coordinator, and a resumed run silently diverges from its own history. Durable execution tells you where a run was when it crashed. It does not remove the need for idempotency, because steps can still run more than once. This is why a workflow engine is the natural home for compensation logic: it already tracks which steps completed, which is the record a saga needs to walk backward. Scaling Workflows: Fan-Out, Fan-In, and Contention Workflows become especially useful when a run branches. Fan-out is the common shape: a lead agent breaks a goal into independent subtasks, sends each to a worker or subagent, and then synthesizes the results.

This buys parallelism and fresh context windows, which is useful for research, document analysis, and other breadth-first work. But the hard part is not dispatching the workers. It is coordinating the fan-in: what happens if one branch fails, how many failures are tolerable, whether to synthesize partial results, and how to avoid waiting forever. A queue can run the workers. A workflow decides what happens next. Fan-out is worth using when subtasks are independent, mostly read-only, and parallelizable. It works poorly when subtasks are tightly coupled or need to share state that keeps changing. Adding agents also adds cost and coordination overhead, so the question of how many agents to use matters more than the mechanics of spawning them. Fan-out also introduces contention. A system can work with three parallel subagents and fail at eleven, even when there is plenty of aggregate provider capacity. Uncoordinated calls can collide on the same rate limit, exhaust the same connection pool, or retry at the same time. Treat providers, databases, and model APIs as shared resources: cap concurrency, use token buckets where needed, and back off when the shared resource pushes back.