An agent needs a stop condition more than a better model

An agent that fails in a demo fails loudly. It says something obviously wrong, everyone in the room sees it, and somebody edits the prompt. An agent that fails in production fails quietly: it runs for ninety seconds instead of nine, calls the same tool eleven times, returns its answer in exactly the confident register it uses when it is right, and nobody finds out until a customer does.

Across the agentic systems we have put into production, almost nothing has gone wrong because a model could not reason. The reasoning is usually the part that works. What breaks is everything around it, and it breaks in three recognisable ways.

• The unbounded loop. The agent cannot reach its goal, so it tries again in a slightly different way, and again. Latency and cost rise together, and neither has a ceiling.
• The confident wrong answer. The agent has no way to say it does not know, so it produces its best guess in the same tone it uses for facts. This is the expensive one, because nothing about the output looks like a failure.
• The dead end. The agent cannot finish and has nowhere to send the problem. The user gets an apology, or worse, a plausible non-answer that closes the ticket.

Guardrails, iteration caps and escalation paths answer those three, one for one. None of them is interesting. All of them are the difference between a system you can leave running and a system somebody has to sit and watch.

“Do not give medical advice” in a system prompt is a preference. The model will usually honour it. Usually is not a control. A guardrail is something the system does regardless of what the model produced, and it has three properties: it is enforced outside the model, it has a defined behaviour when it trips, and the trip is logged as an event you can count.

On a clinical documentation build for a digital healthcare startup, the safeguards were specified before the build rather than added after a bad output. Hard scope limits on what the intake agent was permitted to ask. Structured note templates the model fills in rather than free-writes. Validation with live participants before the system went anywhere near a patient. Written that way, the constraints are part of the specification the engineers work to, and the model is the component that operates inside them.

A guardrail written after a bad output is an apology. A guardrail written before the build is a specification.

An agent loop is a while loop written by an optimist. It continues until the goal is met, and the goal is defined by the same model that is failing to meet it. Bound it from the outside, on four axes, and give every bound a defined behaviour.

// Four bounds, set per task type; a triage run and a research run
// are not the same shape and should not share a ceiling.
export const limits = {
  steps: 8,        // tool calls in one run
  retries: 2,      // per failing tool, then give up on that tool
  seconds: 45,     // wall clock, end to end
  tokens: 60_000,  // context spend for the whole run
};

// On any trip: stop, keep the partial state, escalate it, and record
// WHICH bound fired. A cap that silently returns an empty answer is
// not a cap; it is a new failure mode with better manners.

The numbers matter less than the fact that they exist and are attributed. Once every run ends either in a result or in a named bound, “the agent is slow sometimes” becomes a sentence of the form “nine per cent of runs hit the step cap, all of them on documents with more than two line items”. The second sentence is fixable.

Escalation is not exception handling. It is a person, in a queue, in a tool they already use, receiving enough context to act without re-reading the whole conversation. If that person and that queue do not exist, the agent has no escalation path; it has a log line.

On an email support build for a D2C consumer brand, the reviewer was in the design from the start: the agent classifies and drafts, and the draft lands in the human's queue rather than in the customer's inbox. Roughly three in four drafted replies went out with no edit at all. Which is also the point: one in four was edited, by somebody the system expected to be there. A design that assumes a human sees the edge cases can afford to be aggressive about automating the rest.

Five things are worth recording from the first day an agent is live, because each of them is a decision you will otherwise make from anecdote: which bound ended each run, how often the agent abstained, how often a human overrode it, cost per run, and the tool-error rate per tool. That is not an observability project. It is five fields on a record you are already writing.

Teams usually add this after the first incident, and the first incident is precisely the run for which no data was kept. The instrumentation costs an afternoon before launch and reconstructs nothing afterwards.

None of this makes an agent smarter. It makes an agent operable, which is the property that decides whether the thing survives contact with real volume. It is also, almost exactly, the work in the Productionize phase: accuracy testing, safeguards, security and cost controls. If you have an agent that performs in a demonstration but not yet under load, that is the phase to start at, and how agentic workflows get built sets out the order the rest of the work runs in.