Production failures rarely begin with the model
When an agent fails in production, the root cause is often not that the model was incapable. The agent may have received stale context, selected a tool that exposed too much power, or completed only part of the requested action. The visible mistake happens at the model layer. The cause usually lives somewhere around it.
A production agent is a distributed system with probabilistic behavior. It crosses identity, data, application, and human boundaries. Treating it as a prompt attached to a model leaves the most important engineering questions unanswered.
Failure one: the context was correct yesterday
An agent can reason perfectly from obsolete information. A policy changed, a document was superseded, or an index did not refresh. The answer sounds confident because the source itself looks authoritative. The model has produced a correct answer to an old version of the world.
The control is provenance with a freshness requirement. The system should know which source supported the answer and when that source was last validated. When freshness cannot be established, the safe behavior is not to improvise. It is to say that the evidence is insufficient.
A citation alone does not solve this. A perfectly cited answer can still be wrong if the cited policy expired yesterday. The evaluation must check whether the source was valid for the decision, not merely whether a source was returned.
This is why retrieval quality is an operational concern, not merely a search concern. If the team cannot explain what the agent knew at decision time, it cannot explain the decision.
Failure two: the tool was more powerful than the task
A broad tool makes a prototype easy to build. It also gives the agent more ways to be wrong. An unrestricted database tool may be able to answer a question, but it can also bypass business rules. A general mailbox tool can send a notification, but it may expose messages the task never required.
The answer is not a stronger instruction telling the agent to be careful. The answer is a narrower contract. Expose the specific business action, validate its inputs, and make its side effects explicit. A tool should make the permitted action easy and the prohibited action impossible.
Tool design is policy expressed as software. If a boundary matters, enforce it below the model.
Failure three: the identity could do too much
Teams often connect an agent using an existing service account because the integration already works. That shortcut silently converts the account’s permissions into the agent’s authority. The agent may only need to prepare a change, yet the inherited identity can approve it.
Every action should run with an explicit identity and the least privilege required for that step. Read access and write access are different decisions. Preparing a transaction and committing it are different decisions. The architecture should preserve those differences instead of hiding them behind one credential.
Where possible, the action should preserve the user’s identity instead of disappearing behind a shared service account. That keeps authorization tied to the person who initiated the work and leaves an audit trail people can understand.
Human approval is meaningful only when the approver controls an action the agent cannot already perform.
Failure four: the error looked like success
Agents are good at continuing. That becomes dangerous when a tool returns partial data, an ambiguous status, or a technically successful response that did not complete the business action. The agent may summarize the output as success and continue building on a false assumption.
A dependable tool contract reports what changed and what did not. Consequential actions should return evidence that can be independently checked. When the result is ambiguous, execution must stop rather than allowing fluent language to conceal an incomplete operation.
Batch operations make this especially visible. Updating eight records and failing on the ninth is not success, even if the API returned a response. The tool must expose the partial result, and the workflow must know whether to retry, reverse, or escalate it.
Observability is not a dashboard full of token counts. It is the ability to reconstruct the decision, the action, and the evidence the system used to declare success.
Failure five: nobody designed a test that could disagree
A demo usually proves that the agent can succeed once. Production requires evidence that it fails safely. Happy-path examples will not reveal whether the agent respects a denied action, recognizes stale evidence, or stops after a partial tool failure.
Evaluation must challenge the behavior the team is most worried about. The suite should include boundary cases drawn from the real workflow and should run again when the model, context, or tools change. A model upgrade is a system change even when no application code moved.
The release decision should be based on that evidence. If a new version improves answer quality but becomes less reliable at respecting a denied action, it is not an upgrade for that workflow. Aggregate quality cannot excuse failure at a critical boundary.
A successful answer is not enough. The system must reach the answer through an acceptable path.
The operating model is the product around the model
An operating model assigns ownership before something goes wrong. It defines who can change the tools, who accepts an exception, and who decides whether evidence is strong enough for release. It also gives product teams reusable foundations instead of asking each one to rediscover the same controls.
It should also define how the system changes. Models will be replaced. Sources will move. Tool contracts will evolve. The operating model determines what must be reevaluated before those changes reach users.
Users do not trust a system because its architecture diagram contains a responsible-AI box. They trust it because the system uses current evidence, respects its authority, and fails visibly. They trust it because a person remains accountable when the decision matters.
Model capability determines what an agent might accomplish. The operating model determines what the organization can safely depend on.