Classical software fails honestly. An exception has a stack trace, a crashed process trips an alert, a 500 turns the dashboard red. The entire operational practice of IT (monitoring, alerting, on-call) rests on a quiet assumption: when something breaks, you notice.
AI agents break that assumption. A language model in a loop always produces something, even when its tools come back empty, its context has been clipped, or its stop signal got lost. The failure does not present as an outage, but as plausible-looking output. At scale.
What follows are five failure patterns from our own engineering practice: from agentic systems we build and run on our own hardware with local models. Every one of these stories happened to us. Not one of them threw an exception.
An agent’s most dangerous failure is not the crash. It is the output that looks like work.
The update that removed the stop
One of our pipelines has local language models generate scripts overnight, score them automatically, and refine them over successive rounds. Four parallel inference slots, each with a large context window. One evening, the second round seemed stuck. No error, no crash: the system simply kept running.
The cause was three days old. A chat-template update had accidentally declared a tool token to be an end-of-generation signal. The inference server, llama.cpp in our case, thereby lost the real stop token from its list. The result: generations of over 90,000 tokens that nobody had asked for. All four slots occupied. By then the gateway had long since answered the requests with a timeout. The orphaned generations kept decoding in the background, invisible to anyone watching only the API.
Three consequences have proven themselves since. Model updates are a supply chain: weights, chat template, and tokenizer metadata all change behavior, even when ‘only a small thing’ was patched. After every update, we check the stop-token configuration in the inference server’s log [1]. And a truncated generation (finish_reason: length) has been a first-class alarm signal in monitoring ever since, not a footnote.
# after every model update: is the stop-token list still intact?
podman logs llama-server 2>&1 | grep special_eog_ids
# in the harness: a truncated generation is an alarm, not a footnote
if response.finish_reason == "length":
alert("runaway_generation", model=model, tokens=usage.completion_tokens)The pile called ‘Other’
One of our workflows triages an inbox: a local model classifies every e-mail into defined categories. Twice, this workflow failed quietly: in two different ways, with the same symptom.
The first case: a well-intentioned output cap. A reasoning model spent its token budget mid-thought; the actual answer came back empty, and the pipeline filed every single mail under ‘Other’. It looked like a model quality problem. It was a budget problem. Removing the max_tokens limit fixed it outright.
The second case: the model server was unreachable. Every call failed within milliseconds, and the workflow dutifully recorded every mail as ‘Other (error)’, while the progress display looked frozen, because everything ‘finished’ so fast. Today the workflow aborts hard when the first wave of calls fails across the board, with a red banner instead of a fully ‘processed’ inbox.
# fail fast: if the first wave fails across the board, stop the job
first = [classify(mail) for mail in inbox[:3]]
if all(r.status >= 500 for r in first):
raise PipelineDown("model unreachable, refusing to file everything as 'Other'")The lesson is uncomfortable: nobody double-checks the ‘Other’ pile. A plausible wrong answer caused by a hidden cap is more dangerous than a loud context-length error. And a pipeline whose model is down must stop, not keep ‘working’ uniformly wrong.
Check the tool before you blame the model
An autonomous agent was set up to drive a third-party system through its scripting API. The local 12B model did poorly; switching to a large cloud model brought some improvement. The obvious diagnosis: the small model is too weak.
The real cause sat in the logs. The tool that prepared the system’s API reference for the agent was silently returning zero functions and zero namespaces: a parser tripped up by Windows line endings and a typed stub format. Both models had been working without a real API reference the entire time, reconstructing function signatures from the memory of their training data. The small model had even reported twice that the function list looked empty. It was right – and was ignored. The fix was a few lines in the parser: from zero to 49 functions.
‘The model is too dumb’ is the most expensive misdiagnosis in agent operations, because its therapy is a bigger model and more hardware instead of a bugfix. Our rule since: tool outputs get hard invariants. A reference with zero entries is not an empty result, it is a reason to halt the pipeline. And when a model reports that its input looks empty, it is worth a look.
# a tool output with zero entries is a reason to stop, not an input
reference = parse_api_stubs(raw_docs)
if not reference.functions:
raise ToolOutputError("API reference is empty, halting pipeline")Progress that isn’t
An extraction pipeline was meant to move facts from a large document set into a knowledge base, section by section, in a loop with the termination condition: stop when no new facts are added. A single section yielded 155 ‘new’ facts. On closer inspection, most were rewordings: ten to thirteen variants of the same underlying fact per entity. Each new phrasing was technically a new entry and reset the progress counter. The loop ran in circles, looking productive.
A stricter prompt (‘no rewordings!’) changed nothing. Lexical similarity measures could not reliably separate a paraphrase from a genuinely new, related fact either. What was missing was semantic deduplication: by embedding, not by string comparison.
The pattern is bigger than this example: a model in a loop satisfies the letter of its termination condition, not its intent. Progress measures for agents must measure meaning, not strings. Otherwise the system ‘makes progress’ for as long as the power stays on.
Green dashboards, invented answers
Two shorter finds from the same family. First: in the Model Context Protocol, the open standard for connecting tools to agents, a failed tool call comes back as an error flag inside a successful HTTP response [2]. A logger that only looks at the status code shows an error rate of zero while every single call is failing. The transport reports success; the truth sits one protocol layer deeper.
# MCP: the transport says 200, the truth sits in the payload
result = await session.call_tool(name, args)
if result.isError:
metrics.increment("tool_error", tool=name) # this call was NOT a successSecond: in an assistant system, we had switched off web search, cleanly removed from the tool list. But the system instruction still said: ‘use web search for current information.’ The model resolved the contradiction its own way: it invented search results, including a freely hallucinated weather report, in a tone of complete confidence [3]. The fix was not to hide the tool better, it was to tell the model explicitly that the tool is gone, and to forbid fabrication. Afterwards, the same question got the honest answer: no current data available.
Hallucination lives in the gap between what an agent is allowed to do and what it has been told to do. Anyone switching tools off per tenant, per role, or for compliance reasons – the normal case in enterprise use – must test exactly this case explicitly.
Failing loudly is an architecture decision
Five stories, one pattern: nowhere did an exception fly. Every system kept running and produced something that looked like work: endless generations, a sorted inbox, API calls from memory, a growing knowledge base, a green dashboard. Reliable agents therefore do not come from such failures never happening. They come from failure becoming visible before it has a real cost. Before the GPU spends nights computing scrap. And before people manually correct the very work the AI was meant to take off their hands [4]:
- Runaway detection as infrastructure: count truncated generations, check the stop-token configuration after every model update.
- Circuit breakers: if the first calls fail across the board, the pipeline halts. Loudly.
- Invariants on tool outputs: an empty reference or an empty answer is a reason to stop, not an input.
- Measure progress semantically: termination conditions via embeddings, not strings.
- Evaluate error semantics at every protocol layer, not just the HTTP status.
- Capability changes reach the agent’s instructions, not just its tool list.
One closing observation: not one of these failures was found by looking at ‘the model’. Every single one was found in the logs of a layer we operate ourselves: the inference server, the gateway, the orchestration’s own trace. On-premise does not prevent these failures. But it makes every layer they hide in inspectable. On a managed agent platform, several of those layers are a third party’s black box – and a quiet failure stays quiet. What an appliance looks like where hardware, model, and agent orchestration run under your control: see Product, and what agents concretely take over in mid-sized companies today: see Use cases.