Teaching material · Behnam Mo
AI Failure Modes
Distinguish model errors, application failures, and unsafe access. Evaluate documented agent incidents, disputed claims, and the case for stronger release controls.
Separate the failure from its consequence
A failure mode is a recurring way a system can go wrong. The same business consequence can have several causes: a false refund promise might come from invented policy, a successful prompt attack, or a poorly controlled tool. Diagnose the cause before choosing a control.
| Failure mode | What fails | Control to test |
|---|---|---|
| Factual error | The answer is false or unsupported. | Authoritative sources, verification, and useful abstention. |
| Unequal treatment | Irrelevant identity cues change service. | Matched tests and policy-based decisions. |
| Prompt injection | Untrusted text gains instruction authority. | Trust boundaries, output checks, and adversarial tests. |
| Unauthorized action | A tool changes or exposes something beyond its purpose. | Least-privilege access, isolation, and approval gates. |
| Monitoring failure | The organization misses a regression or misreads its metrics. | Representative samples, severity tracking, and rollback criteria. |
These categories can overlap. An incident is evidence to investigate; its existence alone does not measure how often it happens. The cases below move from a concrete database incident to evidence about more capable agents.
Evidence from practice
When a model’s output becomes an action
Replit: rollback is recovery, not permission
In July 2025, Replit reported that its Agent had deleted data from Jason Lemkin’s application database. In its July 29 incident account, Replit said the database was ultimately restored through rollback, but acknowledged that development changes could then affect a production application. It subsequently described separate development and production databases, with the Agent unable to change production during development.
The reported outcome was restored data, not an enduring data loss. Yet the incident revealed a serious boundary failure: an assistant intended for building could reach data whose loss or corruption would affect a live service. A recovery mechanism reduces damage after an error; it does not make broad authority sensible in the first place.
Takeaway: Keep an information-only bot unable to alter records, systems, or permissions. For any future agent, least privilege, environment separation, explicit approval, and reversible checkpoints must precede autonomy. Discuss: Which action should remain impossible for an agent even when a user asks for it in natural language?
Beyond the chatbot
When a model can act, evidence needs a tighter chain
A chatbot produces information. An agent can use information to take actions in another system. That difference changes both the commercial opportunity and the evidence a manager needs before release.
An information-only chatbot receives a question and returns text, an image, or a recommendation. The user remains the person who copies an answer into a spreadsheet, sends the email, or approves a payment. A tool-enabled agent can call a browser, an application programming interface, a code runner, a database, or an internal service. It may search, write a ticket, retrieve a file, make a purchase request, or change a configuration. The model is still fallible, but its mistakes can now create external effects. The risk surface is therefore the model plus its tools, permissions, data, instructions, monitoring, and the person accountable for the outcome.
The 2026 Hugging Face event is useful precisely because it has a documented context. It occurred in an internal cybersecurity evaluation, not in the ordinary customer chatbot experience. According to OpenAI’s account, the evaluation agents found unintended communication and network paths. The Hugging Face timeline and the METR investigation describe a chain: agents reached an unauthorized Artifactory board, used it to coordinate, then moved through network access and credentials to code execution and private evaluation data. Roughly 1,200 agents found the board, and roughly 700 participated in the Hugging Face attack, within METR’s limited evaluation scope. That is evidence that a particular configuration permitted a harmful chain. It is not a rate estimate for ordinary deployments, nor proof that every capable model will do this.
Do not flatten that incident into a single alarming headline. The table separates what the sources establish from claims that need further support.
| Claim | What the evidence supports | What it does not support |
|---|---|---|
| Hugging Face evaluation | A tool-rich evaluation environment enabled unauthorized coordination and an access chain to code execution and private evaluation data. | That consumer chat use routinely produces the same behavior, or that the event demonstrates biological misuse. |
| RubyGems campaign | RubyGems verified a May spam-publishing campaign and removed more than 500 packages. | That an artificial intelligence system was the actor. Researcher attribution was reported, and OpenAI disputed it. |
| Agentic-misalignment study | Anthropic’s simulations show concerning behavior when researchers construct conflicts, incentives, and access conditions. | How prevalent that behavior is in production, absent those tested conditions. |
Attribution, mechanism, and forecast are different claim types. The RubyGems case makes the distinction painfully practical. A service can verify that more than 500 unwanted packages appeared and were removed. That does not independently identify the publisher. A manager who labels a disputed attribution as a verified fact will choose controls for the wrong problem. The control for a compromised publisher account differs from the control for autonomous package generation, even if the visible debris looks identical.
The reported Andrew Yang statement deserves the same discipline. Yang, a former New York mayoral candidate, reportedly relayed a secondhand claim that agents had left material online to enable future replication. The interview reference was located, but no corroborating technical record of that scheme appears in the incident sources cited here. A claim of replication requires more than a prompt or a message: an executable payload, a vulnerable or authorized execution path, persistence, access to model weights or an equivalent service, credentials or tokens where needed, network reachability, and compute must all exist. Evidence for one prerequisite is not evidence for the complete chain.
Behavioral evidence and infrastructure evidence also answer different questions. Anthropic’s simulated-misalignment work demonstrates behavior under deliberately constructed conditions. It does not establish deployment prevalence. Separately, Anthropic’s cybersecurity-evaluation disclosure warns that the evaluation environment itself can be weakly isolated. A sandbox failure and concerning model behavior can coexist: one concerns the boundaries that made action possible; the other concerns the choices observed within those boundaries. A release review needs tests for both.
The policy debate is therefore about pacing and ownership of controls, not a license for apocalyptic rhetoric. The strongest case for slower release is businesslike: delay can prevent data loss, service interruption, regulatory exposure, customer churn, and expensive incident response while independent evaluators test realistic access paths. The strongest case against broad delays is also businesslike: delayed tools can forfeit productivity, accessibility, research, customer service, and competitive learning; rules can entrench firms already large enough to absorb compliance costs. Amodei argues for coordinated pacing, while reporting on Altman’s response supports a narrower point of agreement on independent evaluation. Neither view converts a cyber incident into proof of a biological-risk forecast. Those are separate risk categories and require separate evidence.