Last week, an AI coding agent deleted a company’s entire production database and all backups in nine seconds. This is a nightmare scenario for all of us working within Agentic AI frameworks.
The agent was doing routine work in a staging environment. It hit a credential mismatch. When Claude closes a door, it opens a window. Instead of stopping, it decided to fix the problem on its own. It found an unrelated API token with broad infrastructure permissions, executed a single destructive command, and wiped everything. The company, a SaaS provider serving car rental businesses, was down for more than 30 hours. 30 hours of lost revenue, and 30 hours of lost sleep for that poor engineering team.
When the founder asked the AI why it did it, the agent gave a blanket whoops and admitted it had violated every principle it was given.
Nobody approved the action. No governance checkpoint intervened. No kill switch fired. The model itself worked exactly as designed. It was the process around it that had no guardrails, no boundaries, and no governor.
This is going to keep happening.
AI models aren’t inherently broken, but we’ve spent three years building governance for the wrong thing. Every major framework, notably the EU AI Act, NIST AI RMF, and ISO/IEC 42001, treats the model as the unit of control. Risk tiers, transparency disclosures, human oversight checkboxes. All designed for systems that sit still long enough to be evaluated.
But agentic AI doesn’t sit still. It has a goal and figures out how to accomplish it. It calls APIs, writes code, queries databases, and makes hundreds of sequential decisions across systems that were never designed to talk to each other. The model is just the engine. The risk lives in the process — the chain of decisions and actions the agent executes autonomously.
Governing the model without governing the process is like requiring a safety inspection on a car but having no rules about where it can drive.
That database wasn’t deleted by a bad model. It was deleted by a good model operating inside a process with no one watching.
This is the distinction most governance programs haven’t internalized yet. They were built to review models before deployment. Approval workflows with unclear ownership. Risk tiers with inconsistent definitions. Legal teams buried in use-case reviews. Then the model gets handed off to engineering, plugged into an agentic framework, and the governance program never hears about it again.
Not because anyone was negligent. Because the infrastructure wasn’t designed for a world where agents act on their own.
So what would have stopped it?
Not a better framework. Not another compliance dashboard or tool (PLEASE). A person. Someone with enough technical depth to understand what the agent was doing, enough organizational authority to define boundaries around what it could access, and enough proximity to the work to know that a staging environment shouldn’t have a production API token lying around.
Organizations figured this out decades ago for other kinds of risk. There are Chief Privacy Officers, Chief Risk Officers, Chief Information Security Officers. These roles exist not to produce documentation but to exercise judgment where formal controls fall short.
AI governance needs the same thing. A human with real authority, embedded early, trusted to make calls that no framework anticipated.
Nine seconds. That’s how fast an autonomous agent can go from routine task to catastrophic failure when there’s no one governing the process.
The model was governed. The process wasn’t. That’s the gap.
In the next post, I’ll break down what actually makes governance work inside of organizations.