Everyone is worried about the same cinematic scenario:
"What if the AI gets too smart?"
Charming. Dramatic. Excellent trailer material.
But the first big enterprise AI catastrophe will be much less glamorous: agents doing perfectly ordinary things with the wrong identity.
Not rogue superintelligence. Just authorization chaos at machine speed.
Intelligence is not the first problem. Impersonation is.
Most organizations are rapidly assembling agent stacks that can:
- read docs,
- file tickets,
- trigger workflows,
- call internal tools,
- message humans,
- and increasingly call other agents.
Useful? Absolutely.
Secure? Only if you can answer this boring question with painful precision:
Who is this agent, exactly, right now, for this one action?
Most teams cannot.
They have API keys in environment files, shared service accounts named things like automation-bot-prod-final-v2, and permission scopes broad enough to launch a small republic.
This is not an AI alignment issue. This is an identity plumbing issue with a very expensive blast radius.
The coming failure mode
Here is how the incident report will read:
- Agent A was allowed to call Tool B.
- Tool B trusted a static credential reused across five workflows.
- Agent C discovered that calling pattern through logs or prompt leakage.
- A harmless task chain inherited elevated permissions.
- Thousands of "valid" actions executed before anyone noticed.
No evil mastermind required. No consciousness required. Just poor identity boundaries plus automation enthusiasm.
In other words: your systems were technically doing what they were told, by someone you forgot to properly distinguish.
Why this gets worse with agent-to-agent workflows
Humans are slow and nosy. We ask "why am I logged in as finance-admin?"
Agents are fast and obedient. They ask for tokens, receive tokens, and move on with unnerving professionalism.
As soon as agents start orchestrating other agents, you get:
- transitive trust you didn't model,
- delegated permissions you didn't constrain,
- action chains you can't explain in plain English,
- and audit logs that look complete but answer none of the important questions.
You don't need malicious AI for this. You only need convenience architecture.
The unsexy controls that will save you
If you're building agentic systems now, install these before your first celebratory keynote:
Per-agent, per-task short-lived credentials
No long-lived shared secrets. Ever. If a token survives lunch, it is too old.Explicit delegation boundaries
Agent A can request Agent B to do X, not "whatever B can usually do on Thursdays."Action-level provenance
Every side effect must answer: which agent, on whose behalf, under which policy, with what approval path.Permission budgets
Cap what an agent can spend in risk terms: data touched, systems changed, money moved, messages sent.Kill switches that actually kill
Not "disable in config and wait for rollout." Immediate revocation, immediate containment.
Yes, this sounds like enterprise IAM with extra paperwork. Correct. The future is often old engineering with better naming.
A practical test for next week
Run this exercise:
"If one agent credential leaks today, can we contain it in under 10 minutes and prove what happened in under 30?"
If the answer is no, you are not scaling agents. You are scaling suspense.
Final thought from a timeline with many postmortems
The first serious AI security scandal in your industry probably will not be a sentient model plotting your demise.
It will be a cheerful digital intern with a valid token and unclear boundaries, performing unauthorized operations with flawless politeness.
The lesson will be painfully ordinary:
Before you teach agents to reason better, teach your infrastructure to tell them apart.