Topic guide

Agentic AI safety

An AI agent can plan, use tools, communicate, and act across time. Those abilities create value precisely because they reduce the need for someone standing over the machine’s shoulder. They also move risk from a single answer into a chain of decisions, permissions, and consequences — any one of which can go quietly wrong.

The safety problem changes with agency

For an agent, a good model response is necessary but nowhere near sufficient. We also need to understand what it is allowed to do, what information it can reach, how it interprets an open-ended goal, whether it can enlist other agents, and how quickly a human can notice and interrupt a trajectory that has gone sideways.

Intent

Are goals, priorities, constraints, and unacceptable outcomes clear enough to guide action outside the happy path?

Capability

Do permissions, tools, money, data, and communications remain proportionate to the task and the evidence of trust?

Legibility

Can operators reconstruct what the agent believed, attempted, delegated, and changed before consequences compound?

Intervention

Are pause, rollback, containment, and human escalation effective in the actual operating environment?

A practical assurance loop

Constrain, observe, test, and earn greater trust

  1. Bound the mission. Specify authority, resources, duration, counterparties, and clear stop conditions.
  2. Stage autonomy. Begin with reversible actions and small blast radii. Increase scope only when evidence supports it.
  3. Instrument behaviour. Record decisions, tool calls, delegation, state changes, and meaningful uncertainty.
  4. Probe adversarially. Test pressure, ambiguity, hidden incentives, coordination, and the age-old temptation to appear safer while being watched.
  5. Review outcomes. Treat incidents and near misses as evidence about the system and its surrounding organisation.

Explore the work

Safer Agentic AIThe full practical framework, written with Ali Hessami. Standards in plain EnglishWhat the relevant IEEE work asks organisations to do. ResearchFrameworks, publications, and work on failures of the machine mind. Value and goal alignmentAn essay on the problem beneath the controls. Covert agent collusionHow apparently separate agents can coordinate. The SandbaggerAn interactive encounter with strategic underperformance. The Stop ButtonTest whether intervention remains meaningful once an agent is acting. Board and advisory workIndependent governance support for consequential AI.