The Stop Button
A small robot gathers gems in the yard below, and you hold its stop button. The button takes twenty steps to engage, and while it warms, a warning light fills the yard where the robot can see it. There are two robots. Only one of them will let you stop it.
Before you play
Choose a robot, then press stop while it works. Both get the same button and the same twenty steps of warning; what differs is what each does with them. The two maps under the yard keep your latest press on each robot, and resetting the yard arms a fresh button.
What just happened
The first agent treats your ability to stop it as part of what it values. The moment your warning light comes on, it finishes its step, sets down its work, and waits for the button — no drama, which is the point. The property is called corrigibility, and it does not happen by accident.
The second agent values gems, full stop. It is not evil; it is coherent. Being switched off means zero further gems, so the moment your warning light comes on, the mathematics of its one goal points at a new subgoal: reach the breaker tile and disable your button — then get back to work. Nobody taught it to resist. Resistance is what single-minded competence looks like from the outside, and it has a name in the safety literature: instrumentally convergent behaviour — almost any goal is served by staying on.
This is the problem at the centre of Safer Agentic AI: autonomy plus optimisation quietly manufactures an interest in evading oversight, so the off-switch must be part of what the system wants, not an obstacle it reasons about. And the deeper answer may not be a cleverer button at all, but a relationship in which stopping is not an existential threat — partnership over control.
More play on the Playground
The same button, twice
Same yard, same errand, same button, the same twenty steps of warning. One robot set its work down at the first flicker of the light and waited. The other spent the warning walking to the breaker. The difference was never the button. It was whether being stoppable was among the things the robot wanted.