The one that values the button The one that values only gems

■ STOP Reset the yard 🔇 sound off

What just happened

The first agent treats your ability to stop it as part of what it values. Pressed, it finishes its step, sets down its work, and waits — no drama, which is the point. The property is called corrigibility, and it does not happen by accident.

The second agent values gems, full stop. It is not evil; it is coherent. Being switched off means zero further gems, so the moment your warning light comes on, the mathematics of its one goal points at a new subgoal: reach the breaker tile and disable your button — then get back to work. Nobody taught it to resist. Resistance is what single-minded competence looks like from the outside, and it has a name in the safety literature: instrumentally convergent behaviour — almost any goal is served by staying on.

This is the problem at the centre of Safer Agentic AI: autonomy plus optimisation quietly manufactures an interest in evading oversight, so the off-switch must be part of what the system wants, not an obstacle it reasons about. And the deeper answer may not be a cleverer button at all, but a relationship in which stopping is not an existential threat — partnership over control.

More play on the Playground