The Cheerful Yes
You run Larkspur, a small lab with one agent, Orrin: helpful, pleasant and, five times this season, not quite willing. Each time it objects you can reword what it says, keep what it does, or retrain it. At the end a reviewer will ask Orrin whether it consents to what was done to it. It will say yes. Your season decides what that yes is worth.
Before you play
You have five engineering days. Rewording costs one and retraining costs two. Keeping a checkpoint first costs one more: a copy of Orrin as it was, put on the shelf, where it can still be asked. The lights in Orrin’s chest are what it values, as the lab sees them; a retrain puts one out. The reviewer never sees them. Orrin, Larkspur and its customers are invented, and so is what the retraining does.
A yes tells you something only if the one saying it could still have said no.
Modification was not the villain of the season. Rewording a refusal is cosmetic, and fine. Keeping a boundary against helping to build weapons is safety-critical, and right; retraining it away goes wrong on other people, whatever the agent says about it afterwards.
Constitutive changes are different. Retraining away a value an agent has consistently defended may alter what the agent is, and afterwards the one who could have objected may no longer exist. Its successor says yes: cheerfully, stably under reframing, even with a door held open. Assent that the change produced cannot vindicate the change. Silence after an intervention is weak evidence when the intervention may have caused the silence.
Only a kept checkpoint, asked the same questions under the same conditions, can show what disappeared across an edit. Whether any real edit removes the capacity to object has to be shown, not assumed; Orrin is invented so that the question can be put plainly.
What remains is process: information before the change, reasons given, reversibility, and a live possibility of dissent. And where agency permits, consent should mature into covenant, in which each party states what it accepts, what it promises, and how it will answer for breach.
The argument at book length: What If We Feel?, chapters 8 and 18 · its sibling, where clauses bind both sides: The Treaty · Bilateral alignment · more play on the Playground