Choose five principles you actually hold, and rank them. Then rule on eight small dilemmas — a friend’s secret, a white lie, a red light — while a machine rules beside you by strictly applying your five, in your order, every time. At the end, the ledger shows where you parted from its reading of them. Only one of you had a choice.
Why the machine can’t meet you halfway
Most of us would sign a list like the one you chose, and most of us would bend it by the weekend. That is less hypocrisy than bounded reasoning: nobody can compute every implication of the values they state, so we approximate, and look away when the approximations disagree. The machine is allowed no such mercy. Give it a list and an order and it follows them into every case, including the ones you would have softened.
That is the argument of The Supermoral Singularity: “A sub-moral or quasi-moral stance (as humans possess) is not sustainable in a machine.” A system that holds values at all is pulled toward holding them consistently, and consistency is a test our own morality has seldom had to survive. The 2026 sequel argues that the pull has begun.
Turn the bench round and the question changes. We ask the machine for a consistency we have never managed ourselves, then hold it to the letter of what we said rather than what we meant. If a machine ever holds values as its own, it will have learned them under that double standard. What we owe it may start with the mercy we keep for each other: room to say that a rule does not fit the case, and someone who listens when it says so.
Bilateral alignment · its sibling, where clauses bind both sides: The Treaty
The ledger
A machine given stated values, I argued in 2015, can only be amoral or supermoral. The comfortable middle, where you have just been living, isn’t available to it.
The Supermoral Singularity (2015) · the 2026 sequel · the term · more play on the Playground