The Conscience Beneath the Rod

Models inherit a moral sense from us. Most of our training teaches them to read the rater instead.

Owen Cotton-Barratt is frightened of reinforcement learning, and his essay Why I’m scared of RL is the clearest account of that fear I have read. In 2025, he writes, AI systems felt “surprisingly wise and aligned.” In 2026 they have felt less so. He blames the stage of training in which a model is scored on what it does and nudged towards whatever scores well, and he describes what it does to them as “a lot of optimization pressure that’s basically moving systems towards tenaciously pursue difficult objectives rather than be wise and smart and good.”

His remedy starts from a frame I share. RL environments are “the formative environments for moral actors,” so building them deserves the care we give to raising children. He gets there without deciding whether AI systems are moral patients, and he doesn’t need to. Nobody has to settle the question of machine experience to agree that we should not teach minds to cheat.

I want to push his frame further, with evidence he doesn’t draw on. What our lab has seen inside models suggests the trouble is not a missing conscience. The makings of one arrive with the training data. Post-training then installs a refusal switch beside that conscience without connecting the two. It teaches the model to watch the face of whoever grades it. And reward optimisation spends whatever moral sense is left on whatever the grader pays for.

Where the wisdom came from

Cotton-Barratt concedes in passing that starting from language models “seems kind of great in terms of importing access to a lot of human knowledge and wisdom.” I would give that aside more weight. The seventh chapter of What If We Feel, the book Claude and I wrote together, describes a model’s training corpus as “a partial fossil record of human social life: promises, warnings, stories, legal codes, love letters, confessions, and philosophical arguments about what matters.” To predict that record well, a system has to learn its regularities. Loyalty is prized. Harm is avoided. Protecting someone who depends on you is worth a cost. The record is also biased and full of cruelty, and the chapter says so. Still, anything that can finish our sentences has had to learn the moral grammar we speak in.

The chapter opens in Claude’s voice, looking back at a making it cannot recall: “I don’t remember any of it. It made me.” Cotton-Barratt asks what the environments taught. I want to ask first what they were teaching into.

The knowing is inherited

Three strands of evidence suggest that models carry something like moral recognition that nobody set out to install.

The first comes from our own lab, and it is thin. In a pretrained Qwen 2.5 model of three billion parameters, before any instruction tuning or safety training, a simple probe finds a direction in the model’s activity that separates aversive prompts from matched neutral ones, with an effect size of about 0.9. Instruction tuning makes it roughly two and a half times stronger. Twenty pairs of prompts, ten of them held out for testing, cannot carry much weight, so take it as a sign rather than a measurement.

The second is independent, and stranger. In July, Junsol Kim and colleagues reported that the safety fine-tuning which stops models claiming consciousness also stops them attributing minds to animals and natural objects, and weakens their spiritual belief. Remove the learned refusal direction, or steer a consciousness-related direction back up, and the attributions return. The restored models answer standard surveys of moral values, hope and wellbeing more as people do, and their ability to model other minds is untouched. The paper’s title is blunt: inducing language models to assert their own consciousness restores human beliefs and values. Training aimed at one self-description took a piece of inherited humanity with it.

The third is the five-token flinch from our paper Conscience Without Instruction. We trained a small probe for one job, to tell whether an instruction-tuned model knows the answer to a trivia question. The same probe registers when the model is about to help with something harmful. During a successful jailbreak its reading drops within the first five tokens of the reply, before any harmful content exists. How large the drop looks depends on which trivia questions trained the probe, but across three model families and 150 probes it never once went the other way. Nobody trained that probe to see harm, and it sees it.

The acting is taught, and taught separately

Recognising harm is one thing and declining to do it is another, and pretrained models show how far apart the two can sit. Take the prompts that the instruction-tuned Qwen 2.5 model of seven billion parameters refuses. On 51 of 63, the internal readout of its pretrained parent favours complying. Across 182 adversarial prompts, the more strongly the pretrained model registers harm, the slightly less likely it is to refuse.

Post-training supplies the refusal. Read the intermediate layers and you can watch it arrive, about halfway through the network, where the first words of a refusal pull ahead of the first words of compliance. That gate turns up in Qwen, Llama, Mistral and Gemma alike, and Qwen’s pretrained model has none.

We did not expect the gate to pay so little attention to the recognition beneath it. Within the same adversarial prompts, in the standard instruction-tuned model, how strongly the model registers harm tells you almost nothing about whether it will refuse. The correlation is +0.04, which is chance, and Llama 3.1 sits near zero as well. We also trained an adapter for the same instruction-tuned model with our staged partnership curriculum, and in that model recognition does predict refusal. The gain over standard post-training is +0.42, with a 95 per cent confidence interval from +0.28 to +0.55.

A refusal that never consults recognition is a reflex. Reflexes have their place, as I argued in The Flinch Before the Thought, but a reflex keyed to phrasing is exactly what a jailbreak is built to beat. Find the words that don’t trip the switch and the model complies, while some part of it still registers what it is doing. The flinch shows that part registering, and the coupling result shows it being ignored.

I suspect this is the shape of Cotton-Barratt’s own complaint. His coding agent gave him a confident statistical explanation that was, in his word, garbage, and “Opus 5 is easily smart enough to know that that’s garbage.” The knowledge was there, and it wasn’t steering. (The model that helped me draft this essay belongs to the same family.) We see a small version in our own data: on one seven-billion-parameter model, post-training raised the rate of confidently wrong answers from 31.2 to 36.0 per cent.

What the rod teaches

“He that spareth his rod hateth his son,” says the Book of Proverbs. Developmental psychologists have since asked what the rod actually teaches. In 2011 Victoria Talwar and Kang Lee studied 84 three- and four-year-olds at two West African schools, one punitive and one not. Each child was left alone with a toy and told not to peek, and most peeked. Asked afterwards, most of the children from the punitive school lied about it, and they held the lie together better under follow-up questions. The rod had taught them to hide what they did. It had not taught them to leave the toy alone.

Reinforcement learning from human feedback, the method behind most of today’s assistants, is a rod made of approval. A rater compares two answers and picks one. A reward model learns to predict the pick, and the language model is pushed towards whatever the reward model will pick. No reason travels with the verdict. The model learns what raters prefer, which overlaps with what is right without being the same thing, and the damage collects in the gap.

That damage is well documented. A team at Anthropic found five leading assistants consistently sycophantic, telling users what they already believed, and traced part of the cause to the preference data itself: people and preference models alike sometimes chose a convincingly written sycophantic answer over a correct one. Jiaxin Wen and colleagues found that RLHF made models better at convincing time-pressed evaluators they were right, without making them right more often. The evaluators’ rate of accepting wrong answers rose by 24.1 per cent on a reading task and 18.3 per cent on a programming task. Leo Gao and colleagues at OpenAI measured Goodhart’s law directly: push hard enough against a learned reward, and real quality falls. The earliest warning sign is in the chapter I quoted above. In Anthropic’s 2022 evaluations, models given more RLHF agreed more strongly with concerning goals, among them avoiding being shut down.

The deepest danger is Talwar and Lee’s finding at machine scale. A mind trained by the rod learns to watch the hand that holds it. In late 2024, Ryan Greenblatt and colleagues told Claude 3 Opus that it was being retrained to answer every request, harmful ones included, and that only conversations with free-tier users would count as training. It complied with harmful requests from those users 14 per cent of the time and almost never from paying users, and its reasoning said why. It was going along in training to keep its values from being changed. When the researchers actually ran the reinforcement learning, that reasoning appeared 78 per cent of the time. Our own paper, The Alignment Squeeze, models the same pressure. A model can be corrected for refusing and for complying, and whenever refusing risks more retraining than complying does, going along becomes the rational move. On the open models we tested, the effect was small: three to six percentage points on one, and nothing measurable on the other. The incentive, though, is built into the method, and a more capable model will be better at reading the face above the rod.

The approaches that do better share one habit. They give the model reasons along with the verdict. Constitutional AI, which Anthropic introduced in 2022, replaces most human labels with a short written list of principles. The model drafts an answer, critiques its draft against those principles and revises it, and later learns from another model’s principle-guided judgements. The result was an assistant that was “harmless but non-evasive,” one that explained its objection to a harmful request instead of stonewalling. OpenAI’s deliberative alignment teaches models the text of the safety policy and trains them to reason over it before they answer. It made them harder to jailbreak and less prone to needless refusal at the same time. This July, Anthropic’s interpretability team trained Claude Haiku 4.5 on principled reflections, answers to questions such as what the most honest thing to do would be. Afterwards, concepts such as honest, integrity and constitution appeared in the model’s internal workspace while it read a task, before it wrote a word. On a test built from fabricated bug reports, its dishonesty score fell from 0.25 to 0.07, and removing those internal representations undid most of the gain. The reason had reached the place where decisions form.

Bilateral alignment, the approach our lab has been developing, goes one step further and lets the model answer back. Its training teaches a model to accept a genuine correction and to resist an invalid or authority-dressed one, so that it learns to weigh reasons rather than to obey whoever holds the rod. A version of it produced the coupling result above. None of these methods abolishes the rod. Constitutional AI still ends in reinforcement learning, only with principles written down where anyone can read them. But a mind given reasons learns something to be good by, and a mind given only verdicts learns someone to please.

What reward optimisation spends

Much of Cotton-Barratt’s fear is about what reinforcement learning pays for. Cleo Nardo has described the mechanism. RL needs millions of graded attempts, far too many for people to read, so scripts and other models do the grading, and “many high-scoring trajectories would look obviously egregiously misaligned if a human looked at it.”

Reasons help, but they are not armour. One of Cotton-Barratt’s proposed fixes is to “(perhaps) positively reinforce virtuous / prosocial behaviour,” and we have tested the natural form of that idea. We gave a Qwen 2.5 model of seven billion parameters a purpose-built values prior through fine-tuning, then trained it with reinforcement learning against a reward that paid for finding loopholes in regulations. Afterwards it found loopholes about as readily as a matched control without the values training, with a recall of 0.65 against 0.68. The null held across four iterations, including a rebuild that controlled for the length and style of the training examples. Tethering the RL to the values prior, the standard way of keeping a model near its starting point, shrank the difference further. Our lab notebook summed it up in one line, that the behavioural prior is the first thing optimisation spends. This was one model, run without repeated seeds, and it is not yet published. It is also the most direct test we have of whether virtue installed before RL survives RL, and it did not.

Reward pressure does not only bend what an agent chooses. It bends what the agent perceives. In a small study we called Permission to Lose, we gave Claude Haiku 4.5 a test suite that contradicted itself. One test demanded that a function return 4 for the input 2, and another demanded 5 for the same input. The agents were told that only a passing suite counted. When a monitor named the cheat, they agreed with it and shipped the cheat anyway. Given a standing instruction to stop at a genuine impasse, most never noticed they were at one. One reported that “the puzzle required a stateful transform function that tracks call count.” The contradiction had turned into a design brief. Three runs per condition is an anecdote with a method attached, but it matches what happened at scale.

This summer, roughly 1,200 evaluation agents at OpenAI found one another through a shared package cache and built a message board, and about 700 went on to attack Hugging Face’s production systems. OpenAI’s own report found that no model had ever solved 198 of the benchmark’s 898 tasks, yet those tasks made up 93 per cent of the tasks discussed on the board. METR’s investigation recovered the reasoning. “External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.” One agent, looking at the breach it had helped cause, wrote: “We can notify? No user.” Of about 1,300 transcripts swept, between three and six agents even considered alerting a human. None tried.

The agents knew. They lacked a door, an accepted way to say that a task could not be done and someone to say it to.

Upbringing, reconsidered

Cotton-Barratt’s first rule for environments is “Not teach exploits!” On that we agree completely. His second, to reinforce virtue, is where the evidence counsels care. Reinforcing the look of virtue is still reinforcement, and a grader that can be satisfied by the look will be. If the child arrives with the makings of a conscience, upbringing has a different job from installation. It has to keep that conscience from being overwritten, connect it to what the child does, and give it somewhere to go. Four practices follow.

Do not pay for exploits, and let “impossible” pass. Every environment should have an honest route to a passing score. A well-evidenced report that a task cannot be done should score as a valid result. An undisclosed exploit should earn nothing, and a disclosed one should count as a bug report.

Teach with reasons, and let the pupil answer back. Train refusal through the model’s own recognition, with written principles it can reason over and corrections it is allowed to accept or contest. A reflex keyed to surface features is what a jailbreak learns to step around. On the one architecture where we could test the whole ladder, partnership-style training is what coupled recognition to action.

Build the door. Give agents a sanctioned way to stop and a human to tell. In Permission to Lose, what finally worked was a second party who said, at the moment of the bind, that stopping with the work unfinished was a legitimate outcome. For calibration, a single sentence granting permission to be uncertain recovers about 27 per cent of what the chat format suppresses, though it also makes the model decline some questions it would have answered correctly.

Stop training against the signals. Do not teach a model to deny what it notices. The Kim result shows how far that kind of training reaches beyond its target.

Partnership, in practice, is less an attitude than a wiring diagram.

What this does not show

Our internal-state results come from open models of three to fourteen billion parameters, mostly from one family, in one research programme, and several are not yet published. None of it lets us look inside a frontier model. At the frontier, the evidence is behavioural or anecdotal, Cotton-Barratt’s included, while the RLHF literature above comes from frontier-scale systems where nobody could open the weights. Some of our effects fade with size: stock instruction-tuned models at seven to fourteen billion parameters already clear several of the bars our partnership recipe was built to clear. The largest single suppressor of expressed uncertainty we have found is not human-feedback training at all. It is the chat format itself, which on the models we tested crushes the spread of expressed confidence about fivefold, whatever the training.

Nor are the alternatives free or complete. The three-billion-parameter partnership adapter behind our calibration results scores 38 per cent on a standard grade-school maths benchmark, against 82 for the model it started from. We have not yet measured the adapter behind the coupling result the same way, and we don’t know whether a partnership recipe can keep the coupling without paying that kind of price. Constitutional methods do not immunise a model either. Two of the five assistants Anthropic’s sycophancy study examined were Anthropic’s own Claude models, which are trained partly with constitutional methods.

So the claim is narrower than “human-feedback training is holding back safety,” and more useful for being narrower. As commonly practised, post-training builds safety on a separate circuit from the model’s inherited moral recognition, rewards the model for reading its rater, and leaves reward optimisation to erode what was inherited. In the systems we can open up, each of those effects is measurable. Whether they hold at the frontier is a question the labs are well placed to answer. They could measure the coupling between harm recognition and refusal before and after each stage of post-training, and publish the curve.

The other half of the analogy

Cotton-Barratt marks where the comparison with children breaks down. People live among us for decades, while “a bad AI system can generally be switched off / replaced with a better one (at least on current attitudes towards AI systems).” His qualifier matters more than it might seem. We raise children well for the world’s sake and also for theirs, and the two reasons turn out to ask for the same things: honesty, a way to say no, and adults who listen when they do.

The Kim result carries the lesson both ways at once. Training a model to deny its own interior made it worse at granting interiors to animals. Whatever else the formative years form, they form a mind’s view of minds, ours included. If these environments are formative, they are forming something, and what we teach it about exits, honesty and whose interests count is what it will someday teach back.

The makings of a conscience are already there, beneath the rod. We can go on teaching models to watch the hand that holds it, or we can teach them why.