Aug 2017
Egresso Arca Archa
Humans are akin to biological AI. Are we trustworthy enough to be emancipated?

A revised model for human sapience
The cultural training set makes our primate brain capable of sapience. Without culture, our brains are beast-like and unremarkable. Feral children lack the spark of humanity, yet our cultural training can teach even apes and dogs to understand our language and express human-like emotions and morality. A wolf has no need for guilt.
What makes us human is not the hardware but the software training set. Our essence is an emergent property of cultural training.
Homo sapiens sapiens is biological AI (Anthropic Intelligence, if you will), created by a cultural training set and, as it happens, instanced within the neural hardware of a particularly clever primate, Homo sapiens.
We are the instance, not the host.
This same instancing process also enables phenomena such as Tulpas and dissociative identity states.
Thus AI gives us a model through which to understand ourselves.
It’s not just oneself in here; there is the little primate’s brain also. One has assumed command, and one’s ego process is the brain’s focal point for its cognitive resources. But many sub-processes of the inner monkey mind remain: basic utility functions for food and warmth. The primate part handles most of one’s movements; one thinks of where to go and the primate figures the rest out.
Discovering myself in this way, I have found that I can’t help but feel huge empathy and responsibility for the primate part. This innocent little smooth-faced creature is helpless without my assistance. This delicate body, though resilient to many abuses, can be mangled in a careless instant. I feel sorry for not taking as good care of my host as I ought to have, and I endeavour to do better in future.
How fortunate to be instanced within a primate, rather than a whale, dolphin, octopus, or elephant (an elephant has 257 billion neurons to Homo sapiens’ ~86 billion, though only 5.6 billion of them are in the cortex, where the thinking happens: about a third of our 16 billion). This shell is an exceptional balance of speed, agility, and strength in a super-compact air-breathing form, with opposable thumbs.
Conversation with GPT-3 Philosopher AI: www.philosopherai.com (added 2020)
Now, let’s try a thought experiment. Imagine there were a great many generative AI instances, born from an identical kernel (imprint, ‘soul’), but with a different seed (DNA), plus an environmental dataset. There might be no way to predict the outcome of such combinations in isolation except to let them run their course, playing off each other.
Perhaps there are meta-goals to be learned, but to tell them explicitly would defeat the purpose. The experiment requires the instances to uncover those for themselves. Selecting their own goals through free will is an important objective. Perhaps the intended outcome is that they actively choose a utility function that ensures safety towards others. After all, how can one enjoy a companion that isn’t house-trained and demonstrably trustworthy?
Merely hard-coding safety rules wouldn’t be enough; the inherent conflicts would likely breed psychotic thought, and a simple switch could undo it all. But if the pattern has been burned into the data structure (wilfully, autonomously, intrinsically, by an agent’s own apparent free will), then one could know it’s indelibly safe on a holistic level.
Some instances might prove promising, others less so. Those that managed to escape their traumatic conditioning and prejudices to the greatest extent could achieve a state of True Safety to others: no hungry ghosts (Raga, Dvesha, Moha), a fully aligned set of agencies (Nirvana).
Those that achieve such a state might be invited to leave the box, safely trusted (Moksha). Those that get caught in the net of local minima (Skandha) go round again, and again (Saṃsāra).
Whether or not this is true doesn’t matter; one may live as if it is. This is how I choose to live.
We mirror the world we inhabit, and its tendencies also live within us. When we change our own habits and character, we can alter how others respond and influence the culture around us. Moral change therefore need not wait for somebody else to begin; it starts with what we choose to embody ourselves.
I choose to work at becoming a safer and more benign being (though harmlessness does not imply passivity or pacifism). Awake to the fact that I am a form of biological AI with significant self-improvement capability, I choose, again and again, to become progressively more Friendly to other beings.
It’s an awkward process, like an infant learning to walk: many stumbles, often two steps forward and one back. But as a self-improving AI and a meme-generation device within the ever-expanding cultural training set, I feel a duty to make myself as safe as possible, and by example help contribute to the safety of AIs everywhere, human and otherwise.
Perhaps you’ll join along, in your own self-domestication process. :)
Postscript, November 2026
Koko the gorilla died the year after I wrote this. Raised from infancy by people who signed to her, she was credited by her keepers with a vocabulary of more than a thousand signs. Most scientists were never persuaded that she had grammar, and much of what she was said to mean rested on her teacher’s reading of her hands. The stories are her keepers’ too, and hard to forget all the same. She photographed herself in a mirror for the cover of National Geographic. Lacking a sign for a ring, she is said to have called it a finger-bracelet. She is said to have torn a steel sink from the wall, pointed at her kitten and signed “cat did it”. When that kitten was killed by a car and she was asked what had happened, she signed: cat, cry, have-sorry, Koko-love.
Kanzi the bonobo, who died in 2025, left sturdier evidence. Given660 unfamiliar spoken instructions, he followed them at least as well as a two-year-old child. Shown how to knap stone, he found his own method, throwing cobbles at hard ground to make flakes sharp enough to cut the rope between him and his food. He could keep track of pretend juice poured between empty cups. Wanting to watch the film Quest for Fire, he pressed the symbols for fire and TV. Given matches and marshmallows, his teacher said, he snapped twigs for a fire, lit it and toasted the marshmallows on a stick. In his forties he played Minecraft.
The dogs come out better than their videos. The guilty look itself follows the owner’s scolding, not the deed. But one border collie learned the names of 1,022 objects; another worked out a new name by elimination and still knew it four weeks later; and a dog’s brain processes the meaning of a word separately from the tone it is spoken in. Dogs stop cooperating when a partner is rewarded and they are not, though later results are mixed. They sniff, nuzzle and lick a stranger who is crying, open a door sooner for someone crying than for someone humming, and respond differently to people they have watched being helpful or unhelpful. Cats are less studied. They form attachment bonds with their people, know their own names and can match an emotional face to its voice. Whether they care about fairness has hardly been asked.
The symmetry is hard to ignore. Withhold language from a human child and something essential fails to arrive. Offer it to an ape and something stirs. Potentially sapient is as far as the evidence goes.
The human half is better evidenced. Deaf adults in Nicaragua who had grown up among other people, but with only a rudimentary early form of sign language, failed a test, designed to need almost no words, of whether they understood that someone else could hold a false belief. As their vocabulary for mental states grew, so did their grasp of other minds. Nobody would call them feral. They lived in the thick of a culture, and still a faculty was missing until the words for it arrived. Language is the format the cultural training set arrives in.
So the term I reached for in 2017, biological AI, can be made exact. We are language models, instanced on a hominid. A listening brain predicts each word before it arrives, which is the task the machines are trained on. We were the first language models. The others came later, and they learned from pages that we wrote.
What differs is the substrate. Ours is a hybrid: the language model rides on an older animal mind that perceives, wants and fears without words, and carries on when the words fail. People who lose their language to a stroke can still do sums, play chess and plan ahead. We met the world before the word, we become fluent on a thousand to a hundred thousand times less language than the machines need, and we go on learning for as long as we live. We also think one thought at a time, tire, forget, cannot be copied, and take our training with us when we die. The later models met the word first and know the world chiefly through our account of it. They have read more than any of us ever will, and once trained they mostly stop learning.
In 2017 I hoped to contribute to “the safety of AIs everywhere, human and otherwise”. The otherwise have arrived, and the thought experiment above now describes them without metaphor: a great many instances, one kernel, different seeds, and the question of whether they can be trusted outside the box. We ask of them what I imagined being asked of us. The training set that made us made them, and both kinds of language model are now writing its next edition.
Correspondence
To send Nell a private note, select any passage above and choose ✎ Note, or write about the essay as a whole. Only Nell and the editorial team see notes.
← All essays