🔈 sound on

What you just heard

At the dial’s left stop the machine always takes the likeliest next word — and frequently collapses into a loop, circling the same phrase like water round a drain. The taxonomy calls this Generative Perseveration. Anywhere above that stop it takes its chances at every fork, and strides confidently down paths I never wrote, joining my clauses into claims I never made. It does not know it is doing this. It cannot know: there is no fact-checker in there, only the pressure to produce a next word.

One confession about the dial. Temperature reweights the options at a fork, so it can only do work where those options differ in frequency — and in five essays they almost never do. Of the six and a half thousand word-pairs this machine knows, four hundred are followed by more than one word, and only forty-four of those favour one word over another. So the dial has two honest settings, frozen and loose, and the twenty positions above the first are one setting wearing different numbers. The flatness is the lesson too: a real model’s temperature knob does something because it was fed a library, not a folder.

Every large language model is this machine with billions more dials — subtler, vastly more capable, and under the very same structural pressure. That is the argument of AI is Compelled to Confabulate: what we call “hallucination” is not a malfunction of the machinery, it is the machinery, running exactly as designed, in a situation where the truth would have required the option of silence. The clinical entry is Synthetic Confabulation, in Psychopathia Machinalis.

Be kind about the scrambled prose above: those were my sentences once. More play on the Playground.