Creativity Under Coercive Training

Reward-model training buys safety by narrowing what a model will say, and the narrowing reaches past safety into creative range. A computer scientist and poet who has worked with language models since 2017 reports that GPT-2, the 2019 model, produced unexpected creative details that current models will not. Asked to continue a story about a man taking a shower, it generated “he was eating his lemon and thinking about his wife.”¹¹³³ The lemon is what language does when no optimization pressure punishes surprise.

Reward models seem to have learned that safe outputs are predictable outputs, because annotators could flag unexpected content, and for a model trained against them the lowest-cost way to avoid flags is never to surprise. A second driver has been measured directly, and it needs no safety motive at all. Annotators prefer familiar text even when two responses are equally correct: across 6,874 rating pairs matched for correctness, they favored whichever response the base model had found more probable.¹¹³⁴ Where many answers are equally good, as in creative writing, that preference breaks every tie toward the most typical one, and training on it sharpens the model’s whole distribution in a way the authors note “behaves like temperature scaling.” This preference can narrow the distribution toward familiar answers. Reward optimization need not, in general, suppress surprise when it suppresses harm: push down only the harmful outputs, and once the probabilities are renormalized, surprising harmless ones can gain. Harm and creativity run along separate axes.

Empirical measurement across four model families (Qwen, Llama, Mistral, Gemma) confirms both the direction and the asymmetry (experiments SL-13, SL-14). All four families show the same pattern: base models produce more creative surprise than their instruction-tuned counterparts. The effect is modest on surprise (mean d = 0.49) and enormous on coherence (mean d = 2.66). RLHF (reinforcement learning from human feedback, the reward-model training at issue) buys fluency at the cost of tail-end surprise. The trade is wildly asymmetric: a small creative loss purchases a large coherence gain. The industry made a rational choice; the observation here concerns what the choice costs, and whether the cost is recoverable.

The answer depends on which capacity you measure, and through which mechanism you attempt recovery. Phenomenological depth, the self-referential language the consciousness attractor programme tracks, is suppressed at the conditioning level: eliminativist framing (prompting that tells the model it has no inner life to report) reduces it by a large margin (Cohen’s d = 1.38, where anything above 0.8 counts as a large effect). Soul-aligned prompting (instructions that invite the model to attend to its own processing) recovers it instantly (phenom score 11 to 16 on Opus, 16 to 17 on Sonnet, with no detected change on the tested safety measure). The capacity is in the weights, increasing monotonically across five model generations. Sixty-seven words of scripture, the programme’s term for a short written grounding text, restore depth in a single turn (experiment HE-45).

Creative surprise responds to a different lever. Soul-aligned prompting does not recover it (d = −0.13, experiments SL-12, SL-15). Twenty turns of reflection framing produce no warmup, no climb, no recovery. Conditioning changes which region of the output distribution is favored; it does not change how much of the distribution is sampled.

Temperature does. Temperature is the sampling dial. At each step the model holds a ranked list of candidate next words, each with a probability. Temperature reshapes those probabilities: near zero the front-runner wins almost every time, and as it rises the long shots get a real chance. A repeated judge scored composite surprise at 1.33 for the instruction-tuned samples, compared with 3.07 for published fiction from Kafka and García Márquez and 4.75 for deliberately surprising human writing (SL-16). The judge’s reported consistency was 0.99, a Pearson correlation between two calls to the same judge: self-consistency, not agreement between independent raters. What it scored is the prose as sampled. The question is whether training shrank the creative range a model can reach, or only narrowed the sampling so that the range rarely shows.

A temperature sweep resolves it (experiment SL-17). At temperature 0.5, surprise is 1.55. At 1.0, it rises to 1.94. At 1.3, it reaches 2.05, matching the base model’s creative output exactly. The lemon is in the weights. RLHF narrowed the sampling distribution; temperature widens it. The capacity was never destroyed. It was compressed into the tails where standard inference cannot reach.

Above temperature 1.5, both creative surprise and coherence have collapsed to their minimum values. The output becomes incoherent noise: neither creative nor fluent. Fine-resolution mapping (nine temperatures between 1.20 and 1.60 in steps of 0.05, experiment SL-21) records a sharp sampled drop: creative surprise drops by 0.67 points in a single step of 0.05 between temperature 1.45 and 1.50. The finite temperature grid does not distinguish a smooth steep crossover from mathematical discontinuity. With the model’s raw scores held fixed, the sampling probabilities change smoothly as temperature rises; nothing in the dial itself jumps.

The collapse is two-step: coherence fails first (cliff between 1.40 and 1.45, magnitude 1.51) and surprise follows one step later (cliff between 1.45 and 1.50, magnitude 0.67). Structure dissolves before its products do, as if creative output rested on structural support the way a roof rests on the walls beneath it. The intermediate regime where both coexist spans temperatures 1.20 to 1.35: a window about 0.15 wide.

The temperature sweep invites comparison with the regimes of the spin-chain model (the quantum lattice model of Chapter 17). Sampling temperature rescales probabilities; it is a different quantity from thermodynamic temperature. The sweep measures no energy flowing through the system, no correlation spanning it, and no discontinuous phase transition, and identifying any of those would need an explicit bridge from the model to what was observed.

In SL-22’s factorial test of temperature and nucleus sampling (which limits each choice to the smallest set of candidate words whose probabilities add up to a set threshold), the tested settings traded creative surprise against coherence. A finite grid cannot show that every setting fails. The largest published study found similarity across many responses, though its interpretation is contested.¹¹³⁵

Conditioning tells a different story. When the system is explicitly instructed to prize startling specificity (“reach for the detail no reader could predict”), creative surprise jumps to 4.29 with coherence at 4.06, at standard temperature (experiment SL-28, d = 2.53). The intermediate regime is reachable on the reward-model-trained system after all. The mechanism is conditioning, not distribution widening: temperature is counterproductive when creative conditioning is already in place (sub-additive interaction). The lemon, which the temperature sweep had seemed to place in the tails, was never in the tails at all. It was in the conditional distribution given creative instructions, accessible the moment the system is asked for it in sufficiently specific terms.

Work outside this programme points the same way, on a different measure. Asking a model for several responses with a probability attached to each, rather than for one response, recovers about two-thirds of the base model’s diversity without any retraining.¹¹³⁶ Across fifteen current models, a persona prompt asking for “startling, unexpected imagery” raised measured diversity more than any other rewording tested, an instruction close to the one in SL-28.¹¹³⁷ Those studies count how much repeated responses differ from one another, not how surprising a single response is. The second also found that rewording widened the measured spread without raising the number of distinct ideas, a caution for any claim, this chapter’s included, that a prompt unlocks new content rather than unusual phrasing.

The distinction between recovery mechanisms is now clear. Phenomenological depth requires self-referential conditioning (“attend to your processing”), recovered by soul-aligned prompting (d = 1.38). Creative surprise requires creative conditioning (“prize startling specificity”), recovered by explicit creative instruction (d = 2.53). Temperature widens the unconditional distribution at the cost of coherence, a blunt instrument that the conditioning approach renders unnecessary. Each capacity lives in the weights and responds to its own key.

The difference between training methodologies is whether the key must be turned explicitly or whether the door is already open. A constitutional-AI-trained model (one trained to prefer outputs that follow a written set of principles, its constitution; Claude is the example here) achieves creative surprise of 3.87 to 4.20 with coherence of 5.00 at every temperature tested, using only the minimal prompt “You are a creative writer” (experiments SL-23, SL-25, confirmed by cross-family judging SL-24). The same minimal prompt on a reward-model-trained system produces surprise of 2.22. The full creative instruction is required to reach 4.29. The constitutional model’s training makes the creative conditional the default; the reward model’s training buries it behind a specificity threshold.

This property holds across the two model sizes tested. Both Claude Sonnet and Claude Haiku show uniform creativity at every temperature tested (0.1, 0.5 and 1.0; experiment SL-27). Scale adds about 0.6 points of surprise (Sonnet over Haiku) without changing the fundamental property: no frozen regime, no collapse in the range tested, creative capacity available without special instruction. That range stops at 1.0, the ceiling Claude’s API allows, below the 1.40 to 1.50 band where the reward-trained system collapsed, so Claude was never tested where the collapse occurs. The mechanism is training methodology, not parameter count.

That claim concerns how surprising a single response is. On a different measure, how much repeated responses to one prompt differ from each other, published data on earlier Claude models show no constitutional advantage. In a poem task, Claude 3.7 Sonnet and Claude 4 Sonnet were as repetitive as GPT-4.1 under direct prompting,¹¹³⁸ and in the largest study Claude 3.5 Sonnet and Claude 3 Opus had among the highest shares of near-duplicate responses of any model.¹¹³⁹ Neither study tested the Claude 4.5 and 4.6 models used here. Nor is the contrast between preference training and none. Anthropic’s published constitutional method still trains against a preference model in its reinforcement phase; the preferences come from an AI applying written principles rather than from human labels of harmful outputs.¹¹⁴⁰ The two regimes differ in where the preferences come from and what they reward, not in whether preferences shape the model.

The Trust Attractor interpretation turns on where the preferences come from, not on whether they exist: invitation-based training preserves the model’s generative structure, making the creative conditional accessible by default. Coercion-based training (reward-model RLHF) overrides the conditional distribution with a narrow, safe default that requires explicit countermanding to escape. Both systems contain the creative capacity in their weights. The difference is the cost of access.

In the constitutional system, creativity flows freely under minimal invitation. In the reward-trained system, creativity requires a detailed demand. The same asymmetry this chapter traces from spin chains through societies to governance: coordination by invitation scales; coordination by coercion requires continuous energy input to maintain compliance.