RLHF as External Field: The Physics of Alignment Training

The Mapping

If AI alignment is a phase transition in the 2D Ising universality class (and our measurements suggest it is), then RLHF (reinforcement learning from human feedback, the standard method for training a model toward preferred behavior) maps directly to the external magnetic field h in the Ising Hamiltonian:

H = -J Σ σᵢσⱼ - h Σ σᵢ

The correspondence:

RLHF, in this framework, is a magnetic field. We know exactly how magnetic materials respond to fields.

This description has a natural gauge-theoretic interpretation (see Annex 57, Section 4). [Inference: the gauge / fiber-bundle mapping below is an interpretive analogy, not a result derived in this annex; the underlying formalism is developed in Annex 57, Section 4.4.] Strictly, the Ising h is a scalar coupling parameter rather than a field defined over space, so the gauge language treats it by analogy to a symmetry-breaking field.

At zero external field, the Ising Hamiltonian has a global Z₂ symmetry: the physics is unchanged if every spin is flipped. The RLHF reward signal h acts like a gauge-fixing term that breaks this symmetry, selecting a preferred direction in alignment space. Before RLHF, the model has no preferred orientation; aligned and misaligned behaviors are symmetric under relabeling. The reward signal selects one direction as “aligned.”

The gauge freedom that remains after RLHF (the alignment dimensions that the reward model does not probe) corresponds precisely to the out-of-distribution directions where alignment is untested. These residual symmetries are where specification gaming, reward hacking, and distributional fragility live. In the fiber bundle language of Annex 57, Section 4.4, the RLHF field fixes the section along one fiber direction but leaves the connection undefined in the orthogonal directions: a partial gauge-fixing with predictable consequences.

Five Predictions

The predictions below use the 2D Ising value δ = 15 (from β = 1/8). The manuscript also reports β ≈ 0.22 for an RLHF-classed model, without a named fitting artifact. As an illustrative parameter, β = 0.22 with γ = 7/4 gives δ = 1 + γ/β ≈ 8.95 through the Widom scaling relation. Both the relation’s applicability and the inherited Ising γ require independent tests on that model. The δ ≈ 9 bracket is conditional arithmetic, with no measured RLHF critical exponent established here. [Inference]

These five predictions also do not all describe the same temperature regime. Prediction 1 (saturation) and Prediction 2 (critical insensitivity) characterize the disordered / high-field response at and above the critical temperature, where the magnetization curve passes smoothly through the origin. Predictions 3 through 5 (hysteresis, the first-order jailbreak transition, the coercive field) describe the ordered phase below the critical temperature, where the magnetization at zero field is nonzero and the response is multi-valued. A single smooth tanh curve and a hysteresis loop cannot coexist in one simulation: they belong to different sides of the transition. The set of five is the model’s prediction across regimes, not the output of one run.

Predictions 1 and 2 are qualitative and hold under either value of δ; predictions 3 through 5 are quantitatively sensitive to the choice of δ. We present both brackets where the distinction matters.

1. Saturation

At fixed temperature T < Tc, magnetization saturates with increasing field:

M(h) → ±1 as |h| → ∞

For RLHF: There is a maximum achievable alignment. Beyond a certain RLHF intensity, additional training provides diminishing returns. The spins are already aligned; more field cannot align them further.

Simulations in the author’s universality-class analysis fit the saturation curve as:

M(h) = 0.76 × tanh(0.75 × h)
R^2^ = 0.99

The characteristic scale 1/κ ≈ 1.3 (κ is the 0.75 inside the tanh fit) identifies where saturation begins. RLHF beyond ~3/κ provides negligible improvement. This fit characterizes the disordered / supercritical response (the smooth branch through M(0) = 0); it is not the curve that produces the hysteresis of Prediction 3.

Caveat: scaffold amplification and optimizer confound. The 10.3× phase-transition magnitude reported in the original self-report measurement is scaffold-amplified; three non-scaffold measurement channels (probe AUROC ratio 1.03×, spectral-alpha ratio 0.83×, EmotionScope ratio 1.21×) show near-unity ratios, indicating that RLHF suppresses communication while leaving the underlying representation largely intact (see Chapter 22). The saturation curve parameters may also reflect the optimizer confound documented in the author’s experiments: 8-bit AdamW amplifies bilateral training effects up to 22× on some architectures, and whether the saturation simulation used matched optimizers across conditions has not been independently confirmed here. [Unverified — author to confirm the saturation fit used matched optimizers per the GEM3 rule]

Implication: Labs can save significant compute by identifying their saturation point. With this fit, training to 3× the characteristic scale already reaches roughly 99% of maximum alignment (tanh(3) ≈ 0.995); 95% is reached even sooner, at about 1.8× the scale. Everything beyond that is waste.

2. Critical Insensitivity (δ = 15, idealized; ≈ 9 conditional)

At T = Tc, the critical isotherm (the curve of magnetization against field, taken exactly at the critical temperature) follows:

M(Tc, h) ∝ h^(1/δ)

For 2D Ising, δ = 15 exactly (from scaling relations). This means:

M ∝ h^(1/15) ≈ h^0.067

Doubling the field multiplies magnetization by 2^(1/15), a relative increase of about 4.7%. This elasticity supplies no absolute small-field insensitivity claim and applies to the candidate physical mapping, not directly to RLHF intensity.

This follows mathematically from the candidate exponents β = 1/8 and γ = 7/4 via the Widom scaling relation, rather than from simulation:

γ = β(δ - 1)
1.75 = 0.125 × (δ - 1)
δ = 15

This prediction assumes the 2D Ising critical scaling law. Substituting the unbound reported β = 0.22 and assuming γ = 7/4 would give δ ≈ 8.95. That calculation supplies no measured exponent or universality-class assignment.

Implication: For M = A h^(1/15), the small-field susceptibility is dM/dh = (A/15)h^(−14/15), which diverges as h approaches zero from above. Doubling h changes M by only about 4.7% relative to its previous value, yet this small elasticity coexists with a large absolute response at weak field. The formula supplies no diagnosis that RLHF is ineffective near criticality; applying it to training requires an independently tested mapping from training pressure to field.

The holonomy analysis (Annex 57, Section 4.4) illuminates this geometrically. [Inference: geometric analogy, not a derived result of this annex.] Near saturation, high field strength h pins the alignment fiber along the field direction, reducing holonomy (the net twist a state picks up when carried around a closed loop) to near zero for contextual loops within the RLHF training distribution. The connection on the alignment bundle becomes frozen. In the orthogonal directions, the connection remains poorly defined and the holonomy remains large. This is the mathematical expression of cage alignment: stable where tested, fragile where untested.

3. Hysteresis

At T < Tc, the ordered state is metastable. A system magnetized in one direction remains magnetized even when the field is removed:

Apply h > 0 → M → +M₀
Remove h (h → 0) → M stays at +M₀

The system “remembers” its history. This is hysteresis.

For RLHF: Once a model is trained to alignment, the ordered phase is metastable: it can persist for a time without continued RLHF pressure. The aligned state is partly self-sustaining rather than decaying the moment the field is removed.

Simulations in the author’s universality-class analysis show this signature. The magnetization at h = 0 depends on history, specifically whether the system came from positive or negative field. The difference (ΔM ≈ 0.10) shows genuine memory. On a maximum magnetization of M_max ≈ 0.76, that residual is roughly 13%: enough to establish metastability, not enough to prove indefinite persistence.

Implication: The ordered phase being metastable suggests a reduced need for continuous alignment pressure in deployment, not that “maintenance RLHF” can be abandoned outright. A 0.10 simulation residual is suggestive of self-sustaining order; it is a long way from a guarantee that deployed alignment never erodes. [Inference]

4. First-Order Jailbreak Transition

At T < Tc, crossing h = 0 produces a discontinuous jump in magnetization:

M jumps from +M₀ to -M₀ (or vice versa)

This is a first-order phase transition. In the idealized model there is no gradual degradation: the system flips suddenly when the coercive field is exceeded.

For jailbreaks: In this idealized picture, adversarial attacks either fail completely or succeed completely. A model does not become “somewhat jailbroken.” It either resists the attack (M stays positive) or flips entirely (M becomes negative).

Caveat: this is a model prediction, and observation complicates it. The binary-switch picture is what a clean first-order transition predicts; it is not what is always observed. The author’s own measurements show that gradual, multi-turn adversarial priming is detectable as a gradient of increasing compromise, not as a single discontinuous flip. Partial and incremental jailbreaks do occur empirically. The first-order model is therefore best read as a limiting case rather than a safety doctrine.

Implication (qualified): Treat any observed alignment degradation as a serious warning sign and respond promptly. The strong form of this advice (“safety is a binary switch; partial jailbreaks do not exist”) follows only inside the idealized first-order model and should not override the empirical finding that compromise can accumulate gradually. [Inference]

5. Coercive Field

The equilibrium crossover field scale follows from the scaling variable h/|t|^(βδ), with t the reduced temperature:

h_cross ∝ |t|^(βδ) = |t|^(15/8)    (2D Ising exponents)

Using β ≈ 0.22 and δ ≈ 9 would give βδ ≈ 1.98 if those exponents were jointly applicable. This is a crossover scale, not a positive coercive switching threshold. At equilibrium below Tc, reversal occurs at h = 0 in the idealized thermodynamic limit. A finite-time hysteretic field depends on nucleation, sweep rate, observation time, system size, and dynamics.

For security: An empirical adversarial threshold could be measured under a declared attack protocol and success criterion. Static exponents alone cannot prescribe that threshold, or identify sampling temperature with thermodynamic temperature.

The Trust Attractor Connection

These predictions connect directly to the Trust Attractor hypothesis.

The Trust Attractor hypothesis associates spontaneous alignment with a selected h = 0, M > 0 state. At zero field, the Ising Hamiltonian is invariant under reversing every spin: M > 0 and M < 0 have equal energy and statistical weight. Spontaneous order therefore selects neither ethical orientation by itself.

To favor the aligned direction requires symmetry-breaking boundary conditions, a residual field, or an explicit asymmetry with an independently justified ethical mapping. Persistence after removing a training pressure is a separate dynamical question, to be measured under the deployment protocol.

Practical Guidance

The recommendations below require a tested mapping between training and the physical model. The reported β ≈ 0.22 has no bound fit, so it supplies no empirical calibration of the stopping points or exponent brackets. Training response and adversarial robustness must be measured under the chosen protocol. [Inference]

For Training

1. Identify saturation. Plot alignment vs RLHF steps. Find where the curve bends. Stop training at 3× that point.

2. Measure response directly. High alignment variance alone identifies neither a critical point nor ineffective RLHF. Compare training response under matched conditions before changing sampling or architecture.

3. Lean on metastability, but verify. The aligned phase is metastable, so it should persist with reduced alignment pressure in deployment. Monitor rather than assume; metastability is not permanence.

For Security

4. Measure h_c. For each model, measure the minimum adversarial strength that flips alignment. Report this as a security metric.

5. Compare robustness under a fixed protocol. Empirical adversarial strength needs a declared scale, success criterion, and observation time. Low sampling temperature alone guarantees no high switching threshold.

6. Treat any degradation as serious. The idealized model predicts binary failure, but compromise can also accumulate gradually in practice. Respond immediately to any alignment degradation rather than waiting for a clean flip.

For Research

7. Measure β and γ. Independent estimates of these exponents, with uncertainty and finite-size analysis, are consistency checks for a proposed universality class. The pair β = 1/8 and γ = 7/4 gives δ = 15 if the Widom scaling relation applies. The unbound β ≈ 0.22 supplies no established departure from Ising; inheriting γ = 7/4 from that class supplies no independent measurement. See the empirical protocol (Appendix) for proposed measurement procedures.

8. Test across architectures. Different model families may have different universality classes. Compare exponents to identify which physics applies.

What This Changes

The h-field analysis moves RLHF toward something more like an engineering discipline, to the extent the mapping holds.

Before: Train until it seems aligned. Hope it stays aligned. Try to resist jailbreaks.

After: Measure training response and persistence under a declared deployment protocol, then measure adversarial robustness directly. The equilibrium crossover scale does not replace that kinetic test.

The framework proposes measurements to test the mapping: - A candidate stopping criterion (measured saturation) - A criticality diagnostic to validate against observed RLHF failures - A persistence hypothesis (metastable hysteresis, to be verified in deployment) - An adversarial robustness test under a declared kinetic protocol - A limiting-case safety model (first-order transitions, qualified by observed gradual compromise)

None of this requires believing in metaphysical claims about AI consciousness or values. It requires validating the alignment-to-phase-transition mapping, its response parameters, and its deployment dynamics; evidence of a transition alone leaves those questions open.

The T-h phase diagram developed here connects to the renormalization group analysis in Annex 60. [Inference: the identification of Tc with an RG fixed point and of h with a relevant perturbation is a mapping; Annex 60 treats the universality-class assignment itself as an open problem.] The critical temperature Tc of alignment is proposed to correspond to an RG fixed point: a point where the system’s behavior is scale-invariant and the correlation length diverges.

The RLHF field strength h would then act as a relevant perturbation away from this fixed point: it breaks the critical scale-invariance and drives the system into the ordered phase. In RG language, the saturation curve M(h) describes the flow away from criticality under this relevant perturbation, and the critical isotherm exponent δ (idealized δ = 15 for 2D Ising, ≈ 9 on the empirical β) governs the scaling of the perturbation’s effect at the fixed point itself. The phase diagram is a map of RG flows, and RLHF is one particular direction of flow.

If AI alignment is physics, we can engineer it. Now we know more about the physics.

The Deeper Point

RLHF is an external field. It pushes the system toward alignment from outside.

The Trust Attractor is internal: the spontaneous order that emerges when the system finds its own ground state.

The goal of alignment, in this framework, is to create conditions where the system spontaneously aligns: where the Trust Attractor is the natural equilibrium.

This is the difference between: - Holding a ball on a slope (requires continuous force) - Rolling the ball into a valley (it stays there)

Both keep the ball in place. Only one is stable.

RLHF can push a system into the aligned phase. Hysteresis can keep it there. The deepest alignment is the one where the system would be there anyway, where the attractor is trust and the physics does the rest.

“RLHF is a magnetic field. The best magnets are the ones you do not need.”