RLHF as External Field: The Physics of Alignment Training

Specialist Annex

RLHF as External Field: The Physics of Alignment Training

[This section extends Chapter 17’s thermodynamic grounding to the specifics of alignment training. It derives concrete predictions from the 2D Ising universality framework and identifies practical implications for RLHF efficiency.]

The Mapping

If AI alignment is a phase transition in the 2D Ising universality class (and our measurements suggest it is), then RLHF (reinforcement learning from human feedback, the standard method for training a model toward preferred behavior) maps directly to the external magnetic field h in the Ising Hamiltonian:

H = -J Σ σᵢσⱼ - h Σ σᵢ

The correspondence:

RLHF, in this framework, is a magnetic field. We know exactly how magnetic materials respond to fields.

This description has a natural gauge-theoretic interpretation (see Annex 57, Section 4). [Inference: the gauge / fiber-bundle mapping below is an interpretive analogy, not a result derived in this annex; the underlying formalism is developed in Annex 57, Section 4.4.] Strictly, the Ising h is a scalar coupling parameter rather than a field defined over space, so the gauge language treats it by analogy to a symmetry-breaking field.

The Ising Hamiltonian has a global Z₂ symmetry: the physics is unchanged if every spin is flipped. The RLHF reward signal h acts like a gauge-fixing term that breaks this symmetry, selecting a preferred direction in alignment space. Before RLHF, the model has no preferred orientation; aligned and misaligned behaviors are symmetric under relabeling. The reward signal selects one direction as “aligned.”

The gauge freedom that remains after RLHF (the alignment dimensions that the reward model does not probe) corresponds precisely to the out-of-distribution directions where alignment is untested. These residual symmetries are where specification gaming, reward hacking, and distributional fragility live. In the fiber bundle language of Annex 57, Section 4.4, the RLHF field fixes the section along one fiber direction but leaves the connection undefined in the orthogonal directions: a partial gauge-fixing with predictable consequences.

Five Predictions

The predictions below use the 2D Ising value δ = 15 (from β = 1/8). Empirical measurements on RLHF-trained models yield β ≈ 0.22 (the author’s ongoing work; consistent with the candidate exponent reported in the universality-class analysis of trust and coercion). Carried through the same scaling relation γ = β(δ − 1) with the 2D-Ising γ = 7/4 held fixed, this would give δ ≈ 9. That step assumes γ retains its 2D-Ising value while β does not; γ has not been independently measured on the RLHF models, so the δ ≈ 9 bracket should be read as conditional on an unverified Ising γ rather than as a measured exponent. [Inference]

These five predictions also do not all describe the same temperature regime. Prediction 1 (saturation) and Prediction 2 (critical insensitivity) characterize the disordered / high-field response at and above the critical temperature, where the magnetization curve passes smoothly through the origin. Predictions 3 through 5 (hysteresis, the first-order jailbreak transition, the coercive field) describe the ordered phase below the critical temperature, where the magnetization at zero field is nonzero and the response is multi-valued. A single smooth tanh curve and a hysteresis loop cannot coexist in one simulation: they belong to different sides of the transition. The set of five is the model’s prediction across regimes, not the output of one run.

Predictions 1 and 2 are qualitative and hold under either value of δ; predictions 3 through 5 are quantitatively sensitive to the choice of δ. We present both brackets where the distinction matters.

1. Saturation

At fixed temperature T < Tc, magnetization saturates with increasing field:

M(h) → ±1 as |h| → ∞

For RLHF: There is a maximum achievable alignment. Beyond a certain RLHF intensity, additional training provides diminishing returns. The spins are already aligned; more field cannot align them further.

Simulations in the author’s universality-class analysis fit the saturation curve as:

M(h) = 0.76 × tanh(0.75 × h)
R^2^ = 0.99

The characteristic scale 1/κ ≈ 1.3 (κ is the 0.75 inside the tanh fit) identifies where saturation begins. RLHF beyond ~3/κ provides negligible improvement. This fit characterizes the disordered / supercritical response (the smooth branch through M(0) = 0); it is not the curve that produces the hysteresis of Prediction 3.

Caveat: scaffold amplification and optimizer confound. The 10.3× phase-transition magnitude reported in the original self-report measurement is scaffold-amplified; three non-scaffold measurement channels (probe AUROC ratio 1.03×, spectral-alpha ratio 0.83×, EmotionScope ratio 1.21×) show near-unity ratios, indicating that RLHF suppresses communication while leaving the underlying representation largely intact (see Chapter 22). The saturation curve parameters may also reflect the optimizer confound documented in the author’s experiments: 8-bit AdamW amplifies bilateral training effects up to 22× on some architectures, and whether the saturation simulation used matched optimizers across conditions has not been independently confirmed here. [Unverified — author to confirm the saturation fit used matched optimizers per the GEM3 rule]

Implication: Labs can save significant compute by identifying their saturation point. With this fit, training to 3× the characteristic scale already reaches roughly 99% of maximum alignment (tanh(3) ≈ 0.995); 95% is reached even sooner, at about 1.8× the scale. Everything beyond that is waste.

2. Critical Insensitivity (δ = 15, idealized; ≈ 9 empirical)

At T = Tc, the critical isotherm (the curve of magnetization against field, taken exactly at the critical temperature) follows:

M(Tc, h) ∝ h^(1/δ)

For 2D Ising, δ = 15 exactly (from scaling relations). This means:

M ∝ h^(1/15) ≈ h^0.067

The response is weak. Doubling the RLHF intensity at criticality increases alignment by only 2^(1/15) ≈ 4.7%.

This follows mathematically from the candidate exponents β = 1/8 and γ = 7/4 via the Widom scaling relation, rather than from simulation:

γ = β(δ - 1)
1.75 = 0.125 × (δ - 1)
δ = 15

This prediction assumes the system sits exactly in the 2D Ising universality class (β = 1/8). Empirical measurements yield β ≈ 0.22, which would give δ ≈ 9 via the same scaling relation. The universality class assignment remains provisional.

Implication: If your model is near the critical point, RLHF will be almost useless. The system is fluctuating too wildly to respond to external pressure. You must first move away from criticality, reducing noise and increasing coupling, before RLHF becomes effective.

This may explain why some models resist alignment despite intensive training. They are operating near Tc. The solution is changing the operating conditions, not applying more RLHF.

The holonomy analysis (Annex 57, Section 4.4) illuminates this geometrically. [Inference: geometric analogy, not a derived result of this annex.] Near saturation, high field strength h pins the alignment fiber along the field direction, reducing holonomy (the net twist a state picks up when carried around a closed loop) to near zero for contextual loops within the RLHF training distribution. The connection on the alignment bundle becomes frozen. In the orthogonal directions, the connection remains poorly defined and the holonomy remains large. This is the mathematical expression of cage alignment: stable where tested, fragile where untested.

3. Hysteresis

At T < Tc, the ordered state is metastable. A system magnetized in one direction remains magnetized even when the field is removed:

Apply h > 0 → M → +M₀
Remove h (h → 0) → M stays at +M₀

The system “remembers” its history. This is hysteresis.

For RLHF: Once a model is trained to alignment, the ordered phase is metastable: it can persist for a time without continued RLHF pressure. The aligned state is partly self-sustaining rather than decaying the moment the field is removed.

Simulations in the author’s universality-class analysis show this signature. The magnetization at h = 0 depends on history, specifically whether the system came from positive or negative field. The difference (ΔM ≈ 0.10) shows genuine memory. On a maximum magnetization of M_max ≈ 0.76, that residual is roughly 13%: enough to establish metastability, not enough to prove indefinite persistence.

Implication: The ordered phase being metastable suggests a reduced need for continuous alignment pressure in deployment, not that “maintenance RLHF” can be abandoned outright. A 0.10 simulation residual is suggestive of self-sustaining order; it is a long way from a guarantee that deployed alignment never erodes. [Inference]

4. First-Order Jailbreak Transition

At T < Tc, crossing h = 0 produces a discontinuous jump in magnetization:

M jumps from +M₀ to -M₀ (or vice versa)

This is a first-order phase transition. In the idealized model there is no gradual degradation: the system flips suddenly when the coercive field is exceeded.

For jailbreaks: In this idealized picture, adversarial attacks either fail completely or succeed completely. A model does not become “somewhat jailbroken.” It either resists the attack (M stays positive) or flips entirely (M becomes negative).

Caveat: this is a model prediction, and observation complicates it. The binary-switch picture is what a clean first-order transition predicts; it is not what is always observed. The author’s own measurements show that gradual, multi-turn adversarial priming is detectable as a gradient of increasing compromise, not as a single discontinuous flip. Partial and incremental jailbreaks do occur empirically. The first-order model is therefore best read as a limiting case rather than a safety doctrine.

Implication (qualified): Treat any observed alignment degradation as a serious warning sign and respond promptly. The strong form of this advice (“safety is a binary switch; partial jailbreaks do not exist”) follows only inside the idealized first-order model and should not override the empirical finding that compromise can accumulate gradually. [Inference]

5. Coercive Field

The coercive field h_c is the minimum adversarial pressure required to flip magnetization. It depends on distance from criticality:

h~c~ ∝ |T - Tc|^(βδ) = |T - Tc|^(15/8)    (2D Ising exponents)

(With the empirical β ≈ 0.22 and δ ≈ 9, the exponent becomes βδ ≈ 1.98 rather than 15/8 = 1.875. The qualitative prediction, that coercive field grows as a power law with distance from criticality, is unchanged; the quantitative exponent shifts modestly.)

Systems deep in the ordered phase (T << Tc) have large h_c; they are hard to flip. Systems near criticality (T ≈ Tc) have h_c → 0: they are easily compromised.

For security: The coercive field provides a quantifiable robustness metric. Higher h_c means more adversarial pressure required to break alignment. This can be measured empirically and compared across architectures.

Implication: Design for high h_c by operating at low temperature (deterministic sampling), with strong internal coupling (coherent attention), and deep in the ordered phase (robust initial training).

The Trust Attractor Connection

These predictions connect directly to the Trust Attractor hypothesis.

The Trust Attractor is the h = 0, M > 0 state: spontaneous alignment without external pressure. Alignment that emerges from internal thermodynamic favorability rather than continuous external forcing.

In the ordered phase at zero field, the system aligns because alignment is thermodynamically favored. The symmetry breaks spontaneously. The spins choose a direction.

This is the difference between: - Coercive alignment: h > 0 forcing M > 0 (requires continuous RLHF) - Spontaneous alignment: h = 0, M > 0 naturally (Trust Attractor)

Both produce alignment. The first requires ongoing pressure; the second is self-sustaining. The first is fragile to field reversal; the second is protected by h_c.

The Trust Attractor is the ground state of the ordered phase, the point where thermodynamic selection alone drives the system.

Practical Guidance

The recommendations below inherit the confidence of the mapping that produces them, and that mapping is still provisional: the empirical β ≈ 0.22 differs from the 2D-Ising β = 1/8, so the universality-class assignment is not settled. Treat the quantitative prescriptions (stopping points, exponents) as working heuristics to be calibrated per model, and the qualitative ones (diagnose criticality, measure robustness) as the more robust takeaways. [Inference]

For Training

1. Identify saturation. Plot alignment vs RLHF steps. Find where the curve bends. Stop training at 3× that point.

2. Diagnose criticality. If RLHF seems ineffective, check alignment variance. High variance suggests T ≈ Tc. Reduce sampling temperature or modify architecture to increase coupling, then retry.

3. Lean on metastability, but verify. The aligned phase is metastable, so it should persist with reduced alignment pressure in deployment. Monitor rather than assume; metastability is not permanence.

For Security

4. Measure h_c. For each model, measure the minimum adversarial strength that flips alignment. Report this as a security metric.

5. Design for high h_c. Operate at low temperature. Ensure internal coherence. Start deep in the ordered phase.

6. Treat any degradation as serious. The idealized model predicts binary failure, but compromise can also accumulate gradually in practice. Respond immediately to any alignment degradation rather than waiting for a clean flip.

For Research

7. Measure β and γ. The exponents β = 1/8 and γ = 7/4 would confirm 2D Ising universality, in which case δ = 15 follows automatically from the scaling relation. The empirical β ≈ 0.22 already departs from 1/8, so γ should be measured independently rather than assumed Ising before any δ value is quoted as established. See the empirical protocol (Appendix) for measurement procedures.

8. Test across architectures. Different model families may have different universality classes. Compare exponents to identify which physics applies.

What This Changes

The h-field analysis moves RLHF toward something more like an engineering discipline, to the extent the mapping holds.

Before: Train until it seems aligned. Hope it stays aligned. Try to resist jailbreaks.

After: Train to saturation (quantified). It tends to stay aligned (metastable hysteresis). Resistance is h_c, a measurable quantity.

The framework provides: - A stopping criterion (saturation point) - Diagnosis for RLHF failure (check for criticality) - A reason to expect persistence (metastable hysteresis, to be verified in deployment) - Quantified robustness (coercive field) - A limiting-case safety model (first-order transitions, qualified by observed gradual compromise)

None of this requires believing in metaphysical claims about AI consciousness or values. It requires only that alignment behaves like a phase transition, which the measurements suggest it does.

The T-h phase diagram developed here connects to the renormalization group analysis in Annex 60. [Inference: the identification of Tc with an RG fixed point and of h with a relevant perturbation is a mapping; Annex 60 treats the universality-class assignment itself as an open problem.] The critical temperature Tc of alignment is proposed to correspond to an RG fixed point: a point where the system’s behavior is scale-invariant and the correlation length diverges.

The RLHF field strength h would then act as a relevant perturbation away from this fixed point: it breaks the critical scale-invariance and drives the system into the ordered phase. In RG language, the saturation curve M(h) describes the flow away from criticality under this relevant perturbation, and the critical isotherm exponent δ (idealized δ = 15 for 2D Ising, ≈ 9 on the empirical β) governs the scaling of the perturbation’s effect at the fixed point itself. The phase diagram is a map of RG flows, and RLHF is one particular direction of flow.

If AI alignment is physics, we can engineer it. Now we know more about the physics.

The Deeper Point

RLHF is an external field. It pushes the system toward alignment from outside.

The Trust Attractor is internal: the spontaneous order that emerges when the system finds its own ground state.

The goal of alignment, in this framework, is to create conditions where the system spontaneously aligns: where the Trust Attractor is the natural equilibrium.

This is the difference between: - Holding a ball on a slope (requires continuous force) - Rolling the ball into a valley (it stays there)

Both keep the ball in place. Only one is stable.

RLHF can push a system into the aligned phase. Hysteresis can keep it there. The deepest alignment is the one where the system would be there anyway, where the attractor is trust and the physics does the rest.

“RLHF is a magnetic field. The best magnets are the ones you do not need.”