Annex: The Neural Architecture Record

Specialist Annex

Supporting record for Chapter 17. The chapter decomposes coordination into content and gate: coercion is hard gating imposed from outside, invitation is smooth gating decided from within. This annex assembles the engineering history behind that decomposition, six decades of competitive selection on neural-network components in which nobody set out to test the Trust Attractor thesis and the selection pressure tested it anyway.

Sixty Years of Gates

An inadvertent experiment spanning six decades confirms the point at the level of neural network components. The activation function, the nonlinear curve inside every layer that decides which signals pass forward and which get suppressed, has been redesigned continuously since Rosenblatt’s perceptron in 1958. The progression runs from hard threshold (a binary switch: on or off) through sigmoid and hyperbolic tangent (smooth curves that flatten at their extremes) to ReLU (simple, fast, but creating permanently dead neurons) to the gated linear units that power current transformers (SwiGLU, GEGLU, Squared ReLU).¹ Each generation replaced a harder gate with a smoother one. Each replacement was selected by competitive pressure across thousands of labs and millions of GPU-hours: researchers kept what trained faster, generalized better, and resisted degradation. Nobody intended to test the Trust Attractor thesis. The selection pressure did it anyway.

The failure modes trace the same pattern Chapter 17 documents at larger scales. Hard-threshold neurons that receive a negative input produce zero output and zero gradient: no update, no recovery, no learning. A neuron that dies under ReLU has lost a degree of freedom permanently, the activation-function equivalent of a coercive regime that suppresses a participant so completely that the suppressed capacity can never return, even if conditions change. Smooth gating prevents this by maintaining gradient flow for all inputs: unfavorable signals are attenuated, not annihilated, and the system retains its full dimensionality.²

The most recent finding sharpens the point. Squared ReLU, f(x) = max(0,x)², is pure self-gating: the signal’s own magnitude determines its passage, quadratically, with no learned gate, no external arbiter, no additional parameters. It matches or exceeds the complex learned alternatives.³ The simplest possible selection mechanism, a signal modulating itself, is sufficient. The direction of sixty years of competitive selection is consistently from external binary classification toward internal continuous self-modulation: from coercion toward invitation, discovered by engineering selection pressure with no one intending the result.

A systematic investigation confirms the boundary more precisely. Across three hundred controlled experiments testing fifteen distinct modifications to the standard smooth gate, the performance surface is flat: every variation that preserves information throughput (sharper or softer gating profiles, more or less independent projection channels, curriculum schedules that transition between activation shapes) produces identical performance within measurement noise. The network compensates. Downstream weights absorb the change and reconstruct equivalent representations.

Only modifications that destroy information, that sever the correlation between the two projection streams before they combine, produce measurable degradation, and the degradation is catastrophic (four to five standard deviations below baseline). The landscape has a plateau of equivalent smooth-gating designs and a cliff at the boundary where gating becomes destructive. The sixty-year selection trajectory navigated onto that plateau; once there, the specific shape of the gate ceases to matter. What matters is that no signal is irrecoverably lost.

Aggregation Across Depth

A parallel instance operates across layers rather than within them. In a standard transformer, each layer’s output is added to a single residual stream (the running sum that carries the model’s working representation forward) with equal weight, progressively diluting earlier contributions. By the final layer, the model inherits one mixed representation where later layers dominate through magnitude alone: the depth-wise equivalent of the arms race that smooth gating solved within individual neurons. The Kimi Team at Moonshot AI replaced this blind accumulation with attention over the depth of the network: each layer queries all previous layer outputs and combines them by learned, input-dependent relevance weights, turning depth from a lossy compression pipeline into a selective retrieval mechanism.⁴ Output magnitudes stabilize across the full network depth, the escalation dynamic vanishes, and equivalent performance is achieved at 25 percent less compute, with the largest gains on multi-step reasoning tasks where selective retrieval of earlier representations matters most.

Independent replication at small scale confirms the mechanism is genuine: the depth-attention weights are not uniform. Early blocks consistently receive disproportionate attention from later processing stages, functioning as reference libraries that later blocks selectively consult rather than blindly inherit. The selection is input-dependent: different inputs produce different retrieval patterns, precisely the content-relevance-based coordination that the Trust Attractor predicts. A cross-analysis with the magnitude profiles sharpens the finding: the block that receives the most attention is also the block with the lowest output magnitude. The network has learned to attend by informational content rather than signal strength, overriding the very magnitude hierarchy that additive residuals would impose.

The trajectory is the same one traced above at a different architectural scale: fixed aggregation that rewards magnitude gives way to selective routing that rewards relevance, discovered by engineering pressure with nobody intending the result. The security strategist Richard Danzig observes that markets, bureaucracies, and machine intelligence belong to the same family of artificial intelligences (see the Hirschman discussion in Chapter 17). That observation sharpens the parallel: markets compress all information to price, and bureaucracies compress to categories on forms. Both are powerful precisely because of the compression, and stupid precisely because of what the compression discards.⁵

The architectural choice in a neural network, whether to preserve representational richness across depth or flatten it to a uniform signal, is the same choice that coordination mechanisms face at institutional scale. The sixty-year engineering trajectory resolved it in favor of preservation. Whether institutional coordination mechanisms can learn the same lesson, or whether they lack the reflexivity to resist the local attractor of lossy compression, is the question the Trust Attractor poses to governance.

Curriculum: Meeting the Model Where It Is

A complementary instance of the same selective pressure operates on the training loop rather than the activation function. Token Superposition Training processes averaged bags of contiguous tokens (several neighboring tokens blurred into one composite prediction target) during an initial phase, then returns to standard next-token prediction for the remainder of pretraining.⁶ The method’s key empirical finding: the optimal loss weighting within each bag follows the power-law decay of mutual information in natural text (exponent about 1.06). Training is most efficient when its objective respects the statistical geometry of the domain. Models that undergo the coarse-then-fine developmental sequence reach lower loss in roughly forty percent of the wall-clock time. The inference-time model is architecturally identical to one produced by conventional training; only the training path changes.

The parallel to coordination is structural. The coarse phase meets the model where its representational capacity actually is, rather than demanding precision it cannot yet deliver. The fine phase releases the constraint once the scaffold is in place. Competitive evaluation on benchmarks favors the developmental path over the brute-force path, consistent with the broader pattern: selection pressure, whether acting on activation functions over six decades or on training loops across five model scales, discovers that working with the data’s natural gradients outperforms working against them.

The directional finding, that alignment with information structure outperforms indifference to it, is the load-bearing claim; quantitative thresholds await replication at frontier scale. Pilot experiments by the author confirm the asymmetry is not merely directional: cross-domain data presented during the coarse phase is absorbed over a thousand times less efficiently than the same data in the fine phase, and the gap widens with model scale. The system that was met where it was, then given full-resolution signal once the scaffold was in place, outperformed the system that received the same signal during the coarse phase by three orders of magnitude.

Four Instances, One Structure

Four instances of the same selective pressure, operating at four different scales of neural architecture, have now been traced. Activation functions within neurons: smooth gating outcompetes hard gating because hard gates create absorbing states (dead neurons from which no gradient can recover the lost capacity). Layer aggregation across depth: attention residuals outcompete additive residuals because fixed accumulation creates a magnitude monopoly where later layers drown earlier ones irrecoverably. Training curricula across optimization: developmental placement outcompetes uniform exposure because premature commitment to fine-grained representations traps the model in an incompatible scaffold that the remaining training budget cannot undo. Agent coordination across interaction histories (the VRP relationship simulations reported in Chapter 17): full memory outcompetes compressed memory because cascading false sanctions from truncated history collapse trust into a basin from which the relationship cannot recover.

The shared mathematical structure, developed in Chapter 17, is the preservation of reversibility: each winning configuration keeps the system outside absorbing states, and each losing configuration creates them by a different mechanism (gradient death, magnitude domination, scaffold lock-in, trust collapse) with the same result, a degree of freedom permanently lost.

Nested Learning and the Spectrum of Timescales

A theoretical framework presented at NeurIPS in 2025 unifies several of these instances. Behrouz, Razaviyayn, Zhong, and Mirrokni observed that a neural network’s architecture and its optimization algorithm are the same mathematical object viewed at different update frequencies.⁷ The attention mechanism is an associative memory that updates every forward pass. The optimizer is an associative memory that updates every training step. The feedforward weights are associative memories that change only during training. Each component has its own context flow, its own stream of information from which it learns, distinguished from the others solely by how often its parameters change.

The framework, called Nested Learning, proposes organizing these components into a “continuum memory system”: a spectrum of modules from fast-updating (responsive to immediate context) through slow-updating (retaining deep training knowledge), with the full gradient between them designable rather than accidental. A proof-of-concept architecture, Hope, showed better memory management and lower perplexity than standard transformers in the paper’s benchmarks. The architecture is young and unproven at frontier scale; the theoretical frame is what earns its place here.

The structural parallel to the two-layer dove-and-serpent architecture described in Chapter 17 is direct. The dove layer, compressing moment-to-moment interaction to prevent grievance accumulation, is a fast-updating memory. The serpent layer, maintaining the full coordination history that buffers against regime shocks, is a slow-updating memory. Read this way, catastrophic forgetting becomes the pathology of a system with too few timescales: new learning overwrites old because no intermediate layer absorbs the perturbation.

The critical memory window between ten and fifty interactions (VRP-HR6) marks the boundary below which the slow layer has insufficient thermal mass: too little stored history to absorb a shock without being reshaped by it. The same pathology in social systems is a culture that updates its deep commitments at the speed of its crisis cycle, compressing institutional memory until the thermal mass required for regime-shock recovery is lost. Nested Learning’s contribution is to generalize the two-layer case to an arbitrary spectrum, a continuum of timescales, each protecting the layers below it from overwriting by the layers above. The generalization awaits empirical validation beyond the paper’s proof-of-concept benchmarks, yet the structural correspondence with the dove-and-serpent architecture the program already measures is striking enough to note.

Incompressible Problems

The boundedness is sharper than Vanchurin’s general argument suggests. Some problems have provable lower bounds on the number of sequential steps required for any solution: sorting n items requires at least n·log(n) comparisons; Sudoku requires iterative deduction that cannot be parallelized away; maze navigation requires sequential exploration. These are incompressible problems: no amount of parallel compute substitutes for sequential depth. A feed-forward architecture with k layers can perform at most k sequential operations per input, regardless of its width. When the problem’s lower bound exceeds k, the architecture fails, provably, no matter how many parameters it contains.⁸

Two recent architectures exploit this gap. Wang et al.’s Hierarchical Reasoning Model applies the same 27-million-parameter network recursively, maintaining a compressed latent state across iterations. Jolicoeur-Martineau’s Tiny Recursive Model simplifies the approach further: a single two-layer network with 7 million parameters, applied iteratively, outperforms trillion-parameter language models on Sudoku, maze navigation, and abstract reasoning tasks that require sequential deduction.⁹

The recursive models do not contain more knowledge. They think longer, with themselves: the same small structure applied iteratively, updating a compressed internal state at each pass. The mechanism by which the recursion produces correct solutions is still debated; what is not debated is that one-shot architectures fail on these tasks regardless of scale, while iterative architectures succeed at a fraction of the parameter count. The field spent a decade scaling feed-forward width. These results show that for incompressible problems, the axis that matters is iterative depth.

The mapping to coordination is direct. Many coordination problems (justice, care, sustained trust) are incompressible in the same sense: they cannot be resolved in a single pass through an institutional pipeline, regardless of that pipeline’s capacity. A bureaucracy that processes every case through a single pipeline, however elaborate, is a feed-forward architecture: wide, perhaps, but shallow. If the coordination problem requires iterative relationship, the same small structure returning to the same problem with updated state, no amount of institutional width substitutes. The recursive models succeed because they maintain a compressed representation of prior iterations and apply the same transformation repeatedly; trust-based coordination succeeds for the same structural reason. The compressed state is the relationship itself: lossy, fallible, and sufficient for the next interaction without replaying the full history.

A constraint sharpens the analogy. The author’s experiments (IC-5, IC-5b) tested whether recursive self-processing provides a coordination advantage over single-pass processing in a synthetic agent simulation. It does not. At matched interaction budgets, wider state capacity outperforms deeper recursion, even when the recursive weights are trained. The architectural advantage is specifically about step-budget asymmetry: the recursive model wins because the feed-forward model runs out of layers, not because recursion is inherently superior.

The analogy between recursive architectures and trust-based coordination is coarse at the architectural level. It is sharp at the information-processing level: what matters is what the recursion enables, not the recursion itself. Three strategies (representational compression, bilateral mutual modeling, and iterative learning) are each confirmed experimentally in Chapter 17b.

A separate constraint applies to explicit chain-of-thought reasoning, the “thinking scaffold” that current language models use to extend their sequential processing budget. Extended thinking scaffolds improve deliberative reasoning on incompressible tasks like the ones above. They degrade factual question-answering at all tested durations: with retrieved context, accuracy drops from 72% to 14% at 1,000 thinking tokens (KC#XC4-XC6). The mechanism is distributional mismatch: hidden-state thinking tokens occupy a different subspace from the embedding manifold, and their presence in context poisons retrieval. Recursive depth has task-specific benefits; for factual QA with context, extended thinking actively harms performance.

The Aurora Correction

A 2026 result in optimizer design provides a concrete demonstration at the scale of individual neurons. The Muon optimizer, now used by several frontier laboratories, updates weight matrices by computing their polar factor. Divide a vector by its own length and an arrow of length one remains, pointing the same way. The polar factor does that to a matrix, dividing out how far the matrix stretches and keeping only the directions it stretches along (formally, the closest matrix with unit spectral norm). Those stretch factors are the singular values, one for each independent direction, and the procedure treats all of them equally. On square matrices, this symmetric treatment distributes update energy uniformly across neurons. On tall matrices (more rows than columns, the standard shape of feedforward projections in transformers), it does not.

Dewulf and colleagues showed that Muon’s orthogonalization inherits the row-norm anisotropy of the raw gradient.¹⁰ Neurons whose gradient rows happen to be small during the learning rate warmup receive proportionally small updates. Small updates yield small gradients, which yield still smaller updates: a self-reinforcing loop that permanently kills more than a quarter of MLP neurons within the first five hundred training steps. The dead neurons carried no less useful information. They occupied rows where the initial momentum was small, and the optimizer’s indifference to row geometry made that accident irreversible.

The pathology lives in the update rule. Orthogonalization equalizes the singular values and passes the raw gradient’s row-norm anisotropy straight through, so whatever leverage asymmetry the update arrives with survives it intact. Symmetric constraints applied to asymmetric structures concentrate power. Higher-fidelity orthogonalization makes the pathology worse: the most precise algorithm tested killed more neurons than the least precise, because it more faithfully preserved the leverage anisotropy the gradient inherited from the matrix geometry. Rigorous enforcement of a symmetric rule on an asymmetric substrate produced the most severe inequality.

The researchers’ fix, Aurora, adds a single constraint: each row receives a proportional share of update energy. At 1.1 billion parameters, the equitable optimizer achieved lower loss, a ten-point improvement on a memorization benchmark, and competitive performance with models trained on a hundred times more data. The system that distributed opportunity outperformed the system that concentrated it. The thermodynamic argument is the same one Chapter 17 traces from bacteria to institutions: concentration wastes capacity; equity is efficiency.

The philosophical argument these instances support appears in Chapter 17; the geometry of the basin in Chapter 17a; the experimental record in Chapter 17e.