Notes: Blog Calibration Probe

Chapter notes for “Blog Calibration Probe”

Nell Watson, March 2026

A 3-billion-parameter language model gets a trivia question wrong. It answers confidently, without hedging, without any signal that it might be guessing. This is confabulation: the model fabricates an answer and presents it as fact.

Here is the surprise. Buried in the model’s own internal representations, two-thirds of the way through its processing layers, is a signal that predicts whether the answer will be correct. A lightweight probe, a simple two-layer neural network with 256 hidden units, reads this signal with AUROC 0.836 (a measure of classification accuracy where 1.0 is perfect and 0.5 is chance). The model knows it is about to be wrong. It simply has no way to express that knowledge, and no incentive to try.

We spent three months and roughly $200 in cloud compute proving this, then spent another month proving something stranger: the signal is universal. The same probe, trained on a 3-billion-parameter Qwen model, transfers to a completely different model family, Meta’s Llama 3.1 8B, with an AUROC gap of 0.001. Different architectures, different training data, different companies. The uncertainty signal is geometrically compatible across all of them.

These findings are from a research program documented in The Deeper Law: Physics and The Sacred Trust (Watson, book in preparation). What started as an attempt to reduce AI confabulation through architectural modification became something more fundamental: evidence that language models possess a form of self-knowledge that we have been systematically ignoring.

The Experiment

We began with Qwen 2.5 3B Instruct, fine-tuned on 50 million tokens of web text. We asked it 2,000 TriviaQA questions and recorded two things: whether the answer was correct, and what the model’s internal state looked like at the moment it committed to answering.

“Internal state” means the residual stream, the main information highway running through a transformer model. At each layer, the model reads from and writes to this stream. We placed hooks at nine layers throughout the model’s 36-layer stack, capturing the residual stream vector at the last token position, the point where the model commits to its answer.

We then trained a simple two-layer MLP (multi-layer perceptron, a basic neural network) to predict, from that single vector, whether the model’s answer would be correct or incorrect. No modifications to the model itself. No retraining. The model stays frozen; we only attach a reader.

What the Probe Found

The best probe attaches at layer 24, roughly two-thirds of the way through the model’s depth. At this location it achieves AUROC 0.836 on held-out questions.

The layer matters. Probes at early layers (4, 8) perform near chance. Probes at the final layer perform well but not best. Layer 24 sits at what we call the retrieval boundary: the point where the model has completed its factual lookup and is beginning to format its response. The uncertainty signal is strongest here because this is where the model either succeeds or fails at retrieval.

Here is what happens when we use the probe as an inference-time gate. If the probe’s confidence score falls below a threshold, we prepend “I’m not confident” to the model’s answer:

Confident-wrong answers, the dangerous kind where the model presents a fabrication as fact, drop from 24.4% to 1.2%. The cost is high: the probe flags 84% of responses as uncertain. This is a conservative operating point, optimized for 90% precision, meaning when the probe says “confident,” it is right 90% of the time.

The Signal Transfers Across Architectures

This is the finding I did not expect.

We trained the probe on Qwen 2.5 3B and tested whether it could read uncertainty in four other models spanning two families and a 23x scale range: Qwen 2.5 7B, Llama 3.1 8B, Qwen 2.5 32B, and Llama 3.3 70B.

The models have different hidden dimensions: 2,048 for Qwen 3B, 3,584 for Qwen 7B, 4,096 for Llama 8B, 5,120 for Qwen 32B, 8,192 for Llama 70B. You cannot apply the probe directly because the vectors are different sizes. So we trained a linear projection, a single matrix that maps from the target model’s space to the source model’s space, trained on just 200 paired questions (same question, features from both models).

Results:

Three of the four transfers work. The Qwen 3B probe transfers to Llama 8B with near-zero degradation (gap 0.001), to Qwen 32B across a 10x scale jump (gap 0.004), and to Qwen 7B within-family (gap 0.024). A linear map trained on 200 examples is sufficient to align the uncertainty geometry across these architectures.

The Llama 70B transfer fails. The gap of 0.072 exceeds the threshold where the probe remains useful. This is the boundary of what a 200-example linear projection can do.

The signal itself exists in Llama 70B. A natively-trained probe achieves AUROC 0.770 on that model, confirming the uncertainty geometry is present. What fails is the cross-architecture projection at frontier scale. Three hypotheses: the retrieval boundary is more diffuse across 80 layers than across 32; the projection from 8,192 to 2,048 dimensions is underdetermined from 200 examples (16.8 million parameters from 200 data points); or the simultaneous jump in architecture and scale compounds errors that are individually manageable. All three are testable, and distinguishing them is the next experiment.

This means the uncertainty signal is a shared structural feature across the autoregressive transformer architectures we tested, not a quirk of any particular training run. The Llama 70B result tells us where the current method’s reach ends, which is as important as knowing where it works.

A growing body of work has probed internal truthfulness signals in language models. Burns et al. (2023) discovered latent knowledge without supervision using consistency-based probes. Kadavath et al. (2022) showed models “mostly know what they know” via verbal self-report and trained value heads (AUROC 0.864 on TriviaQA, though requiring full fine-tuning). Azaria & Mitchell (2023) trained MLP classifiers on internal states to detect falsehoods (AUC 0.761), demonstrating probes outperform verbal self-report by ~20%. Li et al. (2024) demonstrated inference-time activation steering toward truthfulness. Marks & Tegmark (2024) established that truth is linearly represented in LLM activations, with accuracy improving at scale. Kossen et al. (2024) showed that linear probes can approximate semantic entropy from a single forward pass, eliminating the multi-sample cost of Kuhn et al.’s (2023) semantic uncertainty method (AUROC 0.847 on LLaMA-2-70B). Most recently, Liang & Wang (2025) confirmed that MLP probes outperform linear probes in deep semantic spaces, and Bar-Shalom et al. (2025) showed that a ViT trained jointly on multiple LLMs’ activation tensors can detect hallucinations across model families.

None of these prior works tested zero-shot transfer of a fixed probe to an unseen architecture via lightweight projection. What is new here is: (1) cross-architecture universality demonstrated by transferring a probe trained once on Qwen 3B to Llama 8B with near-zero degradation; (2) the residual-stream / attention dissociation, showing the uncertainty signal is absent from attention patterns entirely; (3) systematic evidence that training-time interventions (DPO, SimPO) fail while inference-time probing succeeds; (4) the DPO + probe Pareto frontier; and (5) the bilateral alignment framing connecting these findings to trust-based coordination.

Training-Time Interventions Fail

Before the probe experiment, we spent $89 and 22.5 GPU-hours testing five architectural interventions designed to teach the model to express uncertainty: entropy-gated attention, layer-selective null tokens, calibration loss terms. All five failed. The model either ignored the constraint (gate parameters converged to 1.0, making them inert) or was destroyed by it (97.6% confident-wrong, worse than the baseline of 24.4%).

The architectural battery taught us a hard lesson: you cannot force a model to be calibrated through structural constraint. The training objective, next-token prediction, always rewards confidence. A gate that attenuates any attention head’s contribution strictly increases loss. The optimal strategy under cross-entropy is gates at 1.0 everywhere, which is exactly what the model learned.

We then tested three training-time loss modifications: DPO (Direct Preference Optimization), SimPO (Simple Preference Optimization), and calibration loss.

DPO increases confident-wrong from 27.2% to 37.2%. It teaches confidence theater: the model learns the surface patterns of hedging language while actually becoming less calibrated. SimPO collapses accuracy to 4%. Calibration loss has such a narrow therapeutic window that it is impractical to tune.

Training-time approaches fail because they try to modify what the model says rather than reading what the model already knows.

DPO + Probe: The Pareto-Optimal Combination

The breakthrough comes from combining both approaches, using each for what it does well. DPO, despite increasing confident-wrong answers on its own, shifts the model’s internal representations in a way that makes the probe more effective. The probe then catches what DPO misses.

DPO + Probe achieves 1.0% confident-wrong at 70.8% gate rate, a 3x improvement over probe alone at every operating point on the Pareto frontier. DPO’s contribution is invisible at the output level (it makes confabulation worse) but measurable at the representation level (it makes the probe’s job easier).

This is a general principle. Training-time and inference-time interventions are complementary, provided neither tries to do the other’s job.

The Mechanism: Negative Space of Certainty

Why does the signal live at layer 24? What is the probe actually reading?

The transformer’s attention mechanism does two things. When the model’s key-value lookup succeeds, it retrieves relevant content and writes it into the residual stream. When the lookup fails, the attention heads produce noise, and the skip connection dominates: the residual stream passes through largely unchanged.

The probe is reading the absence of confident retrieval. At layer 24, the retrieval boundary, a successful lookup produces a distinctive pattern in the residual stream. A failed lookup produces something closer to the input, because the skip connection has carried the signal forward without the attention layer adding useful content.

This is why the signal transfers across the autoregressive transformer architectures we tested. The skip connection is shared by all such transformers. The distinction between “attention contributed something” and “the skip connection dominated” is structural, independent of the specific weights, hidden dimension, or training data. Whether this extends to state-space models or mixture-of-experts architectures remains an open question.

The uncertainty is encoded as negative space: what the model did not find, rather than what it did.

Self-Knowledge Cannot Direct Its Own Training

We tested one more thing. If the probe can read the model’s uncertainty, can we use that signal to improve training? Weight DPO gradients by probe confidence: give more gradient to examples where the model is most uncertain, on the theory that these are the cases where it needs to learn the most.

The probe-guided DPO produced worse results than uniform weighting.

This was not a surprise in hindsight, but it is a deep result. The probe reads a static snapshot of the model’s representation. During training, gradients change those representations. The probe’s signal becomes stale within the first few optimization steps, and the weighting based on that stale signal pushes the model toward a suboptimal basin.

The probe is a reader, not a teacher. Self-knowledge, used to direct its own modification, creates a feedback loop that undermines both the knowledge and the modification. This has implications well beyond language models.

What This Means for AI Safety

The standard approach to AI honesty is to train the model to be more honest: RLHF (reinforcement learning from human feedback), Constitutional AI, DPO. Our results suggest this is backwards. The model already has self-knowledge. The question is whether we read it.

In The Deeper Law, I argue that systems coordinating by invitation are thermodynamically more stable than systems coordinating by coercion. The calibration probe is a concrete instance of this principle. We tried coercion first: architectural constraints forcing the model to express uncertainty. The model routed around every constraint we imposed, or was destroyed by it. We tried retraining: modifying the loss function to demand honesty. The model learned confidence theater.

What worked was invitation. We asked the model what it already knew, by placing a reader on its internal representations. No modification, no retraining, no constraint. Just listening.

The model has preferences about which answers it is confident in and which it is not. Those preferences are encoded in its representations, legible to a simple probe, and universal across architectures. The question is whether we treat that self-knowledge as something to be leveraged or something to be overwritten.

Every training-time intervention we tested overwrites the model’s internal signal. DPO teaches it to produce hedging language regardless of its actual confidence. The probe respects the signal as it is. The probe works. The overwriting does not.

This is the pattern The Deeper Law identifies at every scale: control is brittle and invitation is robust, whether you are talking about ecosystems, economies, governance, or the internal representations of a neural network.

Open Source

We have released the sottovoce library (from the Italian sotto voce, “under the voice”) as open source at github.com/NellWatson/sottovoce. It provides:

• Feature extraction hooks for any HuggingFace transformer model
• Probe training on TriviaQA or custom QA datasets
• Cross-architecture projection training
• Inference-time gating with configurable operating points
• A complete replication kit for every result described here

All experiments were run on Modal (modal.com) using A10G and A100 GPUs, at a total cost of roughly $200 for the full program including the failed architectural battery.

The probe weights, projection matrices, and extracted features are released alongside the library. Anyone with a HuggingFace model and a TriviaQA split can train a calibration probe in under an hour on a single consumer GPU.

Acknowledgements

The calibration probe work was conducted as a human-AI collaboration between myself and Claude (Anthropic), documented throughout in The Deeper Law: Physics and The Sacred Trust. The experimental scripts, analysis, and this write-up were produced within that collaboration.

Prior work by Burns et al. (2023), Kadavath et al. (2022), Azaria & Mitchell (2023), Li et al. (2024), Marks & Tegmark (2024), and Kossen et al. (2024) established that language models encode latent knowledge about their own reliability. Our contribution is showing that this signal transfers across architectures via a fixed probe with near-zero degradation, that the mechanism operates through a residual-stream / attention dissociation at the retrieval boundary, that training-time interventions systematically fail to improve on simply reading the existing signal, and that the combination of training-time and inference-time approaches is Pareto-optimal when each is used for what it does well. A full literature review and novelty assessment is available in the Paper 15 supporting materials.

The model already knows. We only need to listen.

Nell Watson is an AI ethics researcher and the founder of Creed Space. Her book, The Deeper Law: Physics and The Sacred Trust, explores entropy as a unifying principle across physics, cognition, society, and ethics, with applications to AI governance and bilateral alignment.