Optimizer Confound

When 95% of a dramatic finding is methodological artifact

Direction

✓ Robust

holds across architectures

Magnitude

✗ 95% Artifact

optimizer-driven

DD-22claimed −0.462
(8-bit AdamW)

GEM-3suspected a confound

GEM-3bconfirmed it: −0.021
(1 run)

DD-22-MATCHEDresolved: −0.003
(3 seeds)

Methodological Lesson

The direction was real. The magnitude was 95% artifact. Good methodology means re-running with matched conditions, even when the original finding supports your thesis.

Scale note: GEM-3b put the amplification at 22×, but that ratio divides by a near-zero denominator from a single unseeded run, so the record (KC#GEM3) treats the ratio itself as unstable. The durable statement is the one the seeded re-run supports: under matched optimizers the Gemma bilateral effect is indistinguishable from zero (DD-22-MATCHED, Δ = −0.003, 3 seeds), and essentially all of DD-22’s reported magnitude was optimizer-driven. Direction is consistent across all three architectures re-tested (bilateral reduces extraction, standard CE increases it), but on Gemma itself the matched-optimizer effect is indistinguishable from zero, so that direction claim rests on Qwen (Δ = −0.351) and Llama (Δ = −0.272).

the bilateral effect optimizer / mechanism direction robust magnitude artifact