Some training methods behave politely when you build a model from nothing and turn vicious when you point them at a model that already works. This is the story of watching that happen under preregistered conditions, and of the design flaw I built into my own follow-up sweep, which I'll get to, because it's the kind of flaw that flatters your results and never announces itself.
The method in question is decoupled training: split a transformer into blocks that are gradient-isolated from each other, give each block its own local next-token objective through a small auxiliary head, and train the blocks without end-to-end backpropagation. In separate from-scratch experiments the approach had looked reasonable, which raised the question that actually matters for anyone who wants to keep updating a model they already have: does it work as a way of continuing to train a pretrained model? Specifically, can full-parameter decoupled training absorb new knowledge that LoRA, the incumbent cheap method, cannot?
Setup
We generated 1,500 synthetic documents about 300 invented people, with fabricated names, towns, and roles, so that nothing is recallable from pretraining. Each entity has four attributes; question-answer pairs are trained for only two of them. The primary probe, qa_unseen, asks about attributes trained only in declarative form: your notes are statements, your queries are questions, and knowledge extraction across that gap is the realistic case. Sixty held-out entities give the chance floor (~0.06), and perplexity on general held-out text measures forgetting.
Four arms: the base model as floor, LoRA (rank 32, all attention and MLP projections) as incumbent, full-parameter end-to-end as ceiling, and full-parameter decoupled (four gradient-isolated blocks, each with an auxiliary head initialised so training starts function-preserving).
One design decision worth stealing: compare frontiers, not points. Calibration showed absorption trades directly against forgetting along learning rate, so any single-learning-rate comparison is gameable, by accident or otherwise. Each arm was swept over four learning rates spanning its useful range, and arms are compared as absorption-versus-forgetting curves.
Result
Best point per arm on the absorption/forgetting frontier, first sweep (no rehearsal):
| arm | absorption | at perplexity |
|---|---|---|
| full end-to-end | 0.314 | 27.9 |
| LoRA | 0.321 | 57.3 |
| full decoupled | 0.175 | 176.6 |
Chance floor ≈ 0.06. Base-model perplexity 5.98 — every arm pays real forgetting, but decoupled pays catastrophically more for less.
Second sweep, with 1:1 rehearsal on general text mixed into every batch:
| arm | absorption | at perplexity |
|---|---|---|
| full end-to-end | 0.562 / 0.317 | 48.4 / 34.5 |
| LoRA | 0.418 / 0.274 | 30.1 / 34.9 |
| full decoupled | 0.399 / 0.158 | 134.9 / 121.0 |
Two representative frontier points per arm. The ordering is unchanged: decoupled needs 3–4× the forgetting for comparable absorption.
Replicated in both sweeps, with and without rehearsal: decoupled training needs three to four times the forgetting to absorb the same knowledge. It is the worst arm in every cell. The diagnosis is structural. A pretrained transformer has a division of labour across depth: early layers build representations that only pay off later. A local next-token loss on every block forces early layers to be immediately predictive, and the gradient pressure that satisfies the local objective tears up exactly the long-range structure pretraining spent its budget building. Training from scratch has no such structure to destroy, which is precisely why the method's from-scratch behaviour was no warning at all.
Both sweeps voided the strict band
Honesty requires reporting what the preregistration actually said. The validity guard required some arm to beat chance by ≥10 points inside the low-forgetting band. Best in-band margin: +2.8 points in sweep one, +3.2 in sweep two. Both sweeps are void under the strict rule, reported as written, band not widened after seeing the data. The frontier ordering above is informative anyway, because it is consistent at every operating point, but the void itself is a finding: knowledge injection of this kind, 20 epochs over 52k tokens of narrow synthetic prose, cannot be had at this scale without material forgetting, no matter which trainer you use.
The flaw I built myself
The rehearsal sweep has a contaminated metric, and I put it there. The rehearsal text and the forgetting eval came from the same distribution (different slices of the same corpus), which means rehearsal directly optimises the forgetting metric: one arm reached perplexity 5.57, better than the base model's 5.98, which should be impossible for a model being pushed away from its pretraining. The cross-arm comparison survives because all arms got identical rehearsal, but the sweep's absolute numbers are lenient and not comparable to the first sweep's. The general lesson costs nothing to state and a lot to relearn: never draw rehearsal data from the distribution you use to measure forgetting. The contamination flatters your result, so nothing about it looks like an error.
What this bought us
A clean decision, cheaply. The follow-up engineering this experiment was gating did not get built, and the effort went where the frontier pointed instead. Eighty runs on a consumer card, two nights, and a preregistered kill is a better outcome than a year of building on an assumption that two tables could refute.