What a weight edit costs that a program edit does not, with the measured numbers. And the one way program loops lose state too.
Sequential edits on CounterFact: GLUE F1 on SST-2, MRPC, NLI and CoLA declines, then drops to zero. ROME on GPT-J fails between about 100 and 1,000 edits depending on the sample; MEMIT lasts until about 1,400 to 1,500 on GPT-J and about 4,000 on GPT2-XL. The cause named by the authors: the edited layer's weight norm drifts until it no longer fits the rest of the network. A single disabling edit early can set the clock.
Across a sequential stream of self-edits, accuracy on earlier passages "gradually declines as the number of edits increases". The paper proposes replay, constrained updates or representational superposition as future fixes, none demonstrated. GRPO and PPO were tried and rejected as unstable; ReST-EM took their place.
Each task gets a fresh LoRA, fit by AdamW, then dropped. Nothing is forgotten because nothing is kept. The price is paid per query: the 53% public-set result fell to 47.5% on the semi-private split, and some ARC Prize tracks disallow the method over leakage concerns.
Program loops keep a text artifact and a frozen model, so nothing in the model degrades. They lose state in one way: when the archive is a single lineage. DGM without open-ended exploration ends at 23.0% on SWE-bench because one bad self-modification blocks every later one. The archive is the program loop's replay buffer.
After a ROME edit, the diff is a change in a weight matrix's norm. After a DGM edit, the diff is a Python patch with the evaluation log that motivated it. After a GEPA step, the diff is a prompt with the diagnosis that produced it. The weight loop can be measured for forgetting; the program loop can be read for it.
If the knowledge must change often, keep it in an artifact or a retrieval store and leave the weights alone. MEMIT's own curve says where "often" begins: about a thousand edits on GPT-J.