Ten mechanisms that appear in more than one approach on this hub. Observed in the papers, not invented. Each names where it is used and the number that shows its cost or its gain.
Keep every validated variant and sample parents from the whole set. One bad step then cannot poison the lineage. DGM measured the cost of dropping this: without open-ended exploration the agent ends at 23.0% on SWE-bench, close to its 20.0% start, because one bad self-modification blocks every later one. The archive is what makes the loop recover.
Every program loop needs a judge that is cheap, exact, and hard to fool. AlphaEvolve's authors state it as the stated limit of the method: a domain without an automated evaluator is out of scope, which excludes most natural science that cannot be simulated. ASI-Arch pays the other price: its judge is a real training run, 20,000 GPU-hours for 1,773 experiments.
What goes back from the judge to the proposer is either one number or the whole trace. A score is cheap and safe; a trace carries the reason for the failure. GEPA reads full execution traces and reaches its results with up to 35x fewer rollouts than GRPO, which sees only a reward. The trace is the "gradient" of the program loop.
Split the program into a discrete part that search handles and a continuous part that an optimizer handles. LLM-SR: the LLM writes f(x, params), BFGS fits params. SINDy: the human writes the library, least squares fits the coefficients. ASI-Arch moves the boundary up one level: the LLM writes the architecture, gradient descent fits the whole weight tensor. The optimizer never has to guess structure; the proposer never has to guess numbers.
FunSearch gives the LLM two scored versions of one subroutine and never the problem statement. Ernest Davis read this as total shallowness: a stronger reasoner would have nothing to reason about. AlphaEvolve inverts it with whole-file context and natural-language problem descriptions. The hidden version resists memorization; the open version reaches further. LLM-SR shows the price of openness: its fast convergence on Feynman equations was recall, which forced the authors to build LLM-SRBench.
FunSearch, AlphaEvolve, LLM-SR, DGM, ADAS, GEPA, DSPy and TextGrad never backpropagate into the LLM. All gain sits in the artifact: code, prompt, equation, architecture. The artifact can be diffed, versioned and read after a failure. The model has nothing to forget because it never changed.
DGM runs each self-modified agent on the benchmark before it enters the archive. ROME and MEMIT overwrite a weight matrix in closed form with no check of compatibility against the rest of the network. Gupta et al. measured the result: GLUE scores decline, then collapse to zero; ROME on GPT-J fails somewhere between 100 and 1,000 edits, MEMIT around 1,400 to 1,500. An unchecked commit is a time bomb in both code and weights.
A prompt tuned on 30 examples is a model trained on 30 examples. MAGE reports GEPA on GSM8K-Hard with 30 training examples at 34.0% held-out against 62.4% for the untouched seed prompt, a 28.4-point regression the authors call a structural risk of any failure-grounded optimizer with small training sets. Reading the artifact makes the overfit visible; it does not prevent it.
When weights must change, both approaches confine the gradient to LoRA adapters and freeze the base. SEAL: rank 64 on MLP and attention projections for knowledge, rank 128 for ARC. TTT: rank 128, fit fresh per task with AdamW. The adapter bounds the blast radius of a bad update and can be discarded. SEAL still reports forgetting across a sequential stream of self-edits; the adapter limits the damage, it does not remove it.
A program can only be found inside the space the prior spans. SINDy cannot recover dynamics outside its candidate library; Al-Khwarizmi shows ODEBench models 4 and 7 failing for exactly this reason. FunSearch can only mutate the one function the human left open. The LLM in LLM-SR widens the library to anything it can write in Python, and the 31.5% ceiling on LLM-SRBench says how far that reaches today.
The loop that works has a frozen proposer, an exact judge, an archive, and a boundary where search hands constants to an optimizer. The loop that breaks commits without validation, overfits a small judge, or lets the prior exclude the answer. The same failures appear on both sides of the weight/program line.