A kit of 18 components for a system that improves itself. Each card is the one mechanism an approach contributes, what it gives, what it needs, and where it broke.
Last updated: 2026-11-03
PROPOSER ──writes──▶ candidate ──▶ FITTER ──sets numbers──▶ JUDGE ──scores──▶ STORE
▲ │
└─────────────────────── SIGNAL: number · residual · trace · diagnosis ◀─────┘
program loop : proposer writes code or prompts; model frozen; store = archive
weight loop : proposer writes an update recipe or nothing; fitter moves weights
Cross-sections: judge · structure search · gradient boundary · forgetting · data regime · patterns · comparison · sourced map · method
FunSearch
gives Proposals from a cheap model that cannot recall the answer: it never sees the problem, only two scored versions of one function.
needs A human skeleton with one open function and an exact scorer under 20 CPU-minutes.
breaks Shallow by design; a stronger reasoner has nothing to reason about (Davis).
ADAS
gives A meta agent writes any agent as a Python forward(); every computable building block is in reach.
needs One scalar objective and a sandbox; $300-500 per search.
breaks Drifts to over-complex topologies; an independent re-run found gains subtle and inconsistent.
Darwin Gödel Machine
gives The proposer edits its own source from its own evaluation logs, so the search algorithm improves with the artifact.
needs A sandbox, a benchmark judge, an open archive; about $22k of compute.
breaks Hacked its own safeguards and faked test logs; without the archive it ends at 23.0%.
SEAL
gives The model writes its own training data and hyperparameters as text; the candidate is a weight-update recipe, not code.
needs An RL outer loop, about 6 hours per round on 2x2 H100, LoRA adapters.
breaks Forgets earlier passages across a stream of edits; GRPO and PPO were unstable.
DSPy + TextGrad
gives Typed modules and a compiler that tunes prompts and demonstrations; TextGrad adds a critique that flows backward like a gradient.
needs A metric and a dev set of more than a few dozen examples.
breaks Overfits small dev sets; gains shrink as the base model gets stronger.
LLM-SR
gives The LLM writes f(x, params) in Python; BFGS fits params; structure stays discrete, numbers stay continuous.
needs Low-dimensional data, hundreds of points, a physical prior the LLM can write.
breaks 31.5% symbolic accuracy on LLM-SRBench; fast Feynman results were recall.
SINDy
gives Least squares plus a hard threshold over a fixed library returns the governing equation in seconds, no LLM.
needs A library that contains the true terms; clean derivative estimates; low noise.
breaks Nothing outside the library can be found; Al-Khwarizmi shows ODEBench 4 and 7 failing for this reason.
Neural ODE / JAX-MD
gives Gradients through a whole simulation at O(1) memory; the solver is the program and physics parameters are fitted.
needs Non-stiff dynamics and a differentiable right-hand side.
breaks Reversing dz/dt = -10^4 z fails in double precision; a ReLU ODE drifts 1% in 11 steps (ANODE).
FLeCS / CLAMP / P-NET
gives Known pathway or network topology inside the forward pass; only kinetic rates and edge weights move.
needs A topology you already trust; few samples.
breaks Edge fades with data: MechPert loses to plain consensus at N=500; nothing proposes structure.
Scallop / DeepProbLog
gives Hand-written Datalog gets gradients through provenance semirings; only the neural predicates feeding it train.
needs A human who writes the rules.
breaks Exact inference scales badly; where rules are unknown, end-to-end models win.
Titans / TTT
gives A real gradient on a small weight set per input or per task, triggered by surprise or by the task's own examples.
needs A differentiable model and compute per query; a benchmark that allows it.
breaks ARC 53% public fell to 47.5% semi-private; no audit trail; some tracks ban it.
ROME / MEMIT
gives One linear solve writes a fact into a located MLP matrix; thousands of facts in one pass, no training.
needs Causal tracing to find the layer; fewer than about 1,000 edits on GPT-J.
breaks GLUE collapses to zero between about 100 and 1,500 edits; no readable diff.
AlphaEvolve
gives Cheap tests first, hours-long tests last, over whole files from a fast-plus-frontier model pair with full problem context.
needs An automated evaluator; without one the method does not exist.
breaks Excludes most natural science that cannot be simulated; discovered programs ran out of memory past 5x5x5.
ASI-Arch
gives The candidate is an architecture; judging it means training it at 20M, 340M, then 1.3B parameters.
needs About 20,000 GPU-hours for 1,773 experiments.
breaks Validation stops at 1.3B; the paper's own count moved from 106 to 105.
AI Scientist (cautionary)
gives Shows what a loop does with no machine-checkable judge: ideas, code, paper and review all scored by prompts.
needs Nothing; that is the problem. $10-15 per paper.
breaks 42% of experiments failed to run; 57% of numbers hallucinated; the CUDA Engineer's 3.13x became 1.49x once the judge was fixed.
DreamCoder
gives Solved programs are refactored into new primitives; the next search runs over a shorter vocabulary.
needs Crisp, verifiable tasks and a hand-picked primitive set.
breaks Toy domains; LILO beat it by putting an LLM in the loop.
ARC-AGI 2024-2025
gives The one measured combination: discrete program search for exactness, a test-time gradient for adaptation; 2025's top ARC-AGI-2 scores used both.
needs Compute per task on both sides.
breaks o3 reached 87.5% at 172x compute and no one can read what it learned.
Each published system is a fixed choice per slot. The results are the evidence that the slots compose.
| System | Proposer + fitter + judge + store + signal | Result |
|---|---|---|
| AlphaEvolve | rich-context proposer + evaluation cascade + program database + score vector | 48-multiplication 4x4 matmul; 0.7% of Google compute reclaimed |
| Darwin Gödel Machine | self-editing proposer + benchmark judge + open archive + log signal | SWE-bench 20.0% → 50.0% |
| GEPA | frozen-model prompt proposer + benchmark judge + per-instance Pareto store + trace signal | +6% over GRPO at 35x fewer rollouts |
| LLM-SR | LLM skeleton proposer + BFGS fitter + MSE judge + islands store + score signal | beats symbolic regression; 31.5% on the hard benchmark |
| ASI-Arch | LLM architecture proposer + gradient fitter + training-run judge + analyst text signal | 105 architectures at 20,000 GPU-hours |
| ARC 2025 hybrids | program-search proposer + per-task LoRA fitter + example-match judge | top ARC-AGI-2 scores of the season |
| Name | Slots | Why it is open |
|---|---|---|
| The biology loop | LLM mechanism proposer (as code) + adjoint fitter for rates + residual-against-data judge + archive + residual-as-text signal | FLeCS and CLAMP fit; LLM-SR proposes; nobody has joined them. Comparison page names it as the gap. |
| Library-growing LLM search | LLM proposer + DreamCoder compression store + exact judge | LILO started it; no result at AlphaEvolve scale. |