Code as prior, gradient as fit. Why an LLM should write a learnable program instead of editing its own weights. Sourced map, 2026-11-03.
Both loops start from a model that proposes a change and a judge that scores it. They differ in what the change is made of.
The model writes training data or an update rule, then applies it to its own parameters. SEAL, Titans, MEMIT.
The model writes a program skeleton. An optimizer fits the parameters to data. A judge scores the result. LLM-SR, AlphaEvolve, Darwin Gödel Machine.
Each result below holds the model fixed and improves code. Numbers come from the cited paper.
The agent rewrites its own code, not its weights, and keeps every validated variant in an open archive. SWE-bench 20.0% → 50.0%, Polyglot 14.2% → 30.7%. Without self-modification it plateaus at 39.0%. The strongest "self-improving agent" result on record edits source.
The LLM reads its own execution traces and rewrites the program, prompts and structure, in natural language, with Pareto selection per instance. Beats GRPO weight-RL by +6% on average, up to +20%, with up to 35x fewer rollouts. Reflection over a program beats gradient over weights at equal budget.
Gemini proposes code mutations, automated evaluators score them, the loop keeps the best. Found a 48-multiplication algorithm for 4×4 complex matrix multiplication, the first gain over Strassen in that setting in 56 years. Also used on datacenter scheduling and on the training of its own base model. The output is a program a human can verify.
Researcher, Engineer and Analyst agents propose linear-attention architectures as code; each one is trained and scored. 1,773 experiments, 20,000 GPU-hours, 105 architectures that beat the existing designs. The gradient runs inside the loop, over the parameters of each proposed program.
ADAS: a meta-agent writes agent code into a growing archive; the found agents beat hand-built ones and transfer across models. FunSearch: a frozen LLM plus an evaluator in an evolutionary loop over code found new cap-set constructions and a better bin-packing heuristic. Both ship compact programs, not weights.
The pattern Kron asked about has a name in equation discovery: the LLM proposes the form, a numeric optimizer fits the constants, the residual says what to change.
Equation skeletons written as code from scientific priors; evolutionary search over skeletons, optimizer over parameters. Beats symbolic-regression baselines, with the largest margin out of domain, where a wrong black box has nothing to fall back on.
239 tasks built to block recall of memorized laws. The best system reaches 31.5% symbolic accuracy. The route is real and it is far from solved: most of the time the LLM proposes the wrong skeleton.
The same idea applied to LLM pipelines. DSPy compiles declared modules into a pipeline and tunes the program against a metric. TextGrad runs autograd over a graph of LLM calls and sends natural-language feedback backward: GPT-4o GPQA 51% → 55%. The program is the thing that learns; the model stays fixed.
A program can learn from data only if an optimizer can reach its parameters. These are the tools that make a program differentiable or fittable.
The model writes its own fine-tuning data and update directives; RL rewards the edits that raise downstream scores after the weight update. Its self-written data beats GPT-4.1-written data on their tasks. The authors list catastrophic forgetting and compute cost as open problems.
Titans updates a memory module's weights at test time from a surprise signal and scales past 2M tokens of context. ROME rewrites one fact with a rank-one update to one MLP layer; MEMIT spreads thousands of edits over several layers. These solve memory and recall. None of them produce a reason you can read.
Forgetting is the documented cost. Three surveys of continual learning in foundation models report the same thing after adaptation: 2404.16789, 2405.18653, 2506.13045. The finding: performance drops on the old domains. A source diff cannot forget; a weight diff can.
The mechanistic side of this exists without LLMs. The LLM side exists without mechanism. Nobody has joined them end to end.
FLeCS wires gene-network structure into coupled differential equations for single-cell dynamics. CLAMP (ICML 2026) fits a differentiable Hill-kinetics regulatory network and treats a perturbation as a steady-state constraint. Φ-ROM puts a differentiable PDE solver inside reduced-order-model training and generalizes to unseen parameters better than data-only models.
MechPert steers an LLM agent toward directed regulatory relationships for few-shot perturbation prediction: +10.5% relative gain over similarity-based prompting. The LLM supplies structure; nothing is fitted.
No paper found puts an LLM inside a differentiable cell or physics simulator with the loop closed: LLM writes the mechanism as code, the optimizer fits the rate constants, the residual goes back to the LLM. The pieces are on this page. The join is unclaimed.
Methods that scale with compute beat methods that encode human structure, and they have done so every time. A hand-written skeleton is human structure. When data is plentiful, expect the black box to win.
Program space is discrete. The optimizer can fit the constants but cannot walk from one skeleton to a better one; that step is search, and search is what costs. LLM-SRBench's 31.5% ceiling is this cost measured. Fixes are relaxations of the program space or neural-guided search, both still open.
Non-experts rarely supply enough examples for a synthesizer to learn the intended program. The "a human can teach it" argument for programs is weaker than it sounds.
Verified against the arXiv abstract or primary page: LLM-SR, LLM-SRBench, ASI-Arch, SEAL, GEPA, Darwin Gödel Machine, AlphaEvolve, ADAS, JAX-MD, Neural ODEs, Pyro, DreamCoder, Bayesian Program Learning, Scallop, ROME, MEMIT, Titans, the bitter lesson. Items marked lead come from search results only and carry no numbers beyond their abstract.