LLM-SR writes equation skeletons as Python code, then fits only the free constants continuously by least squares, leaving structure to discrete LLM search.
lane: program loop · arXiv:2404.18400 (LLM-SR) — ICLR 2025 Oral — Shojaee, Meidani, Farimani, Reddy (Carnegie Mellon / Penn State); arXiv:2504.10415 (LLM-SRBench) — ICML 2025 Oral — Shojaee, Nguyen, Meidani, Farimani, Doan, Reddy
params is continuous and fitted by BFGS or Adam against the MSE residual.Gives. The LLM writes f(x, params) in Python; BFGS fits params; structure stays discrete, numbers stay continuous.
Needs. Low-dimensional data, hundreds of points, a physical prior the LLM can write.
Breaks. 31.5% symbolic accuracy on LLM-SRBench; fast Feynman results were recall.
LLM-SR represents an equation skeleton as a Python function, def f(x, params): ... return y, not an expression tree, so the LLM can carry comments, physical units, and partial derivations inside the hypothesis it proposes. No expression-tree genetic-programming system, such as GPlearn or PySR, carries that. The program is the hypothesis; the LLM edits it; a numeric optimizer then fits only the placeholder constants. This forces the search over code structure, discrete and LLM-proposed, while LLM-SR pushes every continuous unknown into a params vector, fit by gradient or quasi-Newton methods. The split cleanly separates "what shape" from "what number."
1. Propose: an LLM, GPT-3.5-turbo or Mixtral-8x7B, samples b=4 new skeletons per call at temperature 0.8. LLM-SR prompts it with the problem spec, the evaluation and optimization function, and "experience demonstrations": in-context examples of prior skeletons plus their score trajectory (arXiv:2404.18400 Sec. 2.2). 2. Fit: LLM-SR fits each skeleton's params vector against the data, either with numpy+BFGS via SciPy (Fletcher 1987) or torch+Adam (Kingma & Ba 2014); BFGS performed slightly better in ablation (Sec. 3.5). 3. Judge: fitness is negative MSE between predicted and true targets, s = −MSE(ŷ, y). The paper reports results as Normalized MSE (Sec. 2.3, 3.3). 4. Keep: scored skeletons go into an islands-model buffer borrowed directly from FunSearch (Romera-Paredes et al. 2024, cited Sec. 2.4), with m=10 islands. A candidate joins its island only if it beats that island's current best. Within an island, programs cluster by score signature for diversity. Sampling picks an island uniformly, then a cluster by Boltzmann selection. Every roughly 4 hours, the worst half of islands reset from a surviving island's best program.
Everything inside params is continuous and fitted by BFGS or Adam against the MSE residual. Everything else is discrete and searched by LLM sampling plus island selection: which variables appear, which operators compose them, the functional form itself. The only signal crossing that boundary is the scalar fitness score, negative MSE, fed back into the next prompt's "experience demonstrations." No gradient or residual vector crosses into the LLM's proposal step, only the scored trace of past skeletons.
The method targets small, noisy, low-dimensional datasets, one to a handful of input variables and hundreds of points, where a closed-form, physically interpretable equation is the deliverable and a black-box regressor would not be checkable against theory. Each candidate costs one LLM call plus a full BFGS/Adam fit, so cost scales with iteration count, not dataset size. Table 1 and Fig. 3 (Sec. 3.2-3.3) show LLM-SR reaching low NMSE in far fewer iterations than GPlearn/PySR's 2M+ generations, trading per-step LLM compute for fewer total search steps. The authors state this caveat directly: "the computational cost of iterative LLM queries and parameter optimization could be prohibitive for large-scale problems" (Sec. 5).
On Table 1 (arXiv:2404.18400 Sec. 3.3), LLM-SR (Mixtral) reaches OOD NMSE of 0.0002 on Oscillation 1 versus PySR's 0.3106 and GPlearn's 0.5567; on E. coli bacterial growth, LLM-SR (Mixtral) OOD NMSE 0.0037 versus PySR 1.0141 and GPlearn 1.039. The paper states the gap "is more pronounced in the OOD setting" (Sec. 3.3). No independent production deployment found.
1. Memorization on Feynman equations: Sec. 3.1 states that "LLMs have significant memorization issues with these well-known physics equations." App. C attributes LLM-SR's fast convergence there to recall, rather than discovery. This motivated LLM-SRBench's construction. 2. LLM-SRBench ceiling: the best system, LLM-SR with GPT-4o-mini, reaches only 31.53% symbolic accuracy on LSR-Transform (arXiv:2504.10415 Table 1, Sec. 3.2). The paper attributes this to "the fundamental challenges of this task" (Sec. 3.2) rather than a single named failure mode, and Sec. 3.3 explicitly finds the difficulty "is not necessarily due to the structural complexity." 3. Follow-up critique (FunctionEvolve, found via WebSearch): the paper reports that LLM-SRBench has "input-variable linear dependence and weak nonlinear factor variation that undermine structural identifiability." That is a benchmark-construction flaw, not just a model limitation. 4. Backbone sensitivity: on LLM-SRBench Table 1, LLM-SR with GPT-3.5-turbo scores only 10.81% symbolic accuracy on LSR-Transform, versus 30.63% for Llama-3.1-8B and 31.53% for GPT-4o-mini. The same method swings 3x with backbone choice.
LLM-SR borrows the islands and clustering population model directly from FunSearch (Romera-Paredes et al. 2024), and the BFGS constant-fitting idea from classical nonlinear least squares (Fletcher 1987). Its own authors built LLM-SRBench specifically to stress-test LLM-SR-style memorization, via LSR-Transform, pivoted Feynman equations filtered for non-recallable form, and LSR-Synth, GPT-4o-generated synthetic terms, expert-validated. IGSR (arXiv:2605.29184, "Influence-Guided Symbolic Regression," ICML 2026 submission) evaluates against LLM-SRBench as one of its benchmarks. PiT-PO (arXiv:2602.10576, KDD 2026) targets the same LLM-equation-discovery space with RL-based token regularization, but its abstract does not name LLM-SR or LLM-SRBench. This is an unconfirmed lineage link. UNVERIFIED pending full-text read.
Verified: WebFetch on https://arxiv.org/html/2404.18400 (full HTML, Sec. 1-5 read) and https://arxiv.org/html/2504.10415 (full HTML, Sec. 1-3 read); WebFetch on arXiv abstract pages for 2605.29184 and 2602.10576; WebSearch for repo and follow-ups. · code: https://github.com/deep-symbolic-mathematics/LLM-SR (MIT, companion repo deep-symbolic-mathematics/llm-srbench)