Data regime — Programs that learn

Where a written program beats a black box, where scale beats the program, and the numbers that mark the line.

Rule

Programs win when data is small, constraints are hard and the judge is exact. Scale wins when data is large and the judge is soft. The bitter lesson holds on the right side of that line and has no case on the left.

Small data, hard constraints: programs

LLM-SR

Hundreds of points, one to a few variables, a physical prior. The deliverable is a formula someone can check against theory.

+ beats symbolic-regression baselines, largest margin out of domain

− 31.5% symbolic accuracy on LLM-SRBench

P-NET, FLeCS, CLAMP

Known pathway or network topology, few samples, parameters to fit.

+ P-NET's structured sparsity beats a dense net of equal size at ≤500 samples

− the advantage is the small-sample regime; it fades as N grows

FunSearch, AlphaEvolve

No data at all; an exact scorer instead.

+ cap set 512 over 496; 48-multiplication 4×4 matmul

− no scorer, no method

Where the line moves with N

MechPert shows the crossover inside one paper: at N=500, plain binary consensus at 0.606 beats the confidence-weighted mechanistic method at 0.589. In HepG2 and Jurkat, LLM-sourced regulatory hypotheses lose to random gene sampling. The mechanism helps exactly while data is scarce.

Large data, soft judge: weights

Titans keeps learning per token across 2M-token contexts; no program could be written for that. ARC is the contested middle: program synthesis at 42 to 50% with ~8,000 programs per task, test-time training at 53% on an 8B model, o3 at 87.5% with 172x compute. Compute bought the last result, and no one can read what it learned.

Cost per improvement

Approach	Cost stated in the paper	Gain stated in the paper
GEPA	up to 35x fewer rollouts than GRPO	+6% average, +20% max
DGM	about $22k compute	SWE-bench 20.0% → 50.0%
ADAS	$300 to $500 per search	gains an independent re-run called subtle and inconsistent
ASI-Arch	20,000 GPU-hours	105 architectures, validated to 1.3B parameters
SEAL	about 6 hours per RL round on 2×2 H100	self-written data beats GPT-4.1-written data
AI Scientist	$10 to $15 per paper	42% of experiments failed to run; 57% of numbers hallucinated in the audit

What follows

Ask how much data the judge sees and how exact it is. Small and exact: write the program and fit the constants. Large and soft: train. In between, write the skeleton and let the gradient fill it; that is the whole claim of this hub, and LLM-SRBench says it is one third done.