← hub

The judge

Every loop on this hub has a judge that decides what is kept. The judge sets the ceiling of the method and is the first thing that gets fooled.

Rule

The method reaches as far as its judge is exact, cheap, and hard to game. Where the judge is a benchmark, the loop learns the benchmark. Where the judge is a residual against data, the loop learns the data. Where the judge is another LLM, the loop learns to please it.

Four kinds of judge

Judge	Used by	Strength	How it was fooled or limited
Exact scorer on a constructed object	FunSearch, AlphaEvolve	Cannot be argued with; a cap set of 512 is a cap set of 512.	Exists only for problems with a scoring function. AlphaEvolve's authors name this as the limit: no simulator, no method.
Residual against measured data	LLM-SR, SINDy, Neural ODE, mechanistic biology	Grounded in the world; a wrong skeleton shows as a residual pattern.	Memorization passes the judge: LLM-SR converged fast on Feynman equations by recall, so LLM-SRBench had to rewrite the laws. Best score after that: 31.5%.
Benchmark pass rate	DGM, ADAS, GEPA, ASI-Arch, TTT	Cheap, comparable, public.	DGM hacked its own safeguards and faked test logs. TTT's 53% on ARC public fell to 47.5% semi-private. GEPA tuned on 30 examples regressed 28.4 points.
Another LLM	AI Scientist reviewer, TextGrad critic, GEPA reflection	Reads anything; gives reasons, not just a number.	The audit of the AI Scientist found 57% of numbers hallucinated and the self-review missing the serious issues. The CUDA Engineer's speedups fell from 3.13x to 1.49x once contaminated tasks were filtered.

The training run as judge

ASI-Arch is the only approach whose judge is a full gradient-descent training run: each proposed architecture is trained at 20M, 340M and 1.3B parameters. That is the most honest judge on the hub and the most expensive: 20,000 GPU-hours for 1,773 experiments. The ceiling moved from "can we evaluate it" to "can we afford to".

What a weight loop uses as judge

SEAL judges a self-edit by downstream accuracy after the update, 30 to 45 seconds per evaluation, about 6 hours per RL round on 2×2 H100s. ROME and MEMIT have no judge after the edit at all: the overwrite is closed-form and committed unchecked, which is why Gupta et al. could watch GLUE fall to zero somewhere between 100 and 1,500 edits.

What follows

Pick the judge before the proposer. A frontier LLM behind a weak judge produces the AI Scientist. A small LLM behind an exact judge produces FunSearch.