Three LLM agents run a closed research loop where every proposed program is a full neural-network architecture, trained and scored by gradient descent, not a cheap scoring function.
lane: program loop · Xu, Liu, Nan, Ye, Hu, Qin, Liu, "Neural Architecture Discovery via Autonomous Evolution" (v1 title: "AlphaGo Moment for Model Architecture Discovery"), arXiv:2507.18074, accepted COLM 2026 (camera-ready); v1 submitted 2025-07-24, v2 camera-ready submitted 2026-10-08. Shanghai Jiao Tong University et al.
Gives. The candidate is an architecture; judging it means training it at 20M, 340M, then 1.3B parameters.
Needs. About 20,000 GPU-hours for 1,773 experiments.
Breaks. Validation stops at 1.3B; the paper's own count moved from 106 to 105.
Every other program-loop approach in this hub, FunSearch, AlphaEvolve, LLM-SR, scores a candidate program with a fast, deterministic evaluator: run the code, check a numeric output. ASI-Arch's candidate program is a neural-network architecture definition. Judging it means actually training that network. An Engineer agent runs real gradient descent (Sections 3.2-3.3). It starts at 20M parameters on 1B tokens, promotes survivors to 340M parameters on 1B tokens, then scales the best few to 1.3B parameters on 15B tokens (Appendix B.1, B.3). The "evaluator" is therefore itself a full ML training pipeline, with its own stochasticity, cost, and failure modes: training divergence, OOM, loss spikes. It is a qualitatively heavier judge than any scoring function in the rest of this hub.
1. Propose: the Researcher agent drafts a new architecture and implements it as code in one pass. It conditions on retrieved entries from a shared Cognition base, roughly 150 curated insights distilled from about 100 linear-attention papers. It also draws on the Experience Repository's top-50 candidate pool (Sec. 3.1, 3.4). Each proposal passes a novelty check and a sanity check before training. 2. Fit/Judge: the Engineer agent trains and evaluates the proposed architecture. A debugging sub-agent fixes implementation errors, and a monitor kills pathological runs (Sec. 3.2). This is the only stage in this hub's program-loop lane where "fitting" means an actual gradient-descent training run, rather than fitting a handful of scalar constants. 3. Analyze/Keep: the Analyst agent turns each run's results into structured comparative analysis against the parent architecture and sibling runs. It archives both the architecture and the analysis into the Experience Repository (Sec. 3.3). That repository becomes the retrieval source for the next Researcher proposal, closing the loop. 4. Scale-up verification: the pipeline promotes candidates surviving the 20M-parameter exploration stage, 1,350 of 1,773 experiments, to 340M-parameter verification, about 150 candidates. It scales the strongest further still, to 1.3B parameters on 15B tokens. The authors argue this "rules out artifacts of the resource-constrained exploration setting" (Sec. 3.5, Appendix B.3).
The Researcher LLM discretely proposes architecture structure: which operators, which gating, which recurrence form. Gradient descent never touches that structure directly. Unlike FunSearch, AlphaEvolve, and LLM-SR, the thing gradient descent does touch inside this loop is not a handful of constants but the full weight tensor of a trained network. The boundary between discrete search and continuous fitting sits one level higher than in the other approaches. Gradient descent optimizes the weights of each proposed architecture. The Researcher/Analyst exchange of text, novelty checks, comparative analysis, Cognition retrieval, is what crosses back into the next proposal, not a gradient or residual.
This is the most compute-hungry approach in the program-loop lane, by a wide margin. The run totals 1,773 experiments across all three scales. It costs roughly 10,000 GPU-hours for the 20M-parameter exploration stage alone, and roughly 20,000 GPU-hours overall (Sec. 3.5, intro). That is orders of magnitude more compute than FunSearch's millions of cheap CPU-bound evaluations, or AlphaEvolve's hours-long evaluation cascade, because each "evaluation" here is itself a training run. The authors claim an empirical "scaling law for discovery": near-linear growth of cumulative SOTA-architecture count versus number of exploration experiments (Fig. 2). This is their central scaling argument. But it rests on fitting a trend to outputs from a single run of their own system, not a theoretical derivation. No one reported a second independent run to confirm the slope holds elsewhere.
1. Smaller, interference-driven gains on some baseline families: Sec. 5.4 reports that cross-family gains on Mamba2 and HGRN2 were smaller than on DeltaNet. The authors attribute this to baseline saturation, and to externally inserted rules overlapping with or conflicting with those models' own built-in stabilization mechanisms. The discovered modifications do not transfer cleanly across architecture families. 2. Version-to-version number discrepancy, 105 versus 106: the camera-ready paper downgraded its own headline count from 106, v1 preprint, to 105, v2 COLM 2026 accepted version. The GitHub repository and v1 preprint still advertise 106. This is an unresolved inconsistency in the paper's own self-reported central number. 3. No explicit limitations section: the paper's Conclusion (Sec. 6) lists future work — larger hypothesis spaces, family-specific adapters, efficiency optimization. It never frames its own validation scale, 1.3B parameters and 15B tokens, still small by frontier-model standards, as a limitation. This research draws that characterization from what the paper omits, not from a stated caveat. 4. Scaling-law claim is a single-run empirical fit, per secondary synthesis, emergentmind topic page, UNVERIFIED beyond snippet level. The near-linear compute-to-discoveries trend has no theoretical derivation. No one reported a second independent run testing whether the slope generalizes. One summary notes the law ignores the qualitative value of each discovery, counting only SOTA pass/fail. 5. Public skepticism, unverified rigor: anonymous commenters on the alphaXiv discussion page for 2507.18074, found via WebSearch, not independently cross-checked, argued the paper overstates novelty, calling it "basically LLM-based NAS." They questioned whether 20M-parameter exploration models can support claims about scaling, given the reported 20,000 GPU-hour cost makes independent reproduction difficult. These are forum opinions, not a peer-reviewed rebuttal, flagged here as leads rather than citations. 6. No independent third-party replication found: this research found no dedicated 2026 paper that reproduces ASI-Arch's pipeline, or re-validates its 105/106 discovered architectures, at a larger training scale than the paper's own 1.3B/15B-token check.
ASI-Arch's Researcher-Engineer-Analyst loop generalizes the propose/evaluate/archive structure shared with FunSearch and AlphaEvolve. But it swaps their fast code-evaluator for a real training-and-benchmark pipeline. That makes it closer to a fully automated neural architecture search, NAS, system, with an LLM as the proposal generator instead of a reinforcement-learning controller or random search. Its "Cognition base," distilled insights from about 100 linear-attention papers used as retrieval context, plays a role analogous to AlphaEvolve's rich natural-language prompt context. It is sourced from literature, though, rather than from the system's own evaluation history. This research found no later work building directly on ASI-Arch's specific pipeline this session. The closest adjacent 2025-2026 work found by search, ONNX-Net, arXiv:2510.04938, covers fast architecture-performance prediction, but this research could not confirm it cites or extends ASI-Arch directly — flagged UNVERIFIED.
Verified: WebFetch full arXiv HTML at three offsets (0-100000: modules/mechanism/Table 1; 40000-139704: conclusion/future-work; 100000-139704: appendix prompts/training tables); WebFetch arXiv abstract page (title, authors, submission history, venue); WebFetch GitHub repo README; WebSearch standard + extended for the v1/v2 discrepancy, "scaling law for discovery" phrasing, and skeptical commentary (alphaXiv discussion page, Medium secondary commentary — both UNVERIFIED beyond snippet-level). · code: https://github.com/GAIR-NLP/ASI-Arch — pipeline (evolve/eval/analyse), MongoDB architecture database, RAG-based cognition base; README states "we open-source all 106 discovered state-of-the-art linear attention architectures" (count conflicts with the camera-ready paper's 105 — see below)