A meta agent writes agent code as a forward() function in an unbounded search space, not a fixed block library, and the archive keeps only working code, never weights.
lane: program loop · arXiv:2408.08435 "Automated Design of Agentic Systems" — ICLR 2025, submitted 2024-08-15 (v2 2025-03-02) — Shengran Hu, Cong Lu, Jeff Clune (UBC, Vector Institute)
Gives. A meta agent writes any agent as a Python forward(); every computable building block is in reach.
Needs. One scalar objective and a sandbox; $300-500 per search.
Breaks. Drifts to over-complex topologies; an independent re-run found gains subtle and inconsistent.
Every other agent-design-search method on this hub searches over a constrained representation. DGM edits an existing agent's own files. AlphaEvolve mutates a scored program. ADAS's Meta Agent Search gives the meta agent a near-unbounded target instead. It writes arbitrary Python that defines a new agent's forward() function, inside a framework under 100 lines long. That framework exposes only FM query APIs and prompt formatting (Sec. 3). The paper's thesis: code can express any computable building block — prompting, tool use, control flow, multi-agent orchestration. Searching directly in code therefore subsumes prior work that searches only prompts or only fixed topologies.
1. Propose: A meta agent, GPT-4 in reported runs, reads the archive of discovered agents and their recorded performance. It then invents a new agent, writing code for a forward() function (Sec. 3). 2. Self-check: The candidate undergoes two rounds of self-reflection focused on novelty before it ever runs. If it errors during evaluation, it gets up to five automatic repair attempts (Sec. 3). 3. Judge: The candidate agent runs on validation data for the target domain. Scoring is accuracy for MGSM/MMLU/GPQA, F1 for DROP, and exact-match-style scoring for ARC. The score is the sole search signal (Sec. 3, Sec. 4.1-4.2). 4. Keep: Surviving agents, both code and measured performance, join an archive. The archive can be seeded with hand-designed baselines such as Chain-of-Thought and Self-Refine (Algorithm 1, Appendix H: "Initialize archive 𝒜 with baseline agents"). An "Empty Initialization" ablation runs the search with no seed agents at all (Appendix I, Table 6).
Nothing is continuous. The FM weights stay frozen. Every object the loop manipulates is discrete, text-level, and non-differentiable: agent source code, the archive, the self-reflection critique text. What crosses from judge to proposer is a scalar validation score plus the agent's own code. Both feed back into the meta agent's next prompt as context. No gradient and no residual vector exist anywhere in the loop.
ADAS needs a domain with a cheap, automatic scoring function over a modest validation set. The paper used 20 validation and 60 test questions per ARC run (Sec. 4.1). Standard splits with 30 search iterations per domain cover the reasoning benchmarks (Sec. 4.2). Cost scales with iterations times per-candidate evaluation cost. Appendix J states a single ARC search-and-evaluation run "costs approximately $500 USD in OpenAI API costs." A reasoning or problem-solving run "costs about $300 USD." Both cite "the primary expense" as querying gpt-3.5-turbo-0125 during evaluation of discovered agents. This buys a reusable artifact, agent code transferable across tasks and models, rather than a one-off answer. The search itself does not scale down. Every candidate still needs a full evaluation pass, and the paper only ever searched one domain at a time (Sec. 6: "we targeted only one domain during the search").
Table 1 reports Meta Agent Search beating the best hand-designed baseline on every domain tested. DROP F1: 79.4±0.8 versus Role Assignment's 65.8±0.9, a gain of 13.6. MGSM accuracy: 53.4±3.5 versus LLM Debate's 39.0±3.4, a gain of 14.4. MMLU accuracy: 69.6±3.2 versus OPRO's 67.6±3.2, a gain of 2.0 with overlapping confidence intervals. GPQA accuracy: 34.6±3.2 versus Self-Refine's 31.6±3.2, a gain of 3.0. On ARC, Figure 3 and Table 3 show the top discovered agents reaching 13.7±3.9, 13.3±3.8, and 12.7±3.9, versus COT-SC's 8.0±3.2. Transfer results appear in Table 2 and Table 3. The MGSM-discovered "Dynamic Role-Playing Architecture" scores 69.5±3.2 on GSM8K, versus LLM Debate's 43.6±3.4, a gain of 25.9. It scores 31.2±3.2 on GSM-Hard, versus Role Assignment's 18.0±2.7, a gain of 13.2. The ARC-discovered "Dynamic Memory and Refinement Agent" transfers across models too. It scores 9.7±3.3 on Claude-Haiku, 37.0±5.3 on GPT-4, and 48.3±5.7 on Claude-Sonnet, versus Self-Refine's 39.3±5.5 on Claude-Sonnet.
1. Independent reproduction finds ADAS loses to a simple baseline on average: Multi-Agent Design, Zhou et al., arXiv:2502.02533, re-ran ADAS with Gemini 1.5 as both optimizer and evaluator, 30 rounds, 3 validation evals per round (App. B.2). On Gemini 1.5 Pro, ADAS's average score of 69.72 trails Multi-Agent Debate's 70.26 (Table 1). On Gemini 1.5 Flash, ADAS averages 64.75 versus Debate's 65.91. A fixed-topology baseline beats ADAS's searched agents on average under this reproduction. The paper states: "ADAS only brings subtle gains even by already conditioning its meta-agent generation based on the common forms of agents." It diagnoses the cause: "ADAS gets trapped in discovering over-complex topologies that appear to be less effective than the prompt design space," and "the meta-agent keeps proposing complex topologies but without optimizing the prompt design." 2. Per-task losses confirm the pattern: on Gemini 1.5 Pro, ADAS trails Debate on MuSiQue, 41.95 versus 46.00, and on LiveCodeBench, 65.17 versus 73.67 (Table 1, arXiv:2502.02533). On Gemini 1.5 Flash, ADAS trails Self-Consistency on MATH, 68.00 versus 69.33. The original paper's "beats hand-designed baselines" finding does not survive a different optimizer, a different evaluator model, and a stronger set of baselines. 3. Authors admit several limits of their own method (Sec. 6). The paper calls the approach "both expensive and misses a lot of information," referring to evaluation functions that return only a scalar. It states "we only evaluate Meta Agent Search on single-step QA tasks" and "we only consider one objective (i.e., performance) to optimize." It adds "we targeted only one domain during the search." The search algorithm itself is "relatively simple." Generated code "could still act destructively" and required containerized execution plus manual inspection. 4. A field-level critique questions the whole evaluation regime. "AI Agents That Matter," arXiv:2407.01502, is by Kapoor, Stroebl, Siegel, Nadgir, and Narayanan at Princeton, published in TMLR 2025. It argues the broader agent-benchmarking field has a "narrow focus on accuracy without attention to other metrics." That focus produces "SOTA agents [that] are needlessly complex and costly." It argues "many agent benchmarks have inadequate holdout sets, and sometimes none at all." That gap yields agents "fragile because they take shortcuts and overfit to the benchmark in various ways." This paper does not name ADAS directly in its abstract — confirmed only via the abstract page, full text UNVERIFIED this session. Its argument structure is still the one independently-cited critique of the entire category ADAS belongs to.
ADAS is a direct ancestor of DGM, in the same Clune/Vector-Institute line. Both keep an archive of agent code scored by benchmark performance. ADAS's meta agent and the agent population stay separate entities, while DGM collapses that separation — the searching agent edits itself. ADAS itself generalizes earlier narrower searches into unconstrained code search. It subsumes prompt-only optimization, OPRO, used as one of its own baselines, and fixed multi-agent topologies, LLM Debate and Self-Refine. Multi-Agent Design, 2502.02533, and the Mass/AFlow line of later work react directly to ADAS. They narrow the search space back down, optimizing prompts within a fixed or lightly-searched topology. Unconstrained topology search underperforms prompt optimization at comparable cost, per their own finding.
Verified: WebFetch on https://arxiv.org/html/2408.08435 (full HTML, two offsets covering Sec. 1-6, Table 1-6, Appendix C/H/I/J); WebFetch on https://arxiv.org/html/2502.02533 (Multi-Agent Design, independent reproduction of ADAS); WebSearch for "AI Agents That Matter" critique and WebFetch on https://arxiv.org/abs/2407.01502 (abstract only — full text not opened, UNVERIFIED beyond abstract). · code: not independently located this session (paper references a code release; URL not confirmed via WebFetch — UNVERIFIED)