GEPA feeds a reflection LLM a rollout's full execution trace, not a scalar reward, to rewrite prompts, and keeps every Pareto-optimal candidate instead of one running best.
lane: program loop · arXiv:2507.19457 (v2, revised 2026-02-14) — ICLR 2026 Oral — Agrawal, Tan, Soylu, Ziems, Khare, Opsahl-Ong, Singhvi, Shandilya, Ryan, Jiang, Potts, Sen, Dimakis, Stoica, Klein, Zaharia, Khattab (UC Berkeley, Stanford, Notre Dame, Databricks, BespokeLabs.ai, MIT)
Gives. The proposer reads the whole execution trace, writes a diagnosis, then the fix; the store keeps every per-instance leader.
Needs. A strong reflection model and more than a few dozen training examples.
Breaks. 28.4-point regression with 30 examples (MAGE); loses to GRPO on AIME-2025.
Other optimizers in this hub, MIPROv2 and TextGrad, collapse a rollout to a scalar score or one textual gradient. GEPA instead feeds the reflection LLM the full execution trace: reasoning, tool calls, output, evaluator feedback. It has the LLM diagnose in prose why the instruction produced that trace, then rewrite it. The second piece is selection: GEPA uses a per-instance Pareto frontier, not hill-climbing on one aggregate. It tracks which candidate currently leads on each training instance, keeps every non-dominated candidate, and samples the next parent weighted by instance-wins.
1. Propose: GEPA samples a parent by Pareto weighting, runs it on a minibatch, and collects traces and feedback. It round-robins to one pipeline module; a reflection LLM reads that module's instruction, trace, score, and feedback, and writes a replacement (arXiv:2507.19457 Sec. 3). 2. Fit/accept: GEPA keeps the child only if it beats the parent on the same minibatch, then scores it on a held-out Pareto validation set. 3. Judge: a task-specific evaluator, such as exact match, F1, code execution, or LLM-as-judge. GEPA trusts whatever scalar and feedback the harness returns, and trains no separate reward model. 4. Keep: a per-instance best-score table decides survival. GEPA drops candidates that are not best anywhere. The optional "system-aware merge" (Appendix D.1) splices two Pareto-optimal candidates that share an ancestor but have disjoint module changes: per module, the changed side wins, or the higher-scoring side if both changed (ties random). GEPA invokes merge at most 5 times per run (Appendix E.4). 5. Output: the candidate with best mean score on the Pareto validation set.
Nothing in GEPA is gradient-fitted; base weights stay frozen. The only continuous quantity is the scalar evaluator score used to accept or reject a step and rank the Pareto frontier. One LLM call reads another's trace in prose and produces the instruction text itself. The "gradient" is a paragraph of diagnosis, not a vector. GEPA crosses the numeric-to-textual boundary only once, at evaluation: score plus feedback string go in, a revised prompt comes out.
GEPA targets frozen LLM pipelines that must be tuned cheaply against a task metric with few rollouts. The paper frames this as "when weight updates are not the best lever," for example closed models, small compute budgets, or RL instability. Rollout counts per benchmark range from 1,839 to 7,051 (3,593 for IFBench+Merge, 6,871 for HotpotQA), versus 24,000 for GRPO on the same benchmark. The "35x fewer rollouts" figure is the IFBench ratio: 678 rollouts to best prompt versus 24,000 (Sec. 4/Table 1). Dollar cost for the full GPT-4.1 Mini suite (Table 2) was under $500 total: GEPA $86, GEPA+Merge $67, MIPROv2 $76, and Trace+TextGrad $172 combined (Appendix E.3). Reflection-LLM call counts per iteration live in Appendix N, not read this session. UNVERIFIED exact count.
Table 1 (Qwen3 8B, aggregate across HotpotQA, IFBench, HoVer, PUPA, AIME-2025, LiveBench-Math): GEPA 54.85 vs. GRPO 48.91 (+5.94, abstract's "~6% avg"), vs. MIPROv2 47.84. Table 2 (GPT-4.1 Mini): GEPA 65.22 (+12.19 over baseline) vs. MIPROv2 58.67 (+5.64); GEPA+Merge 66.36 (+13.33). Per-benchmark GEPA-vs-GRPO (Table 1): HotpotQA 62.33 vs. 43.33, HoVer 52.33 vs. 38.67, PUPA 91.85 vs. 86.66, IFBench 38.61 vs. 35.88. HotpotQA's +19-point gap is the source of "up to 20%"; the Sec. 4 text itself says "up to 19%," a 1-point internal discrepancy.
1. Loses to GRPO on AIME-2025: Table 1 shows GEPA scoring 32.00 versus GRPO's 38.00 on Qwen3 8B, the one benchmark in the suite where RL-style weight optimization beat reflective prompt evolution outright. 2. Merge is unstable and only helps sometimes: GEPA+Merge degrades relative to plain GEPA on Qwen3 8B IFBench (28.23 vs. 38.61). The authors attribute this to fixed merge hyperparameters that are not tuned per model, and list adaptive merge timing as future work rather than a solved problem (Sec. 4 discussion). 3. Validation-budget inefficiency acknowledged by authors: most of the rollout budget goes to scoring candidates on the Pareto validation set, rather than to generating learning signal. The paper proposes smaller, dynamic validation subsets as future work, unimplemented (Sec. 4/Appendix discussion). 4. Independent critique (alphaxiv reviewer thread, via WebSearch): reviewers note that the headline RL-beating comparison comes from one task/model combination. They also note that GEPA's rollout-count advantage ignores wall-clock time and the extra reflection-LLM calls each iteration costs, a cost dimension the paper's own rollout tables do not report. 5. Practitioner reports (blog posts, via WebSearch, UNVERIFIED beyond snippet): unconstrained GEPA runs can grow prompts past 5,000 characters, "prompt bloat." Results reportedly shift with which model serves as the reflection LLM. One reviewer observed an evolved prompt that emitted a literal placeholder string in its output, a concrete reflection-loop failure mode, though not reproduced independently here.
GEPA borrows the Pareto-archive idea from multi-objective evolutionary search, and the idea of evolving natural-language artifacts via LLM reflection from the same family as FunSearch and AlphaEvolve-style code evolution. But it applies that to prompts and instructions inside a DSPy-style compound-LM program, rather than to code or equations. GEPA's authors built and evaluated it directly against DSPy's own MIPROv2 optimizer and against TextGrad/Trace as baselines (Table 2), positioning it as the reflection-based alternative to both numeric RL, GRPO, and textual-gradient optimization, TextGrad, for tuning frozen-model pipelines. GEPA ships as the default optimizer inside the DSPy library itself (github.com/gepa-ai/gepa; integrated into dspy).
Verified: WebFetch on https://arxiv.org/html/2507.19457 (full HTML, read in two passes covering mechanism, Table 1/2, Appendix D.1 merge rule, Appendix E.3/E.4 cost); WebSearch for venue confirmation and critiques. · code: https://github.com/gepa-ai/gepa