ARC-AGI pits LLM-guided discrete program search against test-time gradient updates to weights, and by 2025 the top ARC-AGI-2 scores came from combining both, not either alone.
lane: head to head · ARC Prize 2024 Technical Report — arXiv:2412.04604, Chollet/Knoop/Kamradt/Landers; ARC Prize 2025 Technical Report — arXiv:2601.10904; ARC-AGI-2 — arXiv:2505.11831, Chollet et al.; Akyürek et al., "The Surprising Effectiveness of Test-Time Training for Abstract Reasoning" — arXiv:2411.07279; Greenblatt, "Getting 50% (SoTA) on ARC-AGI with GPT-4o" — Redwood Research blog, 2024
Gives. The one measured combination: discrete program search for exactness, a test-time gradient for adaptation; 2025's top ARC-AGI-2 scores used both.
Needs. Compute per task on both sides.
Breaks. o3 reached 87.5% at 172x compute and no one can read what it learned.
Program synthesis brings exactness from few examples. Once the search finds a correct DSL program, it generalizes perfectly to every grid the transformation covers, at the cost of combinatorial search that explodes with program size. Test-time training brings adaptation without search. A small gradient update on the task's own examples reshapes the model's weights toward the task's specific transformation, at the cost of needing a differentiable model. It also loses the auditability of an explicit program. Neither alone solved ARC-AGI-2. The single highest verified 2025 score, Poetiq's refinement loop on Gemini 3 Pro at 54%, combines program-synthesis-style iterative refinement with an LLM's learned priors (arcprize.org/blog/arc-prize-2025-results-analysis). The ARC Prize's own 2024 essay argues the two must compose: deep learning as perception and intuition, System 1, and program synthesis as exact step-by-step search, System 2 (arcprize.org/blog/beat-arc-agi-deep-learning-and-program-synthesis).
Program-synthesis camp (Greenblatt, 2024, Redwood Research blog, full text verified): the pipeline shows GPT-4o each ARC task as both images and several text grid representations, inside a ~30k-token few-shot prompt with step-by-step reasoning. It generates roughly 8,000 candidate Python programs per task, about 5,000 initial completions plus about 3,000 revision samples spread across the 12 most promising implementations. The pipeline keeps programs only if they are correct on the given examples. A revision step shows GPT-4o the gap between a program's actual and expected output on the 12 best candidates and asks for fixes. The final answer is a majority vote over surviving programs. JUDGE is exact match against the training examples, a unit test. KEPT is the set of programs that pass, voted over for the final guess.
Test-time-training camp (Akyürek et al., arXiv:2411.07279): the method fine-tunes an 8B-parameter language model at inference time per task, using a loss derived from the task's own input data, per-instance adapter training. That follows initial fine-tuning on synthetic ARC-like tasks and leave-one-out augmented task generation. It applies self-consistency under invertible grid transformations, symmetries, to vote across augmented variants. There is no discrete program kept at the end — only an updated weight state specific to that task, discarded afterward.
o3 (OpenAI, Dec 2024, arcprize.org/blog/oai-o3-pub-breakthrough): OpenAI did not disclose o3's architecture, but ARC Prize treats it as a hybrid of program-synthesis-style internal search and chain-of-thought reasoning at massive sampling scale. That scale means 1,024 samples per task, "low-efficiency" or "high-compute, ~172x," versus 6 samples, "high-efficiency."
In the program-synthesis camp, nothing is continuous: GPT-4o's own weights are frozen. The only "fitting" is discrete program search plus majority voting, with the LLM's pretrained weights supplying the proposal distribution over code. In the test-time-training camp, the boundary flips. There is no discrete search at all, only a gradient step on the model's weights, computed from a loss on the task's own examples. The "program" is implicit in the updated weight tensor and is never materialized as readable code. The signal that crosses from task to model differs accordingly: a scalar pass/fail per candidate program (synthesis) versus a loss gradient computed directly from the input/output pairs (TTT). The 2025 winning combination, Poetiq on Gemini 3 Pro, crosses this boundary explicitly. It iterates a refinement loop that uses an LLM's learned judgment to prune and re-propose candidates, closer to the synthesis camp's discrete loop. But the model driving that loop may itself have had its own weights adapted.
Numbers and dates, chronologically:
Program synthesis's clearest win is cost-efficiency at a fixed accuracy band. NVARC's 24.03% at $0.20/task (2025) is far cheaper per point than o3's 87.5% at $4,560/task (2024) on the easier ARC-AGI-1. The two numbers are not on the same benchmark version, though, and are not directly comparable. Test-time training's clearest win is sample efficiency without search. Akyürek's 53% comes from an 8B model with no search over discrete programs at all, rising to 61.9% (human-average) only once ensembled with program synthesis. The paper's own headline result is a hybrid, not pure TTT. The clearest 2025 evidence for combination over purity is Poetiq's 54%-at-$30/task refinement loop. It beats every pure-synthesis result (NVARC, 24.03%) and every pure-model result (Opus 4.5 Thinking, 37.6%) reported on ARC-AGI-2 this season.
1. Program synthesis's named failure modes (Greenblatt): GPT-4o badly misreads grid colors above 12x12. It makes frequent off-by-one coding errors. Its performance degrades past ~32k-40k tokens of context. The lack of flexible prompt-prefix caching limits multi-round debugging. The method used over 1,000x more runtime compute than prior work, and it was ineligible for the ARC Prize leaderboard for using a closed-source model and too much compute. 2. Brute-force explosion, stated directly by ARC Prize: beating ARC-AGI-1 by pure brute force needs over 100 million programs per task. The essay's verdict: "brute force search, which demonstrates skill, is not a demonstration of intelligence." 3. o3's cost cliff and generalization gap: 6→1,024 samples (172x compute) bought ~12 points on the semi-private set (75.7%→87.5%) at ~175x dollar cost ($26→$4,560/task). The same page's prediction of under-30% on ARC-AGI-2 at high compute proved directionally correct, once NVARC and most LLM approaches scored well under 30% there in 2025. 4. TTT's narrower demonstrated scope: Akyürek's 53% is for an 8B model specifically. The stronger 61.9% requires ensembling with program synthesis. The authors frame pure TTT as necessary but insufficient for human-level accuracy. 5. ARC-AGI-2 reset the leaderboard for everyone: Chollet built it to resist the brute-force and memorization patterns that pushed ARC-AGI-1 past 50%. Per Chollet's contemporaneous public statement (via WebSearch, not reopened as primary), every base LLM and single-CoT reasoner scored 0-1% at launch. Even 2025's best Kaggle program-synthesis entry, NVARC, reached only 24.03% a year later. The same redesign knocked both camps back to near-zero. 6. This research found no 2025-2026 statement in Chollet's own voice declaring one camp the winner. The ARC Prize 2025 results-analysis page attributes its paradigm commentary to co-founder Mike Knoop, not Chollet.
ARC-AGI-1 (Chollet, "On the Measure of Intelligence," 2019) set the original challenge: human-level fluid intelligence tasks resistant to memorization. Program synthesis's entry includes the 2020 DSL-plus-search winners and Greenblatt's LLM-guided search in 2024. It is the direct descendant of classical inductive program synthesis, the DeepCoder and FlashFill lineage, applied to a visual domain. Test-time training's entry, Akyürek et al., 2024, descends from the test-time-adaptation line, fine-tuning on the task's own data at inference. It explicitly cites ARC's structurally novel tasks as the motivating failure case for plain in-context learning. ARC-AGI-2 (2025) and the ARC Prize 2025 season's winning combination, Poetiq's refinement loop, mark the point where the competition stopped rewarding either camp in isolation. It started rewarding architectures that interleave both. The 2025 Paper Prize runner-up, SOAR, is itself explicitly a self-improving program-synthesis system built to combine test-time training and test-time search. That description comes only from the WebSearch-level title and abstract; full text was not opened this session.
Verified: WebFetch on arcprize.org/blog/oai-o3-pub-breakthrough (o3 table), blog.redwoodresearch.org/p/getting-50-sota-on-arc-agi-with-gpt (full post), arxiv.org/abs/2411.07279 (abstract), arcprize.org/blog/beat-arc-agi-deep-learning-and-program-synthesis (full post), arcprize.org/blog/arc-prize-2025-results-analysis (full post); WebSearch cross-checks for ARC Prize 2024 technical report (arXiv:2412.04604) numbers and for a 2025-2026 Chollet verdict (none found stated in his own voice beyond the 2024-2025 posts opened). · code: none consolidated; individual team repos linked from arcprize.org (not independently opened this session)