ARC-AGI

NVIDIA Kaggle Grandmasters' winning approach combines Qwen3 4B with Tiny Recursive Models.

• Core insight: Move heavy reasoning offline into synthetic data pipeline
• Synthetic data: 103K puzzles + 3.2M augmented examples
• Training: All done on single H100; batch size 2 works on 24GB GPUs
• Inference: Qwen-4B alone achieved 24% — ensemble added marginal gains
• Cost: Only $0.20 per task (vs $10K+ for o3)

Samsung researcher Alexia Jolicoeur-Martineau shows that 7M parameters can beat 671B parameter models.

• Architecture: Single transformer block that iterates, not stacks
• Mechanism: Recursively improves predicted answer + latent state
• Results: 45% ARC-1, 8% ARC-2 — beats DeepSeek-R1 (671B), o3-mini, Gemini 2.5 Pro
• Also improved: Sudoku-Extreme 87.4%, Maze-Hard 85.3%

Self-improving evolutionary program synthesis by Julien Pourcel et al. (ICML 2025).

• Method: LLM generates programs → test → refine promising ones → fine-tune LLM on attempts
• Key insight: Failed programs are correct programs for different tasks — use them as training data
• Results: 52% ARC-1 test set — SOTA for open-source LLMs without hand-crafted DSLs
• Dataset released: 5 million ARC solutions

Isaac Liao (CMU) proves lossless compression alone can produce intelligent behavior.

• Approach: Minimum Description Length (MDL) — find shortest program that outputs puzzle + solution
• No pretraining: Models randomly initialized, trained only at inference time
• No dataset: One model trains on one task, outputs one answer
• Size: Only 76K parameters, ~20 min per puzzle on RTX 4070
• Results: 20-34% ARC-1, 4% ARC-2

Major evolution from their 2024 autoregressive approach to masked diffusion.

• Architecture: LLaDA-8B masked diffusion LLM with 2D positional encoding
• Method: Soft-masking turns grid positions into partially noisy states, iteratively refined
• Key change: Autoregressive failed on tasks requiring global structure understanding
• Results: 16.5% on ARC-AGI-2

First to break 50% on ARC-AGI-2 through refinement loops on Gemini 3 Pro.

• Core method: Generate solution → receive feedback → analyze → refine (iterate)
• Self-auditing: System monitors progress, decides when to stop
• Cost: $30/task (vs Gemini 3 Deep Think at $77/task for 45%)
• Model-agnostic: Works with OpenAI, Anthropic, xAI models too

Play the preview games and see how you compare to current AI systems.

• Qwen3 4B + Tiny Recursive Models ensemble
• Synthetic data generation pipeline (SDG)
• 103K synthetic puzzles + 3.2M augmented
• ~24% on ARC-AGI-2 under contest constraints

• Mistral-NeMo-Minitron-8B base model
• Test-time training with augmented data
• Novel augmentations + stability criterion
• 53.5% on private eval

Smaller models benefit disproportionately more from fine-tuning than larger ones.

• Rank (r): Start with 8-16. Lower = fewer params, higher = more capacity.
• Target modules: Usually attention layers (q_proj, v_proj, k_proj, o_proj)
• Alpha: Scaling factor. Common: alpha = 2× rank
• Result: 19MB adapter vs 11GB full model. Same performance.

The original method that powered ChatGPT. Train a reward model from human preferences, then use PPO to optimize the policy.

• Components: Policy model, Reference model, Reward model, Value model (4 LLMs total)
• Pros: Proven at scale, handles complex preferences, can optimize arbitrary rewards
• Cons: Expensive (4 models), unstable training, requires extensive tuning
• Use when: You have compute budget and need maximum control over the reward signal

Simplifies RLHF by treating alignment as a classification problem. Used to train Llama 3, Zephyr, NeuralChat.

• Components: Policy model, Reference model (2 LLMs — no reward/value models)
• Input: Preference pairs (prompt, chosen response, rejected response)
• Pros: Simple, stable, efficient, no reward model needed
• Cons: Offline only (can't explore), requires paired preference data
• Use when: You have preference data and want simple, reliable alignment

The algorithm behind DeepSeek-R1's reasoning capabilities. Cuts PPO compute in half by eliminating the value network.

How GRPO Works

1. Group Sampling: For each prompt, sample G outputs (typically 16)
2. Reward Assignment: Score each output (accuracy + format rewards)
3. Baseline: Use group mean as baseline: A = r - mean(r)
4. Normalize: Divide by group std for stable gradients
5. Policy Update: PPO-style clipped objective + KL penalty

Key Differences from PPO

DeepSeek-R1 Training Config

• Learning rate: 3e-6
• KL coefficient: 0.001
• Clip ratio ε: 10
• Temperature: 1.0
• Samples per prompt: 16
• Max length: 32,768 tokens
• Batch: 32 prompts × 16 samples = 512

GRPO++ — fixes key issues in GRPO. 50 points on AIME 2024 with Qwen2.5-32B, outperforming DeepSeek-R1-Zero.

DAPO's Four Key Improvements

DAPO Results

• AIME 2024: 50% (vs DeepSeek-R1-Zero's 47%)
• Training efficiency: 50% fewer steps than R1-Zero
• Model: Qwen2.5-32B base → reasoning model

Returns to value-based methods with better credit assignment. 60.4 on AIME 2024 — current SOTA.

Why Value-Based?

GRPO/DAPO use final reward only. VAPO learns per-step values for precise credit assignment — crucial for long reasoning chains where early errors compound.

VAPO's Seven Techniques

• Refined value learning with better targets
• Exploration-exploitation balance
• Smoother training dynamics
• Better length scaling
• Faster score growth

VAPO Results

Fixes mathematical biases in GRPO's advantage estimation.

Three Biases Fixed

• Baseline bias: Using biased baseline without correction
• Length normalization: Removes length bias causing longer wrong answers
• Std normalization: Removes division by group std

Result

Prevents models from generating progressively longer incorrect responses. Improves token efficiency.

Simplified alternative to GRPO. More stable training, used by ProRL V2, ScaleRL, Magistral.

Key Difference

• GRPO: Local normalization within group
• REINFORCE++: Global normalization across batch
• Result: GRPO can overfit immediately; REINFORCE++ learns more stably

Single-Sample Efficiency

REINFORCE++ with k=1 achieves top-tier scores while being more token-efficient than group-sampling GRPO.

Bootstraps reasoning ability by learning from self-generated rationales. The model teaches itself to reason.

• Loop: Generate rationales → Keep correct ones → Fine-tune → Repeat
• Rationalization: For wrong answers, generate rationale given the correct answer (reason backward)
• Results: Comparable to 30× larger models on CommonsenseQA
• Limitation: Requires base model with some reasoning ability (GPT-2 too small)

Extends STaR to teach LLMs to "think before speaking" with internal rationales at every token.

• Method: Generate inner rationales in parallel before responding
• Zero-shot gains: GSM8K 5.9%→10.9%, CommonsenseQA 36.3%→47.2%
• Key insight: Reasoning is implicit in almost all text — learn to infer unstated rationales

LLM plays against previous versions of itself in a GAN-like framework.

• Mechanism: Current model learns to distinguish its own outputs from human data
• Roles: Opponent (generates synthetic) + Main Player (discriminates)
• Results: +5% on HuggingFace benchmarks over 3 iterations
• Advantage: No preference data or separate reward model needed

Model acts as both actor AND judge — provides its own rewards via LLM-as-a-Judge prompting.

• Why: Human feedback bottlenecks superhuman performance
• Method: Generate → Self-judge → Iterative DPO
• Extension: Meta-Rewarding — model judges its own judgments
• Results: Llama-3-8B win rate 22.9%→39.4% on AlpacaEval 2

Self-improving evolutionary program synthesis. 52% on ARC-AGI-1 — SOTA for open-source without DSLs.

• Loop: LLM generates programs → Test → Refine best ones → Fine-tune LLM on attempts
• Key insight: Failed programs are correct programs for different tasks — use as training data
• Hindsight: Relabel failed attempts as solutions for tasks they accidentally solve
• Released: 5 million ARC solutions dataset

Use deterministic verifiers (code execution, math checking) as ground-truth rewards instead of learned reward models.

• Domains: Code (unit tests), Math (exact match), Formal proofs
• Advantages: No reward hacking, objective, scales with compute
• Finding: RLVR can extend reasoning boundaries for both math and code
• Caveat: May narrow exploration — models favor known high-reward paths

Expectation-Maximization framework for self-training with binary rewards.

• E-step: Generate samples, filter with binary reward
• M-step: Fine-tune base model on filtered samples
• Key: Always fine-tune base model (not previous iteration) to prevent drift
• Results: Outperforms human-data fine-tuning on MATH and APPS

Combines process reward guidance with tree search for higher-quality reasoning traces.

• Method: MCTS* search + per-step value training
• Advantage: Higher accuracy than Best-of-N, Tree-of-Thought at same compute budget
• Outperforms: ReST-EM and Self-Rewarding LM in iterative improvement

Train harmless AI using self-improvement with a "constitution" of principles. No human labels for harmful outputs.

• SL Phase: Generate → Self-critique → Revise → Fine-tune on revised
• RL Phase: AI evaluates responses → Train preference model → RLAIF
• Constitution: Natural language principles (e.g., "be helpful, honest, harmless")
• Result: More harmless with minimal helpfulness impact, increased transparency

Modern transformer with carefully selected components:

• Rotary Embeddings (RoPE) — no learned position embeddings
• QK Normalization — stabilizes attention scores
• Group-Query Attention (GQA) — memory-efficient attention
• ReLU² activation — relu(x)² in MLP for sparse, efficient computation
• RMSNorm — no learnable params, faster than LayerNorm
• No bias — removed from all linear layers
• Untied embeddings — separate token embedding and lm_head weights
• Flash Attention 3 — H100-optimized with sliding window support

Combined optimizer: Muon for matrix params, AdamW for embeddings/scalars.

• Muon (MomentUm Orthogonalized by Newton-schulz)
  • Standard SGD-momentum + orthogonalization post-processing
  • Uses Polar Express iteration for fast orthogonalization in bfloat16
  • NorMuon variance reduction — per-neuron adaptive learning rate
  • Cautious weight decay — only decays weights aligned with gradient
• Distributed version (ZeRO-2 style)
  • Reduce-scatter gradients → compute locally → all-gather params
  • 3-phase async to overlap communication with compute
  • Sharded optimizer state across ranks