Instead of searching program text, inference-time weight updates take a real gradient step, triggered by a continuous surprise or loss signal from the input itself.
lane: weight loop · Titans — arXiv:2501.00663 — NeurIPS 2025 — Behrouz, Zhong, Mirrokni (Google Research); TTT for abstract reasoning — arXiv:2411.07279v2 — ICML 2025 workshop / arXiv preprint, Akyürek, Damani, Zweiger, Qiu, Guo, Pari, Kim, Andreas (MIT); TTT-for-LLM follow-up — arXiv:2505.20633 ("Test-Time Learning for Large Language Models", TLM) — ICML 2025 — Hu, Zhang, Chen, Wen, Shuai, Luo, Xiao, Li, Tan
Gives. A real gradient on a small weight set per input or per task, triggered by surprise or by the task's own examples.
Needs. A differentiable model and compute per query; a benchmark that allows it.
Breaks. ARC 53% public fell to 47.5% semi-private; no audit trail; some tracks ban it.
Every other approach in this hub keeps the base model frozen and searches over discrete program or prompt text, judged by an external evaluator. Test-time weight updates invert that: the thing that changes per-instance is a numeric weight tensor, updated by a real gradient step from a self-supervised loss derived from the input itself, Titans' "surprise," or TTT's reconstructed-task loss. No LLM reflects on a trace, and no Pareto archive exists. The update signal is intrinsic and continuous, a scalar loss on held-in data, not an external, possibly LLM-generated verdict. What updates is architecture-internal memory weights in Titans, or a tiny per-task adapter in TTT, not the program calling the model.
Titans: a deep neural long-term-memory module, an MLP with at least one layer, sits alongside a short-term attention branch and fixed "persistent memory" parameters. For input x_t, the module computes k_t = x_t W_K, v_t = x_t W_V, and an associative-memory loss ℓ(M_{t-1}; x_t) = ‖M_{t-1}(k_t) − v_t‖² (Eq. 11-12). "Surprise" is ∇ℓ with respect to the memory weights. A momentum-and-forgetting update combines decayed past surprise (η_t) and scaled current surprise (θ_t) with forgetting gate α_t: M_t = (1−α_t)M_{t-1} + S_t, S_t = η_t S_{t-1} − θ_t∇ℓ (Eq. 8, 10, 13-14). The paper states this is literally gradient descent with momentum and weight decay, run online per token, chunk-parallelized (Eq. 16-18). Retrieval is a forward pass with no update (Eq. 15). Three variants, Memory as Context, Gate, or Layer, combine this module with attention differently.
TTT for ARC (Akyürek et al.): starting from a model fine-tuned on synthetic ARC-like tasks, the method expands each task's demonstration pairs at test time via leave-one-out construction, shuffled order, and rule-based invertible geometric transforms, such as rotations, flips, transpose, translation, upscaling, and chains, plus color permutations, capped at 250 examples per task. A fresh LoRA adapter, rank 128, alpha 16, on q/v projections and MLP weights, plus output projection for the 8B model, trains via AdamW for 2 epochs per task on standard LM loss over these self-generated examples. The method discards the adapter after that task's prediction — a new adapter per test task (Sec. 3-4, Table 6).
TTT for LLMs generally (TLM, arXiv:2505.20633): TLM frames test-time learning as minimizing input perplexity on unlabeled test data, updating only LoRA weights to limit forgetting. A "Sample Efficient Learning Strategy" actively selects high-perplexity samples to update on preferentially. The exact LoRA rank or target modules are not confirmed; this session read only the abstract page — UNVERIFIED.
Titans: the gradient runs continuously through the long-term memory MLP's weights at every token, online, forever. There is no discrete search step anywhere; "surprise" is a true gradient norm, not a text score. TTT-for-ARC: the gradient is real and runs through LoRA adapter weights only, rank 128, alpha 16, fit fresh per task by AdamW. Non-LoRA weights stay frozen at test time, though a full fine-tune happens once, offline, beforehand. What is discrete and non-differentiable in the ARC pipeline is task selection and the hand-designed augmentation and transformation rules used to synthesize training examples. Across both methods, nothing crosses a text or symbolic boundary the way it does in GEPA, DSPy, or TextGrad. Feedback is a numeric loss the whole way through.
Titans targets long-context sequence modeling where attention's quadratic cost is prohibitive. The abstract claims scaling "to larger than 2M context window" with higher needle-in-haystack accuracy, but the needle-in-haystack table actually read, Table 2, S-NIAH, only goes to 16K tokens. The 2M claim is stated in the Abstract and Conclusion but not demonstrated in the sections fetched — UNVERIFIED pending the unread back half. TTT-for-ARC targets the opposite regime: very few examples per task, paying steep compute for per-task specialization. That cost is about 12 hours on one A100 for 100 sampled validation tasks (Appendix C.2), a real limitation beyond benchmark-scale evaluation. TLM targets distribution shift at inference, adapting to a target domain using only unlabeled test data and trading inference-time compute for domain generalization.
Titans (Table 1, 760M params, 30B tokens): average accuracy across language-modeling and commonsense tasks is Titans (MAC) 52.51, Titans (MAG) 52.50, versus Transformer++ 48.69 and Gated DeltaNet 49.69. Needle-in-a-haystack at 16K (Table 2): Titans (MAC) 98.4 versus DeltaNet 71.4 and Mamba2 5.4 on S-NIAH-PK; on S-NIAH-W, Titans (MAC) 95.2 versus DeltaNet, TTT, and Mamba2, all 0.0. TTT-for-ARC (Table 1, Sec. 4.6): BARC's fine-tuned model combined with TTT reaches 53.0% on ARC public validation, 61.9% ensembled with program synthesis, versus 60.2% average human. The authors' own 8B model goes from 18.3% to 47.1% with TTT — the source of the abstract's "up to 6x" framing. BBH 10-shot (8B, Fig. 1, 8): 57.8% with TTT versus 50.5% in-context and 40.9% zero-shot. TLM claims "at least 20%" improvement on its own AdaptEval benchmark, abstract only, UNVERIFIED exact figures.
1. ARC semi-private evaluation drop: the headline 53.0% and 61.9% figures are on the public validation set. On the semi-private split, accuracy fell to 47.5%, attributed to distribution shift (Sec. 4 limitations) — a sourced gap between the headline and the harder held-out condition. 2. ARC Prize rules reportedly disallow TTT in some tracks over leakage: a practitioner summary, from WebSearch, states TTT "is disallowed in ARC Prize settings due to leakage risks." That is a secondary source, not confirmed against ARC Prize's own rules — UNVERIFIED, but consistent with item 3 below. 3. ARC data contamination concern, independently argued: ARC-AGI-3 (arXiv:2603.24621, WebSearch) argues frontier models likely absorbed ARC-style data during pretraining, and recommends future private splits stay out-of-distribution from public demonstrations. This is a structural critique of any ARC-benchmarked TTT number — UNVERIFIED, summary only. 4. Compute cost is a real ceiling: roughly 12 GPU-hours for 100 ARC tasks on one A100 (Appendix C.2) means per-task LoRA fitting never amortizes. The authors' own QLoRA variant trades further accuracy for memory, 29 to 26 tasks solved in a 1B ablation (same appendix). 5. Titans' headline scaling claim outruns its own shown evidence: the abstract and conclusion claim effective scaling past a 2M-token context, but needle-in-a-haystack numbers in the sections read stop at 16K (Table 2). The paper's own limitations note that larger-model results were still being finalized "in the next version" — forward-looking, not verified in this fetch. 6. Forgetting magnitude varies wildly by backbone, undercutting any single "small effect" claim: a 2025 comparative study (arXiv:2504.01241, WebSearch, UNVERIFIED) reports forgetting scores of 0.02-0.1 for Phi-3.5-mini and Phi-2, versus 0.59-0.935 for Llama-3.1-8B and Qwen2.5-14B, under continual fine-tuning. Whether TTT forgets has no single answer across backbones. Note: the companion paper this task named for forgetting data, arXiv:2410.15665, turned out on inspection to be about long-term memory for agentic self-evolution, not a forgetting study — a mis-citation in the scouting notes, flagged rather than silently substituted.
Titans sits in the linear-recurrent-memory lineage, with Mamba2, DeltaNet, and Gated DeltaNet as its direct baselines in Table 1, and is compared against prior TTT-style sequence-modeling architectures. Table 2's "TTT" row names a different, older method than the Akyürek ARC approach — the same acronym covers two things in this literature, worth flagging. The ARC TTT line descends from the original "test-time training" idea, Sun et al. 2020, self-supervised adaptation at inference, applied to few-shot symbolic reasoning. The BBH extension, same authors, companion repo, generalizes the recipe to arbitrary few-shot reasoning. TLM (2505.20633) generalizes further to open-domain LLM deployment, framing test-time adaptation as perplexity minimization rather than task-specific augmentation. Google's later "Nested Learning" paper (arXiv:2512.24695, title only, WebSearch) appears to reframe Titans-style memory inside a broader multi-level-optimization story — unconfirmed, UNVERIFIED pending full-text read.
Verified: WebFetch on https://arxiv.org/html/2501.00663 (full HTML, first 100k chars, Sec. 1-4/Table 1-4) and https://arxiv.org/html/2411.07279v2 (full HTML, first 100k chars, Sec. 1-4/Appendix C,E,F); WebFetch on https://arxiv.org/abs/2505.20633 (abstract page only — full text not opened, mechanism details partly UNVERIFIED); WebSearch for NeurIPS 2025 venue confirmation, ARC leakage debate, and forgetting-magnitude data. · code: github.com/ekinakyurek/marc (ARC TTT) ; github.com/adamzweiger/Fewshot-TTT (BBH TTT) ; no official Titans release found