The LLM writes its own finetuning data and training hyperparameters as a text self-edit, then an RL loop rewards self-edits that raise downstream accuracy after a gradient update.
lane: weight loop · arXiv 2506.10943 — NeurIPS 2025 — Zweiger, Pari, Guo, Akyürek, Kim, Agrawal, MIT
Gives. The model writes its own training data and hyperparameters as text; the candidate is a weight-update recipe, not code.
Needs. An RL outer loop, about 6 hours per round on 2x2 H100, LoRA adapters.
Breaks. Forgets earlier passages across a stream of edits; GRPO and PPO were unstable.
SEAL is a weight-loop approach where the thing being searched over is the recipe for updating the weights, not the weights themselves: a natural-language self-edit that specifies synthetic training data and, for ARC, the SFT hyperparameters to apply it with (App. A.2). The LLM is both the thing being updated and the author of its own update instructions. The only training signal is whether the resulting weight update helps on a downstream task (Alg. 1, §3.1).
In the outer loop, the policy model samples a context and task, then generates a self-edit. SEAL applies the self-edit via an inner SFT step and evaluates the updated model on the task. It assigns a binary reward, 1 if the model improved and 0 if not (Eq. 2). The outer loop then updates the policy toward self-edits that earned reward 1. This procedure is ReST-EM, rejection sampling plus SFT, also called filtered behavior cloning (§3.1). The paper reports GRPO and PPO were tried and found unstable. For knowledge incorporation, SEAL runs 2 outer rounds with 50 contexts per batch, 5 self-edit samples per context at temperature 1, and 3 seeds each; it keeps the best edit per context and runs SFT on those 50 (App. B.2). For few-shot ARC, SEAL samples 15 self-edits per training task; only edits whose resulting model solves the held-out test pair get reinforced (§4.1).
Self-edit format differs by task. For knowledge incorporation, it is free text: the model is prompted to list "implications derived directly or indirectly from the content" of a passage (App. B.3). Alternates tested are rewrites and Q&A pairs (App. B.11, Table 10). For ARC, it is a JSON spec choosing augmentation flags, such as use_basic_augmentations and use_size_augmentations, a training-target mode, train_using_all_tokens versus train_using_output_tokens, and a learning rate and num_train_epochs (App. A.2).
The gradient runs only inside the inner SFT step, and only through LoRA adapters, never the base weights. For knowledge incorporation, LoRA uses rank 64, alpha 128, on all MLP and attention projection layers (App. B.2). In the ARC inner loop, LoRA uses rank 128, alpha 16, on q_proj/v_proj/gate_proj/down_proj/up_proj; the outer RL-trained SFT policy uses rank 16, alpha 16, 8 epochs, and a learning rate of 5e-5 (App. A.2). Full fine-tuning appears only in a separate continued-pretraining ablation (Table 2 caption, "full-FT"). The outer loop that picks which self-edit to write is never backpropagated through the SFT step. It only sees the scalar binary reward, so the self-edit content and hyperparameters are discrete, policy-gradient-free, ReST-EM-selected text, not continuously optimized.
SQuAD knowledge incorporation, single passage, Qwer2.5-7B (Table 2): base model 32.7, train-on-passage 33.5, passage+GPT-4.1-synthetic 46.3, passage+base-model-synthetic 39.7, SEAL 47.0. ReST-EM rounds (Table 10, implications prompt): round 0, no RL, 39.7 → round 1, 43.7 → round 2, 47.0. The paper states "two iterations suffice for SEAL to overtake GPT-4.1 data" (§4.2). Few-shot ARC (Table 1): in-context learning 0%, test-time training plus untrained self-edit 20%, SEAL 72.5%, oracle TTT ceiling 100%, across 8 tasks × 5 self-edits.
Catastrophic forgetting: §5 and Figure 6 show accuracy on earlier-seen passages "gradually declines as the number of edits increases" across a sequential self-edit stream. The paper gives no exact percentage drop in the HTML text — UNVERIFIED exact magnitude. It proposes replay, constrained updates, or representational superposition as future fixes, not demonstrated ones. GRPO and PPO were tried and rejected as unstable, forcing the weaker ReST-EM (§3.1). Reward is expensive: each self-edit evaluation costs roughly 30 to 45 seconds end to end (§5). A full knowledge-incorporation RL round takes roughly 6 hours on 2×2 H100s, versus roughly 5 minutes for a cheaper proxy-reward variant that underperforms it, 45.6 versus 47.0 (Table 9, App. B.5). MIT's own announcement states the forgetting problem is unsolved, left to future work.
SEAL builds on test-time training, TTT, for ARC and on DeepMind's ReST-EM filtered-RL recipe, reusing it as the outer-loop optimizer because standard policy gradients were unstable on this reward (§3.1). The authors position SEAL as a step toward models that "self-adapt" rather than be adapted by an external pipeline. Later discussion treats it as the canonical example of the weight-loop lane's cost problem: large RL-outer-loop compute per self-edit, with no scaling story past small models and narrow tasks.
Verified: arxiv.org/html/2506.10943 (full HTML text, body + appendices A–B) · code: github.com/Continual-Intelligence/SEAL