DSPy and TextGrad optimize the program wrapping a frozen LLM, compiling prompts and demonstrations instead of touching model weights.
lane: program loop · DSPy — arXiv:2310.03714 (submitted 2023-10-05), Khattab, Singhvi, Maheshwari, Zhang, Santhanam, Vardhamanan, Haq, Sharma, Joshi, Moazam, Miller, Zaharia, Potts (Stanford, UC Berkeley, CMU, IIT Bombay, Microsoft, Amazon Alexa AI); TextGrad — arXiv:2406.07496 (submitted 2024-06-11), Yuksekgonul, Bianchi, Boen, Liu, Huang, Guestrin, Zou (Stanford, Chan Zuckerberg Biohub)
Gives. Typed modules and a compiler that tunes prompts and demonstrations; TextGrad adds a critique that flows backward like a gradient.
Needs. A metric and a dev set of more than a few dozen examples.
Breaks. Overfits small dev sets; gains shrink as the base model gets stronger.
Both refuse to touch model weights and instead optimize the program wrapping the model. DSPy compiles declarative signatures and modules into concrete prompts plus bootstrapped demonstrations via a teleprompter. TextGrad builds an explicit computation graph of LLM and tool calls, then backpropagates natural-language critique through it the way autograd backpropagates numeric gradients. The piece orthogonal to GEPA, which also evolves prompts, is the abstraction layer. DSPy exposes typed modules and a declarative compiler general enough to optimize multi-stage pipelines with no hand-written prompt strings in source. TextGrad exposes a PyTorch-style .backward(), so arbitrary differentiable-in-principle systems, such as code, molecules, or treatment plans, get "trained" by text feedback using tensor-autograd's call pattern.
DSPy: a program is built from Signatures, input/output fields such as question -> answer, composed into Modules: Predict, ChainOfThought, ReAct, MultiChainComparison. A Teleprompter, the compiler, takes the program, a training set with labels optional, and a metric, and returns a compiled program. BootstrapFewShot, the simplest teleprompter, runs a teacher or zero-shot program on training inputs at high temperature, keeps only multi-stage traces that pass the metric, and uses survivors as per-module few-shot demonstrations (arXiv:2310.03714 Sec. 4). MIPROv2 (arXiv:2406.11695, EMNLP 2024, Opsahl-Ong et al.) factorizes optimization into instructions and demonstrations separately. It proposes task- and data-grounded instruction candidates, fits a surrogate objective model from stochastic minibatch evaluations, and meta-optimizes the proposal-generating LM over time. This comes from the abstract; the full table was not read. UNVERIFIED beyond "5 of 7 multi-stage programs with Llama-3-8B, up to 13% accuracy."
TextGrad: a computation graph whose nodes are Variables, prompts, code, molecule strings, or treatment plans, and whose edges are LLM calls, simulators, or solvers. The graph computes a loss at the output; loss.backward() walks the graph in reverse, and at each LLM-call edge the "gradient operator" builds a prompt from the variable's context plus downstream feedback, then asks an LLM for a natural-language critique of how the variable should change (Eq. 6, Sec. 2). TextGrad aggregates gradients from every successor (Eq. 11) before Textual Gradient Descent (TGD) prompts an LLM to rewrite the variable, incorporating the criticism (Eqs. 7-9, 12). Extensions include minibatch aggregation via tg.sum, natural-language constraints, and momentum: TextGrad shows the optimizer its own prior iterations (Sec. 2, Appendix B).
In both, nothing numeric backpropagates through the base LLM; weights stay frozen. DSPy's "gradient" is categorical: a demonstration passes the metric or doesn't (BootstrapFewShot), or a surrogate score ranks instruction candidates (MIPRO). Either way, it is discrete search over text, scored by a task metric. TextGrad's gradient is a string: a paragraph of LLM-written critique standing in for ∂loss/∂variable. TGD's "step" is another LLM call that reads that critique and emits new text. The one continuous, numeric quantity in either stack is the task metric or loss used to accept or compare candidates. Everything upstream of that score, up to the prompt edit itself, is natural language, not a vector.
Both target small-to-medium labeled, metric-checkable datasets where compiling a pipeline is cheap relative to finetuning. DSPy's GSM8K experiments use 200 training and 300 dev examples (Sec. 7), compiling in "minutes (or tens of minutes)," with the HotPotQA ensemble test run on only 50% of test "due to cost" (Sec. 5). TextGrad needs at most one extra LLM call per graph edge per iteration (Sec. 2). Cost scales with graph size times iterations, not model size, so it is cheap against any frozen API model, including closed ones with no gradient access. The trade-off is sensitivity to training-set size. Optimizers reading failure traces from a handful of examples, such as BootstrapFewShot, MIPRO, and by analogy GEPA, can latch onto patterns specific to that small set rather than the task. DSPy itself flags this risk about its small dev sets (Sec. 7), and later work quantifies it directly, below.
DSPy, Table 1 (GSM8K, GPT-3.5 dev/test): uncompiled 24.0/25.2% to CoT+bootstrap+ensemble 88.3%/81.6%; Llama2-13b-chat 7.0/9.4% to reflection+bootstrap+ensemble 49.0/46.9%. Table 2 (HotPotQA, GPT-3.5, answer EM): few-shot 34.3, multihop+bootstrap 48.7, +ensemble 54.7. Compiled T5-Large (770M) reached 39.3% answer EM / 46.0% passage accuracy on dev with 200 labeled + 800 unlabeled inputs (Sec. 7). TextGrad, Table 1: LeetCode-Hard completion 0.36±0.018 vs. zero-shot 0.26, Reflexion 0.31±0.012. Table 2: GPQA (gpt-4o) 55.0% vs. CoT 51.0%, prior best 53.6%; MMLU ML 88.4% vs. CoT 85.7%; MMLU College Physics 95.1% vs. CoT 91.2%. Table 3: Object Counting (gpt-3.5) 91.9% vs. CoT 77.8%, DSPy 84.9%.
1. GEPA-style reflective optimizers overfit small training sets, quantified: MAGE (arXiv:2607.11944, Singh, 2026, Sec. 5.2/Table 1) shows GEPA on GSM8K-Hard with 30 training examples scoring 34.0%±7.0% on held-out test, versus 62.4%±2.7% for the unoptimized seed prompt, a 28.4-point regression. The paper attributes this to "genuine overfitting to Ntrain=30" and calls it "a structural risk of any failure-grounded optimizer with small training sets" rather than an implementation bug. 2. MIPROv2 can also regress, driven by the scaffold, not the optimizer: the same paper reports MIPROv2+CoT dropping from 71.1% to 68.9%, a 2.2-point drop. The paper attributes this to the CoT scaffold interacting badly with optimization, rather than to MIPROv2 itself. 3. Mean-only reporting hides instability: MAGE concludes multi-component prompt optimizers are "coupled stochastic systems" whose variance must be reported alongside mean score; in low-data regimes a well-designed fixed prompt beat every reflective optimizer tried. 4. DSPy's own stated scope limit: the original paper optimizes only demonstrations, leaving instructions and field descriptions untouched "in this iteration" (Sec. 4). MIPRO was later built to close that gap. 5. TextGrad's own admitted gaps: molecule and radiotherapy results are in-silico only; the paper calls experimental and clinical validation "outside the scope" (Sec. 5). Natural-language constraints get less reliable as more are stacked (Sec. 2), and stability analysis is deferred to future work. 6. Practitioner reports on data-hungriness (WebSearch, UNVERIFIED beyond snippets): a 2025 study on prompting without gold labels found that DSPy showed limited benefit with very few labeled examples, and recommended MIPRO only at roughly 300 or more examples. This is consistent with MAGE's small-set fragility finding.
DSPy's BootstrapFewShot is the direct ancestor MIPROv2 replaces by also optimizing instructions via a learned surrogate model. GEPA, covered separately in gepa.md, pits itself directly against MIPROv2 as the baseline it claims to beat, using trace-level reflective feedback instead of demonstration bootstrapping. TextGrad's "PyTorch for text" framing borrows the autograd computation-graph abstraction wholesale, and GEPA's Table 2 itself uses TextGrad as a comparison baseline. Both sit upstream of the 2025-2026 "prompts as code" literature, for example arXiv:2507.03620, which treats prompt optimization as a software-engineering discipline rather than a one-off trick.
Verified: WebFetch on https://arxiv.org/html/2310.03714 (full HTML, Sec. 1-7, first 100k chars) and https://arxiv.org/html/2406.07496 (full HTML, Sec. 1-5, first 100k chars); WebFetch on https://arxiv.org/abs/2406.11695 (MIPRO abstract page) and https://arxiv.org/html/2607.11944 (MAGE, full text) for optimizer-instability data; WebSearch for 2025-2026 critiques. · code: https://github.com/stanfordnlp/dspy ; https://github.com/zou-group/textgrad