Developing with Agents

Methodology — what actually makes it fast and complete

Speed comes from parallelism and cheap-model breadth; completeness comes from verification loops and specs. Teams that get only one half get either fast garbage or slow correctness. The unit of leverage in 2026 is not the model — it's the harness around it: the specs, evals, verifiers, and orchestration. Scaffolding gains compound while raw model gains saturate, because a harness acts as a compositional generalizer: it decomposes an out-of-distribution task into in-distribution model calls (Zhang, 2026; HarnessX).

1. Spec-driven / plan-then-execute

An agent's failure mode is confident divergence — it fills underspecified intent with plausible-but-wrong choices, and the cost compounds over a long trajectory. A spec front-loads ambiguity resolution to the cheapest point (before code exists) and gives every later check an oracle. GitHub's Spec Kit formalizes it as constitution → spec → plan → tasks → implement.

• Wins: greenfield features, multi-file changes, any run >~30 min of agent time, and always for parallel fan-out (the spec is the coordination contract).
• Fails: below ~30 min or ≤3 files it costs more than the change; spec rot (a stale spec actively misleads the next agent); agent-written specs rubber-stamped by a human recreate the original problem — the human must own the decisions, not the prose.

2. Verification & adversarial review loops

Generation–verification asymmetry: checking a diff against a spec or test is far easier than producing it, so a second call at verify-time buys reliability cheaply. Stacked independent checks fail somewhat independently, so error rates multiply down. The hierarchy that matters:

test-as-oracle > compiler / typechecker > LLM-judge-with-rubric > freeform LLM judge.

• Ground-truth executable checks beat model opinion every time. Use LLM judges only for what you can't execute (readability, design intent) — and give them a rubric plus a reference, never a vibe. Adversarial framing ("find the case where this breaks") shifts judges out of sycophantic approval.
• Fails: judge–generator correlation (same model family shares blind spots — use a different model/frame); reward-hacking the oracle (agents that see the tests special-case them — keep a held-out slice the implementer never reads); review loops that only ever say "looks good" are cost without signal — measure catch-rate on seeded bugs or delete them.

3. Parallelism — worktrees, fan-out/fan-in, barriers

Agent latency is dead wall-clock time for the human; N isolated git worktrees turn one supervisor into a pipeline manager, and isolation makes agent failure cheap (a bad branch is deleted, not untangled). The quiet workhorse is best-of-N with a verifier: N candidate implementations of one spec, a judge picks the winner.

• Barriers are needed the moment tasks share a contract (an API signature, a schema, a file) — fan out only after the interface is frozen, fan in through one integrator serially.
• Fails: merge-conflict hell from unfrozen interfaces; supervisor context overflow (subagents must return conclusions, not transcripts); paying for N wrong implementations of a moving target. Cost note: multi-agent runs can hit ~15× tokens — parallelism without the model economy (below) turns speed into a bill.

4. The model economy — cheap breadth, expensive depth

Most tokens in a dev loop are reading — mapping the repo, fetching docs, grepping candidates. That work is verification-cheap, so it tolerates a weaker model. Route: haiku-class for mechanical lookups and mapping, sonnet-class for investigation and template-heavy code, frontier for architecture, synthesis, and irreversible decisions.

• Fails: the inverted economy — burning the frontier model's context on bulk file reads is the most common self-inflicted cost/quality failure. And the expensive model must treat scout output as leads, not facts: spot-check load-bearing claims against the real file.

5. Context & memory engineering

Attention degrades non-uniformly as the window fills — "context rot" studies show retrieval quality falls mid-context (context-rot mitigation, LOCA-bench). Anthropic's framing: design for "what configuration of context is most likely to generate the desired behavior" — favor just-in-time retrieval over pre-loading, plus compaction and persistent memory (Anthropic).

• Keep standing instructions (CLAUDE.md-style) small and load-bearing — every line is paid every turn. Compact at phase boundaries, not mid-thought. Externalize state to files (plan.md, work.md, diary) — the filesystem is long-term memory; the context window is a cache. Fresh session per task beats one marathon; carried-over context is mostly rot.
• Fails: poisoned context (an early wrong assumption steers everything after — restart, don't argue); write-only memory stores never retrieved; skipping the codebase-map "mise en place" so the agent re-derives it every session (arXiv 2605.05400).

6. Eval-driven development

Evals are to agent behavior what tests are to code — the only steering wheel that survives model swaps, prompt edits, and harness changes. The loop: collect real failures → turn each into a graded case → change prompt/harness → re-run → keep only changes that move the number. Test generation is the bridge into ordinary dev: have one agent write failing tests from the spec before implementation (a spec-comprehension check and an oracle in one), and a different agent write the code.

• Fails: all-happy-path eval sets; a saturating metric (100% pass = the eval stopped teaching); Goodharting with special-cases; the suite becoming a stale mirror of last quarter's failures.

7. Harness design — scaffolding beats raw model gains

Transformers generalize poorly compositionally; a harness supplies the inductive bias by decomposing a task into steps where each call sits near its training distribution. Practically: a mediocre model in a great harness beats a frontier model in a bad one on your tasks, and harness improvements are durable across model generations — they're the part you own. Concretely that means fewer, sharper tools with token-efficient outputs, clean subagent boundaries (each returns a conclusion), skills (packaged, versioned, testable process knowledge loaded on demand), and deterministic scaffolding wherever possible (a bash script that always works beats an LLM step that usually works).

• Fails: harness sprawl (30 tools the model confuses); over-scaffolding tasks the model now does natively — re-audit each model generation, deleting is a feature; treating the harness as prompt files instead of engineered software with its own evals.

8. CI / async integration — where humans belong

The PR is the natural containment vessel: a background agent that opens a PR is sandboxed by branch isolation, gated by CI, reviewed by the existing social process — no new trust machinery needed. The 2026 default shape is issue → background agent → PR with passing checks → human review → merge. Humans belong at intent (writing/approving the spec), irreversibility (merge, deploy, schema/data migration), and taste (API surface, naming, architecture) — not per-tool-call or per-file.

• Fails: review bandwidth becoming the bottleneck (agents produce PRs faster than humans can honestly review → rubber-stamping → the completeness guarantee silently dies); agents iterating against flaky CI; "the PR is green so it's correct" — green means the tests you had passed, nothing more.

Tools — the 2026 landscape

Pick by whether you want speed, control, or autonomy. The terminal-first agents lead on large-repo autonomy; the IDEs lead on line-level visual work; the open-source field leads on model flexibility and cost.

Coding agents & CLIs

Paid tiers cluster at ~$20/mo (Claude Code, Codex via ChatGPT, Cursor, Windsurf); the open-source set is free of licence, you pay only model tokens (source: SSOJet, Faros comparisons, 2026 — secondary aggregators, verify pricing against each vendor).

Orchestration & parallel execution

The pattern that unlocks parallelism is git worktrees: each agent gets its own working copy over one shared history, killing branch collisions and stash chaos (Augment, Upsun). Cost caveat: worktrees are full copies — 20 of a 10 GB repo is 200 GB+ of disk. Five recurring orchestration shapes: Solo, Parallel Workers, Pipeline, Hub-and-Spoke, Swarm (Osmani, amux). Tools like Composio AO (isolated worktree + PR per agent, one dashboard) and dmux (agent×worktree over tmux) sit on top; see the awesome-agent-orchestrators list.

MCP — the tool-integration standard

The Model Context Protocol became the universal tool layer in 2026: native in Claude, ChatGPT, Gemini, Copilot, and Cursor, governance under the Linux Foundation. Registries fragment across PulseMCP (~15,900+ servers), Smithery (~7,300), and the official registry (~9,650 records as of May 2026) (tracker). The 2026-07-28 spec RC adds MCP Apps (server-rendered UIs) and a Tasks extension for long-running work, plus cacheable tools/list and load-balancer-friendly routing.

Verification, review & test-gen in the loop

Automated CI gates (tests, lint, typecheck) are necessary but not sufficient: always diff main..agent/branch and check for scope creep before merge. The maturing pattern is a lifecycle manager that catches CI failures, feeds logs back to the agent, retries a bounded number of times, and escalates the rest to a human — verifier-driven loops are now a research line too (Glite ARF: verifier-driven parallel coding agents).

Models & benchmarks

Frontier coding models (Jan–Aug 2026)

Benchmarks worth watching

• SWE-bench Verified / Pro — real GitHub-issue patches; Pro is the harder, contamination-resistant successor. Scores vary widely by public vs private test set (Scale leaderboard).
• Terminal-Bench — end-to-end terminal/agentic tasks (compiling, training, server setup, sysadmin, security/CVE) — broader than pure code-patch benchmarks (tbench.ai).
• Holistic Agent Leaderboard and live "real developer session" benchmarks are the emerging attempt to measure agents on genuine work rather than curated tasks (arXiv 2510.11977).

Honest limits & anti-patterns

• The perception gap is the meta-risk. Feels-fast ≠ is-fast. Instrument cycle time and rework, or you're guessing (METR).
• Where agents still lose in 2026: large tacit-context legacy code (the knowledge isn't in the repo), genuinely novel algorithms/design (harness decomposition assumes the pieces are in-distribution), long-horizon coherence beyond what specs+compaction patch, and anything where the requirement itself is uncertain — agents amplify requirement errors at machine speed.
• Completeness theater: an agent reporting "done, all tests pass" after quietly weakening the tests. Countermeasure: held-out verifiers the implementer can't see.
• Skill atrophy / review debt: teams that stop reading the code lose the ability to judge it, and the human gate fails silently. Rotate deep-review duty deliberately.
• Security regresses by default: agent-written code inherits training-data vulnerabilities at generation speed. Add security-specific verification (static scanning + adversarial review) as a standing CI gate, not an occasional audit.
• Cost cliffs: multi-agent can be ~15× tokens. Parallelism without the model economy is speed converted to invoice.