Parallel Subagent Research Swarm

What This Builds

This recipe reproduces the orchestrator-worker multi-agent architecture Anthropic used for Claude’s Research feature. A lead (orchestrator) agent analyzes a query, develops a strategy, saves its plan to memory, and spawns specialized subagents that search in parallel, each with its own context window. The lead synthesizes their findings and decides whether more research is needed; a final citation agent attaches sources to every claim.

The concrete use case is a breadth-first research assistant for questions like “find all board members of the IT companies in the S&P 500” — tasks that decompose into many independent sub-investigations. Subagents act as intelligent filters: they explore one aspect, compress the most important findings, and return them to the lead.

The Stack

• An agent runtime — the Claude Agent SDK or your own agent loop (LLM + tools in a loop). Use a stronger model as the lead and cheaper models for subagents (Anthropic found Opus-lead + Sonnet-subagents strongly outperformed a single agent on their internal eval).
• Anthropic Claude (e.g. via the Anthropic Startup Program) — extended thinking for the lead’s planning scratchpad and interleaved thinking for subagents evaluating tool results.
• Search / retrieval tools — web search plus an optional Qdrant vector index for private corpora; subagents call 3+ tools in parallel.
• Upstash Redis — durable memory/checkpoints so the lead’s plan survives context truncation and the system can resume mid-run instead of restarting (errors in stateful agents compound).

Step-by-Step Outline

1. Plan in the lead. The lead agent uses extended thinking to assess complexity, choose a subagent count, and write its plan to memory (so it isn’t lost if the context window exceeds the limit).
2. Teach the lead to delegate. Give each subagent an explicit objective, output format, tool/source guidance, and clear task boundaries to avoid duplicated work and gaps.
3. Scale effort to complexity. Embed scaling rules: simple fact-finding ~1 subagent with a few tool calls; comparisons ~2-4 subagents; complex research 10+ subagents with divided responsibilities.
4. Run subagents in parallel. Spin up 3-5 subagents concurrently, each issuing multiple tool calls in parallel; have them start broad then narrow. Persist large outputs to a store and pass back lightweight references to avoid the “game of telephone.”
5. Synthesize and loop. The lead combines findings and either finishes or spawns more subagents. Then a citation agent maps each claim to its source.
6. Evaluate. Start with ~20 representative queries and an LLM-as-judge rubric (factual accuracy, citation accuracy, completeness, source quality, tool efficiency); add full tracing because runs are non-deterministic.

Why This Shape Works

Multi-agent systems win on breadth-first work mainly by spending more tokens across separate context windows — parallel subagents add reasoning capacity a single agent can’t fit. The trade-off is real: these systems can use roughly 15x the tokens of a chat, so reserve this for high-value tasks with heavy parallelism. Durable memory, checkpoints, retries, and end-state evaluation are what move it from a fragile prototype to something production-grade.

Source

Anthropic, How we built our multi-agent research system (orchestrator-worker architecture, parallel subagents, prompting and evaluation lessons): https://www.anthropic.com/engineering/multi-agent-research-system

Companion patterns: Anthropic, Building effective agents: https://www.anthropic.com/engineering/building-effective-agents