Deep Research & Autonomous Agent Swarms Reference Architecture

Architecture

Building enterprise Deep Research Systems requires coordinating autonomous sub-agents across long execution horizons while maintaining strict context hygiene, factual accuracy, and durable state persistence.

The Deep Research & Autonomous Agent Swarms Reference Architecture compiles research workflows into stateful execution graphs:

The application control plane manages search iterations, evaluates claim confidence scores, enforces scratchpad memory boundaries, and persists state checkpoints to PostgreSQL.

Task Decomposition & Search Iteration Heuristics

Unstructured user requests often lack the specificity needed for single-shot search. Deep research systems decompose queries into targeted sub-vectors:

1. Sub-Query Formulation: Generating $N$ targeted search queries covering technical implementation, trade-offs, and failure modes.
2. Parallel Scraping Fan-Out: Worker crawler sub-agents concurrently fetch web pages, extract body text, and filter out promotional fluff.
3. Factual Evidence Extraction: Converting raw unstructured page text into typed evidence claims ($E = {id, claim, source, confidence}$).

Swarm Topology Comparison

| Swarm Topology | State Isolation | Concurrency | Auditability | Token Overhead | Best For | |---|---|---|---|---|---| | Supervisor Fan-Out | High (Isolated scratchpad) | High (Parallel workers) | High (Superstep traces) | Medium | Deep research & multi-domain teardowns | | Hierarchical Team Trees | Very High (Nested team state) | High (Multi-tier parallel) | Medium (Sub-tree traces) | High | Enterprise pipelines (10+ sub-agents) | | Decentralized Swarms | Low (Shared context buffer) | Low (Sequential hand-off) | Low (Non-deterministic) | Low | Open-ended peer brainstorming |

Scratchpad State Isolation & Context Hygiene

As research sub-agents execute dozens of web searches, raw HTML dumps, DOM trees, and HTTP headers rapidly pollute prompt context windows.

Memory Isolation Principles:

• Global Parent State: Holds overall user intent, verified evidence claims, and final report drafts.
• Worker Ephemeral Scratchpad: Allocated solely for localized web search tool iterations.
• Barrier Flush Reducers: On sub-agent exit, a pure channel reducer (add_messages) sanitizes worker output, returning only high-confidence evidence summaries and discarding raw execution logs.

Durable Checkpointing & Human Interrupt Boundaries

Deep research pipelines running for hours across hundreds of tool calls require fault tolerance and human governance:

CheckpointRecord = AsyncPostgresSaver.serialize(ThreadID, SuperstepIndex, StateDelta)

1. Postgres Serialization: Pregel serializes superstep state to PostgreSQL via JSONPlusSerializer.
2. Interrupt Gates: Configure interrupt_before=["publish_report"] on final publication nodes to freeze thread state until human operator sign-off.
3. Resumption: Operators can inspect active research state, add missing sources, and resume execution from exact thread checkpoints.

Failure Defenses & Safeguards

• Recursion Caps: Enforce recursion_limit: 30 on all research graph executions to prevent infinite search loops.
• Fact Verification Thresholds: Require at least 2 independent primary sources before committing a claim to final synthesis.
• Domain Allowlisting: Restrict web crawlers to validated documentation and research repositories to block prompt injection attacks.

Decisions

| Decision | Required evidence | Review trigger | |---|---|---| | Isolate crawler scratchpad memory from global parent prompt state. | Trace inspection confirming HTML dumps stripped at barrier flush | Prompt context window exceeding token budget | | Enforce Postgres checkpointer serialization on long research runs. | Database audit showing thread checkpoint persistence | Worker process restart during active web crawl | | Require multi-source cross-verification for research claims. | Evidence registry showing source count $\ge 2$ per claim | Hallucination rate exceeding SLA threshold |

Alternatives and trade-offs

Flat supervisor routing provides clear auditability and low latency for 2–5 sub-agents. Decentralized swarms reduce orchestrator token costs but sacrifice deterministic governance and replayability.

Failure modes

• Raw HTML and DOM dumps leaking into the supervisor context window.
• Search agents getting stuck ping-ponging between redundant search queries.
• Unvalidated web page content injecting malicious instructions into sub-agent prompts.

Operational checklist

• [ ] Every research sub-graph has an explicit recursion_limit configured.
• [ ] Raw web page HTML is stripped prior to channel reducer barrier flushes.
• [ ] Postgres checkpointer connection pool handles worker process restarts cleanly.
• [ ] High-consequence publishing steps enforce interrupt_before approval gates.

Connected practice

• Labs: /labs/agent-loop-tool-selection
• System breakdowns: /systems/inside-deerflow-2
• System breakdowns: /systems/inside-langgraph

Sources

• deerflow-repo
• langgraph-repo
• anthropic-effective-agents
• anthropic-trustworthy-agents