Evidence boundary
This system breakdown analyzes the Open Deep Research agent (src/open_deep_research), Supabase auth middleware (src/security/auth.py), and Deep Research Bench evaluation harness (tests/evaluators.py) at commit 1b7d2e80db9faa586165c60e09096dbbfd483a64 (release v0.0.16).
1. Hierarchical 3-Graph Topology & override_reducer State Channels
Open Deep Research decomposes long-horizon web synthesis into three nested LangGraph StateGraph machines: the outer deep_researcher pipeline, the middle supervisor_subgraph, and the inner researcher_subgraph:
Standard LangGraph operator.add reducers only append to list channels, making it impossible to reset a sub-graph's scratchpad or clear accumulated notes after final report synthesis without mutating state in place. In src/open_deep_research/state.py, override_reducer solves this with a tagged dictionary protocol:
When write_research_brief initializes the supervisor or final_report_generation flushes completed notes, it emits {"type": "override", "value": [...]} to deterministically reset the channel while preserving incremental operator.add semantics during normal loop turns.
2. Bounded Supervisor Fan-Out & Deliberate Reflection
After clarify_with_user verifies scope clarity (or halts early with a clarifying question via Command(goto=END)) and write_research_brief synthesizes a ResearchQuestion, control enters supervisor_subgraph.
The Lead Researcher model is bound to three structured tools:
think_tool: Forces a deliberate reflection pause to evaluate gathered evidence and identify missing sub-topics before issuing new searches.ConductResearch: Delegates a detailed, single-topic investigation to an isolated worker subgraph.ResearchComplete: Signals termination of the research loop.
To prevent rate-limit exhaustion when the supervisor emits a wide fan-out of ConductResearch calls in a single turn, supervisor_tools enforces hard concurrency slicing:
Any tool calls beyond max_concurrent_research_units (default 5) immediately receive an explicit backpressure ToolMessage instructing the supervisor to re-queue those topics on the next iteration.
3. URL Deduplication, Subgraph Compression & Progressive Truncation
If five sub-researchers each execute 6 multi-query web searches, raw webpage scrapes easily exceed 500,000 tokens. Open Deep Research applies a three-tier context defense pipeline:
- In-Tool URL Deduplication & Timeout-Bounded Summarization (
utils.py):tavily_searchexecutes multiple queries viaAsyncTavilyClient, deduplicates results by URL across the batch, truncates raw HTML/markdown tomax_content_length(50,000characters), and runs parallelsummarize_webpagecalls wrapped inasyncio.wait_for(..., timeout=60.0). - Subgraph-Level
compress_researchIsolation (deep_researcher.py): When a sub-researcher finishes its ReAct loop,compress_researchinvokescompression_modelto distill the entireresearcher_messagestrajectory into a structured synthesis (ResearcherOutputState.compressed_research). Only this compressed synthesis is returned as theToolMessageobservation tosupervisor_messages, while raw scrapes are sidelined intoraw_notesfor downstream evaluation. - Progressive Context-Overflow Recovery:
Provider exceptions are inspected by is_token_limit_exceeded() across OpenAI (context_length_exceeded), Anthropic (prompt is too long), and Gemini (ResourceExhausted). If compress_research overflows, it strips older turns via remove_up_to_last_ai_message(). If final_report_generation overflows, it looks up MODEL_TOKEN_LIMITS and applies iterative 10% character truncation across up to 3 retries:
4. RFC 8693 MCP OAuth Token Exchange & Multi-Rubric Evaluation
In multi-tenant deployments (src/security/auth.py), LangGraph @auth.authenticate verifies the caller's Supabase JWT in a worker thread (asyncio.to_thread(supabase.auth.get_user, token)) and stamps metadata["owner"] = ctx.user.identity on every thread, assistant, and store namespace.
When a researcher connects to an authenticated Model Context Protocol (MCP) server over streamable_http, get_mcp_access_token() performs an RFC 8693 OAuth 2.0 Token Exchange (urn:ietf:params:oauth:grant-type:token-exchange), trading the user's x-supabase-access-token for a resource-scoped MCP token cached in LangGraph's BaseStore under (user_id, "tokens"). Each MCP tool is wrapped by wrap_mcp_authenticate_tool(), which recursively traverses Python 3.11+ ExceptionGroup chains for McpError code -32003 (interaction required) and surfaces actionable authorization URLs via ToolException.
Finally, tests/evaluators.py benchmarks generated reports across six LLM-as-a-judge dimensions (OverallQualityScore, RelevanceScore, StructureScore, CorrectnessScore, CompletenessScore, and claim-by-claim GroundednessScore verified against raw_notes), utilizing Anthropic cache_control={"type": "ephemeral", "ttl": "1h"} breakpoints to amortize judge prompt tokens across multi-criteria evaluations.
Continue Reading
All Blog PostsInside Agent Protocol: OpenAPI Control Plane, RFC 8610 CDDL Streaming & Thread Checkpoint Architecture
A production teardown of LangChain Agent Protocol (v0.1.6)—deconstructing the 5-router OpenAPI control plane, thread checkpoint branching, 9-channel RFC 8610 CDDL streaming, subgraph causation edges, and AST topological code generation.
Inside Autonomous Agentic Code Editor: AST Indexing, Diff Generation, and Sandbox Verification
A commit-pinned examination of autonomous agentic code refactoring architectures detailing tree-sitter AST symbol indexing, unified diff generation, and isolated sandbox execution loops.
Inside Apache Kafka Partition Storage & Zero-Copy Engine
An evidence-audited, 20-diagram interactive system breakdown tracing sequential commit log append (.log, .index, .timeindex), Linux sendfile() zero-copy page cache transfers, Producer RecordAccumulator memory pools, Consumer Group Cooperative Rebalancing, and KRaft quorum leader fencing.
