Mental model
Sources do not combine themselves. A synthesis is a claim graph: each conclusion points to specific evidence, records how strongly it follows, preserves disagreement, and names what would invalidate it.
Theory
Define the decision and claim before searching. For each source, record identity, version, method, context, result, limitations, and verification date. Distinguish direct observation, author interpretation, vendor claim, standard requirement, local measurement, and your own inference.
Compare sources along the dimensions that explain disagreement: population, workload, metric, baseline, implementation, time, and incentives. Do not average incompatible results. State what is confirmed, what is plausible, what is contested, and what remains unknown. Attach a freshness trigger such as a dependency release, specification change, new model version, or operational incident.
Alternatives and trade-offs
A narrative review communicates context but can hide source-to-claim links. Evidence tables improve comparison but may flatten methodology. Executable experiments produce local evidence but do not automatically generalize. Strong synthesis combines structured extraction with a concise decision narrative.
Failure modes and misconceptions
Many citations do not guarantee strong evidence. Secondary summaries can erase limitations. Vendor benchmarks should not become independent conclusions. A recent source is not always better than a durable specification. Confidence labels without reasons are decorative.
Knowledge check
When two benchmarks disagree, which contextual variables should be compared before drawing a combined conclusion?
Decision scenario
A serving review compares a paper, repository documentation, release notes, and local load tests. The recommendation separates published mechanism, current implementation, measured workload behavior, and untested assumptions, then schedules review when the pinned engine version changes.
Learning outcomes
- Explain Evidence Synthesis as a system mechanism rather than a slogan.
- Compare its alternatives, trade-offs, and production failure modes.
- Apply the concept to a decision and identify evidence that would validate it.
Trade-offs
Using Evidence Synthesis can improve capability or control, but it also introduces cost, latency, complexity, and failure modes that must be measured against an explicit objective.