Mental model
Latency describes time experienced by one request; throughput describes work completed over time. Optimizing one can degrade the other.
Theory
End-to-end latency includes queueing, network time, prompt processing, time to first token, output generation, retrieval, tools, and rendering. Output tokens are generated sequentially, so response length often dominates. Batching can raise throughput while adding queue delay. Report percentiles, not only averages.
Alternatives and trade-offs
Streaming improves perceived latency, caching avoids repeated work, smaller models reduce compute, parallel independent calls reduce critical path, and shorter outputs reduce generation time.
Failure modes and misconceptions
Do not optimize provider latency while ignoring tools, use averages that hide tail behavior, or retry unboundedly under overload.
Knowledge check
Why can batching improve throughput while making an individual request slower?
Decision scenario
For an interactive assistant, track time to first useful output and p95 completion time separately, then shorten responses and parallelize independent retrieval.
Learning outcomes
- Explain Latency and Throughput as a system mechanism rather than a slogan.
- Compare its alternatives, trade-offs, and production failure modes.
- Apply the concept to a decision and identify evidence that would validate it.
Trade-offs
Using Latency and Throughput can improve capability or control, but it also introduces cost, latency, complexity, and failure modes that must be measured against an explicit objective.