Learning outcomes
- Decompose response latency
- Distinguish latency and throughput
- Identify optimization levers
Mental model
Latency describes time experienced by one request; throughput describes work completed over time. Optimizing one can degrade the other.
Theory
End-to-end latency includes queueing, network time, prompt processing, time to first token, output generation, retrieval, tools, and rendering. Output tokens are generated sequentially, so response length often dominates. Batching can raise throughput while adding queue delay. Report percentiles, not only averages.
Alternatives and trade-offs
Streaming improves perceived latency, caching avoids repeated work, smaller models reduce compute, parallel independent calls reduce critical path, and shorter outputs reduce generation time.
Failure modes and misconceptions
Do not optimize provider latency while ignoring tools, use averages that hide tail behavior, or retry unboundedly under overload.
Knowledge check
Why can batching improve throughput while making an individual request slower?
Decision scenario
For an interactive assistant, track time to first useful output and p95 completion time separately, then shorten responses and parallelize independent retrieval.
Relationships
Primary sources
- Latency Optimization - OpenAI, verified 2026-07-16