Mental model
Production quality is a portfolio of outcomes: task success, cost, latency, availability, and safe degradation. A system is not optimized if one metric improves by hiding failure elsewhere.
Theory
Model unit cost includes input, cached input, output, retries, retrieval, tools, storage, evaluation, and operations. Reliability design sets timeouts, retry budgets, idempotency, circuit breakers, fallbacks, and service objectives. Route by task difficulty and consequence. Observe p50 and tail latency, cost per successful task, and fallback quality.
Alternatives and trade-offs
Smaller models reduce unit cost, caching avoids repeated work, batch processing improves utilization, self-hosting changes fixed and variable costs, and graceful degradation preserves core workflows.
Failure modes and misconceptions
Do not retry every error, route solely by token price, report cost per request without success, or use a fallback that violates the original safety contract.
Knowledge check
Why is cost per successful task more informative than cost per API call?
Decision scenario
A document workflow retries transient failures once, falls back to a validated smaller model for low-risk extraction, and queues high-risk cases for review.
Learning outcomes
- Explain Cost Latency and Reliability as a system mechanism rather than a slogan.
- Compare its alternatives, trade-offs, and production failure modes.
- Apply the concept to a decision and identify evidence that would validate it.
Trade-offs
Using Cost Latency and Reliability can improve capability or control, but it also introduces cost, latency, complexity, and failure modes that must be measured against an explicit objective.