Cost Latency and Reliability

Learning outcomes

• Model unit cost
• Set reliability targets
• Design bounded fallback behavior

Mental model

Production quality is a portfolio of outcomes: task success, cost, latency, availability, and safe degradation. A system is not optimized if one metric improves by hiding failure elsewhere.

Theory

Model unit cost includes input, cached input, output, retries, retrieval, tools, storage, evaluation, and operations. Reliability design sets timeouts, retry budgets, idempotency, circuit breakers, fallbacks, and service objectives. Route by task difficulty and consequence. Observe p50 and tail latency, cost per successful task, and fallback quality.

Alternatives and trade-offs

Smaller models reduce unit cost, caching avoids repeated work, batch processing improves utilization, self-hosting changes fixed and variable costs, and graceful degradation preserves core workflows.

Failure modes and misconceptions

Do not retry every error, route solely by token price, report cost per request without success, or use a fallback that violates the original safety contract.

Knowledge check

Decision scenario

A document workflow retries transient failures once, falls back to a validated smaller model for low-risk extraction, and queues high-risk cases for review.

Relationships

Primary sources

• Latency Optimization - OpenAI, verified 2026-07-16
• OpenTelemetry Generative AI Semantic Conventions - OpenTelemetry, verified 2026-07-16