Cost and Reliability Engineering

Architecture

A control system for optimizing cost per successful task while preserving quality, latency, safety, capacity, and fallback behavior.

The architecture separates intent, policy, state, execution, evidence, and response. Each boundary has one accountable owner and an observable contract. Probabilistic components may propose or classify; deterministic software owns permissions, budgets, state transitions, and release conditions.

Decisions

| Decision | Required evidence | Review trigger | |---|---|---| | Optimize cost per accepted outcome rather than price per token. | Versioned test, trace, or architecture record | Material workload, provider, policy, or data change | | Budget quality, latency, reliability, and safety together. | Versioned test, trace, or architecture record | Material workload, provider, policy, or data change | | Route only when eligibility and fallback evidence are explicit. | Versioned test, trace, or architecture record | Material workload, provider, policy, or data change | | Use error budgets to decide when optimization experiments may proceed. | Versioned test, trace, or architecture record | Material workload, provider, policy, or data change |

Alternatives and trade-offs

A centralized control plane simplifies policy and evidence collection but can become a latency and availability dependency. Distributed enforcement reduces central bottlenecks but requires consistent identity, versions, and audit semantics. Prefer the smallest boundary that can enforce the invariant and prove the outcome.

Failure modes

• Retries and human correction are excluded from cost.
• Cheap models receive tasks they cannot complete reliably.
• Caching hides stale or unauthorized outputs.
• Autoscaling reacts after unbounded queues have formed.

Operational checklist

• [ ] Unit economics include failures, retries, review, and platform cost.
• [ ] Routing cohorts and escalation paths are evaluated.
• [ ] Latency budgets reserve time for fallbacks.
• [ ] Provider degradation and quota exhaustion are rehearsed.
• [ ] Optimization changes have paired quality and reliability gates.

Connected practice

• Labs: /labs/cost-latency-reliability, /labs/inference-batching-queueing
• Projects: /projects/inference-cost-latency-project, /projects/architecture-decision-workbook
• System breakdowns: /systems/inside-vllm

Sources

• openai-latency
• sre-book
• kubernetes-hpa
• openai-evals