Architecture
A telemetry and response architecture for tracing model, retrieval, tool, policy, quality, cost, and user-outcome failures.
The architecture separates intent, policy, state, execution, evidence, and response. Each boundary has one accountable owner and an observable contract. Probabilistic components may propose or classify; deterministic software owns permissions, budgets, state transitions, and release conditions.
Decisions
| Decision | Required evidence | Review trigger | |---|---|---| | Trace stable identifiers and versions without logging sensitive content by default. | Versioned test, trace, or architecture record | Material workload, provider, policy, or data change | | Join service indicators with evaluation and outcome evidence. | Versioned test, trace, or architecture record | Material workload, provider, policy, or data change | | Define containment and rollback before a model or prompt release. | Versioned test, trace, or architecture record | Material workload, provider, policy, or data change | | Preserve evidence before changing a failing system. | Versioned test, trace, or architecture record | Material workload, provider, policy, or data change |
Alternatives and trade-offs
A centralized control plane simplifies policy and evidence collection but can become a latency and availability dependency. Distributed enforcement reduces central bottlenecks but requires consistent identity, versions, and audit semantics. Prefer the smallest boundary that can enforce the invariant and prove the outcome.
Failure modes
- Prompts are logged indiscriminately.
- Provider status is treated as complete service health.
- Quality incidents have no severity or owner.
- Rollback destroys the evidence needed to explain the incident.
Operational checklist
- [ ] Trace IDs connect retrieval, model, tool, and outcome events.
- [ ] Sensitive fields are redacted or represented by hashes and labels.
- [ ] SLOs have error-budget response policies.
- [ ] Kill switches and fallbacks are exercised.
- [ ] Post-incident actions update tests and controls.
Connected practice
- Labs: /labs/cost-latency-reliability, /labs/evaluation-metric-comparison
- Projects: /projects/system-design-casebook, /projects/evaluation-control-system-project
- System breakdowns: /systems/inside-vllm
Sources
otel-genaiopentelemetry-tracessre-booknist-ai-rmf
Continue Reading
All Blog PostsBalancing Cost, Latency, Reliability, and Security
Operational trade-offs must be measured at the successful user outcome, inside the same security and quality boundary.
ColBERTv2 Paper Breakdown: Multi-Vector Late Interaction, MaxSim Scoring, and Residual Compression
Definitive retrieval-systems teardown of ColBERTv2 (Santhanam et al., arXiv:2112.01488): bridging single-vector bi-encoder compression loss and cross-encoder latency via token-level MaxSim late interaction, centroid-residual b-bit quantization, and PLAID inverted-list pruning.
The Complete Map of an LLM System
A production LLM system is a chain of contracts, not a model wrapped in a chat box.
