Mental model
A trace is a causal tree for one user-visible outcome. Each span represents a bounded operation; attributes describe its configuration; events capture important state changes; links connect asynchronous or fan-out work.
Theory
Start the trace at the product boundary and propagate context through queues, model calls, retrieval, tools, and background work. Record stable identifiers for model version, prompt template, dataset or index revision, tool name, response status, token usage, and error category. Measure queue, network, retrieval, inference, and application processing separately.
Trace payloads are sensitive. Prefer hashes, counts, classifications, and references over raw prompts or retrieved text. Apply tenant isolation, sampling, retention, and role-based access. Link an evaluation result or incident to the trace so operational evidence feeds regression suites.
Alternatives and trade-offs
Metrics summarize populations and support alerts. Logs preserve discrete events. Traces explain individual paths. Most production systems need all three with shared correlation identifiers. Full sampling improves debugging but increases cost and privacy risk; tail-based or risk-based sampling retains unusual paths more efficiently.
Failure modes and misconceptions
A model-provider request ID is not an end-to-end trace. Logging complete prompts by default can leak sensitive data. High-cardinality labels can overwhelm metric systems. Spans without version metadata make comparison ambiguous. Observability cannot compensate for missing product-level success definitions.
Knowledge check
Which attributes are necessary to reproduce a quality regression without storing raw user content?
Decision scenario
A RAG trace links the authenticated request to query rewriting, index revision, candidate IDs, reranker version, context token count, generation usage, citation validation, and final task rating. A failing citation can be traced to the exact retrieval stage.
Learning outcomes
- Explain AI Tracing as a system mechanism rather than a slogan.
- Compare its alternatives, trade-offs, and production failure modes.
- Apply the concept to a decision and identify evidence that would validate it.
Trade-offs
Using AI Tracing can improve capability or control, but it also introduces cost, latency, complexity, and failure modes that must be measured against an explicit objective.