Mental model
Caching reuses a previous computation or stable context prefix when an application can define when reuse is equivalent enough and still fresh.
Theory
Exact caches key on normalized inputs and versions. Semantic caches use similarity to reuse answers across related queries, increasing false-hit risk. Provider context caching can reduce repeated prefix processing. Keys should include model, prompt, policy, tenant, data version, and relevant settings; entries need TTLs and invalidation.
Alternatives and trade-offs
Exact caching maximizes correctness, semantic caching broadens reuse, prefix caching reduces prompt processing, and precomputation handles predictable workloads.
Failure modes and misconceptions
Do not cross tenant boundaries, cache personalized or volatile outputs without policy, ignore model or prompt versions, or report cached latency as uncached capacity.
Knowledge check
Which dimensions belong in a safe cache key for an LLM response?
Decision scenario
Cache a stable public handbook prefix by document version, but never reuse a personalized compliance answer across users or after policy updates.
Learning outcomes
- Explain Exact and Semantic Caching as a system mechanism rather than a slogan.
- Compare its alternatives, trade-offs, and production failure modes.
- Apply the concept to a decision and identify evidence that would validate it.
Trade-offs
Using Exact and Semantic Caching can improve capability or control, but it also introduces cost, latency, complexity, and failure modes that must be measured against an explicit objective.