Map/AI Engineering and Systems/LLM Application Engineering/Context Strategies/Context Construction
Concept lesson

Exact and Semantic Caching

How reusable computations reduce latency and cost while introducing freshness and correctness risks.

lessonpublicFreshness: current12 min

Learning outcomes

  • Choose a cache layer
  • Define safe cache keys
  • Balance freshness and reuse

Mental model

Caching reuses a previous computation or stable context prefix when an application can define when reuse is equivalent enough and still fresh.

Request
Versioned cache key
Hit policy
Freshness check
Reuse or compute

Theory

Exact caches key on normalized inputs and versions. Semantic caches use similarity to reuse answers across related queries, increasing false-hit risk. Provider context caching can reduce repeated prefix processing. Keys should include model, prompt, policy, tenant, data version, and relevant settings; entries need TTLs and invalidation.

Alternatives and trade-offs

Exact caching maximizes correctness, semantic caching broadens reuse, prefix caching reduces prompt processing, and precomputation handles predictable workloads.

Failure modes and misconceptions

Do not cross tenant boundaries, cache personalized or volatile outputs without policy, ignore model or prompt versions, or report cached latency as uncached capacity.

Knowledge check

Reflect before revealing the guide

Which dimensions belong in a safe cache key for an LLM response?

Decision scenario

Cache a stable public handbook prefix by document version, but never reuse a personalized compliance answer across users or after policy updates.

Relationships

prerequisite

Context Engineering

Caching reuses parts of context construction and model work.

Primary sources