Mental model
An embedding is a learned coordinate representation. Useful relationships become geometric enough that distance or similarity can support prediction, clustering, and retrieval.
Theory
Token embeddings initialize transformer representations; sentence or document embedding models create a single vector intended for comparison. Geometry is task- and model-dependent. Cosine similarity compares direction, dot product mixes direction and magnitude, and Euclidean distance measures absolute separation. None of them proves semantic equivalence or factual agreement.
Alternatives and trade-offs
Sparse lexical vectors preserve exact terms and are inspectable. Dense embeddings capture paraphrase-like similarity. Hybrid retrieval deliberately uses both when exact identifiers and semantic intent matter.
Failure modes and misconceptions
Similarity is not truth, index scores are not calibrated probabilities, and vectors from different model versions are usually not interchangeable. Evaluate retrieval on representative queries.
Knowledge check
Why can two contradictory statements still have high embedding similarity?
Decision scenario
A policy search system misses document codes but finds conceptual matches. Keep dense retrieval, add lexical retrieval and metadata filters, and evaluate the combined candidate set.
Learning outcomes
- Explain Embeddings as a system mechanism rather than a slogan.
- Compare its alternatives, trade-offs, and production failure modes.
- Apply the concept to a decision and identify evidence that would validate it.
Trade-offs
Using Embeddings can improve capability or control, but it also introduces cost, latency, complexity, and failure modes that must be measured against an explicit objective.