Learning outcomes
- Interpret embedding geometry
- Select similarity measures
- Recognize representation limits
Mental model
An embedding is a learned coordinate representation. Useful relationships become geometric enough that distance or similarity can support prediction, clustering, and retrieval.
Theory
Token embeddings initialize transformer representations; sentence or document embedding models create a single vector intended for comparison. Geometry is task- and model-dependent. Cosine similarity compares direction, dot product mixes direction and magnitude, and Euclidean distance measures absolute separation. None of them proves semantic equivalence or factual agreement.
Alternatives and trade-offs
Sparse lexical vectors preserve exact terms and are inspectable. Dense embeddings capture paraphrase-like similarity. Hybrid retrieval deliberately uses both when exact identifiers and semantic intent matter.
Failure modes and misconceptions
Similarity is not truth, index scores are not calibrated probabilities, and vectors from different model versions are usually not interchangeable. Evaluate retrieval on representative queries.
Knowledge check
Why can two contradictory statements still have high embedding similarity?
Decision scenario
A policy search system misses document codes but finds conceptual matches. Keep dense retrieval, add lexical retrieval and metadata filters, and evaluate the combined candidate set.
Relationships
Tokens and Tokenization
Embedding lookup begins with token identifiers.
prerequisiteTransformer Architecture
Transformers operate on learned vector representations.
prerequisiteVector Search
Semantic vector retrieval uses learned representations and similarity.
Primary sources
- Attention Is All You Need - arXiv, verified 2026-07-16
- Sentence-BERT Sentence Embeddings using Siamese BERT-Networks - arXiv, verified 2026-07-16