Learning outcomes
- Trace a transformer block
- Explain attention roles
- Connect architecture to constraints
Mental model
A transformer repeatedly mixes information across sequence positions with attention and transforms each position with feed-forward computation, while residual paths preserve and refine representations.
Theory
Each attention head projects queries, keys, and values. Query-key compatibility produces weights used to mix value vectors. Multi-head attention learns different interaction patterns. Feed-forward layers perform position-wise nonlinear transformation; normalization and residual connections stabilize deep computation. Positional information supplies order that attention alone lacks.
Alternatives and trade-offs
Recurrent networks process sequence state step by step, and convolutional sequence models use local kernels. Transformers parallelize training well and model long-range interactions, but attention and memory costs still shape long-context design.
Failure modes and misconceptions
Attention weights are not a complete explanation, a large context window does not guarantee use of every token, and architecture names alone do not predict application quality.
Knowledge check
What information do queries, keys, and values contribute to attention?
Decision scenario
A team attributes a missed fact to a small context window even though the fact was present. Test position, distractors, prompt framing, and retrieval quality before selecting a larger model.
Relationships
Embeddings
Transformers operate on learned vector representations.
prerequisiteTraining versus Inference
Architecture knowledge clarifies learning and generation computation.
Primary sources
- Attention Is All You Need - arXiv, verified 2026-07-16