Learning outcomes
- Explain probability reshaping
- Choose decoding settings
- Recognize reproducibility limits
Mental model
At each generation step the model produces scores over possible next tokens. Decoding converts that distribution into one continuation, trading determinism, diversity, and error risk.
Theory
Softmax turns logits into probabilities. Temperature rescales logits: lower values sharpen differences; higher values flatten them. Top-p keeps the smallest high-probability set whose cumulative mass reaches a threshold. Greedy decoding chooses the highest-probability token. Even deterministic settings may not guarantee bit-for-bit reproducibility across changing infrastructure or model versions.
Alternatives and trade-offs
Greedy decoding supports stable extraction, constrained decoding enforces syntax, sampling supports variety, and reranking can compare candidates. Settings must follow the task and evaluation evidence.
Failure modes and misconceptions
High temperature does not create knowledge, low temperature does not guarantee facts, and copying settings between models without evaluation is weak practice.
Knowledge check
How do temperature and top-p alter a distribution in different ways?
Decision scenario
Use constrained structured output and conservative decoding for an invoice parser; use measured diversity for ideation, with separate factual verification.
Relationships
Primary sources
- Generation with LLMs - Hugging Face, verified 2026-07-16