Mental model
A language model does not read characters or words directly. A tokenizer converts text into a sequence of IDs drawn from a fixed vocabulary; the model operates on the learned vectors associated with those IDs.
Theory
Tokenization is a reversible encoding step, not linguistic understanding. Subword vocabularies trade sequence length against vocabulary size. Common strings may occupy one token while an uncommon name, code fragment, or non-English phrase can split into many. The result controls context usage, input cost, output limits, and the positions attention must process.
Alternatives and trade-offs
Word tokenization is intuitive but handles new words poorly. Character or byte schemes cover arbitrary text but create longer sequences. Subword and byte-level subword schemes are the common compromise.
Failure modes and misconceptions
Do not assume one token equals one word, count characters as a billing estimate, compare token counts across models without their tokenizer, or split retrieval documents solely by characters.
Knowledge check
Why can two sentences with the same character length consume different context budgets?
Decision scenario
A multilingual support assistant exceeds its context limit only for some customers. Measure with the deployed model tokenizer, compare languages and templates, then budget instructions, retrieved evidence, history, and output separately.
Learning outcomes
- Explain Tokens and Tokenization as a system mechanism rather than a slogan.
- Compare its alternatives, trade-offs, and production failure modes.
- Apply the concept to a decision and identify evidence that would validate it.
Trade-offs
Using Tokens and Tokenization can improve capability or control, but it also introduces cost, latency, complexity, and failure modes that must be measured against an explicit objective.