Tokens and Tokenization

Learning outcomes

• Explain subword tokenization
• Estimate token budget effects
• Diagnose boundary surprises

Mental model

A language model does not read characters or words directly. A tokenizer converts text into a sequence of IDs drawn from a fixed vocabulary; the model operates on the learned vectors associated with those IDs.

Theory

Tokenization is a reversible encoding step, not linguistic understanding. Subword vocabularies trade sequence length against vocabulary size. Common strings may occupy one token while an uncommon name, code fragment, or non-English phrase can split into many. The result controls context usage, input cost, output limits, and the positions attention must process.

Alternatives and trade-offs

Word tokenization is intuitive but handles new words poorly. Character or byte schemes cover arbitrary text but create longer sequences. Subword and byte-level subword schemes are the common compromise.

Failure modes and misconceptions

Do not assume one token equals one word, count characters as a billing estimate, compare token counts across models without their tokenizer, or split retrieval documents solely by characters.

Knowledge check

Decision scenario

A multilingual support assistant exceeds its context limit only for some customers. Measure with the deployed model tokenizer, compare languages and templates, then budget instructions, retrieved evidence, history, and output separately.

Relationships

Primary sources

• Attention Is All You Need - arXiv, verified 2026-07-16
• The Tokenization Pipeline - Hugging Face, verified 2026-07-16