Mental model
Ingestion turns governed source material into versioned retrievable units while preserving enough structure and provenance to reconstruct meaning.
Theory
The pipeline discovers sources, authorizes access, parses formats, normalizes content, segments on semantic boundaries, attaches metadata, deduplicates, embeds, indexes, and handles updates or deletion. Chunk size trades local specificity against sufficient context. Parent-child retrieval can search small units and return larger passages.
Alternatives and trade-offs
Fixed token windows are predictable, structural splitting respects headings or code units, semantic splitting follows topic shifts, and parent-child strategies separate search granularity from answer context.
Failure modes and misconceptions
Do not discard headings, mix access scopes, index obsolete versions forever, split tables blindly, or judge chunking without retrieval evaluation.
Knowledge check
Why might a system search small chunks but return a larger parent section?
Decision scenario
For product manuals, preserve product/version metadata and heading paths, search paragraph-sized chunks, then return the surrounding procedure with citations.
Learning outcomes
- Explain Ingestion and Chunking as a system mechanism rather than a slogan.
- Compare its alternatives, trade-offs, and production failure modes.
- Apply the concept to a decision and identify evidence that would validate it.
Trade-offs
Using Ingestion and Chunking can improve capability or control, but it also introduces cost, latency, complexity, and failure modes that must be measured against an explicit objective.