Ingestion and Chunking

Learning outcomes

• Design an ingestion contract
• Choose chunk boundaries
• Preserve provenance

Mental model

Ingestion turns governed source material into versioned retrievable units while preserving enough structure and provenance to reconstruct meaning.

Theory

The pipeline discovers sources, authorizes access, parses formats, normalizes content, segments on semantic boundaries, attaches metadata, deduplicates, embeds, indexes, and handles updates or deletion. Chunk size trades local specificity against sufficient context. Parent-child retrieval can search small units and return larger passages.

Alternatives and trade-offs

Fixed token windows are predictable, structural splitting respects headings or code units, semantic splitting follows topic shifts, and parent-child strategies separate search granularity from answer context.

Failure modes and misconceptions

Do not discard headings, mix access scopes, index obsolete versions forever, split tables blindly, or judge chunking without retrieval evaluation.

Knowledge check

Decision scenario

For product manuals, preserve product/version metadata and heading paths, search paragraph-sized chunks, then return the surrounding procedure with citations.

Relationships

Primary sources

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks - arXiv, verified 2026-07-16