Index Lifecycle Operations

Mental model

A retrieval index is a derived, versioned projection of authoritative content, not the system of record.

Learning outcomes

• Explain the mechanism and ownership boundaries behind Index Lifecycle Operations.
• Compare the main design alternatives and their operational trade-offs.
• Diagnose common failures and select evidence for a production decision.

Theory

Track source revision, parser, chunker, embedding model, schema, ACL version, and build ID. Use idempotent upserts, tombstones, reconciliation, shadow rebuilds, atomic aliases, and rollback checkpoints.

Trade-offs

Incremental updates minimize cost and staleness but accumulate drift. Full rebuilds restore consistency but consume capacity and complicate cutover.

Failure modes and misconceptions

Updating content without deleting old chunks; mixing embedding versions; losing ACL changes; non-idempotent ingestion; no reconciliation; and switching indexes without rollback.

Decision scenario

A new embedding model is ready while production traffic continues. Plan a shadow rebuild, comparison, cutover, and rollback.

Primary sources

• pgvector-docs
• w3c-prov-overview