Mental model
Retrieval evaluation compares a versioned query set and relevance judgments with ranked results, then segments misses by stage and cohort.
Learning outcomes
- Explain the mechanism and ownership boundaries behind Retrieval Evaluation.
- Compare the main design alternatives and their operational trade-offs.
- Diagnose common failures and select evidence for a production decision.
Theory
Use recall-oriented candidate metrics, precision and ranking metrics, answer-support coverage, abstention cases, latency, and authorization checks. Build judgments from representative tasks and inspect disagreements rather than treating one metric as truth.
Trade-offs
Offline judgments are repeatable but age as corpora and user intent change. Online outcomes are realistic but confounded by presentation, generation, and user behavior.
Failure modes and misconceptions
Evaluating only final answers; tiny synthetic query sets; unlabeled hard negatives; leaking test queries into tuning; ignoring access filters; and averaging across unlike cohorts.
Decision scenario
A RAG assistant answers common questions well but fails rare policy exceptions. Design a retrieval test set and segmented metrics that expose the gap.
Which retrieval metrics distinguish a missing candidate from poor ranking?
Primary sources
information-retrieval-bookopenai-evals