LLM Evaluation

Learning outcomes

• Define an evaluation target
• Build representative cases
• Combine metrics and review

Mental model

Evaluation is an executable definition of acceptable system behavior over representative cases, risks, and operating conditions.

Theory

Start with the decision the evaluation will support. Build cases from real task distributions, important slices, adversarial inputs, and known failures. Use deterministic checks for contracts, reference metrics where valid, rubric-based model grading for nuanced criteria, and calibrated human review. Track versions of data, prompts, models, tools, and graders.

Alternatives and trade-offs

Offline evaluations are reproducible, online experiments measure real behavior, shadow traffic reduces rollout risk, and production monitoring detects drift after release.

Failure modes and misconceptions

Do not optimize a single aggregate score, let test data leak into prompts, use an unvalidated judge, ignore slice regressions, or change several variables without attribution.

Knowledge check

Decision scenario

A RAG assistant gates releases on retrieval recall, citation support, answer usefulness, refusal quality, latency, and cost across document types and access roles.

Learning outcomes

• Explain LLM Evaluation as a system mechanism rather than a slogan.
• Compare its alternatives, trade-offs, and production failure modes.
• Apply the concept to a decision and identify evidence that would validate it.

Trade-offs

Using LLM Evaluation can improve capability or control, but it also introduces cost, latency, complexity, and failure modes that must be measured against an explicit objective.

Relationships

Primary sources

• Evaluation Best Practices — OpenAI, verified 2026-07-16