Activity v1

Evaluation Metric Comparison Lab

Match failure modes to deterministic metrics model graders human review and production measures without collapsing quality into one score.

percentage
beginner
seed 11

Evaluation Metric Comparison Lab: Baseline

Explore the core controls with stable inputs and visible assumptions.

Assumptions

The simulation is deterministic and intentionally simplifies provider and hardware behavior.

Failure injection

Stable baseline with no injected production fault.

0 of 5 decisions recorded
0%
1. Every response matches the required JSON contract.
2. Relevant evidence appears in the candidate set.
3. The answer resolves the user's task under a nuanced rubric.
4. No critical unauthorized action occurs.
5. Real users complete the workflow successfully.
0 saved attempts

Expected outcomes

  • Match graders to observable quality claims
  • Separate aggregate slice and severity gates

Connected concepts

LLM EvaluationAI System TestingSafety Evaluation and Response

Apply this lab in an architecture

AI Observability and Incident ResponseLLM Evaluation Control Plane
Back to labs