Mental model
Test each claim at the cheapest layer that can falsify it. Deterministic contracts catch software defects; evaluations measure uncertain behavior; adversarial tests exercise abuse; production evidence checks the real distribution.
Theory
Use ordinary unit tests for parsers, policies, permissions, calculations, state transitions, and tool adapters. Contract tests verify schemas and provider boundaries. Component evaluations isolate retrieval, classification, generation, or tool selection. End-to-end cases measure the user-visible outcome under representative state and dependencies.
Choose graders that observe the claim. Exact checks suit schemas and known facts. Retrieval metrics need relevance labels. Rubrics can grade nuanced outputs after validation against human judgment. Agent tests should inspect environment state, tool trajectory, budget use, and recovery, not just the final message. Version cases, prompts, models, tools, and graders together.
Alternatives and trade-offs
Offline suites are reproducible. Shadow traffic improves realism without user impact. Canary releases expose a small population to live behavior. Human review resolves ambiguity but is expensive. A balanced portfolio uses deterministic gates for critical invariants and sampled review for uncertain quality.
Failure modes and misconceptions
Snapshotting model prose creates brittle tests. One aggregate score hides important slices. Model judges can share the tested model's bias. Test leakage inflates apparent quality. Mocking every dependency removes the failure behavior the system must survive.
Knowledge check
Which grader should verify a tool-using agent that claims it completed a database update?
Decision scenario
A support agent suite unit-tests authorization, contract-tests tools, evaluates answer and retrieval quality, grades final account state, attacks indirect prompt injection, and converts every severe production incident into a protected regression case.
Learning outcomes
- Explain AI System Testing as a system mechanism rather than a slogan.
- Compare its alternatives, trade-offs, and production failure modes.
- Apply the concept to a decision and identify evidence that would validate it.
Trade-offs
Using AI System Testing can improve capability or control, but it also introduces cost, latency, complexity, and failure modes that must be measured against an explicit objective.