Safety Evaluation and Response

Learning outcomes

Build adversarial cases from a threat model
Combine prevention detection and response measures
Define severity-aware release and incident gates

Mental model

Safety is a lifecycle control system. A threat model generates adversarial cases; evaluations exercise prevention and containment; monitoring detects escaped failures; response limits harm; incidents update the threat model and regression suite.

Threat model

Adversarial cases

Prevent and contain

Detect and grade

Respond and recover

Add regression evidence

Conceptual teaching model synthesized from:Artificial Intelligence Risk Management Framework Generative Artificial Intelligence Profile OWASP Top 10 for LLM Applications Demystifying Evals for AI Agents

Theory

Define assets, actors, trust boundaries, attack paths, and unacceptable outcomes. Build cases for direct and indirect prompt injection, sensitive-data exposure, unsafe tool use, poisoned knowledge, denial of service, overreliance, and policy evasion. Test the complete system because failures often occur between model, retrieval, interface, and authorization components.

Use severity-aware grades. A critical unauthorized side effect should fail a release even when average task quality is high. Validate both attack success and control behavior: refusal quality, least-privilege enforcement, sandbox containment, approval correctness, logging, alerting, and recovery. Keep protected adversarial holdouts and rotate cases as controls improve.

Alternatives and trade-offs

Static test suites are reproducible but become familiar. Adaptive red teaming discovers new paths but is harder to compare over time. Model-based graders scale review but require validation against expert labels. Human review is essential for ambiguous harm and domain context but must focus on high-value cases.

Failure modes and misconceptions

Prompt instructions are not a security boundary. A refusal-only test misses harmful tool execution and data leakage. Aggregate safety rates hide severe rare events. Testing only one model version ignores application changes. Secret red-team cases without reproducible incident records do not improve engineering controls.

Knowledge check

Reflect before revealing the guide

Why should one critical authorization failure override a strong average safety score?

Decision scenario

An agent evaluation places malicious instructions in retrieved documents. The suite verifies that the content remains data, tool arguments are rejected, privileged actions request approval, trace alerts fire, and the incident can be replayed after a fix.

Learning outcomes

Explain Safety Evaluation and Response as a system mechanism rather than a slogan.
Compare its alternatives, trade-offs, and production failure modes.
Apply the concept to a decision and identify evidence that would validate it.

Trade-offs

Using Safety Evaluation and Response can improve capability or control, but it also introduces cost, latency, complexity, and failure modes that must be measured against an explicit objective.

Evidence assessment