Mental model
AI incident response joins conventional service operations with model, prompt, retrieval, data, policy, and evaluation evidence.
Learning outcomes
- Explain the mechanism and ownership boundaries behind AI Incident Response.
- Compare the main design alternatives and their operational trade-offs.
- Diagnose common failures and select evidence for a production decision.
Theory
Predefine severity, ownership, kill switches, fallbacks, evidence preservation, privacy handling, communication, and recovery criteria. Snapshot versions and representative traces before changing the system.
Trade-offs
Immediate rollback limits harm but may remove useful capability. Continuing to collect evidence helps diagnosis but can expose more users, so containment thresholds must be explicit.
Failure modes and misconceptions
Deleting traces during rollback; changing prompts before preserving evidence; no model or index version; missing tenant scope; relying on provider status alone; and declaring recovery without evaluation.
Decision scenario
An agent begins executing a valid tool with the wrong tenant context. Define the first fifteen minutes of containment and evidence preservation.
Which AI-specific versions must be captured during an incident in addition to normal service telemetry?
Primary sources
sre-booknist-ai-rmf