Map/Product Operations and Governance/Security Safety and Governance/AI Security/LLM Threats and Defenses
Concept lesson

Security and Privacy for LLM Systems

How untrusted inputs sensitive data tools and external knowledge expand the threat model.

lessonpublicFreshness: current18 min

Learning outcomes

  • Threat model an LLM workflow
  • Isolate untrusted data
  • Apply least privilege and minimization

Mental model

An LLM is an untrusted probabilistic component processing both instructions and data. Security comes from system boundaries, least privilege, validation, and data governance, not from asking the model to behave.

Threat model
Trust boundaries
Minimize and isolate
Validate and authorize
Monitor and respond

Theory

Threats include direct and indirect prompt injection, sensitive information disclosure, insecure output handling, poisoned knowledge, excessive tool agency, denial of service, and supply-chain compromise. Treat external content as untrusted, isolate tenants, minimize retained data, authorize every action outside the model, sanitize rendering, and maintain audit and incident paths.

Alternatives and trade-offs

Provider filters can complement controls, sandboxing contains execution, allowlisted tools reduce authority, and human approval addresses consequential actions. No single layer is sufficient.

Failure modes and misconceptions

Prompt instructions are not access control, redaction after logging may be too late, RAG sources can contain attacks, and output schemas do not establish authorization.

Knowledge check

Reflect before revealing the guide

Why must tool authorization be independent from the model's stated reasoning?

Decision scenario

A support agent retrieves only documents the signed-in user may access, strips active content, uses read-only scoped tools, and requires approval for account changes.

Relationships

risk-mitigated-by

Tool Calling

Tool authority introduces risks controlled through validation and least privilege.

risk-mitigated-by

Ingestion and Chunking

External content ingestion requires provenance isolation and access controls.

Primary sources