Learning outcomes
- Identify compute memory communication and failure domains
- Compare data tensor and pipeline partitioning
- Design bounded degradation and recovery
Mental model
Distribution exchanges one resource limit for coordination costs. A useful design names what is partitioned, what is replicated, when components communicate, and how partial failure affects user-visible work.
Theory
Data parallelism replicates a model and sends different requests or batches to each replica. Tensor parallelism partitions operations within a layer and requires frequent collective communication. Pipeline parallelism partitions layers into stages and introduces pipeline bubbles and stage balance concerns. Expert parallelism routes tokens to subsets of parameters and adds all-to-all communication.
Placement must account for accelerator memory, interconnect bandwidth, topology, model loading, KV-cache locality, and failure domains. The control plane tracks capacity and health; the data plane moves requests, activations, cache state, and results. Backpressure and admission control prevent a slow or failed partition from creating an unbounded queue.
Alternatives and trade-offs
Scale-up keeps communication local but reaches hardware limits. Scale-out increases capacity and resilience options while raising coordination cost. Replication favors request throughput; model partitioning is required when one model does not fit a device. Disaggregation can specialize prefill and decode capacity but complicates cache transfer and scheduling.
Failure modes and misconceptions
More accelerators do not guarantee linear speedup. Cross-zone traffic can dominate latency and cost. A single coordinator may become a failure or throughput bottleneck. Retrying distributed work without deduplication wastes scarce capacity. Autoscaling replicas cannot fix a model partition that lacks memory.
Knowledge check
When should a serving design replicate the model instead of partitioning one request across devices?
Decision scenario
A serving team keeps interactive replicas within one high-bandwidth domain, uses tensor parallelism only where the model cannot fit one device, separates failure domains across replicas, and sheds low-priority work when collective latency rises.
Learning outcomes
- Explain Distributed AI Systems as a system mechanism rather than a slogan.
- Compare its alternatives, trade-offs, and production failure modes.
- Apply the concept to a decision and identify evidence that would validate it.
Trade-offs
Using Distributed AI Systems can improve capability or control, but it also introduces cost, latency, complexity, and failure modes that must be measured against an explicit objective.
Evidence assessment
Theory and decision mastery
Decision scenario
A production team must adopt Distributed AI Systems while meeting quality, latency, security, and operating constraints.
Which decision process is most defensible?
Relationships
Distributed execution extends a single serving control path across devices and failure domains.
Distributed Inference Parallelism builds on Distributed AI Systems.
Primary sources
- Horizontal Pod Autoscaling — Kubernetes, verified 2026-07-21
- Efficient Memory Management for Large Language Model Serving with PagedAttention — SOSP / arXiv, verified 2026-07-16
- vLLM inference and serving engine — vLLM Project, verified 2026-07-16