Distributed Inference Parallelism

Mental model

Parallelism partitions either requests, model computation, layers, or experts; each choice moves the bottleneck between memory, compute, and communication.

Learning outcomes

• Explain the mechanism and ownership boundaries behind Distributed Inference Parallelism.
• Compare the main design alternatives and their operational trade-offs.
• Diagnose common failures and select evidence for a production decision.

Theory

Replicate when one model fits and request parallelism dominates. Partition when weights or per-request compute require multiple devices. Model topology, interconnect, collective cost, placement, and partial failure must be measured together.

Trade-offs

Partitioning enables larger models but makes each request depend on more devices. Replication isolates failures and scales requests but duplicates model memory.

Failure modes and misconceptions

Assuming more GPUs reduce latency; crossing slow network links; mismatched tensor dimensions; no topology-aware placement; correlated replicas; and ignoring collective tail latency.

Decision scenario

A model barely exceeds one GPU's memory while traffic is moderate. Compare quantization, tensor parallelism, and smaller-model routing before choosing a topology.

Primary sources

• vllm-v0102-model-runner
• kubernetes-resource-management