Mental model
Training changes model parameters from examples and an objective. Inference keeps parameters fixed and computes outputs from the current input, optional tools, and decoding controls.
Theory
During training, a forward pass produces predictions, a loss measures error, backpropagation computes gradients, and an optimizer updates parameters. During autoregressive inference, the model repeatedly predicts a next-token distribution, selects a token, appends it, and continues. Fine-tuning is training; prompting, retrieval, caching, and tool use primarily change inference context or workflow.
Alternatives and trade-offs
Prompting changes instructions, RAG supplies current knowledge, tools perform actions, and fine-tuning changes learned behavior. Choose based on the gap rather than treating every quality problem as a training problem.
Failure modes and misconceptions
Temperature cannot teach durable knowledge, RAG does not update model weights, and a successful training loss does not establish production usefulness.
Knowledge check
Which system changes can be deployed without modifying model parameters?
Decision scenario
A model knows the policy but returns the wrong JSON shape. Strengthen schemas and validation before considering fine-tuning; the issue is an interface contract, not missing knowledge.
Learning outcomes
- Explain Training versus Inference as a system mechanism rather than a slogan.
- Compare its alternatives, trade-offs, and production failure modes.
- Apply the concept to a decision and identify evidence that would validate it.
Trade-offs
Using Training versus Inference can improve capability or control, but it also introduces cost, latency, complexity, and failure modes that must be measured against an explicit objective.