GRPO vs. PPO Reasoning Alignment Simulator
Explore Group Relative Policy Optimization (GRPO) from DeepSeekMath and DeepSeek-R1. Compare traditional 4-model PPO with critic-free group advantage normalization Â_i = (r_i - μ_r)/(σ_r + ε), inspect token-level Schulman k_3 non-negative KL divergence penalties, test zero-variance collapse guards, and observe ~50% GPU cluster VRAM reductions across 7B, 67B, and 671B architectures.
Critic-Free Group Advantage Normalization
Examine how GRPO computes relative advantage Â_i = (r_i - μ_r) / (σ_r + ε) across $G$ sampled reasoning rollouts. By substituting the learned Critic value network $V_\phi(s)$ with empirical group statistics, GRPO eliminates value function training instability and saves ~50% of peak training VRAM.
Zero-Variance Collapse Guard ($\sigma_r < 10^-8$)
Select Zero-Variance Collapse Guard preset. When all $G$ reasoning paths produce identical rewards (e.g. all fail difficult Olympiad theorem questions, $r_i = 0$), $\sigma_r$ vanishes. Witness how GRPO gracefully forces Â_i = 0.0 to safeguard training against division-by-zero gradient explosions.
Schulman k_3 Non-Negative KL Divergence
Inspect the per-token KL heatmap. Traditional sample estimators log(π_θ / π_ref) can be negative, causing reward instability. Schulman's $k_3$ formulation r - log(r) - 1 ≥ 0 guarantees non-negativity at every individual token position while remaining an unbiased estimator of true KL divergence.