Group Relative Policy Optimization

---

Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm introduced by researchers at DeepSeek in 2024. The algorithm modifies the widely-used Proximal Policy Optimization (PPO) approach by eliminating the critic network and instead computing advantage estimates from reward statistics within each batch of sampled actions.

Method

Traditional PPO implementations use an actor-critic architecture with separate policy and value networks. GRPO removes the value network entirely, reducing computational overhead and memory requirements during training.

For a given state, GRPO samples multiple actions and computes advantages by comparing each action's reward to the group statistics. The advantage function is:

$A^{{\pi }_{{\theta }_t}}(s,a_j)=\frac{r(s,a_j)-\mu }{\sigma }$

where $\mu$ and $\sigma$ are the mean and standard deviation of rewards within the sampled group. This normalization ensures that advantages are computed relative to the current batch rather than requiring a separate value function approximation.

The policy update uses a clipped objective similar to PPO:

${{ℒ}}_{\text{GRPO}}(\theta )=\frac{1}{G}\sum _{i=1}^G\mathrm{clip}({\rho }_i,1-ϵ,1+ϵ)A_i-\beta D_{\mathrm{K}\mathrm{L}}({\pi }_{{\theta }_t}\Vert {\pi }_{\theta })$

where ${\rho }_i$ represents the probability ratio between current and old policies, and the KL divergence term prevents excessive policy changes.

Applications

GRPO was first applied to train mathematical reasoning models, including the DeepSeekMath 7B model. The algorithm has since been used in training the DeepSeek-R1 series, which demonstrated improved performance on reasoning benchmarks.

Several machine learning frameworks have incorporated GRPO implementations, including the Hugging Face Transformers Reinforcement Learning (TRL) library and Unsloth's fine-tuning toolkit.

See also

• Proximal Policy Optimization
• Reinforcement learning from human feedback
• Actor-critic method

References

This article "Group Relative Policy Optimization" is from Wikipedia. The list of its authors can be seen in its historical and/or the page Edithistory:Group Relative Policy Optimization. Articles copied from Draft Namespace on Wikipedia could be seen on the Draft Namespace of Wikipedia and not main one.

---