CorrGRPO Solves Multi-Reward Imbalance in GRPO: Pearson Correlation Normalization Boosts Code & Tool Calling

核心背景与行业痛点

Group Relative Policy Optimization (GRPO) forms the reinforcement learning backbone for reasoning architectures like DeepSeek-R1. In enterprise agent development, training models requires balancing diverse multi-reward objectives (e.g., test pass rates, algorithmic complexity, tool grammar compliance, and safety guards). Standard GRPO aggregates rewards and normalizes by overall standard deviation—which corresponds to the sum of pairwise covariances. Consequently, large-scale, high-variance rewards dominate advantage calculations and completely suppress subtle yet vital secondary reward signals.

架构亮点与底层机制

HKUST researchers propose CorrGRPO (Correlation-Normalized GRPO), a mathematically principled multi-reward optimization algorithm:

1. Pearson Correlation Mapping: Normalizes the pairwise covariance matrix into standard Pearson correlation coefficients.
2. Scale-Invariant Normalization: Balances the influence of differently-scaled reward components while leaving the centered total reward advantage direction untouched.
3. Eliminating Reward Dominance: Ensures that small-scale rewards (e.g., fine-grained safety boundaries or syntactic constraints) impart active gradient updates alongside dominant execution rewards.
4. Critic-Free Computational Simplicity: Retains the lightweight, critic-free properties of GRPO, requiring zero architectural modifications or extra hyperparameters.

权威 Benchmark 与实测跑分对比

Benchmarked across models from 0.5B to 8B parameters across code generation, tool calling, and agent security:

1. Code Generation Outperformance: In tri-reward setups (unit tests + complexity + style), CorrGRPO beats standard GRPO by 3.2 to 4.8 percentage points on HumanEval and MBPP Pass@1.
2. Tool-Calling Stability: Increases multi-step tool-use completion rates by 5.4% by preserving syntactic parameter constraint rewards.
3. Robust Safety Guardrails: Lifts defense success against adversarial agent prompt injection from 82.1% to 93.7% without compromising base code problem-solving ability.

开发者实战落地与开箱指南

CorrGRPO is open-sourced on GitHub with modular implementations compatible with TRL, vLLM, and OpenRLHF. Teams training DeepSeek-R1 style reasoning models with multi-reward objectives can replace standard variance normalizers with CorrGRPO in minutes.