TPPO: Tracking-Value PPO for Language Model Reasoning Resolves Critic Collapse Under Sparse Terminal Rewards

Background and the Problem

In post-training language models for complex mathematical reasoning and code synthesis, reinforcement learning with verifiable rewards (RLVR) has become standard. Proximal Policy Optimization (PPO) is the canonical choice due to its policy constraint stability. However, standard PPO relies on a critic network to estimate state-action values along trajectory steps. Under sparse terminal rewards—where a multi-thousand token chain of thought receives only a binary final verification signal—standard critic models struggle to track localized value evolution, resulting in noisy advantage estimates that destabilize policy convergence.

Architecture and How It Works

To resolve critic degradation under sparse terminal feedback, researchers from LINs-lab introduced TPPO (Tracking-Value PPO), fundamentally re-engineering critic value modeling:

1. Bayesian Decomposition with Additive Log-Odds: Mathematical derivation reveals that successive prefix value states in reasoning sequences admit an additive structure when projected into log-odds space;
2. Cumulative Value Parameterization: TPPO models critic representations by accumulating position-wise scalar predictions into trajectory value logits. Supervised solely by terminal outcome verification, this cumulative formulation precisely reconstructs token-level credit dynamics across complex reasoning traces;
3. Noise-Resilient Advantage Estimation: Circumvents the compounding gradient noise of conventional regression heads over long sequences, producing substantially more dependable Generalized Advantage Estimation (GAE) signals.

Benchmarks and Measured Results

Evaluated extensively across mathematical, logical, and code benchmarks on Qwen model families:

• Massive Math Gains: Elevates aggregate mathematical reasoning scores over standard PPO by 7.0 points on Qwen3-4B-Base and by 8.4 points on Qwen3-8B-Base;
• Scientific Coding & OOD Generalization: Yields significant accuracy improvements on countdown puzzles and scientific programming tasks, proving that cumulative value tracking generalizes robustly beyond training distributions;
• Credit Attribution Fidelity: Trajectory probe analysis confirms TPPO's critic reliably pinpoints the exact pivotal tokens where mathematical derivation pivots from flawed to sound logic.

Getting Started for Developers

TPPO is fully open-sourced on GitHub at https://github.com/LINs-lab/TPPO:

• Quickstart Repository: Clone via git clone https://github.com/LINs-lab/TPPO to inspect the custom critic loss definitions and distributed PPO training scripts;
• Seamless Critic Replacement: Teams running RLVR pipelines can swap out conventional token-level value prediction heads with TPPO's cumulative parameterization without modifying policy architectures or inference loops;
• Cost-Efficient RL Scaling: Ideal for code synthesis and formal verification pipelines that rely strictly on compiler/sandbox exit codes, unlocking dense token-level credit assignment without expensive step-by-step annotation.