In reinforcement learning with verifiable rewards (RLVR), rollout trajectories are generated by high-throughput inference engines while backward gradients are computed by training engines, introducing subtle floating-point probability discrepancies that trigger unbounded variance under exact importance sampling. Researchers identified the invariance of logit displacement distributions and introduced Calibrated Importance Sampling (CIS), featuring confidence-aware truncation that replaces unbounded second moments with constant bounds, establishing top scores across five mathematical benchmarks.

Key Takeaways

  • ✓Exposing Engine Probability Discrepancies: Fast inference engines (vLLM) and training backends (Megatron) assign diverging probabilities to identical tokens due to kernel optimizations, inducing severe variance under standard importance sampling.
  • ✓Logit-Displacement Invariance: Formulates that per-logit perturbations before softmax remain approximately invariant to token confidence, enabling a principled confidence-aware importance weight cap.
  • ✓Theoretical Variance Bounds & Benchmark Leadership: Replaces unbounded second moments with strict constant bounds, outperforming existing baselines across three MoE backbones on five rigorous mathematical reasoning benchmarks.
🧭

Finished reading? Explore benchmark rankings & pricing

Real-world SWE-bench scores & $20/mo vs API cost break-even calculator

🏆 Leaderboard→💰 Compare Plans↗
🔗

Project Links & Resources

Direct Access
Direct access to official project resources and documentation
🌐
ArXiv Preprint (2609.32444)
arxiv.org/abs/2609.32444
↗
📖
Hugging Face Daily Papers (CIS RLVR)
huggingface.co/papers/2609.32444
↗
🔬

In-Depth Technical Analysis

核心背景与行业痛点 / Background & Pain Points Scalable reinforcement learning with verifiable rewards (RLVR) relies on disaggregated architectures: decoupled high-throughput inference engines (vLLM/SGLang) sample rollout trajectories at scale, while distributed training engines (Megatron-LM/PyTorch) compute backward policy gradients. Because kernel optimizations, numerical precisions, and operator fusions differ subtly between engines, they assign slightly divergent probabilities to identical tokens. Under standard importance sampling, these discrepancies introduce exploding second moments in policy ratios, destabilizing policy gradient convergence. ### 架构亮点与底层机制 / Architectural Highlights Researchers introduced Calibrated Importance Sampling (CIS): 1. Logit-Displacement Invariance: Identifies an empirical property where per-logit additive perturbations $\varepsilon_t$ before softmax remain invariant to token confidence levels; 2. Confidence-Aware Ratio Truncation: Maps a constant logit-displacement threshold back to dynamic importance-weight caps that naturally tighten as token confidence increases while preserving exploratory weights on uncertain tokens; 3. Bounded Second Moments: Mathematically replaces the unbounded variance of exact importance sampling with a constant upper bound, ensuring numerical stability throughout extended reasoning rollouts. ### 权威 Benchmark 与实测跑分对比 / Benchmark & Evaluation Rigorous testing across three Mixture-of-Experts (MoE) backbones and five mathematical reasoning benchmarks confirmed clear superiority: - Benchmark Leadership: CIS attained the highest five-benchmark average accuracy across all evaluated MoE checkpoints compared to truncated importance sampling and PPO baselines; - Reduced Exploration Bias: Preserves unbiased gradient signals on low-confidence exploratory tokens, whereas heuristic clipping over-penalizes novel reasoning steps; - Zero Overhead: Incurs negligible computational latency, requiring only vectorized tensor clamping operations. ### 开发者实战落地与开箱指南 / Developer Practical Guide - Formal Proofs: Documented in arXiv preprint 2609.32444; - RLVR Training Pipeline Adoption: Recommended as a drop-in replacement for naive probability ratio clipping in open-source RL frameworks (e.g., verl, TRL, OpenRLHF); - Engineering Impact: Eliminates numerical instability and NaN loss spikes in disaggregated vLLM-plus-Megatron training clusters.