Better Supervision Is Nearby: Tencent AI Lab Unveils Neighborhood On-Policy Self-Distillation (N-OPSD) for Math Reasoning

核心背景与行业痛点

On-policy self-distillation (OPSD) has emerged as a premier methodology for bolstering mathematical reasoning without external model weights. In OPSD, a privileged teacher—augmented with ground-truth reference solutions—supervises rollouts sampled by student policies. However, conventional OPSD relies exclusively on a single fixed parameter setting at each token step. Single teachers frequently hit saturation blind spots along extended deductive paths, while ensembling distinct external models fractures latent geometric compatibility and explodes compute costs.

架构亮点与底层机制

Tencent WeChat and Tencent AI Lab researchers introduce Neighborhood OPSD (N-OPSD), demonstrating that 'better supervision is nearby':

1. Local Parameter Perturbations: Uncovers that microscopic weight perturbations around the privileged teacher checkpoint yield diverse, reference-aligned corrections across distinct token steps.
2. Greedy Pruning of Frozen Expert Pools: Filters redundant perturbations offline to retain a compact pool of frozen experts that provide maximal reference token gains.
3. Two-Stage MaxPeak & Quantile Routing: Decouples anchor direction from support levels. MaxPeak identifies the anchor token, while quantile selection routes student states to the optimal teacher distribution among matching experts.
4. Clipped Forward-KL Alignment: Guides the student policy with the chosen expert's distribution, deploying a pure student checkpoint at inference time with zero runtime overhead.

权威 Benchmark 与实测跑分对比

Evaluated on demanding international mathematics competitions across AIME 2024, AIME 2025, and HMMT February 2025:

1. 2.75-Point Gain Across Three Benchmarks: N-OPSD improves Average@12 over standard OPSD by 2.75, 1.67, and 1.94 points on Qwen3-1.7B, 4B, and 8B respectively.
2. Superior Error Mitigation: Neighboring experts rescue 31.4% more student error branches along extended reasoning steps compared to static teachers.
3. Flawless Out-of-Distribution Transfer: The student policy successfully generalizes beyond reference trajectories, demonstrating authentic procedural mastery.

开发者实战落地与开箱指南

N-OPSD offers an ultra-efficient distillation upgrade for reasoning labs. Without training auxiliary models, researchers can generate local teacher perturbations and integrate MaxPeak routing into existing OPSD pipelines, extracting richer supervisory signals from native parameter neighborhoods to conquer frontier Olympiad benchmarks.