RSIPO: Recursive Self-Improving Policy Optimization for Agentic RL Achieves 98.6% Success on ALFWorld via Skill-Policy Co-Evolution

Background and the Problem

Credit assignment across long-horizon interactive episodes remains a fundamental challenge in agentic reinforcement learning (RL). Integrating reinforcement learning with verifiable rewards (RLVR) with on-policy self-distillation (OPSD) has shown considerable promise, using a privileged-skill teacher to provide dense token-level logit supervision over student rollouts. However, two systemic vulnerabilities persist: first, agentic skills are universally treated as static, handcrafted artifacts frozen prior to training, preventing skills from evolving as the policy explores; second, privileged guidance can be inconsistent or noisy, causing uncalibrated student imitation to degrade policy stability.

Architecture and How It Works

To resolve these structural bottlenecks, researchers from CUHK and Tencent introduced Recursive Self-Improving Policy Optimization (RSIPO), a self-evolving agentic RL architecture:

1. Skill-Policy Co-Evolution: Empowers the target policy to recursively extract procedural skills directly from successful execution rollouts, actively optimizing its own skill synthesis capabilities via trajectory replay;
2. Empirical Bayes Shrinkage for Trust Calibration: Implements an adaptive calibration layer that evaluates teacher supervisory variance, dynamically shrinking student reliance on teacher logits when privileged signals demonstrate uncertainty or high entropy;
3. Self-Sustaining On-Policy Architecture: Operates without external proprietary teacher networks, driving autonomous agentic capability growth entirely from internal trajectory reflections.

Benchmarks and Measured Results

Rigorous evaluations across embodied environments and search-based tool tasks establish new state-of-the-art results:

• 98.6% Pinnacle on ALFWorld: With open-weight Qwen2.5-7B as the base model, RSIPO achieves an unprecedented 98.6% average task resolution rate on ALFWorld;
• 17.4-Point Outperformance Over GRPO: Outstrips standard GRPO baselines by 17.4 percentage points and beats the closest competitive on-policy self-distillation variant by 2.5%;
• Robust Multi-Step Tool Coordination: Demonstrates superior navigation efficiency and diminished trajectory loops in open-domain search environments, confirming that co-evolving skills shield agents from policy plateauing.

Getting Started for Developers

RSIPO provides actionable post-training blueprints for developers building autonomous embodied and tool-calling agents:

• Algorithmic Review: Study arXiv:2610.15772 for mathematical derivations of the empirical Bayes trust shrinkage and skill-replay loss terms;
• Dynamic Enterprise Skill Registries: Incorporate self-reflective skill extraction into internal agent frameworks to allow agents to catalog and index their own procedural workflows after successful execution;
• Noise-Resilient Knowledge Distillation: Implement Bayesian trust attenuation whenever distilling from privileged or larger teacher models to prevent hallucinated teacher tokens from corrupting student gradients.