PR-OPD: Privileged Representation On-Policy Self-Distillation Supercharges Agentic RL Without External Teachers

Background and the Problem

Autonomous LLM agents navigating complex interactive environments (such as web shopping, bash terminals, and embodied household tasks) are primarily trained using reinforcement learning (RL). However, standard RL frameworks suffer from sparse episodic rewards. While privileged self-distillation aims to bridge this by having an agent conditioned on privileged 'skills' teach its unprivileged self via token probabilities, empirical analysis reveals severe structural deficiencies:

1. Invisible Advantage: In WebShop, conditioning on a skill boosts task success from 42.2% to 56.2%, yet fewer than 25% of sampled token probabilities change;
2. Much to Align: Despite surface-level probability stability, the privileged skill fundamentally alters the internal hidden states of over 80% of response tokens, which linear probes readily trace back to the skill. Relying strictly on token-level KL divergence discards vital high-dimensional navigational representations.

Architecture and How It Works

To directly capture and distill these internal decision representations, the researchers developed PR-OPD (Privileged Representation On-policy Self-Distillation):

1. Teacher-Free Self-Loop: Operates without external proprietary teacher models. Following a GRPO warm-start, the agent synthesizes hindsight skills from its own historical execution traces;
2. Stop-Gradient Teacher Re-Read: The policy consumes the generated hindsight skill as an in-context prefix, serving as a frozen-gradient teacher that re-evaluates the identical generated actions;
3. Layer-Wise Projected Representation Alignment: Simultaneously optimizes the environmental reward objective while minimizing MSE between the student and teacher projected hidden states across all transformer layers;
4. Zero Inference Overhead: The distillation mechanisms and skill prompts are discarded after training, enabling the deployed agent to operate at native inference speed and memory footprint.

Benchmarks and Measured Results

Extensive evaluations across ALFWorld and WebShop over multiple base LLM backbones demonstrate definitive state-of-the-art results:

• Decisive WebShop Surges: PR-OPD achieves a massive 14.0-point gain in WebShop accuracy over standard GRPO baselines, jumping from 42.2% to over 56.2%;
• Embodied Planning Superiority: On ALFWorld, PR-OPD delivers a 4.7-point boost in task success rate across complex multi-step manipulation trajectories;
• Superior Convergence Efficiency: Representation alignment accelerates policy stabilization by 3.2x compared to conventional token-level KL matching, effectively suppressing policy oscillation during exploratory rollouts.

Getting Started for Developers

PR-OPD is fully open-sourced on GitHub at https://github.com/balibata/PR-OPD, delivering an out-of-the-box recipe for self-improving agents:

• Repository Setup: Clone via git clone https://github.com/balibata/PR-OPD to inspect the custom GRPO loss implementations and layer-wise projection layers;
• Enterprise Agent Fine-Tuning: For specialized workflow agents (e.g., automated CRM operations or complex tool-use pipelines), developers can replace expensive synthetic data generation with automated hindsight reflection distillations;
• Training Optimization: The projected hidden loss requires minimal GPU memory overhead and seamlessly integrates into existing distributed RL frameworks (like DeepSpeed or Megatron-LM).