High-throughput LLM inference engine SGLang released milestone v0.5.20, consolidating 713 merged PRs from 237 contributors. The update introduces native return_sampling_mask for reinforcement learning (RL) rollouts, returning exact token support and log-probabilities per decode step to eliminate post-hoc Top-k/Top-p reconstruction during PPO/GRPO/DPO training. Paired with overlap scheduling, decode throughput increases by 17% at batch size 1 and 52% at batch size 64 on Qwen3-8B. The release also implements a unified radix tree with branching-point caching for sliding window attention (SWA) and brings first-day support for GLM-5.3-Flash, Tencent Hy4-Preview, and Qwen3.8-Flash-Next.
Key Takeaways
- ✓Massive release: 713 merged PRs from 237 contributors covering RL rollouts, overlap scheduling, and multi-model support
- ✓Zero-reconstruction RL rollouts: return_sampling_mask returns token candidates and log-probs to streamline GRPO/PPO training
- ✓52% decode throughput gain: overlap scheduling boosts Qwen3-8B decoding by 17% at batch 1 and 52% at batch 64
- ✓Unified Radix Tree for SWA: enables branch-point caching across sliding-window and hybrid-attention model architectures
- ✓Day-one frontier model support: out-of-the-box support for GLM-5.3-Flash, Tencent Hy4-Preview, Qwen3.8-Flash-Next, and SenseNova MoT
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Core Background & Industry Pain Points
In the era of reinforcement learning post-training (PPO, GRPO, DPO) and test-time compute, inference engines must synchronize tightly with training frameworks. Standard engines only output final sampled tokens during rollouts, forcing trainers to reconstruct Top-k/Top-p truncated distributions at high computational cost. Furthermore, as hybrid-attention and sliding window attention (SWA) architectures proliferate, legacy Radix Tree caches fail across window boundaries, causing severe Time-to-First-Token (TTFT) latency penalties and VRAM fragmentation.
Architecture Highlights & Internals
SGLang v0.5.20 implements core infrastructural upgrades:
- Sampling Masks for RL Rollouts: The
return_sampling_maskparameter instructs decoders to output exact token candidate support sets and log-probabilities at each step, enabling lossless trajectory replay without probability reconstruction; - Overlap Pipeline Scheduling: Masks run under asynchronous overlap scheduling with
--sampling-mask-max-tokens(default 4096), eliminating host-device synchronization stalls; - Unified Radix Tree for SWA: Re-architects cache branching for sliding window models, enabling shared prefix hits comparable to standard causal attention;
- Broad Model Compatibility: Delivers first-day support for GLM-5.3-Flash, Tencent Hy4-Preview, Qwen3.8-Flash-Next, FastH3, and SenseNova MoT.
Authoritative Benchmarks & Measured Scores
- Decoding Throughput: On NVIDIA H100 with Qwen3-8B, overlap scheduling boosts decode throughput by 17% at batch size 1 and 52% at batch size 64;
- RL Training Speedup: Integrated with Ray/DeepSpeed GRPO workflows, removing reverse probability reconstruction shortens single-round training latency by 28.4%;
- SWA Prefix Hit Rate: Over 32K context evaluations, prefix hit rate jumps from 21.3% to 78.9%, slumping TTFT by 63%.
Developer Hands-on Guide
Upgrade via pip install --upgrade sglang and launch with --return-sampling-mask --sampling-mask-max-tokens 4096. Release notes: https://github.com/sgl-project/sglang/releases/tag/v0.5.20.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.