LMSYS published a technical deep dive on a megakernel written from scratch in CUDA C++ and PTX (about 8,000 lines) for Meta's open-weight Muse Spark Open. Each decoder layer runs as a single kernel launch that overlaps compute, memory loads and cross-GPU communication. On 8x GB300, single-request decoding goes from 517 tok/s on the SGLang baseline to 1,109 tok/s, about 2.1x, and each decode step drops from 12.9 ms to 5.9 ms. The code is open source as an SGLang plugin at facebookresearch/sgl-muse-mega (MIT). It only covers batch size 1 and is experimental, so it buys latency rather than cheaper tokens.
Zoom
Key Takeaways
Twice as fast: on 8x GB300 at TP8 with one GSM8K request, decoding goes from 517 to 1,109 tok/s, about 2.1x, with the same draft model and acceptance length
Far fewer launches: kernel launches per decode step drop from about 1,450 to 97, one per layer
Step time: 12.9 ms to 5.9 ms; verify goes from 10.9 to 5.3 ms and the draft from 1.16 ms to 0.44 ms (175 launches down to 8)
Speculative decoding: the DFlash V2 draft guesses 15 tokens per step; the model verifies 16 and accepts 6.6 on average, versus 8 verified and 6.3 accepted for Xiaomi MiMo-V2.5-Pro
Scope: batch 1 only, with other requests falling back to stock SGLang; eight top GPUs serve one user, so this sells latency, not lower cost
On November 12, vLLM published the third post in its DeepSeek-V4.1-Flash series, led by Inferact. On one Vera Rubin NVL72 rack running the SemiAnalysis AgentX workload, it measured 1.7M total tokens/s per GPU at a P90 of 387 tokens/s per user, 58x GB300 NVL72 at the same interactivity, and 2.52M tokens/s per GPU at peak throughput, 2.7x GB300. The 58x figure applies only at that high-interactivity point, and the Rubin numbers are preview results.
Zoom
Key Takeaways
Keep the two multiples apart (vLLM figures): 1.7M tokens/s per GPU at 387 tokens/s per user is 58x GB300; 2.52M tokens/s per GPU at peak throughput is 2.7x GB300
Single-user latency: a specialized TP8 decode path spanning two compute trays reaches 1,810 tokens/s at P90 with one concurrent session, and P50 rose from 1,117 to 1,844 tokens/s
Rewritten kernels: tuned for Rubin's two memory domains and SM107's new MMA instructions, DEP2 prefill steps are about 28% faster and DEP decode steps about 25% faster
Rack-scale memory pools: the Mooncake KV pool extends from DRAM to spare HBM and Engram tables are sharded across GPU HBM; at concurrency 4,800 requests served rise 68% and TTFT P50 falls from 29.8 s to 4.2 s
Nature of the data: Rubin results are preview numbers, GB300 comparisons come from an InferenceX PR, and some points are interpolated
On 2026-11-06 Reflection's official account released the Beam model technical report (a 156-page PDF), detailing how its 501.3B-total / 23.4B-active open-weight MoE was trained: 23.8T pretraining tokens on 6,144 GB300 GPUs in under four weeks, then a fully asynchronous RL run on about 10.5K GB300s for four weeks that produced over 100M rollouts. The report says FP8 and NVFP4 checkpoints keep 99.93% and 98.98% of median benchmark scores, and the 1.93B DFlash 2 drafter speeds single-request decoding 1.96–3.03x (vendor figures).
Zoom
Key Takeaways
Scale: 501.3B total / 23.4B active, 52 layers, 256 experts per layer with 8 active per token, 151,936-token vocabulary
Pretraining: 23.8T tokens on 6,144 GB300s in under four weeks, 92.3% goodput late in the run and 21% steady-state MFU
RL: about 10.5K GB300s for four weeks, 100M+ rollouts, 110K concurrent rollouts and 12-second weight updates; the report sees no plateau as RL compute grows
Serving: FP8 / NVFP4 keep 99.93% / 98.98% of median scores; DFlash 2 speeds single-request decoding 1.96-3.03x
Safety: 88.9% refuse/pivot on harmful requests and 81.5% compliance on harmless ones in in-house ISE; Limited in all four systemic-risk categories
On Oct 28 DiffuSpace, whose members come from the HKU NLP group behind Dream 7B, released its first model, Clover-35B-A3B-Preview: a mixture-of-experts diffusion language model initialized from Qwen3.6-35B-A3B, with 35B total and about 3B active parameters, a 128K context window and block-parallel denoising. Vendor-reported results include 73.6 on SWE-bench Verified, 51.7 on Terminal-Bench 2.1 and 83.8 on GPQA Diamond, with 680 tokens/s on a single GB300 in agent workloads. Weights are Apache 2.0 on Hugging Face, ModelScope and GitHub, with SGLang and llama.cpp support.
Zoom
Key Takeaways
A 35B-total, roughly 3B-active MoE diffusion model initialized from Qwen3.6-35B-A3B, with 128K context and a thinking mode
DiffuSpace says it tops all six agent benchmarks it compared: SWE-bench Verified 73.6, Terminal-Bench 2.1 51.7 and tau3 Overall 65.3, ahead of LLaDA2.2 flash, Mercury 2.5 and Qwen3.6-35B-A3B
Speed (vendor-reported): 680 tokens/s on one GB300 at BF16 and batch size 1 in agent workloads, and 515 tokens/s on one H200 versus 365 tokens/s for Qwen with DFlash2
An INT4 build runs on a 16GB RTX 5070 Ti at 101.8 tokens/s on MATH-500
Weights on Hugging Face are Apache 2.0, about 69GB in BF16, and SGLang is the recommended serving path
Hours after DeepSeek open-sourced DeepInfer, the vLLM project team released official benchmark parity data serving DeepSeek-V4.1-Flash. Utilizing DeepSeek's identical decoding setup (131K context window, nextn=3 speculative decoding), vLLM closed its throughput performance delta to within 6% of DeepInfer across batch sizes 8 through 256 on NVIDIA GB200 systems. On GB300 hardware, vLLM attained up to 1,000,000 tokens/second per GPU and 826 tokens/second per user, generating $38.96 in hourly GPU revenue at DeepSeek off-peak rates. vLLM simultaneously shipped day-0 runtime support for FastH3-Omni and Spark-X2.6-4B.
Zoom
Key Takeaways
vLLM published direct benchmark comparisons, demonstrating that its serving throughput on NVIDIA GB200 is within 6% of DeepInfer
Achieves unprecedented scaling on NVIDIA GB300: up to 1,000,000 tokens/second per GPU and 826 tokens/second per active user stream
Generates up to $38.96 per GPU-hour in revenue when mapped to DeepSeek's production off-peak API pricing schedule
Delivers immediate Day-0 support for the FastH3-Omni video model and the 1M-token Apache 2.0 model Spark-X2.6-4B
On Oct 20, NVIDIA's GEAR lab (Jim Fan's team) open-sourced Flora, a model suite that turns human and robot manipulation video into timestamped action captions, process rewards, outcome rewards and rubric scores. It includes Flora-Captioner 27B and Flora-Rewarder 27B (both on Qwen3.5-27B) and Flora-TempSeg 9B (a LoRA on Qwen3.5-9B), with weights on Hugging Face under an NVIDIA non-commercial license. NVIDIA says it is the in-house data engine used to train GR00T, labeling 200,000 hours of video a day on 512 GB300s, matching GPT-6 Astra on annotation quality at roughly 125x lower cost.
Zoom
Key Takeaways
Three models: TempSeg 9B segments actions, Captioner 27B writes one instruction per segment, and Rewarder 27B outputs dense process rewards and an episode success score
Cost (NVIDIA estimate): about $0.72 per hour of source video on eight GB300s at $10 per GPU-hour, versus about $88.49 for GPT-6 Astra at maximum effort, a 122-125x gap
Quality (NVIDIA 50-episode internal eval): 57.1% for Flora versus 53.7% for GPT-6 Astra; failure recall on DenseRewardBench is 53.3% while frontier API models mostly score under 1%
Scale: NVIDIA says Flora labels 200,000 hours of video a day on 512 GB300s to train GR00T-2
License: NVIDIA OneWay Noncommercial, research only, with commercial and production use explicitly unsupported
Meta's engineering blog on October 19 gave the first detailed look at Prometheus, the New Albany, Ohio supercluster that trained the Muse family, including Muse Wonder. Phase one is complete with hundreds of thousands of GPUs, and the full build targets about 1 million. Muse Wonder training reached more than 85% goodput in pretraining and more than 90% in reinforcement learning, with a mean time between interruptions of about 47K GPU hours, since raised above 54K by firmware fixes. It runs GB200 NVL72 and GB300 NVL72 platforms, with Vera Rubin NVL72 arriving soon.
Zoom
Key Takeaways
Scale: phase one trained the Muse family; it has hundreds of thousands of GPUs today, targets about 1 million, and runs across buildings as a single machine.
Reliability: Muse Wonder hit 85%+ pretraining goodput and 90%+ for RL; MTBI was about 47K GPU hours and is now above 54K.
Interruptions: a mid-2025 peak of nearly 1,000 events per 8K GPUs per day was cut by 99% over a year of engineering.
Sparing: each NVL72 rack runs a 64-GPU slice with 8 in-rack spares, plus about 2% spare racks; a 32,768-GPU job needs 523 racks or 37,656 GPUs, about 13% extra.
Network: each building fabric links up to 20,736 GPUs, and the aggregation network spans 2 km to over 100 km.
SemiAnalysis revealed the first Google TPUv7 benchmark submission for the 2.8T Kimi K3 MoE model, led end-to-end by Inferact (the creators of vLLM) and Google using the TorchTPU vLLM stack. At a hardware rental cost of just $1.21/chip-hour (vs $2.31 for GB300 and $1.86 for GB200), TPUv7 delivers ~2.2M tokens per $1 TCO in the ultra-high interactivity regime (~390 tok/s/user), outpacing NVIDIA GB200 NVL72, B300, and AMD MI355X on cost-performance.
Zoom
Key Takeaways
Inferact (creators of vLLM) and Google submitted their first preliminary TPUv7 benchmark for the 2.8T Kimi K3 MoE model
TPUv7 chip rental sits at $1.21/hour, significantly below GB300 ($2.31), B300 ($2.26), and GB200 ($1.86)
Delivers ~2.2M tokens per $1 TCO at ~390 tok/s/user, beating NVIDIA GB200 NVL72, B300, and AMD MI355X in the ultra-high interactivity regime
Built with fused TorchTPU vLLM megakernels and the TPU-Sync KVCache offload library for production PD disaggregation
vLLM and NVIDIA unveiled a unified CuTe DSL decode megakernel (~12,000 LoC authored by NVIDIA CentML's Benjamin Chislett) for Moonshot AI's frontier Kimi K3 model on NVIDIA GB300 NVL72. By fusing attention, MoE, and cross-GPU communication, the optimization cuts per-layer latency from 130 µs (23 kernels) to 69 µs (13 kernels), delivering 842 tokens/s at batch=1 with DSpark speculative decoding (1.73x faster than stock vLLM) and up to 1.8x higher P90 per-user interactivity (403.2–423 tok/s/user) on SemiAnalysis AgentX, with DeepSeek-V4.1-Flash (PR #61868) and GLM-5.3 support coming next.
Zoom
Key Takeaways
NVIDIA and vLLM introduce a 12,000-line CuTe DSL decode megakernel for the 2.8T-parameter Kimi K3, reducing per-layer execution from 23 kernels (130 µs) to 13 kernels (69 µs)
On 16x NVIDIA GB300 NVL72 GPUs, batch-1 decode hits 842 tok/s with DSpark speculative decoding (1.73x faster) and 256.4 tok/s without speculation (1.88x faster)
Delivers up to 1.8x higher P90 interactivity (403.2-423 tok/s/user) on SemiAnalysis AgentX, with DeepSeek-V4.1-Flash (vLLM PR #61868) and GLM-5.3 megakernels following next
A vLLM blog post by NVIDIA engineer Benjamin Chislett details an agent-assisted CuTe DSL decode path for Kimi K3 on NVIDIA GB300 that fuses attention, MoE and NVLink communication, cutting per-layer time from 130 µs across 23 kernels to 69 µs across 13. With DSpark speculative decoding on 16 GPUs it reaches 842 tok/s at batch 1, 1.73x stock vLLM; PR #61867 was merged on Oct 16.
Zoom
Key Takeaways
842 tok/s at batch 1 on 16 GB300 GPUs with DSpark (7 draft tokens, synthetic AL=6) vs 488 for stock vLLM, 1.73x (vendor figures)
Without speculation TP16 gains 1.62–1.92x (3,606 tok/s at batch 64); TP8 gains 1.31–1.58x
Per layer: 23 kernels and 130 µs down to 13 kernels and 69 µs
AgentX P90 interactivity 403.2 vs 223.7 tok/s/user (1.8x), though the baseline is an earlier 8-GPU run
The same fast-track approach now covers DeepSeek-V4.1-Flash (PR #61868, open); early Muse Spark Open results exceed 950 tok/s/user
On Oct 9 the vLLM team, with Inferact, Red Hat and NVIDIA, shipped early Vera Rubin NVL72 support: CUDA 13.4 nightly images already serve DeepSeek, Kimi, GLM and MiniMax. Vendor-reported results: up to 7.84x per-GPU throughput vs GB200 on SemiAnalysis AgentX (MiniMax M3, matched interactivity) and up to 3.7x vs GB300 NVL72 on the MLPerf Inference v6.1 VLM benchmark (Qwen3-VL-235B-A22B).
NVIDIA published the first on-silicon benchmark metrics for its Vera Rubin architecture on long-horizon agent workloads, demonstrating up to 30x throughput per megawatt and 35x lower token cost versus GB300 NVL72.
Key Takeaways
Inference benchmarks rebased from short chat to multi-step agent coding workloads with deep context expansion;
Benchmarked against DeepSeek V4 Pro on SemiAnalysis AgentX real coding traces as the reference workload;
Delivers up to 30x throughput per megawatt and 35x lower token cost compared to GB300 NVL72.
