Anthropic's Frontier Red Team published a research post on November 2 and open-sourced FlashProve under Apache 2.0. A model provider commits to hidden weights once, then attaches a zero-knowledge proof to each reply, so a user without the weights can check on an ordinary CPU that the reply really came from that model. That targets routing services that quietly swap in cheaper models and tampered token streams. Anthropic reports about 110 tokens/s on one H100 for Llama 4 Maverick (400B MoE, 4-bit), proofs of about 19 MB per 1,024-token request, and roughly 70-95x the GPU time of vLLM inference. It is an unmaintained research prototype limited to 1,024-token contexts and greedy decoding of an integer version of the model. Anthropic engineer Steve Weis shared the release on X.
Zoom
Key Takeaways
Anthropic-reported: on Llama 4 Maverick (400B MoE, 4-bit) one H100 proves about 110 tokens/s (prefill, 24 requests per batch), with proofs of about 19 MB per 1,024-token request.
It is costly: roughly 70-95x the GPU time of vLLM inference; the performance page lists about 270 GB of host memory for serving and about 39 minutes to commit to a set of weights.
Verification needs no GPU: the C++ verifier checks a proof in about 21 seconds on one thread, and a slower Lean verifier plus Lean 4 proofs of the security theorem are included.
Against prior work: VerInf proved the same model at about 0.02 tokens/s; the commercial Attestable reached 53-77 tokens/s on Gemma 4 31B but has no open-source release.
Limits: v0.16.0 is an unmaintained research prototype with 1,024-token contexts, greedy decoding of an integer model only, and no architecture hiding.
On Oct 27 OpenBMB (ModelBest) open-sourced the MiniCPM-OCR suite with two models: MiniCPM-OCR 1.0, a 1.3B-parameter document parser that scores 93.68 on Real5-OmniDocBench and 54.7 on OCRBench v2 (vendor-reported), and MiniCPM-OCR Jev, a 2B OCR quality judge that scores any parser's output region by region from 0 to 1 and flags content that needs human review. Code and weights are Apache 2.0, with day-one support for vLLM, SGLang, llama.cpp, Ollama and multiple chip platforms.
Zoom
Key Takeaways
MiniCPM-OCR 1.0 has just 1.3B parameters and scores 93.68 on Real5-OmniDocBench and 54.7 on OCRBench v2 (vendor-reported, both claimed SOTA)
The 2B MiniCPM-OCR Jev judge reaches 93.6 AUROC on JevOCRBench, above JEV-35B-A3B at 90.1 and close to Qwen3.5-397B-A17B at 93.8 (vendor-reported)
Jev is parser-agnostic: AUROC stays between 91.9 and 93.4 on outputs from MiniCPM-OCR, GLM-OCR, PaddleOCR-VL-1.6 and Xiaomi-OCR-0
Throughput is about 1.49x PaddleOCR-VL-1.6 and 3.62x MinerU2.5 Pro, reaching up to about 128 pages per minute with SDK concurrency (vendor-reported)
Code and weights on GitHub are all Apache 2.0, with GGUF/AWQ/GPTQ quantized builds and a SKILL.md for coding agents
On Oct 26, Google's official @googlegemma account said a set of vLLM optimizations built with Gemini 4 Argon lifted gemma-4-E2B-it throughput on a single A100 from about 37,900 to 108,000+ tokens/sec, roughly 2.85x, with 100% token parity. The PRs were filed by Luciano Martins of Google DeepMind, and most are still under review.
Zoom
Key Takeaways
Google says gemma-4-E2B-it throughput on an A100 rose about 2.85x (about 37.9K to 108K+ tokens/sec) with 100% token parity
The PRs come from Luciano Martins of Google DeepMind and cover fused Triton kernels, MTP speculative decoding and KV-sharing fast prefill
The 2.85x figure is Google's claim for the full set; PR #60674 alone self-reports up to 1.90x prefill and about 1.06x decode
As of Oct 27, only 3 of the 10 PRs in the linked list are merged; the rest are still in review
It is a public example of using a frontier coding model for inference-kernel performance work, still awaiting reproduction by the vLLM community
Google Cloud officially open-sourced TorchTPU, enabling native, zero-compilation-barrier PyTorch execution directly on Google Cloud TPU hardware (TPU v5p and TPU v6e). Overcoming the legacy friction of PyTorch/XLA's rigid graph compilation and dynamic-shape stalls, TorchTPU supports native PyTorch eager execution and custom Triton/C++ kernels. Leading high-throughput serving engines vLLM and SGLang announced immediate Day-0 support. Concurrently, Google Cloud introduced OpenAI models and Codex into its Vertex Model Garden, becoming the first hyperscaler to offer all three top frontier closed models.
Zoom
Key Takeaways
Google Cloud open-sources TorchTPU, delivering native, compilation-free PyTorch execution on TPU v5p and TPU v6e hardware
The vLLM project team and Ant Group officially open-sourced Humming (humming-kernels), a specialized high-performance kernel library designed for serving quantized MoE and dense LLMs at massive scale. Humming natively supports WnA16, W8A8, W4A8, and W4A4 across INT1–INT8 and FP4–FP8 formats, spanning NVIDIA architectures from Turing through Blackwell as well as T-Head Zhenwu 810E silicon. In production benchmarks, Humming kernels outpace cuBLAS by 3–4x at small batch sizes and surpass Marlin by 2x on H200 clusters. Serving DeepSeek-V4.1-Flash across 8x H20 GPUs, Humming drives a +43.3% throughput gain under 64 concurrent requests while reducing time-to-first-token (TTFT) by 32%, pre-integrated directly into vLLM v0.29.0+.
Zoom
Key Takeaways
vLLM and Ant Group open-sourced Humming (humming-kernels), integrated out-of-the-box into vLLM v0.29.0+
Covers WnA16, W8A8, W4A8, and W4A4, compiling JIT kernels in just 1.82 seconds versus 23.35 seconds for CUTLASS
Serving DeepSeek-V4.1-Flash on 8x H20 GPUs yields a +43.3% throughput boost at 64 concurrency (442 to 634 tokens/sec)
Cuts time-to-first-token (TTFT) by 25% to 32% while delivering up to 5x kernel speedups on Blackwell B300 systems
On Oct 21 the vLLM community and Ant Group open-sourced Humming (Apache 2.0), a quantized-inference library that JIT-compiles GEMM kernels on demand. It supports every NVIDIA architecture from Turing on plus T-Head's Zhenwu 810E, and covers weight and activation combinations from INT1-INT8 and FP4-FP8 for dense and MoE workloads. In vLLM's microbenchmarks it runs 2-5x faster than vLLM Marlin, FlashInfer or DeepGEMM in some cases; on 8 H20 GPUs, switching DeepSeek-V4.1-Flash MoE from Marlin W4A16 to Humming W4A8 raised throughput 14-43%. vLLM 0.29.0 ships it built in.
Zoom
Key Takeaways
One JIT kernel framework instead of a hand-written kernel per format: WNA16, W8A8, W4A8, W4A4, INT1-INT8 and FP4-FP8 weights, dense and MoE
Hardware: every NVIDIA GPU from Turing (SM75) on, including Blackwell SM100/103/120/121, plus T-Head Zhenwu 810E
End to end: DeepSeek-V4.1-Flash on 8x H20 at TP=8, Humming W4A8 beats Marlin W4A16 by 14-43% in throughput and cuts mean TTFT 25-32%; at concurrency 64 output goes from 442 to 634 tok/s
Microbenchmarks run by the vLLM community: W4A8-MXFP4 on H200 is about 4.5x DeepGEMM W8A8, and the B300 MoE Grouped path is about 5x Marlin; one W4A16 kernel compiles in 1.82 s versus 23.35 s for CUTLASS
Use: pip install humming-kernels (about a 500 KB wheel); built into vLLM 0.29.0+, enable with --linear-backend humming --moe-backend humming
Hours after DeepSeek open-sourced DeepInfer, the vLLM project team released official benchmark parity data serving DeepSeek-V4.1-Flash. Utilizing DeepSeek's identical decoding setup (131K context window, nextn=3 speculative decoding), vLLM closed its throughput performance delta to within 6% of DeepInfer across batch sizes 8 through 256 on NVIDIA GB200 systems. On GB300 hardware, vLLM attained up to 1,000,000 tokens/second per GPU and 826 tokens/second per user, generating $38.96 in hourly GPU revenue at DeepSeek off-peak rates. vLLM simultaneously shipped day-0 runtime support for FastH3-Omni and Spark-X2.6-4B.
Zoom
Key Takeaways
vLLM published direct benchmark comparisons, demonstrating that its serving throughput on NVIDIA GB200 is within 6% of DeepInfer
Achieves unprecedented scaling on NVIDIA GB300: up to 1,000,000 tokens/second per GPU and 826 tokens/second per active user stream
Generates up to $38.96 per GPU-hour in revenue when mapped to DeepSeek's production off-peak API pricing schedule
Delivers immediate Day-0 support for the FastH3-Omni video model and the 1M-token Apache 2.0 model Spark-X2.6-4B
On Oct 20, inference company Eagle released EAGLE-4, the fourth generation of the EAGLE speculative-decoding family. One drafter resizes its speculative tree for the batch size, hardware and workload while keeping the target model's output distribution. Eagle reports 961.6 tokens/s for one request on Qwen3.8-27B with 4x B300. The code ships as eagle-harness v0.1.0 with plugins for SGLang, vLLM and an experimental TokenSpeed backend. Many reposts call it open source, but the official license is source-available, not OSI open source.
Zoom
Key Takeaways
Vendor figure: 961.6 tokens/s on Qwen3.8-27B with TokenSpeed, BF16, 4x B300, TP4, one request, T=0 (paper Figure 1(a))
Repo measurements: vLLM on 4x B300 goes from 241.6 to 722.8 tok/s at concurrency 1 (2.99x) and 1.70x at concurrency 8; SGLang gets 1.91x at concurrency 8 and 1.35x at 64
AMD numbers too: SGLang on one MI355X gets 2.07x at concurrency 8 and 1.56x at 32 (ROCm 7.2)
Design: sliding-window draft layer, low-rank candidate head and append-only draft cache; the harness calibrates verify latency offline and grows the tree while gain per cost stays high
License: Eagle 4 Source-Available; free for noncommercial use, commercial use only under $20M trailing 12-month revenue and outside competing offerings
At PyTorch Conference 2026 on Oct 20, Google open-sourced TorchTPU on GitHub, co-developed with Meta's PyTorch team, making TPUs a native PyTorch device alongside GPUs and CPUs: eager mode and torch.compile, about 450 ATen ops, and torch.distributed collectives. Google also released vLLM-TorchTPU and SGLang-TorchTPU engines, integrations with TorchTitan, Ray, VeRL and Hugging Face Transformers v5 (device_map="tpu"), and a preview TPU agent suite for coding assistants such as Claude Code and Antigravity. AWS shipped the comparable TorchNeuron a day earlier.
Zoom
Key Takeaways
Porting (official PyTorch blog): existing code runs by changing model.to("cuda") to model.to("tpu") and the distributed backend to tpu_dist; the team notes peak TPU performance still needs TPU-aware shapes and sharding
Coverage (vendor-reported): 98% of about 4,917 popular Hugging Face models (4,821) run without crashing, and 97% of 657 architecture representatives (638) match GPU numerics
Training scale (vendor-reported): Qwen3 MoE 235B on TorchTitan scales from 1,024 to 8,192 TPU v7 chips with MFU roughly flat at 27-29%; on 64 TPU v7 chips it reaches 1,828 TPS versus 1,937 TPS for JAX/MaxText
Inference cost (Google citing SemiAnalysis InferenceX): TPU v7 with vLLM-TorchTPU hits 2.01 cents per million tokens on 8K/1K, down from 3.6 cents at its September debut
Four repos open-sourced: torch_tpu, vllm-torchtpu, sglang-torchtpu and a TPU TorchTitan; the TPU agent suite is in preview for TPU v6e and TPU7x
The Ornith AI team officially released Ornith-1.5-122B under the permissive MIT license, a 122-billion parameter mixture-of-experts (MoE) reasoning model optimized to execute on a single GPU node (such as a single DGX). The model is immediately available across vLLM (0.19.1+), SGLang (0.5.9+), Hugging Face, and Ollama (ornith-1.5:122b) with native FP8, GGUF, and MLX quantization. Across independent agentic evaluations in the OpenHands harness, Ornith-1.5-122B registered 84.5 on SWE-bench Verified, 76.9 on SWE-bench Pro (surpassing Qwen3.8-Flash-Next's 43.9 by 33 points), 81.2 on TerminalBench 2.1, and 74.6 on MCP-Atlas.
Zoom
Key Takeaways
Ornith-1.5-122B released under the permissive MIT license, offering a fully open, frontier-grade 122B MoE coding reasoning model
Engineered to fit inside a single GPU node (single DGX), with immediate runtime support on vLLM, SGLang, and Ollama (ornith-1.5:122b)
Achieves 76.9 on SWE-bench Pro, drastically outperforming the comparable open-weight baseline Qwen3.8-Flash-Next (43.9)
Delivers balanced agentic performance with 81.2 on TerminalBench 2.1, 80.0 on CyberGym, and 74.6 on MCP-Atlas
DeepSeek officially open-sourced DeepInfer under Apache 2.0, its production-grade inference engine serving DeepSeek-V4.1-Flash at scale. Originating from vLLM with architectural refinements from SGLang and TensorRT-LLM, DeepInfer natively targets NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It integrates specialized kernels including FlashMLA, DeepGEMM, DeepEP, and TileKernels. The engine features DSpark confidence-scheduled speculative decoding with 5-token drafts, SWA bounded replay over a 140-token sliding window, and balanced MoE expert parallelism. DeepSeek simultaneously open-sourced deepseek-recipe, delivering an end-to-end open ecosystem across models, kernels, and inference runtimes.
Zoom
Key Takeaways
DeepSeek officially open-sourced DeepInfer under Apache 2.0, releasing the production codebase powering DeepSeek-V4.1-Flash at massive scale
Natively optimized for both NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs with deterministic, batch-independent output guarantees
Integrates custom FlashMLA, DeepGEMM, DeepEP, and TileKernels, alongside DSpark 5-token speculative decoding and 140-token SWA replay
Simultaneously released deepseek-recipe, aligning open-source local deployments perfectly with DeepSeek's official production API tokenization
At PyTorch Conference North America on Oct 20, the CNCF Sandbox project llm-d released version 1.0. It turns inference engines such as vLLM and SGLang into one coordinated serving pool, and 1.0 adds multi-tenant priority and fairness scheduling (flow control), router high availability with zero-downtime rolling upgrades (DisaggregatedSet), a v1 configuration API, and agent-aware session routing with Responses API state. Founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, the project has 500+ contributors, and JetBrains, DigitalOcean and Mistral already run it in production.
Zoom
Key Takeaways
Scale (Red Hat press release, vendor-reported): GLM 5.2, roughly 753B parameters, on NVIDIA GPUs peaked at 6.6M output tokens per minute and served up to 3,000 concurrent coding agents with zero preemptions, 5-10x cheaper per token than equivalent commercial API pricing at cloud rental rates
Cache reuse: IBM says the prefix-aware router captured 97% of available cache reuse on an agentic coding workload; an IBM Research quantum error correction project saw ~98% prefix hits, ~$0.11 per million tokens, and 11.5x lower cost than an all-proprietary setup
Google Cloud figures: on Ironwood TPU, prefix-aware routing plus host memory offload cut median time-to-first-token by more than 85% and raised throughput 2.4x on an agentic coding workload; the Gemini platform doubled its prefix cache hit rate from 35% to 70%
New in 1.0: flow-control multi-tenant QoS, router hot-standby HA, DisaggregatedSet for coordinated prefill/decode rollouts, a v1 config API, and previous_response_id support in the Responses API
Ecosystem: 500+ contributors from 25+ organizations; JetBrains' free Junie Lite, DigitalOcean Serverless Inference and Mistral's tiered commercial traffic run on it; the GitHub v1.0.0 notes admit not every tracked goal shipped
The vLLM Project officially announced that PhysisServe's core physical-world serving techniques are being integrated into vLLM-Omni. Addressing the severe memory bottlenecks and latency spikes of multi-step video diffusion and physical AI generation, PhysisServe introduces cross-denoising-step pipeline parallelism across Transformer blocks, time-shared batching, and virtual-rank joint scheduling. Benchmarked on MiniMax H3 33B across 200 GPUs, it generates a chunk every 4.1 ms (~1,000 FPS without VAE decoding), while cutting single-GPU latency for Wan2.2 5B from 173 ms down to 98 ms.
Zoom
Key Takeaways
The vLLM project announced the upstream integration of PhysisServe's serving architecture into vLLM-Omni for physical AI and video models
Introduces cross-timestep block pipelining and time-shared batching to eliminate GPU communication bubbles in multi-step diffusion
Generates a video chunk every 4.1 ms on MiniMax H3 33B across 200 GPUs (~1,000 FPS), cutting Wan2.2 5B single-card latency to 98 ms
Expands 24-GPU serving concurrency by 1.7x to 3.3x, with full open-source code merging into the vLLM-Omni mainline
On October 19 Ant Ling (inclusionAI) open-sourced five Ming-3.0 models under MIT, spanning speech synthesis (tts-0.5B; tts-tiny, 7.9B total/1.3B active), audio understanding (audio-flash, 126B/6.5B active; audio-tiny, 9.3B/2.7B active) and speech recognition (asr-tiny, 8.6B/2B active). Per the vendor, tts-0.5B ranks #1 among open-weight models on Artificial Analysis Controlled Voice at 1069 Elo and ties #1 overall on pronunciation robustness at 93.2%; asr-tiny averages 4.41% error across four ASR categories. Weights are on Hugging Face and ModelScope, with day-0 SGLang and vLLM support.
Zoom
Key Takeaways
Lineup: five models in three families, two TTS, two audio-understanding and one ASR, all MIT-licensed.
TTS results: tts-0.5B scores 1069 Elo on AA Controlled Voice, #1 among open weights; 1188 Elo and #3 on Provider Voice; ties #1 overall on pronunciation robustness at 93.2% (vendor figures).
TTS features: 7 languages plus Sichuanese and Cantonese, 14 inline paralinguistic tags (laughter, sighs, coughs and more), 44.1 kHz output, built on Qwen2.5-0.5B.
Audio understanding: audio-flash has 126B total and 6.5B active parameters, and handles long-recording summaries, spoken Q&A, audio reasoning and tool-call generation.
ASR: asr-tiny accepts hotwords and dialogue context, is evaluated on 15 dialect test sets, and averages 4.41% error across four categories (vendor figures).
MiniMax confirmed that Space Bunny, the anonymous model that led usage rankings on OpenRouter and elsewhere, was an early version of M3.1 Flash, and launched it on its API on October 19. Pay-as-you-go pricing is $0.15 input, $0.60 output and $0.015 cache read per million tokens, with a 1M context, text, image and video input, and an effort setting for thinking depth. Per MiniMax, it costs one third of M3 and runs 3x faster, scoring 73.7% on DeepSWE and 62.2% on Automation-Bench; the weights went live on Hugging Face on schedule on October 23 (MiniMax Community license, MXFP8), with day-0 support from Together, Inferact (vLLM) and LightSeek TokenSpeed.
Zoom
Key Takeaways
Official pay-as-you-go pricing: $0.15 input / $0.60 output / $0.015 cache read per million tokens; per token that is half of M3 ($0.30/$1.20), and MiniMax does not explain how it gets one third of the cost.
1M context, text/image/video input, five effort levels from low to max, default max, thinking cannot be turned off.
Weights opened October 23 at Hugging Face MiniMaxAI/MiniMax-M3.1-Flash: ungated, MiniMax Community license (not MIT/Apache), MXFP8 quantized, native multimodal MoE with about 428B total and 23B active parameters per token, plus a DSpark speculative-decoding draft head.
Migration: M2.5 and M2.7 closed to new users on October 19 and retire on November 18; M Plan and Token Plan requests now route to M3.1 Flash automatically.
Day-0 serving: Together AI runs it fully managed, MiniMax is partnering with Inferact to contribute production-inference improvements back to vLLM, and LightSeek's TokenSpeed added day-0 support.
