AI Coding Live News

On Oct 26, Google's official @googlegemma account said a set of vLLM optimizations built with Gemini 4 Argon lifted gemma-4-E2B-it throughput on a single A100 from about 37,900 to 108,000+ tokens/sec, roughly 2.85x, with 100% token parity. The PRs were filed by Luciano Martins of Google DeepMind, and most are still under review.

Zoom

Google Cloud officially open-sourced TorchTPU, enabling native, zero-compilation-barrier PyTorch execution directly on Google Cloud TPU hardware (TPU v5p and TPU v6e). Overcoming the legacy friction of PyTorch/XLA's rigid graph compilation and dynamic-shape stalls, TorchTPU supports native PyTorch eager execution and custom Triton/C++ kernels. Leading high-throughput serving engines vLLM and SGLang announced immediate Day-0 support. Concurrently, Google Cloud introduced OpenAI models and Codex into its Vertex Model Garden, becoming the first hyperscaler to offer all three top frontier closed models.

Zoom

The vLLM project team and Ant Group officially open-sourced Humming (humming-kernels), a specialized high-performance kernel library designed for serving quantized MoE and dense LLMs at massive scale. Humming natively supports WnA16, W8A8, W4A8, and W4A4 across INT1–INT8 and FP4–FP8 formats, spanning NVIDIA architectures from Turing through Blackwell as well as T-Head Zhenwu 810E silicon. In production benchmarks, Humming kernels outpace cuBLAS by 3–4x at small batch sizes and surpass Marlin by 2x on H200 clusters. Serving DeepSeek-V4.1-Flash across 8x H20 GPUs, Humming drives a +43.3% throughput gain under 64 concurrent requests while reducing time-to-first-token (TTFT) by 32%, pre-integrated directly into vLLM v0.29.0+.

Zoom

On Oct 21 the vLLM community and Ant Group open-sourced Humming (Apache 2.0), a quantized-inference library that JIT-compiles GEMM kernels on demand. It supports every NVIDIA architecture from Turing on plus T-Head's Zhenwu 810E, and covers weight and activation combinations from INT1-INT8 and FP4-FP8 for dense and MoE workloads. In vLLM's microbenchmarks it runs 2-5x faster than vLLM Marlin, FlashInfer or DeepGEMM in some cases; on 8 H20 GPUs, switching DeepSeek-V4.1-Flash MoE from Marlin W4A16 to Humming W4A8 raised throughput 14-43%. vLLM 0.29.0 ships it built in.

Zoom

Hours after DeepSeek open-sourced DeepInfer, the vLLM project team released official benchmark parity data serving DeepSeek-V4.1-Flash. Utilizing DeepSeek's identical decoding setup (131K context window, nextn=3 speculative decoding), vLLM closed its throughput performance delta to within 6% of DeepInfer across batch sizes 8 through 256 on NVIDIA GB200 systems. On GB300 hardware, vLLM attained up to 1,000,000 tokens/second per GPU and 826 tokens/second per user, generating $38.96 in hourly GPU revenue at DeepSeek off-peak rates. vLLM simultaneously shipped day-0 runtime support for FastH3-Omni and Spark-X2.6-4B.

Zoom

Modal said in an October 19 blog post that it will add NVIDIA's Arm-based Vera CPU to power Modal Sandboxes, its first step beyond an all-x86 fleet. Using Agentless Agent Bench, a 50-task suite built from Terminal-Bench, Modal measured up to 2x the task throughput of a comparable x86 GNR machine under heavy concurrency, with per-task latency equal or lower (vendor figures). On an idle host running tasks serially, Vera's edge was small. The capacity targets background coding agents and reinforcement learning rollouts.

Zoom

The vLLM team announced two major milestones: shipping precompiled Python bytecode via uv in v0.32.0 release images cuts cold-start container launch by 14–18% (~18s on NVIDIA H100) with zero runtime recompiles. Additionally, vLLM integrated the Decision 3.0 (D3) structured reasoning family (0.6B to 27B) into the /v1/systemone endpoint, ranking #1 on the Jev Decision Index and delivering 1.8x req/s on a single AMD MI325X under 64 concurrent clients.

Zoom

The SGLang team released v0.5.22 featuring 950 PRs from 245 contributors: DeepSeek-V4.1 delivers up to 2.3x more throughput per GPU on NVIDIA H200 and 1.9x on AMD MI350X/MI355X via DSpark. The new InstantTensor loader cuts model loading time by 10x. B200 decodes Qwen3.8-Flash-Next 12% faster, while B300 speeds up GLM-5.3-Flash by 10% and yields 1.85x decode throughput for MiMo-V2.6-Pro. SGLang also adds native Apple Silicon PyTorch support and ComfyUI integration.

Zoom

Addressing KV cache bottlenecks in long-context LLM serving, a new production study on 8xB300 nodes serving GLM-5.3 (arXiv:2610.21500) reveals that unsharded remote KV caches inflate hit-heavy p99 TTFT to 83.32s, and resolves it via rank-sharded asynchronous LMCache loading—boosting GET bandwidth 6.3x to 10.76 GB/s and slashing p99 to 0.67s (+34% throughput). Meanwhile, Baidu PaddlePaddle open-sources Virtual Head Attention (VHA, arXiv:2610.17959), folding linear Q-Premix and Postmix into standard GQA-2 weights at inference to halve KV cache while beating MLA (+0.70%) and GQA-4 (+2.60%) on a 30B-A3B MoE and lifting 8K-128K RULER from 73.0 to 84.4.

Zoom

Following the release of DeepSeek-V4.1-Flash-1017, vLLM announced Day-0 serving optimization delivering 1.3B tokens per dollar TCO and 830K tok/s per GPU at 100 tok/s/user on SemiAnalysis AgentX (5x agentic throughput gain since Day 0), while OpenRouter launched live multi-provider routing starting at $0.10/M input and $0.50/M output tokens.

Zoom

As autonomous coding agents achieve surging scores on benchmarks like SWE-bench, the prevailing assumption has been that coding proficiency primarily correlates with lines of code modified when aided by terminal search tools. However, agents routinely fail on subtle modifications in real repositories. Microsoft Research introduces CABRA (arXiv:2610.10610, code: github.com/microsoft/CABRA), a synthetic evaluation blueprint designed to isolate code comprehension from simple editing throughput. CABRA synthesizes programs as Directed Acyclic Graphs (DAGs) and scales task difficulty systematically along four orthogonal axes parameterized by n: Function Traversal, Function Search, Runtime Resolution, and Instruction Following. The findings establish that code comprehension—rather than tool invocation or editing volume—is the true bottleneck of autonomous coding agents. Crucially, the volume of tool calls allocated to code comprehension operations (such as READ and ANALYZE) serves as a substantially stronger predictor of an agent's success on SWE-bench Verified than the number of lines of code edited, with code and datasets fully open-sourced.

Zoom

Token-level LLM routing—routing easy tokens to small models and switching to frontier models for critical reasoning tokens—presents an optimal trade-off between output quality and inference cost. However, current LLM serving engines (such as vLLM and TGI) are architected for single-model execution, suffering severe step desynchronization, batch admission latency, and extreme orchestration complexity when handling high-frequency token transfers across heterogeneous models. Researchers from Tsinghua University's NICS Lab present TokenRouter (arXiv:2610.12242, accepted at NeurIPS 2026, code: github.com/thu-nics/TokenRouter), an efficient and developer-friendly serving system engineered for token-level routed LLM inference. TokenRouter introduces a 'request-centric programming and model-centric execution' paradigm, enabling developers to write routing logic intuitively per request while the runtime handles asynchronous execution across subservers. Incorporating a Delayed-Batching Scheduler parameterized via discrete-time Markov chain modeling, a decoupled tri-loop architecture, and an instant handoff-resume mechanism, TokenRouter achieves 2.01x to 64.15x higher decoding throughput compared to existing serving platforms across diverse routing workloads, with complete code open-sourced.

Zoom

On Oct 9 the vLLM team, with Inferact, Red Hat and NVIDIA, shipped early Vera Rubin NVL72 support: CUDA 13.4 nightly images already serve DeepSeek, Kimi, GLM and MiniMax. Vendor-reported results: up to 7.84x per-GPU throughput vs GB200 on SemiAnalysis AgentX (MiniMax M3, matched interactivity) and up to 3.7x vs GB300 NVL72 on the MLPerf Inference v6.1 VLM benchmark (Qwen3-VL-235B-A22B).

Zoom