AI Coding Live News

On November 2 OpenAI published "Scaling Git at OpenAI", describing Pando, a Git-compatible backend it built for its monorepo. Per OpenAI, Pando cut median push latency by nearly 5x (about 80%), now delivers nearly a petabyte of Git data per day, and has served more than 57 TB in a single hour. It is written in Rust, its core was designed by one engineer and implemented by teams of Codex agents, and it went from kick-off to production in 10 weeks, serving production traffic since April. The trigger: Codex multiplied parallel branches and pushes, and the old system's reliability at times fell below 90%.

Zoom

On Oct 26 Tencent Hunyuan (Tencent Hy) open-sourced Hy-SimulMT, a real-time simultaneous interpreting model for bidirectional Chinese-English speech-to-speech translation. It translates while the speaker talks, outputs both translated text and speech, and can clone the speaker's voice in real time. Six checkpoints landed on Hugging Face (30B-A3B translation-only and source-plus-translation Interleave variants, each in BF16, FP8 and GGUF), with the model card listing Apache-2.0; GitHub ships the SimulInfer serving stack and SimulTrain training tools. Vendor-reported: about 3s StreamLAAL on RealSI long-form audio, the lowest among compared systems.

Zoom

Researchers from Tsinghua University and CUHK introduced a breakthrough paper titled 'Cache Engineering for Agents' (arXiv:2610.28577) alongside the open-source AKV serving engine (thu-nics/AKV). While multi-turn agents rely on traditional 'context engineering' (text trimming and summarization) that invalidates prefix KV-caches and incurs punishing re-prefilling overhead, cache engineering empowers agents to directly manipulate KV-cache states via 'drop' and 'repos' primitives. Coupled with AKV's drop-aware radix-tree engine, the framework slashes agent inference costs by 27.7%, accelerates P95 latency by 1.5x, and boosts benchmark solve scores by 1.7 points.

Zoom

LMSYS Org in collaboration with the NVIDIA Dynamo team officially unveiled Elastic Expert Parallelism (Elastic EP) with NIXL backend integration in SGLang (@lmsysorg). Addressing the critical vulnerability of large-scale MoE deployments (such as DeepSeek-V3/V4) where static EP causes entire cluster crashes upon single-node failures and prevents live capacity resizing, Elastic EP decouples rigid expert-to-GPU bindings. Leveraging NVIDIA NIXL device APIs and Mooncake orchestrators, SGLang enables live scaling of active MoE serving deployments (e.g. dynamically expanding from 20 to 24 GPUs) without draining traffic or restarting clusters, while delivering resilient fault tolerance.

Zoom

Google Cloud officially open-sourced TorchTPU, enabling native, zero-compilation-barrier PyTorch execution directly on Google Cloud TPU hardware (TPU v5p and TPU v6e). Overcoming the legacy friction of PyTorch/XLA's rigid graph compilation and dynamic-shape stalls, TorchTPU supports native PyTorch eager execution and custom Triton/C++ kernels. Leading high-throughput serving engines vLLM and SGLang announced immediate Day-0 support. Concurrently, Google Cloud introduced OpenAI models and Codex into its Vertex Model Garden, becoming the first hyperscaler to offer all three top frontier closed models.

Zoom

On Oct 21, PHP published a security advisory fixing two critical unauthenticated remote code execution flaws (CVE-2026-109618, CVE-2026-109619) affecting every PHP install serving web requests, with mod_php at highest risk. OpenAI disclosed the same day that Calif found a flaw using GPT-5.6 Sol under its Patch the Planet initiative, with Wiz independently finding and reporting both; OpenAI coordinated a two-month disclosure and committed $1B in subsidized Daybreak access for under-resourced defenders.

Zoom

The vLLM project team and Ant Group officially open-sourced Humming (humming-kernels), a specialized high-performance kernel library designed for serving quantized MoE and dense LLMs at massive scale. Humming natively supports WnA16, W8A8, W4A8, and W4A4 across INT1–INT8 and FP4–FP8 formats, spanning NVIDIA architectures from Turing through Blackwell as well as T-Head Zhenwu 810E silicon. In production benchmarks, Humming kernels outpace cuBLAS by 3–4x at small batch sizes and surpass Marlin by 2x on H200 clusters. Serving DeepSeek-V4.1-Flash across 8x H20 GPUs, Humming drives a +43.3% throughput gain under 64 concurrent requests while reducing time-to-first-token (TTFT) by 32%, pre-integrated directly into vLLM v0.29.0+.

Zoom

Hours after DeepSeek open-sourced DeepInfer, the vLLM project team released official benchmark parity data serving DeepSeek-V4.1-Flash. Utilizing DeepSeek's identical decoding setup (131K context window, nextn=3 speculative decoding), vLLM closed its throughput performance delta to within 6% of DeepInfer across batch sizes 8 through 256 on NVIDIA GB200 systems. On GB300 hardware, vLLM attained up to 1,000,000 tokens/second per GPU and 826 tokens/second per user, generating $38.96 in hourly GPU revenue at DeepSeek off-peak rates. vLLM simultaneously shipped day-0 runtime support for FastH3-Omni and Spark-X2.6-4B.

Zoom

DeepSeek officially open-sourced DeepInfer under Apache 2.0, its production-grade inference engine serving DeepSeek-V4.1-Flash at scale. Originating from vLLM with architectural refinements from SGLang and TensorRT-LLM, DeepInfer natively targets NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It integrates specialized kernels including FlashMLA, DeepGEMM, DeepEP, and TileKernels. The engine features DSpark confidence-scheduled speculative decoding with 5-token drafts, SWA bounded replay over a 140-token sliding window, and balanced MoE expert parallelism. DeepSeek simultaneously open-sourced deepseek-recipe, delivering an end-to-end open ecosystem across models, kernels, and inference runtimes.

Zoom

At PyTorch Conference North America on Oct 20, the CNCF Sandbox project llm-d released version 1.0. It turns inference engines such as vLLM and SGLang into one coordinated serving pool, and 1.0 adds multi-tenant priority and fairness scheduling (flow control), router high availability with zero-downtime rolling upgrades (DisaggregatedSet), a v1 configuration API, and agent-aware session routing with Responses API state. Founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, the project has 500+ contributors, and JetBrains, DigitalOcean and Mistral already run it in production.

Zoom

The vLLM Project officially announced that PhysisServe's core physical-world serving techniques are being integrated into vLLM-Omni. Addressing the severe memory bottlenecks and latency spikes of multi-step video diffusion and physical AI generation, PhysisServe introduces cross-denoising-step pipeline parallelism across Transformer blocks, time-shared batching, and virtual-rank joint scheduling. Benchmarked on MiniMax H3 33B across 200 GPUs, it generates a chunk every 4.1 ms (~1,000 FPS without VAE decoding), while cutting single-GPU latency for Wan2.2 5B from 173 ms down to 98 ms.

Zoom

Multimodal LLMs excel at holistic image captioning but struggle with pixel-level spatial grounding due to lack of spatial supervision. Xiaomi AI Lab unveiled Xiaomi-Lens-0 (arXiv:2610.23279), a 4B MLLM based on Qwen3.5 that natively predicts pixel masks. A lightweight decoder integrates patch embeddings, visual latents, and mask-token activations, driving RefCOCO comprehension (REC [email protected]: 94.6% / 92.1% / 92.5%) and segmentation (RES cIoU: 83.4% / 79.9% / 81.1%) past Florence-2 and LISA-7B while preserving a 69.7% MME-RW score.

Zoom

Traditional speculative decoding relies on static target hidden features that fail to supply varying contextual needs across draft positions. HKU researchers introduced ReaDy (arXiv:2610.20503). Designed specifically for Hybrid Linear Attention (HLA) architectures combining Transformers with recurrent linear attention states, ReaDy leverages compact recurrent states as dynamic context for drafting. On a single NVIDIA B200 GPU serving Qwen3.8-27B, ReaDy achieves an acceptance length of 4.53 and 302.78 TPS, accelerating inference by 2.4x over standard autoregressive decoding.

Zoom

The vLLM team announced two major milestones: shipping precompiled Python bytecode via uv in v0.32.0 release images cuts cold-start container launch by 14–18% (~18s on NVIDIA H100) with zero runtime recompiles. Additionally, vLLM integrated the Decision 3.0 (D3) structured reasoning family (0.6B to 27B) into the /v1/systemone endpoint, ranking #1 on the Jev Decision Index and delivering 1.8x req/s on a single AMD MI325X under 64 concurrent clients.

Zoom