AI Coding Live News

On November 12, on-device model company Edge0 released open weights for Edge0-145B and Edge0-120B, along with its C++ inference engine edge0-infer and Edge0 Chat 0.2.0. The 145B model is built on Qwen3.8-Flash-Next as a latent sparse MoE, with about 10.12B active parameters per token, native 4-bit training and a 256K context. Edge0 reports 29GB peak memory locally and 98.3 tok/s agent decode on an RTX 5090 D 24GB. It reports 86.52 on Terminal-Bench 2.1, versus 78.2 for Claude Opus 4.6 (Max). All scores come from Edge0 internal evaluations.

Zoom

On November 12, LMSYS Org, the nonprofit behind the open-source inference engine SGLang, announced the SGLang Consortium in the opening keynote of the first SGLang Summit in San Francisco. Founding members NVIDIA, AMD, Intel, Google, Meta, Cloudflare, Modal and RadixArk will work with the SGLang maintainers on the roadmap, focusing on frontier model support, reasoning and multimodal workloads, distributed inference and hardware optimization. Each member names one technical representative and joins two roadmap meetings a year; SGLang maintainers keep authority over technical decisions and code acceptance, and the project stays open to the whole community.

Zoom

OpenAI introduced Control Room on its official account and blog: a central system across its research infrastructure that connects every research inference workload, including agentic sampling for training and evaluation and data processing. It works with OpenAIs misalignment monitoring systems and halts a jobs model output within 35 seconds of a decision to intervene. The kill switch lives in the inference engine layer rather than in research code: engines check a job_id with Control Room before serving each request, and if Control Room goes down or is tampered with, all inference requests are denied (fail-closed). Fail-closed already covers all frontier model research workloads and will reach the remaining research workloads over the next few days. This is a safety infrastructure announcement, not a new model or public API.

Zoom

DeepSeek officially open-sourced DeepInfer under Apache 2.0, its production-grade inference engine serving DeepSeek-V4.1-Flash at scale. Originating from vLLM with architectural refinements from SGLang and TensorRT-LLM, DeepInfer natively targets NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It integrates specialized kernels including FlashMLA, DeepGEMM, DeepEP, and TileKernels. The engine features DSpark confidence-scheduled speculative decoding with 5-token drafts, SWA bounded replay over a 140-token sliding window, and balanced MoE expert parallelism. DeepSeek simultaneously open-sourced deepseek-recipe, delivering an end-to-end open ecosystem across models, kernels, and inference runtimes.

Zoom

At PyTorch Conference North America on Oct 20, the CNCF Sandbox project llm-d released version 1.0. It turns inference engines such as vLLM and SGLang into one coordinated serving pool, and 1.0 adds multi-tenant priority and fairness scheduling (flow control), router high availability with zero-downtime rolling upgrades (DisaggregatedSet), a v1 configuration API, and agent-aware session routing with Responses API state. Founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, the project has 500+ contributors, and JetBrains, DigitalOcean and Mistral already run it in production.

Zoom

Strata, an open-source C++ inference engine, runs the 125B-parameter (6B active) Qwen3.8-Flash-Next on a 12 GB gaming GPU plus 32–64 GB RAM, exposing OpenAI, Anthropic and Responses-compatible APIs on localhost so Claude Code and Codex CLI can use it. The repo passed 10k GitHub stars and hit the Hacker News front page (507 points).

Zoom

SGLang, a leading open-source high-throughput inference engine for large models, officially releases milestone version v0.5.21, merging 779 PRs across 227 contributors. The release introduces dynamic on-the-fly role switching between Prefill and Decode (PD) instances without restarting, an upgraded high-performance Rust core for the default prefix cache, and 22% faster first-token generation for DeepSeek-V4.1 on long prompts. It also debuts the /v1/decisions and /v1/score APIs, alongside complete support for next-gen models including MiMo-V2.6, DiffusionGemma, and GLM-5.3-Flash on AMD MI355X.

Zoom

In reinforcement learning with verifiable rewards (RLVR), rollout trajectories are generated by high-throughput inference engines while backward gradients are computed by training engines, introducing subtle floating-point probability discrepancies that trigger unbounded variance under exact importance sampling. Researchers identified the invariance of logit displacement distributions and introduced Calibrated Importance Sampling (CIS), featuring confidence-aware truncation that replaces unbounded second moments with constant bounds, establishing top scores across five mathematical benchmarks.

Zoom

High-throughput LLM inference engine SGLang released milestone v0.5.20, consolidating 713 merged PRs from 237 contributors. The update introduces native return_sampling_mask for reinforcement learning (RL) rollouts, returning exact token support and log-probabilities per decode step to eliminate post-hoc Top-k/Top-p reconstruction during PPO/GRPO/DPO training. Paired with overlap scheduling, decode throughput increases by 17% at batch size 1 and 52% at batch size 64 on Qwen3-8B. The release also implements a unified radix tree with branching-point caching for sliding window attention (SWA) and brings first-day support for GLM-5.3-Flash, Tencent Hy4-Preview, and Qwen3.8-Flash-Next.

Zoom