AI Coding Live News

LM Studio posted on X at 15:32 UTC on Oct 29 that users can now download a new oMLX engine inside LM Studio and run it with MLX models they already have. The official changelog lists it in Bionic 1.1.10, alongside speculative decoding for MLX models and an in-app terminal. oMLX is an open-source Apple Silicon inference server maintained by developer jundot (Apache 2.0, about 23,000 GitHub stars) built around continuous batching and a two-tier RAM and SSD KV cache. LM Studio has not published speed comparisons for the engine.

Zoom

On Oct 29 Anthropic introduced self-assessment on X: after each response, Claude produces a structured preliminary safety assessment of the whole conversation for offline review. Anthropic's Safeguards blog says that on Claude Fable 5, at a 1% false-positive rate, it flags 78% of the conversations a stronger prompted classifier flags and 94% of high-priority ones. Because the conversation is already in the KV cache, it adds about 10 output tokens on average and an estimated under 2% of inference compute. It has run in production since June, but Anthropic says it is not robust against an attacker who targets it directly and is only one layer of defense in depth.

Zoom

In an Oct 26 incident report on its blog, Cloudflare said that between Aug 27 and Sep 9, 155 responses from GLM-5.3 on Workers AI (@cf/zai-org/glm-5.3) to prompts longer than 681,280 tokens contained data from other customers' recent requests, all returned with HTTP 200. The cause was a prefill KV cache pool smaller than the 1,048,576-token context window, combined with an SGLang bug in disaggregated prefill/decode mode. Cloudflare found no evidence of deliberate exploitation, has fixed it, and recommends operators upgrade to SGLang v0.5.22 or later.

Zoom

Researchers introduced MemT-1.0 (arXiv:2610.26825), a groundbreaking architecture internalizing memory as an intrinsic component of foundation model computation. Moving beyond exploding KV caches and lossy Vector RAG pipelines, MemT-1.0 employs Gated DeltaNet-2 with stable within-turn token-level updates alongside a novel Mixture-of-Storage mechanism. Trained across a two-stage 34B-token pipeline, the released MemT-1.0-Nano model achieves a 1.86x end-to-end inference speedup and a staggering 66x reduction in persistent storage at 128K tokens relative to full-context KV caching, dramatically elevating procedural task execution in autonomous agents.

Zoom

On Oct 20 (UTC) DeepSeek open-sourced DeepInfer (Apache 2.0), the in-house inference stack that serves DeepSeek-V4.1-Flash at scale. The release covers offline inference with Attention DP/CP and MoE EP+EPLB parallelism, DSpark speculative decoding, SWA bounded replay, batch-invariant inference, deterministic sampling and structured outputs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The KV cache system, prefill/decode disaggregation and load balancing are not included. DeepSeek also released deepseek-recipe (MIT), which converts Chat Completions, Messages and Responses requests and encodes them the same way the official API does.

Zoom

vLLM shipped v0.32.0 on Oct 18 with 814 commits from 363 contributors (121 new). Highlights: a default-served POST /v1/systemone structured-decision endpoint and a generic hf tool/reasoning parser; NVFP4 KV cache on SM8x/SM90/SM12x and FP8 KV by default for GLM-5.x on SM100; Model Runner V2 gains a GPU n-gram speculator and Elastic EP and becomes the ROCm default; three security advisories are fixed, default CORS is restricted, and Python 3.10 is dropped.

Zoom

Addressing KV cache bottlenecks in long-context LLM serving, a new production study on 8xB300 nodes serving GLM-5.3 (arXiv:2610.21500) reveals that unsharded remote KV caches inflate hit-heavy p99 TTFT to 83.32s, and resolves it via rank-sharded asynchronous LMCache loading—boosting GET bandwidth 6.3x to 10.76 GB/s and slashing p99 to 0.67s (+34% throughput). Meanwhile, Baidu PaddlePaddle open-sources Virtual Head Attention (VHA, arXiv:2610.17959), folding linear Q-Premix and Postmix into standard GQA-2 weights at inference to halve KV cache while beating MLA (+0.70%) and GQA-4 (+2.60%) on a 30B-A3B MoE and lifting 8K-128K RULER from 73.0 to 84.4.

Zoom

Standard Transformers route layer communication exclusively via the residual stream h—a superposition channel—forcing every attention layer to project and maintain massive independent key-value caches that scale memory consumption at 2 * L * d_model. Researcher Jakob Eriksson introduces The Extender (arXiv:2609.32759), a log-structured alternative to standard Transformers. The Extender complements the residual superposition channel h with an explicit concatenation channel x. Each layer l emits a standard residual update delta_l to h and appends a minimal extension vector epsilon_l to x (e.g., |epsilon_l| = 32). While FFNs and Query projections attend to h, Key and Value projections take exclusively the cumulative concatenation channel x as input. This structural decoupling slashes persistent attention memory from 2 * L * d_model to sum |epsilon_l|. For a 924M-parameter model with hidden width 1664, The Extender compresses persistent attention memory by 104x relative to Multi-Head Attention (MHA), matching baseline accuracy on short-context CORE benchmarks and outperforming standard Transformers on long-context RULER evaluations.

Zoom

On Oct 5 Reka Labs released a research preview of Rho-1, a 19B omni-reasoning model trained from scratch that unifies text, image, video and robot-action tokens in one network and one KV cache, with no tool calls or second model. Each transformer block has an understanding stream and a generation stream with shared attention, trained jointly with next-token prediction and flow matching. Reka reports base video generation at 0.79x real time (median) and a distilled 8-step variant (vs 99) returning a 5.3s clip in about a second. The checkpoint was trained on 320 H100s for three months. Research preview only; no public weights.

Zoom

Multi-agent systems solving long-context distributed tasks face a fundamental trade-off: textual communication is compact but introduces autoregressive decoding latency and token-level information loss, whereas latent communication transferring raw KV caches avoids token generation but causes explosive linear memory scaling across agents and context tokens. Researchers from Columbia University introduce CacheBack (arXiv:2609.32046), a robust, training-free realization of receiver-conditioned communication. The receiving agent transmits a concise description of its information requirements, which acts as a dynamic mask on the sender's attention weights to filter and compress the KV cache. Benchmarked on FanOutQA with Qwen 3, CacheBack prunes 75% to 94% of redundant cache states, boosts task accuracy by 14.7 percentage points, and slashes median completion latency by 3.2x compared to text messages across dense, Mamba-hybrid, and sliding-window architectures.

Zoom

Conventional LLM agents emit reasoning thoughts and actions token-by-token, making extended interactions slow and compute-heavy. While Jev-style probabilistic models deliver rapid predictions over finite action domains, they mandate pre-specified, static fields, precluding autonomous adaptation in open-ended language tasks. Shanghai Jiao Tong University researchers introduce JevSpawn, a compositional policy framework bridging natural language instructions with finite probabilistic exploration. JevSpawn couples parallel action spawning with feedback-driven branch selection, representation revision, and recovery from cached alternatives. By sharing prefix KV caches and action structures, JevSpawn eliminates repeated context generation without model training, systematically outperforming seven leading agent baselines across eight benchmarks.

Zoom

Chunked KV-cache compression reduces the memory and computational footprints of long-context LLMs by condensing contiguous token windows into fewer cache entries at fixed strides. Researchers from Stanford and Renmin University uncover an intrinsic vulnerability: chunking introduces a new relative positional coordinate—a token's phase relative to compression-window boundaries. The study reveals systematic 'phase sensitivity,' where retrieval accuracy for identical facts varies by up to 40 percentage points depending solely on input token phase alignments. Pretraining transformers across various KV compression configurations and conducting causal interventions, the authors demonstrate that standard aggregate benchmark scores mask recurrent blind spots, establishing a new evaluation standard for long-context compression.

Zoom

Deploying 70B+ LLMs under tight memory constraints necessitates extreme 4-bit weight, activation, and KV-cache quantization (W4A4KV4). However, current methods predominantly focus on suppressing activation outliers, overlooking their geometric alignment with the quantizer grid. Researchers introduce PrismQuant, a quantizer-aware rotation framework that mathematically aligns the leading activation eigenspace with the constant group subspace of asymmetric grouped INT4 quantization. Formulated as a Ky Fan trace maximization problem, PrismQuant derives a closed-form, provably optimal rotation solution without requiring expensive gradient descent. Benchmarked across Llama, Qwen, and Mistral architectures up to 70B parameters and 30B MoE variants, PrismQuant sets new SOTA marks: on Llama-3.1-70B under W4A4KV4, it achieves 72.46% zero-shot accuracy—merely 0.22 percentage points below full FP16 precision. Deployment benchmarks demonstrate a 1.51x prefill speedup and 56.34% reduction in peak decoding VRAM with code open-sourced on GitHub.

Zoom

In collaborative multi-agent systems sharing a common foundation model, distinct system prefixes cause KV cache representations for the shared context to diverge, forcing each agent to redundantly re-prefill the entire conversation history. Researchers from Seoul National University and KAIST introduced KVCMAS, an online KV cache correction framework. By parameterizing cross-agent cache deviations as compact low-rank states and chaining corrections along agent pipelines without external reference prefills, KVCMAS accelerates TTFT by 2.0x while reducing peak GPU memory by up to 3.7x.

Zoom