AI Coding Live News

On 2026-10-23 (15:41 UTC) the official @NVIDIAAI account announced Nemotron 3.5 Bolt E4B 4GB, a small open model for local assistants, interactive characters and robots that can hold a conversation and use tools. The official Hugging Face model card says it is post-trained from Gemma-4-E4B-it, has 8B total parameters (4B effective), ships in Q4_K_M quantization, supports up to 64K input and 64K output tokens, and has a switchable thinking mode, plus a DFlash speculative-decoding drafter and a BF16 version. Weights, data and recipes are open under the OpenMDW-1.1 license, with commercial use allowed. On the model card the 4GB build scores 55.60 on MultiChallenge (Gemma4 E4B-it at the same quantization 39.36, Qwen3.5 4B 52.23), but trails Qwen3.5 4B on Tau2-Bench (52.32) and BFCL-v3 (67.31).

Zoom

In small language models, vocabulary embedding tables often account for over 60% of total parameters, posing severe bottlenecks for edge deployment and ternary quantization. A new study (arXiv:2610.22428) replaces trainable embeddings with a single frozen 16 MB table of 512-bit binary codes derived via PCA and sign binarization. On Gemma 3 270M, trainable parameters collapse from 268.1M to 101.0M (-62.3%) while matching or exceeding original performance across HellaSwag (30.4%), LAMBADA (23.2%), and PIQA (61.5%).

Zoom

Two new frameworks re-architect how weak and strong models collaborate in long-horizon agents: SpecAgent (arXiv:2610.21463) lets a small model draft multi-step read-only tool trajectories grounded in real observations while holding environment-modifying actions, allowing the strong model to verify the entire segment in a single parallel prefill pass—halving wall-clock time and memory traffic on τ²-Bench. Complementarily, LEASH (arXiv:2610.22280) uses training-free latent-state uncertainty to escalate and supervise computer-use agents, reaching 73.66% on OSWorld with Qwen3.5-35B-A3B—beating always-on Claude Opus 5 (72.90%) while cutting inference cost by 19.2%.

Zoom

Token-level LLM routing—routing easy tokens to small models and switching to frontier models for critical reasoning tokens—presents an optimal trade-off between output quality and inference cost. However, current LLM serving engines (such as vLLM and TGI) are architected for single-model execution, suffering severe step desynchronization, batch admission latency, and extreme orchestration complexity when handling high-frequency token transfers across heterogeneous models. Researchers from Tsinghua University's NICS Lab present TokenRouter (arXiv:2610.12242, accepted at NeurIPS 2026, code: github.com/thu-nics/TokenRouter), an efficient and developer-friendly serving system engineered for token-level routed LLM inference. TokenRouter introduces a 'request-centric programming and model-centric execution' paradigm, enabling developers to write routing logic intuitively per request while the runtime handles asynchronous execution across subservers. Incorporating a Delayed-Batching Scheduler parameterized via discrete-time Markov chain modeling, a decoupled tri-loop architecture, and an instant handoff-resume mechanism, TokenRouter achieves 2.01x to 64.15x higher decoding throughput compared to existing serving platforms across diverse routing workloads, with complete code open-sourced.

Zoom

While test-time compute scaling has propelled reasoning capabilities in large models, small reasoning models (sRMs, 4B-8B) frequently fail when scaling thinking tokens: self-refinement merely consolidates probability onto already reachable hypotheses, failing completely when missing parametric knowledge. KAIST researchers introduce FlyBy, a selective querying framework that formally decouples execution bottlenecks (recoverable through internal reflection) from knowledge bottlenecks (requiring external teacher queries). Optimized via cost-aware reinforcement learning, FlyBy teaches small models when to think, when to ask, and how much compute budget to expend. Evaluated across 1,158 hard problems on six benchmarks, FlyBy-4B reaches 45.96% pass@8, outperforming the much larger Qwen3-14B (41.64%) while slashing serving costs by 2.7x (a 63% reduction).

Zoom