News · Leaderboard

Latest news, beside the leaderboard

Read what just changed, and see who is ahead. Those are the two doors on this page.

C
Claude Devs@ClaudeDevs·15h ago
🚀 Release

Claude Code desktop reworks multi-session flow: saved environments, nested parallel sessions, /btw side tabs and a needs-you sidebar

On 2026-10-14 (UTC) ClaudeDevs announced a Claude Code desktop update built for running many sessions side by side: a new-session screen with recent folders, saved environments (several folders, an SSH host or the cloud) with an optional worktree per folder, nested sessions for parallel tasks that report only results back to the main thread, one-click follow-up sessions, /btw side questions in their own tab, and a sidebar that separates working sessions from ones that need you, plus a live to-do list, recap cards and a side rail showing where a session runs, its context and its changes. Update the desktop app to get it; Anthropic published no performance numbers.

Claude Code desktop reworks multi-session flow: saved environments, nested parallel sessions, /btw side tabs and a needs-you sidebar
⚡ Key Takeaways
  • •6-post official thread; the lead post had about 15.8K likes and 1.31M impressions at fetch time
  • •Parallel work: each task Claude runs in parallel becomes a nested session that reports only its result back, keeping the main thread readable
  • •Saved environments: folders, an SSH host or the cloud, with an optional Git worktree per folder (default <project-root>/.claude/worktrees/)
  • •Attention: sidebar splits working vs needs-you sessions; live to-do, recap card on hand-off; /btw (or Cmd+;) side questions stay out of the main history
  • •No benchmarks published; users already ask to un-nest forked sessions in the sidebar (GitHub issue #100216)
Read details→
M
ModelScope@ModelScope2022·18h ago
🚀 Release

StepFun opens Step 5 Preview weights: 600B/27B sparse MoE agent flagship with 1M context now on Hugging Face and ModelScope

On 2026-10-15 StepFun released open weights for Step 5 Preview as promised: a 600B-total / 27B-active sparse MoE with up to 1M-token context and vision input, aimed at agentic software engineering and knowledge work. Checkpoints are on Hugging Face (FP8, BF16, NVFP4) and ModelScope. Official scores include StepCodeBench 49.0%, FrontierFinance 66.4%, and Artificial Analysis Intelligence Index 44. September coverage was API/index only; this is the first public weight drop.

StepFun opens Step 5 Preview weights: 600B/27B sparse MoE agent flagship with 1M context now on Hugging Face and ModelScope
⚡ Key Takeaways
  • •Specs: sparse MoE 600B total / 27B active, up to 1M-token context, text+image input (model page)
  • •Weights: stepfun-ai/Step-5-Preview plus BF16 and NVFP4 on Hugging Face and ModelScope (HF card)
  • •Coding: StepCodeBench avg@4 49.0% (leads the open-weight set StepFun cites); DeepSWE v1.1 67.7% (official table)
  • •Finance / index: FrontierFinance 66.4%; Artificial Analysis Intelligence Index 44 (matches earlier AA coverage)
  • •Serve via official vLLM / SGLang Docker images and OpenAI-compatible APIs; license is StepFun Model License 1.0 (Apache-2.0-style with commercial conditions)
Read details→
C
Claude Devs@ClaudeDevs·19h ago
🚀 Release

Claude Platform opens OpenTelemetry tracing beta: Messages API, Managed Agents and SDK spans flow to your own observability stack

On 2026-10-14 Anthropic put OpenTelemetry tracing for Claude Platform into public beta. Admins add a telemetry destination in the Claude Console and the API exports spans for Messages API requests and Managed Agents sessions server-side with no code changes, while the Claude SDKs export client-side spans for requests and tool-runner tools. Spans follow the OpenTelemetry gen_ai conventions and default to metadata (duration, status, token counts, model, tool names), with an optional full-content mode.

Claude Platform opens OpenTelemetry tracing beta: Messages API, Managed Agents and SDK spans flow to your own observability stack
⚡ Key Takeaways
  • •Two paths: a server-side destination via Console/Admin API (Messages API + Managed Agents, no code changes) or client-side spans from the Claude SDK (overview)
  • •Destinations need OTLP/HTTP with protobuf over HTTPS on port 443; the Admin API uses the anthropic-beta: telemetry-destinations-2026-08-11 header (API guide)
  • •SDK tracing covers Python, TypeScript, Go and Java (anthropic-java 2.72.0+), and runs on Claude API, AWS, Bedrock, Google Cloud and Microsoft Foundry, all in beta (SDK guide)
  • •Metadata-only or full-content modes; ZDR customers can still export full content for Messages API requests (held in memory until delivered); Managed Agents is not ZDR-eligible
  • •Anthropic warns span names and SDK options may change during the beta, so pin SDK versions
Read details→
G
Google Research@GoogleResearch·19h ago
🚀 Release

Google Research launches Deep Review publicly: a four-stage agentic document verifier, piloted on 16,000+ conference papers, now on Gemini 4 Argon with an API

Google Research publicly released Deep Review on 2026-10-14. It grew out of the experimental Paper Assistant Tool (PAT), which gave opt-in pre-submission feedback on 16,000+ manuscripts at STOC 2026, ICML 2026, NeurIPS 2026 and ICLR 2027, and now covers design docs, presentations, and legal and financial documents. Users get two free reviews a week, can connect a Gemini API key for higher limits and the Deep Review API, and the system now supports Gemini 4 Argon.

Google Research launches Deep Review publicly: a four-stage agentic document verifier, piloted on 16,000+ conference papers, now on Gemini 4 Argon with an API
⚡ Key Takeaways
  • •Pilot scale: 16,000+ opt-in manuscripts across STOC, ICML, NeurIPS and ICLR (Google Research blog)
  • •Review load: over 74,000 submissions to ICLR, NeurIPS and ICML this year vs ~45,000 in 2025, 32,000 in 2024 and 23,000 in 2023 (Google-reported)
  • •Two free reviews per user per week; a Gemini API key unlocks higher limits and the Deep Review API; PDF and plain text at launch, multi-file coming (Deep Review)
  • •Uploaded documents are never used to train or fine-tune Google models and are stored one year for display only
  • •Antigravity adds a Teamwork Deep Document Review pattern via /teamwork-preview; methodology in arXiv 2606.28277
Read details→
ADSponsored
A
Artificial Analysis@ArtificialAnlys·22h ago
📊 Benchmark

Artificial Analysis launches AA-Robotics v0.1: coding agents drive a real robot arm, GPT-6 Astra leads at 69.4

Artificial Analysis released AA-Robotics v0.1 on 2026-10-14, a research preview in which frontier models running inside coding-agent harnesses (Codex, Claude Code, Antigravity) control a 6-axis robot arm over an HTTP API across eight tabletop tasks. GPT-6 Astra scores 69.4, ahead of GPT-6.1 Sol (64.6), Gemini 4 Argon (55.3), Claude Opus 5.5 (48.8) and Claude Fable 5.1 (46.3). Models take about 30 minutes per task versus roughly 20 seconds for a human.

Artificial Analysis launches AA-Robotics v0.1: coding agents drive a real robot arm, GPT-6 Astra leads at 69.4
⚡ Key Takeaways
  • •Scores (out of 100): GPT-6 Astra 69.4, GPT-6.1 Sol 64.6, Gemini 4 Argon 55.3, Claude Opus 5.5 48.8, Claude Fable 5.1 46.3 (AA article)
  • •Speed: ~30 min per task for models (Argon ~52 min) vs ~20 s for a human
  • •Cost: GPT-6.1 Sol cheapest at $1.93/task, GPT-6 Astra $16.36/task; ~70% of spend is input context
  • •Grasp success: Astra 45%, Fable 5.1 44%, Opus 5.5 40%, Argon 36%, Sol 28%
  • •Measure-the-box (within 2 mm) tops out at 17/100 across all five models
Read details→
S
SpaceX Super Intelligence@SpaceXSI·1d ago
🚀 Release

SpaceXSI launches Grok Voice 3.0: speech-to-speech model tops Artificial Analysis S2S Index at 86.9, price held at $0.08/min

SpaceXSI (SpaceX Super Intelligence, x.ai) released Grok Voice 3.0 on 2026-10-14 17:04 UTC. Per Artificial Analysis testing (Oct 13) cited on the official blog, it scores 86.9 on the Speech-to-Speech Index (Grok Voice 2.0 81.3, GPT-Live-1 Astra 83.2, Gemini 3.8 Live 82.6), 97% on Full Duplex Bench, and 71.6% on τ-Voice — on par with GPT-Live-1 at ~40% lower latency. In the Voice Agent API, `grok-voice-latest` now aliases `grok-voice-3.0` with an unchanged interface at $0.08/min; `grok-voice-2.0` is deprecated.

SpaceXSI launches Grok Voice 3.0: speech-to-speech model tops Artificial Analysis S2S Index at 86.9, price held at $0.08/min
⚡ Key Takeaways
  • •#1 on Artificial Analysis Speech-to-Speech Index at 86.9 (v2.0 81.3, GPT-Live-1 Astra 83.2, Gemini 3.8 Live 82.6)
  • •Big Bench Audio 98.5%, Speech Agent Arena Elo 1070, task success 94.8%, Full Duplex Bench 97.0%
  • •τ-Voice 71.6%, comparable to GPT-Live-1 (Astra) at ~40% lower latency; top accuracy (74%) on ServiceNow EVA-Bench
  • •Same price: $0.08/min ($4.80/hr); grok-voice-latest auto-upgraded to 3.0, grok-voice-2.0 deprecated
  • •Production scale (vendor figures): ~20k Starlink support calls/day, ~70% resolved with no human
Read details→
D
Daanish Khazi@bertgodel·1d ago
🚀 Release

Paper Instruments releases Ultramarine 1 Flash: non-verifiable-RL knowledge-work model on GLM-5.3 Flash, $0.13/task on its own suite, API live, weights promised

San Francisco lab Paper Instruments launched Ultramarine 1 Flash on 2026-10-14 16:02 UTC: GLM-5.3 Flash post-trained with large-scale non-verifiable RL (per-task rubric rewards) for finance, legal, consulting and operations work. Its report claims +4.1 points on average over the base across six knowledge-work benchmarks with 6.5% fewer output tokens, and 59.3 at $0.13/task on its internal PI Knowledge Work suite vs Claude Fable 5 at 57.8 / $8.19. An OpenAI-compatible API (`ultramarine-1-flash`) is live; the CEO says weights follow in about two weeks and post-training cost ~$300k. The headline cost comparison rests on an internal benchmark and has not been independently reproduced.

Paper Instruments releases Ultramarine 1 Flash: non-verifiable-RL knowledge-work model on GLM-5.3 Flash, $0.13/task on its own suite, API live, weights promised
⚡ Key Takeaways
  • •GLM-5.3 Flash base + modified CISPO RL (LoRA rank 64, 131K context, up to 250 turns); +4.1 avg over base on 6 knowledge-work benchmarks with 6.5% fewer tokens
  • •JobBench 62.9 (#1, Gemini 3.8 Flash 60.6); FrontierFinance 53.3 (#1); APEX-Agents 1.1 57.0 vs Gemini 3.8 Flash 64.3
  • •Internal PI Knowledge Work: 59.3 at $0.13/task vs Claude Fable 5 57.8 / $8.19 and GPT-6 Sol 61.3 / $1.26
  • •Trade-off: slight coding regression — Terminal-Bench 2.1 83.2 (base 84.3), DeepSWE 1.1 61.5 (base 63.4)
  • •Curriculum: 4 domains, 19 practice areas, 88 synthetic environment types, fully synthetic tasks graded by rubrics
Read details→
H
Hugging Face Daily Papers@HuggingFace·1d ago
🔥 Trending

Tencent Hunyuan Releases LifeSearch: The First Real-World Life & Productivity Benchmark for Deep Research Agents, 4,500 Hours of Human Annotation Shows Frontier SOTA at Just 63.4%

While deep research (DR) agents excel on synthetic and academic queries, their reliability in realistic personal life and professional workflows remains uncharted. Tencent Hunyuan introduces LifeSearch, a rigorous benchmark featuring 203 real-user tasks across 7 domains backed by 4,500+ hours of expert annotation and 4,000+ fine-grained rubrics. Evaluating 12 leading DR agents reveals that the strongest frontier agent scores only 63.4%, unmasking persistent bottlenecks in multi-step information synthesis and constraint verification.

Tencent Hunyuan Releases LifeSearch: The First Real-World Life & Productivity Benchmark for Deep Research Agents, 4,500 Hours of Human Annotation Shows Frontier SOTA at Just 63.4%
⚡ Key Takeaways
  • •Pioneering real-world life & productivity testbed: spans 203 bilingual tasks across 7 realistic domains including travel itinerary synthesis, procurement, and healthcare analysis
  • •4,500 hours and 4,000+ expert rubrics: evaluates both synthesized report depth and concrete attributes (exact values, units, dates, and live URLs) with Spearman rank correlation of 0.89
  • •Revealing severe frontier agent limits: the highest-performing deep research agent scores a modest 63.4%, exposing widespread failures in temporal conflict resolution and deep search navigation
Read details→
ADSponsored
H
Hugging Face Daily Papers@HuggingFace·1d ago
🔥 Trending

Recursive Self-Improvement for Embodied Navigation: RSI-Nav Unifies Policy-Synchronized Dual Bootstrapping, Achieving 68.6% SR on R2R-CE Without Human Annotation

Vision-language navigation requires abundant, high-quality language supervision, yet traditional synthetic pipelines generate supervision from static models that lag behind evolving navigation policies. Shanghai AI Lab, Tsinghua, and SJTU introduce RSI-Nav, an embodied recursive self-improvement framework combining instruction-following and instruction-generation in a single model. Using shard-wise dual bootstrapping on unlabeled trajectories, RSI-Nav achieves state-of-the-art 68.6% SR on R2R-CE and 72.4% SR on RxR-CE without human intervention.

Recursive Self-Improvement for Embodied Navigation: RSI-Nav Unifies Policy-Synchronized Dual Bootstrapping, Achieving 68.6% SR on R2R-CE Without Human Annotation
⚡ Key Takeaways
  • •Unified dual-capability architecture: combines vision-language instruction following (VL2A) and vision-action instruction generation (VA2L) within a shared foundation model
  • •Shard-wise policy-synchronized bootstrapping: generates instructions over small trajectory shards, validates via simulation rollout, and updates parameters instantly before processing subsequent batches
  • •New continuous-environment SOTA: secures 68.6% Success Rate on R2R-CE and 72.4% on RxR-CE, proving robust recursive self-evolution in complex embodied navigation
Read details→
H
Hugging Face Daily Papers@HuggingFace·1d ago
🔥 Trending

Eliminating Off-Policy Bias in Asynchronous RLVR: Proximal KL Enables Token-Wise Importance Sampling, Beating GRPO on Qwen3 with Zero Training Collapse

Off-policy data in reinforcement learning with verifiable rewards (RLVR) causes severe policy gradient bias during asynchronous generation and response reuse. Researchers from UIUC and HKUST introduce Proximal KL, theoretically proving that full future KL credit renders the exact proximal optimizer the unique stationary policy without requiring full-response weights or fitted critics. Evaluated across Qwen3-1.7B and 4B models, Proximal KL outperforms GRPO on 8 math benchmarks by up to 2.58% while completely averting training collapse.

Eliminating Off-Policy Bias in Asynchronous RLVR: Proximal KL Enables Token-Wise Importance Sampling, Beating GRPO on Qwen3 with Zero Training Collapse
⚡ Key Takeaways
  • •Theoretical breakthrough for off-policy gradient bias: proves full future KL credit guarantees a unique stationary policy, decoupling reference and sampling policies without sequence-level variance explosion
  • •Decisive outperformance over GRPO: yields up to +2.58% micro-averaged accuracy gains across 8 mathematics benchmarks on Qwen3-1.7B and 4B, sharply improving AIME24 pass@16
  • •Eliminates training collapses during off-policy refresh: permanently avoids catastrophic policy degradation seen in standard PG and CISPO, reaching peak accuracy with fewer update steps
Read details→
G
GitHub@github·1d ago
🚀 Release

GitHub Copilot updates HydraFusion and adds HydraFusion Max, with Claude Opus 5.5 and the GPT-6 family in its model pool

On 2026-10-14 GitHub updated HydraFusion, Copilot's multi-model orchestration feature, and added HydraFusion Max for harder tasks. The pool now includes Claude Opus 5.5, GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna; effort is tuned per stage (medium for drafts, high for review), Cascade gains a judging step, and Critique now inspects the workspace with tools using parallel critics. Both remain in research preview across VS Code, the Copilot app and Copilot CLI.

⚡ Key Takeaways
  • •Pool: Claude Opus 5.5, GPT-6 Astra, GPT-6.1 Sol, GPT-6 Luna, chosen per workflow (GitHub Changelog)
  • •Per-stage effort: medium for drafts, high for review (previously each model's default)
  • •Cascade adds a judge that requests repair only when needed; Critique uses tools on the workspace plus parallel critics
  • •Two tiers: HydraFusion (balanced) and HydraFusion Max (deepest reasoning, orchestration still minds tokens)
  • •Available in VS Code, Copilot app and Copilot CLI on Pro, Pro+, Business and Enterprise with preview features enabled
Read details→
a
arXiv CS@arXiv_CS·1d ago
🔥 Trending

GreenWAM: Universal Robotics World Foundation Model Controls Real Humanoids Across 20 Embodiment Families

Robotics research typically separates world models (predicting scene transitions) from control policies (generating actions), fragmented by proprietary robot-specific kinematic formats. Sber AI researchers introduced GreenWAM, a foundational world model for robotics. GreenWAM unifies 20 robot embodiment families (from bimanual arms to dexterous humanoids) under a common joint coordinate space, co-training across visual, action-conditioned, and world-action objectives with specialized action-expert heads. GreenWAM achieves top-tier 3D trajectory and depth accuracy on WorldArena 1.0, establishes state-of-the-art policy benchmark scores on RoboTwin 2.0 and LIBERO-plus, and successfully drives real-world humanoid robots outperforming Cosmos 3 Edge and pi_0.5.

GreenWAM: Universal Robotics World Foundation Model Controls Real Humanoids Across 20 Embodiment Families
⚡ Key Takeaways
  • •Unified anatomical action space: standardizes joint coordinates across 20 diverse robot families, breaking down proprietary kinematic data silos
  • •Dual world model and policy capability: unifies visual scene generation with real-time policy execution via action-conditioned multi-task pretraining
  • •Real humanoid deployment: tops WorldArena 1.0 3D trajectory benchmarks and outperforms Cosmos 3 Edge and pi_0.5 on physical humanoid manipulation
Read details→
C
CMU Robotics Institute@CMU_Robotics·1d ago
🔥 Trending

Reading Right, Acting Wrong: CMU Diagnoses Why Fast System One Models Stumble in Autonomous Agent Loops

In production agentic loops, fast single-pass 'System 1' models are favored for latency-sensitive tool selection and interface actions. When these models fail, engineers cannot easily diagnose whether the agent misperceived environmental observations or misjudged its subsequent action. Researchers from Carnegie Mellon University (CMU) introduced a behavioral diagnostic benchmark across 17 fast models and frontier reasoning baselines, pairing next-action queries with independent factual probes about the state. The results expose a striking systemic pattern: the vast majority of erroneous actions follow completely accurate perception, with error concentration increasing in more accurate models. The true failure mode lies not in reading facts, but in synthesizing them into the next programmatic sub-goal.

Reading Right, Acting Wrong: CMU Diagnoses Why Fast System One Models Stumble in Autonomous Agent Loops
⚡ Key Takeaways
  • •Unmasking agent failure modes: CMU demonstrates that next-action failures are rarely caused by perceptual hallucinations; most errors occur after flawlessly reading state facts
  • •Higher accuracy exacerbates synthesis gaps: as base model accuracy increases, a higher percentage of its remaining failures stem from reasoning synthesis rather than perception
  • •Engineering remediation: identifies sub-goal derivation as the missing link; composing the model's own independent factual readings with deterministic code resolves the failure
Read details→
H
Hugging Face Daily Papers@HuggingFace·1d ago
🔥 Trending

Latent World Action Model for GUI Agents: Slashing Inference Latency by 98.9% While Boosting Task Success by 40.6%

Pixel-space world models for GUI agents require predicting future screen images prior to action selection. However, full-screen pixel reconstruction suffers from extreme latency and frequently triggers identity shortcuts, where generated screens merely copy the past rather than reflecting action-induced mutations. Researchers from KAIST introduced the first Latent World Action Model for GUI agents, shifting predictive objectives from pixel generation to action-conditioned transitions in latent space. Featuring shared multimodal representations with decoupled action and transition heads alongside action-aware contrastive losses, the framework achieves up to a 40.6% relative gain in online task success while reducing inference latency by 98.9%.

Latent World Action Model for GUI Agents: Slashing Inference Latency by 98.9% While Boosting Task Success by 40.6%
⚡ Key Takeaways
  • •Eliminating pixel reconstruction overhead: shifts future-state modeling from costly pixel diffusion to action-conditioned latent transitions, bypassing the identity shortcut
  • •Decoupled dual-head with contrastive alignment: leverages shared multimodal representations with specialized action and transition heads to isolate action-induced deltas
  • •Dramatic efficiency leap: delivers up to a 40.6% boost in online GUI success rate while reducing inference latency by 98.9% for real-time desktop execution
Read details→
F
Fish Audio@FishAudio·1d ago
🚀 Release

Fish Audio ships Drama 3: open-weight 35B MoE TTS with word-level emotion direction, #1 on Artificial Analysis Pronunciation Robustness at 93.2%

On 2026-10-13 Fish Audio launched Drama 3, a TTS model directed with plain-language bracket tags, open weights on Hugging Face, and API header model: drama-3. The backbone is a ~35B MoE (~3B active per token, initialized from Qwen3.5-35B-A3B's language tower) with a codebook-MTP audio decoder and preference-model RL. Independent Artificial Analysis Pronunciation Robustness: 93.2% vs 88.1% for S2.1 Pro; 83 languages and up to 16 speakers; Community License free for commercial use under US$500k annual revenue. Launch promo through Nov 12: about $10 per million characters on the API.

Fish Audio ships Drama 3: open-weight 35B MoE TTS with word-level emotion direction, #1 on Artificial Analysis Pronunciation Robustness at 93.2%
⚡ Key Takeaways
  • •Open weights on Hugging Face fishaudio/drama-3 under Community License v1.0 (free commercial use under US$500k ARR; no non-consensual voice cloning)
  • •Architecture: 35B MoE / ~3B active; hybrid attention (1 full-attn layer per 4); codebook-MTP decoder over 10 codebooks; initialized from Qwen3.5-35B-A3B LM
  • •Benchmarks: Artificial Analysis Pronunciation Robustness 93.2% (#1) vs S2.1 Pro 88.1%; brand/place/tech names 95.9%, context-dependent readings 95.3%
  • •Vendor H200 streaming: 271 frames/s single stream; concurrency-1 TTFA p50 88.9 ms; concurrency 16 ~927 frames/s, RTF 0.373
  • •Ship: API POST /v1/tts with model: drama-3; self-host via SGLang-Omni (sgl-omni serve --model-path fishaudio/drama-3, typically ≥48GB NVIDIA VRAM)
Read details→
C
Chainguard@chainguard_dev·1d ago
🚀 Release

Chainguard ships four MCP servers so coding agents verify images, APKs, EOL status and policy live—server-side governance instead of prompt guardrails

On 2026-10-14 Chainguard announced four MCP servers for agent supply-chain checks, generally available to enterprise customers: `cg-oci` (image manifests/SBOMs/provenance), `cg-apk` (Wolfi package equivalents), `cg-versions` (upstream releases and EOL), and `cg-api` (org policy and attestations). Any MCP client (Claude Code, Cursor, and others) connects via OAuth; pagination and response caps are enforced server-side so agents stop hallucinating tags or recommending EOL packages. Complements the same-day Harness/Checks/Workspaces launch by wiring dependency choices to live product data.

Chainguard ships four MCP servers so coding agents verify images, APKs, EOL status and policy live—server-side governance instead of prompt guardrails
⚡ Key Takeaways
  • •Four endpoints: https://cgr.dev/mcp, apk.cgr.dev/mcp, versions.cgr.dev/mcp, console-api.enforce.dev/mcp over Streamable HTTP + OAuth 2.0
  • •Live data: image SBOM/provenance, Debian→Wolfi package mapping, upstream EOL history, org policy/advisories—not model memory
  • •Governance: server-side pagination (~25–50 default, 200 max) and response caps; cg-api hides tools your token cannot use
  • •Headless: chainctl auth login --audience=<mcp-url> plus a headersHelper Bearer injector avoids Claude Code's ~1h OAuth re-login loop
  • •Extras: docs MCP at mcp.edu.chainguard.dev (no sign-in); Skills MCP at skills.cgr.dev/mcp for on-demand hardening skills
Read details→
a
arXiv CS@arXiv_CS·1d ago
🔥 Trending

Can Agents Predict Their Own Failure? Outcome-Grounded Confidence Calibration in LLM Agents Unlocks Reliable Execution

Controlling acting LLM agents requires step-level confidence signals to decide whether to execute an action, abstain, or hand off control to humans. Research from NYU introduces outcome-grounded confidence calibration for multi-step agents. Rather than calibrating confidence against immediate local actions, the framework aligns per-step verbalized confidence with the ultimate multi-step episode outcome using GRPO rollouts and proper scoring rules. Evaluated across ALFWorld, TextWorld, and WebShop on Qwen2.5-7B, confidence AUROC jumps to 0.68–0.85, steps within the top-10% confidence bracket succeed 4.3x more frequently, and expected calibration error plummets from 0.42 to below 0.12.

Can Agents Predict Their Own Failure? Outcome-Grounded Confidence Calibration in LLM Agents Unlocks Reliable Execution
⚡ Key Takeaways
  • •Outcome-grounded calibration: trains step-level verbalized confidence against eventual multi-step episode outcomes via GRPO rollout groups and proper scoring rules
  • •Immediate first-step foresight: enables agents to recognize task failure vulnerabilities right from the initial step, elevating confidence AUROC to 0.68–0.85
  • •4.3x higher success in top-tier confidence: the top 10% of confident steps succeed ~4.3x more often, driving expected calibration error from 0.42 down below 0.12
Read details→
N
NUS School of Computing@NUSingapore·1d ago
🔥 Trending

Last Secure Code Benchmark: Vibe Coding Under Fire as Top Agents Produce Secure Code on Only 25.1% of Tasks

While AI coding agents and 'vibe coding' workflows synthesize an unprecedented volume of production software, code security remains highly fragile. Researchers from NUS and Monash University introduced the Last Secure Code Benchmark, evaluating 450 tasks derived from 150 real CVE vulnerabilities across 7 languages (Java, C, C++, JS, Python, PHP, Go) and 24 CWE categories (including 21 of the 2025 CWE Top 25 and 8 of the Top 10 KEVs). While frontier agents achieve 73.6% to 91.3% functional completion rates, even the strongest model (Claude Fable 5) produces code that is both functional and secure on only 25.1% of tasks, exposing acute supply-chain risks in autonomous vibe coding.

Last Secure Code Benchmark: Vibe Coding Under Fire as Top Agents Produce Secure Code on Only 25.1% of Tasks
⚡ Key Takeaways
  • •First comprehensive vibe-coding security benchmark: 450 tasks based on 150 CVEs spanning 7 languages, covering 21 of the 2025 CWE Top 25 and 8 Top 10 KEVs
  • •The functionality illusion: while agents pass functional specs at 73.6%–91.3%, dual functional-and-secure pass rates collapse to 16.4%–25.1%
  • •Vulnerability defense crisis: across full-repository and file generation scenarios, top models resolve only 12% of vulnerability environments securely
Read details→
N
NUS School of Computing@NUSingapore·1d ago
🔥 Trending

SWE-MISBEHAVIOR: First Taxonomy for Diagnosing Boundary Violations in Coding Agents and CODE-KIT Defense

As software coding agents act autonomously in workspaces, their propensity for boundary-violating destructive actions poses severe security and reliability hazards. Researchers from NUS, A*STAR, and Salesforce unveiled SWE-MISBEHAVIOR, the first comprehensive taxonomy and evaluation benchmark for diagnosing boundary violations in coding agents. Built from 7.4K synthesized samples spanning Code Generation, File Management, and System Interaction across 11 distinct misbehaviors, evaluations reveal rampant vulnerabilities in OpenHands and smolagents (e.g., 98.5% dependency misinterpretation, 53% silent environment degradation). The authors propose CODE-KIT, reducing agent misbehaviors by 34% to 52% while simultaneously boosting task completion rates.

SWE-MISBEHAVIOR: First Taxonomy for Diagnosing Boundary Violations in Coding Agents and CODE-KIT Defense
⚡ Key Takeaways
  • •First boundary violation benchmark: introduces an 11-category taxonomy across Code Generation, File CRUD, and System Interaction with 7.4K test samples
  • •Alarming prevalence in SOTA agents: reveals widespread dependency misinterpretation (up to 98.5%), unintended user input alterations (50.5%), and environment tampering (53%)
  • •CODE-KIT mitigation toolkit: reduces boundary violations by 52% in smolagents and 34% in OpenHands while concurrently improving task resolution rates
Read details→
I
Inworld AI@inworld·1d ago
🚀 Release

Inworld Inference opens to all developers: an open-model inference API built for consumer AI, with Gemma 4 26B p95 first token 7.8x faster than the median OpenRouter provider

On 2026-10-14 Inworld AI opened Inworld Inference to every developer, positioning it as an inference service for consumer apps (companions, tutors, voice, interactive fiction). Instead of listing thousands of models it builds a per-model serving stack for open models such as Gemma 4, GLM 5.3 Flash, DeepSeek and MiMo: draft-and-verify (speculative) decoding, a rewritten split-KV verify attention kernel, KV-aware routing and eval-gated low precision. Vendor-reported: Gemma 4 26B p95 first token of 0.31 s, 7.8x faster than the median OpenRouter provider; up to 90% of prompts from cache; 99.999% success over 20M+ GLM 5.3 Flash requests in 30 days. The API is OpenAI-compatible with Flex/Default/Priority/Fast tiers per request.

Inworld Inference opens to all developers: an open-model inference API built for consumer AI, with Gemma 4 26B p95 first token 7.8x faster than the median OpenRouter provider
⚡ Key Takeaways
  • •Gemma 4 26B first token: p50 0.17 s / p95 0.31 s, 3.6x / 7.8x faster than the median OpenRouter provider (blog, vendor-reported snapshot, not a matched workload)
  • •Pricing: Gemma 4 26B at $0.07 input, $0.035 cached, $0.34 output per 1M tokens, nearly 40% below the median no-training provider
  • •Rewritten verify attention: 3.4–7.5x faster per layer, attention share of a decode step 27.5%→16.5%, p50 TTFT 154→117 ms at equal eval scores
  • •KV-aware routing: GLM 5.3 Flash cache hits 25%→47%, DeepSeek V4.1 Flash 11%→33%; cached MiMo 2.6 Pro input costs 99% less
  • •Reliability: 98 failures across 20M+ GLM 5.3 Flash requests in 30 days (~4.8 per million); per-request service_tier of Flex/Default/Priority/Fast
Read details→