Benchmark13d ago
FrontierRefactor Introduces First Code Performance Benchmark: 41% of Agent Patches Pass Tests with Zero Gain or Regressions
Published in arXiv:2610.25834, FrontierRefactor establishes the industry's first evaluation benchmark targeting autonomous coding agents on multi-scale, performance-oriented codebase refactoring. Unlike traditional benchmarks like SWE-bench that measure only functional pass rates, FrontierRefactor spans seven programming languages and 60 Amdahl-qualified tasks ranging from kernels to full repositories (10^1 to 10^4+ LOC). Initial evaluations reveal that over 41% of agent patches pass functional equivalence suites yet yield zero verified speedup—with 19.3% triggering severe runtime performance regressions, highlighting the vital need for multi-turn profiling feedback.
Tooling8d ago
Google's Gemma team says Gemini 4 Argon wrote vLLM optimizations that speed up gemma-4-E2B-it 2.85x on an A100
On Oct 26, Google's official @googlegemma account said a set of vLLM optimizations built with Gemini 4 Argon lifted gemma-4-E2B-it throughput on a single A100 from about 37,900 to 108,000+ tokens/sec, roughly 2.85x, with 100% token parity. The PRs were filed by Luciano Martins of Google DeepMind, and most are still under review.
Tooling48d ago
Vercel: TypeSafe Jev Coming to AI Gateway — Up to 18× Faster Than GPT Luna
Vercel CEO Guillermo Rauch said fx’s default auto mode uses a safety reviewer on every command. That reviewer runs on GPT Luna today; TypeSafe’s Jev is up to about 18× faster at p95 and more accurate, and is coming to Vercel AI Gateway as a likely new default. Earlier, a Vercel engineer reported Jev saturating a classifier eval vs Gemini 2.5 Flash Lite while running about 6× faster.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.