Agents26d ago
TestPrism: Nanjing University Exposes the Single-Reference Illusion in Coding Agent Test Generation, Slashing Pass Rates from 59.7% to 28%
Autonomous coding agents are increasingly leveraged to synthesize test suites for enterprise software. However, the standard practice of evaluating generated tests against a single reference solution overlooks alternative valid implementations and inflates perceived test quality. Researchers from Nanjing University and Hong Kong Polytechnic University unveil TestPrism (arXiv:2610.12289), exposing this systemic evaluation illusion. TestPrism comprises 300 test generation tasks sourced from 17 repositories and pairs them with 3,000 candidate implementations evenly divided into valid and invalid solutions. Its primary metric, the Joint Success Function, mandates that a test suite must simultaneously fail against buggy baselines, pass across all diverse valid implementations, and reject every invalid candidate. Evaluating fourteen frontier coding agent setups reveals that while models register a 59.67% success rate under legacy single-reference scoring, their actual Joint Success Function plunges to just 28.00%. To resolve these foundational flaws, the authors introduce TestHelix, an agentic synthesis harness leveraging peer cross-validation and recursive self-improvement to elevate Joint Success scores by up to 9.00 percentage points.
Agents28d ago
CheckerBench: ECNU and Peking University Introduce the First Executable Benchmark for Coding Agents Synthesizing Static-Analysis Checkers
Prevailing coding agent benchmarks evaluate isolated patch synthesis (e.g. SWE-bench) or vulnerability detection, neglecting whether agents can construct reusable, production-grade static-analysis checkers from scratch. Developing static analyzers requires interpreting defect specifications, traversing multi-file AST semantics, writing engine-specific logic, and refining implementations through iterative compilation and diagnostic feedback. Researchers from East China Normal University, SJTU, and Peking University unveil CheckerBench (arXiv:2610.07557), an executable suite comprising 300 tasks derived from 297 CVEs across 167 repositories, 85 CWEs, and five programming languages. Powered by the CheckerLab harness, evaluation across 21 model-harness configurations reveals a mean Pass@1 rate of only 32.30% (peak 45.33%), demonstrating that end-to-end program analysis tool synthesis represents a profound new frontier for software engineering agents.
Agents29d ago
SafeActBench: NUS Investigates the Broken Evidence-to-Action Chain in Tool-Using Agents Across Six Operational Domains
Tool-using agents execute consequential modifications to external systems, yet nominal task success frequently conceals unverified actions executed without prior evidence. Researchers from the National University of Singapore (NUS) present a rigorous empirical study on the breakdowns along the evidence-to-action chain (arXiv:2610.07753). Evaluating ten model-harness configurations reveals a striking paradox: high static action assessment accuracy routinely masks brittle interactive execution. Failures overwhelmingly originate prior to execution: agents terminate prematurely before collecting requisite evidence, or initiate state-altering actions before foundational proofs are established. The authors formulate SafeActBench, spanning 656 rigorous cases across six operational domains and five protocols progressing from static judgment to complex multi-action workflows. Supported by a provenance-bound Evidence Ledger and deterministic trajectory evaluator, findings demonstrate that while single-step execution is stable once evidence is verified, multi-action workflows frequently collapse under unresolved prerequisites and truncated state transitions.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.