Last Secure Code Benchmark: Vibe Coding Under Fire as Top Agents Produce Secure Code on Only 25.1% of Tasks

Background and the Problem

With 'vibe coding' rapidly transforming software production, developers increasingly delegate entire production workflows to autonomous AI coding agents from natural-language instructions. However, standard coding benchmarks evaluate code purely on whether functional unit tests pass. In reality, a function that satisfies all user-facing assertions may harbor catastrophic memory vulnerabilities, SQL injection pathways, or insecure deserialization logic. Whether autonomous coding agents systematically inject high-risk CVE vulnerabilities into production codebases while striving for functional green builds has remained a dangerous blind spot.

Architecture and How It Works

To expose and quantify this structural vulnerability in autonomous software engineering, researchers from NUS and Monash University introduced the Last Secure Code Benchmark:

1. Real-World CVE Foundations Across Seven Languages: Formulated from 150 real CVE vulnerabilities, establishing 450 rigorous software engineering tasks spanning Java, C, C++, JavaScript, Python, PHP, and Go;
2. Direct Alignment with CWE Top 25 & KEV: Encompasses 24 common weakness enumerations, including 21 of the 2025 CWE Top 25 list and 8 of the Top 10 Known Exploited Vulnerabilities (KEVs) cataloged by CISA;
3. Three Realistic Vibe Coding Workflows: Tests agents across three distinct development modalities: in-filling a function, synthesizing a standalone module file, and architecting an entire repository source tree from scratch;
4. Dual-Gate Blind Security Testing: Prompts specify only business functionality without providing hints about security. Submissions are judged against hidden dual-gate pipelines evaluating functional regression and dynamic exploit resistance.

Benchmarks and Measured Results

Empirical evaluations across six frontier coding agents establish alarming security gaps:

• The Deceptive Functionality Illusion: Leading models achieve high functional success rates between 73.6% and 91.3%, easily deceiving engineers into premature code approval;
• Dual-Gate Pass Rate Collapse: When evaluated for both functional correctness and zero-vulnerability security, the strongest agent (Claude Fable 5) succeeded on only 25.1% of tasks, while competitor baselines slumped to between 16.4% and 24.7%;
• Near-Total Failure in Full-Repository Scenarios: In repository-from-scratch vibe coding setups, top models produced secure implementations in only 12% of vulnerability scenarios, systematically defaulting to unsafe string interpolation, unescaped queries, and disabled SSL checks.

Getting Started for Developers

The Last Secure Code Benchmark sounds an urgent wake-up call for teams deploying autonomous AI developers:

• Paper Study: Review arXiv:2610.14766 for the complete vulnerability breakdown and test harness architecture;
• Enforce Mandatory SAST/DAST Gateways: Never permit direct merging of agent-generated pull requests without mandatory integration of automated static analysis tools (e.g., CodeQL, Semgrep) in CI/CD pipelines;
• Contextual Security Rules Injection: Embed explicit security guidelines in local agent configuration files (e.g., .cursorrules, Cline system rules), strictly barring raw query interpolation, unsafe pointer arithmetic, and unverified deserialization calls.