Benchmark49d ago
Vals Live Terminal-Bench 4.0: GPT-6 Astra Leads at 57.1%
Vals AI launched Terminal-Bench 4.0: 66 new end-to-end terminal tasks (ship a service, prove a theorem, train a GPU kernel, write a forensic report, and more). Median expert effort is about 4 hours; grading is strict (full verifier suite or zero), and scores are avg@3. GPT-6 Astra won at 57.1%, 7.6pp ahead of Claude Fable 5.1 (49.5%) and 11.6pp ahead of Claude Opus 5 (45.5%). No other model cleared 30%; fourteen of 27 models scored zero on both hardware and media. Version 4.0 spans seven categories—software, science, ML, operations, hardware, security, and media—with roughly three-quarters of tasks outside traditional software work and no overlap with v2.1.
Benchmark14d ago
Artificial Analysis Releases HealthBench & MLCR-AA: Exposing Severe Verbosity Bias in Medical AI Rankings
Independent AI evaluation platform Artificial Analysis officially released HealthBench Professional alongside the length-normalized MLCR-AA index (@ArtificialAnlys). The benchmark revealed a pervasive industry blindspot: LLM-as-a-Judge evaluations suffer from acute 'verbosity bias', disproportionately rewarding longer generations regardless of medical conciseness. In raw HealthBench scoring where Anthropic outputs were 4–6x longer, Claude Opus 5.5 led at 71.4%. However, once length bias was scientifically normalized via MLCR-AA, GPT-6 Astra (max) tied Claude Sonnet 5.5 for #1, while Claude Opus 5.5 plummeted from #1 down to #6, setting a new rigorous standard for medical model evaluation.
Release2d ago
OpenAI Plans Its Next Internal Security Sprint With Cathedral, More Than Quadruples Critical Bug Bounty Rewards to Up to $172,000
On November 2 (US time) OpenAI said it is planning its next internal security sprint and partnering with security firm Cathedral, whose researchers will use OpenAI models alongside OpenAI teams to challenge its defenses. It is also changing its Security Bug Bounty program, more than quadrupling rewards for critical findings; OpenAI's X post says exceptional findings can earn up to $172,000 including bonuses. Per OpenAI, the last sprint dedicated 25% of its product engineers to finding and fixing weaknesses, using Codex to reproduce vulnerabilities and prepare patches, and it has committed $1 billion in subsidized access over six months for frontline defenders.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.