On November 5, the BenchFlow team publicly launched FrontierPhysics (arXiv:2611.06910): 56 end-to-end physics-research tasks authored by 42 physicists from their own work across nine domains. Agents must do literature review, planning, research/experiments, deterministic artifacts, and a manuscript. Across 22 models in 26 configurations, no setup passed more than 46 of 168 rollouts; the best reported Pass@1 is about 27.4% for Claude Opus 5.5 in Claude Code. Expert trajectory review finds the main gaps are rigorous reasoning and expert intuition, not missing physics facts. Artificial Analysis said it will partner to host the evaluation.
Key Takeaways
- ✓Scale: 56 tasks by 42 authors at 26 institutions across 9 physics domains; 56 accepted from 182 submissions; 18 based on published papers (including 4 Nature)
- ✓Grading: deterministic verifiers plus expert rubrics (blockers/non-blockers); a pass needs all verifiers and blockers
- ✓Results: 22 models / 26 configs; no configuration passed more than 46 of 168 rollouts; best Pass@1 ~27.4% for Claude Opus 5.5 in Claude Code (paper)
- ✓Failures: rigorous reasoning ~48%, expert intuition ~34%, literature synthesis ~27%; missing physics knowledge is not the main cause (manual audit of GPT-6 Astra)
- ✓Ecosystem: run via BenchFlow; Artificial Analysis will partner to host; paper arXiv:2611.06910
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Core Background & Industry Pain Points
Most physics benchmarks still test fixed questions, isolated subtasks, or paper reproduction with the original solution in view. FrontierPhysics targets end-to-end research: from a PI-style prompt through literature, planning, experiments, deterministic artifacts, and a manuscript, graded for both correctness and publishability.
Architecture Highlights & Internals
Tasks come from contributors' own research and pass general QA, domain peer review, and failure-mode analysis before merge. Scoring combines deterministic verifiers with an agent-as-judge rubric; failing any blocker zeros the task. Runs use BenchFlow across harnesses such as Codex, Claude Code, Grok Build, and Antigravity, with a uniform ~10-hour timeout per rollout.
Authoritative Benchmarks & Measured Scores
Per the paper: 22 models in 26 configurations; no configuration passed more than 46 of 168 rollouts; best Pass@1 is about 27.4% for Claude Opus 5.5 in Claude Code. A manual audit of 168 GPT-6 Astra trajectories finds rigorous reasoning and expert intuition as the main failure modes; authors' own manuscripts score about 96.2% on matching rubrics. Figures are paper/author-reported; no independent third-party replication is published yet.
Developer Hands-on Guide
Read arXiv:2611.06910 and benchflow.ai/research. Use it to stress-test research-agent harnesses, not as a quiz leaderboard. Tasks are physics-heavy and expensive in time/compute. Artificial Analysis said it will partner to host the evaluation; until then rely on the paper and BenchFlow.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.