On November 2, Stanford researchers Owen Queen and James Zou published an interactive report, "The shelf life of LLM findings", retesting 50 widely cited claims about LLM behavior on 20 models released from July 2024 to September 2026 by eight developers. Verdict: 22 claims (44%) expired, 17 (34%) weakened, 8 (16%) persist and 3 (6%) strengthened; the median time from publication to expiration for expired claims was 14.7 months. Capability failures such as letter counting, lost-in-the-middle and fabricated citations have largely faded, while preference issues such as sycophancy and judges favoring their own answers persist.
Key Takeaways
- ✓50 claims, 20 models, eight developers; verdicts reflect fitted effects as of September 30, 2026.
- ✓44% expired, 34% weakened, 16% persist, 6% strengthened; expired claims had a median lifetime of 14.7 months (mean 15.7).
- ✓Kaplan-Meier estimates put the expired share at about 4%, 16% and 39% six months, one year and two years after publication.
- ✓Citation fabrication is expired overall, yet Qwen3.7-Flash still fabricated 69% of references in the same test (about 1% for Sonnet 5.5, 0% for GPT-6.1 Sol).
- ✓The pipelines were built with Claude Code and Codex; 43 of 48 reproduction checks matched the original papers within tolerance.
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and industry pain points
Much of the conventional wisdom about LLMs comes from 2022-2024 papers: models cannot count the r's in strawberry, lose facts in the middle of long contexts, fabricate citations and abandon correct answers when challenged. These findings are still cited to design evals, write safety policies and make product decisions, yet few people systematically check whether they still hold on newer models.
How the study works
Stanford's Owen Queen and James Zou picked 50 widely cited claims, first reproduced each paper's result on the original model or a close substitute, then ran the same setup on 20 models released from July 2024 to September 2026 by eight developers (open and closed, large and small) and fitted trends against release date. Effects are normalized to a relative size e, where 1 matches the paper and 0 means gone; e at or below 0.2 counts as expired. The pipelines were built with Claude Code and Codex, which is how a two-person team covered 50 papers.
Results
All figures are as reported by the researchers: 22 claims (44%) expired, 17 (34%) weakened, 8 (16%) persist and 3 (6%) strengthened. Expired claims are mostly capability failures; those that persist or strengthen concern preferences and judgment, such as sycophancy, judges favoring their own answers, a left political lean and answering when they should abstain. Models differ widely: among July-September 2026 releases, expired counts range from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol, and although citation fabrication is expired overall, Qwen3.7-Flash still fabricated 69% of references.
What it means for developers
Do not treat two-year-old findings as default assumptions; retest evals and guardrails on the models you actually ship. Preference problems like sycophancy and judge self-preference remain real risks, so cross-score with different models when using LLMs as judges. The report lets you filter claim cards by research area and verdict, and the GitHub repo has a summary.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.