On November 22 OpenAI published "Perplexity and Intelligence" by Chief Scientist Jakub Pachocki, disclosing for the first time the log-perplexity of its models on an internal held-out codebase dataset since GPT-3.5: about 0.99 for GPT-3.5, about 0.91 for GPT-4, transformative cybersecurity and math capabilities below 0.8, and breakthrough math results from a model at about 0.76. He proposes perplexity as a shared cross-lab metric and common limits not to be crossed until AI monitoring reaches a concrete milestone.

Key Takeaways

  • ✓Source: an official openai.com post by Chief Scientist Jakub Pachocki, who introduced it on X; Sam Altman reposted it calling it the most important thing OpenAI has published in a while
  • ✓Disclosed data: log-perplexity on an internal held-out codebase set is about 0.99 for GPT-3.5 and about 0.91 for GPT-4; recent models are below 0.8, and a model at about 0.76 began resolving open math problems
  • ✓Forecast: more transformative capabilities are expected in the 0.68-0.7 window, and an avalanche of pretraining algorithm advances could come as soon as 2027
  • ✓Proposal: frontier labs jointly set development limits based on perplexity and other measurements, not crossed until understanding and monitoring of AI behavior reaches a concrete milestone such as a successor to CoT monitors
  • ✓Caveats: labs use different eval sets and data mixtures, and gains from chain-of-thought reasoning do not show up in perplexity
OpenAI Chief Scientist Jakub Pachocki Publishes "Perplexity and Intelligence": Discloses Internal Pretraining Loss Curve and Proposes Shared Industry Limits
🖼️Official Media
Click to view high-res
🧭

Turn your technical choice into a development budget

Compare 40 dev plans & simulate token costs vs $20/mo subscriptions

🔬

In-Depth Technical Analysis

Background

Downstream benchmarks saturate quickly, so they track AI progress poorly over time, and labs lack a comparable shared metric. OpenAI Chief Scientist Jakub Pachocki posted on X on November 22 to introduce his article "Perplexity and Intelligence", saying OpenAI chose to share more about pretraining progress to move toward shared safety bars.

How it works

The post frames modern AI as training a compressor, or predictor, on a huge corpus and then eliciting its latent capabilities with prompting, finetuning and reinforcement learning. OpenAI has tracked this intelligence substrate with perplexity, or cross-entropy loss, on held-out data. Pachocki argues that AI taking part in pretraining algorithm research could cause a step change in pace that alignment and monitoring may not keep up with.

Numbers that matter

The disclosed log-perplexity on an internal held-out codebase set is about 0.99 for GPT-3.5 and about 0.91 for GPT-4. Below 0.8, transformative cybersecurity and math capabilities appeared, and a model at about 0.76 began producing breakthrough math results. The post expects new capabilities around 0.68-0.7 and notes a steep acceleration in 2026.

What it means

This is not a product launch and does not change the API. The key ask is an industry norm: frontier labs set common limits based on perplexity and other measurements and do not cross them until AI monitoring reaches a concrete milestone. Whether other labs will sign on has not been announced.

Action HubReady to adopt this in production?

Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.

Up NextChatGPT Dots Now Follow Custom Instructions: Set Tone and Which Actions Need Your Go-Ahead in the Thread, Plus Faster Replies, Less Jargon and Better Follow-Through
Read Now→