0.91, 0.76, 0.7: What OpenAI's perplexity numbers measure, and why they're a weak brake
OpenAI's chief scientist has published internal loss values for the first time and wants to turn them into limits for every AI lab. We work out what 0.76 actually means. And we explain why the number can't be compared across labs as long as the unit, tokenizer and test data remain undisclosed.

What this is about
On November 22, 2026, Jakub Pachocki, OpenAI's chief scientist, published the essay “Perplexity and Intelligence”. In it, OpenAI discloses internal pretraining metrics for the first time: GPT-3.5 sits at about 0.99, GPT-4 at about 0.91. Below 0.8, “transformative” capabilities in cybersecurity and mathematics are said to have emerged. From about 0.76, a model produced remarkable results on open math problems. Pachocki places the next critical range at 0.68 to 0.7. What will emerge there, he says, he doesn't know.
His proposal: the leading labs should agree on “shared limits based on perplexity and other measurements.” They would only cross these limits once monitoring of AI behavior reaches a concrete milestone, such as a successor to today's chain-of-thought monitors. The rationale is a possible “avalanche” of pretraining advances “as early as 2027.” Sam Altman called the text on X “the most important thing OpenAI has published in a long time.”
Most coverage simply repeats the numbers and the warning. This article asks something else: what is this number, can it be compared across labs, and what would need to be pinned down to turn it into a verifiable speed limit?
What the number measures and what 0.76 actually means
A language model predicts text piece by piece, in tokens, meaning word fragments. For each next token, it assigns probabilities. Cross-entropy loss measures how much probability the model gave to the token that actually comes next: you take the negative logarithm and average it over a test text. According to footnote 2, OpenAI tracks exactly this logarithm of perplexity (“log of perplexity, also called cross-entropy loss”). Perplexity itself has an intuitive reading: on average, the model is as uncertain as if it had to guess between that many equally likely candidates.
From this, you can work backward. Using the natural logarithm, i.e. in nats, e^-0.76 ≈ 0.47. So the 0.76 model gives the correct next token about 47 percent probability on a geometric average. The geometric mean is a multiplicative average: the model predicts many tokens almost with certainty and barely gets others. For GPT-4, it's about 40 percent. Our calculation:
| Model per OpenAI | Log perplexity | Perplexity (if nats) | Geom. hit probability (nats) | Same, if bits |
|---|---|---|---|---|
| GPT-3.5 | ~0.99 | 2.69 | 37 % | 50 % |
| GPT-4 | ~0.91 | 2.48 | 40 % | 53 % |
| Math model | ~0.76 | 2.14 | 47 % | 59 % |
| Next zone | 0.68–0.70 | 1.97–2.01 | 50–51 % | 62 % |
The rightmost column already shows the first problem.
Gap one: the unit
OpenAI doesn't state the base of the logarithm. The text mentions neither nats nor bits. In practice, “loss” usually means nats. But if 0.76 is in bits, i.e. the base-2 logarithm, the hit rate would be 59 percent rather than 47. explainx.ai and Hacker News user londons_explore also point out the missing detail. For a document meant to justify an industry-wide limit, that's a glaring gap.
Gap two: tokenizer and test corpus
Loss is measured per token. How big a token is depends on the tokenizer, the program that splits text into pieces. If a tokenizer packs more characters into one token, the model has to “guess” more per token. Loss per token goes up, even though the model compresses the text just as well. That's why researchers convert to bits per byte (BPB): they divide the total loss by the number of bytes in the raw text. The Pile paper gives the formula BPB = (Tokens/Bytes) × ℓ / ln 2 and explicitly calls BPB “invariant to different tokenizations.” Huang et al. justify their choice of bits per character the same way: bits per token are “not directly comparable” because of different tokenizers.
That's how other labs do it. DeepSeek reports 0.548 BPB for DeepSeek-V3-Base on the public Pile test set, 0.542 for Llama 3.1 405B and 0.638 for Qwen2.5 72B. This invites a thought experiment: if OpenAI's code had the same token-to-byte ratio as GPT-2 on the Pile (0.29335) and the values were in nats, 0.76 would correspond to roughly 0.32 BPB. That calculation is strictly for illustration. Code is far more predictable than mixed web text, and OpenAI's bytes per token are unknown. The math only shows that the number shifts considerably depending on your assumptions. The text also doesn't say whether the same tokenizer was used over the years.
The test corpus is an even bigger issue. According to footnote 3, the held-out data, meaning data withheld from training, has come from OpenAI's internal code since GPT-3.5, measured against a fixed snapshot. No other lab has this code. How OpenAI keeps it from leaking into training isn't explained. On Hacker News, RAWGOD writes that if you accidentally dump the test set into training, you “can reach perplexity 0.” Pachocki concedes in footnote 4 that code was a poor gauge of the general capabilities of GPT-2 and GPT-3. He chose these numbers anyway because they're the ones OpenAI relies on most today.
Research shows how strongly the numbers depend on the corpus. Du et al. found a capability threshold at a loss of about 2.2 for their own models, not 0.76. The authors explicitly write that loss values from models with different training corpora are “not directly comparable.”
Do capabilities really appear at thresholds?
Pachocki's argument assumes that capabilities show up at specific loss values. That's contested. Schaeffer et al. argued in 2023 that emergence is a “mirage”: abrupt capabilities arise from harsh metrics like “answer fully right or wrong” and vanish with continuous metrics. Du et al. countered in 2024: on MMLU, C-Eval, GSM8K and GSM8K-Chinese, models of three sizes stayed at chance level until the loss reached about 2.2. That tipping point persisted even with continuous measures like the Brier score. Mind you, this holds for a fixed corpus and a fixed tokenizer.
The assumption that compression reflects intelligence does have empirical support. Huang et al. compared 31 open base models on 12 benchmarks and found an almost linear relationship: the correlation between bits per character and benchmark scores was roughly −0.93 to −0.95. The caveats, however, are exactly the ones that matter here. Only base models were studied (“fine-tuned models are no longer general-purpose compressors”). The correlation depends on how well the test corpus matches the domain. And Qwen models had to be excluded from the analysis over suspected test-data contamination. Deletang et al. showed how general “compressor” really is: Chinchilla 70B compresses image patches to 43.4 percent (PNG: 58.5) and speech recordings to 16.4 percent (FLAC: 30.3).
Eliezer Yudkowsky names the real catch on X:
“Not a soul on Earth predicted which capabilities would show up where on this y-axis.”
Pachocki himself writes that the thresholds weren't clearly predictable in advance. That “0.68 to 0.7” is the next critical zone is an extrapolation. X user @scaling01 promptly asked whether the range was chosen arbitrarily.
What the number misses
OpenAI names the biggest weakness itself: chain-of-thought, i.e. spelling out solutions step by step, “does not lower perplexity” but greatly increases capabilities. The same goes for reinforcement learning, meaning training via rewards, as well as tools and agent environments. On Hacker News, qphe95 notes that GPT-4 and o3 have similar perplexity even though o3 was a big leap. ACCount39 points out that RL can even make the measured loss go up. theptip warns that capping pretraining perplexity simply shifts the pressure to post-training. Then there's Goodhart's law: when a metric becomes a target, it stops being a good measure. X user @surlymochan writes that labs would then optimize the number rather than the capability.
The existing alternatives have weaknesses too. The EU AI Act presumes systemic risk above 10^25 floating-point operations (FLOP) in training, and California's SB 53 defines frontier models from 10^26 FLOP. Lennart Heim points out that more efficient algorithms loosen the link between compute and capability. Sara Hooker considers the thresholds simply unfounded. Perplexity at least measures the outcome rather than the effort. Our view: but that only moves the problem from “how much compute was used” to “what was it measured on.”
Then there's the imprecision of reading the values. The figures for the newest models come from a chart. CellCog reads GPT-6.2 Astra at about 0.76 and an unnamed internal model at 0.73. In the body text, OpenAI doesn't name any models. At the same time, Pachocki writes that near the dataset's unknown minimum, small loss gains correspond to “bigger leaps.” So a limit would have to be accurate to within a few hundredths, and all of this is far too fuzzy for that.
What would need to be pinned down
The sources point to a list. The weighting is our own assessment:
- Unit and normalization: bits per byte instead of loss per token, so tokenizer differences drop out.
- A shared, secret test corpus: It should be broad, refreshed regularly and held by a neutral party so it doesn't leak into training. Hacker News user ranyume criticizes that the proposal rests on “a metric only labs can compute.” Charlie O'Neill calls on OpenAI on X to release the test set.
- Measurement protocol: context length, when the measurement happens (base model before post-training) and how synthetic training data is handled all need to be defined.
- Independent verification: Who measures, who checks, and what happens if a limit is crossed? The essay says nothing about this.
- Additional limits for post-training, since RL and chain-of-thought would otherwise be left out entirely. Pachocki only hints at this with “and other measurements.”
Our take
It's unusual and valuable for a leading lab to disclose its internal pretraining progress. Will Depue praised “transparency over competition” on X. But in its current form, the number makes a poor speed limit: the unit, tokenizer, corpus and contamination safeguards are all undisclosed, and the values can only be compared within OpenAI. Critics on X such as @da7_tech also accuse OpenAI of tactical motives. There's no evidence for that. Anthropic, Google DeepMind, Meta and xAI had not responded by the time of publication.
Our view: the idea is a useful starting point, but not yet a measurement standard. It would only become verifiable with shared normalization in bits per byte, a neutrally held test corpus and additional limits on post-training. And even then, Yudkowsky's objection stands: a number can tell you that something is changing, but not which capability comes next.
Sources
- OpenAI: Perplexity and Intelligence (Jakub Pachocki) openai.com
- Unite.AI: OpenAI Proposes Shared AI Development Limits unite.ai
- GeekNews: 퍼플렉시티와 지능 news.hada.io
- Techmeme: Reaktionen auf X techmeme.com
- explainx.ai: OpenAI Perplexity Metric – GPT Log-Loss Curve explainx.ai
- CellCog: OpenAI Shows Its Pretraining Progress, Asks for Limits cellcog.ai
- OfficeChai: OpenAI Wants Perplexity To Become The Industry's Shared Yardstick officechai.com
- MLQ.ai: OpenAI publishes its models' perplexity scores mlq.ai
- dev.to: OpenAI wants pretraining loss to become a shared safety yardstick dev.to
- arXiv: Compression Represents Intelligence Linearly (Huang et al.) arxiv.org
- arXiv: Language Modeling Is Compression (Delétang et al.) arxiv.org
- arXiv: Are Emergent Abilities of LLMs a Mirage? (Schaeffer et al.) arxiv.org
- arXiv: Understanding Emergent Abilities from the Loss Perspective (Du et al.) arxiv.org
- The Pile: An 800GB Dataset (Definition Bits pro Byte) ar5iv.labs.arxiv.org
- Hugging Face: DeepSeek-V3-Base (Pile-test BPB) huggingface.co
- arXiv: Training Compute Thresholds (Heim) arxiv.org
- arXiv: On the Limitations of Compute Thresholds (Hooker) arxiv.org
Ask the AI assistant
Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.
The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.
Comments
What do you think? Already tried it? Write it here, no sign-up needed.
Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.