1,000 tokens per second? What fast diffusion language models really deliver

What this is about

Almost every chatbot writes like a typewriter: one token (a word fragment) after another, each one depending on everything before it. The technical term is autoregressive. Since June 2026, several models have arrived that work differently. Diffusion language models start with a block of placeholders and fill it in over several passes, all at once, much like image generators carve a picture out of noise. They include Google's DiffusionGemma (June 10), Inception's Mercury 3 (October 19) and DiffuSpace's Clover-35B-A3B-Preview (late October). Then there's Next-Concept Modeling (NCM) from Stanford and Together AI (October 9), which generates text coarsely first and then refines it step by step.

All of them sell speed: four times, eight times, up to 31 times faster. We looked into where these numbers come from. Our thesis: the advantage mostly holds for single requests on big datacenter GPUs and against weak baselines. It shrinks once many requests run at the same time, once you run on ordinary hardware, or once classic models get speculative decoding. With that technique, a small helper proposes several tokens at once and the large model verifies them in a single pass.

We weren't able to run our own benchmark of DiffusionGemma vs. Gemma 4 on a consumer graphics card for this article. We rely on the technical reports, public measurement data and community reports.

The promises at a glance

The table shows the core problem: every number was measured under different conditions, and almost none has been checked by third parties.

DiffusionGemma: Racing its own sibling

Google's technical report is the most thorough document of the four. It discloses the setup: one H100, FP8 precision, batch size 1 (a single request), 4,096 input tokens and 1,024 output tokens.

Three things stand out.

First, the baseline. The "highly optimized" Gemma 4 variant with MTP (multi-token prediction, a form of speculative decoding) averages just 1.4 tokens per forward pass. The report itself cites typical values of 3 to 6 for speculative decoding. How much more is possible shows up with DFlash on the similarly built Qwen3.6-35B-A3B. According to z-lab's model card, speed on a B200 with a single request rises from about 307 to as much as 1,116 tokens/s, a 2.2x to 3.6x gain. Our back-of-the-envelope math, not a measurement: if Gemma 4 benefited similarly, it would land at roughly 470 to 740 tokens/s. DiffusionGemma would then be two to three times faster, not nearly five.

Second, the load. According to the report, DiffusionGemma only stays ahead up to about 32 concurrent requests; beyond that, classic Gemma 4 with MTP wins. In its blog post, Google itself writes that parallel decoding brings little benefit for cloud services handling many requests and can even raise costs. The authors stress that they haven't optimized batch processing yet and call an analysis under real-world load "future work".

Third, quality. DiffusionGemma trails its sibling on almost every benchmark, most clearly in math (AIME: 69.1 vs. 88.3) and agentic tasks (Tau2). Its answers are also shorter: 4,001 tokens on average in thinking mode, compared with 7,207 for Gemma 4 with MTP. That cuts wait times further, but according to the report it also gives up some of the benefits that longer "thinking" brings.

Data on consumer hardware is thin. The figure of over 700 tokens/s on an RTX 5090 comes from Google. The Nvidia blog lists numbers for the H100 (1,000), DGX Spark (150) and DGX Station (2,000), but none for the 5090. Unsloth cites over 2,000 tokens/s on an RTX 6000, without stating test conditions. Reports from Apple Silicon are sobering. A tester at XDA found DiffusionGemma no faster than Gemma 4 on an M4 Pro. In a llama.cpp issue, one user initially measured 6 to 28 tokens/s on an M3 Max, and 48 to 61 tokens/s after their own optimizations. Google itself warns that unified-memory machines like Macs won't see the same speedup. We couldn't find a properly documented benchmark against Gemma 4 on an RTX 4090 or 5090, with one and with 16 requests.

How parallel is "parallel"?

DiffusionGemma writes in blocks of 256 tokens. On average, though, it commits about 20 tokens per pass, not 256. Thanks to adaptive stopping, it needs around 12 passes per block. The blocks themselves are still generated one after another.

An interpretability study by Asaria, Salomone and Gandhi traces when the model commits to which token. It tested 686 prompts across six task types. The result: decoding is "neither parallel nor block-wise sequential". There's a partial left-to-right tendency that's weak at the token level and grows stronger at coarser granularity. JSON gets filled in almost in arbitrary order. Google's report confirms the pattern in Appendix G.2: on tasks with strict dependencies, the model resolves the logic "predominantly from left to right" and needs more passes. Our take: the more a text builds on what came before, such as math derivations or longer arguments, the less the model can work in parallel.

Mercury 3: 1,068 in the lab, around 200 in practice

Inception quotes 1,068 tokens/s "on Nvidia GPUs". Model size, batch size and prompt length aren't specified. Independent measurements paint a different picture. According to an analysis by eesel.ai, the weekly median on OpenRouter was 221 tokens/s, and the slowest 10 percent of requests hit 79 tokens/s or less. OpenRouter routes all requests directly to Inception. On Hacker News, testers reported 300 to 400 tokens/s, while another saw about 160 tokens/s on a large coding job. Simon Willison called the speed "a bit disappointing" there. Several users also complained about a long time to first token, the wait before the first word appears.

These numbers reflect real-world usage with arbitrary reasoning effort and fluctuating server load, so they aren't directly comparable to a lab measurement. But it's exactly what users experience. Artificial Analysis doesn't list Mercury 3 yet, only Mercury 2 (most recently 292 tokens/s, since discontinued) and Mercury 2.5 (557 tokens/s). According to a Digg summary, Artificial Analysis also cautioned that the scores Inception published for AA-Omniscience and GDPval-AA weren't produced using its methodology. Google's team estimated Mercury 2 at an average of 600 tokens/s via OpenRouter, while Artificial Analysis measured a median of around 987 tokens/s at the time. The spread shows how heavily such numbers depend on hardware, load and test data.

Clover and NCM: Against strong competition, the gap shrinks

DiffuSpace derived Clover from the classic Qwen3.6-35B-A3B and at least compares it against a strong opponent. According to the Hugging Face blog, Clover reaches 515 tokens/s on agent tasks on an H200, while a Qwen model with the speculative method DFlash2 hits 365. That's a factor of roughly 1.4, not 4. The peak of 680 tokens/s comes from a GB300 with a single request on internal tasks. On an RTX 5070 Ti, DiffuSpace cites 101.8 vs. 48.5 tokens/s without saying exactly which model it compared against. Locally, the speed is still theoretical anyway: according to AtomicChat's GGUF page, llama.cpp support merged on November 2 covers only the classic token-by-token mode. The unmerged parallel mode delivered 0.6 to 0.7 tokens/s on an A100, while normal mode managed about 53.

For NCM, the 10x to 31x speedups are genuinely measured: on an H100 at batch 16. But according to the paper, NCM only goes up against other diffusion models such as DUO or AURORA-LM, and the models have around 100 million parameters (up to 725 million in the scaling test). Classic models only appear when counting forward passes, not in the timing measurements. The authors acknowledge limitations themselves: even short texts go through the full hierarchy, the tokenizer is trained separately, and errors occasionally creep in at the transitions between levels.

What researchers still see as unfinished

New papers from November show where things get stuck. SpecCTMC describes the fundamental dilemma: if tokens are sampled independently in parallel, the dependencies between them are lost. Exact sampling, on the other hand, requires one model evaluation per token. Measured Coupling measures which tokens can safely be committed together, at an average cost of 1.1 points of accuracy. Mix-FT targets training, because it isn't suited to a small number of parallel steps. Taken together: so far, writing more in parallel usually costs quality.

Our assessment

Our thesis largely holds. The speed advantage of diffusion language models is real, but it applies under specific conditions. It's biggest for a single request on a fast datacenter GPU and against classic models without good speculative decoding. Against a strong method like DFlash, the available numbers suggest it shrinks to roughly 1.4x to 3x. With many concurrent requests it reverses, and on Macs it hasn't been demonstrated so far. On top of that come quality losses that are significant in math and agentic tasks.

To be fair: Google discloses its test setup, weaknesses and the tipping point at 32 requests. That's more than Inception offers for Mercury 3. For local single users, code completion and structured output, the approach can make sense.

The most important thing is still missing: independent, identically designed benchmarks. That means the same graphics card, the same prompts, one and 16 requests, short and long answers, and the classic model with its best speculative decoding as the opponent. Until those exist, "1,000 tokens per second" is a lab figure, not an everyday one.