“Lossless,” but not quite: why speculative decoding changes your LLM's answers

What this is about

Speculative decoding is the most popular speedup for local language models. A fast helper guesses several words ahead, the big model checks all the proposals in a single pass and only accepts the ones it would have picked itself. Because the big model always has the final say, the technique is considered lossless. Nvidia's Jetson AI Lab, for example, writes that with deterministic output, the result matches what the target model would produce without speculation.

Now that Qwen3.8-27B ships with a so-called MTP head (multi-token prediction, an extra module trained alongside the model that guesses several upcoming words) right inside the GGUF file, a single flag in llama.cpp is all it takes: --spec-type draft-mtp. Several sets of community measurements now show that the text mostly does not stay the same, not even at temperature 0, where the model is supposed to always pick the most likely word. We explain why this happens, why it isn't a quality problem, and who it hurts anyway.

What the measurements show

The most thorough single measurement comes from Italian developer Maurizio Fonte. He had Qwen3.8-27B solve 50 Python tasks on an RTX 5060 Ti with 16 GB, once with and once without the MTP head, at temperature 0, with a fixed seed and thinking mode turned off. Everything else stayed the same. The result:

• Median speed rose from 29.3 to 56.8 tokens per second (a factor of 1.94). All 50 tasks ran faster.
• Only 7 of 50 answers were identical character for character. The differences ranged from a renamed variable to a different algorithm.
• With the head, 34 tasks were solved; without it, 33. Five tasks only worked with the speedup, four only without it. Fonte doesn't see a meaningful difference there.

Other measurement series paint the same picture:

Source	Setup	Differing outputs
Fonte (blog)	Qwen3.8-27B, RTX 5060 Ti, MTP	43 of 50
Reddit r/LocalLLM	Qwen3.8-27B, RTX 3090, llama.cpp b12024, 4 and 7 drafts	113 of 282
thc1006 (preregistered study)	Qwen3.8-27B UD-Q4_K_XL, RTX 3090	92–100% at 1,600 tokens, 76–80% already at 400 tokens
makoni (issue #25618)	Qwen3.8-27B, RTX 5080	4 of 9, 0 of 9 with patch
rikvanriel (issue #31472)	Qwen3.5-35B-A3B, even on the CPU	7 of 8 pairs

The key point: this isn't random noise. According to the Reddit author, llama.cpp without the speedup produces exactly the same bytes from run to run, and two sessions with the speedup also match each other. thc1006 likewise found that the first divergence occurs at exactly the same position in every run. So the speedup reproducibly delivers a different text. Short answers aren't exempt either: according to the Reddit post, 72 of the 113 changed answers were shorter than 1,000 tokens.

Why the math checks out and the text still flips

The core issue is a quirk of computer arithmetic. Floating-point numbers are stored with limited precision. That's why the result of a sum depends on the order of the additions. Thinking Machines Lab illustrates this in a widely cited blog post: (0.1 + 10²⁰) − 10²⁰ yields 0, whereas 0.1 + (10²⁰ − 10²⁰) yields 0.1.

With speculative decoding, the big model checks several words at once. To do that, the software uses different compute kernels (the small programs that do the actual number crunching on the GPU) than it does when generating one word at a time. At the end of October, llama.cpp maintainer danbev broke this down in detail for CUDA in issue #25618:

• In the matrix-vector multiplication for Q4_K and Q6_K weights, a single token is computed with 8 warps (groups of parallel compute units), while a batch of 8 tokens uses 2. That means partial sums are added in a different order. The difference is about one unit in the last decimal place.
• In Flash Attention (an optimized method the model uses to take earlier words into account), a single token goes through the VEC kernel with 32-bit values, while a batch goes through the MMA kernel with 16-bit values.

When danbev forced both paths to match as a test, the outputs were bit-identical. That's not a fix yet, because the MMA kernel slows down normal single-token generation.

Most of the time, tiny deviations like these change nothing. They become dangerous at a near tie, when two candidate words are almost equally likely. Then the last decimal place decides, and from that word on, the model writes a different continuation. Thinking Machines Lab documents a case like this with Qwen3-235B: of 1,000 answers at temperature 0, the paths split at token 103. 992 wrote “Queens, New York,” 8 wrote “New York City.” With batch-invariant kernels, all 1,000 were identical.

The MarginGate paper measured how rare these tipping points are. Across five models, only 0.3 to 1.3 percent of steps flipped when a request ran in a batch rather than alone. That sounds like very little. Our back-of-the-envelope math: at 0.5 percent per token and an answer of 1,000 tokens, the chance that the answer stays completely untouched is only about 0.995¹⁰⁰⁰ ≈ 0.7 percent. That's a gross simplification, since the steps aren't independent of each other. But it explains why long answers almost always diverge.

Experts call the missing property batch invariance: the result for one row shouldn't depend on how many rows are computed at the same time. As the issue shows, this mainly affects quantized models. Several testers stayed bit-identical with BF16 weights. N-gram speculation looked clean in the original report, but not in the sudoingX/qwen38-mtp repo on an AMD Instinct MI210.

The finding hardly anyone knows about: it happens without the speedup too

Contributor Rik van Riel sums it up in the issue:

The bug isn't in the speculative path at all.

In fact, on unmodified code and with no speculation at all, danbev found that just changing the ubatch size (-ub 2 instead of -ub 512, i.e., how many tokens llama.cpp processes internally at once) alters the greedy output. He doesn't give percentages. On the CPU, by contrast, van Riel found no ubatch effect in his case, so the problem depends on the backend.

Numbers from user aditi-p31 are even more striking. On an RTX 4090 with Qwen2.5-7B and no speculation, she compared answers that hit the prefix cache (where an already computed conversation prefix is reused) with fully recomputed answers. Differing outputs:

Weights	Differing episodes
F16	37.4%
Q8_0	59.5%
Q4_K_M	82.5%
Q3_K_M	77.1%

She stresses herself that this isn't the same bug and that she hasn't investigated it at the kernel level. In practice, though, it means this: if you have llama-server answer the same conversation once fresh and once from the cache, you often won't get the same text.

Quality holds, reproducibility suffers

The good news: no test so far has found worse answers. Fonte's came out 34 to 33. The Reddit author got 229 correct out of 282 questions with llama.cpp. His custom engine, which produces no diverging answers at temperature 0, scored 230 and 229. One tester in the issue puts it this way:

The damage is plausible but different text, not garbage.

What takes the hit is comparability. If you compare two quantizations, two llama.cpp versions or two prompts and have the speedup configured differently, you're partly measuring the kernels rather than the model. The Reddit author nails it: if two draft settings produce different outputs, at least one of them doesn't match what the model would say without speculation. Tweak the drafts and you're tweaking the answers. That affects local benchmarks, evals and regression tests, which are exactly the kinds of numbers we regularly cite too.

Some context on the “0 instead of 113”: the Reddit project qwen38-megakernel is a custom CUDA engine built solely for Qwen3.8-27B on a 24 GB card (tested on an RTX 3090, licensed under AGPL v3). It stays bit-identical with itself at 0, 4 and 7 drafts. Compared with llama.cpp, however, it agrees on the most likely token at only 98.74 percent of positions. The README doesn't explain exactly how the engine achieves this.

Where llama.cpp stands

Issue #25618 has been open since July and still carries the “bug-unconfirmed” label. These are the fixes attempted so far:

• PR #26358 (Vulkan) helped in one case but was closed unmerged in August.
• A patch by Apoze fixes the MTP divergence for Qwen3.8-27B. However, it cut normal speed at 67,000 tokens of context from about 29 to 22 tokens/s. makoni's modified version leaves normal speed unchanged but makes MTP roughly 12 percent slower at long context and only covers up to 3 draft tokens.
• PR #31504 by van Riel makes batched matrix multiplication on the CPU bit-identical, with no measurable speed loss according to him. The PR is still awaiting review, and van Riel says a second divergence remains in 4 of 8 test prompts.

Whether bit-exact batch invariance will become a project goal at all, for example as an opt-in mode, is still open.

What the big servers offer and what it costs

Engine	Switch	Measured cost
SGLang	--enable-deterministic-inference (FlashInfer, FA3, Triton)	24–55% slower offline, 34% on average (LMSYS, Qwen3-8B, H200); up to 56% according to the LLM-42 paper
vLLM	VLLM_BATCH_INVARIANT=1 (beta, Nvidia compute capability 8.0 and up, AMD Instinct, Intel XPU)	no official figure, docs warn of performance loss
Thinking Machines prototype	batch_invariant_ops	26 s → 42 s for 1,000 sequences (roughly 62% more time)

Neither set of docs says whether the deterministic mode works with speculative decoding. Research approaches like LLM-42 and MarginGate aim to cut the cost by re-checking only the uncertain steps. With little deterministic traffic, LLM-42 trails normal operation by only 1 to 8 percent. These approaches aren't available for local tools yet, though.

Our take

“Lossless” holds for quality, based on everything measured so far. The output isn't bit-identical, and that's not down to the idea of speculation itself but to kernels that compute batches differently from single tokens.

Our assessment: for chat and coding help, leave the speedup on; it cuts your wait time roughly in half. Turn it off, or keep every setting identical, as soon as you're comparing: benchmarks, A/B tests of prompts or quantizations, regression tests and debugging. Fonte also recommends checking the outputs of both configurations by hash. That alone isn't enough, though: based on the findings in the issue, the ubatch size, cache type, prefix cache usage and the exact build version all belong in your test description too. Most of the numbers here aren't solid yet, since they're one-off measurements with few prompts, some with AI-written analyses. We didn't run a test of our own for this article.