llama.cpp: Intel XMX overhaul doubles speculative decoding speed on B70
A pull request for llama.cpp reworks how Intel GPUs' XMX matrix units are used. One user measures nearly twice as many tokens per second with speculative decoding on an Arc B70 using SYCL.
What's going on
On the Reddit forum r/LocalLLaMA, a user reports significant speed gains from a pull request (#29864) in the llama.cpp project, a widely used piece of software for running language models locally. The contribution comes from "budiony" and reworks the use of XMX matrix units, the compute units in Intel GPUs that specialize in matrix math. The user tested on an Intel B70 with the SYCL backend (Intel's programming interface for GPUs).
Test setup
The test used the "Muse Glimmer" model with 30 billion parameters in Q6_K quantization (weights are compressed to roughly 6 bits) with DFlash2. The focus is speculative decoding: a small draft model guesses several tokens ahead, and the large model checks them in one batch. If the predictions hit, that saves time. Nine test requests, including code, translation, summarization and math, each generated around 192 tokens.
Results
| Task | Before (tok/s) | After (tok/s) |
|---|---|---|
| code_python | 38.0 | 73.0 |
| code_cpp | 36.2 | 68.5 |
| explain_concept | 41.2 | 70.7 |
| summarize | 50.1 | 85.4 |
| qa_factual | 37.0 | 60.3 |
| translation | 45.1 | 87.7 |
| creative_short | 37.1 | 53.6 |
| stepwise_math | 53.2 | 77.6 |
| long_code_review | 38.4 | 64.2 |
Total time fell from 46.86 s to 26.68 s. Accepted draft tokens rose from 1085 to 1272, while total drafts grew from 1383 to 3504.
Why the acceptance rate drops
The user points out that because verifying tokens is now much cheaper, they raised the n-max parameter (the maximum draft length). That means more tokens get proposed, and the acceptance rate falls from 78.5% to 36.3%. According to the post, this isn't a step backward, since overall throughput in tokens per second is higher.
Our take
The numbers come from a single user with one piece of hardware and one model, and there's no independent confirmation. Also, going by its title, the pull request is still just a proposal, and it's unclear whether or when it will be merged into llama.cpp. Still, for owners of Intel Arc GPUs, it's a promising sign that the SYCL backend has plenty of performance left to give.
Sources
- r/LocalLLaMA reddit.com
Ask the AI assistant
Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.
The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.
Comments
What do you think? Already tried it? Write it here, no sign-up needed.
Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.