inferzero: How a one-person project turns Intel laptops into AI machines

Millions of laptops with a Core Ultra chip have an Arc GPU that local AI has largely ignored. The free inferzero engine reads long prompts on these machines many times faster than llama.cpp. That barely helps once the model starts writing its answer, so your choice of model matters.

Analysis Nov 17, 2026 10 min read Original analysis by our editors
Illustrative image, AI-generated

What this is about

A huge number of current laptops run an Intel Core Ultra: Meteor Lake, Arrow Lake, Lunar Lake or Panther Lake. Every one of these chips has an Arc GPU that shares system memory with the CPU. Most people run local AI with llama.cpp, the software behind Ollama and LM Studio. On Intel graphics, though, it gets only a fraction of what the hardware can do. The inferzero project by developer rizagp wants to change that. It's an engine written from scratch that sits directly on Level Zero, Intel's programming interface for low-level access to the GPU.

Our thesis: inferzero makes Intel laptops much faster at reading in long texts. When it comes to writing the answer, not much changes. If you want to benefit, you have to pick the right model. All performance figures below come from the developer or from individual testers on GitHub. Nobody has independently re-measured them yet, and we didn't run our own benchmarks.

What inferzero does differently

According to the README, inferzero reads GGUF files directly, the standard format for shrunk-down (quantized) models. It supports IQ1 through Q8_0 as well as MXFP4. It exposes APIs compatible with OpenAI, Anthropic and Ollama, plus a web UI at 127.0.0.1:8090 and an integration with the Pi coding agent. On top of that you get image and audio input, voice conversations, PDF text recognition and remote access via Cloudflare Tunnel. Supported models include Qwen3.5, Qwen3.6 and Qwen3.8, Gemma 4, DeepSeek-V4.1-Flash, and Whisper and Qwen3-ASR for speech recognition. The license is Apache 2.0. The project stresses that it's independent and not backed by Intel.

At its core are hand-written compute kernels. Older Intel graphics such as Meteor Lake lack XMX units, the dedicated matrix engines comparable to Nvidia's Tensor Cores. They can, however, use DP4a to run many small integer multiplications at once. According to the project docs, the developer reached 6 to 9 TOPS (trillion operations per second) this way. llama.cpp's Vulkan and SYCL backends managed about 1.5 TOPS. On newer chips with XMX (Lunar Lake, Panther Lake, Arc B-series cards), inferzero uses those units as well. Then there's expert streaming: with Mixture-of-Experts models, which activate only some of their specialist modules per word, modules that don't fit in memory are loaded from the SSD on demand. And there's speculative decoding via an MTP head or a DFlash2 draft model: a small helper guesses up to seven tokens ahead, and the large model checks them in a single pass.

The project checks accuracy using KL divergence, a measure of how far the probabilities deviate from an exact reference computation (0 means identical). The target is below 0.015.

The numbers: fast reading, slow writing

Testing was done on a Core Ultra 5 125H with 32 GB of RAM. The developer measured the llama.cpp comparison in October on the same laptop with the same files. We calculated the speedup factors from his numbers:

Test (125H) inferzero llama.cpp Vulkan llama.cpp SYCL Factor
Qwen3.5 4B, 2K prompt 911 tok/s 242 337 2.7–3.8×
Qwen3.5 9B, 2K prompt 402 tok/s 151 189 2.1–2.7×
Qwen3.8 27B, 2K prompt 113 tok/s 23 25 4.5–4.9×
27B, 34K code prompt to first token 327 s 2,418 s 1,624 s 5.0–7.4×
27B, generation afterwards 3.7 tok/s 2.1 1.1 –

So the "four- to fivefold" speedup applies to the 27B. With small models the gap is smaller, with very long prompts it's bigger. On perplexity, a measure of how well a model predicts text (lower is better), inferzero comes out slightly ahead: 6.056 versus 6.141 for the 4B and 4.274 versus 4.311 for the 27B.

The current table from November 9 (125H, 256 output tokens):

Model 8K prompt 131K prompt Generation With drafting
Qwen3.6 35B-A3B (MoE) 851 tok/s, 9.6 s 6.7 min 21.6 tok/s 38.1
Gemma 4 26B-A4B (MoE) 232 tok/s, 35 s 50 min 19.8 tok/s –
Qwen3.8 27B (dense) 135 tok/s, 61 s 27 min 5.1 tok/s 12.9
Qwen3.5 9B 487 tok/s, 17 s 7.8 min 11.8 tok/s 21.6
Gemma 4 E4B 707 tok/s, 12 s 9.8 min 19.1 tok/s 41.8
Qwen3.5 4B 896 tok/s, 9.2 s 6.0 min 21.0 tok/s 35.8

The 35B-A3B reads an eight-page report in about ten seconds. The dense 27B needs a minute for the same document, and almost half an hour for a large codebase of 131,000 tokens.

Why memory bandwidth is the bottleneck

For every token it generates, a dense model has to read practically all of its weights from memory once. The docs cite a measured 75 to 77 GB/s for the 125H, against a theoretical 102. The 27B in the quantization used reads about 11.3 GB per token. Our back-of-the-envelope math: 77 divided by 11.3 gives a ceiling of just under 7 tokens per second, no matter how good the software is. The developer himself puts it at about 6.5. The measured 5.1 is already close to that limit.

An MoE model like Qwen3.6 35B-A3B activates only about 3 billion of its 35 billion parameters per token, so it has to read far less. That's why it generates four times faster than the 27B on the same laptop, even though it's bigger overall. Drafting helps mainly with code. According to the docs, it does much less for prose, because the helper models were trained mostly on English and code.

The November 17 update: Panther Lake

Since release 2026.11.17.01, inferzero uses the same compute paths on Panther Lake (Arc B390) and Lunar Lake as on the dedicated Arc cards. According to the release notes, the 27B's 4K prompt speed on a B390 rose from about 260 to about 550 tokens/s, with a KL divergence of 0.0018. You can switch off the new paths with ARC_H8=0 and ARC_ATTN_FX=0.

All the figures in Issue #1 come from a single tester. He's using a Core Ultra X7 358H with 128 GB of socketed RAM, which is slower than the soldered memory you usually find. The "~590 tok/s" quoted on Reddit (measured at 583 to 592) needs 16 GB of permanently pre-prepared weights. According to release 2026.11.17.02, that setting only becomes the default once the GPU's memory share is 64 GiB or more. On a typical 32 GB laptop, expect something closer to 550. Generation with the 27B stayed at 5.8 to 5.9 tokens/s. The Reddit headline's claim that a 12 GB Arc B580 can now run the 27B is something we couldn't back up with measurements anywhere in the repo, docs or release notes.

Which model on which chip

Chip Notes Measured inferzero figures Our recommendation
Meteor Lake (e.g. 125H) no XMX, ~77 GB/s see table above 32 GB: 35B-A3B, 16 GB: Qwen3.5 4B or Gemma 4 E4B
Arrow Lake H (Arc 140T) XMX, ~81 GB/s 27B: 357 tok/s (4K), 6.1 tok/s (14.6 with MTP) 35B-A3B, MoE figures still missing
Lunar Lake (Arc 140V) XMX, 16 or 32 GB soldered none published 32 GB: try the 35B-A3B, 16 GB: small models
Panther Lake (Arc B390) Xe3 with XMX 27B: 550–590 tok/s, 5.9 tok/s (16.8 with MTP); 35B-A3B: 1,182 tok/s on a 2.5K prompt, 24–34 tok/s 35B-A3B, test llama.cpp for comparison
Arc Pro B60 discrete graphics card 27B: 1,522 tok/s, 22.7–23.3 tok/s (51.6 with MTP); 35B-A3B: 3,536 tok/s, 82.4 tok/s (111.9) 27B becomes usable day to day
Arc Pro B70 discrete graphics card 27B (IQ3_S): 2,772 tok/s (4K), ~32 tok/s; 35B-A3B: 5,612 tok/s (8K), 95–98 tok/s both work well

Our take: on every laptop chip, the 35B-A3B is the best choice, provided you have 32 GB of RAM. Dense 27B models only make sense on integrated GPUs if you don't mind waiting for the answer. You can combine multiple cards with --gpus N. With four B60s, the project says the 27B generated 35.7 to 53 tokens/s. However, one tester on a board with a slow x4 link lost half the prompt-processing speed (Issue #19).

Reality check: llama.cpp, OpenVINO, IPEX-LLM

Independent tests show that the lead depends on the chip and the model. On Panther Lake, llama.cpp with Vulkan hit 633 tokens/s on prompt processing (512-token prompt) and 33 tokens/s generating with the 35B-A3B, according to grigio.org. A second test with a newer version came in at 758 and 30.9. With Intel's OpenVINO, the NoLlama project's benchmark collection reached 41 tokens/s generating on the same chip type. So for MoE models on new chips, inferzero isn't ahead on generation, though it probably is on prompt processing. The test lengths differ, though.

On the B70, too, the gap is smaller than the laptop numbers suggest. Jonathan Mann measured 76 tokens/s with the 35B-A3B and 20 with the 27B (Q4_K_XL) using llama.cpp and Vulkan after a Mesa update. He warns that older benchmarks were off by a factor of 7 after just two months. That applies to inferzero's llama.cpp comparison too, which dates from October. For the B70 there's also a similar engine under the AGPL license, urakozz/intel-arc-inference-server. It reports 9,362 tokens/s prompt processing with the MoE model Ornith 35B-A3B. inferzero plans to borrow ideas from it.

Intel's own IPEX-LLM library, on the other hand, has been archived since January 28, 2026. According to the notice on GitHub, it's no longer supported and has "known security issues". OpenVINO is still maintained, but in grigio.org's test it couldn't run the 35B-A3B as a llama.cpp backend.

What you need to try it

You need a Core Ultra laptop with Arc graphics and either Windows 11 (64-bit) with a current Intel driver or Ubuntu 24.04 or later with Intel's GPU runtime. According to the README, 16 GB of RAM is enough for 4B and E4B models, while 12B to 35B models need 32 GB. On Windows, download the roughly 10 MB ZIP from the Releases page, run zero-launcher.cmd and pick a model. On Linux, run ./zero-launcher.sh --check. The Linux version is only a few days old, and some features are still missing.

The bottom line

inferzero shows how much performance sits untapped in Intel graphics that llama.cpp hasn't been using. That's especially true for older Meteor Lake laptops without XMX, where llama.cpp is at its weakest. If you have it summarize long documents or code there, you wait seconds instead of minutes. But even the best software can't beat memory bandwidth.

There are caveats. The project is the work of one person, Riza Guntur Prakoso according to the Level1Techs forum, and has 815 commits. Some days see five releases. Some replies on GitHub are signed as an "overnight assistant" for the maintainer, so an AI agent is apparently pitching in. Testing was done mostly on a single 125H, and the Panther Lake, B60 and B70 figures come from individual users. Our take: if you like tinkering and own an Intel laptop, it's worth a try, especially with the 35B-A3B. If you just want something that runs reliably, stick with llama.cpp and Vulkan for now.

#intel arc#local ai#llama.cpp#inference engine#open source#moe

Ask the AI assistant

Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.

Connecting …

The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.

Comments

What do you think? Already tried it? Write it here, no sign-up needed.

  1. Loading comments …

Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.