inferzero runs big LLMs fast on Intel laptops, no discrete GPU needed

What it's about

If you own a recent Windows laptop with an Intel Core Ultra chip, you already have a fairly capable integrated GPU: "Arc Graphics." It has seen little use for local AI models so far, because the popular tools are mostly optimized for Nvidia GPUs and Apple silicon. The open-source project inferzero wants to change that. It comes from independent developer rizagp and was called "arcinferzero" until November 10, 2026. Intel isn't involved in the project and doesn't officially support it.

inferzero is an inference engine, meaning the program that runs an already trained language model. The developer wrote it from scratch in C++ and tailored it specifically to Intel's integrated GPUs. Supported generations are Xe-LPG (Meteor Lake) plus Xe2 and Xe3, which covers Lunar Lake and Panther Lake with the Arc B390. Everything was built and tuned on a Core Ultra 5 125H with 32 GB of RAM. That laptop has no discrete GPU and no XMX units, the dedicated matrix engines that only appear in newer Intel chips. The code is on GitHub under the Apache 2.0 license.

How the engine gets so fast

The key trick: every matrix multiplication, which is where a language model spends most of its compute, runs on the iGPU's DP4a units. These compute dot products of 8-bit integers (int8) very efficiently. The engine talks to the hardware through Level Zero, Intel's low-level programming interface for GPUs, and the compute kernels are compiled with OpenCL. Where XMX units are available, the engine uses them for prompt processing, but only after they pass a startup cross-check against the DP4a kernels.

The second building block targets Mixture-of-Experts (MoE) models. In these models, only a small subset of the sub-networks, the "experts," is active for each token. inferzero keeps only the experts it currently needs in GPU memory. While the GPU is still working on the current layer, it reads the missing experts for the next layer from the SSD. That lets you run models larger than your available memory. One example is Qwen3.8 Flash Next, with 52 GB of experts, on a laptop with 32 GB of RAM.

On top of that comes speculative decoding. A small add-on guesses several tokens ahead, and the large model verifies them in a single pass. inferzero uses the model's built-in MTP block (Multi-Token Prediction) and a "DFlash2" drafter, which guess up to seven tokens per step. According to the developer, the output text is identical to normal decoding. With this, the 35B model generates Python code at around 50 tokens per second.

The project puts special emphasis on accuracy. Every model is compared against an independent CPU implementation running in full 32-bit precision, using KL divergence as the metric. It shows how far the next-token probabilities drift from the reference; 0 means identical. Every change has to stay below 0.015. The outputs are also deterministic: the same prompt yields bit-identical values every time.

The numbers

All figures come from the developer himself and were measured on the Core Ultra 5 125H with 32 GB of RAM (as of October 25, 2026). "Prompt" means how fast the input is read in. "Generation" means how fast the response is produced.

The comparison with llama.cpp, the most widely used tool for local models, is striking:

Perplexity measures how "surprised" a model is by a test text. Here inferzero edges out llama.cpp, so the faster computation doesn't make the results worse. As an experimental option, the NPU, the AI accelerator inside the processor, can pitch in too (flag --npu on). On Qwen3.8 27B, that raises prompt speed from 133.5 to 153.8 tokens/s, though the deviation from the reference grows slightly.

More than an engine: web UI, voice, agents

inferzero comes with a server. It offers endpoints compatible with OpenAI, Anthropic and Ollama, so many existing tools can use it straight away. A single command also hooks up the coding agent Pi. Prompt and session caches make sure only the new part gets processed when you send a new message. For example, after a 20,000-token tool result, a one-line follow-up to Gemma 4 E4B takes 0.8 seconds instead of 38. With --slots 2, two Pi sessions can share the same model at once. When both are generating at the same time, total throughput goes up, from 20.2 to 24.7 tokens/s on the 35B model, for example.

The web UI can search the web and read pages. You can also talk to the model by voice: Gemma 4 E4B and 12B understand audio natively, and for other models Qwen3-ASR handles transcription (--whisper auto). The response is read aloud sentence by sentence, and you can interrupt it. There are also dedicated tabs for transcripts and meetings. For long documents, the digest_file function reads them section by section and takes notes, and answers cite page numbers. PDFs and images are processed by Xiaomi-OCR-0, which converts them to Markdown and keeps tables intact. Through a password-protected Cloudflare tunnel, you can reach the system from your phone as well.

Supported model families are Qwen3.5, 3.6 and 3.8 (0.8B to 27B dense, 35B-A3B MoE, Flash Next), Gemma 4 (E4B, 12B, 26B-A4B, 31B in Google's QAT Q4_0 version), plus Whisper large-v3 and Qwen3-ASR for speech recognition. GGUF files load as-is, from IQ1 to Q8_0.

Discrete Arc cards too

Intel's discrete Arc B-series cards (B580, Pro B60, Pro B70) are supported experimentally. One user measured the following on an Arc Pro B60:

• Qwen3.6 35B-A3B: 3,232 tokens/s for the prompt, 78.8 tokens/s for generation and 113.7 with MTP.
• Qwen3.8 27B: 23.2 tokens/s for generation and 49.3 with MTP.

With --gpus N, you can split a model across multiple cards using tensor parallelism. The 27B model reaches 35.7 tokens/s on two GPUs, 43 on three and 53 on four.

What you need to try it

• Operating system: Windows 11 (64-bit) with an up-to-date Intel graphics driver. The project doesn't mention Linux or macOS, and Macs are out anyway since you need an Intel Arc GPU.
• Processor: an Intel Core Ultra with integrated Arc graphics. You don't need a discrete GPU.
• RAM: 16 GB for the small models (4B, Gemma 4 E4B) and 32 GB for the 12B to 35B models.
• Storage: a fast SSD with enough room for the model files. With the large MoE models, experts are read from it while you use them, and Flash Next alone takes up 52 GB.
• Installation: Grab the 6 MB package from Releases. Unzip it, run zero-launcher.cmd, pick a model (the right one is preselected) and click "Download and start." The UI then opens at http://127.0.0.1:8090/. You don't need Python or a compiler. On first launch, Windows may show a warning; click "More info" and then "Run anyway."

Our take

inferzero shows how much performance is hiding in ordinary office laptops when software is tailored to the hardware. The lead is most pronounced when reading long inputs, which is exactly the part that eats the most time with coding agents and long documents. Running 35-billion-parameter MoE models on a 32 GB laptop via SSD streaming is remarkable for this class of device.

You should know the limitations, though. All figures come from the developer himself or from individual users; there are no independent tests yet. The "851 tokens/s" claim for the 35B model from the Reddit announcement doesn't appear in that form in the README, which lists 604 tokens/s for a 32K prompt. Generation speed for dense models is still slow, at 3 to 5 tokens/s for 27B to 31B without a drafter. Things only get really smooth with the MoE models and the smaller Gemma models. The software is also Windows- and Intel-only, and it comes from a solo developer who ships changes at a very rapid pace. Still, if you own a Core Ultra laptop, it's one of the most interesting ways to run models locally without spending money on extra hardware.