Giant AI models off an SSD: why "runs on a laptop" often means 0.1 tokens/s

What this is about

In a short span, we've run four headlines: Kimi K3, with 2.78 trillion parameters, runs on a CPU with 8 GB of RAM. colibri gets GLM-5.2 (744 billion parameters) running on a laptop with 25 GB. Betelgeuse hits 22 tokens per second on an RTX 3060. And a Strata patch starts agent sessions 20 times faster. All of them use the same trick. Mixture-of-experts (MoE) models consist of hundreds of subnetworks, the "experts," and only a few of them do any work for each token. So the engines load just the experts they currently need from the SSD.

Our take: with the really big models, "runs on a laptop" often means 0.05 to 0.8 tokens per second. Expert streaming only becomes usable when a cache keeps the right experts in RAM or VRAM (video memory) and a fast NVMe SSD delivers the rest. We put the projects side by side and worked out the physical limit.

The projects at a glance

The table already shows the split. From around 40 billion active parameters upward, you're looking at seconds per token. With 6 to 16 billion active parameters, double-digit speeds are possible. But that's mostly because the bulk of the experts don't come from the SSD at all. They come from RAM. According to the Strata docs, all of Qwen3.8-Flash-Next's experts normally sit in RAM (23 to 50 GB), and the graphics card keeps copies of the most frequently used ones.

The physics: bytes per token divided by read speed

You can estimate the limit on the back of an envelope. Multiply the routed experts per token by the bits per weight, and you get the gigabytes that have to come off the disk when the cache misses. Divide the SSD's read speed by that amount, and you get the ceiling in tokens per second. Our calculation is based on figures from the sources. colibri gives roughly 11.4 GB per token for GLM-5.2 (75 layers × 8 experts at about 19 MB each). The Kimi README gives 25.83 GB per token without a cache. For Qwen3.8-Flash-Next, we divided Betelgeuse's 35.3 GB expert file by 24,576 experts and multiplied the result by 10 experts times 48 layers. For DeepSeek, it's a rough upper bound (16B × 2 bits, including the dense part, which actually lives in VRAM).

These are best-case figures. They assume the SSD actually holds its spec-sheet speed and that compute takes no time at all. The measurements line up surprisingly well. Better Stack tested GLM-5.2 on a MacBook M2 Max with an external USB-C SSD and got 0.1 tokens/s, with 92 percent of the time spent waiting on the disk. A 10 Gbit/s USB port delivers roughly 1 GB/s, which works out to about 0.09 tokens/s. With an internal NVMe drive, an RTX 4090 and 64 GB of RAM, the testers reached 0.8 tokens/s. In the colibri fork's community table, switching from a Crucial P3 to a Samsung 9100 Pro PCIe 5 SSD on a Ryzen 9 9950X takes you from 0.10 to 0.28 tokens/s. The Kimi README reports the same workload running at 2.7 GB/s on one run and 5.9 GB/s on another. Over a network drive, a token took 2.2 times longer.

One counterexample shows that bandwidth isn't everything. According to the Betelgeuse author, stock llama.cpp reads only 208 to 312 MiB per token via mmap (which maps the file into memory). On paper, that would allow more than 6 tokens/s on his 2.1 GB/s SSD. In practice, he got 1.47 tokens/s, because about 1,500 page faults per token slow the system down. That means lots of small, blocking individual reads. If you issue large, batched reads instead (io_uring, O_DIRECT), you get more out of the same drive.

The cache decides everything, and it's fickle

The formula gets a third factor: the amount actually read is GB per token times (1 minus the hit rate). This is where the projects diverge sharply.

• Kimi K3: According to the README, an LRU cache (which evicts the least recently used expert) stayed flat at a 36 percent hit rate anywhere between 8 and 64 GB. In live runs, the real hit rate was 0 to 30 percent. The "100 percent" figure shown on screen counted preloaded experts. The model deliberately spreads the load evenly across its experts, which defeats LRU.
• GLM-5.2: A Hacker News user reports a 23 percent hit rate on an M5 Max with 128 GB. An EPYC server with 430 GB of RAM and 77.5 GB of pinned experts reached 98 percent, and with it 1.0 tokens/s.
• Qwen3.8-Flash-Next: Betelgeuse reads only 12.43 MiB per token from the SSD. By our math, that means around 98 percent of expert accesses are served from VRAM or RAM.
• Depends on the task: A llama.cpp fork by DavideWasTaken filled its cache using source code. On code, it hit 72 percent, but only 17 percent on Italian text and 9 percent on English. The developer stresses that this is just an initial sample.

That's why the noticeably faster part of the scene doesn't primarily stream from the SSD at all. It caches from RAM into VRAM. According to its author, the cache now merged into llama.cpp (--moe-cache-mib, pull request #29887) delivers 2.2 times the speed on an RTX 5090 at an 89 percent hit rate. Other users, however, reported slowdowns under Vulkan and ROCm, for example from 23 down to 18.1 tokens/s.

Exact or just close?

colibri, Betelgeuse and the Kimi engine stress that where the weights are stored only changes the speed, not the output. Betelgeuse demonstrates byte-exact output over 576 tokens. Others deliberately cut corners. The Strata patch only lets tokens pick from experts that are already in memory while the prompt is being read. The author admits that this slightly changes the answers. The DeepSeek spin-off Xolotl manages 26.8 tokens/s on an RTX 4090 with 90 GB of RAM. But it uses a "route bias" that favors experts in VRAM, and the V4 build matches the original's most likely next token 86.9 percent of the time. So when you read speed comparisons, check whether they're actually running the same model.

The underrated problem: prompt processing and cold starts

For agents, what often matters isn't generation speed but time to first response. With Kimi K3, the first step for a five-token prompt took 53 seconds and 89 GB of reads. With 17 tokens, it was 98 seconds. Better Stack measured more than two minutes to first token for GLM-5.2 on the Mac. Even DeepSeek V4.1 Flash needed around 100 seconds for a 37,000-token agent prompt before the Strata patch. The reason: even a short prompt touches almost every expert. At 1,200 tokens, that meant 58 GB read from the SSD. And Betelgeuse is actually slower than Strata at prompt processing, at 16 to 19 tokens/s.

What about the SSD?

One HN commenter argues that pure reads are harmless, since it's write cycles that are limited. It's not quite that simple. A KAIST study (ASPLOS 2021) of eight current SSDs found that 3D TLC and QLC flash can wear out within the warranty period even under light, read-only workloads. The culprit is "read disturb": reading shifts charges in neighboring cells, so the controller rewrites the affected blocks in the background. And expert streaming reads enormous amounts of data. By our calculation, 1,000 tokens of GLM-5.2 without a cache add up to a good 11 TB of reads. Nobody has measured the wear yet, and the jacar.es test explicitly lists it as an open question.

Our verdict

Here's how we'd rate the options for everyday use:

• Ready for daily use: Qwen3.8-Flash-Next (6B active) on a 12 GB graphics card with 32 to 64 GB of RAM. With Strata, RustingStrata or colibri (51 tokens/s on a 24 GB card), 20 to 80 tokens/s is realistic. Betelgeuse shows it can be done with 16 GB of RAM too, but it's experimental.
• Usable if you're patient: DeepSeek V4.1 Flash with 12 to 24 GB of VRAM, 46 to 90 GB of RAM and an NVMe SSD. That gets you 5 to 27 tokens/s, but long startup times with agent prompts.
• Demo: GLM-5.2 (744B) and Kimi K3 as full models. Even with a PCIe 5 SSD and 128 GB of RAM, you're stuck at 0.3 to 2 tokens/s or several seconds per token, respectively. That may be fine for overnight jobs, but not for chat or coding.

The rule of thumb: tokens per second ≈ SSD speed divided by (GB per token × cache miss rate). If you're upgrading, put your money into RAM first, then an internal PCIe 4/5 SSD. A faster CPU helps the least. Almost all the numbers here are self-reported, each from a single machine. So far, colibri is the main project that's been independently benchmarked, and those results landed at the low end of what was promised.