Betelgeuse runs a 68 GB MoE model at 22 tok/s on an RTX 3060 with 16 GB RAM

What's going on

A user (u/zyxciss) has presented Betelgeuse on Reddit's r/LocalLLaMA. It's a runtime built as a fork of llama.cpp. The goal: make the very large Mixture-of-Experts model (MoE – a model made up of many "expert" sub-networks, only a few of which are active per token) Qwen 3.8 Flash Next usable on an ordinary gaming PC, using a 68 GB quantization (GSQ-RCO-IQ2_XS). All figures come from the author and have not been independently verified.

The author also distances himself from an earlier prototype. It reached high speeds through "gate pruning" (dropping rarely used experts) but hurt quality, so he scrapped it. Betelgeuse, by contrast, skips not a single expert.

Test system

• GPU: RTX 3060 with 12 GB VRAM
• RAM: 16 GB DDR4, single-channel (about 19 GB/s); tested inside a cgroup with MemoryMax=12G
• OS: CachyOS / Arch Linux
• Storage: consumer NVMe SSD at around 2.1 GB/s

Stock llama.cpp loads the model via mmap (the file is mapped into memory), and the OS page cache thrashes: roughly 1,500 large page faults per token and just 1.47 tok/s. The alternative, Strata, reportedly reaches 40+ tok/s but needs 48–64 GB of RAM. On 16 GB it falls back to 8.11 tok/s with --low-ram mmap.

How Betelgeuse works

• Arrival-Slack VRAM Slot Pool: All 49 dense layers, plus GDN, QSA and the LM head, take up about 3.2 GB of VRAM. The rest holds roughly 5,000 expert slots, distributed unevenly: 240 per layer in the early layers 0–4 (where there's no compute slack), 92 in the middle, 60 at the end.
• In-Stream GPU Doorbell + CUDA Graph Replay: Instead of splitting the graph 48 times per token and asking the CPU (about 360 ms per token from cudaStreamSynchronize, according to the author), a custom CUDA kernel checks in about 2 µs whether an expert is in VRAM. It only "rings the doorbell" to the host on a miss. The entire decode pass then runs as a single CUDA Graph replay.
• RAM Victim Cache + io_uring Pager: Experts evicted from VRAM move into a RAM cache. VRAM and RAM are mutually exclusive, with no duplicates in the page cache. SSD reads drop to 12.43 MB per token.
• No duplicate memory: The model is split into Dense.gguf (3.2 GB), experts.bin (35.3 GB, 4 KB-aligned) and PLE.gguf (26.8 GB).

Measurements

On long coding prompts, it reportedly holds 21.8–22.03 tok/s. Even the huge Q8_K_XL variant (~190 GB) runs on the same machine at 8 tok/s cold and 12 tok/s warm (peaks of 21), while stock llama.cpp crashes or stays below 0.1 tok/s. One weak spot: prompt processing is slower than with Strata.

What you need to try it

According to the author, you need an Nvidia GPU with 12 GB of VRAM (RTX 3060), 16 GB of RAM and an NVMe SSD with enough free space (the package is a good 65 GB; the largest variant is about 190 GB). It has only been tested on Linux with CUDA. There's a prepackaged bundle on Hugging Face (Alpha-Orionis-XS), code on GitHub and an upstream PR for llama.cpp.

Our take

The project shows that with large MoE models, memory management and synchronization matter as much as raw compute, if not more. Notably, no quality is lost by dropping experts. On the critical side: all benchmarks come from the author and cover a single hardware configuration, and byte-exactness was demonstrated on 576 tokens. Prompt processing isn't faster than the alternatives, and the author himself says the upstream PR has little chance of being merged. The DeepSeek V4.1 port is still in early development.