colibri runs a 744B model on a 25 GB laptop by streaming experts from disk

What it's about

colibri (spelled colibrì in the project) is an open-source inference engine by JustVugg that is getting attention on GitHub right now. Its motto: "Tiny engine, immense model." It runs very large Mixture-of-Experts (MoE) models on machines you already own. It's written in pure C with no dependencies, has one file per model family, and doesn't require a GPU. The project is free and open source. According to a second source (alphamatch.ai), it is licensed under Apache 2.0.

The idea: a MoE model (Mixture of Experts: the model consists of many specialist sub-networks, of which only a few are active per token) is huge on disk but uses only a small part per token. colibri keeps the always-needed part in RAM and reads the remaining experts from disk on demand.

How expert streaming works

Using GLM-5.2 as an example, the README does the math: 744 billion parameters in total, about 40 billion active per token, and only around 11 GB of those change from token to token, the "routed" experts. That's only about 5.4% of the parameters.

The dense part (attention, shared experts, embeddings) therefore stays in RAM. The routed experts sit on the SSD and are loaded when the router (the selection stage that picks the right experts for each token) asks for them. A cache learns which experts your own workload needs often. If you have a GPU, it also holds the "hottest" experts and the dense layers.

An important point from the README: where a weight lives changes only speed, not the result. The default setting never silently changes model precision or router semantics. Less fast memory may cost you speed, but it won't quietly redefine the model.

According to alphamatch.ai (details that don't come from the README excerpt), the engine also uses the following for GLM-5.2:

• an LRU cache (evicts whatever went unused the longest) per layer, with the operating system's page cache as a second tier;
• Multi-Token Prediction (MTP) as speculative decoding: an auxiliary head guesses the next token ahead and the main model verifies it. The post cites 2.2–2.8 tokens per pass with 39–59% of drafts accepted;
• a compressed KV-cache layout (MLA): 576 instead of 32,768 numbers per token;
• AVX2 integer kernels at 119 GFLOP/s.

These claims come from a third-party blog and haven't been independently verified here.

Which models run

The README lists thirteen engines: ten for language models (GLM-5.2/5.3, which also runs Mistral-Large-4; GLM-5.3-Flash, Inkling, Kimi K3, DeepSeek V4 Flash and V4.1 Flash, MiMo-V2.6 Flash and Pro, Qwen3.8-Flash-Next, Qwen3.6 and OLMoE), one for images (Qwen-Image-2.1) and two "decision models" (Laya, GLiNER2.5-Decide), plus Clef on the Qwen3.6 engine. In addition, the language-model engines read published GGUF files directly, on CPU, via Vulkan or with CUDA.

Excerpt from the README table (vendor figures from the project, measured on the machines named):

On the large models, the disk sets the pace, and a fast NVMe SSD helps most. Mistral-Large-4 (1T, 52B active) from a 233 GB GGUF reaches 1.34–1.36 tok/s with CUDA on test machine D. The small Flint (40B, 2.5B active, Apache 2.0) hits 77 tok/s, and LFM3-VL-2B exceeds 200 tok/s with a GPU.

Installation and use

The setup is meant to be a one-step affair. On Windows, download the ZIP and double-click START-HERE.bat. On Linux and macOS, run ./start-here.sh after cloning the repo. The script checks RAM, free storage, CPU and GPUs, recommends a suitable model, builds the engine (or downloads a prebuilt one), downloads the model with resume support and starts colibri along with a dashboard in your browser. After that, an OpenAI-compatible (/v1) and an Anthropic-compatible interface are available, so you can connect tools like Claude Code or SDKs. Options such as --list, --model ID, --yes, --dir and --backend vulkan|cuda|cpu control the behavior. The README also describes an MCP server and a guide that lets an AI assistant handle the installation.

Other features: tool calling in almost all chat engines, image input for several models, a chat UI with an "expert map," a monitor and profiling page, and a "System One" mode: instead of generating text, the model returns probabilities and a confidence value for predefined options. The interface is compatible with TypeSafe's Jev API.

GPU support

• Vulkan: Any GPU with a Vulkan 1.2 driver (AMD, Intel, NVIDIA, integrated or discrete) can use an expert cache in GPU memory and a "dense chain." On the integrated Radeon 780M, Qwen3.6-35B-A3B rose from 6.0 to 9.9 tok/s. With small models like OLMoE, the iGPU can even be slower (12.8–17.3 vs. 23.1 tok/s on the CPU). On an Intel Iris Xe laptop GPU, Vulkan wasn't faster either.
• CUDA: On Linux, if the CUDA toolkit is installed. Windows has a separate DLL. On an RTX PRO 4000 (24 GB), Qwen3.6-35B-A3B reaches 104.8 tok/s. Running on cards without Resizable BAR is also new, but the README says it hasn't been measured on such cards.
• Apple Silicon: A Metal backend for several engines. The one-step setup, however, initially builds for the CPU.

According to the README, CI checks the Vulkan paths against the CPU results. Because of a different order of computation, a long answer may differ by a word.

What you need to try it

• Minimum: A machine with 8 GB of RAM (16 GB or more is better), at least 22 GB of free storage for the smallest model, and an internet connection. A graphics card is optional.
• Large models: Storage needs grow fast. GLM-5.2 needs 429 GB and Kimi K3 1.56 TB. A fast NVMe SSD is crucial. Under WSL, keep models in the Linux filesystem, not under /mnt/c.
• Mac/CPU: Runs on CPU and Apple Silicon (Metal optional). The source doesn't mention Google Colab.
• A sensible starting point: Qwen3.6-35B-A3B (23 GB) or Qwen3-Coder-30B-A3B, where double-digit speeds are realistic.

Our take

colibri shows that the memory bottleneck of MoE models can partly be sidestepped with clever placement: you can start models that would never fit in RAM. That's valuable for research, experiments and understanding expert routing (the dashboard visualizes every activated expert).

The limits are clear, though. At 0.05–0.1 tokens/s, GLM-5.2 on a 25 GB laptop is an experiment, not a working tool. Even 1–2 tokens/s is only usable if you're patient. The smaller MoE models with double-digit speeds are more practically interesting. All figures come from the project itself, measured on different machines, and the sources contain no independent re-measurements. The second source gushes about "democratization" and cites 241 stars, which is a snapshot from the blog that has probably been overtaken by now. The hardware example of an Apple M5 Max at about 1 token/s also comes only from there. Compared with llama.cpp, colibri offers its own engines per model family and a lot of extra functionality, but the sources contain no direct comparison.