Maple-Preview: Ternary 20B MoE runs fast on CPUs and small GPUs

What it's about

The Deepgrove team has published the Maple-Preview-GGUF collection of files on Hugging Face. Maple-Preview is a reasoning model (it "thinks" in intermediate steps before answering) with 20 billion parameters, of which only about 1 billion are active per token (hence the name 20B-A1B). According to the authors, it was designed from the start to run efficiently on your own device, and it is released under the MIT license. The preview is an interim release; a full version is due after further training.

The trigger for this report was a post on r/LocalLLaMA about another ternary model that refers to Maple (see below).

Architecture and ternary technique

Maple is a Mixture-of-Experts (MoE) model: instead of using the whole network for every token, a router picks a small share of specialized sub-networks ("experts"). Here there are 24 layers and 256 experts, of which 8 are active. That saves compute, but memory still has to hold all the weights.

For attention, Maple uses a 3:1 ratio of sliding-window attention (window size 512) to global attention. In other words, three out of four layers only look at the last 512 tokens, while one looks at the entire context. That keeps memory use lower for long texts.

Ternary means the weights can take only three values (-1, 0, +1), roughly 1.58 bits per weight. This shrinks the model dramatically. The source mentions two packing schemes: TQ1_0 (more compact) and TQ2_0 (slightly larger, but generally faster). The output layer ("LM head") stays at higher precision, either Q4_K or FP16.

Variants and speed

The benchmarks ran on an M5 Pro, CPU only, 16 threads, 512 prompt tokens, 128 generated tokens, three repetitions. "Prefill" is reading in the prompt; "decode" is generating the answer. Notably, the Q4_K head speeds up generation considerably compared with FP16.

Quality, according to the maker

Deepgrove claims Maple-Preview sets a new point on the Pareto frontier, both for memory versus performance and for speed versus performance. The source gives no concrete benchmark numbers, so this is an unsubstantiated vendor claim. The team does concede that the preview focuses on pure reasoning and may score lower on agent benchmarks.

The second ternary model

The Reddit post presents "Ternary Nemotron 30B-A3B", a ternary (1.58-bit) MoE with 30B parameters, 3B active. The author reports 368K context on a 12 GB GPU, 172 tok/s decode and over 7,000 tok/s prefill. One file reportedly runs on both stock llama.cpp (validated from b11555 and b11796 respectively) and the PrismML fork (b10793). According to an update, it is the second ternary MoE after Maple-Preview that works with stock llama.cpp. The author is still looking for CPU-only numbers and results for cards around 10 GB. These figures come from the uploader and have not been independently verified. The main source's Hugging Face page, however, covers Maple.

What you need to try it

• Memory: The files are 4.6 to 5.9 GiB. With context, plan on 8 GB of free RAM or VRAM as a comfortable minimum (an estimate).
• Processor: Per the measurements, it runs on the CPU alone, including on Apple Silicon (M5 Pro tested). You don't need a GPU.
• Disk: Around 6 GB per variant.
• Software: A custom Deepgrove llama.cpp fork (github.com/deepgrove-ai/llama.cpp) with setup instructions is referenced. The page also lists Ollama, LM Studio, vLLM and other tools, but it's unclear whether they support the ternary types.
• Google Colab: Possible in principle given the small size, but not confirmed in the source.

Our take

Ternary MoE models are interesting because they combine a large store of knowledge with very small memory and compute demands. Over 250 tokens/s on a laptop CPU is impressive. On the downside, there are no independent benchmarks, the preview is explicitly unfinished, and its usefulness for agent tasks is limited. It's worth a look for tinkerers, edge applications and anyone testing ternary quantization. For production agents, it's probably not there yet.