2-bit copy in VRAM, rest in RAM: speculative decoding for oversized models

What this is about

On r/LocalLLaMA, a user (u/Happy_Guitar3521) presents an idea of his own that he can't pursue further for lack of time. It is meant to speed up models that don't fit into video memory (VRAM). He stresses that it isn't "AI slop": the idea is his own, and he uses it in a production application that has nothing to do with LLMs. A public repository is on Hugging Face (HappyGuitar123/twobit-residual-spec). This post is the only source, and much of it is self-reported and not independently verified.

The idea

Ordinary speculative decoding uses a small draft model that proposes tokens and a large model that verifies several of them at once. That takes extra memory for the draft model. The proposal here: split the model's weights by bits.

• A 2-bit version of the model sits in the GPU's VRAM.
• The rest of the weights (the "residual", 6–8 bit) sit in system RAM.
• The 2-bit copy drafts tokens, and the full model (2-bit part plus residual) verifies them.
• The GPU computes the 2-bit part and the CPU the residual part. Only activations travel over PCIe (the connection between CPU and GPU), not weights.

The draft therefore costs no extra memory: according to the author, it is literally half of the verifier.

Measured results

Measurements used real weights, but dequantized and in PyTorch, so there are no tokens per second yet.

• 2-bit plus an 8-bit residual lands closer to bf16 than Q8_0, with about half the KL divergence (a measure of how far the output distribution drifts from the original).
• With a 6-bit residual, you reach Q8_0 quality and read about 28% fewer bytes from RAM.
• The 2-bit draft, which reuses the verifier's KV cache, gets 25–30% more accepted tokens per verification than Google's official Gemma-4 drafter (31B, 16-node tree): 5.23 versus 4.20.
• The CPU kernel verifies 16 tokens in about the time it takes to do one, because it is memory-bound.

Scope and limits

The closest work, according to the author, is SubSpec. But it uses a separate 4-bit copy and streams weights over PCIe, rather than splitting the model by bits. The approach only makes sense when Q8 doesn't fit in VRAM. If it does, he says, just use llama.cpp. He points to a possible combination with MTP (multi-token prediction) and advises having a strong LLM analyze the repo.

Our take

The idea is plausible and addresses a real problem for local LLM users: large models with limited VRAM. But the crucial real-world proof is missing. Without actual throughput measurements (tokens/s), it's unclear whether the gain gets eaten up by CPU compute time and PCIe latency. The acceptance rates against Google's drafter come from the author's own measurements. This is an early research approach from one individual, not a finished tool.