4-bit KV cache: The same memory hack can be free – or break everything

Squeezing a language model's cache down to 4 bits is the most popular trick for getting more context out of a GPU with limited VRAM. One test finds no difference at all. Another sees 80 out of 80 correct answers drop to zero. We ran the numbers and sorted through the measurements: it depends on the model family, the task and how the test was run.

Analysis Nov 21, 2026 10 min read Original analysis by our editors
Illustrative image, AI-generated

What's going on

As a language model reads, it stores every token in a cache called the KV cache. For each layer, it holds two vectors per token, the "keys" and the "values," which the model draws on every time it generates a new word. This cache grows with context length and can quickly eat several gigabytes of VRAM on long documents. The obvious trick: instead of storing the entries at 16 bits (f16), you store them more coarsely at 8 bits (q8_0) or 4 bits (q4_0). In Ollama and llama.cpp, that's a single switch.

Whether that switch costs you anything is where the published measurements flatly contradict each other. For Qwen3.6, a recall test finds no difference between 4 and 16 bits. For Qwen2.5, on the other hand, the output collapses. With Hy4 Flash, DeepSeek-V4.1-Flash and other large models, lots of people are currently trying to cram long context into limited memory. So we sorted out the contradictions. To be clear up front: we did not complete our own retest for this piece. Our analysis is based on the published measurements and our own recalculations.

The measurements at a glance

Source Model Engine Metric Result q4_0
Reddit r/LocalLLM (Giraffe) Qwen2.5 7B (Q4_K_M) Ollama 0.40.16, T4 correct JSON extractions 0/80 (q8_0: 80/80)
InventiveHQ Qwen2.5-Coder-7B (Q4_K_M) llama.cpp, 8K context text similarity to f16 8.3% (q8_0: 81.6%)
Localbench Qwen3.6 27B / 35B A3B (BF16) llama.cpp, patched KL divergence 0.087–0.117
Localbench Gemma 4 26B A4B (BF16) llama.cpp, patched KL divergence 1.088 (q8_0: 0.377)
sanmai (report from June 10) Qwen3.6-27B (approx. Q3 weights) llama.cpp, 262K context verbatim recall up to 140K same as bf16 within noise
Anbeeld Qwen3.6 27B (Q5_K_S) llama.cpp fork, 64K 99.9% precision (KLD tail) 89.84% (q8_0: 94.61%)

The Reddit test shows the problem most clearly. The author had Qwen2.5 extract six order details as JSON from each of 80 synthetic support notes, with structured output mode turned off. With q8_0, every field was correct, for example order number "DP-3976," 17 units and $444.42. With q4_0, only backticks and digits came back. Ollama reported both requests as completing normally. The author adds caveats: one run per setting, the cause not isolated, four parallel slots with 8,192 tokens of context each.

Cause 1: The model family

Qwen2.5 is a known problem case in the research. The Q-Filters paper suspects the cause lies in Qwen2.5 adding an extra bias, a learned offset, to its query, key and value projections. A not-yet-peer-reviewed preprint measures key values in some Qwen2.5-7B layers that are more than 250 times larger than the median. Outlier channels like these are poison for simple 4-bit rounding: a block of 32 numbers shares a single scale, and if one number is huge, all the others get squashed down to zero or one. KVTuner concludes that Qwen2.5-7B-Instruct needs at least about 4 bits for the key cache. The Ollama FAQ already warns that models like Qwen2 can lose more precision.

Qwen3.6 is built differently. According to The Kaitchup, only 16 of the 64 layers in the 27B model use classic attention with a full KV cache. The rest are Gated DeltaNet blocks that don't need a growing cache. In llama.cpp issue #21385, one user speculates that the many linear layers shield the few attention layers. Another user in the same thread reports that a custom 3-bit quantization failed badly on Qwen2/2.5, while Qwen3.5 was "immune." Neither claim has been independently confirmed.

Gemma 4 is the third case. According to Localbench, Gemma 4 31B already sits at a KL divergence of 0.108 with q8_0, worse than Qwen3.6 with q4_0 in most categories. The MoE variant 26B A4B hits 0.377 with q8_0 and 1.088 with q4_0, and only 68 percent of the most likely tokens still match f16. In a Hugging Face discussion, one user measured the extreme tail (KLD99.9) of the 26B model at 0.59 with an f16 cache and 17.43 with q8_0. According to the thread, Google hadn't responded there by the end of July.

Cause 2: The task

Plain prose is forgiving. If the model picks a different but fitting word, the answer still reads fine. Strict formats forgive nothing: one misplaced bracket makes the JSON invalid, one wrong digit makes the order number useless. Localbench finds exactly this pattern for Qwen3.6 with q4_0. Most categories stay close to zero, while long documents hit 0.581 and tool calling 0.086. The measurement by bmdpat with Ollama 0.40.6 points the same way on a small scale: Gemma4:12b solved only 14 instead of 16 tasks with q4_0. The sample is tiny, though, and Qwen3.5:4b already slipped at q8_0 in that test.

Cause 3: The measurement method

The three metrics answer different questions.

  • Text similarity under greedy decoding (InventiveHQ): The model always picks the most likely token, and the answer is compared with the f16 answer using Python's difflib. If a single token flips early on, the entire rest of the output goes differently, even if it's just as good in substance. So 8.3 percent means "different text," not necessarily "nonsense." InventiveHQ itself calls the method blunt. Still, the gap to q8_0 (81.6 percent) is striking, on CUDA and Metal alike.
  • KL divergence (Localbench, Anbeeld): It measures how far the probability distribution over the next token shifts, token by token, while reading a fixed text. An average can look good while rare outliers do the damage. That's why Anbeeld evaluates the 99.9 percent tail, and that's exactly where the formats separate, while perplexity stays almost flat.
  • Accuracy and recall (Reddit, sanmai): These check whether the right result comes out at the end. That's closest to real-world use, but it only holds for that specific task. Retrieving a passage verbatim ("single-hop") is a different skill from processing information precisely. In issue #21385, one user reports a path score of 65.70 instead of 100 for Qwen3.6-35B-A3B with q4_0/q4_0, even though the KL divergence stayed close to the original.

Our take: the tests contradict each other less than it seems. They measure different models on different tasks with different yardsticks.

Rotation moves the goalposts

On April 1, 2026, Georgi Gerganov merged PR #21038 into llama.cpp. Since then, Q, K and V are rotated with a normalized Hadamard matrix before being stored. This doesn't change the attention results, but it spreads outliers across all channels, so 4-bit rounding does less damage. The idea comes from work like QuaRot. The rotation kicks in automatically as soon as K or V is quantized and the head size is a multiple of 64. It stays off for MLA models such as the DeepSeek family, and you can disable it with LLAMA_ATTN_ROT_DISABLE=1.

The effect, according to the PR: for Qwen3.5-9B, KL divergence for q4_0/q4_0 dropped from 0.0080 to 0.0051. For gpt-oss-20b, the AIME25 score with a q4_0 cache rose from 2.0 to 21.7 percent, compared with 37.9 at f16. It's not entirely free: one tester measured token generation at about 0.88 to 0.93 times the original speed. For older measurements, that means anyone who tested before April was measuring a different q4_0. According to the Hugging Face discussion, the rotation initially didn't work on Gemma 4, so the day-one numbers were worse.

Whether Ollama uses the rotation is still unclear. The Ollama FAQ and release notes up to version 0.40.17 say nothing about it. An Ollama PR with Hadamard-based types tq3/tq4 (#15090) has been open since March. InventiveHQ says it measured with standard llama.cpp, using build b9430 for Metal and in June, so after the merge, and still saw the collapse. Our take: even with rotation, Qwen2.5 doesn't appear to work reliably with symmetric q4_0. But that's backed by just this one test.

One switch versus two

Ollama only offers OLLAMA_KV_CACHE_TYPE, global for all models and shared by K and V. According to Sam McLeod, who built the feature, the developers deliberately left out separate levels to keep things simple. Quantization only happens with Flash Attention, a memory-efficient way of computing attention. If a model doesn't support it, Ollama falls back to f16. McLeod describes a log warning for this, while ModelPiper calls the fallback "silent."

llama.cpp lets you set separate types with -ctk and -ctv. That pays off because keys are more sensitive than values. The KIVI research already shows this by quantizing keys per channel and values per token, and llama.cpp testers found key norms 6 to 182 times larger than those of the values. Anbeeld's measurements on Qwen3.6 show that putting bits into K first gives you the most benefit. There's a catch, though: according to issue #24485, a CUDA build without GGML_CUDA_FA_ALL_QUANTS=ON silently falls back to CPU attention for mixed types like q8_0/q4_0, and prompt processing (prefill) gets 25 to 45 times slower. For MLA models and DeepSeek V4, llama.cpp has enforced identical types for K and V since build b10206 anyway.

What the savings really get you

We recalculated the numbers from the Reddit test. Qwen2.5-7B has 28 layers, each with 4 KV heads of size 128, which comes to 56 KiB per token in f16. q8_0 needs 8.5 bits per value, q4_0 4.5 bits.

Cache Qwen2.5-7B, 32,768 tokens Qwen3.6-27B, 262,144 tokens
f16/bf16 1,792 MiB 16 GiB
q8_0 952 MiB 8.5 GiB
q8_0 K / q4_0 V 728 MiB 6.5 GiB
q4_0 504 MiB 4.5 GiB

The 952 and 504 MiB match the Ollama logs exactly. So in this case, going from q8_0 to q4_0 saves only 448 MiB, about 3 percent of a 16 GB T4. The big wins come with very long context. For Qwen3.6-27B (16 attention layers, 4 KV heads of size 256, so 64 KiB per token), our calculation matches the 17.2 GB that The Kaitchup reports.

Our recommendation

  • Default: q8_0 for K and V. With Qwen and Llama-like models, that's usually unremarkable, but with Gemma 4 the loss is measurable.
  • Saving more: In llama.cpp, lower V first (q8_0 K with q4_0 V, or q5_0 K with q4_1 V) before taking K down to 4 bits.
  • q4_0 for both only with hybrid models like Qwen3.6, for chat and summaries, and preferably in a current llama.cpp build with rotation.
  • Stay away from q4_0 with Qwen2.5, with Gemma 4, and anywhere you need JSON, tool calls or exact numbers.
  • Test it yourself: Use llama-perplexity to save the logits once with an f16 cache (--kl-divergence-base), then compare against the quantized cache (--kl-divergence). Also run a handful of your own must-pass tasks with automated checks; tools like Giraffe help with that. Speed and logs won't reveal the problem.

The bottom line

"q8_0 is practically lossless, q4_0 costs a little": this rule of thumb from the Ollama FAQ is too optimistic for some models and too pessimistic for others. Three things are well supported: older Qwen models and Gemma 4 are far more sensitive than the hybrid Qwen3.6. Strict formats and long documents suffer first. And llama.cpp's rotation has noticeably improved q4_0 since April. Much of what gets the attention remains unproven: the 0/80 case rests on a single run with four parallel slots, the sanmai report is no longer available, and whether Ollama rotates isn't documented. The most critical issue is that the failure is silent. Nothing in the log warns you, and the token rate looks normal. Our take: if you turn on q4_0, treat it like a change to the model itself and test it against f16 on your own task first.

#kv cache#quantization#llama.cpp#ollama#local ai#qwen

Ask the AI assistant

Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.

Connecting …

The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.

Comments

What do you think? Already tried it? Write it here, no sign-up needed.

  1. Loading comments …

Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.