KV cache on a diet: what learned “bouncers” forget in long contexts

What this is about

As a language model reads, it stores every token in a buffer called the KV cache. Think of it as a notepad with two rows of numbers for every word piece: a “key” for finding it again and a “value” holding the content. The notepad grows with every token. In short chats you won't notice. With a 300-page contract you will: on your own GPU, the problem is usually no longer the size of the model but the size of its short-term memory.

Two pieces of news landed together on November 12, 2026. On the r/LocalLLM subreddit, a developer showed off a self-trained eviction gate that lets Qwen3-4B read 123,000 tokens on a 16 GB card. On the same day, llama.cpp got an FP8 KV cache. Since the summer, several research papers on these learned “bouncers” have come out as well. We explain the three approaches available right now and take a close look at what the bouncers forget along the way.

How big the notepad really gets

You can work out the size: layers × KV heads × head dimension × 2 (key and value) × bytes per number × tokens. For Qwen3-4B, the official config (36 layers, 8 KV heads, dimension 128) gives 147,456 bytes, or 144 KiB per token at 16 bits. Qwen3-8B has the same values, so it needs exactly as much cache even though the model is twice as large.

*Extrapolated linearly from the figures in the vLLM blog (83.9 and 9.62 GiB at 1 million tokens in bf16). Gemma values use the formula from Kaitchup's analysis and leave out any savings from shared key/value storage in the global layers. All values assume 16 bits and a single request.

Here's an example. As a 4-bit file, Qwen3-4B takes up just a few gigabytes, but its cache at 128k is 18 GiB. That makes the notepad more than twice the size of the entire model at full 16-bit precision. It also matches the Reddit post's claim that the full cache comes to “about 9 GB” at 61,000 tokens (the formula gives 8.4 GiB).

Approach 1: Store smaller numbers

The simplest option is KV quantization. Every number on the notepad is stored at lower precision, with 8 or 4 bits instead of 16. Every entry is kept, just less precisely. In llama.cpp you control this with -ctk and -ctv. According to the current server documentation, the options are f32, f16, bf16, f8_e4m3, q8_0, q4_0, q4_1, iq4_nl, q5_0 and q5_1. The FP8 type is new: PR #28392 by ORippler was merged around November 10, 2026. According to a report by Marais Software, FP8 targets Nvidia cards from the Ada Lovelace generation onward, meaning RTX 40 and up. A third-party project notes that the FP8 cache isn't available under Vulkan. The PR doesn't include any quality measurements.

Independent numbers do exist for the older types. The sotaaz blog benchmarked Qwen3-8B on an A100. With q8_0, perplexity (a measure of how “surprised” the model is by a text) rose by at most 0.1 percent. With q4_0, it rose by 0.5 to 1.1 percent. Speed takes the hit instead: with 64,000 tokens in the cache, q8_0 generated output at only 56 percent of f16 speed. Since April 2026, llama.cpp has automatically applied a Hadamard rotation to values before storing them (PR #21038). This spreads out outliers so they round more cleanly. In the PR thread, this lifted gpt-oss-20b with a q4_0 cache from 2.0 to 21.7 percent on the AIME25 math test. It doesn't apply to MLA models.

Microsoft's SCBench benchmark points to an important caveat. In multi-turn conversations, the 2-bit quantization method KIVI kept 12.0 points on exact text retrieval (full cache: 57.1). That sounds weak, but it's twice as much as the best eviction method managed. Our take: quantization is the most robust approach because nothing gets deleted. But it saves a factor of 4 at most.

Approach 2: Throw things out, starting with fixed rules

With eviction, entire entries are removed from the notepad. The first methods worked with fixed rules. H2O (2023) keeps the most recently read tokens plus the “heavy hitters”, tokens that have received a lot of attention so far. SnapKV (2024) checks what the end of the prompt (usually the question) pays attention to and keeps that. RazorAttention (2024) treats heads differently: a few “retrieval heads” keep everything, while the rest discard distant content. The authors report savings of over 70 percent without any training.

SCBench exposes the problem: these methods are tuned to the first question. With Llama-3.1-8B and a 1/32 budget, SnapKV fell to 6.1 points on exact text retrieval in multi-turn conversations and to 0.3 on separate queries about the same document (full cache: 57.1 and 29.5 respectively). The KVzip authors put it this way: query-dependent caches “overfit” to the first question. A study by Chen and colleagues also found that when a prompt contains multiple instructions, some of them are practically ignored after compression.

Approach 2b: Learned bouncers

The new generation trains a small add-on network called a gate. As each token is written to the cache, the gate decides whether it will be needed later. The model itself stays unchanged.

The “94 percent of quality at 88 percent compression” figure comes from the body text of the KVpop paper. The paper's own table works out to about 97 percent for Qwen3-4B, so the text and the table don't agree. For Qwen3-8B at 131,000 tokens, memory use dropped from 36 to about 19 GB. But this was tested on math tasks, where the model writes its own long chains of thought and old intermediate steps can be dropped.

What the gates lose in detail

Needle in a haystack: RAZOR finds the hidden passkey at 31,000, 61,000, 96,000 and 123,000 tokens. The cache takes up 0.72 to 2.84 GB, which means around 83 percent gets discarded. But a passkey is just a single number buried in filler text. On the harder InfiniteBench task kv_retrieval, the model scores only 0.08, just as badly as with the full cache, and with only 12 examples per task. One important detail: according to the README, the RULER scores came from a masking path that saves no memory at all. So accuracy and memory savings were measured in separate runs. All the numbers come from the developer himself, the repo had zero stars when we checked, and the paper isn't on arXiv yet.

Exact retrieval and repetition: This is the real weak spot. RAZOR's own documentation admits it struggles with RULER-cwe, a task that requires every occurrence of a repeated word. In TrimKV's SCBench results, all eviction methods drop to 0 on key-value retrieval, while the full cache reaches 66. The authors themselves call these tasks “incompressible”.

Multiple questions: This is where the gates clearly beat fixed rules. TrimKV scores 49.16 points on multi-turn retrieval in SCBench (full cache 49.60, SnapKV 0.31). On the LongMemEval memory test, however, it falls from 49.4 to 44.8 with a 32k budget and to 34.0 with 8k. Fast KVzip is explicitly trained to stay query-agnostic and tests exactly this scenario with SCBench and MRCR.

German texts: We couldn't find a single measurement. Every gate was trained and tested on English data, mostly math or English web text. Our take: nobody knows yet whether a gate trained on math chains of thought keeps the right clauses in a German rental agreement. If you want to use one, test it on your own documents.

Approach 3: A different architecture altogether

The biggest savings come from an architecture that needs little cache in the first place. DeepSeek's MLA (Multi-head Latent Attention) stores a heavily compressed summary per token instead of full keys and values. DeepSeek V4 goes further: it merges 4 or 128 tokens into a single entry and reads only a selection of those entries (sparse attention). According to the vLLM blog, this means V4 needs only about a ninth of V3.2's cache. Hybrid models like Qwen3.5 use linear attention in three out of four layers, with a fixed-size state that doesn't grow. Gemma 4 lets most of its layers see only the last 1024 tokens. That's essentially eviction, just built in and learned during training.

What works today with a single flag

Ollama's FAQ says q8_0 is usually unnoticeable, while q4_0 becomes more noticeable at large context sizes. You have to set the environment variable on the server service. For FP8, vLLM recommends calibrating with llm-compressor and lets you exclude sensitive layers such as sliding-window attention.

The bottom line

For local users, the options sort themselves out clearly. Our take: q8_0 or the new FP8 are the safe first step. They halve the cache and cost hardly any quality, but they slow down output at long context lengths. q4_0 is the last resort when nothing else fits. The biggest gain comes from picking the right model: at 128k, a hybrid 4B model needs about a quarter of the cache of a classic one.

The learned gates are the most exciting option, since they can deliver factors of 4 to 8 without changing the model. But they're research code for individual Qwen models. They run on Transformers rather than llama.cpp, mostly handle one request at a time and are backed almost entirely by their developers' own numbers. The headlines rest mainly on math benchmarks. On exact retrieval, lists with repetitions and instructions buried deep in the prompt, there's clear evidence that the gates lose information. For German text there's no data at all. If you want to search a long contract on 16 GB today, a quantized cache and a well-chosen model are the safer bet. A bouncer might have thrown out exactly the clause you need for your third question.