Engram explained: why new AI models keep part of their knowledge on your SSD

DeepSeek, Qwen, and now Tencent are building giant lookup tables into their models. For each word, they read only a few dozen small rows from them. We explain how this "Engram" works, why the "total parameters" figure tells you little as a result, and why these tables are the rare case where weights can sit on your SSD without slowing the model down much.

Analysis Nov 16, 2026 10 min read Original analysis by our editors
Illustrative image, AI-generated

What this is about

Tencent unveiled Hy4 flash today. According to Tencent, the model has 224 billion parameters, but only 12 billion are active per token. One detail in the announcement stands out: 64 of the 224 billion are "Engram memory parameters." It's not an isolated case. DeepSeek-V4.1-Flash ships with around 196 billion such parameters. Alibaba's Qwen builds the same principle, under the name PLE, into the architecture the upcoming Qwen4 models are based on. Smaller Qwen4 models are slated to come out as open weights.

These parameters don't compute anything; they get looked up. That raises three questions: What's behind it? How meaningful are figures like "224B" anymore? And what does it mean for your machine at home?

How Engram works: a dictionary for word sequences

DeepSeek described the idea in the paper "Conditional Memory via Scalable Lookup" and released the code as deepseek-ai/Engram. A language model recomputes even trivial associations like "capital of France → Paris" for every token. Engram stores fixed patterns like that in a large table instead.

Here's how it works: the model takes the last two or three tokens, an n-gram (a short word sequence). From that it computes an address via a hash, a kind of fingerprint that points directly to a table row. To keep different word sequences from constantly landing on the same row, Engram uses multiple "heads": several independent hash functions, each with its own table. Before that, token variants such as upper- and lowercase are merged. According to the paper, that shrinks the vocabulary by about 23 percent. The rows that are read pass through a gate, a kind of valve that compares the table content with the current context. If the memory doesn't fit the context, it gets damped.

The key point for everything that follows: the address depends only on the token IDs, not on intermediate results of the computation. So the system knows which rows it needs before computing anything and can prefetch them. That's not possible with Mixture-of-Experts (MoE). There, only the running computation decides which subnetworks ("experts") are needed.

The paper's main findings (DeepSeek's own claims):

  • Engram-27B has the same number of parameters (26.7 billion) and the same compute cost (3.8 billion active) as a pure MoE baseline. In exchange, 17 of 72 experts are dropped, and their budget goes into 5.7 billion table parameters. Results versus the baseline: MMLU +3.4, BBH +5.0, HumanEval +3.0, MATH +2.4 points. On long-context retrieval (Multi-Query-NIAH), the score rises from 84.2 to 97.0.
  • The best results came from a mix where about 20 to 25 percent of the "dormant" budget goes into the tables. The curve is U-shaped; both extremes do worse.
  • If you switch Engram off afterward, the model retains only 29 to 44 percent of its performance on factual knowledge (TriviaQA: 29 percent). On reading comprehension it's 81 to 93 percent. So a large part of the factual knowledge lives in the table.

Three vendors, three names, one idea

Model Total Active per token Of which tables Open weights?
DeepSeek-V4.1-Flash 552 + 196 billion (HF shows 763 billion) 8 (prefill) / 16 (output) 196 billion yes, MIT
Qwen3.8-Flash-Next (Qwen4-Exp architecture) 125 billion 6 billion 51 billion yes
Qwen4-Max 2,400 billion 71 billion 328 billion no, API only
Hy4 flash 224 billion 12 billion 64 billion not yet, announced
Engram-27B (paper) 26.7 billion 3.8 billion 5.7 billion research model

According to the vLLM recipe, DeepSeek places two tables of around 384 million rows each in layers 1 and 14. Qwen calls its approach PLE ("Per-Layer Embedding"). According to the Transformers documentation, by default it hashes three-token sequences with 8 heads per n-gram length, plus a small convolution over neighboring tokens. Google introduced the PLE name with Gemma 3n. There, per-layer embeddings can sit on the CPU: in Gemma 3n E2B, only about 2 billion of 5 billion parameters live in VRAM. Qwen and DeepSeek, however, explicitly address their tables via hashed word sequences. For Hy4 flash, Tencent describes a "LatentMoE" design alongside Engram, plus a KV cache (the buffer for the text so far) that's only one-sixth the size of Hy4's. Tencent hasn't yet disclosed exactly how the 64 billion Engram parameters are split up.

Why "total parameters" needs a third column

With MoE, people got used to two numbers: total (determines memory requirements) and active (determines speed). Engram fits neither column. The tables count toward the total but are barely read per token. By Rohit Ghumare's calculations, DeepSeek-V4.1-Flash needs 2 tables × 3 n-gram lengths × 8 heads = 48 rows per token. In the 4-bit version (IQ4_NL, 144 bytes per row), that's 6,912 bytes. By our math, that's about one sixteen-millionth of the 110.6 GB table. For comparison: at 2 bits, the selected experts move around 2 GB per token.

Our take: an honest spec would have three parts, "active / total compute / lookup tables." Hy4 flash would then be 12 / 160 / 64, and DeepSeek-V4.1-Flash 16 / 552 / 196. Even on Hacker News, one user wrote that they couldn't make sense of V4.1 Flash's actual parameter count.

The special case where the SSD costs almost nothing

Normally, "runs from SSD" means painfully slow for language models. Engram is different, because it only generates tiny read requests that are known in advance.

vLLM, a serving engine, offloads the tables to pinned system memory by default, according to its documentation (cpu_offload is on). The GPU reads the rows via UVA, a shared address space for the graphics card and RAM. Prefetching runs in parallel with computation, and the freed-up VRAM goes to the KV cache. For DeepSeek-V4.1-Flash, that's about 200 GB in FP8 or 104 GB in MXFP4. For Qwen4-Exp there's also disk_offload, which reads the rows directly from the files at every step. All of this runs on CUDA hardware only.

llama.cpp has been able to read large tables "lazily" since PR #27794 (late August). The file is mapped via mmap, mirrored into memory in such a way that the operating system only fetches the pieces that are actually touched from disk. The flag is -lzm/--lazy-mode. The default auto applies to tensors over 4 GiB. llama.cpp also marks the region with MADV_RANDOM so the system doesn't read ahead unnecessarily. llama.cpp has supported DeepSeek V4.1 since build b11870 from October 25. When the model loads correctly, the log reports "lazy read enabled" for per_layer_token_embd.weight.

At 10 tokens per second, Ghumare comes out at about 2 MB/s and 500 random reads per second. That's roughly 0.06 percent of what a 3.6 GB/s NVMe drive can do. Frequent word sequences also end up in the page cache, the OS's disk cache in RAM, so they only cost a disk access the first time they appear. The company Capacity measured latencies on an Apple machine and estimates that a decent NVMe drive keeps up to around 40 to 60 tokens per second.

For home hardware, that means: according to Unsloth, DeepSeek-V4.1-Flash in its smallest variant (UD-IQ1_S, 233 GB on disk) needs around 122 GB for the rest of the model. Without offloading, the 110 GB table would come on top. That's what makes a 128 GB system realistic in the first place. You'll still need the full disk space, though.

Where tools miscalculate

Software that estimates in advance whether a model fits on your machine didn't know about this case at first. In Unsloth Studio, DeepSeek V4.1 wasn't on the list of models read "lazily." Studio therefore counted the entire 110.6 GB table toward RAM requirements. On a test system with 120 GiB RAM and a 32 GB GPU, it warned that around 186 GB would have to run from system memory (PR #14444). Unsloth didn't merge that draft; it replaced it with the more comprehensive PR #14465, merged on October 26. Since then, Studio lists the tables of DeepSeek V4.1, Qwen4-Exp, and Gemma 4 as a separate line item. With 24 GB VRAM and 64 GB RAM, the displayed shortfall shrank from about 132 to 29 GiB. If you use other calculators, treat their numbers for these models with skepticism. Kronk also has an open issue on this.

The limits

  • mlock and no-mmap: According to the orcarouter Model Card, --mlock or --no-mmap pull all 110 GB into RAM. In newer builds, these options are handled via -lm/--load-mode. -lzm off or a build older than b11870 also cancel out the benefit.
  • Long inputs: During prefill, when the prompt is read in, the model needs thousands of rows at once. Every row that isn't loaded yet pulls in an entire 4 KiB page via mmap for around 90 bytes and waits for it individually. In PR #28136, an RTX 5090 on Windows hit 167 instead of 621 tokens/s this way. That direct-read variant wasn't merged, and follow-ups are still open. Unsloth reports prefill speeds between 140 and 3,158 tokens/s for streamed tables, and server throughput that's 1.6 to 2 times higher with the table in RAM.
  • The 3 percent promise: The less-than-3-percent loss from the paper (Dense-4B: −1.9%, Dense-8B: −2.8%) applies to a table in an H800's DRAM. It's no evidence for SSDs. Small models lose out too: on Gemma 4 E4B, lazy reading in PR #27794 cost about 8 to 11 percent of output speed.
  • Outliers and drive type: Capacity measured individual reads of up to 36.8 ms. A hard drive at around 30 ms misses every time window. Unsloth doesn't check the drive type. On integrated GPUs, llama.cpp reads the whole table into memory anyway.
  • The experts remain the problem: Figures like 4 to 11 tokens/s for V4.1 Flash on 12 GB cards come from the engine developer Strata itself. What's slow there is loading the experts, not the table.
  • Conceptual criticism: The tables are frozen after training and don't learn anything new. The authors of Memory Grafting call training the tables from scratch "expensive and sometimes ineffective." NVIDIA describes converting the table from FP8 to MXFP4 as lossy.

Our assessment

Engram is the second big efficiency idea after MoE. MoE saves compute; Engram saves expensive VRAM by moving memorized knowledge into cheap storage. That DeepSeek, Qwen, and Tencent adopted it within a few months is a strong signal. According to The Next Web, Qwen explicitly designs the tables for accelerators with little memory. So far, though, the benchmark gains come almost entirely from the vendors themselves, and an independent replication of the paper is still missing.

Our take: for you at home, Engram is good news with fine print. The tables really can live on your NVMe drive without noticeably slowing down generation. But you need up-to-date software, mmap has to stay on, and you'll have to accept slower processing of long prompts. The big headline numbers still don't change much: a 224B model with 64B of Engram needs roughly as much RAM as a 160B model. Things get interesting once the announced small Qwen4 models and an open Hy4 flash arrive. That's when we'll see whether the third column makes the difference on regular PCs.

Sources

#engram#deepseek#qwen#tencent#llama.cpp#local ai

Ask the AI assistant

Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.

Connecting …

The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.

Comments

What do you think? Already tried it? Write it here, no sign-up needed.

  1. Loading comments …

Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.