Looped models explained: they save memory, not compute time

What this is about

Within a few weeks, two open-weight language models have shipped that use an unusual trick: NVIDIA Dory1.0-4B (weights out since mid-October, paper dated November 3/4) and SenseTime's SenseNova-Loop-9B-Preview (October 30). Both send every token through the same block of layers several times. These are called looped or recurrent-depth models: models with loops in their depth.

Our take: looping saves memory for weights, but it doesn't save compute time. Every loop costs as much compute as the layers it repeats. That's why the approach needs new tricks to pay off in practice. Dory is the best example so far, because NVIDIA puts numbers on the costs itself. Alongside parameters and training data, the number of passes becomes a third knob for scaling.

The basic idea: one tool, applied several times

A standard transformer sends each token through a stack of layers exactly once. If you want more depth, you need more layers, and therefore more parameters. A looped model splits the stack into three parts: an input block, a repeated core block and an output block. On each pass, the core keeps working on the intermediate result from the previous pass. Think of a tool you apply again and again to a half-finished workpiece.

The idea isn't new. Universal Transformers (Dehghani et al., 2018) combined the parallelism of the transformer with the recurrent principle of older networks. They added a mechanism that decided, per position, when to stop. In 2025, Geiping et al. trained Huginn-3.5B on 800 billion tokens. According to the paper, it keeps improving on math and coding tasks with more passes, up to a compute budget equivalent to 50 billion parameters. ByteDance followed with Ouro (1.4 and 2.6 billion parameters, 7.7 trillion tokens). Its authors explicitly attribute the gains to better processing of knowledge, not to more stored knowledge. Keep that point in mind.

A footnote on the numbers: Sebastian Raschka counts Dory as 8/20/8 blocks, the paper as 16/40/16 layers. Both are right. Dory has 36 pure MLP layers and 36 attention layers, so one block consists of two layers.

The catch: memory saved, compute time not

With Dory, the number of layer passes per token rises from 72 (one loop) to 112 (two) and 192 (four). That's 1.56 and 2.67 times as much work per token, respectively. It shows in practice, too. Hugging Face user cloudnathan5 built his own GGUF files (GGUF is the file format used by llama.cpp). On an RTX PRO 6000 with Q5_K_M, he measures 193.5 tokens/s with one loop and 80.9 tokens/s with four, roughly 2.4 times slower.

Then there's the KV cache, where the model keeps attention results for every previous token. By default, each loop gets its own. According to cloudnathan5, Dory in Q4_K_M with four loops needs 3.6 GiB at 4,000 tokens of context, 6.2 GiB at 32,000 and 27.4 GiB at 256,000 tokens. At full context, this 4-billion-parameter model needs more memory than the weights of many 30B models.

SenseNova looks similar. 56 instead of 32 layer passes means 1.75 times as much compute per token. According to a code analysis on note.com, each pass gets its own cache here as well. That's softened somewhat because three out of four layers use linear attention. This variant gets by with fixed-size memory instead of one entry per token. SenseTime doesn't give concrete memory figures.

Trick 1: Think harder, talk less

NVIDIA's key argument is in the paper: with four loops, Dory generates 1.3 to 2.5 times fewer tokens on agentic tasks than with one. On GDPval-AA v2, turns per attempt drop from 116 to 49 and output from 212,000 to 85,000 tokens. On Terminal-Bench v2.1, output falls from 109,000 to 54,000 tokens.

We ran the numbers: total compute = tokens × layer passes. For GDPval, that gives 85/212 × 192/72 ≈ 1.07 times the compute for four loops; for Terminal-Bench, 54/109 × 192/72 ≈ 1.32 times. According to the paper, the range runs from 1.07 to 2.04 with four loops and from 1.03 to 1.25 with two.

Our take: this works mainly on long agentic tasks, where a smarter model avoids detours. A short chat question leaves hardly any tokens to save, so you pay nearly the full 2.4x compute time. Developer Laurence Moroney finds the trade-off plausible but unproven. He hasn't seen measured latencies on laptops at two to four loops anywhere.

Trick 2: Last-loop KV caching

With last-loop caching, there's only one cache per layer, and each loop overwrites the previous one. Memory stays as small as with a single loop. The catch: for older tokens, early loops look at values from the last loop instead of their own, which isn't how they were trained. According to NVIDIA, results shift by −2.12 to +0.42 points. AA-LCR (−2.12) and Terminal-Bench (−1.97) take the biggest hits. In cloudnathan5's measurements, memory at 32,000 tokens drops from 6.2 GiB (four loops, separate caches) to 4.1 GiB. The llama.cpp PR notes another quirk: in last-loop mode, predictions depend on the chunk sizes the prompt is processed in.

Trick 3: The model as its own draft helper

Standard speculative decoding uses a small helper model that quickly guesses several tokens. The big model then verifies them in one go. Dory doesn't need a second model for this. The same file drafts four tokens with one loop, and the four-loop variant verifies them. Both share the cache of the input block and the first loop, and intermediate results from the draft get reused. According to the paper, the four-loop variant's output distribution is mathematically preserved. Decoding throughput rises by 1.37 to 1.71 times, with the advantage shrinking at larger batch sizes. Per request, latency drops by 1.11x (τ³-Banking) to 1.53x (AA-Omniscience). The paper doesn't say how many drafts get accepted. The pull requests don't make clear whether llama.cpp or vLLM already support self-speculation.

Elastic depth: one checkpoint, four models

During training, the number of loops was picked at random from {1, 2, 3, 4}, with harder tasks getting more loops more often. That's why a single checkpoint works with any loop count. NVIDIA calls this "elastic inference compute." The vendor's numbers show where it helps and where it doesn't:

Two loops already capture 80 percent of the gain on SWE-Bench and 86 percent on GDPval. After that, the curve flattens. For factual knowledge (AA-Omniscience), more loops do nothing at all, which matches the Ouro findings. On GPQA, Moroney notes that Dory's 72.41 trails Qwen3.5-4B (77.71). All numbers come from NVIDIA; we found no independent replication.

What matters in practice: for Dory, "elastic" means you set the loop count at launch. It doesn't adapt on its own per request or per token. SenseNova runs with a fixed two passes. For Ouro, Raschka says the released code always computes all four passes and only picks an output afterwards, so the promised savings remain theoretical there. True variable depth fits poorly with today's serving systems like vLLM, because tokens then exit at different points. The paper "Continuous Depth Batching" (Schwethelm et al.) tackles this by regrouping batches between loops. On Ouro and Huginn, it reaches up to 99 percent of the estimated achievable speedup.

Pitfalls when running locally

• llama.cpp: Dory support lives in draft PR #30306, which hasn't been merged yet. Standard builds and Ollama won't load the model. NVIDIA's own GGUF requires a pinned commit from a fork. According to the PR, Metal is untested, and context shifting (set --no-context-shift) and more than four loops aren't supported.
• Virtual layers: llama.cpp counts every loop pass as its own layer: 16 + 40 × loops + 16, so 192 with four loops. If you set -ngl 40 out of habit, you'll quietly offload almost everything to the CPU. One tester in the PR reports that only the output head ended up on the GPU. The right setting is -ngl 999.
• Loops via override: You set the count with --override-kv dory.num_loops=int:2. The cloudnathan5 files default to four loops if you don't specify anything; NVIDIA recommends starting with two.
• Contradictory guides: Several blogs call vLLM PR #60149 "open," but according to GitHub it was merged on October 25. Transformers, on the other hand, has no dory.
• SenseNova and trust_remote_code: The loader executes the Python file modeling_sensenova_loop.py from the repo. AiCybr recommends reading it first and pinning a specific revision. Transformers 5.18.0 is required. The download is roughly 38 GB, well above what 9 billion parameters in BF16 would suggest. The vLLM and LightLLM examples use four GPUs, and there are no official GGUF files.

What research still leaves open

The new papers show where things still snag. The Nanbeige4.2 team found that more than two passes make training much more expensive without much benefit. The paper "Allocating Recurrent Compute" repeats only the attention part instead of the whole block. That saves 45.9 percent of the operations in the core but, at 110 million parameters, keeps only 41.5 percent of the gain. "Readout Blind Spot" shows on small models (44 and 129 million parameters) that internal values can grow into the thousands without normalization between loops. Our take: Dory normalizes the state before and after each loop, which is exactly the direction the paper recommends. That link hasn't been proven, though. A solo author retrofitted loops into Qwen2.5-0.5B, but only for one narrowly defined arithmetic task.

The bigger picture

Looped models aren't a free upgrade. They trade memory for compute time. That's interesting for local users whose GPU is short on VRAM but has plenty of compute. Dory in Q4_K_M gets by with 2.4 GB of weights, but with four loops, decoding takes about 2.4 times as long. Whether that's worth it comes down to Trick 1: does the model actually save as many tokens on your tasks as it does on NVIDIA's agentic benchmarks?

Our take: NVIDIA is the first major vendor to put honest cost numbers and workable countermeasures on the table. For SenseNova, a paper, memory figures and compute costs are still missing entirely. Three things remain open: independent benchmarks, self-speculation in mainstream tools, and depth that truly adapts automatically to each request. Until then, "recurrence" is a promising third scaling knob, but one for tinkerers.