NVIDIA Dory: A 4B model that thinks in loops and beats bigger rivals

What's going on

NVIDIA has released a new small language model called Dory. It comes out of the GPU maker's Nemotron research, has 4 billion parameters and ships under the Open-MDW license, which allows both research and commercial use. One of the people who put it on the map is AI explainer Sam Witteveen, who covered it in a detailed video. Geeky Gadgets wrote up a summary, and the r/LocalLLaMA community has been discussing it as a new release.

What makes Dory interesting isn't its size but its design. It's a looped transformer: a model that runs part of its layers several times in a row instead of using each layer just once. NVIDIA is tackling an old question here: how do you get more computational depth without the model needing more memory?

How the looping works

A classic transformer, the base architecture behind almost every language model today, sends each token through a stack of layers exactly once, from bottom to top. More layers usually mean a smarter model, but also more parameters and therefore more memory.

Dory splits its layers into three blocks:

• an input block that prepares the text,
• a recurrent block that runs multiple times,
• an output block that produces the result.

The middle block works like a revision pass: each loop further refines the model's internal representations of the words, a bit like proofreading a text several times. According to Geeky Gadgets, one, two or four passes are typical. Because the same layers are reused every time, the parameter count doesn't grow.

To keep the model from "drifting off course" over multiple rounds, Dory borrows an idea from ResNet image models: the original input is fed back in on every pass. That's meant to keep things stable and prevent information from getting lost along the way.

Elastic compute: speed vs. quality

The most practically exciting feature is what NVIDIA calls "elastic inference compute". It means you decide at runtime how many times the middle block loops:

• Fewer passes give you fast answers, for example for simple questions or real-time applications.
• More passes cost more compute time but should deliver better results on harder tasks.

The memory needed for the model weights stays the same either way. A single model can act as a quick "short thinker" or a more thorough "long thinker" depending on what you need, without you having to keep two different models around. Geeky Gadgets describes this as a lean alternative to mixture-of-experts models, which are fast but need a lot of memory.

Training and performance

Dory was pretrained on 6 trillion tokens. That was followed by supervised fine-tuning (SFT), where the model learns from example answers, and reinforcement learning, where it gets rewarded for good answers.

On benchmarks, Dory reportedly shines in two areas: following instructions and understanding long texts. According to the report, it even beats much larger models like Gemma 4 26B here. Geeky Gadgets doesn't give concrete numbers, though.

The weaknesses are spelled out just as clearly: on knowledge-heavy tasks, meaning questions where the model has to know lots of facts by heart, Dory can't keep up with bigger models. That makes sense. More loops make the model think more thoroughly, but knowledge is stored in its parameters, and it only has 4 billion of those. On top of that, extra loops don't improve every task equally.

What you need to try it

At 4 billion parameters, Dory is small. With 4-bit quantization (shrinking the model by storing its numbers at lower precision), the weights fit into roughly 3 to 4 GB of VRAM; at full 16-bit precision you need about 8 to 9 GB. A regular 8 GB graphics card should basically do the job. Apple Silicon Macs with 16 GB of memory should also be enough, and the size works for Google Colab's free T4 GPU too. A 4B model will run on the CPU alone, but slowly, and every extra loop makes it slower still.

One catch: according to Geeky Gadgets, the KV cache, the buffer where the model keeps track of the tokens it has already processed, grows with more passes. With four loops and long context, you'll need noticeably more memory than the raw model size suggests. Leave yourself some headroom.

Since the looped architecture is new, popular tools like llama.cpp or Ollama may not support it yet. The sources don't say whether GGUF files exist. Your safest bet is the official guide on NVIDIA's model page, most likely using Hugging Face Transformers.

The bigger picture

Dory is first and foremost a research release, and Geeky Gadgets also stresses that it probably needs more optimization before it's ready for broad production use. Looped transformers aren't a brand-new idea, but here a major vendor has implemented one as a fully trained, openly licensed model. That puts the concept within reach of anyone who wants to experiment with it.

For people with modest hardware, the approach is appealing: a small model that "thinks longer" when needed, instead of a big one that always eats a lot of memory. That fits edge devices, smartphones, or applications that sometimes need fast answers and sometimes thorough ones.

Plenty remains open. The sources give no concrete benchmark numbers, no context length and no speed measurements. The comparison with Gemma 4 26B covers only selected disciplines. How well Dory holds up in everyday use, say in languages other than English or for coding, is something independent testing still has to show. Another intriguing question is how the concept scales to larger models. Geeky Gadgets sees potential there, but that hasn't been proven yet.