LatentLoop: Looped MoE saves compute by shortening the first pass

What it's about

A research paper titled "LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation" has appeared (arXiv 2610.19975, submitted October 14, 2026). It deals with language models that run some of their layers several times, and asks whether every pass needs the same amount of compute. The authors say no. They link code on GitHub, a Hugging Face entry and a project page. The available sources don't say what exactly is hosted there, for example whether finished model weights are included.

The basics: looped MoE

A Mixture-of-Experts (MoE) model has many specialized sub-networks, the "experts." A router picks only a few of them for each token (a chunk of text). That lets a model store a lot of knowledge while computing only a fraction of it per token. In a looped model, layers are reused several times in a row. This saves stored parameters (the dials adjusted during training), but each extra pass still costs compute.

The baseline is SMELT: the middle 50% of the Transformer layers run twice, and both passes share attention, router and experts. LatentLoop is compared with this approach and with a non-looped MoE. All have the same stored parameters, the same KV cache (a memory of the text so far), the same FLOPs per step and the same total training FLOPs. FLOPs are floating-point operations, a measure of compute.

The observation: two passes, two roles

The authors added a learnable scaling factor for the residual branches to each pass. Residual branches are the shortcuts through which each layer adds its output to the existing state. The result: the first pass gets much smaller factors than the second, for example about (0.27 / 0.87) at 1.3B and about (0.36 / 1.09) at 3B.

Control runs with fixed factors, trained from scratch, show the same thing. With (0.25 / 1), validation NLL is lowest: about 0.008 better than (1 / 1) at 1.3B and about 0.003 at 3B. If you shrink the second factor instead, results get worse. (0.25 / 1) also beats the symmetric (0.5 / 0.5) from earlier work. So the two passes seem to want changes of different strength. That raised the question of whether they should also get different amounts of compute. According to the analysis, the experiments don't show that the scaling factor itself explains the success of the final design.

How LatentLoop works

• Compress: After the input layers, a parameter-free encoder averages every two neighboring token states into one "latent" position (compression factor k=2). The first pass therefore runs on only half as many positions.
• Write back: The first pass's update is applied to the full sequence using learned coefficients per channel and position. Causality is preserved: a pair's update reaches its last token and the next one, but never earlier tokens. This costs only 2d extra parameters (d = model width).
• Refine: The second pass processes all tokens at full resolution, followed by the output layers.
• Reallocate: The saved compute is invested in a larger model width (hidden size).
• Share: MoE router and experts stay shared between the passes. The attention blocks have separate parameters because the sequence lengths differ.

In ablations, a learned decoder beat direct write-back. Learning an encoder as well brought no gain.

Numbers and results

The key test is where to compress. At 3B and 40B training tokens:

Here, NLL is the cross-entropy loss when predicting the next token (lower is better). It is measured without the router auxiliary loss on a validation set of 0.1B tokens. According to the authors, the "only first" and "only second" variants have identical parameter counts, KV memory, FLOPs per step and training updates. So placement makes the difference.

Across all 15 tuned endpoints (1.3B, 3B, 7B), LatentLoop has a lower NLL than both MoE Base and MoE Loop. Fitted compute-optimal curves give a requirement of 1.85×10^20 FLOPs at NLL 1.988. That is 19.0% less than MoE Base and 7.9% less than MoE Loop, within the observed range.

For downstream tasks, nine benchmarks were averaged with equal weight: LAMBADA OpenAI, LAMBADA Standard, HellaSwag, PIQA, WinoGrande, ARC-Easy, ARC-Challenge, SciQ and MMLU (continuation format), each 0-shot and 5-shot. LatentLoop has the highest average in all six size/shot combinations, though not in every single benchmark cell. For the 7B model trained on 140B tokens, the 5-shot average is 63.40% versus 62.55% for MoE Loop.

Our take

The paper delivers a clear, well-controlled result: under the same budget, it pays to shorten the first pass and run the second at full length. That matters most for researchers training more efficient architectures. There are limits: the savings are predictions from fitted curves for training FLOPs. They aren't measured wall-clock time, they don't mean faster inference, and they say nothing about using fewer tokens. The models are comparatively small at up to 7B parameters, and the benchmark gains are modest (around one percentage point). The authors call asymmetric compute allocation a "promising direction," not a fully explained mechanism. Independent verification is still pending.