4-bit training: Six times faster on paper, 8 percent in practice
Thinking Machines trained its 594-billion-parameter model Inkling 2 Small largely in 4 bits. The spec sheet promises six times the compute throughput. During pretraining, the lab measured an 8 percent gain. We explain where the rest goes, and why the real winners are the people who run the finished model.

What this is about
Thinking Machines Lab, the startup founded by former OpenAI CTO Mira Murati, explained on November 13 in its blog Connectionism how it trained Inkling 2 Small. It's a mixture-of-experts (MoE) model with 594 billion parameters, 17 billion of which are active per token. In an MoE, a router picks just a few specialist blocks, the so-called experts, for each token. The lab says Inkling 2 Small is the largest known model trained with a native 4-bit forward pass from the first step of pretraining to the last reinforcement learning run. The forward pass is the computation in which the model turns an input into a prediction.
The main sales pitch for this format comes from Nvidia's spec sheets. According to Thinking Machines, a GB300 hits up to 15 petaflops in NVFP4, 5 in MXFP8 and 2.5 in BF16. That's six times the BF16 figure. Yet the lab measured only an 8 percent time saving in pretraining and 20 percent in RL. This article explains where that gap comes from. It also makes our case: for the big labs, 4-bit training saves little time. It matters more for everyone who runs the finished model later.
What "4 bits" really means here
A 4-bit number can only take 16 distinct values. In the E2M1 format used by NVFP4, those are 0, 0.5, 1, 1.5, 2, 3, 4 and 6, each with a sign. To still represent both large and small numbers, NVFP4 uses block scales. Every 16 values share an 8-bit factor they get multiplied by, like a map where each section has its own scale. On top of that comes an FP32 factor for the whole tensor. Effectively, that's 4.5 bits per value. The 8-bit format MXFP8 shares one factor across 32 values, which comes to 8.25 bits.
Thinking Machines' recipe uses 4 bits selectively:
| Part of training | Format |
|---|---|
| Forward pass of the expert layers | NVFP4 |
| Backward pass (gradients, both matrix multiplications) | MXFP8 with stochastic rounding |
| Final 16.6% of layers | MXFP8 |
| Outputs before they're sent back between GPUs ("combine") | S2F6 after Hadamard rotation, 8.25 bits |
| Master copy of the weights, optimizer | FP32 |
The backward pass works out which direction each weight needs to be nudged. It stays in 8 bits. According to the blog, early attempts at full NVFP4 training led to "very large" degradation. The catch: every stored activation has to be unpacked, transposed and then repacked in MXFP8. This requantization costs compute time. Stochastic rounding helps here: instead of always rounding to the nearest value, it rounds randomly. 2.25 becomes 2 with 75 percent probability and 3 with 25 percent. On average, the value stays correct and no systematic bias creeps in. For activations, the lab also uses Four Over Six (arXiv 2512.02010). Some blocks are scaled so their largest value lands on 4 instead of 6, where the number grid is finer.
Where the speedup goes
The 6x applies against BF16. But Thinking Machines compares against a run done entirely in MXFP8, and against that, NVFP4 delivers only 3x on paper. The lab likewise estimates the theoretical gain for the forward pass of the MoE layers at around 3x, before communication limits.
Our back-of-the-envelope math: in a training step, roughly a third of the matrix math happens in the forward pass and two thirds in the backward pass. Even if everything in the forward pass ran three times faster and nothing else slowed you down, you'd save about 29 percent (1 / (2/3 + 1/9)). In reality, only part of it runs in 4 bits, namely the experts minus the final layers. Then there are the brakes the blog itself names:
- Communication: Once the compute cores get faster, the link between GPUs (NVLink) becomes the bottleneck. According to the blog, each GPU could perform up to 16,000 operations in the time it takes one byte to arrive. That's why the lab ships tokens in NVFP4. For the return trip, it compresses data with a Hadamard transform, a rotation that spreads outliers across 32 values, and the fixed-point format S2F6. That halves the traffic from 16 to 8.25 bits. MXFP8 wasn't an option here because it handles outliers too poorly.
- Overhead: Computing scales, requantizing and stochastic rounding all load up the compute units. The tensor cores sometimes end up waiting on them.
- Quality: The lab says the recipe costs about 2 percent in compute efficiency. The model reaches the loss an MXFP8 run would have hit with roughly 2 percent less compute.
Our take: the net saving in pretraining comes to about 6 percent for the same quality (1.08 / 1.02). For the first pretraining phase, which ran 68 hours on GB300 racks, that's roughly four to five hours. Murati herself called 4 bits "one of many improvements across the stack" that cut pretraining from weeks to 68 hours. It makes sense that RL benefits far more, at 20 percent. There, the model constantly generates text in rollouts tens of thousands of tokens long, which is pure forward computation. The blog stresses that every token has to load the weights from memory, and fewer bits per weight speed up exactly that.
The replication on a B200
A user on r/LocalLLaMA recreated the core recipe on a single B200: NVFP4 forward, FP8 with stochastic rounding backward. He trained GPTs with 219 million, 840 million and 3.3 billion parameters on TinyStories. Code and measurements are on 4bit.vizuara.ai and GitHub.
| Model (840M, 2,000 steps) | Time | Validation loss |
|---|---|---|
| BF16 | 336 s | 1.688 |
| All 8-bit | 274 s | 1.686 |
| 4-bit forward, 8-bit backward | 259 s | 1.691 |
Versus BF16, the per-step saving was 18, 28 and 38 percent, depending on model width. A single matrix multiplication ran 3.6 times faster in 4 bits than in BF16. The fair comparison, though, is against 8 bits, and there the 4-bit recipe was only 1 to 6 percent faster depending on the run. Both reached the same loss at practically the same time (262 vs. 264 seconds). On an H100 without FP4 cores, every step was 3.4 times slower. The author is upfront about the limits: one GPU only, no expert communication, no Four Over Six, mostly a single seed. So the numbers confirm the trend, but not any detail of the big run.
Who really benefits: the people running the model
The more important line in the blog is almost an aside. Because a single numerics recipe applies from pretraining to the final RL run, the lab says there's no calibration step when shipping a quantized checkpoint at the end. The model isn't shrunk after the fact; it never computed any other way. The replication shows the effect in miniature: quantizing a finished BF16 model to 4 bits after training raised the loss from 1.690 to 1.695. The model trained in 4 bits for the same number of steps landed at 1.692.
Other labs go halfway. OpenAI post-trained gpt-oss with MoE weights in MXFP4 (4.25 bits) so the 120B variant fits on an 80 GB GPU. For Kimi K2 Thinking, Moonshot uses quantization-aware training in post-training, meaning the model already "feels" the later rounding during training, to get INT4 expert weights. Moonshot's reasoning: thinking models write such long answers that post-training quantization often costs significant performance. The result is supposed to be roughly twice-as-fast text generation. In research, others go further than Thinking Machines, though on smaller models:
| Approach | What runs in 4 bits | Size | Reported gap |
|---|---|---|---|
| Nvidia (arXiv 2509.25149) | all three matrix computations, 16% of layers at higher precision | 12B, 10T tokens | loss gap under 1%, ending just over 1.5% above FP8; MMLU 76.57 vs. 77.36% |
| Quartet II (arXiv 2601.22813) | forward and backward entirely | up to 1.9B | 1.44% above BF16 (Nvidia recipe in the same setup: 1.92%) |
| Thinking Machines | forward only, experts | 594B MoE | approx. 2% compute efficiency vs. MXFP8 |
The percentages aren't directly comparable, since some measure loss gap and others compute efficiency.
The catch: "4 bits" doesn't mean an eighth of the memory
According to Hugging Face, the Inkling checkpoint is 394 GB. With 594 billion parameters, that's about 5.3 bits on average, because the scales, the attention layers and the higher-precision final layers all add up. The Model Card calls for at least 500 GB of VRAM, for example two B300s or four H200s. The MXFP8 variant needs 800 GB and, according to the lab, is only 1 to 2 percentage points better. For smaller machines, there are GGUF versions from Unsloth starting at 171.7 GB. These require a llama.cpp build with Inkling 2 support (PR 32187). But that brings back post-training rounding: the smallest variant picks the same most likely token as the original in only 89.7 percent of cases.
Fine-tuning gets even tighter. Patronus AI reports on an RL run with the NVFP4 model GLM-5.2 (744 billion parameters). Nvidia's Transformer Engine library stores every weight matrix twice, once normally and once transposed for the backward pass, because the block scales are tied to the direction of computation. Instead of the expected 55 GB, the base model took up 110 GB per GPU, on a B200 with 180 GB. According to Patronus, the 4-bit model was thus "silently back at roughly 8 bits." Only a patch brought it down to 56 GB, at the cost of a slower backward pass.
Then there's the hardware question. Only Blackwell chips can actually compute in FP4. On Hopper or RTX 40 cards, vLLM's own docs say it unpacks the 4-bit weights for computation (Marlin, "W4A16"). That saves memory but gives you no FP4 speed boost.
Our take
The 8 percent isn't a failure. It's an honest number. On runs across thousands of GPUs, even a few percent is a lot of money. But it shows that FP4 spec-sheet figures tell you very little about time-to-finished-model. Even Nvidia quotes 50 petaflops of NVFP4 for Rubin in inference, but only 35 for training. The real value lies elsewhere: Inkling 2 Small ships with no post-training quality loss, in a format that modern servers run fast natively. As Sameer Singh put it on X, it makes no sense to train a model in FP16 that's never meant to run at that size.
Plenty remains open. For the 2 percent, the lab gives no absolute loss values; for inference, no speed measurements of its own; and for the ablations, no model sizes. So far, only the basic principle has been independently verified, on a single GPU. Thinking Machines itself has announced the next step: moving the backward pass to FP4 as well. Only then would the numbers come close to the spec sheet.
Sources
- Thinking Machines: End-to-End 4-Bit Training (Connectionism) thinkingmachines.ai
- Thinking Machines: Introducing Inkling 2 Small thinkingmachines.ai
- Hugging Face: thinkingmachines/Inkling-2-Small huggingface.co
- Hugging Face: Dateiliste Inkling-2-Small huggingface.co
- Vizuara: 4-Bit-Training auf einer B200 (Nachbau) 4bit.vizuara.ai
- GitHub: VizuaraAI/4bit-training-b200 github.com
- Patronus AI: GLM-5.2 NVFP4 QLoRA RL cdn.patronus.ai
- Nvidia Dynamo: Inkling-Rezepte docs.nvidia.com
- arXiv: Pretraining Large Language Models with NVFP4 arxiv.org
- arXiv: Quartet II arxiv.org
- arXiv: Four Over Six arxiv.org
- Unsloth: Inkling 2 Small lokal ausführen unsloth.ai
- Hugging Face: unsloth/Inkling-2-Small-GGUF huggingface.co
- Kimi: Introducing Kimi K2 Thinking kimi.com
- arXiv: gpt-oss-120b & gpt-oss-20b Model Card arxiv.org
- vLLM-Doku: ModelOpt-Quantisierung docs.vllm.ai
- Nvidia Developer Blog: 3 Ways NVFP4 Accelerates AI Training and Inference developer.nvidia.com
- Sameer Singh auf X x.com
- Latent Space: AI News latent.space
- SiliconANGLE: Nvidia auf der SC26 siliconangle.com
Ask the AI assistant
Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.
The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.
Comments
What do you think? Already tried it? Write it here, no sign-up needed.
Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.