Mid-training: What coding models read before they become agents

What this is about

The usual story is that a language model is built in two steps. In pretraining, it reads trillions of tokens from the web. In post-training, fine-tuning (SFT, supervised learning on example dialogues) and reinforcement learning (RL, learning through rewards) shape it into an assistant. For coding agents, meaning models that read code, run commands and fix bugs on their own, there's almost always a third phase in between: mid-training. Qwen3-Coder-Next, GLM-5 and Kimi-Dev all have one, and Cursor's Composer 2 uses something similar called "continued pretraining".

None of those reports explained how the data for this phase is put together. NVIDIA's Nemotron team addresses this in its new paper "Priming Base Models for Agentic Post-Training" (arXiv 2610.22019, October 15, 2026):

"Technical reports describe this phase only in broad strokes, but rarely explain how its data is constructed."

The paper lays out the recipe down to the ablation level. An ablation means deliberately removing one ingredient and measuring what changes. The core thesis: a model should first read what real software work looks like before it practices agent behavior. Japanese tech blogs like Zenn sum it up as: "Let it study the job site first, then teach it the format."

Why a third phase is needed

The authors point to three gaps. First, pretraining usually runs with only 4,000 to 8,000 tokens of context. Entire repositories, commit histories or long tool logs simply don't fit. Yet this data is too large for the short phase at the end of pretraining when the context gets extended. Second, pretraining has no loss masking, meaning a way to exclude certain parts of the text from what the model is scored on. So it's unclear how to handle tool outputs in agent trajectories. Third, according to the authors, very long SFT on synthetic dialogues can slow down later RL. Mid-training is meant to take learning work off the fine-tuning stage so that stage can stay short.

The term itself isn't standardized. At OLMo 2, the comparable phase was called a "late-stage curriculum" during annealing, the final stretch where the learning rate drops toward zero. Survey papers from 2025 explicitly criticize this inconsistent terminology. A CMU study (Liu, Neubig, Xiong) distinguishes mid-training from plain continued pretraining: in mid-training, you mix specialized data into general data instead of training on specialized data alone. In their experiments with small models, the mix performed better on code and math and caused less forgetting.

The recipe: What the model gets to read

NVIDIA developed the method on Nemotron 3.5 Lightning (30 billion parameters, 3 billion active per token) and tested it on Nemotron 3.5 Super (120B, 12B active). Training uses packed sequences 256,000 tokens long. The mix for the 500-billion-token run:

Do the math and about 65 percent is natural software data: human-written code plus its changes and logs. Only a tenth is in chat format. According to Zenn, the share of trajectories is deliberately kept at 10 to 15 percent so the model doesn't lock into the rigid dialogue format too early. Some of the trajectories come from Perplexity, a partner in NVIDIA's "Nemotron Coalition".

Two details show the thinking behind it. Repositories only make it into the mix if at least 20 percent of their code files import other files from the project (10 percent for Ruby). The files are then arranged so that each file appears only after the files it depends on. That way, the model reads a codebase in the order in which its structure makes sense. The authors justify this cautiously: dependency ordering separates strong models from weak ones better than alphabetical sorting does. In other words, it exposes cross-file relationships that weaker models haven't learned yet. The second detail concerns the agent trajectories. The model is trained only on the assistant's turns. It sees what tools return, such as terminal output or file contents, but doesn't have to predict it.

What the ablations show

The key individual experiments, each followed by identical fine-tuning:

Without trajectories, then, reading real software alone gains less than a point on SWE-bench Verified, the test where a model has to fix real GitHub bugs. Most of the gain only appears in combination with the trajectories. Trajectories alone, the authors argue, would essentially just be a long SFT and would hinder later RL. That's their reasoning, not a measured result. Training on the often long and noisy tool outputs costs about ten points. According to Zenn, they apparently pull learning capacity away from the actual agent actions.

The learning rate reveals the method's costs. It determines how much the model is reshaped with each step. Higher values deliver more agent performance but wreck math and general knowledge: MATH-500 falls from 79.7 to 61.7, MMLU from 79.4 to 76.8. NVIDIA deliberately made different choices: Lightning runs at 5e-5, prioritizing agent gains over general knowledge, while Super uses a cautious 1e-5. With that setting, all general benchmark groups for the Super model stayed within 1.2 points of baseline after 65 billion tokens.

What it adds up to

For Lightning, mid-training was gradually extended to 500 billion tokens:

That's +3.8 on Verified, +7.0 on Multilingual and +3.5 on Pro. The pass@k metric counts how many tasks get solved at least once across several attempts. It rises by 2.3 to 4.7 points, so the model really can solve more tasks. Context stands out: on SWE-bench Pro, 91 of 2,193 runs hit the 256K limit without mid-training, versus just 13 afterward. That's about seven times fewer. Summed across all four benchmarks, the count drops from 164 to 54, roughly a third. The "seven times fewer" applies only to SWE-Pro.

The team also measured across different harnesses, the agent programs the model works in. For the Super model after 65 billion tokens and a short fine-tune, the average gain on Verified was 8.6 points. For Lightning, the score grew by 19.0 points in Codex and 14.9 in Claude Code, but only 3.8 in OpenHands. An initial RL test with 20 steps shows that the lead of about two points survives early RL. Format errors fell from 0.34 to 0.03 percent. The authors themselves stress that this doesn't prove mid-training raises RL's ceiling.

The phase really is part of Nemotron 3.5 Super's production training: 65.4 billion tokens at 256K (the Model Card says 67 billion), with a 41/20/10/3/1/20/5 percent mix. There was a side effect, though. The ability to handle a million tokens collapsed to around 33 points on the long-context test RULER. Only a short recovery phase of about 9 billion tokens brought it back to 81.

What others reveal, and what they don't

Kimi-Dev already used the idea in 2025: merged PR diffs and commit sequences served as a "skill prior", prior knowledge about finding bugs and editing code that the later agent builds on. One contradiction remains unresolved. Cursor picked the base model for Composer 2 using internal metrics such as perplexity on codebases. Perplexity measures how surprised a model is by a text. According to Cursor, cross-entropy loss does predict later RL success. NVIDIA, by contrast, found that such base-model metrics (bits per byte, LoCoDiff) do not reliably predict results after fine-tuning. So the team evaluates every data decision with a short, fixed SFT run of 33 billion tokens.

Our verdict

The paper's value lies less in the point gains than in its openness. For the first time, you can read which ingredients a coding model gets between pretraining and fine-tuning, and what each one does. That partly explains why open agent models like Xing4.0 or Reflection Beam now reach high SWE-bench scores. Whether their makers do it the same way, though, they don't say.

Our take: the numbers are solid but modest, and they haven't been independently verified. Almost every comparison rests on a single training run. The authors themselves put the run-to-run variance for the Super model at about one point on Verified and two on DeepSWE. The Verified gain of +1.4 for the Super model and +1.0 on Terminal-Bench for Lightning therefore barely clear the noise. For the large model, Multilingual (−1.1) and Pro (−2.1) actually dropped after full fine-tuning. Between 100 and 500 billion tokens, Verified moves within a single point. So more data only visibly helps on Multilingual at this point. There's no comparison where the same token budget goes into longer pretraining or fine-tuning instead. And testing was limited to Nemotron models.

Also, only the description is "open" so far. NVIDIA's documentation explicitly states that the data mixes aren't public and the phase can't be reproduced. The paper only says processing scripts and metadata will appear in a later version; there's no mention of the data itself. On Hugging Face, you'll currently find only NVIDIA's older SFT and RL datasets for software tasks, not the mid-training material. If you want to cook this recipe yourself, you'll have to source the ingredients on your own for now.