Does the trick really help? What 127,000 AI papers reveal about their own tweaks
A new, free atlas used AI to tally how often 37 common tricks actually help across 126,758 research papers, according to the papers themselves. Components that authors add "help" almost every time. Simply turning up a setting helps far less often. We re-ran the numbers, compared them with independent replications and spot-checked twelve entries against the original papers.

What this is about
Almost every new AI model and almost every research paper comes with an "ablation." That's a control experiment in which the authors remove or add individual components to show that their component is what drives the improvement. The free scime atlas from young company Latentic Labs has now counted these experiments at scale. According to the operators, an AI read the full text of 126,758 papers from the major conferences and journals between 2016 and 2026 and extracted 1,273,440 results. The page "Does it actually help?" shows, for 37 common tricks, how often they help according to the papers themselves.
The most striking pattern is a gap. Components you add to a model "help" almost every time. Knobs you simply turn up help far less often. Our thesis: this gap mainly measures what authors want to show, not what actually works. But as our check shows, it's not quite that simple.
What the atlas counts, and how
According to the methodology, an AI model extracted, for each tested change, what it was tested on, which metric counted, what number came out and whether the change helped. This was explicitly not checked by hand. Mapping to the 37 tricks is done "by wording." Only changes that add, increase or use something are counted. Removals, reductions and swaps are excluded. Each paper counts once per trick, with the outcome its findings on that trick show by majority.
In total, the atlas covers 621,064 added changes from 110,393 papers. 69 percent of them helped, 10 percent showed no clear effect, 16 percent made results worse and 5 percent were mixed. 66 percent of papers report at least one change that didn't help. The operators themselves warn that failures get published less often, so the true failure rate is higher.
A few inconsistencies jump out right on the site. The methodology section talks about "37 patterns," but the table of "usual suspects" shows only 31. 137,217 papers are listed, 126,758 were read in full. The site doesn't explain why more than 10,000 are missing. There's no raw data to download.
The numbers: components vs. knobs
The atlas splits the 37 tricks into "components" (adding a part, such as a loss or a module) and "knobs" (turning up a setting, such as learning rate or model size). A selection:
| Trick | Type | Papers | Helped | Worse |
|---|---|---|---|---|
| Add contrastive loss | Component | 1,317 | 95% | 2% |
| Add attention module | Component | 492 | 90% | 5% |
| Knowledge distillation | Component | 2,108 | 89% | 7% |
| Add dropout | Component | 795 | 69% | 18% |
| Chain-of-thought prompting | Component | 1,107 | 69% | 18% |
| Gradient clipping | Component | 88 | 64% | 12% |
| More training data | Knob | 1,169 | 78% | 6% |
| More layers | Knob | 4,253 | 69% | 17% |
| Larger LoRA rank | Knob | 233 | 46% | 20% |
| Higher learning rate | Knob | 452 | 44% | 37% |
| Higher sampling temperature | Knob | 122 | 34% | 42% |
A quick primer: a contrastive loss is a training objective that pulls similar examples closer together inside the model and pushes dissimilar ones apart. The LoRA rank sets how big the small add-on module is when you fine-tune efficiently with LoRA. Temperature determines how randomly a language model picks its next word. Chain-of-thought means the model writes out its reasoning steps before answering.
The 82 and 64 percent figures aren't on the atlas page. We calculated them from the 37 rows. Weighted by number of papers, the 20 components come to 82.3 percent and the 17 knobs to 64.6 percent. Unweighted, it's 80 vs. 58 percent. The knob figure leans heavily on "more layers" with 4,253 papers. Without that entry, it drops to just under 62 percent.
Pushback: not every gap is spin
There's also a purely mechanical reason knobs score worse. Learning rate, temperature or LoRA rank are usually swept, meaning tested across a range of values. Such sweeps almost always have a best value somewhere in the middle. If the default is already well tuned, turning it up further will inevitably make results worse. So the fact that a higher temperature hurts more often than it helps isn't a sign of unusual honesty. It's the expected result of testing around the optimum. The 64 percent isn't a neutral truth you can simply hold up against the 82 percent.
On top of that, the atlas doesn't distinguish by who invented a component. "Component" doesn't automatically mean "the authors' own creation." Indirectly, though, that's exactly what supports the thesis. Our assessment rests on our own grouping: components that are typically a paper's own contribution, namely contrastive loss, attention module, auxiliary loss and distillation, come to about 91 percent weighted. Standard ingredients that almost everyone adds anyway, namely dropout, label smoothing, gradient clipping, normalization and cosine schedule, land at around 70 percent, close to the knobs. Chain-of-thought also scores a modest 69 percent. It's often tested by third parties as a baseline, for instance in the paper "To CoT or not to CoT?". Where authors have no stake in the outcome, the success rate drops noticeably.
One caveat remains: contrastive losses apparently do work. Of all things, the critical replication by Musgrave and colleagues showed that the classic 2006 contrastive loss is very robust in a fair comparison. So the 95 percent probably isn't just spin.
What independent replications show
How much of published progress survives when outsiders re-measure it has been well studied:
- Narang et al. 2021 (arXiv 2102.11972) reimplemented around 50 Transformer variants in a shared codebase. Their conclusion:
"Surprisingly, we find that most modifications do not meaningfully improve performance."
What did help were mostly small changes such as SwiGLU or RMSNorm, or variants developed in the same codebase. Of the creators of 12 techniques the authors contacted, 6 confirmed the implementation was correct. - Musgrave et al. 2020 (arXiv 2003.08505) found unfair comparisons, weak baselines and hyperparameters chosen on the test set in metric learning papers. Some newer loss functions claimed more than double the performance of older methods. In a fair comparison, they came out only about 1 to 4 points ahead of the 2006 contrastive loss, and some even fell behind it. - Lucic et al. 2018 (arXiv 1711.10337) showed for GANs, image generators built from two networks trained against each other, that with enough tuning and restarts most models reach similar scores. None reliably beat the original method. - Melis et al. 2018 found that well-regularized standard LSTMs outperformed newer language model architectures.
Lipton and Steinhardt (arXiv 1807.03341) call the pattern by its name: papers like to credit gains to new architectures when they actually come from hyperparameter tuning. That's exactly the pattern a tally like the atlas's would book as "component helps."
Spot check: does the AI read correctly?
We compared twelve entries from atlas topic pages on chain-of-thought, LoRA and contrastive learning with the original papers. We were able to check ten of them against the original tables:
| Paper | Atlas entry | Original | Verdict |
|---|---|---|---|
| SIM-CoT (ICLR 2026) | 3B instead of 1B decoder: 56.1 → 50.4 | Tab. 4a identical | correct |
| KAM-CoT (AAAI 2024) | Knowledge graph: 87.74 → 92.62 | Tab. 5 identical | correct |
| T-SciQ (AAAI 2024) | 91.75 → 85.99 | identical | correct |
| VeriCoT (ICLR 2026) | Premises: 3.3 → 45.2% | Tab. 1 identical | correct |
| StreamingThinker (ICLR 2026) | 0.437 → 0.871 | Tab. 1 identical | correct |
| MiSS (ICLR 2026) | Rank 16 → 128: 45.90 → 53.49 | Tab. 7 identical | correct |
| RandLoRA (ICLR 2025) | Rank 32: 85.98 → 83.62 | Tab. 2 identical | correct |
| LoRA-RITE (ICLR 2025) | Rank 4 instead of 16: 68.8 → 70.4 | identical | correct, but incomplete |
| CopyNCE (ICCV 2025) | 68.9 → 70.5 µAP | Tab. 2 identical | correct |
| DoRA (ICML 2024) | "Rank halved": 74.7 → 77.5 | 74.7 is LoRA, not DoRA | misattributed |
So the numbers are almost always right. The bigger problem is what the numbers mean. For the DoRA paper, the atlas compares LoRA at rank 32 against DoRA at half the rank. Within DoRA, halving the rank actually costs something: the result drops from 78.4 to 77.5. For LoRA-RITE, the smaller rank only helps the authors' own method. For regular Adam training, rank 16 is better (64.0 vs. 63.0). Another entry, on a CVPR paper about geospatial adapters, contradicts itself: "worse," even though the number rises from 85.86 to 87.14. We couldn't retrieve the original there. The same goes for the paper "Chain-of-Thought Reasoning Without Prompting." Twelve entries are too few to establish an error rate. But they show where the limits are.
Just how hard ablations are to see through, even for strong models, is shown by AblationBench: the best system tested found on average only 45 percent of the ablations the authors had actually run. We weren't technically able to access the reactions in the Reddit thread about the atlas, so we can't assess the community's criticism.
Our take
The atlas is a useful, freely accessible tool and a rare wide-angle view. In our sample, its raw numbers were mostly transcribed correctly. The core claim that "26 percent of changes don't help" is plausible, though the operators themselves consider it more of a lower bound.
Our verdict on the thesis: it partly holds. The fact that authors' own contributions almost always "help" while standard ingredients do so only about 70 percent of the time fits publication bias, the tendency to publish mainly successes. It also fits what replications like Narang's or Musgrave's regularly find. But part of the gap to the knobs is simply mechanics: testing around an optimum inevitably produces losses.
Open questions remain: how often the AI mapping gets it wrong, how "without X" ablations are counted, and why more than 10,000 papers are missing. What this means for you: when a new model shows off an ablation table, treat it as a hint, not proof. It only becomes solid once someone else rebuilds the trick in a different codebase.
Sources
- scime Dispatch: What didn't work 2026 (Methodik, übliche Verdächtige) atlas.scime.ai
- scime atlas Startseite atlas.scime.ai
- scime atlas: Chain-of-thought training and analysis atlas.scime.ai
- scime atlas: LoRA variants and theory atlas.scime.ai
- scime atlas: Self-supervision in supervised pipelines atlas.scime.ai
- scime atlas: Empirical contrastive visual learning atlas.scime.ai
- Latentic Labs latenticlabs.com
- Reddit r/MachineLearning: We logged 621k changes from 127k papers' ablations reddit.com
- Narang et al. 2021: Do Transformer Modifications Transfer? (arXiv) arxiv.org
- Narang et al. 2021 (ar5iv-Volltext) ar5iv.labs.arxiv.org
- Musgrave et al. 2020: A Metric Learning Reality Check (ar5iv-Volltext) ar5iv.labs.arxiv.org
- Lucic et al. 2018: Are GANs Created Equal? arxiv.org
- Lipton & Steinhardt 2018: Troubling Trends in ML Scholarship arxiv.org
- AblationBench (arXiv 2507.08038) arxiv.org
- Melis et al.: On the State of the Art of Evaluation in Neural Language Models mlanthology.org
- SIM-CoT (arXiv HTML) arxiv.org
- KAM-CoT (arXiv HTML) arxiv.org
- T-SciQ (arXiv) arxiv.org
- VeriCoT (arXiv HTML) arxiv.org
- StreamingThinker (arXiv HTML) arxiv.org
- DoRA (arXiv HTML) arxiv.org
- LoRA Done RITE (arXiv HTML) arxiv.org
- MiSS (arXiv HTML) arxiv.org
- RandLoRA (arXiv HTML) arxiv.org
- CopyNCE / Tracing Copied Pixels (arXiv HTML) arxiv.org
Ask the AI assistant
Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.
The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.
Comments
What do you think? Already tried it? Write it here, no sign-up needed.
Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.