Given free rein to design a neural net, three frontier AIs all skip the Transformer
POSTECH researchers let three current frontier models design a neural network however they wanted. Working independently, all three built sparse lookup tables instead of a Transformer. The best design clearly beats a Transformer of the same size, but it still falls far behind small pretrained models.

What's going on
What happens when you give an AI complete freedom to design a neural network itself? Researchers Hyunjong Ok and Jaeho Lee of POSTECH in South Korea set out to answer that question. Their paper came out on November 18, 2026, and got noticed through a post on the Reddit forum r/artificial. Their test subjects were three current frontier models: Claude Opus 5.5 from Anthropic, plus GPT-6.1 Sol and GPT-6 Astra from OpenAI.
All three got the same setup: ten labeled example questions, a code sandbox (an isolated environment where they could write and run programs) and instructions to design any architecture they liked. There was one catch: gradient descent wasn't part of the process. That's the standard training method, where a network adjusts its weights over millions of small steps until its errors shrink. So the language models had to set the weights more or less by hand.
The result is surprising. None of the three built a Transformer, the architecture behind practically every large language model today, themselves included. Instead, all three independently landed on the same idea: they turned facts from ordinary text into sparse lookup tables, which are then queried by a fixed procedure.
How the table networks work
Think of a system like this as a huge card index. Each row in the table holds one fact. "Sparse" means most entries are empty or zero, and only a few rows matter for any given query. When a question comes in, a fixed, predefined procedure pulls out the matching entries and derives the answer from them. A Transformer, by contrast, spreads its knowledge across billions of numbers that can't be traced back to any single fact.
That gives the approach an advantage the authors highlight: every prediction can be traced back to exactly one fact row, and you can edit that row. If the model gives a wrong answer, you can see which entry caused it and fix it. With ordinary language models, doing that is still a difficult research field in its own right.
The numbers
The Claude Opus 5.5 design performed best. The baseline was a Transformer of the same size, trained the classic way with gradient descent.
| Model | Parameters | ARC test accuracy |
|---|---|---|
| Table model (Claude Opus 5.5 design) | 3.35M | 54.4% |
| Two-layer Transformer, trained with gradient descent | approx. 3.35M | 29.8% |
| Qwen3.5-0.8B (pretrained language model) | 0.8B | 82.1% |
The Reddit post doesn't say exactly which version of the ARC benchmark was used. The reported scores, including 82 percent for a small language model, point to the AI2 Reasoning Challenge. That's a multiple-choice test of science questions, not the well-known ARC-AGI puzzle benchmark. That isn't certain, though.
According to the paper, the table model also beats the most accurate Transformer written by any of the language models, and it does so with roughly 1,800 times fewer parameters. Working backward, that Transformer would have had around six billion parameters. That's only an extrapolation from the reported numbers, though. The table model is also said to run up to 17 times faster on a GPU.
The big caveats
The Reddit post itself spells out the limitations, and they're significant:
- Narrow scope: On MMLU, a broader knowledge test covering dozens of subjects, the model scores about 25 percent. With four answer choices, that's pure guessing.
- Far behind pretrained models: The small language model Qwen3.5-0.8B scores 82.1 percent on the same ARC test. The table model reaches only about two-thirds of that.
- A convenient comparison: A tiny two-layer Transformer without large-scale pretraining is a weak opponent. A fact-lookup system beating it on knowledge questions is less impressive than the headline suggests.
The authors say the code is published on GitHub, though the post doesn't give a link. A more detailed summary is available at emergent.sh.
The bigger question
The Reddit discussion focuses on a question that goes beyond the experiment. Maybe the Transformer became the standard mainly because it works so well with gradient descent. It's highly parallel and scales well, meaning it keeps improving with more compute and data. If language models that are allowed to write the weights by hand don't want this architecture at all, that suggests the choice of architecture depends heavily on the training method, not just on the task.
You don't have to buy that reading. For a language model asked to build a network by hand, a fact table is also simply the obvious solution. You can generate it directly from text, it needs no training and it's easy to inspect. Still, it's notable that three models from two different companies arrived at the same solution independently.
The takeaway
The study doesn't mean Transformers are about to become obsolete. On general knowledge and language understanding, the table models fall far short. It's interesting for two other reasons. First, it shows that language models can design working learning systems without human guidance, and that those systems can be very small and fast on specific tasks. Second, the result has a practical upside: every answer can be traced back to a single fact and corrected. That matters for applications where traceability counts for more than breadth, such as verified domain knowledge.
Plenty of questions remain open. How much does the result depend on the ten example questions and how the task was worded? How well does the approach carry over beyond the ARC test? And is a fact store queried by a fixed procedure even a neural network in the usual sense, or more of a well-built database? So far, the results come only from the authors' own paper. There's no independent replication yet, but the published code makes one possible.
Sources
- Reddit r/artificial reddit.com
- siliconangle.com siliconangle.com
Ask the AI assistant
Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.
The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.
Comments
What do you think? Already tried it? Write it here, no sign-up needed.
Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.