Big AI builds tiny AI: What floppy-disk models really know
Hobbyists are now letting Claude write not just the code, but the training data and run the training itself. The results are models for DOS PCs, Game Boys and microcontrollers that know facts instead of children's stories. We lined up the projects, ran the numbers on what they really deliver, and checked whether any of this is even allowed.

What this is about
A language model on a microcontroller, a Commodore or a Game Boy has almost always meant the same demo. It ran a checkpoint trained on TinyStories, a dataset of simple children's stories. The model produced sentences like "Lily went to the park," and that was all it could do. So the usual criticism ("that's not a chatbot") was almost always on point.
In the past few days, something new has emerged. Hobbyists are letting a large Claude model not only write the inference code, but also generate the training data and run the training. The best-known example is Tiny-2.1: a question-answering model with 367,000 parameters that runs on DOS from a floppy disk. Then there's Tim Hwang's 6-million-parameter Game Boy model. Our question: is there real, compressed knowledge in these tiny models, or are they just very expensively built lookup tables?
Tiny-2.1: What the README says and what's missing
The repository github.com/WollyMonjac/Tiny-2.1 describes a transformer written in Turbo C 2.0. It has 4 layers with a width of 96 and works at the character level: it reads and writes one letter at a time instead of computing with word pieces. The weights are stored in 8 bits, with one scaling factor per row. The TINY.BIN file is about 380 KB. To run, the program needs about 540 KB of free conventional memory, the classic DOS RAM below 640 KB. The README puts it simply:
"Written by Fable 5.2 and trained by Opus 5.5."
According to README.TXT, the model knows about 3,500 facts about countries, chemical elements, animals, science and history. On top of that come simple foreign words, arithmetic and spelling. The limits are tight: addition only goes up to 20 + 20, multiplication only up to the 12 times table. With a math coprocessor (FPU, an extra chip for floating-point math, built in from the 486DX onward), an answer takes "a few seconds"; without one, "several minutes." There's no figure in tokens per second.
The widely quoted accuracy figures come from the README itself. For questions phrased exactly as in training, about 90 percent of the answers are correct. For 780 test questions with "new phrasings and new combinations," it's about 61 percent. The first version scored 6 percent on the same test. The README lists weaknesses including questions about two countries at once, rare elements, division and made-up names. For the latter, the model often invents an answer instead of admitting it doesn't know.
The README doesn't explain how the test questions were created. It's also unclear whether the same model that wrote the training data phrased them. The README does refer to a "TRAINKIT" archive with a data generator and training script, but it's missing from the repository. An open issue from November 16 asks exactly about that. Pasquale Pillitteri criticizes that initially only a video served as evidence. His assessment:
"It shows that the architecture fits on the floppy, not that it can answer new questions."
WindowsForum complains that nobody has independently benchmarked the project. The binaries and source code are now public, but we haven't found any independent external tests.
The projects compared
| Project | Parameters | Size | Data source | Speed | What it can really do |
|---|---|---|---|---|---|
| Tiny-2.1 (DOS) | 367,000 | approx. 380 KB (8-bit) | approx. 3,500 facts, reportedly generated by Opus 5.5 | "a few seconds" per answer with FPU | Factual questions: 90 % with training phrasing, 61 % when reworded |
| Tim Hwang's Game Boy model | 6M | 3.2 MB | trained by Claude, data source unknown | not stated | Conversation with a finch that knows a lot about the Revolutionary War |
| esp32-ai (ESP32-S3) | 28.9M (only 556,416 in the core) | 14.9 MB (4-bit) | 300 MB TinyStories | 9.88 tokens/s, 22.6 in InsiderLLM's fork | Children's stories |
| Habr article (ESP32-S3) | 260,000 | 274 KB (8-bit) | TinyStories (Karpathy) | 406 tokens/s | Children's stories |
| vellm (DOS) | 15M / 42M | 8-bit (Q8_0) | TinyStories | 0.27 / 0.11 tokens/s on Pentium OverDrive 83 MHz | Children's stories; code "100 %" from Claude Code per README |
| Single Floppy 346K | 346,104 | 1.32 MB (FP32) | 1.36 MB Capybara-ShareGPT | not stated | "still nonsense," per the author |
| tiny-voca (TTS) | 4.5M | 18–21 MB | 17.6 hrs of synthetic speech from Qwen3-TTS | approx. 70x real time on 4 CPU threads | English speech output, 2 voices |
| Hugging Face PyBot | no neural network | approx. 500,000 lines of Python | written by more than 1,000 agents | approx. 1 s per answer | approx. 26 % win rate, roughly on par with SmolLM-360M |
So far, the only source on Tim Hwang's project is his post on X: Claude needed "a full week and a ridiculous amount of Fable credits" to train a 6-million-parameter model in 3.2 MB for the Game Boy. We haven't found a repository, weights or benchmarks. Do the math and you get about 4.3 bits per parameter (3.2 MB × 8 ÷ 6M). So our guess is 4-bit quantization. Quantization stores a model's numbers at lower precision to make it smaller. For context: the original Game Boy runs at 4.19 MHz and has 8 KiB of RAM. An earlier demo running TinyStories-260K on the Game Boy Color managed about 0.006 tokens per second, according to one commenter, and produced "gibberish," according to its builder.
Why tokens/s mean nothing without a parameter count
The table reveals a fundamental problem. The 406 tokens/s from the Habr headline and the roughly 20 tokens/s others measure on the ESP32 come from the same chip. The difference is the models: 260,000 versus 28.9 million parameters. In the Habr project, the 260K model fits almost entirely into fast internal memory at 274 KB after conversion to 8 bits. The author sped it up 18x over his naive starting version using vector instructions and both processor cores, and verified step by step that the same tokens come out at the end.
Even the parameter count can be misleading. According to its Model Card, 25.2 million of the 28.9-million-parameter model belong to a per-layer embedding table. That's a kind of lookup table in flash memory, of which only a few rows are read per token. The actual compute core has only 556,416 parameters, barely more than Tiny-2.1.
Old PCs show the same pattern. According to dosllam2's benchmarks, a 200 MHz Pentium MMX under DOS manages 15.32 tokens/s with the 260K model, but only 0.43 tokens/s with the 15M model. That's a factor of 36. Our take: a speed figure without parameter count, quantization and memory location is worthless for comparison.
Distillation or lookup table?
Distillation means a small model learns from the outputs of a large one. That's nothing new in itself. According to its paper, TinyStories is also a synthetic dataset written by GPT-3.5 and GPT-4. So the old wave of mini demos was distilled too, just from OpenAI models and on children's stories.
What's new is the specialization, and you can roughly calculate it. In 2024, Allen-Zhu and Li measured that language models store at most about 2 bits of knowledge per parameter, even with 8-bit quantization. For Tiny-2.1, that works out to about 734,000 bits, or roughly 92 KB. Divided by 3,500 facts, that leaves about 210 bits per fact, or about 26 characters. That's enough for "Capital of Peru: Lima," but not much more. Our take: the model is pretty much exactly as big as it needs to be for its set of facts. There's not much room left for generalization.
The arithmetic points the same way. Sums up to 20 + 20 are 441 combinations; the 12 times table covers 144 problems. A model can memorize all of that. The fact that division works worst fits this picture as well.
So is it just a lookup table? Not quite. A pure table with exact text matching would score zero percent on reworded questions. Tiny-2.1 manages 61 percent and tolerates typos. Version 1 was at 6 percent, so the progress is real. That said, PyBot shows that agents can program robust pattern matching even without a neural network. Our take: Tiny-2.1 is a learned, fuzzy search over a fixed set of facts that Opus provided. It can retrieve the answers, but it hasn't inherited the teacher's reasoning ability. Where the model knows nothing, it confidently makes something up, according to the README. In that respect, it's no different from its bigger relatives.
The counterexample is the Hugging Face model Single Floppy 346K. It has almost the same number of parameters but was trained on general chat data and, according to its author, "still produces nonsense." According to RuntimeWire, he says it even performed worse than a model with 1,000 parameters. He doesn't name a measurement method. The difference from Tiny-2.1 isn't size, but the narrowly tailored training material.
What does it cost, and is it allowed?
Hwang doesn't put a number on his "ridiculous amount of Fable credits." What we do know: according to Founderz, Fable 5.2 costs $8 per million input tokens and $40 per million output tokens via the API. A single eight-hour agent run to build a game came to about $550 there. Since November 9, Pro and Team users have been able to use Fable for up to half of their weekly limit at no extra cost. That free week ends on November 16 at 11:59 p.m. Pacific Time. After that, Fable is billed against usage credits again. Weeks-long training runs then become a real cost factor.
The legal situation is trickier. Anthropic's Usage Policy (effective November 12, 2026) explicitly prohibits, under "Do Not Abuse Our Platform," using outputs to train an AI model, "such as 'model distillation'," unless Anthropic has given prior permission. No competing product needs to be involved. The Consumer Terms (Section 3) and the Commercial Terms (D.4) additionally prohibit using the services to build competing products, "including" training competing AI models.
Having Claude write the training code, as with vellm, isn't affected. But if Opus generated the facts Tiny-2.1 was trained on, that falls under the ban by its wording. Whether Anthropic will act against hobby projects is an open question. According to Pillitteri, the company hadn't commented as of November 15.
Our verdict
The new wave is a genuine step beyond TinyStories. A large model carves out a narrow slice of knowledge and compresses it enough to fit on a floppy disk. With reworded questions, that works 61 percent of the time, if you take the creator's claims at face value. Our take: this is distillation of knowledge, not of intelligence. "Trainable encyclopedia with 3,500 entries" is closer to the truth than "chatbot."
In practice, that's more interesting than it sounds. A microcontroller that tolerantly answers 200 fixed questions about a machine or a museum exhibit needs exactly this kind of thing. Three things remain open: independent testing (the test data and generator are missing), the actual cost of the agent-driven training runs, and whether such projects are even compatible with Anthropic's own rules. The next time you read about a tokens/s record, ask first about the parameter count, and then about what the model actually knows.
Sources
- GitHub: WollyMonjac/Tiny-2.1 github.com
- Tiny-2.1 README.TXT raw.githubusercontent.com
- GitHub-Issue #1: Missed TRAINKIT? github.com
- MakeUseOf: Someone used Claude to build a chatbot that fits on a floppy disk makeuseof.com
- WindowsForum: Claude-Assisted 435 KB Language Model for DOS windowsforum.com
- Slashdot: Floppy Disk-Based AI Chatbot for MS-DOS tech.slashdot.org
- Tim Hwang auf X x.com
- GitHub: slvDev/esp32-ai github.com
- Hugging Face: slvDev/esp32-ai-tinystories huggingface.co
- InsiderLLM: LLM on a microcontroller (ESP32-S3) insiderllm.com
- Habr: Sprachmodell auf ESP32 mit 400 Token/s habr.com
- GitHub: ecliptik/vellm github.com
- GitHub: yeokm1/dosllam2 github.com
- Hugging Face: NILKNARFGonzo/single-floppy-346k huggingface.co
- RuntimeWire: Nilky documents a floppy-disk model runtimewire.com
- StartupFortune: Game Boy Color AI demo startupfortune.com
- GitHub: terryguo3180-eng/Gemma.c github.com
- GitHub: karpathy/llama2.c github.com
- arXiv: TinyStories (Eldan, Li) arxiv.org
- arXiv: Physics of Language Models – Knowledge Capacity Scaling Laws arxiv.org
- Pan Docs: Game Boy Specifications gbdev.io
- Anthropic Usage Policy anthropic.com
- Anthropic Consumer Terms anthropic.com
- Anthropic Commercial Terms anthropic.com
- Founderz: Claude Fable 5.2 – Preise und Projekte founderz.com
Ask the AI assistant
Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.
The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.
Comments
What do you think? Already tried it? Write it here, no sign-up needed.
Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.