A 16B model on a cheap TV box: smolmaxxing squeezes AI out of ancient hardware

What it's about

Nearly every home has devices with a processor, RAM and flash storage that sit idle most of the day: TV boxes, old phones and tablets, single-board computers, routers. The smolmaxxing project sets out to find which language model such a device can actually still run. Then the developers try to make it faster with their own optimizations. The code and extensive documentation are on GitHub at github.com/hyperionaidev/smolmaxxing. The project was recently featured on the Korean tech platform GeekNews.

The team stresses that every figure was measured directly on the hardware in question. So far there are two test devices: a third-generation Xiaomi TV Box S and a first-generation iPad Air from 2013. More TV boxes, phones, single-board computers and routers are set to follow.

Device 1: A TV box with just one gigabyte of RAM

The Xiaomi box runs on an Amlogic S7D chip with four Cortex-A55 cores at 2.5 GHz. These are frugal, fairly slow ARM cores that execute instructions strictly in order. On top of that you get 1.9 GB of RAM, of which only about 1 GB is actually usable, eMMC flash storage and Android 14 without root access. The box runs models ranging from a tiny 260,000 parameters up to 16 billion. Here are the key results, measured in tokens per second (a token is roughly a syllable or a short word):

The huge 77x factor for the 2B model has a simple explanation: the original file didn't fit in memory and had to be reloaded from flash constantly. The first version that fit in RAM already ran at 4.8 tokens per second. Measured from that point, the gain is still 1.6x. According to the developers, the box's speed fluctuates by about 10 percent over the course of a day. That's why every change was measured head-to-head against the previous version.

The project recommends Qwen3.5-2B as the best trade-off between speed and quality on this box. Bigger isn't automatically better. Ling-mini-2.0 is twice the size of Moon-Beam, yet it scores worse on quality per byte and runs slower. The project measures quality in "bits per byte", meaning how well a model can predict a fixed 22 KB English text. Lower is better. Because the figure is per byte, models with different tokenizers can be compared fairly. The best score goes to Qwen3.5-9B at 0.885, but it only manages 0.27 tokens per second.

Why flash storage is the real problem

The developers spell out exactly where the box hits its limits. A single process can use about 0.96 GB before Android starts killing apps. In so-called headless mode, where the launcher and Google services are forcibly stopped, that rises to around 1.15 GB. The flash storage reads at no more than 278 MB per second, and a single random read of about 1 MB takes roughly 6 milliseconds.

That leads to a simple calculation: if a model has to load X megabytes from flash per token, it can never run faster than 278 divided by X tokens per second. According to the project, memory rather than the CPU caps everything above 2 billion parameters. This is where mixture-of-experts models (MoE) come in. They consist of many specialized subnetworks, the "experts", and only a few of them are active per token. Moon-Beam-8B-A1B has 128 experts, 6 of which are used. So each token only needs about 1 billion of its 8 billion parameters.

To exploit this, the team built an "expert pager". It loads experts from flash only when they're needed, a bit like a librarian who fetches only the books someone has just asked for from the stacks. The routing is also cache-aware and favors experts that are already in RAM. On top of that, the next layer's experts are requested ahead of time. This cut the share of experts not already in RAM from 13 to about 2 percent, and Moon-Beam sped up from 4.4 to 7.2 tokens per second. Ling-mini-2.0, with 256 experts of 1 MB each and 8 active, runs with the same recipe unchanged.

Can you hit 10 tokens per second at 8B? The team tried. Moon-Beam reaches 10.4 tokens per second with only 2 experts per token, but then it produces gibberish. The best setting that still yields coherent text lands at around 7 to 8 tokens per second.

The toolbox: custom kernels, 1-bit models, memory tricks

The optimizations are grouped by the bottleneck they tackle:

• Compute: Ten measures in total, including repacked Q2_K and ternary formats for special ARM instructions (vdotq_laneq) and a lookup-table kernel for 1-bit models. A kernel is a hand-optimized compute routine. With the lookup-table approach, results are read from a precomputed table instead of being recalculated every time. There's also 8-bit attention and a fused Gated DeltaNet computation, an architectural building block of the new Qwen3.5 models. As a result, the box processes Q2_K weights at 13.9 billion weights per second instead of 7.8 billion. Long answers no longer halve the speed.
• RAM: Five measures, including a trimmed output head that only knows about 60,000 to 65,000 tokens instead of the full vocabulary (1.3 percent of tokens drop out as a result) and lossless 1.875-bit ternary storage. Ternary means each weight can only take the values −1, 0 or +1. Add to that headless mode and a cache for repacked files in flash. Together these save 0.2 to 0.5 GB per model. Free memory rose from about 1,000 to about 1,300 MB.
• Flash: Streaming weights straight past the system cache (O_DIRECT), plus MTP speculation, where the model predicts several tokens at once and then verifies them. Then there's the expert pager and prefetching. This made the 9B model 69 percent faster.
• Output quality: A custom test detects when long answers spiral into repetition loops. A repetition penalty of 1.15 over 256 tokens cut the number of derailed answers from 12 out of 13 to 1 out of 13.

According to the project, the techniques aren't tied to any particular device. The 1-bit kernel only needs basic NEON instructions, which virtually every ARM chip has. The other kernels run on any ARM chip with the dotprod extension. The streaming engine works on any Linux or Android device where the model is bigger than the available RAM. Measurements were taken with llama-server on real chat requests and with llama-bench, both part of the well-known llama.cpp project.

Device 2: A 13-year-old iPad

The second test device is a first-generation iPad Air with an Apple A7 chip, two cores and 1 GB of RAM. The chip has NEON but no dotprod instructions. The iPad runs iOS 12.5.7 and was jailbroken with the Amethyst tool. So without a jailbreak, none of this works. Inference runs on the CPU only, with 2 threads and a 1,024-token context. The software is cross-compiled for the iPad on a Mac, and the chat runs as a web page in the iPad's own Safari. The results:

• Bonsai-1.7B (1-bit): 24 to 26 tokens per second, 18 to 22 in long chats
• Bonsai-4B: about 10 tokens per second
• Moon-Beam-8B-A1B (MoE): about 3.5 tokens per second, 5 with half the experts, but with lower quality
• Ling-mini-2.0 (16B MoE): 2.6 to 4.5 tokens per second, experts paged in from flash

Build instructions are in the ipad-air/ folder of the repository.

Openly documented

What stands out is how thoroughly the project is documented. The EXPERIMENT.md file describes 160 individual experiments (E1 to E160) with measurements, including the ones that didn't work. One example is IBM's Granite 4.0 H 1B. It came closest to the 2B model's quality, but after conversion it only ran at 5.29 tokens per second and only fit after freeing up memory. The project provides the optimized model files on Hugging Face. Only the 9B variant has to be built yourself.

Our take

smolmaxxing shows that careful work on memory and compute routines can get far more out of cheap hardware than the spec sheets would suggest. On a box with 1 GB of RAM, models with 8 and 16 billion parameters run at reading speed. The key takeaway goes beyond the novelty: on memory-constrained devices, flash storage is the main brake. That makes MoE models the obvious path to bigger models, because they only need a small fraction of their weights per token.

There are some caveats you should know about. All the numbers come from the project itself, and there are no independent measurements yet. Quality was measured on a single English text, which says little about other languages or real-world tasks. Models with 1 to 2 billion parameters still fall clearly short of large cloud models in substance. The iPad needs a jailbreak, and the TV box's headless mode forcibly kills system services. For everyday use, all of this is fairly impractical. But as an openly documented case study for tinkerers and developers who want to run AI on edge devices, the project is extremely valuable.