Taobao open-sources a tiny 0.3B phone model that hits 350 tokens per second

What's going on

On November 6, 2026, the MNN team at Chinese e-commerce giant Taobao (part of Alibaba Group) open-sourced a language model called MNN-Pocket-0.3B. MNN is the team's in-house inference engine, the software that runs AI models directly on smartphones and other devices. The new model has only about 0.3 billion parameters and was built for mobile apps from the start. The goals: fast responses, a small memory footprint, and very little battery drain.

The team calls it an "app-native large model." In plain terms, it's a language AI that lives inside an app like Taobao and works without an internet connection. According to the team, it runs fully offline on an iPhone 18 Pro at an average of about 350 tokens per second. A token is roughly a piece of a word. That's faster than most cloud chatbots stream their text.

Four design goals

The team set four targets up front:

• Speed: More than 200 tokens/s during decoding, the step where the model writes its answer token by token. Fast cloud services served as the benchmark.
• Memory: Peak RAM use should stay under 300 MB. The reasoning: even a phone with 8 to 16 GB of RAM shares that memory with the OS and lots of other apps. If you budget about 500 MB for the model, a 300 MB peak leaves room for longer texts and for integrating it into the app.
• Energy: A typical request (512 to 1,024 tokens in, 128 tokens out) should use no more than one ten-thousandth of the battery's energy. That way, 100 requests cost at most 1 percent of your battery.
• Quality: In the use cases it was built for, the model should match an 8-billion-parameter model. The focus areas are Chinese and English, general knowledge, math, code, and above all instruction following, meaning how precisely it sticks to your instructions.

How they made it so small and fast

The first step was picking the size. The team tested dense models (where every part of the network does work for every token) from about 50 million to 1.3 billion parameters on an iPad Pro M5. About 300 million parameters was the largest size that met both targets: under 300 MB of memory and over 200 tokens/s.

Next came the shape of the network at that fixed size. Going from 16 to 28 layers made the model better. At 36 layers the benchmark scores stopped improving, but speed fell from 355 to 323 tokens/s. So the team settled on 28 layers plus Grouped Query Attention (GQA). With GQA, several parts of the attention mechanism share the same caches, which saves memory and compute.

The vocabulary was another lever. The team started from Qwen3's tokenizer, the tool that splits text into tokens, and cut its vocabulary from about 151,000 to 64,000 entries, roughly 57 percent smaller. Since the model is mainly meant for Chinese and English, the team accepted that other languages get split into tokens less efficiently. The payoff is lower memory use and less compute for every token generated.

For on-device use, the model is quantized to INT4, which shrinks it by storing its numbers at lower precision. Specifically, it uses W4A16 with QAT (quantization-aware training): weights are 4-bit, activations are 16-bit, and the model was prepared for this compression during training so it loses less quality.

Numbers and benchmarks

All figures come from the team. Measurements used MNN 3.6.2 and INT4: one warm-up run, five measured runs, median reported.

Prefill is the step where the model reads your input. Decoding is the step where it writes the answer. According to the team, that makes the file about 49 percent smaller than Qwen3-0.6B and peak memory about 47 percent lower. Decoding is about 1.63 times as fast.

The team measured power draw on a OnePlus 13 (Snapdragon 8 Elite) using the CPU with 4 threads, 1,024 input tokens, and 128 output tokens. After subtracting idle consumption, one request used about 2.7 joules on average. With a 6,000 mAh battery, 100 of these requests use about 0.3 percent of a charge, well under the team's own 1 percent target.

On capabilities, the technical report lists these scores: IFEval (English instruction following) 73.20 and IFEval-zh (a Chinese version the team built itself) 74.59. Both are the best results among the models compared. Averaged across 11 tests covering knowledge, reasoning, math, code, and instruction following, the model scores 50.00. The report says that puts it ahead of models its own size and also ahead of bigger ones like MiniCPM4-0.5B and Qwen3-0.6B. Other individual scores: MGSM-zh (Chinese math problems) 51.20 and HumanEval (coding) 53.66.

Six variants for different jobs

The team is releasing the model in several variants, covering everything from research to shipping in apps:

Speculative decoding works like this: a very small draft model guesses several tokens ahead, and the main model checks them all in one pass. When the draft is right, output comes out much faster. The Dec variant uses its own decision head to choose from a set of given answers instead of writing free text. That's a good fit for classification or selection tasks inside an app.

What you need to try it

The model is so small that almost any current device will do. The on-device version is 173 MB, and the team says it needs just over 200 MB of extra memory while running. You don't need a GPU. It runs fine on a regular laptop CPU, a Mac with Apple Silicon, or Google Colab. To get the top measured speeds, though, you need the MNN engine on the right hardware: Metal on Apple devices, OpenCL or CPU on Android. On a PC, the Base, SFT, and Instruct variants will probably work with common tools like Hugging Face Transformers, but the source doesn't say so explicitly. It's also unclear whether GGUF files for llama.cpp or Ollama are available.

Our take

MNN-Pocket-0.3B isn't trying to compete with the big chatbots. It's a tool for app developers who want to ship offline, fast, power-efficient AI features on phones, for things like search queries, forms, summaries, or selection tasks. What stands out isn't any single benchmark score. It's that everything was optimized together: model size, vocabulary, quantization, and engine were all tuned as one package for real devices. The memory and power measurements also go further than in many other releases.

There are caveats, though. All the numbers come from the team itself, and it built the Chinese IFEval-zh test on its own. The goal of matching an 8B model applies only to "specific business scenarios" and hasn't been shown for general tasks. A model with 0.3 billion parameters has little room to store knowledge, so expect mistakes and made-up facts when you ask factual questions. German and other languages were deliberately deprioritized. The source doesn't mention the license. Still, for developers and researchers experimenting with small on-device models, it's an interesting new starting point, especially since the Base and SFT versions are part of the release.