Meta paper: small AI models read raw bytes instead of tokens

What's it about?

A post titled "Meta trained small AI models to read raw bytes instead of tokens" on r/LocalLLaMA presented an arXiv paper (ID 2609.12303). The source is a user's summary, not the paper itself. Everything here is based on that summary, and the attribution to Meta comes from the post's title. I couldn't read the paper itself.

The basic idea: today's language models first split text into tokens. These are word pieces from a fixed list, and a token typically covers about 4–5 letters. Here, the models read raw bytes instead, which is essentially letter by letter, and no longer need a huge vocabulary list.

How the experiment was set up

The researchers trained small models with around 1 billion parameters. Parameters are, roughly speaking, the knobs a model learns during training. Some of the models worked with tokens, some with bytes.

Training partly used distillation: a small "student" model learns by imitating a larger "teacher" model, here Llama 3 8B. That was the tricky part, because the teacher thinks in tokens while the student reads bytes. The authors therefore had to find a clean way to translate the outputs of both sides into each other. The source doesn't give details.

What they found

• With little training, the token models win by a wide margin.
• With more training, the token models flatten out while the byte models keep getting better.
• If you extend the trend lines, the byte model beats Llama 3.2 1B and Gemma 3 1B by a few points.
• The byte models reached the same level with about 6x less text.

The caveats

The post's author names several catches themselves:

• The edge over Llama and Gemma is a prediction. The curves were extended beyond what was actually trained. In the real runs, the token model was still ahead.
• Reading byte by byte is slower. Since one token equals 4–5 letters, you need roughly 5x more steps for the same text. So it takes less data but more compute.
• Tokens haven't disappeared entirely, because the teacher still uses them.
• Only small models were tested; whether this holds for larger models is an open question.

Our take

For small models on phones and laptops, getting more quality out of the same size would be valuable. The post's author also speculates that dropping the big vocabulary could ease weaknesses like miscounting letters in words. That's a guess, not a demonstrated result. What matters now is whether the trend holds up with real, longer training runs and whether byte models can run fast enough. The author is asking the community exactly that. Until the paper has been vetted, treat the results as promising but preliminary research.