WorldTUI: A 360M-parameter model plays a terminal with no real shell

What this is about

A developer (blog: simedw.com) has trained a small language model with about 360 million parameters to play a terminal. There is no shell and no file system behind it. Every keystroke goes in, and the model predicts how the screen changes. It can "create" files, operate vim and tmux, and play Snake. Most of the time, it also remembers what you typed several commands earlier. The demo runs in a browser with WebGPU (best in Chrome or Edge) and requires a 208 MB download. The author's tip: type pythoon and you'll see it isn't a real shell. He plans to release the weights "soon".

The motivation was Google's Genie 3: the author imagined applications that aren't programmed but learned. A "world model" (a model that predicts an environment's next state from inputs) for games was too big a task for him. A terminal is a grid of characters rather than pixels, and you can generate as much training data as you like with real shells. The central question: can the model hold a consistent world, or does it only produce convincing-looking output?

How the model "sees" a terminal

Ordinary language models append text at the end. A terminal works differently: Escape in vim acts on its own, and clear wipes the screen. So the model predicts changes to the screen instead.

Each session begins with a snapshot (size, contents, cursor position). After that, every keystroke arrives as the raw bytes a real terminal sends (Enter is \r, Escape is 0x1b, Left arrow is ESC [ D). The model replies with the lines to overwrite and the new cursor position, for example: write "l" on line 0 starting at column 17, move the cursor to column 18. Nothing is erased beforehand; new characters simply overwrite old ones. Time is also an input: when no key is pressed, the model receives a WAIT token with a duration, which is how Snake moves by itself.

Qwen already has tokens for all 256 byte values, which are reused for characters, coordinates and small numbers. A COPY operation is added on top. Instead of re-emitting every line when scrolling (about 3,000 tokens), the model copies lines from the previous screen, at roughly three tokens per line instead of about 96. This also helps with clearing the screen, vim lines and man pages. The whole vocabulary has just 333 tokens (256 byte tokens, 64 line tokens, a few special operations). Decoding is grammar-constrained, so every prediction is a valid update. The display is monochrome, and tmux's Unicode borders are mapped to ASCII.

Training data and training

The author couldn't find suitable datasets (such as asciinema) with an exact key-to-screen mapping, so he generated his own: real commands in Docker containers, with every key and every screen recorded. He randomly varied usernames, hostnames, prompts, terminal sizes (mostly 32×96, also 24×80 and 40×120), and directories and files based on fictional personas (developer, sysadmin). He also varied 6–24 weighted activities (editing files, vim, python, tmux) and human-like typing with a 1.2% typo rate. 35% of sessions start midway through. The end result was 336,000 episodes with about 2B tokens. For comparison, the base model saw roughly 18 trillion.

The base is Qwen2.5-Coder-0.5B, because it is small enough for real time and already knows CLI usage. After ordinary supervised fine-tuning, errors piled up in long sessions, such as a wrong file name that kept propagating. The fix is inspired by DAgger: the model learns from the states it produces itself. It predicts 1–4 events, the result is compared with the real screen, and when the deviation is small, a correction is built that teaches the model to repair its own screen. Importantly, corrected words must have appeared earlier on a real screen or in the input, otherwise you would reward hallucinations. About a third of sessions yielded usable examples, repeated over three rounds. This was one of the most effective changes: together with more data, PresetBench at int4 rose from 24/47 to 38/47. All training, including refinements, took about 75 GPU hours on an RTX 4090.

Measurements

Three tests, none included in training: ShellBench (106 commands in 24 hand-written bash sessions, exact output), PresetBench (47 cases drawn from failures in the demo) and a memory test with 32 worlds, in which two random values are written to two files and read back in separate sessions.

The comparison models got three examples each and predict whole command outputs rather than individual screen updates, so the task isn't identical. WorldTUI is not a reliable file system: recombine familiar commands in new ways and it quickly goes wrong. For example, cat colors often fails to show the appended line after echo green >> colors.

Speed and shrinking

Generating one token at a time gave ~38 ms per keystroke, but up to 12 s for full-screen redraws (vim, man, clear) on the RTX 4090. The fix: lines are largely independent, so each line gets its own batch element with the same history (KV cache), generated in parallel. That cuts the cost from up to 3,000 steps to about 100. The cursor follows as a separate step. This required a matching attention mask during training. The p95 latency for large frames fell from 6.7 s to 0.87 s.

The downside: independent lines don't coordinate. vim opened on line 3 instead of 0, and Snake spawned food several times. The remedy is speculative decoding with the same model: the parallel lines provide a draft, which the model checks sequentially. The draft stays up to the first mismatch, then the model corrects it and proposes the rest again. On an M4 Mac in the browser, a single character takes about 60 ms. A larger update like man test takes 2.5 s sequentially or 0.9 s with draft and verification. According to the author, the method helps much more on the 4090 than on the M4. He only guesses at the cause (compute vs. memory bandwidth).

To shrink the model, the embedding table (151,936 tokens, 136M parameters) was cut down to the 333 required tokens (padded to 384). That takes the model from 494M to about 360M parameters (990 MB → 716 MB). With int4, it comes to 208 MB. The demo uses a custom JavaScript implementation of the Qwen decoder with WGSL compute shaders over WebGPU. Weights are unpacked at inference time, and activations and the attention cache are 32-bit floats.

The author gave up on distilling into fewer layers, but learned something: removing layer 7 lowered next-token accuracy only from 99.3% to 98.9%, yet exact memory readbacks dropped from 64/64 to 0/64. A student that kept layers 1, 7, 15 and 22 (16 layers, about 138 MB at 4 bits) passed all the memory tests, but managed only 65 ShellBench commands instead of 78.

What it can do, and what it can't

Memory mattered most to the author: create a file, edit it, do something else, and find it unchanged. He started tmux, changed a file in vim, saved it, and read it back correctly with cat in another tmux window. In vim, mode switching with Esc, saving and quitting work. In tmux, multiple panes work. Environment variables and variable expansion work. In the Python REPL, small additions are mostly right, while list comprehensions produce a list, often with the wrong contents. pythoon starts Python, ython doesn't.

Weaknesses: the containers had no internet access, so tools like lynx are boring. Harder Python tasks are reminiscent of hallucinations from the GPT-3 era. With nested directories, the model sometimes loses the path. In testing, the demo ran about 20× slower in Firefox than in Chrome, for reasons unknown.

Context

Back in 2022, people got ChatGPT to play a Linux terminal. GameNGen simulated Doom with diffusion, and NeuralOS simulated a desktop interface. WorldTUI is closer to the ChatGPT trick, but it is trained specifically on keystrokes and screen updates, including full-screen applications. The author himself calls it "not really useful" and a weak world model. What's interesting is that a 360M model in the browser holds a reasonably consistent world without implementing the rules. The figures come from the author's own tests on self-built benchmarks and haven't been independently verified. Next, he wants to try diffusion instead of autoregressive decoding, and to test with thousands of randomly generated applications whether the model can predict unfamiliar apps from context cues.