WorldTUI: A 360M Model That Simulates a Terminal Without a Shell

WorldTUI: A 360M Model That Simulates a Terminal Without a Shell

Simon Edwardsson, a developer known for quirky AI experiments, has unveiled WorldTUI, a 360M-parameter language model fine-tuned to simulate a terminal. Unlike traditional terminal emulators, there's no shell or filesystem behind it—every keypress goes in, and the model predicts how the screen changes. The result is a convincing, interactive terminal that can create files, run vim and tmux, play snake, and even remember what you wrote a few commands later.

The demo is available in the browser (208 MB download) and runs entirely locally via WebGPU. It's best experienced in Chrome or Edge on a desktop with a physical keyboard. As Edwardsson puts it, "Try it in your browser and type pythoon to see that it's not a real shell."

From Genie 3 to a Terminal World Model

Edwardsson's inspiration came from DeepMind's Genie 3, which generates game worlds from text or images. He wanted to train his own world model but lacked massive GPU clusters and video data. So he scaled down: instead of a game world, he chose a terminal—a grid of characters rather than pixels. This choice was pragmatic: he could generate unlimited labeled training data by running real shells, and a pretrained language model already knows what ls or cat should print.

The core question was whether the model could maintain a consistent world, not just generate convincing output. The final model, based on Qwen2.5-Coder-0.5B, was fine-tuned on terminal sessions recorded from real programs inside Docker. The training took about 75 GPU-hours on an RTX 4090.

How It Sees a Terminal

Standard LLMs predict text tokens sequentially, but a terminal isn't linear. A single keypress like esc in vim changes modes, and clear wipes the screen. So Edwardsson treated the terminal as a grid of characters, with the model predicting changes to the grid rather than a stream of text.

Each session starts with a snapshot: screen size, contents, and cursor position. For example, a prompt might be encoded as BOS SCREEN_START 32 96 ROW_0 0 'simedw@laptop:~$ ' CURSOR 0 17 1 SCREEN_END. After that, each keypress is fed as raw bytes (e.g., Enter as \r, Escape as 0x1b). The model outputs which rows to rewrite and where the cursor ends up.

To handle scrolling efficiently, Edwardsson added a COPY operation that copies rows from the previous screen. A scroll now costs three tokens per row instead of ~96. The model uses a tiny vocabulary of 333 tokens: 256 byte tokens, 64 row tokens, and a few special operations. Decoding is grammar-constrained, ensuring every prediction is a valid screen update.

Training Data: Synthetic Sessions from Docker

Edwardsson couldn't find existing datasets with precise keystroke-to-screen alignment, so he created his own. The recipe involved:

• Randomizing usernames, hostnames, prompt styles, and terminal sizes.
• Randomizing directories and files based on fictional personas (developer, sysadmin, etc.).
• Picking 6–24 weighted activities (modify files, vim, python, tmux).
• Tracking state so sessions mostly open files that exist.
• Typing like a human: one keypress at a time, with tab completion and a 1.2% typo rate.
• Starting 35% of sessions partway through to force learning from any state.

The final dataset included 336k episodes, roughly 2B tokens. For comparison, the base model saw ~18T tokens during pretraining.

Training and Self-Correction

Edwardsson used straightforward supervised fine-tuning initially: given a screen history and a keypress, predict the screen delta. Ordinary typing was learned quickly, but errors compounded in longer sessions. To fix this, he adopted a DAgger-style approach:

1. Pick a point in a recorded session.
2. Let the model predict the next 1-4 events.
3. Compare the resulting screen with the real one.
4. If the difference is small, construct a correction that repairs the model's screen on the next keypress.
5. Continue training on the corrected frame.

This taught the model to repair its own mistakes. About a third of sampled sessions produced useful corrections. This technique, along with additional data, improved PresetBench scores from 24/47 to 38/47 at int4 quantization.

Performance and Memory

Edwardsson tested the model on three benchmarks: ShellBench (106 commands), PresetBench (47 scripted cases), and a memory test (32 worlds with file readbacks). The results were impressive:

• ShellBench: WorldTUI scored 80/106, vs. 52 for the prompted Qwen2.5-Coder and 60 for Qwen3-0.6B.
• Memory test: WorldTUI got 9/13 readbacks right, vs. 2/13 and 3/13 for baselines.

However, it's still far from a reliable filesystem. Combining commands in new ways can break things; for example, after echo green >> colors, running cat colors often doesn't show the appended line.

Making It Fast: Parallel Rows and Speculative Decoding

Initial autoregressive decoding was too slow: a full redraw took up to 12 seconds on an RTX 4090. Edwardsson's breakthrough was realizing that most rows in a frame are independent. By running one batch element per row, decoding cost dropped from the sum of all rows (~3,000 steps) to the length of the longest row (~100 steps). Large-frame p95 latency fell from 6.7s to 0.87s.

But independence caused coordination issues: vim might open on row 3 instead of row 0, or multiple rows would add food in snake. The solution was speculative decoding, where the parallel rows produce a draft, and the model verifies it sequentially. In the browser on an M4 Mac, typing takes ~60ms, and a larger update like man test takes 0.9s with parallel drafting.

Shrinking the Model for the Browser

To run in a browser, Edwardsson dropped almost the entire 151,936-token embedding table (136M parameters), keeping only the 333 tokens needed. This reduced the model from 494M to ~360M parameters (990MB to 716MB). Int4 quantization brought it down to 208MB, small enough for a custom WGSL/WebGPU decoder.

He also experimented with distillation, but found that removing layer 7 barely affected next-token accuracy (99.3% to 98.9%) yet collapsed memory readbacks from 64/64 to 0/64. A 16-layer student that kept key layers passed all memory tests but scored lower on ShellBench (65 vs. 78). Ultimately, he gave up on distillation.

What It Can Do and What It Can't

WorldTUI excels at maintaining a consistent world in short sessions. It can create files, edit them in vim, save, and read them back in a different tmux pane. It handles Python REPL with simple operations, though complex ones produce hallucinations. It's also forgiving of typos: pythoon launches Python, but ython doesn't.

Limitations include: no internet access (so tools like lynx are boring), nested directories can confuse it, and Firefox runs ~20x slower than Chrome due to WebGPU differences.

Why It Matters

WorldTUI is a proof-of-concept that world models can work for interactive applications without explicit rules. As Edwardsson notes, "It's obviously a pretty weak world model, and I'd love to see how far this approach could go with better consistency and a different way of generating screen updates." He plans to train on thousands of randomly generated applications to see if the model can predict behavior of unseen apps.

The project also highlights practical techniques for on-device AI: vocabulary trimming, int4 quantization, and parallel decoding. These are lessons that could apply to any interactive AI system running in resource-constrained environments.

Weights will be posted soon, and the demo is live. Try it, try to break it, and see for yourself.