Microsoft's Mage-VL-Cowork: 4B model controls your desktop by click and shell

What's this about?

A Microsoft research team has released Mage-VL-Cowork, a model with 4 billion parameters (parameters are, roughly speaking, the dials the model tuned during training). It operates a computer the way a person would: it looks at screenshots, clicks and types. But it can also run shell commands and Python scripts. The model decides for itself which tool to use. The weights are on Hugging Face under the Apache-2.0 license. The AI Tracker service logged them on October 30, 2026 at 8:20 a.m. US Eastern time. That is when they were discovered, not necessarily when they were officially announced.

Some background: agents that work only from screenshots burn through lots of tokens (the text chunks a model computes with). Changing a single spreadsheet cell can take many steps through menus. Rendering delays and slightly wrong coordinates make tasks fail. Pure shell agents, on the other hand, struggle with layouts or drawing canvases. Earlier solutions paired several agents or external routers, which meant extra calls and lost context.

One model, two channels

At each step, Mage-VL-Cowork chooses between two channels:

Click coordinates are normalized to a range of 0 to 999, and the environment converts them to the real resolution (for example 1920×1080). Here's an example: to merge cells in a spreadsheet and set a heading, a pure GUI agent would have to launch LibreOffice Calc, wait, and format. Mage-VL-Cowork can instead run a Python script with openpyxl and save the file directly. That saves steps and sidesteps pop-ups.

In a demo on the project page, the GUI model Qwen3.6-27B needed 39 steps and 9:56 minutes for a Calc task. Mage-VL-Cowork needed 6 steps and 39 seconds (roughly 15 times faster). According to the source, that's a single example and doesn't carry over to every task.

Five-stage training

The base is a 4B intermediate checkpoint called "Mage-VL". Rather than teaching both skills at once, the team uses a "Specialize-then-Unify" curriculum:

SFT (supervised fine-tuning) means the model learns from examples. RL (reinforcement learning) means it learns by being rewarded for successful outcomes.

For the training data, the team built an engine called Mage-CUA-Forge. It bundles reusable core skills and predefined task and environment scripts, instead of having agents search by trial and error each time. According to Microsoft, token use per task dropped by 93.3% (from 1,276,000 to 85,000), and cost fell by 82.6% (from $0.906 to $0.158). The engine produced 293,806 tasks from 6,915 core skills, plus 318,837 verified hybrid trajectories.

Particularly interesting: the tool environment of the teacher model, GPT-5.4, strongly influenced the small model. In the lean environment (without overlapping aids such as the accessibility tree and xdotool, but with openpyxl and pandas preinstalled), the teacher's success rate rose from 89.6% to 92.0%, and the step count fell by 29.2% (35.33 to 25.00). The student trained on data from the cluttered environment reached 25.1% at 39.40 steps, while the one trained on the lean data reached 37.6% at 30.91 steps. That's a 12.5-point gap. Whether this holds for other teachers and datasets is an open question.

Benchmarks

On OSWorld-Verified, the model scores 66.04%. It was measured on 361 tasks excluding Google Drive tasks, with a maximum of 50 steps, as the average of 65.33% and 66.75% from two runs. The agent used was "MAGEHybridAgent", with three screenshots, 1920×1080, temperature 0.7, top_p 0.8 and at most 4,096 generated tokens.

For the larger models, Microsoft's table gives no details such as step limits, so the comparison isn't necessarily fair. Here is how the training stages compare:

So RL added 10.2 points, with 13% fewer calls and 25.4% fewer output tokens. The reference model GPT-5.4-mini scores higher at 69.57%, but needs more than twice as many output tokens.

On web tasks, the model is weaker than specialists: on WebVoyager-Static, the SFT checkpoint (before RL) scores 58.6%, while the browser-specialized Fara1.5-4B reaches 68.0%. In return, Fara1.5-4B manages only 13.9% on OSWorld-Verified. Both figures come from different training stages and tests, so they aren't directly comparable.

What you need to try it

• Memory: The bfloat16 weights are 9.48 GB (two safetensors files). KV cache, vision encoder and system overhead come on top, so real requirements are higher. A GPU with at least 12 to 16 GB of VRAM is realistic; the source doesn't mention quantized variants.
• Software: Microsoft recommends an SGLang environment via the official Docker image: SGLang 0.5.2, PyTorch 2.8.0+cu128, Transformers 5.7.0, FlashInfer 0.3.1 and SGLANG_MAX_PIXELS=4000000. The Model Card also lists transformers>=4.57.3.
• Custom code: The "llava_onevision2" architecture is custom Python code in the repository, so trust_remote_code=True is required. Review that code first.
• Context: 32,768 tokens in training and inference; the architecture is said to support up to 262,144. Old steps are compressed into one-line "Action:" summaries, while the last three screenshots and four responses stay in full.
• Safety: Test only in disposable containers or VMs with a snapshot, with no access to host files, the local network, SSH keys, tokens or cookies. The source doesn't cover Mac, CPU or Colab.

Safety

The bash channel executes arbitrary commands. Wrong decisions or injected instructions (indirect prompt injection, meaning hidden commands in web pages or documents) can destroy files or leak data. On GitHub, Microsoft describes the model family as "for research purposes only", not for commercial products. For consequential actions such as sending data, logging in, payments or permission changes, a human should approve, and outbound connections should be restricted to allowed addresses.

Takeaway

Mage-VL-Cowork shows that model size isn't the only thing that matters for computer agents: channel selection, clean teacher data and efficient history management add a lot. At 66% on OSWorld-Verified, a 4B model is competitive, but it doesn't beat every open model (ScaleCUA-9B is ahead) and trails GPT-5.4-mini. A caveat: the number depends on the specific agent setup, the 50-step limit and Microsoft's own measurement, and independent reproductions are still pending. The permissive license doesn't change the fact that Microsoft advises against production use. It's also not Microsoft's "Copilot Cowork" product, which runs on its own as a cloud service for Microsoft 365. For researchers and tinkerers who want to study agents locally, the model is interesting. For everyday use, it's not.