"Vibe Robotics": Google DeepMind lets AI agents program robot arms

Google DeepMind has released a plugin for its AI development environment Antigravity. With it, an AI agent writes control code for robot arms, tests it in the MuJoCo physics simulator and then runs it on real hardware. You describe what the robot should do in plain language. You don't even need a robot, since the simulation works on its own.

Tools Nov 18, 2026 6 min read Source: Show HN (neueste)
Illustrative image, AI-generated

What it's about

Google DeepMind has released a plugin called Robotics on GitHub, in the repository google-deepmind/robotics-antigravity-plugin. It was announced on Hacker News as "Show HN: Try Vibe Robotics!". The name is a nod to "vibe coding", where you tell an AI in everyday language what a program should do and the AI writes the code. "Vibe robotics" works the same way, except the code ends up driving a robot arm.

The plugin is built for Google Antigravity, Google's development environment where AI agents handle programming tasks on their own. An AI agent here is a language model that runs through several steps in a row, uses tools and checks its own results. With the plugin, the agent writes the controller, the program that steers the robot. It tests the controller in the MuJoCo physics simulator and then sends the same code to a real robot arm. If you don't have robot hardware, you can stick to the simulation. The code is released under the permissive Apache 2.0 license.

How the workflow runs

The README describes a loop with five steps: bring-up, perceive, plan, act and verify.

  • Bring-up is the robotics term for powering up new hardware for the first time: connecting to it, addressing the joints and checking that everything responds.
  • Perceive: Multimodal Gemini models analyze camera images and work out, for example, where a blue pen is lying on the table. "Multimodal" means a model handles images as well as text.
  • Plan and act: The agent generates code that computes and executes the movements. DeepMind talks about "state-of-the-art spatial reasoning" plugged directly into the robot's real-time control loop.
  • Verify: The result gets checked, for example through multiple test runs in simulation or a recorded video.

The idea behind this is "Sim2Real", going from simulation to reality. You try out a control program safely in a virtual world first and then move it to the real machine. MuJoCo is a good fit for this. The name stands for "Multi-Joint dynamics with Contact". The software simulates joints, forces and contact between objects very precisely, and DeepMind develops it as open source.

According to the README, the plugin covers perception, planning, control, Sim2Real, the "MuJoCo Studio" tool and hardware bring-up. DeepMind says this makes closed-loop control, fast bring-up of new hardware and always-on robot applications "fast and cheap enough for everyday use". That's the vendor's claim. There are no independent tests yet.

The examples: cookies, coffee and a cycling pelican

The showcase example is a cookie factory. An I2RT YAM Pro robot arm picks cookies off a moving conveyor belt and sorts them by type into two boxes. The repository shows animations of the same task in MuJoCo and with the real arm. The prompt behind it stands out: the agent is told to rebuild the setup in simulation from photos, including the arm, the conveyor belt, two kinds of cookies and two boxes. It then has to test the task over several runs and show a video.

Other example prompts that only need the simulation:

  • A MuJoCo simulation of a "pelican riding a bicycle". That's a nod to a well-known test where language models have to draw exactly this scene as a vector graphic.
  • A grocery-sorting task with the low-cost hobbyist arm LeRobot SO-101. Groceries of different shapes and sizes have to go into containers without crushing anything fragile or knocking anything over, across randomized layouts.
  • A café with two Franka Research 3 arms working together (bimanually). They place an empty cup under the espresso machine and serve the finished drink on a tray. The rule: every movement of an object has to come from contact with the robot. Nothing is allowed to simply "teleport".

For real robots, DeepMind suggests simpler tasks: picking a blue marker up off the table with an SO-101, waving with a YAM arm, or pushing a pen several centimeters deep into a tall box without touching the walls. A video is recorded every time.

Installation and requirements

The official route goes through Google Antigravity. Install the latest version and add the plugin with this command:

agy plugin install https://github.com/google-deepmind/robotics-antigravity-plugin

You can also clone the repository and copy the plugin files to ~/.gemini/config/plugins. After that, restart Antigravity. If you install the plugin from the Antigravity Marketplace, you also get seven MuJoCo skills: mujoco-python, mujoco-spec-editing, mujoco-rendering, mujoco-accelerated, mujoco-parity, mujoco-studio and mujoco-gui. Skills are ready-made bundles of instructions and tools that show the agent how to use a specific API correctly.

You don't strictly need Antigravity: you can install the skills one at a time with npx skills add google-deepmind/robotics-antigravity-plugin. The MuJoCo skills come in via npx skills add google-deepmind/mujoco. They also ship with the MuJoCo Python package. In principle, that means you can use them with other agent tools that understand this skill format.

One thing to know: skills that call Gemini models for image recognition need an API key in the GEMINI_API_KEY environment variable. Those parts don't run locally. They go through Google's cloud and can cost money depending on how much you use them. Some skills also download third-party software and robot models, which come with their own licenses.

Safety with real robots

The README is explicit about the risks:

A moving robot can injure people and damage equipment. Stay next to the robot, keep the workspace clear and make sure you can cut the power at any time. You are responsible for operating your robot safely.

That warning matters even more here, because AI-generated code turns straight into physical motion. A bug that would just throw an error message in a normal app can knock things over on a robot arm, or worse.

The bigger picture

The plugin isn't a new AI model. It's a toolkit that ties existing building blocks together: Gemini as the agent and for visual perception, MuJoCo as the simulator, and low-cost arms like the SO-101 as hardware. It's still exciting. Until now, programming robots meant you had to know kinematics, control engineering and simulator APIs. Now a single plain-language sentence is supposed to be enough to build a scene, write a controller and test it. That lowers the barrier to entry a lot for hobbyists, students and small companies, especially since the simulation runs without any hardware at all.

The project also pulls developers into Google's own ecosystem of Antigravity, the Gemini API and Gemini Robotics. You can read more about DeepMind's robotics models at deepmind.google/models/gemini-robotics.

A few questions are still open:

  • How reliably do the agents handle tasks that go beyond the showcase examples?
  • How well does what works in simulation actually carry over to real hardware? This "Sim2Real gap" is a well-known problem in robotics.
  • How much do the API calls cost?

The animations in the repository show hand-picked successes. Success rates and independent tests are still missing. Users can share their own results in the GitHub Discussions, which should give a more realistic picture soon.

Sources

#robotics#google deepmind#mujoco#ai agents#antigravity#open source
Read the original source

Ask the AI assistant

Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.

Connecting …

The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.

Comments

What do you think? Already tried it? Write it here, no sign-up needed.

  1. Loading comments …

Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.