Robots that imagine before they move

Ask a person to pick up a mug and they do something quietly remarkable before their hand moves: they picture it. Where the handle is, how heavy it will feel, what happens if it slips. That picture takes a fraction of a second and nobody notices it.

Robots have mostly skipped that step. They see, then they act. The most interesting shift in robotics right now is giving them the missing step, and the tool for it is the same one that makes AI video: a model that can generate what happens next.

This post explains the idea without the jargon: what a world model is, why generated video is turning into robot training data, and why the newest robot brains are being squeezed onto the robot itself instead of living in a data center.

Video models are secretly physics guessers

To generate a convincing clip of a glass tipping off a table, a video model has to get a lot right. The glass falls, it does not float. Water spills in the direction of the fall. The table does not move. Nobody wrote those rules into the model. It picked them up from watching millions of hours of footage.

Researchers noticed that this makes a video model a rough simulator of the real world. Give it the current camera frame and a planned action, like "move the gripper 10 cm left", and ask for the next few seconds of video. If the generated clip shows the mug being knocked over, the robot has learned something without breaking anything.

Google researchers showed this early with UniSim, a model that turned real video into an interactive simulator robots could practice in. NVIDIA then released Cosmos, a family of open world foundation models trained specifically on physical scenes for robots and self-driving cars. Google DeepMind’s Genie 3 pushed further, generating explorable worlds in real time that stay consistent for minutes.

Dreaming up training data

The biggest bottleneck in robotics is data. A language model can read the internet. A robot that folds laundry needs thousands of recordings of real hands folding real laundry, captured by people operating the robot by remote control. That is slow and expensive, and it never covers every shirt.

Generated video offers a shortcut. NVIDIA’s DreamGen work showed the recipe: fine-tune a video model on a small set of real robot recordings, prompt it with new tasks in words, and let it generate videos of the robot doing things it was never shown. A second model then works out the motor commands that would produce those videos, turning them into training examples.

This is where image and video generation stops being a creative tool and becomes infrastructure. Every generated clip is a lesson. The quality bar is not "does it look nice" but "does it obey physics well enough to teach something true".

Why the brain is moving onto the robot

Training happens on big GPU clusters. Running the result is a different story. A robot arm that waits half a second for a reply from a data center will drop the mug. A warehouse robot that loses Wi-Fi cannot stop working. So the models that decide what to do next are moving onto chips inside the robot. That is edge AI.

• Speed: control loops run many times a second. A network round trip eats most of that budget before the model even starts.
• Reliability: a robot has to keep working, or at least stop safely, when the connection drops.
• Privacy: cameras in homes and hospitals see things that should not leave the building.
• Cost: streaming video from a fleet of robots to the cloud around the clock adds up fast.

Google DeepMind’s Gemini Robotics On-Device is one clear example: a vision, language and action model built to run entirely on the robot, adapting to a new task from as few as 50 to 100 demonstrations. NVIDIA’s GR00T N1 takes a similar two-part approach for humanoids, pairing a slower model that understands the scene with a fast one that produces smooth movement.

The next step: imagining on the robot

Today the imagining mostly happens in the cloud, during training. The frontier is moving it onto the robot too: a small world model that previews a few candidate moves, a fraction of a second ahead, and picks the one whose imagined future looks best.

If that sounds familiar, it is the same trick we wrote about yesterday for image generation: try several possibilities, judge them, keep the best. For robots the judge is not "which picture is prettiest" but "which future has the mug still in my hand".

The cloud teaches the robot what the world is like. The edge lets it check its guess before it commits.

Doing that on a battery means making video models dramatically smaller and faster: fewer cleanup steps, lower resolution where it does not matter, and predicting compact summaries of the scene instead of every pixel. Expect most of the progress in the next year to come from that kind of distillation rather than from bigger models.

What this means if you build with generative video

• Physical accuracy is becoming a selling point. Models judged on whether objects fall, slide and collide correctly will matter as much as models judged on beauty.
• Generation volume is about to jump. Synthetic training sets are measured in hundreds of thousands of clips, which turns video generation into a bulk, cost-sensitive GPU workload.
• Cloud and edge are partners, not rivals. The heavy generating happens on rented GPUs. The small, distilled model ships to the device.

That middle point is why we built ClockingCloud around image and video generation. Whether you are generating ads or teaching a robot to pour water, the job is the same: run open video models like Wan on fast GPUs, keep every output, and only pay for the seconds you use.

Recommended setups on ClockingCloud

If you want to try this yourself, here is what we would pick for each part of a robotics pipeline. Every setup below launches from the console in a few clicks. Inference endpoints scale to zero, so you pay nothing while no requests are running.

1. Generate: launch a Wan 2.2 endpoint and send it your task prompts. Every clip is stored privately and listed under Assets.
2. Vary: use FLUX.1 schnell or Z-Image Turbo to create new rooms, lighting and objects, then feed those back in as starting frames.
3. Train: start a bare H100 with your SSH key, pull the generated clips and fine-tune your policy or world model.
4. Shrink: distill the result on an A100 into a model small enough for the chip in your robot, then stop the GPU so the meter stops.