MolmoBot: Large-Scale Simulation Enables Zero-Shot Manipulation

Say you want a robot arm to put a banana in a black bowl, in a room it's never seen. Can a policy trained only in simulation do that? That's sim-to-real: train in a simulator, run on a physical robot. MolmoBot goes zero-shot, meaning no real robot data and no fine-tuning first. It leans on domain randomization: lighting, textures, camera pose, friction and more change at random across program-generated houses. Scripted experts, planners that read the simulator's true object poses, record 1.7 million demonstrations on two robots, and MolmoBot learns to imitate them. On real tabletop pick-and-place, it succeeds in 79.2% of 120 trials across 4 rooms, against 39.2% for π0.5-DROID, which trained on more than 10,000 hours of real robot demos. On a mobile robot, a door-only version opened a real pull door in 2 of 9 tries, with the robot's emergency stop tripping in 5 of them.

MolmoBot learns by behavior cloning, copying the expert's actions. Its action head uses flow matching, which learns to turn random noise into an action chunk a little at a time. The paper takes this from π0; written out as in the released code, training minimizes $$\mathcal{L}(\theta) = \mathbb{E}_{(o,\ell,q,A)\sim\mathcal{D},\;\tau,\;\epsilon}\Big[\big\|v_\theta(A^\tau, \tau, o, \ell, q) - (A - \epsilon)\big\|^2\Big], \quad A^\tau = \tau A + (1-\tau)\,\epsilon$$$$\begin{aligned}&\mathcal{L}(\theta) = \mathbb{E}\Big[\big\|v_\theta(A^\tau, \tau, o, \ell, q) - (A - \epsilon)\big\|^2\Big]\\&\text{over } (o,\ell,q,A)\sim\mathcal{D},\ \tau,\ \epsilon\\&A^\tau = \tau A + (1-\tau)\,\epsilon\end{aligned}$$ Here $\mathcal{D}$ is the Franka demonstrations (about 1.5 million), mixed by task; $o$ is two frames from each of a wrist camera and an external camera; $\ell$ is the instruction, for example “put the banana in the black bowl”; and $q$ is the joint angles. $A$ is the expert's next 16 actions, each a target position for every joint and the gripper, called an action chunk. A mixing time $\tau$ between 0 and 1 blends it with Gaussian noise $\epsilon$ of the same shape into the noisy chunk $A^\tau$: at $\tau = 0$ it's pure noise, and at $\tau = 1$ it's the expert's chunk. The action head $v_\theta$, a DiT (diffusion transformer) that reads the Molmo2 vision-language model layer by layer, predicts the direction from noise toward the clean chunk. At run time it follows $v_\theta$ in small steps from pure noise to a chunk, runs 8 of the 16 actions, then plans a new chunk. The head is cheap next to the vision-language model, so the authors draw 8 values of $\tau$ for each example and train on all of them at once. Nothing in the loss mentions the real world. Every action the policy learns comes from a scripted expert in simulation; only its starting weights for vision and language, from SigLIP2 and Molmo2, come from pretraining on image and text data.

← All writing