Why LLM reinforcement learning breaks on robots, and how the frontier is fixing it

Reinforcement learning is how language models got good at math and code. The recipe is surprisingly short. You give the model a prompt, sample 64 answers, score each one, and push up the probability of the answers that beat the group's average. That's GRPO. Its older sibling, PPO, trained the first chat models.

Now look inside a modern robot policy. It's built from the same parts: a pretrained vision-language transformer with a small action head on top. People call this a VLA, for vision-language-action model. Same transformer, same kind of pretraining, so the obvious move is to take the RL that worked for language models and run it on the robot.

So let's try it. Take a Unitree G1 humanoid with 29 joints and ask it to fold a shirt, sending motor commands 50 times a second (50 Hz). What goes wrong?

Here's the short version. Classical policy-gradient RL comes in two families. One family, which includes the PPO and GRPO that train LLMs, needs the exact log-probability of each action. The other, which includes SAC, needs to differentiate the critic's score back through the policy. A modern robot policy makes the first intractable and the second unstable, which is why the field had to invent a third way: condition the policy on the reward, and train it with behavior cloning (BC), the plain supervised learning that copies recorded actions.

Robots trigger the whole deadly triad

In Breaking Bellman and the deadly triad part of my RL series, I looked at three ingredients that make value learning blow up when they meet: function approximation, bootstrapping and off-policy data. A toy problem can usually avoid at least one of them. A robot needs all three, and each one is forced on us for a concrete reason.

Function approximation. A humanoid like Figure 02 or the G1 has 23 to 43 continuous degrees of freedom (DoF, the joints you can command), and it sees the world through camera pixels. No table has a row for every combination of 43 joint angles and a camera image, so everything has to live inside neural networks that generalize from the states they've seen to the ones they haven't.

Bootstrapping. At 50 Hz, a task like folding a shirt or busing a table is thousands of frames long. Suppose you wait until the end of the episode to see how it went, and then hand that one score back to every frame. Thousands of random decisions all fed into that single number, so the estimate for any one frame is hugely noisy. Instead, you bootstrap. You score a step by its immediate reward $r$ plus the critic's guess $V(s')$, the predicted return from the next state $s'$, discounted by a factor $\gamma$ just below 1:

$$r + \gamma V(s')$$

The catch is that the training target now contains the critic's own guess, so the critic's errors feed back into its own training.

Off-policy replay. Robot hours are expensive. Motors heat up, things break, and a person has to walk over and reset the objects after every attempt. You can't afford to use a rollout for one gradient step and throw it away, which is what on-policy methods do. So modern robot RL is mostly the art of off-policy sample efficiency: squeezing as much signal as you can out of every trajectory in the replay buffer, whether it's a human demo, an autonomous success, a clumsy near-miss or an outright failure.

Why not copy LLM RL?

What about simulation? For some tasks it works beautifully. Train a G1 to walk in Isaac Gym with 4,096 parallel environments, and rollouts are basically free. If one attempt fails, 4,095 others are running, and you can reset any of them instantly. PPO works great there.

Part of the reason it works is the policy. A walking policy is a simple unimodal Gaussian MLP: a small multi-layer perceptron that outputs a mean and a spread for each joint, and the torque is sampled from that bell curve. For a Gaussian you can write down the log-probability of any action exactly. I'll write it as $\log \pi_\theta(a \mid s)$, the log-probability that the policy $\pi$ with weights $\theta$ picks action $a$ in state $s$. PPO and GRPO both need this number, and for a Gaussian it costs one line of math.

Now move to dexterous manipulation with a VLA like $\pi_0$, and you hit two walls.

1. The math wall, even in simulation. Manipulation needs policies that can hold several good answers at once (section 2 shows exactly why), so $\pi_0$ generates its actions with flow matching, a cousin of diffusion. A flow policy starts from random noise and refines it over several steps into an action $a_1$. The trouble is that nobody can write down $\log \pi_\theta(a_1 \mid s)$ for the action it outputs. There's no closed form. So PPO and GRPO can't even compute their loss, and we haven't touched a real robot yet.
2. The reset and replay wall, in the real world. GRPO's trick is to compare 64 answers to the same prompt. An LLM can clone a prompt 64 times in GPU memory, so all 64 attempts start from exactly the same state, and the group's average is a fair baseline without any critic. A robot can't reset a crumpled shirt to the exact same wrinkles 64 times. And once the robot is deployed in someone's home, it can't pause to collect fresh on-policy data for every update. It has to learn off-policy, from a replay buffer of whatever happened.

I contribute to Hugging Face's LeRobot, and that's where I first noticed how big this gap is. Open source has good imitation learners ($\pi_0$, ACT) and a 2018-era SAC. None of the 2025 to 2026 off-policy generative RL stack in this article is implemented yet.

The route

The route through the article follows the three sentences above:

1. Section 1 derives the two classic families from the product rule. Family 1, the score-function family, pushes up the probability of sampled actions that did well. Family 2, the pathwise family, climbs the critic's slope, which makes it Q-learning for continuous actions. The section ends with what each family needs from the policy.
2. Section 2 shows why a robot needs a policy that can hold two good answers at once, and how flow matching ($\pi_0$) does it.
3. Section 3 shows why a flow policy can't give either family what it needs.
4. Section 4 is family 3, the third way: condition the policy on the reward, and train it with BC. RECAP, $\pi_{0.7}$ and GVC, a generative value classifier, all work this way.
5. Section 5 is family 4, which goes beyond the data the robot already has. It searches only among realistic motions (the manifold of real data), either by steering a frozen flow or by tilting its velocity. It ends with my own proposal for where this goes next.

The board on the side follows along as you scroll. Its first panel is the map: four families of policy improvement, from REINFORCE in 1992 to $\pi_{0.7}$, GVC and QF3 in 2026.

1. Score-function versus pathwise gradients

The cliff paradox

Let's shrink the robot down to a single joint, so we can draw everything. The action is one number $a$, say a wrist torque.

We also need a critic. A critic $Q_\phi(s, a)$ is a network with weights $\phi$ that predicts the return (the total future reward) you'll get if you take action $a$ in state $s$. On the board, the critic's curve rises to a peak at $a = +1.5$, where it reads 8, and then drops off a cliff just past $a = +2.0$. Think of a wrist that pushes too far and knocks the cup off the table.

The policy is a Gaussian bell centered at $\mu_\theta = 0$. The critic reads 2 there, so 2 is roughly what the policy earns right now, and we'll use it as our baseline, $V(s) = 2$.

Now sample one action from the bell, and suppose it lands at $a = +2.0$. The critic says $Q_\phi(s, 2.0) = 7$. That's a lot better than the baseline, so the advantage, which measures how much better this action was than average, is

$$A = Q_\phi(s, 2.0) - V(s) = 7 - 2 = +5$$

But look at the slope of the curve at $a = 2.0$. We're already past the peak at 1.5, so the curve slopes down there: $\partial Q_\phi / \partial a < 0$. Moving further right lowers the return, and a little further right is the cliff.

So which way should the bell move?

The two classic families of RL give opposite answers. PPO looks at the +5, concludes that +2.0 was a good action, and slides the bell to the right, toward the cliff. SAC looks at the slope, sees that the return goes up to the left, and slides the bell left, toward the real peak at +1.5.

The underlying score-function and pathwise estimators are unbiased estimates of the same gradient, for a fixed critic. Averaged over many samples, they agree. On this one sample they point in opposite directions, because they measure different things: PPO asks how good the sampled action was, and SAC asks which way is uphill. Let's derive both and see where each answer comes from.

Family 1: score-function gradients (REINFORCE, PPO, GRPO)

We want to change the policy weights $\theta$ so that the expected return goes up. For the derivation, pretend the wrist has a finite menu of torques (we'll come back to continuous ones in a moment). Write the expected return $J(\theta)$ as a weighted sum over all the actions the policy might pick:

$$J(\theta) = \sum_a \pi_\theta(a \mid s)\, Q(s, a)$$

Each term is "how often the policy picks $a$", $\pi_\theta(a \mid s)$, times "how much $a$ earns", $Q(s, a)$. Each term is a product, so take the gradient with the product rule, $(u \cdot v)' = u'v + uv'$:

$$\nabla_\theta J(\theta) = \sum_a \Big[\big(\nabla_\theta \pi_\theta(a \mid s)\big)\, Q(s, a) + \pi_\theta(a \mid s)\, \underbrace{\nabla_\theta Q(s, a)}_{=\,0}\Big]$$

Why is $\nabla_\theta Q(s, a)$ zero? Each $a$ in the sum, like $a = +2.0$, is a fixed point on the axis. Changing the policy weights changes how often the robot picks +2.0. It doesn't change how much reward +2.0 earns in the physical world: the cup falls off the table or it doesn't, whatever the network's weights are. So only the first term survives, and the whole gradient flows through the probabilities.

One practical problem is left. With continuous actions we can't sum over every $a$. We want an expectation instead, something we can estimate by sampling a few actions and averaging. So multiply and divide by $\pi_\theta(a \mid s)$, and use the identity $\nabla_\theta \pi_\theta / \pi_\theta = \nabla_\theta \log \pi_\theta$ (the derivative of a log). The probability factors out in front, and the sum becomes an expectation:

$$\nabla_\theta J(\theta) = \sum_a \pi_\theta(a \mid s)\, \nabla_\theta \log \pi_\theta(a \mid s)\, Q(s, a) = \mathbb{E}_{a \sim \pi_\theta}\Big[A(s, a)\, \nabla_\theta \log \pi_\theta(a \mid s)\Big]$$

On the right I also swapped $Q$ for the advantage $A(s, a) = Q(s, a) - V(s)$. Subtracting the baseline $V(s)$ leaves the expectation unchanged, because $\mathbb{E}_{a \sim \pi_\theta}[\nabla_\theta \log \pi_\theta] = 0$. It does cut the variance a lot: a typical action now gets a number near zero, and only unusually good or bad actions get a big push.

This is the log-derivative trick, and it is family 1. REINFORCE (1992), PPO (2017) and GRPO (2024) are all built on it. A nice way to read it is as supervised learning. Pretend the action you sampled was the correct label, compute the usual gradient that makes that label more likely, and scale that gradient by the advantage. Good actions get pushed up and bad ones get pushed down.

Back to the cliff. For a Gaussian with mean $\mu$ and spread $\sigma$, the gradient of the log-probability with respect to the mean is

$$\frac{\partial \log \pi}{\partial \mu} = \frac{a - \mu}{\sigma^2}$$

At $a = 2.0$ and $\mu = 0$ that's positive, and the advantage $A = +5$ is positive too. Their product is positive, so the mean moves right, toward the cliff. Nowhere in this calculation did we look at the critic's slope. The estimator only ever asks how good the sampled action was.

Why PPO wins in simulation and flatlines on an old buffer

That blindness to the slope has an upside, and it's why people love PPO when rollouts are free. (The cliff picture drew a $Q$ curve so that we could see both directions at once. Ordinary PPO gets its advantage from the rollout itself.) PPO's critic is a state critic $V_\phi(s)$, trained only on fresh rollouts that just happened. The actor uses these state values to estimate advantages, and it never climbs an action critic's slope $\nabla_a Q_\phi$.

Why is that a good thing? A critic can be badly wrong about actions nobody has tried. Somewhere in the huge space of untried actions, it can predict a big return for no reason at all. I'll call such a spike a critic mirage. PPO never asks the critic about untried actions, so it's immune to mirages.

Now feed PPO a replay buffer from 3 days ago and watch what happens. PPO doesn't use $\log \pi_\theta$ directly. It compares the current policy with $\pi_{\text{old}}$, the policy that collected the sample, through the probability ratio

$$r(\theta) = \frac{\pi_\theta(a \mid s)}{\pi_{\text{old}}(a \mid s)}$$

A ratio of 1.5 means the current policy picks this action 50% more often than the old one did. PPO clips the ratio to $[0.8, 1.2]$, so that no single update moves the policy too far. For one sample, the objective is

$$L(\theta) = \min\big(r A,\ \text{clip}(r, 0.8, 1.2)\, A\big)$$

and the board plots it. Read it one case at a time. For a good action ($A > 0$), the objective climbs with $r$ until $r = 1.2$ and then goes flat: PPO stops paying you for making a good action even more likely. For a bad action ($A < 0$), it's flat below $r = 0.8$: PPO stops paying you for making a bad action even less likely. On those flat plateaus, the gradient is exactly zero.

On fresh data that's exactly what you want. The ratios start at 1, and the clip is a guardrail. But three days of training moved $\pi_\theta$ far from the $\pi_{\text{old}}$ that collected the buffer. The old samples have ratios far outside $[0.8, 1.2]$, and every one that lands on a plateau contributes a gradient of zero. It teaches the policy nothing. That's the expensive robot data you couldn't afford to throw away, and PPO throws it away.

Where Q-learning sits, and family 2: pathwise gradients (DDPG, TD3, SAC)

Before family 2, where does the famous Q-learning (DQN) fit in? In discrete Q-learning there's no actor network at all. If the game has $K$ buttons, you ask the critic about each one and press the best, $a^* = \arg\max_a Q_\phi(s, a)$. That's easy when $K$ is a handful of Atari buttons.

On a G1 with 29 continuous joints there are $\infty^{29}$ possible actions, and you can't loop over them to find the max. So you train a second network to find the max for you. That's the actor $a_\theta(s)$. Pathwise RL is continuous Q-learning: the actor is an amortized argmax finder, trained to climb the critic's slope $\nabla_a Q_\phi(s, a)$. "Amortized" means it pays for the search once, during training, so that at run time the best action costs one forward pass.

A policy still needs a dice roll, for two reasons. The first is exploration: jiggling the torques around the mean is how the robot discovers that slightly different actions earn more. The second is multimodality: one state can branch into several valid futures, like going left or right around an obstacle.

But you can't backpropagate through a dice roll. If $a \sim \mathcal{N}(\mu_\theta, \sigma_\theta^2)$, the sampled number is not a differentiable function of $\theta$, so there's no $\partial a / \partial \theta$ to chain the critic's slope into. The fix takes one line, and it's the same reparameterization trick that trains variational autoencoders (VAEs). Move the dice roll outside the network, into standard Gaussian noise $\epsilon \sim \mathcal{N}(0, I)$ that has no parameters ($I$ is the identity matrix, so every coordinate is independent with unit spread). Then the action is a deterministic formula:

$$a_\theta(s, \epsilon) = \mu_\theta(s) + \sigma_\theta(s)\, \epsilon$$

The distribution of $a$ is the same as before. The difference is that the randomness now lives in $\epsilon$, and $\theta$ enters only through smooth functions. The objective becomes an average over the noise, whose density is $p(\epsilon) = \mathcal{N}(0, I)$, and the product rule goes straight through. Here $N_{\text{weights}}$ is the number of weights in the actor:

$$J(\theta) = \mathbb{E}_{\epsilon \sim p(\epsilon)}\Big[Q_\phi\big(s, a_\theta(s, \epsilon)\big)\Big] \implies \nabla_\theta J(\theta) = \mathbb{E}_{\epsilon \sim p(\epsilon)}\Big[\underbrace{\frac{\partial Q_\phi(s, a)}{\partial a}\Big|_{a = a_\theta}}_{1 \times 29} \cdot \underbrace{\nabla_\theta a_\theta(s, \epsilon)}_{29 \times N_{\text{weights}}}\Big]$$

The other product-rule term would be $\nabla_\theta p(\epsilon)$, and it vanishes at once, because $p(\epsilon) = \mathcal{N}(0, I)$ has no $\theta$ in it.

The top of the panel shows the shapes. The critic's slope $\partial Q_\phi / \partial a$ is a row of 29 numbers, one per joint: for each joint, a sign that says which way is uphill and a size that says how steep. The actor Jacobian $\nabla_\theta a_\theta$ is a $29 \times N_{\text{weights}}$ matrix that says how each weight moves each joint. Multiply them and you get one gradient over all of the actor's weights. This is the bell sliding left in the cliff picture.

The price is the loop at the bottom of the panel. It's the deadly triad in one drawing, and it makes training an off-policy actor and critic together unstable:

1. The critic $Q_\phi(s, a)$ grows a random spike at an action the data never covered: a critic mirage.
2. The actor climbs $\nabla_a Q_\phi$ straight onto the mirage.
3. The Bellman backup $y = r + \gamma Q_\phi\big(s', a_\theta(s')\big)$ uses the hallucinated action at the next state $s'$ to update earlier states, and the mirage sucks the surrounding states in like a black hole.

Each classic algorithm adds a band-aid to this loop.

• DDPG (2015) is the bare loop.
• TD3 (2018) trains twin critics and backs up the smaller of the two, $\min(Q_1, Q_2)$. A random spike rarely shows up in both critics at the same action, so the other critic vetoes it. TD3 also updates the actor less often than the critics, which gives the critics time to fix a spike before the actor chases it.
• SAC (2018) keeps a stochastic Gaussian and adds an entropy bonus $-\alpha \log \pi_\theta$ to the objective, with $\alpha$ a weight. Without it, the spread $\sigma_\theta$ collapses to 0 at the peak, where $\nabla_a Q_\phi = 0$ and nothing else keeps the policy exploring. The bonus pays the policy for staying uncertain.

What each family needs from the policy

Each family asks the policy for one thing:

• Family 1 needs $\log \pi_\theta(a \mid s)$. The gradient is $A \cdot \nabla_\theta \log \pi_\theta$, so you have to be able to compute the exact log-probability of the action you sampled, and differentiate it.
• Family 2 needs a short differentiable path from $\theta$ to $a$. The gradient is $\nabla_a Q_\phi \cdot \nabla_\theta a_\theta$, so the critic's slope has to flow back through the policy into its weights.

The algorithms we met also differ in the data they can use. Here they are side by side:

A robot wants the right column, because it learns from any replay buffer, but without the mirages. The panel shows the bad news. A robot needs a flow policy, for reasons section 2 explains, and a flow policy fails both "Needs" entries. We already met the first failure: a flow has no closed-form $\log \pi_\theta$. The second is that the critic's slope has to travel back through all of the flow's refinement steps, and section 3 shows why that is unstable. So the field built a third family that needs neither.

2. Multimodality versus tractable likelihoods

The mug in the way

Back to the whole robot. The G1 has to reach past a coffee mug on the table. In the demonstrations, half the people went left of the mug ($a = -1$) and half went right ($a = +1$). Both are fine.

Now train a unimodal Gaussian MLP on those demos with a squared error. The best single bell for data at −1 and +1 is a bell centered at their average, $a = 0$. That's straight into the mug. (I worked through this failure in mode averaging.) So manipulation needs a policy that can hold two answers at once, left and right, without blending them.

An LLM has a big advantage here that's easy to miss. Its output is a discrete softmax, say over 1,024 token bins, and the probabilities have to sum to one:

$$\sum_{k=1}^{1024} \pi_\theta(k \mid s) = 1$$

That constraint makes every update of cross-entropy, the standard classification loss, a two-for-one. The softmax turns raw scores, called logits, into these probabilities. When you push up the logit of the correct token, the softmax's denominator grows, so the probabilities of all 1,023 wrong tokens go down automatically. You never had to name a wrong answer to suppress it. The board shows the loss gradient on each logit: $\pi(k \mid s) - 1$ for the correct token and $\pi(k \mid s)$ for each wrong one.

On top of that built-in contrast, a softmax gives you multimodality for free, because two good bins can both hold mass, so left and right can coexist. And it gives you an exact $\log \pi_\theta$ in a single forward pass. That's why Physical Intelligence built FAST. It compresses continuous action chunks with a JPEG-style discrete cosine transform and turns them into discrete tokens, so a robot policy can use the LLM's softmax machinery.

Continuous flow matching, which we look at next, has no such contrast. Its training pulls the output toward each recorded action, and nothing automatically pushes down the wrong actions elsewhere. Section 4 is largely about putting that push-down back.

Flow matching: iterative test-time thinking

So how does $\pi_0$ produce actions? Its output $a_1$ is now a whole action chunk: the joint targets for the next 50 control steps (ticks of the 50 Hz clock), for all 29 joints at once, $50 \times 29 = 1{,}450$ continuous numbers. (The chunk is recomputed as the robot moves, and overlapping chunks are merged.)

Diffusion and flow matching bring iterative test-time compute to continuous space. The same network is called $K = 10$ times in a loop. It starts from pure noise $a_0$, and each call moves the chunk a little closer to a clean action chunk $a_1$.

The network is a velocity field, $v_\theta(a_t, t, s)$. Give it a partly finished chunk $a_t$, a time $t$ between 0 and 1, and the state $s$, and it tells you which direction to move the chunk, and how fast. Here $t$ is the sampler's own clock, from pure noise at $t = 0$ to a finished chunk at $t = 1$. It has nothing to do with the robot's 50 Hz clock: each of the 10 sampler steps updates all 1,450 numbers at once.

Training is self-supervised on the replay buffer, and it's surprisingly simple:

1. Take any recorded chunk $a_1$ from the buffer.
2. Draw a fresh noise sample $a_0 \sim \mathcal{N}(0, I)$ of the same shape.
3. Pick a random time $t \in [0, 1]$, and put a point on the straight line between them, $a_t = (1 - t)\, a_0 + t\, a_1$.
4. Regress the network's velocity at that point, with a squared error, onto the whole arrow from noise to data, $a_1 - a_0$.

That's conditional flow matching (CFM), with the loss

$$\mathcal{L}_{\text{CFM}}(\theta) = \mathbb{E}_{t, a_0, a_1}\Big[\big\| v_\theta(a_t, t, s) - (a_1 - a_0) \big\|^2\Big]$$

The training step and the sampler fit in a few lines:

# train: one step on a batch of B recorded chunks
a1 = buffer.sample_chunks()   # (B, 50, 29)
a0 = randn_like(a1)           # noise
t  = rand(B, 1, 1)            # per-sample time in [0, 1]
at = (1 - t) * a0 + t * a1    # a point on the line
loss = ((v(at, t, s) - (a1 - a0)) ** 2).mean()

# act: 10 Euler steps from noise to a chunk
a = randn(50, 29)
for k in range(10):
    a = a + 0.1 * v(a, k / 10, s)

One thing confused me when I first saw this loss. Why is the target still the full arrow $a_1 - a_0$ at $t = 0.9$, when the point has almost arrived? Shouldn't it be the small distance that's left?

The answer is in the target of the loss. At every $t$, the loss asks $v_\theta$ for the same whole arrow $a_1 - a_0$: the trip from noise to data, driven at constant speed. So $v_\theta$ is a speedometer. It measures speed in mph, while the distance left is in miles. Drive a 60-mile straight road from $t = 0$ to $t = 1$ hour at constant speed. The speedometer reads 60 mph at $t = 0$, at $t = 0.5$, and still at $t = 0.9$, when you're at mile 54. You don't overshoot mile 60, because the last step only lasts $\Delta t = 0.1$ hour: $54 + 0.1 \times 60 = 60$. The Euler update in the code, a + 0.1 * v, is exactly that speed-times-time step.

There's a training benefit too. A constant-velocity target has the same size at every $t$, so the gradients never vanish near $t = 1$.

But if every training path is a straight line, why take 10 steps at test time? Why not one big jump along the line?

Look at the panel. Straight lines run from a Gaussian cloud at $t = 0$ to two targets at $t = 1$, $a_1 = -1$ (left of the mug) and $a_1 = +1$ (right of it). Near $t = 0$ the lines cross each other. A noise point there could sit on a line headed left or on a line headed right, so during training the network saw arrows going both ways from the same spot, and a squared-error regression answers with their average. So $v_\theta(a_0, 0, s)$ points at the middle, and one big step from $t = 0$ would put every sample on the average, which is the mug.

Small steps of $0.1\, v_\theta$ fix this. The lower drawing shows the sampling paths, which are new curves, separate from the straight training pairs above. At every position, the network predicts the average of the training arrows that pass nearby. After each small step, a particle asks the network again from its new position. A particle that drifted a little to the right is now in a region where most training lines were headed right, so its arrow bends right. Step by step, the samples split into two clean branches, one on each side of the mug.

3. Why flow matching breaks both standard RL families

Back to the panel from section 1, now that we know what a flow policy is. Family 1 needs $\log \pi_\theta(a_1 \mid s)$, the log-probability of the chunk the flow produces. Family 2 needs to push the critic's slope $\nabla_{a_1} Q_\phi$ back through those same ten steps and into $\theta$.

Each of the next two subsections takes one red cross and shows where it comes from. For family 1, we have to track how much the flow squeezes probability along the way. For family 2, the backward pass has to cross all ten steps.

Family 1 breaks: squeezed volume and the Jacobian trace

PPO and GRPO need $\log \pi_\theta(a_1 \mid s)$. So what is the probability density of the chunk a flow produces?

Start in one dimension. Take a slice of the noise of width $\Delta a_0 = 1.0$ that holds 30% of the probability mass. As the flow carries it, the arrows $v_\theta$ converge, and at $t = 1$ the same 30% sits in a slice of width $\Delta a_1 = 0.01$. The flow only moves probability around, so the mass stays 30% the whole way. Density is mass over width, so it went from $0.3 / 1.0 = 0.3$ to $0.3 / 0.01 = 30$, a 100× jump. To know the density at the end, you have to track how much the flow squeezed the volume along the whole path.

The sampler follows an ODE, an ordinary differential equation, $da_t/dt = v_\theta(a_t, t, s)$, and the Euler steps approximate its path. To track the density, we need the local rate of squeezing. In one dimension that rate is $\partial v / \partial a$. If it's negative, the points ahead move slower than the points behind, and the slice squeezes. If it's positive, the slice stretches.

With many coordinates, the same idea becomes the divergence of the velocity field. The divergence is the trace, meaning the sum of the diagonal entries, of the Jacobian $\partial v_\theta / \partial a_t$, the matrix of how each coordinate's velocity changes when you nudge each coordinate's position. Each diagonal entry is the one-dimensional rate for one coordinate, so the sum is the total squeezing rate. To keep the numbers small, take one 29-joint action instead of the whole chunk. Its Jacobian is $29 \times 29$ (the full chunk's would be $1{,}450 \times 1{,}450$).

For the continuous flow that the 10 Euler steps approximate, start from the Gaussian noise density $p_0$, which we know exactly, and integrate the divergence along the path:

$$\log \pi_\theta(a_1 \mid s) = \log p_0(a_0) - \int_0^1 \left(\sum_{i=1}^{29} \frac{\partial v_{\theta,i}(a_t, t, s)}{\partial a_{t,i}}\right) dt$$

Here $v_{\theta,i}$ is the velocity of coordinate $i$. Check it on the 1-D slice. The integral of the divergence is the log of how much the width changed, $\ln(\Delta a_1 / \Delta a_0) = \ln 0.01 \approx -4.6$. Subtracting it adds 4.6 to the log density, a factor of $e^{4.6} \approx 100$, which matches the picture.

Now count the cost. One backward pass through a network gives you one row of its Jacobian, and you need a different row for each diagonal entry. So each diagonal entry costs its own backward pass. With 29 joints, that's 29 backward passes at every one of the 10 ODE steps, for every sample. And PPO then wants to differentiate all of it again with respect to $\theta$. At VLA scale it's intractable.

What if we constrain the Jacobian?

Researchers have tried constraining flow layers so that the Jacobian is triangular or the flow is reversible: SAC-NF (arXiv:1905.06893), NFPO (arXiv:2603.11470), PrT-Flow (arXiv:2610.17667) and GenPO++ (arXiv:2606.06967). Others bound it with spectral normalization so the flow is Lipschitz (arXiv:2504.08246). The catch is that triangular or bi-Lipschitz Jacobians restrict what each layer can express (arXiv:2107.07232), and they rule out the standard unconstrained transformer action experts like the one in $\pi_0$.

Family 2 breaks: ten Jacobians multiply into a mirage

Fine, skip the likelihood and climb the critic like SAC. The sampler is ten Euler steps, all differentiable, so backpropagate $\nabla_{a_1} Q_\phi$ through it. Write out the first two steps:

$$a_{0.1} = a_0 + 0.1\, v_\theta(a_0, 0, s), \qquad a_{0.2} = a_{0.1} + 0.1\, v_\theta(a_{0.1}, 0.1, s)$$

Look at where $a_{0.1}$ appears. The output of step 1 goes inside $v_\theta$ as the input of step 2. So the 10 steps are nested functions, $f_{10}(f_9(\dots f_1(a_0)))$, and the chain rule multiplies their Jacobians.

The same weights $\theta$ are used at all ten steps, so $\theta$ enters the chain ten times. Follow only the part that enters at step 1. Its effect has to pass through the nine steps after it before it reaches the critic, so it picks up nine Jacobian factors. With $t_k = k/10$, this is $g_1$, the first step's contribution to the sampled gradient (for one noise draw):

$$g_1 = \frac{\partial Q_\phi(s, a_1)}{\partial a_1} \cdot J_9 J_8 \cdots J_1 \cdot \frac{\partial a_{0.1}}{\partial \theta}, \qquad J_k = I + 0.1\, \frac{\partial v_\theta(a_{t_k}, t_k, s)}{\partial a_{t_k}}$$

Each $J_k$ is the Jacobian of one Euler step: the identity $I$ (the chunk carries over) plus a small correction from the velocity network. The sampled gradient sums ten such terms, one per step.

That product is backpropagation through a 10-layer recurrent neural network (RNN), and it has the same problem RNNs have. If each of the nine factors stretches the propagated direction by 1.3, the total stretch is $1.3^9 \approx 10.6$. If each one scales it by 0.7, the total is $0.7^9 \approx 0.04$. The gradient explodes in some directions and vanishes in others.

Now look at what gets amplified. The signal at the end of the chain is the critic's slope, in a 1,450-D space. Real, coordinated motions fill only a thin manifold of that space: most random combinations of 1,450 numbers look nothing like a recorded motion. Off that manifold the critic has never seen a sample, and that's where mirages live. The amplified gradient steers the chunk off the manifold and into a critic mirage. It's the black hole from section 1, with ten amplifiers in front of it.

4. Family 3: condition the policy on the reward

Patching PPO onto flows: πRL and FPO

The first camp kept PPO and patched the flow so that it had a likelihood to give.

Patch 1 is πRL (and DPPO before it). The starting noise makes the final action random, but once that noise is drawn, every Euler step is deterministic. A deterministic step has $\sigma = 0$, and a Gaussian $\log \pi_\theta$ divides by $\sigma$. So πRL injects Gaussian noise into all 10 denoising steps. Each step is then a small Gaussian policy with an exact log-probability, so you can treat each one as a step of a Markov decision process (MDP) and run PPO on it.

The cost is the horizon. Every decision the robot makes now contains 10 noisy denoising steps, so an episode of 50 decisions becomes $50 \times 10 = 500$ steps. The reward still arrives at the end, and credit assignment over 500 steps is much harder than over 50.

Patch 2 is FPO. It keeps the deterministic flow and uses the change in flow-matching loss as a stand-in for the change in $\log \pi_\theta$. This is an ELBO-style proxy, from the evidence lower bound on the log-likelihood: a lower loss on an action roughly means a higher likelihood. The proxy then goes into PPO's $[0.8, 1.2]$ clip.

Physical Intelligence tested FPO and PPO on real-robot laundry folding and reported what happened in Appendix D of the $\pi^*_{0.6}$ paper (arXiv:2511.14759). The noisy flow-loss proxy lied about how far the policy had moved. It failed to trigger the clip, the updates wrecked the pretrained VLA weights, and training collapsed unless the learning rate was turned down so low that the robot barely improved. On the board, the proxy's needle stays inside the guardrail while the real ratio has already left it.

RECAP ($\pi^*_{0.6}$): the reward becomes an input

The second camp started from what already works. BC on a flow is the plain loss $\|v_\theta - (a_1 - a_0)\|^2$, and it never needed $\log \pi_\theta$ or $\nabla_a Q_\phi$. So keep that loss, and give the network a label of how well each trajectory went, as an extra input. The policy is conditioned on the reward, and the reward never enters the gradient. That's RECAP:

1. It trains a state critic $V_\phi(s)$ offline, with 201 value bins (the critic predicts a distribution over 201 values instead of one number). This critic only scores states that are actually in the buffer, so nobody ever asks it about an untried action, and it's safe from mirages.
2. The critic labels every trajectory in the buffer. The top 30% by advantage get the label $I = \text{Good}$, and the rest get $I = \text{Bad}$.
3. RECAP runs plain, unweighted BC with the label as an extra input:

$$\mathcal{L}_{\text{RECAP}}(\theta) = \big\| v_\theta(a_t, t, s, I) - (a_1 - a_0) \big\|^2$$

The label works like a prompt for an LLM: one network, and you pick the behavior you want at test time.

Why train on the bad trajectories at all? Why not clone only the good ones? There are two reasons to keep them.

The first is representation. The standard way to use rewards in BC is advantage-weighted regression (AWR): multiply each sample's squared error by a weight $w = e^{A/\beta}$, with $\beta$ a temperature. The weight has to stay positive. A negative weight on a squared error would reward the network for making the error huge, and it would happily send $\|v_\theta\| \to \infty$. So bad samples can only get a weight near zero, and in practice AWR throws away 80 to 90% of the buffer. RECAP keeps 100% of the trajectories at weight 1.0. That forces the VLA's representations to separate what makes a motion succeed from what makes it fail, across every physical state in the data.

The second reason is classifier-free guidance (CFG). The name says what it skips: there's no separate classifier or critic to differentiate at test time. Instead, you query the flow network twice at every step. Once with no label, which gives $v_{\text{all}}$, the average over Good and Bad. And once with $I = \text{Good}$, which gives $v_{\text{good}}$. (Dropping the label on some training steps teaches the same network both.)

If you know Anthropic's work on sparse autoencoders and activation steering, this will look familiar. There, subtracting a "sycophancy" direction from an LLM's activations steers its behavior. Here, the difference $v_{\text{good}} - v_{\text{all}}$ is a concept vector in velocity space. It points from the mixed average through Good, and therefore directly away from Bad. With a guidance weight $\beta$ (a different $\beta$ from AWR's temperature), CFG extrapolates along it:

$$v_{\text{guided}} = v_{\text{all}} + \beta\, \big(v_{\text{good}} - v_{\text{all}}\big)$$

At $\beta = 0$ you get the plain average, at $\beta = 1$ you get $v_{\text{good}}$, and above 1 you go past Good.

Put numbers on it, as on the board. The unlabeled arrow at a point mixes Good and Bad by how likely each is to have produced that point, and in general that mix changes from point to point. At our point, suppose it matches the dataset: Good has weight 30% and pulls with $v_{\text{good}} = (0.7, 1)$, and Bad has weight 70% and pulls with $v_{\text{bad}} = (-0.3, 1)$. The unlabeled average is

$$v_{\text{all}} = 0.3\,(0.7, 1) + 0.7\,(-0.3, 1) = (0, 1)$$

which points straight ahead, a blend of good and bad. The concept vector is $v_{\text{good}} - v_{\text{all}} = (0.7, 0)$, and with $\beta = 2$:

$$v_{\text{guided}} = (0, 1) + 2\,(0.7, 0) = (1.4, 1)$$

The guided arrow goes twice as far sideways as the Good demos did. The policy extrapolates past $v_{\text{good}}$ and actively repels the failure modes.

# label once, with the 201-bin state critic V
for traj in buffer:
    good = advantage(V, traj) in top_30_percent
    traj.I = "Good" if good else "Bad"

# train: plain flow-matching BC on 100% of the data,
# with the label as an input
# (dropping I on some steps teaches the unlabeled v_all)
loss = ((v(at, t, s, I) - (a1 - a0)) ** 2).mean()

# act: two network calls per Euler step
for k in range(10):
    v_all  = v(a, k / 10, s, I=None)
    v_good = v(a, k / 10, s, I="Good")
    a = a + 0.1 * (v_all + 2.0 * (v_good - v_all))

The simulator below is a toy version you can play with. Seen from above, each action is a 2D point: where the gripper passes the mug. The clean passes on the left and on the right are the top 30% of the demonstrations, labeled Good. A crash through the mug and two clumsy brushes past it are Bad. The velocity field is the exact flow-matching solution for this data, which is what a perfect network would learn, so the only knobs are the number of Euler steps $K$ and the guidance $\beta$.

Try three things. First, set $\beta = 0$ and one Euler step: every sample lands on the average of the data, inside the mug, as the one-big-step argument in section 2 predicted. Second, raise the steps, and the samples spread out over all five clusters of the data, crash included. Third, set $K = 2$ and move $\beta$ from 0 (all data) through 1 (Good only) to 2. With the default 160 samples (seed 7), $\beta = 1$ still leaves 34% of the samples nearest a Bad cluster, and $\beta = 2$ cuts that to 18%.

Richer labels ($\pi_{0.7}$) and actions in the output (GVC)

RECAP's label is a single bit. $\pi_{0.7}$ (arXiv:2604.15483) replaces the 1-bit tag $I \in \{0, 1\}$ of $\pi^*_{0.6}$ with three richer metadata labels, plus an image:

• Speed,
• Quality, from 1 to 5,
• Mistake, true or false,
• and a visual subgoal image of the scene 4 s ahead.

That distills all the task specialists into one steerable generalist: you choose the behavior by choosing the labels.

The Generative Value Classifier, GVC (Swerdlow, Dong and Finn, arXiv:2610.22497), pushes the same flip into the critic. It starts from a rule of deep learning that's easy to forget: inputs create no supervision. Only the outputs in the loss do.

Look at a standard critic with that rule in mind. $Q_\phi(s, a) \to \hat{R}$ takes the 1,450-D action chunk as an input and predicts one scalar, the return $\hat{R}$, trained with $(\hat{R} - R)^2$ against the observed return $R$. All that rich action data sits on the input side, and each sample supervises exactly one number. One number per sample starves the critic of signal.

GVC flips it. It moves the return into the input, as one of $M$ discrete return bins $z_1, \dots, z_M$, and puts the 1,450-D action velocity in the output target:

$$\big\|v_\theta(a_t, t, s, z_i) - (a_1 - a_0)\big\|^2$$

It's the RECAP loss again, with a return bin where RECAP had Good or Bad. Every sample now supervises 1,450 numbers instead of 1, which is 1,450× more supervision.

But a critic has to answer "how good is this action?", and this network answers "what would you do if you were aiming for return $z_i$?" How do you get a score out of it?

With Bayes' rule, and the use case is Best-of-$N$ at test time. The policy proposes $N = 16$ candidate chunks. For each candidate $a$, GVC asks every return bin how well it explains the chunk. It checks which bin $z_i$ gives the smallest flow-matching error on $a$, using the error as a stand-in for $-\log p(a \mid s, z_i)$.

It's the same ELBO proxy as FPO, with a smaller job: here it only ranks 16 candidates, and it no longer decides how far a training update may move the policy. If $a$ looks like what the network does when it's conditioned on a high return, the high bins win.

Bayes' rule combines these per-bin likelihoods with a prior $p(z_i \mid s)$, how likely each return bin is from state $s$ before looking at the action, and turns them into probabilities over the bins:

$$p(z_i \mid s, a) = \frac{p(a \mid s, z_i)\, p(z_i \mid s)}{\sum_{j=1}^{M} p(a \mid s, z_j)\, p(z_j \mid s)}$$

The sum over all $M$ bins in the denominator makes the probabilities add up to one. The score is the expected return under them, $Q_{\text{GVC}}(s, a) = \sum_i p(z_i \mid s, a)\, z_i$. A temporal-difference (TD) anchor keeps the numbers calibrated: the loss $(Q_{\text{GVC}} - y_{\text{TD}})^2$, where $y_{\text{TD}} = r + \gamma V(s')$ is the bootstrapped target from the start of the article. The robot executes the best of the 16.

cands = [policy.sample(s) for _ in range(16)]

def q_gvc(a):
    # a small error means bin z explains the chunk a
    err = array([flow_error(a, s, z) for z in bins])
    # -err stands in for log p(a | s, z)
    p = softmax(-err + log_prior(s))
    return (p * bins).sum()   # expected return

execute(max(cands, key=q_gvc))

5. Beyond cloning the dataset: family 4

Breaking the behavior-cloning ceiling

Family 3 has a ceiling. Reward-conditioned BC like RECAP and $\pi_{0.7}$ can only interpolate between trajectories that already exist in the replay buffer. Guidance and Best-of-16 choose among behaviors the network learned from that buffer. To beat the buffer, the critic has to drive a search again. How do we do that without backpropagating through 10 ODE steps and into a mirage? The frontier has two answers, one on each half of the board.

4A: freeze the VLA and steer it. The first answer leaves the big pretrained model untouched.

One way is a residual. Policy Decorator (Yuan et al., arXiv:2412.13630), PLD (arXiv:2511.00091) and MiDAS (arXiv:2608.11363) freeze the pretrained VLA and train a tiny residual edit $\Delta a$ that's added to its action. The VLA does the heavy lifting, and a small network learns the correction.

The other way steers the noise. DSRL (Wagenmaker, Levine et al., arXiv:2506.15799) and FRS (arXiv:2606.13675) freeze the flow VLA $\theta$ completely and train a lightweight RL policy to pick which starting noise vector $a_0$ goes into the ODE. The small policy's action is $a_0$, the frozen VLA decodes it into a chunk, and the small policy learns from the return, so no gradient has to pass back through the ten steps. Every $a_0$ in the Gaussian ball decodes into a realistic motion, because that's what the flow was trained to do. So the RL policy can choose any noise it likes, the output stays on the manifold of real motions, and critic mirages in action space become impossible.

4B: tilt the velocity with a one-step preview. The second answer keeps training the flow itself, but never backpropagates $\nabla_a Q_\phi$ through all 10 steps. QF3 (Kim, McAllister, Goldberg, Abbeel, Kanazawa, arXiv:2610.08789) and FlowDPG (arXiv:2606.22303) work this way. They pull the velocity toward the critic's slope, an idea that started with Q-score matching.

The trick reuses the speedometer. At any single step $t$, you can preview where the flow is heading by driving at the current speed for the time that's left:

$$\hat{a}_1 = a_t + (1 - t)\, v_\theta(a_t, t, s)$$

Then ask the critic which direction improves $\hat{a}_1$, and tilt the velocity arrow at that step a little in that direction. The gradient passes through one step, so the product of ten Jacobians never forms.

Fixing $\pi_{0.7}$'s pixel subgoals, coarse tags and amnesiac states

$\pi_{0.7}$ leaves three things to fix, and the fixes lead into the loop at the end of the article.

Its subgoals are pixels. To make the 4 s subgoal, a 14B image model takes 1.25 s on 4× H100 to render an RGB picture of the future. The policy then re-encodes that picture into ViT tokens (the patch embeddings of a vision transformer), which is the form it actually uses. So we render pixels only to turn them straight back into embeddings. That's a temporary stepping stone. Predicting subgoals directly in the latent ViT embedding space skips the rendering and is 100× faster.

Latent subgoals also keep hindsight experience replay (HER) working. HER relabels a trajectory with the goal it actually reached, so a failed attempt still teaches something. Say the arm smoothly grabs the wrong corner of the shirt. Choosing that subgoal was a mistake, so the subgoal gets Mistake: true. For the low-level flow controller, HER relabels the corner it actually reached as its goal. Relative to that goal, the controller did its job smoothly, so it still gets Quality: 5.

Its tags are coarse. Quality and Speed label whole episodes or coarse segments. An episode with one perfect grasp and one clumsy fold gets a single tag, and there's no dense credit for each step.

Its state has no memory. Current VLAs see only the current frame $o_t$, or 2 frames. Suppose a heavy box slips out of the grasp at $t = 1$ s. At $t = 2$ s the box is back on the table, and the camera frame looks identical to the one before the first try. The policy can't know it already tried, so it repeats the failed grasp forever. The fix is a compressed episodic memory-imprint vector in the state. It encodes the actions the robot recently tried and the critic's surprise $\Delta V$ (how far the predicted value jumped when the box slipped). With it, the policy can adapt to the physics in context at test time.

The finale: simulate before you act

Max Bennett's A Brief History of Intelligence describes one of the defining leaps in vertebrate brain evolution: the ability to pause and simulate candidate actions inside an internal world model before acting. A cat or a squirrel on a branch pauses, mentally tries three different jumps, scores whether each one lands safely or falls, and only then executes the winner. LLMs do something like it with token-level chain of thought. Physical robots have lacked the equivalent: fast internal physical simulation with action scoring.

This last part is my own extrapolation of where the field is going. Put the pieces of this article together and you get a design for that loop, which I'll call family 4C. It's AlphaZero's loop for physical robots: AlphaZero searches over candidate moves, scores them with a learned value, and trains its policy to imitate the result of the search.

while True:
    s = observe() + memory_imprint()
    # propose
    cands = [flow_policy.sample(s) for _ in range(16)]
    # simulate, in latent space
    future = [world_model.rollout(s, a) for a in cands]
    # score the imagined outcomes
    best = argmax([gvc_score(f) for f in future])
    # act, then distill on the next pass
    execute(cands[best])
    buffer.add(s, cands[best], quality=5)

1. Propose. Given the state and its memory imprint, the multimodal flow policy proposes $N = 16$ diverse candidate chunks: a left grasp, a right grasp, a fast push, a gentle hook.
2. Simulate. A fast latent world model rolls the 16 chunks forward in embedding space in milliseconds, and GVC scores the 16 outcomes in parallel.
3. Act and distill. The robot executes the best of the 16 and logs the winning trajectory back into the replay buffer as a new Quality: 5 experience, so the base flow reflex absorbs it on the next training pass. AlphaZero's policy learns the same way, by imitating the moves its search selected.

Parts of this loop already exist:

• GPC (Qi et al., arXiv:2502.00622) samples candidate actions from a frozen diffusion policy, forecasts each one with an action-conditioned world model, and executes the best.
• FOREWARN (Wu et al., arXiv:2502.01828) imagines candidate plans in a latent world model and lets a fine-tuned vision-language model decide which ones to keep.
• VeriVLA (Sun et al., arXiv:2610.21985) does the same for a flow VLA: a latent world model that predicts future image embeddings ranks candidate chunks by the task progress it predicts.
• Ctrl-World (Guo et al., arXiv:2510.10125) and Verify, Act, Distill (Hiratsuka et al., arXiv:2610.14885) close the distill step. They fine-tune the policy with the ordinary flow-matching loss on imagined successes, or on the chunks a verifier chose.

I haven't found a paper that puts all of it in one loop: a GVC score on imagined latent futures, a memory imprint in the state, and every winner distilled back into the flow. That's the experiment I'd run next.