Mode averaging: six blackboard panelsPanel 1: seen from above, a left route and a right route around a box, both safe; their point-by-point average goes straight through the box. Panel 2: demonstrated offsets at minus 1 and plus 1; the squared-error answer is 0, inside the box band, and a single bell curve fitted to them peaks at 0 with 38 percent of samples in the box; an arrow points from the label squared-error fit to that answer. Panel 3: with one command and a new look each time, a small lean grows into one route; with a chunk of commands and no looking, the averaged chunk runs through the box. In the game's fork level with 10 commands per decision, a Gaussian copy reached the goal 33 percent of the time, a deterministic copy 67 percent and a flow copy 85 percent. Panel 4: four kinds of model that can hold two answers: a mixture, action tokens, a hidden style variable, and diffusion or flow. Panel 5: three noise starts follow a flow over integration time to plus 1 or minus 1; the robot moves only after integration ends. Panel 6: the learned flow splits the noise starts into a region for the left route, a region for the right route, and a route in 5 percent of the data may get only a sliver or nothing; noise near the divide between regions can land between the routes, and the game's flow copy with 10-command chunks still hit the obstacle 3.65 times per test; a flow copies routes without judging them, a chunk still runs without looking, and every chunk costs 10 network passes. 1. Two safe routes, one bad average box start cup left route, saferight route, safe average, point by point: through the box 2. Squared error picks the middle box: |y| < 0.5 demosdemos bell curve squared-error fit −10+1 offset y, in 10 cm units one bell curve: 38% of samples in the box 3. Chunks remove the chance to correct one command, look again10 commands, no looking a small lean grows into one routethe averaged chunk runs through game fork level, 10 commands per decision, reached the goal: Gaussian 33%deterministic 67%flow 85% 4. Models that can hold two answers mixture of bell curvesaction tokens (bins)hidden style (CVAE)diffusion or flow style rightleft 5. A flow turns noise into one plan integration time τ, inside the computer start 0.3start 0.05start −0.8 rightboxleft τ = 00.51 The robot moves only after τ = 1. 6. What a flow still doesn't guarantee noise starts, split by the learned flow left routeright routerare near the divide: can land between a route in 5% of the data may get a sliver, or none game, 10-command chunks: still 3.65 hits per test It copies routes, it doesn't judge them.A chunk still runs without looking.10 network passes for every chunk.

Suppose you teleoperate a robot arm to reach a cup that sits behind a box. Half the time you go around the left side and half the time around the right, and both are good demonstrations. Train a policy that predicts one action to be as close as possible to every demonstration, and it can learn the average, straight through the box. This is mode averaging, and it doesn't come from a bad reward or a wrong value estimate. It comes from asking one answer to stand in for two different good answers.

Part 8. Previously: learning from a fixed dataset. The failure map in part 0 puts this problem under "copying the wrong thing", next to copying bad demonstrations from part 7.

Two safe routes, one bad average

Seen from above, each route is a safe plan on its own. Each dot is one waypoint of a five-step plan. Average the left plan and the right plan point by point and you get five waypoints on the center line, and the straight segment between them crosses the box. The space of safe plans isn't convex, so the midpoint of two safe plans doesn't have to be safe.

A mode is a distinct cluster of good actions or plans for the same situation. Here there are two, the left route and the right route, and the demonstrations are split evenly between them.

Squared error picks the middle

Let's make it small enough to calculate. Measure how far the gripper passes to the side of the box with one number yy, in units of 10 cm. The left route passes at y=−1y=-1 and the right route at y=+1y=+1. The box covers ∣y∣<0.5|y|<0.5, so anything closer to the center than half a unit hits it.

Now train a policy f(o)f(o) that outputs one number for the observation oo, using squared error over the NN demonstrated offsets yiy_i. The fit chooses the output that minimizes

1N∑i=1N(f(o)−yi)2.\frac{1}{N}\sum_{i=1}^{N}\big(f(o)-y_i\big)^2.

The minimizer of a squared error is the mean, and with half the demonstrations at −1-1 and half at +1+1, that mean is 00, the center of the box. More data only makes the estimate of the mean more precise, and the mean is the wrong summary of these demonstrations.

A single bell curve doesn't fix it either. Fit a Gaussian with a learned mean and spread to the same data by maximum likelihood, and you get mean 00 and standard deviation 11. It still peaks at the box, and if you sample from it once and commit, about 38% of samples fall inside ∣y∣<0.5|y|<0.5. A single bell curve has one peak, so it can't put its weight on −1-1 and +1+1 while leaving the middle empty.

These are teaching numbers on idealized data. They don't say that every policy of this kind hits the obstacle on a real robot, and the next section shows why the outcome depends a lot on how often the robot gets to look again.

Chunks remove the chance to correct

Suppose the robot picks one small command, moves a little, looks again and picks the next command. If the first command leans slightly to the right, the next observation shows the gripper a bit to the right of center. That state looks more like the right-route demonstrations, so the next command leans further right. The exact middle is unstable, so a small lean grows and the robot commits to one side.

Now let the policy output several future commands at once and execute them before it asks again. That's an action chunk. Chunks are useful: they make motion smoother, they reduce how often a large model has to run, and ACT (Zhao et al., 2023) uses them to cut the number of decisions in a task. But within a chunk, the robot can't use feedback to break the tie, and the longer the chunk, the more of the motion is decided by that midpoint before anyone looks again.

My game Breaking Bellman has a level with two ways around an obstacle. It trains three copies of the same demonstrations: a Gaussian copy, a deterministic copy that outputs one action trained with squared error, and a flow copy. With one command per decision, all three reached the goal in every test. With 10 commands per decision played without looking, they reached it 33%, 67% and 85% of the time. That's one toy, not a ranking of methods.

What the game measured, and what it doesn't show

Fork level, 3 seeds, simulation. With one command per decision, all three copies reached the goal in every test. With 10 commands per decision played without looking, the Gaussian copy reached the goal 33% of the time with 15.0 obstacle hits per test, the deterministic copy 67% with 13.7 hits, and the flow copy 85% with 3.65 hits. The flow's 85% uses the level's own settings (step size 0.003, batches of 128). With the game's default settings (step size 0.001, batches of 64), the flow copy reached 63%, and the log doesn't give the other two copies at those settings.

The game's log notes that the gap depends on the training budget, since with 10,000 updates the deterministic copy reached 100%. It also doesn't record where the chunked copies hit the obstacle, so these numbers don't show that the failures went through the middle.

Chunks don't cause the problem by themselves. If the policy can sample a whole chunk from one mode, all five waypoints come from the left route or all from the right. The trouble is a chunk produced by a model that can only express the average.

Models that can hold two answers

What we need is a policy that, given the same observation, can output −1-1 on one call and +1+1 on another, and almost never 00. Several kinds of models can represent this:

  • A mixture of several Gaussians, each one able to sit on a different mode.
  • Discrete action tokens, which turn the action into a classification problem with one probability per bin.
  • A conditional variational autoencoder, which ACT uses so that a hidden variable can select a style of demonstration.
  • Diffusion and flow models, which start from random noise and transform it into an action.

Diffusion Policy (Chi et al., 2023) shows the effect in a pushing task where the end effector can go left or right around a T-shaped block. The authors report that their policy learns both modes and commits to one within each rollout, while two of their baselines lean toward one mode and another fails to commit. The rest of this article follows the flow-model branch, because it's the one used by π0 (Black et al., 2024), a robot foundation model that outputs action chunks.

A flow turns noise into one plan

A flow policy doesn't output an action directly. It outputs a direction to move in. Let xx be a candidate action chunk, τ\tau a number from 0 to 1 that we'll call integration time, and oo the observation. A network vθv_\theta with weights θ\theta predicts a velocity, and the candidate follows it:

dxdτ=vθ(x,τ,o).\frac{dx}{d\tau}=v_\theta(x,\tau,o).

Start with xx drawn from random noise at τ=0\tau=0 and follow the velocity until τ=1\tau=1, which π0 does in ten small steps. The end point is the action chunk the robot executes.

Integration time is not robot time. All ten of those steps happen inside the computer, before the robot moves at all, and the intermediate values of xx are not positions the gripper visits. In π0 a chunk holds 50 future actions. On its 50 Hz robots, the authors execute 25 of them, half a second, and then run the model again.

Where does the choice of mode come from? From the starting noise. In the drawing, with the exact velocity field a perfect learner would end up with for our even two-route split, a start at 0.3 ends on the right route, a start at −0.8 on the left route, and a start at 0.05 hesitates near the middle and then commits to the right. Only a start almost exactly at 0 ends at the average. So the velocity field splits the noise into regions, and each region flows to a different mode.

The trajectories in the drawing, step by step

In the exact, continuous flow, the only start that ends at the average is exactly 0, which a continuous noise distribution draws with probability zero. Ten coarse steps leave a thin band: in this example, a start closer to 0 than about 0.00012 still finishes inside the box, which a standard normal draws about once in 10,000 tries.

Each of π0's ten steps uses the simplest rule, forward Euler: x←x+0.1 vθ(x,τ,o)x \leftarrow x+0.1\,v_\theta(x,\tau,o) and τ←τ+0.1\tau\leftarrow\tau+0.1. The table uses starting noise drawn from a standard normal, the exact velocity field for an even split between −1-1 and +1+1, and ten Euler steps of size 0.1:

Starting noise After 5 steps After 10 steps Plan
0.3 0.21 1.00 right route
−0.8 −0.56 −1.00 left route
0.05 0.04 1.00 right route
0 exactly 0 0 the average
How the velocity is trained, and the exact field in the example

π0 trains the network with a flow matching loss. Take a demonstrated chunk AA, draw noise ϵ\epsilon from a standard normal and a time τ\tau between 0 and 1, then blend them:

xτ=τA+(1−τ)ϵ.x_\tau=\tau A+(1-\tau)\epsilon.

The training target is the direction from the noise to the demonstration, u=A−ϵu=A-\epsilon, and the loss is the squared distance between the network's velocity and that target:

L(θ)=E∥vθ(xτ,τ,o)−(A−ϵ)∥2.L(\theta)=\mathbb E\big\|v_\theta(x_\tau,\tau,o)-(A-\epsilon)\big\|^2.

This is still a squared error, so it is still minimized by a mean. The difference is what gets averaged. The network learns the average direction at a given noisy point xτx_\tau, and at most noisy points the demonstrations that could have produced it mostly belong to one route. So the averaged direction points toward one mode, not toward the middle. Lipman et al. (2022) introduced flow matching as a general method. π0 sets it up in its Section IV, with a chunk of H=50H=50 actions and 10 integration steps. The current arXiv version (v4) prints the target as A−ϵA-\epsilon, as here. Versions 1 to 3 printed ϵ−A\epsilon-A, which has the opposite sign from both the blend above and the forward Euler step, so check the version if you compare against the paper.

For the table, the data are A=+1A=+1 or A=−1A=-1 with equal probability and one-dimensional noise. Let m(x,τ)m(x,\tau) be the expected AA given xτ=xx_\tau=x, computed from the two Gaussians centered at ±τ\pm\tau with spread 1−τ1-\tau. The exact field is

v(x,τ)=m(x,τ)−x1−τ,v(x,\tau)=\frac{m(x,\tau)-x}{1-\tau},

and the table applies the Euler rule above from τ=0\tau=0 to τ=1\tau=1. This is a calculation with a perfect field, not a trained network. The thin band comes from the same calculation: with these ten steps, a start of 0.0001 ends at about 0.42, while with 20 steps even a start of 0.000001 ends at a route.

What a flow still doesn't guarantee

Being able to represent two modes is not the same as learning both of them.

  • Coverage is not guaranteed. If one route appears in 5% of the data, the learned field may give it a tiny region of the noise, or none.
  • The dividing line is where errors live. A learned field is approximate, and noise near the boundary between regions is the most likely to end in between. In the game's fork level, the flow copy still hit the obstacle 3.65 times per test with 10-command chunks.
  • It copies, it doesn't judge. A flow policy reproduces the routes in the data, including slow or risky ones. Choosing the better mode needs a reward or a value estimate, the subject of the previous part on mixed-quality data.
  • Chunks still run open loop. One consistent chunk fixes averaging inside the chunk, but the robot can't react to something that changes during those 25 commands. π0's authors report that temporal ensembling, which averages overlapping chunks, hurt performance, so they execute chunks without blending.
  • It costs computation. Every chunk needs 10 network evaluations of the action part of the model instead of one.

So the failure and the remedy are different things. Mode averaging is what happens when a model that can only express one answer is trained on demonstrations with two. A flow policy is one way to express both answers: it chooses between them using random noise, and the training data decides how much of the noise goes to each.

Policy outputs compared on the two-route example
Policy output Two even routes, sampled once and executed as a chunk Needs
One action, squared error The mean: 0, inside the box Nothing extra
One Gaussian Peak at 0, 38% of samples in the box A spread estimate
Mixture or tokens Can put weight on both routes Choosing the number of components or bins
Flow or diffusion Each noise start flows to one route Several integration steps per chunk

The table uses the idealized two-route example from this article. Closed-loop feedback, the size of the network and the training budget can change the outcome on a real robot.

This is the last part of the series. To see where mode averaging sits among the other failures, go back to the failure map in part 0.

Sources and what each one supports
  • Squared error and the mean. A standard result: the constant that minimizes the average squared distance to a set of numbers is their mean. The Gaussian numbers come from fitting mean and variance to an even split at ±1 (mean 0, variance 1), and the 38% is the probability that a standard normal lies within ±0.5. Teaching calculation.
  • π0. Black et al., 2024, Section IV of arXiv v4 (8 January 2026): action chunks of H=50H=50, flow matching loss with xτ=τA+(1−τ)ϵx_\tau=\tau A+(1-\tau)\epsilon and target A−ϵA-\epsilon (v1 to v3 print ϵ−A\epsilon-A), forward Euler with 10 steps of 0.1, and the statement that flow matching gives the model "multimodal modeling capability". Appendix D: 50 Hz robots run inference every 0.5 seconds after 25 actions, chunks run open loop, and temporal ensembling hurt performance.
  • Diffusion Policy. Chi et al., 2023, Section 4.1 and Figure 3: the Push-T left or right example, with the authors' description of how their baselines behave.
  • ACT. Zhao et al., 2023: action chunking to shorten the effective horizon, and a conditional variational autoencoder because human demonstrators use different trajectories for the same task.
  • Flow matching. Lipman et al., 2022.
  • Breaking Bellman. Fork level numbers from the game's experiment log and web/measured.js at commit 681cae3, batch 23, 3 seeds (runs tw-gauss-c10, tw-det-c10, tw-flow-c10 and their one-command versions). The log records that earlier fork results (batches 8 to 18) were wrong: the demonstrator skimmed the obstacle and hit it in 6 of 50 episodes, so the copies failed from imprecision near the edge rather than from averaging. These numbers come from the corrected world. Simulation.
  • Flow trajectories. The starts 0.3, −0.8 and 0.05, the 0.00012 band and the once-in-10,000 rate are recomputed with the exact field above (ten Euler steps; the band edge is about 0.000123, and a standard normal falls within it with probability about 0.0001).

← All writing