Learning from a fixed dataset: six blackboard panelsEach row puts a problem next to its fix. Top row, a hole in the data. Panel 1: above the mug, lower slowly has estimate 4 from 60 logged attempts, lower fast 3 from 40, and sweep sideways has an unchecked estimate of 9 from 0 attempts while its true value is minus 5; the max picks the sweep, so the target is 8.1 instead of 3.6. Panel 2: BCQ and IQL only consider recorded actions, giving a next-state value of 4 and about 3.93, and CQL pushes the unrecorded estimate down. Middle row, mistakes in the data. Panel 3: 70 attempts rotate then grasp, 30 grasp straight in; behavior cloning copies both. Panel 4: advantage weights move the copy from 70 percent good to 81 percent with temperature 5 and 98 percent with temperature 1. Bottom row. Panel 5: the reward-aware fixes need rewards that tell good from bad, values right on the data, good actions where it matters and true records; on the game's mixed log of 50 good and 50 random episodes, final distance to the target is 0.22 m for IQL, 0.33 for CQL, 0.35 for copying the best episodes, 0.50 for TD3+BC, 0.51 for the one-bit advantage tag, 0.56 for a Gaussian copy of everything and 0.57 for AWR; IQL and AWR both weight by advantage but estimate it differently. Panel 6: a hole, bad behavior and false records need different fixes. 1. A hole: an action nobody tried estimates with the gripper just above the mug 439 ? true −5 max lower slowlylower fastsweep sideways 60 in log40 in log0 in log target: 0.9 × 9 = 8.1, not 0.9 × 4 = 3.6 2. Fix the hole: stay with the data only ask about what was recorded BCQ, IQL look only here 43 was 9 CQL: down slowlyfastsweep next value: BCQ max = 4IQL: V ≈ 3.93 Behavior cloning copies actions and never asks Q at all. 3. Mistakes: bad actions in the log at the mug handle, 100 recorded attempts 70 rotate, then graspworks 30 straight graspoften knocks the mug behavior cloning copies both: the straight grasp in about 30% of visits robomimic nut task: 58.7% → 46.7% with worse operators added 4. Fix the mistakes: weight by advantage advantage: rotate +1, straight −2 plain copyAWR, β = 5AWR, β = 1 70%81%98% weight = exp(A / β) only as good as the advantage estimates 5. What the reward-aware fixes need • rewards that tell good from bad• values right on the data• good actions where it matters• true records game, mixed log of 50 good and 50 random episodes: final distance to the target in m, lower is better IQL 0.22CQL 0.33best episodes 0.35TD3+BC 0.50one-bit tag 0.51Gaussian copy 0.56AWR 0.57 copy IQL and AWR both weight by advantage, but estimate it differently: 0.22 m vs 0.57 m. 6. Bad behavior or bad records? a hole in the datastay near the data: BCQ, CQL, IQL, TD3+BC bad actions, true labelsweight by reward: AWR, IQL, filtering false recordsfix the pipeline: clocks, executed actions,audited success labels ? ≠ No learning method repairs a false label.

Suppose you've recorded a few hundred teleoperated attempts at picking up a mug with a robot arm. Some operators were experienced and some were new, and you can't collect more data right now. Two different things can go wrong when you learn from that log. A learner that tries to improve on it asks about actions nobody tried, which is called extrapolation error, and a learner that copies it copies the mistakes too, the problem of mixed-quality data. They sit next to each other, but they are different problems with different fixes.

Part 7. Previously: policy gradients. This part extends data coverage in part 4 to a log you can't add to. In the failure map, extrapolation error belongs to "value estimates that answer the wrong question", next to the deadly triad.

A hole: an action nobody tried

Take one moment in the log: the gripper is just above the mug. Operators either lowered it slowly (60 times) or fast (40 times). Nobody swept the gripper sideways, because that knocks the mug over. Train a Q-function on this log, the same kind of action-value estimate as in part 1, and real outcomes keep correcting its estimates for the two recorded actions. The sweep's estimate is whatever the network happens to produce there, say 9, while its true value is −5.

Q-learning's target for the move that brought the gripper above the mug takes the largest next-action estimate. With reward 0 for that move and discount γ=0.9\gamma=0.9, that's 0.9×9=8.10.9\times9=8.1, against 0.9×4=3.60.9\times4=3.6 if you only count recorded actions. The earlier state now inherits a value that came from an action nobody took, and the policy that reads it chooses the sweep. A robot that could act would try the sweep, see the mug fall and correct the estimate. With a fixed log nothing ever corrects it, and the next target built from the inflated value carries the error further back. It's the same feedback as in the deadly triad, with the missing correction coming from a dataset that can't grow.

Fujimoto, Meger and Precup (2019) named this extrapolation error: an error introduced by the mismatch between the dataset and the states and actions the learned policy would visit. It isn't limited to actions with zero examples, since an action with two noisy examples can also get an estimate the data barely constrains. So every method that learns from a fixed log has to say what it does about actions the data doesn't support.

The numbers in the drawing
Action above the mug Times in the log Estimate Q^\hat Q True value
Lower slowly 60 4 4
Lower fast 40 3 3
Sweep sideways 0 9 −5

max over all actions:0+0.9×9=8.1,max over recorded actions:0+0.9×4=3.6.\begin{aligned} \text{max over all actions:}&\quad 0+0.9\times 9=8.1,\\ \text{max over recorded actions:}&\quad 0+0.9\times 4=3.6. \end{aligned}

These estimates are made up to show the mechanism.

Fix the hole: stay with the data

The fixes share one idea: limit how far the learner can move from what the data supports. Behavior cloning (BC) copies the recorded actions and never asks a Q-function anything, so it has no extrapolation error, but it can't improve on the data either. It can still drift into states the log never covered, which is a different problem. Batch-Constrained Q-learning (BCQ), from the 2019 paper above, only considers actions that a learned model finds likely, so the max above the mug runs over the two recorded actions and the target is 3.6 instead of 8.1. Implicit Q-Learning (IQL) (Kostrikov, Nair and Levine, 2021) never evaluates an unrecorded action at all, so its next-state value comes out about 3.93 here, and never 9. Conservative Q-Learning (CQL) (Kumar et al., 2020) pushes the sweep's estimate down until it stops attracting the policy. TD3+BC (Fujimoto and Gu, 2021) adds a term that pulls the policy back toward the recorded action.

Each one gives something up. BCQ and IQL can only find the best of what was recorded. CQL's pessimism can also hold back good actions with few examples. And TD3+BC's pull toward the data is a pull toward whatever was recorded in that state.

How each method sets its limit, and the game's version with good data

My game Breaking Bellman has a level that shows both sides with good data. A simulated arm learns to reach a target from 100 episodes of a controller that ends 0.01 m from the target. DDPG, an actor-critic method that climbs its own critic, ended 0.84 m away. A deterministic copy of the same 100 episodes ended 0.065 m away (a Gaussian copy 0.11 m), IQL 0.12 m, TD3+BC 0.20 m and CQL 0.24 m (3 seeds, simulation). When the log is already good, a reward-aware method has little left to add, and it can cost some precision.

BCQ learns a model of which actions appear in each state. When it builds a target, it only considers actions that model finds likely, plus a small learned adjustment. The cost is that improvement is limited to actions near the data, and the result depends on how well the action model fits.

TD3+BC. In state ss with recorded action aa, the policy π\pi maximizes

λ Q(s,π(s))−(π(s)−a)2,λ=α1N∑i∣Q(si,ai)∣.\begin{aligned} &\lambda\,Q(s,\pi(s))-\big(\pi(s)-a\big)^2,\\ &\lambda=\frac{\alpha}{\frac1N\sum_i|Q(s_i,a_i)|}. \end{aligned}

The first term pulls the action toward higher estimates. The second pulls it back toward the recorded action. Dividing by the average size of QQ over the NN recorded pairs keeps the balance independent of the reward scale, and the authors use α=2.5\alpha=2.5. If a critic error is large enough, the first term can still win.

CQL adds a term that pushes estimates down on actions the policy would choose and up on actions in the data. The authors show this makes the learned value a lower bound on the policy's true value. A weight controls how strongly unrecorded actions are pushed down, and too much of it also holds back good actions with few examples.

IQL fits V(s)V(s) to an upper expectile of the recorded actions' estimates instead of taking a max. An expectile is a weighted average that counts values above it more heavily, controlled by τ\tau between 0 and 1. With τ=0.5\tau=0.5 you get the ordinary mean, and as τ\tau approaches 1 it approaches the largest recorded value. Above the mug, with 60% of the log at 4 and 40% at 3:

τ\tau VV above the mug Reading
0.5 3.6 Average of what operators did
0.9 about 3.93 Close to the best recorded action
any never 9 The sweep is not in the data

The Q-function then learns from r+γV(s′)r+\gamma V(s'), so the target never touches the sweep. IQL keeps the value of the transition separate from the action choice, which also stops one lucky transition from looking like a great action. To turn these values into a policy, it uses advantage weighting, the main tool for the second problem below.

Mistakes: bad actions in the log

Now change the log. Say 70 attempts came from experienced operators and 30 from a new operator. At the moment the gripper reaches the mug handle, the experienced operators rotate the wrist first and then grasp, while the new operator grasps straight in, which often knocks the mug. Both actions are in the data, and both are labeled correctly: the log shows exactly what happened.

Behavior cloning makes every recorded action likely, so an expressive policy reproduces the straight grasp in about 30% of visits to that state. A policy that outputs one action can end up between the two, which is mode averaging again. This has nothing to do with extrapolation error. There's no unrecorded action and no inflated estimate, just content you didn't want in the data. And a method built for the first problem can make this one worse, because BCQ and TD3+BC deliberately keep the policy close to the recorded actions, mistakes included.

There's real evidence that this matters with human teleoperation. In robomimic (Mandlekar et al., 2021), plain behavior cloning on a task where the robot puts a square nut on a rod reached 58.7% success with the better operators' 100 demonstrations, and 46.7% once the worse operators' 100 were added (3 seeds).

More of the robomimic result

A recurrent copier, BC-RNN, went the other way on the same mixture, from 66.0% to 73.3%, and the batch RL methods they tried, BCQ and CQL, did worse than plain behavior cloning on these mixed human datasets. The paper doesn't isolate why BC-RNN gained, so it doesn't show that a better copier always absorbs bad demonstrations.

Operators were grouped by skill, and each group recorded 100 demonstrations. On a task where the robot moves a can into a target bin, the same change took plain behavior cloning from 83.3% to 76.0%, and BC-RNN from 99.3% to 100.0%. BC-RNN sees a history of observations instead of only the current one. The authors sum it up as most algorithms declining when the worse demonstrations were added, while BC-RNN improved. The rewards were binary task completion. These are their results, the best success rate during training averaged over 3 seeds, in their 2021 setup.

Fix the mistakes: weight by advantage

Off-policy learning is what lets you learn from data someone else collected, but permission to reuse data doesn't say which of the recorded actions were good. The usual measure is the advantage, A(s,a)=Q(s,a)−V(s)A(s,a)=Q(s,a)-V(s): how much better an action did than what usually happens from that state. Say the rotate-then-grasp action has advantage +1 and the straight grasp −2.

Advantage-Weighted Regression (AWR) (Peng et al., 2019) still fits the recorded actions, but it multiplies each one's importance by exp⁡(A/β)\exp(A/\beta), where β>0\beta>0 is a temperature. With β=1\beta=1, the good grasp's total weight is its 70% share times e1e^{1}, about 1.90, and the bad one's is 0.3 e−2≈0.040.3\,e^{-2}\approx0.04, so about 98% of the copy goes to the good grasp. With β=5\beta=5 it's 81%, and a large β\beta flattens the weights back toward plain copying. IQL extracts its policy the same way, with advantages from its in-sample critic.

You can also give the policy the advantage as an extra input and ask for a high one at test time. Reward-Conditioned Policies (Kumar, Peng and Levine, 2019) introduced an advantage-conditioned variant, and π*0.6 (Physical Intelligence, 2025) applies it to a robot foundation model with one bit: is this action's advantage above a threshold?

AWR, advantage tags and TD3+BC in detail

The advantage-conditioned variant rests on the idea that a mediocre action is a perfect example of mediocre behavior, so nothing has to be thrown away. π*0.6 sets its threshold so that about 30% of the demonstration data counts as positive.

AWR's objective, over the recorded state and action pairs:

max⁡π  ∑(s,a)∈loglog⁡π(a∣s) exp⁡ ⁣(A(s,a)/β).\max_\pi\;\sum_{(s,a)\in\text{log}}\log\pi(a\mid s)\,\exp\!\big(A(s,a)/\beta\big).

Training Share of the copy going to the good grasp
Plain behavior cloning 70%
AWR, β=5\beta=5 81%
AWR, β=1\beta=1 98%

The authors clip very large weights because they can make training unstable. IQL writes the factor as exp⁡(β(Q−V))\exp(\beta(Q-V)), so its β\beta is an inverse temperature.

π*0.6 marks human corrections as positive, which, in their words, is reasonable if experts always provide good corrective actions. Later in the same paper they qualify that assumption: even expert operators can't guarantee a consistent quality of interventions, so the corrections mainly fix large mistakes.

TD3+BC's QQ term also uses the reward, by tilting the copy toward actions with higher estimates. Its squared-distance term still pulls toward every recorded action in that state.

What the reward-aware fixes need

All of them work by trusting a number attached to each recorded action. So they need rewards that separate good from bad, value estimates that are right on the data, good actions in the states that matter, and true records. With a binary success reward, for example, a hesitant attempt that still succeeds scores the same as a clean one unless a discount or a cost per step tells them apart. And weighting only chooses among recorded actions, so if every recorded attempt fumbles one particular step, no weighting produces a clean version of it.

None of this is guaranteed by a method's name. In the game's version of the problem, the log mixes 50 episodes of a good controller with 50 episodes of random commands. IQL ended 0.22 m from the target, and AWR ended at 0.57 m, no better than a Gaussian copy of everything at 0.56 m (3 seeds, simulation). Both weight actions by an advantage, so in this toy how the advantage was estimated mattered more than the idea of weighting.

What each method uses, and where it helps

On the game's mixed log, the other methods ended at 0.33 m (CQL), 0.35 m (copying only the best episodes), 0.50 m (TD3+BC) and 0.51 m (the one-bit advantage tag), after 8,000 training steps. The length of an episode alone doesn't separate a hesitant success from a clean one: with no discount, both returns are 1.

Method Uses Helps when Limits
Behavior cloning Actions only The log is uniformly good Copies mistakes and can't beat the data
Copy the best episodes Episode returns Good and bad episodes are easy to rank Throws data away, and ranks whole episodes, not steps
AWR Advantage from fitted returns Advantages are accurate Temperature to tune, and noisy returns give noisy weights
IQL In-sample Q and expectile V Good actions are recorded somewhere Best of the data at most, with τ\tau and β\beta to tune
Advantage conditioning Advantage as an input Many data sources of different quality Depends on the advantage estimate and threshold
TD3+BC Q gradient plus copy term Data near good actions, modest critic error Copy term pulls toward recorded mistakes too

Bad behavior or bad records?

Everything above assumed the log tells the truth. The new operator's straight grasp really happened, in the recorded state, with the recorded result. That's poor behavior, bad actions with correct labels, and reward-aware reuse can handle it under the conditions above.

Corrupted data is different, because the record itself is false. On a robot that can look like camera frames and arm commands logged with a clock offset, a command the arm couldn't follow stored as if it had happened, or a failed grasp marked as a success. A reward-aware method takes every record at face value, so a false success gets a large weight from AWR and a positive tag from advantage conditioning. Off-policy learning can't repair it either, since it can reuse a transition from another policy only if that transition is real. These fixes belong to the data pipeline: synchronize the clocks, record the executed action next to the command, audit success labels, and drop episodes that fail consistency checks.

The three problems side by side
Extrapolation error Poor behavior Corrupted records
What's wrong A question the data can't answer Bad actions, true labels False states, actions or rewards
Example above the mug Sweep never recorded, estimate 9 Straight grasp recorded, often fails Failed grasp labeled success
What helps BCQ, CQL, IQL, TD3+BC, BC AWR, IQL, advantage conditioning, filtering Fixing the data pipeline
What doesn't Replaying the log more often Copying everything Any method that trusts the labels

The next part looks at a problem that remains even when every demonstration is good and correctly recorded: two good ways to do the same thing. Part 8: mode averaging.

Sources and what each one supports
  • Extrapolation error and BCQ. Fujimoto, Meger and Precup, 2019, Section 3 (definition and causes: absent data, model bias, training mismatch) and Section 4 (batch-constrained action selection).
  • TD3+BC. Fujimoto and Gu, 2021, Section 5, equations 3 to 5, with α=2.5\alpha=2.5 and state normalization.
  • CQL. Kumar, Zhou, Tucker and Levine, 2020, abstract and Section 3: a conservative Q-function whose expected value lower-bounds the policy's value.
  • IQL. Kostrikov, Nair and Levine, 2021, Section 4: expectile loss, VV fitted over dataset actions (equation 5), QQ fitted to r+γV(s′)r+\gamma V(s') (equation 6), advantage-weighted extraction (equation 7). The 3.93 is a calculation for the two-value example: the 0.9 expectile mm solves 0.9⋅0.6(4−m)=0.1⋅0.4(m−3)0.9\cdot0.6(4-m)=0.1\cdot0.4(m-3).
  • AWR. Peng, Kumar, Zhang and Levine, 2019: weights exp⁡(A/β)\exp(A/\beta), clipped at a maximum, usable with fully off-policy data. The 70%, 81% and 98% are calculations for the illustrative advantages.
  • Advantage conditioning. Kumar, Peng and Levine, 2019, Section 4 (advantage-conditioned variant). π*0.6, Physical Intelligence, 2025, Section IV equation 3, Section V on the limits of human corrections, and the appendix on thresholds (about 30% positive in pre-training, about 40% of rollouts in fine-tuning).
  • robomimic. Mandlekar et al., 2021, Section 2 (operator proficiency, missed grasps) and Section 4.2, Table 2 (success rates over 3 seeds, with 100 demonstrations in the Better and Worse subsets and 200 in the mixtures; BC and BC-RNN columns, and the authors' statement that most algorithms decline while BC-RNN improves). Task descriptions from Section 3.1. Binary task-completion rewards, Appendix C. Author-reported, in their 2021 setup and hyperparameters.
  • Breaking Bellman. Offline reach levels from the game's experiment log and web/measured.js at commit 681cae3. Expert data: 100 controller episodes. Mixed data: 50 controller and 50 random-command episodes. 3 seeds, and distances are final tip distances. Simulation of a two-link arm, not a real robot.
  • Mug example. The estimates 4, 3 and 9, the 70/30 split and the advantages +1 and −2 are made up to show the mechanisms.

← All writing