On-policy and off-policy: whose future are we learning?

Your favorite route runs beside a cliff, and while you're still exploring, one random move can make you fall. Should the route's value include those random moves? If it does, you're learning on-policy: the value describes the moves you actually make. If it doesn't, you're learning off-policy: the value describes a different policy, here a greedy one that never slips. You can see the difference by holding one next-state row fixed and calculating three targets (Sutton and Barto, §§5.5 and 6.4–6.6).

Part 4. A value estimate credits earlier actions with rewards that arrive later, so it has to assume which actions come next. The task, the rewards and the table stay the same.

Two policies, one task

The rule that collects your moves is the behavior policy, written $b$. The rule whose future you want to value is the target policy, written $\pi$. Both give the probability of action $a$ in state $s$: $b(a\mid s)$ and $\pi(a\mid s)$.

So a down move can show up in the data even though you're valuing a future that always goes right. Learning is on-policy when the two rules match and off-policy when they differ. Q-learning is off-policy even when it learns from each move as it happens: the exploratory rule picks the move, but the target assumes greedy actions afterwards. Off-policy is about whose future the target describes, not about how old the data is.

One row, three targets

Hold the next-state row fixed: right is worth $10$, down is worth $-10$, the behavior picks them with probabilities $0.9$ and $0.1$, the reward is $-1$ and $\gamma=0.9$. Every method moves $Q(S_t,A_t)$ toward a target $Y_t$. They only differ in how the next action enters that target:

$$\begin{aligned} Y_t^{\mathrm{Q}}&=R_{t+1}+\gamma\max_{a'}Q(S_{t+1},a'),\\ Y_t^{\mathrm{Sarsa}}&=R_{t+1}+\gamma Q(S_{t+1},A_{t+1}),\\ Y_t^{\mathrm{Expected}}&=R_{t+1}+\gamma\sum_{a'}\pi(a'\mid S_{t+1})Q(S_{t+1},a'). \end{aligned}$$

Q-learning takes the best next action: $-1+0.9(10)=8$. Sarsa uses the next action it actually sampled, so its target is $8$ after right or $-10$ after down. Expected Sarsa with $\pi=b$ averages over the next action: $-1+0.9[0.9(10)+0.1(-10)]=6.2$. That removes the noise from sampling the next action. With a greedy $\pi$, Expected Sarsa becomes Q-learning. §§6.4–6.6.

Exploration has a cost

Q-learning's target values a future without exploratory slips, so it learns the route right along the cliff edge. Sarsa includes the slips in the behavior it evaluates, so it can prefer a longer route that leaves room to recover. That's the book's cliff example, with exploration that never stops (Example 6.6). Neither target is a safety guarantee: Sarsa's value is an expected return for its own policy, and that policy can still choose dangerous actions.

Data must cover the question

Suppose a stored dataset never takes a bridge that the target policy would use. Replaying the other transitions more often won't tell you where the bridge leads. For action correction, the coverage requirement is

$$\pi(a\mid s)>0\;\Longrightarrow\;b(a\mid s)>0:$$

every action the target might take must also be possible under the behavior. The states have to show up in the updates too, and a buffer can skip them or sample them very unevenly. That freedom to learn about another policy is what off-policy methods are for, and its price is that coverage becomes part of the learning problem (§5.5). Part 7 takes this to a fixed robot log, where a missing action becomes a target nothing corrects.

Correcting action probabilities

Say the data choose right and down equally, but the target chooses right four times as often. A plain average then answers the wrong policy's question. The importance ratio $\rho(a\mid s)=\pi(a\mid s)/b(a\mid s)$ fixes each action's share, because behavior probability times the ratio leaves target probability. For any fixed quantity $f(s,a)$ attached to an action:

$$\mathbb E_{A\sim b}[\rho(A\mid s)f(s,A)]=\sum_a\pi(a\mid s)f(s,a).$$

With behavior $(0.5,0.5)$, target $(0.8,0.2)$ and quantities $(10,-10)$, the ratios are $1.6$ and $0.4$, and $0.5(1.6)(10)+0.5(0.4)(-10)=6$, the same as $0.8(10)+0.2(-10)$. That's a one-step identity. Correcting a whole return needs a product of ratios, and a rare behavior action can get a huge weight. §5.5.

Which guarantees survive?

On-policy sampling fixes one mismatch, not every problem. The familiar convergence result covers fixed-policy prediction with fixed linear features, enough visits and shrinking step sizes (Tsitsiklis and Van Roy, 1997). Sample the states off-policy and the same linear TD can diverge, because one shared weight vector $w$ moves a prediction and its own target together. Part 5 shows how. For a changing control policy or a neural network there's no general guarantee, so each method needs its own analysis.

choose A_t using behavior b at S_t
observe R_(t+1), S_(t+1), terminal
if terminal: Y_t = R_(t+1)
else: Y_t = R_(t+1) + gamma * sum_a pi(a|S_(t+1)) Q(S_(t+1),a)
move Q(S_t,A_t) a fraction alpha toward Y_t

Is off-policy a must at scale?

A growing dataset fills up with old runs, other controllers and failed attempts, and using all of it means learning off-policy. Whether you need that depends on the domain.

Robots. Real robot time is expensive, so real-robot RL stores old attempts in a replay buffer or a dataset and trains on them again. QT-Opt pooled over 580,000 grasps from 7 robots (2018), and RECAP in π*0.6 learns from good and bad runs together (2025), with a Monte Carlo value instead of Q-learning. In simulation, parallel environments make fresh data cheap, and Proximal Policy Optimization (PPO) is still the usual choice (FastTD3). In the table of real-world successes in a 2025 survey, locomotion has 37 on-policy, 6 off-policy and 0 offline entries, and manipulation has 9, 24 and 6 (my count).

LLMs. The main algorithms, PPO and Group Relative Policy Optimization (GRPO), learn from fresh samples of a recent version of the model. InstructGPT used PPO, and DeepSeekMath, which introduced GRPO, took a single update on each fresh batch. Large systems drift a little from fresh data. DeepSeek-R1 made one pass over each batch of 8,192 outputs in 16 minibatch steps, so its later steps used slightly older samples. Others reuse a batch for several updates or let rollouts lag a few steps, then correct with clipped or truncated probability ratios. In a controlled study, results generally got worse as the data grew more off-policy, while an asynchronous setup with a one-step lag matched the synchronous results and trained faster (Noukhovitch et al.).

There's no defensible count of the off-policy share of RL methods. A rough proxy points the other way: among arXiv abstracts that name PPO, TRPO, A2C or A3C versus DQN, DDPG, TD3 or SAC, the off-policy-only share fell from 61% in 2022 to 35% in 2025. Off-policy mentions still grew, from 244 to 416 abstracts, while on-policy mentions grew faster. The proxy counts mentions, not use, but its share shows no rise.

Part 5: the deadly triad follows the shared-weight feedback all the way to a growing number. Later, Part 6 turns action credit into the policy updates that PPO and GRPO make.