Policy gradients: six blackboard panelsPanel 1: one attempt with reach, carry and place; a good attempt makes every action more likely, a bad one less likely, by an amount set by a score. Panel 2: rewards minus 2, 0 and plus 10; the whole return gives every action 8, reward-to-go gives 8, 10 and 10. Panel 3: four returns 10, 10, 10 and 0 with a baseline of 7.5; scores become plus 2.5 three times and minus 7.5. Panel 4: the PPO clipped objective against the probability ratio for advantage plus 2 and minus 2, flat outside 0.8 to 1.2 only in the direction the advantage wants, with dashed lines for the objective without the clip; green dots at ratio 1.3 and 0.7 sit on the flat part, red dots sit on the slope and are pushed back toward 1; KL, how far the policy moved per update in the game's CartPole level, was 1.5 without clipping and 0.008 with it. Panel 5: a group of rewards 1, 1, 1, 1 gives advantages 0, 0, 0, 0; a group of 1, 0, 0, 0 gives plus 1.73 and minus 0.58 three times. Panel 6: four problems, each with a small sketch from its panel, its fix and what the fix costs. 1. Push the policy one attempt reachcarryplace went wellevery action ↑ a bit more likely went badlyevery action ↓ a bit less likely Δθ ∝ Σ ∇log π(A | S) × score The score sets how hard each action is pushed. 2. Credit only for what follows reachcarryplace bump −20+10 whole return: every action gets 8 888 reward-to-go: rewards from this action on 81010 The bump came first. It can't be the carry's fault. 3. Compare with the usual result returnscore = return − 7.5 1010100 1234 dropped dashed: the usual return, 7.5 +2.5+2.5+2.5 −7.5 Now the drop is pushed down. Same average update. The noise can drop a lot. 4. Reuse a batch, but clip advantage +2advantage −2 objectiveobjective 0.81.20.81.2ratio rratio r green: flat, no pushred: pushed back toward 1 dashed: the objective without the clip KL, how far the policy moved per update: 1.5 plain, 0.008 clipped 5. Same reward, no signal group A: all four placedgroup B: one placed 11111000 mean 1, std 0mean 0.25, std 0.43 advantages 0000 +1.73 −0.58−0.58−0.58 Group A says nothing about the task. 6. Four fixes, four costs 81010 all 0 past blame → reward-to-gono comparison → baselinereuse overshoots → PPO clipidentical group → filter it still one noisy sample per scoreneeds a good V or several attemptsnot a hard bound: step size, passes, KLextra attempts, a varying number per batch

Suppose you're training a robot arm to place a block on a shelf. Instead of learning a value table and acting greedily, as in the earlier parts, you adjust the probabilities of its actions directly, so actions from attempts that went well become more likely and actions from attempts that went badly become less likely. That's a policy-gradient method. The idea is simple, but the score that decides how hard to push each action can go wrong in four ways, and each fix has a cost.

Part 6. Previously: the deadly triad. In the failure map, these belong to "too little signal", "credit in the wrong place" and "updates that overshoot".

Push the policy

Let πθ(a∣s)\pi_\theta(a\mid s) be the policy's probability of action aa in state ss, with adjustable weights θ\theta. The gradient ∇θlog⁡πθ(a∣s)\nabla_\theta\log\pi_\theta(a\mid s) points in the direction that makes aa more likely in ss. The simplest policy-gradient update, REINFORCE, collects one attempt and pushes each action's probability up in proportion to a score. If the score is the whole attempt's return GG, the update for an attempt with actions A0,…,AT−1A_0,\dots,A_{T-1} is

Δθ  ∝  ∑t=0T−1∇θlog⁡πθ(At∣St) G.\Delta\theta\;\propto\;\sum_{t=0}^{T-1}\nabla_\theta\log\pi_\theta(A_t\mid S_t)\,G.

So everything depends on that score: which rewards it counts, what it's compared with, and how far one batch of attempts is allowed to push.

Credit only for what follows

Take one attempt with three actions. The arm reaches toward the block and bumps the shelf on the way (reward −2), carries the block (reward 0), and places it (reward +10). The whole return is 8, so every action gets a score of 8, including the place action, which had nothing to do with the bump.

A fairer score for action tt is the reward-to-go, the rewards from that action onward: Gt=∑k≥tRk+1G_t=\sum_{k\ge t}R_{k+1}, with no discount to keep the numbers simple. The reach still gets 8, the carry and the place get 10 each, and only the reach is charged for the bump. This is the return REINFORCE uses in Sutton and Barto, Section 13.3.

Dropping the past rewards doesn't change the average update. Given what already happened, those rewards are fixed numbers, and the gradient term averages to zero over the policy's own actions, so in a single attempt they only add a random push. OpenAI's Spinning Up calls this "don't let the past distract you".

Why the past rewards average to zero

The policy's gradient term has zero mean over its own actions:

Ea∼πθ[∇θlog⁡πθ(a∣s)]=∑aπθ(a∣s)∇θπθ(a∣s)πθ(a∣s)=∇θ∑aπθ(a∣s)=∇θ1=0.\begin{aligned} &\mathbb E_{a\sim\pi_\theta}\big[\nabla_\theta\log\pi_\theta(a\mid s)\big]\\ &\quad=\sum_a\pi_\theta(a\mid s)\frac{\nabla_\theta\pi_\theta(a\mid s)}{\pi_\theta(a\mid s)}\\ &\quad=\nabla_\theta\sum_a\pi_\theta(a\mid s)=\nabla_\theta 1=0. \end{aligned}

So a fixed number times this term also averages to zero. In any single attempt, though, it isn't zero: it adds a push that has nothing to do with the action. Leaving it out keeps the same average update with less noise.

Action Reward after it Whole-return score Reward-to-go score
Reach −2 8 −2 + 0 + 10 = 8
Carry 0 8 0 + 10 = 10
Place +10 8 10

Compare with the usual result

Reward-to-go fixes the timing, but here every score is still positive. Say the arm makes four attempts from the same starting position. Three place the block and score 10, and one drops it and scores 0. With raw returns, the three good attempts are pushed up by 10 and the failed one by 0. Nothing gets pushed down, so the failure is only less encouraged, and the difference has to come out of many noisy updates.

Subtract a baseline b(s)b(s), a number that depends on the state but not on the action. A natural choice is the usual return from that state, here the average of the four attempts, b=7.5b=7.5. Now the placements score +2.5 and the drop scores −7.5, so the drop is pushed down and the placements up.

The baseline can't change the average update, for the same reason as before, but it can have a large effect on the noise. That isn't a promise of less noise, since a badly chosen baseline can add some. In practice the baseline is a learned state value V(s)V(s), and the score becomes an estimate of the advantage, how much better the action did than usual.

What a baseline does and doesn't promise

Strictly, with NN attempts, a baseline that includes the attempt itself scales the expected update by (N−1)/N(N-1)/N, here 3/4. The mean of the other attempts avoids this, and the direction is the same either way. Sutton and Barto, Section 13.4, put it this way: the baseline leaves the expected value of the update unchanged, but it can have a large effect on its variance.

Generalized Advantage Estimation in part 2 is the standard way to build the advantage estimate one step at a time, and it inherits the learned value's errors, including the time-limit problem in part 2.

Reuse a batch, but clip

Robot attempts are slow, so you'd like to take several gradient steps on each batch instead of one. After the first step, though, the batch no longer comes from the current policy. The usual correction, as in part 4's action probabilities, weights each action by the probability ratio rr, the new policy's probability for that action divided by the old one's, and maximizes rr times the advantage. Schulman et al. (2017) note that without a constraint, maximizing this objective "would lead to an excessively large policy update". If an action had a positive advantage, raising its probability always looks better.

Proximal Policy Optimization (PPO), from the same paper, clips the ratio, with the paper's setting to the interval from 0.8 to 1.2. In the left plot of the drawing, an action with advantage +2 whose ratio already reached 1.3 sits on the flat part, so nothing pushes it up any further. In the right plot, an action with advantage −2 whose ratio drifted up to 1.3 is still on the slope, so it gets pulled back down. At ratio 0.7 the two cases swap. Clipping removes the incentive to keep moving in the direction the advantage wants, and it keeps the full correction in the other direction.

That isn't a hard limit, though. Several passes over the data and shared weights can still push the ratio out of the interval, so PPO implementations also control the step size and the number of passes. Many also stop a batch early when the KL divergence, a measure of how far the new policy has drifted from the old one, grows too large.

My game Breaking Bellman has a CartPole level that shows the difference. With 10 passes over each batch at step size 0.03, the plain update moved the policy by a KL divergence of about 1.5 per update, and at the end of training the pole stayed up for 158 steps on average. With clipping at the same settings, the KL was 0.008 per update and the average was about 395 steps, out of a 500-step limit (simulation, 3 seeds). The step size matters as much as the clip here: at step size 0.001, the plain update with 10 passes also stayed at a KL of 0.008 and averaged about 379 steps.

The clipped objective and the four cases in the drawing

With r(θ)=πθ(a∣s)/πθold(a∣s)r(\theta)=\pi_\theta(a\mid s)/\pi_{\theta_\text{old}}(a\mid s) and A^\hat A the advantage estimate from the batch, PPO maximizes

LCLIP=min⁡ ⁣(r(θ)A^,clip⁡(r(θ),1−ϵ,1+ϵ)A^).\begin{aligned} L^{\text{CLIP}}=\min\!\Big(&r(\theta)\hat A,\\ &\operatorname{clip}\big(r(\theta),1-\epsilon,1+\epsilon\big)\hat A\Big). \end{aligned}

With ϵ=0.2\epsilon=0.2 the clipped ratio stays between 0.8 and 1.2, and the min⁡\min keeps whichever term is lower. After a few steps on one batch:

A^\hat A Ratio rr rA^r\hat A Clipped LCLIPL^{\text{CLIP}} Gradient
+2 1.3 2.6 2.4 2.4 none: already raised enough
−2 0.7 −1.4 −1.6 −1.6 none: already lowered enough
−2 1.3 −2.6 −2.4 −2.6 pushes the ratio back down
+2 0.7 1.4 1.6 1.4 pushes the ratio back up

The paper describes clipping as removing the incentive for moving the ratio outside the interval, and it only ignores the change when that change would make the objective improve. The paper's entropy bonus is a separate term for exploration, not part of the clip. For early stopping on the KL, Spinning Up's PPO uses target_kl.

Same reward, no signal

A learned value V(s)V(s) is a second network to train, and if it's wrong, every advantage is wrong. Another option is to measure the usual result directly: put the robot in the same starting position, make several attempts, and compare each one with its group. Group Relative Policy Optimization (GRPO), introduced in DeepSeekMath (Shao et al., 2024) for language models, scores each attempt by how far its reward sits above or below its group's mean, measured in the group's standard deviations.

With reward 1 for a placed block and 0 for anything else, a group of 1, 0, 0, 0 gives the one success +1.73 and each failure −0.58. A group of 1, 1, 1, 1 gives every attempt an advantage of 0, and so does a group of four failures. That's four attempts and nothing learned about the task.

The GRPO advantage, and why an identical group stays empty

With GG samples from the same starting point and rewards r1,…,rGr_1,\dots,r_G:

A^i=ri−mean⁡(r1,…,rG)std⁡(r1,…,rG).\hat A_i=\frac{r_i-\operatorname{mean}(r_1,\dots,r_G)}{\operatorname{std}(r_1,\dots,r_G)}.

The numbers above use the population standard deviation. Implementations add a small number to the standard deviation to avoid dividing by zero, and zero divided by anything is still zero. GRPO keeps PPO's clipped objective and adds a separate penalty, β\beta times the KL divergence from a fixed reference policy, so with β>0\beta>0 the update isn't empty, it just doesn't say anything about the task.

The two cases mean different things. All successes on an easy start is mostly harmless, since there's little left to learn there. All failures on a hard start is the exploration problem from part 1 again: without at least one success to compare against, there's no direction to move. In the game's pass-or-fail goal grid, a group-mean baseline solved all of the tasks, and by the end of training 92.4% of the groups had identical rewards (3 seeds, simulation). Since every task was solved by then, those groups were most likely all successes.

Two published responses, and what each one costs
  • Filter the groups. DAPO (Yu et al., 2025) over-samples and drops prompts where every sample is right or every sample is wrong, then keeps sampling until the batch is full of groups with a signal. The cost is the extra samples, and the number needed changes from batch to batch. For a language model those are extra generations. For a robot, each one would be a real attempt with a reset. DAPO measured language-model generations, so that robot cost is untested. DAPO also removes GRPO's KL penalty, so the pull toward the reference doesn't apply to it.
  • Remove the normalization biases. Dr. GRPO (Liu et al., 2025) shows that dividing by the response length and by the group's standard deviation biases GRPO, and removes both. That fixes a different problem. An identical group still has every reward equal to its mean, so its advantages are still zero.
Group Mean Std Advantages
1, 1, 1, 1 1 0 0, 0, 0, 0
1, 0, 0, 0 0.25 0.43 +1.73, −0.58, −0.58, −0.58

Four fixes, four costs

Each fix repairs one part of the score, and none of them is free. Reward-to-go gets the timing right, but each score is still one noisy sample. A baseline adds the comparison, as long as you have a good value estimate or several attempts from the same start. The PPO clip makes batch reuse safer, but it's not a hard bound, so the step size, the number of passes and the KL still matter. Filtering identical groups brings the signal back at the price of extra attempts, a different number each batch.

The same summary as a table
Problem What goes wrong Fix What it costs or leaves open
Credit for the past Actions share blame for earlier rewards Reward-to-go Same average update, and each score is still one noisy sample
No comparison All actions in decent attempts go up Baseline, advantage Needs a good VV or several attempts
Overshooting on reused data Ratio grows with every pass PPO clip Not a hard bound, so learning rate, passes and KL still matter
Identical groups Every advantage is zero DAPO filtering Extra attempts, a different number each batch

Part 7 leaves the policy's own data behind and asks what can be learned from a fixed log of someone else's attempts. Part 7: learning from a fixed dataset.

Sources and what each one supports
  • REINFORCE and reward-to-go. Sutton and Barto, 2nd edition, Section 13.3: REINFORCE uses GtG_t, the return from time tt. The zero-average argument for past rewards is the same identity as the baseline proof. Spinning Up, "Don't let the past distract you" gives it in full.
  • Baselines. Sutton and Barto, Section 13.4, equations 13.10 and 13.11: any baseline that doesn't depend on the action leaves the expected update unchanged.
  • PPO. Schulman et al., 2017, Section 3 (without a constraint, maximizing the ratio objective leads to an excessively large update) and equation 7 (clipped objective, ϵ=0.2\epsilon=0.2); Section 5, equation 9 (entropy bonus as a separate term).
  • KL early stopping. Spinning Up PPO implementation, target_kl.
  • GRPO. Shao et al., 2024, Sections 4.1.1 and 4.1.2: group-normalized advantages and a KL penalty added to the loss.
  • DAPO. Yu et al., 2025, Section 2.3 (KL penalty removed) and Section 3.2 (dynamic sampling).
  • Dr. GRPO. Liu et al., 2025, Section 3.1: length and standard-deviation normalization biases.
  • Breaking Bellman. CartPole reuse level and pass-or-fail goal grid from the game's experiment log at commit 681cae3. The CartPole numbers are the runs st-plain-e10-lr03, st-clip-e10-lr03 and st-plain-e10-lr001 in its web/measured.js: mean final evaluation return and mean KL per update over 3 seeds. The 92.4% is the share of groups with identical rewards in the run gg-ppo-grpo. Simulation, not a robot.
  • Block example. The rewards −2, 0 and +10 and the four attempts are made up to show the arithmetic. The robot reset cost of DAPO filtering is an untested consequence, not a measured result.

← All writing