The deadly triad: how a guess can train itself upward

How can value estimates grow forever when every reward is zero?

Part 5. Three ideas make RL practical on big problems. Function approximation, or shared weights, lets one set of weights cover states you've never seen, bootstrapping lets you learn from a guess before an episode ends, and off-policy learning lets you reuse old data. Every method on the first board uses the first idea, but only the deep Q-network (DQN) and soft actor-critic (SAC) use all three, and they stand for a family of off-policy value learners that includes TD3. Shared weights and bootstrapping let a prediction feed its own target, and off-policy data can remove the correction that would pull it back down. Sutton and Barto call the combination the deadly triad: it names a risk, not a rule that every combination diverges.

Who keeps all three?

The board checks six methods against their papers. Only DQN and SAC have the whole triad: they replay old data with targets from their own networks, and both slow those targets down. PPO bootstraps on a fresh batch that it reuses for only a few epochs. GRPO and REINFORCE score whole answers or episodes on fresh data and never train a value on their own guesses. RECAP, in Physical Intelligence's π*0.6, reuses old robot data, but it trains its value on complete returns. Its authors call that value less optimal than an off-policy Q-function, but "simple and highly reliable".

The six methods: what each paper does

The table follows the variant in each primary paper. All six share weights: a Q-network in DQN, neural networks in SAC and PPO, a language model in GRPO and RLOO, and a vision-language-action model (VLA) with a separate value network in RECAP. Off-policy here has the meaning from Sutton and Barto, §11.3: training on transitions from a different distribution than the one the target policy produces. Bootstrap here means that a learned value trains toward its own estimates.

Method	Bootstrap	Off-policy
DQN, 2015	yes, from a frozen copy	yes, replay
SAC, 2018	yes, from a slow copy	yes, replay
PPO, 2017	yes, the critic fits λ-returns ( $\lambda=0.95$ )	almost no: a fresh batch, reused for a few epochs
GRPO, 2024	no value model	no, fresh samples
REINFORCE, 1992, and RLOO, 2019 (for language models, 2024)	no, complete returns	no, fresh episodes
RECAP in π*0.6, 2025	no for the value, which fits complete returns; the policy's advantage adds that value 50 steps ahead	yes, for the policy

DQN. Its paper describes the same loop as the next panel: an update that raises $Q(s_t,a_t)$ often raises the next state's values too, and so the target, "possibly leading to oscillations or divergence of the policy". The frozen copy is its fix, and replay is why it must learn off-policy (Methods).
SAC. Its introduction names the risk: "the combination of off-policy learning and high-dimensional, nonlinear function approximation with neural networks presents a major challenge for stability and convergence". It uses a slowly moving target network and the smaller of two Q-functions (Section 4.2).
PPO. The advantage bootstraps: $\delta_t=r_t+\gamma V(s_{t+1})-V(s_t)$ in the paper's Eq. 12. The paper doesn't say which target the critic fits; the OpenAI baselines code fits it to λ-returns, which contain the critic's own estimates. Each batch gets 10, 15 or 3 epochs in the paper's MuJoCo, Roboschool and Atari settings, and the clipped ratio removes the gain from moving far from the policy that collected it. The critic learns the value of that collecting policy.
GRPO. Algorithm 1 allows several updates on one batch, and then the later updates use slightly stale data. The DeepSeekMath run made "a single update following each exploration stage" (Section 4.2), and a learned reward model scored each answer (Algorithm 1). The paper drops the value model to save memory, and because a per-token value is hard to train when usually only the last token gets a reward. It doesn't mention the triad.
REINFORCE and RLOO. Williams (1992) introduced REINFORCE. It waits for the complete return, so Sutton and Barto call it "a Monte Carlo algorithm" (§13.3). RLOO, a REINFORCE variant from Kool et al. (2019) that Ahmadian et al. (2024) apply to language models, keeps that and compares several samples for the same prompt.
RECAP. Its value predicts the return of the policies behind the data (Section IV-A, Eq. 1), so the value learner itself is on-policy for that data mix. The policy then improves on the same old data by conditioning on the advantage, which adds the learned value 50 steps ahead (Appendix F). That is the step Part 7 covers.

Three pressures, one loop

Read the loop on the board clockwise. Raising one prediction also raises a successor prediction that shares its weights. That successor supplies the next target, so the next update raises the first prediction again. If no update ever trains the successor, nothing pulls it back down. Van Hasselt et al., §2.

Two states, one weight

Take two states where every reward is zero, so both true values are zero. One weight $w$ and fixed features $\phi(s_1)=1$ and $\phi(s_2)=2$ give the estimates $v_w(s_1)=w$ and $v_w(s_2)=2w$ . The policy moves from $s_1$ to $s_2$ and then stays at $s_2$ .

Now replay only the transition $s_1\to s_2$ , forever. The semi-gradient TD update treats the target as fixed and only moves the current prediction. With temporal-difference error $\delta$ and step size $\alpha$ :

\begin{aligned} \delta&=0+\gamma(2w)-w=(2\gamma-1)w,\\ w^+&=w+\alpha\delta\phi(s_1)=[1+\alpha(2\gamma-1)]w. \end{aligned}

With $\gamma=0.9$ and $\alpha=0.1$ , the multiplier is $1.08$ . Starting from $w_0=1$ , the weight after $n$ replays is exactly $w_n=1.08^n$ :

Replay update $n$	Weight $w_n$	Estimate at $s_1$	Estimate at $s_2$
0	1	1	2
1	1.08	1.08	2.16
2	1.1664	1.1664	2.3328

No reward ever arrives. The growth comes only from the shared weight feeding its own target. The mismatch is in which states get updated: the policy ends up at $s_2$ , but replay never starts there. So this example breaks state coverage, and it isn't a counterexample to the stable pairs below. The example goes back to Tsitsiklis and Van Roy (1996, 1997), as presented in van Hasselt et al., §2.

Notation used in this article

$S_t$ and $A_t$ are the state and action at step $t$ , and $R_{t+1}$ is the reward after that action. The discount is $0\leq\gamma<1$ , and $\alpha_t>0$ is a learning rate. A superscript $+$ marks an updated quantity. Here $t$ is a step index, not an exponent.

Remove any one ingredient and there is a setting where learning is stable. Each of the next three panels keeps two ingredients and shows what it restores. Each result needs its own assumptions; none of them says a pair is always safe.

Drop off-policy data

Keep the shared weights and bootstrapping, but sample states from the policy's own trajectories. Linear TD then settles on one projected solution. The projected Bellman step can't increase the distance between two candidate predictions when both are measured with the same stationary state weighting: with $\gamma=0.9$ , two predictions 2 apart end up at most 1.8 apart. Off-policy sampling changes that weighting, and the argument no longer holds. Tsitsiklis and Van Roy. PPO's critic is close to this case, since it fits fresh data from the policy that collected it, but a neural network gets no guarantee from this linear result.

Drop bootstrapping

Keep the shared weights and off-policy data, but wait for complete returns, correct them with importance weights and freeze them as labels. The labels no longer depend on the predictions, so fitting them is ordinary regression. With one feature equal to 1, a label of 2 and a step size of 0.5, the weight goes $0\to1\to1.5\to1.75$ and approaches 2. Bootstrapping would let the label move with the weight. Sutton and Barto, §§5.5, 9.2. RECAP's value learner also fits complete returns as fixed labels. It needs no importance weights, because it learns the value of the policies that collected the data.

Drop shared weights

Keep bootstrapping and off-policy data, but give every state-action pair its own table entry. With reward 0, a next-row maximum of 5 and $\gamma=0.9$ , the target is 4.5, and a step size of 0.5 moves an old entry of 2 to 3.25. The successor's 5 doesn't move during that update, so the target can't chase the prediction. Tabular Q-learning converges under the conditions of Watkins and Dayan (1992). The derivations and conditions for all three pairs are in the drawer at the end.

Slow the target feedback

DQN keeps all three ingredients, so it slows the loop instead of removing it. It trains on transitions sampled from a replay buffer, which breaks up runs of nearly identical states, and it computes targets with a frozen copy of the network that it refreshes every $C$ updates. Replay doesn't make the data on-policy, and a frozen target delays the feedback without guaranteeing convergence. SAC also slows its targets, with a moving average of the weights. Mnih et al., Methods; van Hasselt et al..

The DQN target and training loop

Let $Q_\theta$ be the online action-value network with weights $\theta$ , and $Q_{\theta^-}$ its frozen copy. Let $d$ be one for a terminal transition and zero otherwise, $y$ the target, and $\nabla_\theta$ the gradient with respect to the online weights. Here $s,a,r,s'$ are the current state, action, observed reward and successor, $a'$ ranges over next actions, and $\theta^+$ is the updated online network. The superscript minus labels the frozen copy, not subtraction.

\begin{aligned} y&=r+\gamma(1-d)\max_{a'}Q_{\theta^-}(s',a'),\\ \theta^+&=\theta+\alpha_t[y-Q_\theta(s,a)]\nabla_\theta Q_\theta(s,a). \end{aligned}

The terminal mask removes the nonexistent future after an episode ends. This is a semi-gradient update: gradients flow through the current prediction while $y$ stays fixed. With $C$ the number of updates between target copies:

store each observed transition in replay
sample a transition (s, a, r, s′, d)
compute y using frozen weights theta_minus
update online weights theta toward y
if C updates have passed: theta_minus = theta

The published DQN samples minibatches of 32, uses RMSProp and clips the error to [−1, 1]; this loop keeps only the target logic. Freezing delays the target feedback. It doesn't prove convergence, even for the two-state example above.

Check estimates beside rewards

A good task score tells you how the policy acts, not whether its values are right, so check the values against what is possible. If every reward has magnitude at most $R_{\max}$ , the discounted return $G_t$ can't be larger than a geometric sum:

|G_t|\leq\frac{R_{\max}}{1-\gamma}.

With rewards bounded by 1 and $\gamma=0.9$ , the bound is 10, so a prediction of 100 is wrong for that task. Check the reward scale and terminal handling first, then sampling and target feedback. A finite spike proves an error; divergence to infinity is a stronger claim.

Coverage: which states and actions do the updates skip?
Coupling: which other predictions move after one update?
Targets: when do labels change, and where do gradients stop?

The three pairs: detailed stability arguments and conditions

Each pair below has a particular stable construction under stated assumptions. They explain why removing one source of feedback can help. The deadly triad names a risk when all three ingredients are present; it is not a rule that every pair is safe or every triple diverges.

Shared predictions, guessed targets and off-policy data can reinforce an error. Each pair has a different stability argument.

In the equations below, $t$ is the time step; $S_t$ and $A_t$ are the state and action; $R_{t+1}$ is the next reward. The target policy $\pi(a\mid s)$ gives the probability of action $a$ in state $s$ ; $b(a\mid s)$ is the policy that collected the data. The discount is $0\leq\gamma<1$ and $\alpha_t>0$ is a learning step size. A superscript $+$ means the next parameter or table value.

1 · Approximation + bootstrapping

Learn a fixed policy's values with on-policy linear TD. A fixed feature vector $\phi(s)$ describes state $s$ ; the weights $w$ produce $v_w(s)=\phi(s)^\top w$ . The superscript $\top$ means transpose. Let $\delta_t$ be the TD error, and let $w^+$ denote the updated weights. One observed transition gives this update:

\begin{aligned} \delta_t &= R_{t+1}+\gamma v_w(S_{t+1})-v_w(S_t),\\ w^+ &= w+\alpha_t\delta_t\phi(S_t). \end{aligned}

Why can this settle down? For a candidate value vector $v$ , let $T^\pi$ be the Bellman prediction operator:

(T^\pi v)(s)=\mathbb E_\pi[ R_{t+1}+\gamma v(S_{t+1})\mid S_t=s].

The expectation uses the fixed target policy and the environment's transition probabilities. Let $D$ put its stationary state probabilities $d(s)$ on the diagonal; $\|x\|_D^2=\sum_s d(s)x(s)^2$ weights a value vector $x$ by those probabilities. Let $\Pi$ be least-squares projection onto the space spanned by the fixed features, using those same weights. For value vectors $v$ and $u$ :

\|\Pi T^\pi v-\Pi T^\pi u\|_D \leq\gamma\|v-u\|_D.

On-policy averaging cannot enlarge this norm; neither can the orthogonal projection. Multiplying by $\gamma<1$ makes the combined map a contraction, with one projected fixed point $v=\Pi T^\pi v$ . TD's sampled weight update is not literally repeated projection. Under the conditions below, it converges to that same projected solution. Tsitsiklis and Van Roy, Sections IV and VII.

Add off-policy sampling: projection may now weight states by the behavior distribution, while the Bellman target follows $\pi$ . The matched-norm argument can fail. The two-state example earlier in this article shows the resulting feedback.

Read the bound: if two candidate value vectors are distance 2 apart and $\gamma=0.9$ , one projected Bellman map leaves them at most 1.8 apart. It compares two whole predictions under the same policy and state weighting. It does not say every noisy TD sample improves its estimate.

Conditions behind this prediction result

This is a fixed-policy prediction result. A finite irreducible, aperiodic chain supplies a stationary distribution and repeated on-policy visits. Rewards are bounded, the discount is below 1, and the fixed feature columns are linearly independent.

Positive diminishing learning rates have infinite sum and finite squared sum: they retain enough total adjustment while suppressing persistent sampling noise. These assumptions support the linear result cited above; they do not cover arbitrary neural networks or a changing control policy.

2 · Approximation + off-policy

Wait for the complete return, then fit fixed targets. Let $H$ be the terminal time, $k$ index future steps, $G_t$ be the full discounted return and $\rho_t$ be the trajectory weight correcting behavior $b$ to target policy $\pi$ . The product $\prod$ multiplies the action-probability ratios along that recorded future. Its return and importance weight are

\begin{aligned} G_t &= \sum_{k=0}^{H-t-1}\gamma^k R_{t+k+1},\\ \rho_t &= \prod_{k=t}^{H-1} \frac{\pi(A_k\mid S_k)}{b(A_k\mid S_k)}. \end{aligned}

Crucially, $G_t$ contains rewards, not a guessed future value. Freeze the recorded data. For example $i$ , use its corrected return $y_i=\rho_i G_i$ as the label; here $G_i$ and $\rho_i$ mean the return and weight for that recorded starting point. Let $y$ stack these labels, and let row $i$ of the feature matrix $X$ be $\phi(s_i)^\top$ , with $s_i$ the example's starting state. Let $\eta>0$ be the batch step size; $X^\top$ is the transposed feature matrix. Batch gradient descent is

w^+=w-\eta X^\top(Xw-y).

The Euclidean norm $\|\cdot\|_2$ measures vector length. The fixed squared-error loss is $L(w)=\tfrac12\|Xw-y\|_2^2$ . Let $w_*$ be its empirical minimizer, $\succ0$ mean positive definite, and $^{-1}$ mean matrix inverse. If $X$ has full column rank,

X^\top X\succ0 \quad\Longrightarrow\quad w_*=(X^\top X)^{-1}X^\top y.

Full rank gives one empirical least-squares minimizer. Choose $\eta$ below 2 divided by the largest eigenvalue of $X^\top X$ : then the error in each curvature direction shrinks. The labels stay fixed while the weights move. Rank-deficient data need not have a unique fit. This is a well-posed regression argument, not a blanket convergence claim for off-policy Monte Carlo. Sutton and Barto, published second edition, §§5.5 and 9.2.

Add bootstrapping: replace the complete return with a target containing $v_w$ at a future state. Now the labels move with $w$ . Treating them as constants during each update no longer gives gradient descent on the fixed regression above.

One fixed label: take one feature equal to 1 and a corrected return label of 2. The loss is $(w-2)^2/2$ . With batch step size $\eta=0.5$ , weights $0,1,1.5,1.75$ approach 2 because the label stays 2. Bootstrapping would let that destination move with the predictions.

Conditions behind this regression example

Freeze the target and behavior policies, linear features and recorded data. Use independent finite-horizon episodes with bounded rewards and features. Behavior must cover the target actions and the starting states whose values are being fitted; corrected labels need finite variance for statistical claims.

Full column rank gives a unique empirical fit. The batch step-size bound stated above makes optimization approach it. Finite data need not recover the true target-policy value; large importance weights can make that statistical estimate poor.

3 · Bootstrapping + off-policy

Keep a separate table entry for every state-action pair. Let $Q$ be the action-value table. In a transition $(s,a,r,s')$ , $s$ and $a$ are the state and action, $r$ the reward and $s'$ the next state; $a'$ ranges over next actions. Tabular Q-learning updates just one entry:

\begin{aligned} Q^+(s,a)=Q(s,a)+\alpha_t\big[&r+\gamma\max_{a'}Q(s',a')\\ &-Q(s,a)\big]. \end{aligned}

All other entries stay unchanged. For any candidate table $q$ , its expected target is the Bellman optimality operator $T_*$ :

(T_*q)(s,a)=\mathbb E\!\left[ r+\gamma\max_{a'}q(s',a')\mid s,a\right].

The expectation averages the environment's rewards and transitions. The max norm $\|x\|_\infty=\max_{s,a}|x(s,a)|$ is the largest absolute entry of a table $x$ . For two candidate tables $q$ and $u$ ,

\|T_*q-T_*u\|_\infty \leq\gamma\|q-u\|_\infty.

Taking a maximum and averaging cannot amplify the largest entrywise difference; discounting shrinks it. Thus $T_*$ has one fixed point, the optimal table. With enough visits and suitable step sizes, sampled Q-learning reaches it. Watkins and Dayan's convergence theorem.

Add function approximation: a weight update can also change other states' values, including a value in the bootstrap target. The tabular Bellman operator still contracts, but that alone does not control the added shared fitting step. Its guarantee does not transfer to an arbitrary approximator.

Two separate entries: if the successor maximum is 5, the reward is 0 and $\gamma=0.9$ , the target is 4.5. Moving a predecessor from 2 toward 4.5 leaves the successor at 5 during that update. A shared weight can move both; a table entry cannot.

Conditions behind tabular convergence

The task is a finite Markov decision process with fixed transition and reward laws, bounded rewards and discount below 1. Every nonterminal state-action pair must receive infinitely many updates.

Each pair's learning rates are in [0, 1), have infinite sum and finite squared sum. A replay buffer omitting a pair fails the visit assumption; a constant noisy step size fails the shrinking-step assumption. The theorem cited above concerns the resulting infinite update sequence, not a finite successful route.

Across the series, we've used values to connect late rewards to earlier choices. The triad matters because sharing that credit can also share an error, which then becomes the next target. Return to the method tree, or the opening problems.

The next three parts look at failures outside the triad: policy-gradient credit and update size, learning from a fixed dataset, and mode averaging. The failure map groups them all.

← All writing