I'm learning reinforcement learning because it seems key to understanding AI and robotics right now. It's part of how today's language models get better, and part of how robots learn from their own attempts.
I learn it in a few ways. I made a small game, Breaking Bellman, where you watch an RL algorithm fail, work out why, and fix it by changing one word of its update rule. I contribute to open-source robot learning libraries such as Hugging Face's LeRobot, and you can find that work on my GitHub. I also write blog articles like this series as I go.
This series maps out the key problems that drive the algorithms. Most methods in this series exist because an earlier one broke on something: a reward that arrives late, a lucky first result, too many situations to keep in a table. Follow those problems and you also get a genealogy of the algorithms: which one grew out of which, and why.
What I'm really after is taste. To me, RL is more of an art than a recipe: the methods are brittle, small choices decide whether training works at all, and yet the same tools are behind big gains in language models and in robotics. Taste is the judgment to look at a problem and know which tool fits it.
The six problems on the canvas come back throughout the series. Part 5 reaches the deadly triad, where three of the fixes (learning from guesses, sharing what you learn across situations, and reusing old data) combine into a loop where an error feeds the next prediction.
Part 0. Next: a value-based family tree and a Q table.
In reinforcement learning, we have an agent that takes actions, observes what happens and receives rewards. Its policy is the rule it uses to choose actions. We want the agent to earn more reward over time, but it has to learn which choices will get it there.
Delayed reward: which move earned it?
Suppose you win a game after a hundred moves. You get a final reward, so you know the game went well. But which moves actually contributed to the win? Maybe the last move was obvious, while a choice much earlier made the winning route possible.
Connecting that outcome to the decisions that contributed to it is called credit assignment. The same problem comes up when a robot reaches for an object. Even if it only knows whether it succeeded at the end, it needs to learn about both the approach and the grasp.
We will estimate future reward and use those estimates to carry information back to earlier choices. In Part 1, we can see those estimates in a table.
Exploration: which choice have we missed?
Imagine you have ten slot machines, and you don’t know their average payouts. You try one and get a good payout. Do you keep playing that machine, or spend some turns trying the others?
If you repeat the choice that looks best so far, that is exploitation. If you try a less familiar choice, that is exploration. The problem is that rewards can be noisy. A lucky first payout might keep you playing a mediocre machine. Exploring has a cost now, but what you learn can help you make better decisions later.
We compare these choices in Part 1’s bandit example, then add states and a longer future to consider.
Waiting: can we learn before the end?
We could wait until each game ends before learning from it. With a long episode, though, we would be waiting a long time. And that one outcome can depend on many chance events.
With bootstrapping, we combine a reward we have observed with an estimate of what comes next, so we can learn before the final result arrives. Information can travel sooner, but an error in that estimate can also become part of what we learn from next. Part 2 compares waiting, guessing and looking several steps ahead.
Scale: can one lesson help elsewhere?
In a small grid, we have a manageable number of cells. But with camera images or a robot’s joint positions, there are far more situations than we can visit one by one. If we had to learn a separate answer for each situation, almost everything would remain unknown.
With function approximation, we use a model, such as a neural network, to share learning across situations. Learning from one situation can improve a prediction somewhere else, but it can also change a prediction that was already good. In Part 3, we follow both effects through one shared weight.
Reusing data: whose choices does the data show?
A robot might have stored experience collected by an older controller. Or an agent might be taking random actions to explore, while trying to learn how a more decisive policy would behave.
So the policy collecting the data can be different from the policy we want to evaluate. When we use data from one policy to learn about another, that is off-policy learning. With on-policy learning, we evaluate the policy that produces the experience. Reusing data is useful, but we still need enough information about the choices we want to evaluate. If an action was never tried, it can remain unknown. Part 4 shows this difference at the edge of a cliff.
Feedback: what if a guess changes its target?
Now consider what happens when we combine the three ideas. With bootstrapping, the target we learn toward contains a prediction. With function approximation, one update can change several predictions. Off-policy data can leave some of those predictions without enough correction. So an update can enlarge an error, and the next target can carry that error forward.
We call this combination the deadly triad. It can make learning unstable, although it does not cause every learner using these three ideas to fail. In Part 5, we build the feedback loop using two states and one weight, then look at ways to interrupt it.
How learning goes wrong
Each of these ideas fixes one problem and opens the door to others. Think of the six problems as causes. What you actually see when training goes wrong is a symptom, like a learner that never finds the better slot machine, or a policy that copies a bad demonstration. The failures in this series come down to five symptoms, and the list below sorts them, with the problems behind each one and the part that explains it.
Five symptoms: each failure, the problems behind it, the part that explains it and the methods that address it
Too little signal. The learner can't improve because nothing in its experience points to a better choice. Behind it: exploration, delayed reward and waiting.
- A lucky first result keeps a greedy learner on a worse choice. Random exploration: Part 1's bandit. Optimistic starting values and upper confidence bounds: Part 1, "Two other ways to explore".
- A late reward takes many updates to reach early choices. Multi-step returns, which trade error from the estimate for noise from later outcomes: Part 1's corridor and Part 2.
- A group of attempts that all score the same gives every attempt zero credit. Filtering those groups, at the cost of extra attempts: Part 6.
Credit in the wrong place. The learner gets a signal, but it rewards or blames the wrong action. Behind it: delayed reward.
- An action shares blame for rewards that came before it. Reward-to-go: Part 6.
- Every action in a decent attempt gets pushed up. A baseline, and advantages estimated with GAE: Part 6 and Part 2.
Updates that overshoot. One change goes further than the evidence supports. Behind it: scale and reusing data.
- Reusing one batch for many steps moves the policy too far. PPO's clipped objective, which is not a hard limit: Part 6.
- One update to shared weights changes predictions that were already right. Part 3.
Value estimates that answer the wrong question. The target looks reasonable but measures something other than what we want. Behind it: waiting, reusing data and feedback.
- A time limit is treated as the end of the task, or the learner can't see how much time is left. Time-aware inputs or partial-episode bootstrapping, depending on what the limit means: Part 2.
- A target asks about actions the data never tried, so nothing corrects the answer. Staying close to the data with BCQ, TD3+BC, CQL or IQL: Part 7.
- A prediction feeds its own error through bootstrapping, shared weights and off-policy data. Part 5.
Copying the wrong thing. A policy learned from demonstrations does something the demonstrations shouldn't have taught it. Behind it: reusing data, here a log of demonstrations.
- The log includes bad attempts, so copying it copies the mistakes. Reward-aware reuse with AWR, IQL, advantage conditioning or TD3+BC, when the rewards and values are good enough: Part 7.
- The log itself is wrong, with misaligned timestamps or false success labels. Fixing the data, not the learner: Part 7.
- Two good ways to do the task get averaged into one bad one. Policies that can represent both, such as flow policies: Part 8. Representing both routes still doesn't guarantee that a sample is safe or that every route is covered: Part 8's limits.
The route through the series
- A value-based family tree and a Q table: connect one branch of methods to their problems, then watch one table learn.
- Bootstrapping: choose how much observed reward to use before a guess. This also leads to Generalized Advantage Estimation (GAE), which mixes return lengths to assign action credit.
- Function approximation: see what changes when predictions share weights.
- On-policy and off-policy learning: separate the policy collecting data from the one being evaluated.
- The deadly triad: bring the three mechanisms together and examine stability.
- Policy gradients: give each action fair credit and keep each update small.
- Learning from a fixed dataset: actions the data never tried, and demonstrations that include mistakes.
- Mode averaging: two good answers, one bad average, and what flow policies change.
One loop sits under most of these methods: judge the current policy, then act greedily on that judgment, and repeat. A side article, generalized policy iteration, works it through on the Part 1 cliff grid and follows it to AlphaGo Zero.
We start with these questions. Next, we will look at the tools people developed to answer them.
Sources
The progression draws on Sutton and Barto, Reinforcement Learning: An Introduction, second edition: exploration in Chapter 2, returns and policies in Chapter 3, temporal-difference learning in Chapter 6, approximation in Chapter 9 and the deadly triad in Section 11.3. Van Hasselt and colleagues examine the interaction between the triad’s ingredients. The measured slot-machine comparisons and their settings are in Part 1.