Why can't robots remember?

The classic robot policy is Leonard from Memento.

Leonard can't form new memories. Everything he knows about the last hour sits in Polaroids, notes and tattoos, and he reads them to find out what he is doing. A policy that decides from its last frame or two is in a similar spot, and many of the systems below live this way.

His workarounds look a lot like what robots try. A Polaroid with a caption is a text note. A photo he keeps because it matters is a key frame. The film also shows the catch: a note can be wrong or out of date. In a robot that is a stale entry.

Coding agents have no memory either

Coding agents have the same gap. They don't learn on the job. Each session starts from the model's weights plus whatever notes and files you put in front of it. It is like a coworker who gets handed a huge textbook before every shift and remembers nothing outside of it.

I don't think that is where agents end up. A coworker who learns your codebase, your customers and your habits gets better every week. Maybe the end state is a model fine-tuned on the company.

OpenAI is moving the other way on its self-serve fine-tuning platform. Its deprecations page says that since 7 May 2026 organizations that never fine-tuned a model can't start there. From 6 January 2027, active customers can't create new jobs on it. A model that is already tuned keeps serving until its base model is retired.

The page gives no reason. My guess is batching. A provider can serve requests from every customer on one shared model at once, which is far more efficient than hosting a separate model for each company.

Robots have it worse

Most robots don't even have a memory file. A robot is controlled by a policy, a network that turns the current camera frames and an instruction into the next action. Many standard policies use only the current frame or a short fixed history, with no persistent episode memory. There is no textbook to reread.

Some systems do better. RT-1 (Brohan et al., 2022) reads six frames. ViNT (Shah et al., 2023) and NoMaD (Sridhar et al., 2023) build a graph of places they have seen. MemER (Sridhar et al., 2025) and the systems in RoboMME (Dai et al., 2026) keep key frames or text notes. They are the exceptions, and the rest of this article looks at what they store and what it costs.

The hall problem

Say you ask a robot to walk down a hall, take the second door on the left, pick up the blue box and bring it back to where it started. It does the first part. Now it holds the box. The first door is far behind, the starting point is out of sight, and the camera shows a box and a wall.

Which way is home? Nothing in that frame says. The answer lives in something the robot saw a minute ago, so the real question is where a robot keeps it.

In October 2026 there is no agreed answer. People are trying frames, text notes, vectors, a few saved key frames, maps and libraries of code. The papers disagree on which one wins, and the biggest benchmark I found for this says the winner changes with the task. That is why I wanted to write down where things stand today.

Four things people call memory

The word covers very different things, and mixing them causes most of the confusion:

• Weights. What pretraining baked in. A robot that knows what a mug is has a kind of memory, and it never changes during a task.
• Recent history. The last few frames, or a recurrent state. It lasts seconds.
• Within-task memory. Key frames, a map, a log of what happened so far. It lasts minutes and is thrown away when the task ends.
• Across-task memory. Skills and notes that survive the episode and help the next task. It lasts as long as someone keeps the file.

Most of this article is about the third kind, because that is where the hall example breaks. The fourth shows up at the end with ASPIRE (Lu et al., 2026).

Many policies run on one to six frames

Short memory is still memory, and it works better than you might expect. This is how many frames some well-known policies read at each step:

History helps where it is used well. RT-1 loses accuracy when its six-frame history is removed: 97 to 82 percent on seen tasks and 76 to 62 percent on unseen ones (its Table 13, real robot). Each frame costs RT-1 only 8 tokens, thanks to a compression layer, so six frames stay cheap.

The five-minute hall is another matter. The RoboPapers episode on MemER puts it bluntly: most robot policies today largely lack memory and decide from what they can see right now. That is the hosts' summary, and it fits the table above.

When one frame is not enough

Each of these compares the same system with and without memory, in the paper's own setup.

• SAM2Act (Fang et al., 2025), version 4, three memory tasks in simulation: 55.0 percent without its memory bank and 94.3 percent with it. The earlier version printed 53.7 and 91.3 before the authors fixed one task, so the version matters.
• RoboMME (Dai et al., 2026), 16 tasks in simulation: a plain π0.5 (Physical Intelligence, 2025) policy averages 17.93 percent. The best design the authors tried averages 44.51. Humans score 90.5.
• MemER (Sridhar et al., 2025), real robot, searching for 3 objects over 20 trials per task. The chart shows how many objects it found out of 60.

Why not just add more frames?

Two separate reasons keep coming up, and they are different problems.

Cost

Every frame becomes tokens, and the model attends over all of them. A good approximation from the appendix of CronusVLA (Li et al., 2025):

$$N = (M + 1)\,P + I$$

Here $M$ is the number of past frames, $P$ is the tokens per frame and $I$ is the instruction length. Attention cost grows roughly with $N^2$. The HAMLET paper (Koo et al., 2025) measured it for a robot policy (GR00T N1.5 on RoboCasa, one A100):

Navigation models face the same bill, so they spend tokens unevenly. NavFoM (Zhang et al., 2025) gives the newest frame 64 tokens and each older frame 4, under a fixed budget. When the budget runs out it drops the oldest frames.

Training costs too. Mistral's Robostral Navigate (Bounhar et al., 2026) trains on whole episodes packed into a single sequence with a special attention mask. The paper reports 22 times fewer training tokens, turning months into days.

Copycat behavior

The second problem has a name. Imagine an expert whose joint command is 0.10, 0.12, 0.14, 0.16, and so on. A policy that sees two frames can work out the last command, and copying it is wrong by only 0.02 each step. The training loss looks great. Then the cup moves, and the policy keeps copying.

Wen and colleagues (2020) measured this on six simulated control tasks: the cloned policy's actions were easier to predict from its own past actions than the expert's were, in all six. De Haan and colleagues (2019) showed that handing a cloned policy its previous action can lower its validation error and its reward at the same time (Pong, Hopper and others). Chuang and colleagues (2022) trace the same effect to how information flows through the network, and propose an architecture that avoids it by design.

Nobody should read this as "history always hurts". The same Wen paper finds two frames beat one in five of six partially observed tasks. The picture is mixed:

So the cause depends on the model. The authors of ContextVLA read their result as the pretrained VLM start doing the work, but that is their interpretation of an ablation, not a separate test. A 2026 audit by Liao and Cao looked inside three frozen VLAs. Past-frame content stays decodable, but information unique to the history is nearly absent, and the models use history mainly when the current frame is heavily degraded.

Six ways to remember, and what each one stores

Each row comes from a different paper, with its own benchmark, robot, trial count and baselines. RoboMME's numbers for MemER, SAM2Act+ and MemoryVLA are the RoboMME authors' own re-implementations.

For one latent-vector design up close, see my Imprint explainer (Wu et al., 2026). Instead of picking what to keep, it compresses video, actions and joint states into 256 memory tokens trained to rebuild the past, and it takes π0.5 from 25.4 to 72.8 percent success on RMBench's nine memory tasks.

Drawing the problem

Take the hall again. The policy reads a short window directly. Everything older has to go through one of the routes below, and each route fails in its own way.

The two models I started from

I first wrote "Robostroll" for a navigation model I remembered as seeing only a few frames. The model is Mistral's Robostral Navigate (Bounhar et al., 2026).

Its paper says the model reads "a history of monocular RGB frames". It never says how long that history is. My memory of "a few frames" is not backed by the paper, and the training recipe suggests long histories are the point: each step is trained to see all the frames before it. On R2R-CE (unseen validation, a single RGB camera, no depth) the paper reports 77.4 percent success. Mistral's launch post reports 76.6 percent on the same split, probably from an earlier checkpoint. That is my inference.

The other one is SOLE-R1 (Schroeder et al., 2026), and the memory question is sharper there. It is not a policy but a reward model that watches video and scores progress toward a goal. It reads the goal, the first frame, a window of the latest frames and its own previous progress number. In the RoboPapers interview, first author Philip Schroeder says it reads the first, previous and current time steps from two cameras, so six frames, and that the reasoning text from one step is forgotten at the next. He suggests compressed text could serve as memory. That is an idea from the interview, not a result in the paper.

Here is the first-frame trick in pseudocode, my paraphrase of the paper's input definition:

at each step t:
  input = [goal, first_frame, last K frames, progress_at(t-1)]
  reasoning, progress_at(t) = model(input)
  keep only progress_at(t)    # the reasoning text is dropped

A key frame memory in eight lines

MemER shows how small a working memory can be. This is my paraphrase of its method:

memory = []                          # at most 8 key frames
every step:
  look = last N frames + memory
  subtask, nominated = high_level_vlm(look)   # text for the low-level policy
  group nominated frames: gaps of 5 frames or less form one cluster
  for each cluster:
    add its median frame to memory
  low_level_policy(subtask) -> actions

The high-level model runs at about 1 Hz and writes a text subtask. A separate low-level π0.5 policy turns it into motion. Memory only grows. Tasks that need hours of memory could overflow it, and the authors say so.

Which design wins depends on the task

RoboMME is the closest thing to a common test. It has 16 tasks in four groups (counting, permanence, reference and imitation) and 14 memory variants on one π0.5 backbone. The authors report that no single memory design wins everywhere. The chart compares two of their designs on five tasks: text subgoals from a helper model against sampled visual tokens used as a modulator.

Reading it: a text note like "two cubes in" is enough to count cubes, and useless for how many times a cube has swung left and right. Counting the swings needs the motion itself. The authors also report 25 of 40 real-robot trials for the visual design and 19 of 40 for the text design on four tasks, against 4 of 40 for no memory, from their Table 4. The tasks are just 4, with 10 trials each.

The library that remembers across tasks

ASPIRE (paper, Lu et al., 2026) is a different kind of memory. There is no learned policy. A coding agent writes Python for each task, and each verified fix is saved as a text skill: a failure signature, a trigger and a repair. A later task reads the matching notes.

That is across-task, procedural memory. It remembers how to recover from a planner error near a table edge. It does not remember where the box is in this room. The paper does not describe a map or an episodic store for the robot at run time, and whether the generated programs keep state between steps is not stated. Its authors list stale and redundant entries as open problems.

More work worth knowing

A literature sweep of 148 robot-memory papers (titles, authors and years checked against arXiv and Crossref) shows how fast this field moves: 88 of them were posted in 2026. These 22 matter most for the argument above. Each description in the last column comes from the paper's own abstract.

Two of them change the picture above:

• The RoboMME top score moved. SimpleARM (Yuyou Zhang and colleagues, 2026) reports 67.17 percent mean success on the 16 RoboMME tasks. The strongest non-oracle baseline in its comparison scores 44.51.
• Having memory is not the same as using it. Three 2026 audits ask whether history changes what a policy does. Liao and Cao found little history-unique information inside three frozen VLAs. Jiajie Zhang and colleagues found actions that change with the past but do not reliably pick the behavior the past calls for. Luo and colleagues found, in cross-episode tests, that swapping key facts in a learned memory bank had no clear extra effect over fact-preserving replacements.

Where it stands

In the RoboMME paper no memory design wins every task, and the best average is 44.51 percent against 90.5 for humans. A later training-free layer, SimpleARM, reports 67.17 percent on the same 16 tasks. That is a large jump and still well below the human score.

ASPIRE's skill library is one answer. To me it feels very clunky and not the most elegant solution. A text note is easy to read and fix, and it is also the kind of memory that goes stale.

What I would watch next: benchmarks that test several memory types on one robot, like RoboMME does in simulation, and navigation models that say plainly how long their history is.

← All writing