I've been interested in memory systems for a while, so I looked for one I could port to LeRobot, which I love, and I picked this one. It's an interesting system because the memory is a set of vectors that's more expressive than text, and I feel like that's where memory for robots is going. As I said in my post on robot memory , I think memory is a big bottleneck for long-horizon tasks. Reproducing Imprint taught me a lot. I found a couple of bugs and opened pull requests #10 and #11 to fix them, and the release is missing the recorded scenes the RoboCasa evaluation replays, so I emailed the authors to ask for them. In this short article I just want to share what the method is about and why I find it so interesting. In the Cover Blocks task, red, green and blue blocks sit in a random order. The robot covers them with three lids, then must uncover them in red, green, blue order. Once they're covered, every order looks the same, so how does the policy know which to open first? It has to remember. Most memory methods pick what to keep, but Imprint compresses the video, actions and joint states into a fixed set of 256 memory tokens, updated as the episode runs. With Imprint, π0.5 goes from 25.4% to 72.8% success on RMBench's nine memory tasks, the best among memory methods compared.
Imprint: a robot memory that keeps every stream and learns what matters Panel 1, the problem: in the RMBench Cover Blocks task, red, green and blue blocks sit in a random order and the robot covers them with three identical lids, then must uncover red first. Two episodes, A and B, that started in different orders end with the same camera view, yet the lid to lift differs, so only the past can tell. Panel 2: the history has three streams, camera video, the actions sent and the joint states. Other memory methods keep only the video: the last frames, keyframes (some with a text summary), or a learned summary; the last frames here show only lids. Imprint keeps all three streams and learns what matters. Panel 3: the history arrives in chunks of four frames from each of three cameras, plus the actions and joint states. A frozen video model turns the frames of each chunk into 12 tokens, and the actions and joint states of the chunk join those tokens at the state. Each chunk updates a fixed-size running state once; this recurrent encoder is the default. Learned queries guided by the task read that state plus the newest frames into the memory m_t, 256 tokens. The memory never grows. A full-attention variant instead reads every stored past token into the same 256-token memory. Panel 4, stage 1: with no policy and no human labels, a decoder that sees only m_t must rebuild image features, actions and joint states at random past steps, and estimate progress t/(T-1) through the demonstration. On Cover Blocks training demonstrations, a linear probe reads the color order from the memory of the full-attention variant at 83.7% balanced accuracy after stage 1, against 17.3% for a memory trained on the action loss alone; chance is 16.7%. Panel 5, stage 2: the decoder is dropped and the stage 1 encoder is plugged into the vision-language-action model. Every layer of its action expert reads m_t through cross-attention and a FiLM scale and shift, and encoder and policy train together on the action loss. RMBench success, memory bars with the full-attention variant: 25.4% for no memory, 50.8% for no stage 1, 58.8% for stage 1, then frozen, 72.7% for stage 1, then tuned. Panel 6, the result on RMBench, success against latency per 50-step action chunk, log scale: π0.5 alone 25.4% at 91 ms, HAMLET 28.1% at 144 ms, MemoryVLA 41.7% at 321 ms, EventVLA 67.8% at 960 ms, Imprint 72.8% at 122.8 ms. Imprint uses its default recurrent encoder here. Latency is measured on an L40S GPU, except for EventVLA, whose 960 ms is a reported value. The EventVLA and MemoryVLA success rates come from prior work on other backbones, and the EventVLA result is for an anchors-only variant (first frame plus recent frames). HAMLET runs on GR00T N1.6, trained by the Imprint authors. RoboCasa is the overall average over 12 tasks. real robot · π0: 25.6% without memory, 75.6% with Imprint; RoboCasa · π0.5: 52.2% without memory, 66.7% with Imprint. 1 · same view, two pasts start now A lids on lift B lids on lift same view uncover red first: which lid? only the past can tell RMBench · Cover Blocks 2 · keep every stream video actions joint states 0 now last frames: only lids others keep only video · last frames · keyframes, some + text · a learned summary Imprint keeps all three and learns what matters 3 · squeeze it as you go chunks: 4 frames × 3 cameras, plus actions and joint states newest frozen video model: frames → 12 tokens running state same size · recurrent (the default encoder) task memory m t 256 tokens the state updates once per chunk; the memory never grows 4 · stage 1: rebuild the past m t decoder sees only the memory 0 now random past steps image features action joint state progress 0 1 no policy, no labels: the recording is the target color order read from m t · Cover Blocks linear probe, balanced accuracy · full-attention variant no stage 1 17.3% after stage 1 83.7% chance 16.7% 5 · stage 2: plug it into the VLA camera task VLM action expert actions encoder from stage 1 decoder dropped memory m t every layer reads it train encoder + policy together on the action loss RMBench success · memory bars: full-attention variant no memory 25.4% no stage 1 50.8% stage 1, then frozen 58.8% stage 1, then tuned 72.7% 6 · better, and still fast 0 40 80 100 300 1000 ← faster · ms per 50-step action chunk, log scale success % → π0.5 alone HAMLET MemoryVLA† EventVLA†* Imprint (recurrent) 72.8% · 122.8 ms † success from prior work, other backbones * latency from prior work gray: no memory · orange: with Imprint real robot · π0 25.6% → 75.6% RoboCasa · π0.5 52.2% → 66.7% the memory · every stream, a fixed size the training, then the result m t Imprint: a robot memory that keeps every stream and learns what matters Panel 1, the problem: in the RMBench Cover Blocks task, red, green and blue blocks sit in a random order and the robot covers them with three identical lids, then must uncover red first. Two episodes, A and B, that started in different orders end with the same camera view, yet the lid to lift differs, so only the past can tell. Panel 2: the history has three streams, camera video, the actions sent and the joint states. Other memory methods keep only the video: the last frames, keyframes (some with a text summary), or a learned summary; the last frames here show only lids. Imprint keeps all three streams and learns what matters. Panel 3: the history arrives in chunks of four frames from each of three cameras, plus the actions and joint states. A frozen video model turns the frames of each chunk into 12 tokens, and the actions and joint states of the chunk join those tokens at the state. Each chunk updates a fixed-size running state once; this recurrent encoder is the default. Learned queries guided by the task read that state plus the newest frames into the memory m_t, 256 tokens. The memory never grows. A full-attention variant instead reads every stored past token into the same 256-token memory. Panel 4, stage 1: with no policy and no human labels, a decoder that sees only m_t must rebuild image features, actions and joint states at random past steps, and estimate progress t/(T-1) through the demonstration. On Cover Blocks training demonstrations, a linear probe reads the color order from the memory of the full-attention variant at 83.7% balanced accuracy after stage 1, against 17.3% for a memory trained on the action loss alone; chance is 16.7%. Panel 5, stage 2: the decoder is dropped and the stage 1 encoder is plugged into the vision-language-action model. Every layer of its action expert reads m_t through cross-attention and a FiLM scale and shift, and encoder and policy train together on the action loss. RMBench success, memory bars with the full-attention variant: 25.4% for no memory, 50.8% for no stage 1, 58.8% for stage 1, then frozen, 72.7% for stage 1, then tuned. Panel 6, the result on RMBench, success against latency per 50-step action chunk, log scale: π0.5 alone 25.4% at 91 ms, HAMLET 28.1% at 144 ms, MemoryVLA 41.7% at 321 ms, EventVLA 67.8% at 960 ms, Imprint 72.8% at 122.8 ms. Imprint uses its default recurrent encoder here. Latency is measured on an L40S GPU, except for EventVLA, whose 960 ms is a reported value. The EventVLA and MemoryVLA success rates come from prior work on other backbones, and the EventVLA result is for an anchors-only variant (first frame plus recent frames). HAMLET runs on GR00T N1.6, trained by the Imprint authors. RoboCasa is the overall average over 12 tasks. real robot · π0: 25.6% without memory, 75.6% with Imprint; RoboCasa · π0.5: 52.2% without memory, 66.7% with Imprint. 1 · same view, two pasts start now A lids on lift B lids on lift same view uncover red first: which lid? only the past can tell RMBench · Cover Blocks 2 · keep every stream video actions joint states 0 now last frames: only lids others keep only video · last frames · keyframes, some + text · a learned summary Imprint keeps all three and learns what matters 3 · squeeze it as you go chunks: 4 frames × 3 cameras, plus actions and joint states newest frozen video model: frames → 12 tokens running state same size · recurrent (the default encoder) task memory m t 256 tokens the state updates once per chunk; the memory never grows 4 · stage 1: rebuild the past m t decoder sees only the memory 0 now random past steps image features action joint state progress 0 1 no policy, no labels: the recording is the target color order read from m t · Cover Blocks linear probe, balanced accuracy · full-attention variant no stage 1 17.3% after stage 1 83.7% chance 16.7% 5 · stage 2: plug it into the VLA camera task VLM action expert actions encoder from stage 1 decoder dropped memory m t every layer reads it train encoder + policy together on the action loss RMBench success · memory bars: full-attention variant no memory 25.4% no stage 1 50.8% stage 1, then frozen 58.8% stage 1, then tuned 72.7% 6 · better, and still fast 0 40 80 100 300 1000 ← faster · ms per 50-step action chunk, log scale success % → π0.5 alone HAMLET MemoryVLA† EventVLA†* Imprint (recurrent) 72.8% · 122.8 ms † success from prior work, other backbones * latency from prior work gray: no memory · orange: with Imprint real robot · π0 25.6% → 75.6% RoboCasa · π0.5 52.2% → 66.7% the memory · every stream, a fixed size the training, then the result m t Imprint: a robot memory that keeps every stream and learns what matters Panel 1, the problem: in the RMBench Cover Blocks task, red, green and blue blocks sit in a random order and the robot covers them with three identical lids, then must uncover red first. Two episodes, A and B, that started in different orders end with the same camera view, yet the lid to lift differs, so only the past can tell. Panel 2: the history has three streams, camera video, the actions sent and the joint states. Other memory methods keep only the video: the last frames, keyframes (some with a text summary), or a learned summary; the last frames here show only lids. Imprint keeps all three streams and learns what matters. Panel 3: the history arrives in chunks of four frames from each of three cameras, plus the actions and joint states. A frozen video model turns the frames of each chunk into 12 tokens, and the actions and joint states of the chunk join those tokens at the state. Each chunk updates a fixed-size running state once; this recurrent encoder is the default. Learned queries guided by the task read that state plus the newest frames into the memory m_t, 256 tokens. The memory never grows. A full-attention variant instead reads every stored past token into the same 256-token memory. Panel 4, stage 1: with no policy and no human labels, a decoder that sees only m_t must rebuild image features, actions and joint states at random past steps, and estimate progress t/(T-1) through the demonstration. On Cover Blocks training demonstrations, a linear probe reads the color order from the memory of the full-attention variant at 83.7% balanced accuracy after stage 1, against 17.3% for a memory trained on the action loss alone; chance is 16.7%. Panel 5, stage 2: the decoder is dropped and the stage 1 encoder is plugged into the vision-language-action model. Every layer of its action expert reads m_t through cross-attention and a FiLM scale and shift, and encoder and policy train together on the action loss. RMBench success, memory bars with the full-attention variant: 25.4% for no memory, 50.8% for no stage 1, 58.8% for stage 1, then frozen, 72.7% for stage 1, then tuned. Panel 6, the result on RMBench, success against latency per 50-step action chunk, log scale: π0.5 alone 25.4% at 91 ms, HAMLET 28.1% at 144 ms, MemoryVLA 41.7% at 321 ms, EventVLA 67.8% at 960 ms, Imprint 72.8% at 122.8 ms. Imprint uses its default recurrent encoder here. Latency is measured on an L40S GPU, except for EventVLA, whose 960 ms is a reported value. The EventVLA and MemoryVLA success rates come from prior work on other backbones, and the EventVLA result is for an anchors-only variant (first frame plus recent frames). HAMLET runs on GR00T N1.6, trained by the Imprint authors. RoboCasa is the overall average over 12 tasks. real robot · π0: 25.6% without memory, 75.6% with Imprint; RoboCasa · π0.5: 52.2% without memory, 66.7% with Imprint. 1 · same view, two pasts start now A lids on lift B lids on lift same view uncover red first: which lid? only the past can tell RMBench · Cover Blocks 2 · keep every stream video actions joint states 0 now last frames: only lids others keep only video · last frames · keyframes, some + text · a learned summary Imprint keeps all three and learns what matters 3 · squeeze it as you go chunks: 4 frames × 3 cameras, plus actions and joint states newest frozen video model: frames → 12 tokens running state same size · recurrent (the default encoder) task memory m t 256 tokens the state updates once per chunk; the memory never grows 4 · stage 1: rebuild the past m t decoder sees only the memory 0 now random past steps image features action joint state progress 0 1 no policy, no labels: the recording is the target color order read from m t · Cover Blocks linear probe, balanced accuracy · full-attention variant no stage 1 17.3% after stage 1 83.7% chance 16.7% 5 · stage 2: plug it into the VLA camera task VLM action expert actions encoder from stage 1 decoder dropped memory m t every layer reads it train encoder + policy together on the action loss RMBench success · memory bars: full-attention variant no memory 25.4% no stage 1 50.8% stage 1, then frozen 58.8% stage 1, then tuned 72.7% 6 · better, and still fast 0 40 80 100 300 1000 ← faster · ms per 50-step action chunk, log scale success % → π0.5 alone HAMLET MemoryVLA† EventVLA†* Imprint (recurrent) 72.8% · 122.8 ms † success from prior work, other backbones * latency from prior work gray: no memory · orange: with Imprint real robot · π0 25.6% → 75.6% RoboCasa · π0.5 52.2% → 66.7% the memory the training, then the result m t Stage 1 needs no policy and no human labels, since the recording itself is the target. The encoder packs the history and task into the memory m t m_t m t m t m_t m t L mem = 1 K ∑ k [ λ v ∥ z ^ k v − z k v ∥ 2 + λ a ∥ a ^ k − a k ∥ 2 + λ q ∥ q ^ k − q k ∥ 2 ] + λ ρ ( ρ ^ t − ρ t ) 2 \mathcal{L}_{\text{mem}} = \frac{1}{K}\sum_{k}\Big[\lambda_v\big\|\hat z^{v}_{k}-z^{v}_{k}\big\|^{2}+\lambda_a\big\|\hat a_{k}-a_{k}\big\|^{2}+\lambda_q\big\|\hat q_{k}-q_{k}\big\|^{2}\Big]+\lambda_\rho\big(\hat\rho_t-\rho_t\big)^{2} L mem = K 1 k ∑ [ λ v z ^ k v − z k v 2 + λ a a ^ k − a k 2 + λ q q ^ k − q k 2 ] + λ ρ ( ρ ^ t − ρ t ) 2 L mem = 1 K ∑ k [ λ v ∥ z ^ k v − z k v ∥ 2 + λ a ∥ a ^ k − a k ∥ 2 + λ q ∥ q ^ k − q k ∥ 2 ] + λ ρ ( ρ ^ t − ρ t ) 2 \begin{aligned}\mathcal{L}_{\text{mem}} = {}&\frac{1}{K}\sum_{k}\Big[\lambda_v\big\|\hat z^{v}_{k}-z^{v}_{k}\big\|^{2}\\&+\lambda_a\big\|\hat a_{k}-a_{k}\big\|^{2}+\lambda_q\big\|\hat q_{k}-q_{k}\big\|^{2}\Big]\\&+\lambda_\rho\big(\hat\rho_t-\rho_t\big)^{2}\end{aligned} L mem = K 1 k ∑ [ λ v z ^ k v − z k v 2 + λ a a ^ k − a k 2 + λ q q ^ k − q k 2 ] + λ ρ ( ρ ^ t − ρ t ) 2 k k k K K K z k v z^v_k z k v a k a_k a k q k q_k q k ρ t = t / ( T − 1 ) \rho_t = t/(T-1) ρ t = t / ( T − 1 ) T T T λ v = 1 \lambda_v=1 λ v = 1 λ a = λ q = λ ρ = 0.5 \lambda_a=\lambda_q=\lambda_\rho=0.5 λ a = λ q = λ ρ = 0.5 π θ \pi_\theta π θ − log π θ ( a t ∣ o t , ℓ , m t ) -\log \pi_\theta(\mathbf{a}_t \mid o_t, \ell, m_t) − log π θ ( a t ∣ o t , ℓ , m t ) a t \mathbf{a}_t a t o t o_t o t ℓ \ell ℓ
What gets rebuilt, and at what rate? + Question 1 board Q1: a past camera frame goes through the frozen DINOv3 ViT-B/16 image model. Its 16 by 16 patch grid is averaged to 8 by 8, which gives 64 patch features of 768 numbers per camera, three cameras per step. That grid is the target. The decoder sees the history only through the memory m_t and predicts the same grid. Both get a parameter-free LayerNorm patch by patch, then a mean squared error. Pixels are never rebuilt. Q1 · features, not pixels past frame DINOv3 frozen ViT-B/16 target 64 × 768 × 3 cameras m t decoder prediction compare LayerNorm each patch, then mean squared error Is the visual target pixels, or a frame embedding? Not pixels. The target z k v z^v_k z k v λ v = 1 \lambda_v = 1 λ v = 1 m t m_t m t
Paper: Sec. 4.2.1, Eq. 4.4 · App. A.1.1, recent visual history · App. A.1.3, Eqs. A.5 and A.6pretrain_loss.py:79-87 · build_dinov3_store.py:92-95 · visual_streams.py:51-85 · train_stage1.sh:53-56 · config:20 · config:62-63
Question 2 board Q2: frames, joint states and actions share one simulator step index. The memory keeps every second step of all three streams together and drops the rest; in training, each example uses one of eight stored frame lists instead. Four kept steps, eight simulator steps, make one chunk: in the recurrent encoder, 12 video tokens plus 4 action and joint-state pairs. In training, the action stored with a kept step is the one executed at the kept step before. The paper does not state the simulator rate in Hz or any camera frame rate. Q2 · one clock for all three frames joint states actions 4 kept = 8 steps recurrent: 12 tokens + 4 pairs 0 simulator steps bright: kept · faint: dropped blue arrow: in training, the action from the kept step before not stated in the paper: simulator Hz, camera fps How do streams at different rates line up? No stream is resampled to another's rate, and nothing is interpolated. In RMBench, video, joint states and actions share one step index, and the memory keeps every second step of all three. Four kept steps, eight simulator steps, make a video chunk, and up to three leftover frames go in as DINOv3 tokens with their actions and joint states. Only actions that already ran go in: in training, each kept frame gets the previous kept step's action. Each training example uses one of eight stored, time-sampled frame lists, while evaluation keeps exactly every second step. The memory is written and read once per policy call, after each 50-action chunk runs.
The paper's only rate is the real robot's: each 30-action chunk runs at a nominal 30 Hz, about a second of motion. It doesn't give the simulator's rate, any camera frame rate, or whether the real robot keeps every second step, and the simulation-only public code can't fill that in.
Code observation, not stated in the paper: the released video encoder is causal, so chunk 0 holds kept frame 0 alone and each later chunk c c c 4 c − 3 4c-3 4 c − 3 4 c 4c 4 c c c c 4 c 4c 4 c 4 c + 3 4c+3 4 c + 3 Proposed fix, off by default .
Paper: App. A.1.1, observations, consolidated and recent visual history, Eq. A.1 · App. A.1.2, recurrent bottleneck · App. B.5.1 and B.5.3config:22-27 · validate_store.py:39-40 · pretrain_bottleneck.py:653-657 · variant_actions.py:27-29 · cosmo_buffer.py:144-151
Question 3 board Q3: the joint-state stream goes into the memory m_t. The decoder rebuilds the raw recorded joint state from m_t at up to 16 random past steps, all at or before now, with a mean squared error of weight 0.5. The decoder forecasts nothing after now. Q3 · joint states: in, and rebuilt joint states in m t decoder 0 now future the decoder never forecasts rebuilt at up to 16 random past steps raw joint vector · mean squared error, weight 0.5 Do they rebuild joint states, predict them, or neither? Yes, it rebuilds them. In RMBench, joint states (the paper's proprioception) are an input stream and a Stage 1 target. The target is the raw joint vector, 14 values (arm joints and grippers) zero-padded to 16, unnormalized, and the loss averages the squared error over all 16, with weight λ q = 0.5 \lambda_q = 0.5 λ q = 0.5 m t m_t m t
Paper: Sec. 4.1 · App. A.1.1, robot signals · App. A.1.3, Eqs. A.5 and A.6cosmo_decoder.py:155-162 · cosmo_decoder.py:225-228 · pretrain_bottleneck.py:726-727 · pretrain_bottleneck.py:39 · pretrain_bottleneck.py:773 · pretrain_loss.py:89-90 · train_stage1.sh:53-56 · config:24
Wu, Shah, Huang, Sontakke, Bharadhwaj, Bajcsy, Simchowitz, Shi · Amazon FAR, CMU, UT Austin, JHU · arXiv · 9 Oct 2026
Paper ↗ Project ↗ Code ↗ ← All writing