WorksComputer Science
F: What Makes Good Experience in World-Action Models?
Abstract
Deployed world-action models (WAMs) can keep improving only if they make good use of their own experience. Existing methods supply retrieved rollouts or demonstrations to the policy as context, yet such experience often turns previously solved tasks into failures. What, then, makes good experience for a WAM? We show that experience is valuable only in use: its value depends not on whether the source rollout succeeded but on what is stored, when it is used, and where it enters the policy. We therefore propose F, which makes these three decisions with Form, Fit, and Fusion. Form compresses successful and failed rollouts alike into short, verified operation records; Fit teaches the policy to read a record and to predict when it helps; Fusion injects the record's images, text and actions into the policy as a prefix and an action prior. Trained once on only 200 examples and deployed with a growing experience library, the model achieves state-of-the-art results on MolmoSpaces and RoboLab without any training on RoboLab: over native Cosmos 3 Nano, it raises success by 12.0 pp and 10.8 pp, reduces steps to success by 12.2% and 11.3%, and cuts regressions by 57%.
Introduction
Post-deployment self-improvement is essential for world-action models (WAMs) to operate autonomously over the long term in real-world environments [1]. Every rollout shows a WAM the consequences of its own actions, and this experience could guide later attempts at the same task and inform decisions in other tasks under similar conditions [2], but how a WAM should use it remains open.
Existing methods supply such experience, or demonstrations from other sources, to the policy as context. Zero-WAM [3] specifies an unseen task with a human video, WATCH [4] predicts actions and frames from one demonstration of the same task, RA-VLA [5] retrieves demonstrations by behavioral similarity, AED [6] feeds retrieved action embeddings to the policy, and Zeva [2] retrieves action effects by task phase. Each selects a record, a retrieved piece of experience, by its source or by its match to the task, behavior or phase, assuming that such a record will help.
Our experiments, however, suggest otherwise. In paired closed-loop trials, giving Cosmos 3 Nano [7], the strongest baseline in our evaluation, the most similar record from its own past rollouts as text and frames breaks 21 episodes it would otherwise solve and rescues only 7, lowering its success from 55% to 41% (Figure 2c). We therefore ask what makes good experience for a world-action model.

Figure 2. F3 succeeds more often and in fewer steps. (a) Success rate on MolmoSpaces and RoboLab; the dotted line marks native Cosmos 3 and hatching the gain of F3. (b) Mean steps on episodes that both policies complete; hatching marks saved steps. (c) Paired outcomes against native Cosmos 3 on the 100 MolmoSpaces episodes: regressions (S→F), rescues (F→S) and net change. Direct input breaks three times as many episodes as it rescues, whereas F3 cuts regressions by 57%.
We show that experience is valuable only in use: a record is valuable at a state if supplying it raises the probability that the policy completes the task from there. This value follows neither from the record’s source nor its similarity to the task. It depends on three choices: what is stored, since failures hold useful operations (§ 4.3); when it is used, since injecting by predicted gain, not similarity, breaks fewer successes (§ 4.4); and where it enters the policy, since raw records hurt, whereas the same records help once they enter through trained readers (§ 4.5).
We therefore propose F³, built on three components that
decide what to store (Form), when to use it (Fit), and where it enters the policy (Fusion). Form compresses successful and failed rollouts alike into short operation records, each verified against the recorded frames and states, so that failures still contribute the operations that made progress. Fit teaches the policy to read records and injects one only when its predicted gain exceeds a margin, learning that gain from paired continuations with and without the record rather than from similarity alone. Fusion encodes the record’s images, text and actions and feeds them to both streams of the WAM, as a prefix that tells the Reasoner which operation to perform and an action prior that tells the Generator how to move. On the knife task in Figure 1b, the native WAM times out, whereas F³ finishes in 139 steps by reusing fork, ladle and spatula records, none of which involves a knife.
Figure 1. Which past experience a WAM reuses decides whether it succeeds. (a) Form turns a rollout into short multimodal records. (b) On the same knife-to-bowl task, the native WAM times out, whereas reusing the right records succeeds in 237 and then 139 steps. (c) Fusion feeds the images, text and actions of a selected record to the WAM as a prefix and an action prior.
On MolmoSpaces and RoboLab, F³ achieves state-of-the-art results (Figure 2), although it is never trained on RoboLab: it raises the success of Cosmos 3 Nano from 55% to 67% and from 29.2% to 40.0%, reduces steps to success by 12.2% and 11.3%, and cuts regressions against direct input by 57%.
Our contributions are threefold.
We show that experience supplied directly as context can hurt a WAM, and propose to judge a record by its gain for the receiving policy at the current state rather than by its source. This view yields three requirements for reuse: what to store, when to use it, and where it enters the policy.
We propose F³, which meets each requirement with one component. Form compresses successful and failed rollouts into verified operation records, with a verification bound that links local checks to the whole rollout; Fit learns when a record helps from paired continuations rather than from similarity; and Fusion injects its images, text and actions into both the reasoning and action streams.
We demonstrate that F³, trained once on 200 examples and fixed at deployment, achieves state-of-the-art results on MolmoSpaces and on RoboLab, where it is never trained, and that ablations support each requirement.
Related Work
Learning from experience. Agents have long improved from past attempts. Case-based reasoning retrieves, reuses and revises prior solutions [8], language agents store verbal reflections, distilled lessons [9, 10] or reusable skills scored by swapping them out [11], and ICAL [12] extends this to VLM agents. Robot policies instead update their weights, from expert labels on visited states [13] or from successful and failed deployment rollouts [1, 14]. F³ keeps the policy fixed at deployment and stores experience in the robot’s images, states and actions, which the policy learns to read.
Representing experience. Classical planners describe an action by its preconditions and effects [15], and the information bottleneck keeps only what an input reveals about a target [16]. Form combines the two: each record is an operation with a precondition and an effect, and a rollout is compressed to the fewest steps that preserve the evidence of task progress, verified clip by clip.
Deciding when to retrieve. Retrieval-augmented language models found that retrieving regardless of need can hurt, and now retrieve on demand: FLARE [17] retrieves when an upcoming sentence has low-confidence tokens, and Self-RAG [18] trains the model to emit tokens that decide when to retrieve. Fit makes this decision for a WAM, but learns it from the measured effect of a record on paired continuations.
Experience as context for robot policies. Human videos and demonstrations specify goals for WAMs or supervise them across executions [3, 4], VLA policies retrieve demonstrations by behavior or by subtask phase and geometry [5, 19], and ReCAP [20] corrects a motion retrieved from another embodiment. Within WAMs, which jointly predict observations and actions [21, 22, 23], MaP-WAM [24] plans from earlier stages of a rollout, AED [6] supplies retrieved action embeddings, and Zeva [2] retrieves action effects by task phase. All select experience by its match to the current task, phase or motion. F³ also checks whether a record’s precondition holds and its effect is still pending, and injects it only when its predicted gain exceeds a margin. § 4.1 reimplements these mechanisms as baselines.
Method
Experience is not what happens to a man; it is what a man does with what happens to him.
ALDOUS HUXLEY, Texts and Pretexts
A rollout is what happens to a WAM; experience is what the WAM can do with it. Handing a WAM its own successful trajectory can even make it fail (§ 4.5), so F³ treats experience as something to be made: it decides what to keep, when to use a record, and how the policy reads it (Figure 3).
Figure 3. Overview of F³. Form (left) compresses successful and failed rollouts into short verified records; Fit (middle) teaches the WAM to read a record and learns when it helps; Fusion (right) injects the selected record’s images, text and actions as a prefix and an action prior. The WAM is trained once and stays fixed at deployment, while new rollouts grow the library.
Problem formulation: what is good experience?
Setting. A WAM policy maps a query (current image, robot state and instruction) to a chunk of 32 actions. We build on Cosmos 3 Nano [7], which couples two transformers through shared attention: a Reasoner that reads the instruction and images, and a Generator that denoises the action chunk by flow matching [25]. A rollout records one attempt with context (instruction and initial observation) and outcome . The policy’s own rollouts, successful or failed, form the data .
Good experience. An experience is a record extracted from . Let and be the probabilities of completing the task from , with the same action budget, when is supplied and when the policy receives only its native inputs. We call good at if
Goodness is thus a property of three things together: the record, the state and the policy that receives it. A grasp record is good before the object is held and harmful afterwards, and it is worthless to a policy that cannot read it.
From the definition to the method. We seek a library, a rule that selects a record and an operator that supplies it to , with native behavior when nothing is selected. Equation (1) yields a gain only if three conditions hold, and each motivates one component (Figure 3): (i) the library must contain records with , so Form (§ 3.2) decides what to store; (ii) a record must be used only where , so Fit (§ 3.3) teaches the policy when and how to use it; (iii) a record must be able to change the predicted actions, since otherwise , so Fusion (§ 3.4) decides where it enters the policy.
Form: what to store
Motivation. Prior work reuses whole trajectories [3, 4] or segments retrieved from them [19, 26]. But the outcome of a rollout says little about the value of its steps. A successful rollout carries steps that do not advance the task, such as empty reaches before the grasp that mattered, and a failed rollout can contain a sound grasp that achieved a local goal. The unit worth keeping is therefore the step that carries evidence of task progress, whatever the outcome of its rollout, which makes choosing experience a compression problem.
Minimal sufficient experience. The information bottleneck [16] formalizes such compression: it keeps what an input reveals about a target and discards the rest. Taking task progress as the target, let be progress variables, one per goal, such as whether a grasp was established, and let be the records with total length . Form seeks the shortest records that keep the evidence of progress,
Two-level compression. Solving Eq. (2) requires deciding which goals a rollout reached, which defines , and which clip shows each most briefly; the WAM’s own Reasoner does both, so Form needs no external model.
(1) Goal level. The Reasoner reviews the rollout against its instruction and divides its progress into the reached goals , discarding steps that serve no goal.
(2) Clip level. For each goal, the Reasoner proposes the clip that achieved it. The clip is kept only if the Reasoner, now acting as verifier, confirms the goal from its frames and state trace, e.g., a grasped object rises with the gripper, so a verified grasp from a failed rollout is kept as well. If several verified clips support a goal, we link clip to clip when the effect of satisfies the precondition of and keep the chain with the fewest actions, found by the recursion with . Minimality thus holds over the candidate clips the Reasoner proposes; kept clips are then stored as 32-action windows, one policy chunk.
Packaging as experience. A clip is reusable only if the policy can tell where it applies and what it achieves. The Reasoner therefore annotates each kept clip with a precondition and an effect , tests that on the observation and state , and an action that names the operation and the goal it serves. The record and its consistency conditions are
where the underbraced terms are native signals.
At a new query , the record applies only if and . For the fork grasp in Figure 1a, holds with the gripper open beside the fork, and once it is lifted.
What compression preserves. Equation (2) constrains the whole rollout, whereas verification checks each clip from its signals ; local checks suffice ( is the binary entropy).
Theorem 1 (Local checks certify the global constraint) If takes at most values and a verifier predicts from with error probability , then
where .
Whenever every clip is verified correctly, which holds with probability at least , the records determine every progress variable, and replacing a rollout by its records raises the minimum log loss for predicting progress by exactly (Theorem 2, Appendix A). This is why Form keeps a clip only after the Reasoner confirms its goal. Form does not optimize , which depends on the receiving policy and state; it keeps the evidence of progress on which a positive can rest, and Fit estimates itself.
Fit: learning to use a record, and when
Motivation. A good library is not enough. Supplied directly, a record from the WAM’s own success can turn a native success into a failure (§ 4.5): the policy was never trained to read it, and a record that helps at one state disrupts at another. Fit learns both how to use a record and when.
Learning to use a record. To teach reading, the supervision must reward applying a record under new conditions instead of copying it. Each training triple therefore pairs a verified record with a window in another rollout that starts under compatible conditions and exhibits the same operation, with query at the start of that window and target the 32 actions executed next. An encoder maps the images, states, actions and annotation of to memory tokens , which condition the policy through the two readers of § 3.4. We jointly train policy, encoder and readers on
where flow matching supervises action denoising and is the negative log-likelihood of under the predicted prior. On a fraction of examples we bypass memory (), so the policy keeps its native behavior without a record. This training runs once, offline; at deployment all parameters stay fixed (values of , and later constants are in Appendix B).
Deciding when to inject. The ideal rule injects a candidate exactly when its gain is positive and attains , with the success probability under bypass. As is unobservable at deployment, Fit approaches this rule through three checks of rising cost before each chunk:
(1) Similar. A record can help only in a situation like the one it came from. With the cosine similarity between the query and the start of a record in modality , we rank records by
and keep the top records whose scores reach , writing for any of them. In the knife task of Figure 1b, this retrieves the fork-grasp record of Figure 1a.
(2) Applicable. A similar scene may already be past the operation. We require the precondition of to hold and its effect to be unrealized, and (Eq. (3)), so the fork record applies while the gripper is open beside the knife and is skipped once it is lifted.
(3) Useful now. An applicable record can still disrupt the action the policy is about to take. We inject only if the predicted gain exceeds a margin , so the fork record guides the grasp only when it helps. Failed checks revert to native inputs and need no WAM call.
The gain predictor is trained on the quantity it must estimate. With the policy, encoder and readers held fixed, we continue the policy twice from each eligible state with matched seeds, injecting the candidate in one branch and bypassing it in the other, and regress the outcome difference with a small MLP,
whose minimizer is the true gain , which ties the gate to the ideal rule.
Proposition 1 Among candidates that pass the first two checks, let be the expected gain given and its estimate. Injecting when yields
The gap to is thus set by the gain estimate’s error, so Fit regresses on paired outcomes, not on similarity.
Algorithm 1. F³ training and deployment.
Offline training
Form: build the library from successful and failed rollouts
Goal level: the Reasoner divides each rollout into reached goals and discards steps that serve no goal.
Clip level: propose a clip per goal; keep it only if its frames and states confirm the goal (Theorem 1).
Keep the shortest chain per goal as 32-action windows.
Package into .
Fit (1): fit the policy to records
Pair a record with a window of another rollout that shows the same operation.
Encode ; bypass it on a fraction of examples ().
Read the prefix and prior from ; update , and the readers on (Eq. (5)).
Fit (2): learn when a record helps
Hold , and the readers fixed. From each eligible , continue with and without it to obtain and .
Fit to (Eq. (7)).
Online deployment (before every action chunk)
Fit (3): select a record
Observe and take the top records by (Eq. (6)).
Similar? If , use native inputs.
Applicable? If or , use native inputs.
Useful now? Encode ; if , use native inputs.
Fusion: inject and execute
Reasoner: place the prefix before the instruction (score-weighted if several records pass).
Generator: add the prior, (Eq. (10)).
Denoise the chunk and execute it; repeat until termination.
After the rollout, grow the library: .
Fusion: where a record enters the policy
Motivation. A record changes behavior only if it enters where the WAM forms its actions. It carries two kinds of guidance, which operation to perform and how to move, and Fusion supplies each to the tower that uses it, the Reasoner and the Generator (Figure 1c). Two readers, trained in Fit, attend to the memory tokens given the query , whose image tokens ground the record in the present scene.
(1) What to do: a prefix for the Reasoner. A memory token pooled from precedes the instruction, as in soft prompting [27], and conditions the chunk on the record, for the fork record on grasping and lifting.
(2) How to move: a prior for the Generator. On the flow-matching path between a clean chunk and noise , the optimal velocity given and memory is
so a record changes the optimal denoising direction only by shifting the expected clean chunk. The reader supplies this shift directly: it reads per-step prior means from and adds them to the noisy action tokens,
where is a projection and a learned gate; for the fork record, carries its lifting motion.
When several records pass all three checks, their memory tokens are averaged with weights proportional to their scores before both readers. Without injection, the WAM receives exactly its native inputs.
Algorithm 1 summarizes the procedure. At deployment all parameters stay fixed, while Form adds records after every rollout, so the library grows with use (Appendix § 8).
Experiments
We first ask whether F3 improves success and efficiency (§ 4.2), and then each question of Sec. 3.1: what to store (§ 4.3), when to use it (§ 4.4) and where it enters (§ 4.5). A case study shows all three (§ 4.6).
Setup
Benchmarks. MolmoSpaces evaluation uses 100 fixed instances sampled from its task families in proportion to their size: 27 MS-Pick, 43 MB-Pick and 30 MS-PnP, with action budgets of 455, 304 and 682 steps [33, 34]. The experience library comes from 50 other instances, disjoint from these, and is not changed within a full evaluation pass. RoboLab covers 120 tasks, 64 simple, 39 moderate and 17 complex [35]. We run RoboLab v0.2.1 on NVIDIA A100 GPUs, whereas most leaderboard entries used v0.1.0 on RTX GPUs. Cosmos 3 Nano and our controls run in this setup, and the other baselines are taken from the leaderboard (Appendix § 9).
Policies. We compare against VLA and WAM baselines (Table 1) and build four controls on the same Cosmos 3 Nano, fitted once as in § 3.3, so all of them share what Fit teaches the policy about reading records. Each adds one component to the one before, so the difference between adjacent rows isolates that component. Memory off runs without experience; Form adds the record library and supplies the most similar record as direct input; Form × Fusion instead routes it through the trained readers into both towers whenever its similarity exceeds ; and Form × Fusion × Fit, i.e., F3, replaces this threshold with the Fit gate. Table 2 (left) also feeds records directly to native Nano. Prior experience-conditioned WAMs [2, 6] use other backbones or training tasks, confounding an end-to-end comparison, so we reimplement their mechanisms on our fitted Nano and library. Zeva-style, the same run as Form, supplies a record as context, but selects it by similarity rather than task phase and lacks Zeva’s trained causal encoder; AED-style feeds retrieved actions through the trained encoder (Table 2, right, A†).
The same experience hurts or helps depending on whether the WAM has learned to read it. Success on the 100 MolmoSpaces episodes when retrieved records are given to native Cosmos 3 Nano-DROID directly (left) or through the trained encoder and readers (right). T/V/A: text/vision/action; vision inserts three retrieved frames into a camera slot. Right: T+V+A is the epoch-5 full-input run, the other inputs replay its injection schedule, and the last row adds the Fit gate. †AED-style input (§ 4.1).
| Input | Direct input (native WAM) | Trained encoder and readers | ||||||
| MS-Pick | MB-Pick | MS-PnP | All | MS-Pick | MB-Pick | MS-PnP | All | |
| None | 63.0 | 55.8 | 46.7 | 55.0 | 63.0 | 55.8 | 46.7 | 55.0 |
| T | 63.0 | 55.8 | 40.0 | 53.0 | 74.1 | 67.4 | 36.7 | 60.0 |
| V | 66.7 | 62.8 | 46.7 | 59.0 | 63.0 | 62.8 | 30.0 | 53.0 |
| A | 29.6 | 25.6 | 3.3 | 20.0 | 66.7 | 62.8 | 30.0 | 54.0† |
| T+V | 51.9 | 51.2 | 16.7 | 41.0 | 70.4 | 58.1 | 36.7 | 55.0 |
| T+A | 40.7 | 39.5 | 16.7 | 33.0 | 63.0 | 60.5 | 36.7 | 54.0 |
| V+A | 37.0 | 20.9 | 0.0 | 19.0 | 70.4 | 62.8 | 46.7 | 60.0 |
| T+V+A | 14.8 | 18.6 | 0.0 | 12.0 | 74.1 | 62.8 | 50.0 | 62.0 |
| F³ (T+V+A) | — | — | — | — | 70.4 | 67.4 | 63.3 | 67.0 |
Table 2.
Metrics. SR is the fraction of completed episodes. Because all policies start from the same states, we also count per-episode changes against native Nano, a rescue (F→S) or a regression (S→F). Steps are compared on jointly solved episodes. Each configuration is run several times; we report the run closest to the mean, for paired analysis.
Main results
F³ is the strongest policy on both benchmarks, without any training on RoboLab (Table 1, Figure 2a). It raises native Nano, itself the strongest baseline, from 55% to 67% on MolmoSpaces and from 29.2% to 40.0% on RoboLab. Fit uses only 200 examples, none from RoboLab, so RoboLab’s tasks and scenes are unseen in training, and the gain there shows that how F³ reads and injects records generalizes.
| Policy | Type | MS-Pick | MB-Pick | MS-PnP | Overall | Simple | Moderate | Complex | Overall |
| Baseline policies | |||||||||
| π0-DROID [29] | VLA | 3.7 | 7.0 | 10.0 | 7.0 | 7.2 | 3.6 | 0.0 | 5.0 |
| π0-FAST-DROID [30] | VLA | 14.8 | 2.3 | 13.3 | 9.0 | 20.2 | 13.3 | 2.9 | 15.5 |
| π0.5-DROID [31] | VLA | 18.5 | 25.6 | 6.7 | 18.0 | 29.7 | 31.5 | 13.5 | 28.0 |
| DreamZero-DROID [32] | WAM | 40.7 | 27.9 | 16.7 | 28.0 | 26.1 | 30.0 | 14.1 | 25.7 |
| Cosmos 3 Edge-DROID [7] | WAM | 29.6 | 44.2 | 13.3 | 31.0 | 25.6 | 23.3 | 11.8 | 22.9 |
| Cosmos 3 Nano-DROID [7] | WAM | 63.0 | 55.8 | 46.7 | 55.0 | 34.4 | 25.6 | 17.6 | 29.2 |
| Prior mechanisms, reimplemented on our fitted Nano | |||||||||
| Zeva-style context [2] | WAM | 59.3 | 65.1 | 43.3 | 57.0 | 28.1 | 38.5 | 23.5 | 30.8 |
| AED-style action input [6] | WAM | 66.7 | 62.8 | 30.0 | 54.0 | — | — | — | — |
| Our method | |||||||||
| Memory off | WAM | 51.9 | 60.5 | 50.0 | 55.0 | 35.9 | 30.8 | 11.8 | 30.8 |
| Form‡ | WAM | 59.3 | 65.1 | 43.3 | 57.0 | 28.1 | 38.5 | 23.5 | 30.8 |
| Form × Fusion | WAM | 74.1 | 62.8 | 50.0 | 62.0 | 43.8 | 30.8 | 17.6 | 35.8 |
| Form × Fusion × Fit (Mol) | WAM | 70.4 | 67.4 | 63.3 | 67.0 | 48.4 | 33.3 | 23.5 | 40.0 |
| Form × Fusion × Fit (Rob) | WAM | 74.1 | 69.8 | 53.3 | 66.0 | 48.4 | 33.3 | 17.6 | 39.2 |
Table 1. Closed-loop manipulation performance. Success rate (SR, %) on 100 MolmoSpaces instances and 120 RoboLab tasks. Gray: RoboLab results from the official leaderboard [28]; all other numbers are from our runs. The middle rows reimplement prior mechanisms on our fitted Cosmos 3 Nano and library (§ 4.1); our rows add one component at a time to it. (Mol)/(Rob): library source. ‡Same run as the Zeva-style row. Bold: best.
The gain comes from new successes. Against native Nano, F³ rescues 21 MolmoSpaces episodes and breaks 9. The gain is largest on MS-PnP (+16.7 pp), the only family chaining grasp, transport and placement. On RoboLab, simple, moderate and complex tasks gain 14.0, 7.7 and 5.9 pp.
Experience transfers across benchmarks. A library built on either benchmark gives success within 1 pp on both, as records describe operations rather than scenes.
Every component is needed in our controls. Without memory the fitted policy stays near native Nano, and retrieving records without trained readers (Zeva-style) adds at most 2 pp. On MolmoSpaces, F³ exceeds the Zeva-style control by 10 pp (57% to 67%) and the AED-style action input by 13 pp (54% to 67%). Routing records through Fusion’s readers and then Fit’s decision of when to inject each add about 5 pp. The gain thus needs all three, as Sec. 3.1 predicts; the next three sections examine them in turn.
F³ also acts more efficiently. On the episodes that both policies complete, mean steps fall by 12.2% on MolmoSpaces and by 11.3% on RoboLab (Figure 2b).
What to store: failed rollouts matter
If Form × Fusion and F³’s outcome-blind criterion (§ 3.2) is right, records from failed rollouts should matter as much as those from successes. They do (Figure 4). Nearly a third of the 55 MolmoSpaces records come from failed rollouts. When retrieved, they are injected as often as records from successes (55% vs. 57%) and account for over a third of all injections. Of the 67 episodes that F³ completes, 35 inject at least one failed-rollout record. A success-only library would lack them all.
Figure 4. Failed rollouts are as useful as successful ones. Left: growth of the MolmoSpaces library by source. Middle: the 55 records by operation and object. Right: injection rate when retrieved, share of the 490 injections, and the 35 of 67 successful episodes that use a failed-rollout record.
When to use: similarity is not usefulness
Predicted gain beats a similarity threshold. Form × Fusion and F³ share the library, encoder and readers and differ only in the rule that decides when to inject, so their gap measures that decision. Replacing the threshold with the Fit gate raises success from 62% to 67% on MolmoSpaces and from 35.8% to 40.0% on RoboLab, and the gate leads at every Fit checkpoint (Figure 5).

Figure 5. Fit training converges and improves success. (a) Training losses over five epochs. (b) Success of each saved checkpoint: F³ stays at or above the similarity threshold and native Nano throughout.
The gate disrupts less and rescues more. It breaks fewer of native Nano’s successes (9 vs. 11) and rescues more of its failures (21 vs. 18). Similarity to the current scene is thus a poor proxy for usefulness: the gate keeps the similar records that help and withholds those that would disrupt.
The gate matters most on long tasks. The gain over the threshold concentrates where tasks chain operations: on MS-PnP, success rises from 50.0% to 63.3%, against small changes on the shorter families (Table 1).
Where it enters: through readers the WAM has learned
To test whether Fusion’s readers, trained in Fit, are necessary, we supply the same records with and without them.
The native WAM can already use some experience. We first give records to the native policy without any training. At each replan, the most similar record over the enabled modalities is supplied if its score exceeds (Table 2, left). Vision alone, which places three retrieved frames in one camera slot, raises success from 55% to 59%. Text alone, which describes an operation from another task, barely changes success. Both reach the policy through channels it was trained on: the WAM already takes several camera views, so a retrieved frame arrives as one more view of a similar scene, and its Reasoner grounds language in the current images, which outweigh a mismatched description. Both changes are small, yet the WAM uses experience in known channels.
Retrieved actions are what hurts. Every input that contains actions loses 22 to 43 successes, and the richest input is the worst: with text, vision and actions together, success falls to 12%. A likely cause is that retrieved actions enter the same policy denoises, where an untrained policy cannot separate them from its own plan. This is the gap that the trained action reader of Fusion closes.
Learning to read reverses this ordering. With the trained encoder and readers, the same modalities no longer disrupt the policy (Table 2, right). Every combination stays within −2 to +7 successes of native Nano, and the input that was most harmful becomes the most helpful: all three modalities give the best fixed-schedule result. In Figure 6, direct inputs fall in the loss region, the trained encoder moves them to the diagonal, and F lies furthest beyond it. Whether records harm or help thus depends on learned reading: without trained readers (Form, Table 1), MS-PnP drops by 6.7 pp.
Figure 6. Learning to read experience turns losses into gains. Each point is one input type on the 100 MolmoSpaces episodes, placed by the episodes it breaks (S→F) versus rescues (F→S) relative to native Nano; above the diagonal is a net gain. Direct inputs fall in the loss region, the trained encoder brings them close to the diagonal, and F³ lies furthest in the better direction.
Case study: from grasp to placement
Figure 7 follows the knife-to-bowl episode of Figure 1b. All runs grasp the knife but differ afterwards: native Nano and Form time out at 682 steps. Reading the fork-grasp record at the second call finishes in 237 steps (Form × Fusion). F injects the fork and ladle records together at that call and the spatula record over the next two calls, then completes placement without memory in 139 steps. No record involves a knife, yet each covers the next sub-goal.
Figure 7. How F reuses experience within one episode. On the knife-to-bowl task of Figure 1b, injecting the fork, ladle and spatula records at the right calls lets the WAM grasp, carry and place the knife in 139 steps, while native Nano times out at 682. (a) Knife paths. (b) Gripper path and injected records. (c) Each record and the sub-goal it completes. (d) Placement error. (e) Steps per stage.
Conclusion
We presented F, which turns a WAM’s own successful and failed rollouts into usable experience through Form, Fit and Fusion. On both benchmarks, it makes Cosmos 3 Nano succeed more often and in fewer steps, even on RoboLab, where it is never trained, whereas the same records given directly hurt it. What makes experience good is thus not where it comes from or how much of it is supplied, but what is stored, when it is used and where it enters the policy.
LLM Usage Statement
Large language models were used to revise wording, correct grammar and improve readability. OpenAI Codex assisted code development, including drafting code, diagnosing errors and suggesting fixes during debugging.
Proofs for Form and Fit
All logarithms are natural, so information is in nats.
Proof of Theorem 1. Cosmos 3 supplies a candidate , and a separate check bounds its information loss. Mutual information is at most entropy, entropy is subadditive, and conditioning on less evidence cannot decrease it. Since is a function of ,
Fano’s inequality [36] bounds each term. With the verifier’s error indicator, coding costs at most nats. When , the prediction determines . When , at most alternatives remain. Thus , which gives Eq. (4). Here is fixed by the goal-level split, and the verifier judges each clip from its frames and state trace alone, so should be measured against labels independent of the Reasoner. On the event that every verifier prediction is correct, is a function of and its conditional entropy vanishes; by a union bound this event has probability at least . For binary goals the Fano bound is informative only for small ; its role is to show that the loss vanishes as .
Estimating the bound. Fix the verifier, then label held-out rollouts independently. By Hoeffding’s inequality and a union bound over the goals, with probability at least every is at most . The function is increasing on , since its derivative is . On that event, and when each , the bound holds with in place of .
Cost of compression. Let be the minimum expected log loss for predicting from [37], and the Bayes risk of any decision with loss in that depends on the rollout only through .
Proof of Theorem 2. Log loss. For any predictor given evidence and context , its expected log loss is plus the expected KL divergence from to . The true conditional distribution attains the minimum.
the minimum. Because , conditioning on in addition to leaves this distribution unchanged. Thus
This proves Eq. (12). Equality to zero gives agreement of the two predictive distributions almost surely.
Decision risk. Write and . Every decision rule based on is also a rule based on , so . Let and minimize the expected loss under and , respectively. For a loss with values in , expectations under two distributions differ by at most their total variation distance . Hence, for every rollout,
The expectation of the left side is and that of the first term on the right is . By Pinsker’s inequality and Jensen’s inequality [36],
which proves Eq. (13) and completes the proof of Theorem 2.
Shortest supported chain. We link verified clip to clip when and the effect of satisfies the precondition of [15]. With cost and a source node 0 linked to the clips whose preconditions hold initially, backtracking from the goal clip with the smallest in the recursion of § 3.2 yields the chain with the fewest retained steps among the candidate clips.
Minimality. Use for an empty predecessor set. Every path ending at clip consists of a path to an admissible predecessor and the final clip of cost . Conversely, appending to any such path yields an admissible path. Since the graph is ordered in time, induction gives the minimum cost at every node. Nonoverlap makes this cost exactly the number of retained candidate steps. For ordered intermediate goals, the same argument holds on nodes augmented with the number of goals already reached.
Proof of Proposition 1. Condition on , so that , and are fixed. The best decision given injects when and attains . The gated rule differs from it in two cases. If and , the rule bypasses and loses . If and , the rule injects and loses . In every case the loss is at most . Taking the expectation over decision points gives Eq. (8). Since , the first two terms are at least , the values of always and never injecting. For the regression in Eq. (7), the squared loss is minimized by the conditional mean , because and are indicators of completion under the two choices.
Implementation details
The encoder extracts image and text features with a fixed visual VAE and Qwen3-Embedding-0.6B. Each modality has its own projection to 256 dimensions, learned temporal positions and a two-layer, four-head Transformer, and a learned query pools each sequence into one token. The experience library is built from rollouts on 50 MolmoSpaces instances that are disjoint from the 100 evaluation instances. The cross-episode and same-window constructions yield 100 examples each and cover 160 distinct query windows across 36 task instances. Fit training runs five epochs, 125 updates with a global batch of eight, and saves a checkpoint every 25 updates (Figure 5). Retrieval searches only this library, never the evaluation instances, with similarity threshold , shared by the Similar check, the threshold control and the direct-input runs of Table 2, and Fusion uses . The retrieval score uses and , and Fit training uses and a bypass fraction . When several records pass all checks, as in Figure 1b, we take an average of corresponding modality tokens, weighted by the scores , before both readers. The same-window reconstruction examples are used only for training, with support from the window that holds the target actions.
Additional experimental protocols
Policy and benchmark protocols. The OpenPI baselines and DreamZero receive wrist and exterior views. OpenPI policies replan every eight actions, initializing the sampling key to 42 once per episode and advancing it with each call. DreamZero replans every 24 actions. DreamZero receives the same exterior view in both exterior-camera slots. The other baseline rows in Table 1 use published RoboLab Default-instruction success rates from ten episodes per task (1,200 total). The leaderboard records the Edge submission on RoboLab v0.2.0 and the other reported baselines on v0.1.0 [28]. The local RoboLab evaluation of native Nano and of all our controls runs v0.2.1 on A100 GPUs and uses default instructions. Each configuration is run with seeds 0, 1 and 2, and we report the seed whose success is closest to the three-seed mean, i.e., one full pass of 120 episodes, so that episodes can be paired across policies. An episode succeeds only with both simulator success and a score of 1.
References
- [1]Wenxuan Guo, Yingping Liang, Zhaoguo Wang, Ziyuan Li, Angyuan Ma, Zhenyu Wu, Chengkun Wang, Bingyao Yu, Ze Chen, Borui Zhang, Xiwei Xu, Meng Zhang, Erjin Zhou, and Jiewen Lu. LoopWAM: Closed-loop generalist policy improvement from deployment rollouts. arXiv preprint arXiv:2610.18873, 2026. 1, 2arxiv.org/abs/2610.18873
- [2]Fu Chen, Xin Ding, Bingjia Huang, Xiangyu Li, Mingju Wang, Jiawei He, Kun Li, Wei Sun, Yunxin Liu, Hao Wu, and Ting Cao. Zeva: In-context causal learning for generalizable embodied manipulation. arXiv preprint arXiv:2608.30880, 2026. 1, 2, 3, 6arxiv.org/abs/2608.30880
- [3]Jiming Zhou, Qihang Zhang, Gangwei Xu, Cunxin Fan, Yujie Zhao, Ruilin Wang, Yiming Luo, Shuai Yang, Xing Zhu, Yujun Shen, Junwei Liang, and Yinghao Xu. Zero-WAM: In-context world-action modeling from human videos for open-ended task generalization. arXiv preprint arXiv:2608.26103, 2026. 2, 3, 4arxiv.org/abs/2608.26103
- [4]Jeongeun Park, Taekyung Kim, Dongyoon Han, and Sangdoo Yun. Learning transferable world-action models from task-paired human videos. arXiv preprint arXiv:2610.14351, 2026. 2, 3, 4arxiv.org/abs/2610.14351
- [5]Sanghwan Jang, Minjin Jeon, Minsoo Kim, Seongjin Choi, Dongha Kim, and Hwanjo Yu. R-VA-LA: Retrieval-augmented VLA for test-time adaptation. In Proceedings of the 43rd International Conference on Machine Learning, 2026. 2, 3
- [6]Qi Lyu, Jiahua Dong, Hao Shen, Xudong Wang, Hongyuan Yu, Baichen Liu, Henghui Ding, Zhi Han, Nicu Sebe, Ivan Laptev, Fahad Shahbaz Khan, and Salman Khan. Learning skills from historical action trajectories: Action experience dictionary for world action models. arXiv preprint arXiv:2609.40219, 2026. 2, 3, 6
- [7]NVIDIA. Cosmos 3: Omnimodal world models for physical AI. arXiv preprint arXiv:2606.02800, 2026. 2, 3, 6
- [8]Agnar Aamodt and Enric Plaza. Case-based reasoning: Foundational issues, methodological variations, and system approaches. AI Communications, 7(1):39–59, 1994. 2DOI
- [9]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, 2023. 2arxiv.org/abs/2303.11366
- [10]Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 19632–19642, 2024. 2arxiv.org/abs/2308.10144
- [11]Yang. SkillRubricsRL: Evolving skill-grounded agents with rubric-guided reinforcement learning. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 9870–9885, 2026. 2DOI
- [12]Gabriel Sarch, Lawrence Jang, Michael J. Tarr, William W. Cohen, Kenneth Marino, and Katerina Fragkiadaki. VLM agents generate their own memories: Distilling experience into embodied programs of thought. In Advances in Neural Information Processing Systems, 2024. 2arxiv.org/abs/2406.14596
- [13]Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 627–635, 2011. 2
- [14]Ali Amin, Rachelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace B. Connors, James Darpinian, Karan Dhabalia, Jared Di Carlo, Danny Driess, Michael Robert Equi, Adnan Esmail, Yunhao Fang, Chelsea Finn, Catherine Glossop, Thomas Godden, Ivan Goryachev, Lachy Groom, Hunter Hancock, Karol Hausman, Gashon Hussein, Brian Ichter, Szymon Jakubczak, Rowan Jen, Tim Jones, Benjamin Katz, Liyiming Ke, Chandra Kuchi, Marinda Lamb, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Yao Lu, Vishnu Mano, Mohith Muthukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Charvi Sharma, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, Will Stoeckle, Alexander Swerdlow, James Tanner, Marcel Torne, Quan Vuong, Anna Walling, Haohuan Wang, Blake Williams, Sukwon Yoo, Lili Yu, Ury Zhilinsky, and Zhiyuan Zhou. π₀.₆: A VLA that learns from experience. In Proceedings of Robotics: Science and Systems, 2026. 2
- [15]Richard E. Fikes and Nils J. Nilsson. STRIPS: A new approach to the application of theorem proving to problem solving. Artificial Intelligence, 2(3–4):189–208, 1971. 2, 11
- [16]Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. In Proceedings of the 37th Annual Allerton Conference on Communication, Control, and Computing, pages 368–377, 1999. 2, 4
- [17]Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969–7992, 2023. 3
- [18]Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In International Conference on Learning Representations, 2024. 3arxiv.org/abs/2310.11511
- [19]Songhua Yang, Ziyu Liu, Xuetao Li, Ruqi Xiao, Kangxin Zhu, and Miao Li. ICL-VLA: In-context imitation with spatiotemporally aligned demonstrations for vision-language-action models. arXiv preprint arXiv:2609.07581, 2026. 3, 4
- [20]Jeongeun Park, Juhan Park, Taekyung Kim, Sungjoon Choi, Dongyoon Han, and Sangdoo Yun. Retrieve, don’t retrain: Extending vision-language-action models to new tasks at test time. In Advances in Neural Information Processing Systems, 2026. 3arxiv.org/abs/2606.15631
- [21]Lin Li, Qihang Zhang, Yiming Luo, Shuai Yang, Ruilin Wang, Luyao Zhang, Mingrui Yu, Zelin Gao, Nan Xue, Boyu Zhou, Xing Zhu, Mingyu Ding, Yujun Shen, and Yinghao Xu. Causal world modeling for robot control. In Proceedings of Robotics: Science and Systems, 2026.arxiv.org/abs/2601.21998
- [22]Pengfei Zhou, Shengcong Chen, Di Chen, Jiaxu Wang, Rongjun Jin, Bingwen Zhu, Yike Pan, Songen Gu, Kuan-ning Wang, Shufeng Nan, Xingyu Qiu, Chenhao Qiu, Pu Yang, Yunuo Cai, Jianxiang Gao, Yifan Li, Yanwei Fu, Xiangyu Yue, Zhi Chen, and Jianlan Luo. 7o-WM: A unified video-action world model for robotic manipulation. arXiv preprint arXiv:2606.01027, 2026.
- [23]Tianyuan Yuan, Zibin Dong, Yicheng Liu, and Hang Zhao. Fast-WAM: Do world action models need test-time future imagination? arXiv preprint arXiv:2603.16666, 2026.
- [24]Sizhe Zhao, Haozhe Xie, Weiyu Zhao, Chenhu Zhang, Huan Wang, Chenyang Wang, Qinglin Liu, and Shengping Zhang. Memory as plans: World-action modeling with memory-grounded planning. arXiv preprint arXiv:2609.11561, 2026.
- [25]Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.02747
- [26]Marius Memmel, Jacob Berg, Bingqing Chen, Abhishek Gupta, and Jonathan Francis. STRAP: Robot sub-trajectory retrieval for augmented policy learning. In International Conference on Learning Representations, 2025.
- [27]Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, 2021.
- [28]RoboLab Team. RoboLab-120 leaderboard. https://research.nvidia.com/labs/srl/projects/robolab/, 2026. Accessed November 2026.
- [29]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, Laura Smith, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. π₀: A vision-language-action flow model for general robot control. In Proceedings of Robotics: Science and Systems, 2025.
- [30]Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. FAST: Efficient action tokenization for vision-language-action models. In Proceedings of Robotics: Science and Systems, 2025.
- [31]Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. π₀.₅: A vision-language-action model with open-world generalization. In Proceedings of the Conference on Robot Learning, pages 17–40, 2025.
- [32]Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, Sihyun Yu, George Kurian, Suneel Indupuru, You Liang Tan, Chuning Zhu, Jiannan Xiang, Ayaan Malik, Kyungmin Lee, William Liang, Nadun Ranawaka, Jiasheng Gu, Yinzhen Xu, Guanzhi Wang, Fengyuan Hu, Avnish Narayan, Johan Bjorck, Jing Wang, Gwanghyun Kim, Dantong Niu, Ruijie Zheng, Yuqi Xie, Jimmy Wu, Qi Wang, Ryan Julian, Danfei Xu, Yilun Du, Yevgen Chebotar, Scott Reed, Jan Kautz, Yuke Zhu, Linxi Fan, and Joel Jang. World action models are zero-shot policies. arXiv preprint arXiv:2602.15922, 2026.
- [33]Yejin Kim, Wilbert Pumacay, Omar Rayyan, Max Argus, Winson Han, Eli VanderBilt, Jordi Salvador, Abhay Deshpande, Rose Hendrix, Snehal Jauhari, Shuo Liu, Nur Muhammad Mahi Shafullah, Maya Guru, Ainaz Eftehkar, Karen Farley, Donovan Clay, Jiafei Duan, Arjun Guru, Piper Wolters, Alvaro Herrasti, Ying-Chun Lee, Georgia Chalvatzaki, Yuchen Cui, Ali Farhadi, Dieter Fox, and Ranjay Krishna. MolmoSpaces: Large-scale open ecosystem for robot manipulation and navigation. In Proceedings of Robotics: Science and Systems, 2026.
- [34]Abhay Deshpande, Maya Guru, Rose Hendrix, Snehal Jauhari, Ainaz Eftehkar, Rohun Tripathi, Max Argus, Jordi Salvador, Haoquan Fang, Matthew Wallingford, Wilbert Pumacay, Yejin Kim, Quinn Pfeifer, Ying-Chun Lee, Piper Wolters, Omar Rayyan, Minglong Zhang, Jiafei Duan, Karen Farley, Winson Han, Eli Vanderbilt, Dieter Fox, Ali Farhadi, Georgia Chalvatzaki, Dhruv Shah, and Ranjay Krishna. MolmoBOT: Large-scale simulation enables zero-shot manipulation. arXiv preprint arXiv:2603.16861, 2026.
- [35]Xuning Yang, Rishit Dagli, Alex Zook, Hugo Hadfield, Ankit Goyal, Stan Birchfield, Fabio Ramos, and Jonathan Tremblay. RoboLab: A high-fidelity simulation benchmark for analysis of task generalist policies. In Proceedings of Robotics: Science and Systems, 2026.
- [36]Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley-Interscience, 2 edition, 2006.
- [37]Jiantaо Jiao, Thomas A. Courtade, Kartik Venkat, and Tsachy Weissman. Justification of logarithmic loss via the benefit of side information. IEEE Transactions on Information Theory, 61(10):5357–5365, 2015.




