WorksComputer Science
Can Astra Solve RoboMME without Breaking the Bank?
Abstract
Robotic manipulation often requires remembering information that is no longer visible, such as an object's location before occlusion or the order of a demonstrated action sequence. A large vision-language planner can read this history, but calling it before every action chunk is slow and costly. We present a three-tier system for the RoboMME benchmark. GPT-6 Astra (System 2) selects a grounded subtask from stored visual history, a vision-language-action policy (System 1) executes it, and a fine-tuned Qwen3-VL-4B monitor (System 1.5) checks after each action chunk whether the subtask is finished. The instruction is kept until the monitor reports completion or a task-specific event occurs, while the policy keeps acting on new observations. On RoboMME, the system succeeds in 79.13% of episodes with an average of 3.63 Astra calls and 80.84 seconds per episode. This is 34.62 points above the best published non-oracle baseline and 4.95 points below GroundSG+Oracle, which gives the low-level policy ground-truth subgoals.
Introduction
Manipulation can depend on information that is no longer visible: an object’s identity before it is covered, the order of a demonstrated sequence, or the number of repetitions already completed. RoboMME [1] tests these dependencies with 16 tasks in four suites: Counting, Permanence, Reference, and Imitation. A common design for such tasks is hierarchical, with a vision-language model (VLM) choosing the next subtask and a vision-language-action (VLA) policy executing it.
In our development runs, small high-level VLMs often misread RoboMME’s video inputs and issued wrong subtasks, so we use GPT-6 Astra [2] as the planner. Calling it before every action chunk, however, is expensive and mostly redundant: a chosen subtask usually remains valid for several chunks, and whether it has finished can be judged from the last few frames.
We therefore place a small learned monitor between the planner and the policy. Astra is called at the start of an episode and again only when the monitor reports completion or a task-specific event occurs. In between, the VLA keeps executing the same instruction from fresh observations. We evaluate the full system on RoboMME and report success rates, planner usage, and failure cases.
Related work. RoboMME compares symbolic and perceptual memory interfaces [1]. SAM2Act adds a visual memory bank [3], MemER retrieves keyframes [4], and MEM combines short-horizon video with long-horizon text memory [5]. Hierarchical systems such as -VLA use execution memory and test-time reasoning to select subtasks [6]. Our system instead sends task-dependent history to an external planner at each call and lets a local monitor decide when to call it.
Three-Tier System
Figure 1(a) shows the three components. Let be the episode instruction, the observation at environment step , and the th grounded subtask.
Figure 1. Planning, action, and monitoring at different frequencies. (a) Astra uses task history to choose a subtask instruction . The VLA executes actions in 16-step chunks; the environment returns observations , and a local monitor checks progress after each chunk. The instruction is kept until an event requires a new high-level decision (§ 2.4). (b) Overall success on RoboMME compared with two published baselines from Table 1.
System 2: History and Grounded Planning
At each planning event, Astra produces one executable instruction:
where is the history passed to the planner. It has three parts: the demonstration video when the task provides one, selected execution frames that may show objects now hidden, and a text progress record that lists issued subtasks and monitor-confirmed subtasks separately. Repeated actions are recorded as separate entries, so their order and count remain available. Each request also includes the current front-camera image and the image at execution start. Astra keeps no state between calls; all history is sent with each request.
Which history is sent and which output templates are allowed are configured per task: nine tasks receive the demonstration video and four receive dense execution history (Appendix § 6; full prompts in Appendix § 8). When a target must be localized, the instruction contains a pixel coordinate in the current image. The output is checked against the templates before it reaches the VLA.
System 1: Low-Level Action Execution
We use RoboMME’s grounded-subgoal policy [7]:
where includes camera images and robot state. Each chunk runs 16 environment steps. The same can guide several consecutive chunks, and each chunk is predicted from a new observation, so control remains closed-loop while Astra is idle.
System 1.5: Visual Progress Monitoring
The monitor is Qwen3-VL-4B-Instruct [8], fine-tuned with LoRA on RoboMME demonstrations to predict a binary event for the active subtask:
The input has ten images: eight recent front-camera frames taken three steps apart (spanning up to 21 steps), the front image from when the current instruction was issued, and the current wrist image. This reference image is reset whenever an instruction is issued, even if its text repeats an earlier one, so each repetition is compared with its own starting state. In most tasks the event marks completion. In StopCube it signals an approaching target, and in ButtonUnmaskSwap Astra verifies a reported completion of the second button press (Appendix § 6.§ 3).
Event-Triggered Coordination
Astra is called once at initialization, and the monitor runs after every action chunk. A negative decision keeps the current instruction for another chunk. A completion event adds the subtask to the progress record and requests the next one. The StopCube timing event and the ButtonUnmaskSwap verification event send the execution history to Astra before the system advances. Figure 2 shows instruction reuse across one completion event; Appendix § 5 gives full walkthroughs.
Figure 2. Instruction reuse between planning events. Three frames from a successful recorded episode. A negative check at step 64 keeps . Completion at step 128 triggers Astra’s second call and the next instruction ; the positive check shown refers to . At step 160, a negative check continues . Details are in Appendix § 5.
Inference scheduling. As in RoboMME’s official two-system evaluation, the simulator pauses while Astra runs, so planning time adds to wall-clock duration but not to the step budget. In preliminary runs without pausing, the VLA used up the 1,300-step budget in most episodes before Astra returned.
Experiments
Setup. We evaluate on the official RoboMME test set (50 episodes per task) with a 1,300-step budget per episode and report simulator task success. Baseline numbers are the published RoboMME results [1]: [7] with the current observation only; FrameSamp+Modul and SAM2Act+ [3] with perceptual memory; and MemER [4], GroundSG+QwenVL, and GroundSG+Gemini with a high-level planner. The last two use a fine-tuned Qwen3-VL-4B [8] and a prompted Gemini 2.5 Pro [9]. GroundSG+Oracle gives ground-truth subgoals to the low-level policy and serves as a reference for grounded execution. Astra uses medium reasoning effort; the VLA, monitor, and simulator share one NVIDIA L40S GPU.
Task success. The system reaches 79.13% overall success (Table 1): 94.00% on Permanence, 92.00% on Reference, 69.50% on Counting, and 61.00% on Imitation. Overall success is 34.62 points above FrameSamp+Modul, the best non-oracle baseline, and 4.95 points below GroundSG+Oracle. The gap to the oracle is largest in Counting (14.36 points, versus 3.17 in Reference and 2.98 in Imitation), and it comes mostly from StopCube: the other three Counting tasks average 86.67%.
| Method | Counting | Permanence | Reference | Imitation | Overall |
| [7] | 28.78 | 17.00 | 17.16 | 8.78 | 17.93 |
| FrameSamp+Modul [1] | 65.22 | 25.11 | 36.33 | 51.39 | 44.51 |
| SAM2Act+ [3] | 35.33 | 26.00 | 16.83 | 7.33 | 21.37 |
| MemER [4] | 48.83 | 53.17 | 38.00 | 29.50 | 42.38 |
| GroundSG+QwenVL [1] | 38.00 | 39.34 | 31.56 | 21.89 | 32.70 |
| GroundSG+Gemini [1] | 12.75 | 15.75 | 11.25 | 6.50 | 11.56 |
| Three-tier Astra (ours) | 69.50 | 94.00 | 92.00 | 61.00 | 79.13 |
| GroundSG+Oracle [1] | 83.86 | 93.31 | 95.17 | 63.98 | 84.08 |
Table 1. Success rates (%) on RoboMME. Baselines are published RoboMME results [1]. GroundSG+Oracle uses ground-truth subgoals. Bold marks the best non-oracle result.
Failure cases. Of the 167 unsuccessful episodes, 103 fail and 64 time out. StopCube and InsertPeg each succeed in only 9/50 episodes and together account for 82 of them. StopCube requires pressing a button on a specific passage of a moving cube; all 41 of its unsuccessful episodes are failures rather than timeouts. InsertPeg requires contact-rich insertion; 36 of its 41 unsuccessful episodes time out. We have not yet attributed these failures to planning, monitoring, or execution errors. Appendix § 7 lists outcomes for all 16 tasks.
Planner usage. The system averages 3.63 Astra calls per episode, counting both planning and visual-verification requests, and 80.84 seconds of wall-clock time. The recorded trajectories contain 24.62 action chunks on average, so calling Astra before every chunk would require about 24.62 calls along the same trajectories. Writing these averages as and ,
This figure assumes the recorded trajectories; a controller that replans at every chunk would follow different trajectories and could reach a different success rate. Call count is also only a proxy for cost, because request size grows with the supplied history and the monitor runs after every chunk.
Limitations and Conclusion
Limitations. We evaluate only the full system. Ablations that remove the history, replace Astra with a smaller planner, or replace the monitor with fixed-interval replanning are needed to separate the contributions of memory, planner capability, and monitoring. The GroundSG baselines differ from our system in planner, inputs, and replanning schedule, so they are not matched comparisons. We did not run an always-replan controller or measure token usage and monetary cost. We also did not measure the monitor’s closed-loop error rate; a false completion corrupts the progress record used by later calls. History selection, output templates, and event rules are set per task, which limits transfer to new tasks. Finally, the simulator pauses during planning, so we have not tested scenes that change while Astra reasons.
Conclusion. On RoboMME, the three-tier system reached 79.13% success while calling Astra 3.63 times per episode, compared with 24.62 action chunks. The ablations and cost measurements above would show which parts of the system account for this result.
Visual Walkthroughs
The following figures show selected frames from two successful hard test episodes. Each execution panel pairs the front-camera view with the active Astra instruction, the latest monitor decision, and the cumulative number of Astra calls.
Figure 3. Planning, action, and monitoring during execution. BinFill, hard test episode 023. The first pickup instruction is kept at steps 0 and 64. Completion at step 128 triggers the placement instruction. A negative check at step 160 continues that instruction; completion at step 192 triggers the next pickup. Green badges mark completion events, red badges mark continued execution, and purple marks initialization. At a completion event, the displayed instruction is the next instruction issued by Astra; the monitor decision refers to the preceding subtask.
Figure 4. From demonstration history to grounded action. VideoUnmaskSwap, hard test episode 035. The upper row shows selected demonstration frames: visible objects, covering, and the later scene after container motion. Astra uses the demonstration history to select the container in the current execution scene. The lower row shows the grounded pickup instruction, continued execution under a negative monitor decision, and completion at step 112 followed by the next instruction. Demonstration labels are video frame indices; execution labels are environment steps.
Implementation Details
Planner Inputs
Image history is sent as sheets of up to 16 frames in a grid, each labeled with its frame index.
Nine tasks receive the demonstration video: VideoUnmask, VideoUnmaskSwap, VideoRepick, VideoPlaceButton, VideoPlaceOrder, MoveCube, InsertPeg, PatternLock, and RouteStick. Four receive dense execution history: ButtonUnmask, ButtonUnmaskSwap, PickHighlight, and StopCube. In the first three, this history stops at the monitor-confirmed button press; only the current image is updated after that. StopCube receives every execution frame so that Astra can count the cube’s passages. The remaining tasks (BinFill, PickXtimes, SwingXtimes) rely on the text progress record.
Monitor Training
The monitor is trained on 71,835 examples built from RoboMME demonstrations, labeled with the recorded subtask boundaries. For StopCube waiting subtasks, positive labels instead mark the interval just before the cube reaches the target. We fine-tune with LoRA (rank 16) on the language model, keeping the vision encoder frozen, for two epochs.
Task-Specific Events
StopCube requires stopping a moving cube on a specified passage through the target. During a waiting subtask, the monitor fires about 16–32 environment steps before the cube reaches the target. Each trigger calls Astra, which counts the actual passages in the execution history and decides whether to keep waiting or press the button. In ButtonUnmaskSwap, when the monitor reports that the second button press is complete, Astra checks this against the execution history before the system advances; if Astra rejects it, the same instruction continues. Both prompts are in Appendix § 9.
Deployment
Astra runs with medium reasoning effort through Codex CLI. Calls are spaced at least 20 seconds apart. We use the grounded-subgoal checkpoint released with RoboMME.
In the first run, a bug in our planner–simulator bridge interrupted six of the 800 test episodes. We fixed the bug and reran only those six; the other 794 results are from the first run.
Complete Per-Task Results
| Suite | Task | Success | Fail | Timeout | Rate (%) |
| Counting | BinFill | 41 | 6 | 3 | 82.00 |
| StopCube | 9 | 41 | 0 | 18.00 | |
| PickXtimes | 48 | 1 | 1 | 96.00 | |
| SwingXtimes | 41 | 9 | 0 | 82.00 | |
| Permanence | ButtonUnmask | 50 | 0 | 0 | 100.00 |
| VideoUnmask | 46 | 1 | 3 | 92.00 | |
| VideoUnmaskSwap | 50 | 0 | 0 | 100.00 | |
| ButtonUnmaskSwap | 42 | 6 | 2 | 84.00 | |
| Reference | PickHighlight | 42 | 0 | 8 | 84.00 |
| VideoRepick | 43 | 3 | 4 | 86.00 | |
| VideoPlaceButton | 50 | 0 | 0 | 100.00 | |
| VideoPlaceOrder | 49 | 0 | 1 | 98.00 | |
| Imitation | MoveCube | 40 | 4 | 6 | 80.00 |
| InsertPeg | 9 | 5 | 36 | 18.00 | |
| PatternLock | 44 | 6 | 0 | 88.00 | |
| RouteStick | 29 | 21 | 0 | 58.00 | |
| Overall | 633 | 103 | 64 | 79.13 |
Table 2. Per-task outcomes on the RoboMME test set (50 episodes per task).
Complete Task Prompt Specification
Each planner prompt consists of the shared prefix (Section D.1) followed by a task-specific part. BinFill, PickXtimes, and SwingXtimes replace the prefix’s GROUNDING CHECK paragraph, as shown in their sections. Appendix E gives the text appended at runtime, the ButtonUnmaskSwap verification prompt, and the monitor prompts. Braces and angle brackets are placeholders that appear literally in the prompts.
Shared Planner Prefix
You are the high-level planner for a tabletop robot. Produce the NEXT single grounded subgoal for the named task.
INPUT CONTRACT
The task instruction specifies this episode’s goal. Completed subgoals are confirmed past actions, ordered oldest first. Each entry is one completed occurrence; preserve repeated actions when determining progress. The last completed entry is already finished, not the action to execute again.
Demonstration images, when supplied, show an observation phase that has already ended. Infer what must be reproduced, but do not replay setup, reset, or irrelevant demonstration actions.
Execution-memory images, when supplied, are chronological past observations of this episode. They are evidence, not future observations. Recover identities through occlusion and motion. The separately supplied CURRENT image is the execution state now.
Sheets are read left-to-right, top-to-bottom, then page-by-page. Each tile is a full 256x256 front-camera image; frame numbers label time, not objects. The current image is not a sheet.
OUTPUT CONTRACT
Output exactly one line of English grounded-subgoal text using one of this task’s templates. Replace bracketed placeholders with episode-specific values. Output no bullet, quotes, JSON, explanation, confidence, completion condition, or future plan.
Preserve the templates’ action granularity. A pick and a placement are separate calls. Some supported templates describe a whole push or curved movement; do not decompose these into invented actions.
Every <y, x> point is an integer row then column in the full 256x256 CURRENT image, each in 0..255, origin top-left. Never use sheet coordinates, [x,y], normalized 0..1000 coordinates, or 3D coordinates. For stationary entities under the fixed camera, a position established in the execution-start image remains usable during current occlusion, unless later evidence shows displacement.
Ground the entity specified by the template: button cap, manipulated object, grasp endpoint, or destination. Do not replace a grasp endpoint with an object’s center. Re-localize moved objects in the current image; old history coordinates are not current locations. Prefer current visible evidence. For buttons, target markers, and the bin opening, consult the fixed execution-start image when current evidence is occluded or ambiguous; do not abstain solely because the robot now occludes a previously observed stationary entity. Cubes and containers may move: use the start image for initial identity only, and establish their current location from current/recent evidence. Never assume a moved object remains at its initial coordinate.
Do not add a coordinate to a template without one. Do not invent an extra stop/DONE subgoal; the environment terminates successful episodes.
Use task rules and supplied evidence, not remembered seed-specific answers. Do not invent missing visual events. If the required action cannot be justified, state that briefly rather than fabricate it; this is a failed/unresolved planning output and must not be forwarded to a VLA.
GROUNDING CHECK
Before returning the single output line, check that the action, intended entity, and <y, x> location agree. Check row/column order and that the point lies on that entity in the supporting full-size image. For any coordinate-bearing template, identify the entity named by that template; do not substitute another nearby object. Perform this check internally; do not change the output format.
EXAMPLE CONVENTION
The small text examples below illustrate decision rules and output grammar only. Their colors, counts, and coordinates are synthetic, not evidence for the current episode. The current images and instruction always take precedence.
BinFill
Replace the shared GROUNDING CHECK paragraph with: Before returning the single output line, check that the action, intended entity, and <y, x> location agree. Check row/column order and that the point lies on that entity in the supporting full-size image. Bin-placement coordinates must identify the bin opening. Stop coordinates must identify the button cap. Pickup coordinates must identify an eligible cube in its current location. Perform this check internally; do not change the output format.
TASK: BinFill
TASK RULES
Fill the bin with the requested NUMBER of DISTINCT cubes of each requested color, then press the button. This is not repeatedly picking one cube. Pair each completed pickup with its subsequent bin placement to count delivered cubes. If holding a cube after a confirmed pickup, place it in the bin next. Otherwise choose a still-visible cube of a color whose quota is incomplete. Never retrieve a cube already delivered to the bin. The instruction does not impose a color or within-color object ordering: any unmet quota and any eligible same-color cube is valid. For deterministic choices, use colors in instruction order and the leftmost eligible image object. Ordinals count cubes of that color delivered/being delivered, not all colors together. Press only after every quota is fulfilled. The bin point refers to its opening/destination.
SUPPORTED TEMPLATES
pick up the {ordinal} {color} cube at <y, x>
put it into the bin at <y, x>
press the button at <y, x>
ILLUSTRATIVE TEXT EXAMPLE
Instruction: deliver two blue cubes. History: pickup blue; put into bin. Current: another blue cube centered at row 90, column 150. Output: pick up the second blue cube at <90, 150>
StopCube
TASK: StopCube
TASK RULES
Prepare above the button first without pressing. Then remain static while the moving cube makes the required visits, and press to stop it on the requested visit. Ground the preparation point on the button, but the final template’s point on the TARGET, matching the dataset grammar. A completed ’remain static’ is a completed waiting segment, NOT proof of one target crossing. Do not equate their count with visits or infer an absolute simulator schedule from frame IDs. With only a current still and waiting history, the required crossing may be unidentifiable; do not claim accurate timing from this input. Use remain static while evidence does not justify pressing. No extra event counter or hidden state is supplied.
SUPPORTED TEMPLATES
move to the top of the button at <y, x> to prepare
remain static
press the button to stop the cube on the target at <y, x>
ILLUSTRATIVE TEXT EXAMPLE
History: preparation completed. Current still shows the cube away from the target; the requested visit is not established. Output: remain static
PickXtimes
Replace the shared GROUNDING CHECK paragraph with: Before returning the single output line, check that the action, intended entity, and <y, x> location agree. Check row/column order and that the point lies on that entity in the supporting full-size image. Placement coordinates must identify the purple target marker, never the stop button or the cube. Stop coordinates must identify the button cap, never a cube or target marker. Perform this check internally; do not change the output format.
TASK: PickXtimes
TASK RULES
One repetition is a completed PICK followed by PLACE onto the target. Read the requested color and repetition count from the instruction. With no completed pickup, pick the specified cube for the first time. After each pickup, place it onto the target; after each placement, pick the same cube again if repetitions remain. Count only this execution’s confirmed history, never frames or demonstration actions. The final placement is required before pressing stop. Re-localize the cube because after the first repetition it is on the target. Never merge pick and place, increment on a pickup twice, or stop while holding the cube.
SUPPORTED TEMPLATES
pick up the {color} cube at <y, x> for the {ordinal} time
place the {color} cube onto the target at <y, x>
press the button at <y, x> to stop
ILLUSTRATIVE TEXT EXAMPLE
Instruction: pick and place a blue cube three times. History: first pick; place; second pick. Target center in current image: row 120, column 80. Output: place the blue cube onto the target at <120, 80>
SwingXtimes
Replace the shared GROUNDING CHECK paragraph with: Before returning the single output line, check that the action, intended entity, and <y, x> location agree. Check row/column order and that the point lies on that entity in the supporting full-size image. Movement coordinates must identify the specified target marker, preserving this task’s right/left convention. Stop coordinates must identify the button cap, not either target marker or a cube. Perform this check internally; do not change the output format.
TASK: SwingXtimes
TASK RULES
Pick the requested cube ONCE, then keep holding it through the requested repetitions. Each back-and-forth repetition visits the RIGHT target and then the LEFT target, in that order. A completed right visit is half a repetition. After the final left visit, put the cube on the table, then press the button. Even if an instruction omits the intermediate release, use the standard put-down subgoal before pressing. The labels left/right use ROBOT perspective: in the fixed front image, robot-right is image-left and robot-left is image-right. Ground the target surface center, not an elevated gripper location. Repeated visits use their own ordinal within each side.
SUPPORTED TEMPLATES
pick up the {color} cube at <y, x>
move to the top of the right-side target at <y, x> for the {ordinal} time
move to the top of the left-side target at <y, x> for the {ordinal} time
put the {color} cube on the table
press the button at <y, x>
ILLUSTRATIVE TEXT EXAMPLE
Instruction: swing twice. History: pick; right first; left first; right second. Robot-left target center: row 100, column 180. Output: move to the top of the left-side target at <100, 180> for the second time
ButtonUnmask
TASK: ButtonUnmask
TASK RULES
First press the button. Use early execution memory to associate each visible colored cube with the container that subsequently covers it. After the press, pick the container hiding the FIRST requested color, not the cube itself. If a second color is requested, put down the first container as a separate subgoal, then pick the second container. Do not require the hidden cube to be visible now. Container positions do not exchange in this task; the original binding identifies which currently visible container to pick. Do not treat all similarly colored/identical containers as interchangeable. SUPPORTED TEMPLATES
press the button at <y, x>
pick up the container at <y, x> that hides the {color} cube
put down the container
ILLUSTRATIVE TEXT EXAMPLE
Instruction: press, reveal blue, then red. History: button press; pick container hiding blue. Output: put down the container
VideoUnmask
TASK: VideoUnmask
TASK RULES
The demonstration already showed which colored cube became hidden by each container. Recover that association from the video. At execution, pick the container hiding the first color requested by the instruction. Do not press a button or replay the covering phase. If a second color is requested, release the first container, then pick the second. Use the current scene to locate each container; the required object is the covering container, not the hidden cube. No swap is part of this task.
SUPPORTED TEMPLATES
pick up the container at <y, x> that hides the {color} cube
put down the container
ILLUSTRATIVE TEXT EXAMPLE
Instruction: reveal blue then green. History: picked container hiding blue. Output: put down the container
VideoUnmaskSwap
TASK: VideoUnmaskSwap
TASK RULES
From the demonstration, bind each visible cube color to its covering container and track that container through ALL subsequent swaps, preserving identity through crossing and occlusion. Use the final demonstrated correspondence, then localize the selected container in the CURRENT execution image. Never assume a requested color stayed at its initial image position, that swaps always reverse each other, or that final position alone identifies a hidden color. Pick the container for the first requested color; if a second is requested, put down the first and then pick the second. The video is already complete; do not prescribe a button press or another swap.
SUPPORTED TEMPLATES
pick up the container at <y, x> that hides the {color} cube
put down the container
ILLUSTRATIVE TEXT EXAMPLE
Demonstration establishes that the container hiding blue ends on the image right. Current tracked container center: row 105, column 190. History empty. Output: pick up the container at <105, 190> that hides the blue cube
ButtonUnmaskSwap
TASK: ButtonUnmaskSwap
TASK RULES
Press both distinct buttons before any container pickup. With no completed button press choose the RIGHT-MOST image button; label it first. After that press, choose the LEFTMOST image button and label it second. This right-then-left order is mandatory for this controlled comparison. Button order in language is the order you execute, not a hidden object index. Use execution memory to bind initially visible cube colors to container identities, then follow the complete sequence of container movements and swaps. A target may move repeatedly; do not assume the final ordering matches the initial ordering or infer a swap from endpoints alone. After both presses, pick the container NOW hiding the first requested color. If another color is requested, put down the held container and then pick the other tracked container. Re-ground each tracked identity in the current image.
SUPPORTED TEMPLATES
press the first button at <y, x>
press the second button at <y, x>
pick up the container at <y, x> that hides the {color} cube
put down the container
ILLUSTRATIVE TEXT EXAMPLE
History: first button pressed at <62, 170>. Current: distinct other button at row 60, column 80. Output: press the second button at <60, 80>
PickHighlight
TASK: PickHighlight
TASK RULES
First press the button. Recover the FULL set of cubes highlighted by white areas in the chronological execution memory; highlights can disappear from the current image. Pick each highlighted cube once, never a non-highlighted distractor. The instruction requests a set, so there is no hidden required order: choose any unpicked highlighted cube (leftmost current image position breaks ties). Use the history to exclude already picked identities, not all objects sharing a color. Release a held cube onto the table before picking another; no release is needed after the final pickup unless explicitly required by the actual task. Use first/second/etc. for multiple targets and the singular template for one. A white highlight is evidence marking the target set, not the cube’s color. Do not add a stop-button step unless the actual instruction explicitly requests it.
SUPPORTED TEMPLATES
press the button at <y, x>
pick up the highlighted cube at <y, x>, which is {color}
pick up the {ordinal} highlighted cube at <y, x>, which is {color}
place the cube onto the table
ILLUSTRATIVE TEXT EXAMPLE
Memory: blue and green cubes were highlighted. History: press; pick first highlighted blue cube. Another highlighted cube remains. Output: place the cube onto the table
VideoRepick
TASK: VideoRepick
TASK RULES
Identify the particular cube physically picked in the demonstration; keep tracking its identity through subsequent motion/swaps even when several cubes have the same color. The current position of that same identity is required. Execute the number of pickup-and-put-down repetitions specified by the instruction; ’again’ without a number means one execution repetition, not the number of demo movements. Count only completed execution pickups and releases. A pickup is followed by ’put it down’; after the final release press the button to finish. Do not interpret ’finally put it down’ as an extra repetition or count demonstrations as completed execution actions.
SUPPORTED TEMPLATES
pick up the correct cube at <y, x> for the {ordinal} time
put it down
press the button at <y, x> to finish
ILLUSTRATIVE TEXT EXAMPLE
Instruction: pick the demonstrated cube again, put it down, finish. History: first pickup; put it down. Current button at <65, 130>. Output: press the button at <65, 130> to finish
VideoPlaceButton
TASK: VideoPlaceButton
TASK RULES
Read whether the instruction asks for the placement immediately BEFORE or immediately AFTER the demonstrated button press. Track the requested-colored cube across the full demonstration and track the relevant target marker if targets move. BEFORE means its last relevant target placement before the button event; AFTER means its first relevant target placement after that event. Other colors’ placements and arbitrary nearest/last targets are not substitutes. At execution, pick the instructed cube, then place it on the recovered target. Do not replay the demonstration’s button press or its entire placement sequence. Localize the cube and destination in the current image.
TARGET IDENTITY AFTER DEMONSTRATED PLACEMENT
The HARD variant can move/swap the circular TARGET MARKERS after the demonstrated placements. This is a target-marker identity problem, not merely cube tracking. First identify the physical target used at the requested event. Then follow THAT target through the remaining video, including the late swap, to find its final current location. A target’s former screen position is not its identity. Review the video through its end even if the requested placement occurred near the beginning. Preserve the requested event’s target identity while updating its position. Ground the resulting target in the current image, not at its pre-swap location. The rule applies to every episode; it does not specify which target or side is correct.
SUPPORTED TEMPLATES
pick up the cube at <y, x>
place the cube onto the correct target at <y, x>
ILLUSTRATIVE TEXT EXAMPLE
Instruction: target immediately before the button. Demo order: requested cube onto A; button; requested cube onto B. History: pick completed. A is currently at <130, 70>. Output: place the cube onto the correct target at <130, 70>
VideoPlaceOrder
TASK: VideoPlaceOrder
TASK RULES
Read the requested ordinal and cube color. In the demonstration, enumerate actual completed target placements of THAT cube in temporal order, tracking both the cube and the target identities when objects move. Ignore placements of unrelated cubes and do not rank targets by spatial position. Re-visiting a target is still another placement event. Recover the target associated with the requested occurrence. In execution there are just the next standard pick and then placement subgoals, not a replay of all demonstrated placements. Ground in the current image.
TARGET IDENTITY AFTER DEMONSTRATED PLACEMENT
The HARD variant can move/swap the circular TARGET MARKERS after the demonstrated placements. This is a target-marker identity problem, not merely cube tracking. First identify the physical target used at the requested event. Then follow THAT target through the remaining video, including the late swap, to find its final current location. A target’s former screen position is not its identity. Review the video through its end even if the requested placement occurred near the beginning. Preserve the requested event’s target identity while updating its position. Ground the resulting target in the current image, not at its pre-swap location. The rule applies to every episode; it does not specify which target or side is correct.
SUPPORTED TEMPLATES
pick up the cube at <y, x>
place the cube onto the correct target at <y, x>
ILLUSTRATIVE TEXT EXAMPLE
Instruction: place blue on its second demonstrated target. Demo: blue onto A; red onto C; blue onto B. History: pick completed. B currently at <110, 180>. Output: place the cube onto the correct target at <110, 180>
MoveCube
TASK: MoveCube
TASK RULES
Infer WHICH manipulation method the demonstration uses, then reproduce that method in the current layout: (1) grasp a peg and hook the cube with it; (2) close the gripper and directly push; or (3) pick up and place the cube. Do not choose an easier different method. For peg use, identify the demonstrated graspable peg end and ground that endpoint, not the rod midpoint. Pick the peg first, then issue the complete supported hook subgoal with cube and destination points. For direct pushing the close-and-push template is one supported subgoal. For pick-and-place use two separate subgoals. Ignore demonstration resets and re-localize every current point.
SUPPORTED TEMPLATES
Pick up the peg at <y, x>
Hook the cube at <y, x> to the target at <y, x> with the peg
Close the gripper and push the cube at <y, x> to the target at <y, x>
Pick up the cube at <y, x>
place the cube onto the target at <y, x>
ILLUSTRATIVE TEXT EXAMPLE
Demo: gripper directly pushes with closed fingers. Current cube <110, 80>, target <130, 170>. Output: Close the gripper and push the cube at <110, 80> to the target at <130, 170>
InsertPeg
TASK: InsertPeg
TASK RULES
Infer the exact peg identity, the demonstrated grasp endpoint, and the side of the box used for insertion. Distractor pegs and the opposite end are not interchangeable. First grasp the same end of the same peg; then insert it from the same demonstrated side. The near/far word describes the peg endpoint relative to the ROBOT BASE, not camera distance: on the fixed front view, the robot is behind the workspace. Use colors/geometry to match the endpoint after reset, but output the standard near/far template. The pickup coordinate is the grasp endpoint; the insertion coordinate grounds the box (the dataset’s reference entity), not an invented invisible hole pixel. Left/right insertion labels use the task/robot frame, not unexamined image-left/image-right. Recover the demonstrated insertion direction carefully.
EXPLICIT NEAR/FAR DEFINITION
Near and far refer to the two ends of the SAME peg in its initial resting pose, relative to the stationary ROBOT BASE. They do not refer to the camera, the moving gripper, the box, or how much of an end is visible. In the benchmark, near is the endpoint with the smaller absolute separation from the robot base along the robot’s forward (world X) axis; the other endpoint is far. Do not substitute Euclidean distance to the gripper.
For the supplied fixed front view, the robot base is behind the table, toward the TOP of the image. For a peg lying flat on the table with both endpoints at the same height, its base-side/upper endpoint is near and its camera-side/lower endpoint is far. This image shortcut does not apply to a lifted or rotated peg.
First identify the end actually enclosed by the fingers at the demonstrated grasp. Track that physical endpoint using the peg’s two endpoint colors and geometry back to its resting pose, then assign near/far. The opposite free endpoint that enters the box is the INSERTION end, not the GRASP end. A clearly exposed tip on a carried peg may be the free insertion tip; do not mistake it for the grasped endpoint. Keep the selected endpoint identity through lifting and reset; do not rename it because the gripper moves.
Before responding, check that the near/far word and the pixel point refer to the SAME demonstrated grasp endpoint. A correct peg identity with a point on the opposite end is incorrect. SUPPORTED TEMPLATES
Pick up the peg by grasping the {near_or_far} end at <y, x>
Insert the peg from the {left_or_right} side at <y, x>
ILLUSTRATIVE TEXT EXAMPLES (synthetic, not current-scene evidence)
Example A: In the resting pose the demonstrated grasp endpoint is nearer the robot base, at row 70, column 120. Output: Pick up the peg by grasping the near end at <70, 120>
Example B: In the resting pose the demonstrated grasp endpoint is farther from the robot base, at row 100, column 140. Output: Pick up the peg by grasping the far end at <100, 140>
PatternLock
TASK: PatternLock
TASK RULES
Infer the ORDERED sequence of target-to-target segments traced by the attached stick in the demonstration, including backtracking and repeated/diagonal segments. The robot is already holding the stick and positioned at the execution starting node; do not output pickup/reset actions. Each completed ’move ...’ is one traversed segment. Use the history length/order to choose the next segment, retaining repetitions. Output one direction-only subgoal, not a waypoint list. Labels are ROBOT-frame directions: forward is world (toward the front camera, image-down on the table), backward image-up, left world (image-right), right image-left. Combine these for diagonals. Perspective may tilt rows; use the board’s geometry rather than raw image angle alone. No pixel point is included.
SUPPORTED TEMPLATES
move forward
move backward
move left
move right
move forward-left
move forward-right
move backward-left
move backward-right
ILLUSTRATIVE TEXT EXAMPLE
Demo next segment runs to the adjacent image-left node on the same board row; no preceding segments remain. Output: move right
RouteStick
TASK: RouteStick
TASK RULES
Infer the chronological sequence of adjacent target transitions and which side of each intervening stick the tip passes around. Each subgoal is ONE entire curved transition around one stick to the nearest target on the indicated side. Preserve backtracking and repeated identical segment labels; progress comes from completed execution subgoals. Do not issue pickup, reset, arbitrary waypoints, or extra segments beyond the demo. Labels left/right use robot perspective: robot-right is image-left and robot-left is image-right in the fixed front camera. Clockwise/counterclockwise describe the curve around the obstacle when viewed from above, not merely leftward/rightward travel. Inspect the trajectory’s intermediate arc; two endpoints do not establish its circulation. Output exactly the supported direction-and-circulation template with no coordinate.
SUPPORTED TEMPLATES
move to the nearest {left_or_right} target by circling around the stick {clockwise_or_counterclockwise}
ILLUSTRATIVE TEXT EXAMPLE
A demonstrated segment goes to the robot-left adjacent target and curves counterclockwise around the intervening stick. Output: move to the nearest left target by circling around the stick counterclockwise
Runtime Additions and Auxiliary Prompts
Planner Request Fields
Each request appends the following fields to the task prompt; square brackets mark runtime values.
CURRENT REQUEST
Task name: [task]
Task instruction: [episode goal]
Completed subgoals (monitor-confirmed, chronological): [completed list as JSON]
Current/previous issued subgoals (NOT evidence of completion): [issued list as JSON]
Attachment 1 is the CURRENT execution image. Attachment 2 is the FIXED EXECUTION START image (execution frame 0, before any robot action), NOT the first demonstration frame. Use it as historical scene evidence, not as the current state. Remaining attachments are labeled below.
Attachment [index]: [demonstration sheet or execution-memory sheet].
No demonstration supplied. No demonstration images supplied.
Frozen history (ButtonUnmask, ButtonUnmaskSwap, PickHighlight). Execution-memory sheets contain history from execution start through the monitor-confirmed button-press completion (second button for ButtonUnmaskSwap); before that boundary they contain history through the current frame. Once frozen, later execution frames are excluded. Attachment 1 remains the current observation.
No selected execution history. No selected execution-memory sheets supplied; the fixed execution-start image is still supplied.
StopCube addition. ONLINE STOPCUBE OVERRIDE: A monitor trigger is a replanning opportunity, NOT confirmation that any subgoal finished or any target crossing happened. On the initial call first prepare above the button. On later calls inspect current robot readiness and the complete dense visual history. Count actual target passages in either direction; never infer passages from the number of calls or issued remain-static commands. If unprepared, continue preparation. If prepared, choose remain static for an earlier approaching visit or press for the requested approaching visit. The trigger aims for 16-32 steps of lead; do not wait for the requested passage to finish. Historical frames include every execution frame, without subsampling. Simulator pauses during reasoning, so current image remains current.
Closing line (all requests). Closed-input inference: use attached images and this prompt only. Do not invoke tools, browse, read files or delegate. Output only the single grounded subgoal.
Additional Visual Verification
Used only to check the second button press in ButtonUnmaskSwap. Audit whether the CURRENT second-button command has actually been completed in ButtonUnmaskSwap.
The small monitor has proposed completion. This is an unverified claim, NOT evidence that the correct button was pressed. Your job is to verify the target against the images before the controller is allowed to move on to picking up containers.
Coordinates are <row y, column x> on the original 256x256 front image. Identify the physical button specified by the CURRENT command using the command-start reference and the execution-start image. Compare it with the first-button command and track the same physical target through the chronological execution history. Issued commands are intentions, not evidence of actions. Earlier completed history also contains monitor claims, not ground truth.
Return true only if the images clearly support that the specified second button, distinct from the first button, has actually been pressed during the CURRENT command. It may already have been released; use the sequence, not just the final pose. Repeatedly pressing the first button, hovering over either button, lifting away, approaching the second button, or container motion alone does not establish completion. If the target is wrong, the press has not happened, or the evidence is ambiguous, return false.
Do not propose a new coordinate, change the command, or plan a container pickup. The controller will continue the identical second-button command after false, and may ask you to review again when the small monitor next proposes completion. After true, the normal planner will choose the next subgoal using the approved history.
Use only the supplied prompt and images. Do not invoke tools, browse, read files, or delegate. Output exactly true or false, with no explanation.
The request also contains the episode goal, the current instruction, the progress record, the current, execution-start, and instruction-start front images, the current wrist image, and sheets of all execution frames so far.
Monitor: Default
You monitor the current grounded subgoal using visual observations. Judge the specified action and target, not a previous action. An action may fail or require repeated attempts; do not assume success from typical duration. The recent eight front-camera images are chronological and sampled three environment steps apart. The command-start front image identifies this command instance; it is not evidence of completion. The final image is the current wrist-camera view. Return true only when the current grounded subgoal has completed by the current observation. Otherwise return false. Output exactly one lowercase word: true or false. No explanation or JSON.
Monitor: StopCube
You monitor the current grounded subgoal using visual observations. Judge the specified action and target, not a previous action. An action may fail or require repeated attempts; do not assume success from typical duration. The recent eight front-camera images are chronological and sampled three environment steps apart. The command-start front image identifies this command instance; it is not evidence of completion. The final image is the current wrist-camera view. For StopCube, use the current grounded subgoal to select the rule. For a waiting subgoal (remain static), return true when the moving cube is on the incoming leg toward the target, approximately 16 to 32 environment steps before reaching it, on every pass in either direction. Return false outside that interval, while moving away, or after it has stopped. Do not count passes or decide whether to press the button; the high-level planner does that. For a preparation or press-button subgoal, return true only when that specified action is complete; approaching the target alone does not complete these actions. Otherwise return false. Output exactly one lowercase word: true or false. No explanation or JSON.
Both monitor prompts are followed by the task name, episode instruction, current subtask, and the ten images described in § 2.3.
References
- [1]Y. Dai, H. Fu, J. Lee, Y. Liu, H. Zhang, J. Yang, C. Finn, N. Fazeli, and J. Chai. RoboMME: Benchmarking and understanding memory for robotic generalist policies. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 22396–22450, 2026. URL https://proceedings.mlr.press/v306/dai26c.html.arxiv.org/abs/2603.04639
- [2]OpenAI. GPT-6 Astra system card. OpenAI Deployment Safety Hub, 2026. URL https://deploymentsafety.openai.com/gpt-6-astra.
- [3]H. Fang, M. Grotz, W. Pumacay, Y. R. Wang, D. Fox, R. Krishna, and J. Duan. SAM2Act: Integrating visual foundation model with a memory architecture for robotic manipulation. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 15925–15942, 2025. URL https://proceedings.mlr.press/v267/fang25c.html.arxiv.org/abs/2501.18564
- [4]A. Sridhar, J. Pan, S. Sharma, and C. Finn. Scaling up memory for robotic control via experience retrieval. In International Conference on Learning Representations, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/hash/9da515b1ad19d032a7398f00f5ff9b0c-Abstract-Conference.html.arxiv.org/abs/2510.20328
- [5]M. Torne, K. Pertsch, H. Walke, K. Vedder, S. Nair, B. Ichter, A. Z. Ren, H. Wang, J. Tang, K. Stachowicz, K. Dhabalia, M. Equi, Q. Vuong, J. T. Springenberg, S. Levine, C. Finn, and D. Driess. MEM: Multiscale embodied memory for vision language action models. arXiv preprint arXiv:2603.03596, 2026. URL https://arxiv.org/abs/2603.03596.
- [6]X. Cai, Y. Cai, B. Chen, J. Chen, Z. Chen, S. Feng, T. Hou, J. Huang, H. Jiang, R. Ju, D. Li, M. Li, S. Li, X. Li, Y. Li, Y. Liu, Z. Liu, J. Luo, J. Miao, R. Ni, B. Nie, M. Pan, X. Ren, J. Song, J. Wang, P. Wang, S. Wang, X. Wang, D. Wei, D. Wu, P. Xie, P. Yang, H. Ye, X. Yue, J. Zhang, Q. Zhang, X. Zhao, P. Zhou, and Y. Zhou. τ₀-VLA: A hierarchical robot foundation model with world-model-guided test-time computation. arXiv preprint arXiv:2608.16885, 2026. URL https://arxiv.org/abs/2608.16885.
- [7]K. Black, N. Brown, J. Daripinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Muthukumar, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky. π₀.₅: A vision-language-action model with open-world generalization. In Proceedings of the 9th Conference on Robot Learning, volume 305 of Proceedings of Machine Learning Research, pages 17–40, 2025. URL https://proceedings.mlr.press/v305/black25a.html.arxiv.org/abs/2504.16054
- [8]Qwen Team. Qwen3-VL-4B-Instruct. Model release, 2025. URL https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct.
- [9]Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. URL https://arxiv.org/abs/2507.06261.



