betaFeedback

WorksComputer Science

Robocasa-Mem: Benchmarking Environment and Process Memory in Long-Horizon Mobile Manipulation

Abstract

In the real world, mobile manipulation robots must perform complex object interactions and make multi-step decisions, often relying on task-relevant information that is no longer observable. However, existing benchmarks provide limited evaluation of such memory-dependent decision-making in mobile manipulation. To address this gap, we introduce RoboCasa-Mem, a new benchmark comprising nine mobile manipulation tasks that require agents to use interaction history at key decision points. Specifically, the tasks are grouped into two categories of memory: One is environment memory, which captures spatial locations, object attributes, and object relationships. The other is process memory, which captures single-task progress and multi-task scheduling. Particularly, we introduce Behavior-Conditioned Evaluation. For tasks with multiple valid execution paths, instead of fixing a single target or trajectory, the evaluator derives later task constraints from the robot's earlier actions. This allows us to evaluate whether later behavior is consistent with the robot's own interaction history while reducing shortcuts based on fixed target assignments. We provide 1,800 successful demonstrations for policy training. We also develop a baseline on our benchmark, namely Hierarchical Memory-Augmented Policy(HMAP). Its high-level planner uses structured entity and process memories to generate subtasks. Its low-level VLA controller uses subgoal-conditioned queries to incorporate VGGT-Omega geometry into action generation. Together, RoboCasa-Mem and HMAP baseline support systematic evaluation of memory-dependent behavior and the development of memory-aware mobile manipulation policies.

Project Page: https://sihhangchen.github.io/RoboCasa-Mem/

Code: https://github.com/SiHhangChen/RoboCasa-Mem

Data: https://modelscope.cn/datasets/ouhebai/RoboCasa-Mem/tree/master/data

Introduction

Mobile manipulation requires robots to navigate between locations and interact with objects [18] (Nasiriany et al. 2024). While tabletop manipulation often concentrates task-relevant information within a fixed workspace, mobile manipulation extends interactions across different work areas. As the robot moves and interacts with the environment, previously observed information may become inaccessible yet remain essential for subsequent decisions. For example, when cooking, a robot may need to remember whether it has already added salt to the pot after the salt has dissolved and is no longer visually distinguishable. Such situations motivate maintaining historical information about both the environment and task execution, beyond what is available in the current observation [19]. Memory dependence is not equivalent to long-horizon manipulation. A task may involve multiple sequential steps without requiring historical information, provided that the current observation is sufficient for decision-making. Conversely, even a short task may require memory when critical information is no longer observable. From a partially observable decision-making perspective, historical information is needed when different interaction histories yield indistinguishable current observations but require different subsequent decisions [6]. We therefore evaluate memory based on the necessity of historical information for correct decision-making, rather than task length or execution complexity alone.

Existing mobile-manipulation benchmarks, such as RoboCasa (Nasiriany et al., 2024), RoboCasa365 (Nasiriany et al., 2026), MS-HAB [16], BEHAVIOR-1K [8], and the suite of works based on Habitat [13, 18, 14], primarily evaluate task execution, long-horizon control, generalization, and policy learning, without explicitly isolating decisions that require historical information. In parallel, robotic memory benchmarks such as RMBench [2], RoboMME [5], RoboMemArena [7], and RoboTwin-MeM [22] study history-dependent manipulation, predominantly in tabletop or fixed-base settings. Mobile manipulation brings these memory demands into settings where robots move between work areas, lose visual access to previously observed entities, and must resume unfinished activities. Two requirements are particularly relevant: retaining object states established through earlier interactions but no longer visually distinguishable, and tracking progress and pending actions across interleaved activities in different regions. Although neither requirement is unique to mobile manipulation, their joint evaluation under changing viewpoints and cross-region execution remains limited.

To systematically characterize these requirements, we organize memory into two complementary dimensions: Environment Memory and Process Memory. Environment Memory captures information about the physical environment that cannot be reliably recovered from the current observation, encompassing Spatial Memory, Attribute Memory, and Relational Memory. Process Memory tracks task execution states and information needed to determine subsequent actions, encompassing Single-Task Progress and Multi-Task Scheduling. This taxonomy explicitly accounts for visually unobservable object attribute changes and the scheduling of interleaved tasks, providing a systematic characterization of memory requirements in mobile manipulation.

Based on this taxonomy, we introduce RoboCasa-Mem, a benchmark for memory in mobile manipulation. RoboCasa-Mem contains nine mobile-manipulation tasks covering the five memory categories above. Each task is designed to introduce observation-insufficient decision points, where the current observation alone cannot determine the appropriate action, requiring the robot to retrieve and utilize information from its interaction history. This design explicitly evaluates memory dependence rather than relying on task horizon as its proxy. For tasks with multiple valid execution paths, we introduce Behavior-Conditioned Evaluation, which derives later constraints from the policy’s own earlier interactions rather than fixing a target in advance. This allows history-consistent behavior to be evaluated without prescribing a single execution trajectory.

To provide a baseline, we develop a Hierarchical Memory-Augmented Policy (HMAP) that combines structured memory for planning with geometry-augmented control. A high-level planner maintains Entity Memory and Process Memory to track environment states, execution progress, and pending actions, and generates executable subgoals. A low-level VLA controller executes these subgoals using current observations and multi-view geometric features extracted by VGGT-Ω\mathrm{VGGT}\text{-}\Omega [20], which are incorporated through task-conditioned queries and action modulation. Across the nine tasks, HMAP improves average success from 16.2% to 24.2% over π0.5\pi_{0.5} [12], with gains concentrated in hidden-attribute recall. Improvements on process-memory tasks remain inconsistent, highlighting challenges in progress tracking and interleaved execution.

Our contributions are threefold:

  • Benchmark and dataset. We introduce RoboCasa-Mem, comprising nine mobile-manipulation tasks organized into five memory categories across environment-state and task-state memory, together with 1,800 successful demonstration trajectories for policy training.

  • Evaluation protocol. We introduce Behavior-Conditioned Evaluation, which derives subsequent semantic constraints from the robot’s actual interactions to assess history-consistent behavior across multiple valid execution paths. We report both final task success and stage-level completion.

  • Baseline and findings. We develop Hierarchical Memory-Augmented Policy (HMAP) that combines structured semantic memory for planning with 3D geometric feature for control. Our

BenchmarkMobileHistory dependenceSpatialAttributeRelationProgressSchedulingStage eval
MIKASA-Robo [3]✗✓✓✗✓✓✗✗
RMBench [2]✗✓✓✗✓✓✗✗
RoboMME [5]✗✓✓✗✓✓✗✗
RoboTwin-Mem [22]✗✓✓✗✓✓✗✗
RoboMemArena [7]✗✓✓✓✓✓✗✓
RoboCasa-Mem (Ours)✓✓✓✓✓✓✓✓

Table 1. Comparison of simulated robotic memory benchmarks. Coverage denotes at least one qualifying task, not uniform requirements across tasks or equivalent taxonomies.

experiments reveal gains concentrated in hidden-attribute recall and identify persistent challenges in task-progress tracking and interleaved multi-task execution.

Related Work

Mobile-manipulation benchmarks. Existing benchmarks study diverse aspects of mobile manipulation, including task diversity, long-horizon execution, generalization, and scalable policy learning. RoboCasa and RoboCasa365 provide large-scale household environments, tasks, and demonstrations (Nasiriany et al., 2024; 2026); Habitat 2.0 and MS-HAB focus on long-horizon household rearrangement and low-level mobile-manipulation control [18, 16]; and BEHAVIOR-1K covers a broad range of household activities and physical interactions [8]. These benchmarks do not explicitly isolate memory dependence as an evaluation target. In contrast, RoboCasa-Mem focuses on decisions for which the current observation is insufficient and earlier information must be retained.

Memory benchmarks for manipulation. Recent benchmarks explicitly evaluate memory in robotic manipulation. MIKASA-Robo and RMBench study memory-intensive manipulation tasks [3, 2]; RoboMME evaluates temporal, spatial, object, and procedural memory [5]; RoboMemArena studies long-horizon manipulation with memory-related annotations [7]; and RoboTwin-Mem considers non-Markovian tasks in which later actions depend on earlier visual observations [22]. Most of these benchmarks focus on tabletop or fixed-base manipulation. RoboCasa-Mem instead targets mobile manipulation and emphasizes memory demands arising from changing environment states and interleaved task execution. It also evaluates history-dependent decisions without requiring a single predefined execution path.

Memory-augmented embodied policies. Memory-augmented policies differ mainly in how historical information is selected and represented. MemER retrieves historical keyframes for high-level subtask generation, while EventVLA, KEMO, and UniMem retain task-relevant or event-triggered observations for later prediction and control [17, 22, 23, 11]. Other methods combine memory across temporal or semantic scales: MemoryVLA uses working and perceptual-cognitive memory, MEM combines short-term video memory with long-term textual memory, and Mimir separates world memory from task memory [15, 19, 21]. Our method similarly uses memory at high levels and combines persistent structured memory for high-level task decisions with current 3D visual features for low-level control.

The RoboCasa-Mem Benchmark

RoboCasa-Mem contains nine household mobile-manipulation tasks covering five memory types under two groups: Environment Memory and Process Memory. Figure 1 gives an overview of the benchmark.

Figure 1. Overview of RoboCasa-Mem. Top: a history-dependent cup-selection example illustrates why the current observation alone may not determine the correct action. Middle: five memory categories cover spatial, attribute, and relational Environment Memory, together with single-task progress and multi-task scheduling in Process Memory. Bottom: demonstration construction and semantic evaluation support analysis of final outcomes and stage-level completion. Table 2 lists the nine core tasks.

Memory Demand and Task Design

Memory-dependent task design. We design tasks such that information required for a later decision is observed earlier but cannot be reliably recovered from the current observation. For example, after sugar dissolves in water, the current image does not reveal whether sugar was previously added to a cup. The policy must therefore retain information from earlier interactions rather than relying only on the current observation. This distinguishes memory dependence from task horizon: a task may contain many steps without requiring memory if each decision can still be made from the current observation.

We formalize this design principle using observation aliasing in partially observable environments [6]. Let xx denote the task instruction, oto_t the current observation, and ht=(o1,a1,…,ot−1,at−1,ot)h_t=(o_1,a_1,\ldots,o_{t-1},a_{t-1},o_t) the interaction history. For a semantic query qq, let Gq∗(ht,x)\mathcal{G}^{*}_{q}(h_t,x) denote the set of acceptable responses. Historical information is required when two feasible histories satisfy

ot(ht)=ot(ht′),Gq∗(ht,x)∩Gq∗(ht′,x)=∅.(1)o_t(h_t)=o_t(h'_t),\qquad\mathcal{G}^{*}_{q}(h_t,x)\cap\mathcal{G}^{*}_{q}(h'_t,x)=\varnothing. \tag*{(1)}

That is, similar current observations may require different decisions because of different interaction histories. We use this condition as an idealized task-design criterion rather than requiring observations to be exactly identical at the pixel level.

Task Classification

The nine tasks are grouped according to the historical information required for later decisions. Environment Memory concerns information about the environment that may no longer be recoverable from the current observation. It includes Spatial Memory, which retains earlier object locations; Attribute Memory, which retains object properties or states; and Relational Memory, which retains associations between entities.

Process Memory concerns information about task execution. It includes Single-Task Progress, which tracks completed actions, search progress, or counts within one task, and Multi-Task Scheduling, which tracks pending actions and task progress across interleaved activities. Each task is assigned to its primary memory category, although successful execution may involve more than one type of memory. Table 2 summarizes the nine core tasks and their main memory requirements. Detailed task definitions, stage predicates, and success conditions are provided in Appendix § 7.

Table 2: Overview of the nine RoboCasa-Mem tasks, organized into five memory categories under Environment Memory and Process Memory. Quoted descriptions summarize the tasks rather than reproduce the exact policy instructions.

TaskMemory typeKey challengeDescription
Environment Memory
PutBackCupSpatialLocation recall“Place the cup at the previously cleaned spot.”
SweetCupAttributeHidden attribute“Prepare one sweetened and one unsweetened coffee, then select the sweetened cup.”
TempReportAttributeAttribute recall“Recall whether hot or cold water was dispensed, then press the matching button.”
CanRecallRelationalObject–container binding“After closing the cabinets, recall the requested items’ locations and place them back on the tabletop.”
MilkMatchRelationalSource–target binding“Transport milk boxes in two trips, placing each on the plate matching its source tray’s color.”
Process Memory
DrawerSearchSingle-task progressSearch progress“Count the drawers checked while searching for the red cube, then press the answer button that many times.”
SugarCountSingle-task progressAction counting“Add KK sugar cubes to the hot chocolate, then press the completion button.”
HeatAndInspectMulti-task schedulingPending obligation“Inspect drawers while heating a pan, then return to turn off the stove and submit.”
FillAndRetrieveMulti-task schedulingPending obligation“Retrieve a can while dispensing water, then return to turn off the faucet and complete both tasks.”

Table 2.

Benchmark Construction

Environment and policy interface. We implement all tasks in RoboCasa, combining mobile navigation with object and fixture manipulation (Nasiriany et al., 2024). Each episode is associated with a language instruction. At each recorded timestep, the dataset contains three synchronized 224×224224 \times224 RGB views, a 37-dimensional robot state, and a 13-dimensional action at 20 Hz. The three views are captured by the left external camera, right external camera, and wrist-mounted camera. The standard policy receives the language instruction, RGB observations, and robot state, and predicts the robot action.

Demonstration collection. We collect demonstrations using a rule-based expert that combines MPLib for arm motion planning with a closed-loop controller for mobile-base navigation. Each task is decomposed into navigation and manipulation stages, and the corresponding controller is used to execute each stage. To increase data diversity, we randomize task-relevant scene configurations, including object positions and appearance, manipulation order, navigation targets, and manipulation poses where applicable. Some task parameters are also varied across episodes, such as the requested number of sugar cubes in SugarCount.

We collect 200 successful demonstrations for each task, resulting in 1,800 trajectories in total. Only trajectories that satisfy the required stage sequence and final semantic predicates are retained. Detailed state and action definitions, controller settings, randomization procedures, trajectory statistics, validation procedures, and dataset format are provided in Appendix § 8.

Evaluation Protocol

Behavior-conditioned evaluation. We evaluate each task over 50 held-out scene seeds using the same randomized environments for all policies. For tasks with multiple valid execution paths, the evaluator derives later constraints from the policy’s earlier interactions rather than assigning a fixed target in advance. For example, in SweetCup, either cup may receive sugar, and the evaluator later checks whether the policy selects the same cup. Success is defined by task semantics rather than by reproducing the scripted expert trajectory. Detailed stage and terminal predicates are in Appendix § 7.

Evaluation metrics. We report two complementary metrics. Task Success Rate (SR) measures the fraction of episodes in which the complete task is successfully completed. Stage Completion Rate (SCR) summarizes task progress as the average success rate over the KK predefined evaluation stages, i.e., SCR=1K∑k=1Krk\mathrm{SCR} = \frac{1}{K}\sum_{k=1}^{K} r_k, where rkr_k denotes the success rate of the kk-th evaluation stage. SR captures final task completion, while SCR provides a stage-level measure of how far a policy progresses through the task.

Hierarchical Memory-Augmented Policy

Our policy consists of a high-level symbolic memory planner and a low-level subgoal-guided 3D-aware controller. The planner maintains entity memory and process memory to reason over long-horizon task progress and select an executable subgoal. Conditioned on this subgoal, the controller retrieves task-relevant information from current multi-view geometry and uses it to guide action generation. This separation allows symbolic memory to support long-horizon reasoning while geometric context is used for low-level execution. At each planning time tkt_k, the planner updates entity memory eke_k and process memory pkp_k and produces the current subgoal gkg_k, which is passed to the low-level controller. Figure 2 illustrates the overall architecture.

Figure 2. Overview of our Hierarchical Memory-Augmented Policy. The high-level planner maintains entity and process memory to select an active subgoal. The low-level subgoal-guided 3D-aware controller conditions on this subgoal and retrieves task-relevant features from current multi-view geometry to modulate action generation.

High-Level Symbolic Memory Planner

Building on language memory for high-level planning [19], we maintain two complementary forms of symbolic memory. Entity memory eke_k stores task-relevant properties and spatial information about entities, while process memory pkp_k tracks execution progress and pending steps. Entity memory is intentionally sparse, retaining only information relevant to future task decisions.

We use Qwen3-VL-4B-Instruct [1] as the planner. At planning step kk, it takes the overall instruction xx, previous memories (ek−1,pk−1)(e_{k-1}, p_{k-1}), and observations from the current and an earlier timestep:

(ek,pk,gk)=Hθ(c,x,ek−1,pk−1,Itk,Itk−50),(2)(e_k, p_k, g_k) = H_\theta(c, x, e_{k-1}, p_{k-1}, I_{t_k}, I_{t_k-50}), \tag*{(2)}

where cc denotes the fixed planner prompt. The two observations provide evidence of recent state changes. The updated memories are recursively passed to subsequent planner calls, while only the executable subgoal gkg_k is provided to the low-level controller. This recurrent memory allows the planner to preserve task-relevant information that may no longer be observable from the current scene.

The planner is supervised offline using structured annotations derived from demonstration trajectories. The memory schema, annotation procedure, planner prompts, and additional examples are provided in Appendix § 9.

Low-Level subgoal-guided 3D-aware Controller

The low-level subgoal-guided 3D-aware controller builds on π0.5\pi_{0.5} [12] and augments it with geometric features from a frozen VGGT-Ω\Omega encoder [20]. VGGT-Ω\Omega jointly encodes the current RGB observations from the left external, right external, and eye-in-hand cameras, producing multi-view geometric tokens that remain separate from the standard PaliGemma visual–language–state representation.

To incorporate geometry selectively, we introduce task-conditioned queries into the PaliGemma stream. Conditioned on the current observations, robot state, overall instruction, and active subgoal, these queries retrieve relevant information from the VGGT-Ω\Omega representation through cross-attention. The retrieved geometric features are then used to modulate the corresponding action-expert layers through learned scale and shift transformations. Thus, geometry influences action generation through task-conditioned retrieval and layerwise modulation rather than direct token concatenation.

The planner and controller are trained separately: the planner learns symbolic subgoal prediction and memory updates, while the controller is optimized with the standard flow-matching objective for action generation. At inference, the planner provides the active subgoal, and the controller combines it with current multi-view geometry to execute the task. Detailed architecture, optimization settings, and inference procedures are provided in Appendix § 9.

Experiments

We evaluate our method on the nine core RoboCasa-Mem tasks, analyze its performance across different memory requirements, validate it on a physical AgileX Cobot Magic Platform, and conduct an ablation study to examine the contributions of the main policy components.

Experimental Setup

Baselines and training. We compare ACT, Diffusion Policy (DP), π0.5\pi_{0.5}, MemER-style, and our hierarchical memory policy [24, 4, 12, 17]. All methods are trained on the same task demonstrations. For a controlled comparison, MemER-style uses the same Qwen3-VL-4B-Instruct high-level model and low-level policy backbone as our method, while differing in high-level memory training. Detailed training schedules, sampling strategies, and per-method hyperparameters are provided in Appendix § 9.

Main results

Simulation Results

Overall performance. Table 3 summarizes performance on the nine core tasks. Our method achieves the highest average SR and SCR, reaching 0.24 and 0.47 compared with 0.16 and 0.41 for π0.5\pi_{0.5}. ACT, DP, and MemER-style rarely complete the full tasks despite making partial progress on several stages. In particular, inspection of MemER-style failures shows that many occur around subtask transitions, suggesting that successful execution requires not only manipulation ability but also reliable maintenance and use of task-relevant history.

ModelPutBackCup SRPutBackCup SCRSweetCup SRSweetCup SCRTempReport SRTempReport SCRCanRecall SRCanRecall SCRMilkMatch SRMilkMatch SCR
ACT0.000.480.000.070.000.090.000.090.000.03
DP0.000.200.000.040.080.210.000.050.000.05
π0.5\pi_{0.5}0.680.850.040.270.220.570.000.240.120.47
MemER-style0.100.220.000.210.000.580.000.210.000.19
HMAP (Ours)0.700.790.100.460.860.930.000.260.160.47
ModelDrawerSearch SRDrawerSearch SCRSugarCount SRSugarCount SCRHeatAndInspect SRHeatAndInspect SCRFillAndRetrieve SRFillAndRetrieve SCRAvg. SRAvg. SCR
ACT0.000.080.040.070.000.250.000.260.000.16
DP0.000.080.140.190.000.230.000.180.020.14
π0.5\pi_{0.5}0.120.320.280.230.000.320.000.390.160.41
MemER-style0.000.250.080.100.000.320.000.200.020.25
HMAP (Ours)0.080.310.280.200.000.500.000.340.240.47

Table 3. Results on the nine RoboCasa-Mem tasks. SR denotes task success rate, and SCR denotes stage completion rate. Avg. is the macro-average across all nine tasks. Best nonzero values are bolded, including ties at the reported precision.

Environment memory. The clearest gains appear when the information needed for a later decision is no longer recoverable from the current observation. On TempReport, our method improves SR from 0.22 to 0.86 and SCR from 0.57 to 0.93 relative to π0.5\pi_{0.5}. Among episodes reaching the reporting stage, the conditional success rate of recalling the correct temperature increases from 0.41 to 0.90. A similar improvement is observed on SweetCup, where the corresponding conditional recall success increases from 0.22 to 0.45.

The gains are smaller on PutBackCup and MilkMatch at the task level, but stage-level analysis again shows improved history-dependent decisions. For example, the conditional success of returning the cup to its original location increases from 0.72 to 0.83, while the average conditional success of the two color-matched milk placements increases from 0.47 to 0.79. These results indicate that explicit memory primarily improves decisions that require retrieving earlier environment states, rather than uniformly improving all manipulation stages. Stage-wise results are detailed in the Appendix.

Process memory. A different pattern emerges for Process Memory. On Single-Task Progress tasks, our method and π0.5\pi_{0.5} perform similarly, suggesting that explicit memory provides limited benefit when progress remains partially inferable from the ongoing execution state. Multi-Task Scheduling is substantially harder: on HeatAndInspect, our method improves SCR from 0.32 to 0.50, but no method completes the full task. FillAndRetrieve remains similarly challenging. These results suggest that multi-task execution requires not only retaining unfinished obligations, but also correctly switching between and resuming task threads.

Figure 3. An illustration of our real-world experiments conducted on AgileX Cobot Magic Platform. The task is set as: the robot first observes and picks up an object, then turns to transport it and places it into the corresponding container.

Real-Robot Evaluation

We further evaluate the policies on the AgileX Cobot Magic platform using an Object Transport task. The robot picks an object from a colored source container, turns toward the opposite workspace, and places it into a target container matching the source container’s color. Once the robot turns, the source container leaves the camera view, requiring the policy to retain its color for subsequent target selection. We collect 100 demonstrations and train ACT, DP, π0.5\pi_{0.5}, and HMAP using the same policy interface as in simulation. As shown in Table 4, HMAP completes 8 of 10 trials, compared with 2 of 10 for π0.5\pi_{0.5}, with the difference concentrated at the final target-selection stage.

MethodPickMove & PlaceRight Container
π0.5\pi_{0.5}10/1010/102/10
Ours10/109/108/10

Table 4. AgileX Cobot Magic Platform Object Transport results over 10 trials per method.

Ablation Study

We conduct an ablation study on the simulated SingleMilkMatch task to examine three components: planner-generated subtasks, VGGT-Ω\Omega conditioning, and learnable query tokens. Starting from π0.5\pi_{0.5}, we progressively add the high-level planner, direct VGGT-Ω\Omega modulation, and finally task-conditioned learnable queries with cross-attention. Table 5 reports the resulting stage-level performance. Terminal success increases from 0.20 for π0.5\pi_{0.5} to 0.34 with planner-generated subtasks, showing that high-level memory substantially improves target selection. Adding VGGT-Ω\Omega conditioning further increases success to 0.40 and primarily improves low-level execution. The full model reaches 0.46 terminal success; among episodes in which the object is successfully placed, correct target selection rises to 0.82. These results suggest complementary roles for the two levels of the policy: structured memory supports history-dependent decision making, while task-conditioned geometric features improve execution and target-specific control.

VariantPickMove & PlaceRight Container
π0.5\pi_{0.5}0.700.600.20
+ Planner0.660.500.34
+ VGGT-Ω\Omega0.700.640.40
Ours0.740.560.46

Table 5. Simulated SingleMilkMatch ablation. Stage rates are cumulative.

Conclusion and Limitations

We introduced RoboCasa-Mem, comprising nine mobile-manipulation tasks and 1,800 successful demonstrations organized around Environment Memory and Process Memory. Behavior-Conditioned Evaluation checks whether subsequent behavior satisfies constraints established by actual interactions, accommodating multiple valid execution paths. Our HMAP baseline combines structured semantic memory for planning with current multi-view geometry for control and improves average task success from 16.2% to 24.2% over π0.5\pi_{0.5}. These gains are concentrated in hidden-attribute recall, while improvements on process-memory tasks remain inconsistent and neither interleaved scheduling task is completed by the evaluated policies. These findings motivate further work on reliable memory updates, subtask switching, and resumption of pending actions.

Our evaluation is primarily conducted in simulation, with only one physical task evaluated over 10 trials per method, limiting conclusions about broader real-world performance. The five memory categories are not exhaustive, and the dataset contains only successful expert demonstrations. Stage-level metrics help localize failures but do not fully separate memory errors from execution errors. In SugarCount, stage applicability also varies across episodes, so SCR summarizes task progress over different episode subsets. More diagnostic evaluation protocols are therefore needed to distinguish history-dependent decision errors from physical execution failures.

Task Specifications

The nine core tasks are summarized in Table 2, and their workflows are illustrated in Figure 4. This section describes the information acquired during execution, why it may become unavailable at a later decision point, and what the reported evaluation stages assess. These descriptions summarize task behavior rather than reproduce the exact policy instructions. A stage is credited only after its required predecessors have been satisfied. Final success additionally requires the task-specific terminal semantic condition.

Figure 4. Execution workflows of the nine RoboCasa-Mem tasks.

Environment Memory Tasks

PutBackCup (spatial memory). The robot observes the cup’s initial location before moving it aside to clean the area it occupied. The cup’s current position no longer identifies its original location, which the robot must recall when returning it. The reported stages check, in order, stain removal, placement of the cleaning tool, and return of the cup to its original location.

SweetCup (attribute memory). The robot prepares one sweetened and one unsweetened cup. It may choose which cup receives sugar, and the evaluator records this choice from the actual rollout rather than assigning a cup identity in advance. After the sugar dissolves, the current RGB observation does not reliably identify the sweetened cup. The reported stages check sugar addition, staging of both cups, and selection of the cup that received sugar.

TempReport (attribute memory). The robot dispenses hot or cold water and later reports which temperature was dispensed. At the reporting stage, the filled cup’s appearance does not directly reveal this attribute. The stages check filling the cup, placing it in the service area, and final task success, including the correct temperature report.

CanRecall (relational memory). The robot observes the associations between the requested cans and their containing cabinets. After closing both cabinet doors to hide their contents, it must reopen the appropriate cabinet, retrieve the requested items, and place them back on the tabletop. The four reported stages check closure of either cabinet door (Close-1), simultaneous closure of both doors (Close-2), reopening of the correct cabinet after the contents have been hidden (Reopen), and retrieval and placement of the requested items back on the tabletop (Success). Correct reopening assesses part of the recall decision, while terminal success also depends on subsequent execution.

MilkMatch (relational memory). Two milk boxes are transported from colored source trays on one side of the coffee machine to matching-color target trays on the other side. Turning toward the targets removes the source trays from the immediate view. Either box may be transported first. The evaluator assigns “first” and “second” according to the policy’s actual pickup order and checks pickup, placement, and correct color-matched placement for each box. Terminal success requires both boxes to reach their corresponding target trays.

Process Memory Tasks

DrawerSearch (single-task progress). The robot searches drawers for a red cube while tracking the number of drawers inspected. Finding the target determines when to stop searching, and the inspection count determines the required number of button presses. The stages check opening and closing a drawer, finding the target, stopping the search after discovery, pressing the button, and reporting the correct count. The answer must reflect the drawers inspected during the actual rollout.

SugarCount (single-task progress). The instruction specifies an episode-dependent target count KK. The robot tracks successful sugar-cube additions, stops at the requested count, and presses the completion button. The reported stages assess the first, second, and third additions, followed by final task success. Each addition stage applies only to episodes requesting at least that many cubes, so its denominator may differ from those of other addition stages.

HeatAndInspect (multi-task scheduling). The robot turns on the stove to heat a pan while inspecting drawers. During inspection, it must retain the obligation to return and turn off the stove before submission. The stages check turning on the stove, inspecting a drawer, completing either task thread, completing both threads, and final task success.

FillAndRetrieve (multi-task scheduling). The robot starts dispensing water and retrieves a can while the faucet is running. Completion requires returning to turn off the faucet and finishing the retrieval task. The stages check turning on the faucet, retrieving the can, completing either task thread, completing both threads, and final task success.

Data and Evaluation Details

Observation and expert interfaces. Simulation trajectories are recorded at 20 Hz. Each timestep contains three synchronized 224×224224 \times224 RGB images, a 37-dimensional robot state, and a 13-dimensional action. The images are captured by the left and right external cameras and the wrist-mounted camera. The standard policy receives these RGB images, the robot state, and the language instruction. A rule-based expert combines MPLib for arm motion planning with closed-loop mobile-base navigation. Episode configurations vary task-relevant object positions and appearances, navigation targets, manipulation order and poses where applicable, and parameters such as the requested sugar count. The expert’s internal task state is not included in the standard policy inputs.

Storage and validation. Native HDF5 recordings store trajectories, camera recordings, calibration information, and episode metadata. Conversion to LeRobot preserves the alignment between each observation and its subsequent action. Termination and success flags are retained for bookkeeping and evaluation but are excluded from the standard policy inputs. We retain 200 successful demonstrations per core task. Validators check the relevant grasp and release, containment, fixture-state, count, and ordering conditions. Trajectories that fail validation are excluded. An action-only replay audit samples 20 accepted demonstrations per task, yielding 180 sampled trajectories in total.

Control schema. Each 13-dimensional action comprises seven arm joint targets, one gripper command, three mobile-base commands, one torso command, and one base/arm mode indicator. The exact axis order, units, mode encoding, and definitions of the 37 robot-state dimensions should accompany the released dataset.

Evaluation and aggregation. Methods are evaluated on 50 held-out scene seeds per core task, using the same randomized environments across methods. For task jj, let NjN_j denote the number of valid evaluation episodes and zijz_{ij} indicate terminal semantic success in episode ii. Let KjK_j denote the number of evaluation stages. For stage kk, let AjkA_{jk} be the applicable episode set and sijks_{ijk} indicate whether the stage predicate and its required predecessors have been satisfied. We compute

SRj=1Nj∑i=1Njzij,rjk=∑i∈Ajksijk∣Ajk∣,SCRj=1Kj∑k=1Kjrjk.(3)\mathrm{SR}_j = \frac{1}{N_j}\sum_{i=1}^{N_j} z_{ij}, \qquad r_{jk} = \frac{\sum_{i \in A_{jk}} s_{ijk}}{\lvert A_{jk}\rvert}, \qquad\mathrm{SCR}_j = \frac{1}{K_j}\sum_{k=1}^{K_j} r_{jk}. \tag*{(3)}

An episode that fails before reaching an applicable stage remains in that stage’s denominator. Sugar-Count addition stages have different applicable episode sets when target counts differ, so their rates need not decrease monotonically. The overall SR and SCR in Table 3 are the unweighted means of the nine per-task values.

When adjacent stages share the same applicable episode set, have a strict predecessor relation, and satisfy rj,k−1>0r_{j,k-1} > 0, the ratio rjk/rj,k−1r_{jk}/r_{j,k-1} measures completion of stage kk among episodes that completed stage k−1k - 1. For π0.5\pi_{0.5} and HMAP, respectively, the conditional rates are 0.72 and 0.83 for returning the cup in PutBackCup, 0.22 and 0.45 for selecting the sweetened cup in SweetCup, and 0.41 and 0.90 for the final reporting stage in TempReport. For MilkMatch, the conditional rates of correct placement given placement are 0.48 and 0.77 for the first box and 0.46 and 0.80 for the second. A predicate that combines target selection with manipulation cannot attribute every failure uniquely to memory.

Method and Annotation Details

Planner state. The Qwen3-VL-4B-Instruct planner (Bai et al. [1]) receives the instruction, previous entity and process memories, and observations at times tkt_k and tk−50t_k - 50, as defined in Eq. (2). Entity memory records relevant properties, locations, and relations, such as which cup received sugar or a milk box’s source tray. Process memory records completed steps and pending obligations, such as inspected drawers or a faucet that remains to be turned off. Each planner call returns updated memories and one executable language subgoal. The memories are passed to the next planner call, while only the subgoal is supplied to the low-level controller. Structured supervision is derived offline from demonstration trajectories and checked for consistency with task-event ordering.

Geometric conditioning. The low-level controller builds on π0.5\pi_{0.5} [12]. A frozen VGGT-Ω\Omega encoder [20] produces multi-view geometric tokens separately from the standard visual–language–state representation. Eight learnable query embeddings are processed together with the current task context. The resulting representations attend to the geometric tokens through cross-attention, and the fused features provide scale and shift modulation for the action expert. The planner and controller are trained separately, with the controller optimized using the standard flow-matching objective for action generation.

Reported training schedules. The reported batch size is 48. ACT and DP are trained for 40,000 optimization steps with a sampling stride of 10. The reported training budgets for π0.5\pi_{0.5} and HMAP are 20,000 steps for PutBackCup, TempReport, DrawerSearch, and SugarCount, and 30,000 steps for SweetCup, CanRecall, MilkMatch, and HeatAndInspect. The available schedule does not specify the training budget for FillAndRetrieve. The reported replanning interval is 40 control steps for MilkMatch, DrawerSearch, SugarCount, and PutBackCup. Optimizer settings, LoRA configurations, checkpoint-selection criteria, and the remaining task-specific settings remain to be specified from the final run configurations.

Complete Results

Tables 6 and 7 report stage-level completion rates for all nine benchmark tasks, with stage descriptions provided below each subtable. Final task success rates are summarized separately in Table 3.

tbl:6: Stage-level completion rates for PutBackCup, SweetCup, and TempReport. Each value is the fraction of evaluated episodes satisfying the corresponding stage and its required predecessors.

MethodCleanToolReturn
ACT0.720.720.00
DP0.300.300.00
π0.5\pi_{0.5}0.940.940.68
MemER-style0.280.280.10
HMAP (Ours)0.840.840.70

Table a. PutBackCup Stages. Clean the stain, put down the cleaning tool, and return the cup.

MethodSugarStage-1Stage-2Select
ACT0.220.040.020.00
DP0.100.040.000.00
π0.5\pi_{0.5}0.480.380.180.04
MemER-style0.720.120.000.00
HMAP (Ours)0.820.680.220.10

Table b. SweetCup Stages. Add sugar, stage the first cup, stage the second cup, and select the sweetened cup.

MethodFillPlaceSuccess
ACT0.260.000.00
DP0.380.180.08
π0.5\pi_{0.5}0.940.540.22
MemER-style0.980.760.00
HMAP (Ours)0.980.960.86

Table c. TempReport Stages. Fill the cup, place it in the service area, and achieve final task success.

tbl:7: Stage-level completion rates for the remaining six RoboCasa-Mem tasks. Abbreviated stage names are explained below each subtable.

MethodClose-1Close-2ReopenSuccess
ACT0.340.000.000.00
DP0.180.000.000.00
π0.5\pi_{0.5}0.760.140.040.00
MemER-style0.800.020.000.00
HMAP (Ours)0.800.240.000.00

Table d. CanRecall Stages. Close either cabinet door (Close-1), have both doors closed simultaneously (Close-2), reopen the correct cabinet after hiding its contents (Reopen), and retrieve the requested items and place them back on the tabletop (Success).

MethodP1D1C1P2D2C2
ACT0.100.100.000.000.000.00
DP0.200.120.000.000.000.00
π0.5\pi_{0.5}0.920.800.380.340.260.12
MemER-style0.520.420.140.020.020.00
HMAP (Ours)0.840.700.540.360.200.16

Table e. MilkMatch Stages. P, D, and C denote pickup, placement, and correct placement, respectively. The indices 1 and 2 refer to execution order.

MethodOpen/CloseFindStopButtonCount
ACT0.340.040.040.000.00
DP0.220.100.060.000.00
π0.5\pi_{0.5}0.840.380.160.120.12
MemER-style0.860.380.020.000.00
HMAP (Ours)0.840.380.160.080.08

Table f. DrawerSearch Stages. Open and close a drawer, find the target, stop searching, press the button, and press it the correct number of times.

MethodCube-1Cube-2Cube-3Success
ACT0.220.000.000.04
DP0.500.1250.000.14
π0.5\pi_{0.5}0.460.040.1250.28
MemER-style0.260.040.000.08
HMAP (Ours)0.480.040.000.28

Table g. SugarCount Stages. Add the first, second, and third sugar cubes, followed by final task success. Sugar-addition stages may apply to different episode subsets.

MethodStoveInspectEitherBothSuccess
ACT1.000.240.000.000.00
DP0.980.180.000.000.00
π0.5\pi_{0.5}0.980.540.080.000.00
MemER-style1.000.580.040.000.00
HMAP (Ours)1.000.880.620.000.00

Table h. HeatAndInspect Stages. Turn on the target stove, inspect a drawer, complete either task thread, complete both threads, and achieve final task success.

MethodFaucetRetrieveEitherBothSuccess
ACT1.000.180.080.020.00
DP0.900.000.000.000.00
π0.5\pi_{0.5}1.000.460.400.080.00
MemER-style1.000.000.000.000.00
HMAP (Ours)1.000.400.320.000.00

Table i. FillAndRetrieve Stages. Turn on the faucet, retrieve the can, complete either task thread, complete both threads, and achieve final task success.

References

  1. [1]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, et al. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025. URL https://arxiv.org/abs/2511.21631.
  2. [2]Tianxing Chen, Yuran Wang, Mingleyang Li, Yan Qin, Hao Shi, et al. RMBench: Memory-dependent robotic manipulation benchmark with insights into policy design. arXiv preprint arXiv:2603.01229, 2026. URL https://arxiv.org/abs/2603.01229.
  3. [3]Egor Cherepanov, Nikita Kachaev, Alexey K. Kovalev, and Aleksandr I. Panov. Memory, benchmark & robots: A benchmark for solving complex tasks with reinforcement learning. arXiv preprint arXiv:2502.10550, 2025. URL https://arxiv.org/abs/2502.10550.
  4. [4]Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin CM Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems, 2023. doi: 10.15607/RSS.2023.XIX.026. URL https://roboticsproceedings.org/rss19/p026.html.DOI
  5. [5]Yinpei Dai, Hongze Fu, Jayjun Lee, Yuejiang Liu, Haoran Zhang, Jianing Yang, Chelsea Finn, Nima Fazeli, and Joyce Chai. RoboMME: Benchmarking and understanding memory for robotic generalist policies. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pp. 22396–22450. PMLR, 2026. URL https://proceedings.mlr.press/v306/dai26c.html.arxiv.org/abs/2603.04639
  6. [6]Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1–2):99–134, 1998. doi: 10.1016/S0004-3702(98)00023-X. URL https://doi.org/10.1016/S0004-3702(98)00023-X.
  7. [7]Huashuo Lei, Wenxuan Song, Huarui Zhang, Jieyuan Pei, Jiayi Chen, Haodong Yan, Han Zhao, Pengxiang Ding, Zhipeng Zhang, Lida Huang, Donglin Wang, Yan Wang, and Haoang Li. RoboMemArena: A comprehensive and challenging robotic memory benchmark. arXiv preprint arXiv:2605.10921, 2026. URL https://arxiv.org/abs/2605.10921.
  8. [8]Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabrael Levine, Wensi Ai, Benjamin Martinez, Hang Yin, Michael Lingelbach, Minjune Hwang, Ayano Hiranaka, Sujay Garlanka, Arman Aydin, Sharon Lee, Jiankai Sun, Mona Anvari, Manasi Sharma, Dhruva Bansal, Samuel Hunter, Kyu-Young Kim, Alan Lou, Caleb R. Matthews, Ivan Villa-Renteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Yunzhu Li, Silvio Savarese, Hyowon Gweon, C. Karen Liu, Jiajun Wu, and Li Fei-Fei. BEHAVIOR-1K: A human-centered, embodied ai benchmark with 1,000 everyday activities and realistic simulation. arXiv preprint arXiv:2403.09227, 2024. URL https://arxiv.org/abs/2403.09227.
  9. [11]Lars Osterberg, Maggie Wang, and Mac Schwager. UniMem: Unifying visual and textual memory in VLAs for non-markovian tasks. arXiv preprint arXiv:2608.22869, 2026. URL https://arxiv.org/abs/2608.22869v2.
  10. [12]Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, et al. π₀.₅: A vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025. URL https://arxiv.org/abs/2504.16054.
  11. [13]Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander Clegg, Michal Hlavac, So Yeon Min, Vladimír Vondruš, Theophile Gervet, Vincent-Pierre Berges, John M Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, and Roozbeh Mottaghi. Habitat 3.0: A co-habitat for humans, avatars, and robots. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=4znwzG92CE.arxiv.org/abs/2310.13724
  12. [14]Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9339–9347, 2019. URL https://openaccess.thecvf.com/content_ICCV_2019/html/Savva_Habitat_A_Platform_for_Embodied_AI_Research_ICCV_2019_paper.html.arxiv.org/abs/1904.01201
  13. [15]Hao Shi, Bin Xie, Yingfei Liu, Lin Sun, Fengrong Liu, Tiancai Wang, Erjin Zhou, Haoqiang Fan, Xiangyu Zhang, and Gao Huang. MemoryVLA: Perceptual-cognitive memory in vision-language-action models for robotic manipulation. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=54U3XHf7qq.arxiv.org/abs/2508.19236
  14. [16]Arth Shukla, Stone Tao, and Hao Su. ManiSkill-HAB: A benchmark for low-level manipulation in home rearrangement tasks. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=6bKEWegSd.arxiv.org/abs/2412.13211
  15. [17]Ajay Sridhar, Jennifer Pan, Satvik Sharma, and Chelsea Finn. MemER: Scaling up memory for robot control via experience retrieval. arXiv preprint arXiv:2510.20328, 2025. URL https://arxiv.org/abs/2510.20328.
  16. [18]Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems (NeurIPS), 2021. URL https://arxiv.org/abs/2106.14405.
  17. [19]Marcel Torne, Karl Pertsch, Homer Walke, Kyle Vedder, Suraj Nair, Brian Ichter, Allen Z. Ren, Haohuan Wang, Jiaming Tang, Kyle Stachowicz, Karan Dhabalia, Michael Equi, Quan Vuong, Jost Tobias Springenberg, Sergey Levine, Chelsea Finn, and Danny Driess. MEM: Multi-scale embodied memory for vision language action models. arXiv preprint arXiv:2603.03596, 2026. URL https://arxiv.org/abs/2603.03596.
  18. [20]Jianyuan Wang, Minghao Chen, Shangzhan Zhang, Nikita Karaev, Johannes Schönberger, Patrick Labatut, Piotr Bojanowski, David Novotny, Andrea Vedaldi, and Christian Rup-precht. VGGT-Ω. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21486–21499, 2026. URL https://openaccess.thecvf.com/content/CVPR2026/html/Wang_VGGT_ohm_CVPR_2026_paper.html.DOI
  19. [21]Haoming Xu, Zhenlin He, Hengyi Wang, Jiafeng Xu, and Hao Dong. Mimir: A neuro-symbolic memory system with dynamic grounding for embodied agents in interactive environments. arXiv preprint arXiv:2608.04933, 2026. URL https://arxiv.org/abs/2608.04933.
  20. [22]Ganlin Yang, Zhangzheng Tu, Yuqiang Yang, Sitong Mao, Junyi Dong, Tianxing Chen, Jiaqi Peng, Jing Xiong, Jiafei Cao, Jifeng Dai, Wengang Zhou, Yao Mu, and Tai Wang. EventVLA: Event-driven visual evidence memory for long-horizon vision-language-action policies. arXiv preprint arXiv:2606.20092, 2026. URL https://arxiv.org/abs/2606.20092.
  21. [23]Yihan Zeng, Minghao Ye, Yiyuan Chen, Yide Shentu, Philipp Wu, Zike Yan, and Zhongyu Li. KEMO: Event-driven keyframe memory for long-horizon robot manipulation with VLA policies. arXiv preprint arXiv:2606.23589, 2026. URL https://arxiv.org/abs/2606.23589.
  22. [24]Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In Proceedings of Robotics: Science and Systems, 2023. doi: 10.15607/RSS.2023.XIX.016. URL https://roboticsproceedings.org/rss19/p016.html.DOI

Paper details

vlnc:W_cr3a7dr5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents