WorksComputer Science
MESA: An Evaluation Framework for Compositional, Semantic, and Spatial Generalization in Robotics
Abstract
Despite growing interest in foundation models for robotic manipulation, systematic evaluation of their capabilities remains elusive. Existing benchmarks are either narrow and saturated, offering little signal on SOTA methods, broadly scoped, providing diversity at the cost of controlled experimentation, or require training on large, messy open-source datasets. We introduce MESA, an evaluation framework designed for precise, reproducible measurement of language-conditioned policy generalization. MESA provides five evaluation suites that isolate in-distribution performance and four distinct axes of generalization: spatial configuration, object instance, object category, and subtask composition, enabling researchers to diagnose where and why policies fail, not just whether they fail. To populate these suites at scale, we develop MESA-Gen, a pipeline for task and demonstration generation built by improving MimicGen. We instantiate the framework on two platforms and release both: the single-arm MESA suite, and BiMESA, the first large-scale benchmark for bimanual generalization. We evaluate seven baseline methods spanning diffusion policies, VLM-based approaches, and pretrained VLAs, finding that the strongest model achieves only 52% on MESA and 49% on BiMESA, providing meaningful signal while leaving ample room for progress. MESA rankings are broadly consistent with RoboArena scores, suggesting that MESA captures method-level differences relevant beyond simulation. Project website: https://pairlab.github.io/MESA.

Figure 1. MESA (Left) is an evaluation framework designed to enable researchers to probe language steering and generalization capabilities in settings accessible to researchers with academic budgets. We introduce MESA-Bench (Right), which includes single-arm and bimanual datasets and test sets designed to probe generalization to unseen spatial configurations, object instances, object categories and subtask compositions.
Introduction
10th Conference on Robot Learning (CoRL 2026), Austin TX, USA.
Introduction
Benchmarks shape the trajectory of research. They define what the community measures, and hence what it optimizes for. In robotic manipulation, recent progress in robotic foundation models (RFMs) has produced open-ended language-conditioned policies, long-horizon policies, and impressively dexterous manipulation policies [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]. Despite rapid progress, the capabilities, limitations, and failure modes of these models remain poorly understood. Current evaluation practices make systematic understanding of RFMs difficult, with prominent VLA efforts like [3, 8, 13], GR00T [6] and MolmoAct [7, 16] using completely different backbones, training data, architectural choices and evaluation protocols. This makes it difficult to isolate which design choices actually matter for generalization, sample efficiency, or language grounding.
Simulation offers a path toward reproducible evaluation, but existing benchmarks suffer from several complementary limitations. Some are narrow and saturated: LIBERO [17] is still used frequently despite state-of-the-art RFMs exceeding 98% success rate [18]. Other simulation benchmarks are diverse, but imprecise: RoboCasa [19] and Behavior-1K [20] provide myriad assets, but offer no clear methodology for experimentation with controlled distribution shifts. Recent work proposes evaluating real data-trained policies with a digital twin [21, 22, 23, 24, 25], but this requires training on the large, messy DROID [26] dataset. Finally, despite growing interest in bimanual manipulation, there are no multitask bimanual manipulation benchmarks commonly used by RFM researchers.
We propose that evaluation sets should isolate specific axes of generalization, enabling diagnosis of where and why policies fail. Evaluation should require policies to follow language instructions, rather than memorize task-specific behaviors or deduce tasks from visual observation alone. Frameworks should be extensible, enabling researchers to generate new tasks and evaluation suites as model capabilities evolve. Finally, they should support fast iteration, with training and evaluation feasible under limited compute.
To meet these objectives, we introduce MESA, an extensible evaluation framework for language-conditioned robotic manipulation. Our contributions are as follows:
MESA-Bench, a benchmark containing single-arm Franka and bimanual YAM embodiments, each instantiated with a canonical training set and five evaluation suites targeting in-distribution performance, spatial generalization, instance generalization, category generalization, and compositional generalization. We introduce a language-following metric based on distractor-task completion to disentangle instruction-grounding failures from control-execution failures.
MESA-Gen, a framework for scalable task and dataset generation. Researchers can curate targeted evaluations for emerging questions and synthesize the corresponding data to power their experiments. We introduce improvements to MimicGen [27] that enable higher-quality data generation under greater spatial variation and support generation for novel tasks.
Rigorous experiments evaluating seven methods spanning diffusion policies, VLM-based approaches, and pretrained VLAs. In the bimanual setting, our strongest baseline achieves only 48.7% success rate, dropping to 18.6% on novel object categories, suggesting substantial headroom for future improvement. Results correlate with real-world rankings, providing preliminary evidence that MESA captures method-level differences relevant beyond simulation.
Related Work
Simulation offers reproducible, scalable evaluation without the cost and variability of real-world rollouts. However, existing simulated benchmarks for RFMs are insufficient, and we share a detailed taxonomy in Tab. 1.
Simulated Benchmarks. A useful benchmark for modern RFMs should provide axis-aligned splits that isolate specific generalization axes, remain tractable to train and evaluate on academic compute, and target a setting compatible with RFM pretraining distributions. No existing simulated benchmark satisfies all three. LIBERO [17] and its predecessors [32, 28, 31] are tractable and embodiment-
| Benchmark | Num. Tasks | Obj. / Insts | MT | Action Space | Train Data | OOD | Bim. |
| RLBench [28] | 100 | 28 / 28 | ✓ | Keyframe | Scripted | ✗ | ✓ |
| PerAct2 [29] | 13 | 15 / 15 | ✓ | Keyframe | Scripted | ✗ | ✓ |
| COLOSSEUM [30] | 20 | 28 / 28 | ✗ | Keyframe | Scripted | ✓ | ✗ |
| CALVIN [31] | 34 | 5 / 30 | ✓ | Low-level | Human | ✗ | ✗ |
| MetaWorld [32] | 50 | 38 / 38 | ✗ | Low-level | None | ✗ | ✗ |
| LIBERO [17] | 130 | 51 / 75 | ✓ | Low-level | Human | ✗ | ✗ |
| SIMPLER [33] | 8 | 17 / 17 | ✓ | Low-level | External | ✗ | ✓ |
| Behavior-1K [20] | 1000 | 2211 / 9331 | ✓ | Both | None | ✗ | ✓ |
| VLABench [34] | 100 | 163 / 2164 | ✓ | Keyframe | Scripted | ✓ | ✗ |
| Robomimic [35] | 8 | 12 / 12 | ✗ | Low-level | Human | ✗ | ✓ |
| RoboCasa [19] | 100 | 153 / 2509 | ✓ | Low-level | MG 1→1 | ✗ | ✗ |
| RoboCasa365 [36] | 365 | 153 / 2509 | ✓ | Low-level | MG 1→1 | ✓ | ✗ |
| RoboTwin 2.0 [37] | 50 | 147 / 731 | ✗ | Low-level | Scripted | ✗ | ✓ |
| VLA-Arena [38] | 170 | 153 / 2509 | ✓ | Low-level | Human | ✓ | ✗ |
| MolmoSpaces [24] | 8 | 2.8k / 130k | ✓ | Both | Scripted | ✗ | ✗ |
| REALM [25] | 7 | 3500+ | ✓ | Low-level | External | ✗ | ✗ |
| RoboLab [21] | 120 | ∼96 / 298 | ✓ | Low-level | External | ✗ | ✗ |
| MESA | 1021 | 153 / 2509 | ✓ | Low-level | MG few→many | ✓ | ✓ |
caption: Comparison of robot manipulation evaluation suites. MESA is, to our knowledge, the first benchmark to combine large-scale multitask language-conditioned evaluation with low-level control, controlled OOD splits, and bimanual tasks. It uses MESA-Gen to generate demonstrations for many tasks from a small set of source demonstrations and reusable subtasks, enabling benchmark-specific training data at scale. For MESA, Num. Tasks counts all single-arm and bimanual training and evaluation tasks. Obj. / Insts: object categories / unique object instances; MT: multitask language-conditioned; Train Data: source of provided training demonstrations, where MG 1→1 uses MimicGen to expand human demonstrations of each task and MG few→many generates data for many tasks from source demonstrations of a few; OOD: controlled out-of-distribution evaluation splits; Bim.: bimanual tasks.
:::
compatible but saturated, with recent RFMs exceeding 98% success [18]. OOD follow-ups attempt to address this [39, 40] but inherit the original overfit-prone training data and yield near-zero success rate for all current RFMs, offering little signal for ranking relative progress. On the other hand, broad suites such as RoboCasa [19, 36] and Behavior-1K [20] target large mobile-manipulation or long-horizon household activities that fall outside typical RFM pretraining regimes and lack targeted OOD evaluation sets. Orthogonally, several benchmarks provide bimanual settings without sufficient diversity to benchmark language following and generalization [29, 35, 37, 41, 42]. MESA satisfies all these requirements by providing axis-aligned evaluation suites in a tabletop manipulation setting, with a training set that is tractable to train on academic compute while still providing meaningful signal for modern RFMs. Additionally, MESA is, to our knowledge, the first benchmark to test language steerability and axis-aligned generalization splits for both single-arm and bimanual embodiments.
Real-to-Sim Benchmarks. Many recent works evaluate policies trained from large-scale real datasets in simulation [33, 23, 22, 24, 25, 21]. In particular, several involve training on the expansive DROID dataset [26] and evaluating on a simulated digital twin [22, 24, 25, 21]. This paradigm has several drawbacks. First, DROID’s scale and opaque coverage make it expensive to train on and difficult to construct OOD settings that are guaranteed to be absent from training. Second, the dataset is messy, with many inaccurate language instructions or long meaningless actions, such that simply filtering it can yield outsized performance gains [16], confounding the true source of any reported improvement. On the other hand, MESA provides small, canonical datasets with sufficient diversity to benchmark language grounding and generalization without excessive scale slowing iteration.
Generating Benchmarks with MESA-Gen
MESA-Gen is a framework for procedurally generating manipulation tasks and corresponding demonstration data (Figure 2), which we use to instantiate MESA-Bench (§ 5). A core principle is that every demonstration and evaluation trajectory uses a unique scene with a unique set of distractor objects, making language following critical even in-distribution: the policy must disambiguate between multiple plausible goals. Concretely, we specify each task as in- and out-of-distribution BDDL files [43, 44] and use MimicGen [27] to generate in-distribution demonstrations.

Figure 2. MESA-Gen experiment building framework. (1) If a suitable task skeleton does not already exist, define it in Python. (2) Instantiate a task from the skeleton and generate BDDL files for the training and evaluation sets. (3) Optionally, collect a small number of source demonstrations. (4) Generate a large dataset for imitation learning. Using subtask stitching enables data generation for tasks without human data.
Generating Diverse Manipulation Tasks
The first step in MESA-Gen is to procedurally generate task variants for training and evaluation from a task skeleton: a Python class specifying initialization, demonstration, and goal predicates (App. § 11.§ 6). Given a skeleton and a set of object categories, MESA emits BDDL files for the training and evaluation sets, each with a unique scene layout, distractor set, and, optionally, varied textures, camera pose, and robot embodiment (Figure 2).
Ensuring Visual Confusion: To benchmark language steerability, we must prevent the policy from inferring the task solely from visual observations. If a scene contains only one interactable object, the policy may ignore the language instruction and simply act on that object, a form of causal confusion [45]. To ensure high observation-conditioned task distribution entropy, we sample distractors such that every scene contains multiple seen object-destination pairs, as elaborated in App. § 11.§ 5.
Scalable Data Generation with MimicGen
To generate demonstrations at scale, MESA-Gen builds on MimicGen [27], an object-centric data generation pipeline. We extend it along two axes. First, we introduce subtask stitching: rather than requiring a human demonstration for every new task, MESA-Gen composes object-centric subtasks (e.g., Grasp(apple), On(·, plate)) drawn from a shared library indexed by predicate type and object semantics, enabling data generation for tasks with no human source data (App. § 12.§ 4). Second, we introduce trajectory optimizations that improve data generation quality under large source-to-target object pose differences, avoiding the unnatural sweeping motions and self-collisions that vanilla MimicGen produces under large spatial variation (App. § 12.§ 5). Together, these enable MESA-Gen to generate high-quality data for novel tasks under substantially greater spatial variation than prior MimicGen-based pipelines.
Evaluating Language Following
Recent VLA benchmarks have emphasized language following as a key capability [4, 8, 46, 6], but the term is often defined imprecisely. To address this, MESA-Gen uses the BDDL predicate system to explicitly measure whether a policy completed the intended task or an alternative distractor task. For each scene, we enumerate feasible distractor tasks from pairs of distractor objects and detect when a failed rollout completes one. We define the language following rate as where is the success rate and is the distractor task completion rate. Intuitively, this measures the probability that, conditioned on completing some feasible task, the policy completed the correct one.
MESA-Bench
We use MESA-Gen to instantiate MESA-Bench, a fixed benchmark to support accessible and reproducible policy evaluation. MESA-Bench includes two embodiments: a DROID-like single-arm Franka Panda [26] and a bimanual YAM setup. For each embodiment we provide a training set (§ 5.1) and five evaluation suites (§ 5.2) probing in-distribution performance and OOD performance along four axes of generalization. Benchmark construction and per-suite details are in App. § 13.
Training Set
We provide one shared training set per embodiment, used for all evaluation sets. The single-arm set, MESA-70, contains 7,000 demonstrations across 70 tasks, while the bimanual set, BiMESA-57, contains 6,833 demonstrations across 57 tasks. We additionally provide 736 auxiliary single-arm tasks, MESA-Aux, for studying diversity scaling (§ 6.4). Further dataset details are provided in App. § 13.§ 5.
Evaluation Suites
For all trials, we sample an unseen set of distractor objects and poses and generate a fixed set of initial states for unbiased comparisons across policies. The full task lists and partition definitions are in App. § 13, and per-task success rates are in App. § 10.8.
MESA-ID tests policy performance on in-distribution tasks: 70 single-arm tasks (MESA-70) and 57 bimanual tasks (BiMESA-57). Each scene has an unseen set of distractor objects, meaning that policies must precisely follow language instructions to succeed in this setting.
MESA-Spatial evaluates policies under changes in object location. The single-arm setting contains 10 tasks: we partition the table into two halves and (one left, one right) and only initialize the target objects in during training and during evaluation. The bimanual setting contains 38 tasks and tests a form of spatial generalization with no single-arm analogue: relocating the scene changes which arm must act and how the arms coordinate, requiring policies to solve held-out arm assignments, reversed handoff directions, and cross-body interactions (App. § 13.§ 4).
MESA-Instance tests generalization to unseen object instances with 20 single-arm and 18 bimanual tasks. For each task, we train on one subset of object assets and evaluate on a disjoint held-out subset.
MESA-Composite consists of unseen compositions of seen subtasks with 20 single-arm and 15 bimanual tasks. These include tasks with familiar structure but novel object combinations and longer-horizon tasks with novel structure (e.g., two-step pick-and-place).
MESA-Category consists of tasks involving object categories never seen during training with 20 single-arm and 17 bimanual tasks. Policies must rely on broad vision-language understanding beyond the finetuning dataset.
Experiments
We organize our experiments around the following questions: (1) How do state-of-the-art RFMs and baseline policies perform under controlled distribution shifts? (2) To what extent is poor language following responsible for policy failures? (3) How does naïvely scaling data quantity and diversity influence policy performance? (4) Are the results from MESA well-correlated with results from real-world policy evaluations?
We evaluate seven baselines on MESA-Bench spanning three families: (i) a tabula rasa Multitask Diffusion Policy (DP) [47, 48]; (ii) VLM finetuning baselines that adapt a PaliGemma-3B [49] backbone with either action binning (PG-Bin) [50, 1, 2] or a flow-matching action head (PG-FM, equivalent to [3] without robot pretraining); and (iii) pretrained VLA models, namely GROOT-N1.6 [6], [3], -FAST [4], and [8]. Full architectural and training details are in App. § 14. We finetune each baseline on the MESA-70 and BiMESA-57 datasets.
Baseline Results
We present success rates for MESA and BiMESA in Table 2, with confidence intervals and paired significance tests in App. § 13.§ 7. In the single-arm setting, a clear hierarchy emerges: achieves the strongest average performance, followed by -FAST, , PG-FM and GR00T-N1.6. The BiMESA results are more nuanced, with GR00T-N1.6 achieving more competitive performance and underperforming PG-FM despite its robot pretraining. DP’s poor performance, especially in the single-arm setting, reflects poor multi-task scaling properties (App. § 10.§ 2).
| Setting | DP | PG-Bin | PG-FM | GR00T-N1.6 | FAST | GPT-6.1-Sol† | ||
| MESA-ID | 1.6 | – | 30.7 | 25.0 | 43.1 | 46.5 | 62.8 | – |
| MESA-Spatial | 1.0 | – | 35.0 | 23.2 | 42.2 | 53.6 | 68.6 | – |
| MESA-Instance | 1.9 | – | 22.7 | 22.7 | 34.0 | 39.1 | 55.4 | – |
| MESA-Composite | 0.3 | – | 17.7 | 15.0 | 27.5 | 28.9 | 46.7 | – |
| MESA-Category | 0.7 | – | 7.7 | 14.1 | 11.6 | 8.4 | 26.4 | – |
| MESA Avg. | 1.1 | – | 22.8 | 20.0 | 31.7 | 35.3 | 52.0 | – |
| BiMESA-ID | 13.0 | 26.7 | 43.8 | 47.0 | 45.9 | 53.7 | 68.7 | 41.2 |
| BiMESA-Spatial | 18.8 | 30.3 | 50.8 | 46.2 | 49.2 | 58.1 | 70.5 | 41.8 |
| BiMESA-Instance | 9.4 | 13.9 | 28.6 | 36.7 | 23.9 | 36.7 | 48.1 | 43.9 |
| BiMESA-Composite | 5.7 | 14.4 | 17.9 | 22.1 | 18.4 | 28.8 | 37.5 | 34.7 |
| BiMESA-Category | 5.8 | 6.0 | 10.9 | 26.1 | 10.6 | 19.8 | 18.6 | 52.4 |
| BiMESA Avg. | 10.6 | 18.3 | 30.4 | 35.6 | 29.6 | 39.4 | 48.7 | 42.8 |
Table 2. Table 2: Success rates (%) on the single-arm and bimanual suites. achieves the strongest average performance on both embodiments. FAST also performs well, while GR00T-N1.6 is particularly competitive on BiMESA despite weaker single-arm performance. DP performs poorly, and PG-Bin underperforms PG-FM.
Robot pretraining does not uniformly transfer. The comparison between and PG-FM is particularly informative: they share a backbone and architecture, differing primarily in ’s pretraining on a large robot dataset that includes both single-arm Franka and bimanual YAM-like embodiments [4]. In the single-arm setting, robot pretraining leads to a consistent improvement, but in the bimanual setting PG-FM slightly outperforms on average. This surprising reversal suggests that embodiment similarity alone does not guarantee transfer from pretraining.
Action representation matters. PG-Bin’s performance is worse than PG-FM on BiMESA despite sharing its VLM backbone. This indicates that directly discretizing continuous action dimensions into language tokens is a poor fit for the fine-grained control required by BiMESA.
MESA-Category represents an open frontier. Performance drops sharply for all finetuned models on novel object categories. GR00T-N1.6 outperforms and -FAST here despite weaker single-arm performance, possibly because its frozen Cosmos backbone provides good priors for novel objects. In contrast, the zero-shot GPT-6.1-Sol agent (App. § 10.7) is reasonable at pick-and-place (64.1% on BiMESA-ID) but weak on handoffs (26.5%) and articulated manipulation (13.1%), yet does not degrade on unseen instances or categories (43.9% and 52.4% vs. 41.2%). This suggests that open-vocabulary grounding in current open VLAs has substantial room to improve.
MESA-Bench provides strong signal. The strongest model achieves 52.0% on MESA and 48.7% on BiMESA, indicating substantial headroom. At the same time, modern VLA baselines make meaningful progress across all settings, making MESA-Bench a high-signal benchmark for future work.
Controlled Distribution Shifts
Since MESA-Spatial and MESA-Instance each contain tasks with in-distribution counterparts in MESA-ID, we can directly compare success rates with and without distribution shift (Figure 3).
Figure 3. Controlled spatial and instance shifts reduce performance. For tasks with matched in-distribution counterparts, policies generally perform worse under held-out spatial configurations and object instances, showing that MESA isolates failures under these distribution shifts. Error bars are 95% Wilson intervals.
Across both spatial and instance-level shifts, performance generally drops relative to matched in-distribution tasks, indicating that MESA evaluates more than task memorization. For spatial shifts, GR00T-N1.6 and show large degradations, while , FAST and PG-FM are comparatively robust. This pattern is consistent with the hypothesis that preserving the base VLM’s spatial priors, either through knowledge insulation or direct finetuning from the VLM without broad robot pretraining, can improve transfer to unseen spatial configurations. With unseen object instances, degradation is more uniform, indicating that instance generalization remains a broadly challenging axis. These comparisons show how MESA can isolate the effect of specific distribution shifts: rather than only reporting aggregate success, it reveals which models generalize and when they do so.
Probing Language Following
Reliably following language instructions is a core objective for RFMs, but existing benchmarks offer no way to study language following beyond using success rate as a proxy. We present language following rate (LFR) (§ 4.2) in Table 3.
| Setting | DP | PG-Bin | PG-FM | GR00T-N1.6 | FAST | GPT-6.1-Sol | ||
| MESA-ID | 18.9 | – | 80.0 | 71.4 | 79.7 | 83.7 | 87.6 | – |
| MESA-Spatial | 11.4 | – | 77.4 | 53.7 | 73.0 | 78.6 | 84.5 | – |
| MESA-Instance | 23.5 | – | 66.6 | 60.5 | 66.5 | 69.0 | 76.9 | – |
| MESA-Composite | 6.4 | – | 65.1 | 56.2 | 65.5 | 66.3 | 81.5 | – |
| MESA-Category | 12.1 | – | 32.6 | 42.5 | 33.1 | 28.0 | 44.1 | – |
| MESA Avg. | 14.4 | – | 64.3 | 56.9 | 63.6 | 65.1 | 74.9 | – |
| BiMESA-ID | 34.9 | 78.7 | 84.6 | 87.7 | 73.4 | 86.9 | 92.9 | 73.2 |
| BiMESA-Spatial | 45.3 | 75.7 | 86.2 | 77.5 | 72.6 | 83.0 | 89.1 | 69.7 |
| BiMESA-Instance | 25.5 | 53.0 | 64.4 | 73.3 | 48.0 | 64.8 | 74.0 | 65.8 |
| BiMESA-Composite | 14.5 | 65.1 | 66.7 | 71.2 | 42.3 | 66.7 | 80.1 | 74.3 |
| BiMESA-Category | 15.7 | 27.4 | 30.1 | 55.6 | 21.2 | 35.1 | 28.7 | 79.5 |
| BiMESA Avg. | 27.2 | 60.0 | 66.4 | 73.1 | 51.5 | 67.3 | 73.0 | 72.5 |
Table 3. Language following rate (%, § 4.2). Models with VLM backbones exhibit strong language following, while models trained from scratch do worse. Lower performance in MESA-Category reflects that with unfamiliar objects, our baselines are bottlenecked by their ability to ground language to novel visual referents rather than low-level execution. 1
Strong VLM backbones are important for language following. On in-distribution tasks (MESA-ID and BiMESA-ID), models with VLM backbones exhibit strong LFR: PG-FM, , and -FAST all exceed 70%, and reaches 92.9% for BiMESA-ID. When these models fail, they typically fail to complete any task rather than completing the wrong one. DP, which has weaker language conditioning, achieves a very low language following rate.
Furthermore, we find that pretraining recipe has an outsize effect on LFR, and this is especially obvious with unseen object categories, which present the greatest language following challenge. GR00T-N1.6, which uses a frozen (and thus better-preserved) VLM backbone, consistently achieves strong LFR here, and is strongest by far among finetuned policies in BiMESA-Category. This suggests that preserving open-vocabulary visual-language knowledge can be important when policies must ground instructions involving object categories not seen during robot training.
Our language following metric provides a diagnostic that success rate alone cannot: it distinguishes successful execution of a distractor task from completion of the intended task, revealing grounding failures despite competent execution. Episodes in which neither task is completed remain ambiguous, since they may involve grounding errors, execution failures, or both.
Scaling Data Quantity and Diversity
We study the effect of increasing demonstration quantity for and in the single-arm setting (Figure 4) with a fixed compute budget. A striking result is that trained with 10 demonstrations per task consistently outperforms with 100. In this regime, pretraining quality dominates data quantity, suggesting that investing in better pretraining or representation preservation yields greater returns than collecting larger fine-tuning datasets.
Figure 4. Scaling data quantity. Scaling data quantity almost monotonically improves performance. However, pretraining dominates as with 10 demonstrations consistently outperforms with 100.
We investigate whether co-training on diverse auxiliary tasks improves single-arm generalization under a fixed compute budget by adding 10 demonstrations from each of 736 auxiliary tasks. These tasks reuse the MESA-70 training verbs with different objects and do not overlap the held-out evaluation settings. We observe no consistent gains across models and suites (Table 4). One exception is on the 10-task Spatial suite, where success increases from 42.2% to 54.4%; however, this isolated result does not establish a general benefit for spatial generalization. On the remaining axes, shows no statistically significant improvements, and Category performance decreases from 11.6% to 8.7%. For , no statistically significant changes are observed across the five suites. Overall, increasing auxiliary task diversity alone does not yield consistent generalization gains in this evaluation setting.
| ID | Spatial | Instance | Composite | Category | |
| DP | 1.6 | 1.0 | 1.9 | 0.3 | 0.7 |
| DP+Aux | 2.3 | 2.4 | 2.2 | 0.6 | 0.3 |
| 43.1 | 42.2 | 34.0 | 27.5 | 11.6 | |
| +Aux | 43.4 | 54.4 | 33.2 | 28.8 | 8.7 |
| 62.8 | 68.6 | 55.4 | 46.7 | 26.4 | |
| +Aux | 59.0 | 69.2 | 53.6 | 45.6 | 23.0 |
Table 4. Scaling task diversity via auxiliary co-training yields negligible gains. We double the training set size by adding auxiliary demonstrations from MESA-Aux and train for the same number of steps. We find that the co-trained policies show little if any improvement, and hypothesize that naively scaling co-training task diversity does not automatically improve generalization.
Studying Cross-Arm Transfer
Because BiMESA-Spatial holds out the arm assignment rather than relocating objects corresponding to a single end-effector, it probes whether a policy has acquired arm-symmetric skills or has instead memorized per-arm trajectories, a failure mode with no analogue in the single-arm setting. Each task has an in-distribution counterpart in BiMESA-57, enabling a direct trained-vs.-arm-switched comparison. We find that for tasks involving one arm (all task-relevant objects are on one side) or handoff tasks, policies are relatively capable of switching arms or handoff directions. However, they struggle to transfer articulated manipulation skills across arms. This suggests that current policies can often reuse reaching and object-transfer behaviors across arms, but struggle when cross-arm transfer requires precise contacts with articulated objects.
Figure 5. BiMESA reveals that cross-arm transfer is hardest for articulated manipulation. BiMESA-Spatial holds out arm assignments, allowing us to compare in-distribution tasks with matched arm-switched variants. Policies show modest degradation when switching single-arm tasks (left) or reversing handoff direction (middle), but much larger drops when transferring articulated manipulation skills across arms (right). DP, PG-Bin and -FAST are exceptions and do not degrade on reversed handoffs. Error bars are 95% Wilson intervals.
Comparison with Real-World Performance
While simulated benchmarks primarily provide controlled and reproducible diagnostics, their results are more useful when they align with trends observed in real-world evaluations. Inspired by Jain et al. [22], in Table 5 we compare MESA success rates to Elo scores on RoboArena [51], a crowdsourced evaluation platform where higher Elo scores reflect aggregate pairwise blind human preferences.
| PG-Bin | PG-FM | -FAST | |||
| RoboArena | 771 | 1527 | 1449 | 1564 | 1594 |
| MESA | – | 22.8 | 31.7 | 35.3 | 52.0 |
| BiMESA | 18.3 | 30.4 | 29.6 | 39.4 | 48.7 |
Table 5. Comparing RoboArena [51] Elo with MESA success rates. MESA and BiMESA exhibit broadly similar stack rankings to RoboArena, suggesting that MESA captures method-level differences relevant beyond simulation.
The rankings are broadly consistent: and -FAST are the strongest methods across evaluations, while PG-FM and occupy the middle of the ranking. The main disagreement is the ordering of PG-FM and , which differs between RoboArena and the single-arm MESA suite. Overall, the agreement between simulated benchmark rankings and RoboArena scores provides preliminary evidence that MESA captures method-level differences relevant beyond simulation. Since RoboArena evaluates policies trained on real data while MESA evaluates the same architectures finetuned on MESA data, we also evaluate the real-trained DROID checkpoints zero-shot in MESA; these runs had success rates too low for reliable ranking (App. § 10.§ 7).
Conclusion and Limitations
We introduced MESA, a language-conditioned robotic manipulation benchmarking framework consisting of MESA-Gen, a scalable scene and data generation pipeline, and MESA-Bench, a fixed, reproducible benchmark. Our experiments show that current RFMs leave substantial room for improvement in language following and out-of-distribution generalization, and that MESA rankings are broadly consistent with real-world evaluation trends. These results position MESA as a scalable testbed for measuring progress in general-purpose robotic manipulation.
A key limitation of MESA is its task distribution: most tasks involve pick-and-place or simple articulated manipulation, leaving more difficult settings such as dexterous, long-horizon, and mobile manipulation to future extensions. Additionally, our language-following metric’s simplicity makes it robust at the cost of granularity, since it does not capture partial task completion or cases where a policy attempts the correct instruction but fails during execution.
MESA prioritizes controlled distribution shifts and accessible experimentation. Establishing how its method rankings and axis-specific findings carry over to real robots remains an important direction for future validation. Rankings are conditional on the supplied training data and evaluation protocol; we have not tested whether alternative data sources, such as a matched human-teleoperation dataset, would preserve the ordering.
Acknowledgments
The authors would like to acknowledge the State of Georgia, the Agricultural Technology Research Program at Georgia Tech, and microagi for supporting the work described in this paper. We are thankful to all the reviewers whose comments helped to strengthen the paper.
Supplement Overview and FAQs
In Appendix § 10, we provide additional experimental results.
In Appendix § 11, we provide detailed information about the MESA-Gen framework.
In Appendix § 12, we provide additional information about our data generation pipeline.
In Appendix § 13, we provide additional details about MESA-Bench.
In Appendix § 14, we provide detailed policy training information.
Frequently Asked Questions (FAQs)
Why should I use MESA? MESA enables quick iteration while giving strong signal about language steering and performance along our axes of generalization. Furthermore, it is an easily extensible platform for designing new experimental settings.
What explains diffusion policy’s unexpectedly poor performance? We study this question in depth in App. § 10.§ 2. To summarize, we find that our implementation works well in LIBERO and that DP simply struggles to fit the diverse setting presented by the benchmark.
What is this benchmark testing? We designed this benchmark to test language steering and policy performance under changes in spatial arrangement, object instance, object category and with unseen subtask compositions. BiMESA additionally tests bimanual generalization through held-out arm assignments, reversed handoff directions, and cross-arm articulated manipulation.
What does this benchmark not test for? This benchmark is not geared towards dexterous or precise tasks, open-ended language steerability, long-horizon tasks or mobile manipulation.
What motivates the lack of scene and camera pose diversity? We purposely limit diversity along axes we don’t test in order to minimize the amount of training required. That being said, this is implemented and easy to add back if desired.
What are the limits of MimicGen data? MimicGen data has several limitations, especially for our domain with such scene variety. It is not well suited to deal with dense clutter, and even with our optimizations it sometimes outputs suboptimal trajectories. We do not claim in this paper to present a general solution for robot data generation, or that our data is optimal. Rather, we claim that MimicGen data is useful for inexpensively generating data for experiments, and that the stack ranking of policies trained on this data is broadly consistent with real-world rankings.
What types of tasks are included in the benchmark? The benchmark mainly focuses on pick and place and articulated manipulation tasks. This limited scope allows us to quickly generate data with MimicGen and train policies that achieve meaningful generalization under reasonable compute budgets.
Additional Experimental Results
Why Diffusion Policy Scales Poorly in Multi-Task MESA
One surprising finding is that, although we expected the diffusion policy to underperform relative to its VLA counterparts, it performs substantially worse than anticipated on the single-arm MESA benchmark, achieving an average success rate of 1.1%. We investigate this in Figure 6, where we train a DiT policy [48] for a single task with increasing numbers of demonstrations and cotraining tasks. We see that it achieves a reasonable success rate with 1000 demonstrations, but performance drops quickly as the number of demonstrations decreases. Furthermore, we see that it especially struggles when switching from a single-task to multi-task setting, consistently achieving a 0% success rate when cotraining with other tasks. A likely explanation is that, compared to the baselines using a VLM backbone, the DiT policy has a much weaker vision-language conditioning pathway and cannot adequately fit the diverse data included in the MESA dataset.
Figure 6. We study the performance of diffusion policy on a single task (“put the lime in the bowl”) as we scale the number of demonstrations and cotraining tasks. (a) We see that with 1000 demonstrations it achieves reasonable performance, but with 100 demonstrations, the number present for each task in MESA-70, its performance drops sharply. (b) With 100 demonstrations we see that as we increase the number of cotraining tasks, performance quickly drops to zero.
Verifying with LIBERO
To verify our DiT Policy implementation we train and evaluate on LIBERO-90 [17]. We achieve a success rate of 90.7%, which is comparable to results reported elsewhere [52, 53] and provides evidence against a general implementation failure. We hypothesize that MESA’s substantial variation in object sets and placements across episodes increases the demands on vision-language grounding and contributes to this implementation’s poor multi-task performance.
Ablating Language Instructions
A core claim from our work is that the diverse initial configurations create a consistent requirement for language steerability. To verify this claim, we rerun single-arm evaluations for , the strongest baseline, while scrambling the natural language instructions (precisely, for each evaluation we sample a random instruction from the benchmark), and present results in Figure 7. We see that this change leads to a substantial drop in success rate, with successes only occurring when the policy successfully guesses the correct instruction. This verifies our claim that accurate language steering is essential to perform well in the MESA benchmark.
Figure 7. We evaluate with the original and randomized language instructions in the single-arm MESA setting. This leads to a substantial performance drop, indicating the necessity of language following for MESA.
Ablating MimicGen Changes
In App. § 12 we present several modifications to MimicGen to improve data generation in our setting with many tasks and objects. These changes include subtask stitching and the trajectory starting point optimization procedure. The optimization procedure has two main components: (i) we remove roll, pitch and optionally yaw from the source and target object poses and (ii) we optimize the starting point of the trajectory as described in App. § 12.§ 5. Subtask stitching has a clear benefit – enabling data generation across over 800 tasks after only collecting data from 80 – which is difficult to ablate. We instead ablate the changes to the optimization procedure in Figure 8, running the experiment in the single-arm setting. As shown in Figure 8a, these changes both lead to an improvement in data generation rate, reducing compute requirements when generating large datasets. In Figure 8b, we additionally observe that removing the optimization procedure leads to a substantial drop in performance. This is likely because without optimizing the start of the trajectory, the generated data can include large, unnecessary interpolations to distant regions before moving towards the target object. In effect, the data includes many suboptimal motions that can confuse the policy. This interpretation is further supported by trajectory-length statistics: removing the optimization procedure increases the average trajectory length from 141.7 to 177.9.
Figure 8. We study the effects of our MimicGen changes when generating and training on 1000 demos for the “put the lime in the bowl” task. We ablate the omission of roll, pitch and yaw (− RPY) and the starting point optimization procedure (− Opt). In (a) we study data generation rate, or the success rate of MimicGen at data generation time, and see that our changes lead to a consistent improvement. In (b) we study downstream policy performance and see that especially the starting point optimization leads to a substantial policy improvement.
Bias in Generated Data Toward High Success-Rate Initial Conditions
MimicGen works by initializing the environment, attempting to solve the task using its trajectory transformations, and then reinitializing it once it exceeds some retry budget. This process can bias the dataset towards initial configurations where MimicGen is likely to succeed, creating issues for the downstream policy. We study this issue by computing normalized initial position frequency for the single-arm setting in Figure 9. We observe minor bias in the dataset with more data in the lower-right quadrant and fewer in the upper-left quadrant. However, within each tile, the deviation from an unbiased distribution is at most about 15%, reflecting a relatively evenly distributed dataset.
Figure 9. We study whether MimicGen led to a biased dataset by visualizing the frequency with which different target object initial positions appear in our dataset. In (a), we show this value, and in (b) we show the frequency with which initial states spawn before MimicGen. Since these distributions are highly correlated, we show the normalized distribution in (c). We see that there is some bias towards the lower right corner of the region and away from the upper left corner. In (c), values greater than one reflect higher frequency in the dataset than an unbiased dataset would have and values less than one are the opposite.
Visualizing Confusion Matrices
In Figure 10 we visualize confusion matrices for DP and from the single-arm setting, which we can compute directly using the logged distractor task completion data. One notable observation is that, despite being more performant overall, is confused more often. This is likely because it is simply more capable of successfully grasping and placing objects; as a result, even if it occasionally takes an incorrect action, it is still likely to complete the distractor task and register as a confusion. A second observation is that the confusion matrix for is much more “spiky”, with a handful of frequently repeated mistakes, whereas diffusion policy’s confusions are scattered more uniformly across the confusion matrix. Importantly, the high-frequency confusions made by are intuitive. For example, peaches and tomatoes look visually similar to apples and are therefore often confused with them. This suggests that is generally making reasonable attempts to select the intended object, with errors driven by visual similarity. In contrast, diffusion policy seems to select incorrect objects more indiscriminately, consistent with behavior that resembles grabbing random objects.
Figure 10. We compare confusion matrices for diffusion policy and . Entry in these plots shows the frequency with which a given policy completed a distractor task by grasping when it was supposed to grasp (rows are target objects; columns are the objects picked up). For example, both confusion matrices show a lot of confusion for tomato, apple because the policy frequently grasped an apple when it was supposed to grasp a tomato. We condensed them to a reasonable size by selecting the top 15 most confusing entries in each matrix and removing rows and columns corresponding to objects not involved in those entries. For example, no object was commonly confused for mushrooms, so their rows and columns were removed.
Zero-Shot Evaluation of Real-Trained Policies
We evaluate the DROID checkpoints of , , and in the single-arm MESA environment without finetuning on MESA data. We match action and observation spaces, normalization, action horizon, and joint dynamics, and use system identification to tune the controller gain from to 234; this adjustment yields no observed performance benefit. The intermediate subtask completion rates are 0.60% for , 0.23% for , and 1.77% for . These measurements concern intermediate subtask completion rather than the full-task success metric used in our main results. The low absolute rates provide insufficient signal for reliable ranking conclusions, so this experiment does not establish alignment with real-world performance without MESA finetuning.
Zero-Shot Evaluation of an LLM Agent
We evaluate GPT-6.1-Sol on BiMESA without BiMESA training data, using the RoboDojo L3 Inspect EEF harness [54]. Each call, the model receives the instruction, the egocentric and both wrist images at , each arm’s end-effector pose, joint angles, and measured jaw opening, and the remaining step budget; it receives no object poses, depth, or detections. It responds with one tool call that sets end-effector and gripper targets for either or both arms, or gives up. We replace the harness’s CuRobo planner with damped least-squares IK, which does not check collisions; the resulting joint-position waypoints drive the same controller as the baselines. We adapt the harness’s embodiment notes to the YAM arms and drop RoboDojo’s return-to-home requirement. Otherwise, we use the published configuration (170 calls, medium reasoning effort) without per-task recipes, and the baselines’ 500-step limit. We evaluate rollouts 0–9 of each task (1,450 in total) to reduce API call costs. Model latency is a median the simulated robot time, so the agent does not run in real time.
Per-Task Success Rates for MESA
Here we list success rates across all tasks present in the benchmark. We present success rates for MESA-ID in Table 6, MESA-Spatial in Table 7, MESA-Instance in Table 8, MESA-Composite in Table 9, and MESA-Category in Table 10.
| MESA-ID | DP | PG-FM | GR00T-N1.6 | FAST | ||
| put the apple on the tray | 2.0 | 54.0 | 62.0 | 66.0 | 64.0 | 66.0 |
| put the avocado in the basket | 0.0 | 58.0 | 18.0 | 50.0 | 70.0 | 88.0 |
| put the bagel on the cutting board | 8.0 | 26.0 | 36.0 | 48.0 | 36.0 | 86.0 |
| put the bar in the bottom drawer of the cabinet and close it | 0.0 | 20.0 | 12.0 | 22.0 | 30.0 | 46.0 |
| put the bar soap in the top drawer of the cabinet and close it | 0.0 | 24.0 | 20.0 | 24.0 | 58.0 | 60.0 |
| put the beer in the slide cabinet and close it | 0.0 | 6.0 | 6.0 | 10.0 | 10.0 | 34.0 |
| put the bell pepper in the box and close it | 0.0 | 10.0 | 10.0 | 14.0 | 26.0 | 52.0 |
| put the bottled water on the tray | 4.0 | 36.0 | 30.0 | 42.0 | 66.0 | 84.0 |
| put the bowl in the microwave and close it | 0.0 | 22.0 | 10.0 | 34.0 | 12.0 | 40.0 |
| put the broccoli in the pan | 4.0 | 56.0 | 52.0 | 64.0 | 62.0 | 72.0 |
| put the candle on the tray | 6.0 | 48.0 | 30.0 | 52.0 | 48.0 | 62.0 |
| put the carrot in the bowl | 0.0 | 30.0 | 16.0 | 44.0 | 50.0 | 68.0 |
| put the cheese on the plate | 0.0 | 32.0 | 24.0 | 46.0 | 70.0 | 84.0 |
| put the corn on the cutting board | 6.0 | 56.0 | 30.0 | 72.0 | 76.0 | 88.0 |
Table 6. Success rates for all tasks in the MESA-ID task set.
| MESA-Spatial | DP | PG-FM | GR00T-N1.6 | FAST | ||
| open the slide cabinet and put the tomato in it | 2.0 | 34.0 | 30.0 | 58.0 | 54.0 | 74.0 |
| open the slide cabinet and put the water bottle in it | 0.0 | 12.0 | 6.0 | 16.0 | 28.0 | 46.0 |
| put the rolling pin in the right side of the dish drainer | 0.0 | 4.0 | 6.0 | 8.0 | 32.0 | 38.0 |
| put the rolling pin in the pan | 4.0 | 12.0 | 36.0 | 26.0 | 46.0 | 62.0 |
| put the rolling pin on the tray | 4.0 | 28.0 | 22.0 | 28.0 | 54.0 | 72.0 |
| put the tomato on the cutting board | 0.0 | 54.0 | 28.0 | 58.0 | 72.0 | 78.0 |
| put the tomato in the pan | 0.0 | 64.0 | 42.0 | 76.0 | 74.0 | 90.0 |
| put the tomato on the tray | 0.0 | 54.0 | 46.0 | 68.0 | 76.0 | 86.0 |
| put the water bottle in the basket | 0.0 | 40.0 | 6.0 | 44.0 | 58.0 | 76.0 |
| put the water bottle on the tray | 0.0 | 48.0 | 10.0 | 40.0 | 42.0 | 64.0 |
Table 7. Success rates for all tasks in the MESA-Spatial task set.
| MESA-Instance | DP | PG-FM | GR00T-N1.6 | FAST | ||
| put the apple on the tray | 10.0 | 38.0 | 46.0 | 48.0 | 56.0 | 64.0 |
| put the beer in the slide cabinet and close it | 0.0 | 6.0 | 4.0 | 10.0 | 6.0 | 14.0 |
| put the bell pepper in the box and close it | 0.0 | 10.0 | 4.0 | 10.0 | 14.0 | 30.0 |
| put the cheese on the plate | 0.0 | 4.0 | 2.0 | 2.0 | 12.0 | 28.0 |
| put the fish in the pan | 4.0 | 32.0 | 54.0 | 60.0 | 76.0 | 80.0 |
| put the jam in the basket | 0.0 | 50.0 | 24.0 | 52.0 | 44.0 | 78.0 |
| put the mushroom in the bowl | 0.0 | 28.0 | 24.0 | 46.0 | 46.0 | 72.0 |
| open the top drawer of the cabinet and put the carrot in it | 2.0 | 16.0 | 16.0 | 28.0 | 30.0 | 64.0 |
| open the top drawer of the cabinet and put the peach in it | 0.0 | 8.0 | 2.0 | 22.0 | 30.0 | 60.0 |
| open the microwave and put the squash in it | 0.0 | 2.0 | 2.0 | 8.0 | 26.0 | 30.0 |
| open the slide cabinet and put the wine in it | 2.0 | 2.0 | 8.0 | 10.0 | 12.0 | 40.0 |
| put the orange on the plate | 2.0 | 22.0 | 32.0 | 40.0 | 44.0 | 56.0 |
| put the peach in the basket | 0.0 | 38.0 | 36.0 | 74.0 | 58.0 | 82.0 |
| put the peach on the cutting board | 0.0 | 40.0 | 48.0 | 40.0 | 50.0 | 64.0 |
| put the peach on the tray | 0.0 | 54.0 | 50.0 | 74.0 | 78.0 | 78.0 |
| put the potato in the microwave and close it | 0.0 | 2.0 | 2.0 | 18.0 | 12.0 | 30.0 |
| put the squash in the pan | 2.0 | 12.0 | 30.0 | 20.0 | 44.0 | 56.0 |
| put the squash on the tray | 4.0 | 24.0 | 26.0 | 36.0 | 36.0 | 52.0 |
| put the wine in the box | 2.0 | 22.0 | 18.0 | 34.0 | 40.0 | 50.0 |
| put the wine on the tray | 10.0 | 44.0 | 26.0 | 48.0 | 68.0 | 80.0 |
Table 8. Success rates for all tasks in the MESA-Instance task set.
| MESA-Composite | DP | PG-FM | GR00T-N1.6 | FAST | ||
| put the apple on the plate | 0.0 | 66.0 | 54.0 | 70.0 | 64.0 | 84.0 |
| put the broccoli and mushroom on the cutting board | 0.0 | 0.0 | 0.0 | 0.0 | 2.0 | 4.0 |
| put the carrot in the pan | 2.0 | 30.0 | 42.0 | 58.0 | 60.0 | 74.0 |
| put the cheese in the basket | 0.0 | 28.0 | 6.0 | 36.0 | 62.0 | 62.0 |
| put the cheese and mug on the tray | 0.0 | 0.0 | 0.0 | 2.0 | 0.0 | 2.0 |
| put the cup in the left side of the dish drainer | 4.0 | 22.0 | 34.0 | 24.0 | 28.0 | 34.0 |
| put the fish and garlic in the pan | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| put the lemon on the cutting board | 0.0 | 34.0 | 40.0 | 54.0 | 44.0 | 76.0 |
| put the lemon and lime in the basket | 0.0 | 0.0 | 0.0 | 2.0 | 6.0 | 12.0 |
| put the mushroom on the tray | 0.0 | 52.0 | 48.0 | 52.0 | 56.0 | 74.0 |
| put the onion in the bowl | 0.0 | 40.0 | 20.0 | 70.0 | 62.0 | 80.0 |
| open the middle drawer of the cabinet and put the apple in it | 0.0 | 14.0 | 12.0 | 38.0 | 24.0 | 50.0 |
| open the box and put the apple in it and close it | 0.0 | 0.0 | 0.0 | 4.0 | 2.0 | 44.0 |
| open the box and put the beer in it | 0.0 | 6.0 | 10.0 | 10.0 | 14.0 | 38.0 |
| MESA-Composite | DP | PG-FM | GR00T-N1.6 | FAST | ||
| open the middle drawer of the cabinet and put the broccoli in it | 0.0 | 14.0 | 6.0 | 30.0 | 34.0 | 66.0 |
| open the top drawer of the cabinet and put the carrot in it and close it | 0.0 | 0.0 | 4.0 | 10.0 | 14.0 | 46.0 |
| open the slide cabinet and put the jam in it | 0.0 | 20.0 | 16.0 | 52.0 | 48.0 | 74.0 |
| open the middle drawer of the cabinet and put the mushroom in it and close it | 0.0 | 2.0 | 0.0 | 6.0 | 10.0 | 40.0 |
| open the microwave and put the potato in it and close it | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 2.0 |
| put the sponge in the right side of the dish drainer | 0.0 | 26.0 | 8.0 | 32.0 | 48.0 | 72.0 |
Table 9. Success rates for all tasks in the MESA-Composite task set.
| MESA-Category | DP | PG-FM | GR00T-N1.6 | FAST | ||
| put the beet in the pot | 0.0 | 0.0 | 12.0 | 8.0 | 10.0 | 38.0 |
| put the brussel sprout in the pan | 0.0 | 2.0 | 6.0 | 8.0 | 4.0 | 8.0 |
| put the chili pepper in the bowl | 0.0 | 20.0 | 38.0 | 22.0 | 4.0 | 26.0 |
| put the cupcake in the basket | 2.0 | 14.0 | 28.0 | 14.0 | 8.0 | 26.0 |
| put the ginger on the cutting board | 0.0 | 10.0 | 12.0 | 0.0 | 26.0 | 20.0 |
| put the grapes in the basket | 2.0 | 12.0 | 16.0 | 24.0 | 8.0 | 28.0 |
| put the ice cream in the bowl | 0.0 | 4.0 | 2.0 | 34.0 | 4.0 | 32.0 |
| put the kebabs in the pan | 0.0 | 20.0 | 18.0 | 12.0 | 16.0 | 44.0 |
| put the olive oil in the basket | 0.0 | 4.0 | 2.0 | 6.0 | 6.0 | 46.0 |
| put the pear in the pot | 4.0 | 0.0 | 20.0 | 12.0 | 2.0 | 16.0 |
| put the pomegranate in the top drawer of the cabinet | 0.0 | 8.0 | 6.0 | 2.0 | 2.0 | 4.0 |
| put the potato on the baking sheet | 2.0 | 4.0 | 16.0 | 34.0 | 8.0 | 42.0 |
| put the radish on the cutting board | 2.0 | 4.0 | 8.0 | 8.0 | 8.0 | 32.0 |
| put the raspberry in the basket | 0.0 | 14.0 | 10.0 | 8.0 | 6.0 | 18.0 |
| put the salt shaker in the box and close it | 0.0 | 2.0 | 2.0 | 4.0 | 4.0 | 34.0 |
| put the sausage on the baking sheet | 2.0 | 8.0 | 20.0 | 6.0 | 0.0 | 14.0 |
| put the scone on the plate | 0.0 | 0.0 | 12.0 | 2.0 | 14.0 | 12.0 |
| put the strawberry in the bowl | 0.0 | 28.0 | 48.0 | 20.0 | 34.0 | 74.0 |
| put the sushi on the plate | 0.0 | 0.0 | 2.0 | 8.0 | 2.0 | 10.0 |
| put the thermos in the slide cabinet | 0.0 | 0.0 | 4.0 | 0.0 | 2.0 | 4.0 |
Table 10. Success rates for all tasks in the MESA-Category task set.
| MESA-ID | DP | PG-FM | GR00T-N1.6 | FAST | ||
| put the croissant in the slide cabinet and close it | 0.0 | 16.0 | 10.0 | 24.0 | 28.0 | 36.0 |
| put the cucumber in the bottom drawer of the cabinet and close it | 0.0 | 8.0 | 12.0 | 16.0 | 24.0 | 48.0 |
| put the cup in the right side of the dish drainer | 2.0 | 32.0 | 20.0 | 42.0 | 48.0 | 62.0 |
| put the egg in the bowl | 0.0 | 24.0 | 24.0 | 40.0 | 42.0 | 74.0 |
| put the eggplant on the cutting board | 0.0 | 38.0 | 28.0 | 44.0 | 68.0 | 78.0 |
| put the fish in the pan | 2.0 | 40.0 | 24.0 | 46.0 | 56.0 | 72.0 |
| put the fish on the plate | 0.0 | 40.0 | 14.0 | 48.0 | 52.0 | 78.0 |
| put the garlic in the middle drawer of the cabinet and close it | 0.0 | 56.0 | 54.0 | 52.0 | 50.0 | 70.0 |
| put the garlic in the pan | 2.0 | 68.0 | 32.0 | 76.0 | 58.0 | 78.0 |
| put the jam in the basket | 0.0 | 46.0 | 20.0 | 64.0 | 68.0 | 88.0 |
| put the jug in the basket | 0.0 | 10.0 | 20.0 | 10.0 | 32.0 | 64.0 |
| put the kiwi in the top drawer of the cabinet and close it | 0.0 | 28.0 | 28.0 | 42.0 | 34.0 | 60.0 |
| put the lemon on the plate | 0.0 | 36.0 | 32.0 | 48.0 | 48.0 | 74.0 |
| put the lime in the bowl | 0.0 | 70.0 | 60.0 | 80.0 | 84.0 | 96.0 |
| put the mango in the box and close it | 0.0 | 20.0 | 2.0 | 24.0 | 34.0 | 56.0 |
| put the mug in the left side of the dish drainer | 0.0 | 22.0 | 22.0 | 24.0 | 42.0 | 48.0 |
| put the mushroom in the bowl | 0.0 | 38.0 | 30.0 | 38.0 | 48.0 | 66.0 |
| put the onion on the tray | 0.0 | 56.0 | 52.0 | 64.0 | 66.0 | 70.0 |
| open the top drawer of the cabinet and put the avocado in it | 0.0 | 10.0 | 6.0 | 28.0 | 40.0 | 58.0 |
| open the slide cabinet and put the book in it | 6.0 | 58.0 | 32.0 | 68.0 | 64.0 | 70.0 |
| open the bottom drawer of the cabinet and put the broccoli in it | 0.0 | 0.0 | 4.0 | 16.0 | 0.0 | 22.0 |
| open the box and put the can in it | 2.0 | 22.0 | 20.0 | 32.0 | 42.0 | 62.0 |
| open the slide cabinet and put the candle in it | 2.0 | 24.0 | 14.0 | 12.0 | 20.0 | 32.0 |
| open the microwave and put the canned food in it | 0.0 | 24.0 | 20.0 | 18.0 | 24.0 | 52.0 |
| open the top drawer of the cabinet and put the carrot in it | 0.0 | 12.0 | 12.0 | 42.0 | 34.0 | 64.0 |
| open the box and put the cheese in it | 0.0 | 14.0 | 8.0 | 30.0 | 38.0 | 58.0 |
| open the microwave and put the eggplant in it | 0.0 | 14.0 | 4.0 | 28.0 | 18.0 | 50.0 |
| open the middle drawer of the cabinet and put the lemon in it | 0.0 | 12.0 | 6.0 | 40.0 | 46.0 | 62.0 |
| open the bottom drawer of the cabinet and put the mushroom in it | 0.0 | 2.0 | 2.0 | 10.0 | 2.0 | 20.0 |
| open the middle drawer of the cabinet and put the onion in it | 2.0 | 14.0 | 6.0 | 28.0 | 32.0 | 46.0 |
| open the top drawer of the cabinet and put the peach in it | 2.0 | 6.0 | 18.0 | 20.0 | 22.0 | 52.0 |
| open the microwave and put the squash in it | 0.0 | 6.0 | 4.0 | 10.0 | 28.0 | 64.0 |
Table 1.
| MESA-ID | DP | PG-FM | GR00T-N1.6 | FAST | ||
| open the slide cabinet and put the tomato in it | 4.0 | 44.0 | 60.0 | 80.0 | 68.0 | 90.0 |
| open the slide cabinet and put the water bottle in it | 4.0 | 16.0 | 54.0 | 60.0 | 60.0 | 72.0 |
| open the slide cabinet and put the wine in it | 0.0 | 18.0 | 22.0 | 34.0 | 32.0 | 46.0 |
| put the orange on the cutting board | 2.0 | 22.0 | 14.0 | 54.0 | 48.0 | 56.0 |
| put the orange on the plate | 2.0 | 40.0 | 36.0 | 64.0 | 70.0 | 74.0 |
| put the peach in the basket | 0.0 | 52.0 | 24.0 | 66.0 | 64.0 | 88.0 |
| put the peach on the cutting board | 2.0 | 52.0 | 32.0 | 50.0 | 52.0 | 58.0 |
| put the peach on the tray | 0.0 | 50.0 | 46.0 | 60.0 | 56.0 | 74.0 |
| put the plate in the left side of the dish drainer | 0.0 | 2.0 | 2.0 | 8.0 | 14.0 | 16.0 |
| put the potato in the microwave and close it | 0.0 | 8.0 | 8.0 | 24.0 | 22.0 | 38.0 |
| put the rolling pin in the right side of the dish drainer | 0.0 | 10.0 | 8.0 | 20.0 | 22.0 | 36.0 |
| put the rolling pin in the pan | 2.0 | 34.0 | 30.0 | 58.0 | 56.0 | 60.0 |
| put the rolling pin on the tray | 2.0 | 30.0 | 26.0 | 62.0 | 50.0 | 50.0 |
| put the sponge in the middle drawer of the cabinet and close it | 0.0 | 12.0 | 0.0 | 10.0 | 12.0 | 24.0 |
| put the squash in the pan | 0.0 | 26.0 | 28.0 | 46.0 | 66.0 | 64.0 |
| put the squash on the tray | 2.0 | 24.0 | 26.0 | 56.0 | 68.0 | 66.0 |
| put the tomato on the cutting board | 8.0 | 72.0 | 58.0 | 88.0 | 76.0 | 86.0 |
| put the tomato in the pan | 8.0 | 64.0 | 92.0 | 86.0 | 82.0 | 88.0 |
| put the tomato on the tray | 8.0 | 72.0 | 74.0 | 90.0 | 70.0 | 96.0 |
| put the water bottle in the basket | 4.0 | 34.0 | 20.0 | 52.0 | 66.0 | 78.0 |
| put the water bottle on the tray | 10.0 | 30.0 | 38.0 | 50.0 | 50.0 | 74.0 |
| put the wine in the box | 0.0 | 34.0 | 18.0 | 58.0 | 70.0 | 76.0 |
| put the wine on the tray | 2.0 | 30.0 | 28.0 | 44.0 | 66.0 | 68.0 |
| put the yogurt in the basket | 2.0 | 26.0 | 10.0 | 44.0 | 48.0 | 56.0 |
Table 21.
Per-Task Success Rates for BiMESA
Several BiMESA tasks share a language instruction and differ only in arm assignment, so each row is annotated with its configuration: which arm performs a single-arm task, the pick and place sides for cross-arm pick-and-place, the handoff direction (when held out), or the acting arm and fixture side for articulated tasks.
| BiMESA-ID | DP | PG-Bin | PG-FM | GR00T-N1.6 | FAST | GPT-6.1-Sol | ||
| put the apple in the bowl (left arm) | 16.0 | 76.0 | 76.0 | 86.0 | 78.0 | 94.0 | 98.0 | 80.0 |
| put the apple in the box and close it | 0.0 | 0.0 | 8.0 | 16.0 | 16.0 | 50.0 | 46.0 | 10.0 |
| put the apple on the tray (right arm) | 28.0 | 48.0 | 74.0 | 78.0 | 66.0 | 88.0 | 92.0 | 70.0 |
| put the beer on the plate (left arm) | 26.0 | 70.0 | 70.0 | 76.0 | 56.0 | 88.0 | 98.0 | 60.0 |
| put the beer in the slide cabinet and close it | 0.0 | 0.0 | 0.0 | 6.0 | 10.0 | 22.0 | 20.0 | 0.0 |
| put the bell pepper in the basket (right arm) | 4.0 | 16.0 | 62.0 | 46.0 | 62.0 | 76.0 | 80.0 | 40.0 |
| put the bell pepper on the tray (right arm) | 22.0 | 24.0 | 74.0 | 66.0 | 80.0 | 88.0 | 90.0 | 70.0 |
| put the bottled drink in the basket (left arm) | 12.0 | 52.0 | 36.0 | 54.0 | 54.0 | 64.0 | 64.0 | 40.0 |
| put the can in the pan (left arm) | 10.0 | 78.0 | 58.0 | 64.0 | 62.0 | 84.0 | 92.0 | 80.0 |
| put the candle on the tray (right arm) | 20.0 | 28.0 | 58.0 | 74.0 | 60.0 | 86.0 | 88.0 | 30.0 |
| put the coffee cup on the tray (right arm) | 10.0 | 22.0 | 52.0 | 78.0 | 56.0 | 80.0 | 80.0 | 50.0 |
| put the garlic in the pan (right arm) | 12.0 | 22.0 | 64.0 | 62.0 | 44.0 | 72.0 | 80.0 | 60.0 |
| put the alcohol in the basket (handoff) | 10.0 | 14.0 | 26.0 | 34.0 | 36.0 | 6.0 | 36.0 | 0.0 |
| put the beer on the plate (handoff) | 20.0 | 8.0 | 46.0 | 52.0 | 64.0 | 30.0 | 68.0 | 40.0 |
| put the bottled drink in the basket (handoff) | 18.0 | 28.0 | 42.0 | 52.0 | 50.0 | 28.0 | 62.0 | 0.0 |
| put the bottled water in the basket (handoff) | 6.0 | 16.0 | 32.0 | 18.0 | 12.0 | 10.0 | 32.0 | 10.0 |
| put the can in the pan (handoff) | 10.0 | 26.0 | 50.0 | 38.0 | 48.0 | 26.0 | 70.0 | 60.0 |
| put the candle on the tray (handoff) | 18.0 | 22.0 | 12.0 | 40.0 | 22.0 | 42.0 | 40.0 | 10.0 |
| put the coffee cup on the tray (handoff) | 14.0 | 12.0 | 22.0 | 16.0 | 34.0 | 38.0 | 68.0 | 50.0 |
| put the condiment on the plate (handoff) | 16.0 | 20.0 | 46.0 | 60.0 | 68.0 | 36.0 | 74.0 | 40.0 |
| put the jam in the right side of the dish drainer (handoff) | 4.0 | 2.0 | 22.0 | 6.0 | 30.0 | 14.0 | 60.0 | 10.0 |
| put the jug in the basket (handoff) | 14.0 | 10.0 | 18.0 | 30.0 | 10.0 | 12.0 | 32.0 | 20.0 |
| put the ketchup on the plate (handoff) | 28.0 | 16.0 | 46.0 | 64.0 | 62.0 | 30.0 | 86.0 | 20.0 |
| put the milk on the tray (handoff) | 6.0 | 20.0 | 22.0 | 24.0 | 26.0 | 10.0 | 48.0 | 40.0 |
| put the soap dispenser on the tray (handoff) | 16.0 | 24.0 | 36.0 | 38.0 | 30.0 | 20.0 | 56.0 | 30.0 |
| put the spray in the basket (handoff) | 4.0 | 16.0 | 18.0 | 22.0 | 34.0 | 12.0 | 52.0 | 20.0 |
| put the water bottle on the tray (handoff) | 8.0 | 18.0 | 60.0 | 26.0 | 60.0 | 20.0 | 74.0 | 40.0 |
| put the wine on the cutting board (handoff) | 4.0 | 16.0 | 64.0 | 44.0 | 56.0 | 20.0 | 68.0 | 30.0 |
| put the yogurt in the bowl (handoff) | 8.0 | 10.0 | 32.0 | 24.0 | 34.0 | 22.0 | 56.0 | 30.0 |
| put the jam in the basket (right arm) | 10.0 | 14.0 | 62.0 | 60.0 | 46.0 | 70.0 | 76.0 | 40.0 |
| put the kiwi on the cutting board (right arm) | 28.0 | 54.0 | 86.0 | 84.0 | 70.0 | 88.0 | 94.0 | 60.0 |
| put the kiwi on the plate (right arm) | 34.0 | 66.0 | 88.0 | 86.0 | 76.0 | 92.0 | 96.0 | 80.0 |
| put the lemon on the plate (right arm) | 16.0 | 30.0 | 88.0 | 84.0 | 70.0 | 92.0 | 96.0 | 90.0 |
| put the lemon in the box and close it | 0.0 | 6.0 | 6.0 | 8.0 | 20.0 | 66.0 | 70.0 | 0.0 |
| put the lime in the bowl (right arm) | 22.0 | 52.0 | 64.0 | 84.0 | 70.0 | 88.0 | 98.0 | 70.0 |
| put the lime on the tray (left arm) | 26.0 | 82.0 | 44.0 | 80.0 | 76.0 | 92.0 | 90.0 | 70.0 |
| put the mango in the basket (right arm) | 14.0 | 14.0 | 50.0 | 48.0 | 52.0 | 74.0 | 90.0 | 40.0 |
Table 11. Success rates for all tasks in the BiMESA-ID task set.
| BiMESA-ID | DP | PG-Bin | PG-FM | GR00T-N1.6 | FAST | GPT-6.1-Sol | ||
| put the mango on the plate (left arm) | 26.0 | 62.0 | 68.0 | 52.0 | 58.0 | 86.0 | 90.0 | 70.0 |
| put the mango in the slide cabinet and close it | 0.0 | 8.0 | 16.0 | 24.0 | 48.0 | 34.0 | 52.0 | 0.0 |
| put the mushroom in the bowl (right arm) | 2.0 | 22.0 | 42.0 | 46.0 | 52.0 | 62.0 | 68.0 | 30.0 |
| put the onion on the cutting board (right arm) | 28.0 | 38.0 | 92.0 | 70.0 | 58.0 | 86.0 | 92.0 | 60.0 |
| put the onion in the pan (left arm) | 26.0 | 66.0 | 82.0 | 66.0 | 56.0 | 86.0 | 94.0 | 60.0 |
| put the onion on the tray (right arm) | 26.0 | 32.0 | 74.0 | 76.0 | 58.0 | 76.0 | 84.0 | 70.0 |
| open the box and put the apple in it | 0.0 | 2.0 | 4.0 | 4.0 | 16.0 | 32.0 | 54.0 | 10.0 |
| open the slide cabinet and put the beer in it | 0.0 | 2.0 | 10.0 | 20.0 | 26.0 | 38.0 | 38.0 | 50.0 |
| open the slide cabinet and put the book in it | 0.0 | 0.0 | 0.0 | 28.0 | 24.0 | 20.0 | 36.0 | 10.0 |
| open the slide cabinet and put the can in it | 0.0 | 10.0 | 16.0 | 34.0 | 32.0 | 54.0 | 64.0 | 50.0 |
| open the box and put the can in it | 0.0 | 4.0 | 8.0 | 18.0 | 20.0 | 34.0 | 48.0 | 10.0 |
| open the box and put the can in it and close it | 0.0 | 0.0 | 0.0 | 0.0 | 2.0 | 6.0 | 10.0 | 0.0 |
| open the slide cabinet and put the candle in it | 0.0 | 0.0 | 0.0 | 22.0 | 26.0 | 54.0 | 58.0 | 10.0 |
| open the box and put the mango in it and close it | 0.0 | 0.0 | 2.0 | 0.0 | 6.0 | 16.0 | 26.0 | 0.0 |
| open the slide cabinet and put the orange in it and close it | 0.0 | 0.0 | 0.0 | 18.0 | 16.0 | 30.0 | 42.0 | 20.0 |
| put the orange on the cutting board (right arm) | 34.0 | 64.0 | 82.0 | 76.0 | 68.0 | 88.0 | 100.0 | 80.0 |
| put the orange on the plate (right arm) | 32.0 | 52.0 | 82.0 | 80.0 | 72.0 | 86.0 | 94.0 | 90.0 |
| put the tangerine on the cutting board (left arm) | 18.0 | 72.0 | 72.0 | 90.0 | 78.0 | 90.0 | 90.0 | 100.0 |
| put the tangerine on the tray (right arm) | 28.0 | 52.0 | 78.0 | 76.0 | 72.0 | 86.0 | 88.0 | 90.0 |
| put the yogurt in the basket (right arm) | 8.0 | 6.0 | 56.0 | 52.0 | 28.0 | 66.0 | 66.0 | 50.0 |
Table 22.
| BiMESA-Spatial | DP | PG-Bin | PG-FM | GR00T-N1.6 | FAST | GPT-6.1-Sol | ||
| put the apple in the bowl (right arm) | 24.0 | 56.0 | 76.0 | 84.0 | 76.0 | 88.0 | 92.0 | 50.0 |
| put the apple on the tray (left arm) | 24.0 | 78.0 | 72.0 | 82.0 | 58.0 | 86.0 | 80.0 | 70.0 |
| put the beer on the plate (right arm) | 22.0 | 36.0 | 64.0 | 72.0 | 42.0 | 80.0 | 88.0 | 50.0 |
| put the beer in the slide cabinet and close it (left arm, right fixture) | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 4.0 | 6.0 | 0.0 |
| put the bell pepper in the basket (pick L, place R) | 20.0 | 32.0 | 52.0 | 44.0 | 56.0 | 60.0 | 80.0 | 10.0 |
| put the bell pepper on the tray (left arm) | 36.0 | 58.0 | 76.0 | 54.0 | 60.0 | 80.0 | 70.0 | 70.0 |
| put the bottled drink in the basket (right arm) | 2.0 | 12.0 | 42.0 | 38.0 | 32.0 | 60.0 | 58.0 | 0.0 |
| put the can in the pan (right arm) | 14.0 | 46.0 | 68.0 | 68.0 | 60.0 | 86.0 | 86.0 | 70.0 |
| put the candle on the tray (pick R, place L) | 14.0 | 22.0 | 20.0 | 38.0 | 28.0 | 32.0 | 36.0 | 30.0 |
Table 12. Success rates for all tasks in the BiMESA-Spatial task set.
| BiMESA-Spatial | DP | PG-Bin | PG-FM | GR00T-N1.6 | FAST | GPT-6.1-Sol | ||
| put the coffee cup on the tray (pick R, place L) | 10.0 | 16.0 | 30.0 | 22.0 | 26.0 | 28.0 | 48.0 | 20.0 |
| put the garlic in the pan (pick L, place R) | 10.0 | 24.0 | 48.0 | 34.0 | 48.0 | 58.0 | 90.0 | 70.0 |
| put the beer on the plate (handoff R→L) | 50.0 | 26.0 | 48.0 | 56.0 | 56.0 | 62.0 | 90.0 | 30.0 |
| put the can in the pan (handoff R→L) | 22.0 | 22.0 | 62.0 | 44.0 | 64.0 | 62.0 | 80.0 | 30.0 |
| put the condiment on the plate (handoff L→R) | 16.0 | 32.0 | 26.0 | 42.0 | 38.0 | 12.0 | 56.0 | 30.0 |
| put the jug in the basket (handoff L→R) | 2.0 | 2.0 | 12.0 | 22.0 | 24.0 | 14.0 | 30.0 | 10.0 |
| put the milk on the tray (handoff L→R) | 4.0 | 12.0 | 36.0 | 22.0 | 14.0 | 30.0 | 52.0 | 30.0 |
| put the water bottle on the tray (handoff R→L) | 8.0 | 16.0 | 24.0 | 26.0 | 22.0 | 24.0 | 26.0 | 50.0 |
| put the wine on the cutting board (handoff R→L) | 14.0 | 18.0 | 20.0 | 46.0 | 46.0 | 44.0 | 52.0 | 40.0 |
| put the yogurt in the bowl (handoff L→R) | 8.0 | 30.0 | 30.0 | 18.0 | 34.0 | 42.0 | 60.0 | 10.0 |
| put the jam in the basket (pick L, place R) | 14.0 | 34.0 | 30.0 | 36.0 | 54.0 | 44.0 | 72.0 | 40.0 |
| put the kiwi on the cutting board (pick L, place R) | 26.0 | 24.0 | 66.0 | 46.0 | 66.0 | 74.0 | 84.0 | 30.0 |
| put the kiwi on the plate (pick R, place L) | 26.0 | 52.0 | 70.0 | 44.0 | 68.0 | 68.0 | 88.0 | 40.0 |
| put the lemon on the plate (pick R, place L) | 14.0 | 28.0 | 74.0 | 38.0 | 72.0 | 80.0 | 94.0 | 90.0 |
| put the lime in the bowl (left arm) | 20.0 | 78.0 | 74.0 | 82.0 | 70.0 | 94.0 | 92.0 | 20.0 |
| put the lime on the tray (right arm) | 28.0 | 34.0 | 88.0 | 86.0 | 76.0 | 82.0 | 96.0 | 70.0 |
| put the mango in the basket (left arm) | 8.0 | 44.0 | 62.0 | 64.0 | 62.0 | 72.0 | 86.0 | 40.0 |
| put the mango on the plate (right arm) | 42.0 | 38.0 | 88.0 | 64.0 | 80.0 | 94.0 | 98.0 | 80.0 |
| put the mushroom in the bowl (pick L, place R) | 6.0 | 14.0 | 38.0 | 22.0 | 40.0 | 56.0 | 66.0 | 10.0 |
| put the onion on the cutting board (pick R, place L) | 32.0 | 32.0 | 82.0 | 36.0 | 58.0 | 78.0 | 94.0 | 40.0 |
| put the onion in the pan (right arm) | 22.0 | 34.0 | 80.0 | 58.0 | 62.0 | 78.0 | 96.0 | 80.0 |
| put the onion on the tray (pick L, place R) | 34.0 | 30.0 | 44.0 | 50.0 | 52.0 | 60.0 | 78.0 | 80.0 |
| open the slide cabinet and put the candle in it (right arm, left fixture) | 0.0 | 2.0 | 0.0 | 4.0 | 8.0 | 18.0 | 24.0 | 10.0 |
| open the slide cabinet and put the orange in it and close it (right arm, left fixture) | 0.0 | 0.0 | 0.0 | 2.0 | 2.0 | 6.0 | 0.0 | 20.0 |
| put the orange on the cutting board (left arm) | 30.0 | 60.0 | 76.0 | 76.0 | 78.0 | 88.0 | 84.0 | 90.0 |
| put the orange on the plate (pick L, place R) | 46.0 | 18.0 | 58.0 | 60.0 | 64.0 | 62.0 | 94.0 | 50.0 |
| put the tangerine on the cutting board (right arm) | 22.0 | 44.0 | 86.0 | 78.0 | 62.0 | 84.0 | 84.0 | 90.0 |
| put the tangerine on the tray (pick R, place L) | 38.0 | 26.0 | 66.0 | 44.0 | 64.0 | 76.0 | 88.0 | 30.0 |
| put the yogurt in the basket (pick R, place L) | 8.0 | 20.0 | 42.0 | 52.0 | 48.0 | 42.0 | 80.0 | 10.0 |
Table 27.
| BiMESA-Instance | DP | PG-Bin | PG-FM | GR00T-N1.6 | FAST | GPT-6.1-Sol | ||
| put the apple in the bowl | 12.0 | 56.0 | 54.0 | 62.0 | 42.0 | 64.0 | 72.0 | 40.0 |
| put the apple in the box and close it | 0.0 | 0.0 | 4.0 | 14.0 | 6.0 | 28.0 | 52.0 | 0.0 |
| put the apple on the tray | 18.0 | 30.0 | 20.0 | 62.0 | 22.0 | 54.0 | 42.0 | 80.0 |
| put the beer on the plate | 12.0 | 2.0 | 20.0 | 52.0 | 4.0 | 40.0 | 54.0 | 60.0 |
| put the bell pepper in the basket | 10.0 | 6.0 | 28.0 | 36.0 | 14.0 | 32.0 | 34.0 | 20.0 |
| put the bell pepper on the tray | 6.0 | 16.0 | 38.0 | 38.0 | 26.0 | 34.0 | 48.0 | 80.0 |
| put the beer on the plate (handoff) | 4.0 | 4.0 | 18.0 | 26.0 | 20.0 | 10.0 | 38.0 | 80.0 |
| put the can in the pan (handoff) | 18.0 | 8.0 | 26.0 | 24.0 | 38.0 | 20.0 | 60.0 | 40.0 |
| put the candle on the tray (handoff) | 10.0 | 10.0 | 14.0 | 14.0 | 6.0 | 24.0 | 20.0 | 30.0 |
| put the jam in the right side of the dish drainer (handoff) | 0.0 | 4.0 | 24.0 | 4.0 | 8.0 | 6.0 | 10.0 | 0.0 |
| put the water bottle on the tray (handoff) | 10.0 | 4.0 | 26.0 | 30.0 | 36.0 | 20.0 | 62.0 | 40.0 |
| put the jam in the basket | 4.0 | 18.0 | 32.0 | 62.0 | 18.0 | 46.0 | 62.0 | 60.0 |
| put the mushroom in the bowl | 6.0 | 16.0 | 58.0 | 62.0 | 40.0 | 52.0 | 78.0 | 60.0 |
| open the box and put the apple in it | 2.0 | 6.0 | 6.0 | 8.0 | 12.0 | 30.0 | 32.0 | 0.0 |
| open the slide cabinet and put the beer in it | 0.0 | 0.0 | 0.0 | 14.0 | 18.0 | 34.0 | 26.0 | 30.0 |
| open the slide cabinet and put the orange in it and close it | 0.0 | 0.0 | 4.0 | 10.0 | 10.0 | 20.0 | 18.0 | 20.0 |
| put the orange on the cutting board | 26.0 | 34.0 | 80.0 | 70.0 | 60.0 | 76.0 | 84.0 | 60.0 |
| put the orange on the plate | 32.0 | 36.0 | 62.0 | 72.0 | 50.0 | 70.0 | 74.0 | 90.0 |
Table 13. Success rates for all tasks in the BiMESA-Instance task set.
| BiMESA-Composite | DP | PG-Bin | PG-FM | GR00T-N1.6 | FAST | GPT-6.1-Sol | ||
| put the apple on the plate | 32.0 | 62.0 | 76.0 | 78.0 | 74.0 | 76.0 | 90.0 | 90.0 |
| put the broccoli and the mushroom on the cutting board | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 2.0 | 0.0 | 30.0 |
| put the carrot in the pan | 2.0 | 4.0 | 4.0 | 14.0 | 2.0 | 2.0 | 32.0 | 100.0 |
| put the fish and the garlic in the pan | 0.0 | 0.0 | 0.0 | 4.0 | 0.0 | 2.0 | 2.0 | 20.0 |
| put the lemon on the cutting board | 28.0 | 54.0 | 84.0 | 70.0 | 72.0 | 90.0 | 94.0 | 100.0 |
| put the lemon and the lime in the basket | 0.0 | 0.0 | 0.0 | 2.0 | 0.0 | 2.0 | 2.0 | 10.0 |
| put the mushroom on the tray | 6.0 | 38.0 | 40.0 | 64.0 | 32.0 | 72.0 | 82.0 | 40.0 |
| put the onion in the bowl | 18.0 | 58.0 | 60.0 | 74.0 | 52.0 | 74.0 | 92.0 | 40.0 |
| open the box and put the apple in it and close it | 0.0 | 0.0 | 0.0 | 4.0 | 0.0 | 16.0 | 28.0 | 10.0 |
| open the slide cabinet and put the beer in it and close it | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 2.0 | 8.0 | 20.0 |
| open the box and put the beer in it | 0.0 | 0.0 | 4.0 | 0.0 | 6.0 | 14.0 | 24.0 | 10.0 |
| open the slide cabinet and put the book in it and close it | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 10.0 |
| open the slide cabinet and put the jam in it | 0.0 | 0.0 | 0.0 | 16.0 | 32.0 | 48.0 | 70.0 | 20.0 |
| open the box and put the lemon in it and close it | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 24.0 | 14.0 | 10.0 |
| open the slide cabinet and put the mushroom in it and close it | 0.0 | 0.0 | 0.0 | 6.0 | 6.0 | 8.0 | 24.0 | 10.0 |
Table 14. Success rates for all tasks in the BiMESA-Composite task set.
| BiMESA-Category | DP | PG-Bin | PG-FM | GR00T-N1.6 | FAST | GPT-6.1-Sol | ||
| put the beet in the pot | 2.0 | 0.0 | 4.0 | 16.0 | 2.0 | 2.0 | 8.0 | 30.0 |
| put the brussel sprout in the pan | 20.0 | 14.0 | 18.0 | 42.0 | 6.0 | 24.0 | 14.0 | 100.0 |
| put the chili pepper in the bowl | 0.0 | 0.0 | 0.0 | 8.0 | 0.0 | 2.0 | 2.0 | 40.0 |
| put the cupcake in the basket | 2.0 | 4.0 | 26.0 | 32.0 | 10.0 | 20.0 | 26.0 | 10.0 |
| put the ginger on the cutting board | 6.0 | 20.0 | 14.0 | 24.0 | 8.0 | 28.0 | 10.0 | 90.0 |
| put the grapes in the basket | 2.0 | 0.0 | 6.0 | 24.0 | 0.0 | 16.0 | 12.0 | 20.0 |
| put the olive oil bottle on the tray (handoff) | 18.0 | 4.0 | 34.0 | 12.0 | 30.0 | 6.0 | 34.0 | 90.0 |
| put the pomegranate in the bowl (handoff) | 20.0 | 20.0 | 30.0 | 40.0 | 52.0 | 34.0 | 26.0 | 60.0 |
| put the thermos in the basket (handoff) | 4.0 | 14.0 | 18.0 | 30.0 | 20.0 | 6.0 | 12.0 | 70.0 |
| open the box and put the cupcake in it | 0.0 | 0.0 | 0.0 | 0.0 | 6.0 | 8.0 | 26.0 | 0.0 |
| open the box and put the grapes in it and close it | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 4.0 | 0.0 | 0.0 |
| open the slide cabinet and put the pear in it | 0.0 | 4.0 | 6.0 | 38.0 | 20.0 | 44.0 | 52.0 | 40.0 |
| open the slide cabinet and put the potato in it | 0.0 | 0.0 | 0.0 | 40.0 | 4.0 | 38.0 | 28.0 | 60.0 |
| put the pear in the pot | 12.0 | 14.0 | 18.0 | 40.0 | 6.0 | 38.0 | 50.0 | 90.0 |
| put the radish on the cutting board | 0.0 | 0.0 | 4.0 | 34.0 | 0.0 | 22.0 | 6.0 | 70.0 |
| put the sausage on the baking sheet | 2.0 | 0.0 | 0.0 | 4.0 | 0.0 | 0.0 | 0.0 | 50.0 |
| put the scone on the plate | 10.0 | 8.0 | 8.0 | 60.0 | 16.0 | 44.0 | 10.0 | 70.0 |
Table 15. Success rates for all tasks in the BiMESA-Category task set.
MESA Details
Background: BDDL
The BEHAVIOR Domain Definition Language (BDDL) [43, 44] allows users to specify task initial and goal states using logical predicates, and has been used for several high-profile benchmarks [17, 20]. More recently, Saxena et al. [55] adapts it to MimicGen, specifying the object-centric subtasks using demonstration predicates and enabling MimicGen-ready environment generation purely in the BDDL language.
Implemented Task Skeletons
We implement task skeletons as simple Python classes specifying task-relevant objects, initialization predicates, demonstration predicates, and goal predicates. Our pipeline currently includes the PickAndPlace skeleton, which encompasses all pick-and-place tasks; MultiStepArticulatedManipulation, which encompasses tasks involving articulated objects, such as opening or closing them and placing objects in them; and, for the bimanual setting, BimanualHandoff.
Simulation Platform
We build on MimicLabs [55], which integrates infrastructure from robosuite [56], MimicGen [27], LIBERO [17] and RoboCasa [19], and uses the MuJoCo simulator [57]. For our single-arm Franka setup, we use the Franka Panda embodiment [58] asset from robosuite with a Robotiq 2F-85 gripper whose asset was sourced from MuJoCo Menagerie [59]. The dataset is designed to be a digital cousin of the DROID [26] dataset. Our environment includes left and right shoulder cameras and a wrist camera, but we only use the left shoulder and wrist perspectives for our experiments. For our bimanual YAM setup, we use the YAM asset from MuJoCo Menagerie [59] and include two wrist cameras and an egocentric camera.
Algorithm 1 Distractor sampling with visual confusion guarantees.
Require: target distractor object counts , , task-relevant objects , , maps between objects and destinations ,
Ensure: list of distractor object keys
,
while or do
if then
Sample object
if then
Sample destination
return
Ensuring Visual Confusion via Distractor Sampling
This section expands on the distractor sampling procedure referenced in the main paper, which aims to maintain high observation-conditioned task entropy and reduce causal confusion.
Each scene contains a mixture of task-relevant objects and task-irrelevant distractor objects. For example, for the task “put the apple in the basket”, the task-relevant objects would be the apple and the basket, and the task-irrelevant objects might include objects such as a plate. The goal is that for each initial state corresponding to a task , there exists some other task , , such that the task-relevant objects for appear in . To achieve this, we define as the set of graspable objects (e.g., apples) and to be the set of destination objects (e.g., baskets). For each object , we define a set as the set of destination objects in which is placed during some task in MESA-70. Similarly, for each destination , we define to be the set of objects placed in . From here, we start by placing the task-relevant objects and then iteratively add more objects, sampling from the combined sets according to the placed items, until we have the desired number of distractor objects. This process is summarized for pick and place tasks in Alg. 1, and we apply a similar algorithm to choose valid articulated objects.
Example: PickAndPlace Skeleton
This section provides a sample PickAndPlace skeleton.
from dataclasses import dataclass, field
from typing import List, Optional, Tuple
from .objects import (
get_grasp_object_spec,
get_dest_object_spec,
)
from .task import BaseTask, BaseTaskConfig, register_task
from .utils import (
make_named_grasp_object_spec,
make_named_dest_object_spec,
NamedGraspObjectSpec,
NamedDestObjectSpec,
NamedArticulatedObjectSpec,
)
@dataclass(frozen=True, kw_only=True)
class PickAndPlaceTaskConfig(BaseTaskConfig):
"""Configuration for generating pick-and-place task suites."""
dest_object: str
pick_objects: str | List[str]
dest_region: str
goal_predicate: str
class_name: str = "PickAndPlaceTask"
@register_task("pick_and_place", config_cls=PickAndPlaceTaskConfig)
class PickAndPlaceTask(BaseTask):
def __init__(self, config: PickAndPlaceTaskConfig) -> None:
super().__init__(config=config)
if isinstance(config.pick_objects, str):
pick_objects = [config.pick_objects]
else:
pick_objects = config.pick_objects
pick_objects = [get_grasp_object_spec(object_name) for
object_name in pick_objects]
self._pick_objects: List[NamedGraspObjectSpec] = []
for i, spec in enumerate(pick_objects):
object_name = f"object_{i}"
self._pick_objects.append(make_named_grasp_object_spec(object_name,
spec))
dest_object = get_dest_object_spec(config.dest_object)
self._dest_object: NamedDestObjectSpec = \
make_named_dest_object_spec("dest_object", dest_object)
self._dest_region = config.dest_region
self._goal_predicate = config.goal_predicate
def build_task_objects(
self
) -> Tuple[List[NamedGraspObjectSpec], List[NamedDestObjectSpec],
List[NamedArticulatedObjectSpec]]:
return self._pick_objects, [self._dest_object], []
def build_goal_state(
self
) -> List[object]:
goal_state: List[object] = ["and"]
for pick_object in self._pick_objects:
if self._goal_predicate == "on":
goal_state.append(["on", pick_object.id,
self._dest_object.id])
else:
assert self._dest_region is not None
contain_region_name = \
f"{self._dest_object.id}_{self._dest_region}"
goal_state.append(["in", pick_object.id,
contain_region_name])
return goal_state
def build_demonstration_states(
self,
) -> List[list]:
steps: List[list] = []
for pick_object in self._pick_objects:
steps.append(["grasp", pick_object.id])
if self._goal_predicate == "on":
steps.append(["on", pick_object.id,
self._dest_object.id])
else:
assert self._dest_region is not None
contain_region_name = \
f"{self._dest_object.id}_{self._dest_region}"
steps.append(["in", pick_object.id, contain_region_name])
return steps
Example: BDDL File
Here we provide a sample BDDL file for the task “open the box and put the apple in it and close it”. Note that whereas prior work saves these in the .bddl file format, we found this to add unnecessary implementation complexity since they are ultimately parsed into Python dictionaries. To avoid this, we simply save them as JSON files as we see below. This BDDL file shows how we instantiate several distractor objects, including a bell pepper, tray, bottled water, pan, tomato and squash. In distractor_goal_specs we see how the possible distractor tasks are enumerated. In demonstration_states we see the predicates that are used when decomposing the task for MimicGen.
Notice in regions that the initialization regions all cover the same area of the table. Unlike prior works such as LIBERO [17], which constrains each object to a small region without inter-object overlap, we allow objects to spawn across the full table and filter out overlapping initial configurations. To do this, we start by labeling each object with a 2D bounding box of its projection onto the table, which is already done for RoboCasa [19] objects and we do manually for other objects. Next, we sort objects by the sizes of their bounding boxes and iterate through them, randomly placing an object until it doesn’t intersect with any already placed objects.
{
"problem_name": "mimiclabs_lab1_tabletop_manipulation",
"fixtures": {
"table": [
"table"
],
"sliding_top_box_631": [
"sliding_top_box_631_1"
]
10 },
11 "regions": {
12 "table_sliding_top_box_631_1_init_region": {
13 "target": "table",
14 "ranges": [
15 [
16 0.05,
17 -0.4,
18 0.4,
19 0.4
20 ]
21 ],
22 "extra": [],
23 "yaw_rotation": [
24 -1.5707963267948966,
25 1.5707963267948966
26 ],
27 "rgba": [
28 0.0,
29 0.0,
30 1.0,
31 0.0
32 ]
33 },
34 "table_object_1_init_region": {
35 "target": "table",
36 "ranges": [
37 [
38 0.05,
39 -0.4,
40 0.4,
41 0.4
42 ]
43 ],
44 "extra": [],
45 "yaw_rotation": [
46 -1.5707963267948966,
47 1.5707963267948966
48 ],
49 "rgba": [
50 0.0,
51 0.0,
52 1.0,
53 0.0
54 ]
55 },
56 "table_robocasa_bell_pepper_1_1_1_init_region": {
57 "target": "table",
58 "ranges": [
59 [
60 0.05,
61 -0.4,
62 0.4,
63 0.4
64 ]
65 ],
66 "extra": [],
67 "yaw_rotation": [
68 -1.5707963267948966,
69 1.5707963267948966
70 ],
71 "rgba": [
72 0.0,
73 0.0,
74 1.0,
75 0.0
]
},
"table_wooden_tray_1_2_init_region": {
"target": "table",
"ranges": [
[
0.05,
-0.4,
0.4,
0.4
]
],
"extra": [],
"yaw_rotation": [
-1.5707963267948966,
1.5707963267948966
],
"rgba": [
0.0,
0.0,
1.0,
0.0
]
},
"table_robocasa_bottled_water_12_1_3_init_region": {
"target": "table",
"ranges": [
[
0.05,
-0.4,
0.4,
0.4
]
],
"extra": [],
"yaw_rotation": [
-1.5707963267948966,
1.5707963267948966
],
"rgba": [
0.0,
0.0,
1.0,
0.0
]
},
"table_robocasa_pan_24_1_4_init_region": {
"target": "table",
"ranges": [
[
0.05,
-0.4,
0.4,
0.4
]
],
"extra": [],
"yaw_rotation": [
-1.5707963267948966,
1.5707963267948966
],
"rgba": [
0.0,
0.0,
1.0,
0.0
]
},
"table_robocasa_tomato_0_1_5_init_region": {
"target": "table",
"ranges": [
[
0.05,
-0.4,
0.4,
0.4
]
],
"extra": [],
"yaw_rotation": [
-1.5707963267948966,
1.5707963267948966
],
"rgba": [
0.0,
0.0,
1.0,
0.0
]
},
"table_robocasa_squash_1_1_6_init_region": {
"target": "table",
"ranges": [
[
0.05,
-0.4,
0.4,
0.4
]
],
"extra": [],
"yaw_rotation": [
-1.5707963267948966,
1.5707963267948966
],
"rgba": [
0.0,
0.0,
1.0,
0.0
]
},
"sliding_top_box_631_1_contain_region": {
"target": "sliding_top_box_631_1",
"ranges": [],
"extra": [],
"yaw_rotation": [
0,
0
],
"rgba": [
0,
0,
1,
0
]
}
},
"objects": {
"robocasa_apple_0": [
"object_1"
],
"robocasa_bell_pepper_1": [
"robocasa_bell_pepper_1_1_1"
],
"wooden_tray": [
"wooden_tray_1_2"
],
"robocasa_bottled_water_12": [
"robocasa_bottled_water_12_1_3"
],
"robocasa_pan_24": [
"robocasa_pan_24_1_4"
],
"robocasa_tomato_0": [
"robocasa_tomato_0_1_5"
],
"robocasa_squash_1": [
"robocasa_squash_1_1_6"
]
},
"textures": {},
"camera": {
"rightshoulder": {
"r": 1.2,
"theta": 1.0471975511965976,
"phi": 0.7853981633974483,
"r_radius": 0.0,
"theta_radius": 0.0,
"phi_radius": 0.0
},
"leftshoulder": {
"r": 1.2,
"theta": 1.0471975511965976,
"phi": -0.7853981633974483,
"r_radius": 0.0,
"theta_radius": 0.0,
"phi_radius": 0.0
}
},
"lighting": {},
"styles": {},
"scene_properties": {},
"initial_state": [
[
"on",
"sliding_top_box_631_1",
"table_sliding_top_box_631_1_init_region"
],
[
"on",
"object_1",
"table_object_1_init_region"
],
[
"on",
"robocasa_bell_pepper_1_1_1",
"table_robocasa_bell_pepper_1_1_1_init_region"
],
[
"on",
"wooden_tray_1_2",
"table_wooden_tray_1_2_init_region"
],
[
"on",
"robocasa_bottled_water_12_1_3",
"table_robocasa_bottled_water_12_1_3_init_region"
274 ],
275 [
276 "on",
277 "robocasa_pan_24_1_4",
278 "table_robocasa_pan_24_1_4_init_region"
279 ],
280 [
281 "on",
282 "robocasa_tomato_0_1_5",
283 "table_robocasa_tomato_0_1_5_init_region"
284 ],
285 [
286 "on",
287 "robocasa_squash_1_1_6",
288 "table_robocasa_squash_1_1_6_init_region"
289 ]
290 ],
291 "goal_state": [
292 "and",
293 [
294 "close",
295 "sliding_top_box_631_1"
296 ],
297 [
298 "in",
299 "object_1",
300 "sliding_top_box_631_1_contain_region"
301 ]
302 ],
303 "distractor_goal_specs": [
304 [
305 "on",
306 "object_1",
307 "wooden_tray_1_2"
308 ],
309 [
310 "on",
311 "object_1",
312 "robocasa_pan_24_1_4"
313 ],
314 [
315 "in",
316 "robocasa_bell_pepper_1_1_1",
317 "sliding_top_box_631_1_contain_region"
318 ],
319 [
320 "on",
321 "robocasa_bell_pepper_1_1_1",
322 "wooden_tray_1_2"
323 ],
324 [
325 "on",
326 "robocasa_bell_pepper_1_1_1",
327 "robocasa_pan_24_1_4"
328 ],
329 [
330 "in",
331 "robocasa_bottled_water_12_1_3",
332 "sliding_top_box_631_1_contain_region"
333 ],
334 [
335 "on",
336 "robocasa_bottled_water_12_1_3",
337 "wooden_tray_1_2"
338 ],
339 [
340 "on",
341 "robocasa_bottled_water_12_1_3",
342 "robocasa_pan_24_1_4"
343 ],
344 [
345 "in",
346 "robocasa_pan_24_1_4",
347 "sliding_top_box_631_1_contain_region"
348 ],
349 [
350 "on",
351 "robocasa_pan_24_1_4",
352 "wooden_tray_1_2"
353 ],
354 [
355 "on",
356 "robocasa_pan_24_1_4",
357 "robocasa_pan_24_1_4"
358 ],
359 [
360 "in",
361 "robocasa_tomato_0_1_5",
362 "sliding_top_box_631_1_contain_region"
363 ],
364 [
365 "on",
366 "robocasa_tomato_0_1_5",
367 "wooden_tray_1_2"
368 ],
369 [
370 "on",
371 "robocasa_tomato_0_1_5",
372 "robocasa_pan_24_1_4"
373 ],
374 [
375 "in",
376 "robocasa_squash_1_1_6",
377 "sliding_top_box_631_1_contain_region"
378 ],
379 [
380 "on",
381 "robocasa_squash_1_1_6",
382 "wooden_tray_1_2"
383 ],
384 [
385 "on",
386 "robocasa_squash_1_1_6",
387 "robocasa_pan_24_1_4"
388 ]
389 ],
390 "demonstration_states": [
391 [
392 "open",
393 "sliding_top_box_631_1"
394 ],
395 [
396 "grasp",
397 "object_1"
398 ],
399 [
400 "in",
401 "object_1",
402 "sliding_top_box_631_1_contain_region"
403 ],
404 [
405 "close",
"sliding_top_box_631_1" ] ], "language_instruction": [ "open", "the", "box", "and", "put", "the", "apple", "in", "it", "and", "close", "it" ] }
Example: Initial State Distribution
In Figure 11 we visualize 20 initialization states for the “open the box and put the apple in it and close it” task. The top left image corresponds to the BDDL file shown in App. § 11.§ 7. Notice that each one has a unique set of distractor objects, posing substantial vision-language grounding challenges.

Figure 11. We visualize 20 initialization states for the “open the box and put the apple in it and close it” task.
Data Generation Details
In this section, we describe MESA-Gen’s pipeline for scalable data generation with substantial variety in tasks, objects and initial configurations. We first provide background on MimicGen in App. § 12.§ 2, then describe our protocol for collecting human source demonstrations in App. § 12.§ 3, then describe our method for enabling data generation with unseen tasks via subtask stitching in App. § 12.§ 4, and finally describe our trajectory optimizations for handling large source-target object pose differences in App. § 12.§ 5.
Background: MimicGen
MimicGen [27] is a framework for generating large robot demonstration datasets from a small set of human-collected seed trajectories. The core idea is to decompose a demonstration trajectory into object-centric segments where and denote the object and end-effector poses, respectively. For each segment, MimicGen computes the relative transform , which decouples motion from the original global frame. Given a new target object pose , the corresponding adapted end-effector pose is reconstructed as . By stitching these transformed segments together in their original temporal order, MimicGen generates new demonstrations that are physically consistent in novel contexts.
Source Demonstration Collection Protocol
We collect human demonstrations for source tasks to seed MimicGen and subtask stitching. For the single-arm setting, we collect 1125 demonstrations across 80 tasks using a Meta Quest VR setup [60]. During data collection, control commands are executed at 20 Hz. For the Quest controller, translational and rotational inputs are normalized to along axes and passed directly to the controller without action clipping, using a gain of 1.5. Source tasks are selected to broadly cover the objects and predicates supported in the benchmark. We filter tasks by a data generation rate cutoff of 10% after collecting 5 demonstrations per task. The 2,152 teleoperated BiMESA demonstrations (cross-arm pick-and-place and handoffs; App. § 13.§ 2) are collected with the same Meta Quest hardware.
Subtask Stitching for Unseen Tasks
While MimicGen is built to enable scalable data generation, it still requires human data collection for each task. This requires linear human effort in the number of created tasks, which is cumbersome for researchers with limited resources. To alleviate this, we use subtask stitching which, instead of using only demonstrations from the target task, uses MESA’s predicate structure to enable data generation for unseen tasks by composing object-centric subtasks sourced from other tasks with compatible predicates. For example, for a target task pick up the apple and put it on the plate, MESA-Gen might combine the Grasp(apple) subtask from put the apple in the drawer with the On(banana, plate) subtask of put the banana on the plate.
Demonstration Lookup System: We construct a hierarchical index offline by parsing all demonstration HDF5 files and extracting subtask-level metadata. This index is organized primarily by predicate type (e.g., grasp, in, on, open, close, stack, turnon, turnoff), and further refined by using object-centric semantic attributes such as shape category, object category, insertion type, destination region, and object identity. The exact hierarchy depth and key structure depend on the predicate semantics; for example, grasp predicates are indexed by the grasped object’s shape and category, whereas in predicates additionally encode both the source object and destination container properties.
Each leaf node of the index contains a list of concrete subtask segments extracted from demonstrations, including the path to the source HDF5 file, the demonstration identifier, object references involved in the interaction, the subtask’s index within the original trajectory, and the termination signal defining the subtask boundary. This is sufficient to recover the full state-action segment corresponding to the subtask and to re-contextualize it for a new task.
At generation time, we query this index using a predicate-specific set of search keys from the current subtask specification. These keys are computed using object metadata functions that map object instances to semantic categories like shape class or insertion type. Retrieval is performed by a recursive tree walk that follows the exact key if present. If a key is not present, we take the union across all children at that depth, which broadens the search to semantically related subtasks.
From the retrieved candidate list, we select using either RandomStrategy or one of several nearest-neighbor strategies: NearestNeighborObjectStrategy (weighted object pose distance), NearestNeighborRobotDistanceStrategy (robot travel distance to the first transformed pose), or NearestNeighborEefObjectTranslationStrategy (relative end-effector-object translation). Nearest-neighbor selection computes distances for all candidates and samples uniformly among the top- closest (default ).
Stitching Procedure: Each subtask is first trimmed from its source demonstration using start and end indices derived from predicate-specific termination signals stored in the HDF5 logs. For non-final subtasks, boundaries are detected via the first valid rising edge of the corresponding terminal signal following the previous subtask; the final subtask extends to the end of the demonstration. To increase diversity, subtask end boundaries may be randomly offset within a task-specified range, except for the final subtask, which is fixed to prevent truncation.
When concatenating subtasks, the system does not resample trajectories globally but instead inserts short transition segments to ensure temporal continuity. Transitions optionally insert i) a linear interpolation segment that smoothly moves the end-effector from the final pose of the previous subtask to the initial pose of the next, and ii) a fix-pose hold segment that maintains the first pose of the next subtask for a fixed number of steps. To avoid duplicate commands, the first waypoint of each appended segment is removed prior to concatenation. After all subtasks are merged, their actions are concatenated into a single action sequence.
The environment is reset once at the beginning of the stitched trajectory, and all subsequent subtasks execute sequentially without intermediate resets. We transform subtask trajectories to the current scene by preserving the relative pose between the end-effector and the reference object. When objects differ in canonical rotation axes, an additional rotation correction is applied to align coordinate conventions.
Demonstrations with missing or invalid subtask termination signals are skipped during dataset parsing. We enforce non-empty subtasks, strictly increasing subtask indices, and valid ordering of subtasks. On generation, if no compatible subtasks are found even after using the broader semantic fallbacks mentioned previously, generation terminates.
Trajectory Optimizations for Large Pose Differences
As described in App. § 12.§ 2, MimicGen works by applying a rigid transformation based on the target object pose in the source and current task. This is effective when source and target object poses are relatively close to one another and the target object is quasistatically stable, but can lead to unpredictable behaviors when either of these assumptions is broken. In this subsection, we outline changes to the MimicGen pipeline leading to behaviors that quantitatively lead to higher data generation rates and qualitatively lead to more sensible behaviors.
Removing Roll and Pitch in MimicGen Transforms: MimicGen can produce unstable trajectories for objects without stable resting poses. We observe that most tabletop objects either have effectively fixed roll and pitch under our initialization distribution or admit roll- and pitch-invariant grasps, so we ignore roll and pitch when computing MimicGen transformations. Concretely, given an input rotation , we extract static XYZ Euler angles (roll, pitch, yaw) via mat2euler(R, axes="sxyz"), and reconstruct a pure-yaw rotation by setting roll and pitch to zero. We enable this when grasping or placing objects and ignore it for other predicates (for example opening or closing articulated objects). Additionally, we ignore yaw for on predicates, or when interacting with objects rotationally symmetric about the vertical axis. These include, for example, bottles, cans and apples.
Trajectory Start Optimization: Another issue with MimicGen arises when there is a large change between the source and target object poses, sometimes leading to unnatural behaviors. We show an example of this in Figure 12, where the source trajectory comes from a configuration where the soap was farther away from the robot, leading to a trajectory beginning with a large unnecessary interpolation and self-collision. In the best case this is unnatural and suboptimal; in the worst case it leads to dangerous behaviors and kinematic singularities. To address this, we optimize the trajectory by rotating it about the vertical axis going through the target object to minimize the distance between the current end-effector pose and the start of the trajectory, and then truncate any additional frames so that the starting frame is the closest to the end-effector.

Figure 12. Due to the large change in object pose, the original MimicGen trajectory (green) starts from a position near the robot base, leading to a large interpolation and self-collision. We optimize the trajectory (blue) by rotating it about the vertical axis through the object position and truncating unnecessary frames (red) to minimize the Euclidean distance from the end-effector to a point along the trajectory.
Let be a sequence of end-effector poses output by MimicGen, and let and be the poses of the target object and end-effector at the beginning of the rollout. For each pose , we solve for an angle defining a rotation about the vertical axis through that best aligns the rotated pose with the current end-effector position:
where denotes a rotation about the world vertical axis and extracts the translational component of an transform. We then select the starting index as the waypoint from the first half of the trajectory that can be made closest to the current end-effector position under such a rotation:
and execute the rotated, truncated trajectory . This ensures execution begins from the point along the optimally rotated trajectory that is closest to the current end-effector pose, avoiding large initial deviations.
While this operation leads to more natural trajectory starting points, it may also lead to incorrect grasps for objects where end-effector yaw is important, while producing natural motions without modification for objects where it is not. To alleviate this, we use an LLM to label objects based on whether they are geometrically symmetric under yaw rotation, and slerp end-effector rotation back to the original MimicGen output when they are not.
The next issue is that the current end-effector rotation is sometimes far from the rotation at the beginning of the MimicGen trajectory, even after optimizing the starting pose, causing unnecessary rapid rotations at the beginning of the subtask. To alleviate this, we compute an offset rotation aligning the beginning of the trajectory with the current end-effector rotation. We then compute a sequence of offsets slerping from to the identity and left multiply these by the end-effector orientations in our trajectory.
MESA-Bench Details
All MESA tasks are listed in App. § 10.8.
Benchmark Construction
We use the MESA-Gen pipeline (App. § 4) to generate hundreds of tasks for MESA-Bench. We define a handful of task skeletons such as PickAndPlace, MultiStepArticulatedManipulation, and (for the bimanual setting) BimanualHandoff, which requires picking with one arm and placing with the other. We use GPT-5.1 to label each object with container compatibility (e.g., it is reasonable to place an apple but not a bottle of wine in a bowl) and generate BDDL files for all compatible task instantiations.
We then collect human source demonstrations covering a wide range of subtasks, run MESA-Gen on the full task set, and iterate on subtasks where data generation succeeds at a low rate. After iterating, we collect 1125 human demonstrations across 80 single-arm tasks. We then keep all tasks with a data generation rate above 10%, yielding 100 demonstrations each for 806 single-arm tasks. A potential concern is that generated data is biased toward regions with high data generation success rates; we study this in App. § 10.§ 5.
The bimanual dataset, BiMESA-57, consists of a mixture of MimicGen and manually collected data. MESA-Gen can automatically synthesize spatial variants of a source demonstration whenever the manipulation is carried out by a single arm, but cross-arm motions—a pick with one arm placed by the other, or a hand-to-hand handoff—require coordinated bimanual control that the single-arm trajectory transform cannot reproduce. We therefore generate single-arm task variants with MimicGen and teleoperate the bimanual ones, then merge both sources into one dataset of 6,833 demonstrations across 57 tasks.
We organize the tasks into three groups. Spatial pick-and-place (27 tasks, 4,690 demos) instantiates each PickAndPlace task under four side configurations defined by which arm picks and on which side the destination lies: the two aligned configurations (one arm both picks and places) are self-contained single-arm motions and are generated with MimicGen, while the two cross configurations require coordinated two-arm motion and are teleoperated. For each task we hold out one configuration for evaluation and train on the remaining three. Handoff (17 tasks, 843 demos) covers BimanualHandoff tasks, which are inherently cross-arm and thus collected entirely by teleoperation (48–50 demos per task); we train a single handoff direction and reserve the reverse direction for evaluation. Single-arm articulated manipulation (13 tasks, 1,300 demos) covers MultiStepArticulatedManipulation tasks—opening, placing into, and closing sliding cabinets and boxes—which a single arm performs end to end and are generated with MimicGen (100 demos per task, balanced 50/50 across left- and right-arm execution). In total, 4,681 demonstrations are MimicGen-generated and 2,152 are teleoperated for the BiMESA set.
Single-Arm Suite Construction
Here we list details about all of our single-arm task suites.
MESA-70
This evaluation set is composed of the 70 tasks from the MESA-70 training dataset. Notably, it includes in-distribution variants of all the tasks from MESA-Spatial and MESA-Instance.
MESA-Spatial
By default, coordinates are sampled from Uniform(0.05, 0.4) and from Uniform(, 0.4), which we empirically found to be the subset of the table where the robot could reliably interact with objects. We partition the table into two regions, for the training set and MESA-70 evaluation set, and for the MESA-Spatial out-of-distribution evaluation set. For we instead sample coordinates from Uniform(, 0.0) and for we sample them from Uniform(0.0, 0.4).
MESA-Instance
For each task, we partition the set of object assets into disjoint in-distribution and out-of-distribution subsets, training exclusively on the in-distribution assets and evaluating on held-out instances at test time. As with MESA-Spatial, the in-distribution asset variants are included in MESA-70, enabling direct comparison under changes in object appearance and geometry.
For three objects, namely the peach, squash and wine bottle, we hold out some assets from the RoboCasa-objaverse object set. For the rest, we replace the original object instance with RoboCasa AI-generated assets, which are never seen during training.
MESA-Composite
These are unseen composite tasks composed of subtasks seen in the MESA-70 dataset.
We include 12 tasks with familiar structure. Some are easy, such as “put the apple on the plate”, which composes skills from seen tasks “put the apple on the tray” and “put the cheese on the plate”. Others have familiar structure but are designed adversarially. For example, “put the cup in the left side of the dish drainer” is similar to the seen task “put the cup in the right side of the dish drainer”, but to complete it correctly the policy must notice this small change in task description and adjust the placement accordingly.
We include 8 tasks with unfamiliar structure. Four of these are two-step pick and place tasks, such as “put the fish and garlic in the pan”, which are unfamiliar since the training data only includes one-step pick and place tasks such as “put the fish in the pan”. The other four are articulated manipulation tasks involving opening objects, placing things in them, and then closing them. These are unfamiliar since the training tasks only opened or closed objects, but never both.
MESA-Category
These tasks all involve unseen grasp objects sourced from the RoboCasa AI-generated assets. In addition, some include the pot and baking sheet placement locations, which are also unseen. For example, for the task “put the beet in the pot”, the policy has never seen a beet or pot during finetuning.
Bimanual Suite Construction
The bimanual setting uses a YAM-style setup. Tasks fall into three families: bimanual pick-and-place (objects spawn on either side of the table; the policy chooses which arm picks and which arm places), single-arm articulated manipulation (one arm opens/closes a cabinet or sliding-top box and the same arm picks and places), and bimanual handoffs (one arm picks the object and hands it to the other arm, which places it).
BiMESA-57
This evaluation set is composed of the 57 tasks from the BiMESA-57 training dataset, evaluated on held-out initial-state seeds drawn from each task’s evaluation split. It includes the in-distribution variants of all tasks in BiMESA-Spatial.
BiMESA-Spatial
All 38 BiMESA-Spatial tasks require the policy to use a different arm than was used during training. Because two arms are available, relocating the scene changes which arm must act and how the arms coordinate, rather than simply where a single end-effector must reach—a form of spatial generalization with no single-arm analogue.
Pick-and-place held-out side combination (27 tasks): each task is trained with three of four combinations of and evaluated on the held-out fourth combination.
Handoff direction reversal (8 tasks): the policy is trained on handoffs in one direction (e.g., pick-left then place-right) and evaluated in the opposite direction. Four tasks are evaluated on unseen left-to-right handoffs and four on unseen right-to-left handoffs.
Articulated fixture-side misalignment (3 tasks): training pairs the pick-side and fixture-side (e.g., pick with the left arm while operating a left-side cabinet); evaluation places the fixture on a different side, requiring cross-body coordination.
BiMESA-Instance
For 18 tasks (9 pick-and-place, 5 handoff, 4 articulated), the pick object or destination is replaced with an AI-generated variant (suffixed _ai) at evaluation time. The in-distribution variants of these objects appear in BiMESA-57, enabling direct comparison.
BiMESA-Composite
These 15 unseen tasks compose subtasks seen during training:
5 pick-and-place pairs where the individual pick and destination objects were seen, but never together.
3 two-step pick-and-place tasks that place two seen objects into a shared destination (e.g., “put the lemon and lime in the basket”).
4 unseen open-and-place pairs and 3 longer-horizon open-place-close sequences with unseen target objects, drawing on seen open/close skills.
BiMESA-Category
These 17 tasks involve pick objects (and sometimes destinations) drawn from categories never seen during training, sourced primarily from RoboCasa AI-generated assets. They span 10 pick-and-place, 3 handoff, and 4 articulated manipulation tasks.
Training and Auxiliary Datasets
Our main MESA-70 training set includes 7,000 demonstrations across the 70 tasks in MESA-70, which is about 1M frames. We also include an auxiliary dataset of 100 demonstrations per task for 736 tasks, and train on a subset of this with 10 demonstrations per task, which also has about 1M frames. The bimanual training set, BiMESA-57, contains 6,833 demonstrations across the 57 in-distribution bimanual tasks (roughly 120 per task on average).
Evaluation Details
To minimize bias caused by unlucky starting configurations, we generate a fixed set of starting configurations. We use these configurations for all of our experiments and include them in our open-source code release. Within each embodiment and task, rollout restores the same object set and starting configuration for each method, ensuring comparisons use identical starting conditions. When generating these starting configurations, we ensure that the target object is not occluded or out of frame and that the target task has not already been completed.
We evaluate each policy for 50 rollouts for each task, with each rollout consisting of a different set of distractor objects. Note that since the distractor objects and initial positions are sampled completely independently of the training configurations, every single initial observation in the benchmark is unique, and solving this benchmark through memorization alone is infeasible.
Statistical Uncertainty
With 50 rollouts per task, each finetuned method receives 3,500 / 500 / 1,000 / 1,000 / 1,000 rollouts on MESA-ID / Spatial / Instance / Composite / Category and 2,850 / 1,900 / 900 / 750 / 850 on the corresponding BiMESA suites; GPT-6.1-Sol receives 570 / 380 / 180 / 150 / 170 (10 per task). We report 95% Wilson intervals on the pooled success counts of each suite (Table 16 and Table 17) and on the successful and distractor-completing rollouts for the language following rate (Table 18 and Table 19). Suite averages use a normal interval from per-task binomial variances.
| Setting | DP | PG-FM | GR00T-N1.6 | FAST | ||
| MESA-ID | 1.6 [1.2, 2.1] | 30.7 [29.2, 32.2] | 25.0 [23.6, 26.5] | 43.1 [41.5, 44.7] | 46.5 [44.9, 48.2] | 62.8 [61.2, 64.4] |
| MESA-Spatial | 1.0 [0.4, 2.3] | 35.0 [30.9, 39.3] | 23.2 [19.7, 27.1] | 42.2 [37.9, 46.6] | 53.6 [49.2, 57.9] | 68.6 [64.4, 72.5] |
| MESA-Instance | 1.9 [1.2, 2.9] | 22.7 [20.2, 25.4] | 22.7 [20.2, 25.4] | 34.0 [31.1, 37.0] | 39.1 [36.1, 42.2] | 55.4 [52.3, 58.5] |
| MESA-Composite | 0.3 [0.1, 0.9] | 17.7 [15.5, 20.2] | 15.0 [12.9, 17.3] | 27.5 [24.8, 30.3] | 28.9 [26.2, 31.8] | 46.7 [43.6, 49.8] |
| MESA-Category | 0.7 [0.3, 1.4] | 7.7 [6.2, 9.5] | 14.1 [12.1, 16.4] | 11.6 [9.8, 13.7] | 8.4 [6.8, 10.3] | 26.4 [23.8, 29.2] |
| Average | 1.1 [0.8, 1.4] | 22.8 [21.7, 23.8] | 20.0 [19.0, 21.0] | 31.7 [30.5, 32.8] | 35.3 [34.1, 36.5] | 52.0 [50.7, 53.2] |
Table 16. Single-arm success rates (%) with 95% Wilson intervals for Table 2.
| Setting | DP | PG-Bin | PG-FM | GR00T-N1.6 | FAST | GPT-6.1-Sol | ||
| BiMESA-ID | 13.0 [11.8, 14.3] | 26.7 [25.1, 28.4] | 43.8 [42.0, 45.7] | 47.0 [45.2, 48.9] | 45.9 [44.1, 47.7] | 53.7 [51.8, 55.5] | 68.7 [66.9, 70.3] | 41.2 [37.3, 45.3] |
| BiMESA-Spatial | 18.8 [17.1, 20.7] | 30.3 [28.2, 32.4] | 50.8 [48.5, 53.0] | 46.2 [43.9, 48.4] | 49.2 [47.0, 51.5] | 58.1 [55.9, 60.3] | 70.5 [68.4, 72.5] | 41.8 [37.0, 46.9] |
| BiMESA-Instance | 9.4 [7.7, 11.5] | 13.9 [11.8, 16.3] | 28.6 [25.7, 31.6] | 36.7 [33.6, 39.9] | 23.9 [21.2, 26.8] | 36.7 [33.6, 39.9] | 48.1 [44.9, 51.4] | 43.9 [36.8, 51.2] |
| BiMESA-Composite | 5.7 [4.3, 7.6] | 14.4 [12.1, 17.1] | 17.9 [15.3, 20.8] | 21.1 [19.3, 25.2] | 18.4 [15.8, 21.3] | 28.8 [24.7, 32.7] | 37.5 [34.1, 40.6] | 34.7 [27.5, 42.6] |
| BiMESA-Category | 5.8 [4.4, 7.5] | 6.0 [4.6, 7.8] | 10.9 [9.0, 13.2] | 26.1 [23.3, 29.2] | 10.6 [8.7, 12.8] | 19.8 [17.2, 22.6] | 18.6 [16.1, 21.3] | 52.4 [44.9, 59.7] |
| Average | 10.6 [9.9, 11.2] | 18.3 [17.5, 19.0] | 30.4 [29.5, 31.3] | 35.6 [34.6, 36.6] | 29.6 [28.7, 30.5] | 39.4 [38.0, 40.4] | 48.7 [47.7, 49.7] | 42.8 [40.5, 45.1] |
Table 17. Bimanual success rates (%) with 95% Wilson intervals for Table 2.
| Setting | DP | PG-FM | GR00T-N1.6 | FAST | ||
| MESA-ID | 18.9 [14.8, 23.7] | 80.0 [77.8, 82.1] | 71.4 [68.8, 73.8] | 79.7 [77.8, 81.4] | 83.7 [82.0, 85.3] | 87.6 [86.3, 88.8] |
| MESA-Spatial | 11.4 [5.0, 24.0] | 77.4 [71.6, 82.4] | 53.7 [47.0, 60.2] | 73.0 [67.7, 77.8] | 78.6 [73.9, 82.6] | 84.5 [80.6, 87.7] |
| MESA-Instance | 23.5 [15.6, 33.8] | 66.6 [61.4, 71.4] | 60.5 [55.5, 65.3] | 65.5 [62.3, 70.5] | 69.0 [65.0, 72.6] | 76.9 [73.7, 79.9] |
| MESA-Composite | 6.4 [2.2, 17.2] | 65.1 [59.2, 70.5] | 56.2 [50.2, 62.0] | 65.5 [60.8, 69.9] | 66.3 [61.7, 70.6] | 81.5 [78.1, 84.5] |
| MESA-Category | 12.1 [6.0, 22.9] | 32.6 [27.0, 38.8] | 42.5 [37.3, 47.8] | 33.1 [28.4, 38.2] | 28.0 [23.2, 33.3] | 44.1 [40.1, 48.1] |
| Average | 14.4 [10.9, 18.0] | 64.3 [62.1, 66.6] | 56.9 [54.5, 59.2] | 63.6 [61.7, 65.5] | 65.1 [63.3, 66.9] | 74.9 [73.5, 76.3] |
Table 18. Single-arm language following rates (%) with 95% Wilson intervals for Table 3.
| Setting | DP | PG-Bin | PG-FM | GR00T-N1.6 | FAST | GPT-6.1-Sol | ||
| BiMESA-ID | 34.9 [32.1, 37.9] | 78.7 [76.0, 81.2] | 84.6 [82.7, 86.4] | 87.7 [86.0, 89.2] | 73.4 [71.3, 75.4] | 86.9 [85.3, 88.4] | 92.9 [91.7, 93.9] | 73.2 [68.1, 77.8] |
| BiMESA-Spatial | 45.3 [41.8, 48.7] | 75.7 [72.5, 78.6] | 86.2 [84.0, 88.1] | 77.5 [74.9, 79.8] | 72.6 [70.1, 75.0] | 83.0 [80.9, 84.9] | 89.1 [87.5, 90.6] | 69.7 [63.5, 75.3] |
| BiMESA-Instance | 25.5 [21.1, 30.5] | 53.0 [46.6, 59.2] | 64.4 [59.6, 69.0] | 73.3 [69.1, 77.2] | 48.0 [43.4, 52.6] | 64.8 [60.6, 68.9] | 74.0 [70.3, 77.4] | 65.8 [57.0, 73.7] |
| BiMESA-Composite | 14.5 [11.0, 19.0] | 65.1 [57.5, 71.9] | 66.7 [59.9, 72.8] | 71.2 [65.1, 76.7] | 42.3 [37.1, 47.8] | 66.7 [61.4, 71.6] | 80.1 [75.6, 83.9] | 74.3 [63.0, 83.1] |
| BiMESA-Category | 15.7 [12.1, 20.2] | 27.4 [21.5, 34.2] | 30.1 [25.3, 35.4] | 55.6 [50.7, 60.4] | 21.2 [17.6, 25.3] | 35.1 [30.9, 39.4] | 28.7 [25.1, 32.6] | 79.5 [71.1, 85.9] |
| Average | 27.2 [25.5, 28.9] | 60.0 [57.5, 62.4] | 66.4 [64.4, 68.4] | 73.1 [71.2, 74.9] | 51.5 [49.8, 53.2] | 67.3 [65.6, 69.0] | 73.0 [71.6, 74.3] | 72.5 [69.1, 75.9] |
Table 19. Bimanual language following rates (%) with 95% Wilson intervals for Table 3.
Since rollout uses the same initial state for every method, we compare finetuned methods with an exact McNemar test on paired rollouts, uncorrected for multiple comparisons, and call a difference significant at . On MESA-ID, all five adjacent pairs in the ranking differ significantly, and is significantly ahead of every other method on all five single-arm suites. On BiMESA-ID, 4 of 6 adjacent pairs differ significantly, and is significantly ahead on every BiMESA suite except Category, where GR00T-N1.6 is. OOD rankings are less certain: 5 of 20 single-arm and 6 of 24 bimanual adjacent pairs cannot be separated. Most of these are near-ties within 1.6 points; the other two are limited by suite size, as the MESA-Composite and Category intervals are about twice as wide as those of MESA-ID.
These intervals and tests capture only rollout variation for a single trained checkpoint per method, not variation across training seeds or task selection.
| ID | Spatial | Instance | Composite | Category | |
| DP | 1.6 [1.2, 2.1] | 1.0 [0.4, 2.3] | 1.9 [1.2, 2.9] | 0.3 [0.1, 0.9] | 0.7 [0.3, 1.4] |
| DP+Aux | 2.3 [1.8, 2.8] | 2.4 [1.4, 4.1] | 2.2 [1.5, 3.3] | 0.6 [0.3, 1.3] | 0.3 [0.1, 0.9] |
| McNemar | 0.03 | 0.07 | 0.76 | 0.51 | 0.34 |
| 43.1 [41.5, 44.7] | 42.2 [37.9, 46.6] | 34.0 [31.1, 37.0] | 27.5 [24.8, 30.3] | 11.6 [9.8, 13.7] | |
| +Aux | 43.4 [41.7, 45.0] | 54.4 [50.0, 58.7] | 33.2 [30.4, 36.2] | 28.8 [26.1, 31.7] | 8.7 [7.1, 10.6] |
| McNemar | 0.79 | <0.001 | 0.68 | 0.44 | 0.02 |
| 62.8 [61.2, 64.4] | 68.6 [64.4, 72.5] | 55.4 [52.3, 58.5] | 46.7 [43.6, 49.8] | 26.4 [23.8, 29.2] | |
| +Aux | 59.0 [57.4, 60.6] | 69.2 [65.0, 73.1] | 53.6 [50.5, 56.7] | 45.6 [42.5, 48.7] | 23.0 [20.5, 25.7] |
| McNemar | <0.001 | 0.86 | 0.34 | 0.55 | 0.04 |
Table 20. Auxiliary co-training success rates (%) with 95% Wilson intervals for Table 4. McNemar compares each model with its +Aux counterpart.
Training Details for Baselines
Here we share additional details about our baselines.
Shared Details
Unless noted otherwise, all VLA / VLM-based policies except for GR00T are trained for 50k gradient steps with a batch size of 128. For GR00T, we use a batch size of 1024 as recommended by the authors on single-arm MESA and a batch size of 128 on BiMESA. The PaliGemma baselines and models are trained using the JAX version of the official openpi repository, and GR00T is trained using the official Isaac-GR00T repository. We use the default optimization parameters from these repositories for all experiments. In the single-arm setting, all models are trained with a left shoulder camera image and a wrist camera image. The bimanual setup provides two wrist cameras and an egocentric camera. All models are evaluated using the final model at the end of training. All models use relative joint angle action space, which is a common choice in the pretraining data for our main VLA baselines. Unless otherwise noted, all models are trained with a chunk size matching the control frequency of 20 Hz and we replan after executing the first 5 actions. In general, we didn’t tune hyperparameters unless performance was surprisingly poor.
Policy inputs and privileged information. – During training and evaluation, policies are conditioned only on camera images, proprioception (joint positions and gripper widths), and the language instruction; demonstration actions provide the training targets. No baseline receives ground-truth object poses, object-identity metadata, task predicates, or symmetry-derived grasp yaw as policy inputs. Privileged simulator information is used by the demonstration-generation pipeline and by the evaluator to score task completion, not supplied to the policies.
Baseline-Specific Details
Multitask Diffusion Policy (DP) [47]: We use the DiT Block Policy from Dasari et al. [48] with 6 blocks, an embed dimension of 512 and feedforward dim of 3200, yielding a total of 80M parameters. This architecture uses a ResNet-18 [61] backbone with FiLM [62] to extract feature vectors from each image. These image features are concatenated with the projected state and language embedding before being fed to a transformer encoder. The outputs of the encoder are pooled and used together with the diffusion timestep as AdaLN [63] conditioning for the diffusion transformer decoder. We use the AdamW [64] optimizer with a learning rate of , weight decay of and cosine annealing learning rate decay. We train for 320,000 gradient steps.
PaliGemma-Binning (PG-Bin): This baseline finetunes a PaliGemma-3B [49] backbone to predict actions discretized using RT-2-style [1] binning. We use the implementation in the
openpirepository, originally intended for use as a baseline in Atreya et al. [51]. The baseline discretizes each action dimension into 256 bins. Since this baseline is extremely token-inefficient, we only predict chunks of 5 actions to save compute. The model autoregressively predicts these binned actions.
PG-Bin single-arm omission. – We omit PG-Bin results on the single-arm MESA suites due to a bug in our PG-Bin inference implementation that affected only the single-arm setting. The issue did not affect PG-Bin training, the BiMESA PG-Bin evaluation, or any other baseline. We therefore report PG-Bin only on BiMESA and exclude it from single-arm averages and comparisons.
PaliGemma-Flow Matching (PG-FM): This baseline finetunes a PaliGemma-3B backbone with a flow matching head; i.e., [3] without robot pretraining. It is implemented in the
openpirepository and is identical to the baseline, but using PaliGemma pretrained weights for the backbone and initializing the action head from scratch.GR00T-N1.6 [6]: This baseline attaches a flow matching action head to a Cosmos-2B VLM [65]. It is implemented in the official
Isaac-GR00Trepository and is trained using the default configurations. Notably, this includes leaving the backbone frozen. On single-arm MESA, we originally trained with a batch size of 128 and retrained with a batch size of 1024, which we found to lead to a modest improvement. BiMESA results use a batch size of 128.
[3], -FAST [4], [8]: attaches a flow matching head to a PaliGemma-3B backbone and pretrains on a large cross-embodiment robot dataset. -FAST replaces the flow matching head with discretized action chunk prediction. trains a -like architecture with knowledge insulation [46]. These were all trained using the default configuration from the
openpirepository. For we finetuned with and without knowledge insulation, found this not to make much difference, and report the version without.
Training Compute Usage
An important goal for our framework is to allow academic researchers to quickly iterate with minimal compute requirements. We find that MESA’s dataset was small enough to achieve saturation after a relatively quick training cycle. For the openpi repository models, the full 50k training steps took 48 hours on four L40S GPUs or 21 hours on two H100s. We also saw that achieved strong performance even after 5k gradient steps, and that it achieved strong performance with only 10% of our data, meaning it may be possible to shrink the budget even further. For GR00T, the version with a batch size 128 took about 12 hours on one H100 and the version with batch size 1024 took about 30 hours with four H100s.
References
- [1]A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023.arxiv.org/abs/2307.15818
- [2]M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024.arxiv.org/abs/2406.09246
- [3]K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Muthukumar, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky. π0: A vision-language-action flow model for general robot control, 2024. URL https://arxiv.org/abs/2410.24164.
- [4]K. Pertsch, K. Stachowicz, B. Ichter, D. Driess, S. Nair, Q. Vuong, O. Mees, C. Finn, and S. Levine. Fast: Efficient action tokenization for vision-language-action models, 2025. URL https://arxiv.org/abs/2501.09747.
- [5]G. R. Team, S. Abeyruwan, J. Ainslie, J.-B. Alayrac, M. G. Arenas, T. Armstrong, A. Balakrishna, R. Baruch, M. Bauza, M. Blokzijl, S. Bohez, K. Boumsalis, A. Brohan, T. Buschmann, A. Byravan, S. Cabi, K. Caluwaerts, F. Casarini, O. Chang, J. E. Chen, X. Chen, H.-T. L. Chiang, K. Choromanski, D. D’Ambrosio, S. Dasari, T. Davchev, C. Devin, N. D. Palo, T. Ding, A. Dostmohamed, D. Driess, Y. Du, D. Dwibedi, M. Elabd, C. Fantacci, C. Fong, E. Frey, C. Fu, M. Giustina, K. Gopalakrishnan, L. Graesser, L. Hasenclever, N. Heess, B. Hernaez, A. Herzog, R. A. Hofer, J. Humplik, A. Iscen, M. G. Jacob, D. Jain, R. Julian, D. Kalashnikov, M. E. Karagozler, S. Karp, C. Kew, J. Kirkland, S. Kirmani, Y. Kuang, T. Lampe, A. Laurens, I. Leal, A. X. Lee, T.-W. E. Lee, J. Liang, Y. Lin, S. Maddineni, A. Majumdar, A. H. Michaely, R. Moreno, M. Neunert, F. Nori, C. Parada, E. Parisotto, P. Pastor, A. Pooley, K. Rao, K. Reymann, D. Sadigh, S. Saliceti, P. Sanketi, P. Sermanet, D. Shah, M. Sharma, K. Shea, C. Shu, V. Sindhwani, S. Singh, R. Soricut, J. T. Springenberg, R. Sterneck, R. Surdulescu, J. Tan, J. Thompson, V. Vanhoucke, J. Varley, G. Vesom, G. Vezzani, O. Vinyals, A. Wahid, S. Welker, P. Wohlhart, F. Xia, T. Xiao, A. Xie, J. Xie, P. Xu, S. Xu, Y. Xu, Z. Xu, Y. Yang, R. Yao, Y. Yaroshenko, W. Yu, W. Yuan, J. Zhang, T. Zhang, A. Zhou, and Y. Zhou. Gemini robotics: Bringing ai into the physical world, 2025. URL https://arxiv.org/abs/2503.20020.
- [6]NVIDIA, J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. J. Fan, Y. Fang, D. Fox, F. Hu, S. Huang, J. Jang, Z. Jiang, J. Kautz, K. Kundalia, L. Lao, Z. Li, Z. Lin, K. Lin, G. Liu, E. Llontop, L. Magne, A. Mandlekar, A. Narayan, S. Nasiriany, S. Reed, Y. L. Tan, G. Wang, Z. Wang, J. Wang, Q. Wang, J. Xiang, Y. Xie, Y. Xu, Z. Xu, S. Ye, Z. Yu, A. Zhang, H. Zhang, Y. Zhao, R. Zheng, and Y. Zhu. Gr00t n1: An open foundation model for generalist humanoid robots, 2025. URL https://arxiv.org/abs/2503.14734.
- [7]J. Lee, J. Duan, H. Fang, Y. Deng, S. Liu, B. Li, B. Fang, J. Zhang, Y. R. Wang, S. Lee, W. Han, W. Pumacay, A. Wu, R. Hendrix, K. Farley, E. VanderBilt, A. Farhadi, D. Fox, and R. Krishna. Molmoact: Action reasoning models that can reason in space, 2025. URL https://arxiv.org/abs/2508.07917.
- [8]P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Muthukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky. π₀.₅: a vision-language-action model with open-world generalization, 2025. URL https://arxiv.org/abs/2504.16054.
- [9]G. R. Team, A. Abdolmaleki, S. Abyeruwan, J. Ainslie, J.-B. Alayrac, M. G. Arenas, A. Balakrishna, N. Batchelor, A. Bewley, J. Bingham, M. Bloesch, K. Bousmalis, P. Brakel, A. Brohan, T. Buschmann, A. Byravan, S. Cabi, K. Caluwaerts, F. Casarini, C. Chan, O. Chang, L. Chappellet-Volpini, J. E. Chen, X. Chen, H.-T. L. Chiang, K. Choromanski, A. Collister, D. B. D’Ambrosio, S. Dasari, T. Davchev, M. K. Dave, C. Devin, N. D. Palo, T. Ding, C. Doersch, A. Dostmohamed, Y. Du, D. Dwbibedi, S. T. Egambaram, M. Elabd, T. Erez, X. Fang, C. Fantacci, C. Fong, E. Frey, C. Fu, R. Gao, M. Giustina, K. Gopalakrishnan, L. Graesser, O. Groth, A. Gupta, R. Hafner, S. Hansen, L. Hasenclever, S. Haves, N. Heess, B. Hernaez, A. Hofer, J. Hsu, L. Huang, S. H. Huang, A. Iscen, M. G. Jacob, D. Jain, S. Jesmonth, A. Jindal, R. Julian, D. Kalashnikov, M. E. Karagozler, S. Karp, M. Kecman, J. C. Kew, D. Kim, F. Kim, J. Kim, T. Kipf, S. Kirmani, K. Konyushkova, L. Y. Ku, Y. Kuang, T. Lampe, A. Laurens, T. A. Le, I. Leal, A. X. Lee, T.-W. E. Lee, G. Lever, J. Liang, L.-H. Lin, F. Liu, S. Long, C. Lu, S. Maddineni, A. Majumdar, K.-K. Maninis, A. Marmon, S. Martinez, A. H. Michaely, N. Milonopoulos, J. Moore, R. Moreno, M. Neunert, F. Nori, J. Ortiz, K. Oslund, C. Parada, E. Parisotto, A. Paryag, A. Pooley, T. Power, A. Quaglino, H. Qureshi, R. V. Raju, H. Ran, D. Rao, K. Rao, I. Reid, D. Rendleman, K. Reymann, M. Rivas, F. Romano, Y. Rubanova, P. P. Sampedro, P. R. Sanketi, D. Shah, M. Sharma, K. Shea, M. Shridhar, C. Shu, V. Sindhwani, S. Singh, R. Soricut, R. Sterneck, I. Storz, R. Surdulescu, J. Tan, J. Thompson, S. Tunysuvunakool, J. Varley, G. Vesom, G. Vezzani, M. B. Villalonga, O. Vinyals, R. Wagner, A. Wahid, S. Welker, P. Wohlhart, C. Wu, M. Wulfmeier, F. Xia, T. Xiao, A. Xie, J. Xie, P. Xu, S. Xu, Y. Xu, Z. Xu, J. Yan, S. Yang, S. Yang, Y. Yang, H. H. Yu, W. Yu, W. Yuan, Y. Yuan, J. Zhang, T. Zhang, Z. Zhang, A. Zhou, G. Zhou, and Y. Zhou. Gemini robotics 1.5: Pushing the frontier of generalist robots with advanced embodied reasoning, thinking, and motion transfer, 2025. URL https://arxiv.org/abs/2510.03342.
- [10]D. Qu, H. Song, Q. Chen, Z. Chen, X. Gao, X. Ye, Q. Lv, M. Shi, G. Ren, C. Ruan, M. Yao, H. Yang, J. Bao, B. Zhao, and D. Wang. Eo-1: Interleaved vision-text-action pretraining for general robot control. arXiv preprint, 2025. URL https://arxiv.org/abs/2508.21112.
- [11]C. Cheang, S. Chen, Z. Cui, Y. Hu, L. Huang, T. Kong, H. Li, Y. Li, Y. Liu, X. Ma, H. Niu, W. Ou, W. Peng, Z. Ren, H. Shi, J. Tian, H. Wu, X. Xiao, Y. Xiao, J. Xu, and Y. Yang. Gr-3 technical report, 2025. URL https://arxiv.org/abs/2507.15493.
- [12]M. Shukor, D. Aubakirova, F. Capuano, P. Kooijmans, S. Palma, A. Zouitine, M. Aractingi, C. Pascal, M. Russi, A. Marafioti, S. Alibert, M. Cord, T. Wolf, and R. Cadene. SmolVLA: A vision-language-action model for affordable and efficient robotics, 2025. URL https://arxiv.org/abs/2506.01844.
- [13]P. Intelligence, B. Ai, A. Amin, R. Aniceto, A. Balakrishna, G. Balke, K. Black, G. Bokinsky, S. Cao, T. Charbonnier, V. Choudhary, F. Collins, K. Conley, G. Connors, J. Darpinian, K. Dhabalia, M. Dhaka, J. DiCarlo, D. Driess, M. Equi, A. Esmail, Y. Fang, C. Finn, C. Glossop, T. Godden, I. Goryachev, L. Groom, H. Habeeb, H. Hancock, K. Hausman, G. Hussein, V. Hwang, B. Ichter, C. Jacobsen, S. Jakubczak, R. Jen, T. Jones, G. Kammerer, B. Katz, L. Ke, M. Khadikov, C. Kuchi, M. Lamb, D. LeBlanc, B. LeCount, S. Levine, X. Li, A. Li-Bell, V. Lialin, Z. Liang, W. Lim, Y. Lu, E. Luo, V. Mano, N. Marwaha, A. Mongush, L. Murphy, S. Nair, T. Patterson, K. Pertsch, A. Z. Ren, G. Schelske, C. Sharma, B. Shi, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, W. Stoeckle, J. Tang, J. Tanner, J. Tankeste, M. Torne, K. Vedder, Q. Vuong, A. Walling, H. Wang, J. Wang, X. Wang, C. Whalen, S. Whitmore, B. Williams, C. Xu, S. Yoo, L. Yu, W. Zhang, Z. Zhang, and U. Zhilinsky. π₀.₇: a steerable generalist robotic foundation model with emergent capabilities, 2026. URL https://arxiv.org/abs/2604.15483.
- [14]S. Ye, Y. Ge, K. Zheng, S. Gao, S. Yu, G. Kurian, S. Indupuru, Y. L. Tan, C. Zhu, J. Xiang, A. Malik, K. Lee, W. Liang, N. Ranawaka, J. Gu, Y. Xu, G. Wang, F. Hu, A. Narayan, J. Bjorck, J. Wang, G. Kim, D. Niu, R. Zheng, Y. Xie, J. Wu, Q. Wang, R. Julian, D. Xu, Y. Du, Y. Chebotar, S. Reed, J. Kautz, Y. Zhu, L. J. Fan, and J. Jang. World action models are zero-shot policies, 2026. URL https://arxiv.org/abs/2602.15922.
- [15]J. Pai, L. Achenbach, V. Montesinos, B. Forrai, O. Mees, and E. Nava. mimic-video: Video-action models for generalizable robot control beyond vlas. arXiv preprint 2512.15692, 2025.arxiv.org/abs/2512.15692
- [16]H. Fang, J. Duan, D. Clay, S. Wang, S. Liu, W. Huang, X. Fan, W.-C. Tsai, S. Chen, Y. R. Wang, S. Xing, J. Cho, J. S. Park, A. Eftekhar, P. Sushko, K. Farley, A. Wadhwa, C. Harrison, W. Han, Y.-C. Lee, E. VanderBilt, R. Hendrix, S. Ellawela, L. Ngoo, J. Chai, Z. Ren, A. Farhadi, D. Fox, and R. Krishna. Molmoact2: Action reasoning models for real-world deployment, 2026. URL https://arxiv.org/abs/2605.02881.
- [17]B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36, 2024.arxiv.org/abs/2306.03310
- [18]M. J. Kim, Y. Gao, T.-Y. Lin, Y.-C. Lin, Y. Ge, G. Lam, P. Liang, S. Song, M.-Y. Liu, C. Finn, and J. Gu. Cosmo: Fine-tuning video models for visuomotor control and planning, 2026. URL https://arxiv.org/abs/2601.16163.
- [19]S. Nasiriany, A. Maddikuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots. arXiv preprint arXiv:2406.02523, 2024.arxiv.org/abs/2406.02523
- [20]C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martín-Martín, C. Wang, G. Levine, W. Ai, B. Martinez, H. Yin, M. Lingelbach, M. Hwang, A. Hiranaka, S. Garlanka, A. Aydin, S. Lee, J. Sun, M. Anvari, M. Sharma, D. Bansal, S. Hunter, K.-Y. Kim, A. Lou, C. R. Matthews, I. Villa-Renteria, J. H. Tang, C. Tang, F. Xia, Y. Li, S. Savarese, H. Gweon, C. K. Liu, J. Wu, and L. Fei-Fei. Behavior-1k: A human-centered, embodied ai benchmark with 1,000 everyday activities and realistic simulation, 2024. URL https://arxiv.org/abs/2403.09227.
- [21]X. Yang, R. Dagli, A. Zook, H. Hadfield, A. Goyal, S. Birchfield, F. Ramos, and J. Tremblay. Rolobal: A high-fidelity simulation benchmark for analysis of task generalist policies. In Robotics: Science and Systems (RSS), 2026. URL https://arxiv.org/abs/2604.09860.
- [22]A. Jain, M. Zhang, K. Arora, W. Chen, M. Torne, M. Z. Irshad, S. Zakharov, Y. Wang, S. Levine, C. Finn, W.-C. Ma, D. Shah, A. Gupta, and K. Pertsch. Polaris: Scalable real-to-sim evaluations for generalist robot policies, 2025. URL https://arxiv.org/abs/2512.16881.
- [23]Y. Jangir, Y. Zhang, P.-C. Lo, K. Yamazaki, C. Zhang, K.-H. Tu, T.-W. Ke, L. Ke, Y. Bisk, and K. Fragkiadaki. Robotarena ∞: Scalable robot benchmarking via real-to-sim translation, 2025. URL https://arxiv.org/abs/2510.23571.
- [24]Y. Kim, W. Pumacay, O. Rayyan, M. Argus, W. Han, E. VanderBilt, J. Salvador, A. Deshpande, R. Hendrix, S. Jauhri, S. Liu, N. M. M. Shafiullah, M. Guru, A. Guru, A. Eftekhar, K. Farley, D. Clay, J. Duan, P. Wolters, A. Herrasti, Y.-C. Lee, G. Chalvatzaki, Y. Cui, A. Farhadi, D. Fox, and R. Krishna. Molmospaces: A large-scale open ecosystem for robot navigation and manipulation, 2026.
- [25]M. Sedlacek, P. Yefanov, G. Ponimatkin, J. Bardhan, S. Pilc, M. Fourmy, E. Kazakos, C. G. M. Snoek, J. Sivic, and V. Petrik. Realm: A real-to-sim validated benchmark for generalization in robotic manipulation, 2025. URL https://arxiv.org/abs/2512.19562.
- [26]A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Sr irama, L. Y. Chen, K. Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945, 2024.arxiv.org/abs/2403.12945
- [27]A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. In 7th Annual Conference on Robot Learning, 2023.arxiv.org/abs/2310.17596
- [28]S. James, Z. Ma, D. Rovick Arrojo, and A. J. Davison. RLbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 2020.arxiv.org/abs/1909.12271
- [29]M. Grotz, M. Shridhar, T. Asfour, and D. Fox. Percat2: Benchmarking and learning for robotic bimanual manipulation tasks, 2024. URL https://arxiv.org/abs/2407.00278.
- [30]W. Pumacay, I. Singh, J. Duan, R. Krishna, J. Thomason, and D. Fox. The colosseum: A benchmark for evaluating generalization for robotic manipulation. 2024.
- [31]O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L), 7(3):7327–7334, 2022.arxiv.org/abs/2112.03227
- [32]T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094–1100. PMLR, 2020.arxiv.org/abs/1910.10897
- [33]X. Li, K. Hsu, J. Gu, K. Pertsch, O. Mees, H. R. Walke, C. Fu, I. Lunawat, I. Sieh, S. Kirmani, S. Levine, J. Wu, C. Finn, H. Su, Q. Vuong, and T. Xiao. Evaluating real-world robot manipulation policies in simulation. arXiv preprint arXiv:2405.05941, 2024.
- [34]S. Zhang, Z. Xu, P. Liu, X. Yu, Y. Li, Q. Gao, Z. Fei, Z. Yin, Z. Wu, Y.-G. Jiang, and X. Qiu. Vlabbench: A large-scale benchmark for language-conditioned robotics manipulation with long-horizon reasoning tasks, 2024. URL https://arxiv.org/abs/2412.18194.
- [35]A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint arXiv:2108.03298, 2021.
- [36]S. Nasiriany, S. Nasiriany, A. M. Maddukuri, and Y. Zhu. Robocasa365: A large-scale simulation framework for training and benchmarking generalist robots. In International Conference on Learning Representations (ICLR), 2026.
- [37]T. Chen, Z. Chen, B. Chen, Z. Cai, Y. Liu, Z. Li, Q. Liang, X. Lin, Y. Ge, Z. Gu, W. Deng, Y. Guo, T. Nian, X. Xie, Q. Chen, K. Su, T. Xu, G. Liu, M. Hu, H. ang Gao, K. Wang, Z. Liang, Y. Qin, X. Yang, P. Luo, and Y. Mu. Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robot bimanual robotic manipulation, 2025. URL https://arxiv.org/abs/2506.18088.
- [38]B. Zhang, J. Li, J. Shen, Y. Cai, Y. Zhang, Y. Chen, J. Dai, J. Ji, and Y. Yang. Vla-arena: An open-source framework for benchmarking vision-language-action models, 2025. URL https://arxiv.org/abs/2512.22539.
- [39]S. Fei, S. Wang, J. Shi, Z. Dai, J. Cai, P. Qian, L. Ji, X. He, S. Zhang, Z. Fei, J. Fu, J. Gong, and X. Qiu. Libero-plus: In-depth robustness analysis of vision-language-action models, 2025. URL https://arxiv.org/abs/2510.13626.
- [40]X. Zhou, Y. Xu, G. Tie, Y. Chen, G. Zhang, D. Chu, P. Zhou, and L. Sun. Libero-pro: Towards robust and fair evaluation of vision-language-action models beyond memorization, 2025. URL https://arxiv.org/abs/2510.03827.
- [41]Y. R. Wang, C. Ung, G. Tannert, J. Duan, J. Li, A. Le, R. Oswal, M. Grotz, W. Pumacay, Y. Deng, R. Krishna, D. Fox, and S. Srinivasa. RoboEval: Where robotic manipulation meets structured and scalable evaluation, 2025. URL https://arxiv.org/abs/2507.00435.
- [42]J. Gu, F. Xiang, X. Li, Z. Ling, X. Liu, T. Mu, Y. Tang, S. Tao, X. Wei, Y. Yao, X. Yuan, P. Xie, Z. Huang, R. Chen, and H. Su. Manipulation skills: A unified benchmark for generalizable manipulation skills, 2023. URL https://arxiv.org/abs/2302.04659.
- [43]D. McDermott, M. Ghallab, A. E. Howe, C. A. Knoblock, A. Ram, M. M. Veloso, D. S. Weld, and D. E. Wilkins. Pddl-the planning domain definition language. 1998. URL https://api.semanticscholar.org/CorpusID:59656859.
- [44]S. Srivastava, C. Li, M. Lingelbach, R. Martín-Martín, F. Xia, K. Vainio, Z. Lian, C. Gokmen, S. Buch, C. K. Liu, S. Savarese, H. Gweon, J. Wu, and L. Fei-Fei. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments, 2021. URL https://arxiv.org/abs/2108.03332.
- [45]P. de Haan, D. Jayaraman, and S. Levine. Causal confusion in imitation learning, 2019. URL https://arxiv.org/abs/1905.11979.
- [46]D. Driess, J. T. Springenberg, B. Ichter, L. Yu, A. Li-Bell, K. Pertsch, A. Z. Ren, H. Walke, Q. Vuong, L. X. Shi, and S. Levine. Knowledge insulating vision-language-action models: Train fast, run fast, generalize better, 2025. URL https://arxiv.org/abs/2505.23705.
- [47]C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.arxiv.org/abs/2303.04137
- [48]S. Dasari, O. Mees, S. Zhao, M. K. Srirama, and S. Levine. The ingredients for robotic diffusion transformers. arXiv preprint arXiv:2410.10088, 2024.arxiv.org/abs/2410.10088
- [49]L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, et al. Paligemma: A versatile 3b vlm for transfer, 2024. URL https://arxiv.org/abs/2407.07726.
- [50]A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.arxiv.org/abs/2212.06817
- [51]P. Atreya, K. Pertsch, T. Lee, M. J. Kim, A. Jain, A. Kuramshin, C. Eppner, C. Neary, E. Hu, F. Ramos, et al. Roboarena: Distributed real-world evaluation of generalist robot policies. In Proceedings of the Conference on Robot Learning (CoRL 2025), 2025.arxiv.org/abs/2506.18123
- [52]A. Mete, H. Xue, A. Wilcox, Y. Chen, and A. Garg. Quest: Self-supervised skill abstractions for learning continuous control, 2024. URL https://arxiv.org/abs/2407.15840.
- [53]A. Wilcox, M. Ghanem, M. Moghani, P. Barroso, B. Joffe, and A. Garg. Adapt3r: Adaptive 3d scene representation for domain transfer in imitation learning, 2025. URL https://arxiv.org/abs/2503.04877.
- [54]W. Zhang, K. Wang, Y. Ouyang, X. Huang, L. Li, K. Su, W. Jin, W. Chai, H. Liang, Z. Dou, Y. Chen, and T. Chen. An unexpected robot policy: Early evaluations of gpt-6 astra on robodojo and beyond. arXiv preprint arXiv:2609.24170, 2026.arxiv.org/abs/2609.24170
- [55]V. Saxena, M. Bronars, N. R. Arachchige, K. Wang, W. C. Shin, S. Nasiriany, A. Mandlekar, and D. Xu. What matters in learning from large-scale datasets for robot manipulation. In The Thirteenth International Conference on Learning Representations, 2025. URL https://arxiv.org/pdf/2506.13536.
- [56]Y. Zhu, J. Wong, A. Mandlekar, R. Martín-Martín, A. Joshi, S. Nasiriany, Y. Zhu, and K. Lin. robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009.12293, 2020.arxiv.org/abs/2009.12293
- [57]E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033. IEEE, 2012. doi:10.1109/IROS.2012.6386109.DOI
- [58]Franka Emika GmbH. Panda User Manual, 2023. Available at https://www.franka.de.
- [59]K. Zakka, Y. Tassa, and MuJoCo Menagerie Contributors. MuJoCo Menagerie: A collection of high-quality simulation models for MuJoCo, 2022. URL http://github.com/google-deepmind/mujoco_menagerie.
- [60]F. E. Jedrzej Orbik. Oculus reader: Robotic teleoperation interface, 2021. URL https://github.com/rail-berkeley/oculus_reader. Accessed: YYYY-MM-DD.
- [61]K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.arxiv.org/abs/1512.03385
- [62]E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.arxiv.org/abs/1709.07871
- [63]W. Peebles and S. Xie. Scalable diffusion models with transformers, 2023. URL https://arxiv.org/abs/2212.09748.
- [64]I. Loshchilov and F. Hutter. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/1711.05101.
- [65]NVIDIA, A. Azzolini, J. Bai, H. Brandon, J. Cao, P. Chattopadhyay, H. Chen, J. Chu, Y. Cui, J. Diamond, Y. Ding, L. Feng, F. Ferroni, R. Govindaraju, J. Gu, S. Gururani, I. E. Hanafi, Z. Hao, J. Huffman, J. Jin, B. Johnson, R. Khan, G. Kurian, E. Lantz, N. Lee, Z. Li, X. Li, M. Liao, T.-Y. Lin, Y.-C. Lin, M.-Y. Liu, X. Lu, A. Luo, A. Mathay, Y. Ni, L. Pavão, W. Ping, D. W. Romero, M. Smelyanskyi, S. Song, L. Tchapmi, A. Z. Wang, B. Wang, H. Wang, F. Wei, J. Xu, Y. Xu, D. Yang, X. Yang, Z. Yang, J. Zhang, X. Zeng, and Z. Zhang. Cosmos-reason1: From physical common sense to embodied reasoning, 2025. URL https://arxiv.org/abs/2503.15558.







