WorksComputer Science
Benchmarking Real-World Robotic Manipulation through Stochastic Challenges
Abstract
A challenging bottleneck in manipulation research is reproducibility and comparability of methods in real-world environments. Since manipulation is an interactive task, it cannot be evaluated on a static dataset of observations. Benchmarking in simulation is a common remedy, but cannot cover the whole complexity of real-world tests. This paper proposes a new paradigm of stochastic challenges. Like the well-known YCB objects, it is based on reproducible real-world scenes and therefore allows system-level innovation and comparability between different platforms. Due to their stochastic nature, our challenges however, mimic the rigor and surprise of hidden test sets. To this end, we show how different robots in different labs can be compared based on their success rate over different random task samples, if these samples come from the same distribution. We propose stochastic challenges as a general paradigm for robot benchmarking and implement this paradigm in a novel benchmark for mobile manipulation, on which we compare two robots tested in two different labs.
Project Website: Stochastic Challenges — Benchmark: KLX Benchmark
Introduction
Reliable comparability between robotic manipulation methods is a common bottleneck in current robotics research. Manipulation is an inherently interactive task: the same model can produce different outcomes on the same physical setup, and different model outputs can lead to the same outcome, so evaluation on static datasets cannot capture method quality. Existing alternatives each carry limitations. Centralized competitions such as RoboCup require physically transporting robots to a single site. Simulation benchmarks [1] provide scale and repeatability but exhibit physical realism gaps. Real-world benchmarks built around fixed object sets [2] have driven important progress, but a single hand-picked setup risks overfitting.
Recent efforts have identified the same problem and proposed real-world distributed benchmarks [3, 4]. However, both rely on standardizing either the robot embodiment [3] or the task [4], and both constrain themselves to tabletop scenarios. This prevents precisely the comparisons that matter as manipulation systems scale to deployment: comparing different platforms - humanoids, wheeled mobile manipulators, quadrupeds, on real tasks that require both mobility and interaction with real scenes.
We address this gap with stochastic challenges, a benchmarking framework that standardizes a distribution of scenes and tasks rather than a fixed test set. This extends the idea of YCB [2] from a fixed object set to a sampled distribution: any lab can reconstruct any specific scene from a seed, but participants see only a small fraction of all possible scenes and do not know the correct response to a challenge in advance, similar to the test set of a dataset benchmark. This allows evaluation instances to be sampled only at test time, unlike a fixed paired test set. A system must therefore be prepared for the broader task distribution even though only a small subset of configurations is evaluated in practice. This reduces the opportunity to overfit to a known physical test set. We hypothesize that, if tasks and scenes are sampled from the same distribution, sufficient samples allow methods to be compared at a chosen statistical confidence level independent of the robot platform, software stack, or policy used.

Figure 1. Stochastic Challenges target several desired aspects of robotic benchmarking: (i) system-level innovation, i.e. not restricting evaluation to specific hardware; (ii) local evaluation, avoiding the effort to transport systems and reducing the chance of errors when trying to reproduce methods; (iii) reduced overfitting, testing true generalization instead of pre-known tasks.
We instantiate this paradigm in a concrete benchmark built on IKEA KALLAX shelves: a widely available household furniture system that provides standardized geometry, modular inserts (doors, drawers, open shelves), and a natural set of interaction primitives, such as opening, retrieving, placing, and searching. A seeded randomizer generates configurations and object placements, each identified by a seed and exported as a spec from which any partner lab can reconstruct the same physical setup. We demonstrate the paradigm by comparing two robots tested in two different labs and analyzing how many randomized samples are required to draw confident conclusions.
In summary, our main contributions are:
The stochastic challenges framework for real-world manipulation benchmarking, together with a variance-decomposition framework that quantifies how many randomized samples are needed for statistically grounded cross-method comparisons.
The KALLAX benchmark – comprising a seeded configuration randomizer, a task distribution, and a web platform for cross-lab distribution and reproducibility.
A study comparing two robotic systems with different embodiments in two different labs on the benchmark, validating the statistical framework and characterizing the sample complexity required to separate methods.
Related Work
Simulation benchmarks
Because simulation offers repeatability at low cost, many manipulation benchmarks are built in simulation. RLBench [5] established a large-scale standardized suite spanning many manipulators and learning paradigms, while later benchmarks probe different aspects of generalization. Colosseum [1] studies robustness under visual and physical perturbations, [6] evaluate LLM-guided 3D policies for category-level generalization across articulation and placement, and [7, 8] examine vision-language reasoning for low-level and spatially grounded tabletop tasks. CALVIN [9], LIBERO [10], and ARNOLD [11] further evaluate generalization across predefined tasks, environments, objects, or goal states, while RoboCasa [12] and BEHAVIOR-1K [13] broaden evaluation toward large-scale realistic household activities and environments beyond the tabletop settings.
These benchmarks therefore already contain stochasticity or controlled variation, but the source of randomness is typically defined within a predetermined simulator, task suite or test split. In contrast, stochastic challenges sample the evaluation examples themselves: complete physical scenes and tasks are drawn at evaluation time from a specified distribution. This distinction is important in real-world manipulation, where perception errors, contact dynamics, execution failures, and environment-specific variation introduce uncertainty that cannot be fully reproduced in simulation.
Real-world manipulation benchmarks
Real-world benchmarks aim to provide comparability across physical setups by standardizing objects, layouts, tasks, or hardware. Some focus on reproducible tabletop manipulation through common object sets and physical arrangements, including YCB, GRASPA, Box-and-Blocks, and SceneReplica [2, 14, 15, 16]. However, repeatedly evaluating on the same objects and scenes can encourage overfitting and constrain the range of tasks. Centralized competitions such as RoboCup instead enforce comparability by gathering robots at a single site, which limits participation to those able to physically attend. Others standardize broader task protocols or hardware platforms, such as RB2, NIST Assembly, OCRTOC, TOTO, and Real Robot Challenge [17, 18, 19, 20, 21]. More recent benchmarks extend this direction toward complex manipulation or distributed evaluation: FurnitureBench [22] evaluates RL and IL models on miniature 3D-printed furniture, where miniature-scale interactions may not fully reproduce full-scale contact dynamics, and FMB [23] benchmarks multi-object manipulation with engineered blocks and multiple sensing modalities, though with tasks centered on functional manipulation rather than semantically rich household interaction. ManipulationNet and RoboArena [4, 3] extend evaluation across platforms or institutions, but they remain confined to tabletop manipulation. RoboDojo [24] further combines simulation and real-world evaluation across a broader set of manipulation tasks.
Despite this breadth, comparability is typically obtained through matched objects, layouts, tasks, or embodiments, often under tabletop or fixed-base settings. Such paired evaluation provides strong experimental control, but requires systems to be evaluated on matched physical instances, which in fixed benchmarks may already be known during method development and experiment preparation.
Our framework instead standardizes the distribution from which real-world scenes and tasks are sampled. Robots may therefore execute different samples drawn from the same distribution, enabling unpaired comparison across laboratories and embodiments without requiring identical hardware, software or scene. Seeded generation supports matched configurations when desired, while independent seeds enable unpaired evaluation. Moreover, whereas prior physical benchmarks commonly report aggregate success rates, standard deviations, or ranking statistics, our variance-based framework (Section 3) determines whether differences observed over finite stochastic trials are statistically distinguishable.
Method: Stochastic Challenges
Figure 2 summarizes the proposed stochastic-challenge paradigm. Rather than evaluating methods on fixed setups, we evaluate them on random configurations sampled from a defined distribution. Aggregating binary success outcomes across sampled configurations estimates expected performance over the full distribution. Sampling configurations at evaluation time gives the benchmark its rigor: participants cannot tune to known setups, while experiments remain independently reproducible across labs without shared hardware constraints. However, because results depend on the sampled configurations and preventing overfitting requires new samples across experiments, a key question arises: how many configurations and trials are needed to draw statistically significant conclusions? The remainder of this section develops the framework that answers this.

Figure 2. Concept overview. Seeded randomization generates reproducible KALLAX manipulation scenes; robots across labs (Stretch 3 and Spot) execute sampled configurations, with outcomes statistically aggregated for robust comparison.
The framework supports three comparison regimes: independently sampled configurations across systems, fully shared configurations, and mixed designs in which only a fraction of seeds is shared. These correspond to different trade-offs between generalization to unseen configurations and statistical efficiency, developed in § 3.1–§ 3.5.
Setup and benchmark score
We evaluate a robot on scenes drawn from the randomized distribution created by the seeded configuration generator (§ 4). A seed therefore fixes a complete scene layout / configuration. We further assume that is uniform, i.e. each scene has the same likelihood of being sampled. For a given scene, we define a set of tasks that a robot may attempt. Each task may be attempted several times, and every attempt yields a binary outcome indicating task success. As is commonly done in robotic benchmarking, we model attempts for each task as independent Bernoulli trials with an unknown per-task success probability ; the elementary attempt- and task-level moments are standard and are deferred to Appendix § 9. In practice, we approximate this assumption by resetting the robot software and task state between repeated attempts.
The quantity we ultimately report is the robot’s expected success over the distribution of scenes and tasks. Denoting as the robot’s true mean success on seed , the benchmark objective is , the expectation taken over seeds drawn from the benchmark distribution; Measuring rather than performance over a manually chosen scene or set of scenes is what makes the benchmark a property of the distribution rather than of specific setups.
Because and are unobservable, we estimate them by a three-level average – over repeated attempts per task, tasks within a scene, and over scenes:
where is the number of repeated attempts per task and the number of sampled seeds / scenes. Averaging to a seed-level score before averaging over seeds ensures that scenes with more possible tasks, or tasks that happened to be attempted more often, do not dominate the benchmark mean. Under the sampling assumptions of § 3.5, is unbiased for (Appendix § 9).
The two choices the experimenter controls are and : how often each task is attempted, and how many randomized seeds are evaluated. The remainder of this section asks how the uncertainty of depends on these two quantities, and how they should be chosen.
Two sources of variance
The uncertainty in has two qualitatively different origins. The first is trial noise: even on a single fixed configuration a robot succeeds on some attempts and fails on others, so a task’s success rate is estimated from only Bernoulli trials. The second is configuration difficulty: randomized scenes are likely to differ in how difficult they are. Therefore, we cannot control for the difficulty at benchmark level and the per-seed performance of a given robot varies from seed to seed. The spread of per-seed performance across the distribution then measures how strongly depends on which scenes happen to be sampled.
Treating a single seed-level score as random over both the sampled seed and the trial noise on it, the law of total variance separates these two contributions exactly (Appendix § 10). Under approximately equal trials per task, the result is a planning variance
where aggregates the per-task Bernoulli variance , reduced by averaging over the tasks of a seed and over the seed distribution; its exact form is given in Appendix § 10. Averaging over independently sampled seeds gives the variance of the reported benchmark mean,
This expression makes the allocation problem explicit. Both terms are divided by the number of seeds , but only the second is further reduced by the number of attempts . In particular, as the benchmark variance does not vanish but approaches a floor , set entirely by the number of evaluated configurations: no amount of per-configuration repetition can beat this floor, and it can be lowered only by sampling more seeds. The framework, however, does not require configurations to have equal difficulty. Greater variation across seeds instead appears as a larger between-configuration variance and consequently increases sample complexity.
Sample complexity
The variance expression of § 3.2 turns directly into experiment-planning rules. For a given robot, a two-sided confidence interval for has half-width , where denotes the -quantile of the standard normal distribution, so reporting the benchmark score to a target precision requires seeds.
A benchmark’s primary purpose, however, is comparison. For two robots and we estimate their performance gap ; when each robot is evaluated on its own set of independently sampled seeds, the two estimates are independent, so the comparison variance is (assuming comparable planning variance for both). Under the normal approximation, detecting a true gap of size at significance level and power requires
seeds per robot. The significance level is the probability of a Type 1 error: concluding a significant advantage for one robot that does not truly exist. is the probability of a Type 2 error: Not being able to find an advantage through the benchmark that actually exists.
Equation (1) exposes the cost structure of the benchmark. The required number of seeds grows with the configuration-difficulty spread and the trial noise , and falls quadratically as the gap to be resolved grows: small differences between strong systems are expensive to certify. The trial-noise contribution can be reduced by raising , but the contribution cannot: it is fixed by the distribution and, under independent seeds, imposes a hard lower bound that persists even as . Because this dominant term arises purely from testing the two robots on different sampled configurations, it can be eliminated by testing them on the same ones; we develop this in § 3.4. Finally, since and are not known in advance and must be estimated from a pilot set of seeds, (1) should be read as a planning estimate rather than an exact requirement (§ 3.5).
Comparing robots across labs
The dominant cost in (1) is the between-configuration term , and it arises only because the two robots were evaluated on different sampled configurations. If instead both robots are run on the same seeds, the difficulty of each configuration acts on both equally and cancels in their per-seed difference. Writing the per-seed gap under a paired random-effects model in which a common seed-difficulty effect enters both robots’ scores (Appendix § 11), that shared effect subtracts out, leaving only the robots’ relative skill and the residual trial noise.
More generally, let be the fraction of seeds shared by both robots. The comparison variance then interpolates between the independent and fully paired designs:
At this recovers the independent-seed case of § 3.3; at the between-configuration term vanishes entirely, leaving – a fully paired design pays only the trial-noise price.
Substituting (2) into the power requirement gives the number of seeds needed to detect a gap under partial sharing:
Fully paired evaluation () is statistically efficient because both systems encounter the same configurations. However, when the complete physical test set is known in advance, repeated evaluation may encourage adaptation to those particular objects and scenes rather than measuring generalization over the broader challenge distribution. At the other extreme, independent sampling () better evaluates generalization to unseen configurations but may require more trials. Intermediate values provide a practical compromise between these objectives, with the appropriate choice depending largely on the cost of reconstructing physical scenes.
Assumptions
We assume independent seed sampling from the same benchmark distribution and require a central service (e.g. web service) to prevent selective reporting. We model repeated attempts as independent Bernoulli trials and approximate this assumption by resetting the task and software state. We capture variations in seed difficulty through and assume a common difficulty effect only for paired comparisons. Finally, we use a normal approximation of aggregated seed-level scores for sample-complexity analysis.
The KALLAX Benchmark
Based on the benchmarking methodology in § 3, we propose a mobile manipulation benchmark for robotic furniture interaction. We focus on an easily reproducible physical environment (the IKEA KALLAX shelf) and randomized configuration generation to obtain a large number of comparable real-world trials.
Randomizer We randomly configure the KALLAX grid by first uniformly selecting a specified number of empty positions without replacement. In our experiments, an empty fraction of results in six empty positions. Each remaining position is assigned an insert from a configurable pool of up to 82 KALLAX-compatible options, including doors and drawers. The evaluated configurations contain approximately 2.5 times as many door as drawer inserts, with each drawer insert comprising two drawers. Object-search tasks additionally sample from 79 YCB objects. Each configuration has a unique seed and is exported in machine- and human-readable form with a rendered visualization.
Tasks and success criteria. On top of the randomized setups we define manipulation tasks that reflect household interactions and are executable on a mobile manipulator, for example: (1) Open: open a specified door/drawer to a target state, (2) Pick: retrieve a specified object from a specified shelf/drawer, (3) Place/Move: move an object to a target cubby or shelf position. Objects to move/search for may be randomized as well as the shelf’s general configuration. This creates many trials with controlled variability while keeping tasks understandable and close to real-world use cases.
Web interface and cross-lab reproducibility We provide a publicly accessible website that (i) generates configurations (with seeds), (ii) presents assembly instructions and object lists, and (iii) exports task instances and records reported outcomes. The generator is available through the KLX Benchmark2.
A partner lab may reproduce a configuration by following the same specification, execute the same task set, and upload results. This supports direct replication studies (same configuration) and generalization studies (different samples from the same distribution). The platform records generated configurations and reported outcomes to support transparency and discourage selective reporting.
Reliable success labeling Some failures are ambiguous (partial opening, object grasped but dropped, slight miss on the knob). Clearly defined success criteria per task and failure type are essential. Long-term automated verification (e.g., VLM-based video checks) is promising, but for this project, we validate labels through careful logging and human review.
Experiments
Our experiments evaluate the benchmarking methodology itself rather than any single manipulation method, and are organized around three questions: (i) reproducibility – whether seed-defined configurations can be reconstructed across different laboratories and robotic platforms under a common seed; (ii) robustness – how strongly success rates vary across randomized configurations, and whether a capable system maintains consistent performance across them; and (iii) comparability – whether the statistical framework of § 3 lets us draw controlled, confident comparisons between systems evaluated in different labs.
Benchmarked Systems and Tasks
We benchmark two heterogeneous robots in labs across two countries: a Hello Robot Stretch 3 and a Boston Dynamics Spot with a manipulator arm. Their differing locomotion, reach, and kinematics support cross-embodiment evaluation. Each platform uses a distinct but related autonomy stack. Stretch 3 uses Stretch-Compose [25], which is based on Spot-Compose [26]. Spot uses a stack based on SpotLight [27], which in turn builds on Spot-Compose. Both stacks provide environment reasoning, perception for handle and object detection, and navigation and manipulation functionality conditioned on location or natural-language goals. Figure 3 shows the evaluation protocol: a randomizer generates seeded configurations that both labs reconstruct from a shared specification before executing the common task set, recording outcomes per attempt. For the shared-seed experiments, a single researcher reconstructed the scenes in both laboratories. Thus, the experiments vary the laboratory, robot embodiment, and software stack while controlling for variability introduced by different reconstruction teams.

Figure 3. Experimental setup. Our KALLAX benchmark draws sample configs from the same distribution, which are then recreated in two different labs to evaluate the two different robotic systems.
Within the KALLAX Benchmark, we evaluate two task types: door and drawer opening. For each, we consider attempts successful only if the robots open a door or drawer by at least 10 cm. We analyze the two task types separately, using all doors or all drawers within a sampled configuration as the corresponding task set .
Overall, we conduct 568 experimental trials. For doors, we run 62 trials over 6 seeds with Spot and 130 trials with 7 seeds for Stretch. For drawers, we run 88 trials over 6 seeds with Spot and 288 trials over 8 seeds with Stretch. As a reference for experimental effort, approximately 150 trials with Spot were conducted by a single experimenter over three days.
Variance Analysis
Table 1 shows results for different subsets of evaluation seeds. Random subset estimates exhibit high seed variance, especially for Spot, confirming substantial variation in seed difficulty. Comparing the standard same-configuration protocol to random-seed subsets, we find that small-sample noise can even reverse the apparent robot ranking (e.g., Drawers, Spot subset 1 vs. Stretch subset 3). This highlights the need for statistical significance measures beyond raw success rates in stochastic benchmarks.
| Test Scenes | Drawer Opening | Door Opening | ||
| Spot | Stretch | Spot | Stretch | |
| same seeds | 92.7% | 58.1% | 86.3% | 12.8% |
| subset 1 | 65.0% | 61.9% | 66.7% | 16.8% |
| subset 2 | 100.0% | 51.3% | 75.0% | 20.2% |
| subset 3 | 87.5% | 70.7% | 100.0% | 30.0% |
Table 1. Testing on same vs. different shelf configurations. Averaged success rates vary substantially with the selected scene subset when few seeds are used.
Detectable performance margin
We invert the planning framework of § 3: rather than asking how many seeds are needed for a target margin, we ask, given the seeds we have already evaluated, what is the smallest performance difference between the two robots that we can resolve? Throughout this analysis we adopt the pilot-study settings and – yielding – which trade a slightly higher Type I rate for power at small samples. We follow the per-task aggregation of § 3.1.
For an unpaired comparison between Spot and Stretch, the standard error of the difference in success rates for task type is and the smallest reliably detectable difference is .
Table 2 reports the observed gaps and detectable margins. Both task types are distinguishable at the chosen significance and power: Spot outperforms Stretch by 0.531 on doors against a detectable margin of 0.333, and by 0.244 on drawers against a margin of 0.232. As a complementary uncertainty estimate, Welch 95% confidence intervals for the Spot–Stretch difference are 23.4–82.8 percentage points for doors and 4.2–44.6 points for drawers, with Welch degrees of freedom and respectively. The drawer comparison thus sits close to the decision boundary, and the framework’s scaling implies roughly more seeds would be needed to halve the margin. These results show that the framework can distinguish robotic systems evaluated in different laboratories and on different scenes, while also illustrating the cost of statistical resolution: smaller detectable margins require substantially more evaluation data.
| Task | Spot | Stretch | difference | different random seeds | 50/50 same and different seeds | ||
| detectable margin | result | detectable margin | result | ||||
| Door | 0.7109 | 0.1802 | 0.5307 | 0.333 | distinguishable | 0.256 | distinguishable |
| Drawer | 0.8792 | 0.6352 | 0.2439 | 0.232 | distinguishable | 0.183 | distinguishable |
Table 2. Observed robot success rates and detectable margins. Margins are computed for and 80% power. Partial successes are counted as failures, and all configuration labels are included.
Mixed shared-randomized comparison. As described in § 3.4, a potential way to achieve lower detectable margins is to partially test robots on the same seeds. The right half of Table 2 reports a mixed design with , where half of the comparison is performed on shared configurations and half on independently chosen configurations. We observe that this reduces the detectable margin for both task types: from 33.3 to 25.6 percentage points for doors and from 23.2 to 18.3 percentage points for drawers. Mixed sharing therefore provides a practical compromise: it reduces the required benchmarking effort while preserving evaluation on independently sampled configurations. Importantly, both the fully independent and mixed designs lead to the same qualitative conclusion for the two systems in our experiments.
Limitations
A key limitation is that a considerable number of trials is required to obtain a statistically detectable performance margin. Moreover, because the same researcher reconstructed the shared-seed scenes in both laboratories, the present study does not quantify reconstruction variability that may arise when independent teams reconstruct the same configuration. Finally, the current evaluation relies on honest and well-calibrated reporting of success. Requiring proof with uploaded execution videos, and eventually automated VLM-based evaluation, could improve verifiability and standardization.
Conclusion
We introduce Stochastic Challenges, a novel framework for comparing heterogeneous robotic systems on real-world tasks sampled from a shared distribution. Unlike fixed paired test sets, the framework supports independent, shared, and mixed configuration sampling, while explicitly accounting for execution noise and configuration difficulty. The KALLAX benchmark demonstrates the approach in mobile manipulation, and our two-lab study shows that statistically grounded cross-embodiment comparison is feasible without requiring identical robotic platforms or software stacks.
Acknowledgments
We thank Laurent Côté for support with the experiments at the University of Bonn and for valuable discussion on the statistical analysis, and Tim Engelbracht for assistance with the experiments at ETH Zurich. This work was partially supported by the Lamarr Institute for Machine Learning and Artificial Intelligence, the Robotics Institute Germany, and Google. Gülhan Şikaroğlu is supported by the YLSY Scholarship Program of the Republic of Türkiye Ministry of National Education under Law No. 1416.
Estimators
This appendix derives the elementary properties of the per-task, seed-level, and benchmark-mean estimators introduced in § 3.1.
Per-trial model. For a fixed robot , seed , and task , repeated trials are independent trials with unknown success probability :
so that and .
Per-task estimator. For repeated trials of task in seed ,
Conditional on ,
so is unbiased for and its variance shrinks as .
Seed-level estimator. The seed-level score averages task estimates uniformly,
which is conditionally unbiased for the true seed-level performance :
The uniform average over tasks, rather than over raw trials, prevents tasks retried more often from exerting disproportionate influence on .
Benchmark mean. Over independent seeds drawn from the benchmark distribution,
Taking expectation over both trial noise and seed sampling,
so is an unbiased estimator of the benchmark objective .
Variance
We derive the planning variance used in § 3.2.
Within-seed variance. Conditional on a seed , treating per-task estimates as conditionally independent,
Defining the per-seed Bernoulli-variance aggregate
this becomes . Between-seed variance. Configurations differ in difficulty, so the true seed-level performance varies across seeds:
Total variance via law of total variance. Treating as random over the joint distribution of seeds and trial noise, the law of total variance gives
Substituting (Appendix § 9) and ,
where . We denote this combined quantity the planning variance .
Benchmark mean variance. Under independent seed draws, averaging over seeds gives
Comparison
We derive the variance of the difference estimator under the unpaired, fully paired, and mixed designs of § 3.3–§ 3.4.
Unpaired comparison. If robots and are evaluated on independent sets of seeds each, and are independent, and
where the last step assumes comparable planning variance . Substituting into the normal-approximation power requirement and solving for yields the sample-size formula (1) of § 3.3.
Paired comparison. If both robots are evaluated on the same seeds, we model each seed-level score as
with a common seed-difficulty effect satisfying and residual within-seed trial noise with . The per-seed difference
cancels exactly. Assuming the residuals are independent across robots given the seed,
removing the between-seed variance term entirely.
Mixed design. For a fraction of seeds shared between the two robots, the variance of interpolates between the unpaired and fully paired endpoints. Treating shared seeds as paired and unshared seeds as independent, an additive approximation gives
which recovers at and at . Substituting into the power requirement gives the mixed-design sample-size formula of § 3.4.
Extensions
We describe three refinements of the analysis used in the experiments as stratified reports, a model-based robustness check, and the basis for equivalence claims.
Task-type stratification. Different interaction types differ systematically in difficulty – a recessed bottom drawer is not comparable to an open shelf at chest height. The variance components and benchmark scores can therefore be estimated separately for each task class
Restricting the per-task estimators of Appendix § 9 to tasks of class yields stratified estimates , , and . These allow different task classes to have different mean success rates and variance components, and prevent unbalanced trial mixes from biasing the overall comparison.
Logistic mixed-effects model. The variance decomposition of Appendix § 10 relies on a normal approximation to averaged Bernoulli outcomes. As a more exact alternative, we fit a logistic mixed-effects model directly to the trial-level binary outcomes:
with fixed robot and task-type effects, their interaction , a random seed effect , and a random task-within-seed effect . The model is used as a robustness check on the headline comparison.
Equivalence testing. Failure to detect a difference at significance level is not evidence that two robots are equivalent. To claim practical equivalence we specify in advance an equivalence margin and conclude that and are practically equivalent if a confidence interval for lies entirely within . The choice of reflects the smallest performance difference considered practically meaningful for the benchmark.
References
- [1]W. Pumacay, I. Singh, J. Duan, R. Krishna, J. Thomason, and D. Fox. The colosseum: A benchmark for evaluating generalization for robotic manipulation, 2024. URL https://arxiv.org/abs/2402.08191.
- [2]B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 International Conference on Advanced Robotics (ICAR), pages 510–517, 2015. doi:10.1109/ICAR.2015.7251504.DOI
- [3]P. Atreya, K. Pertsch, T. Lee, M. J. Kim, A. Jain, A. Kuramshin, C. Eppner, C. Neary, E. Hu, F. Ramos, J. Tremblay, K. Arora, K. Ellis, L. Macesanu, M. T. Villasevil, M. Leonard, M. Cho, O. Aslan, S. Dass, J. Wang, W. Reger, X. Yuan, X. Yang, A. Gupta, D. Jayaraman, G. Berseth, K. Daniilidis, R. Martin-Martin, Y. Lee, P. Liang, C. Finn, and S. Levine. Roboarena: Distributed real-world evaluation of generalist robot policies, 2025. URL https://arxiv.org/abs/2506.18123.
- [4]Y. Chen, K. Kimble, E. H. Adelson, T. Asfour, P. Chanrungmaneekul, S. Chitta, Y. Chitambar, Z. Chen, K. Goldberg, D. Kragic, H. Li, X. Li, Y. Li, A. Prather, N. Pollard, M. A. Roa-Garzon, R. Seney, S. Sha, S. Wang, Y. Xiang, K. Zhang, Y. Zhu, and K. Hang. Manipulationnet: An infrastructure for benchmarking real-world robot manipulation with physical skill challenges and embodied multimodal reasoning, 2026. URL https://arxiv.org/abs/2603.04363.
- [5]S. James, Z. Ma, D. R. Arrojo, and A. J. Davison. RIbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019–3026, 2020. doi:10.1109/LRA.2020.2974707.DOI
- [6]R. Garcia, S. Chen, and C. Schmid. Towards generalizable vision-language robotic manipulation: A benchmark and llm-guided 3d policy. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 8996–9002, 2025. doi:10.1109/ICRA55743.2025.11127315.DOI
- [7]E. Zhao, V. Raval, H. Zhang, J. Mao, Z. Shangguan, S. Nikolaidis, Y. Wang, and D. Seita. Manipbench: Benchmarking vision-language models for low-level robot manipulation, 2025. URL https://arxiv.org/abs/2505.09698.
- [8]S. Zhang, Z. Xu, P. Liu, X. Yu, Y. Li, Q. Gao, Z. Fei, Z. Yin, Z. Wu, Y.-G. Jiang, and X. Qiu. Vlabench: A large-scale benchmark for language-conditioned robotics manipulation with long-horizon reasoning tasks, 2024. URL https://arxiv.org/abs/2412.18194.
- [9]O. Mees, L. Hermann, E. Rosete-Victoria, and W. Burgard. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L), 7(3):6323–6330, 2022. doi:10.1109/LRA.2022.3173719. URL https://arxiv.org/abs/2112.03227.
- [10]B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning, 2023. URL https://arxiv.org/abs/2306.03310.
- [11]R. Gong, J. Huang, Y. Zhao, H. Geng, X. Gao, Q. Wu, W. Ai, Z. Zhou, D. Terzopoulos, S.-C. Zhu, B. Jia, and S. Huang. ARNOLD: A benchmark for language-grounded task learning with continuous states in realistic 3D scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20483–20495, 2023. doi:10.1109/ICCV51070.2023.01873. URL https://arxiv.org/abs/2304.04321.
- [12]S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots, 2024. URL https://arxiv.org/abs/2406.02523.
- [13]C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martín-Martín, C. Wang, G. Levine, W. Ai, B. Martinez, H. Yin, M. Lingelbach, M. Hwang, A. Hiranaka, S. Garlanka, A. Aydin, S. Lee, J. Sun, M. Anvari, M. Sharma, D. Bansal, S. Hunter, K.-Y. Kim, A. Lou, C. R. Matthews, I. Villa-Renteria, J. H. Tang, C. Tang, F. Xia, Y. Li, S. Savarese, H. Gweon, C. K. Liu, J. Wu, and L. Fei-Fei. Behavior-1k: A human-centered, embodied ai benchmark with 1,000 everyday activities and realistic simulation, 2024. URL https://arxiv.org/abs/2403.09227.
- [14]F. Bottarel, A. Altobelli, U. Pattacini, and L. Natale. Graspa-fying the panda: Easily deployable, fully reproducible benchmarking of grasp planning algorithms. IEEE Robotics & Automation Magazine, 31(4):21–31, 2024. doi:10.1109/MRA.2023.3256272.DOI
- [15]A. S. Morgan, K. Hang, W. G. Bircher, F. M. Alladkani, A. Gandhi, B. Calli, and A. M. Dollar. Benchmarking cluttered robot pick-and-place manipulation with the box and blocks test. IEEE Robotics and Automation Letters, 5(2):454–461, 2020. doi:10.1109/LRA.2019.2961053.DOI
- [16]N. Khargonkar, S. H. Allu, Y. Lu, J. J. P, B. Prabhakaran, and Y. Xiang. Scenereplica: Benchmarking real-world robot manipulation by creating replicable scenes, 2024. URL https://arxiv.org/abs/2306.15620.
- [17]S. Dasari, J. Wang, J. Hong, S. Bahl, Y. Lin, A. Wang, A. Thankaraj, K. Chahal, B. Calli, S. Gupta, D. Held, L. Pinto, D. Pathak, V. Kumar, and A. Gupta. Rb2: Robotic manipulation benchmarking with a twist. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.arxiv.org/abs/2203.08098
- [18]W. Lian, T. Kelch, D. Holz, A. Norton, and S. Schaal. Benchmarking off-the-shelf solutions to robotic assembly tasks. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1046–1053, 2021. doi:10.1109/IROS51168.2021.9636586.DOI
- [19]Z. Liu, W. Liu, Y. Qin, F. Xiang, M. Gou, S. Xin, M. A. Roa, B. Calli, H. Su, Y. Sun, and P. Tan. Ocrtoc: A cloud-based competition and benchmark for robotic grasping and manipulation. IEEE Robotics and Automation Letters, 7(1):486–493, 2022. doi:10.1109/LRA.2021.3129136.DOI
- [20]G. Zhou, V. Dean, M. K. Srirama, A. Rajeswaran, J. Pari, K. Hatch, A. Jain, T. Yu, P. Abbeel, L. Pinto, C. Finn, and A. Gupta. Train offline, test online: A real robot learning benchmark. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9197–9203, 2023. doi:10.1109/ICRA48891.2023.10160594.DOI
- [21]S. Bauer, F. Widmaier, M. Wüthrich, A. Buchholz, S. Stark, A. Goyal, T. Steinbrenner, J. Akpo, S. Joshi, V. Berenz, V. Agrawal, N. Funk, J. U. De Jesus, J. Peters, J. Watson, C. Chen, K. Srinivasan, J. Zhang, J. Zhang, M. R. Walter, R. Madan, C. Schaff, T. Maeda, T. Yoneda, and D. Yarats. Real robot challenge: A robotics competition in the cloud. arXiv preprint arXiv:2109.10957, 2021. doi:10.48550/arXiv.2109.10957.DOI
- [22]M. Heo, Y. Lee, D. Lee, and J. J. Lim. Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation, 2023. URL https://arxiv.org/abs/2305.12821.
- [23]J. Luo, C. Xu, F. Liu, L. Tan, Z. Lin, J. Wu, P. Abbeel, and S. Levine. Fmb: a functional manipulation benchmark for generalizable robotic learning, 2024. URL https://arxiv.org/abs/2401.08553.
- [24]T. Chen, Y. Chen, Z. Li, J. Tang, K. Su, W. Wan, B. Chen, H. Lu, H. Yan, H. Su, Z. Dou, K. Wang, D. Zhang, Y. Liu, Y. Qin, Q. Liang, Q. Wu, Z. Lin, W. Lin, Y. Wang, M. He, T. Wu, R. Wu, J. Zhou, K.-C. Lei, H. Yu, Y. Ji, W. Jin, G. Lin, X. Li, Q. Xiong, R. Xu, Z. Li, W. Chai, E. Xie, Z. Wang, Y. Mu, H. Dong, W. Matusik, M. Ding, W. Ding, P. Luo, and M. Tomizuka. Robodojo: A unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies, 2026. URL https://arxiv.org/abs/2607.04434.
- [25]Robot Perception and Learning Lab. Open-vocabulary and semantic-aware search and retrieval of objects using the stretch robot. https://github.com/rpl-bonn/stretch-compose, 2026. Accessed: 2026-02-26.
- [26]O. Lemke, Z. Bauer, R. Zurbrügg, M. Pollefeys, F. Engelmann, and H. Blum. Spot-compose: A framework for open-vocabulary object retrieval and drawer manipulation in point clouds, 2024. URL https://arxiv.org/abs/2404.12440.
- [27]T. Engelbracht, R. Zurbrügg, M. Pollefeys, H. Blum, and Z. Bauer. Spotlight: Robotic scene understanding through interaction and affordance detection, 2024. URL https://arxiv.org/abs/2409.11870.