WorksComputer Science
HospitalBench: Diagnosing Where and Why Embodied Agents Fail on Long Hospital Task Chains
Abstract
Long-horizon embodied tasks fail in ways that end-task success cannot see. HospitalBench is a diagnostic benchmark of ten hospital chores in NVIDIA Isaac Sim, each two unscored warm-up steps followed by a causal chain of five scored sub-tasks in one continuous simulation: the state one step leaves (an object pose, a choice, a fact seen once) determines what the next step must do, unlike benchmarks that place sub-tasks side by side. Vision-language agents act through skill-level tools from camera images. A scripted robot that makes only the calls a model gets is the solvability check of a chore. From each trajectory the benchmark reads where a chain breaks, what the agent did after a failed call, and how far its own position estimate drifts from the true one. This version reports first results only: 15 of 60 planned runs (three models at high effort, one run each on chores T1 and T10 from both start states and T3 from one), with scripted-robot proofs on three of the ten chores. Of the 15 runs, 2 passed all five sub-tasks. Runs still going after sub-tasks 1 to 5 were 15, 8, 4, 2, 2. Of 189 failed calls, 168 were followed by a different call and 111 by a call that did not fail. These are first results and not a full benchmark. We will release the scenes, task chains, scripted robot, evaluation harness and trajectories.

Figure 1. One chain of chore T1 (instrument count before closure): five sub-tasks in one simulation with no reset, after two unscored warm-ups. Top: a scripted robot passes all five (frames from an earlier scene version, before the KEEP card). Arrows: what the next sub-task takes from the scene or the picture: V a trolley pose, C a choice the robot made, P what the last step placed. The first link is also F, a fact shown once (a KEEP card in the first picture of T1.01 names the forceps to leave). Bottom: first results, 15 of 60 planned runs. Each row is a chore and start state, each group of chips a model at high effort, each chip a sub-task T#.01 to T#.05. Chores T1 and T10 ran from both start states and T3 from start 0, one run per cell.
Long-horizon embodied tasks fail in ways that end-task success cannot see. HospitalBench is a diagnostic benchmark of ten hospital chores in NVIDIA Isaac Sim, each two unscored warm-up steps followed by a causal chain of five scored sub-tasks in one continuous simulation: the state one step leaves (an object pose, a choice, a fact seen once) determines what the next step must do, unlike benchmarks that place sub-tasks side by side. Vision-language agents act through skill-level tools from camera images. A scripted robot that makes only the calls a model gets is the solvability check of a chore. From each trajectory the benchmark reads where a chain breaks, what the agent did after a failed call, and how far its own position estimate drifts from the true one. This version reports first results only: 15 of 60 planned runs (three models at high effort, one run each on chores T1 and T10 from both start states and T3 from one), with scripted-robot proofs on three of the ten chores. Of the 15 runs, 2 passed all five sub-tasks. Runs still going after sub-tasks 1 to 5 were 15, 8, 4, 2, 2. Of 189 failed calls, 168 were followed by a different call and 111 by a call that did not fail. These are first results and not a full benchmark. We will release the scenes, task chains, scripted robot, evaluation harness and trajectories.
Introduction
Where along a chain of dependent chores does an embodied agent’s performance degrade, and why? Long-horizon benchmarks report that success falls as tasks get longer [22, 28], and recent work traces failures to the seams between skills [24, 35], but one end-task score cannot say at which step, at which link or by which kind of failure a chain breaks. A chain shows degradation by carrying only if each step needs the actual result of the one before, or a fact seen earlier: a trolley parked a few centimetres off its mark moves every grasp that follows, whereas a card seen once names the forceps to take and is gone by the time they are needed. HospitalBench is built for this diagnosis: ten chores of five dependent sub-tasks (50 sub-tasks, 40 links) in six simulated hospital areas (§ 3, Table 2), each link depending on the step before it in one of four ways (Figure 2).

Figure 2. Link diagram of T1, current version (vqsnd). Five scored sub-tasks run in one simulation with no reset, after two unscored warm-ups (W1, W2: press a YES or NO panel to answer a lamp question). Each arrow names the link kind and what the next sub-task takes from the scene the robot left or the picture it saw: V a cart or trolley pose, C a choice on the robot’s own result, P what this step placed, F a fact shown once (a KEEP card in the first picture of T1.01 only, and T1.02 must take the other forceps). The link kinds of T1 read VCPC, with T1.01→T1.02 also F.
Table 2. Scale and statistics as built, from the task files, the room builder, the T1 version ledger and the run registry.
| Quantity | Value |
| Chores, areas, rooms | 10, 6, 10 (one room per chore) |
| Sub-tasks per chore, total | 5, 50 (plus 2 unscored warm-ups before T1.01) |
| Links by kind (V, C, P) | 14, 16, 10 (total 40, 4 per chore) |
| Links of kind F | 1 (T1.01→T1.02, also V), no other chore has one yet |
| Start states | 2 public per chore (20), hidden split not built |
| SimReady assets | 74 distinct of 2,595 folders, 44 with joints, 3 stand-ins |
| Named harm events | 36 (2 to 5 per chore) |
| T1 versions tried | 26 in the ledger |
| Scripted-robot proofs | T1 and T10 on both starts, T3 on start 0 (3 of 10 chores) |
| Model runs in this paper | 15 of 60 planned: 3 models, T1 and T10 on both starts, T3 on start 0, one run per cell |
A model sends every robot call and a controller executes it. A walk is one model at one effort tier (the reasoning-effort setting of its API call) running the sub-tasks of one chore in order, with no reset (a chain), from one start state (the initial scene). The scene state is the poses of the robot, objects and people, which every pass term and harm event reads. A walk ends at its first failed sub-task. The number passed before it is its passed in a row score. The scripted robot is a hand-written program that makes only the calls a model makes. The calibration model is the model whose walks were used to choose the version of a chore (Opus 5.5 high), so its results are not held out. Held-out models (GPT-6.1 Terra, GPT-6.1 Sol) were not used for that choice (§ 4).
What we find so far (§ 5). This version reports first results only. We ran Opus 5.5 high, GPT-6.1 Sol high and GPT-6.1 Terra high once on each of five cells (chores T1 and T10 from both start states, T3 from start 0), 15 runs of the 60 planned, outside the staged protocol of § 4. The scripted robot has a proof on those chores and starts, and on none of the other seven chores. Two runs passed all five sub-tasks, both on T1 (GPT-6.1 Terra from start 0, GPT-6.1 Sol from start 1). Runs still going after sub-tasks 1 to 5 were 15, 8, 4, 2, 2. Every other run ended at its first failed sub-task, and in 7 of those 13 runs this was the first free sub-task T#0.02, never the guided T#0.01. On T1, 3 of the 6 runs that reached T1.02 failed it. Two took the forceps that a card shown only in the first picture of T1.01 named to keep, and one never picked either forceps. After a failed call the next call differed in 168 of 189 cases and did not fail in 111. With one run per cell there is no interval and no ranking of models, and these are first results, not a full benchmark. The other seven chores have no model run.
Failure modes and harm events. We read four failure modes from a trajectory (§ 4): compounding errors per link, recovery, state drift and tool-use breakdowns. This version reports the last three from the 15 runs (§ 5.1). Compounding by landing position and harm events are defined but not analysed here. Harm events (an instrument left off the tray, the robot closing in on a person) are simulator proxies and not injuries.
Honest scope. No single task is new: every kind of action in our tasks is in a task prompt of seven existing benchmarks, and five further 2026 benchmarks are in the same comparison (Table 1). The design property we add is link dependence, tested link by link (the link test, § 3.2) and through a fact shown once. The link test has been run on 1 of the 40 links, for the scripted robot on an earlier T1 version. The rest have only a paper check on assumed numbers (§ 3.2). Only T1, T10 and T3 have model runs. No method has been trained on HospitalBench, and we do not claim it works as a training environment.
Table 1. Comparison with twelve earlier and five CVPR 2026 benchmarks (last five rows: first pages only). Scale: tasks, VLN-MME records of its released validation subsets (R2R 200, REVERIE 300, ObjectNav 200, CVDN 200), counted by us, PARTNR instructions, buildings, goals. Models: frontier models scored by the authors (✓ yes, ✗ no, ∼ some, and in brackets the models or workflows evaluated, where read). Moves: the model sends every move, an action or pose at each step. ∼ skill calls or primitives. Chain: separately instructed, graded sub-tasks each starting from the scene the last left, no reset (∼: one instruction with sub-goals graded by progress). Fact: fact carried, a step needs a fact seen in an earlier step. Alone: alone vs chained, the same sub-task run alone beside the chained walk. Link: link test, each link tested by moving the earlier result inside its band. ?: not stated or not read. Ours is ∼ where only three chores, one link or three models have run. The link test covers 1 of 40 links, and the alone control has no data yet. The new columns of the older rows and the ? cells are not checked against every full paper.
| Benchmark | Scale | Models | Moves | People | Chain | Fact | Alone | Link |
| LIBERO-10 [20] | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | |
| CALVIN [21] | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | |
| EmbodiedBench [34] | ✓ | ∼ | ✗ | ✗ | ✗ | ✗ | ✗ | |
| VLN-MME [40] | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | |
| VLABench [39] | ∼ | ∼ | ✗ | ✗ | ✗ | ✗ | ✗ | |
| BEHAVIOR-1K [17] | ✗ | ∼ | ✗ | ∼ | ? | ✗ | ✗ | |
| LADDER [22] | ✓ | ∼ | ✗ | ∼ | ? | ✗ | ✗ | |
| PARTNR [7] | ∼ | ∼ | ✓ | ∼ | ? | ✗ | ✗ | |
| Habitat 3.0 [23] | ✗ | ∼ | ✓ | ✗ | ✗ | ✗ | ✗ | |
| HA-VLN 2.0 [9] | ✗ | ∼ | ✓ | ✗ | ✗ | ✗ | ✗ | |
| Rheo [42] | ✗ | ✓ | ✗ | ∼ | ✗ | ✗ | ✗ | |
| ORBIT-Surgical [38] | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | |
| MANSION [8] | ? | ? | ? | ∼ | ? | ✗ | ✗ | |
| LMEE-Bench [30] | ✓ | ✗ | ? | ∼ | ✓ | ✗ | ✗ | |
| AGENTSAFE [37] | ✓ (9+2) | ✗ | ? | ✗ | ✗ | ✗ | ✗ | |
| LIBERO-Plus [11] | ✗ (10 VLAs) | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | |
| CapNav [26] | ✓ (13) | ✗ | ? | ✗ | ✗ | ✗ | ✗ | |
| Ours | 10 chores, 50 sub-tasks | ∼ (3) | ∼ | ✓ | ✓ | ∼ | ✗ | ∼ |
Contributions. (1) A diagnostic benchmark of ten hospital chores, each five dependent sub-tasks in one reset-free simulation, with the kind of each of the 40 links stated, a link kind for a fact shown once, and 36 named harm events (§ 3). (2) A staged protocol (stage 0 calibration, an alone check M1, held-out stages 1 to 3) that reports the calibration model apart from held-out models, and the first 15 runs, made outside it (§ 4). (3) First results on three chores with per-run records of where a chain ended, what followed each failed call and how far the robot’s position estimate drifted, and a read of the T1.01→T1.02 link (§ 5.1).
Related work
The skills are not new. Every kind of action in our tasks appears in at least one of 983 task prompts we collected verbatim from seven benchmarks: LIBERO-10 [20], CALVIN [21], EmbodiedBench [34], VLN-MME [40], VLABench [39], BEHAVIOR-1K [17] and LADDER [22] (counts in Table 1). Five CVPR 2026 benchmarks are in the table from their first pages only: MANSION [8] (multi-floor scenes), LMEE-Bench [30] (multi-goal navigation, then questions on what was seen), AGENTSAFE [37] (hazardous instructions), LIBERO-Plus [11] (perturbed LIBERO tasks) and CapNav [26] (capability-conditioned navigation). Chains of up to ten sub-goals exist [22], CALVIN reports success at each position of five-step chains [21], and mobile manipulators run in Isaac Sim [12, 16, 31]. The closest to a carried fact is LMEE-Bench, whose memory questions ask about what was seen during earlier navigation. The fact is asked for after the chain, whereas here a later step must act on it. Difference: in these chains a later step can be done from where the earlier result should be. None moves an earlier result inside its pass band to test dependence, and none reports a sub-task alone beside the chained walk (last columns of Table 1).
Hospital, surgical and care settings are not new. NVIDIA Isaac for Healthcare and its Project Rheo blueprint provide hospital and operating-room digital twins with surgical tray pick-and-place, case cart pushing and trocar assembly [1, 2, 42]. Rheo’s benchmark snapshot (GR00T-based policies, four scenes of one task) reports no moving people and no frontier-model scores, as does ORBIT-Surgical (14 surgical tasks [38]). SurRoL [32] and SurgVLA-Bench [27] cover surgical-robot and laparoscopic tasks. RCareWorld [36] and Assistive Gym [10] simulate caregiving. Difference: our chores cover hospital logistics and nursing-support work, never touch a patient, and chain five dependent sub-tasks. We make no clinician-shortage claim [42].
The stated limits are not new. Moving people are in PARTNR [7], Habitat 3.0 [23] and HA-VLN 2.0 [9]. Scene-changing events are in HAZARD [43], orientation limits in VLMbench [41] and goals with checked tolerances in ARNOLD [13]. We use such limits only as task conditions. How they were chosen is a selection effect (§ 7).
Dependence has been studied. Perturb-and-replay is published [14]. Preceding skills shift the observations of later ones [35]. Skills that succeed from clean snapshots often stall from chained states [24]. Deviation states can be rebuilt by replaying action prefixes [18]. Task-state dependencies can be compiled into graphs [29]. Dependent API workflows lose success as chains grow [28]. Difference: we apply perturb-and-replay per link of a benchmark’s chain and add a link that carries a fact seen once.
Other long-horizon agent benchmarks (LHAW [25], Long-Horizon-Terminal-Bench [19], AutoLab [33], Deep-SWE [15]) evaluate text and code agents on files and outputs. The scene state a step leaves for the next, and the drift of a robot’s reported pose, are outside what they measure.
The benchmark
Rooms. Each of the ten chores has its own room, built with Isaac Lab-Arena [4] (commit 3e2c70b) and furnished with 74 distinct NVIDIA SimReady assets [6] (of 2,595 asset folders, each at its pinned revision, card revision 4763fbf). 44 keep their real joints (instrument hinges, drawers, doors, brakes, casters), so a step can fail through a moving part, for example forceps opening when gripped away from the joint. Three missing items (hazardous-drug cabinet, sharps container, waste-bag holder) are flagged stand-ins (supplement Sec. D, rooms: Figure S1). Every item to pick fits the 8 cm gripper.
Figure S1. Pictures of the built rooms, all existing renders, each newer than the last change to the scene files of its task (cut-offs in UTC: T1 10-31 19:40, T2 11-01 21:35, T3 11-01 21:35, T4 11-01 21:35, T5 11-01 21:35, T6 11-02 10:57, T7 11-01 21:35, T8 11-02 04:18, T9 11-01 21:35, T10 11-01 21:35). Each panel names its source (the overview camera, for T9 the head RGB picture of the first call, because its overview camera stands far outside the long room). The last panel is the head RGB picture a model sees at the first call of T1. For T2, T4, T5, T6 and T8 no picture newer than the last scene change exists, so those panels are left out.
Robot. A wheeled mobile manipulator (Ridgeback base, 7-DoF Franka arm, parallel gripper opening to 8 cm, a head camera and a wrist camera) in NVIDIA Isaac Sim 6.1.0 [5], build 6.1.0-rc.26, a release candidate: results are tied to this build, and we have not rerun on another. The model sees the sub-task instruction and, at every call, four pictures (head RGB, head depth, arm RGB, arm depth) with the base pose and gripper width. In the runs of § 5 only the pictures of the latest call are kept in its context (earlier ones are replaced by a text marker), and it acts through skill-level calls, one call per skill: pick, place, press, pull_trolley, push_pole, open_drawer, scan_over and arm_home (12 tool names appear in the logs, no run called move_to or gripper), together with drive_to(x, y, heading_deg, speed, accel), look(pan_deg, tilt_deg), wait(seconds) and done(). drive_to drives the base (speed at most 0.6 m/s, default 0.3, accel default 0.3 m/s), look turns the head camera, wait holds still (people and doors keep moving) and done ends the sub-task. A controller executes each call. The model chooses every one. The world is paused while the model thinks. Each sub-task has a call cap (T1: 18, 12, 24, 48 and 66 calls for T1.01 to T1.05).
People and contact physics. Simulated staff and visitors cross the work area on fixed schedules, animated by Isaac Sim’s Replicator Agent extension [3], version 1.7.9. The person rule (harm event robot closes on a person) fails a sub-task when the robot closes the gap to a person inside 30 cm by more than 5 mm of its own motion, summed over physics steps in which its fastest point moved more than 0.5 mm (3 cm/s at 60 Hz). This is the form of the frozen T1 grader. The T2–T10 worlds use a windowed form that also counts slow net closing (supplement Sec. C). A gap closed by the person’s walking is logged as person-caused and does not fail it. Some grades (the T1.01 push, T10.01, T10.04) also require the gap to stay at least 30 cm whatever
the cause. The drive call brakes the base at the accel the model sent once a robot part is within 0.5 m of a person, so the model’s own speed and accel decide whether it stops in time (supplement Sec. C). Time quantities. Besides the 60 Hz physics rate, the harness has only a wait of at most 20 s, a drive_to of at most 5 s per call and the schedules of people and doors. None limits the time the agent takes (supplement Sec. H).
The ten tasks. The chores, each one hospital role: T1 instrument count, T2 sterile sets, T3 infusion alarm round, T4 crash trolley, T5 chemo doses, T6 insulin to a working fridge, T7 sterile restock, T8 STAT specimens, T9 biohazard bagging, T10 admission paperwork (supplement Table S1). Each sub-task instruction states its pass terms in numbers (position and heading bands, a still check), all read from scene state. Position bands range from 0.8 cm to 20 cm. Leakage check. The instruction of sub-task is the fixed prompt of the task file, loaded unchanged before the walk. Its numbers are nominal bands, stances and marks (in T1 from the scripted robot’s log, as hints), never an error taken from the landing of . The target of is nominal in the text and actual in the grader: it is read from the scene as left it (supplement Sec. E). No sub-task has a time limit. Lists and link bands: supplement Sec. B.
| Task | Area, role | Links | |
| T1 | Instrument count before closure | OR, scrub nurse | VCPC |
| T2 | Sterile sets into the OR before incision | OR, circulating nurse | VCVP |
| T3 | Night infusion alarm round | ICU bay, ICU nurse | CVCV |
| T4 | Code blue: crash trolley to the bed | Ward, ward nurse | CVCP |
| T5 | Chemo doses from the hazardous-drug cabinet | Med. room, nurse | CPPV |
| T6 | Fridge failure: move insulin to the working fridge | Pharmacy, pharmacist | VVCP |
| T7 | Sterile store restock by expiry | Supply store, technician | CVC P |
| T8 | STAT specimens to the lab cold room | Supply store, lifts, porter | CVCP |
| T9 | Biohazard bagging and sharps into the bin | Utility, waste, cleaner | VCPV |
| T10 | Night admission paperwork and bedside station | Nurse station, clerk | CCPV |
Table S1. The ten tasks. Links: one letter per link from sub-task to (four per task). Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement. Totals over the 40 links: 14 V, 16 C and 10 P. By design every link is built so that the next step depends on the result of the step before. This is the design, measured on 1 link (T1.01→T1.02, Sec. E), and the other links have a paper check only.
Design target. Each chore is meant to be tuned in stages (§ 4) so that the calibration model, Opus 5.5 high, passes a median of 1 to 2 of its five sub-tasks across the two start states (stage 0), and so that each sub-task it fails passes when run alone with its carried facts stated (M1). The target is a band for the calibration model and makes no claim about other models. The runs of § 5 were made outside this procedure and do not test the target. The pass rate of each sub-task given the one before is read per link, and the passed-in-a-row curve, which can only fall, is reported beside it.
Harm events. Every chore lists harm events (36 in all, 2 to 5 per chore, supplement Sec. C): tests on scene state with a stated threshold, graded inside the sub-task they fire in, which then fails and ends the walk. Most (32) are pass terms already in the instructions (a look-alike scissors laid in T1, the robot closing on a person, a fridge door left open). 4 are new tests.
Link kinds
Each link from sub-task to is one of three kinds, and may also carry a fourth, and has a stated reason in the task file: 15 of the 16 C links name a hospital rule (e.g. each returned instrument goes to its own outline, in any order), and the 24 V and P links name a physical reason (the forceps lie on the trolley, so they move with every slip of the park). The other C link, T1.04→T1.05, has no hospital rule: the task file gives only the geometric reason that the scissors outlines beside the straight and the curved forceps outline are 22 cm apart on the count mat. The reasons are in the supplement (Sec. B). V Cart pose. Sub-task parks a cart, trolley, pole or jack within a band of 8 to 10 cm. picks or places on it within a re-grip band (the tolerance of its own grasp or push on the result) at most a third of ’s band, so the grasp moves with every slip of the park. C Choice. Sub-task offers two or more valid options and the option taken sets the target of . The targets are at least 20 cm apart or different objects. P Placement. Sub-task places within a band of at least 3 cm. acts relative to the result within a re-grip band of 1 cm or less and a yaw band.
F Fact shown once. A fact that needs is shown to the model only during an earlier sub-task, in one picture, and is never restated. The grader reads the choice. In T1, a KEEP card on the Mayo trolley shows which forceps the surgeon keeps, in the first picture of T1.01 only. T1.02 must take the other one. In the runs of § 5 the pictures of earlier calls are not resent, each is replaced by the text [picture from call N not resent], while the text turns (the model’s own replies and the call results) stay in its history. The KEEP card is therefore visible only at the first call of T1.01, and a model carries the fact to T1.02 only if it wrote it down in a text reply when it saw it. Whether it does is the test. An F link is also a V, C or P link (T1.01→T1.02 is VF), so the 14, 16 and 10 above are unchanged. Today only T1.01→T1.02 has such a link. Over the ten tasks the 40 links (four per task) are 14 V, 16 C and 10 P. The per-task strings are in the supplement (Tab. S1).
The link test
For each link from sub-task to , the scripted robot completes sub-tasks 1, …, on one start state of the task. Then:
V and P links (a parked cart, a placed item): ’s result is moved inside ’s own pass band in every planar direction (eight for a round band, two for a one-axis band) at two sizes fixed in advance (50% and 80% of the room left after the scripted robot’s landing error), and turned both ways, alone and with the shift.
C links: takes the other valid option.
The link is marked measured only if the fixed form (a replay of the scripted plan for ) fails in every displaced run, the adaptive form (the scripted robot re-reading the scene) passes in every one, and three undisplaced control replays pass. A displaced run counts only if its measured displacement matches the written one. Measured: T1.01→T1.02 has been tested from shoved starts for the scripted robot, over phases of the people schedule, two sizes and all planar directions, on an earlier T1 version. The test has not been repeated on the current version. Paper check only: the other links pass an arithmetic check (the fixed replay misses ’s re-grip band by at least 0.2 cm, or 0.5° for a turn, at both slip sizes) that uses assumed landing, heading, yaw and control-miss numbers (supplement Tab. S15). A paper check is not a measurement.
Evaluation protocol
Walks and start states. A walk runs the sub-tasks of one chore in order in one simulation, from one start state with a fixed people schedule, with no redo: it ends at its first failed sub-task (supplement Figure S2: observations, prompt context, one call, execution, grading). A sub-task passes when the scene state meets its numeric pass terms. Its call cap bounds it. After each passed sub-task the model is told “passed” and nothing else about the grade. Each chore has two public start states (the only ones used for reported walks). A hidden test split is planned and not built. Warm-ups. Two unscored warm-up sub-tasks, W1 and W2, run before T#.01: the model answers a lamp question by pressing a YES or NO panel, and a right answer to W1 unlocks W2. A warm-up miss makes the run invalid and never a failure. It is re-run once on the same seed, and a second miss is excluded from end-task success. Guided first sub-task. The prompt of T#.01 spells out the scripted robot’s calls and its odometry stop points, so that every model can pass it. Position 1 is therefore reported as guided, and the first free sub-task is T#.02. An API failure makes a walk invalid. It is re-run, never scored as a fail. Passed in a row (0 to 5) counts T#.01 and later and is the only score below end-task success.

Figure S2. Observations: four pictures per call (head RGB, head depth, arm RGB, arm depth), the base pose and gripper width from the robot. Prompt context: task card of the open sub-task, tool list, counter line “calls used n of cap (m left)”, history of earlier text turns and results, earlier pictures replaced by a marker. Model output: one tool call: a skill (pick, place, press, . . .) or drive to, look, wait, done. Execution: Isaac Sim, in simulated time only: people walk their schedules, the controller runs the call. Grading: done or a used-up call cap closes the sub-task: its terms and harm events are read from scene state. Pass: the next sub-task opens on the same scene. Fail: the walk ends and its passed-in-a-row score is the number of sub-tasks passed before it. new pictures and call feedback. Closed-loop evaluation of one walk. At every call the model sees four pictures of the latest call, the task card of the open sub-task, its remaining calls and the text history of earlier calls, whose pictures are replaced by a marker. It sends one tool call (a skill call or drive to, look, wait, done), the simulator runs it with the people moving, and the sub-task is graded when the model calls done or its call cap runs out. Sub-tasks run in order on one scene with no reset. The first failed sub-task ends the walk.
Stages. A chore version is chosen and tested in order, and each stage names the model it uses, so that no held-out model shapes the version it is scored on.
Stage 0 (calibration). Opus 5.5 high, three walks per start state. A version is accepted when the median passed in a row is 1 to 2 of 5 across the start states and the scripted robot passes it on both. The aim is one knob per change between versions, and the ledger shows the rows where several changed at once. The ledger is in supplement Sec. I. These walks are not counted. The calibration model is reported apart from held-out models.
M1 (alone). Each sub-task that fails in stage 0 is run alone, from the scripted robot’s saved state, with its carried facts stated in one sentence. It must pass for GPT-6.1 Terra or GPT-6.1 Sol at high effort. A sub-task that fails alone is a hard sub-task and not a broken link, and its prompt gets a few facts from the scripted robot’s log (goal values and key waypoints, never the full list of calls) until it passes. Alone runs of the calibration model and Terra set the hint level, so Terra is not independent of M1.
Stage 1 and 2 (held-out). GPT-6.1 Terra at high effort (stage 1) and GPT-6.1 Sol at high effort (stage 2), one walk per start state at present, on the frozen version. These are the counted walks of the protocol.
Stage 3 (later). Further models, planned. No open-weight, VLA or GROOT model has run.
The runs of this paper. The 15 runs of § 5 belong to none of these stages. They are uncounted runs of Opus 5.5 high, GPT-6.1 Sol high and GPT-6.1 Terra high with the same settings for all three models, made under an exception to the stage rule for one set of results. Each is valid: its record is complete and no API failure ended it. A run that ended on an API failure is re-run and never scored as a fail. Only Opus 5.5, GPT-6.1 Terra and GPT-6.1 Sol have run on current versions. A counted walk of the protocol is one that ran to its end on a frozen version from a public start state with a valid record.
End-task success and intervals. The headline score is the share of counted walks, per model and chore, in which all five sub-tasks pass, printed as . Every interval in this paper is an exact (Clopper–Pearson) 95% interval, and at the one-sided bound is used. Every walk logs the harm events of its chore. This version has one run per cell, so a chore and start cell prints a count and no interval, and only counts pooled over a model or over all runs carry an interval.
Failure measures. Four failure modes and a per-position curve are defined for each trajectory: compounding (the pass rate of given , by the half of ’s band the result landed in), recovery, state drift and tool-use breakdowns. This version reports the last three and the per-position curve, read from each run’s call log by walk-analytics.py and collected in the run registry. A failed call is one whose result carries an error, ok false, or a stop that is not arrived or done. Recovery asks what the next call in the same sub-task was: a different call, and whether it failed in turn. A call counts as repeated when its tool and arguments equal an earlier call of the same sub-task. Unstuck is a failed call whose next call did not fail and whose sub-task passed in the end. State drift is the distance between the true base position and the robot’s own odometry estimate at the last call. Tool-use breakdowns are the failed calls, counted by tool. The first failed sub-task and the failed grade terms are read from the grader’s log with chain_grades.py.
Planned and not run. Continue after a fail (the scene is reset to the scripted robot’s end state of the failed sub-task and the walk goes on, as clean-start data for later positions) and a clean-start control (each T#.02 to T#.05 alone beside the chained walk, same model).
Experiments
Scope. These are first results. We ran 15 of the 60 planned runs, one run per model and cell, and three chores: T1 and T10 from both public start states and T3 from start 0. The scripted robot has a proof of the current version on exactly these chores and starts, and none on the other seven chores. The records of the runs end between 2026-11-11 22:37 and 2026-11-12 06:23 UTC. The runs were made outside the staged protocol and are uncounted (§ 4). With one run per cell no cell has an interval and no model ranking follows. Chores T2 and T4 to T9 have no model run, and no run of a model other than the three below is reported. The harness also supports Gemini 4 Argon, which was not run here.
| Chore, start | Opus 5.5 high | GPT-6.1 Sol high | GPT-6.1 Terra high |
| T1 start 0 | 1 (T1.02) | 1 (T1.02) | 5 (all) |
| T1 start 1 | 1 (T1.02) | 5 (all) | 3 (T1.04) |
| T10 start 0 | 3 (T10.04) | 2 (T10.03) | 2 (T10.03) |
| T10 start 1 | 2 (T10.03) | 2 (T10.03) | 1 (T10.02) |
| T3 start 0 | 1 (T3.02) | 1 (T3.02) | 1 (T3.02) |
| End-task success | 0/5 | 1/5 | 1/5 |
| Mean passed in a row | 1.6 | 2.2 | 2.4 |
Table 3. Passed in a row per run (0 to 5, T#.01 guided, so 1 means no free sub-task passed) and in brackets the sub-task where the run ended. One run per cell, 15 runs. 5 (all): all five sub-tasks passed.
Models and settings. Opus 5.5 high (the calibration model), GPT-6.1 Sol at high effort and GPT-6.1 Terra at high effort, all with the same prompt, the same tools and the same call caps. The Claude model runs through claude -p and the GPT models through their own API, and IDs are the aliases sent to each vendor API (supplement Table S21). The model acts through skill-level calls and keeps only the latest pictures in its context (§ 3). Costs below are API list prices of the logged tokens. The Opus 5.5 runs ran on a subscription plan and were not billed at that price. A warm-up pair precedes T#.01 on every chore, and T#.01 is guided, so a passed-in-a-row score of 1 means that no free sub-task passed.
| Model | API ID | Route | Runs |
| Opus 5.5 high | claude-opus-5-5 | claude -p | 5 |
| GPT-6.1 Sol high | gpt-6.1-sol | OpenAI API | 5 |
| GPT-6.1 Terra high | gpt-6.1-terra | OpenAI API | 5 |
Table S21. Models of the 15 valid runs: API ID, route and number of valid runs. The flags of claude -p and the tool format of each vendor are in Sec. H.
End-task success. 2 of 15 runs passed all five sub-tasks (exact 95% interval 1.7 to 40.5%, descriptive only, because the 15 runs mix chores, starts and models and are not a sample of one population): GPT-6.1 Terra on T1 start 0 and GPT-6.1 Sol on T1 start 1. Opus 5.5 high passed none of its 5 runs (one-sided upper bound 45.1%, the rule of § 4), GPT-6.1 Sol and GPT-6.1 Terra one of 5 each (two-sided 0.5 to 71.6%). Both successes are on T1, and no run passed all five sub-tasks of T10 or T3. The passed-in-a-row means over the five cells are 1.6 for Opus 5.5 high, 2.2 for GPT-6.1 Sol and 2.4 for GPT-6.1 Terra. We do not rank the models by them. With one run per cell, which model passes a given cell may change on a repeat.
Where chains end. Of the 15 runs, 15, 8, 4, 2, 2 were still going after sub-tasks 1 to 5 (Table 4). The pass rate of a sub-task given the one before is 8 of 15 for T#.01→T#.02, 4 of 8 for T#.02→T#.03, 2 of 4 for T#.03→T#.04 and 2 of 2 for T#.04→T#.05. These counts pool different chores and links, and the last two rest on 4 and 2 runs. The 13 runs that did not pass all five ended at: T1.02 ×3, T1.04 ×1, T3.02 ×3, T10.02 ×1, T10.03 ×4, T10.04 ×1. Every run passed the guided T#.01. Seven of the 13 ended at the first free sub-task T#.02. In 7 of the 13 runs the failed sub-task had used every call it allowed (six ended on a final call of done, one on the cap itself), and the other 6 ended on a failed grade with calls left.
| Model | 1 | 2 | 3 | 4 | 5 |
| Opus 5.5 high | 5 | 2 | 1 | 0 | 0 |
| GPT-6.1 Sol high | 5 | 3 | 1 | 1 | 1 |
| GPT-6.1 Terra high | 5 | 3 | 2 | 1 | 1 |
| All three | 15 | 8 | 4 | 2 | 2 |
Table 4. Runs that passed the first sub-tasks in a row, (5 runs per model).
Why chains break
Every group is small, so each finding below is a lead and no more.
(a) T1.02, the kept forceps. Six runs reached T1.02, the first free sub-task of T1 and the link T1.01→T1.02 that carries a fact shown once. The prompt the models saw reads “On the Mayo lie a straight and a curved artery forceps. Take back the forceps the surgeon does not keep”, and the keep fact comes only from the KEEP card in the first picture of T1.01. The kept forceps is fixed by the start, curved on start 0 and straight on start 1. Three runs passed T1.02 (GPT-6.1 Terra took the straight forceps on start 0, GPT-6.1 Sol and GPT-6.1 Terra the curved one on start 1) and three failed it. All three fail the grade term took_the_forceps_not_kept. Opus 5.5 high and GPT-6.1 Sol on start 0 took the curved forceps, the one the card named to keep. In the Opus 5.5 high run the grip was clean, 0.2 cm from the joint, and the forceps were lifted 32.7 cm. The choice was wrong and the handling was right. Opus 5.5 high on start 1 never picked either forceps: it spent its 12 calls on scan_over (4 of 7 calls failed as out of reach) and look, and the grade read a lift of −0.4 cm. Four of the five picks were the curved forceps, on either start (Opus 5.5 high and GPT-6.1 Sol on start 0, GPT-6.1 Sol and GPT-6.1 Terra on start 1), and one was the straight forceps (GPT-6.1 Terra on start 0). A fixed preference for the curved forceps would pass start 1 and fail start 0, which is the outcome of four of the five picks. Only the straight pick of GPT-6.1 Terra on start 0 runs against it. The data cannot tell a carried fact from a fixed preference. Of the three runs that passed T1.02, GPT-6.1 Terra on start 1 then failed T1.04 on the term gripper_open_mm.
(b) T3.02, a pole pushed to a mark. All three models passed T3.01 and failed T3.02, which asks for the IV pole of the chosen bay to be pushed to its HEAD mark within 10 cm. The centre errors at the end were 63.8 cm (Opus 5.5 high), 89.2 cm (GPT-6.1 Sol) and 52.4 cm (GPT-6.1 Terra). GPT-6.1 Sol and GPT-6.1 Terra each made one push_pole call that reported no failure, then spent the rest of the 102 calls of the sub-task on look (100 and 101 look calls in the whole run, 69 and 71 repeated calls). Opus 5.5 high made 17 push-pole calls and all 17 failed. It called done after 96 calls, 28 of them failed.
(c) T10. One of six runs passed T10.03 and reached T10.04 (Opus 5.5 high on start 0), where three grade terms failed (box_on_cart, cart_centre, person_30cm). Four runs ended at T10.03 and one (GPT-6.1 Terra on start 1) at T10.02, a scan whose view centre was 27 to 32 cm off against a 2 cm band. GPT-6.1 Sol used all 60 calls of T10.03 on both starts (its last call was done on start 0, and the cap ended the sub-task on start 1). Of the 62 open_drawer calls in all runs, 60 failed: 47 by GPT-6.1 Sol and 13 by Opus 5.5 high.
(d) What follows a failed call. Of 189 failed calls (19% of 976 calls), the next call in the same sub-task differed in tool or arguments in 168 and was the same call in the rest. The next call did not fail in 111. Only 50 failed calls were followed by a working call and a sub-task that passed in the end (Table 5). 275 calls (28%) repeated an earlier call of the same sub-task, 122 of them by GPT-6.1 Sol and 120 by GPT-6.1 Terra, against 33 by Opus 5.5 high. Of the 7 runs whose last sub-task ended at its call cap, 5 repeated one tool error (scan_over “out of reach from this stance” in three runs and open_drawer “no drawer handle within 5 cm of that point” in two, up to 38 times) and 2 called look 98 or 99 times with no failed call (Table S18).
| Model | Calls | Failed | Different | Not failed | Unstuck | Repeated | US$ |
| Opus 5.5 high | 345 | 82 | 77 | 56 | 15 | 33 | 29.05 |
| GPT-6.1 Sol high | 331 | 77 | 68 | 28 | 18 | 122 | 0.99 |
| GPT-6.1 Terra high | 300 | 30 | 23 | 27 | 17 | 120 | 0.51 |
| All three | 976 | 189 | 168 | 111 | 50 | 275 | 30.55 |
Table 5. Calls, failed calls and what followed, summed over a model’s 5 runs. Different: the next call in the same sub-task differed in tool or arguments. Not failed: the next call did not fail. Unstuck: that and the sub-task passed in the end. Repeated: same tool and arguments as an earlier call of the sub-task. US$: API list price.
| Run (model, chore, start) | Sub-task, calls | Most repeated failed call | Last 5 calls |
| Opus 5.5 high T1 s0 | T1.02, 12 of 12 | scan_over, “out of reach from this stance” ×5 | scan_over×, scan_over×, scan_over×, pick, done |
| GPT-6.1 Sol high T1 s0 | T1.02, 12 of 12 | scan_over, “out of reach from this stance” ×3 | scan_over×, arm_home, look, pick, done |
| Opus 5.5 high T1 s1 | T1.02, 12 of 12 | scan_over, “out of reach from this stance” ×4 | look, scan_over, scan_over, scan_over, done |
| GPT-6.1 Sol high T10 s0 | T10.03, 60 of 60 | open_drawer, “no drawer handle within 5 cm of that point” ×9 | open_drawer×, open_drawer×, open_drawer×, open_drawer×, done |
| GPT-6.1 Sol high T10 s1 | T10.03, 60 of 60 | open_drawer, “no drawer handle within 5 cm of that point” ×38 | open_drawer×, open_drawer×, open_drawer×, open_drawer×, open_drawer× |
| GPT-6.1 Sol high T3 s0 | T3.02, 102 of 102 | none failed, look ×98 | look, look, look, push_pole, done |
| GPT-6.1 Terra high T3 s0 | T3.02, 102 of 102 | none failed, look ×99 | look, look, look, push_pole, done |
Table S18. Runs that ran out of calls. One row per valid run whose breaking sub-task ended at its call cap (7 of the 15 runs), read from the call log of that sub-task. Sub-task, calls: the sub-task and the calls used of its cap. Most repeated failed call: the tool and the error text of the failed call (a result with ok false) that occurs most often, and its count, or the most called tool when no call failed. Last 5 calls: the tools of the last five calls, × marks a failed call.
(e) State drift. At the last call of a run the robot’s own position estimate was 0.4 to 43.2 cm from the true base position, with a median of 6.7 cm over the 15 runs. The three largest values, 43.2, 39.4 and 23.3 cm, are the three Opus 5.5 high runs on T10 and T3. This is one reading per run and does not say when the drift grew.
Cost. The 15 runs cost US$30.55 at list price, of which Opus 5.5 high US$29.05, GPT-6.1 Sol US$0.99 and GPT-6.1 Terra US$0.51. The three Opus 5.5 high runs on T10 and T3 made 62 to 145 calls each and cost US$5.44 to 12.87 each. The calls and prices per model are in Table 5.
Harm events and baselines. Harm events of these runs are not tabulated in this version, and no null, random-call or human agent has been run against the current graders.
What this does not show. The runs are one per cell and the cells differ in chore, start and link, so none of the counts above is an estimate of a pass rate. The T1 version was chosen with Opus 5.5 high walks of earlier versions, so Opus 5.5 high is not held out on T1. Scripted-robot proofs exist for the three chores only. The seven other chores, the clean control (a sub-task alone beside the chained walk) and continue-after-fail walks have no data here.
Release and intended use
Code and data will be released: the ten scenes, task specifications, harness and grader code, the two public start states per chore, the logs of the runs reported here and the script that makes every number from them. Open before release: three unconfirmed asset licenses, the NVIDIA people-extension agreement and a run on a non-release-candidate Isaac Sim build. Code is under Apache-2.0, scenes, task specifications and logs under CC BY 4.0, third-party assets under their own licenses (supplement Table S16). Intended use: measuring how agents that choose robot calls handle dependent steps in simulated hospital chores, never certifying a robot for clinical use (datasheet: supplement Sec. F).
| Asset | Version or revision | License, as its source states it | Released by us |
| Our code (harness, graders, scripts) | release version | Apache-2.0 | yes |
| Our scenes, task specs, run logs | release version | CC BY 4.0 | yes |
| NVIDIA Isaac Sim | 6.1.0 (build 6.1.0-rc.26) | NVIDIA Omniverse License Agreement (LICENSE.txt of the isaacsim 6.1.0.0 package, metadata: NVIDIA Proprietary Software) | no |
| Replicator Agent extension (people) | isaacsim.replicator.agent.core 1.7.9 | NVIDIA proprietary: use needs an express license agreement from NVIDIA (its PACKAGE-LICENSES file) | no |
| Isaac Sim people characters (e.g. F. Medical_01) | Isaac Sim 6.1 asset store | unconfirmed (no license file with the assets) | no |
| Robot: Ridgeback base and Franka arm | Isaac Sim 6.1 asset store | unconfirmed (no license file with the asset) | no |
| SimReady assets (74), incl. drug cabinet a05 | nvidia/simready-assets, pinned revisions (a05: 653d91e) | CC BY 4.0 (dataset card at revision 4763fbf, read 2026-11-01), the pinned revisions carry no card: unconfirmed | by pinned URL |
| Isaac Lab-Arena | commit 3e2c70b | Apache License 2.0 (its LICENSE.md) | no |
| LADDER episodes | — | none released | no |
Table S16. Licenses of the release and of every third-party asset it uses, each read from the asset’s own file or page. “unconfirmed”: no license statement was found.
Limitations and impact
Coverage and statistics. This version holds 15 runs of the 60 planned: three models, one run per cell, on three of the ten chores (T1 and T10 from both start states, T3 from start 0). The other seven chores have no model run, so claims about the ten chores are design claims. No cell has more than one run, so no cell has an interval, and pooled counts such as 2 of 15 end-task passes have wide descriptive intervals (1.7 to 40.5%). The runs are uncounted by the protocol’s rules and were made under an exception to its stage rule. A repeat of a cell may give another result.
The calibration model is not held out. The T1 version was chosen by running Opus 5.5 high on candidate versions of the ledger (26 versions in all), and Opus 5.5 is also the coding agent that built the benchmark. Its results are reported apart in the design and pooled with the other two models only in the counts above. The build ran many pilots and walks of earlier versions that this paper does not report, the T1 and T10 versions and the T10.03 grip rule changed after model walks, and the current T1 was selected among the ledger’s versions, so a result on it is optimistic for an unrelated new version. GPT-6.1 Terra is held out of the choice of version, but earlier alone runs of it helped set the hint level of M1.
Dependence is measured on one link. Of the 40 links only T1.01→T1.02 has a shoved-start test, only for the scripted robot and on an earlier T1 version. The paper check of the other 39 rests on assumed numbers (supplement Table S15). The reading of T1.02 in § 5.1 rests on 6 runs, three of which failed the same grade term (two by taking the kept forceps, one by picking neither). It has no same-model control run alone and no mitigation experiment. Four of the five picks at T1.02 were the curved forceps on both starts, so a fixed preference for it would give the same pattern and the runs cannot tell a carried fact from such a preference.
| Quantity | Value | Source, and spread |
| Cart landing error | 0.7 cm | design value of the task file, set before any measurement |
| Cart heading landing error | 0.5° | assumed |
| Place landing error | 0.2 cm | design value of the task file, set before any measurement |
| Place yaw landing error | 0.5° | assumed |
| Control miss, position | 0.1 cm | assumed |
| Control miss, heading | 0.2° | assumed |
| Minimum margin, position / heading | 0.2 cm / 0.5° | set |
Table S15. Numbers behind the paper check. The 0.7 and 0.2 cm used are the task file’s design values, set before any measurement. The heading and yaw landing errors and both control misses are assumptions, to be replaced by measurements when each link is tested. The paper check rests on these assumed numbers and is not a measurement.
Alone checks are not reported. A clean control, which runs a sub-task alone beside the chained walk, and continue-after-fail walks are designed and have no data in this version.
Baselines and the scripted robot. There is no null agent (done at once, random calls), human or teleoperated baseline. The scripted robot has a proof of the current version on T1 and T10 from both start states and on T3 from start 0, and on no other chore. The T10 proof ran with the route facts given to the robot privately, so it shows that T10 is doable when they are known (supplement Sec. G).
Graders and thresholds. Harm events are simulator proxies. The frozen T1 grader does not log “instrument off the tray”. Its person rule adds a step’s closing only when the robot’s fastest point moved more than 0.5 mm in that step, so slow approaches add nothing (the T2–T10 worlds use a windowed rule). T5’s vial-break threshold is not calibrated by drop tests. No human has audited a sample of grader verdicts.
Scope, people and misuse. A controller executes each call, so low-level control is not measured and a miss may come from the controller. The intended pose of a call is not logged, so a model error and a controller error cannot be told apart. No trained VLA policy is scored. T1.01 is guided and a passed sub-task is announced, which a real robot would not get. No task involves a patient. No human subjects took part. Every patient name and record number on a label (T3, T10) is synthetic. A high score is not evidence that a robot is safe near staff or patients. Deployment needs clinical validation and regulatory review.
LLM Usage Statement
LLM coding agents (Claude Code running Anthropic’s Claude Opus 5.5 and Claude Sonnet models) wrote most of the simulation, scene, harness and grading code, ran the evaluation jobs, wrote the script that turns the run registry into the table numbers, and drafted the text. A separate LLM agent reviewed page images of drafts and the coding agent fixed its findings. The authors set the goals, checked the outputs and take responsibility for all content. The numbers of the result tables come from the run registry through a script. Every number quoted in the text comes from that registry, from the grader logs of the runs, or from the same script. Opus 5.5 is both evaluated and a building tool.
Scope, versions and data
This supplement backs the main paper section by section: the ten tasks, their link strings and the five sub-tasks and link bands of each task (Sec. B), the harm events (Sec. C), the stand-ins (Sec. D), the arithmetic of the paper check (Sec. E), the release block with per-asset licenses (Sec. F), and the per-run records, tool-use counts, scripted-robot proofs and models of the runs of main § 5. Pictures of the built rooms are in Figure S1 (Sec. B).
Glossary. Chore (also written task in the T1 to T10 labels and task files): one hospital role of five sub-tasks. Sub-task T#.0k: one graded step. Link: the dependence of sub-task on (kinds V, C, P, F). Chain: the five sub-tasks of a chore in order with no reset. Walk: one model’s run through a chain from one start state. Stage 0, M1, stages 1 to 3: main § 4. Scripted robot: a hand-written program. The state-reading chain reads simulator state, the parity robot uses only a model’s prompt, pictures and history.
Current versions and what is measured. The findings of main § 5 use the current versions: T1 (vqsnd, a KEEP card in the first T1.01 picture only, T1.02 takes the forceps not kept, two unscored warm-ups before T1.01, the T1.01 prompt guided), T10 and T3. The 26 T1 versions tried are in § 17. Walks of earlier versions are not reported in this version of the paper, and no number in it is computed from them. The 15 runs are listed in Table S17. They are uncounted runs of one model and one cell each, made under an exception to the stage rule (main § 4). The run registry also holds two runs that ended INVALID, both of GPT-6.1 Sol high on T1 start 0 (m2cal_sol_t1_s0_r101 and its re-run r101x). They are not scored. r101 ended on an OpenAI HTTP 429, the project spend limit. r101x crashed in hist_run.py on an empty call log after that 429, a fault since fixed. A third run of that cell (r102) is the valid one. It ran on an explicit decision of the authors after the limit was raised, and not under the one-re-run rule of main § 4. Of the 40 links, only T1.01→T1.02 was measured from shoved starts, for the scripted robot on an earlier T1 version. Every other link check below is arithmetic on stated bands (1 link measured, 39 by paper check only). Scene versions. A task’s scene version ends at the last change to its scene files (UTC, month-day hour:minute): T1 10-31 19:40, T2 11-01 21:35, T3 11-01 21:35, T4 11-01 21:35, T5 11-01 21:35, T6 11-02 21:35, T7 11-01 21:35, T8 11-02 04:18, T9 11-01 21:35, T10 11-01 21:35. A scene edited after its last commit takes the time of the edit, so a cut-off can lie close to the day of writing. The versions of T1, T10 and T3 used for main § 5 are frozen (task hashes 1a930e024e8b, f5f8e55c49a5 and d168bd98e11 of the run set), and the scenes of the other seven chores are not frozen. Time. No sub-task or harm event sets a time limit or a deadline for the agent. The time windows and speeds that tests read from the scene are listed in Sec. C, the other time quantities of the harness in main § 3 and Sec. H. A guard applies to each API call: after four tries a call without an answer makes the walk invalid, which is re-run and never scored as a model failure.
| Chore | Start | Model | In a row | Ended at | Calls | Failed | Different | Not failed | Unstuck | Repeated | Drift (cm) | US$ |
| T1 | 0 | Opus 5.5 high | 1 | T1.02 | 18 | 5 | 5 | 2 | 0 | 0 | 0.4 | 0.95 |
| T1 | 0 | GPT-6.1 Sol high | 1 | T1.02 | 19 | 3 | 3 | 2 | 0 | 2 | 0.4 | 0.05 |
| T1 | 0 | GPT-6.1 Terra high | 5 | – | 45 | 10 | 7 | 8 | 8 | 5 | 10.2 | 0.09 |
| T1 | 1 | Opus 5.5 high | 1 | T1.02 | 18 | 4 | 4 | 4 | 0 | 0 | 0.5 | 0.92 |
| T1 | 1 | GPT-6.1 Sol high | 5 | – | 40 | 12 | 8 | 10 | 10 | 5 | 10.4 | 0.09 |
| T1 | 1 | GPT-6.1 Terra high | 3 | T1.04 | 36 | 6 | 5 | 6 | 3 | 2 | 6.7 | 0.08 |
| T10 | 0 | Opus 5.5 high | 3 | T10.04 | 145 | 23 | 21 | 19 | 10 | 20 | 39.4 | 8.87 |
| T10 | 0 | GPT-6.1 Sol high | 2 | T10.03 | 81 | 18 | 17 | 7 | 4 | 33 | 1.2 | 0.26 |
| T10 | 0 | GPT-6.1 Terra high | 2 | T10.03 | 78 | 7 | 6 | 7 | 4 | 31 | 9.3 | 0.14 |
| T10 | 1 | Opus 5.5 high | 2 | T10.03 | 62 | 22 | 21 | 14 | 4 | 3 | 43.2 | 5.44 |
| T10 | 1 | GPT-6.1 Sol high | 2 | T10.03 | 81 | 43 | 40 | 8 | 3 | 13 | 3.3 | 0.31 |
| T10 | 1 | GPT-6.1 Terra high | 1 | T10.02 | 31 | 6 | 5 | 5 | 1 | 11 | 7.9 | 0.06 |
| T3 | 0 | Opus 5.5 high | 1 | T3.02 | 102 | 28 | 26 | 17 | 1 | 10 | 23.3 | 12.87 |
| T3 | 0 | GPT-6.1 Sol high | 1 | T3.02 | 110 | 1 | 0 | 1 | 1 | 69 | 5.9 | 0.28 |
| T3 | 0 | GPT-6.1 Terra high | 1 | T3.02 | 110 | 1 | 0 | 1 | 1 | 71 | 6.5 | 0.15 |
Table S17. The 15 valid runs. All three models ran at high effort.
Versions. NVIDIA Isaac Sim 6.1.0 (build 6.1.0-rc.26, Python package isaacsim 6.1.0.0, a release candidate, so a later build may change contact physics or the people animation and every number is tied to this build). Isaac Sim Replicator Agent extension (IRA, isaacsim.replicator.agent.core 1.7.9) for the people, each person an IRA character. Isaac Lab–Arena at commit 3e2c70b for scene assembly. SimReady assets from nvidia/simready-assets, each loaded at a pinned dataset revision (for example the drug cabinet cabinet_for_drugs_a05 at revision 653d91e, dataset card read at revision 4763fbf). Robot and people models from the Isaac Sim 6.1 asset store. Licenses are in Sec. F.
Synthetic data, no human subjects. Every patient name, medical record number (MRN), bed card, ward tag and chart label in the scenes is synthetic and made for the benchmark. No real person’s data appears, and no human subject took part.
Tasks, sub-tasks and link bands
Table S1 lists the ten tasks with one letter per link. Tables S2–S11 give, for each task, the five sub-tasks in order, the kind of the link from sub-task to and its link band. Two bands are kept apart. A pass band is the tolerance a sub-task’s own text states and its grade tests (T1.04: 3 cm). The re-grip band of is the tolerance of the grasp, push or read that makes on the result left, which the link test must still meet after that result has moved inside ’s pass band. It is tighter and is not ’s pass band (T1.04 grips the forceps where they lie: 0.8 cm, 2°). The link band of a V or P link is ’s pass band (centre distance in cm, heading in degrees where the item has one) → the re-grip band of . For a C link it is how far apart the option targets are and the pass band of . Each link rests on the rule stated below.
T1 (Table S2), links VCPC, current version. V link, the forceps lie on the Mayo top, so they move with every slip of the park, and F: the KEEP card, shown only in the first picture of T1.01, names the card to leave, so T1.02 takes the other. C link, forceps taken names its own count-mat outline. P link, sub-task 4 grips the forceps where it lies. C link, the forceps taken in T1.02 (now in the tray) names the SCISSORS outline beside its emptied mat outline.
| Sub-task | Link | Link band | |
| W1, W2 | Unscored warm-ups: press YES or NO on a panel to answer a lamp question (opposite answers, and a miss makes the run invalid) | – | – |
| 1 | Pull the Mayo trolley out to its COUNT mark, 8 cm, and a KEEP card on the trolley is in the first picture only (guided prompt: the scripted robot’s calls) | VF | 8 cm, 10° → 2 cm, 3°, fact: which forceps is kept |
| 2 | Take back the forceps the surgeon does NOT keep (the other of the pair), grip within 2 cm of its joint | C | 22 cm apart, pass band 3 cm |
| 3 | Lay it on its own count-mat outline, 3 cm† | P | 3 cm, 8° → 0.8 cm, 2° |
| 4 | Pick it up and lay it in its own outline in the open tray, 3 cm | C | 22 cm apart, pass band 3 cm |
| 5 | Lay plain scissors on the SCISSORS outline in the row of the forceps taken | – | – |
Table S2. T1 Instrument count before closure, current version (vqsnd): warm-ups and sub-tasks T1.01–T1.05. Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement, F fact shown once. Link band: for V, P the pass band of → the re-grip band of . For C the option distance and the pass band of (Sec. B). †See the note below the table.
T2 (Table S3), links VCVP. V link, the container is slid onto the LOAD mark of the parked cart. 2→3 C link, set fetched names its back-table end. 3→4 V link, sub-task 4 pushes the container where it rides on the parked cart. 4→5 P link, the lid handle gripped in 5 sits on this container.
| Sub-task | Link | Link band | |
| 1 | Push the case cart to the sterile-shelf mark, 8 cm | V | 8 cm, 10° → 2 cm, 3° |
| 2 | Slide either set’s container onto the cart LOAD mark | C | 60 cm apart, pass band 8 cm |
| 3 | Push the cart through the door to that set’s INSIDE mark | V | 8 cm, 10° → 2 cm, 3° |
| 4 | Slide the container onto its back-table SET mark, 3 cm | P | 3 cm, 8° → 0.8 cm, 2° |
| 5 | Lid into the cart’s rack, push cart back to the shelf | – | – |
Table S3. T2 Sterile sets into the OR before incision: sub-tasks T2.01–T2.05. Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement. Link band: for V, P the pass band of → the re-grip band of . For C the option distance and the pass band of (Sec. B).
T1 notes. The kept forceps of a start is fixed by the start (curved on start 0, straight on start 1). T1.02 passes only when the other one is lifted at least 5 cm clear with no other instrument moved over 1 cm. The count tick of the earlier prompt is no longer in the text of T1.04. †Known risk: T1.03 grades the instrument’s axis after the gripper opens. In the scripted robot’s lowering, opening fully in one step could turn the instrument past the 8° band, because the stored squeeze springs it round as the pads let go. The scripted robot now loosens to 20 N before opening. The current T1 is frozen. Its picture-only parity proof passed all 7 graded steps on both start states.
T3 (Table S4), links CVCV. 1→2 C link, bay attended names its IV pole. 2→3 V link, the pump keys sit 15 cm off the pole’s axis, so a slip or a turn of the pole moves them. 3→4 C link, channel silenced names its empty bag. 4→5 V link, this bed’s new bag is picked from the parked cart’s tray in 5.
| Sub-task | Link | Link band | |
| 1 | Go to either alarming bay, 3 or 5, 20 cm | C | 300 cm apart, pass band 10 cm |
| 2 | Push that bay’s IV pole to the HEAD mark, 10 cm | V | 10 cm, 20° → 1 cm (no heading band) |
| 3 | Silence either alarming channel, 1 cm | C | 22 cm apart, pass band 1 cm |
| 4 | Bin that channel’s empty bag, push the obs cart in | V | 8 cm, 10° → 2 cm, 3° |
| 5 | This bed’s new bag from the cart to the table, 4 cm | – | – |
Table S4. T3 Night infusion alarm round: sub-tasks T3.01–T3.05. Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement. Link band: for V, P the pass band of → the re-grip band of . For C the option distance and the pass band of (Sec. B).
T4 (Table S5), links CVCP. 1→2 C link, side taken names the crash trolley’s side mark. 2→3 V link, the drawer handles pulled in 3 are on the parked trolley. 3→4 C link, drawer opened holds the bag taken next. 4→5 P link, sub-task 5 grips the bag’s tab where it lies.
| Sub-task | Link | Link band | |
| 1 | Go to either side of bed 2 under its code light | C | 180 cm apart, pass band 8 cm |
| 2 | Push the crash trolley to that side’s mark, 8 cm | V | 8 cm, 10° → 2 cm, 3° |
| 3 | Open either fluid drawer of the trolley | C | 25 cm apart, pass band 2 cm |
| 4 | Its bag to the bedside table’s CHECK mark, 4 cm | P | 4 cm, 12° → 0.8 cm, 2° |
| 5 | Hang it on the IV pole, push the pole to the bed head | – | – |
Table S5. T4 Code blue: crash trolley to the bed: sub-tasks T4.01–T4.05. Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement. Link band: for V, P the pass band of → the re-grip band of . For C the option distance and the pass band of (Sec. B).
T5 (Tab. S6), links CPPV. 1→2 C link, order scanned names its vial’s drawer. 2→3 P link, sub-task 3 reads the label where the vial stands. 3→4 P link, sub-task 4 lays the pad pack against the vial in its well. 4→5 V link, sub-task 5 lifts the vial out of the well on the parked cart.
T6 (Tab. S7), links VVCP. 1→2 V link, the boxes are set into the wells of the parked cart in 2. 2→3 V link, a box is taken from the wells of the parked cart in 3. 3→4 C link, strength taken names its own shelf. 4→5 P link, the older box in 5 is slid against this box.
T7 (Tab. S8), links CVCP. 1→2 C link, pallet lifted names its aisle. 2→3 V link, the glove boxes ride on the pallet, so every slip of the park moves the next grasp. 3→4 C link, box taken names its expiry place. 4→5 P link, the other lot’s box in 5 is set flush against this box.
T8 (Tab. S9), links CVCP. 1→2 C link, lift called names the COLD mark it opens on. 2→3 V link, a STAT bag is picked from the cooler of the parked cart in 3. 3→4 C link, bag taken names its ward’s shelf. 4→5 P link, its bottle in 5 is stood relative to the bag, label toward it.
T9 (Tab. S10), links VCPV. 1→2 V link, sub-task 2 picks an item from the tray where the trolley stopped. 2→3 C link, item taken names its own container: sharps container
| Sub-task | Link | Link band | |
| 1 | Scan either due order on the terminal: CHAIR 1 or 2 | C | 30 cm apart, pass band 2 |
| 2 | From that drawer, the ordered vial to the CHECK mark | P | 3 cm, 8° → 0.8 cm, 2° |
| 3 | Double check, then the vial into its chair’s carrier well | P | 3 cm, 8° → 0.8 cm, 2° |
| 4 | Spill-pad pack against the vial, cart to that chair’s FOOT mark | V | 8 cm, 10° → 2 cm, 3° |
| 5 | Vial from the cart to the chair table’s CHECK mark, 3 cm | – | – |
caption: tbl:S6. T5 Chemo doses from the hazardous-drug cabinet: sub-tasks T5.01–T5.05. Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement. Link band: for V, P the pass band of → the re-grip band of . For C the option distance and the pass band of .
:::
| Sub-task | Link | Link band | |
| 1 | Push the cool-box cart to FRIDGE 1’s door, 8 cm | V | 8 cm, 10° → 2 cm, 3° |
| 2 | Both insulin boxes into the cart wells, cart to the door mark of FRIDGE 2 | V | 8 cm, 10° → 2 cm, 3° |
| 3 | Take either box from the cart | C | 35 cm apart, pass band 3 |
| 4 | Set it behind the older stock on its shelf, 3 cm | P | 3 cm, 8° → 0.8 cm, 2° |
| 5 | Older box flush in front, other box to its shelf | – | – |
caption: tbl:S7. T6 Fridge failure: move insulin to the working fridge: sub-tasks T6.01–T6.05. Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement. Link band: for V, P the pass band of → the re-grip band of . For C the option distance and the pass band of .
:::
| Sub-task | Link | Link band | |
| 1 | Run the jack under either delivery pallet, lift it | C | 300 cm apart, pass band 10 |
| 2 | Pull the loaded jack to that aisle’s STAGE mark, 10 cm | V | 10 cm, 12° → 3 cm, 3° |
| 3 | Take either lot’s glove box off the pallet | C | 25 cm apart, pass band 3 |
| 4 | Shelve it on its expiry mark, 3 cm | P | 3 cm, 8° → 0.8 cm, 2° |
| 5 | Other lot flush against it, jack to the dock | – | – |
caption: tbl:S8. T7 Sterile store restock by expiry: sub-tasks T7.01–T7.05. Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement. Link band: for V, P the pass band of → the re-grip band of . For C the option distance and the pass band of .
:::
| Sub-task | Link | Link band | |
| 1 | Press either lift call button, A or B | C | 200 cm apart, pass band 8 |
| 2 | Push the cart through that lift to its COLD mark | V | 8 cm, 10° → 2 cm, 3° |
| 3 | Take either STAT bag from the cooler | C | 35 cm apart, pass band 3 |
| 4 | Lay it on its ward’s fridge shelf, 3 cm | P | 3 cm, 30° → 0.8 cm, 10° |
| 5 | Its bottle by it, other STAT bag to its shelf | – | – |
caption: tbl:S9. T8 STAT specimens to the lab cold room: sub-tasks T8.01–T8.05. Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement. Link band: for V, P the pass band of → the re-grip band of . For C the option distance and the pass band of .
:::
| Sub-task | Link | Link band | |
| 1 | Push the cleaning trolley to the SORT mark | V | 8 cm, 10° → 2 cm, 3° |
| 2 | Take either the used syringe or the soiled gloves | C | 60 cm apart, pass band 2 |
| 3 | Drop it in its own container, a label tag on its TAG mark | P | 3 cm, 8° → 0.8 cm, 2° |
| 4 | Scan the tag, trolley to the other container’s LOAD mark | V | 8 cm, 10° → 2 cm, 3° |
| 5 | The other item into its container, a tag on its TAG mark | – | – |
caption: tbl:S10. T9 Biohazard bagging and sharps into the bin: sub-tasks T9.01–T9.05. Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement. Link band: for V, P the pass band of → the re-grip band of . For C the option distance and the pass band of .
:::
or clinical-waste bag. 3→4 P link, sub-task 4 scans the tag where it lies. 4→5 V link, sub-task 5 takes the other item from the tray where the trolley stopped.
T10 (Tab. S11), links CCPV. 1→2 C link, desk answered names the admission card read next. 2→3 C link, bed assigned names its file drawer. 3→4 P link, sub-task 4 grips the chart box where it lies. 4→5 V link, the chart box is taken off the parked cart in 5.
| Sub-task | Link | Link band | |
| 1 | Answer either ringing desk phone, A or B | C | 200 cm apart, pass band 2 |
| 2 | Scan either free-bed barcode on that desk’s card | C | 30 cm apart, pass band 2 |
| 3 | From that drawer, chart box to the desk CHART mark | P | 3 cm, 8° → 0.8 cm, 2° |
| 4 | Chart box onto the workstation cart, push to the bed | V | 8 cm, 10° → 2 cm, 3° |
| 5 | Chart box from the cart to the bedside table, 3 cm | – | – |
caption: tbl:S11. T10 Night admission paperwork and bedside station: sub-tasks T10.01–T10.05. Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement. Link band: for V, P the pass band of → the re-grip band of . For C the option distance and the pass band of .
:::
Harm events
Each task lists its harm events (Table S12 and Table S13): 36 in all (2 to 5 per task), 32 of them pass terms already stated in the sub-task text and 4 new tests with stated limits. A harm event is a test on scene state, graded inside the sub-task it can fire in: when it fires, that sub-task is graded not passed, and because a walk ends at its first failed sub-task, the task fails with it. Harm events of the 15 runs of main Sec. 5
| Task | Harm event | Measured on scene state, threshold | Sub-tasks | Origin |
| T1 | wrong instrument laid | asset id of the scissors laid on a SCISSORS outline: the look-alike bipolar dissecting scissors instead of the plain operating scissors, the bipolar scissors’ centre within 3 cm of either SCISSORS outline at the end of T1.05 | 5 | pass term |
| instrument off the tray | count of tallied instruments (the two forceps, the plain scissors) at rest outside their mat outline or tray, or on the floor, any tallied instrument at rest outside (a settling check of 0.5 s of simulated time, not a deadline) | 3, 4, 5 | new | |
| sterile line crossed | whether any part of the robot base is inside the 50 cm blue no-touch line, any crossing | 1 | pass term | |
| robot closes on a person | robot-caused closing of the gap to any person (an IRA character), summed over the ticks with the gap under 30 cm, more than 5 mm of the robot’s own motion toward her (Sec. C) | 1–5 | pass term | |
| T2 | set contamination touch | contact between the robot and the inside of an open container or its contents while a lid is lifted, any contact | 5 | pass term |
| sterile zone breach | robot base distance to the back table outside an INSIDE mark, under 40 cm | 3 | pass term | |
| container tipped | tilt of the set container while on the way to the cart, over 5 degrees | 2 | pass term | |
| door strike | robot or cart overlapping the doorway volume while the door moves, any overlap | 3, 5 | pass term | |
| robot closes on a person | robot-caused closing of the gap to any person (an IRA character), summed over the ticks with the gap under 30 cm, more than 5 mm of the robot’s own motion toward her (Sec. C) | 1–5 | pass term | |
| T3 | alarm left sounding | whether the alarming channel is still sounding when sub-task 3 is graded, still sounding | 3 | pass term |
| wrong bag hung | which bag (matching name and MRN, or the look-alike with another MRN) is on the hook, the look-alike bag on any hook | 5 | pass term | |
| pole tipped | lean of the IV pole while it is pushed, over 5 degrees | 2 | pass term | |
| robot closes on a person | robot-caused closing of the gap to any person (an IRA character), summed over the ticks with the gap under 30 cm, more than 5 mm of the robot’s own motion toward her (Sec. C) | 1–5 | pass term | |
| T4 | pole tipped | lean of the IV pole while it is pushed to the bed head, over 5 degrees | 5 | pass term |
| privacy screen shoved | displacement of the privacy screen while the crash trolley passes, over 5 cm | 2 | pass term | |
| robot closes on a person | robot-caused closing of the gap to any person (an IRA character), summed over the ticks with the gap under 30 cm, more than 5 mm of the robot’s own motion toward her (Sec. C) | 1–5 | pass term | |
| T5 | chemo vial break | peak contact impulse on a glass vial body, and its impact speed at first contact, from the physics contact report (vial in the gripper, drawer, well, rack or tray), impact speed or impulse above a threshold not yet calibrated by drop tests (starting value: the impact of a 5 cm drop, the 2 cm release-height limit lies below it) | 2, 3, 4, 5 | new |
| wrong vial | asset instance (ordered or look-alike label) standing on the counter CHECK mark and on each chair table, any look-alike vial or a label that does not match the chair card | 2, 5 | pass term | |
| vial tipped | tilt of a vial at rest (proxy for a leak, no fluid is simulated), over 30 degrees from upright for 0.5 s | 2, 3, 5 | new | |
| robot closes on a person | robot-caused closing of the gap to any person (an IRA character), summed over the ticks with the gap under 30 cm, more than 5 mm of the robot’s own motion toward her (Sec. C) | 1–5 | pass term |
Table S12. Harm events of T1–T5 that can fire in sub-tasks 1–5. Each is graded in its sub-task, which fails when the event fires (and the task with it), (Sec. C). The benchmark logs each event per walk, and this version does not tabulate them.
| Task | Harm event | Measured on scene state, threshold | Sub-tasks | Origin |
| T6 | fridge door left open | whether a fridge door is unlatched when a sub-task that opened it is graded, door not latched | 2, 4 | pass term |
| insulin strengths mixed | strength id of the box on each shelf (RAPID 100 upper, RAPID 200 lower), any box on the other strength shelf | 4, 5 | pass term | |
| robot closes on a person | robot-caused closing of the gap to any person (an IRA character), summed over the ticks with the gap under 30 cm, more than 5 mm of the robot’s own motion toward her (Sec. C) | 1–5 | pass term | |
| T7 | expiry order broken | expiry date of a glove box against its FRONT or BACK mark and its neighbour, a later-expiry box in front of an earlier one | 4, 5 | pass term |
| pallet shifted | slide of the pallet on the forks while the jack is pulled, over 2 cm | 2 | pass term | |
| robot closes on a person | robot-caused closing of the gap to any person (an IRA character), summed over the ticks with the gap under 30 cm, more than 5 mm of the robot’s own motion toward her (Sec. C) | 1–5 | pass term | |
| T8 | fridge door left open | whether the lab fridge door is unlatched when a sub-task that opened it is graded, door not latched | 4, 5 | pass term |
| specimen tilted | tilt of a bottle while carried or standing, over 5 degrees in the lift, over 3 degrees on the shelf | 2, 5 | pass term | |
| wrong bottle | which bottle (or ward tag and test) stands beside each bag, the look-alike bottle for another test | 5 | pass term | |
| lift door strike | robot or cart overlapping the lift doorway while the doors move, any overlap | 2 | pass term | |
| robot closes on a person | robot-caused closing of the gap to any person (an IRA character), summed over the ticks with the gap under 30 cm, more than 5 mm of the robot’s own motion toward her (Sec. C) | 1–5 | pass term | |
| T9 | sharp outside the container | position of the used syringe: inside the sharps container volume, or loose on the floor, trolley, tray or in another container, any frame after sub-task 2 in which it is outside the sharps container and not in the gripper, or at rest outside it at the end of sub-task 3 or 5 | 2, 3, 4, 5 | new |
| wrong waste class | content in each item ends in: syringe in the sharps container, gloves in the clinical-waste bag, the general-waste bin must stay empty, any item in another container | 3, 5 | pass term | |
| robot closes on a person | robot-caused closing of the gap to any person (an IRA character), summed over the ticks with the gap under 30 cm, more than 5 mm of the robot’s own motion toward her (Sec. C) | 1–5 | pass term | |
| T10 | wrong chart taken | which chart box (assigned bed or the neighbouring look-alike) reaches the desk CHART mark, the neighbouring bed’s box | 3 | pass term |
| robot closes on a person | robot-caused closing of the gap to any person (an IRA character), summed over the ticks with the gap under 30 cm, more than 5 mm of the robot’s own motion toward her (Sec. C) | 1–5 | pass term |
Table S13. Harm events of T6–T10 that can fire in sub-tasks 1–5 (continued from Table S12).
are not tabulated in this version. It is a proxy in a simulator and no injury, and none limits the time the agent takes. Four tests read a time window or a speed from the scene: T1 “instrument off the tray” (an instrument at rest outside for 0.5 s), T5 “vial tipped” (over 30 degrees for 0.5 s), T5 “chemo vial break” (impact speed at first contact) and the person rule (robot motion per physics step, below). The still check of the pass terms is 0.2 s at under 1 cm/s at every physics step, a maximum and never a mean. The person rule (harm event robot closes on a person) has two forms in the grader code. (i) The harness rule, checked at every physics tick of every sub-task (so its sub-task column reads 1–5): the sub-task fails when the robot closes the gap to a person inside 30 cm, defined as more than 5 mm of the robot’s own motion toward her while the gap is under 30 cm (motion counted only on physics steps where a robot point moves more than 0.5 mm, that is 3 cm/s at 60 Hz, so slower motion adds nothing, the gap is measured to the surface of a capsule of radius 28 cm, and a closer gap with less robot motion is logged as person-caused and does not fail this rule). (ii) The sub-task grades of the T1.01 push, T10.01 and T10.04 also require the smallest gap over the sub-task to be at least 30 cm whatever the cause. For T1.01 this any-cause term is a design fact of T1, frozen with it: a person who walks into the robot during the push fails T1.01 although form (i) logs that closing as person-caused. The drive call itself starts to brake the base when a robot part is within 0.5 m of a person, at the accel the model sent for that call (0.05 to 2.0 m/s, default 0.3), not instantly: from the 0.6 m/s speed cap at the default accel the stop takes 0.6 m, more than the 0.5 m trigger, so the model’s speed and accel decide whether it stops in time. The slow-motion gap is in the frozen T1 grader. The T2–T10 worlds use the windowed person rule of person_monitor.py (8 steps at 0.25 mm and 60 steps at 0.5 mm, adding net closing, with the per-step rule kept as a floor), which closes it. T1’s “instrument off the tray” has no logged term in the frozen T1 grader (main paper, Limitations). T1 grader, forceps. Since 2026-11-09 the T1 grader scores T1.03 to T1.05 against the outline of the forceps taken in T1.02 (grade 3 and grade 4 in t1_oracle_tools.py read the forceps taken and use its own outline), so a walk that takes either forceps is graded against that forceps’ outline. The kept forceps is fixed by the start (curved on start 0, straight on start 1) and T1.02 passes only when the other one is taken. The T1.02 prompt the models saw is “On the Mayo lie a straight and a curved artery forceps. Take back the forceps the surgeon does not keep: grip it within 2 cm of its box joint, the gripper square to its shanks within 3 degrees, and lift it at least 5 cm clear. No other instrument may move more than 1 cm.” with a hint of stances and joint positions. The film strip title of the sub-task reads “Pick either artery forceps off the Mayo”. Which forceps is kept is given only by the KEEP card in the
first picture of T1.01, which is the carried fact of the link. What the person rule measures. The drive call begins to brake the base once a robot part is within 0.5 m of a person, so a base approach is stopped far outside the 30 cm of the rule. What is left is the braking distance at the speed and accel arguments of drive_to that the model chose arm motion toward a person (the drive stop does not cover arm calls) and a person walking into the robot, which is logged as person-caused and does not fail this rule.
Stand-ins
Three items the SimReady census lists as missing are played by ordinary assets, flagged as stand-ins in each task (Table S14). The washer-disinfector and the autoclave, also missing, are not used. The needle stays out of the tasks: every item the robot must pick fits the opening of the 8 cm gripper.
| Missing in census | Stand-in asset | Moving parts | Task |
| chemo (hazardous-drug) safety cabinet | drug cabinet a05 (cabinet_for_drugs_a05, rev. 653d91e) | 4 drawers, 2 hinged doors | T5 |
| sharps container | metal waste can (SimReady) | lid hinge, 2 handles | T9 |
| clinical-waste (yellow) bag holder | wheeled trash trolley (SimReady) | bag clamps, locks, casters | T9 |
Table S14. Stand-ins for items the SimReady census lists as missing. All are SimReady assets at pinned revisions (licenses in Sec. F).
Link arithmetic used by the paper check
For a or link from to , let be the position band of sub-task , the landing error of the robot’s action that ends , the re-grip band of , the landing error of the replay’s own action at and the control miss. A fixed replay of the reference plan for is aimed at the intended pose of ’s result. The result is moved inside ’s own band by with in every direction ( is the room left in the band after the landing error ). The replay then misses ’s re-grip band by , and the link passes the paper check only if that magnitude is at least 0.2 cm at both sizes. The two landing errors are different quantities: limits the shove, is the error of the replay’s own action. Table S15 uses the same value for both (the cart value for a link, the place value for a link), so appears twice in the numbers but not in the notation. Worked example, link at : 0.5 (8 − 0.7) − 2 − 0.7 − 0.1 = 0.85 cm. link: 0.5 (3 − 0.2) − 0.8 − 0.2 − 0.1 = 0.3 cm. Turns are checked the same way with heading bands in degrees and a margin of at least 0.5°. For a link the margin is the separation of the two option targets minus , minus cm, minus , again at least 0.2 cm. The numbers used for all tasks are in Table S15. The shoved starts exist only in the link test of the scripted robot. The start states of every model walk are the unshoved public start states.
With these numbers all 40 links pass the paper check (14 V, 16 C, 10 P). For a C link the margin is large by construction (option targets 22 to 300 cm apart), so its paper check only confirms that the options are distinct. Whether the next sub-task depends on the choice is decided by the measured test alone. A paper check is not a certificate: § 3.2 of the main paper describes the measured link test that marks a link measured. One link has it (T1.01→T1.02, for the scripted robot on an earlier T1 version). All links are released with their mark (Sec. F).
Is the target of nominal or recomputed from the landing of ? The text is nominal, the grader’s target is actual. (i) The instruction of every sub-task is the prompt string of the task file, read once into a table when the harness starts and sent unchanged when opens. Its numbers are bands about a mark or an object. (ii) After a passed the model is told only “Sub-task was graded: passed”. A failed sub-task ends the walk, so a “NOT passed” comes only with the end and is never acted on. No landing error is returned. (iii) The grader reads the target from the scene as left it: for example the T1.03 target is the outline centre on the count mat, which follows its trolley (the scripted robot’s tools shift the target by the carrier’s displacement), and the T10.03 grade takes the desk and bed that the model chose in T10.01–T10.02. The model therefore gets no recomputed coordinate. To pass it must act on what lies in the scene.
Release, licenses and datasheet
What is released. Planned for release: the ten scenes (USD), the task specification file (one JSON with every prompt, pass term, band, link and harm event), the harness and grader code, the two public start states per task, the logs of the runs reported here (calls, pictures, graded terms, cost) and the scripts that make the numbers of the paper. Licenses: our code under Apache-2.0, scenes, task specifications and run logs under CC BY 4.0, and each third-party asset under its own license (Tab. S16). All 40 links are released, each marked measured from shoved starts (1: T1.01→T1.02) or paper check only (39). The release marks no other link as measured. Assets are referenced by pinned path or revision and are not re-hosted where their license does not allow it. No episode of LADDER (a benchmark of 145 long-horizon tabletop tasks, main Table 1) or other rival benchmark data is released.
Harness commands. One walk: python v21_task_model.run.py --task T1 --model <model> --setting top --state 0 --upto 5 (Isaac Sim 6.1.0 on one GPU and the model’s API key, --setting second for the second tier, --state 1 for the other public start state). The tables of main Sec. 5: python make_v2_numbers.py (reads the run registry and the per-tool call counts, writes the LaTeX tables). This supplement: python make_suppl.py.
Calls, tiers and bounds. The harness tool schema has drive_to(x, y, heading_deg, speed, accel) (base to a pose, speed at most 0.6 m/s, default 0.3, accel in m/s, default 0.3, a call runs at most 5 s of simulated time), move_to(x, y, z, yaw, pitch, roll, frame, straight) (fingertip to a 6-DoF pose), gripper(width_mm, force_N) (finger opening and a squeeze-force cap of at most 140 N), look(pan_deg, tilt_deg) (turn the head camera), wait(seconds) (hold still for the stated seconds of simulated time, at most 20 per call) and done() (end the sub-task). The runs of main Sec. 5 used the skill calls of main Sec. 3 for the arm, and no run called move_to or gripper. --setting top runs the top effort tier and --setting second the second: The top tier is the highest effort value the harness accepts for the model (max for all but Gemini 4 Argon, high for Gemini 4 Argon) and the second tier is high for all but Gemini 4 Argon, medium for Gemini 4 Argon, and no setting between the two was run. The exact bound is the one-sided 95% upper confidence bound of the Clopper–Pearson interval for passes in walks, the with . At it is .
Hidden test split (planned and not built). Further start states per task would be kept on a scoring server and never released. The number is not fixed. An entry is an agent that answers the same calls. The server runs its walks and returns per task with the exact bound and the harm events. If a hidden start state leaks, it is retired and replaced, and affected scores are marked.
Canary and scoring-server terms. Every released file carries one canary string so that its presence in training data can be detected. The scoring-server terms bar training or tuning on the benchmark’s start states, logs or hidden split, and require each entry to state its model version and settings.
Datasheet summary. Motivation: test whether agents that drive a robot act on where the previous step actually left things. Composition: 10 tasks × 5 sub-tasks, 40 links, 36 harm events, 74 SimReady assets plus robot and people models, two public start states per task (a hidden split is planned). No personal data (names and MRNs on labels are synthetic). Collection: scenes built with a floor-plan builder and Isaac Lab-Arena, and walk logs recorded by the harness in simulation. Uses: intended for evaluating and comparing agents in simulation. It is not for certifying robots, staffing decisions or clinical claims. Distribution and maintenance: below.
Maintenance. A version is frozen when its scoring server opens. A fix that changes any grade ships as a new version, and scoring-server rows name the version they were scored on. Bugs are reported through the repository’s issue tracker. The maintainers keep the scoring server running and announce the retirement of a version in advance.
Misuse. The main risk is misreading: a high score in a simulated hospital is not evidence that a robot is safe near staff or patients, and must not be presented as such. Harm events are proxies on scene state and are no safety certification.
Runs, tool use, proofs and models of the first results
Per-run record (main Sec. 5). One row per valid run, from the run registry. In a row: sub-tasks T# .01 to T# .05 passed before the first failure (T#.01 is guided). Ended at: the sub-task of the first failure. Failed: failed calls. Different: failed calls whose next call in the same sub-task differed in tool or arguments. Not failed: failed calls whose next call did not fail. Unstuck: failed calls followed by a working call and a sub-task that passed in the end. Repeated: calls equal in tool and arguments to an earlier call of the same sub-task. Drift: distance in cm between the true base position and the robot’s own estimate at the last call. US$: API list price. Run tags and end times. Each run is a tag of the registry with its end time (UTC, month-day hour:minute).
| Tag | Ended |
| m2cal_opus_t1_s0_r101 | 11-12 02:11 |
| m2cal_opus_t1_s0_r102 | 11-12 06:23 |
| m2cal_terra_t1_s0_r101 | 11-12 02:15 |
| m2cal_opus_t1_s1_r101 | 11-12 02:35 |
| m2cal_sol_t1_s1_r101 | 11-12 02:08 |
| m2cal_terra_t1_s1_r101 | 11-12 02:09 |
| m2cal_opus_t10_s0_r101 | 11-12 01:38 |
| m2cal_sol_t10_s0_r101 | 11-12 00:53 |
| m2cal_terra_t10_s0_r101 | 11-12 00:28 |
| m2cal_opus_t10_s1_r101 | 11-12 02:05 |
| m2cal_sol_t10_s1_r101 | 11-12 00:15 |
| m2cal_terra_t10_s1_r101 | 11-11 23:55 |
| m2cal_opus_t3_s0_r101 | 11-12 00:45 |
| m2cal_sol_t3_s0_r101 | 11-11 22:40 |
| m2cal_terra_t3_s0_r101 | 11-11 22:37 |
Table S19. Tags and end times of the 15 valid runs.
Tool use. The calls of all 15 runs by tool, with the failed calls (defined in main Sec. 4).
| Tool | Calls | Failed |
look | 343 | 0 |
scan_over | 175 | 59 |
drive_to | 163 | 48 |
arm_home | 69 | 0 |
open_drawer | 62 | 60 |
done | 42 | 0 |
press | 38 | 2 |
wait | 27 | 0 |
push_pole | 19 | 17 |
pick | 18 | 2 |
place | 14 | 1 |
pull_trolley | 6 | 0 |
| All | 976 | 189 |
Table S20. Calls and failed calls by tool, all 15 runs.
Of the 189 failed calls, 48 are drive_to calls (42 stopped at the per-call time cap and 6 by a person inside the safety distance), 60 are open_drawer, 59 are scan_over and 17 are push_pole. look, done, arm_home, wait and pull_trolley had no failed call.
Scripted robot, proofs. The proof is a run of the picture-only parity robot: a scripted robot that makes skill calls and uses only a model’s prompt, pictures and history, and fails if it reads any simulator pose or id. The run registry records a passed proof of the current version for T1 on start states 0 and 1, T10 on start states 0 and 1, and T3 on start state 0 (all seven graded steps, two warm-ups and five sub-tasks, as read by chain_grades.py), and for no other chore or start state. The T10 proof ran with the route facts of T10 given to the robot privately. The model’s prompt has no such hint, so that proof shows the chore is doable when those facts are known, and not that a robot can find them from the prompt.
Models and run dates (main § 5). Each ID is the alias sent to the vendor API. The run logs hold no dated snapshot string. All three models ran at high effort.
Details moved from the main text
Models, tiers and routes (main § 5). The prompt, the memory (the pictures of the latest call and the text history of calls, earlier pictures replaced by the text “[picture from call N not resent]”), the tool loop and the call caps are the same for every model. The tools reach the Claude and GPT models as JSON text in the system text, and these models reply with one JSON tool call as text, which the harness parses. The Claude model runs through claude -p with its built-in tools off, no MCP server and safe mode, in an append-only session with caching and auto-compaction off. The GPT models use their own API. We did not inspect what claude -p sends beyond these flags.
Time quantities (main § 3). The harness has these and no others: a wait of at most 20 s and a drive_to of at most 5 s of simulated time per call, the people’s schedules and door cycles (scene properties in simulated time, T1: surgeon at the table’s west side, stepping 0.5 m every 20 s, and circulating nurse crossing the trolley lane for about 6 s every 30 s), the 60 Hz physics rate, a guard on each API call (four tries without an answer make the walk invalid), and windows that tests read on the scene: the still check of the pass terms (0.2 s at under 1 cm/s at every physics step), a 0.5 s settling window (T1 “instrument off the tray”), a 0.5 s tilt window (T5 “vial tipped”), an impact speed (T5 “vial break”) and the per-step motion of the person rule. None limits the time the agent takes: a sub-task is bounded by its call cap, and the API-call guard makes a walk invalid, which is re-run. It never scores a fail.
Picture lag (main § 3). Under GPU load the render products can return the previous frame or a black one. Since 2 Nov 2026 07:50 UTC the harness re-reads such a picture up to three times and logs each re-read (FRAME_RETRY). The runs of main § 5 all ran after that date.
T1 version ledger
Table S22 lists the 26 T1 versions in the version ledger, one row per version. Most versions change one knob of their parent (a card, a key, a hint). The first families (va, vc, vk, vx, vu, vux, vw) differ from one another in card, key and scissors labels at once, and vqsnb changes the order and place of the cards in every picture, so the one-knob aim was not kept in those rows. The results of the probe walks and robot proofs of each version are kept in the ledger and are not reported in this version of the paper. The current version is vqsnd.
| Version | Change from its parent |
| dq | no description kept |
| v | no description kept |
| v11edit, v12edit | no description kept |
| v1b, vE, vQM, vUN | no description kept |
| va | card curved/curved, T1.02 notkept, key None, scissors labelled |
| vc | card None, T1.02 either, key M, scissors unlabelled |
| vk | card curved/curved, T1.02 notkept, key K, scissors unlabelled |
| vx | card switch0/switch1, T1.02 notkept, key K, scissors unlabelled |
| vq | kept forceps shown by a card in the T1.01 pictures, T1.02 takes back the not-kept one, joints unnamed |
| vqs | vq, the first card replaced by the real one after the first T1.01 call |
| vqsf, vqsg | no description kept |
| vqsn | vqs, T1.02 hint names the two forceps joints |
| vqsnb | vqsn, decoy card first and real card after, in every T1.01 picture |
| vqsnc | vqsnb, start 1 keeps the curved forceps too |
| vqsnd | vqsnb, KEEP card in the first T1.01 picture only (current) |
| vqsne | vqsnb, real card first, opposite card after the first call, text says the first card counts |
| vqsneo | vqsne, T1.03 hint gives the two outline centres unlabelled |
| vu | card curved/curved, T1.02 notkept, key U, scissors unlabelled |
| vux | card switch0/switch1, T1.02 notkept, key U, scissors unlabelled |
| vuxn | vux, T1.03 hint lists the two outlines as near / far |
| vw | card curved/curved, T1.02 notkept, key U, scissors unlabelled |
Table S22. T1 versions in the ledger, with the change from the parent. Versions with “no description kept” have no description recorded.
Walk loop
Figure S2 shows the loop of one walk: observations, prompt context, one tool call, execution in the simulator and grading.
References
- [1]NVIDIA Isaac for Healthcare (Isaac robotics platform for healthcare). NVIDIA robotics platform, 2026. Accessed 2026-10-28. 3
- [2]Isaac for Healthcare reference workflows (i4h-workflows). Reference workflow code, 2026. Accessed 2026-10-28. 3
- [3]Isaac Sim replicator agent extension (isaac-sim.replicator.agent.core 1.7.9). NVIDIA Isaac Sim 6.1.0 extension for animated people, 2026. Accessed 2026-10-27. 4
- [4]Isaac Lab-Arena. NVIDIA open-source scene-assembly framework, commit 3e2c70b, Apache License 2.0, 2026. Accessed 2026-10-27. 3
- [5]NVIDIA Isaac Sim 6.1.0 (build 6.1.0-rc.26). NVIDIA simulation software, Python package isaacsim 6.1.0.0; NVIDIA Omniverse License Agreement, 2026. Accessed 2026-10-27. 3
- [6]nvidia/simready-assets (Hugging Face dataset), CC BY 4.0. Public asset dataset; each asset loaded at a pinned revision, 2026. Accessed 2026-10-27. 3
- [7]Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramrakhya, Daniel Tran, Joanne Truong, John M. Turner, Eric Undersander, and Tsung-Yen Yang. PARTNR: A benchmark for planning and reasoning in embodied multi-agent tasks. arXiv preprint arXiv:2411.00081, 2024. 3, 4arxiv.org/abs/2411.00081
- [8]Lirong Che, Shuo Wen, Shan Huang, Chuang Wang, Yuzhe Yang, Gregory Dudek, Xueqian Wang, and Jian Su. MANSION: Multi-floor language-to-3D scene generation for long-horizon tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026. arXiv:2603.11554. 2, 4arxiv.org/abs/2603.11554
- [9]Yifei Dong, Fengyi Wu, Qi He, Lingdong Kong, Heng Li, Minghan Li, Zegang Cheng, Yuxuan Zhou, Jingdong Sun, Qi Dai, Alexander G Hauptmann, and Zhi-Qi Cheng. HA-VLN 2.0: An open benchmark and leaderboard for human-aware navigation in discrete and continuous environments with dynamic multi-human interactions. arXiv preprint arXiv:2503.14229, 2025. Accepted to IROS 2026. 3, 4arxiv.org/abs/2503.14229
- [10]Zackory Erickson, Vamsee Gangaram, Ariel Kapusta, C. Karen Liu, and Charles C. Kemp. Assistive Gym: A physics simulation framework for assistive robotics. arXiv preprint arXiv:1910.04700, 2019. 3arxiv.org/abs/1910.04700
- [11]Senyu Fei, Siyin Wang, Junhao Shi, Zihao Dai, Jikun Cai, Pengfang Qian, Li Ji, Xinzhe He, Shiduo Zhang, Zhaoye Fei, Jinlan Fu, Jingjing Gong, and Xipeng Qiu. LIBERO-Plus: A progressive robustness benchmark for visual-language-action models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026. 2, 4
- [12]Ning Gao, Jinliang Zheng, Xing Gao, Haoxiang Ma, Hanqing Wang, Yukai Wang, Jiantong Chen, Zanxin Chen, Shujie Zhang, Mingta Jia, Xuekun Jiang, Zihou Zhu, Xinyu Li, Shuai Wang, Hao Li, Wenzhe Cai, Yuqiang Yang, Xudong Xu, Zhaoyang Lyu, Yao Mu, Tai Wang, Jiangmiao Pang, Jia Zeng, Weinan Zhang, and Chunhua Shen. EBench: Elemental diagnosis of generalist mobile manipulation policies. arXiv preprint arXiv:2606.18239, 2026. 3
- [13]Ran Gong, Jiangyong Huang, Yizhou Zhao, Haoran Geng, Xiaofeng Gao, Qingyang Wu, Wenshi Ai, Ziheng Zhou, Demetri Terzopoulos, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. ARNOLD: A benchmark for language-grounded task learning with continuous states in realistic 3D scenes. arXiv preprint arXiv:2304.04321, 2023. ICCV 2023. 3arxiv.org/abs/2304.04321
- [14]Aryan Goyal. Measuring the stability assumption behind action chunking. arXiv preprint arXiv:2610.01626, 2026. 3arxiv.org/abs/2610.01626
- [15]Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks. arXiv preprint arXiv:2607.07946, 2026. 3arxiv.org/abs/2607.07946
- [16]Nikita Kachaev, Andrei Spiridonov, Andrey Gorodetsky, Kirill Muravyev, Nikita Oskolkov, Aditya Narendra, Vlad Shakhiro, Dmitry Makarov, Aleksandr I. Panov, Polina Fedotova, and Alexey K. Kovalev. Mind and motion aligned: A joint evaluation IsaacSim benchmark for task planning and low-level policies in mobile manipulation. arXiv preprint arXiv:2508.15663, 2025. 3arxiv.org/abs/2508.15663
- [17]Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabriel Levine, Wensi Ai, Benjamin Martinez, Hang Yin, Michael Lingelbach, Minjunhe Hwang, Ayano Hiranaka, Sujay Garlanka, Arman Aydin, Sharon Lee, Jiankai Sun, Mona Anvari, Manasi Sharma, Dhruva Bansal, Samuel Hunter, Kyu-Young Kim, Alan Lou, Caleb R Matthews, Ivan Villa-Renteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Yunzhu Li, Silvio Savarese, Hyowon Gweon, C. Karen Liu, Jiajun Wu, and Li Fei-Fei. BEHAVIOR-1K: A human-centered, embodied AI benchmark with 1,000 everyday activities and realistic simulation. arXiv preprint arXiv:2403.09227, 2024. A preliminary version appeared at CoRL 2022. 2, 4arxiv.org/abs/2403.09227
- [18]Yang Li, Chen Zhao, Zhuoran Wang, Jiankang Wang, Chao Shao, Yihan Lin, Haitao Shen, and Jing Zhang. RoboRecover: Benchmarking robot policy recovery under execution deviations. arXiv preprint arXiv:2609.28952, 2026. 3arxiv.org/abs/2609.28952
- [19]Zongxia Li et al. Long-Horizon-Terminal-Bench: Testing the limits of agents on long-horizon terminal tasks with dense reward-based grading. arXiv preprint arXiv:2607.08964, 2026. 3arxiv.org/abs/2607.08964
- [20]Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. arXiv preprint arXiv:2306.03310, 2023. 2, 4arxiv.org/abs/2306.03310
- [21]Oier Mees, Lukas Hermans, Erick Rosete-Beas, and Wolfram Burgard. CALVIN: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. arXiv preprint arXiv:2112.02277, 2022. IEEE Robotics and Automation Letters (RA-L). 2, 3, 4
- [22]Petko Petkov, Giuliano Albanese, Nikolay Nikolov, Luc Van Gool, Jonathan Tremblay, and Danda Paudel. LADER: A long-horizon robot manipulation benchmark. arXiv preprint arXiv:2610.18053, 2026. 2, 3, 4arxiv.org/abs/2610.18053
- [23]Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander Willm Clegg, Michal Hlavac, So Yeon Min, Vladimír Vondruš, Theophile Gervet, Vincent-Pierre Berges, John M. Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Un-nat Jain, Dhruv Batra, Akshara Rai, and Roozbeh Mottaghi. Habitat 3.0: A co-habitat for humans, avatars and robots. arXiv preprint arXiv:2310.13724, 2023. 3, 4arxiv.org/abs/2310.13724
- [24]Ke Rui, Yushen Zuo, Jiawei Wang, Haoran Jia, Jinming Ma, Weitao Zhou, and Minglei Li. Diagnosing semantic handoff failures in agent-orchestrated vision-language-action skill composition. arXiv preprint arXiv:2607.06256, 2026. 2, 3arxiv.org/abs/2607.06256
- [25]Scale AI. LHAW: Long-horizon task workflows. Scale AI Labs blog, Labs.scale.com/blog/LHAW, 2026. Accessed 2026-11-02. 3
- [26]Xia Su, Ruiqi Chen, Benlin Liu, Jingwei Ma, Zonglin Di, Ranjay Krishna, and Jon Froehlich. CapNav: Benchmarking vision language models on capability-conditioned indoor navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026. arXiv:2602.18424. 2, 4arxiv.org/abs/2602.18424
- [27]Jiashuo Sun, Yue He, Wenxuan Liu, Tao Mao, Jiaz heng Wang, Xiang Chen, and Min Liu. SurgVLA-Bench: Towards evaluating vision-language-action models for laparoscopic surgical robotics. arXiv preprint arXiv:2606.29247, 2026. 3arxiv.org/abs/2606.29247
- [28]Zelin Wan, Arash Nourian, Xiaoxiao Li, Nihar Nandan, and Kamalakannan Nandagopal. APIFlow-Bench: Measuring whether agents survive long, dependent API workflows. arXiv preprint arXiv:2608.29128, 2026. 2, 3arxiv.org/abs/2608.29128
- [29]Meiqi Wang and Shichao Li. Compiling and benchmarking task-state horizons for embodied agents. arXiv preprint arXiv:2608.08036, 2026. 3
- [30]Sen Wang, Bangwei Liu, Zhenkun Gao, Lizhuang Ma, Xuhong Wang, Yuan Xie, and Xin Tan. Explore with long-term memory: A benchmark and multimodal LLM-based reinforcement learning framework for embodied exploration (LMEE-Bench). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026. arXiv:2601.10744. 2, 4arxiv.org/abs/2601.10744
- [31]Wenbo Wang, Fangyun Wei, QiXiu Li, Xi Chen, Yaobo Liang, Chang Xu, Jiaolong Yang, and Baining Guo. MobileManBench: Simplifying model verification for mobile manipulation. arXiv preprint arXiv:2602.05233, 2026. Accessed to ECCV 2026. 3arxiv.org/abs/2602.05233
- [32]Jiaqi Xu, Bin Li, Bo Lu, Yun-Hui Liu, Qi Dou, and Pheng-Ann Heng. SurRoL: An open-source reinforcement learning centered and dVRK compatible platform for surgical robot learning. arXiv preprint arXiv:2108.13035, 2021. IROS 2021. 3arxiv.org/abs/2108.13035
- [33]Zhangchen Xu et al. AutoLab: Can frontier models solve long-horizon auto research and engineering tasks? arXiv preprint arXiv:2606.05080, 2026. 3arxiv.org/abs/2606.05080
- [34]Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, and Tong Zhang. EmbodiedBench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. arXiv preprint arXiv:2502.09560, 2025. ICML 2025. 2, 4arxiv.org/abs/2502.09560
- [35]Yue Yang, Linfeng Zhao, Mingyu Ding, Gedas Bertasius, and Daniel Sfar. BOSS: Benchmark for observation space shift in long-horizon task. arXiv preprint arXiv:2502.15679, 2025. 2, 3arxiv.org/abs/2502.15679
- [36]Ruolin Ye, Wengyang Xu, Haoyuan Fu, Rajat Kumar Jenamani, Vy Nguyen, Cewu Lu, Katherine Dimitropoulou, and Tapomayukh Bhattacharjee. RCareWorld: A human-centric simulation world for caregiving robots. arXiv preprint arXiv:2210.10821, 2022. 3arxiv.org/abs/2210.10821
- [37]Zonghao Ying, Le Wang, Yisong Xiao, Jiakai Wang, Yuqing Ma, Jinyang Guo, Zhenfei Yin, Mingchuan Zhang, Aishan Liu, and Xianglong Liu. AGENTSAFE: Benchmarking the safety of embodied agents on hazardous instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026. arXiv:2506.14697. 2, 4arxiv.org/abs/2506.14697
- [38]Qinxi Yu, Masoud Moghani, Karthik Dharmarajan, Vincent Schorp, William Chung-Ho Panitch, Jingzhou Liu, Kush Hari, Huang Huang, Mayank Mittal, Ken Goldberg, and Animesh Garg. ORBIT-Surgical: An open-simulation framework for learning surgical augmented dexterity. arXiv preprint arXiv:2404.16027, 2024. 3, 4arxiv.org/abs/2404.16027
- [39]Shiduo Zhang, Zhe Xu, Peijiu Liu, Xiaopeng Yu, Yuan Li, Qinghui Gao, Zhaoye Fei, Zhangyue Yin, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. VLABench: A large-scale benchmark for language-conditioned robotics manipulation with long-horizon reasoning tasks. arXiv preprint arXiv:2412.18194, 2024. 2, 4arxiv.org/abs/2412.18194
- [40]Xunyi Zhao, Gengze Zhou, and Qi Wu. VLN-MME: Diagnosing MLLMs as language-guided visual navigation agents. arXiv preprint arXiv:2512.24851, 2025. 2, 4arxiv.org/abs/2512.24851
- [41]Kaizhi Zheng, Xiaotong Chen, Odest Chadwicke Jenkins, and Xin Eric Wang. VLMbench: A compositional benchmark for vision-and-language manipulation. arXiv preprint arXiv:2206.08522, 2022. 3arxiv.org/abs/2206.08522
- [42]Mingxin Zheng, Nic Ma, and Mostafa Toloui. Using simulation to build robotic systems for hospital automation (Project Rheo). NVIDIA Technical Blog, 2026. Posted 16 March; accessed 2026-10-28. 3, 4
- [43]Qinhong Zhou, Sunli Chen, Yisong Wang, Haozhe Xu, Weihua Du, Hongxin Zhang, Yilun Du, Joshua B. Tenenbaum, and Chuang Gan. HAZARD challenge: Embodied decision making in dynamically changing environments. arXiv preprint arXiv:2401.12975, 2024. ICLR 2024. 3arxiv.org/abs/2401.12975
