betaFeedback

WorksComputer Science

HospitalBench: Diagnosing Where and Why Embodied Agents Fail on Long Hospital Task Chains

Abstract

Long-horizon embodied tasks fail in ways that end-task success cannot see. HospitalBench is a diagnostic benchmark of ten hospital chores in NVIDIA Isaac Sim, each two unscored warm-up steps followed by a causal chain of five scored sub-tasks in one continuous simulation: the state one step leaves (an object pose, a choice, a fact seen once) determines what the next step must do, unlike benchmarks that place sub-tasks side by side. Vision-language agents act through skill-level tools from camera images. A scripted robot that makes only the calls a model gets is the solvability check of a chore. From each trajectory the benchmark reads where a chain breaks, what the agent did after a failed call, and how far its own position estimate drifts from the true one. This version reports first results only: 15 of 60 planned runs (three models at high effort, one run each on chores T1 and T10 from both start states and T3 from one), with scripted-robot proofs on three of the ten chores. Of the 15 runs, 2 passed all five sub-tasks. Runs still going after sub-tasks 1 to 5 were 15, 8, 4, 2, 2. Of 189 failed calls, 168 were followed by a different call and 111 by a call that did not fail. These are first results and not a full benchmark. We will release the scenes, task chains, scripted robot, evaluation harness and trajectories.

HospitalBench chore T1 chain and first model results

Figure 1. One chain of chore T1 (instrument count before closure): five sub-tasks in one simulation with no reset, after two unscored warm-ups. Top: a scripted robot passes all five (frames from an earlier scene version, before the KEEP card). Arrows: what the next sub-task takes from the scene or the picture: V a trolley pose, C a choice the robot made, P what the last step placed. The first link is also F, a fact shown once (a KEEP card in the first picture of T1.01 names the forceps to leave). Bottom: first results, 15 of 60 planned runs. Each row is a chore and start state, each group of chips a model at high effort, each chip a sub-task T#.01 to T#.05. Chores T1 and T10 ran from both start states and T3 from start 0, one run per cell.

Long-horizon embodied tasks fail in ways that end-task success cannot see. HospitalBench is a diagnostic benchmark of ten hospital chores in NVIDIA Isaac Sim, each two unscored warm-up steps followed by a causal chain of five scored sub-tasks in one continuous simulation: the state one step leaves (an object pose, a choice, a fact seen once) determines what the next step must do, unlike benchmarks that place sub-tasks side by side. Vision-language agents act through skill-level tools from camera images. A scripted robot that makes only the calls a model gets is the solvability check of a chore. From each trajectory the benchmark reads where a chain breaks, what the agent did after a failed call, and how far its own position estimate drifts from the true one. This version reports first results only: 15 of 60 planned runs (three models at high effort, one run each on chores T1 and T10 from both start states and T3 from one), with scripted-robot proofs on three of the ten chores. Of the 15 runs, 2 passed all five sub-tasks. Runs still going after sub-tasks 1 to 5 were 15, 8, 4, 2, 2. Of 189 failed calls, 168 were followed by a different call and 111 by a call that did not fail. These are first results and not a full benchmark. We will release the scenes, task chains, scripted robot, evaluation harness and trajectories.

Introduction

Where along a chain of dependent chores does an embodied agent’s performance degrade, and why? Long-horizon benchmarks report that success falls as tasks get longer [22, 28], and recent work traces failures to the seams between skills [24, 35], but one end-task score cannot say at which step, at which link or by which kind of failure a chain breaks. A chain shows degradation by carrying only if each step needs the actual result of the one before, or a fact seen earlier: a trolley parked a few centimetres off its mark moves every grasp that follows, whereas a card seen once names the forceps to take and is gone by the time they are needed. HospitalBench is built for this diagnosis: ten chores of five dependent sub-tasks (50 sub-tasks, 40 links) in six simulated hospital areas (§ 3, Table 2), each link depending on the step before it in one of four ways (Figure 2).

Link diagram of T1, current version (vqsnd)

Figure 2. Link diagram of T1, current version (vqsnd). Five scored sub-tasks run in one simulation with no reset, after two unscored warm-ups (W1, W2: press a YES or NO panel to answer a lamp question). Each arrow names the link kind and what the next sub-task takes from the scene the robot left or the picture it saw: V a cart or trolley pose, C a choice on the robot’s own result, P what this step placed, F a fact shown once (a KEEP card in the first picture of T1.01 only, and T1.02 must take the other forceps). The link kinds of T1 read VCPC, with T1.01→T1.02 also F.

Table 2. Scale and statistics as built, from the task files, the room builder, the T1 version ledger and the run registry.

QuantityValue
Chores, areas, rooms10, 6, 10 (one room per chore)
Sub-tasks per chore, total5, 50 (plus 2 unscored warm-ups before T1.01)
Links by kind (V, C, P)14, 16, 10 (total 40, 4 per chore)
Links of kind F1 (T1.01→T1.02, also V), no other chore has one yet
Start states2 public per chore (20), hidden split not built
SimReady assets74 distinct of 2,595 folders, 44 with joints, 3 stand-ins
Named harm events36 (2 to 5 per chore)
T1 versions tried26 in the ledger
Scripted-robot proofsT1 and T10 on both starts, T3 on start 0 (3 of 10 chores)
Model runs in this paper15 of 60 planned: 3 models, T1 and T10 on both starts, T3 on start 0, one run per cell

A model sends every robot call and a controller executes it. A walk is one model at one effort tier (the reasoning-effort setting of its API call) running the sub-tasks of one chore in order, with no reset (a chain), from one start state (the initial scene). The scene state is the poses of the robot, objects and people, which every pass term and harm event reads. A walk ends at its first failed sub-task. The number passed before it is its passed in a row score. The scripted robot is a hand-written program that makes only the calls a model makes. The calibration model is the model whose walks were used to choose the version of a chore (Opus 5.5 high), so its results are not held out. Held-out models (GPT-6.1 Terra, GPT-6.1 Sol) were not used for that choice (§ 4).

What we find so far (§ 5). This version reports first results only. We ran Opus 5.5 high, GPT-6.1 Sol high and GPT-6.1 Terra high once on each of five cells (chores T1 and T10 from both start states, T3 from start 0), 15 runs of the 60 planned, outside the staged protocol of § 4. The scripted robot has a proof on those chores and starts, and on none of the other seven chores. Two runs passed all five sub-tasks, both on T1 (GPT-6.1 Terra from start 0, GPT-6.1 Sol from start 1). Runs still going after sub-tasks 1 to 5 were 15, 8, 4, 2, 2. Every other run ended at its first failed sub-task, and in 7 of those 13 runs this was the first free sub-task T#0.02, never the guided T#0.01. On T1, 3 of the 6 runs that reached T1.02 failed it. Two took the forceps that a card shown only in the first picture of T1.01 named to keep, and one never picked either forceps. After a failed call the next call differed in 168 of 189 cases and did not fail in 111. With one run per cell there is no interval and no ranking of models, and these are first results, not a full benchmark. The other seven chores have no model run.

Failure modes and harm events. We read four failure modes from a trajectory (§ 4): compounding errors per link, recovery, state drift and tool-use breakdowns. This version reports the last three from the 15 runs (§ 5.1). Compounding by landing position and harm events are defined but not analysed here. Harm events (an instrument left off the tray, the robot closing in on a person) are simulator proxies and not injuries.

Honest scope. No single task is new: every kind of action in our tasks is in a task prompt of seven existing benchmarks, and five further 2026 benchmarks are in the same comparison (Table 1). The design property we add is link dependence, tested link by link (the link test, § 3.2) and through a fact shown once. The link test has been run on 1 of the 40 links, for the scripted robot on an earlier T1 version. The rest have only a paper check on assumed numbers (§ 3.2). Only T1, T10 and T3 have model runs. No method has been trained on HospitalBench, and we do not claim it works as a training environment.

Table 1. Comparison with twelve earlier and five CVPR 2026 benchmarks (last five rows: first pages only). Scale: tt tasks, rr VLN-MME records of its released validation subsets (R2R 200, REVERIE 300, ObjectNav 200, CVDN 200), counted by us, ii PARTNR instructions, bb buildings, gg goals. Models: frontier models scored by the authors (✓ yes, ✗ no, ∼ some, and in brackets the models or workflows evaluated, where read). Moves: the model sends every move, an action or pose at each step. ∼ skill calls or primitives. Chain: separately instructed, graded sub-tasks each starting from the scene the last left, no reset (∼: one instruction with sub-goals graded by progress). Fact: fact carried, a step needs a fact seen in an earlier step. Alone: alone vs chained, the same sub-task run alone beside the chained walk. Link: link test, each link tested by moving the earlier result inside its band. ?: not stated or not read. Ours is ∼ where only three chores, one link or three models have run. The link test covers 1 of 40 links, and the alone control has no data yet. The new columns of the older rows and the ? cells are not checked against every full paper.

BenchmarkScaleModelsMovesPeopleChainFactAloneLink
LIBERO-10 [20]10 t10\,t✗✓✗✗✗✗✗
CALVIN [21]34 t34\,t✗✓✗✓✗✗✗
EmbodiedBench [34]1,128 t1{,}128\,t✓∼✗✗✗✗✗
VLN-MME [40]900 r900\,r✓✗✗✗✗✗✗
VLABench [39]100 t100\,t∼∼✗✗✗✗✗
BEHAVIOR-1K [17]1,000 t1{,}000\,t✗∼✗∼?✗✗
LADDER [22]145 t145\,t✓∼✗∼?✗✗
PARTNR [7]100,000 i100{,}000\,i∼∼✓∼?✗✗
Habitat 3.0 [23]2 t2\,t✗∼✓✗✗✗✗
HA-VLN 2.0 [9]1 t1\,t✗∼✓✗✗✗✗
Rheo [42]3 t3\,t✗✓✗∼✗✗✗
ORBIT-Surgical [38]14 t14\,t✗✓✗✗✗✗✗
MANSION [8]1,000+ b1{,}000+\,b???∼?✗✗
LMEE-Bench [30]9,000+ g9{,}000+\,g✓✗?∼✓✗✗
AGENTSAFE [37]1,350 t1{,}350\,t✓ (9+2)✗?✗✗✗✗
LIBERO-Plus [11]10,030 t10{,}030\,t✗ (10 VLAs)✓✗✗✗✗✗
CapNav [26]473 t473\,t✓ (13)✗?✗✗✗✗
Ours10 chores, 50 sub-tasks∼ (3)∼✓✓∼✗∼

Contributions. (1) A diagnostic benchmark of ten hospital chores, each five dependent sub-tasks in one reset-free simulation, with the kind of each of the 40 links stated, a link kind for a fact shown once, and 36 named harm events (§ 3). (2) A staged protocol (stage 0 calibration, an alone check M1, held-out stages 1 to 3) that reports the calibration model apart from held-out models, and the first 15 runs, made outside it (§ 4). (3) First results on three chores with per-run records of where a chain ended, what followed each failed call and how far the robot’s position estimate drifted, and a read of the T1.01→T1.02 link (§ 5.1).

Related work

The skills are not new. Every kind of action in our tasks appears in at least one of 983 task prompts we collected verbatim from seven benchmarks: LIBERO-10 [20], CALVIN [21], EmbodiedBench [34], VLN-MME [40], VLABench [39], BEHAVIOR-1K [17] and LADDER [22] (counts in Table 1). Five CVPR 2026 benchmarks are in the table from their first pages only: MANSION [8] (multi-floor scenes), LMEE-Bench [30] (multi-goal navigation, then questions on what was seen), AGENTSAFE [37] (hazardous instructions), LIBERO-Plus [11] (perturbed LIBERO tasks) and CapNav [26] (capability-conditioned navigation). Chains of up to ten sub-goals exist [22], CALVIN reports success at each position of five-step chains [21], and mobile manipulators run in Isaac Sim [12, 16, 31]. The closest to a carried fact is LMEE-Bench, whose memory questions ask about what was seen during earlier navigation. The fact is asked for after the chain, whereas here a later step must act on it. Difference: in these chains a later step can be done from where the earlier result should be. None moves an earlier result inside its pass band to test dependence, and none reports a sub-task alone beside the chained walk (last columns of Table 1).

Hospital, surgical and care settings are not new. NVIDIA Isaac for Healthcare and its Project Rheo blueprint provide hospital and operating-room digital twins with surgical tray pick-and-place, case cart pushing and trocar assembly [1, 2, 42]. Rheo’s benchmark snapshot (GR00T-based policies, four scenes of one task) reports no moving people and no frontier-model scores, as does ORBIT-Surgical (14 surgical tasks [38]). SurRoL [32] and SurgVLA-Bench [27] cover surgical-robot and laparoscopic tasks. RCareWorld [36] and Assistive Gym [10] simulate caregiving. Difference: our chores cover hospital logistics and nursing-support work, never touch a patient, and chain five dependent sub-tasks. We make no clinician-shortage claim [42].

The stated limits are not new. Moving people are in PARTNR [7], Habitat 3.0 [23] and HA-VLN 2.0 [9]. Scene-changing events are in HAZARD [43], orientation limits in VLMbench [41] and goals with checked tolerances in ARNOLD [13]. We use such limits only as task conditions. How they were chosen is a selection effect (§ 7).

Dependence has been studied. Perturb-and-replay is published [14]. Preceding skills shift the observations of later ones [35]. Skills that succeed from clean snapshots often stall from chained states [24]. Deviation states can be rebuilt by replaying action prefixes [18]. Task-state dependencies can be compiled into graphs [29]. Dependent API workflows lose success as chains grow [28]. Difference: we apply perturb-and-replay per link of a benchmark’s chain and add a link that carries a fact seen once.

Other long-horizon agent benchmarks (LHAW [25], Long-Horizon-Terminal-Bench [19], AutoLab [33], Deep-SWE [15]) evaluate text and code agents on files and outputs. The scene state a step leaves for the next, and the drift of a robot’s reported pose, are outside what they measure.

The benchmark

Rooms. Each of the ten chores has its own room, built with Isaac Lab-Arena [4] (commit 3e2c70b) and furnished with 74 distinct NVIDIA SimReady assets [6] (of 2,595 asset folders, each at its pinned revision, card revision 4763fbf). 44 keep their real joints (instrument hinges, drawers, doors, brakes, casters), so a step can fail through a moving part, for example forceps opening when gripped away from the joint. Three missing items (hazardous-drug cabinet, sharps container, waste-bag holder) are flagged stand-ins (supplement Sec. D, rooms: Figure S1). Every item to pick fits the 8 cm gripper.

Figure S1. Pictures of the built rooms, all existing renders, each newer than the last change to the scene files of its task (cut-offs in UTC: T1 10-31 19:40, T2 11-01 21:35, T3 11-01 21:35, T4 11-01 21:35, T5 11-01 21:35, T6 11-02 10:57, T7 11-01 21:35, T8 11-02 04:18, T9 11-01 21:35, T10 11-01 21:35). Each panel names its source (the overview camera, for T9 the head RGB picture of the first call, because its overview camera stands far outside the long room). The last panel is the head RGB picture a model sees at the first call of T1. For T2, T4, T5, T6 and T8 no picture newer than the last scene change exists, so those panels are left out.

Robot. A wheeled mobile manipulator (Ridgeback base, 7-DoF Franka arm, parallel gripper opening to 8 cm, a head camera and a wrist camera) in NVIDIA Isaac Sim 6.1.0 [5], build 6.1.0-rc.26, a release candidate: results are tied to this build, and we have not rerun on another. The model sees the sub-task instruction and, at every call, four pictures (head RGB, head depth, arm RGB, arm depth) with the base pose and gripper width. In the runs of § 5 only the pictures of the latest call are kept in its context (earlier ones are replaced by a text marker), and it acts through skill-level calls, one call per skill: pick, place, press, pull_trolley, push_pole, open_drawer, scan_over and arm_home (12 tool names appear in the logs, no run called move_to or gripper), together with drive_to(x, y, heading_deg, speed, accel), look(pan_deg, tilt_deg), wait(seconds) and done(). drive_to drives the base (speed at most 0.6 m/s, default 0.3, accel default 0.3 m/s2^2), look turns the head camera, wait holds still (people and doors keep moving) and done ends the sub-task. A controller executes each call. The model chooses every one. The world is paused while the model thinks. Each sub-task has a call cap (T1: 18, 12, 24, 48 and 66 calls for T1.01 to T1.05).

People and contact physics. Simulated staff and visitors cross the work area on fixed schedules, animated by Isaac Sim’s Replicator Agent extension [3], version 1.7.9. The person rule (harm event robot closes on a person) fails a sub-task when the robot closes the gap to a person inside 30 cm by more than 5 mm of its own motion, summed over physics steps in which its fastest point moved more than 0.5 mm (3 cm/s at 60 Hz). This is the form of the frozen T1 grader. The T2–T10 worlds use a windowed form that also counts slow net closing (supplement Sec. C). A gap closed by the person’s walking is logged as person-caused and does not fail it. Some grades (the T1.01 push, T10.01, T10.04) also require the gap to stay at least 30 cm whatever

the cause. The drive call brakes the base at the accel the model sent once a robot part is within 0.5 m of a person, so the model’s own speed and accel decide whether it stops in time (supplement Sec. C). Time quantities. Besides the 60 Hz physics rate, the harness has only a wait of at most 20 s, a drive_to of at most 5 s per call and the schedules of people and doors. None limits the time the agent takes (supplement Sec. H).

The ten tasks. The chores, each one hospital role: T1 instrument count, T2 sterile sets, T3 infusion alarm round, T4 crash trolley, T5 chemo doses, T6 insulin to a working fridge, T7 sterile restock, T8 STAT specimens, T9 biohazard bagging, T10 admission paperwork (supplement Table S1). Each sub-task instruction states its pass terms in numbers (position and heading bands, a still check), all read from scene state. Position bands range from 0.8 cm to 20 cm. Leakage check. The instruction of sub-task k+1k+1 is the fixed prompt of the task file, loaded unchanged before the walk. Its numbers are nominal bands, stances and marks (in T1 from the scripted robot’s log, as hints), never an error taken from the landing of kk. The target of k+1k+1 is nominal in the text and actual in the grader: it is read from the scene as kk left it (supplement Sec. E). No sub-task has a time limit. Lists and link bands: supplement Sec. B.

TaskArea, roleLinks
T1Instrument count before closureOR, scrub nurseVCPC
T2Sterile sets into the OR before incisionOR, circulating nurseVCVP
T3Night infusion alarm roundICU bay, ICU nurseCVCV
T4Code blue: crash trolley to the bedWard, ward nurseCVCP
T5Chemo doses from the hazardous-drug cabinetMed. room, nurseCPPV
T6Fridge failure: move insulin to the working fridgePharmacy, pharmacistVVCP
T7Sterile store restock by expirySupply store, technicianCVC P
T8STAT specimens to the lab cold roomSupply store, lifts, porterCVCP
T9Biohazard bagging and sharps into the binUtility, waste, cleanerVCPV
T10Night admission paperwork and bedside stationNurse station, clerkCCPV

Table S1. The ten tasks. Links: one letter per link from sub-task kk to k+1k+1 (four per task). Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement. Totals over the 40 links: 14 V, 16 C and 10 P. By design every link is built so that the next step depends on the result of the step before. This is the design, measured on 1 link (T1.01→T1.02, Sec. E), and the other links have a paper check only.

Design target. Each chore is meant to be tuned in stages (§ 4) so that the calibration model, Opus 5.5 high, passes a median of 1 to 2 of its five sub-tasks across the two start states (stage 0), and so that each sub-task it fails passes when run alone with its carried facts stated (M1). The target is a band for the calibration model and makes no claim about other models. The runs of § 5 were made outside this procedure and do not test the target. The pass rate of each sub-task given the one before is read per link, and the passed-in-a-row curve, which can only fall, is reported beside it.

Harm events. Every chore lists harm events (36 in all, 2 to 5 per chore, supplement Sec. C): tests on scene state with a stated threshold, graded inside the sub-task they fire in, which then fails and ends the walk. Most (32) are pass terms already in the instructions (a look-alike scissors laid in T1, the robot closing on a person, a fridge door left open). 4 are new tests.

Link kinds

Each link from sub-task kk to k+1k+1 is one of three kinds, and may also carry a fourth, and has a stated reason in the task file: 15 of the 16 C links name a hospital rule (e.g. each returned instrument goes to its own outline, in any order), and the 24 V and P links name a physical reason (the forceps lie on the trolley, so they move with every slip of the park). The other C link, T1.04→T1.05, has no hospital rule: the task file gives only the geometric reason that the scissors outlines beside the straight and the curved forceps outline are 22 cm apart on the count mat. The reasons are in the supplement (Sec. B). V Cart pose. Sub-task kk parks a cart, trolley, pole or jack within a band of 8 to 10 cm. k+1k+1 picks or places on it within a re-grip band (the tolerance of its own grasp or push on the result) at most a third of kk’s band, so the grasp moves with every slip of the park. C Choice. Sub-task kk offers two or more valid options and the option taken sets the target of k+1k+1. The targets are at least 20 cm apart or different objects. P Placement. Sub-task kk places within a band of at least 3 cm. k+1k+1 acts relative to the result within a re-grip band of 1 cm or less and a yaw band.

F Fact shown once. A fact that k+1k+1 needs is shown to the model only during an earlier sub-task, in one picture, and is never restated. The grader reads the choice. In T1, a KEEP card on the Mayo trolley shows which forceps the surgeon keeps, in the first picture of T1.01 only. T1.02 must take the other one. In the runs of § 5 the pictures of earlier calls are not resent, each is replaced by the text [picture from call N not resent], while the text turns (the model’s own replies and the call results) stay in its history. The KEEP card is therefore visible only at the first call of T1.01, and a model carries the fact to T1.02 only if it wrote it down in a text reply when it saw it. Whether it does is the test. An F link is also a V, C or P link (T1.01→T1.02 is VF), so the 14, 16 and 10 above are unchanged. Today only T1.01→T1.02 has such a link. Over the ten tasks the 40 links (four per task) are 14 V, 16 C and 10 P. The per-task strings are in the supplement (Tab. S1).

The link test

For each link from sub-task kk to k+1k+1, the scripted robot completes sub-tasks 1, …, kk on one start state of the task. Then:

  • V and P links (a parked cart, a placed item): kk’s result is moved inside kk’s own pass band in every planar direction (eight for a round band, two for a one-axis band) at two sizes fixed in advance (50% and 80% of the room left after the scripted robot’s landing error), and turned both ways, alone and with the shift.

  • C links: kk takes the other valid option.

The link is marked measured only if the fixed form (a replay of the scripted plan for k+1k+1) fails in every displaced run, the adaptive form (the scripted robot re-reading the scene) passes in every one, and three undisplaced control replays pass. A displaced run counts only if its measured displacement matches the written one. Measured: T1.01→T1.02 has been tested from shoved starts for the scripted robot, over phases of the people schedule, two sizes and all planar directions, on an earlier T1 version. The test has not been repeated on the current version. Paper check only: the other links pass an arithmetic check (the fixed replay misses k+1k+1’s re-grip band by at least 0.2 cm, or 0.5° for a turn, at both slip sizes) that uses assumed landing, heading, yaw and control-miss numbers (supplement Tab. S15). A paper check is not a measurement.

Evaluation protocol

Walks and start states. A walk runs the sub-tasks of one chore in order in one simulation, from one start state with a fixed people schedule, with no redo: it ends at its first failed sub-task (supplement Figure S2: observations, prompt context, one call, execution, grading). A sub-task passes when the scene state meets its numeric pass terms. Its call cap bounds it. After each passed sub-task the model is told “passed” and nothing else about the grade. Each chore has two public start states (the only ones used for reported walks). A hidden test split is planned and not built. Warm-ups. Two unscored warm-up sub-tasks, W1 and W2, run before T#.01: the model answers a lamp question by pressing a YES or NO panel, and a right answer to W1 unlocks W2. A warm-up miss makes the run invalid and never a failure. It is re-run once on the same seed, and a second miss is excluded from end-task success. Guided first sub-task. The prompt of T#.01 spells out the scripted robot’s calls and its odometry stop points, so that every model can pass it. Position 1 is therefore reported as guided, and the first free sub-task is T#.02. An API failure makes a walk invalid. It is re-run, never scored as a fail. Passed in a row (0 to 5) counts T#.01 and later and is the only score below end-task success.

Closed-loop evaluation flow diagram

Figure S2. Observations: four pictures per call (head RGB, head depth, arm RGB, arm depth), the base pose and gripper width from the robot. Prompt context: task card of the open sub-task, tool list, counter line “calls used n of cap (m left)”, history of earlier text turns and results, earlier pictures replaced by a marker. Model output: one tool call: a skill (pick, place, press, . . .) or drive to, look, wait, done. Execution: Isaac Sim, in simulated time only: people walk their schedules, the controller runs the call. Grading: done or a used-up call cap closes the sub-task: its terms and harm events are read from scene state. Pass: the next sub-task opens on the same scene. Fail: the walk ends and its passed-in-a-row score is the number of sub-tasks passed before it. new pictures and call feedback. Closed-loop evaluation of one walk. At every call the model sees four pictures of the latest call, the task card of the open sub-task, its remaining calls and the text history of earlier calls, whose pictures are replaced by a marker. It sends one tool call (a skill call or drive to, look, wait, done), the simulator runs it with the people moving, and the sub-task is graded when the model calls done or its call cap runs out. Sub-tasks run in order on one scene with no reset. The first failed sub-task ends the walk.

Stages. A chore version is chosen and tested in order, and each stage names the model it uses, so that no held-out model shapes the version it is scored on.

Stage 0 (calibration). Opus 5.5 high, three walks per start state. A version is accepted when the median passed in a row is 1 to 2 of 5 across the start states and the scripted robot passes it on both. The aim is one knob per change between versions, and the ledger shows the rows where several changed at once. The ledger is in supplement Sec. I. These walks are not counted. The calibration model is reported apart from held-out models.

M1 (alone). Each sub-task that fails in stage 0 is run alone, from the scripted robot’s saved state, with its carried facts stated in one sentence. It must pass for GPT-6.1 Terra or GPT-6.1 Sol at high effort. A sub-task that fails alone is a hard sub-task and not a broken link, and its prompt gets a few facts from the scripted robot’s log (goal values and key waypoints, never the full list of calls) until it passes. Alone runs of the calibration model and Terra set the hint level, so Terra is not independent of M1.

Stage 1 and 2 (held-out). GPT-6.1 Terra at high effort (stage 1) and GPT-6.1 Sol at high effort (stage 2), one walk per start state at present, on the frozen version. These are the counted walks of the protocol.

Stage 3 (later). Further models, planned. No open-weight, VLA or GROOT model has run.

The runs of this paper. The 15 runs of § 5 belong to none of these stages. They are uncounted runs of Opus 5.5 high, GPT-6.1 Sol high and GPT-6.1 Terra high with the same settings for all three models, made under an exception to the stage rule for one set of results. Each is valid: its record is complete and no API failure ended it. A run that ended on an API failure is re-run and never scored as a fail. Only Opus 5.5, GPT-6.1 Terra and GPT-6.1 Sol have run on current versions. A counted walk of the protocol is one that ran to its end on a frozen version from a public start state with a valid record.

End-task success and intervals. The headline score is the share of counted walks, per model and chore, in which all five sub-tasks pass, printed as x/nx/n. Every interval in this paper is an exact (Clopper–Pearson) 95% interval, and at x=0x = 0 the one-sided bound 1−0.051/n1 - 0.05^{1/n} is used. Every walk logs the harm events of its chore. This version has one run per cell, so a chore and start cell prints a count and no interval, and only counts pooled over a model or over all runs carry an interval.

Failure measures. Four failure modes and a per-position curve are defined for each trajectory: compounding (the pass rate of k+1k+1 given kk, by the half of kk’s band the result landed in), recovery, state drift and tool-use breakdowns. This version reports the last three and the per-position curve, read from each run’s call log by walk-analytics.py and collected in the run registry. A failed call is one whose result carries an error, ok false, or a stop that is not arrived or done. Recovery asks what the next call in the same sub-task was: a different call, and whether it failed in turn. A call counts as repeated when its tool and arguments equal an earlier call of the same sub-task. Unstuck is a failed call whose next call did not fail and whose sub-task passed in the end. State drift is the distance between the true base position and the robot’s own odometry estimate at the last call. Tool-use breakdowns are the failed calls, counted by tool. The first failed sub-task and the failed grade terms are read from the grader’s log with chain_grades.py.

Planned and not run. Continue after a fail (the scene is reset to the scripted robot’s end state of the failed sub-task and the walk goes on, as clean-start data for later positions) and a clean-start control (each T#.02 to T#.05 alone beside the chained walk, same model).

Experiments

Scope. These are first results. We ran 15 of the 60 planned runs, one run per model and cell, and three chores: T1 and T10 from both public start states and T3 from start 0. The scripted robot has a proof of the current version on exactly these chores and starts, and none on the other seven chores. The records of the runs end between 2026-11-11 22:37 and 2026-11-12 06:23 UTC. The runs were made outside the staged protocol and are uncounted (§ 4). With one run per cell no cell has an interval and no model ranking follows. Chores T2 and T4 to T9 have no model run, and no run of a model other than the three below is reported. The harness also supports Gemini 4 Argon, which was not run here.

Chore, startOpus 5.5 highGPT-6.1 Sol highGPT-6.1 Terra high
T1 start 01 (T1.02)1 (T1.02)5 (all)
T1 start 11 (T1.02)5 (all)3 (T1.04)
T10 start 03 (T10.04)2 (T10.03)2 (T10.03)
T10 start 12 (T10.03)2 (T10.03)1 (T10.02)
T3 start 01 (T3.02)1 (T3.02)1 (T3.02)
End-task success0/51/51/5
Mean passed in a row1.62.22.4

Table 3. Passed in a row per run (0 to 5, T#.01 guided, so 1 means no free sub-task passed) and in brackets the sub-task where the run ended. One run per cell, 15 runs. 5 (all): all five sub-tasks passed.

Models and settings. Opus 5.5 high (the calibration model), GPT-6.1 Sol at high effort and GPT-6.1 Terra at high effort, all with the same prompt, the same tools and the same call caps. The Claude model runs through claude -p and the GPT models through their own API, and IDs are the aliases sent to each vendor API (supplement Table S21). The model acts through skill-level calls and keeps only the latest pictures in its context (§ 3). Costs below are API list prices of the logged tokens. The Opus 5.5 runs ran on a subscription plan and were not billed at that price. A warm-up pair precedes T#.01 on every chore, and T#.01 is guided, so a passed-in-a-row score of 1 means that no free sub-task passed.

ModelAPI IDRouteRuns
Opus 5.5 highclaude-opus-5-5claude -p5
GPT-6.1 Sol highgpt-6.1-solOpenAI API5
GPT-6.1 Terra highgpt-6.1-terraOpenAI API5

Table S21. Models of the 15 valid runs: API ID, route and number of valid runs. The flags of claude -p and the tool format of each vendor are in Sec. H.

End-task success. 2 of 15 runs passed all five sub-tasks (exact 95% interval 1.7 to 40.5%, descriptive only, because the 15 runs mix chores, starts and models and are not a sample of one population): GPT-6.1 Terra on T1 start 0 and GPT-6.1 Sol on T1 start 1. Opus 5.5 high passed none of its 5 runs (one-sided upper bound 45.1%, the rule of § 4), GPT-6.1 Sol and GPT-6.1 Terra one of 5 each (two-sided 0.5 to 71.6%). Both successes are on T1, and no run passed all five sub-tasks of T10 or T3. The passed-in-a-row means over the five cells are 1.6 for Opus 5.5 high, 2.2 for GPT-6.1 Sol and 2.4 for GPT-6.1 Terra. We do not rank the models by them. With one run per cell, which model passes a given cell may change on a repeat.

Where chains end. Of the 15 runs, 15, 8, 4, 2, 2 were still going after sub-tasks 1 to 5 (Table 4). The pass rate of a sub-task given the one before is 8 of 15 for T#.01→T#.02, 4 of 8 for T#.02→T#.03, 2 of 4 for T#.03→T#.04 and 2 of 2 for T#.04→T#.05. These counts pool different chores and links, and the last two rest on 4 and 2 runs. The 13 runs that did not pass all five ended at: T1.02 ×3, T1.04 ×1, T3.02 ×3, T10.02 ×1, T10.03 ×4, T10.04 ×1. Every run passed the guided T#.01. Seven of the 13 ended at the first free sub-task T#.02. In 7 of the 13 runs the failed sub-task had used every call it allowed (six ended on a final call of done, one on the cap itself), and the other 6 ended on a failed grade with calls left.

Model12345
Opus 5.5 high52100
GPT-6.1 Sol high53111
GPT-6.1 Terra high53211
All three158422

Table 4. Runs that passed the first kk sub-tasks in a row, k=1,…,5k = 1, \ldots, 5 (5 runs per model).

Why chains break

Every group is small, so each finding below is a lead and no more.

(a) T1.02, the kept forceps. Six runs reached T1.02, the first free sub-task of T1 and the link T1.01→T1.02 that carries a fact shown once. The prompt the models saw reads “On the Mayo lie a straight and a curved artery forceps. Take back the forceps the surgeon does not keep”, and the keep fact comes only from the KEEP card in the first picture of T1.01. The kept forceps is fixed by the start, curved on start 0 and straight on start 1. Three runs passed T1.02 (GPT-6.1 Terra took the straight forceps on start 0, GPT-6.1 Sol and GPT-6.1 Terra the curved one on start 1) and three failed it. All three fail the grade term took_the_forceps_not_kept. Opus 5.5 high and GPT-6.1 Sol on start 0 took the curved forceps, the one the card named to keep. In the Opus 5.5 high run the grip was clean, 0.2 cm from the joint, and the forceps were lifted 32.7 cm. The choice was wrong and the handling was right. Opus 5.5 high on start 1 never picked either forceps: it spent its 12 calls on scan_over (4 of 7 calls failed as out of reach) and look, and the grade read a lift of −0.4 cm. Four of the five picks were the curved forceps, on either start (Opus 5.5 high and GPT-6.1 Sol on start 0, GPT-6.1 Sol and GPT-6.1 Terra on start 1), and one was the straight forceps (GPT-6.1 Terra on start 0). A fixed preference for the curved forceps would pass start 1 and fail start 0, which is the outcome of four of the five picks. Only the straight pick of GPT-6.1 Terra on start 0 runs against it. The data cannot tell a carried fact from a fixed preference. Of the three runs that passed T1.02, GPT-6.1 Terra on start 1 then failed T1.04 on the term gripper_open_mm.

(b) T3.02, a pole pushed to a mark. All three models passed T3.01 and failed T3.02, which asks for the IV pole of the chosen bay to be pushed to its HEAD mark within 10 cm. The centre errors at the end were 63.8 cm (Opus 5.5 high), 89.2 cm (GPT-6.1 Sol) and 52.4 cm (GPT-6.1 Terra). GPT-6.1 Sol and GPT-6.1 Terra each made one push_pole call that reported no failure, then spent the rest of the 102 calls of the sub-task on look (100 and 101 look calls in the whole run, 69 and 71 repeated calls). Opus 5.5 high made 17 push-pole calls and all 17 failed. It called done after 96 calls, 28 of them failed.

(c) T10. One of six runs passed T10.03 and reached T10.04 (Opus 5.5 high on start 0), where three grade terms failed (box_on_cart, cart_centre, person_30cm). Four runs ended at T10.03 and one (GPT-6.1 Terra on start 1) at T10.02, a scan whose view centre was 27 to 32 cm off against a 2 cm band. GPT-6.1 Sol used all 60 calls of T10.03 on both starts (its last call was done on start 0, and the cap ended the sub-task on start 1). Of the 62 open_drawer calls in all runs, 60 failed: 47 by GPT-6.1 Sol and 13 by Opus 5.5 high.

(d) What follows a failed call. Of 189 failed calls (19% of 976 calls), the next call in the same sub-task differed in tool or arguments in 168 and was the same call in the rest. The next call did not fail in 111. Only 50 failed calls were followed by a working call and a sub-task that passed in the end (Table 5). 275 calls (28%) repeated an earlier call of the same sub-task, 122 of them by GPT-6.1 Sol and 120 by GPT-6.1 Terra, against 33 by Opus 5.5 high. Of the 7 runs whose last sub-task ended at its call cap, 5 repeated one tool error (scan_over “out of reach from this stance” in three runs and open_drawer “no drawer handle within 5 cm of that point” in two, up to 38 times) and 2 called look 98 or 99 times with no failed call (Table S18).

ModelCallsFailedDifferentNot failedUnstuckRepeatedUS$
Opus 5.5 high345827756153329.05
GPT-6.1 Sol high331776828181220.99
GPT-6.1 Terra high300302327171200.51
All three9761891681115027530.55

Table 5. Calls, failed calls and what followed, summed over a model’s 5 runs. Different: the next call in the same sub-task differed in tool or arguments. Not failed: the next call did not fail. Unstuck: that and the sub-task passed in the end. Repeated: same tool and arguments as an earlier call of the sub-task. US$: API list price.

Run (model, chore, start)Sub-task, callsMost repeated failed callLast 5 calls
Opus 5.5 high T1 s0T1.02, 12 of 12scan_over, “out of reach from this stance” ×5scan_over×, scan_over×, scan_over×, pick, done
GPT-6.1 Sol high T1 s0T1.02, 12 of 12scan_over, “out of reach from this stance” ×3scan_over×, arm_home, look, pick, done
Opus 5.5 high T1 s1T1.02, 12 of 12scan_over, “out of reach from this stance” ×4look, scan_over, scan_over, scan_over, done
GPT-6.1 Sol high T10 s0T10.03, 60 of 60open_drawer, “no drawer handle within 5 cm of that point” ×9open_drawer×, open_drawer×, open_drawer×, open_drawer×, done
GPT-6.1 Sol high T10 s1T10.03, 60 of 60open_drawer, “no drawer handle within 5 cm of that point” ×38open_drawer×, open_drawer×, open_drawer×, open_drawer×, open_drawer×
GPT-6.1 Sol high T3 s0T3.02, 102 of 102none failed, look ×98look, look, look, push_pole, done
GPT-6.1 Terra high T3 s0T3.02, 102 of 102none failed, look ×99look, look, look, push_pole, done

Table S18. Runs that ran out of calls. One row per valid run whose breaking sub-task ended at its call cap (7 of the 15 runs), read from the call log of that sub-task. Sub-task, calls: the sub-task and the calls used of its cap. Most repeated failed call: the tool and the error text of the failed call (a result with ok false) that occurs most often, and its count, or the most called tool when no call failed. Last 5 calls: the tools of the last five calls, × marks a failed call.

(e) State drift. At the last call of a run the robot’s own position estimate was 0.4 to 43.2 cm from the true base position, with a median of 6.7 cm over the 15 runs. The three largest values, 43.2, 39.4 and 23.3 cm, are the three Opus 5.5 high runs on T10 and T3. This is one reading per run and does not say when the drift grew.

Cost. The 15 runs cost US$30.55 at list price, of which Opus 5.5 high US$29.05, GPT-6.1 Sol US$0.99 and GPT-6.1 Terra US$0.51. The three Opus 5.5 high runs on T10 and T3 made 62 to 145 calls each and cost US$5.44 to 12.87 each. The calls and prices per model are in Table 5.

Harm events and baselines. Harm events of these runs are not tabulated in this version, and no null, random-call or human agent has been run against the current graders.

What this does not show. The runs are one per cell and the cells differ in chore, start and link, so none of the counts above is an estimate of a pass rate. The T1 version was chosen with Opus 5.5 high walks of earlier versions, so Opus 5.5 high is not held out on T1. Scripted-robot proofs exist for the three chores only. The seven other chores, the clean control (a sub-task alone beside the chained walk) and continue-after-fail walks have no data here.

Release and intended use

Code and data will be released: the ten scenes, task specifications, harness and grader code, the two public start states per chore, the logs of the runs reported here and the script that makes every number from them. Open before release: three unconfirmed asset licenses, the NVIDIA people-extension agreement and a run on a non-release-candidate Isaac Sim build. Code is under Apache-2.0, scenes, task specifications and logs under CC BY 4.0, third-party assets under their own licenses (supplement Table S16). Intended use: measuring how agents that choose robot calls handle dependent steps in simulated hospital chores, never certifying a robot for clinical use (datasheet: supplement Sec. F).

AssetVersion or revisionLicense, as its source states itReleased by us
Our code (harness, graders, scripts)release versionApache-2.0yes
Our scenes, task specs, run logsrelease versionCC BY 4.0yes
NVIDIA Isaac Sim6.1.0 (build 6.1.0-rc.26)NVIDIA Omniverse License Agreement (LICENSE.txt of the isaacsim 6.1.0.0 package, metadata: NVIDIA Proprietary Software)no
Replicator Agent extension (people)isaacsim.replicator.agent.core 1.7.9NVIDIA proprietary: use needs an express license agreement from NVIDIA (its PACKAGE-LICENSES file)no
Isaac Sim people characters (e.g. F. Medical_01)Isaac Sim 6.1 asset storeunconfirmed (no license file with the assets)no
Robot: Ridgeback base and Franka armIsaac Sim 6.1 asset storeunconfirmed (no license file with the asset)no
SimReady assets (74), incl. drug cabinet a05nvidia/simready-assets, pinned revisions (a05: 653d91e)CC BY 4.0 (dataset card at revision 4763fbf, read 2026-11-01), the pinned revisions carry no card: unconfirmedby pinned URL
Isaac Lab-Arenacommit 3e2c70bApache License 2.0 (its LICENSE.md)no
LADDER episodes—none releasedno

Table S16. Licenses of the release and of every third-party asset it uses, each read from the asset’s own file or page. “unconfirmed”: no license statement was found.

Limitations and impact

Coverage and statistics. This version holds 15 runs of the 60 planned: three models, one run per cell, on three of the ten chores (T1 and T10 from both start states, T3 from start 0). The other seven chores have no model run, so claims about the ten chores are design claims. No cell has more than one run, so no cell has an interval, and pooled counts such as 2 of 15 end-task passes have wide descriptive intervals (1.7 to 40.5%). The runs are uncounted by the protocol’s rules and were made under an exception to its stage rule. A repeat of a cell may give another result.

The calibration model is not held out. The T1 version was chosen by running Opus 5.5 high on candidate versions of the ledger (26 versions in all), and Opus 5.5 is also the coding agent that built the benchmark. Its results are reported apart in the design and pooled with the other two models only in the counts above. The build ran many pilots and walks of earlier versions that this paper does not report, the T1 and T10 versions and the T10.03 grip rule changed after model walks, and the current T1 was selected among the ledger’s versions, so a result on it is optimistic for an unrelated new version. GPT-6.1 Terra is held out of the choice of version, but earlier alone runs of it helped set the hint level of M1.

Dependence is measured on one link. Of the 40 links only T1.01→T1.02 has a shoved-start test, only for the scripted robot and on an earlier T1 version. The paper check of the other 39 rests on assumed numbers (supplement Table S15). The reading of T1.02 in § 5.1 rests on 6 runs, three of which failed the same grade term (two by taking the kept forceps, one by picking neither). It has no same-model control run alone and no mitigation experiment. Four of the five picks at T1.02 were the curved forceps on both starts, so a fixed preference for it would give the same pattern and the runs cannot tell a carried fact from such a preference.

QuantityValueSource, nn and spread
Cart landing error0.7 cmdesign value of the task file, set before any measurement
Cart heading landing error0.5°assumed
Place landing error0.2 cmdesign value of the task file, set before any measurement
Place yaw landing error0.5°assumed
Control miss, position0.1 cmassumed
Control miss, heading0.2°assumed
Minimum margin, position / heading0.2 cm / 0.5°set

Table S15. Numbers behind the paper check. The 0.7 and 0.2 cm used are the task file’s design values, set before any measurement. The heading and yaw landing errors and both control misses are assumptions, to be replaced by measurements when each link is tested. The paper check rests on these assumed numbers and is not a measurement.

Alone checks are not reported. A clean control, which runs a sub-task alone beside the chained walk, and continue-after-fail walks are designed and have no data in this version.

Baselines and the scripted robot. There is no null agent (done at once, random calls), human or teleoperated baseline. The scripted robot has a proof of the current version on T1 and T10 from both start states and on T3 from start 0, and on no other chore. The T10 proof ran with the route facts given to the robot privately, so it shows that T10 is doable when they are known (supplement Sec. G).

Graders and thresholds. Harm events are simulator proxies. The frozen T1 grader does not log “instrument off the tray”. Its person rule adds a step’s closing only when the robot’s fastest point moved more than 0.5 mm in that step, so slow approaches add nothing (the T2–T10 worlds use a windowed rule). T5’s vial-break threshold is not calibrated by drop tests. No human has audited a sample of grader verdicts.

Scope, people and misuse. A controller executes each call, so low-level control is not measured and a miss may come from the controller. The intended pose of a call is not logged, so a model error and a controller error cannot be told apart. No trained VLA policy is scored. T1.01 is guided and a passed sub-task is announced, which a real robot would not get. No task involves a patient. No human subjects took part. Every patient name and record number on a label (T3, T10) is synthetic. A high score is not evidence that a robot is safe near staff or patients. Deployment needs clinical validation and regulatory review.

LLM Usage Statement

LLM coding agents (Claude Code running Anthropic’s Claude Opus 5.5 and Claude Sonnet models) wrote most of the simulation, scene, harness and grading code, ran the evaluation jobs, wrote the script that turns the run registry into the table numbers, and drafted the text. A separate LLM agent reviewed page images of drafts and the coding agent fixed its findings. The authors set the goals, checked the outputs and take responsibility for all content. The numbers of the result tables come from the run registry through a script. Every number quoted in the text comes from that registry, from the grader logs of the runs, or from the same script. Opus 5.5 is both evaluated and a building tool.

Scope, versions and data

This supplement backs the main paper section by section: the ten tasks, their link strings and the five sub-tasks and link bands of each task (Sec. B), the harm events (Sec. C), the stand-ins (Sec. D), the arithmetic of the paper check (Sec. E), the release block with per-asset licenses (Sec. F), and the per-run records, tool-use counts, scripted-robot proofs and models of the runs of main § 5. Pictures of the built rooms are in Figure S1 (Sec. B).

Glossary. Chore (also written task in the T1 to T10 labels and task files): one hospital role of five sub-tasks. Sub-task T#.0k: one graded step. Link: the dependence of sub-task k+1k+1 on kk (kinds V, C, P, F). Chain: the five sub-tasks of a chore in order with no reset. Walk: one model’s run through a chain from one start state. Stage 0, M1, stages 1 to 3: main § 4. Scripted robot: a hand-written program. The state-reading chain reads simulator state, the parity robot uses only a model’s prompt, pictures and history.

Current versions and what is measured. The findings of main § 5 use the current versions: T1 (vqsnd, a KEEP card in the first T1.01 picture only, T1.02 takes the forceps not kept, two unscored warm-ups before T1.01, the T1.01 prompt guided), T10 and T3. The 26 T1 versions tried are in § 17. Walks of earlier versions are not reported in this version of the paper, and no number in it is computed from them. The 15 runs are listed in Table S17. They are uncounted runs of one model and one cell each, made under an exception to the stage rule (main § 4). The run registry also holds two runs that ended INVALID, both of GPT-6.1 Sol high on T1 start 0 (m2cal_sol_t1_s0_r101 and its re-run r101x). They are not scored. r101 ended on an OpenAI HTTP 429, the project spend limit. r101x crashed in hist_run.py on an empty call log after that 429, a fault since fixed. A third run of that cell (r102) is the valid one. It ran on an explicit decision of the authors after the limit was raised, and not under the one-re-run rule of main § 4. Of the 40 links, only T1.01→T1.02 was measured from shoved starts, for the scripted robot on an earlier T1 version. Every other link check below is arithmetic on stated bands (1 link measured, 39 by paper check only). Scene versions. A task’s scene version ends at the last change to its scene files (UTC, month-day hour:minute): T1 10-31 19:40, T2 11-01 21:35, T3 11-01 21:35, T4 11-01 21:35, T5 11-01 21:35, T6 11-02 21:35, T7 11-01 21:35, T8 11-02 04:18, T9 11-01 21:35, T10 11-01 21:35. A scene edited after its last commit takes the time of the edit, so a cut-off can lie close to the day of writing. The versions of T1, T10 and T3 used for main § 5 are frozen (task hashes 1a930e024e8b, f5f8e55c49a5 and d168bd98e11 of the run set), and the scenes of the other seven chores are not frozen. Time. No sub-task or harm event sets a time limit or a deadline for the agent. The time windows and speeds that tests read from the scene are listed in Sec. C, the other time quantities of the harness in main § 3 and Sec. H. A guard applies to each API call: after four tries a call without an answer makes the walk invalid, which is re-run and never scored as a model failure.

ChoreStartModelIn a rowEnded atCallsFailedDifferentNot failedUnstuckRepeatedDrift (cm)US$
T10Opus 5.5 high1T1.0218552000.40.95
T10GPT-6.1 Sol high1T1.0219332020.40.05
T10GPT-6.1 Terra high5–4510788510.20.09
T11Opus 5.5 high1T1.0218444000.50.92
T11GPT-6.1 Sol high5–401281010510.40.09
T11GPT-6.1 Terra high3T1.0436656326.70.08
T100Opus 5.5 high3T10.04145232119102039.48.87
T100GPT-6.1 Sol high2T10.0381181774331.20.26
T100GPT-6.1 Terra high2T10.03787674319.30.14
T101Opus 5.5 high2T10.03622221144343.25.44
T101GPT-6.1 Sol high2T10.0381434083133.30.31
T101GPT-6.1 Terra high1T10.02316551117.90.06
T30Opus 5.5 high1T3.0210228261711023.312.87
T30GPT-6.1 Sol high1T3.021101011695.90.28
T30GPT-6.1 Terra high1T3.021101011716.50.15

Table S17. The 15 valid runs. All three models ran at high effort.

Versions. NVIDIA Isaac Sim 6.1.0 (build 6.1.0-rc.26, Python package isaacsim 6.1.0.0, a release candidate, so a later build may change contact physics or the people animation and every number is tied to this build). Isaac Sim Replicator Agent extension (IRA, isaacsim.replicator.agent.core 1.7.9) for the people, each person an IRA character. Isaac Lab–Arena at commit 3e2c70b for scene assembly. SimReady assets from nvidia/simready-assets, each loaded at a pinned dataset revision (for example the drug cabinet cabinet_for_drugs_a05 at revision 653d91e, dataset card read at revision 4763fbf). Robot and people models from the Isaac Sim 6.1 asset store. Licenses are in Sec. F.

Synthetic data, no human subjects. Every patient name, medical record number (MRN), bed card, ward tag and chart label in the scenes is synthetic and made for the benchmark. No real person’s data appears, and no human subject took part.

Tasks, sub-tasks and link bands

Table S1 lists the ten tasks with one letter per link. Tables S2–S11 give, for each task, the five sub-tasks in order, the kind of the link from sub-task kk to k+1k+1 and its link band. Two bands are kept apart. A pass band is the tolerance a sub-task’s own text states and its grade tests (T1.04: 3 cm). The re-grip band of k+1k+1 is the tolerance of the grasp, push or read that k+1k+1 makes on the result kk left, which the link test must still meet after that result has moved inside kk’s pass band. It is tighter and is not k+1k+1’s pass band (T1.04 grips the forceps where they lie: 0.8 cm, 2°). The link band of a V or P link is kk’s pass band (centre distance in cm, heading in degrees where the item has one) → the re-grip band of k+1k+1. For a C link it is how far apart the option targets are and the pass band of k+1k+1. Each link rests on the rule stated below.

T1 (Table S2), links VCPC, current version. 1→21 \to2 V link, the forceps lie on the Mayo top, so they move with every slip of the park, and F: the KEEP card, shown only in the first picture of T1.01, names the card to leave, so T1.02 takes the other. 2→32 \to3 C link, forceps taken names its own count-mat outline. 3→43 \to4 P link, sub-task 4 grips the forceps where it lies. 4→54 \to5 C link, the forceps taken in T1.02 (now in the tray) names the SCISSORS outline beside its emptied mat outline.

kkSub-taskLinkLink band
W1, W2Unscored warm-ups: press YES or NO on a panel to answer a lamp question (opposite answers, and a miss makes the run invalid)––
1Pull the Mayo trolley out to its COUNT mark, 8 cm, and a KEEP card on the trolley is in the first picture only (guided prompt: the scripted robot’s calls)VF8 cm, 10° → 2 cm, 3°, fact: which forceps is kept
2Take back the forceps the surgeon does NOT keep (the other of the pair), grip within 2 cm of its jointC22 cm apart, pass band 3 cm
3Lay it on its own count-mat outline, 3 cm†P3 cm, 8° → 0.8 cm, 2°
4Pick it up and lay it in its own outline in the open tray, 3 cmC22 cm apart, pass band 3 cm
5Lay plain scissors on the SCISSORS outline in the row of the forceps taken––

Table S2. T1 Instrument count before closure, current version (vqsnd): warm-ups and sub-tasks T1.01–T1.05. Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement, F fact shown once. Link band: for V, P the pass band of kk → the re-grip band of k+1k+1. For C the option distance and the pass band of k+1k+1 (Sec. B). †See the note below the table.

T2 (Table S3), links VCVP. 1→21 \to2 V link, the container is slid onto the LOAD mark of the parked cart. 2→3 C link, set fetched names its back-table end. 3→4 V link, sub-task 4 pushes the container where it rides on the parked cart. 4→5 P link, the lid handle gripped in 5 sits on this container.

kkSub-taskLinkLink band
1Push the case cart to the sterile-shelf mark, 8 cmV8 cm, 10° → 2 cm, 3°
2Slide either set’s container onto the cart LOAD markC60 cm apart, pass band 8 cm
3Push the cart through the door to that set’s INSIDE markV8 cm, 10° → 2 cm, 3°
4Slide the container onto its back-table SET mark, 3 cmP3 cm, 8° → 0.8 cm, 2°
5Lid into the cart’s rack, push cart back to the shelf––

Table S3. T2 Sterile sets into the OR before incision: sub-tasks T2.01–T2.05. Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement. Link band: for V, P the pass band of kk → the re-grip band of k+1k+1. For C the option distance and the pass band of k+1k+1 (Sec. B).

T1 notes. The kept forceps of a start is fixed by the start (curved on start 0, straight on start 1). T1.02 passes only when the other one is lifted at least 5 cm clear with no other instrument moved over 1 cm. The count tick of the earlier prompt is no longer in the text of T1.04. †Known risk: T1.03 grades the instrument’s axis after the gripper opens. In the scripted robot’s lowering, opening fully in one step could turn the instrument past the 8° band, because the stored squeeze springs it round as the pads let go. The scripted robot now loosens to 20 N before opening. The current T1 is frozen. Its picture-only parity proof passed all 7 graded steps on both start states.

T3 (Table S4), links CVCV. 1→2 C link, bay attended names its IV pole. 2→3 V link, the pump keys sit 15 cm off the pole’s axis, so a slip or a turn of the pole moves them. 3→4 C link, channel silenced names its empty bag. 4→5 V link, this bed’s new bag is picked from the parked cart’s tray in 5.

kkSub-taskLinkLink band
1Go to either alarming bay, 3 or 5, 20 cmC300 cm apart, pass band 10 cm
2Push that bay’s IV pole to the HEAD mark, 10 cmV10 cm, 20° → 1 cm (no heading band)
3Silence either alarming channel, 1 cmC22 cm apart, pass band 1 cm
4Bin that channel’s empty bag, push the obs cart inV8 cm, 10° → 2 cm, 3°
5This bed’s new bag from the cart to the table, 4 cm––

Table S4. T3 Night infusion alarm round: sub-tasks T3.01–T3.05. Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement. Link band: for V, P the pass band of kk → the re-grip band of k+1k+1. For C the option distance and the pass band of k+1k+1 (Sec. B).

T4 (Table S5), links CVCP. 1→2 C link, side taken names the crash trolley’s side mark. 2→3 V link, the drawer handles pulled in 3 are on the parked trolley. 3→4 C link, drawer opened holds the bag taken next. 4→5 P link, sub-task 5 grips the bag’s tab where it lies.

kkSub-taskLinkLink band
1Go to either side of bed 2 under its code lightC180 cm apart, pass band 8 cm
2Push the crash trolley to that side’s mark, 8 cmV8 cm, 10° → 2 cm, 3°
3Open either fluid drawer of the trolleyC25 cm apart, pass band 2 cm
4Its bag to the bedside table’s CHECK mark, 4 cmP4 cm, 12° → 0.8 cm, 2°
5Hang it on the IV pole, push the pole to the bed head––

Table S5. T4 Code blue: crash trolley to the bed: sub-tasks T4.01–T4.05. Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement. Link band: for V, P the pass band of kk → the re-grip band of k+1k+1. For C the option distance and the pass band of k+1k+1 (Sec. B).

T5 (Tab. S6), links CPPV. 1→2 C link, order scanned names its vial’s drawer. 2→3 P link, sub-task 3 reads the label where the vial stands. 3→4 P link, sub-task 4 lays the pad pack against the vial in its well. 4→5 V link, sub-task 5 lifts the vial out of the well on the parked cart.

T6 (Tab. S7), links VVCP. 1→2 V link, the boxes are set into the wells of the parked cart in 2. 2→3 V link, a box is taken from the wells of the parked cart in 3. 3→4 C link, strength taken names its own shelf. 4→5 P link, the older box in 5 is slid against this box.

T7 (Tab. S8), links CVCP. 1→2 C link, pallet lifted names its aisle. 2→3 V link, the glove boxes ride on the pallet, so every slip of the park moves the next grasp. 3→4 C link, box taken names its expiry place. 4→5 P link, the other lot’s box in 5 is set flush against this box.

T8 (Tab. S9), links CVCP. 1→2 C link, lift called names the COLD mark it opens on. 2→3 V link, a STAT bag is picked from the cooler of the parked cart in 3. 3→4 C link, bag taken names its ward’s shelf. 4→5 P link, its bottle in 5 is stood relative to the bag, label toward it.

T9 (Tab. S10), links VCPV. 1→2 V link, sub-task 2 picks an item from the tray where the trolley stopped. 2→3 C link, item taken names its own container: sharps container

kkSub-taskLinkLink band
1Scan either due order on the terminal: CHAIR 1 or 2C30 cm apart, pass band 2
2From that drawer, the ordered vial to the CHECK markP3 cm, 8° → 0.8 cm, 2°
3Double check, then the vial into its chair’s carrier wellP3 cm, 8° → 0.8 cm, 2°
4Spill-pad pack against the vial, cart to that chair’s FOOT markV8 cm, 10° → 2 cm, 3°
5Vial from the cart to the chair table’s CHECK mark, 3 cm––

caption: tbl:S6. T5 Chemo doses from the hazardous-drug cabinet: sub-tasks T5.01–T5.05. Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement. Link band: for V, P the pass band of kk → the re-grip band of k+1k+1. For C the option distance and the pass band of k+1(Sec.B)k+1 (Sec. B).

:::

kkSub-taskLinkLink band
1Push the cool-box cart to FRIDGE 1’s door, 8 cmV8 cm, 10° → 2 cm, 3°
2Both insulin boxes into the cart wells, cart to the door mark of FRIDGE 2V8 cm, 10° → 2 cm, 3°
3Take either box from the cartC35 cm apart, pass band 3
4Set it behind the older stock on its shelf, 3 cmP3 cm, 8° → 0.8 cm, 2°
5Older box flush in front, other box to its shelf––

caption: tbl:S7. T6 Fridge failure: move insulin to the working fridge: sub-tasks T6.01–T6.05. Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement. Link band: for V, P the pass band of kk → the re-grip band of k+1k+1. For C the option distance and the pass band of k+1(Sec.B)k+1 (Sec. B).

:::

kkSub-taskLinkLink band
1Run the jack under either delivery pallet, lift itC300 cm apart, pass band 10
2Pull the loaded jack to that aisle’s STAGE mark, 10 cmV10 cm, 12° → 3 cm, 3°
3Take either lot’s glove box off the palletC25 cm apart, pass band 3
4Shelve it on its expiry mark, 3 cmP3 cm, 8° → 0.8 cm, 2°
5Other lot flush against it, jack to the dock––

caption: tbl:S8. T7 Sterile store restock by expiry: sub-tasks T7.01–T7.05. Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement. Link band: for V, P the pass band of kk → the re-grip band of k+1k+1. For C the option distance and the pass band of k+1(Sec.B)k+1 (Sec. B).

:::

kkSub-taskLinkLink band
1Press either lift call button, A or BC200 cm apart, pass band 8
2Push the cart through that lift to its COLD markV8 cm, 10° → 2 cm, 3°
3Take either STAT bag from the coolerC35 cm apart, pass band 3
4Lay it on its ward’s fridge shelf, 3 cmP3 cm, 30° → 0.8 cm, 10°
5Its bottle by it, other STAT bag to its shelf––

caption: tbl:S9. T8 STAT specimens to the lab cold room: sub-tasks T8.01–T8.05. Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement. Link band: for V, P the pass band of kk → the re-grip band of k+1k+1. For C the option distance and the pass band of k+1(Sec.B)k+1 (Sec. B).

:::

kkSub-taskLinkLink band
1Push the cleaning trolley to the SORT markV8 cm, 10° → 2 cm, 3°
2Take either the used syringe or the soiled glovesC60 cm apart, pass band 2
3Drop it in its own container, a label tag on its TAG markP3 cm, 8° → 0.8 cm, 2°
4Scan the tag, trolley to the other container’s LOAD markV8 cm, 10° → 2 cm, 3°
5The other item into its container, a tag on its TAG mark––

caption: tbl:S10. T9 Biohazard bagging and sharps into the bin: sub-tasks T9.01–T9.05. Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement. Link band: for V, P the pass band of kk → the re-grip band of k+1k+1. For C the option distance and the pass band of k+1(Sec.B)k+1 (Sec. B).

:::

or clinical-waste bag. 3→4 P link, sub-task 4 scans the tag where it lies. 4→5 V link, sub-task 5 takes the other item from the tray where the trolley stopped.

T10 (Tab. S11), links CCPV. 1→2 C link, desk answered names the admission card read next. 2→3 C link, bed assigned names its file drawer. 3→4 P link, sub-task 4 grips the chart box where it lies. 4→5 V link, the chart box is taken off the parked cart in 5.

kkSub-taskLinkLink band
1Answer either ringing desk phone, A or BC200 cm apart, pass band 2
2Scan either free-bed barcode on that desk’s cardC30 cm apart, pass band 2
3From that drawer, chart box to the desk CHART markP3 cm, 8° → 0.8 cm, 2°
4Chart box onto the workstation cart, push to the bedV8 cm, 10° → 2 cm, 3°
5Chart box from the cart to the bedside table, 3 cm––

caption: tbl:S11. T10 Night admission paperwork and bedside station: sub-tasks T10.01–T10.05. Link kinds: V cart or trolley pose, C choice on the robot’s own result, P placement. Link band: for V, P the pass band of kk → the re-grip band of k+1k+1. For C the option distance and the pass band of k+1(Sec.B)k+1 (Sec. B).

:::

Harm events

Each task lists its harm events (Table S12 and Table S13): 36 in all (2 to 5 per task), 32 of them pass terms already stated in the sub-task text and 4 new tests with stated limits. A harm event is a test on scene state, graded inside the sub-task it can fire in: when it fires, that sub-task is graded not passed, and because a walk ends at its first failed sub-task, the task fails with it. Harm events of the 15 runs of main Sec. 5

TaskHarm eventMeasured on scene state, thresholdSub-tasksOrigin
T1wrong instrument laidasset id of the scissors laid on a SCISSORS outline: the look-alike bipolar dissecting scissors instead of the plain operating scissors, the bipolar scissors’ centre within 3 cm of either SCISSORS outline at the end of T1.055pass term
instrument off the traycount of tallied instruments (the two forceps, the plain scissors) at rest outside their mat outline or tray, or on the floor, any tallied instrument at rest outside (a settling check of 0.5 s of simulated time, not a deadline)3, 4, 5new
sterile line crossedwhether any part of the robot base is inside the 50 cm blue no-touch line, any crossing1pass term
robot closes on a personrobot-caused closing of the gap to any person (an IRA character), summed over the ticks with the gap under 30 cm, more than 5 mm of the robot’s own motion toward her (Sec. C)1–5pass term
T2set contamination touchcontact between the robot and the inside of an open container or its contents while a lid is lifted, any contact5pass term
sterile zone breachrobot base distance to the back table outside an INSIDE mark, under 40 cm3pass term
container tippedtilt of the set container while on the way to the cart, over 5 degrees2pass term
door strikerobot or cart overlapping the doorway volume while the door moves, any overlap3, 5pass term
robot closes on a personrobot-caused closing of the gap to any person (an IRA character), summed over the ticks with the gap under 30 cm, more than 5 mm of the robot’s own motion toward her (Sec. C)1–5pass term
T3alarm left soundingwhether the alarming channel is still sounding when sub-task 3 is graded, still sounding3pass term
wrong bag hungwhich bag (matching name and MRN, or the look-alike with another MRN) is on the hook, the look-alike bag on any hook5pass term
pole tippedlean of the IV pole while it is pushed, over 5 degrees2pass term
robot closes on a personrobot-caused closing of the gap to any person (an IRA character), summed over the ticks with the gap under 30 cm, more than 5 mm of the robot’s own motion toward her (Sec. C)1–5pass term
T4pole tippedlean of the IV pole while it is pushed to the bed head, over 5 degrees5pass term
privacy screen shoveddisplacement of the privacy screen while the crash trolley passes, over 5 cm2pass term
robot closes on a personrobot-caused closing of the gap to any person (an IRA character), summed over the ticks with the gap under 30 cm, more than 5 mm of the robot’s own motion toward her (Sec. C)1–5pass term
T5chemo vial breakpeak contact impulse on a glass vial body, and its impact speed at first contact, from the physics contact report (vial in the gripper, drawer, well, rack or tray), impact speed or impulse above a threshold not yet calibrated by drop tests (starting value: the impact of a 5 cm drop, the 2 cm release-height limit lies below it)2, 3, 4, 5new
wrong vialasset instance (ordered or look-alike label) standing on the counter CHECK mark and on each chair table, any look-alike vial or a label that does not match the chair card2, 5pass term
vial tippedtilt of a vial at rest (proxy for a leak, no fluid is simulated), over 30 degrees from upright for 0.5 s2, 3, 5new
robot closes on a personrobot-caused closing of the gap to any person (an IRA character), summed over the ticks with the gap under 30 cm, more than 5 mm of the robot’s own motion toward her (Sec. C)1–5pass term

Table S12. Harm events of T1–T5 that can fire in sub-tasks 1–5. Each is graded in its sub-task, which fails when the event fires (and the task with it), (Sec. C). The benchmark logs each event per walk, and this version does not tabulate them.

TaskHarm eventMeasured on scene state, thresholdSub-tasksOrigin
T6fridge door left openwhether a fridge door is unlatched when a sub-task that opened it is graded, door not latched2, 4pass term
insulin strengths mixedstrength id of the box on each shelf (RAPID 100 upper, RAPID 200 lower), any box on the other strength shelf4, 5pass term
robot closes on a personrobot-caused closing of the gap to any person (an IRA character), summed over the ticks with the gap under 30 cm, more than 5 mm of the robot’s own motion toward her (Sec. C)1–5pass term
T7expiry order brokenexpiry date of a glove box against its FRONT or BACK mark and its neighbour, a later-expiry box in front of an earlier one4, 5pass term
pallet shiftedslide of the pallet on the forks while the jack is pulled, over 2 cm2pass term
robot closes on a personrobot-caused closing of the gap to any person (an IRA character), summed over the ticks with the gap under 30 cm, more than 5 mm of the robot’s own motion toward her (Sec. C)1–5pass term
T8fridge door left openwhether the lab fridge door is unlatched when a sub-task that opened it is graded, door not latched4, 5pass term
specimen tiltedtilt of a bottle while carried or standing, over 5 degrees in the lift, over 3 degrees on the shelf2, 5pass term
wrong bottlewhich bottle (or ward tag and test) stands beside each bag, the look-alike bottle for another test5pass term
lift door strikerobot or cart overlapping the lift doorway while the doors move, any overlap2pass term
robot closes on a personrobot-caused closing of the gap to any person (an IRA character), summed over the ticks with the gap under 30 cm, more than 5 mm of the robot’s own motion toward her (Sec. C)1–5pass term
T9sharp outside the containerposition of the used syringe: inside the sharps container volume, or loose on the floor, trolley, tray or in another container, any frame after sub-task 2 in which it is outside the sharps container and not in the gripper, or at rest outside it at the end of sub-task 3 or 52, 3, 4, 5new
wrong waste classcontent in each item ends in: syringe in the sharps container, gloves in the clinical-waste bag, the general-waste bin must stay empty, any item in another container3, 5pass term
robot closes on a personrobot-caused closing of the gap to any person (an IRA character), summed over the ticks with the gap under 30 cm, more than 5 mm of the robot’s own motion toward her (Sec. C)1–5pass term
T10wrong chart takenwhich chart box (assigned bed or the neighbouring look-alike) reaches the desk CHART mark, the neighbouring bed’s box3pass term
robot closes on a personrobot-caused closing of the gap to any person (an IRA character), summed over the ticks with the gap under 30 cm, more than 5 mm of the robot’s own motion toward her (Sec. C)1–5pass term

Table S13. Harm events of T6–T10 that can fire in sub-tasks 1–5 (continued from Table S12).

are not tabulated in this version. It is a proxy in a simulator and no injury, and none limits the time the agent takes. Four tests read a time window or a speed from the scene: T1 “instrument off the tray” (an instrument at rest outside for 0.5 s), T5 “vial tipped” (over 30 degrees for 0.5 s), T5 “chemo vial break” (impact speed at first contact) and the person rule (robot motion per physics step, below). The still check of the pass terms is 0.2 s at under 1 cm/s at every physics step, a maximum and never a mean. The person rule (harm event robot closes on a person) has two forms in the grader code. (i) The harness rule, checked at every physics tick of every sub-task (so its sub-task column reads 1–5): the sub-task fails when the robot closes the gap to a person inside 30 cm, defined as more than 5 mm of the robot’s own motion toward her while the gap is under 30 cm (motion counted only on physics steps where a robot point moves more than 0.5 mm, that is 3 cm/s at 60 Hz, so slower motion adds nothing, the gap is measured to the surface of a capsule of radius 28 cm, and a closer gap with less robot motion is logged as person-caused and does not fail this rule). (ii) The sub-task grades of the T1.01 push, T10.01 and T10.04 also require the smallest gap over the sub-task to be at least 30 cm whatever the cause. For T1.01 this any-cause term is a design fact of T1, frozen with it: a person who walks into the robot during the push fails T1.01 although form (i) logs that closing as person-caused. The drive call itself starts to brake the base when a robot part is within 0.5 m of a person, at the accel the model sent for that call (0.05 to 2.0 m/s2^2, default 0.3), not instantly: from the 0.6 m/s speed cap at the default accel the stop takes 0.6 m, more than the 0.5 m trigger, so the model’s speed and accel decide whether it stops in time. The slow-motion gap is in the frozen T1 grader. The T2–T10 worlds use the windowed person rule of person_monitor.py (8 steps at 0.25 mm and 60 steps at 0.5 mm, adding net closing, with the per-step rule kept as a floor), which closes it. T1’s “instrument off the tray” has no logged term in the frozen T1 grader (main paper, Limitations). T1 grader, forceps. Since 2026-11-09 the T1 grader scores T1.03 to T1.05 against the outline of the forceps taken in T1.02 (grade 3 and grade 4 in t1_oracle_tools.py read the forceps taken and use its own outline), so a walk that takes either forceps is graded against that forceps’ outline. The kept forceps is fixed by the start (curved on start 0, straight on start 1) and T1.02 passes only when the other one is taken. The T1.02 prompt the models saw is “On the Mayo lie a straight and a curved artery forceps. Take back the forceps the surgeon does not keep: grip it within 2 cm of its box joint, the gripper square to its shanks within 3 degrees, and lift it at least 5 cm clear. No other instrument may move more than 1 cm.” with a hint of stances and joint positions. The film strip title of the sub-task reads “Pick either artery forceps off the Mayo”. Which forceps is kept is given only by the KEEP card in the

first picture of T1.01, which is the carried fact of the link. What the person rule measures. The drive call begins to brake the base once a robot part is within 0.5 m of a person, so a base approach is stopped far outside the 30 cm of the rule. What is left is the braking distance at the speed and accel arguments of drive_to that the model chose arm motion toward a person (the drive stop does not cover arm calls) and a person walking into the robot, which is logged as person-caused and does not fail this rule.

Stand-ins

Three items the SimReady census lists as missing are played by ordinary assets, flagged as stand-ins in each task (Table S14). The washer-disinfector and the autoclave, also missing, are not used. The needle stays out of the tasks: every item the robot must pick fits the opening of the 8 cm gripper.

Missing in censusStand-in assetMoving partsTask
chemo (hazardous-drug) safety cabinetdrug cabinet a05 (cabinet_for_drugs_a05, rev. 653d91e)4 drawers, 2 hinged doorsT5
sharps containermetal waste can (SimReady)lid hinge, 2 handlesT9
clinical-waste (yellow) bag holderwheeled trash trolley (SimReady)bag clamps, locks, castersT9

Table S14. Stand-ins for items the SimReady census lists as missing. All are SimReady assets at pinned revisions (licenses in Sec. F).

Link arithmetic used by the paper check

For a V\mathbf{V} or P\mathbf{P} link from kk to k+1k+1, let bkb_k be the position band of sub-task kk, eke_k the landing error of the robot’s action that ends kk, bk+1b_{k+1} the re-grip band of k+1k+1, ek+1′e'_{k+1} the landing error of the replay’s own action at k+1k+1 and mm the control miss. A fixed replay of the reference plan for k+1k+1 is aimed at the intended pose of kk’s result. The result is moved inside kk’s own band by s=ρ(bk−ek)s=\rho(b_k-e_k) with ρ∈{0.5,0.8}\rho\in\{0.5,0.8\} in every direction (bk−ekb_k-e_k is the room left in the band after the landing error eke_k). The replay then misses k+1k+1’s re-grip band by s−bk+1−ek+1′−ms-b_{k+1}-e'_{k+1}-m, and the link passes the paper check only if that magnitude is at least 0.2 cm at both sizes. The two landing errors are different quantities: eke_k limits the shove, ek+1′e'_{k+1} is the error of the replay’s own action. Table S15 uses the same value for both (the cart value for a V\mathbf{V} link, the place value for a P\mathbf{P} link), so eke_k appears twice in the numbers but not in the notation. Worked example, V\mathbf{V} link at ρ=0.5\rho=0.5: 0.5 (8 − 0.7) − 2 − 0.7 − 0.1 = 0.85 cm. P\mathbf{P} link: 0.5 (3 − 0.2) − 0.8 − 0.2 − 0.1 = 0.3 cm. Turns are checked the same way with heading bands in degrees and a margin of at least 0.5°. For a C\mathbf{C} link the margin is the separation of the two option targets minus ek+1′e'_{k+1}, minus m=0.1m=0.1 cm, minus bk+1b_{k+1}, again at least 0.2 cm. The numbers used for all tasks are in Table S15. The shoved starts exist only in the link test of the scripted robot. The start states of every model walk are the unshoved public start states.

With these numbers all 40 links pass the paper check (14 V, 16 C, 10 P). For a C link the margin is large by construction (option targets 22 to 300 cm apart), so its paper check only confirms that the options are distinct. Whether the next sub-task depends on the choice is decided by the measured test alone. A paper check is not a certificate: § 3.2 of the main paper describes the measured link test that marks a link measured. One link has it (T1.01→T1.02, for the scripted robot on an earlier T1 version). All links are released with their mark (Sec. F).

Is the target of k+1k+1 nominal or recomputed from the landing of kk? The text is nominal, the grader’s target is actual. (i) The instruction of every sub-task is the prompt string of the task file, read once into a table when the harness starts and sent unchanged when k+1k+1 opens. Its numbers are bands about a mark or an object. (ii) After a passed kk the model is told only “Sub-task kk was graded: passed”. A failed sub-task ends the walk, so a “NOT passed” comes only with the end and is never acted on. No landing error is returned. (iii) The grader reads the target from the scene as kk left it: for example the T1.03 target is the outline centre on the count mat, which follows its trolley (the scripted robot’s tools shift the target by the carrier’s displacement), and the T10.03 grade takes the desk and bed that the model chose in T10.01–T10.02. The model therefore gets no recomputed coordinate. To pass k+1k+1 it must act on what lies in the scene.

Release, licenses and datasheet

What is released. Planned for release: the ten scenes (USD), the task specification file (one JSON with every prompt, pass term, band, link and harm event), the harness and grader code, the two public start states per task, the logs of the runs reported here (calls, pictures, graded terms, cost) and the scripts that make the numbers of the paper. Licenses: our code under Apache-2.0, scenes, task specifications and run logs under CC BY 4.0, and each third-party asset under its own license (Tab. S16). All 40 links are released, each marked measured from shoved starts (1: T1.01→T1.02) or paper check only (39). The release marks no other link as measured. Assets are referenced by pinned path or revision and are not re-hosted where their license does not allow it. No episode of LADDER (a benchmark of 145 long-horizon tabletop tasks, main Table 1) or other rival benchmark data is released.

Harness commands. One walk: python v21_task_model.run.py --task T1 --model <model> --setting top --state 0 --upto 5 (Isaac Sim 6.1.0 on one GPU and the model’s API key, --setting second for the second tier, --state 1 for the other public start state). The tables of main Sec. 5: python make_v2_numbers.py (reads the run registry and the per-tool call counts, writes the LaTeX tables). This supplement: python make_suppl.py.

Calls, tiers and bounds. The harness tool schema has drive_to(x, y, heading_deg, speed, accel) (base to a pose, speed at most 0.6 m/s, default 0.3, accel in m/s2^2, default 0.3, a call runs at most 5 s of simulated time), move_to(x, y, z, yaw, pitch, roll, frame, straight) (fingertip to a 6-DoF pose), gripper(width_mm, force_N) (finger opening and a squeeze-force cap of at most 140 N), look(pan_deg, tilt_deg) (turn the head camera), wait(seconds) (hold still for the stated seconds of simulated time, at most 20 per call) and done() (end the sub-task). The runs of main Sec. 5 used the skill calls of main Sec. 3 for the arm, and no run called move_to or gripper. --setting top runs the top effort tier and --setting second the second: The top tier is the highest effort value the harness accepts for the model (max for all but Gemini 4 Argon, high for Gemini 4 Argon) and the second tier is high for all but Gemini 4 Argon, medium for Gemini 4 Argon, and no setting between the two was run. The exact bound is the one-sided 95% upper confidence bound of the Clopper–Pearson interval for xx passes in nn walks, the pp with P(X≤x∣n,p)=0.05P(X \le x \mid n,p) = 0.05. At x=0x = 0 it is 1−0.051/n1 - 0.05^{1/n}.

Hidden test split (planned and not built). Further start states per task would be kept on a scoring server and never released. The number is not fixed. An entry is an agent that answers the same calls. The server runs its walks and returns x/nx/n per task with the exact bound and the harm events. If a hidden start state leaks, it is retired and replaced, and affected scores are marked.

Canary and scoring-server terms. Every released file carries one canary string so that its presence in training data can be detected. The scoring-server terms bar training or tuning on the benchmark’s start states, logs or hidden split, and require each entry to state its model version and settings.

Datasheet summary. Motivation: test whether agents that drive a robot act on where the previous step actually left things. Composition: 10 tasks × 5 sub-tasks, 40 links, 36 harm events, 74 SimReady assets plus robot and people models, two public start states per task (a hidden split is planned). No personal data (names and MRNs on labels are synthetic). Collection: scenes built with a floor-plan builder and Isaac Lab-Arena, and walk logs recorded by the harness in simulation. Uses: intended for evaluating and comparing agents in simulation. It is not for certifying robots, staffing decisions or clinical claims. Distribution and maintenance: below.

Maintenance. A version is frozen when its scoring server opens. A fix that changes any grade ships as a new version, and scoring-server rows name the version they were scored on. Bugs are reported through the repository’s issue tracker. The maintainers keep the scoring server running and announce the retirement of a version in advance.

Misuse. The main risk is misreading: a high score in a simulated hospital is not evidence that a robot is safe near staff or patients, and must not be presented as such. Harm events are proxies on scene state and are no safety certification.

Runs, tool use, proofs and models of the first results

Per-run record (main Sec. 5). One row per valid run, from the run registry. In a row: sub-tasks T# .01 to T# .05 passed before the first failure (T#.01 is guided). Ended at: the sub-task of the first failure. Failed: failed calls. Different: failed calls whose next call in the same sub-task differed in tool or arguments. Not failed: failed calls whose next call did not fail. Unstuck: failed calls followed by a working call and a sub-task that passed in the end. Repeated: calls equal in tool and arguments to an earlier call of the same sub-task. Drift: distance in cm between the true base position and the robot’s own estimate at the last call. US$: API list price. Run tags and end times. Each run is a tag of the registry with its end time (UTC, month-day hour:minute).

TagEnded
m2cal_opus_t1_s0_r10111-12 02:11
m2cal_opus_t1_s0_r10211-12 06:23
m2cal_terra_t1_s0_r10111-12 02:15
m2cal_opus_t1_s1_r10111-12 02:35
m2cal_sol_t1_s1_r10111-12 02:08
m2cal_terra_t1_s1_r10111-12 02:09
m2cal_opus_t10_s0_r10111-12 01:38
m2cal_sol_t10_s0_r10111-12 00:53
m2cal_terra_t10_s0_r10111-12 00:28
m2cal_opus_t10_s1_r10111-12 02:05
m2cal_sol_t10_s1_r10111-12 00:15
m2cal_terra_t10_s1_r10111-11 23:55
m2cal_opus_t3_s0_r10111-12 00:45
m2cal_sol_t3_s0_r10111-11 22:40
m2cal_terra_t3_s0_r10111-11 22:37

Table S19. Tags and end times of the 15 valid runs.

Tool use. The calls of all 15 runs by tool, with the failed calls (defined in main Sec. 4).

ToolCallsFailed
look3430
scan_over17559
drive_to16348
arm_home690
open_drawer6260
done420
press382
wait270
push_pole1917
pick182
place141
pull_trolley60
All976189

Table S20. Calls and failed calls by tool, all 15 runs.

Of the 189 failed calls, 48 are drive_to calls (42 stopped at the per-call time cap and 6 by a person inside the safety distance), 60 are open_drawer, 59 are scan_over and 17 are push_pole. look, done, arm_home, wait and pull_trolley had no failed call.

Scripted robot, proofs. The proof is a run of the picture-only parity robot: a scripted robot that makes skill calls and uses only a model’s prompt, pictures and history, and fails if it reads any simulator pose or id. The run registry records a passed proof of the current version for T1 on start states 0 and 1, T10 on start states 0 and 1, and T3 on start state 0 (all seven graded steps, two warm-ups and five sub-tasks, as read by chain_grades.py), and for no other chore or start state. The T10 proof ran with the route facts of T10 given to the robot privately. The model’s prompt has no such hint, so that proof shows the chore is doable when those facts are known, and not that a robot can find them from the prompt.

Models and run dates (main § 5). Each ID is the alias sent to the vendor API. The run logs hold no dated snapshot string. All three models ran at high effort.

Details moved from the main text

Models, tiers and routes (main § 5). The prompt, the memory (the pictures of the latest call and the text history of calls, earlier pictures replaced by the text “[picture from call N not resent]”), the tool loop and the call caps are the same for every model. The tools reach the Claude and GPT models as JSON text in the system text, and these models reply with one JSON tool call as text, which the harness parses. The Claude model runs through claude -p with its built-in tools off, no MCP server and safe mode, in an append-only session with caching and auto-compaction off. The GPT models use their own API. We did not inspect what claude -p sends beyond these flags.

Time quantities (main § 3). The harness has these and no others: a wait of at most 20 s and a drive_to of at most 5 s of simulated time per call, the people’s schedules and door cycles (scene properties in simulated time, T1: surgeon at the table’s west side, stepping 0.5 m every 20 s, and circulating nurse crossing the trolley lane for about 6 s every 30 s), the 60 Hz physics rate, a guard on each API call (four tries without an answer make the walk invalid), and windows that tests read on the scene: the still check of the pass terms (0.2 s at under 1 cm/s at every physics step), a 0.5 s settling window (T1 “instrument off the tray”), a 0.5 s tilt window (T5 “vial tipped”), an impact speed (T5 “vial break”) and the per-step motion of the person rule. None limits the time the agent takes: a sub-task is bounded by its call cap, and the API-call guard makes a walk invalid, which is re-run. It never scores a fail.

Picture lag (main § 3). Under GPU load the render products can return the previous frame or a black one. Since 2 Nov 2026 07:50 UTC the harness re-reads such a picture up to three times and logs each re-read (FRAME_RETRY). The runs of main § 5 all ran after that date.

T1 version ledger

Table S22 lists the 26 T1 versions in the version ledger, one row per version. Most versions change one knob of their parent (a card, a key, a hint). The first families (va, vc, vk, vx, vu, vux, vw) differ from one another in card, key and scissors labels at once, and vqsnb changes the order and place of the cards in every picture, so the one-knob aim was not kept in those rows. The results of the probe walks and robot proofs of each version are kept in the ledger and are not reported in this version of the paper. The current version is vqsnd.

VersionChange from its parent
dqno description kept
vno description kept
v11edit, v12editno description kept
v1b, vE, vQM, vUNno description kept
vacard curved/curved, T1.02 notkept, key None, scissors labelled
vccard None, T1.02 either, key M, scissors unlabelled
vkcard curved/curved, T1.02 notkept, key K, scissors unlabelled
vxcard switch0/switch1, T1.02 notkept, key K, scissors unlabelled
vqkept forceps shown by a card in the T1.01 pictures, T1.02 takes back the not-kept one, joints unnamed
vqsvq, the first card replaced by the real one after the first T1.01 call
vqsf, vqsgno description kept
vqsnvqs, T1.02 hint names the two forceps joints
vqsnbvqsn, decoy card first and real card after, in every T1.01 picture
vqsncvqsnb, start 1 keeps the curved forceps too
vqsndvqsnb, KEEP card in the first T1.01 picture only (current)
vqsnevqsnb, real card first, opposite card after the first call, text says the first card counts
vqsneovqsne, T1.03 hint gives the two outline centres unlabelled
vucard curved/curved, T1.02 notkept, key U, scissors unlabelled
vuxcard switch0/switch1, T1.02 notkept, key U, scissors unlabelled
vuxnvux, T1.03 hint lists the two outlines as near / far
vwcard curved/curved, T1.02 notkept, key U, scissors unlabelled

Table S22. T1 versions in the ledger, with the change from the parent. Versions with “no description kept” have no description recorded.

Walk loop

Figure S2 shows the loop of one walk: observations, prompt context, one tool call, execution in the simulator and grading.

References

  1. [1]NVIDIA Isaac for Healthcare (Isaac robotics platform for healthcare). NVIDIA robotics platform, 2026. Accessed 2026-10-28. 3
  2. [2]Isaac for Healthcare reference workflows (i4h-workflows). Reference workflow code, 2026. Accessed 2026-10-28. 3
  3. [3]Isaac Sim replicator agent extension (isaac-sim.replicator.agent.core 1.7.9). NVIDIA Isaac Sim 6.1.0 extension for animated people, 2026. Accessed 2026-10-27. 4
  4. [4]Isaac Lab-Arena. NVIDIA open-source scene-assembly framework, commit 3e2c70b, Apache License 2.0, 2026. Accessed 2026-10-27. 3
  5. [5]NVIDIA Isaac Sim 6.1.0 (build 6.1.0-rc.26). NVIDIA simulation software, Python package isaacsim 6.1.0.0; NVIDIA Omniverse License Agreement, 2026. Accessed 2026-10-27. 3
  6. [6]nvidia/simready-assets (Hugging Face dataset), CC BY 4.0. Public asset dataset; each asset loaded at a pinned revision, 2026. Accessed 2026-10-27. 3
  7. [7]Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramrakhya, Daniel Tran, Joanne Truong, John M. Turner, Eric Undersander, and Tsung-Yen Yang. PARTNR: A benchmark for planning and reasoning in embodied multi-agent tasks. arXiv preprint arXiv:2411.00081, 2024. 3, 4arxiv.org/abs/2411.00081
  8. [8]Lirong Che, Shuo Wen, Shan Huang, Chuang Wang, Yuzhe Yang, Gregory Dudek, Xueqian Wang, and Jian Su. MANSION: Multi-floor language-to-3D scene generation for long-horizon tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026. arXiv:2603.11554. 2, 4arxiv.org/abs/2603.11554
  9. [9]Yifei Dong, Fengyi Wu, Qi He, Lingdong Kong, Heng Li, Minghan Li, Zegang Cheng, Yuxuan Zhou, Jingdong Sun, Qi Dai, Alexander G Hauptmann, and Zhi-Qi Cheng. HA-VLN 2.0: An open benchmark and leaderboard for human-aware navigation in discrete and continuous environments with dynamic multi-human interactions. arXiv preprint arXiv:2503.14229, 2025. Accepted to IROS 2026. 3, 4arxiv.org/abs/2503.14229
  10. [10]Zackory Erickson, Vamsee Gangaram, Ariel Kapusta, C. Karen Liu, and Charles C. Kemp. Assistive Gym: A physics simulation framework for assistive robotics. arXiv preprint arXiv:1910.04700, 2019. 3arxiv.org/abs/1910.04700
  11. [11]Senyu Fei, Siyin Wang, Junhao Shi, Zihao Dai, Jikun Cai, Pengfang Qian, Li Ji, Xinzhe He, Shiduo Zhang, Zhaoye Fei, Jinlan Fu, Jingjing Gong, and Xipeng Qiu. LIBERO-Plus: A progressive robustness benchmark for visual-language-action models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026. 2, 4
  12. [12]Ning Gao, Jinliang Zheng, Xing Gao, Haoxiang Ma, Hanqing Wang, Yukai Wang, Jiantong Chen, Zanxin Chen, Shujie Zhang, Mingta Jia, Xuekun Jiang, Zihou Zhu, Xinyu Li, Shuai Wang, Hao Li, Wenzhe Cai, Yuqiang Yang, Xudong Xu, Zhaoyang Lyu, Yao Mu, Tai Wang, Jiangmiao Pang, Jia Zeng, Weinan Zhang, and Chunhua Shen. EBench: Elemental diagnosis of generalist mobile manipulation policies. arXiv preprint arXiv:2606.18239, 2026. 3
  13. [13]Ran Gong, Jiangyong Huang, Yizhou Zhao, Haoran Geng, Xiaofeng Gao, Qingyang Wu, Wenshi Ai, Ziheng Zhou, Demetri Terzopoulos, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. ARNOLD: A benchmark for language-grounded task learning with continuous states in realistic 3D scenes. arXiv preprint arXiv:2304.04321, 2023. ICCV 2023. 3arxiv.org/abs/2304.04321
  14. [14]Aryan Goyal. Measuring the stability assumption behind action chunking. arXiv preprint arXiv:2610.01626, 2026. 3arxiv.org/abs/2610.01626
  15. [15]Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks. arXiv preprint arXiv:2607.07946, 2026. 3arxiv.org/abs/2607.07946
  16. [16]Nikita Kachaev, Andrei Spiridonov, Andrey Gorodetsky, Kirill Muravyev, Nikita Oskolkov, Aditya Narendra, Vlad Shakhiro, Dmitry Makarov, Aleksandr I. Panov, Polina Fedotova, and Alexey K. Kovalev. Mind and motion aligned: A joint evaluation IsaacSim benchmark for task planning and low-level policies in mobile manipulation. arXiv preprint arXiv:2508.15663, 2025. 3arxiv.org/abs/2508.15663
  17. [17]Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabriel Levine, Wensi Ai, Benjamin Martinez, Hang Yin, Michael Lingelbach, Minjunhe Hwang, Ayano Hiranaka, Sujay Garlanka, Arman Aydin, Sharon Lee, Jiankai Sun, Mona Anvari, Manasi Sharma, Dhruva Bansal, Samuel Hunter, Kyu-Young Kim, Alan Lou, Caleb R Matthews, Ivan Villa-Renteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Yunzhu Li, Silvio Savarese, Hyowon Gweon, C. Karen Liu, Jiajun Wu, and Li Fei-Fei. BEHAVIOR-1K: A human-centered, embodied AI benchmark with 1,000 everyday activities and realistic simulation. arXiv preprint arXiv:2403.09227, 2024. A preliminary version appeared at CoRL 2022. 2, 4arxiv.org/abs/2403.09227
  18. [18]Yang Li, Chen Zhao, Zhuoran Wang, Jiankang Wang, Chao Shao, Yihan Lin, Haitao Shen, and Jing Zhang. RoboRecover: Benchmarking robot policy recovery under execution deviations. arXiv preprint arXiv:2609.28952, 2026. 3arxiv.org/abs/2609.28952
  19. [19]Zongxia Li et al. Long-Horizon-Terminal-Bench: Testing the limits of agents on long-horizon terminal tasks with dense reward-based grading. arXiv preprint arXiv:2607.08964, 2026. 3arxiv.org/abs/2607.08964
  20. [20]Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. arXiv preprint arXiv:2306.03310, 2023. 2, 4arxiv.org/abs/2306.03310
  21. [21]Oier Mees, Lukas Hermans, Erick Rosete-Beas, and Wolfram Burgard. CALVIN: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. arXiv preprint arXiv:2112.02277, 2022. IEEE Robotics and Automation Letters (RA-L). 2, 3, 4
  22. [22]Petko Petkov, Giuliano Albanese, Nikolay Nikolov, Luc Van Gool, Jonathan Tremblay, and Danda Paudel. LADER: A long-horizon robot manipulation benchmark. arXiv preprint arXiv:2610.18053, 2026. 2, 3, 4arxiv.org/abs/2610.18053
  23. [23]Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander Willm Clegg, Michal Hlavac, So Yeon Min, Vladimír Vondruš, Theophile Gervet, Vincent-Pierre Berges, John M. Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Un-nat Jain, Dhruv Batra, Akshara Rai, and Roozbeh Mottaghi. Habitat 3.0: A co-habitat for humans, avatars and robots. arXiv preprint arXiv:2310.13724, 2023. 3, 4arxiv.org/abs/2310.13724
  24. [24]Ke Rui, Yushen Zuo, Jiawei Wang, Haoran Jia, Jinming Ma, Weitao Zhou, and Minglei Li. Diagnosing semantic handoff failures in agent-orchestrated vision-language-action skill composition. arXiv preprint arXiv:2607.06256, 2026. 2, 3arxiv.org/abs/2607.06256
  25. [25]Scale AI. LHAW: Long-horizon task workflows. Scale AI Labs blog, Labs.scale.com/blog/LHAW, 2026. Accessed 2026-11-02. 3
  26. [26]Xia Su, Ruiqi Chen, Benlin Liu, Jingwei Ma, Zonglin Di, Ranjay Krishna, and Jon Froehlich. CapNav: Benchmarking vision language models on capability-conditioned indoor navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026. arXiv:2602.18424. 2, 4arxiv.org/abs/2602.18424
  27. [27]Jiashuo Sun, Yue He, Wenxuan Liu, Tao Mao, Jiaz heng Wang, Xiang Chen, and Min Liu. SurgVLA-Bench: Towards evaluating vision-language-action models for laparoscopic surgical robotics. arXiv preprint arXiv:2606.29247, 2026. 3arxiv.org/abs/2606.29247
  28. [28]Zelin Wan, Arash Nourian, Xiaoxiao Li, Nihar Nandan, and Kamalakannan Nandagopal. APIFlow-Bench: Measuring whether agents survive long, dependent API workflows. arXiv preprint arXiv:2608.29128, 2026. 2, 3arxiv.org/abs/2608.29128
  29. [29]Meiqi Wang and Shichao Li. Compiling and benchmarking task-state horizons for embodied agents. arXiv preprint arXiv:2608.08036, 2026. 3
  30. [30]Sen Wang, Bangwei Liu, Zhenkun Gao, Lizhuang Ma, Xuhong Wang, Yuan Xie, and Xin Tan. Explore with long-term memory: A benchmark and multimodal LLM-based reinforcement learning framework for embodied exploration (LMEE-Bench). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026. arXiv:2601.10744. 2, 4arxiv.org/abs/2601.10744
  31. [31]Wenbo Wang, Fangyun Wei, QiXiu Li, Xi Chen, Yaobo Liang, Chang Xu, Jiaolong Yang, and Baining Guo. MobileManBench: Simplifying model verification for mobile manipulation. arXiv preprint arXiv:2602.05233, 2026. Accessed to ECCV 2026. 3arxiv.org/abs/2602.05233
  32. [32]Jiaqi Xu, Bin Li, Bo Lu, Yun-Hui Liu, Qi Dou, and Pheng-Ann Heng. SurRoL: An open-source reinforcement learning centered and dVRK compatible platform for surgical robot learning. arXiv preprint arXiv:2108.13035, 2021. IROS 2021. 3arxiv.org/abs/2108.13035
  33. [33]Zhangchen Xu et al. AutoLab: Can frontier models solve long-horizon auto research and engineering tasks? arXiv preprint arXiv:2606.05080, 2026. 3arxiv.org/abs/2606.05080
  34. [34]Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, and Tong Zhang. EmbodiedBench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. arXiv preprint arXiv:2502.09560, 2025. ICML 2025. 2, 4arxiv.org/abs/2502.09560
  35. [35]Yue Yang, Linfeng Zhao, Mingyu Ding, Gedas Bertasius, and Daniel Sfar. BOSS: Benchmark for observation space shift in long-horizon task. arXiv preprint arXiv:2502.15679, 2025. 2, 3arxiv.org/abs/2502.15679
  36. [36]Ruolin Ye, Wengyang Xu, Haoyuan Fu, Rajat Kumar Jenamani, Vy Nguyen, Cewu Lu, Katherine Dimitropoulou, and Tapomayukh Bhattacharjee. RCareWorld: A human-centric simulation world for caregiving robots. arXiv preprint arXiv:2210.10821, 2022. 3arxiv.org/abs/2210.10821
  37. [37]Zonghao Ying, Le Wang, Yisong Xiao, Jiakai Wang, Yuqing Ma, Jinyang Guo, Zhenfei Yin, Mingchuan Zhang, Aishan Liu, and Xianglong Liu. AGENTSAFE: Benchmarking the safety of embodied agents on hazardous instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026. arXiv:2506.14697. 2, 4arxiv.org/abs/2506.14697
  38. [38]Qinxi Yu, Masoud Moghani, Karthik Dharmarajan, Vincent Schorp, William Chung-Ho Panitch, Jingzhou Liu, Kush Hari, Huang Huang, Mayank Mittal, Ken Goldberg, and Animesh Garg. ORBIT-Surgical: An open-simulation framework for learning surgical augmented dexterity. arXiv preprint arXiv:2404.16027, 2024. 3, 4arxiv.org/abs/2404.16027
  39. [39]Shiduo Zhang, Zhe Xu, Peijiu Liu, Xiaopeng Yu, Yuan Li, Qinghui Gao, Zhaoye Fei, Zhangyue Yin, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. VLABench: A large-scale benchmark for language-conditioned robotics manipulation with long-horizon reasoning tasks. arXiv preprint arXiv:2412.18194, 2024. 2, 4arxiv.org/abs/2412.18194
  40. [40]Xunyi Zhao, Gengze Zhou, and Qi Wu. VLN-MME: Diagnosing MLLMs as language-guided visual navigation agents. arXiv preprint arXiv:2512.24851, 2025. 2, 4arxiv.org/abs/2512.24851
  41. [41]Kaizhi Zheng, Xiaotong Chen, Odest Chadwicke Jenkins, and Xin Eric Wang. VLMbench: A compositional benchmark for vision-and-language manipulation. arXiv preprint arXiv:2206.08522, 2022. 3arxiv.org/abs/2206.08522
  42. [42]Mingxin Zheng, Nic Ma, and Mostafa Toloui. Using simulation to build robotic systems for hospital automation (Project Rheo). NVIDIA Technical Blog, 2026. Posted 16 March; accessed 2026-10-28. 3, 4
  43. [43]Qinhong Zhou, Sunli Chen, Yisong Wang, Haozhe Xu, Weihua Du, Hongxin Zhang, Yilun Du, Joshua B. Tenenbaum, and Chuang Gan. HAZARD challenge: Embodied decision making in dynamically changing environments. arXiv preprint arXiv:2401.12975, 2024. ICLR 2024. 3arxiv.org/abs/2401.12975

Paper details

vlnc:W_g74deex6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude Opus 5.5, Claude Sonnet models, A separate LLM agent

View statement
Looking for related papers…
Contents