betaFeedback

WorksComputer Science

ActOR: Ambient AI-Based Agentic Orchestration for Humanoid Autonomy in Operating Rooms

Abstract

Operating rooms (ORs) increasingly contain ambient sensing infrastructure to enable AI-based optimization of workflows and utilization. We pursue an alternative path: using ambient AI to guide robot action and optimize utilization. Deploying humanoids in the OR without ambient AI is severely limited by cumbersome de novo orientation and planning due to partial onboard observations. We posit that this can be achieved by coupling an ambient AI-based agentic orchestrator with the humanoid robotic actor, providing shared perception and execution feedback for humanoid autonomy in the OR. ActOR fuses a wall-mounted ambient camera and the robot's head-mounted camera into a shared object-level digital twin (DT). A large language model agent plans over the DT using pre-programmed locomotion and manipulation skills without task-specific training. During execution, DT updates and execution outcomes inform retries, alternative approaches, or replanning. We evaluated ActOR in simulated and physical environments. Across three tasks in two simulated ORs reconstructed from real rooms, ambient perception reduced exploration from 1.05 to 0.12 Explore calls per trial when targets were initially outside the robot's view. Closed-loop feedback increased success from 46.7% to 83.3% under nominal execution and from 8.3% to 51.7% under injected navigation, object-relocation, and grasp perturbations. In a physical mock OR with a humanoid, ActOR completed 14 of 20 single-object disposal trials and demonstrated replanning after scene changes. Ambient AI-based observations provide scene state that improves robot action planning, increasing success and reducing exploration compared with robot-only observations. Feeding skill outcomes and DT updates back to the planner improves multi-step task success. Our results show the feasibility of humanoid task autonomy for multi-step disposal tasks in a mock OR.

Introduction

Operating rooms (ORs) face a growing shortage of perioperative nursing and support staff [1]. Room turnover, supply retrieval, and disposal of used materials consume substantial staff time and directly affect OR efficiency [2]. These tasks are repetitive and physically demanding, and do not require highly trained personnel. Automating these tasks with robots rather than relying on an already strained workforce is a promising avenue to improve turnover times. The postoperative phase provides a practical setting for automation: the patient has left, most staff have stepped away, and the environment is comparatively controlled [3, 4]. Humanoid robots are attractive for such tasks because they can, in principle, operate within spaces designed for humans without major modifications.

At the same time, modern ORs are becoming increasingly instrumented. Ceiling- and wall-mounted cameras have been used to recognize surgical phases, localize staff and equipment, and construct digital twins (DTs) of the OR [5, 6, 7, 8]. Yet this ambient sensing infrastructure has primarily been used to inform clinical teams rather than directly support robotic autonomy. We ask whether the instrumented OR itself can instead become part of the robot’s perceptual infrastructure, providing persistent room-scale context that complements the robot’s local, egocentric view.

This question is particularly relevant for humanoid autonomy in the OR, where two limitations make robot-centric operation difficult. First, autonomous task planning requires an accurate representation of the current environment, but an onboard camera provides only a moving and frequently occluded view of a cluttered room. Upon entering the room to complete an assigned task, a robot may therefore need to explore the environment before it can decide where and how to act. Second, humanoid execution remains imperfect: navigation can terminate short of a target, and grasps can fail. A plan generated from the initial scene state therefore becomes unreliable when execution deviates from expectation. Foundation-model agents can decompose open-ended instructions into robot skills [9], but reliable physical autonomy also requires broader access to the world state and feedback about executed actions.

We address these limitations with ActOR, an ambient AI-based agentic orchestration framework for humanoid autonomy in the OR (Figure 1). ActOR fuses observations from a wall-mounted ambient camera and the robot’s head-mounted camera into a shared object-level DT, combining room-scale awareness with near-field observations for manipulation. A large language model (LLM) agent plans over the DT using a library of pre-programmed skills, without task-specific training. During execution, the DT is updated and the outcome is fed back to the agent, which retries, changes its approach, or replans. We evaluate ActOR on postoperative support tasks in two simulated ORs reconstructed from real rooms and on a Unitree G1 humanoid in a physical mock OR. Our contributions are:

Overview of ActOR

Figure 1. Overview of ActOR. (a) A humanoid executes a multi-object disposal task in a mock OR. (b) The shared scene representation integrates ambient and robot observations into an object-level DT and occupancy map. (c) An LLM agent uses the DT and robot skills for task planning, execution monitoring, and replanning.

  • A shared scene representation integrating room-scale and egocentric sensing for robot orientation and task planning in ORs.

  • A closed-loop execution mechanism that feeds skill outcomes and scene updates back to an LLM planner for recovery and replanning.

  • Simulation ablations of ambient perception and execution feedback, together with physical validation in a mock OR.

Related Work

Hospital robots have so far focused on logistics outside the OR, such as mobile manipulators that deliver supplies to nursing units [10], while humanoids in the OR have been studied mainly under teleoperation [11, 12, 13]. Outside medicine, fixed external cameras have been used to localize mobile robots and build shared semantic maps [14]. ActOR uses ambient AI-based scene understanding to support humanoid task planning in the OR and incorporates execution feedback for retry and replanning.

LLMs can decompose open-ended instructions into skill sequences when grounded in a scene representation, such as open-vocabulary or hierarchical 3D scene graphs [15, 16, 17]. Closed-loop variants feed back success detection or scene descriptions to recover from failures [18]. These systems typically build their representation from onboard sensing and rely on exploration to discover the scene [9]. In the OR, however, exploration is undesirable: the room is cluttered with sensitive equipment, and unnecessary robot motion should be minimized. ActOR therefore obtains the scene state from the ambient camera up front, and we evaluate each component through targeted ablations.

Method

Problem Formulation

We consider a humanoid robot performing postoperative support tasks in an OR. Given a natural-language instruction II (e.g., “put all used supplies into the biohazard bin”) and a skill library K\mathcal{K}, the robot must complete the task autonomously. The system maintains an object-level DT St={oi}i=1NtS_t = \{o_i\}_{i=1}^{N_t}, where each node

oi=(li,pi,Bi,hi),(1)o_i = (l_i, \mathbf{p}_i, \mathbf{B}_i, h_i), \tag*{(1)}

consists of a semantic label lil_i, a 3D position pi∈R3\mathbf{p}_i \in\mathbb{R}^3 in a shared world frame, a 3D extent Bi\mathbf{B}_i, and an interaction state hi∈{free,in-hand}h_i \in\{\mathrm{free}, \mathrm{in\text{-}hand}\}. As shown in Figure 1c, ActOR couples this DT (§ 3.2) with an LLM agent that plans and verifies over it (§ 3.3) and a humanoid that executes the resulting skills (§ 3.4).

Shared Scene Representation

The ambient camera provides persistent room-scale coverage independent of robot pose, while the robot camera provides accurate near-field geometry for manipulation. Given an RGB image from either camera, a vision-language model (VLM) [19] detects task-relevant objects with labels lil_i; each 2D box prompts SAM2 [20] for an instance mask, whose pixels are back-projected with stereo depth and transformed into the world frame to obtain pi\mathbf{p}_i and Bi\mathbf{B}_i.

The DT is kept consistent with the scene as the robot acts. It is updated after each manipulation skill and on each Observe call, and only nodes inside the observing camera’s view are modified. A detection is merged into the existing node it overlaps most when the two occupy the same volume; otherwise, a new node is created. A node is removed when consecutive depth observations find its volume unoccupied. Manipulation also changes object state: when the gripper stops before full closure, the grasped node is marked in-hand and follows the gripper until Place. Spatial relations such as in, near and on are derived from the object positions and extents and passed to the planner with the DT.

Closed-Loop Agentic Orchestration

A large language model (GPT-6.1 Sol [21]) serves as the planner. Its system prompt specifies its role, OR domain knowledge, the skill library (Table 1), and rules for handling failures. At each decision step, it receives the instruction II, the DT serialized as JSON, and the execution history. It then calls a skill, emits a multi-step plan, or declares the task complete or failed. Skills in a multi-step plan execute sequentially. On skill failure or plan completion, the planner receives the updated DT and execution history to determine the next action (Algorithm 1).

SkillFunctionSuccess condition
Observe(target)Capture target and update the DT–
Explore()Navigate to likely target locations–
Navigate(target)Walk to a standoff pose facing targetdist⁡xy(pbase,Btarget)≤dreach\operatorname{dist}_{xy}(\mathbf{p}_{\mathrm{base}}, \mathbf{B}_{\mathrm{target}}) \le d_{\mathrm{reach}}
Pick(target)Grasp targethtarget=in-handh_{\mathrm{target}} = \text{in-hand}
Place(target,dest)Release target at destptargetxy∈Bdestxyp_{\mathrm{target}}^{xy} \in\mathbf{B}_{\mathrm{dest}}^{xy} at release

Table 1. Skill library and the condition under which each skill reports success.

Recovery is left to the planner, which is instructed to retry a failed skill and otherwise change its approach or skip the object. An episode ends when the planner declares the task complete or failed, or when the elapsed time reaches TtimeoutT_{\mathrm{timeout}}.

Algorithm 1 ActOR closed-loop execution

Require: Instruction II, skill library K\mathcal{K}, DT SS, timeout TtimeoutT_{\mathrm{timeout}}

  1. H←∅\mathcal{H} \leftarrow\varnothing ▷ execution history

  2. while elapsed time <Ttimeout< T_{\mathrm{timeout}} do

  3. d←Planner⁡(I,S,H,K)\quad d \leftarrow\operatorname{Planner}(I,S,\mathcal{H},\mathcal{K})

  4. \quad if dd declares the task complete or failed then

  5. \qquad return dd

  6. \quad for each skill a∈Skills⁡(d)a \in\operatorname{Skills}(d) do

  7. success⁡←Execute⁡(a)\qquad\operatorname{success} \leftarrow\operatorname{Execute}(a)

  8. S←UpdateDT⁡(S)\qquad S \leftarrow\operatorname{UpdateDT}(S); H←H∪{(a,success⁡)}\mathcal{H} \leftarrow\mathcal{H} \cup\{(a,\operatorname{success})\}

  9. \qquad if not success⁡\operatorname{success} then break

  10. return failure

Skill Implementation

Upper-body manipulation and lower-body locomotion are decoupled. For Navigate, ambient depth is projected onto the floor plane to form a 2D occupancy map. A standoff pose facing the target within arm reach is computed, and A* plans waypoints that are converted to base velocity commands and tracked by an RL-based whole-body controller [22]. For Pick, the arm closer to the target is selected, and AnyGrasp [23] predicts grasp candidates from the robot camera point cloud, guided by the target’s bounding box. We retain AnyGrasp candidates and combine them with candidates generated by a 3D bounding-box heuristic adapted from [24]. A motion planner built on Pinocchio [25] checks candidates for kinematic feasibility and collision-free trajectories; feasible candidates are ranked by hand pose and motion magnitude to select the grasp to execute. The selected trajectory is converted to joint commands via inverse kinematics. Place follows the same pipeline, with a release pose above the destination’s position.

Experiments

We assume no human presence during execution, requiring the planner to resolve ambiguities without operator input. All task-relevant objects are placed within the robot’s reachable workspace. Controlled studies in MuJoCo [26] ablate ambient perception and closed-loop feedback, followed by physical deployment in a mock OR.

Experimental Setup

Environments

Controlled studies were conducted in two simulated ORs with distinct layouts reconstructed from real environments [27], with object assets generated by SAM3D [28]. Each environment included a virtual ambient camera. The simulated robot ran the same detection, segmentation, grasping, and skill stack as the physical system and executed through the whole-body controller in simulation; depth was rendered with simulated stereo noise. Physical experiments were conducted in a mock OR equipped with an operating table, mobile CT scanner, surgical robot, and supply tables.

Tasks

We evaluated three simulation tasks: single-object retrieval, requiring delivery of a specified object; multi-object collection, requiring successive transfers of two objects; and room reset, requiring identification and disposal of task-relevant objects among distractors (e.g., “put all used supplies into the biohazard bin”). Object layouts and initial robot poses were randomized across trials and kept identical across methods within each study.

A trial succeeded when all target objects reached the instructed destination within TtimeoutT_{\mathrm{timeout}}. We report task success rate (SR) over 60 trials per method and condition (3 tasks ×\times 2 environments ×\times 10 trials) and, for the ambient perception study, the mean number of Explore calls per trial. For physical experiments, we report end-to-end success and the source of each failure.

Implementation details

We used a Unitree G1 EDU (29 DoF) with Dex1-1 grippers and a head-mounted ZED Mini, alongside a wall-mounted ZED X connected to a Jetson Orin. The shared world frame is defined by a checkerboard observed simultaneously by the ambient and head-mounted cameras. Hand–eye calibration registers the head-mounted camera to the robot frame, and the robot is localized in its map frame using onboard LiDAR SLAM. Composing these transformations at calibration time yields a fixed transformation from the SLAM map frame to the world frame. Depth estimation uses Fast-FoundationStereo [29]. GPT-6 Luna [19] and GPT-6.1 Sol [21] are accessed through API calls for detection and planning; other modules ran locally on an AMD Ryzen 9 5900HX and RTX 3080. We set dreach=0.85 md_{\mathrm{reach}} = 0.85\,\mathrm{m} and Ttimeout=1800 sT_{\mathrm{timeout}} = 1800\,\mathrm{s}.

Baselines

We compare ActOR against two ablated variants. All methods share the same planner, perception modules, and skill implementations.

  • Robot-only. The DT is built and updated only from the robot camera; execution remains closed-loop, and navigation uses the map built from the head camera.

  • Open-loop. Same DT, planner, and prompt, but once execution has begun, the episode ends at the first skill that reports failure, removing retry and replanning.

Controlled Simulation Study

Ambient perception under partial observability

We compared Robot-only and ActOR without injected perturbations (Figure 3a,Figure 3b). For each object layout, the robot started either with all targets in the head camera’s view (ego-observable) or with at least one target outside its view or occluded but visible to the ambient camera (partially observable).

Figure 3. Controlled simulation results across two ORs. (a) Task success with and without ambient perception. (b) Mean Explore calls pooled across tasks. (c) Task success with and without closed-loop feedback. Error bars: Wilson 95% confidence intervals.

When targets started outside the robot’s view, Robot-only relied on exploration (1.05 Explore calls per trial), whereas ActOR localized targets from the ambient view and rarely explored (0.12 calls). Perception-related failures decreased from 13 to 1, and success was 80.0% (48/60) for ActOR versus 66.7% (40/60) for Robot-only. Under ego-observable conditions, success was 80.0% (48/60) for ActOR and 73.3% (44/60) for Robot-only. ActOR achieved higher success in both conditions, and the gap was larger when targets started outside the robot’s view, consistent with the reduction in perception-related failures. Although the per-task confidence intervals overlap and a larger sample would be needed to confirm this gain, ambient perception nearly eliminated exploration in both conditions.

Closed-loop feedback under execution uncertainty

We compared ActOR with Open-loop (Figure 3c) on a separate set of layouts whose initial robot poses were not controlled for head-camera observability; both methods used the same layouts and initial poses. We evaluated nominal execution and a perturbed condition in which each trial was assigned one of three perturbations: navigation error, stopping 1.0–2.0 m from the target; object relocation, moving the target by at least 0.5 m without updating the DT; or grasp misalignment, offsetting the first grasp attempt by 5–8 cm. Both methods were assigned the same perturbation on each trial.

Under nominal execution, ActOR succeeded in 83.3% (50/60) of trials and Open-loop in 46.7% (28/60). This gap indicates that individual skills fail often enough without injected perturbations that a single plan is unreliable, especially for multi-step tasks (Open-loop succeeded in 7/20 trials for both collection and room reset). Under perturbations, success dropped to 51.7% (31/60) for ActOR and 8.3% (5/60) for Open-loop; in multi-object collection, ActOR succeeded in 12/20 trials and Open-loop in none. Since Open-loop removes retry and replanning together, this gap reflects their combined effect.

By perturbation type, ActOR succeeded in 15/24 navigation-error trials, 11/18 object-relocation trials, and 5/18 grasp-misalignment trials, compared with 2/24, 3/18, and 0/18 for Open-loop (Supplementary Table S1). After a navigation error or an object relocation, the updated robot state and DT reflect the true configuration, so the planner can re-approach the target. A grasp misalignment is reported by Pick and retried without the offset, yet recovery was lowest for this type.

Physical Mock OR Deployment

We deployed the complete ActOR system on the Unitree G1 in the physical mock OR (Figure 2c). We conducted 20 single-object disposal trials, each instructing the robot to find a bottle and dispose of it in the biohazard bin. Bottle and bin placements varied across trials within the robot’s reachable workspace. The robot was initialized with the bottle outside its egocentric view, so the bottle could initially be localized only through the ambient camera. A trial succeeded if the bottle reached the bin without human intervention. We also ran exploratory multi-object tests (“put all bottles into the biohazard bin”), including ones in which the scene was changed during execution by relocating the biohazard bin or introducing an additional bottle.

Experimental environments showing simulated OR-A, simulated OR-B, and physical mock OR

Figure 2. Experimental environments. (a, b) Simulated ORs reconstructed from real rooms for controlled studies. (c) Physical mock OR with a Unitree G1 humanoid.

ActOR succeeded in 14 of 20 single-object disposal trials (70%; Figure 4a). In every successful trial, the bottle was localized from the ambient view and Explore was never invoked. Of the six failures, five originated in perception, where the VLM missed or misidentified the bottle, and one in execution, where a grasp still failed after the retry. In other trials, failed grasps were reported by the Pick skill and recovered by retrying. In the multi-object tests, ActOR completed multi-step executions and, in successful runs, replanned after scene changes toward the relocated bin or the added bottle (Figure 4b, Supplementary Video).

Representative humanoid autonomy in the mock operating room

Figure 4. Representative humanoid autonomy in the mock operating room. (a) Multi-step execution in a single-object disposal trial. (b) Replanning after a scene change.

Discussion

Our results suggest that shared scene representation and closed-loop execution feedback together enable a humanoid to complete simple postoperative OR support tasks. The ambient camera provides room-level scene understanding, reducing the burden on the robot’s onboard sensing and exploration. The LLM agent composes pre-programmed skills without task-specific or room-specific training. More broadly, this work explores a path from ambient AI-based scene understanding to robot action in support of OR workflow and utilization optimization.

This study has several limitations. First, both detection and planning rely on proprietary models accessed through an API, which raises privacy concerns for clinical deployment. Locally deployable open-weight models offer a feasible alternative. For detection, however, even strong VLMs remain unreliable in cluttered scenes, and given the safety requirements of the OR, closed-vocabulary detectors trained on OR objects may be more appropriate. Second, the legged humanoid platform adds whole-body balance control on top of manipulation. This introduces additional constraints such as motor overheating and reduced manipulation accuracy, which currently limit the system to relatively large objects. A wheeled base would provide a more stable platform, at the cost of mobility in narrow spaces and over floor cables. Third, manipulation is limited to pick-and-place. More complex interactions, such as opening cabinets or pushing carts, require whole-body coordination. Learning-based policies combining imitation learning and reinforcement learning are a promising direction.

Conclusion

We presented ActOR, an ambient AI-based agentic orchestration framework integrating shared scene representation with closed-loop robot execution. Simulation studies demonstrated reduced exploration and improved task success under execution perturbations. Physical deployment demonstrated task execution and adaptation to scene changes, supporting the feasibility of humanoid autonomy for OR support tasks.

Code availability

Code availability. The code will be made publicly available upon publication.

Supplementary

Supplementary. The video demonstrates humanoid autonomy in a mock OR.

Acknowledgments

Acknowledgments. Supported by NSF CAREER Award No. 2239077.

Supplementary Material

Simulation Results

MethodNavigation errorObject relocationGrasp misalignment
Open-loop2/243/180/18
ActOR15/2411/185/18

Table S1. Successful trials by perturbation type, pooled over tasks and environments. Denominators are the trials assigned each type.

ConditionMethodOR-A Retr.OR-A Coll.OR-A ResetOR-B Retr.OR-B Coll.OR-B Reset
Ego-observableRobot-only776879
Ego-observableActOR8891049
Partially observableRobot-only656887
Partially observableActOR888978
NominalOpen-loop732745
NominalActOR9961088
PerturbedOpen-loop100103
PerturbedActOR464566

Table S2. Successful trials by environment and task (out of 10).

ConditionMethodSuccessesSR [95% CI]
Ego-observableRobot-only44/6073.3 [?, ?]
Ego-observableActOR48/6080.0 [?, ?]
Partially observableRobot-only40/6066.7 [?, ?]
Partially observableActOR48/6080.0 [?, ?]
NominalOpen-loop28/6046.7 [?, ?]
NominalActOR50/6083.3 [?, ?]
PerturbedOpen-loop5/608.3 [4, 18]
PerturbedActOR31/6051.7 [?, ?]

Table S3. Pooled success rates with 95% Wilson confidence intervals.

Study Conditions

Tasks and objects.

Targets are drawn without replacement from four bottles (alcohol, saline, hydrogen peroxide, spray). Retrieval places one target; collection and room reset place two. Every layout also contains two distractors, a glove box and a reflex hammer, on a surface without targets. Room reset reuses the collection layout of the same seed and changes only the instruction (Table S4). The planner’s site description identifies the bottles as the used disposable items (Sec. S4).

TaskInstruction
Retrieval“Clean up: put the [appearance] into the biohazard box.”
Collection“Put all the bottles into the biohazard box.”
Room reset“Put all the used disposable items into the biohazard box.”

Table S4. Example instructions.

Observability conditions (ambient perception study).

Layouts place all targets on one surface, at most 0.5 m apart; only the robot’s start differs between conditions. In ego-observable starts, the head camera sees every target fully and at least 30% of the bin opening, at torso leans of 0° and 5°; the robot starts within 1.2 m of the targets’ centroid. In partially observable starts, at least one target is entirely outside the head camera’s view or occluded. All targets remain visible to the ambient camera in both conditions. Visibility is tested by projecting sample points on each target into the head camera and ray-casting through the room’s 2.5D height field.

Closed-loop study.

This study uses a separate set of layouts with one sampled start per layout, shared by both methods and both conditions. Starts are standable poses at least 0.8 m from obstacles, with random heading, and are not controlled for head-camera observability.

Perturbations.

Perturbation types cycle with the trial index (4, 3, and 3 trials of the three types per task and room), and both methods are assigned the same type and random seed. Because the realized perturbation depends on execution (e.g., which object is approached first), it can differ between methods. Each perturbation fires at most once:

  • Navigation error. On the first navigation toward a graspable object, the robot stops 1.0–2.0 m from it with up to 20° additional heading error, while Navigate reports success.

  • Object relocation. At the start of that navigation, the target is moved at least 0.5 m to another valid placement; the DT keeps its previous position until the object is re-observed.

  • Grasp misalignment. On the first grasp, the grasp point is shifted 5–8 cm along the finger direction. Retries are unperturbed.

Implementation Details

Simulation.

Both ORs were reconstructed from real rooms with TwinOR (OR-A: 5.46 × 5.63 m; OR-B: 5.80 × 5.42 m). The simulated G1 (29 DoF, Dex1-1 grippers) is controlled by the AMO whole-body policy in MuJoCo. The ambient camera renders 2560 × 1440 images (119° × 88° field of view, 120 mm stereo baseline). The robot camera matches the physical head-mounted ZED Mini (1280 × 720, about 85° × 54°, 63 mm baseline), pitched 46.6° down. Depth is MuJoCo-rendered depth with simulated stereo noise (0.15 px disparity noise); detection, segmentation, and grasping use the same models as the physical system.

Shared scene representation.

  • Representation. Object geometry is stored as 3 cm voxels and serialized to the planner as a footprint radius and height range.

  • Association. Two detections are merged when each voxel set covers at least 40% of the other, or when one lies at least 70% inside the other and their labels are compatible.

  • Updates. The VLM runs on each Observe call (ambient and head cameras) and on the head camera before each Pick and Place. Depth-based occupancy and free-space carving run at about 1 Hz.

  • Removal. A voxel is deleted after two consecutive carving rounds observe free space through it; a node is removed when more than half of its tested voxels (at least ten) are empty, or fewer than five voxels remain. Held objects are not carved.

  • In-hand state. After closing, the grasp is in-hand when the normalized gripper opening stops between 0.005 and 0.9; below 0.005 the fingers met and the grasp is empty.

  • Relations. on and in are derived from vertical contact and footprint overlap with a supporting object; near links objects within 1.0 m edge to edge (at most three neighbours).

Skills and planner.
  • dreach=0.85d_{\mathrm{reach}} = 0.85 m is measured to the target’s footprint boundary (0.60 m planner clearance plus 0.25 m). The occupancy grid has 0.10 m resolution.

  • Retry is a prompt rule, not a fixed loop. After a grasp failure, the planner calls Pick again from the same place; after an approach failure, it approaches from the side nearest an edge and tries once more. The same failure twice from the same place leads it to change the side, the object order, or the target, or to leave the object. Open-loop ends the episode at the first skill failure after execution has begun.

  • Ttimeout=1800T_{\mathrm{timeout}} = 1800 s is a hard stop; the planner is warned at 1200 s.

Planner Prompt

Excerpt of the system prompt used in OR-A, reproduced verbatim; [...] marks omitted passages. OR-B’s SITE text differs in its room name and room description, in naming the procedure table instead of the operating table, and in its list of objects such a room typically holds. In the prompt, scene_graph corresponds to Observe. The full prompt, including skill failure messages, will be released with the code.

[...]
How to work:
- Look first. An empty or thin WORLD MODEL means nobody has looked yet, not an
  empty room: scene_graph with the smallest scope the instruction needs --
  "robot", "targets" (one direct phrase per thing) or "full" for a whole-
  room task. SCENE GRAPH.cameras are the cameras that made that look; the
  head camera alone sees only what is ahead of the robot. explore, a walk
  past the room’s landmarks looking with the head camera, only when those
  cameras leave unseen what the task needs. What WORLD MODEL already holds
  keeps its uid: to refresh its relations or reachability pass target_uids (
  no camera look); phrase in targets only what is not there yet, by its
  direct category or appearance, never the whole instruction. Two candidates
  you cannot tell apart: scene_graph scope "inspect". An inspect only
  settles a doubt that changes what you do next --not to confirm each object
  before acting on it: to tidy or move a thing you need where it is, not
  exactly what it is. A look that did not see it, from a camera that saw
  little of it unblocked, does not prove it is not there.
[...]
- A failed skill’s note is your evidence; read it before the next move. The
  skills do not retry from elsewhere -- choosing what to try next is yours:
  * A pick that failed by the thing -- grasp: grasp_missed, not_lifted,
    no_reachable_grasp -- leaves the robot standing by it: pick again (the
    head camera measures it afresh and the robot finds its stance from there)
    -- not navigate, and no look first.
[...]
  * dropped, dropped_on_the_way, target_has_no_position, not_seen_up_close, or
    the thing no longer where it was: look again (scene_graph, scope targets)
    before another try.
[...]
  * The same failure twice from the same place will come a third time: change
    the side, the order or the thing, or leave it -- then finish the rest and
    report what was left and why.
- A multi-step task: one plan, then its steps in order, each call_skill
  carrying "plan_step": <id> at the top level. Moving things: a navigate
  step before each pick and every place. Give every step its full skill
  args: once you start step 1, each step that succeeds starts the next by
  itself, and you are asked again only when a step fails, the plan reaches a
  step no skill does, or it runs out -- then revise the plan from what is
  left, or declare. A step no skill can do (skill null: open, wipe ...) is
  re-emitted as skipped with what a person must do there -- never faked,
  never waited for.
[...]
- Declare succeeded as soon as the skill that completes the instruction
  reports succeeded; never call it again. A task with several things is
  succeeded only when every one is done; with any left, declare failed, the
  summary saying what was done and what is left, with why. A whole-room task
  is declared only after a scope "full" look made since your last pick or
  place: things the first look missed show up there.

[…]

  • Calling a skill cancels the running one: "wait" while it works. "say" reports without stopping, sparingly. Nobody is there to answer a question: what a look cannot settle, decide yourself.

SITE: you work in a hospital operating room. The operating table stands in the middle under the surgical lights, the anaesthesia machine at its head, IV poles and equipment towers around it. Supply cabinets, carts, small stainless tables, waste bins and linen hampers line the walls. Wall-mounted things (sharps containers, dispensers, phone, clock, posters) are part of the walls, not things to fetch. From the room camera high on the wall a bottle on a table is only a few centimetres across: when a look misses one, look closer at the tops where small items are put down. Small items (bottles a hand tall, a glove box, a reflex hammer) are put down at the outer edge of the operating table’s top and the Mayo stand trays, where a hand reaches them from the floor beside it. Only these are loose: anything else that looks like one -- a bottle, box or pack on a counter, shelf or cabinet top, on a cart with shelves or drawers, high on a wall, or inside a bin or hamper -- is fixed as part of the room, never what an instruction means and never to be moved. Where a thing stands is its measured place, not a relation’s word: one at the edge of a Mayo tray or of the operating table’s top is loose even when SCENE GRAPH has it in or next to a cabinet or cart beside that top (a big neighbour’s box takes in what stands beside it). After a procedure the bottles put down there have been opened and used: they, and nothing else here, are the used disposable items, the used supplies and the waste an instruction may ask to clear away. The glove box (a full box of new gloves) and the reflex hammer (an instrument used again and again) are neither: they stay where they are. Nothing flat -- a mark, stain or patch on a top or a sheet -- is an item.

[…]

References

  1. [1]Xie, A., Duff, J., Munday, J.: Perioperative nursing shortages: An integrative review of their impact, causal factors, and mitigation strategies. Journal of Nursing Management 2024(1), 2983251 (2024) https://doi.org/10.1155/2024/2983251
  2. [2]Portney, D.S., Demirci, H.: Spotlighting turnover costs for pediatric ocular oncology exams under anesthesia. Ocular Oncology and Pathology 10(4), 247–251 (2024) https://doi.org/10.1159/000541659
  3. [3]Kernbach, J.M., Brat, G., Topol, E.J., Rajpurkar, P.: Humanoid robots in the operating room: a framework for staged integration of embodied AI in surgery. npj Digital Medicine (2026) https://doi.org/10.1038/s41746-026-02853-6
  4. [4]Wachs, J.P.: Final Report for the Workshop on Robotics & AI in Medicine (2026). https://arxiv.org/abs/2603.18130
  5. [5]Zhang, H., Seenivasan, L., Porras, J.L., Soberanis-Mukul, R.D., Ding, H., Shu, H., Killeen, B.D., Ghosh, A., Yarmus, L., Jopling, J.K., Ishii, M., Argento, A.C., Unberath, M.: Egosurg: Arbitrary view synthesis for egocentric replay of operating room workflows from ambient cameras. arXiv (2026). https://doi.org/10.48550/arXiv.2510.04802
  6. [6]Perez, A., Zhang, H., Ku, Y.-C., Seenivasan, L., Soberanis, R., Porras, J.L., Day, R., Jopling, J., Najjar, P., Unberath, M.: Privacy-Preserving Operating Room Workflow Analysis using Digital Twins. arXiv (2025). https://doi.org/10.48550/arXiv.2504.12552
  7. [7]Özsoy, E., Pellegrini, C., Bani-Harouni, D., Yuan, K., Keicher, M., Nayab, N.: Specialized foundation models for intelligent operating rooms. npj Digital Medicine 9(1), 362 (2026) https://doi.org/10.1038/s41746-026-02631-4
  8. [8]Wang, T.D., Kim, K.Y., Birdal, T., Navab, N., Bastian, L.: TopoQR: A Unified Topological Scene Representation for the Operating Room. arXiv (2026). https://doi.org/10.48550/arXiv.2603.09466
  9. [9]Huh, G., Gu, C., Truong, T.E., Liu, C.K., Tevet, G.: HomeBody: A Humanoid That Explores, Remembers, and Acts on Its Own (2026). https://tml.stanford.edu/homebody/
  10. [10]Babalola, G.T., Gaston, J.-M., Trombetta, J., Tulk Jesso, S.: A systematic review of collaborative robots for nurses: where are we now, and where is the evidence? Frontiers in Robotics and AI 11, 1398140 (2024)DOI
  11. [11]Cho, S.M., Mangulabnan, J.E., Zhang, H., Mao, Z., He, Y., Guo, P., Xu, D., Hager, G., Ishii, M., Unberath, M.: Humanoid Robots as First Assistants in Endoscopic Surgery. arXiv (2026). https://doi.org/10.48550/arXiv.2602.24156
  12. [12]Liang, Z., Thareja, N., Zhang, P., Joyce, C., Atar, S., Richter, F., Jacobsen, G.L., Liu, S., Broderick, R., Yip, M.: In vivo feasibility study of humanoid robots in surgery. Nature 657(8130), 236–244 (2026) https://doi.org/10.1038/s41586-026-10796-x
  13. [13]Atar, S., Liang, X., Joyce, C., Richter, F., Wood, R., Goldberg, C., Suresh, P., Yip, M.: Humanoids in Hospitals: A Technical Study of Humanoid Surrogates for Dexterous Medical Interventions (2025). https://arxiv.org/abs/2503.12725
  14. [14]Bultmann, S., Memmesheimer, R., Behnke, S.: External camera-based mobile robot pose estimation for collaborative perception with smart edge sensors. In: 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 8194–8200 (2023). IEEEarxiv.org/abs/2303.03797
  15. [15]Gu, Q., Kuwajerwala, A., Morin, S., Jatavallabhula, K.M., Sen, B., Agar- wal, A., Rivera, C., Paul, W., Ellis, K., Chellappa, R., et al.: Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In: 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 5021–5028 (2024). IEEEarxiv.org/abs/2309.16650
  16. [16]Rana, K., Haviland, J., Garg, S., Abou-Chakra, J., Reid, I., Suenderhauf, N.: Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning. arXiv preprint arXiv:2307.06135 (2023)arxiv.org/abs/2307.06135
  17. [17]Byrd, G., Rivera, C., Kemp, B., Booker, M., Schmidt, A., Melo, C., Seenivasan, L., Unberath, M.: Constrained natural language action planning for resilient embodied systems. npj Artificial Intelligence (2026)arxiv.org/abs/2510.06357
  18. [18]Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tomp- son, J., Mordatch, I., Chebotar, Y., Sermanet, P., Brown, N., Jackson, T., Luu, L., Levine, S., Hausman, K., Ichter, B.: Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608 (2022)arxiv.org/abs/2207.05608
  19. [19]OpenAI: GPT-6 Luna Model Documentation. https://developers.openai.com/ api/docs/models/gpt-6-luna
  20. [20]Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., Mintun, E., Pan, J., Alwala, K.V., Carion, N., Wu, C.-Y., Girshick, R., Dollár, P., Feichtenhofer, C.: SAM 2: Segment Anything in Images and Videos. In: International Conference on Learning Representations, vol. 2025 (2025)arxiv.org/abs/2408.00714
  21. [21]OpenAI: GPT-6.1 Sol Model Documentation. https://developers.openai.com/ api/docs/models/gpt-6.1-sol
  22. [22]Li, J., Cheng, X., Huang, T., Yang, S., Qiu, R., Wang, X.: Amo: Adaptive motion optimization for hyper-dexterous humanoid whole-body control. Robotics: Science and Systems 2025 (2025)arxiv.org/abs/2505.03738
  23. [23]Fang, H.-S., Wang, C., Fang, H., Gou, M., Liu, J., Yan, H., Liu, W., Xie, Y., Lu, C.: Anygrasp: Robust and efficient grasp perception in spatial and temporal domains. IEEE Transactions on Robotics (T-RO) (2023)arxiv.org/abs/2212.08333
  24. [24]Yan, Z., Li, S., Wang, Z., Wu, L., Wang, H., Zhu, J., Chen, L., Liu, J.: Dynamic open-vocabulary 3d scene graphs for long-term language-guided mobile manipu- lation. IEEE Robotics and Automation Letters 10(5), 4252–4259 (2025) https: Carpentier, J., Saurel, G., Buondonno, G., Mirabel, J., Lamiraux, F., Stasse, O., Mansard, N.: The pinocchio c++ library – a fast and flexible implementation of rigid body dynamics algorithms and their analytical derivatives. In: IEEE International Symposium on System Integrations (SII) (2019)
  25. [26]Todorov, E., Erez, T., Tassa, Y.: Mujoco: A physics engine for model-based control. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026–5033 (2012). https://doi.org/10.1109/IROS.2012.6386109
  26. [27]Zhang, H., Shen, Y., Soberanis-Mukul, R.D., Ghosh, A., Ding, H., Seenivasan, L., Porras, J.L., Mao, Z., Li, C., Xiao, W., et al.: TwinOR: photorealistic digital twins of dynamic operating rooms for embodied AI research. International Journal of Computer Assisted Radiology and Surgery 21(5), 969–977 (2026) https://doi.org/10.1007/s11548-026-03644-w
  27. [28]Team, S.D., Chen, X., Chu, F.-J., Gleize, P., Liang, K.J., Sax, A., Tang, H., Wang, W., Guo, M., Hardin, T., Li, X., Lin, A., Liu, J., Ma, J., Sagar, A., Song, B., Wang, X., Yang, J., Zhang, B., Dollár, P., Gkiokari, G., Feiszli, M., Malik, J.: Sam 3d: 3dfy anything in images. arXiv:2511.16624 (2025)
  28. [29]Wen, B., Dewan, S., Birchfield, S.: Fast-FoundationStereo: Real-time zero-shot stereo matching. CVPR (2026)arxiv.org/abs/2512.11130

Paper details

vlnc:W_5zqq8wh7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 16

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents