WorksComputer Science
SimRetract: Zero-Shot Sim-to-Real Robotic Assistance for Retractor Take-Over in Open Surgery
Abstract
We demonstrate the first zero-shot sim-to-real transfer for a surgeon-robot collaboration task in open surgery: retractor take-over. A robot arm takes a retractor from the surgeon's hand and preserves retraction at handoff, using a policy trained only on simulated demonstrations, without human demonstrations or real-world fine-tuning. Imitation learning from real demonstrations is limited by collection and labeling costs, and by performance degradation when the environment changes. Our framework, SimRetract, generates simulated demonstrations designed for transfer: the policy observes masked instrument images and grasp geometry, learns to recover from off-nominal states, and acts with margin for execution drift. Deployed zero-shot, it grasps the retractor in 90% of 50 surgeon-defined phantom presentations and preserves retraction at handoff in 78%. Results on unseen retractors suggest potential for generalization to instruments with similar grasp geometry. Across ten policy configurations and 1,200 simulated and real trials, we localize the sim-to-real gap to components: in simulation, grasp rates across ablations span 80-96%, while on the real robot removing the instrument mask alone drops performance to 22% grasped and 12% held. Transfer-oriented design thus makes zero-shot sim-to-real transfer feasible for a surgical assistance task without human demonstrations. Code is available here: https://anonymous.4open.science/r/simretract-58D2.
Introduction
Open surgery relies on human assistants to provide anatomical exposure and maintain tissue tension through retraction. Although an experienced assistant can sometimes independently establish effective retraction, the primary surgeon commonly places the retractor to create the exposure and tension needed to operate, after which the assistant takes over and maintains the selected retraction. Here we focus on taking over a surgeon-positioned retractor, with the surgeon responsible for selecting the desired exposure. Because surgical personnel are scarce and unevenly distributed, robotic assistance could help extend the capacity of existing teams [5, 8]. The same scarcity limits expert time for collecting robot demonstrations, motivating simulation as a scalable source of training data. This handover requires perceiving an instrument partly occluded by the surgeon’s hand and grasping it without disturbing retraction across unfamiliar environments.
We call this hand-off retractor take-over. The surgeon establishes the desired tissue retraction and holds the instrument under load while the robot approaches and grasps the exposed shaft. A successful take-over requires more than grasping the retractor: the established retraction must be preserved at handoff, after the surgeon releases it.
The instrument is presented at varied positions and orientations and is partly occluded by the surgeon’s hand. Only a limited portion of the shaft is available for grasping. A parallel gripper must align its pads with the shaft’s flat faces and center the grasp site between them. Otherwise, closure can shift or rotate the instrument and alter the exposure even if the grasp succeeds. The robot must also approach without disturbing the loaded instrument and move smoothly and predictably beside the surgeon.
Imitation learning from real demonstrations is a natural route to such a skill [2] (Wang et al. 2026), but prior work documents performance degradation under changes in background, lighting and viewpoint (Xie et al., 2024; Pumacay et al., 2024). Generalization to these shifts benefits from greater demonstration diversity [12]. Collecting this diversity across differently draped, lit and equipped surgical settings requires substantial expert time.
Simulation supports scalable data generation with precise labels and controlled variation [15] (Tobin et al. 2017). However, rendered observations differ from real ones, and surgical sim-to-real studies have largely focused on minimally invasive robotic platforms and surgical training tasks [14] (Xu et al. 2021; Yu et al. 2024; Scheikl et al. 2023). With the arm, gripper and wrist-camera configuration fixed, we ask whether a policy trained for retractor take-over on simulated demonstrations can perform the task in a previously unseen real environment without real task demonstrations or environment-specific policy fine-tuning. Wrist-camera sensing and robot joint state provide the policy input, without external cameras or a model of the room.
We propose SimRetract, a framework for learning zero-shot retractor take-over from simulated demonstrations. The expert selects grasps and generates complete approach and closure trajectories, retaining demonstrations only after physical stability checks. Smooth, aligned approaches leave margin for execution drift, while perturbation-recovery adds corrective trajectories from perturbed poses. In both domains, the policy receives masked RGB-D, robot joint state and the grasp-plane normal. Training-time perception augmentation, including input dropout, mimics sensing imperfections.
We fine-tune [2] using a dataset of 1 500 synthetic episodes and evaluate it in held-out simulation and on 50 surgeon-defined real presentations, with 1 200 trials including ablations and unseen instruments. On the training instrument, the full policy achieves 90% grasped and 78% held at handoff in real-world trials. Masking has the largest effect on transfer, while the recovery-augmented dataset improves preservation of retraction.
Contributions
The first zero-shot sim-to-real transfer for a surgeon-robot collaboration task in open surgery. A policy trained on synthetic task demonstrations, with no real task demonstrations or real-world policy fine-tuning, grasps the retractor in 90% of 50 surgeon-defined presentations and preserves the established exposure at handoff in 78%.
A framework for learning a surgical assistance task in simulation. The framework generates and validates complete retractor take-over demonstrations, combines trajectories with execution margin and corrective recovery, and trains on task-relevant inputs with perception augmentation.
A dissection of the sim-to-real gap for an open-surgery task. Our comparisons cover ten policy configurations, 500 simulated and 700 real trials, including unseen instruments. In the main comparison, simulated grasp rates span 80–96%, while real rates span 22–90%; comparing both domains identifies which design choices support transfer.
Related Work
Robot assistance in surgery. Surgical autonomy research has concentrated on the minimally invasive setting and the da Vinci Research Kit: transformer policies from teleoperation [9] and hierarchical imitation learning that automates the clipping and cutting stages of ex-vivo cholecystectomy [10]; see Schmidgall et al. (2025). Learning-based robotic assistance in open surgery remains less explored than minimally invasive surgical automation. Existing work includes robotic scrub nurses, from gesture-driven systems (Jacob et al. [6]) to VLA-based delivery (Li et al. [11]), and an evaluation of imitation-learning policies on suture following (Wang et al. (2026)). Retractor take-over differs from delivery in that the robot must take over an instrument held in a load-bearing pose while preserving the established retraction through the surgeon’s release.
Generalization in imitation learning for robotics. In imitation learning a robot acquires a skill by reproducing demonstrated behavior from recorded observation–action pairs (Ross et al. (2011)), with policies ranging from task-specific action decoders (Zhao et al. (2023); Chi et al. [4]) to pretrained vision–language–action (VLA) models such as (Black et al. [2]). Across architectures, policies degrade under changes of background, lighting and viewpoint (Xie et al. (2024); Pumacay et al. (2024)), fail when their own errors carry them into states the demonstrations never covered (Ross et al. (2011)). Generalization benefits from diverse training environments and objects (Lin et al. [12]).
Simulation for scalable data and environment generalization. Simulation enables high-throughput data collection (Makoviychuk et al. [15]) and scalable expansion of small human demonstration sets (Mandlekar et al. (2023)). Domain randomization uses controlled variation to support transfer (Tobin et al. (2017); James et al. [7]). Videos of manual open surgery do not directly provide robot action targets, whereas instrumented research robots can record synchronized video and kinematics (Kim et al. [9]). SurRoL (Xu et al. (2021)) and ORBIT-Surgical (Yu et al. (2024)) provide dVRK-compatible surgical learning environments. Surgical sim-to-real transfer has been demonstrated using SOFA (Scheikl et al. (2023)) and SurRoL (Long et al. [14]). SimRetract takes a different strategy for the open field: rather than randomizing the whole scene, it removes environment-specific content from the policy input and corrupts the task-relevant perception signals that remain, keeping the policy end-to-end from masked pixels to joint targets.
Method
Retractor take-over requires a grasp that preserves retraction and an approach without premature contact. Figure 1 outlines demonstration generation, policy inputs, training and deployment; Figure S1 shows representative sequences, with implementation details in Appendix B.
Figure 1. Overview of SimRetract. (A) A scripted expert generates complete approach and closure trajectories and retains physically validated demonstrations, including perturbation-recovery episodes. (B) Training inputs combine masked RGB, metric depth, robot joint state and grasp-plane normal, with perception augmentation. (C) Sampled observation–action pairs train to predict chunks of joint and gripper targets. (D) The same checkpoint is evaluated in simulation and deployed on the real robot without fine-tuning. Real outcomes at handoff are held, grasped only, or missed.
Figure S1. Simulated demonstration and real rollout across five stages. Inputs include masked RGB, metric depth, plane normal and joint state. Plane overlays show the fitted plane (cyan) and normal (yellow). Simulation inputs precede augmentation. Real RGB-D matches recorded queries. Plane overlay illustrations are recomputed from recorded RGB-D. Input panels end before release; the simulated hand proxy remains visible.
Synthetic demonstration generation
Our scripted expert combines grasp constraints, trajectories with execution margin and corrective recovery to generate complete take-over demonstrations.
Scene. Isaac Sim models the physical setup: a Trossen WidowX AI arm, parallel gripper and wrist-mounted RealSense D405 (Figure 2). A rigid Volkmann-type rake retractor is prescribed within a surgical-field-sized workspace in front of the arm (Figure S2; Appendix § 9.§ 1). A randomized simulated hand occludes the shaft and holds it compliantly, allowing displacement on contact; a second occluder hides the hook end as tissue does. Retraction is represented by instrument pose, without deformable tissue mechanics. Low pad friction rejects grasps that rely on friction to compensate for poor alignment.
Figure 2. Real experimental setup and instruments. (A) The arm beside the draped abdominal phantom. (B) The wrist-mounted RGB-D camera and parallel gripper. (C) Left to right: Richardson and Army–Navy, both unseen during training, and the Volkmann-type training retractor.

Figure S2. Instrument-pose sampling domain relative to the robot. The green box bounds the proposed instrument centers; red lines show example shaft positions and orientations. The shaft can extend outside the center-sampling box.
Grasp. The expert uses the simulator’s exact instrument pose to select mesh-derived grasp candidates. The target centers the grasp site between the pads and aligns the closing axis with the shaft-plane normal, keeping pad faces parallel to the shaft faces to limit displacement during closure (Figure 1A). Candidates must avoid approaching from below and clear the hand and tissue occluders.
Trajectories with execution margin. The policy predicts two-second action chunks of joint targets, so abrupt direction changes between observations can cause overshoot or premature contact. Smoothing combines a smooth reach curve with limited joint-target changes. A standoff ends the curve short of the grasp for an aligned straight entry (Figure S3). Together these leave margin for execution drift. Inverse kinematics (IK) converts sampled gripper poses into arm joint targets. Only complete trajectories with undisturbed approaches and physically stable grasps are retained as demonstrations (Figure 1A; Appendix § 9.§ 4).

Figure S3. TCP paths reconstructed from joint states for the same presentation and grasp. SimRetract (teal): smooth curve, then a 3 cm aligned straight entry; w/o smoothing (orange): straight reach and aligned entry; w/o standoff (red): curve directly to the grasp.
Perturbation-recovery (PR). To teach recovery from states outside nominal expert trajectories (Ross et al. (2011); Spencer et al. (2021)), we interrupt an accepted demonstration at a uniformly sampled reach point, apply a small random pose perturbation and plan a fresh path to the grasp. Only the recovery segment is retained under the same physical criteria. These corrective data augment 500 expert demonstrations with 1 000 PR episodes.
Training data for precise sim-to-real grasping
Policy inputs. The policy observation is (Figure 1B), where is wrist RGB, metric depth, the binary instrument mask, the six arm-joint positions and gripper position, and the shaft-plane unit normal. Here denotes pixelwise masking and concatenation. Masking removes surrounding clutter, while depth and the normal provide geometry for precise grasp alignment. The normal is estimated by total-least-squares plane fitting with outlier rejection on the point cloud back-projected from masked depth pixels (Appendix § 9.§ 6).
Perception augmentation. Training-time augmentation introduces random mask erosion, dilation, holes or leaks, depth speckle and edge artifacts, and plane-normal perturbations (Appendix § 9.§ 5). Close-range input dropout jointly blanks both images and the plane cue.
Policy training
We fine-tune (Black et al., [2]), a pretrained vision–language backbone (Beyer et al., [1]) with an action expert that generates chunks by flow matching (Lipman et al., [13]). The two masked images fill its camera slots, and fills its state input. Each chunk contains targets at 25 Hz, covering two seconds (Figure 1C). The dataset is generated entirely in simulation: 1 500 episodes and 167 216 training frames after preprocessing (Appendix § 9.§ 4).
Zero-shot deployment
The wrist RGB-D camera supplies color and depth. SAM 3 (Carion et al., [3]) tracks the retractor from a text prompt, and plane fitting on the masked depth point cloud gives . With matched hardware, the same checkpoint predicts joint-target chunks without environment-specific policy fine-tuning (Figure 1D). Inference requires no external camera, room model or room-registration step; camera preprocessing and controller details are in Appendix § 9.§ 6.
Experimental Protocol
Ablation design
SimRetract (full) is trained on 500 expert and 1 000 perturbation-recovery (PR) episodes using all inputs and training augmentation. The seven variants in Table 1 test three design elements (§ 3.1–§ 3.2).
| Model | Simulation | Real robot | ||||
| Grasped (%) | Offset (mm) | Twist (°) | Grasped (%) (−pp vs. sim) | Held (%) | ||
| SimRetract (full) | 94 | 8.0 ± 6.0 [6.7] | 3.8 ± 6.4 [2.1] | 90 (−4) | 78 | |
| w/o dropout | 96 | 9.9 ± 7.5 [8.1] | 5.9 ± 11.5 [2.5] | 88 (−8) | 68 | |
| w/o recovery | 80 | 15.6 ± 21.0 [7.1] | 6.0 ± 8.8 [4.8] | 78 (−2) | 34 | |
| half data (250 + 500) | 82 | 13.1 ± 13.6 [9.5] | 6.8 ± 8.2 [3.1] | 58 (−24) | 18 | |
| w/o plane normal | 84 | 20.0 ± 21.2 [12.0] | 10.9 ± 18.3 [3.9] | 76 (−8) | 58 | |
| w/o mask | 88 | 9.3 ± 7.7 [6.2] | 4.6 ± 3.7 [3.5] | 22 (−66) | 12 | |
| w/o smoothing | 86 | 8.8 ± 7.7 [7.3] | 4.5 ± 4.3 [3.8] | 54 (−32) | 42 | |
| w/o standoff | 84 | 10.2 ± 14.8 [5.5] | 2.6 ± 1.9 [1.9] | 68 (−16) | 36 |
Table 1. Simulation and real-robot results (50 presentations each; outcomes in § 4.2–§ 4.3). Variants are defined in § 4.1: w/o = without; half-data counts denote expert + recovery episodes. Offset and twist are post-release tip displacement and axial rotation-vector error over grasped simulated trials: mean ± SD [median]. Parentheses give descriptive real-minus-simulation grasp-rate differences (pp). Held denotes preserved retraction at handoff; paired comparisons are in Table S6.
Perception. w/o mask supplies unmasked RGB-D; w/o plane normal removes the normal from the policy input. w/o dropout disables joint image-and-plane input dropout.
Demonstration coverage. w/o recovery retains only the 500 expert episodes. half data (250 + 500) selects 250 unique grasp positions, retaining their 250 expert and 500 PR episodes. These compare corrective experience and coverage of grasp positions.
Execution margin. w/o smoothing replaces the curved reach with two straight segments through a fixed 6 cm pre-grasp and removes joint-rate limiting. w/o standoff curves directly to the grasp without a separate aligned entry. All variants use 15 000 optimizer updates and 960 000 training draws, including the data ablations (Appendix § 9.§ 5). Additional architectures are compared in Appendix § 11.§ 1.
Simulation experiments
As a reference for task competence, we evaluate each policy on the same 50 held-out simulated presentations (Figure 1D) using deployment code with the renderer’s instrument mask replacing the real segmenter. A trial is grasped if the retractor drops by less than 20 mm after the simulated operator releases it. For grasped trials, we measure post-release instrument-tip offset and axial twist relative to the presented pose. Twist is the axial component of the relative rotation vector; it complements tip offset by capturing changes in the retractor head’s orientation. Scoring and termination rules are in Appendix § 10.§ 2.
Real-robot experiments
Real trials use a Trossen WidowX AI arm with a wrist RealSense D405 at a draped abdominal phantom under operating-room lighting (Figure 2). Two attending surgeons each defined 25 presentations by establishing retraction at varied instrument positions and orientations, giving 50 presentations spanning plausible retraction directions. Each was photographed from two fixed viewpoints and re-created for every policy, with position and orientation checked against both views; outcomes are paired by presentation ID.
In each trial, the surgeon holds the retractor under load during the robot’s approach and releases it after acquisition. Each trial permits one grasp attempt; robot motion stops at gripper closure, premature contact, or when the approach stalls or moves away from the retractor (Appendix § 10.§ 3). We also evaluate the full model and its no-plane ablation on two instruments unseen in simulation, Army–Navy and Richardson, with 50 paired presentations per instrument (25 per surgeon).
Simulation measures pose change through offset and twist; real trials assess its effect on tissue retraction and exposure. A trial is missed if the robot fails to secure the retractor or contacts the instrument, hand or phantom before grasping. Otherwise, it is grasped when the instrument remains in the gripper after release, and held when tissue retraction and exposure are preserved without repositioning at handoff; grasped includes both held and grasped-only trials (Figure 1D). Miss subtypes are analyzed in §5.3 and Appendix § 10.§ 3.
Results
A policy trained only in simulation takes over a surgeon-held retractor and preserves retraction on the real robot
Trained only on simulated demonstrations, SimReTRACT grasps the retractor in 90% of 50 surgeon-defined real presentations and preserves retraction at handoff in 78% (Table 1; Figure 3). The same checkpoint grasps 94% of held-out simulated presentations, with a post-release tip offset of 8.0 ± 6.0 mm and axial twist of 3.8 ± 6.4° over grasped trials. On the 45 grasped real trials, time to grasp is 10.08 ± 1.69 s (median 10.10 s; Appendix § 11.§ 3).

Figure 3. Generalization to unseen retractors (50 presentations each): grasped and held rates at handoff for the full model and the no-plane ablation.
Simulated success does not fully predict real success; masking has the largest effect on transfer
Across the eight configurations in Table 1, simulated grasp rates span 80–96%, while real rates span 22–90%. Removing the instrument mask reduces grasp success by six percentage points in simulation (94% to 88%), compared with 68 points on the real robot (90% to 22%; Holm-adjusted for the real comparison). The sim-to-real grasp gap consequently expands from four percentage points to 66.
Motion design also has larger effects after transfer, although the contrast is smaller than for masking. Removing smoothing costs eight percentage points of grasp success in simulation and 36 on the real robot; removing standoff costs ten and 22 points, respectively.
Corrective demonstrations and execution margin turn a grasp into preserved retraction
Recovery and standoff produce larger improvements in held success than in grasped success. Recovery-augmented training raises grasp success from 78% to 90% and held success from 34% to 78%. Including standoff raises grasp success from 68% to 90% and held success from 36% to 78% (both held comparisons: Holm-adjusted ; Table S6).
Table S6: Paired held success: full model minus each ablation, in percentage points (50 presentations). Brackets show unadjusted 95 % paired bootstrap intervals; exact McNemar values are Holm-adjusted across seven comparisons.
| Ablation | Held difference [95 % CI], pp | |
| w/o dropout | +10 [−8, 28] | 0.383 |
| w/o recovery | +44 [26, 60] | < 0.001 |
| half data | +60 [42, 76] | < 0.001 |
| w/o plane | +20 [2, 38] | 0.128 |
| w/o mask | +66 [52, 80] | < 0.001 |
| w/o smoothing | +36 [16, 56] | 0.006 |
| w/o standoff | +42 [24, 60] | < 0.001 |
Table S6.
Among presentations grasped by both the full model and the no-recovery variant, retraction was preserved only by the full model in 17 cases and only by the no-recovery variant in one. For the full model versus the no-standoff variant, the corresponding counts were 13 and two.
Removing smoothing also lowers held success, from 78% to 42% (Holm-adjusted ). The ablations exhibit distinct approach failures: short reaches account for 7 of 11 misses without recovery, lateral misses for 13 of 23 without smoothing, and premature interference for 14 of 16 without standoff (Figure 4; contact locations in Table S3).

Figure 4. Failure types in real trials (50 per model; bar labels: missed/evaluated). Interference is pooled; contact locations are in Table S3. Grasped-only trials are excluded.
| Model | Inside | Outside | Hand | Phantom | Total |
| Training retractor | |||||
| SimRetract | 0 | 0 | 1 | 0 | 1 |
| w/o dropout | 3 | 0 | 0 | 0 | 3 |
| w/o recovery | 1 | 1 | 0 | 1 | 3 |
| half data | 6 | 3 | 0 | 0 | 9 |
| w/o plane | 2 | 1 | 0 | 0 | 3 |
| w/o mask | 18 | 9 | 2 | 0 | 29 |
| w/o smoothing | 0 | 3 | 1 | 1 | 5 |
| w/o standoff | 5 | 3 | 4 | 2 | 14 |
| SmolVLA | 2 | 3 | 2 | 0 | 7 |
| ACT | 6 | 0 | 0 | 0 | 6 |
| Army–Navy | |||||
| SimRetract | 2 | 0 | 0 | 0 | 2 |
| w/o plane | 5 | 0 | 3 | 1 | 9 |
| Richardson | |||||
| SimRetract | 3 | 1 | 0 | 1 | 5 |
| w/o plane | 7 | 0 | 0 | 1 | 8 |
Table S3. Interference counts by contact location, with 50 trials per condition. Each interference miss receives one subtype.
Despite having fewer episodes, the 500-episode nominal-only model, covering 500 grasp positions, achieves higher grasp success than the 750-episode half-data model covering 250 positions: 78% versus 58% (unadjusted ).
Transfer extends to unseen retractors, with higher success for similar grasp geometry
On the unseen Army–Navy, which shares the training instrument’s flat grasp surface, the full model achieves 80% grasped and 64% held. On Richardson’s cylindrical handle, it achieves 58% grasped and 50% held (Figure 3). Removing the plane cue lowers Army–Navy performance to 66% grasped and 18% held. Richardson grasp rates change little with and without the cue (58% versus 56%), while held rates are 50% versus 38%. Short reaches dominate Richardson misses both with and without the plane cue, accounting for 15 of 21 and 11 of 22 misses, respectively.
Discussion
SimRetract supports a form of surgical assistance in which the surgeon establishes the desired retraction and exposure, and the robot takes over the hold. Learning this behavior from simulated demonstrations offers a route to developing assistance skills without collecting demonstrations from surgeons. Its significance lies in preserving a surgeon-selected state during handoff, beyond acquiring and retaining an instrument.
Task-relevant perception is central to zero-shot transfer. Masking reduces dependence on surrounding appearance while retaining the instrument and its grasp geometry. Its disproportionate benefit on the real robot supports designing observations around information that remains useful across simulation and deployment. The mask ablation identifies this input design as decisive for transfer in the tested framework, while the motion ablations motivate a complementary focus on accommodating deviations during execution.
Demonstrations need margin for execution drift. The paired recovery and standoff comparisons link demonstration design to preserving retraction even when acquisition succeeds in both conditions. Perception errors and differences in controller response can push real rollouts away from the trajectories seen during training. Recovery episodes provide experience correcting these deviations; smoothing combines curved reaches and joint-rate limiting, while standoff leaves room to align before contact. These designs allow correction before misalignment at closure disrupts the established retraction. The nominal-only and half-data comparison further suggests that broader grasp-position coverage can be more valuable than additional episodes around fewer grasps.
Shared grasp geometry provides a reusable basis for transfer. The Army–Navy result supports the grasp-plane normal as a reusable cue when instruments share a local grasp surface, despite differences in their overall form. Richardson illustrates how changes in appearance can influence the use of these geometric inputs: its short reaches suggest that the policy may interpret a wider mask as closer and
underuse metric depth. Preserving meaningful grasp cues while strengthening the use of depth could extend transfer across instrument shapes.
Limitations and future work. We designed the phantom evaluation to test robustness across a wide range of plausible retractor positions and orientations. The full model failed to preserve retraction at six corresponding presentations across all three retractors (Appendix § 11.§ 2), identifying shared difficult configurations. Studies with biological tissue or animal models should next assess preservation of exposure during active surgeon–robot collaboration.
Collision avoidance is currently implicit in the learned grasp and approach behavior: demonstrations favor exposed-shaft grasps, smooth motion and standoff, while deployment lacks explicit online collision monitoring. Interference with the gripper’s long rail and wrist camera also motivates a more compact end-effector. Future work should combine contact sensing and active collision monitoring with safe stopping, recovery and return of control when the surgeon, another instrument or the tissue moves.
Conclusion
SimRetract demonstrates zero-shot sim-to-real retractor take-over on a general-purpose robot arm, preserving surgeon-established retraction at handoff with a policy trained only on simulated demonstrations and no human demonstrations or real-world policy fine-tuning. The findings identify task-relevant perception, corrective demonstrations and execution margin as key elements of transfer, with shared grasp geometry supporting reuse across instruments. These results support simulation-based learning of surgical assistance and motivate further evaluation with biological tissue and active surgeon–robot interaction.
Representative rollouts
Figure S1 shows a simulated demonstration and real take-over.
Implementation details
Simulation environment
Robot and scene. We use Isaac Sim 5.1 / Isaac Lab 2.3 (Mittal et al., 2025) with the manufacturer’s Trossen WidowX AI model and the flat-pad fingers used on the real arm. The wrist camera renders at 640 × 480 with the real D405 intrinsics ( vertical field of view); physics runs at 500 Hz, observations and actions at 25 Hz. The instrument is a rigid mesh of a 21.6 cm Volkmann-type rake retractor, with a pronged hook and flat ring handle, rendered as stainless steel. Its mass is randomized over 60–160 g and pad friction is deliberately low to reject grasps that rely on high friction. One seed fixes the presented pose, mass, hand occluder and lighting, including dome and key intensity, color temperature, key direction and softness, and up to three fill lights.
Presentation region. Instrument centers are sampled uniformly in a 15 × 24 × 15 cm box representing a draped surgical field in front of the robot: 35–50 cm forward of the base, 12 cm laterally and 15–30 cm above the base-frame origin. These limits keep the presentations within the arm’s reach and also bound the real phantom presentations. The box bounds the instrument’s center; the 21.6 cm retractor can extend about 11 cm beyond it, giving an approximate outer envelope of 37 × 46 × 37 cm. Tilt from vertical is sampled uniformly up to , with uniform azimuth and axial spin (Figure S2).
Operator hold and occlusion. The simulated hold is a spring–damper on the presented pose, with yield-capped torque about the two tilt axes and a weaker cap about shaft spin. Under the instrument’s weight it sags by a fraction of a millimeter; a firm push at the tip (about 5 N) rotates it by tens of degrees. Thus a misaligned approach moves the instrument rather than being absorbed by a rigid attachment. A randomized fist or pinch occludes the shaft below the grasp region. A second visual occluder, a 1–3 cm ellipsoid or multi-primitive blob in tissue and drape tones, covers the hook end to represent the loaded tip being visually occluded by tissue. Neither enters the instrument mask. Both receive a 6 mm safety margin in clearance tests; robot entry into either region aborts demonstration generation. The scene models instrument pose and hand compliance, without deformable tissue mechanics.
Grasp generation
A bank of 432 candidate grasps is computed once from the instrument mesh, using face-on clamps on near-parallel opposing faces and antipodal pairs on non-flat parts. For each presentation, only the exposed instrument region included in the mask is treated as graspable. Grasps and approach paths avoid interference with the hand and tissue regions excluded from the mask. The closing axis is aligned with the shaft-plane normal, making the pad faces parallel to the shaft faces. The approach axis lies approximately in that plane; candidates vary the remaining rotation about the closing axis. A target grasp height is sampled uniformly over 5 cm along the shaft; candidates are tried in order of distance from that height. For each candidate, the expert generates and tests a complete approach and closure trajectory; the first passing the physical checks in Appendix § 9.§ 4 is retained as a demonstration.
Each of the 500 poses used for nominal demonstrations has 432 mesh-derived grasp candidates, giving 216 000 candidates in total. Only 5.89% passed physical grasp trials, highlighting the need for execution-level validation of geometry-based grasp candidates.
Trajectory generation
Each selected grasp defines a target gripper position and orientation . The expert first constructs a path of gripper poses from the current pose to this target, then converts them into arm joint targets using inverse kinematics (IK).
Gripper path. The grasp orientation defines the approach direction . The full expert places the pre-grasp 3 cm before the grasp, , and a control point 6 cm farther back, (coordinates in meters). From the starting position , it follows the quadratic Bézier curve
for , then enters straight along for the final 3 cm. The control point aligns the curve’s final tangent with this entry. Gripper orientation interpolates from its initial orientation to over the first 70% of the curve and remains fixed during entry. The path is sampled at a nominal gripper speed of about 0.11 m/s.
IK and joint-rate limiting. At each path sample, a damped least-squares IK controller ((Wampler, 1986)) uses the arm’s current state and the error to the desired gripper position and orientation to compute six arm joint-position targets. Smooth gripper motion does not guarantee smooth motion at every joint: the mapping depends on the arm’s configuration, so even a nearly constant gripper speed can require abrupt rotation of an individual joint. To limit these spikes, large changes between successive joint targets are subdivided into intermediate commands, limiting each commanded joint-position increment to . This slows the arm locally where needed, leaving margin for execution drift. At the grasp, the gripper closes until both pads report contact, then presses. The resulting arm and gripper commands form the demonstration actions; the learned policy subsequently predicts joint-target chunks directly.
Trajectory variants. The w/o smoothing expert uses two straight segments: a reach to a pre-grasp 6 cm before the grasp, followed by an aligned straight entry. It uses neither a Bézier curve nor joint-rate limiting. The w/o standoff expert follows one Bézier curve directly to the grasp, using its candidate pre-grasp as the control point, with no separate straight entry. Both use IK to convert their gripper paths into joint targets. Figure S3 illustrates the three constructions.
Training-set construction
Physical filtering. During demonstration generation, an approach is rejected if it moves the instrument by more than 3 mm or 3° before closure, balancing grasp precision with generation efficiency. After closure and release of the simulated hold, an episode is retained if translation from the presented pose is below 10 mm and rotation is below 10°. The 500 unperturbed (nominal) expert trajectories are balanced across four 2.5 cm grasp-height bands spanning ±5 cm along the shaft, with 125 episodes per band.
Perturbation-recovery episodes. Four recovery draws are proposed per nominal seed. Each perturbs the robot 25–75% of the way along the approach trajectory. The lower cutoff targets recovery after the initial action chunk, which always starts from home; the upper cutoff reduces the risk of random perturbations driving the gripper into the retractor near contact. The arm is offset by 3–8 cm (scaled down for later perturbations), rotated by 8–25°, and its gripper part-closed with 40% probability. A fresh trajectory is generated from the perturbed state and kept under the same physical criteria; only post-perturbation frames are recorded. The training subset uses interruptions in the first half of the reach. Its 1 000 PR episodes use the same pool of 500 nominal presentations.
Final dataset and preprocessing. The curated dataset contains 500 nominal expert demonstrations and 1 000 PR episodes. Episodes are trimmed one second after the gripper reaches approximately its final closed width, then resampled by retaining frames when joint motion exceeds a small quantum, reducing repeated stationary observations. The resulting 167 216 training frames comprise 65 784 nominal and 101 432 recovery frames. Training samples the two components equally, irrespective of their episode or frame counts. Data generation uses one RTX 5090 with up to eight simulator instances in parallel.
Policy training
Policy hyperparameters. All reported models use 15 000 optimizer updates with an effective batch size of 64, totaling 960 000 sampled observation–action-chunk pairs, including repeated samples. The data ablations change the episode pool while preserving this training budget § 4.1; dataset composition and sampling weights are given in Appendix § 9.§ 4. We report the final checkpoint without model selection. The main policy uses ; SmolVLA (Shukor et al., 2025) is fully fine-tuned with the same optimizer settings and training budget (Table S1). During training and deployment, both VLA policies receive the fixed instruction “take over the tool from the operator and hold its pose”.
| Optimizer | AdamW |
| Peak learning rate | |
| Schedule | 1 k warm-up, cosine decay |
| Effective batch size | 64 (16 × 4 accumulation) |
| Optimizer updates | 15 000 |
| Sampled training pairs | 960 000 |
| Action chunk | 50 targets (2 s at 25 Hz) |
Table S1. Shared policy-training hyperparameters. Sample counts include repeated draws from each model’s training pool.
On one H200, training takes 1.65 s per optimizer update (about 7 h for 15 000 updates), and SmolVLA 0.92 s. ACT with input dropout takes approximately 0.26 s per update.
Training augmentation. Augmentation is sampled online during training: instrument masks and
depth images are corrupted, and the estimated grasp-plane normal is perturbed (Table S2). In the latter half of the reach, plane perturbations become stronger and close-range input dropout becomes more likely, reaching 25 % probability at closure. Dropout jointly blanks both image streams and zeroes the plane cue, preparing the policy for loss of instrument perception near the grasp. The w/o dropout ablation disables this input dropout. Augmentation and random dropout are applied only during training.
| Component | Parameters |
| Clean draw | |
| Severity | 0.4 (), 0.8 (0.35), 1.4 (0.10), 1.0 (rest) |
| Opening fill | (enclosed holes filled) |
| Delate/erode | ellipse ; dilate |
| Boundary wobble | SDF noise amplitude m px, feature size 10–45 px |
| False-positive blobs | 0.1+1.3 blobs on the boundary, radius |
| Interior holes | 0..2, radius |
| Half-plane cut | |
| RGB leak fill | 85 % clutter (4–14 shapes, uneven light), 15 % plain; blur , noise |
| Depth leak fill | random level [0.05, 1] + low-frequency field + 0–6 flat objects |
| Depth warp | in normalized depth; is smooth noise |
| Speckle | |
| Quantization | step |
| Edge artifacts | Sobel band, 20–60 % flying pixels, 20–70 % dropout |
| Occlusion shadow | one-sided kernel half-width 2–7 px, drop |
| Invalid patches | 0–4 blobs, radius 4–25 px |
| Plane normal | ; gross 10–45° with ; ramped toward closure, with joint input dropout |
Table S2. Training-time perception augmentation. Ranges are uniform unless stated; denotes severity.
Deployment in simulation and on the real robot
Policy inputs and plane fit. Both deployments back-project masked depth pixels into a point cloud and estimate its normal by total least squares over the retained points :
where is their centroid. The normal is the smallest right-singular vector of the centered cloud, oriented toward the camera. In simulation, depth filtering and residual trimming suppress outliers before refitting. On the real robot, the metric crop below removes the instrument tip.
Simulation. The evaluator obtains RGB-D and the instrument mask from the renderer, builds the same policy observation, and executes predicted joint-target chunks through the simulated arm controller at 25 Hz. It queries again after each chunk. A completed close requires bilateral finger contact and a stalled gripper carriage for 12 control ticks; the simulated operator then releases the instrument, followed by 90 simulation updates for the retention check. Outcome criteria are given in Appendix § 10.§ 2.
Arm. The arm is a Trossen WidowX AI: six revolute joints plus a parallel two-finger gripper (40 mm maximum finger travel, 100 N maximum grip), 0.77 m reach, 1.5 kg payload, 4 kg mass, Ethernet position control with 500 Hz feedback and a manufacturer-specified repeatability of about 1 mm.
Camera. The wrist camera is an Intel RealSense D405 stereo depth camera: 18 mm baseline, global-shutter imagers, depth field of view, 7–50 cm operating range with a minimum depth of 7 cm at 848 × 480, and a manufacturer-specified Z accuracy of ±2 % of distance with RMS spatial noise ≤ 1 % and temporal noise ≤ 0.5 %. At the 35–50 cm working distance of this task these bounds correspond to up to 7–10 mm of absolute depth error and 3.5–5 mm of per-frame noise, which together with the arm’s repeatability set the floor of the execution and perception variability discussed in §6.
Real perception and control. D405 RGB and aligned depth are cropped to the simulated camera’s field of view. SAM 3 tracks the instrument in video mode from the text prompt “metal rake”; the largest detection is locked at first sighting and detections overlapping the gripper are discarded.
For all retractor types, we exclude the uppermost 1 cm of the visible instrument mask, measured along its estimated three-dimensional longitudinal axis using depth and camera intrinsics, rather than a fixed pixel distance. Here, uppermost denotes the end higher in the wrist-camera image. The cropped mask is used for both plane estimation and the masked RGB-D observations supplied to the policy, so the plane is fitted only to the remaining visible body. This consistent preprocessing reduces orientation bias from bent tips, hooks and rake heads across retractor geometries.
SAM 3 tracking runs asynchronously; each policy query uses the latest completed RGB-D–mask observation. On an RTX 5090, warmed-up requests average 0.087 s for and 0.078 s for SmolVLA, excluding camera acquisition and logging. ACT with input dropout infers a 50-action chunk in approximately 10 ms. The controller executes all 50 targets with light exponential smoothing and a per-tick joint clamp before requesting a new chunk. A completed close is detected when the commanded opening is below 20 mm and the fingers close firmly until their motion stops. The policy then stops and the arm holds while the surgeon releases the instrument. The experimental protocol is given in Appendix § 10.§ 1.
Evaluation and experimental details
Test environments and presentations
Simulation. All models use the same frozen set of 50 held-out presentations in the environment of Appendix § 9.§ 1. The rendered instrument mask replaces the real segmenter, while the arm, gripper and camera configuration match deployment. The simulated hand and hook occluders provide local clutter; the scene has no abdominal phantom, operator or full operating-room surround.
Real experiment. The professionally fabricated abdominal phantom is designed for surgical-department simulation, with tissue-like appearance and texture. The arm operates beside the draped phantom under operating-room lighting (Figure 2). The surgeon establishes retraction and passively holds the instrument while the robot approaches, then releases after the robot closes. Two attending surgeons defined 25 presentations each (50 total), spanning practical retraction directions. Each was photographed from two fixed viewpoints. Before each policy trial, the surgeon re-created the presentation and its instrument position and orientation were checked against both reference views. Appendix § 8 illustrates a simulated demonstration and a real roll-out. The full model and its no-plane ablation also use 50 presentations per instrument (25 per surgeon) for the unseen Army–Navy and Richardson retractors. Presentation IDs denote corresponding surgeon-defined poses across all three instruments and all models to support paired comparisons.
Simulation scoring
A simulated trial is grasped when the gripper closes on the retractor and its vertical drop after release is less than 20 mm. Drop is measured from the instrument body’s vertical position immediately before the simulated operator releases it to its position after 90 simulation updates. Failure to close on the instrument within 18 s, or a drop of 20 mm or more, counts as a failure.
For grasped trials, we report post-release instrument-tip offset and axial twist relative to the presented pose as mean standard deviation and median. These describe the motion introduced by take-over. Let and map instrument coordinates to world coordinates before the approach and after release. Axial twist is the absolute -component of , expressed in degrees; the instrument’s local -axis follows its shaft.
Real outcomes and failure definitions
Outcomes. Each real trial receives one of three mutually exclusive outcome labels.
Held: the retractor remains in the gripper after release, maintaining tissue displacement and preserving the established retraction and exposure without repositioning at handoff.
Grasped only: the retractor remains in the gripper after release, but tissue displacement causes loss of the established retraction or exposure at handoff.
Missed: the robot does not secure the retractor, or the robot contacts the phantom, hand or retractor before the grasp begins, as defined below.
The reported grasped rate includes both held and grasped-only trials; held is the stricter task outcome (§ 4.3).
Failure types. Each missed trial receives one failure type, assigned from the first decisive event:
Interference: any robot contact with the phantom, surgeon’s hand or retractor before the grasp begins. Recorded subtypes are inside interference (contact with the instrument inside the gripper opening, including excessive insertion, contact with a bent instrument section at an incorrect angle, or a long instrument striking the pads, inner gripper structures or wrist camera), outside interference (contact outside the gripper opening), hand, and phantom. Interference is always counted as a miss, regardless of any subsequent grasp.
Lateral miss: closure, or the run being stopped, with the shaft outside the pad opening and no contact.
Short reach: closure with the shaft inside the pad opening’s projection along the approach axis, but not reached.
Observed mechanisms. Misaligned or off-center closure can push or rotate the shaft as the pads make contact, changing the established exposure even when the instrument is acquired. This is recorded as grasped only when acquisition succeeds but retraction is not preserved; any robot contact before the grasp begins instead counts as interference under the rule above.
Table S3 breaks down interference by contact location. The annotations group excessive insertion and instrument-length contact together. On the Army–Navy, trial 43 involves the bent section contacting the gripper at an incorrect angle. Without smoothing, 13 of 23 misses are lateral, including abrupt terminal motions that carry the gripper past the shaft before closure. On the training instrument, no inside-interference misses were observed with the full model, compared with three without dropout (unadjusted exact McNemar ). This pattern is consistent with the intended role of dropout under close-range perception ambiguity.
Trial conduct and annotation. Each trial permits one closure attempt. Motion stops at gripper closure or premature contact. Operators stop a trial if the approach stalls, moves away from the retractor, or the gripper closes without the retractor inside it; these trials count as missed. A separate operator runs the system and records the trial outcome.
Additional results
Other policy architectures
Table S5 reports additional policy architectures using the same transfer framework. SmolVLA grasps 80% in simulation and 74% on the real robot, and holds 54%. Its 13 real misses comprise seven interference and six lateral misses. ACT ((Zhao et al., 2023)) grasps 52% in simulation and 48% on the real robot, with 16% real held success: eight held, 16 grasped only, and 26 missed (17 lateral, six inside interference and three short reaches). Both real ACT rates are lower than the full model’s (exact McNemar for each, unadjusted; Appendix D.2). SmolVLA and ACT show descriptive simulation-to-real grasp-rate decreases of six and four percentage points, respectively, alongside lower simulated task success than .
Paired outcome comparisons
We compare the full model with each ablation on 50 corresponding presentations, paired by ID. The primary outcome is held versus not held, where not held includes both misses and grasped-only trials. The secondary outcome is grasped versus missed, where grasped includes both held and grasped-only trials. Each uses an exact McNemar test, which compares presentations where only one model succeeds. Holm correction accounts for the seven ablation comparisons, separately for each outcome. Other paired comparisons report unadjusted values.
The full model has higher held success than the no-recovery, half-data, no-mask, no-smoothing and no-standoff variants (all adjusted ; Table S6). For the secondary grasped outcome, the full model outperforms the no-mask, half-data and no-smoothing variants after correction.
For the full model, six corresponding presentations are not held on any of the three retractors; one of these is missed on all three. Across the eight main models, one presentation is missed by six; no presentation is missed by all eight. These shared failures suggest that photo-guided replication preserved challenging pose characteristics across instruments. The protocol deliberately samples a broad pose range to assess robustness across presentation configurations.
Real-robot time to grasp
Table S4 reports time to grasp and query counts for grasped trials at default demonstration playback speed. Time to grasp is wall-clock time from the first mask pass until the gripper stops closing. Control-tick timing quantifies inference overhead. SAM 3 tracking is asynchronous. For the full model, query overhead is about 0.89 s of the 10.08 s mean grasping time (8.8%); execution and observation-related holds dominate. Playback speed can be increased without retraining, but hardware stability limits usable speeds: faster execution can increase control-induced drift with an unchanged policy.
Table S4: Real-robot timing at default playback speed on grasped trials: mean SD [median]. Mean component latencies on an RTX 5090 are 86.6 ms for requests, 77.6 ms for SmolVLA requests, 152 ms for SAM 3 first-frame detection, and 160 ms for asynchronous tracking (median 98 ms). ACT inference takes approximately 10 ms per 50-action chunk.
| Model | Time to grasp (s) | Policy queries (count) |
| SimRetract | 10.08 1.69 [10.10] | 3.82 0.49 [4] |
| SimRetract, Army–Navy | 12.41 2.59 [12.30] | 3.90 0.71 [4] |
| SimRetract, Richardson | 8.66 1.10 [9.09] | 3.07 0.26 [3] |
| w/o dropout | 9.59 1.62 [9.60] | 3.75 0.49 [4] |
| w/o recovery | 9.00 1.69 [8.22] | 3.56 0.55 [4] |
| half data | 10.42 1.93 [10.56] | 3.79 0.56 [4] |
| w/o plane normal | 9.70 1.79 [9.56] | 3.76 0.60 [4] |
| w/o plane normal, Army–Navy | 11.70 3.80 [10.15] | 3.82 0.85 [4] |
| w/o plane normal, Richardson | 10.15 2.94 [9.35] | 3.43 0.69 [3] |
| w/o mask | 17.11 4.79 [15.97] | 5.18 0.87 [5] |
| w/o smoothing | 9.14 1.82 [9.42] | 3.52 0.58 [4] |
| w/o standoff | 8.52 1.41 [8.31] | 3.21 0.41 [3] |
| SmolVLA | 10.25 2.13 [9.88] | 3.92 0.49 [4] |
| ACT | 8.81 1.39 [8.85] | 3.62 0.58 [4] |
Table S4.
Table S5: Additional policy architectures, with 50 trials per domain. Offset and twist over grasped simulated trials (Appendix § 10.§ 2) are mean SD, with medians beneath in brackets.
| SmolVLA | ACT | |
| Sim. grasped | 80 % | 52 % |
| Offset (mm) | 20.8 17.2 [16.2] | 12.3 10.3 [9.8] |
| Twist (°) | 6.2 10.0 [3.2] | 6.4 11.5 [3.9] |
| Real grasped | 74 % | 48 % |
| Real held | 54 % | 16 % |
Table S5.
References
- [1]Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. PaLiGemma: A versatile 3B VLM for transfer. arXiv preprint arXiv:2407.07726, 2024.arxiv.org/abs/2407.07726
- [2]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024.arxiv.org/abs/2410.24164
- [3]Nicolas Carion et al. SAM 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719, 2025.arxiv.org/abs/2511.16719
- [4]Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. Robotics: Science and Systems (RSS), 2023.
- [5]Hampus Holmer, Adam Lantz, Teena Kunjumen, et al. Global distribution of surgeons, anaesthesiologists, and obstetricians. The Lancet Global Health, 3:S9–S11, 2015.DOI
- [6]Mithun George Jacob, Yu-Ting Li, and Juan P Wachs. Gestonsurse: a multimodal robotic scrub nurse. In ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 153–154, 2012.
- [7]Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.arxiv.org/abs/1812.07252
- [8]Julius M. Kernbach, Gabriel Brat, Eric J. Topol, and Pranav Rajpurkar. Humanoid robots in the operating room: a framework for staged integration of embodied AI in surgery. npj Digital Medicine, 2026. doi: 10.1038/s41746-026-02853-6. URL https://doi.org/10.1038/s41746-026-02853-6.
- [9]Ji Woong Kim, Tony Z. Zhao, Samuel Schmidgall, Anton Deguet, Marin Kobilarov, Chelsea Finn, and Axel Krieger. Surgical robot transformer (SRT): Imitation learning for surgical tasks. Conference on Robot Learning (CoRL), 2024.arxiv.org/abs/2407.12998
- [10]Ji Woong Kim, Juo-Tung Chen, Pascal Hansen, Lucy Xiaoyang Shi, Antony Goldenberg, Samuel Schmidgall, Paul Maria Scheikl, Anton Deguet, Brandon M. White, De Ru Tsai, Richard Jaeyeong Cha, Jeffrey Jopling, Chelsea Finn, and Axel Krieger. SRT-H: A hierarchical framework for autonomous surgery via language-conditioned imitation learning. Science Robotics, 10(104):eadt5254, 2025.arxiv.org/abs/2505.10251
- [11]Shunlei Li, Jin Wang, Rui Dai, Wanyu Ma, Wing Yin Ng, Yingbai Hu, and Zheng Li. RoboNurse-VLA: Robotic scrub nurse system based on vision-language-action model. arXiv preprint arXiv:2409.19590, 2024.arxiv.org/abs/2409.19590
- [12]Fanqi Lin, Yingdong Hu, Pingyue Sheng, Chuan Wen, Jiacheng You, and Yang Gao. Data scaling laws in imitation learning for robotic manipulation. International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.18647
- [13]Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2210.02747
- [14]Yonghao Long et al. Surgical embodied intelligence for generalized task autonomy in laparoscopic robot-assisted surgery. Science Robotics, 10: eadt3093, 2025.DOI
- [15]Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac Gym: High performance GPU-based physics simulation for robot learning. NeurIPS Datasets and Benchmarks, 2021.arxiv.org/abs/2108.10470


