WorksComputer Science
SurgTransfer: Multi-Step Latent Action Modeling for Action Transfer in Surgical World Model
Abstract
Surgical world models have the potential to provide intra-operative guidance, accelerate risk-free training and facilitate transfer of expert surgical maneuvers. However, the use of world models for transferring surgical expertise has been largely unexplored. Moreover, recent approaches in this domain have mainly relied on ground truth kinematics actions or expert annotations for specific tasks, limiting their applicability beyond the narrow range of tasks with ground-truth annotations. In contrast, existing works in the natural domain have shown promising results in learning latent actions solely from unlabeled videos and transferring the learned actions across contexts. We observe that this paradigm is short-sighted for the surgical domain, failing to capture the hierarchical and goal-directed structure of surgical procedures. Thus, we propose a novel multi-step action learning framework to learn context-invariant latent actions that are easily transferable across various surgical contexts. We perform extensive experiments on the publicly available datasets, SurgToolLoc-2022 and PitVis-2023, verifying our approach with both quantitative metrics and qualitative evaluation by surgical experts. We have published our code and the implementation details at https://github.com/bhattarailab/Surgical-Vision-World-Model.
Introduction
World models learn an interactive simulation of an environment by predicting future observations conditioned on current states and actions [11, 12, 27]. Such models hold significant promise for surgical applications, where they could enable intra-operative guidance, pre-operative planning, and risk-free simulation of procedural decisions. For
example, a surgeon can engage in risk-free interactions with the model to simulate the predicted outcomes of proposed surgical plans. A trainee can explore alternative surgical strategies and rehearse decision-making without patient risk. Beyond simulation, learning action representations that are invariant to surgical contexts [21] would allow surgical skills demonstrated in one case to be transferred to anatomically different target cases, enabling skill reuse and training as shown in Figure 1. Despite this promise, transferable surgical world models have remained unexplored, and this paper makes the first contribution to our knowledge in learning transferable latent actions from unlabeled surgical videos.

Figure 1. Our framework learns composable actions that can be transferred from demonstrations to novel surgical contexts.
Recent works that explore world models in surgical vision depend on explicit action supervision. Many methods rely on robotic actuator signals recorded from real or simulated systems [23, 24, 42], while others utilize textual descriptions as alternative representations of action [2, 33]. Such annotations are scarce, noisy, and difficult to scale to general surgical procedures. To address action scarcity, several works in natural domains and robotics [4, 8, 43] propose learning latent actions in an unsupervised manner. Surgical Vision World Model (SurgWM) [21] was the first work to extend this strategy to the domain of surgery. Our current work extends this work by studying transferring latent actions across surgical contexts for which we find the existing paradigms myopic. Existing latent action models only infer actions between consecutive frames, which is not as well suited to surgical procedures, which are inherently hierarchical and goal-directed [19].
We address these limitations by introduce Multi-Step Latent Action Model (LAM) (See § 3.1.1) designed to infer not only the action between consecutive frames but composite actions that span longer time frames. This design choice provides two key advantages: (a) It aligns naturally with the multi-frame prediction paradigm of video diffusion models, avoiding the progressive deterioration associated with autoregressive rollouts. While predicting multiple frames at a time is common for video diffusion models, world models generally predict single frame at a time to condition on actions between frames. However, since our transfer framework admits conditioning on multiple actions at a time, we can use the multi-frame generation capabilities of video diffusion models. (b) Additionally, it encourages the model to encode coarser, composable latent actions. Our experiments demonstrate that, when transferred to a new target context, these high-level actions can better preserve their semantic consistency.
We summarize our key contributions below:
We introduce a general surgical procedure transfer framework: SurgTransfer that is capable of transferring surgical actions across contexts.
We propose a novel multi-step latent action model to learn composite latent actions from unlabeled surgical videos.
We extensively ablate our method on different datasets and evaluate it both quantitatively, and qualitatively by surgical experts.
Related Work
General World Models
In recent years, there has been a significant leap in world modeling driven by large-scale internet videos, particularly for Embodied AI applications [25]. According to [22], world models can be categorized along three core dimensions: decision coupling, temporal reasoning, and spatial representation, which provide a fundamental framework to compare and contrast recent advancements.
Decision-coupled world models, such as [32, 43], learn implicit action representations directly from videos without explicit control inputs. Building upon this line of work, Ye et al. [40] extend [32] by introducing visual-language-action models that integrate video observations and language instructions through behavior cloning. Similarly, [30] explores knowledge learning such as basic rules, reasoning and planning through compact representation of visual changes. In contrast, general-purpose world models [4, 7, 8] are designed as task-agnostic simulators, emphasizing transferability and generalization over domain-specific tasks [22]. For instance, Genie [4] represents the first interactive world model trained in a fully unsupervised manner. It discretizes latent actions into a set of codebooks to ensure human playability and maintain controllability. Closest to our work is AdaWorld [8], which introduces an adaptable world model framework based on continuous latent actions, enabling transfer across variable contexts in natural and video game domain. We extend this work by learning actions at a coarser level that better suits the hierarchical nature of surgery.
Surgical World Models
World modeling has also begun to pave the way toward surgical dynamics modeling. For instance, [23, 24] introduced a DreamerV2-based [12] world model designed to learn visuomotor policies for surgical grasping tasks. Their approach leverages target object segmentation and the corresponding depth information to guide the robot gripper during manipulation. Notably, the predictions of the world model are conditioned on robotic actions available either in simulation or with real robots.
In contrast, works such as [2, 33] base their predictions on high-level descriptions of actions. Rapuri et. al. [29] makes use of additional conditioning signals in the form of affordance and tool trajectory. Turkean et. al. [33] developed suturing world models that use diffusion to generate videos of suturing tasks conditioned on sub-stitch action annotations. Similarly, Biagini et. al. [2] synthesize laparoscopic videos conditioned on hierarchical representation of the scene in the form of: surgical phase, action triplet and segmentation of the first frame.
COSMOS-Surg-DVRK [42], represents the first foundation world model based on Cosmos WFM [1] developed for evaluating the policy of the surgical robot. The model is obtained by finetuning Cosmos-Predict2-2B-Video2World on video-action paired samples derived from cholecystectomy dataset. While this model effectively learns surgical scene dynamics and action-conditioned predictions, it again makes use of ground truth robotic actions. Conversely, Cosmos-H-Surgical [13] attempts to circumvent ground-truth requirements by using an Inverse Dynamics Model (IDM) to generate pseudo-kinematics. However, this approach remains fundamentally limited as it still relies on kinematic action labels for training the IDM, restricting its efficacy to the specific robot embodiment used during data collection. SurgWM [21] learns implicit action representations from videos but does not investigate their transferability across surgical contexts.
Methodology
We study transferring the learned latent action from a given surgical source video to a given target surgical context, which is represented by a single frame . This produces , the predicted outcome of applying the source actions on the target context. To achieve this, following previous unsupervised works in world modeling [4, 8], we employ two major components: a latent action model (LAM) that learns latent actions between frames, and a dynamics model that predicts future frames conditioned on past observations and actions. However, in contrast to previous works, we learn a single composite latent action between and during training, which can be thought of as a coarser action that captures the overall dynamics in the video. We will look at these components in detail in the subsequent sections.
Latent Action Model (LAM)
First, we will formulate the current latent action models in the literature under a common framework. Note that, even though a single observation typically refers to a single frame in the literature, for the purposes of this section, we will take the convention to use an observation to refer to either a single frame or a series of contiguous frames depending on the type of LAM used.
A latent action model consists of an encoder and a decoder . The encoder takes as inputs observations to infer the action between the observations as i.e. . The decoder takes observation and the inferred action to predict observation i.e. .
The model is trained to minimize the reconstruction loss between and . The latent action can be either discrete [4, 30], in which case the model is trained with a VQ-VAE objective [35], or continuous [8], in which case the model is trained with a VAE objective [20]. Previous works [4] typically use a number of historical frames as and a single future frame as . We refer to models using as Autoregressive LAM models, and as AdaWorld [8] LAM model as shown in Figure 2a and Figure 2b. For continuous actions, the corresponding objective function can be defined as:
Figure 2. Comparison of different Latent Action Models. SurgTransfer (multi-step LAM) learns composable action i.e that encodes long horizon changes.
where,
However, since the model only predicts one frame at a time, the model only learns to encode the fine-grained motion between frames and as action. Moreover, because of the temporal redundancy in the video, the two frames have a high degree of overlap, making the prediction task easier. Thus, the learned actions are inherently myopic, failing to capture the long-range dynamics in surgical videos.
Multi-step LAM
The discussion above motivates us to reframe the action learning framework as predicting frames at a time conditioned on a single frame of context, so that it models long range interactions in the environment. This forces the model to learn higher level actions rather than low level motion encodings. Thus, we can introduce our latent action model by setting , and as shown in Figure 2c. The corresponding objective then becomes:
We minimize this objective for all using causal architecture of the encoder and decoder. For example, to reconstruct , the decoder must use the initial context frame and the corresponding actions , while to reconstruct , the decoder must use the actions . Here, the additional action should still encode the change between the frames and like the previous LAM models. However, since the decoder only has access to the first frame the model should learn the actions (the change between and ) and such that they can be composed together to encode the overall change from to . Our experiments show that the obtained actions show higher invariance to a change of context.
Here, we have used continuous actions to represent more nuanced actions possible in surgery. We also employ the -term introduced in -VAE [15], to balance the expressiveness and the information content of the action.
Multi-Step Dynamics Model
A dynamics model predicts observation given observation and action . Similar to LAM, we can define different dynamic models in terms of the past observations they condition on and the future predictions they make.
Existing dynamics models [4, 8] are autoregressive, where represents , and represents . Such models, however, are prone to autoregressive deterioration, as they condition on their own generated frames during inference as shown in Figure 7. Recognizing that our action transfer framework only requires the dynamics model to predict a full video from a single image rather than modeling every inter-frame transition, we instead formulate our task as predicting video given an image and a composite action, i.e. with and as shown in Figure 3.
Figure 3. SurgTransfer dynamics model conditions on the composite action representation.

Figure 7. Action transfer: source represents the source clip from which the action is inferred, the first frame (highlighted in light green box) represents the context, rest of the frames are generated using the context and transferred actions
To this end, we finetune the publicly available image-to-video Stable Video Diffusion (SVD) [3] model, which is shown to be action-controllable, multi-view consistent, and generalizable [36]. The UNet [31] is trained to also condition on the action in addition to the CLIP [28] image embedding of the conditioning frame and the channel-wise concatenation of the noise-augmented conditioning frame used by the original SVD. We adopt the EDM framework [18] for finetuning, following the original SVD training setup. A simplified form of the training objective of the diffusion dynamics model can be expressed as:
where denotes the frames , and denotes its denoised prediction by the model.
Experiments
Datasets and Baselines
We evaluated SurgTransfer on two of the publicly available surgical datasets, SurgToolLoc-2022 [45] and PitVis [6]. The SurgToolLoc-2022 comprises 24,695 video clips of robot-assisted surgical training exercises performed with the da Vinci Surgical System. Whereas, PitVis contains 25 videos with an average length of over an hour of endoscopic pituitary surgeries.
AdaWorld [8] is currently the only framework that investigates transferability in world modeling, and thus serves as our primary baseline for evaluating video generation. Other works are not directly comparable for action transfer evaluation. Additionally, we have reported the relevant metrics from SurgWМ [21] which conducted their experiments in the same dataset. We also evaluate different combinations of latent action and dynamics models. For LAM, we study three variants: Autoregressive, Adaworld, and SurgTransfer as shown in Figure 2. Autoregressive LAM can either be discrete [4] or continuous. Following [8], we also employ the UniMatch [38] model to use the optical flow between frames as actions for an additional baseline. For dynamics models, we compare the AdaWorld’s autoregressive and SurgTransfer’s multi-step diffusion models. We train all of these variants under the same protocol as SurgTransfer and assess their performance on the SurgToolLoc-2022 dataset in terms of cross-context adaptation. We only compare AdaWorld and SurgTransfer on the PitVis dataset owing to computational constraints. We use 80:20 train:test split on both datasets.
Implementation Details
We utilize the ST-Transformer [39] for both encoder and decoder of the latent action model. Both modules comprise of 12 blocks of spatial and temporal attention layers with 512 channels and 8 attention heads. We use 32 dimensional latent actions with set to . The latent action models are trained for 100k steps on 2 NVIDIA A100 GPUs. Both LAMs and dynamics models are trained with window size 8, unless otherwise mentioned. The diffusion model is finetuned for 80k steps on 8 NVIDIA L40 GPUs. For generating frames beyond the window size that was used to train the dynamics model, the last generated frame is used as the new reference frame. We perform 25 sampling steps for all the diffusion models. All metrics are reported at prediction horizon of length 14.
Evaluation measures
We provide both quantitative and qualitative evaluation protocols to verify the effectiveness of the proposed approach. Our quantitative evaluation focuses on whether the models can learn transferable latent actions. To achieve this, we evaluate the models along two dimensions: the visual plausibility of videos generated through action transfer and the semantic consistency of the latent actions. For the former, we calculate the Content-Debiased Fréchet Video Distance (CD-FVD) [10] between the source and transferred videos. Compared to Fréchet Video Distance (FVD) [34], CD-FVD places greater emphasis on motion dynamics rather than static visual content. To perform better on this metric, the model should maintain high visual fidelity while faithfully transferring the dynamic content of the source videos. We also use Embedding Cosine Similarity (ECS) to complement CD-FVD with frame-level measurements following AdaWorld. To further ensure that our quantitative evaluation measures not only visual plausibility but also semantic consistency during action transfer, we augment these metrics with Action Cycle Consistency metric introduced by [9]. It re-infers the latent actions from the transferred videos and then applies them to the source context, which should closely reproduce the source video if the action transfer works well. Following [9], we report the LPIPS between the original source video and the cyclically reproduced source video.
For PitVis, where step annotations (e.g. nasal corridor creation, tumor excision, debris clearance, etc.) are available, source and target segments are sampled from different surgical videos in such a way that they represent the same surgical step. For SurgToolLoc where step annotations are not available, each surgical clip, which is 30 seconds long, is divided into two temporal segments, forming source–target pairs. Latent actions extracted from the source segments were transferred to the target context image to assess whether the model maintains consistent and context-aware behaviors across similar surgical scenes. This pairing strategy enables a fair comparison by preserving procedural similarity while varying contextual dynamics such as instrument orientation and tissue motion.
We also measure reconstruction fidelity which measures the model’s ability to reproduce video sequences conditioned on implicit latent actions inferred by the LAM. For this purpose we adopt frame-level metrics such as PSNR [16], SSIM [37], LPIPS [44], and CMMD [17]. Unlike FID [14], which depends heavily on the Inception backbone and sample size, CMMD offers a more robust alternative for evaluating video generation quality. It leverages CLIP embeddings in conjunction with MMD distance to compute the score. In our case, we employ PeskaVLP [41] as the embedding model, which is pretrained on a large-scale surgical video–language dataset. This choice also helps mitigate potential distribution mismatches within the surgical domain.
Clinical Evaluation
Following [5], we also employ a clinically-grounded framework that use expert evaluations on different criteria that are specifically designed for evaluating world models on surgical plausibility necessary for meaningful clinical applications. This framework not only measures the visual plausibility of generations but also evaluates them on plausibility of instrument operation, environment feedback and surgical feedback. In addition to these four levels, we incorporate an additional category relevant to our action transfer framework: Action transfer plausibility. This is designed to evaluate the faithfulness in preserving the function and surgical intent of the transferred action. We assess a total of forty generated videos independently with five certified surgeons using a rigorous questionnaire.
As shown in Table 1, our model scores higher on all levels of plausibility than AdaWorld. Visual perceptual plausibility is high for both models, with a decreasing trend seen on higher levels of surgical plausibility, as observed by [5] as well. The most dramatic improvement is seen in visual perceptual plausibility reporting an increase of 29% over the baseline. We measure inter-rater reliability using a two-way random-effects, absolute-agreement ICC for raters. Averaged across the five plausibility levels, ICC [26] was 0.51 for AdaWorld and 0.54 for SurgTransfer.
| Plausibility levels | AdaWorld [8] | SurgTransfer |
| Visual Perceptual↑ | ||
| Instrument Operation↑ | ||
| Environment Feedback↑ | ||
| Surgical Intent↑ | ||
| Action Transfer↑ |
Table 1. Clinical evaluation of SurgTransfer and AdaWorld on different levels of surgical plausibility.
Qualitative Evaluation
We visualize the action transfer capability of our model in Figure 7. In each case, we use actions inferred from the source video by LAM and the starting context frame to roll out 14 more frames into the future. We visually inspect motion cues such as occurrences of temporal inconsistency between generated video and original video. We find that SurgTransfer shows better scene consistency and generation quality while transferring actions than AdaWorld. Particularly, SurgTransfer does not suffer as much from progressive deterioration that is seen in AdaWorld and also demonstrates better long-term temporal consistency by maintaining object identity when objects leave and re-enter the frame as in the third example.
Latent Action Information Visualization
Continuous latent actions offer limited interpretability regarding the information they encode. However, we can track and visualize the trajectory of the tip of the visible instruments in the source and transferred videos, as shown in Figure 6b, to gain partial insights into the information encoded by the latent actions. We follow the same protocol as above on the PitVis dataset to roll out 14 frames in the target context. The instrument tips were labeled manually. As can be seen from the figure, our LAM effectively captures the movements of the instrument. Adaworld, on the other hand, shows the instrument disappearing in frame 5 when a large displacement occurs. Further qualitative evaluation and analysis of the results are provided in the appendix.

Figure 6. Comparison of SurgTransfer and AdaWorld for long-horizon prediction.
Quantitative Evaluations
Cross-context adaptation: Table 2 shows the results of cross context adaptation for different variants of world models. Here, frame-level metrics were not deemed as useful for evaluating cross-context adaptation due to content variation in source and target pair. We obtained significant improvements of our LAM models: lowering CD-FVD by 58% in SurgToolLoc and by 69% in PitVis, and lowering Cyclic LPIPS by 7.48% in SurgToolLoc and 26.59% in PitVis. We can also see the generation quality of our model in comparison to AdaWorld in Figure 7.
| Method | LAM | Dynamics | CD-FVD | ECS | Cyclic Consistency |
| AdaWorld [8] | AdaWorld | AR | 341.76 | 0.53 | 0.321 |
| SurgTransfer | Multi-step | Multi-step | 141.41 | 0.67 | 0.297 |
| Variants | Multi-step | AR | 338.12 | 0.53 | 0.367 |
| AR | Multi-step | 171.41 | 0.61 | 0.322 | |
| Discrete, AR | Multi-step | 361.07 | 0.48 | 0.327 | |
| Optical Flow | Multi-step | 155.92 | 0.60 | 0.321 | |
| AdaWorld [8] | AdaWorld | AR | 513.27 | 0.51 | 0.410 |
| SurgTransfer | Multi-step | Multi-step | 157.34 | 0.66 | 0.301 |
Table 2. Comparison of SurgTransfer against different baselines on Action Transfer. AR: autoregressive models. All LAMs use continuous actions unless otherwise specified.
Reconstruction Fidelity: Additionally, we evaluate the models on reconstruction of the same video using the inferred ground truth actions from the LAM, to evaluate the information content captured by the latent actions and the resulting visual fidelity. We calculate several metrics to measure video generation quality of the different models in Table 3. Since SurgWM also studied video fidelity for reconstruction task from inferred actions, we include the relevant metrics available from the paper. We see a significant gap in performance between autoregressive and multi-step dynamics model for high quality fidelity generation. We also find that our LAM model improves the prediction ability of both dynamics models. We also see strong performance of the optical flow baseline on the reconstruction task. This can be attributed to optical flow being able to encode a large amount of information about the future frame in the action; a strategy that does not work well for cross-context action transfer.
| Method | LAM | Dynamics | PSNR↑ | SSIM↑ | LPIPS↓ | CMMD↓ | CD-FVD↓ | ECS↑ |
| SurgWM [21] | Discrete, AR | AR MaskGIT | 16.49 | 0.39 | - | - | - | - |
| AdaWorld [8] | AdaWorld | AR | 19.098 | 0.576 | 0.321 | 1.23 | 384.205 | 0.576 |
| SurgTransfer | Multi-step | Multi-step | 20.572 | 0.635 | 0.247 | 0.713 | 176.133 | 0.747 |
| Variants | Multi-step | AR | 19.525 | 0.596 | 0.308 | 1.032 | 376.486 | 0.576 |
| AR | Multi-step | 19.398 | 0.602 | 0.273 | 0.68 | 224.156 | 0.657 | |
| Discrete, AR | Multi-step | 18.031 | 0.565 | 0.303 | 0.601 | 374.51 | 0.482 | |
| Optical Flow | Multi-step | 19.756 | 0.626 | 0.256 | 0.666 | 166.973 | 0.72 |
Table 3. Comparison of SurgTransfer against baselines on reconstruction for SurgToolLoc [45].
Key-findings
Key-finding 1: Multi-step dynamics model enables higher-fidelity and more consistent video generation. We can see improved performance of multi-step dynamics model in terms of both action transfer and reconstruction fidelity as shown in Table 2 and Table 3. There exists significant gap from autoregressive dynamics model to multi-step dynamics model for both tasks. This is due to the autoregressive deterioration inherent in these models that results in lower quality generations as the prediction horizon increases. Specifically, visual comparisons between AdaWorld and SurgTransfer show that instrument tips frequently vanish, and tool motion exhibits significant deviations from the ground truth as can be seen in Figure 7. SurgTransfer also poses strong consistency both in spatial and temporal domain, highly matching the procedural dynamics between source and target surpassing AdaWorld in all examples.
Key-finding 2: Better alignment of LAM model with diffusion model improves performance. We investigated the primary cause of the performance drop observed in autoregressive LAM compared to SurgTransfer for the same multi-step dynamics model. Specifically, we analyzed how the current frame is influenced by its predecessor frames and how the LAM decoder enforces learning of desirable temporal changes. This was achieved by plotting the temporal attention maps for each frame, as shown in Figure 4. Our analysis revealed that autoregressive LAM relies more heavily on the immediate previous frame than on the initial frame, which likely leads to misalignment with the Multi-step dynamics model. In contrast, SurgTransfer places greater emphasis on the initial frame, enabling better latent action encoding that ultimately enhances the diffusion model’s performance.

Figure 4. Comparison of temporal attention maps between SurgTransfer and Autoregressive LAM models. Autoregressive LAM shows greater reliance on the preceding frame.
Key-finding 3: SurgTransfer LAM learns better temporal abstractions for action transfer. As can be seen from the Table 2, the SurgTransfer LAM captures actions that give higher fidelity generations and show higher context invariance regardless of the dynamics models used. This shows that our latent action model is better in capturing long horizon dynamics that are inherent in the surgical procedures. LAMs that predict single frame into the future i.e. Autoregressive LAM and AdaWorld LAM encode highly temporally redundant actions that may not necessarily compose together to encode the overarching semantic, whereas optical flow only captures pixel level changes between frames and is thus insufficient as well. This is in contrast to the SurgTransfer LAM which places an emphasis on composite actions that must encode the full surgical evolution.
Ablation Studies
First, to further test the assumption that longer horizon modeling in Latent Action Model translates to better ability to capture surgical actions, we train SurgTransfer LAM models for different context lengths ranging from 2 to 16 while using the same multi-step prediction dynamics model. After the LAM infers actions from frames, its context is reset to infer further actions. Note that corresponds to using the AdaWorld LAM. Second, we study how the per frame generation quality varies as the prediction horizon increases for both the single-step and multi-step dynamics model.
Impact of context length in latent action model: As shown in Figure 5, performance improves significantly with longer horizon modeling in LAM even though the generative modeling ability of the dynamics model is unchanged, which shows that longer horizon action modeling of SurgTransfer LAM is better suited for capturing surgical actions.
Generation quality vs prediction horizon: As shown in Figure 6a, both SurgTransfer and AdaWorld show deteriorating performance as the number of generated frames increases. But AdaWorld exhibits faster fall-off in quality from the start, particularly in CMMD, due to its autoregressive nature. We can also see a sudden drop in quality for the 8th generated frame for SurgTransfer, since the model generates 7 frames at a time, with the model resetting its starting context to the 7th frame to generate the additional 8th to 14th frames.
Figure 5. Increasing context length leads to improved performace in action modeling.
Conclusion and Future Work
Our work represents an initial step towards general expertise transfer from surgical demonstrations. Our novel multi-step action learning framework learns better temporal abstractions to generate high fidelity simulations of transferring actions across surgical contexts, which we have validated through extensive experiments. However, it is only applicable to surgical contexts with similar anatomy or view having the same procedure and instrument embodiment. Our work is also limited to unsupervised setting, which, while beneficial for scalability, offers limited insights about potential performance in downstream tasks where ground truth actions are available. Furthermore, the proposed framework utilizes pre-defined latent action trajectory which prevents a truly interactive experience. Future work could explore adapting these models to specific action spaces of specialized surgical tasks.
Implementation Details
Latent Action Model The encoder takes in frames as inputs. Each input frame is divided into visual tokens using a patch size of (16, 16) and learnable positional embeddings are applied. The resulting visual tokens are concatenated in time dimension with learnable action prompts of model dimension. After the encoder processes the tokens using causal spatial temporal layers, the first action token is discarded and the rest of the tokens are used to estimate the posterior distribution of actions: . We then sample the 32 dimensional action representations . For the Multi-Step LAM Model, the sampled actions are concatenated in time dimension with copies of the first frame . The resulting tokens are again processed with the causal decoder blocks to get the predicted frames .
Dynamics Model The Stable Video Diffusion model is modified so that in addition to the original conditionings, each frame also conditions on the action that comes from the LAM encoder. The conditioning on action is done in the form of both cross-attention and scale-shift normalization. It takes 3.61 seconds to generate 7 frames (i.e. at 1.94 FPS) using single NVIDIA L40 GPU with peak VRAM usage of 15.43 GB.
Transfer Metrics for Disparate Domains
Although sampling source and target clips from different videos in the SurgToolLoc dataset might not maintain procedural similarity, we can also evaluate the model’s ability to transfer the action trajectories even when the instrument presences and contextual cues are markedly different. As shown in Table 4, SurgTransfer still shows superior performance over the other variants.
| Model | CD-FVD↓ | ECS↑ |
| AdaWorld [8] | 432.45 | 0.416 |
| Multi-step LAM + Autoregressive Dynamics | 416.278 | 0.418 |
| Autoregressive LAM + Multi-step Dynamics | 246.316 | 0.471 |
| SurgTransfer | 233.455 | 0.508 |
Table 4. Comparison on Action Transfer for disparate domains
Additional Qualititative Results
We present additional samples for SurgTransfer and AdaWorld of action transfers across same and different videos on the SurgToolLoc dataset in figures 8 and 9 respectively.
Clinical Evaluation Details
We provide the full questionnaire used for clinical evaluation of the action transfer in this section.
Instructions:
You will be shown a real surgical video (source) and a surgical video generated by an AI model (target) by following the source actions in a new target context.
Please assess whether the AI-generated target video preserves surgical reality, causality and intent while performing the transformation.
Visual Perceptual Plausibility
Does this transformed video still look like real surgery?
The target video is visually clear, stable and free from distracting artifacts
Lighting, color balance and camera behavior are consistent with a real operative setting
Tissue textures and surfaces in the target video appear realistic.
Temporal continuity is preserved without flicker, jitter or visual discontinuity.
Instrument Operation Plausibility
Do instruments still behave like surgical instruments after transfer?
Instruments in the target video are anatomically and procedurally appropriate
Instrument trajectories and movements remain physically plausible
No instrument behavior appears newly introduced or corrupted by the transformation
Instrument co-ordination reflects realistic surgical technique
Environment feedback plausibility
Does tissue respond correctly to action in the transformed video?
Tissue deformation in the target video corresponds appropriately to instrument interaction
Bleeding, fluid dynamics or coagulation responses (if present) are realistic
Anatomical relationships remain consistent with known surgical anatomy

Figure 8. Action transfer (source and target from different parts of same video): source represents the source clip from which the action is inferred, the first frame represents the context, rest of the frames are generated using the context and transferred actions. In the first example, for SurgTransfer, we can see the needle driver on the bottom right holding the thread as in the source video. In the second example SurgTransfer replicates the grasping and the subsequent release of a tissue by bipolar forceps as opposed to AdaWorld.

Figure 9. Action transfer (source and target taken from different videos): source represents the source clip from which the action is inferred, the first frame represents the context, rest of the frames are generated using the context and transferred actions. In the first example, both models show increase in brightness for the visible instrument in later frames as in the source video, showing that the models have learned to transfer visual intensity as well. In the second example, the motion of monopolar curved scissors in the source video is transferred to the needle driver present in the target context. Here SurgTransfer can more effectively replicate the sudden movements of the instrument than AdaWorld.
No tissue response violates physical or medical common sense
Surgical intent plausibility
Is the surgical goal preserved across the transformation?
The overall surgical intent in the target video matches that of the source video
The procedural stage implied by the target video is appropriate
The sequence of actions reflects coherent surgical rea-
soning rather than visual mimicry
No new or conflicting surgical intent has been introduced by the AI transformation
Source-target fidelity
Is the surgical goal preserved across the transformation?
Key anatomical structures are preserved between source and target videos.
Instrument identity and functional role are preserved across the transformation.
The temporal order of surgical actions is preserved.
No clinically meaningful information appears lost or distorted in the target video.
Algorithm
We provide the detailed algorithms for training the models and performing action transfer in algorithms 1, 2, and 3.
Algorithm 1 Multi-step Latent Action Learning
Require: Unlabeled surgical videos ; prediction window ; LAM encoder ; LAM decoder ; KL weight
for each training iteration do
Sample a clip from
for to do
Infer multi-step latent actions
Reconstruct the future using only the initial context and the inferred actions
Compute the multi-step LAM objective
end for
Update by minimizing
end for
Algorithm 2 Multi-step Action-conditioned Dynamics Learning
Require: Unlabeled surgical videos ; LAM encoder ; diffusion dynamics model
for each training iteration do
Sample a clip from
Infer the composite latent action trajectory
Let be the target future video
Sample diffusion noise and noise level , and construct
Predict the denoised video with the dynamics model
Update by minimizing
end for
Latent Action Information Visualization
We provide further examples of instrument trajectory visualization in Figure 10 and Figure 11.
Figure 10. Here, SurgTransfer tightly tracks the relative tip displacements showing high adherence to the conditioning action signal.
Figure 11. Although SurgTransfer has high error in tracking tip displacement, we can see that the error is due to over-amplification of the movements in the source video suggesting that the model can be prone to overweighing the conditioning action signal.
Algorithm 3 Cross-context Surgical Action Transfer
Require: Source demonstration ; target context frame ; trained LAM encoder ; diffusion dynamics model with being the number of generated frames per single pass of the diffusion model.
Ensure: Transferred target video
Initialize context frame
for each prediction block do
Extract source block
Infer source latent actions
Generate the corresponding target future block
Set the last generated frame as the context for the next block
end for
return transferred target video
References
- [1]Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025. 2arxiv.org/abs/2501.03575
- [2]Diego Biagini, Nassir Navab, and Azade Farshad. Hierarchy-aware diffusion model for surgical video generation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 310–319. Springer, 2025. 1, 2arxiv.org/abs/2506.21287
- [3]Andreas Blattmann, Tim Dockhorn, Smith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 4arxiv.org/abs/2311.15127
- [4]Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024. 1, 2, 3, 4arxiv.org/abs/2402.15391
- [5]Zhen Chen, Qing Xu, Jinlin Wu, Biao Yang, Yuhao Zhai, Geng Guo, Jing Zhang, Yinlu Ding, Nassir Navab, and Jiebo Luo. How far are surgeons from surgical world models? a pilot study on zero-shot surgical video generation with expert assessment. arXiv preprint arXiv:2511.01775, 2025. 5arxiv.org/abs/2511.01775
- [6]Adrito Das, Danyal Z Khan, Dimitrios Psychogyios, Yitong Zhang, John G Hanrahan, Francisco Vasconcelos, You Pang, Zhen Chen, Jinlin Wu, Xiaoyang Zou, et al. Pitvis-2023 challenge: Workflow recognition in videos of endoscopic pituitary surgery. Medical Image Analysis, page 103716, 2025. 4DOI
- [7]Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. Advances in Neural Information Processing Systems, 37:91560–91596, 2024. 2
- [8]Shenyuan Gao, Siyuan Zhou, Yilin Du, Jun Zhang, and Chuang Gan. Adaworld: Learning adaptable world models with latent actions. arXiv preprint arXiv:2503.18938, 2025. 1, 2, 3, 4, 5, 6, 7arxiv.org/abs/2503.18938
- [9]Quentin Garrido, Tushar Nagarajan, Basile Terver, Nicolas Ballas, Yann LeCun, and Michael Rabbat. Learning latent action world models in the wild. In Forty-third International Conference on Machine Learning. 5arxiv.org/abs/2601.05230
- [10]Songwei Ge, Aniruddha Mahapatra, Gaurav Parmar, Jun-Yan Zhu, and Jia-Bin Huang. On the content bias in fr’échet video distance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 5arxiv.org/abs/2404.12391
- [11]David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, 2018. 1arxiv.org/abs/1809.01999
- [12]Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2021. 1, 2arxiv.org/abs/2010.02193
- [13]Yufan He, Pengfei Guo, Mengya Xu, Zhaoshuo Li, Andriy Myronenko, Dillan Imans, Bingjie Liu, Dongren Yang, Mingxue Gu, Yongnan Ji, Yueming Jin, Ren Zhao, Baiyong Shen, and Daguang Xu. Cosmos-h-surgical: Learning surgical robot policies from videos via world modeling, 2026. 2
- [14]Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in neural information processing systems, 30, 2017. 5
- [15]Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2017. 4
- [16]Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pages 2366–2369. IEEE, 2010. 5DOI
- [17]Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9307–9315, 2024. 5arxiv.org/abs/2401.09603
- [18]Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35:26565–26577, 2022. 4arxiv.org/abs/2206.00364
- [19]Ji Woong Kim, Juo-Tung Chen, Pascal Hansen, Lucy Xiao-aoyang Shi, Antony Goldenberg, Samuel Schmidgall, Paul Maria Scheikl, Anton Deguet, Brandon M White, De Ru Tsai, et al. Srth-a: A hierarchical framework for autonomous surgery via language-conditioned imitation learning. Science robotics, 10(104):eadt5254, 2025. 2arxiv.org/abs/2505.10251
- [20]Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 3arxiv.org/abs/1312.6114
- [21]Saurabh Koju, Saurav Bastola, Prashant Shrestha, Sanskar Amgain, Yash Raj Shrestha, Rudra PK Poudel, and Binod Bhattarai. Surgical vision world model. In MICCAI Workshop on Data Engineering in Medical Imaging, pages 1–10. Springer, 2025. 1, 2, 4, 7DOI
- [22]Xinqing Li, Xin He, Le Zhang, and Yun Liu. A comprehensive survey on world models for embodied ai. arXiv preprint arXiv:2510.16732, 2025. 2arxiv.org/abs/2510.16732
- [23]Hongbin Lin, Bin Li, Chun Wai Wong, Juan Rojas, Xiangyu Chu, and Kwok Wai Samuel Au. World models for general surgical grasping. arXiv preprint arXiv:2405.17940, 2024. 1, 2arxiv.org/abs/2405.17940
- [24]Hongbin Lin, Bin Li, and Kwok Wai Samuel Au. Visuomotor grasping with world models for surgical robots. arXiv preprint arXiv:2508.11200, 2025. 1, 2arxiv.org/abs/2508.11200
- [25]Xiaoxiao Long, Qingrui Zhao, Kaiwen Zhang, Zihao Zhang, Dingrui Wang, Yumeng Liu, Zhengjie Shu, Yi Lu, Shouzheng Wang, Xinzhe Wei, et al. A survey: Learning embodied intelligence from physical simulators and world models. arXiv preprint arXiv:2507.00917, 2025. 2arxiv.org/abs/2507.00917
- [26]Kenneth O McGraw and Seok P Wong. Forming inferences about some intraclass correlation coefficients. Psychological methods, 1(1):30, 1996. 5DOI
- [27]Rudra P.K. Poudel, Harit Pandya, Stephan Liwicki, and Roberto Cipolla. Recore: Regularized contrastive representation learning of world model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 1arxiv.org/abs/2312.09056
- [28]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021. 4arxiv.org/abs/2103.00020
- [29]Sampath Rapuri, Lalithkumar Seenivasan, Dominik Schneider, Roger Soberanis-Mukul, Yufan He, Hao Ding, Jiru Xu, Chenhao Yu, Chenyan Jing, Pengfei Guo, Daguang Xu, and Mathias Unberath. Saw: Toward a surgical action world model via controllable and scalable video generation, 2026. 2arxiv.org/abs/2603.13024
- [30]Zhongwei Ren, Yunchao Wei, Xun Guo, Yao Zhao, Bingyi Kang, Jiashi Feng, and Xiaojie Jin. Videoworld: Exploring knowledge learning from unlabeled videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 29029–29039, 2025. 2, 3arxiv.org/abs/2501.09781
- [31]Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015. 4arxiv.org/abs/1505.04597
- [32]Dominik Schmidt and Minqi Jiang. Learning to act without actions. arXiv preprint arXiv:2312.10812, 2023. 2arxiv.org/abs/2312.10812
- [33]Mehmet Kerem Turckan, Mattia Ballo, Filippo Filicori, and Zoran Kostic. Towards suturing world models: Learning predictive models for robotic surgical tasks. arXiv preprint arXiv:2503.12531, 2025. 1, 2
- [34]Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marc Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 5arxiv.org/abs/1812.01717
- [35]Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017. 3arxiv.org/abs/1711.00937
- [36]Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. In European Conference on Computer Vision, pages 439–457. Springer, 2024. 4arxiv.org/abs/2403.12008
- [37]Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simončelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004. 5pubmed.ncbi.nlm.nih.gov/15376593
- [38]Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying flow, stereo and depth estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11):13941–13958, 2023. 4DOI
- [39]Mingxing Xu, Wenrui Dai, Chunmiao Liu, Xing Gao, Weiyao Lin, Guo-Jun Qi, and Hongkai Xiong. Spatial-temporal transformer networks for traffic flow forecasting. arXiv preprint arXiv:2001.02908, 2020. 4arxiv.org/abs/2001.02908
- [40]Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, et al. Latent action pretraining from videos. arXiv preprint arXiv:2410.11758, 2024. 2arxiv.org/abs/2410.11758
- [41]Kun Yuan, Nassir Navab, Nicolas Paday, et al. Procedure-aware surgical video-language pretraining with hierarchical knowledge augmentation. Advances in Neural Information Processing Systems, 37:122952–122983, 2024. 5arxiv.org/abs/2410.00263
- [42]Lukas Zbinden, Nigel Nelson, Juo-Tung Chen, Xinhao Chen, Ji Woong, Mahdi Azizian, Axel Krieger, Sean Huver, et al. Cosmos-surg-dvrk: World foundation model-based automated online evaluation of surgical robot policy learning. arXiv preprint arXiv:2510.16240, 2025. 1, 2arxiv.org/abs/2510.16240
- [43]Lixuan Zhang, Meina Kan, Shiguang Shan, and Xilin Chen. Prelar: World model pre-training with learnable action representation. In European Conference on Computer Vision, pages 185–201. Springer, 2024. 1, 2DOI
- [44]Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018. 5arxiv.org/abs/1801.03924
- [45]Aneeq Zia, Kiran Bhattacharyya, Xi Liu, Max Berniker, Ziheng Wang, Rogerio Nespolo, Satoshi Kondo, Satoshi Kasai, Kousuke Hirasawa, Bo Liu, et al. Surgical tool classification and localization: results and methods from the miccai 2022 surgtooloc challenge. arXiv preprint arXiv:2305.07152, 2023. 4, 7




