betaFeedback

WorksComputer Science

Complexity-Aware Surgical Video Reasoning Segmentation with Digital Twin and Context Memory

Abstract

Conventional surgical segmentation relies on closed-set categories or explicit text queries, which cannot resolve implicit queries that require reasoning about workflows, instrument functions, and temporal relationships. To address these limitations, we propose SurgReasonSeg, a complexity-aware method that brings reasoning segmentation over digital twin (DT) representations to surgical video. Specifically, SurgReasonSeg constructs DT representations from surgical foundation models and trains two low-rank adapters (LoRA) of one large language model with reinforcement learning (RL), each specialized for a different query complexity. A lightweight router, also trained with RL, selects between the two adapters to balance latency against accuracy. SurgReasonSeg also sets the temporal context with an adaptive sliding window conditioned on query complexity and workflow phase, and retrieves relevant historical cases from a surgical context memory by workflow similarity and procedural relevance. We further introduce EndoVis-ReasonSeg, a surgical reasoning segmentation benchmark with 1,460 training and 961 test samples across semantic, spatial, and temporal categories. On EndoVis-ReasonSeg, SurgReasonSeg improves J&F over the strongest baseline, ReSurgSAM2, by 0.178, 0.237, and 0.297 on semantic, spatial, and temporal queries, and it also achieves the best results on the Ref-EndoVis17 and Ref-EndoVis18 referring segmentation benchmarks. A backbone-matched attribution protocol separates the contribution of the upstream segmenter from that of the reasoning modules. Our method runs at 18.8 FPS averaged over the query distribution, enabling real-time reasoning segmentation in surgical videos.

Introduction

Surgical autonomy requires precise segmentation of anatomical structures and instruments in endoscopic videos [1], yet surgeons often express their intent through functional and contextual descriptions tied to the ongoing procedure rather than through explicit anatomical labels [23]. Conventional segmentation methods rely on pre-defined closed-set categories [12], restricting them to a fixed vocabulary that cannot accommodate such intent-driven communication. Referring segmentation [10] partially mitigates this by identifying objects through explicit text queries [15, 27, 35], yet still requires the surgeon to name the target. In practice, many surgical tasks require interpreting implicit queries that involve reasoning about workflows, instrument functions, and temporal procedural relationships [17]. For instance, rather than “the grasper”, a surgeon may ask for “the instrument retracting tissue for dissection”, expecting the target to be inferred from workflow context and instrument roles, a task that reasoning segmentation addresses through multi-step inference [31].

Existing reasoning segmentation methods such as LISA [11] and VISA [31] pair multimodal large language models (LLMs) with segmentation decoders, yet are trained on general-domain data and lack surgical knowledge [23]. While fine-tuning on surgical reasoning segmentation data could offer a partial solution, it requires costly annotations and yields a model whose knowledge is fixed at training time. Existing methods also cannot retrieve relevant historical context from long procedures, either processing all information uniformly or discarding past observations [3, 8]. Finally, surgical queries vary in reasoning complexity, from identifying an object to understanding the workflow [30], yet existing methods allocate the same computation to every query and spend unnecessary latency on simple ones.

To address these limitations, we propose SurgReasonSeg, an online video reasoning segmentation method built on digital twin (DT) representations [20, 21, 22, 24], which serve as intermediate abstractions between raw video and reasoning. SurgReasonSeg constructs the DT with surgical foundation models such as SurgSAM2 [14] and RASO [12], incorporating surgical domain knowledge directly into the representation. The DT-as-JSON encoding and the reduction of segmentation to index retrieval are inherited from prior DT reasoning segmentation methods [20, 21, 23, 24],

Adaptive Window Sensitivity

not proposed here. Tab. 1 lists what is inherited and what is new, and Tab. 5 quantifies the contribution of each part. To handle varying query complexity, we train two low-rank adapters (LoRA) [6] on the same base LLM with reinforcement learning (RL) [19], one producing extended reasoning chains and the other concise ones. A lightweight router, also trained with RL, selects between the two adapters to balance latency against accuracy. We also propose an adaptive sliding window that adjusts the length of the temporal context to the query complexity and the workflow phase, so that the LLM receives only as much context as the query requires. To reuse experience accumulated across cases, we introduce a surgical context memory that stores past DT representations together with their queries and outcomes. At inference, the memory retrieves past cases by workflow similarity and procedural relevance and supplies them to the LLM as in-context examples.

The major contributions are threefold, summarized as follows. First, we propose SurgReasonSeg, an online video reasoning segmentation method that adapts DT-based reasoning segmentation to surgical video by constructing the DT from surgical foundation models. Specifically, two RL-trained LoRA adapters handle queries of different complexity, and a router selects between them at inference. Second, we introduce an adaptive sliding window that sets the temporal context from the query complexity and the surgical workflow phase, and a surgical context memory that retrieves past cases by workflow similarity and procedural relevance. Finally, we contribute EndoVis-ReasonSeg, a surgical reasoning segmentation benchmark for training and evaluation. We also report a backbone-matched attribution study that separates the contribution of the upstream segmenter from that of the reasoning modules, together with a template-disjoint evaluation that tests for template-level leakage.

Related Work

Reasoning Segmentation. Reasoning segmentation aims to identify and segment objects from implicit text queries that do not name the target explicitly. LISA [11] attaches a segmentation decoder to a multimodal LLM through a learned [SEG] token, GSVA [29] generalizes it to multiple and empty targets, CoReS [2] interleaves chain-of-thought reasoning with mask decoding, and SegZero [16] trains the reasoning chain through RL. For videos, VISA [31] selects a keyframe before propagating masks, while video LLMs [13, 18, 34] compress frames into token sequences. However, these methods encode the video as visual tokens and are trained on general-domain corpora, leaving the surgical vocabulary and procedural priors that implicit surgical queries depend on unmodeled.

Digital Twin Representations. Digital twins provide a structured, symbolic abstraction of a physical scene [7, 9] that recent work uses as the interface between perception and LLM reasoning [22]. In surgery, DT scene representations built from foundation models support task-level automation [5] and robust phase recognition [4], and Shen et al. [23] first perform reasoning segmentation over DTs for operating room workflow analysis. The DT-as-JSON encoding with masks stored as file paths, together with the reduction of segmentation to index retrieval, was introduced by JiT [24] and subsequently refined for general visual reasoning [20] and for efficiency [21]. Outside vision, generative transformers have also modeled structured sequences such as entity trajectories [25]. SurgReasonSeg inherits this DT formulation, and Table 1 lists which of its components are inherited, adapted, or new.

Surgical Video Segmentation. Surgical segmentation has progressed from closed-set categories [12, 14] to referring formulations, where RSVIS [27] exploits video-level consistency and ResSAM2 [15] couples a surgical encoder with SAM2 tracking to reach real-time speed. Yet both require the surgeon to name the target, and neither reasons over workflow phases or instrument function. Temporal context has been modeled through memory banks for phase recognition [3, 8] and adaptive retrieval for long videos [30], yet neither is conditioned on query complexity.

Methods

As shown in Figure 1, SurgReasonSeg combines four components, namely DT construction, an adaptive sliding window, a surgical context memory, and complexity-aware reasoning with a router. The window selects the temporal context, the memory supplies similar past cases as in-context examples, and the router sends each query to one of two LoRA adapters, which generates a reasoning chain followed by the index of the target mask.

Figure 1. Overview of SurgReasonSeg. Given a surgical video and an implicit text query, our method constructs DT representations from surgical foundation models, applies an adaptive sliding window for temporal context selection, and routes each query to the LoRA adapter matched to its complexity, which generates the reasoning chain.

Digital Twin Construction. Given a surgical video clip with frames {ft}t=1T\{f_t\}_{t=1}^{T} and text query QQ, SurgReasonSeg first constructs the DT representation DtD_t at each time step tt to capture the surgical scene. First, we apply SurgSAM2 [14] for instrument and tissue instance segmentation, generating instance masks Mt={mit}i=1NtM_t = \{m_i^t\}_{i=1}^{N_t} where each mitm_i^t is the binary @ends_mid=1 mask of object ii in frame tt. The tracking of SurgSAM2 keeps instance identities consistent across frames. Second, we estimate depth with DepthAnything2 [33] and compute the average depth ditd_i^t of each instance. Third, RASO [12] assigns each instance a surgical semantic label litl_i^t. The DT representation at frame tt is formalized as Dt={(mit,dit,lit,cit,bit)}i=1NtD_t = \{(m_i^t, d_i^t, l_i^t, c_i^t, b_i^t)\}_{i=1}^{N_t}, where each tuple contains the mask, average depth, semantic label, centroid citc_i^t, and bounding box coordinates bitb_i^t for object ii. The centroid and bounding box, both derived from the mask, allow the LLM to reason about spatial relations without reading the pixel-level masks. Following prior work [20, 21, 23, 24], we serialize D={Dt}D = \{D_t\} as JSON that stores each mask as a file path instead of a pixel array, which turns reasoning segmentation into mask retrieval in which the LLM identifies the target object by its JSON index.

Complexity-Aware Reasoning with Router

Following prior work [20, 21], our LLM takes the DT representation Dt−w:tD_{t-w:t} from the sliding window and the implicit text query QQ as input to generate a two-component rollout. The rollout first gives a reasoning chain within <think> and </think> tokens that analyzes object attributes and spatial relations in the DT representation. It then gives the JSON index of the target object within <answer> and </answer> tokens. We train the LoRA adapters on the LLM through RL with the reward R=Rformat+RaccR = R_{\mathrm{format}} + R_{\mathrm{acc}}. The format reward RformatR_{\mathrm{format}} assigns 0 to a well-formed rollout and −1-1 otherwise, so that malformed outputs are penalized without format compliance being rewarded on its own. The accuracy reward RaccR_{\mathrm{acc}} equals 1 when the IoU between the retrieved mask and the ground truth exceeds 0.5 and 0 otherwise. The IoU criterion is applied per referent, and each query is authored so that exactly one DT instance satisfies the description at the queried frame. Set-like phrasings are rewritten at the template level to designate a single instance, for example “the primary grasping tool” rather than “instruments active during dissection phase”. To handle varying query complexity, we train two LoRA adapters [6] on the same base LLM that differ in the length of their reasoning chain within the <think> and </think> tokens. The adapter LcomplexL_{\mathrm{complex}} generates extended reasoning for complex queries, while LsimpleL_{\mathrm{simple}} produces concise reasoning for straightforward ones. Each adapter adds an efficiency reward RlengthR_{\mathrm{length}} to RR, defined as

Rlength={−α⋅max⁡(0,∣Len∣−τshort)for LsimpleβLen⋅1(∣Len∣≥τlong)for Lcomplex(1)R_{\mathrm{length}} = \begin{cases} -\alpha\cdot\max(0, \lvert Len\rvert-\tau_{\mathrm{short}}) & \text{for } L_{\mathrm{simple}} \\ \beta_{\mathrm{Len}} \cdot\mathbb{1}(\lvert Len\rvert\ge\tau_{\mathrm{long}}) & \text{for } L_{\mathrm{complex}} \end{cases} \tag*{(1)}

where ∣Len∣\lvert Len\rvert denotes the length of the reasoning chain, τshort\tau_{\mathrm{short}} and τlong\tau_{\mathrm{long}} are the length thresholds, α\alpha and βLen\beta_{\mathrm{Len}} are the weighting coefficients, and 1(∣Len∣≥τlong)\mathbb{1}(\lvert Len\rvert\ge\tau_{\mathrm{long}}) is an indicator function. To select between the two adapters at inference, we then train a lightweight router network ϕrouter\phi_{\mathrm{router}}, implemented as Qwen3-0.6B [32], with RL while both adapters stay frozen. The router takes the query embedding eQe_Q and the DT window Dt−w:tD_{t-w:t} as input and produces a selection score s=ϕrouter(eQ,Dt−w:t)s = \phi_{\mathrm{router}}(e_Q, D_{t-w:t}), where s>0.5s > 0.5 activates LcomplexL_{\mathrm{complex}} and s≤0.5s \le0.5 activates LsimpleL_{\mathrm{simple}}. The router is trained with the reward function Rrouter=Racc−λ⋅TlatencyR_{\mathrm{router}} = R_{\mathrm{acc}} - \lambda\cdot T_{\mathrm{latency}}, where TlatencyT_{\mathrm{latency}} measures the inference time of the selected adapter and λ\lambda balances the segmentation accuracy against computational efficiency. This reward encourages the router to select LsimpleL_{\mathrm{simple}} for straightforward queries to minimize latency while choosing LcomplexL_{\mathrm{complex}} only when extended reasoning is necessary for accuracy. No query is labeled simple or complex by hand, and the specialization is induced only by RlengthR_{\mathrm{length}} with τshort=50\tau_{\mathrm{short}} = 50 and τlong=200\tau_{\mathrm{long}} = 200 tokens, which are reasoning-chain length thresholds rather than complexity labels, and the router learns its decision boundary from RrouterR_{\mathrm{router}} alone. We define the “no Router” configuration used in the ablation as a single LoRA adapter trained on all queries with R=Rformat+RaccR = R_{\mathrm{format}} + R_{\mathrm{acc}} and without the length reward RlengthR_{\mathrm{length}}, so that removing the router also removes the complexity specialization it selects between.

Adaptive Sliding Window. The adaptive sliding window adjusts the temporal context length of DT representations fed to the LLM based on query complexity and surgical workflow phases. At each time step tt, the window size ww that determines Dt−w:tD_{t-w:t} is computed as w=wbase+Δwquery+Δwphasew = w_{\mathrm{base}} + \Delta w_{\mathrm{query}} + \Delta w_{\mathrm{phase}}, where wbasew_{\mathrm{base}} provides a minimal temporal context. The query complexity adjustment Δwquery\Delta w_{\mathrm{query}} extends the window when the router identifies a complex query, namely Δwquery=wquery⋅I(s>0.5)\Delta w_{\mathrm{query}} = w_{\mathrm{query}} \cdot I(s > 0.5). Because this adjustment depends on ss, the router reads the window of wbase+Δwphasew_{\mathrm{base}} + \Delta w_{\mathrm{phase}} frames, both known before routing, and only the selected adapter reads the full window. The workflow phase adjustment Δwphase\Delta w_{\mathrm{phase}} expands the context during surgical transitions, which are detected from changes in the active instruments recorded in the DT. Specifically, Δwphase=wphase\Delta w_{\mathrm{phase}} = w_{\mathrm{phase}} if ∣It⊕It−1∣≥θchange\lvert I_t \oplus I_{t-1} \rvert\ge\theta_{\mathrm{change}} and 0 otherwise, where ItI_t denotes the set of active instruments in frame tt extracted from the DT representation. The operator ⊕\oplus is the symmetric difference, which returns the instruments present in exactly one of the two frames, and θchange\theta_{\mathrm{change}}, set to 1, is the minimum number of instrument changes that triggers a workflow phase transition.

Surgical Context Memory. We maintain a surgical context memory bank M\mathcal{M} that stores DT representations alongside queries and segmentation results. The bank retains only DT attributes, mask paths, and queries instead of raw frames, which limits pixel-level exposure, though these attributes are still derived from patient recordings and require the same governance as the source videos. Each memory entry is a tuple (Dj,Qj,Ij,vj)(\mathcal{D}_j,Q_j,\mathcal{I}_j,v_j) where Dj={Dt(j)}t=1Tj\mathcal{D}_j = \{D^{(j)}_t\}_{t=1}^{T_j} represents the complete DT sequence for video jj, QjQ_j is the text query, Ij\mathcal{I}_j contains segmentation results, and vj∈{0,1}v_j \in\{0,1\} indicates unsuccessful or successful segmentation. The label vjv_j is produced by an external VLM judge, instantiated as GPT-4o, that is shown the retrieved mask overlaid on the frame together with the query and returns a binary verdict on whether the mask satisfies the description. Table S3 of the supplementary replaces vjv_j by the ground-truth IoU to measure how much memory quality depends on the judge. The memory stores both successful and unsuccessful results, updated as M←M∪{(Dj,Qj,Ij,vj)}\mathcal{M} \leftarrow\mathcal{M} \cup\{(D_j,Q_j,\mathcal{I}_j,v_j)\}. To avoid leakage, the bank is built only from training videos and frozen during evaluation, so the update rule applies only at deployment and is not test case can retrieve another. For a query QnewQ_{\mathrm{new}} at frame tt, we retrieve relevant historical context by the workflow similarity Sworkflow(Dt,Dj)=∣It∩Ij∣/∣It∪Ij∣S_{\mathrm{workflow}}(D_t,D_j) = \lvert I_t \cap I_j\rvert/\lvert I_t \cup I_j\rvert, where ItI_t and IjI_j are sets of instruments. We also compute the procedural relevance Sproc(Qnew,Qj)=cos⁡(eQnew,eQj)S_{\mathrm{proc}}(Q_{\mathrm{new}},Q_j) = \cos(e_{Q_{\mathrm{new}}},e_{Q_j}), the cosine similarity between the query embeddings. The retrieval score incorporates quality judgment: Stotal=[β⋅Sworkflow+(1−β)⋅Sproc]⋅(1+η⋅vj)S_{\mathrm{total}} = [\beta\cdot S_{\mathrm{workflow}} + (1-\beta) \cdot S_{\mathrm{proc}}]\cdot(1+\eta\cdot v_j), where η\eta weights successful examples while retaining unsuccessful ones as contrasting references. We retrieve the top-kk memories and prepend their temporal segments with corresponding results as few-shot examples, namely Dt−w:t′=[(Dj1seg,Ij1,vj1);…;(Djkseg,Ijk,vjk);Dt−w:t]D'_{t-w:t} = [(D^{\mathrm{seg}}_{j_1},\mathcal{I}_{j_1},v_{j_1});\ldots;(D^{\mathrm{seg}}_{j_k},\mathcal{I}_{j_k},v_{j_k});D_{t-w:t}], where each tuple provides the DT representation, segmentation result, and quality label, so the LLM sees both successful and unsuccessful cases as in-context examples.

AdaptersRoutervjv_jSemantic ↑\uparrowSpatial ↑\uparrowTemporal ↑\uparrow
SFTRLJudge.742.688.615
RLRuleJudge.731.706.643
RLClassifierJudge_ .762__ .710__ .648_
RLRLUniform.758.702.640
RLRLJudge.771.717.654
RLRLGT IoU.775.721.658

Table S3. Every design choice of SurgReasonSeg (Ours) outperforms its replacement, and the VLM judge is within 0.004 of the ground-truth IoU bound. Adapters: RL or supervised fine-tuning (SFT); Router: RL, query-category rule, or supervised classifier; vjv_j: VLM judge, uniform weighting, or ground-truth IoU; gray: as in the full model. Values are J&F\mathcal{J}\&\mathcal{F}; bold: best; underline: second best; the ground-truth IoU row is an upper bound and is not ranked.

Dataset Construction. We construct a surgical reasoning segmentation dataset called EndoVis-ReasonSeg using the videos and instance segmentation masks from Ref-EndoVis17, Ref-EndoVis18 [15], and RSVIS [27]. We segment each video sequence into overlapping clips of 20 to 40 consecutive frames, where clip boundaries align with surgical workflow transitions detected through changes in active instruments. For each clip, we manually create implicit text queries spanning three reasoning categories: (1) semantic queries requiring understanding of instrument roles without explicit names (e.g., “the tool used for tissue retraction”), (2) spatial queries demanding relative positioning and anatomical context (e.g., “the instrument closest to the liver”), and (3) temporal queries involving procedural phases and workflow understanding (e.g., “the instrument active during the dissection phase”). Each sample consists of a video clip, an implicit query, and the corresponding ground-truth mask. The training set contains 7 video sequences from EndoVis17 (215 clips, 873 samples) and 11 video sequences from EndoVis18 (164 clips, 587 samples), totaling 18 video sequences. The test set includes 3 video sequences from EndoVis17 (105 clips, 365 samples) and 4 video sequences from EndoVis18 (70 clips, 596 samples). At 2,421 query-mask pairs, the benchmark is about twice the size of ReasonSeg [11], the reference benchmark for this task, which contains 1,218 image-query pairs. Training and test videos are disjoint at the sequence level. Since the 47 query templates are shared across splits, we also build a template-disjoint split with 10 templates held out from training, reported in Table 5, to test for template-level memorization.

SettingMasksSemantic ↑Spatial ↑Temporal ↑
ReSurgSAM2 [15]own.593.480.357
ReSurgSAM2 [15]DT.607.492.371
LISA [11]DT.572.224.258
ReSurgSAM2, fine-tunedown.621.515.388
Base LLM, zero-shotDT.618.442.379
SurgReasonSeg (Ours)DT.771.717.654
Oracle index (upper bound)DT.852.841.829
Candidate recall (%)DT92.491.890.5
Held-out templatesDT.746.692.628
Cholec80 [26] transferDT.721.665.583
Per-video σ\sigma (lower is better)DT.028.035.041

Table 5. Matched masks and fine-tuning explain at most 16% of the margin. Masks: the method’s own masks or the DT masks of SurgReasonSeg; values are J&F\mathcal{J}\&\mathcal{F} unless noted. Bold: best; underline: second best; the last block is not ranked.

Figure 2(a) visualizes the dataset composition across the three reasoning categories.

Figure 2. Dataset composition and adaptive window sizes. The left panel shows the number of EndoVis-ReasonSeg samples per reasoning category in the training and test splits, and the right panel shows the distribution of adaptive sliding window sizes across query types, with diamond markers indicating means. Temporal queries during phase transitions receive the largest windows (mean 6.1 frames), while semantic queries remain close to wbase=2w_{\mathrm{base}} = 2.

Experiments

Implementation Details. We implement SurgReasonSeg with Qwen3-4B [32] as the base LLM, training on 8 NVIDIA RTX 4090 GPUs using LoRA [6] with rank 64 and scaling factor 128. Both LoRA adapters and the router are trained with GRPO [19] in TRL at a learning rate of 4×10−54 \times10^{-5}, and the router reward weights latency with λ=0.3\lambda= 0.3. The two adapters are trained sequentially, LcomplexL_{\mathrm{complex}} then LsimpleL_{\mathrm{simple}} for 3 epochs each, followed by the router for 2 epochs with frozen LoRA weights, and reward variance drops below 0.05 after the first epoch.

For Eq. (1) we set α=0.01\alpha= 0.01, βLen=0.3\beta_{\mathrm{Len}} = 0.3, τshort=50\tau_{\mathrm{short}} = 50, τlong=200\tau_{\mathrm{long}} = 200 tokens; the window uses wbase=2w_{\mathrm{base}} = 2, wquery=2w_{\mathrm{query}} = 2, wphase=3w_{\mathrm{phase}} = 3, θchange=1\theta_{\mathrm{change}} = 1, and the memory β=0.6\beta= 0.6, η=0.5\eta= 0.5, k=3k = 3. The bank holds 500 entries, evicting the lowest-scoring one when full. Retrieval uses FAISS and takes 1.2ms for 500 entries and 2.1ms for 1,000. Two annotators independently created queries with a third resolving disagreements, reaching a Cohen’s κ\kappa of 0.87 across 47 templates.

We evaluate reasoning segmentation on EndoVis-ReasonSeg and referring segmentation on Ref-EndoVis17 and Ref-EndoVis18 [15]. Performance is measured using region similarity (J\mathcal{J}) and contour accuracy (F\mathcal{F}), with FPS evaluated on a single NVIDIA RTX 4090 GPU. All experiments are repeated over three random seeds and we report mean±standard deviation.

Reasoning Segmentation Performance. Table 2 compares the performance on EndoVis-ReasonSeg across three reasoning categories. General-domain methods trained without surgical data perform poorly, with LISA [11] reaching 0.562 J&F\mathcal{J\&F} on semantic and 0.245 on temporal queries. This reflects the domain gap, where general multimodal LLMs lack knowledge of instrument functions. Video-based VISA [31] improves over image-based LISA [11] in spatial (0.469 vs. 0.207) and temporal (0.329 vs. 0.245) rea-

MethodTypeSemantic J↑\mathcal{J} \uparrowSemantic F↑\mathcal{F} \uparrowSemantic J&F↑\mathcal{J\&F} \uparrowSpatial J↑\mathcal{J} \uparrowSpatial F↑\mathcal{F} \uparrowSpatial J&F↑\mathcal{J\&F} \uparrowTemporal J↑\mathcal{J} \uparrowTemporal F↑\mathcal{F} \uparrowTemporal J&F↑\mathcal{J\&F} \uparrowFPS↑\uparrow
LISA [11]Reas..556±.031.568±.024.562±.025.201±.042.213±.035.207±.036.238±.039.251±.033.245±.0341.33
VISA [31]Reas..528±.023.547±.021.538±.019.457±.032.481±.028.469±.027.319±.037.338±.031.329±.0321.87
LLM-Seg [28]Reas..387±.034.394±.031.391±.029.245±.043.258±.038.252±.037.176±.048.189±.041.183±.0420.29
SegZero [16]Reas..509±.025.526±.022.518±.021.398±.033.421±.029.410±.028.312±.038.331±.032.322±.0330.11
CoReS [2]Reas..560±.022.580±.019.570±.018.439±.028.463±.024.451±.023.351±.033.372±.029.362±.0280.27
JiT [24]Reas..548±.024.565±.021.557±.020.452±.029.476±.025.464±.024.364±.032.385±.028.375±.0270.03
RSVIS [27]Ref..542±.023.559±.020.551±.019.391±.031.412±.027.402±.026.296±.035.315±.030.306±.02922.1
ReSurgSAM2 [15]Ref..584±.019.601±.017.593±.016.468±.024.492±.021.480±.020.347±.028.367±.025.357±.02461.2
SurgReasonSeg (Ours)Reas..762±.013.779±.011.771±.010.704±.018.729±.014.717±.013.641±.021.667±.018.654±.01718.8

Table 2. Table 2. SurgReasonSeg achieves the best J\mathcal{J}, F\mathcal{F}, and J&F\mathcal{J\&F} on all three reasoning categories of EndoVis-ReasonSeg. Reas./Ref.: reasoning/referring methods; ±: standard deviation over three random seeds; the FPS of SurgReasonSeg includes DT construction and is averaged over routing. Bold: best; underline: second best.

reasoning, yet still falls short without surgical priors. LLM-Seg [28] achieves the lowest overall performance (0.183 temporal J&FJ\&F) with high variance (±0.042). Among surgical methods, ReSurgSAM2 [15] reaches 0.593, 0.480, and 0.357 J&FJ\&F. Compared to ReSurgSAM2 [15], SurgReasonSeg improves J&FJ\&F by 0.178, 0.237, and 0.297 on semantic, spatial, and temporal reasoning. All baselines are evaluated from their released weights without fine-tuning on EndoVis-ReasonSeg and without access to the DT, so this comparison measures the end-to-end system rather than the reasoning modules in isolation, which the attribution analysis below addresses.

Efficiency Analysis

Most reasoning segmentation methods operate below 2 FPS, with JiT [24] at 0.03 FPS from agent-based overhead, while ReSurgSAM2 [15] reaches 61.2 FPS but cannot handle implicit queries at all. Specifically, on a single RTX 4090, DT construction with SurgSAM2, DepthAnything2, and RASO takes 10.6ms, the router 0.8ms, the memory stage 3.2ms, and LoRA inference 34.2ms for LsimpleL_{\mathrm{simple}} and 48.5ms for LcomplexL_{\mathrm{complex}}. The memory stage consists of the FAISS lookup, 1.2ms at 500 entries in Tab. S1 of the supplementary, and the 2.0ms serialization of the k=3k = 3 retrieved DT segments into the prompt. The total latency is 48.8ms (20.5 FPS) when LsimpleL_{\mathrm{simple}} is selected, which accounts for 69% of queries, and 63.2ms (15.8 FPS) when LcomplexL_{\mathrm{complex}} is activated. Averaged

over the empirical routing distribution, the expected latency is 0.69 × 48.8 + 0.31 × 63.2 = 53.3ms, corresponding to 18.8 FPS. These budgets include DT construction for the incoming frame only, since DT entries for frames already inside the window are cached and reused. Consequently, the 10.6ms DT construction cost is paid once per frame and does not grow with the window size. Figure 3(a,Figure 3b) visualizes the trade-off and the per-component latency.

Figure 3. Accuracy, latency, and component analysis. The left panel plots the average J&F\mathcal{J\&F} over the three reasoning categories against FPS for all methods, where SurgReasonSeg (star) gives the best balance between reasoning accuracy and speed. The middle panel breaks down the latency of the LsimpleL_{\mathrm{simple}} and LcomplexL_{\mathrm{complex}} inference paths per component on a single RTX 4090 GPU. The right panel shows the drop in J&F\mathcal{J\&F} when the router, ASW, or SCM is removed, and each of the three components has its largest effect on temporal reasoning.

Qualitative Results and Failure Cases

In Figure 4, SurgReasonSeg keeps its mask predictions consistent across frames, whereas the other methods produce fragmented masks. For temporal queries, the adaptive sliding window stabilizes predictions across frames, while methods without temporal modeling produce masks that fluctuate from frame to frame. For spatial queries, the centroid and depth fields of the DT resolve ambiguities that appearance-only baselines miss in cluttered scenes. We observe two failure modes of SurgReasonSeg. The first is a missing candidate, where SurgSAM2 does not instantiate the referent so that no index in the DT can be correct, upper-bounding achievable accuracy at the candidate recall reported in Tab. 5. The second is an attribute error, where a RASO label redirects the reasoning chain to a similar instrument or a DepthAnything2 error inverts a near-far relation; the memory occasionally corrects such errors by retrieving a contrasting unsuccessful case. On the test set, missing candidates account for 29.7% of the overall failures (consistent with the 8.4% average candidate miss rate), and attribute errors account for 42.3%, with the remainder stemming from boundary ambiguity.

Qualitative comparison of reasoning segmentation on EndoVis-ReasonSeg

Figure 4. Qualitative comparison of reasoning segmentation on EndoVis-ReasonSeg. Each row shows the predictions of one method on five consecutive frames, with the frame index at the top left and the J\mathcal{J} score at the top right of each frame. Ground truth is shown in green and predictions in purple, and SurgReasonSeg obtains the highest J\mathcal{J} on every frame of the three examples.

Referring Segmentation Performance. Table 3 compares performance on referring surgical video segmentation, where queries name the target directly. General-domain methods transfer poorly, with LISA [11] at 0.388, 0.293, and 0.237 J&F\mathcal{J} \& \mathcal{F}, while RSVIS [27] reaches 0.612, 0.684, and 0.657 through video-level consistency and ReSurgSAM2 [15] ranks second through its surgical encoder. SurgReasonSeg consistently achieves the best J&F\mathcal{J} \& \mathcal{F} scores of 0.801, 0.829, and 0.778 across Ref-EndoVis17 (tool), Ref-EndoVis18 (tool), and Ref-EndoVis18 (tissue) datasets. Compared to ReSurgSAM2 [15], SurgReasonSeg improves performance by 0.024, 0.023, and 0.027 J&F\mathcal{J} \& \mathcal{F} on the three datasets, suggesting that referring segmentation also benefits from accurate DT construction.

Table 3. SurgReasonSeg achieves the best J&FJ\&F on all three referring segmentation settings. ±\pm: standard deviation over three random seeds; bold: best; underline: second best.

MethodRef-EndoVis17 [15] (tool)Ref-EndoVis18 [15] (tool)Ref-EndoVis18 [15] (tissue)
J&F↑J\&F \uparrowJ↑J \uparrowF↑F \uparrowJ&F↑J\&F \uparrowJ↑J \uparrowF↑F \uparrowJ&F↑J\&F \uparrowJ↑J \uparrowF↑F \uparrow
LISA [11].388±.032.420±.029.355±.034.293±.041.284±.043.303±.037.237±.044.244±.042.229±.045
VISA [31].507±.022.479±.026.535±.019.431±.028.398±.031.464±.024.384±.031.354±.033.414±.027
LLM-Seg [28].307±.039.177±.047.437±.031.177±.048.105±.052.249±.041.153±.049.103±.053.203±.043
SegZero [16].468±.027.388±.032.548±.022.388±.031.349±.034.427±.028.349±.033.321±.035.377±.030
CoReS [2].495±.023.415±.028.575±.019.415±.027.377±.030.453±.024.377±.029.351±.031.403±.026
JiT [24].509±.021.441±.026.577±.018.429±.027.403±.029.455±.023.391±.028.361±.031.421±.025
RSVIS [27].612±.018.610±.019.615±.017.684±.016.683±.017.685±.016.657±.019.729±.015.585±.022
ReSurgSAM2 [15].777±.013.777±.014.777±.013.806±.011.809±.012.803±.011.751±.016.809±.012.693±.019
SurgReasonSeg (Ours).801±.011.798±.012.805±.010.829±.009.832±.010.826±.009.778±.013.835±.010.721±.014

Table 3.

The margins on referring datasets (0.024–0.027 J&F\mathcal{J} \& \mathcal{F}) are about an order of magnitude smaller than on reasoning queries (0.178–0.297). When the target is named, ReSurgSAM2 operates close to the segmenter ceiling, so the residual gain reflects DT construction quality rather than reasoning.

Ablation Study. Table 4 reports the ablation of each component. The baseline without any proposed component, namely the same Qwen3-4B backbone consuming raw

Table 4. Removing any component lowers J&FJ\&F, most on temporal queries. Router: routing between two LoRA adapters; ASW: adaptive sliding window; SCM: surgical context memory; ±\pm: standard deviation over three random seeds. Bold: best; underline: second best; FPS is a cost and is not ranked.

DTRouterASWSCMSemantic J↑J \uparrowSemantic F↑F \uparrowSemantic J&F↑J\&F \uparrowSpatial J↑J \uparrowSpatial F↑F \uparrowSpatial J&F↑J\&F \uparrowTemporal J↑J \uparrowTemporal F↑F \uparrowTemporal J&F↑J\&F \uparrowFPS
✗✗✗✗.587±.031.601±.028.594±.027.445±.038.468±.033.457±.033.382±.041.405±.037.394±.0361.5
✓✗✓✓.718±.021.734±.018.726±.017.651±.024.675±.021.663±.020.578±.027.602±.023.590±.02212.5
✓✓✗✓.738±.018.754±.015.746±.014.678±.022.702±.019.690±.018.589±.025.613±.022.601±.02117.8
✓✓✓✗.745±.015.761±.013.753±.012.685±.020.710±.017.698±.016.598±.023.623±.020.611±.01920.0
✓✓✓✓.762±.013.779±.011.771±.010.704±.018.729±.014.717±.013.641±.021.667±.018.654±.01718.8

Table 4.

frames as visual tokens and decoding a mask without the DT, reaches 0.594, 0.457, and 0.394 J&F\mathcal{J} \& \mathcal{F} at 1.5 FPS, where the low rate reflects pixel-level decoding inside the LLM. Introducing the DT with ASW and SCM improves J&F\mathcal{J} \& \mathcal{F} to 0.726, 0.663, and 0.590 at 12.5 FPS. Removing only the adaptive sliding window from the full model gives 0.746, 0.690, and 0.601 at 17.8 FPS, and removing only the surgical context memory gives 0.753, 0.698, and 0.611 at 20.0 FPS. The adaptive sliding window contributes 0.053 J&F\mathcal{J} \& \mathcal{F} on temporal queries against 0.027 on spatial queries, consistent with spatial queries resting on single-frame relations. The surgical context memory likewise contributes most on temporal queries, 0.043 J&F\mathcal{J} \& \mathcal{F}, by retrieving hishistorical procedural patterns. Figure 3(c) visualizes the contribution of each component across reasoning categories.

We further compare the router with single-adapter variants. Using only LcomplexL_{\mathrm{complex}} yields J&F\mathcal{J}\&\mathcal{F} of 0.758, 0.706, and 0.643 at 16.0 FPS, while using only LsimpleL_{\mathrm{simple}} achieves 0.731, 0.679, and 0.612 at 20.8 FPS, since neither variant runs the router. The full model with routing achieves 0.771, 0.717, and 0.654 at 18.8 FPS. The router selects LsimpleL_{\mathrm{simple}} for 69% of queries (mostly semantic) and LcomplexL_{\mathrm{complex}} for 31% (mostly spatial and temporal). Its accuracy difference from the always-LcomplexL_{\mathrm{complex}} variant is 0.013, 0.011, and 0.011 J&F\mathcal{J}\&\mathcal{F}, which falls inside the seed-to-seed deviations of ±0.010\pm0.010–0.017 and should therefore be read as parity rather than as an accuracy gain. The router instead brings a latency benefit at matched accuracy, namely 18.8 against 16.0 FPS, a 17% throughput increase over always invoking LcomplexL_{\mathrm{complex}}. On the semantic queries that the router sends to LsimpleL_{\mathrm{simple}}, LcomplexL_{\mathrm{complex}} scores 0.747±0.0110.747 \pm0.011 and LsimpleL_{\mathrm{simple}} scores 0.750±0.0100.750 \pm0.010, so extended reasoning on these queries adds latency without changing accuracy. The “no Router” row of Table 4 trains a single unspecialized adapter without RlengthR_{\mathrm{length}}, which is why it reaches only 0.726 on semantic queries and falls below both specialized adapters. Figure 5 visualizes the routing analysis across reasoning categories.

Figure 5. Routing analysis. The left panel compares J&F\mathcal{J} \& \mathcal{F} across reasoning categories for the single-adapter variants and full routing, with the FPS of each variant in the legend. The right panel shows the percentage of queries routed to LsimpleL_{\mathrm{simple}} and LcomplexL_{\mathrm{complex}} in each category, with the dashed line indicating the overall average.

Attribution and Protocol Analysis. Because segmentation is reframed as index retrieval over SurgSAM2 and RASO outputs, Table 2 conflates the surgical segmenter with the reasoning modules. Table 5 separates the two by supplying the identical DT masks to ReSurgSAM2 and LISA so that any residual difference is attributable to index selection rather than mask quality. The fine-tuned row removes the training-data asymmetry, and the oracle row bounds achievable accuracy by always selecting the correct index, providing a direct measure of reasoning quality. Handing ReSurgSAM2 the identical DT masks lifts it by only 0.014, 0.012, and 0.014 J&F\mathcal{J}\&\mathcal{F}, so the masks explain less than 10% of the margin.

of the margin. Even fine-tuning it on our training split, the strongest control available, still trails SurgReasonSeg by 0.150, 0.202, and 0.266. A zero-shot prompted base LLM over the same DT reaches only 0.618, 0.442, and 0.379, showing that neither the masks nor prompting alone suffices. Measured against the strongest control, masks and surgical training explain 16%, 15%, and 10% of the total margins of 0.178, 0.237, and 0.297, while the reasoning modules explain the remaining 84%, 85%, and 90%. The oracle bounds of 0.852, 0.841, and 0.829 place SurgReasonSeg at 90%, 85%, and 79% of attainable accuracy, and the candidate recall of 90.5–92.4% marks the ceiling the retrieval reframing imposes. On the protocol, the template-disjoint model loses only 0.025–0.026 J&F\mathcal{J}\&\mathcal{F} on the held-out templates and the per-video deviation is 0.028–0.041, both far below the 0.150–0.266 margin over the strongest control, so neither template memorization nor a single sequence carries the effect. Transfer to Cholec80 [26] reaches 0.721, 0.665, and 0.583 without retraining, above every baseline of Table 2 despite the unseen procedure.

Three further design decisions are ablated in Table S3 of the supplementary, replacing GRPO by supervised fine-tuning, the RL router by a category rule and a supervised classifier, and the VLM judge for vjv_j by uniform weighting and by a ground-truth IoU upper bound. The VLM judge agrees with the two human annotators at a Cohen’s κ\kappa of 0.82.

Limitations. Accuracy is bounded by the candidate recall of SurgSAM2, since reasoning cannot recover a referent that the segmenter never instantiates. The window spans only a few frames, so questions that depend on events minutes earlier can be answered only through the memory, and we do not evaluate such procedure-level questions. On temporal queries whose evidence lies entirely outside the adaptive window, SurgReasonSeg reaches 0.523 J&F\mathcal{J}\&\mathcal{F} through memory retrieval alone. The memory is validated on EndoVis videos only, so whether it supports continual adaptation across institutions remains to be tested.

Conclusion

We propose SurgReasonSeg, which brings DT-based reasoning segmentation to surgical video, constructing the DT from surgical foundation models to bridge the gap between general and surgical domains. Complexity-aware routing between two LoRA adapters adapts computation to query demands, while the adaptive window and surgical context memory provide workflow-aware temporal modeling. We also contribute the EndoVis-ReasonSeg benchmark. SurgReasonSeg achieves the best performance across all reasoning and referring benchmarks at 18.8 FPS averaged over queries, and Table 5 decomposes the gain into segmenter and reasoning contributions. Future work targets open surgery and faster LoRA inference, the main per-frame cost.

Complexity-Aware Surgical Video Reasoning Segmentation with Digital Twin and Context Memory

Supplementary Material

This supplementary material reports the adaptive window sensitivity, the memory bank and backbone scaling, design choice ablations, per-video and per-template results, and the Cholec80 transfer subset. Figure S1 reproduces the reasoning and judge prompts verbatim, and the judge sees only the overlaid mask and the query, without the ground truth or the reasoning chain.

The reasoning prompt and the memory quality judge prompt used by SurgReasonSeg

Figure S1. The reasoning prompt (left) and the memory quality judge prompt (right) used by SurgReasonSeg.

Adaptive Window Sensitivity

We further evaluate the sensitivity of adaptive sliding window hyperparameters by testing wbase∈{1,2,3}w_{\mathrm{base}} \in\{1, 2, 3\}, wquery∈{1,2,3}w_{\mathrm{query}} \in\{1, 2, 3\}, and wphase∈{2,3,4}w_{\mathrm{phase}} \in\{2, 3, 4\}. Across these settings, J&F\mathcal{J}\&\mathcal{F} varies by at most 0.015. Figure 2 of the main paper shows the resulting window sizes, where semantic queries stay near wbasew_{\mathrm{base}} and temporal queries during phase transitions receive larger windows.

Memory Bank and Backbone Scalability

Memory Bank Scalability. Table S1 evaluates the effect of surgical context memory bank size on performance. Without memory (∣M∣=0|\mathcal{M}| = 0), temporal reasoning drops by 0.043 J&F\mathcal{J}\&\mathcal{F} compared to the full model, showing that historical procedural context matters most for temporal queries. The impact on semantic reasoning is smaller (−0.018 J&F\mathcal{J}\&\mathcal{F}), as

∣M∣|\mathcal{M}|Semantic J&F↑\mathcal{J}\&\mathcal{F}\uparrowSpatial J&F↑\mathcal{J}\&\mathcal{F}\uparrowTemporal J&F↑\mathcal{J}\&\mathcal{F}\uparrowLatency (ms)
0.753±.012.698±.015.611±.0190.0
100.762±.011.709±.014.637±.0180.5
250.768±.011.714±.014.649±.0180.8
500.771±.010.717±.013.654±.0171.2
1000_ .770±.011__ .716±.014__ .653±.017_2.1

Table S1. J&F\mathcal{J}\&\mathcal{F} of SurgReasonSeg (Ours) peaks at 500 memory entries. ±: standard deviation over three seeds; latency is a cost and is not ranked.

semantic queries are often resolved from single-frame DT attributes.

Performance improves consistently up to 500 entries and slightly decreases at 1,000. Beyond 500 entries, retrieval noise from marginally similar cases outweighs the added coverage, suggesting that the quality-aware scoring with vjv_j becomes less discriminative. At 500 entries, the FAISS lookup takes 1.2ms, under 2.5% of the total inference time. At 1,000 entries, the retrieval latency of 2.1ms remains compatible with real-time operation. All memory results come from EndoVis videos, so multi-institution deployment remains to be tested.

LLM Backbone. Table S2 compares different Qwen3 model sizes as the base LLM backbone. Scaling from 0.6B to 4B parameters yields consistent improvements across all reasoning categories, with the largest gains in temporal reasoning (+0.056 J&F\mathcal{J}\&\mathcal{F}), as larger models better capture complex procedural dependencies.

LLM backboneSemantic J&F↑\mathcal{J}\&\mathcal{F}\uparrowSpatial J&F↑\mathcal{J}\&\mathcal{F}\uparrowTemporal J&F↑\mathcal{J}\&\mathcal{F}\uparrowFPS
Qwen3-0.6B.721±.016.668±.019.598±.02424.7
Qwen3-1.7B.748±.013.695±.016.631±.02019.5
Qwen3-4B (Ours)_ .771±.010__ .717±.013__ .654±.017_18.8
Qwen3-8B.774±.010.720±.013.659±.0178.6

Table S2. Qwen3-8B adds at most 0.005 J&F\mathcal{J}\&\mathcal{F} over Qwen3-4B at less than half the FPS. ±: standard deviation over three seeds; FPS is end-to-end, a cost, and not ranked.

However, scaling to 8B provides only marginal improvements (+0.003–0.005 J&F\mathcal{J}\&\mathcal{F}) while lowering the FPS from 18.8 to 8.6, below the real-time threshold. This saturation suggests that, once visual perception is offloaded to the foundation models that build the DT, a 4B model is large enough for the reasoning step. We therefore use Qwen3-4B, which balances reasoning accuracy against real-time inference.

Design Choice Ablations

Table S3 ablates three design decisions independently. Replacing GRPO with supervised fine-tuning on the same rollouts tests whether RL is required or whether the reward merely selects an output length that supervision could imitate. Replacing the RL router with a category rule and a supervised classifier tests whether the router must be learned against a latency-aware reward. Varying vjv_j between uniform weighting, a ground-truth IoU bound, and the external judge makes the cost of an imperfect judge measurable. The VLM judge agrees with the human annotators at a Cohen’s κ\kappa of 0.82, which explains why the performance with the VLM judge approaches that of the ground-truth IoU upper bound.

Per-Video and Per-Template Breakdown

Table 5 of the main paper reports the aggregate per-video deviation, and Table S4 resolves it into the individual test sequences. No single video carries the reported margins. The weakest sequence still reaches 0.728, 0.663, and 0.591 J&FJ\&F, exceeding the strongest control of Table 5, ReSurgSAM2 fine-tuned on our training split, by 0.107, 0.148, and 0.203, so the ordering holds sequence by sequence rather than only in aggregate. The 0.075–0.110 spread tracks the source dataset, with the three EndoVis17 sequences below the four EndoVis18 ones, consistent with the older subset’s lower annotation density. Table S5 reports the template inventory and the ten templates held out for the template-disjoint split. The ten held-out templates contribute 205 of the 961 test samples, on which the template-disjoint model scores 0.746, 0.692, and 0.628, only 0.025 to 0.026 below the full model on the standard split. Unseen phrasings therefore yield comparable accuracy to seen ones, indicating that the model resolves queries through the DT rather than through template memorization.

SequenceSourceSemanticSpatialTemporal
seq-1EndoVis17.728.663.591
seq-2EndoVis17.743.682.613
seq-3EndoVis17.763.707.642
seq-4EndoVis18.776.723.661
seq-5EndoVis18.789.740.680
seq-6EndoVis18.795.747.689
seq-7EndoVis18.803.757.701
MeanAll.771.717.654
Std. dev.All.028.035.041

Table S4. Every test video stays above the strongest control of the main paper. Per-sequence J&FJ\&F of SurgReasonSeg.

SemanticSpatialTemporalAll
Templates20151247
Held-out templates43310
Test samples, held out856852205
J&FJ\&F, held out.746.692.628.698

Table S5. Ten of the 47 templates are held out and cover 205 test samples. The last row is the template-disjoint model; All is weighted by sample count.

Cholec80 Transfer Subset

The transfer row of Table 5 uses a reasoning segmentation subset built from Cholec80, a laparoscopic cholecystectomy corpus disjoint from EndoVis in procedure and instruments. We sample 10 sequences and segment them into clips of

20 to 40 frames at instrument-change boundaries, following the same protocol as EndoVis-ReasonSeg, and author 420 implicit queries from the held-out template pool (Cohen’s κ=0.85\kappa= 0.85). The model is evaluated without retraining and with the memory bank populated only from EndoVis training videos, so the setting measures cross-procedure transfer rather than adaptation. Accuracy drops by 0.050, 0.052, and 0.071 J&FJ\&F relative to the in-domain result, consistent with the instrument vocabulary shift, yet the transferred model still exceeds every baseline of Table 2.

References

  1. [1]Aleks Attanasio, Bruno Scaglioni, Elena De Momi, et al. Autonomy in surgical robotics. Annu. Rev. Control Robot. Auton. Syst., 4:651–679, 2021. 1DOI
  2. [2]Xiaoyi Bao, Siyang Sun, Shuailei Ma, et al. CoReS: Orchestrating the dance of reasoning and segmentation. In ECCV, pages 187–204, 2024. 2, 5, 6arxiv.org/abs/2404.05673
  3. [3]Xiaoyi Bao, Xu Lian, and Mathias Unberath. MoSFormer: Augmenting temporal context with memory of surgery for surgical phase recognition. arXiv:2503.00695, 2025. 1, 2
  4. [4]Hao Ding, Yuqian Zhang, Wenzheng Cheng, et al. Towards robust algorithms for surgical phase recognition via digital twin representation. In MICCAI Workshops, pages 119–129, 2025. 2arxiv.org/abs/2410.20026
  5. [5]Hao Ding, Lalithkumar Seenivasan, Hongchao Shu, et al. Towards robust surgical automation via digital twin representations from foundation models. In MICCAI Workshops, pages 21–31, 2026. 2arxiv.org/abs/2409.13107
  6. [6]Edward P. Hu, Yelong Shen, Phillip Wallis, et al. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. 2, 3, 5
  7. [7]Zhenyu Jiang, Cheng-Chun Hsu, and Yuke Zhu. Ditto: Building digital twins of articulated objects from interaction. In CVPR, pages 5606–5616, 2022. 2arxiv.org/abs/2202.08227
  8. [8]Yueming Jin, Yonghao Long, Cheng Chen, et al. Temporal memory relation network for workflow recognition from surgical video. IEEE Trans. Med. Imaging, 40:1911–1923, 2021. 1, 2arxiv.org/abs/2103.16327
  9. [9]David Jones, Chris Snider, Aydin Nassehi, et al. Characterising the digital twin: A systematic literature review. CIRP J. Manuf. Sci. Technol., 29:36–52, 2020. 2DOI
  10. [10]Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, et al. ReferItGame: Referring to objects in photographs of natural scenes. In EMNLP, pages 787–798, 2014. 1DOI
  11. [11]Xin Lai, Zhuotao Tian, Yukang Chen, et al. LISA: Reasoning segmentation via large language model. In CVPR, pages 9579–9589, 2024. 1, 2, 5, 6, 7, 8arxiv.org/abs/2308.00692
  12. [12]Jiajie Li, Brian R. Quaranto, Chenhui Xu, et al. Recognize any surgical object: Unleashing the power of weakly-supervised data. In ICLR, 2025. 1, 2, 3arxiv.org/abs/2501.15326
  13. [13]Yanwei Li, Chengyao Wang, and Jaya Jia. LLaMA-VID: An image is worth 2 tokens in large language models. In ECCV, pages 323–340, 2024. 2arxiv.org/abs/2311.17043
  14. [14]Haofeng Liu, Erli Zhang, Junde Wu, et al. Surgical SAM 2: Real-time segment anything in surgical video by efficient frame pruning. arXiv:2408.07931, 2024. 1, 2arxiv.org/abs/2408.07931
  15. [15]Haofeng Liu, Mingqi Gao, Xuxiao Luo, et al. ReSurgSAM2: Referring segment anything in surgical video via credible long-term tracking. In MICCAI, pages 435–445, 2025. 1, 2, 4, 5, 6, 7, 8arxiv.org/abs/2505.08581
  16. [16]Yuqì Liu, Bohao Peng, Zhisheng Zhong, et al. Seg-Zero: Reasoning-chain guided segmentation via cognitive reinforcement. arXiv:2503.06520, 2025. 2, 5, 6arxiv.org/abs/2503.06520
  17. [17]Chang Han Low, Ziyue Wang, Tianyi Zhang, et al. Sur-GRAW: Multi-agent workflow with chain of thought reasoning for robotic surgical video analysis. IEEE Robot. Autom. Lett., 11:4857–4864, 2026. 1
  18. [18]Muhammad Maaz, Hanoona Rasheed, Salman Khan, et al. Video-ChatGPT: Towards detailed video understanding via large vision and language models. In ACL, pages 12585–12602, 2024. 2arxiv.org/abs/2306.05424
  19. [19]Zhihong Shao, Peiyi Wang, Qihao Zhu, et al. DeepSeek-Math: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300, 2024. 2, 5arxiv.org/abs/2402.03300
  20. [20]Yiqing Shen and Mathias Unberath. Constructing and interpreting digital twin representations for visual reasoning via large language models and reinforcement learning. In ECCV, pages 178–194, 2026. 1, 2, 3DOI
  21. [21]Yiqing Shen and Mathias Unberath. Fast reasoning segmentation for images and videos. In CVPR, 2026. 1, 2, 3arxiv.org/abs/2511.12368
  22. [22]Yiqing Shen, Hao Ding, Lalithkumar Seenivasan, et al. Position: Foundation models need digital twin representations. arXiv:2505.03798, 2025. 1, 2arxiv.org/abs/2505.03798
  23. [23]Yiqing Shen, Chenjia Li, Bohan Liu, et al. Operating room workflow analysis via reasoning segmentation over digital twins. In MICCAI, pages 415–424, 2025. 1, 2, 3arxiv.org/abs/2503.21054
  24. [24]Yiqing Shen, Bohan Liu, Chenjia Li, et al. Online reasoning video segmentation with just-in-time digital twins. In ICCV, pages 24698–24706, 2025. 1, 2, 3, 5, 6arxiv.org/abs/2503.21056
  25. [25]Nicholas Stroh. TrackGPT–a generative pre-trained transformer for cross-domain entity trajectory forecasting. arXiv:2402.00066, 2024. 2arxiv.org/abs/2402.00066
  26. [26]Andru P Twinanda, Sherif Shehata, Didier Mutter, et al. EndoNet: A deep architecture for recognition tasks on laparoscopic videos. IEEE Trans. Med. Imaging, 36:86–97, 2017. 8arxiv.org/abs/1602.03012
  27. [27]Hongqiu Wang, Guang Yang, Shichen Zhang, et al. Video-instrument synergistic network for referring video instrument segmentation in robotic surgery. IEEE Trans. Med. Imaging, 43:4457–4469, 2024. 1, 2, 4, 5, 6, 7DOI
  28. [28]Junchi Wang and Lei Ke. LLM-Seg: Bridging image segmentation and large language model reasoning. In CVPR Workshops, pages 1765–1774, 2024. 5, 6arxiv.org/abs/2404.08767
  29. [29]Zhuofan Xia, Dongchen Han, Yizeng Han, et al. GSVA: Generalized segmentation via multimodal large language models. In CVPR, pages 3858–3869, 2024. 2arxiv.org/abs/2312.10103
  30. [30]Zhucun Xue, Jianqiang Zhang, Xurong Xie, et al. AdaVideo-RAG: Omni-contextual adaptive retrieval-augmented efficient long video understanding. NeurIPS, 38:6094–6117, 2025. 1, 2
  31. [31]Cilin Yan, Haochen Wang, Shilin Yan, et al. VISA: Reasoning video object segmentation via large language models. In ECCV, pages 98–115, 2024. 1, 2, 5, 6arxiv.org/abs/2407.11325
  32. [32]An Yang, Anfeng Li, Baosong Yang, et al. Qwen3 technical report. arXiv:2505.09388, 2025. 3, 5arxiv.org/abs/2505.09388
  33. [33]Lihe Yang, Bingyi Kang, Zilong Huang, et al. Depth anything V2. NeurIPS, 37:21875–21911, 2024. 3arxiv.org/abs/2406.09414
  34. [34]Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An instruction-tuned audio-visual language model for video understanding. In EMNLP System Demonstrations, pages 543–553, 2023. 2arxiv.org/abs/2306.02858
  35. [35]Zijian Zhou, Oluwatosin Alabi, Meng Wei, et al. Text promptable surgical instrument segmentation with vision-language models. NeurIPS, 36:28611–28623, 2023. 1arxiv.org/abs/2306.09244

Paper details

vlnc:W_f6m8nhqm
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 14

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents