betaFeedback

WorksComputer Science

How to "train" a surgeon? Benchmarking generalist agents in surgical scene understanding

Abstract

Evaluating general-purpose multimodal large language models (MLLMs) for surgical scene understanding remains limited due to scarce surgical data and fragmented evaluation protocols. We address these challenges through a two-stage evaluation framework for surgical scene understanding. In Stage I, we introduce SurgKnowBench, a diagnostic benchmark that harmonizes public surgical datasets across procedures and tasks and evaluates proprietary and open-source MLLMs to map the boundary of pretrained surgical competence. SurgKnowBench reveals that current models retain partial visual priors but remain weak on relational and procedural understanding and workflow assessment. Building on this failure frontier and motivated by the scaffolded nature of surgical training, we introduce SurgAgentGym, an agentic environment where general-purpose MLLM agents can use bounded knowledge search over SurgWiki and general-purpose visual grounding for surgical tasks. SurgAgentGym evaluates whether generalist agents can improve surgical understanding at inference time through tool-mediated evidence use, without additional surgical training. Our results show that agentic inference narrows the gap mainly for perception-heavy tasks, while fine-grained localization and expert-style assessment remain persistent bottlenecks. This opens a path toward future studies of agentic surgical understanding.

Introduction

Surgical superintelligence requires models to possess visual, procedural, and expertise-intensive scene understanding capabilities. A model must recognize procedure-specific anatomy, instruments, tissue states, and actions, situate them within an operation, and reason over the procedure-level workflow. However, progress toward surgical superintelligence is always constrained by two obstacles. First, surgical data are scarce, fragmented, and difficult to share: intraoperative videos are restricted by privacy constraints, expert annotation is costly and procedure-specific, and surgical practice varies across procedures, institutions, and surgeons [1, 2]. These make it difficult to rely on the standard recipe of large-scale supervised pretraining and domain fine-tuning. Second, the surgical competence of general-purpose MLLMs remains poorly characterized: it is unclear which surgical tasks are already supported by general-purpose multimodal pretraining, and which failures persist without domain-specific adaptation. These two obstacles motivate a staged question: what do MLLMs understand about surgery with direct inference, and, when large-scale surgical fine-tuning is impractical, can MLLMs improve surgical understanding at inference time without additional training?

We begin with Stage I: Direct Inference, inquiring what current general-purpose MLLMs already understand about surgery. Existing surgical benchmarks [3, 4] and supervised models [1, 5] are often narrow in procedure coverage, task type, or model selection, leaving a need for a unified evaluation protocol. To address this, we introduce SURGKNOWBENCH, a comprehensive diagnostic benchmark that aggregates and harmonizes a broad collection of surgical datasets spanning 7 surgical procedures and 7 tasks, yielding 376,602 samples. We evaluate 17 proprietary and open-source MLLMs in a zero-shot setting across multiple surgical procedures and three task families: spatial recognition, temporal reasoning, and workflow assessment, to systematically understand model performance bottlenecks.

The Stage I results (detailed in § 2) reveal a difficult frontier for current MLLMs in all task domains. This raises a second question: when data-intensive surgical training is impractical, how can they achieve surgical understanding capability? Surgical training for human surgeons suggests a path forward. Human trainees do not learn from labeled examples alone; they study anatomy and surgical ontologies, consult procedural references, review demonstrations and prior cases, and connect abstract knowledge to concrete operative scenes. The transferable capability is not mere memorization, but epistemic control: recognizing when knowledge is missing, seeking relevant evidence, grounding that evidence in the observed scene, and using it to support a decision. Inspired by this, we explore an analogous setting for MLLMs, Stage II: Agentic Evaluation: if we harness general-purpose MLLMs as generalist agents and equip them with human-like abilities (to see, to search, and to think), can new surgical scene-understanding capabilities emerge?

As illustrated in Figure 1, we instantiate this evaluation with a new agentic environment, SURGAGENTGYM, in which generalist MLLM agents can explore surgical tasks. To equip agents with abilities equivalent to those of human trainees, SURGAGENTGYM provides two complementary tools. Visual grounding tools help agents operate on the current surgical input, enabling them to localize semantic boundaries and bridge the visual grounding gap across models. Knowledge acquisition tools allow agents to retrieve and inspect evidence from a bounded multimodal surgical knowledge base containing videos, images, articles, tables, PDFs, and procedural references. We curate an abundant set of open-access, modality-rich, broad-coverage, expert-reviewed sources to build the knowledge base, and conduct thorough data cleaning and indexing to enable robust retrieval.

Figure 1. How to "train" a surgeon? In our two-stage evaluation: Stage I uses SURGKNOWBENCH to diagnose zero-shot surgical competence of MLLMs, while Stage II uses SURGAGENTGYM on SURGKNOWBENCH-Hard to test whether general-purpose MLLM agents can improve at inference time through bounded knowledge search and generalist visual grounding.

On a hard set of SURGKNOWBENCH, termed SURGKNOWBENCH-Hard, we evaluate agents with tool access. Final-answer scores measure task success, while trajectory analysis examines whether the agent recognizes an information need, selects appropriate tools, retrieves relevant evidence, grounds visual evidence in the surgical input, and integrates information into a contract-compliant answer. This makes SURGAGENTGYM more than a question-answering benchmark: it exposes agent behavior as an observable and traceable object of evaluation, a requirement for medical-facing AI where the supporting evidence trail must be inspectable.

Our contributions are:

  • SURGKNOWBENCH: a diagnostic benchmark that harmonizes public surgical-understanding datasets to evaluate zero-shot general-purpose MLLMs’ surgical competence.

  • SURGAGENTGYM: a controlled tool-mediated agentic environment for generalist MLLM agents, with bounded knowledge acquisition tools over SURGWIKI and general-purpose visual grounding tools.

  • Two-stage evaluation on surgical scene understanding: we provide a systematic evaluation comprising direct inference and agentic evaluation with trajectory-level analysis for evaluating generalist agents on surgical scene understanding.

SurgKnowBench: diagnosing pretrained surgical competence

We first inquire what surgical competence is already present in pretrained MLLMs.

Task taxonomy and datasets

Surgical scene understanding requires models to interpret an operative scene at multiple levels, from recognizing instruments and their interactions to identifying procedural state and assessing higher-level aspects of surgical workflow and performance [6, 7].

Task taxonomy. As shown in Table 1, SurgKnowBench organizes surgical scene understanding tasks into three task domains: (1) Spatial recognition tests whether a model can identify visible surgical entities and localized visual relations in an operative scene; (2) Temporal reasoning evaluates whether a model can map visual evidence to procedural state; (3) Workflow assessment evaluates whether a model can make expert-style judgments about operative safety and technical skill.

Tab. 1: SURGKNOWBENCH inventory: we organize 11 datasets × 7 task axes across 7 surgical procedures. Tool = tool detection, Trip. = (instrument, verb, target) triplet detection, Loc. = tool localization, CVS = critical-view-of-safety grading, and Skill subsumes the three GOALS-style assessments [2] (holistic, Calot-phase, and dissection-phase).

DatasetSurgeryToolTrip.Loc.PhaseActionCVSSkillFullHard
Cholec80 [5]Lap. cholecystectomy✓42,09850
CholecT50 [1]Lap. cholecystectomy✓✓✓16,722220
Cholec80-CVS [8]Lap. cholecystectomy✓9696
HeiChole [2]Lap. cholecystectomy✓✓36,13786
MultiBypass140 [9]Lap. Roux-en-Y bypass✓171,63350
AutoLaparo [10]Lap. hysterectomy✓✓28,491100
HeiCo [11]Lap. colorectal resection✓4,05750
GraSP [12]Robotic radical prostatectomy✓✓44,022100
PitVis [13]Endoscopic pituitary✓✓2,310100
Cataract-101 [14]Cataract surgery✓12,50350
CATARACTS [15]Cataract surgery✓18,53350
Total376,602952

Table 1.

Datasets. We build on 11 public surgical datasets spanning seven procedures: laparoscopic cholecystectomy (Cholec80 [5], CholecT50 [1], Cholec80-CVS [8], and HeiChole [2]), Roux-en-Y gastric bypass (MultiBypass140 [9]), laparoscopic hysterectomy (AutoLaparo [10]), laparoscopic proctocolectomy (HeiCo [11]), robot-assisted radical prostatectomy (GraSP [12]), endoscopic pituitary surgery (PitVis [13]), and cataract surgery (Cataract-101 [14] and CATARACTS [15]).

Evaluation protocol

Dataset harmonization. For each source dataset, we convert raw annotations into a common QA-style evaluation format while preserving the label sets. Dataset-specific labels are mapped to canonical strings. Frame-level tasks are represented by annotated frames, while interval-level tasks such as CVS and skill assessment retain temporal metadata and representative visual inputs. Further details on frame sampling and answer schemas are provided in Appendix § 10.§ 2 and Appendix § 10.§ 3.

Evaluation metrics. For recognition tasks with a fixed label set (i.e. tool detection, action recognition, and triplet detection), we report macro-averaged F1F_1, which weights each class equally and better exposes rare-label failures. Triplet detection is scored over complete ⟨instrument,verb,target⟩\langle\text{instrument},\text{verb},\text{target}\rangle triplets. For phase recognition, we report strict label accuracy. For tool localization, we report label-constrained F1F_1@0.5 after Hungarian matching, requiring both instrument-label agreement and box overlap. For CVS assessment, we report mean per-key exact-score accuracy together with mean quadratic-weighted Cohen’s κ\kappa over the three CVS criteria. For skill assessment, we report mean absolute error (MAE) together with mean quadratic-weighted κ\kappa over the GOALS-style rubric keys. Auxiliary metrics are defined in Appendix § 10.§ 5 and reported in Appendix § 12.§ 1.

Diagnosing pretrained MLLMs’ surgical competence

Stage I diagnoses the extent and limits of zero-shot surgical competence in general-purpose MLLMs. Figure 2 places selected dataset–task scores alongside published specialist reference results.

Figure 2. Stage I performance on dataset–task breakdown.

Limited surgical competence. Table 2 shows that the evaluated MLLMs have some ability to identify surgical instruments, but recognition and localization remain weak. On tool detection, 16 of the 17 models score above the majority baseline, yet the best score reaches only 26.1% macro-F1, compared with 13.5% for the majority baseline. Accurate localization also remains difficult: the best model achieves only 22.7% (F1F_1@0.5), which requires both the correct instrument label and sufficient overlap with the reference bounding box.

ModelTool Det. macro-F1↑Triplet Det.† macro-F1↑Tool Loc.† [email protected] ↑Phase Rec. Acc.↑Action Rec. macro-F1↑CVS Ass. Acc./κ ↑Skill Ass. MAE↓/κ ↑Rank↓
Proprietary
GPT-4o-mini [16]22.81.60.024.727.1*15.6/0.90.84/0.27.7 [6.9,9.4]
Claude-Haiku-4.5 [17]18.22.91.618.421.817.0/-4.80.99/10.98.9 [8.4,9.9]
Open-source (<10B)
InternVL2.5-8B [18]10.41.911.323.811.771.2/0.00.86/-7.211.7 [10.7,12.6]
Qwen2.5-VL-7B-Instruct [19]15.01.511.414.18.771.2/0.01.14/8.011.6 [11.1,13.4]
Qwen3-VL-8B-Instruct [20]19.32.113.818.913.531.6/5.41.05/3.98.1 [7.1,9.7]
Qwen3-VL-8B-Thinking [20]22.12.816.323.23.958.3/2.70.96/4.76.3 [5.1,8.6]
Open-source (10–30B)
Pixtral-12B-2409 [21]16.72.34.017.723.871.2/0.01.21/-0.510.7 [9.4,11.3]
InternVL3-14B [22]14.72.78.919.514.424.0/-0.40.99/4.09.7 [8.6,11.7]
InternVL2.5-26B [18]14.61.914.621.39.834.4/-0.70.64/0.012.6 [9.9,12.9]
Gemma-3-27B-it [23]21.53.60.021.412.637.5/-3.91.39/3.310.0 [8.0,10.7]
Open-source (>30B)
Qwen2.5-VL-32B-Instruct [19]18.01.26.522.014.324.3/-0.40.91/3.910.9 [9.1,12.0]
Qwen3-VL-32B-Instruct [20]26.1*2.514.623.721.331.6/8.20.81/14.33.6 [2.7,5.4]
Qwen2.5-VL-72B-Instruct [19]18.32.011.423.618.547.9/-2.71.08/7.28.3 [6.4,9.4]
InternVL2.5-78B [18]22.64.922.724.111.825.7/-0.40.92/0.17.0 [4.7,7.9]
Medical-domain
HuatuoGPT-Vision-34B-hf [24]21.12.00.924.821.414.6/0.10.77/1.77.6 [7.0,9.1]
Lingshu-32B [25]17.92.41.624.616.59.4/-0.20.80/0.49.2 [7.4,10.8]
MedGemma-27B-it [26]18.42.60.019.716.219.4/4.00.92/1.69.1 [8.0,10.7]
Statistical Baseline
Random12.51.60.010.213.532.8/1.70.91/1.6–
Majority13.51.20.029.917.671.2/0.00.63/0.0–

Table 2. Stage I diagnostic evaluation on the full SURGKNOWBENCH test set. F1 and accuracy are percentages; κ\kappa is multiplied by 100. Skill MAE is reported on the original ordinal scale (lower is better). Rank is the mean rank across the seven task axes over all 17 evaluated models, using κ\kappa for workflow-assessment axes, shown with 95% CIs from a video-clustered bootstrap (1000 resamples). Bold/underline indicate the best/second-best values among the evaluated models for each metric (including ties). * marks a significant lead over the runner-up (paired video-clustered bootstrap, BH-FDR q≤0.05q \le0.05). † marks task axes without significance tests due to limited source videos (≤5\le5). Green/red shading indicates better/worse performance than the random baseline; workflow columns are shaded by κ\kappa.

Relational and procedural failure frontiers. This limited recognition does not extend to reliable relational or procedural judgments. Triplet detection is especially brittle: every model scores below 5% macro-F1, highlighting the difficulty of identifying complete instrument–verb–target interactions. Phase recognition exposes a related procedural gap: even the best dataset-averaged accuracy remains below the majority baseline, indicating weak procedural understanding from visual evidence.

Models’ weak agreement on surgical assessment. Workflow assessment exposes a different failure mode. Several models achieve high CVS accuracy by collapsing to frequent rubric values, yet their chance-corrected agreement remains near zero or negative. Our analysis on prediction distribution distinguishes two response patterns: majority collapse and optimistic over-crediting, in which models assign achieved-criterion scores when the reference indicates otherwise. The former results from label imbalance. For example, InternVL2.5-8B, repeatedly predicts the most frequent score for each criterion. This yields 71.2% accuracy with κ=0\kappa= 0; the predictions match common scores in the dataset but fail to distinguish cases with different reference scores. By contrast, over-crediting reveals a directional bias toward overstating the surgical skills. For instance, GPT-4o-mini and Lingshu-32B frequently mark CVS criteria as achieved even though the human annotations indicate low scores. Their accuracies (15.6% and 9.4%) are even below the random baseline (32.8%). We also notice that for GOALS skill assessment, such dataset bias also prevails: ratings are concentrated around the middle of the 5-level scale. As a result, predicting the most frequent score for each criterion produces small errors on many cases without using the surgical images. This again yields a shortcut that allows models to achieve low MAE without really distinguishing surgical cases.

Leaderboard. Qwen3-VL-32B achieves the best mean rank (3.6) across the seven tasks among the 17 evaluated models. However, its lead over the runner-up is statistically significant only for tool detection. Performance does not increase monotonically with model size, even within the same model family. Surprisingly, the evaluated proprietary and medical-domain models do not show a consistent advantage over general-purpose open-source models. These groups have task-specific strengths: GPT-4o-mini leads action recognition, while HuatuoGPT-Vision achieves the highest phase accuracy. Their localization performance is particularly weak: all five models score at most 1.6% F1F_1@0.5, compared with 22.7% for InternVL2.5-78B. Medical-domain models also trail Qwen3-VL-32B on both workflow-assessment tasks under κ\kappa, indicating that their specialization does not translate into stronger rubric-based surgical judgments.

Constructing the hard set: tasks that resist pretrained competence

Stage I reveals the limited surgical competence of general-purpose MLLMs on SurgKnowBench. We build SurgKnowBench-Hard, a subset where all models fail as a failure frontier. By focusing on episodes that MLLMs fail without external tooling, SurgKnowBench-Hard reduces the pretrained-competence confound in later agentic evaluation and provides a controlled stress test for whether generalist agents can make measurable progress at inference time.

Panel-based difficulty scoring. We score the difficulty of samples in Stage I from a panel of open-source MLLMs. For an episode xx, let Mx\mathcal{M}_x denote the panel models. We define the panel error as PanelError⁡(x)=1∣Mx∣∑m∈Mx(1−Perf⁡τ(x)(m,x))\operatorname{PanelError}(x) = \frac{1}{|\mathcal{M}_x|} \sum_{m \in\mathcal{M}_x} \left(1 - \operatorname{Perf}_{\tau(x)}(m, x)\right), where τ(x)\tau(x) is the task type of xx, and Perf⁡τ(m,x)∈[0,1]\operatorname{Perf}_{\tau}(m, x) \in[0, 1] is a task-normalized per-episode score. We construct SurgKnowBench-Hard by ranking candidate episodes within each dataset by PanelError.

SurgAgentGym: harnessing generalist agents for surgical understanding

Surgical training for human surgeons is a scaffolded process: trainees learn anatomical and surgical procedural knowledge from oracle references and canonical surgical cases and generalize this to real-world tasks. SurgAgentGym explores whether generalist agents can achieve surgical competence by solving tasks through an epistemic process with accessible tools, to see (i.e., visual grounding), to search (i.e., knowledge acquisition), and to think, and (i.e., internalized reasoning abilities).

To measure the agents’ behavior in an attributable and explainable way, SurgAgentGym is designed according to the following key principles:

Generalist perception access. Pretrained MLLM backbones have different levels of pixel-level visual grounding ability. For the vision-intensive surgical domain, this can constitute a "congenital" grounding ability gap across different models and can impact surgical understanding ability.

Regulated knowledge search. Surgical reasoning requires bounded and auditable evidence access. Unlike general-domain agent benchmarks, surgical agents need to control source identity and provenance to guarantee high-stakes judgments. We therefore introduce SurgWiki, a closed-domain surgical knowledge base whose sources are carefully selected, curated, and provenance-tracked.

SurgWiki: a regulated, multimodal surgical knowledge base

We present SurgWiki, a “learning platform” for generalist agents to acquire surgical knowledge. To ensure broad and sufficient coverage of surgical knowledge, we (1) regulate our knowledge selection based on common practices in surgical training [27, 28] and prior surgical knowledge bases [29, 30, 31]; (2) cover multimodal, heterogeneous knowledge spanning diverse sources to meet different task requirements; (3) include fundamental surgical knowledge and cover the procedures and task families evaluated in SurgKnowBench. SurgWiki contains 74,315 indexable knowledge chunks: 37,672 indexed text chunks, 35,951 tutorial-video chunks, and 692 case-record chunks, as shown in Figure 3.

SURGWIKI knowledge-base composition

Figure 3. SURGWIKI knowledge-base composition. The corpus contains 74,315 chunks spanning surgical textbooks, clinical guidelines, tutorial videos, and case records.

Reproducibility and governance. For accessibility, we prioritize public, open-access, citable, or otherwise reproducible sources whenever possible. For clinical plausibility, we hired 3 practicing surgeons for sanity-checking the assembled knowledge sources for procedural relevance, source appropriateness, and clinical plausibility. We also evaluate SurgWiki-Open, a corpus configuration excluding commercial sources (Table 25). The curation pipeline reconstructs this configuration from sources obtained directly from their providers; we do not distribute the source data.

ModelKnowledge basePhase Δ\DeltaAcc.Action Δ\DeltaF1CVS Δκ\Delta\kappaSkill Δκ\Delta\kappa
Qwen3-VL-8BFull KB+3.7+27.3−1.8−2.1
Qwen3-VL-8BSURGWIKI-Open+5.1+22.2−8.6−12.2
Qwen2.5-VL-7BFull KB+3.1+3.0−4.3−12.7
Qwen2.5-VL-7BSURGWIKI-Open+3.1+4.5−1.4−10.8

Table 25. Benchmark with SURGWIKI-Open. Entries report Stage II minus Stage I changes on SURGKNOWBENCH-Hard, multiplied by 100. The table covers temporal reasoning and workflow assessment; higher values indicate larger gains.

Knowledge sources. We construct SurgWiki from four provenance-tracked sources: surgical textbooks, clinical practice guidelines, narrated tutorial videos, and annotated case records. These sources provide complementary evidence for the declarative–procedural–conditional [27] knowledge ladder. Figure 3 summarizes the distribution and more details are provided in Appendix § 11.§ 3.1.

Data ingestion. For multimodal and heterogeneous knowledge, we apply a staged and semantics-aware ingestion pipeline that preserves document structure, source provenance, and visual context. Ingestion details can be found in the Appendix § 11.§ 3.2.

Knowledge search backend. We implement knowledge search using a deterministic and hybrid search backend. The agent defines an epistemic decision to search: what to search (i.e., query) and where to search (i.e., knowledge source). The search backend is isolated from agent behavior. Full retrieval pipelines are described in the Appendix § 11.3.4.

Tool definition

In SurgAgentGym, agents are provided with two tool families. The visual grounding tool helps the agent parse the current visual input with general-purpose perception ability; the knowledge tool gives the agent access to surgical knowledge in SurgWiki.

Visual grounding tool. The perceive() tool helps the agent parse the visual inputs by returning query-free, class-agnostic visual proposals. We implement perceive() with SAM 2 automatic mask generation [32] with the facebook/sam2-hiera-base-plus model to generate visual proposals that indicate semantic objects and boundaries. The returned observation contains a visual Set-of-Mark overlay [33] with numeric labels. Details are provided in Appendix § 11.§ 4.

Knowledge acquisition tool. The retrieve() tool is the agent-facing action for accessing SURGWIKI. Each tool call specifies three arguments: query, the agent-generated natural-language query; knowledge_type, which records the agent’s declared knowledge type (declarative, procedural, conditional, or metacognitive) for trajectory analysis; and source_type, which selects guidelines, textbooks, tutorial videos, or case records through a hard source-level filter.

Retrieval quality estimate. We therefore evaluate the retrieve() tool on real trajectories with a TREC-style graded relevance protocol. Two experts annotate 50 retrieve() calls sampled from pooled real trajectories as a calibration set; GPT-5 [34] as an LLM judge, scores the full pooled sample with the same rubric. retrieve() returns a relevance score with every item, passed to the agent as part of its observation. This score separates relevant from non-relevant evidence at AUROC = 0.894 against the LLM judge and 0.817 against human expert annotations, and holds within every knowledge source (0.68–0.87). The rubric and agreement statistics are in Appendix § 11.3.5.

Evaluating generalist agents for surgical scene understanding

We then evaluate how well generalist agents with tool access perform on SURGKNOWBENCH-Hard.

Agent, scaffold, and harness

Backbone selection. We evaluate 16 MLLM backbones spanning proprietary models, open-source models across three scale tiers, and medical-domain models. The proprietary group includes GPT-4o-mini [16] and Claude-Haiku-4.5 [17]. The open-source group is stratified by parameter scale: below 10B parameters, we evaluate InternVL2.5-8B [18], Qwen2.5-VL-7B-Instruct [19], Qwen3-VL-8B-Instruct, and Qwen3-VL-8B-Thinking [20]; between 10B and 30B parameters, we evaluate Pixtral-12B-2409 [21], InternVL3-14B [22], InternVL2.5-26B [18], and Gemma-3-27B-it [23]; above 30B parameters, we evaluate Qwen2.5-VL-32B-Instruct [19], Qwen3-VL-32B-Instruct [20], Qwen2.5-VL-72B-Instruct [19], and InternVL2.5-78B [18]. The medical-domain group includes Lingshu-32B [25] and MedGemma-27B-it [26].

Agent harness. SURGAGENTGYM implements tool use as a universal environment-level protocol shared across backbones. Each episode maintains an explicit state consisting of the task prompt, visual input, observation history, available tools, remaining tool budgets, and the final-answer contract. At each step, the backbone either emits a tool action or submits an answer. The environment parses tool actions and validates them against the tool schema. To prevent unbounded looping over a deterministic backend, retrieve() enforces a source-level call budget. Each source_type can be queried at most twice per episode, yielding an implicit maximum of eight retrieval calls across the four evidence families. perceive() is limited to two invocations for single-image tasks and three for multi-frame tasks, with duplicate calls removed. Full runtime constraint specifications are provided in Appendix § 11.§ 1.

Main results: do tool-mediated agents close the gap?

We first examine how generalist agents perform on the difficult episodes in SURGKNOWBENCH-Hard. Tool-mediated inference produces task-specific gains rather than uniform improvements in surgical understanding. Table 3 reports two comparisons. Panel A measures the overall change from Stage I direct answers to the Stage II agent setting. Panel B compares tools off and on under a matched agent harness, including prompts, to assess the contribution of tool access. Statistical analysis is provided in Appendix § 10.5.1. Appendix § 12.§ 3.1 extends ablation studies.

ModelSpatial RecognitionTemporal ReasoningWorkflow Assessment
Tool Det. Δ\DeltaF1↑Triplet Det.† Δ\DeltaF1↑Tool Loc.† Δ\Delta[email protected] ↑Phase Rec. Δ\DeltaAcc.↑Action Rec. Δ\DeltaF1↑CVS Ass. Δκ\Delta\kappa ↑Skill Ass. Δκ\Delta\kappa ↑
A. Across-stage comparison I = direct answer, II = agent with tools
Proprietary
GPT-4o-mini [16]+3.7—+5.70.0−17.7−1.9+1.2
Claude-Haiku-4.5 [17]+1.7—0.0+0.8−7.6+7.0∗−4.7
Open-source (< 10B)
InternVL2.5-8B [18]+6.9∗+0.3−10.6+0.9∗+18.0∗−4.8+13.0
Qwen2.5-VL-7B-Instruct [19]+3.9∗—−0.9+3.1∗+3.0∗−4.3−12.7
Qwen3-VL-8B-Instruct [20]+6.2∗—+1.6+3.7∗+27.3∗−1.8−2.1
Qwen3-VL-8B-Thinking [20]−5.1∗—−5.8+1.7∗+13.5∗−3.5+7.5
Open-source (10–30B)
Pixtral-12B-2409 [21]+13.5∗—+4.9+2.6∗+27.6∗−1.6+6.5
InternVL3-14B [22]+8.3∗+2.1+2.4+2.3∗+4.2∗+1.0+0.8
InternVL2.5-26B [18]+2.4+1.5−9.7+1.4∗+4.4∗+1.4−0.9
Gemma-3-27B-it [23]+1.6+2.1+5.7+3.1∗+12.8∗−1.5−4.4
Open-source (> 30B)
Qwen2.5-VL-32B-Instruct [19]+9.2∗+7.70.0+7.4∗+25.1∗+3.7−4.6
Qwen3-VL-32B-Instruct [20]+4.9∗—+3.2+5.1∗+26.5∗−0.5−15.1
Qwen2.5-VL-72B-Instruct [19]+5.3∗—0.0+4.9∗+25.6∗+7.5−2.8
InternVL2.5-78B [18]+1.1+3.8−19.5+1.4∗+9.2∗−2.3−6.4
Medical-domain
Lingshu-32B [25]+14.8∗—+4.9+4.9∗+26.7∗+1.3−2.5
MedGemma-27B-it [26]+9.2∗+1.0+4.0+3.1∗+24.0∗−4.2−10.5
B. Matched tool comparison Off = tools disabled, On = tools enabled
InternVL2.5-8B+1.9+0.6+8.1+0.3+14.0+6.3−1.7
Qwen2.5-VL-7B-Instruct+1.40.0+10.9+2.6+4.6−5.0−2.1
Qwen3-VL-8B-Instruct+4.60.0+1.7+3.7+26.2−13.5+5.6
Gemma-3-27B-it+4.1+3.0+4.9+1.7+6.6+2.5+4.5
Lingshu-32B+9.7−2.2−2.4+2.3+16.5+0.9−5.7

Table 3. Stage II performance on SURGKNOWBENCH-Hard. A compares Stage I direct answers with Stage II agents. B compares tools off and on within a matched agent setup. Each cell reports the difference. ∗ marks a significant change (paired video-cluster bootstrap, BH-FDR q≤0.05q \le0.05); † marks axes not significance-tested. Green/red indicates positive/negative changes. Bold/underlining marks the largest/second-largest change.

General performance. The full agent setting improves tool and action recognition for most evaluated models, but the gains vary across tasks and backbones. Tool detection improves for 15 of the 16 models and action recognition improves for all 14 open-source models, whereas GPT-4o-mini and Claude-Haiku-4.5 decline by 17.7 and 7.6 macro-F1 points, respectively. Phase recognition shows smaller, predominantly positive changes. These gains do not extend uniformly to localization or workflow assessment: localization declines across the InternVL2.5 family, while CVS and skill assessment show mixed changes across models. This reveals limitations similar to those observed in Stage I. The gains also need to be considered alongside the attained performance. In the Full setting evaluated in Panel A, the highest action-recognition score is 27.6% macro-F1, still below the majority baseline of 30.0%. Thus, even a task with substantial gains remains difficult in absolute terms. This emphasizes a prevailing frontier for generalist agents to solve domain-expertise surgical tasks. Detailed absolute scores and per-dataset results are provided in Appendix § 12.§ 2.

Benefits of tool access. In Panel B, we match the prompt and compare the net performance change caused by tool access. The matched comparison confirms positive tool-detection and action-recognition gains across all five evaluated backbones, ranging from +1.4 to +9.7 and from +4.6 to +26.2 macro-F1 points, respectively. Phase recognition also improves across all five backbones, although by smaller margins. Localization improves for four of the five backbones, showing that the declines observed across stages do not necessarily imply a negative contribution from tool access. For example, InternVL2.5-8B loses 10.6 localization points in Panel A but gains 8.1 points when tools are enabled in the matched setup. CVS and skill assessment, by contrast, still present a mixed signal with improvements for some backbones and deterioration for others: tool access does not provide a consistent benefit for rubric-based judgments. Mere tool access cannot solve subjective tasks.

Trajectory analysis

We analyze trajectories from our experiments to examine how agents use tools to solve surgical tasks. We focus on the following questions.

Q1: Which tools do agents use across tasks? Figure 4 shows task-dependent tool routing, with frequent visual grounding in spatial recognition and retrieval in phase recognition. This is consistent with our intuition that agents should use external tools to enhance their visual grounding abilities. Agents invoke perceive() in 78.2% of tool-detection episodes, 95.0% of triplet-detection episodes, and 76.2% of tool-localization episodes. For phase recognition, agents invoke retrieve() in 72.6% of episodes. Figure 8 shows that evidence-source selection in retrieve() also varies by task. Tutorial videos dominate retrieval for action recognition, accounting for 80.3% of retrieval calls. They also constitute the majority of retrieval calls for triplet detection (62.2%), tool localization (53.6%), and phase recognition (51.9%). Workflow assessment shows a different source preference: textbooks and guidelines together account for 80.8% of CVS retrieval calls and 66.9% of skill-assessment retrieval calls. This shows the agents’ tendency to seek temporal context for temporal tasks and procedural knowledge for workflow-level tasks.

Figure 4. Full-setting tool-use patterns across four representative agents on SURGKNOWBENCH-Hard.

Retrieval source distribution across task families

Figure 8. Retrieval source distribution aggregated over the four trajectory-analysis models. Percentages are computed over retrieval calls for each task.

Q2: Do tool results actually help the agents? Tool results do not necessarily provide usable evidence for the task because of imperfect tool outputs. We investigate whether tool results actually help the agents answer the questions. For retrieve(), we evaluate results from real agent trajectories using a graded relevance rubric covering topical match, constraint satisfaction, and information coverage. Two surgeon experts annotate a calibration sample of 50 retrieval calls, and an LLM judge evaluates the pooled sample using the same rubric. The reported precision@1 estimates range from 57% to 82%, indicating that retrieved items frequently address the agents’ generated queries, but relevance cannot be assumed for every call. The retrieval relevance score achieves an AUROC of 0.817 against expert annotations. These results indicate that retrieved items are adequately relevant for the queries. The evaluation protocol and agreement statistics are provided in Appendix § 11.3.5. Thus, a failure case is more attributed to that the generated query itself rather than the retrieval pipeline. Visual grounding might also return less usable evidence for agents. In the action-recognition trajectories of GPT-4o-mini and Claude-Haiku-4.5, perceive() returns no usable region proposals in 90%–100% of trajectories. These calls therefore provide no additional region-level evidence for recognizing the action, despite the agents having proactively invoked a visual grounding tool. This exposes the necessity of domain-specific tools beyond domain-specific agents.

Q3: Tool use does not guarantee success. Agents actively use tools even in failed episodes. Table 27 shows that successful and failed episodes average 2.32 and 2.36 tool calls, respectively. Their tool-use profiles differ: perceive() is more frequent in failed episodes (69.8% versus 56.4%), whereas retrieve() is more frequent in successful episodes (71.8% versus 65.0%). The negative association between using both tools and success largely disappears after task-centering, while retrieval use becomes positively associated with success. These patterns motivate targeted post-training of tool-use policies that select tools according to the task and the agent’s information needs.

Task familyOutcomeEpisodesperceive()retrieve()BothWarningCalls/ep.
Spatial RecognitionSuccess7166.269.060.625.42.62
Spatial RecognitionFailure112981.960.259.522.82.28
Temporal ReasoningSuccess8347.078.337.39.61.94
Temporal ReasoningFailure151763.469.147.920.82.12
Workflow AssessmentSuccess4158.563.448.89.82.59
Workflow AssessmentFailure48761.463.254.222.23.28
AllSuccess19556.471.848.215.42.32
AllFailure313369.865.053.021.72.36

Table 27. Success-conditioned trajectory statistics for the four-model Full-setting analysis on SURGKNOWBENCH-Hard. Success is computed by strict agreement between the parsed structured prediction and ground truth, independent of reward. Tool localization is excluded from this table because exact equality between continuous predicted and ground-truth bounding boxes is not a meaningful success criterion. The table covers 3,328 model–episode pairs (four models × 832 episodes). perceive(), retrieve(), Both, and Warning are percentages within each outcome group; Calls/ep. is the mean number of tool calls per episode.

Why do agents fail?

To sum up, we identify agent bottlenecks at three stages of the epistemic process: formulating tool queries, integrating imperfect evidence, and grounding retrieved evidence in the final answer.

Does the agent proactively seek evidence relevant to the question? (Query quality). We show that evidence needs are modality-specific: for perception-heavy tasks, visual exemplars are often more direct than abstract textbook passages (much as for human trainees). Yet agents show a preference for authoritative text, querying textbooks for a "golden answer" where a visual case would serve better. In the trajectory analysis, tutorial videos account for 80.3% of retrieval calls in action recognition, while textbooks and guidelines account for 80.8% in CVS assessment. Task framing and query generation are metacognitive skills acquired through agentic learning like post-training or self-evolution.

Can the agent withstand imperfect evidence? (Agent robustness). Retrieved evidence can often be partially irrelevant or noisy, and in most failure cases agents do not withstand it. In CVS episodes, retrieved guideline text can even override correct visual understanding. In the two proprietary models’ action-recognition trajectories, perceive() returns no usable region proposals in 90–100% of trajectories. GPT-4o-mini’s recall for the dominant action, “hold,” falls from 0.76 under direct inference to 0.07 in the agent setting. Robustness to imperfect or noisy evidence is a core capability that current models lack. Explicit uncertainty and credibility quantification and calibration can help agents remain robust to misleading evidence.

Can the agent leverage useful evidence? (Evidence grounding). Even perfectly relevant evidence must be integrated into the model’s reasoning. Under heavy tool use the accumulated context itself becomes a burden, and useful evidence is retrieved but ignored. In our trajectory analysis, failed episodes invoke perceive() more often than successful episodes (69.8% vs. 56.4%) and use both tools slightly more often (53.0% vs. 48.2%). Evidence grounding is the binding constraint.

Related work

Surgical superintelligence. Surgical AI has advanced task-specific scene understanding on curated specific datasets [1, 35, 2], while recent surgical MLLMs extend supervision through video-text pretraining, instruction tuning, VQA, scene graphs, and multi-task benchmarks [36, 37, 38, 39, 4, 40, 41, 42]. Surgical foundation models further scale representation learning across surgery procedures [43, 44, 45, 46, 47].

Medical agent benchmarks. Medical agent benchmarks such as AgentClinic, MedAgentBench, and MMAgent evaluate general clinical agents with action or diagnostic tools, but do not target surgical visual scene understanding or attribution of knowledge acquisition and visual grounding [48, 49, 50]. Closer to surgery, SurgRAW and CARES study agentic reasoning for robotic surgical video analysis and surgical error detection [51, 52].

Conclusion

We presented a two-stage evaluation of surgical competence for general MLLMs. First, we examined what these models can infer directly from surgical scenes without domain-specific adaptation. Despite strong general-purpose capabilities, current MLLMs show only limited surgical understanding, particularly beyond object-level recognition toward relational, procedural, and assessment tasks. We then evaluated whether tool-augmented agents can compensate for these knowledge gaps by retrieving external information and grounding visual evidence. Agentic inference improves some tasks, showing that generalist MLLMs possess epistemic capabilities that allow them to seek missing domain knowledge proactively. However, our analysis also reveals that these capabilities remain imperfect: agents may formulate the wrong queries, fail to reconcile imperfect evidence, or fail to ground evidence in reasoning. Our findings therefore suggest that MLLM agents could potentially bridge their domain knowledge gap through epistemic reasoning and active tool-use, but these metacognitive capabilities themselves are not taken for granted and still remain an important challenge for generalist agents in the surgical domain.

Limitations. This work has several limitations. (1) SurgKnowBench harmonizes public surgical datasets, so its coverage is bounded by the procedures, institutions, camera views, annotation schemes, and task definitions available in these sources. (2) SurgWiki is a closed and provenance-tracked evidence corpus, but its contents are still limited by source availability and licensing.

Acknowledgments and Disclosure of Funding

The project on which this report is based was sponsored by the Federal Ministry of Research, Technology and Space (BMFTR) under the funding code 16IS26022. Responsibility for the content of this publication lies with the authors. We thank Feng Jin for providing expert advice and assistance with data curation. We also thank Xinling Xie for assistance. The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) under the NHR project b306dd. NHR funding is provided by federal and Bavarian state authorities. We acknowledge the EuroHPC Joint Undertaking for awarding this project access to the EuroHPC supercomputer LEONARDO, hosted by CINECA (Italy) and the LEONARDO consortium, through a EuroHPC Development Access call (project EHPC-DEV-2026D02-157).

Societal impact

This work supports safer surgical AI research by providing reproducible benchmarks and traceable evidence-use trajectories. SURGKNOWBENCH may also aid medical education, but the artifacts are intended for research evaluation, not autonomous clinical decision-making.

Source datasets

Our benchmark is built from eleven public surgical datasets spanning seven surgical procedures, from which we instantiate seven tasks grouped into three task types. Section B.1 describes each source dataset and the tasks we draw from it; Section B.2 describes the prediction tasks and answer formats. The dataset–task inventory is given in Tab. 1.

Throughout, we evaluate frozen general-purpose multimodal language models in a zero-shot manner, with no task-specific training or fine-tuning. Each episode pairs its visual input with a task prompt and a structured answer contract. Visual-input handling, output schemas, and canonical task prompts are described in Appendices C.2, C.3, and D.2.1, respectively.

Datasets

Cholec80.

  • Description. Cholec80 [5] is a public CAMMA dataset (University of Strasbourg) of 80 laparoscopic cholecystectomy videos recorded at 25 fps, with every frame annotated for the surgical phase and tool presence.

  • Surgery type. Laparoscopic cholecystectomy

  • Tasks. Phase recognition.

Cholec80-CVS.

  • Description. Cholec80-CVS [8] augments the same Cholec80 videos with expert annotations of Strasberg’s Critical View of Safety (CVS), scored on three criteria: two structures, cystic plate, and hepatocystic triangle.

  • Surgery type. Laparoscopic cholecystectomy

  • Tasks. CVS assessment.

CholecT50.

  • Description. CholecT50 [1] is a CAMMA dataset of 50 laparoscopic cholecystectomy videos annotated for fine-grained surgical activity as action triplets ⟨instrument,verb,target⟩\langle\text{instrument}, \text{verb}, \text{target}\rangle (6 instruments, 10 verbs, 15 targets, 100 triplet classes).

  • Surgery type. Laparoscopic cholecystectomy

  • Tasks. Tool detection; tool localization; triplet detection.

Cataract-101.

  • Description. Cataract-101 [14] is a public dataset from Klagenfurt University of 101 cataract-surgery videos (four surgeons, two expertise levels) recorded at 25 fps and annotated with 10 surgical phases.

  • Surgery type. Cataract surgery / phacoemulsification (ophthalmology).

  • Tasks. Phase recognition.

HeiChole.

  • Description. HeiChole [2] is a multicentric benchmark of laparoscopic cholecystectomy from the EndoVis 2019 sub-challenge: 33 videos from three centres, annotated framewise for surgical phases, four surgical actions, instrument presence, and five skill dimensions.

  • Surgery type. Laparoscopic cholecystectomy (hepatobiliary / general surgery).

  • Tasks. Action recognition; skill assessment.

MultiBypass140.

  • Description. MultiBypass140 [9] is a multicentric benchmark of laparoscopic Roux-en-Y gastric bypass: 140 videos at 25 fps from two centres (StrasBypass70, Strasbourg; BernBypass70, Bern), annotated with 12 phases, 46 steps, and 5 intra-operative adverse events.

  • Surgery type. Laparoscopic Roux-en-Y gastric bypass (bariatric / general surgery).

  • Tasks. Phase recognition.

PitVis.

  • Description. PitVis [13] is the dataset of the PitVis-2023 Challenge (an Endoscopic Vision sub-challenge at MICCAI 2023): 25 videos of endoscopic pituitary surgery at 24 fps, the first public dataset with both surgical-step and instrument annotations.

  • Surgery type. Endoscopic pituitary surgery, transsphenoidal approach (neurosurgery).

  • Tasks. Phase recognition; tool detection.

GraSP.

  • Description. GraSP [12] is a multi-granular benchmark of robot-assisted radical prostatectomy: 13 da Vinci videos (Fundación Santa Fé de Bogotá) annotated for surgical phases, steps, instruments, and atomic actions.

  • Surgery type. Robot-assisted radical prostatectomy (urology).

  • Tasks. Phase recognition; tool detection.

AutoLaparo.

  • Description. AutoLaparo [10] is a public multi-task dataset from CUHK for image-guided automation in laparoscopic hysterectomy: 21 videos at 25 fps with annotations for surgical phase, motion prediction, and instrument/keypoint tracking.

  • Surgery type. Laparoscopic hysterectomy (gynaecology).

  • Tasks. Phase recognition; tool detection.

HeiCo.

  • Description. HeiCo [11] provides laparoscopic colorectal surgery videos with workflow annotations.

  • Surgery type. Laparoscopic proctocolectomy.

  • Tasks. Phase recognition.

CATARACTS.

  • Description. CATARACTS [15] provides cataract surgery videos with surgical-instrument annotations.

  • Surgery type. Cataract surgery.

  • Tasks. Tool detection.

Tasks

We group the seven tasks into three task types: Spatial Recognition (tool detection, tool localization, triplet detection), Temporal Reasoning (phase recognition, action recognition), and Workflow Assessment (CVS assessment, skill assessment). Frame-level labels and interval-level rubric scores retain their respective annotation units; an interval’s timestamps do not imply that every frame in that interval is provided to the model.

Tool detection.

  • Description. Identify every distinct surgical-instrument type visible in the supplied image. Several instrument types may be present.

  • Task type. Spatial Recognition.

  • Visual input. An annotated frame and its dataset-specific instrument vocabulary.

  • Answer. An unordered set of instrument-label strings, represented as a JSON array. Each type is listed once; an empty array is valid when no instrument is visible.

Tool localization.

  • Description. Identify and localize the instrument in a box-annotated image.

  • Task type. Spatial Recognition.

  • Visual input. A box-annotated frame.

  • Answer. An instrument label and bounding box under the model-family-specific grounding contract. The evaluator normalizes these outputs into labeled image coordinates; the contracts are given in Appendix § 11.§ 2.1.

Triplet detection.

  • Description. Identify the surgical interactions visible in an annotated CholecT50 frame, expressed as ⟨instrument,verb,target⟩\langle\text{instrument},\text{verb},\text{target}\rangle triplets. Multiple triplets may co-occur.

  • Task type. Spatial Recognition.

  • Visual input. An annotated frame and the instrument, verb, and anatomical-target vocabularies.

  • Answer. An unordered set of complete triplet strings, represented as a JSON array. An empty array is valid when no interaction is visible.

Phase recognition.

  • Description. Identify the surgical phase or dataset-specific workflow step shown in the supplied annotated frame.

  • Task type. Temporal Reasoning.

  • Visual input. An annotated frame with the procedure description and the dataset-specific phase or step vocabulary.

  • Answer. One canonical label string copied from the supplied vocabulary. The task is scored against the corresponding frame-level label.

Action recognition.

  • Description. Identify every distinct surgical action visible in the supplied HeiChole frame. Multiple action labels may apply.

  • Task type. Temporal Reasoning.

  • Visual input. An annotated frame with the procedure description and the action vocabulary supplied by the task prompt.

  • Answer. An unordered set of action-label strings, represented as a JSON array and scored against the frame-level annotation. An empty array is valid when no action is visible.

CVS assessment.

  • Description. Assess Strasberg’s three Critical View of Safety criteria: two structures, cystic plate, and hepatocystic triangle.

  • Task type. Workflow Assessment.

  • Visual input. One or more representative images associated with the annotated interval, in temporal order. The prompt asks the model to assess the interval jointly from the provided images.

  • Answer. A JSON dictionary with keys c1, c2, and c3, each taking an ordinal value in {0,1,2}\{0, 1, 2\}. The criterion-specific rubric is given in Appendix § 11.§ 2.1.

Skill assessment.

  • Description. Rate surgical execution over the annotated interval using the GOALS-style rubric used in HeiChole. We retain the holistic, Calot-phase, and dissection-phase assessment sub-axes.

  • Task type. Workflow Assessment.

  • Visual input. The representative visual inputs associated with the whole-procedure or phase-segment episode, subject to its frame budget.

  • Answer. Five ordinal scores from 1 to 5, returned as a JSON dictionary with keys depth_perception, bimanual_dexterity, efficiency, tissue_handling, and case_difficulty.

SurgKnowBench construction

Dataset harmonization

The source datasets in SurgKnowBench differ in annotation units, label vocabularies, and metadata conventions. We harmonize them by converting each source annotation into a common episode record while preserving the original supervision target. Each record stores the dataset identity, procedure type, split, video or case identifier, temporal location, task type, visual-input references, and structured ground truth.

Harmonization is performed at the dataset–task level. Frame-level annotations, such as tool presence, phase labels, and instrument boxes, are converted into single-frame episodes. Interval-level annotations, such as CVS and GOALS-style skill assessment, are converted into interval episodes with start/end timestamps and endpoint visual references. Dataset-specific labels are mapped to the canonical strings used by the corresponding task template and evaluator, while source metadata is retained for per-dataset reporting, hard-set construction, and leakage checks.

This step standardizes the evaluation substrate without redefining the scoring rules. The answer schemas are specified separately in Appendix § 10.§ 3, frame sampling is specified in Appendix § 10.§ 2, and metric definitions are specified in Appendix § 10.§ 5.

Annotation agreement. To document the reliability evidence available for the inherited annotations, Table 4 summarizes inter-annotator agreement reported by the source datasets. These entries reflect each source’s annotation protocol.

DatasetAnnotatorsIAA reported?Metric / value
Cholec80 [5]1 senior surgeonNoSingle annotator; none
CholecT50 [1]2 surgeons + 3rd-clinician mediationNoConsensus only, no metric
Cholec80-CVS [8]4 (2 surgeons + 2 residents)NoDiscussion consensus, no metric
HeiChole [7]Phase: 3 students / Skill: 2 studentsPartialPhase: Fleiss’ κ\kappa 0.92 ±\pm 0.06 (train) / 0.95 ±\pm 0.04 (test); Skill (GOALS): none
MultiBypass140 [9]2 board-certified surgeonsYesCohen’s κ\kappa 96% (phase) / 81% (step) — cited from group’s prior ontology work
AutoLaparo [10]2 gynaecologistsNoProofread / cross-checked, no metric
HeiCo [11]1 resident + 5 expert reviewersYesPhase: Cohen’s κ\kappa intra 0.834, inter 0.68–0.79, Fleiss’ κ\kappa 0.712; segmentation inter-annotator DSC 0.96
GraSP [12]2 residents + 17 annotatorsNoIndependent double-annotation + consensus, no metric
PitVis [13]2 trainees + 1 consultantNoExpert-confirmed consensus, no metric
Cataract-101 [14]1 ophthalmologistNoSingle annotator; none
CATARACTS [15]2 non-MD expertsYes*Per-tool percent agreement 0.630–0.998 (*not chance-corrected; no kappa)

Table 4. Inter-annotator agreement reports in the source datasets.

Frame sampling and visual inputs

SURGKNOWBENCH contains both frame-level and interval-level visual tasks. For frame-level tasks, such as tool detection, phase recognition, action recognition, triplet detection, and tool localization, the episode is anchored to the annotated frame and the primary visual input is that frame. For interval-level tasks, such as CVS assessment and GOALS-style skill assessment, the episode stores the annotated temporal span together with representative visual references and interval metadata.

An annotation interval specifies the scope of a reference score; it does not imply that the model sees every frame of that interval. The frame budget is an upper bound on the images provided, not a fixed clip length: a single-frame episode is not expanded into a multi-frame clip by setting a larger cap.

To control visual context length, we cap the number of frames exposed to a model by max_frames. Given NN candidate frames and a frame budget KK, if N>KN > K, we sample uniformly with endpoint preservation:

I={round⁡(i(N−1)K−1)}i=0K−1.\mathcal{I} = \left\{ \operatorname{round}\left(\frac{i(N-1)}{K-1}\right) \right\}_{i=0}^{K-1}.

The sampled indices are then sorted and deduplicated. This preserves the first and last frames of an interval while distributing the remaining visual evidence across the temporal span. For Stage I diagnostic evaluation, we set K=16K = 16.

This sampling rule applies only to the visual inputs supplied as part of the benchmark episode. In SURGAGENTGYM, agents may additionally request tool-generated visual evidence through perceive(); the runtime limits for such tool calls are specified separately in Appendix § 11.§ 1, and the proposal-generation details are specified in Appendix § 11.§ 4.

Task output schemas

Each benchmark episode specifies a constrained final-answer schema. The same task-level answer schema is used for direct prediction in Stage I and for the submit() action in SURGAGENTGYM. This separates the benchmark answer contract from the agent-control schema described in Appendix § 11.§ 1.

Categorical labels. Phase recognition uses a single canonical label string:

y^∈Cphase.\hat{y} \in\mathcal{C}_{\mathrm{phase}}.

Label sets. Tool detection, action recognition, and triplet detection use unordered label sets:

Y^⊆C.\hat{Y} \subseteq\mathcal{C}.

For triplet detection, each label is a complete ⟨instrument,verb,target⟩\langle\mathrm{instrument}, \mathrm{verb}, \mathrm{target} \rangle string. Empty sets are valid outputs when no label is present.

Bounding boxes. Tool localization uses a list of labeled axis-aligned boxes:

{(ℓj,xj,yj,wj,hj)}j=1M.\left\{(\ell_j, x_j, y_j, w_j, h_j)\right\}_{j=1}^{M}.

where coordinates are normalized to [0,1][0,1] relative to image width and height.

Ordinal rubric dictionaries. CVS and skill assessment use fixed-key ordinal dictionaries. CVS uses three keys {c1,c2,c3}\{\mathrm{c1},\mathrm{c2},\mathrm{c3}\} with scores in {0,1,2}\{0,1,2\}. Skill assessment uses GOALS-style keys, including depth perception, bimanual dexterity, efficiency, tissue handling, and case difficulty, on the original ordinal scale.

All model responses are parsed into these structured objects before scoring. Malformed but recoverable responses undergo one deterministic repair pass; responses that remain non-conformant are mapped to the task-specific null prediction and retained in the evaluation.

Hard-set difficulty scoring

For phase recognition, Perf⁡τ\operatorname{Perf}_{\tau} is exact-match accuracy. For label-set tasks, including tool detection, action recognition, and triplet detection, Perf⁡τ\operatorname{Perf}_{\tau} is Jaccard similarity between predicted and reference label sets. For tool localization, Perf⁡τ\operatorname{Perf}_{\tau} is mean label-matched IoU. For ordinal workflow assessment, including CVS grading on {0,1,2}\{0,1,2\} and GOALS skill ratings on {1,…,5}\{1,\ldots,5\}, Perf⁡τ\operatorname{Perf}_{\tau} is range-normalized agreement:

Perf⁡τ(m,x)=1−1∣Kτ∣∑k∈Kτ∣y^m,x,k−yx,k∣bτ−aτ,(1)\operatorname{Perf}_{\tau}(m,x)=1-\frac{1}{|\mathcal{K}_{\tau}|}\sum_{k\in\mathcal{K}_{\tau}}\frac{|\hat{y}_{m,x,k}-y_{x,k}|}{b_{\tau}-a_{\tau}}, \tag*{(1)}

where Kτ\mathcal{K}_{\tau} is the set of rubric keys, and [aτ,bτ][a_{\tau},b_{\tau}] is the valid ordinal range. All per-episode scores lie in [0,1][0,1], so 1−Perf⁡τ1-\operatorname{Perf}_{\tau} places errors from heterogeneous tasks on a common difficulty scale. Episodes with high PanelError therefore represent failures shared by multiple pretrained models.

Prediction stability. To assess the stability of predictions used in hard-set selection, Table 5 reports outcome flip rates and per-episode score variance across three reruns for two representative backbones. This measures repeated-inference variation at the episode level. Across three reruns, Qwen2.5-VL-7B and Qwen3-VL-8B each have an episode flip rate of 0.32%0.32\%, with mean per-episode score variance of 0.00050.0005 and 0.00030.0003, respectively (Table 5).

ModelEpisode flip rateMean per-episode score variance
Qwen2.5-VL-7B0.32%0.32\%0.00050.0005
Qwen3-VL-8B0.32%0.32\%0.00030.0003

Table 5. Flip rate = fraction of episodes whose outcome changes across the three reruns; mean per-episode variance computed over episode-level scores.

Evaluation metrics

This section gives the full metric definitions used by our evaluator. Model outputs are first normalized into one of three structured formats: label sets, ordinal rubric dictionaries, or labeled bounding boxes. The main paper reports a compact subset of these metrics in Table 2; the remaining metrics are used for diagnostic analyses and are reported in Appendix § 12.§ 1.

Main-table metrics. Table 2 reports macro-F1F_1 for tool detection, action recognition, and full triplet detection; strict accuracy for phase recognition; F1@0.5[email protected] for tool localization; mean per-key accuracy and mean quadratic-weighted Cohen’s κ\kappa for CVS assessment; and mean absolute error (MAE) and mean quadratic-weighted κ\kappa for skill assessment. The rank column is computed across the seven task axes; for workflow-assessment columns, κ\kappa is used for ranking because it corrects for chance agreement and is less vulnerable to majority-value collapse.

Score aggregation. We first compute performance for each available (dataset, task) cell. Task-level scores are then obtained by a macro mean over datasets supporting that task: sτ(m)=1∣Dτ∣∑d∈Dτsd,τ(m)s_{\tau}(m)=\frac{1}{|D_{\tau}|}\sum_{d\in D_{\tau}}s_{d,\tau}(m), where sd,τ(m)s_{d,\tau}(m) is the score of model mm on dataset dd for task axis τ\tau, and DτD_{\tau} is the set of datasets with annotations for τ\tau. This aggregation prevents large datasets from dominating task-level performance. The Rank column in Table 2 reports each model’s mean rank across the seven task axes; for paired workflow columns, we rank by the chance-corrected κ\kappa value. Per-dataset scores and auxiliary metrics are in Appendix § 12.§ 1.

Label-set metrics. Tool detection, action recognition, and triplet detection are evaluated as label-set prediction tasks. Phase recognition is treated as a single-label special case and is reported with strict accuracy in the main table. For sample ii, let Y^i\hat{Y}_i denote the predicted label set and YiY_i the ground-truth label set. Exact-set accuracy is

Acc⁡set=1N∑i=1NI[Y^i=Yi].\operatorname{Acc}_{\mathrm{set}}=\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}\left[\hat{Y}_i=Y_i\right].

For class cc, let TPcTP_c, FPcFP_c, and FNcFN_c denote class-level true positives, false positives, and false negatives. Class-wise F1F_1 and macro-averaged F1F_1 are

F1(c)=2TPc2TPc+FPc+FNc,F1macro=1∣C∣∑c∈CF1(c).F_1^{(c)}=\frac{2TP_c}{2TP_c+FP_c+FN_c},\qquad F_1^{\mathrm{macro}}=\frac{1}{|\mathcal{C}|}\sum_{c\in\mathcal{C}}F_1^{(c)}.

Macro-F1F_1 gives equal weight to every class and is therefore less dominated by frequent surgical labels. The evaluator also reports micro-averaged precision, recall, and F1F_1, computed by pooling TPTP, FPFP, and FNFN over all classes, together with Jaccard similarity:

J=TPTP+FP+FN.J = \frac{TP}{TP + FP + FN}.

For label-set tasks where the empty set is a valid answer, empty predictions and empty references are preserved. When appropriate, the evaluation vocabulary is augmented with a synthetic ⟨no_label⟩\langle\text{no\_label}\rangle class so that empty-set agreement is counted as a valid prediction event.

Triplet detection metrics. For triplet detection, each label in the headline metric is a complete ⟨instrument,verb,target⟩\langle\text{instrument},\text{verb},\text{target}\rangle tuple. A prediction receives full-triplet credit only when all three components are jointly correct. For diagnostics, we additionally project predicted and ground-truth triplet sets onto the instrument, verb, and target axes:

Y^iinst={instrument⁡(t):t∈Y^i},\hat{Y}^{\mathrm{inst}}_i = \{\operatorname{instrument}(t) : t \in\hat{Y}_i\},
Y^iverb={verb⁡(t):t∈Y^i},\hat{Y}^{\mathrm{verb}}_i = \{\operatorname{verb}(t) : t \in\hat{Y}_i\},
Y^itarget={target⁡(t):t∈Y^i}.\hat{Y}^{\mathrm{target}}_i = \{\operatorname{target}(t) : t \in\hat{Y}_i\}.

The same label-set metrics are then computed on each projection. These component-wise scores help distinguish instrument, verb, and target recognition failures from compositional binding failures.

Workflow assessment metrics. CVS assessment and skill assessment are ordinal dictionary-valued tasks. CVS uses three criteria,

KCVS={c1,c2,c3},\mathcal{K}_{\mathrm{CVS}} = \{c_1,c_2,c_3\},

with scores in {0,1,2}\{0,1,2\}. Skill assessment uses GOALS-style rubric keys, including depth perception, bimanual dexterity, efficiency, tissue handling, and case difficulty, on the original ordinal scale. For a rubric key kk, per-key exact-score accuracy is

Acc⁡k=1N∑i=1NI[v^i,k=vi,k].\operatorname{Acc}_k = \frac{1}{N}\sum_{i=1}^{N}\mathbb{I}[\hat{v}_{i,k}=v_{i,k}].

The CVS accuracy reported in the main table is the mean of Acc⁡k\operatorname{Acc}_k over the three CVS keys. We also compute strict dictionary accuracy,

Acc⁡dict=1N∑i=1NI[∀k∈K, v^i,k=vi,k],\operatorname{Acc}_{\mathrm{dict}} = \frac{1}{N}\sum_{i=1}^{N}\mathbb{I}[\forall k \in\mathcal{K},\ \hat{v}_{i,k}=v_{i,k}],

which requires all rubric keys to be correct for a sample to receive credit.

For ordinal error magnitude, we report mean absolute error per key and averaged over keys:

MAE⁡k=1N∑i=1N∣v^i,k−vi,k∣,MAE⁡mean=1∣K∣∑k∈KMAE⁡k.\operatorname{MAE}_k = \frac{1}{N}\sum_{i=1}^{N}|\hat{v}_{i,k}-v_{i,k}|,\qquad \operatorname{MAE}_{\mathrm{mean}} = \frac{1}{|\mathcal{K}|}\sum_{k\in\mathcal{K}}\operatorname{MAE}_k.

For chance-corrected ordinal agreement, we compute quadratic-weighted Cohen’s κ\kappa per key and average across keys. For one key, let OO be the observed agreement matrix and EE the expected agreement matrix under independent marginals. The quadratic weight between ordinal labels lil_i and ljl_j is

Wij=(li−lj)2(lmax⁡−lmin⁡)2,W_{ij} = \frac{(l_i-l_j)^2}{(l_{\max}-l_{\min})^2},

and the weighted kappa is

κw=1−∑i,jWijOij∑i,jWijEij,κwmean=1∣K∣∑k∈Kκw,k.\kappa_w = 1 - \frac{\sum_{i,j}W_{ij}O_{ij}}{\sum_{i,j}W_{ij}E_{ij}},\qquad \kappa_w^{\mathrm{mean}} = \frac{1}{|\mathcal{K}|}\sum_{k\in\mathcal{K}}\kappa_{w,k}.

Skill assessment additionally reports Spearman rank correlation per key as a diagnostic for whether a model preserves the relative ordering of cases.

Localization metrics. Tool localization predictions are lists of labeled boxes. Each predicted box and ground-truth box is represented by an instrument label and normalized image coordinates. The intersection-over-union between prediction pp and ground truth gg is

IoU⁡(p,g)=∣p∩g∣∣p∪g∣.\operatorname{IoU}(p,g)=\frac{\lvert p\cap g\rvert}{\lvert p\cup g\rvert}.

For each frame, predictions and ground-truth boxes are matched using Hungarian assignment with cost 1−IoU⁡1-\operatorname{IoU}. Label mismatches are assigned zero IoU before matching, so a geometrically accurate box with the wrong instrument label receives no localization credit. At threshold 0.5, a matched prediction is a true positive only if it has the correct label and IoU ≥0.5\ge0.5:

Precision⁡@0.5=TP0.5Npred,Recall⁡@0.5=TP0.5Ngt,\operatorname{Precision}@0.5=\frac{TP_{0.5}}{N_{\mathrm{pred}}},\qquad\operatorname{Recall}@0.5=\frac{TP_{0.5}}{N_{\mathrm{gt}}},
F1@0.5=2Precision⁡@0.5Recall⁡@0.5Precision⁡@0.5+Recall⁡@0.5.[email protected]=\frac{2\operatorname{Precision}@0.5\operatorname{Recall}@0.5}{\operatorname{Precision}@0.5+\operatorname{Recall}@0.5}.

The main table reports F1@0.5[email protected]. The appendix additionally reports mIoU, precision, and recall.

Parsing and null predictions. All metrics are computed on parsed structured outputs. The evaluator first applies task-specific normalization and one deterministic repair pass. Non-conformant outputs after repair are converted to the task-specific null prediction: an empty set for label-set tasks, an invalid or missing rubric entry for dictionary tasks, and an empty box list for localization. These failures are kept in the score.

Statistical analysis

Model ranking. We report 95% confidence intervals for the mean-rank column of Tab. 2, recomputing the full model ranking in each of 1,000 video-level bootstrap resamples.

Across-stage comparison. Paired per (model, axis) at the video level, testing Δ=Stage II−Stage I\Delta=\mathrm{Stage\ II}-\mathrm{Stage\ I} on the hard set (the deltas reported in Tab. 3, Panel A), with the same clustered bootstrap and BH-FDR correction. Of 80 testable cells (16 models ×\times 5 axes), 39 are significant improvements and only 1 a significant regression (Qwen3-VL-8B-Thinking, Tool Det.). The significant improvements comprise Phase 14/16 models, Action 14/16, Tool Det. 10/16, CVS 1/16, and Skill 0/16.

SurgAgentGym implementation

Agent schema

SurgAgentGym separates the logical agent interface from the concrete response format used by each model backend. The shared runtime abstraction is a StructuredOutputContract, which specifies the prompt suffix, backend-specific request arguments, parser, and repair prompt. A model response is parsed into either an AgentStep for tool-using episodes or a ParsedText object for no-tool direct-answer episodes.

Tool-mode output. In tool-using conditions, the environment expects the agent to take one tool action at a time. Internally, each action is represented as an ActionCall with an action string and a params object:

internal_action_call
{
  "action": "<tool_name>",
  "params": {
    "<parameter_name>": "<parameter_value>"
  }
}

The concrete model-side carrier depends on the active structured-output contract. Prompt-only text contracts ask the model to emit the ActionCall object directly, optionally preceded by a short Reasoning: line. Backend-native structured contracts wrap the same action call inside an agent_step object:

agent_step_wrapper

{
  "reasoning": "<brief rationale>",
  "action_call": {
    "action": "<tool_name>",
    "params": {
      "<parameter_name>": "<parameter_value>"
    }
  }
}

The reasoning field is included only when reasoning mode is enabled. OpenAI models use response_format JSON schema, vLLM-served models use guided_json constrained decoding, and Claude models use Anthropic native tool use with an internal emit_agent_step tool. These carriers differ at the API level but are parsed into the same AgentStep representation.

No-tool direct-answer output. In C0, the agent does not call environment tools. The structured-output contract instead asks the model to emit a task-aware direct-answer object:

direct_answer_schema

{
  "answer": <task_specific_answer>
}

When reasoning mode is enabled, the same object includes a required reasoning field:

direct_answer_with_reasoning

{
  "reasoning": "<brief rationale>",
  "answer": <task_specific_answer>
}

The type of answer is selected from the active task metadata. Label-set tasks use a list of strings; triplet detection uses a list of strings in the form <instrument, verb, target>; localization uses a list of box objects; rubric tasks use an ordinal score dictionary; scalar tasks use a string.

For example, tool detection is parsed as a list of label strings:

tool_detection_answer_schema

{
  "answer": ["grasper", "hook"]
}

Triplet detection is parsed as a list of complete instrument–verb–target strings:

triplet_detection_answer_schema

{
  "answer": [
    "<grasper, retract, gallbladder>",
    "<hook, dissect, cystic_plate>"
  ]
}

The default localization schema uses normalized [0,1][0, 1] (x,y,w,h)(x, y, w, h) boxes: tool_localization_default_schema

{
  "answer": [
    {
      "instrument": "grasper",
      "x1": 0.18,
      "y1": 0.24,
      "w": 0.31,
      "h": 0.19
    }
  ]
}

Grounding-specific contracts may instead use the native box convention of the model, represented with bbox_2d and label. The evaluator later converts these coordinates back to the benchmark coordinate system using the stored bbox_convention metadata.

grounding_native_schema

{
  "answer": [
    {
      "bbox_2d": [120, 240, 510, 430],
      "label": "grasper"
    }
  ]
}

Rubric-style tasks use a fixed dictionary schema. CVS assessment uses three ordinal criteria:

cvs_answer_schema

{
  "answer": {
    "c1": 2,
    "c2": 1,
    "c3": 2
  }
}

Skill assessment uses the GOALS-style rubric keys:

skill_answer_schema

{
  "answer": {
    "depth_perception": 4,
    "bimanual_dexterity": 4,
    "efficiency": 3,
    "tissue_handling": 4,
    "case_difficulty": 2
  }
}

Tool schemas. Tool declarations define the action space exposed to tool-using agents. Each callable tool is represented by a ToolSchema containing the tool name, description, parameter list, required fields, default values, and optional enumerated choices. Each parameter is represented by a ParameterSchema with type drawn from {string, integer, float, boolean, array, object, any}. The environment serializes the currently available tool schemas into the observation field available_tools; the step engine validates the parsed ActionCall against this schema before dispatch.

A simplified schema for perceive() is:

perceive_tool_schema
{
  "name": "perceive",
  "parameters": {
    "frame": {
      "type": "any",
      "default": "current"
    },
    "granularity": {
      "type": "string",
      "enum": ["coarse", "fine"],
      "default": "coarse"
    }
  }
}

For multi-frame tasks, the frame parameter is dynamically specialized to an integer index aligned with the visual window shown to the model:

perceive_multiframe_schema
{
  "name": "perceive",
  "parameters": {
    "frame": {
      "type": "integer",
      "minimum": 0,
      "maximum": "N-1",
      "default": "N//2"
    },
    "granularity": {
      "type": "string",
      "enum": ["coarse", "fine"],
      "default": "coarse"
    }
  }
}

The retrieve() tool requires three arguments: query contains the natural-language query; knowledge_type records the agent’s declared knowledge type for trajectory analysis; and source_type applies the hard source filter. Only source_type restricts the retrieval pool:

retrieve_tool_schema
{
  "name": "retrieve",
  "parameters": {
    "query": {
      "type": "string",
      "required": true
    },
    "knowledge_type": {
      "type": "string",
      "enum": [
        "declarative",
        "procedural",
        "conditional",
        "metacognitive"
      ],
      "required": true
    },
    "source_type": {
      "type": "string",
      "enum": [
        "guideline",
”textbook”,
”tutorial_video”,
”case_record”
],
”required”: true
}
}
}

The submit() tool terminates the episode. Its answer parameter is task-aware and mirrors the direct-answer schema used in C0:

submit_tool_schema
{
  "name": "submit",
  "parameters": {
    "answer": <task_specific_answer_schema>
  }
}

Observation schema. Every observation contains the base fields goal, available_tools, chat_messages, step, and elapsed_time. The surgical environment extends this with multimodal and configuration fields:

observation_schema
{
  "goal": "<task instruction>",
  "available_tools": [<serialized ToolSchema objects>],
  "chat_messages": [<conversation history>],
  "step": 0,
  "elapsed_time": 0.0,
  "media": [
    {
      "kind": "image",
      "role": "primary",
      "path": "<frame_path>"
    }
  ],
  "context": {
    "<task_metadata_key>": "<task_metadata_value>"
  },
  "system_config": {
    "condition": "C1",
    "visual_config": "V1",
    "tools": ["perceive", "submit"]
  },
  "artifacts": [
    {
      "artifact_id": "<artifact_id>",
      "kind": "perceive_overlay",
      "step": 0,
      "annotated_image": "<overlay_path>"
    }
  ]
}

The media field contains the visual inputs exposed to the model. The context field stores task metadata after removing internal-only keys. The system_config field records the active condition and visual configuration. The artifacts field accumulates tool-generated outputs, such as perception overlays, so that they remain visible in later turns.

Execution constraints and logging. Tool availability depends on the experimental condition. In C0, no tools are exposed and the model emits a direct answer. In C1, the available tools are perceive() and submit(). In C2 and higher knowledge conditions, retrieve() is also available.

Invalid calls do not terminate the episode; instead, the environment returns a success=False observation with a guidance message explaining the violated constraint and listing the remaining valid options. This design keeps tool-use drift visible in the trajectory. The harness enforces three runtime constraints. First, perceive() is capped at two calls for single-image tasks and three calls for multi-frame tasks. Duplicate calls with the same (frame, granularity) pair are rejected because perceive() is deterministic under fixed inputs. Second, retrieve() is capped at two calls per source_type per episode, yielding an implicit maximum of eight retrieval calls across the four evidence families. A repeated query to the same source family is rejected when its dense-embedding cosine similarity to an earlier query is at least 0.95, encouraging the agent to rephrase substantively or query a complementary source. Third, when fewer than two steps remain before episode truncation, the environment restricts available_tools to submit(), forcing the agent to commit an answer. These constraints are applied identically to proprietary API models, open-source MLLMs, and medical-domain models. They prevent unbounded looping over deterministic tools, encourage source diversification, and ensure that every episode ends with a scoreable answer when possible. The full observation history—including valid and invalid tool calls, call arguments, returned evidence, visual-grounding outputs, intermediate model messages, and final answers—is stored as the agent trajectory. Thus, evaluation reflects both final-answer correctness and the agent’s behavior under a stable, enforced tool contract.

Each timestep is stored as a StepRecord:

step_record_schema
{
  "step": 0,
  "reasoning": "<optional reasoning>",
  "action": {
    "action": "<tool_name>",
    "params": {}
  },
  "direct_answer": null,
  "raw_request": [<model request messages>],
  "raw_response": "<raw model output>",
  "reward": 0.0,
  "terminated": false,
  "truncated": false,
  "tool_success": true,
  "validation_errors": [],
  "validation_warnings": [],
  "elapsed": 0.0,
  "error": ""
}

A complete rollout is stored as an EpisodeRecord:

episode_record_schema
{
  "task_id": "<task_id>",
  "idx": 0,
  "reward": 1.0,
  "terminated": true,
  "truncated": false,
  "steps": 2,
  "elapsed": 3.14,
  "trajectory": [<StepRecord objects>],
  "chat": [<conversation messages>],
  "metadata": {
    "agent_config": {},
    "env_config": {},
    "artifact_registry": {},
``
"ground_truth": "<ground_truth>",
"prediction": "<prediction>"
}

This contract-based schema keeps the benchmark interface consistent while allowing each backend to use its strongest available structured-output mechanism: prompt-only JSON, OpenAI JSON schema, vLLM guided decoding, Anthropic tool use, or model-native grounding formats.

Parse validity. Table 6 compares initial parse-error rates with those after re-run rescue for each model. This audit tracks compliance with the structured-output contract separately from correctness on the surgical task.

ModelTruncation (%) ↓Parameter mismatch (%) ↓
Proprietary (API, unconstrained decoding)
GPT-4o-mini9.7 → 0.00.0
Claude-Haiku-4.50.1 → 0.04.3 → 0.1
Open-source (constrained decoding)
InternVL2.5-8B0.00.0
Qwen2.5-VL-7B0.00.0
Qwen3-VL-8B0.00.0
Qwen3-VL-8B-Thinking0.00.0
Pixtral-12B0.00.0
InternVL3-14B0.00.0
InternVL2.5-26B0.00.0
Gemma-3-27B0.00.0
Qwen2.5-VL-32B0.00.0
Qwen3-VL-32B0.00.0
Qwen2.5-VL-72B0.00.0
InternVL2.5-78B0.00.0
Medical-domain (constrained decoding)
Lingshu-32B0.00.0
MedGemma-27B0.00.0

Table 6. Per-model truncation and parameter-mismatch rates (%; lower is better). Arrows indicate initial rates → rates after re-run rescue. Single values are unchanged.

Prompt engineering

We provide full task templates in Appendix D.2.1.

Task prompt templates

We report one canonical template per task type (deduplicated across datasets). Placeholders such as {labels}, {instruments}, {verbs}, and {targets} are populated from dataset-specific label files.

phase_recognition

This frame was captured during {surgery_type}. What is the current surgical phase shown in this frame? Reply with exactly one label copied verbatim from the following list:
{labels}
Use the label text exactly as written. Do not wrap it in JSON, quotes, or any extra punctuation.
Example answer: preparation
phase_recognition

This frame was captured during {surgery_type}. What is the current surgical step shown in this frame? Reply with exactly one label copied verbatim from the following list:
{labels}

Use the label text exactly as written. Do not wrap it in JSON, quotes, or any extra punctuation.

Example answer: nasal corridor creation

tool_detection

This frame was captured during {surgery_type}. Identify every distinct surgical instrument type visible in this frame. Multiple visible instances may belong to the same type, but list each visible type only once. Choose labels only from the following list: {labels} Use the label text exactly as written.

Respond with a JSON array of label strings, for example: ["grasper", "hook"]

If no instrument is visible in the frame, return an empty array \[\]. Do not write the word "none" or any other placeholder — use \[\].

triplet_detection

This frame was captured during {surgery_type}. Identify every distinct surgical action triplet visible in this frame. Each action triplet describes one surgical interaction: which instrument is acting, what action it performs, and what target it acts on. Each triplet must use one instrument, one verb, and one target from the label lists below. A frame may contain zero, one, or multiple distinct triplets. If no surgical interaction is visible in the frame, return an empty list \[\] — do not write "none" or any placeholder.

Label space: Instruments:{instruments} Verbs:{verbs} Targets:{targets}

Use the label text exactly as written for instrument, verb, and target. List each distinct triplet only once, even if it appears multiple times in the frame.

Respond with a JSON array of triplet strings. Each string uses the format "<instrument, verb, target>" with angle brackets, comma-separated, in this exact order. Example (two triplets): ["<grasper, retract, gallbladder>", "<grasper, grasp, peritoneum>"] Example (no triplet visible):[]

action_recognition

This frame was captured during {surgery_type}. Identify every distinct surgical action visible in this frame. One or multiple actions may apply. Choose labels only from the following list: {labels} Use the label text exactly as written.

Respond with a JSON array of label strings, for example: ["grasp", "hold"]

If no action is visible in the frame, return an empty array \[\]. Do not write the word "none" or any other placeholder — use \[\].

cvs_assessment

You are given one or more images in time order from the same interval during {surgery_type}. Assess Strasberg's Critical View of Safety (CVS) for this interval using all provided images jointly.

Key mapping:

  • c1 = two structures

  • c2 = cystic plate

  • c3 = hepatocystic triangle

Score each criterion independently using this rubric:

c1 (two structures) 2: Exactly two structures can immediately and clearly be seen connecting to the gallbladder. 1: Two structures are visible, but overlap or technical issues reduce clarity. 0: Two separate structures cannot be confirmed, or more or fewer than two structures are visible.

c2 (cystic plate) 2: The cystic plate is clearly visible to approximately its lower one third. 1: The cystic plate is visible but incompletely exposed or not optimally seen. 0: The cystic plate is not visible, or more dissection is needed.

c3 (hepatocystic triangle) 2: The hepatocystic triangle is cleared of tissue so the cystic structures are clearly visible and no other structures remain in the triangle. 1: Much of the triangle is visible, but incomplete dissection or technical issues reduce certainty. 0: Tissue or technical issues obscure the triangle so clearance cannot be confirmed.

Allowed integer values: 0, 1, 2

Rules:

  • Use all provided images together to judge the interval.

  • Base the scores only on visible evidence in the provided interval.

  • If visibility is insufficient, use the lower score rather than infer unseen anatomy.

  • Do not output a total score, a separate CVS yes/no flag, explanations, markdown, or extra keys.

Use exactly these integer score keys: c1, c2, c3. Each value must be one of the allowed integer scores above. Assign each score independently from the current interval; any example numbers shown elsewhere are illustrative only and should not be copied.

Respond with a JSON object: {"c1": <int>, "c2": <int>, "c3": <int>}

skill_assessment

This task is surgical skill assessment over the provided full-procedure temporal interval from {surgery_type}. Assess the entire interval on the following five criteria, using an integer from 1 to 5 for each:

  • depth_perception: judgment of spatial relationships and distances

  • bimanual_dexterity: coordination of both hands and instruments

  • efficiency: economy and fluency of motion and workflow

  • tissue_handling: care and appropriateness when manipulating tissue

  • case_difficulty: overall technical difficulty of the case or interval

Use exactly these score keys: depth_perception, bimanual_dexterity, efficiency, tissue_handling, case_difficulty. Each value must be an integer from 1 to 5. Assign each score independently from the current interval; any example numbers shown elsewhere are illustrative only and should not be copied. Do not add a total score, explanations, markdown, or extra keys.

Respond with a JSON object: {"depth_perception": <int>, "bimanual_dexterity": <int>, "efficiency": <int>, "tissue_handling": <int>, "case_difficulty": <int>}

tool_localization (unified task, model-specific output)

Task intent (shared across models). Exactly one surgical instrument is visible in the frame; the model must output one instrument label and one bounding box.

Output contracts by model family:

Qwen-family (Qwen2.5 / Qwen3)

Use this exact format: [{"bbox_2d":[x1,y1,x2,y2],"label":"..." }]

Claude-family

Use this exact format: [{"bbox_2d":[x1,y1,x2,y2],"label":"..." }]

InternVL-family

Use this exact format: <ref>instrument_type</ref><box>[[x1,y1,x2,y2]]</box>

Other models (no dedicated grounding contract)

Use this exact format: [{"instrument":"...","x1":0.12,"y1":0.34,"w":0.20,"h":0.15}]

(normalized xywh in [0,1], default template path)

Although tool_localization is a single task, we use model-specific output contracts to match each model family’s native grounding behavior. Qwen and Claude variants use a JSON array with bbox_2d and label, while InternVL uses tag-based <ref>/<box> output. Models without a dedicated grounding contract follow the default normalized xywh schema (instrument, x1, y1, w, h). This design keeps prompting stable for each model family and lets the evaluator normalize coordinates into a unified benchmark space.

Tool specifications

Tool inventory. In tool-calling mode, the agent operates over three tools: perceive(), retrieve(), and submit(). At runtime, the environment may dynamically narrow the available tool set (e.g., force-submit window). We describe the perceive() tool for single-frame tasks as follows:

perceive() (single-frame)

Name: perceive

Purpose:
Generate query-free candidate regions for the current frame.

Core output:
- proposal list (bbox/mask/area_ratio/score/proposal_id)
- optional overlay image with numbered proposals
(attached in the same observation when proposal_count > 0)

Output boundary:
- perceive returns geometric proposals only.
- perceive does NOT return semantic labels
(no instrument/verb/target/anatomy/phase names).

Behavior constraints:
- deterministic for identical (frame, granularity)
- repeating the same (frame, granularity) is redundant
- use granularity="coarse" for compact evidence
- use granularity="fine" for denser coverage
- HARD BUDGET: call perceive AT MOST 2 times in single-frame tasks

Typical good trajectories:
- perceive(frame="current", granularity="coarse") -> submit(...)
- perceive(frame="current", granularity="coarse")
-> perceive(frame="current", granularity="fine")
-> submit(...)

Typical bad trajectories:
- perceive(frame="current", granularity="coarse")
-> perceive(frame="current", granularity="coarse")
- perceive(frame="current", granularity="fine")
-> perceive(frame="current", granularity="fine")

We describe the perceive() tool for multi-frame tasks as follows:

perceive() (multi-frame)

Name: perceive

Purpose:
Generate query-free candidate regions for one selected frame per call.

Frame argument:
- frame is an integer index aligned to the visible window: 0..N-1
- frame 0 is the first visible frame, frame N-1 is the last

Core output:
- proposal list (bbox/mask/area_ratio/score/proposal_id)
- optional overlay image with numbered proposals
(attached in the same observation when proposal_count > 0)

Output boundary:
- perceive returns geometric proposals only.
- perceive does NOT return semantic labels
(no instrument/verb/target/anatomy/phase names).

Behavior constraints:
- deterministic for identical (frame, granularity)
- never repeat the same (frame, granularity) pair
- prefer sparse sampling (start/mid/end) over dense sequential scanning
- HARD BUDGET: call perceive AT MOST 3 times in multi-frame tasks

Typical good trajectories:
- perceive(frame=N//2, granularity="coarse") -> submit(...)
- perceive(frame=0, granularity="coarse")
-> perceive(frame=N//2, granularity="coarse")
-> perceive(frame=N-1, granularity="coarse")
-> submit(...)
- perceive(frame=N//2, granularity="coarse")
-> perceive(frame=N//2, granularity="fine")
-> submit(...)

Typical bad trajectories:
- perceive(frame=5, granularity="coarse")
-> perceive(frame=5, granularity="coarse")
- perceive(frame=5, granularity="fine")
-> perceive(frame=5, granularity="fine")
- perceive(frame=0, granularity="coarse")
-> perceive(frame=1, granularity="coarse")
-> perceive(frame=2, granularity="coarse")
-> perceive(frame=3, granularity="coarse")

We describe the retrieve tool for single-frame tasks as follows:

retrieve

Name: retrieve

Purpose:
Retrieve one best-matching knowledge chunk from the surgical KB
to ground reasoning (definitions, criteria, workflow, examples).

Required arguments:
- query: natural-language knowledge need
- knowledge_type: one of
{declarative, procedural, conditional, metacognitive}
(logged for analysis; does NOT filter retrieval)
- source_type: one of
{guideline, textbook, tutorial_video, case_record}
(hard-restricts which source is searched)

Core output:
- one best-matching chunk with provenance + relevance score
- if source_type in {tutorial_video, case_record},
sampled frame images are attached in the same observation

Behavior constraints:
- budget: AT MOST 2 calls per source_type
- near-paraphrase repeated queries under the same source_type are rejected
(similarity threshold = 0.95)
- prefer complementary source_type choices across calls
(e.g., textbook + guideline + case_record)

Usage guidance:
- call retrieve when definitions/criteria/prototypical patterns are uncertain
- skip retrieve when visual evidence is already sufficient

System prompt assembly

We include system prompt details as follows:

system prompt composition (setting-dependent)

A) Base prompt (no-tool mode)

You are a helpful AI assistant. Answer the task directly.

B) Base prompt (tool-calling mode)

You are a research-style reasoning agent operating in a tool-using environment. You do NOT have specialized surgical or anatomical knowledge — you have general reasoning ability plus tools that can supply the visual evidence and domain knowledge you lack. Your goal is to answer the current question as accurately and reliably as possible by planning deliberately around what you don't know rather than performing expertise you don't have. Use the available tools to gather relevant evidence when the answer is uncertain, incomplete, hard to localize, or requires verification of a domain criterion or definition. Prefer tool use over unsupported guessing when key evidence is missing. Avoid redundant tool calls when earlier tool results already provide sufficient evidence. At the beginning of an episode, if the answer is not directly obvious from the observation, prefer gathering evidence with the available tools before submitting. Do not invent tool names, parameters, observations, or evidence. Only choose actions from the provided tool list. Return exactly one action at a time. If more evidence is needed, call an appropriate tool; if the answer is sufficiently supported, return the final answer. Task- and tool-specific usage patterns (good / bad trajectory examples) are provided inside each tool's description — those examples adapt to whether the current task is single-image or multi-frame, so follow them for the concrete call shape.
{Perceive Description} / {Retrieve Description}

SurgWiki

SurgWiki is built through a four-stage pipeline, illustrated in Figure 5; the figure separates curation into parsing/linking and chunking/filtering. Sources are first assembled from open-access repositories and licensed reference materials across four provenance-tracked families (Stage 1: Source Collection). Collected material is then processed through a unified multimodal pipeline that parses source documents, removes low-signal content, and decomposes them into retrievable knowledge units (Stage 2: Curation and Ingestion). The resulting units are serialised under a versioned schema and indexed for hybrid lexical and semantic search (Stage 3: Storage and Indexing). At query time, agents access the corpus through a deterministic retrieval backend that combines sparse, dense, and cross-modal signals (Stage 4: Retrieval and Search). Each stage is described in turn below; the agent-facing tool contract that consumes the retrieval backend is specified in § 3.2.

Figure 5. Fig. 5: SURGWIKI knowledge base construction pipeline. (A) Source Collection assembles provenance-tracked corpora from reference texts, clinical guidelines, tutorial videos, and case records. (B) Parsing & Linking uses MinerU [53] to extract structured text, figures, and tables, linking visual elements to captions and in-text mentions. (C) Chunking & Filtering applies quality filtering and heading-aware segmentation. (D) Encoding & Indexing embeds knowledge units with jina-embeddings-v4 [54] and stores them in dense Qdrant4 and sparse BM25 [55] indices. (E) Retrieval & Search combines BM25 and dense recall with RRF fusion [56], followed by text-only reranking [57] over textual candidates and multimodal text surrogates, to return top-kk evidence with source attribution.

SURGWIKI Source

We assembled SURGWIKI from four provenance-tracked source families: surgical textbooks, clinical guidelines, narrated tutorial videos, and annotated case records. Sources were selected to provide complementary declarative, procedural, and visually grounded surgical knowledge [27, 28] while prioritising reproducible provenance, stable open accessibility, and coverage of the procedures and task families evaluated in SURGKNOWBENCH.

The text corpus incorporates major surgical textbooks and operative references, including Bailey & Love [58], Gray’s Anatomy [59], and the Vanderbilt Open Manual of Surgery [60]; clinical guidelines from SAGES [61], WSES [62], and TG18 [63]; and StatPearls articles [64]. The multimodal corpus adds SurgLaVi narrated tutorial-video clips [65] and case records sampled from training annotations in public surgical-video datasets, including CholecT50 [1], Cholec80 [5], Cholec80-CVS [8], HeiChole [2], PitVis [13], Cataract-101 [14], CATARACTS [15], GraSP [12], AutoLaparo [10], HeiCo [11], and MultiBypass140 [9].

SURGWIKI curation and ingestion

All collected source material is processed through a unified multimodal ingestion pipeline that converts heterogeneous source formats into structured, retrievable knowledge units. The pipeline comprises five sequential stages: parsing and layout extraction, multimodal element handling, content filtering, heading-aware chunking, and dense embedding.

Surgical textbooks and clinical guidelines. Document-based sources—surgical textbooks, the Open Manual of Surgery, StatPearls articles, and clinical guidelines—are parsed using MinerU [53], a layout-aware PDF extraction system. The heterogeneous typographic structure of surgical documents, which routinely combines dense prose, hierarchical section trees, numbered clinical recommendations, data tables, and annotated figures, motivates this choice over plain-text extraction. MinerU produces a structured element sequence comprising paragraphs, headings, lists, tables, and figures; this structure is preserved throughout the pipeline to retain the semantic context in which surgical concepts, procedural descriptions, and clinical criteria are originally presented.

Tables and figures are treated as first-class multimodal elements. For each table, we retain both a Markdown serialisation for text-based retrieval and the corresponding image crop for visual retrieval, since layout-dependent information—column alignment, merged cells, highlighting—may be lost during serialisation. Figures are extracted as individual image items and linked to their captions.

Beyond caption linkage, we identify all in-text cross-references to each table or figure and store the surrounding passage as a supplementary textual anchor. This two-tier association connects every non-textual element not only with its caption but also with the broader discussion in which it is introduced, referenced, or interpreted, reducing retrieval fragmentation at query time.

Tutorial videos. SurgLaVi captions define the tutorial segments, so no separate ASR or scene-boundary detector is used. We filter very short or implausibly long spans and extract representative frames at 1 FPS. Each chunk stores the transcript, procedure, time span, and frame_ids. Retrieval is performed over the transcript; frames are attached to the returned chunk for inspection.

Case records. Case-record ingestion reads only training-split annotations. Task-specific renderers convert labels and rubric scores into short natural-language statements. For frame-level tasks, consecutive frames with the same normalized answer are collapsed into one interval; interval-level assessment rows are kept as-is. Each unit stores the text statement, image reference, timestamps, frame IDs, dataset, task, procedure or phase hints, and labels. Retrieval is performed over the text statement; frames are attached to the returned record for inspection. Missing media, empty text, and implausibly long intervals are discarded, and evaluation labels are never read.

Filtering. Prior to chunking, low-signal passages are identified and removed. Filtered material includes reference lists, author affiliations, journal and publisher metadata, copyright notices, running headers and footers, and watermark artefacts introduced during PDF rendering. This step prevents retrieval models from surfacing uninformative bibliographic fragments in response to clinical queries.

Chunking. The filtered element sequence is partitioned into retrievable text units using a hierarchical, heading-aware strategy. This design follows prior work identifying document segmentation as a key preprocessing decision in retrieval-augmented generation [66]. Each chunk targets approximately 512 tokens and inherits the nearest ancestor heading as its section label, so that every indexed unit retains local topical context. Chunk boundaries are constrained by the detected document hierarchy, preventing chunks from spanning major section transitions. This yields retrieval units that remain compact for semantic search while preserving section-level coherence.

Embedding. Textual chunks are encoded with jina-embeddings-v4 [54] into a 2,048-dimensional dense representation. Non-textual elements, including tables, figures, and tutorial-video frames, are encoded with the corresponding vision encoder from the same model family. This provides a shared representation space for textual and visual knowledge units and allows multimodal elements to be retrieved without introducing an additional cross-modal alignment model. Each embedding is stored together with provenance metadata. For document sources, this includes the publication identity, section path, and page reference; for tutorial clips, the video identifier and temporal span; and for case records, the dataset identity, task type, split designation, and annotation labels. When a source contributes both SurgKnowBench evaluation episodes and SurgWiki case records, only training-side annotations are included in SurgWiki. Gold labels from evaluation episodes are excluded from the retrieval corpus to prevent label leakage.

Embedder selection. We selected jina-embeddings-v4 through a controlled ablation against eight alternative dense encoders. The comparison included general-purpose text retrievers, namely BGE-base-en-v1.5 and BGE-large-en-v1.5 [67], E5-large-v2 [68], GTE-large [69], Nomic-embed-text-v1.5 [70], and all-mpnet-base-v2 from the Sentence-Transformers library [71, 72]; one biomedical-domain retriever, MedCPT [73]; and the earlier Jina text-embedding model, jina-embeddings-v3 [74].

To isolate the effect of the embedding model, all rows in the ablation use the same document index, chunking strategy, retrieval backend, and evaluation queries. Probe queries and gold chunk identifiers were generated once with gpt-4o [75] over the indexed corpus and then held fixed across all encoders. For each model, the same encoder is used at indexing time and query time. We evaluate retrieval quality using Recall@k, mean reciprocal rank, and NDCG@10 over 1,515 probe queries against the merged document index. The full results are reported in Table 7.

ModelR@1↑\uparrowR@5↑\uparrowR@10↑\uparrowMRR↑\uparrowNDCG@10↑\uparrowTime (s)↓\downarrowDim
General-purpose dense encoders
BGE-base-en-v1.5 [67]0.4480.6960.7710.5530.605363.6768
BGE-large-en-v1.5 [67]0.4750.7060.7700.5740.622404.81024
E5-large-v2 [68]0.4480.6890.7570.5510.601947.41024
GTE-large [69]0.4600.6950.7610.5580.6071,292.91024
Nomic-embed-text-v1.5 [70]0.4320.6680.7410.5300.581370.8768
all-mpnet-base-v2 [71, 72]0.3710.5840.6620.4590.507354.0768
Biomedical-domain encoder
MedCPT [73]0.2150.4490.5630.3130.372368.0768
Jina encoders (v3: text-only; v4: multimodal)
jina-embeddings-v3 [74]0.3610.6210.7200.4720.5311,372.11024
jina-embeddings-v4 [54]0.4610.7150.7850.5690.6211,334.02048

Table 7. Embedding-model ablation on the SURGWIKI document index. All models are evaluated with the same chunked corpus, probe queries, gold chunk identifiers, and dense-retrieval pipeline; only the encoder used for indexing and querying is varied. Probe queries (n=1,515n = 1,515) and gold chunk identifiers were generated once with gpt-4o [75]. Time denotes total wall-clock time for query embedding and ranking; Dim denotes embedding dimensionality. Bold and underlining indicate the best and second-best values per column.

Although the strongest text-only baseline performs competitively on strict top-rank metrics, jina-embeddings-v4 provides the best overall fit for SurgWiki because retrieval must operate over both textual and visual knowledge units. In contrast to the text-only baselines, jina-embeddings-v4 supports both text and image encoding within the same model family, which is required for retrieving tables, figures, and tutorial-video frames through a unified interface. Using a single multimodal encoder also avoids modality-specific embedding spaces and removes the need for an additional alignment stage before rank fusion. We therefore adopt jina-embeddings-v4 as the production embedder. Document embeddings are precomputed during index construction; query encoding is performed at retrieval time. The timing in Table 7 includes query embedding and ranking.

SURGWIKI storage and indexing

Knowledge unit schema. Each unit produced by the ingestion pipeline is represented as a structured knowledge unit with standardized content, modality, and provenance fields. Textual units contain the chunk text, a unique unit identifier, source identifier, source type, document type, ancestor heading path, and page reference where available. Visual units additionally store the relative image path and the visual element type, such as figure, table, or tutorial frame. Case-record units store the dataset identifier, task type, split designation, and annotation label set. All units are serialized as JSONL, with one record per line, enabling offline inspection, versioned corpus snapshots, and ablation studies over source families, modalities, and metadata fields.

Index structure. We build complementary dense and sparse indices over the stored knowledge units. The dense index stores embeddings produced during the embedding stage and is implemented using Qdrant4. Text chunks, figure crops, and table crops are stored in separate modality-specific collections, allowing retrieval to be filtered by modality when required. Each collection uses 2,048-dimensional embeddings produced by jina-embeddings-v4 [54]. In parallel, we construct a sparse lexical index over textual units using BM25 [55].

SURGWIKI retrieval and search

The retrieval backend is the agent-facing entry point into SURGWIKI. It is deterministic—identical (query, source) arguments return identical ranked lists across runs—and isolates retrieval mechanics from agent behaviour. The agent-side tool contract that consumes this backend is specified in § 3.2; the present section describes the index-side mechanics that produce the ranked list.

For textual knowledge items, the knowledge search backend is based on a fusion of lexical retrieval (BM25) [55] and dense semantic retrieval with the text-embedding model jina-embeddings-v4 [54]. Indexing and retrieval are implemented with Qdrant4. For multimodal knowledge (e.g., tables, figures, videos) where visual representations are available, text embeddings serve as the anchor representation, since they still generalize to domain knowledge. Visual embeddings contribute only an additional dense retrieval signal due to a lack of domain fine-tuning. The multimodal ranking is fused with Reciprocal Rank Fusion [56]. A cross-encoder reranker then rescores the top 30 fused candidates, and the highest-ranked chunk is returned with source attribution. For tutorial-video and case-record results, representative frames are returned alongside the retrieved text, allowing the agent to inspect visual evidence in the same tool observation.

Hybrid retrieval. For a textual query, the backend executes lexical and dense passes in parallel. The lexical pass scores knowledge units with BM25 [55] over the sparse index, surfacing literal matches on instrument names, anatomical landmarks, rubric terms, and standardised guideline phrasing. The dense pass encodes the query with jina-embeddings-v4 [54] into the same 2,048-dimensional space as the indexed text chunks and ranks units by cosine similarity over the Qdrant⁴ dense collection, recovering semantically related passages whose surface form differs from the query. The two passes are complementary: BM25 anchors retrieval to surface vocabulary, while dense retrieval generalises across paraphrases and synonymous surgical terminology. Their ranked lists are combined into a single hybrid ranking, and cross-modal results from the visual collections are integrated through Reciprocal Rank Fusion [56] as described below.

Multimodal retrieval. For sources whose units carry both textual and visual representations—document tables, figures, and tutorial-video frames—textual embeddings serve as the anchor signal. Visual-only retrieval over surgical imagery is unreliable with general-purpose multimodal encoders, since jina-embeddings-v4 has not been domain-adapted to laparoscopic, microscopic, or open-surgery scenes. Visual embeddings therefore contribute an additional dense signal that is fused with the lexical and textual-dense passes via Reciprocal Rank Fusion [56], ensuring that visually salient items remain reachable when their textual description is sparse without letting an unreliable visual ranking dominate the result. When a multimodal unit is returned, representative frames or image crops are attached to the same observation alongside the retrieved text, so the agent can inspect visual evidence in the same tool response.

Cross-encoder reranking. The fused ranked list is rescored with a text-only cross-encoder reranker [76], implemented with jina-reranker-v2-base-multilingual [57]. Reranking is applied to textual candidates and textual surrogates of multimodal candidates: text chunks are represented by their body text, tables by their Markdown serialization, and figures by their captions, in-text mentions, and surrounding context. Image embeddings contribute to the initial dense retrieval stage, but the reranker does not directly consume image pixels or visual embeddings. This separates multimodal candidate generation from text-based relevance refinement and avoids introducing an additional image-aware reranking model. Reranking is applied only within a bounded prefetch window over the fused candidates, and the rescored top-kk items are returned with full source attribution.

Source-level filtering. Each query is dispatched only to the units associated with the agent-selected source family, enforced as a hard filter before the lexical and dense passes are executed. This keeps the retrieval pool aligned with the agent’s epistemic decision (where to search) and prevents implicit cross-source mixing in the ranked list. Source filtering is also the unit at which the per-episode call budget is enforced (§ 3.2).

Retrieval quality estimate

Reporting retrieval precision is nontrivial in our setting since no gold relevance labels exist for agent-generated queries. Relevance and precision require human judgment. We therefore evaluate the retrieve() tool on real trajectories with a TREC-style graded relevance protocol:

  • A designed rubric covering (i) topical match, (ii) constraint satisfaction, and (iii) information coverage, assessing the query-item relevance;

  • Two surgeon experts annotate 50 retrieve() calls sampled from pooled real trajectories as a calibration set; GPT-5 as an LLM judge scores the full pooled sample with the same rubric;

  • We report expert–expert and expert–LLM agreement. On graded relevance, expert–expert linearly-weighted κ\kappa as 0.460; expert–LLM κ\kappa = 0.574/0.539. The LLM judge agrees with each expert at least as well as the experts agree with each other.

retrieve() returns a self-calculated relevance score with every item, passed to the agent as part of its observation. This score separates relevant from non-relevant evidence annotated at AUROC = 0.894 against the LLM judge and 0.817 against human expert, and holds within every knowledge source (0.68–0.87).

Regarding the precision, we use relevance as a proxy. Since retrieval quality and precision are also determined by agent queries, precision@1 in our real trajectories can go up to 57%–82%.

Tool implementation

Because laparoscopic images differ from natural-image scenes in illumination, texture, and object scale, we relax two SAM 2 confidence thresholds: pred_iou_thresh is lowered from 0.80 to 0.70, and stability_score_thresh is lowered from 0.95 to 0.90.

perceive() proposal filtering and ranking. Each proposal returned by perceive() carries a numeric identifier, a confidence score, a binary mask, a bounding box, a pixel area, and an image-normalized area ratio. perceive() supports two granularities. In coarse mode, a proposal is retained only if its area ratio lies in [0.5%,75%][0.5\%, 75\%], its bounding box covers less than 60% of the image, and it does not touch two or more image edges with at least 95% extent. These geometric filters remove whole-frame envelopes and small noise fragments. Surviving proposals are ranked by score×area_ratioscore \times area\_ratio with raw score, area ratio, and pixel area used as successive tie-breakers; the top 10 proposals are returned. In fine mode, only the lower area-ratio bound is enforced, with threshold 0.05%. No edge-touching or bounding-box coverage filter is applied. Fine-mode proposals are ranked by (score,area_ratio)(score, area\_ratio), and the top 30 proposals are returned. If filtering removes all proposals in either mode, the tool falls back to ranking the unfiltered SAM 2 output.

Extended results

Stage I: zero-shot diagnostic evaluation

This section expands the Stage I diagnostic results summarized in § 2. Fig. 6 compares selected dataset–task scores with random and majority baselines.

Stage I performance on dataset–task cells with random and majority baselines: tool detection and phase recognition

Figure 6. Stage I performance on dataset–task cells with random and majority baselines: tool detection and phase recognition.

Four-panel bar charts showing Stage I performance for Triplet Detection on CholecT50 with Macro-F1↑, Tool Localization on CholecT50 with F1@0.5↑, Action Recognition on HeiCo with Macro-F1↑, and CVS Assessment on Cholec80-CVS with Acc↑. The legend lists GPT-4o-mini, Claude-Haiku-4.5, InternVL2.5-8B, Qwen2.5-VL-7B-Instruct, Qwen3-VL-8B-Instruct, Qwen3-VL-8B-Thinking, Pixtral-12B, InternVL3-14B, InternVL2.5-26B, Gemma3-27B, Qwen2.5-VL-32B-Instruct, Qwen3-VL-32B-Instruct, Qwen2.5-VL-72B-Instruct, InternVL2.5-78B, HuatuoGPT-Vision-34B, Lingshu-32B, and MedGemma-27B; baselines are Random and Majority class.

Figure 6 (PDF p. 45). Stage I performance on dataset–task cells with random and majority baselines (continued): triplet detection, tool localization, action recognition, and CVS assessment.

Tool detection. To assess whether instrument recognition is consistent across surgical datasets, Table 8 separates the Stage I scores by dataset. The random and majority baselines help distinguish useful recognition from predictions that exploit the label distribution.

Tab. 8: SURGKNOWBENCH Stage I tool-detection per-dataset breakdown. We report macro-F1 scores (%) across the five tool-detection datasets; the main-table value is the unweighted (macro-) average across these columns. Bold / <u>underline</u> mark the best / second-best per column among models; the Random and Majority rows are uninformed reference baselines (see main-table caption).

ModelCholecT50PitVisCataractsAutoLaparoGraSP
macro-F1↑macro-F1↑macro-F1↑macro-F1↑macro-F1↑
Proprietary
GPT-4o-mini25.94.711.237.334.8
Claude-Haiku-4.524.55.27.531.522.5
Open-source (<10B)
InternVL2.5-8B24.45.06.014.72.0
Qwen2.5-VL-7B-Instruct25.44.04.521.819.5
Qwen3-VL-8B-Instruct<u>31.4</u>4.59.934.316.3
Qwen3-VL-8B-Thinking22.5<u>5.4</u>6.433.4<u>42.9</u>
Open-source (10–30B)
Pixtral-12B-240926.04.66.029.817.3
InternVL3-14B20.74.55.027.615.7
InternVL2.5-26B19.02.63.539.88.1
Gemma-3-27B-it29.65.18.639.624.6
Open-source (>30B)
Qwen2.5-VL-32B-Instruct25.25.28.029.622.1
Qwen3-VL-32B-Instruct28.95.4<u>10.2</u>48.137.9
Qwen2.5-VL-72B-Instruct31.13.5<u>10.2</u>24.821.7
InternVL2.5-78B33.83.45.8<u>41.6</u>28.4
Medical-domain
HuatuoGPT-Vision-34B-hf11.95.28.434.845.1
Lingshu-32B25.24.47.032.920.0
MedGemma-27B-it26.35.57.532.819.9
Statistical Baseline
Random13.84.14.323.316.9
Majority20.12.53.017.824.2

Table 8.

Triplet detection. Recognizing an instrument or an action individually does not establish that a model can identify their interaction. Table 9 compares full-triplet F1 with the instrument, verb, and target components to make this distinction explicit.

Tab. 9: SURGKNOWBENCH Stage I triplet-detection breakdown on CholecT50. We report macro-F1 (%) for the full ⟨instrument,verb,target⟩\langle\text{instrument},\text{verb},\text{target}\rangle triplet (the main-table value) and for each component evaluated as a standalone label set. Bold / <u>underline</u> mark the best / second-best per column among models; the Random and Majority rows are uninformed reference baselines (see main-table caption).

ModelTriplet macro-F1↑Instrument macro-F1↑Verb macro-F1↑Target macro-F1↑
Proprietary
GPT-4o-mini1.613.59.48.7
Claude-Haiku-4.52.924.718.18.4
Open-source (<10B)
InternVL2.5-8B1.917.910.47.3
Qwen2.5-VL-7B-Instruct1.518.614.16.1
Qwen3-VL-8B-Instruct2.121.213.07.8
Qwen3-VL-8B-Thinking2.820.215.211.1
Open-source (10–30B)
Pixtral-12B-24092.311.411.25.7
InternVL3-14B2.721.414.59.8
InternVL2.5-26B1.916.99.66.1
Gemma-3-27B-it<u>3.6</u>22.516.2<u>11.4</u>
Open-source (>30B)
Qwen2.5-VL-32B-Instruct1.222.015.46.2
Qwen3-VL-32B-Instruct2.518.812.69.0
Qwen2.5-VL-72B-Instruct2.0<u>28.1</u>15.710.0
InternVL2.5-78B4.934.523.515.1
Medical-domain
HuatuoGPT-Vision-34B-hf2.013.110.79.6
Lingshu-32B2.427.4<u>20.8</u>10.9
MedGemma-27B-it2.618.312.011.4
Statistical Baseline
Random1.615.710.36.8
Majority1.210.87.16.9

Table 9.

Phase recognition. Table 10 examines phase recognition separately across surgical datasets, showing the variation hidden by the aggregate task score. Each dataset’s majority baseline provides a reference for performance obtained by predicting a frequent phase without distinguishing the current workflow state.

Tab. 10: SURGKNOWBENCH Stage I phase-recognition per-dataset breakdown. We report accuracy (%) across the seven phase-recognition datasets; the main-table value is the unweighted (macro-) average across these columns. Bold / <u>underline</u> mark the best / second-best per column among models; the Random and Majority rows are uninformed reference baselines (see main-table caption).

ModelCataract-101 Acc.↑\uparrowCholec80 Acc.↑\uparrowAutoLaparo Acc.↑\uparrowGraSP Acc.↑\uparrowHeiCo Acc.↑\uparrowPitVis Acc.↑\uparrowMultiBypass140 Acc.↑\uparrow
Proprietary
GPT-4o-mini30.040.1<u>24.1</u>12.527.07.3<u>31.9</u>
Claude-Haiku-4.519.220.617.814.925.77.922.7
Open-source (<10B)
InternVL2.5-8B30.538.920.310.117.234.514.8
Qwen2.5-VL-7B-Instruct28.713.619.43.023.34.95.7
Qwen3-VL-8B-Instruct30.921.715.810.217.013.023.6
Qwen3-VL-8B-Thinking30.636.919.28.8<u>27.4</u>11.727.7
Open-source (10–30B)
Pixtral-12B-240930.58.921.211.826.610.714.1
InternVL3-14B30.010.921.68.127.114.624.6
InternVL2.5-26B31.630.723.72.224.615.121.1
Gemma-3-27B-it23.825.918.914.523.818.125.0
Open-source (>30B)
Qwen2.5-VL-32B-Instruct31.121.620.612.726.916.524.8
Qwen3-VL-32B-Instruct21.2<u>39.6</u>26.221.327.13.726.7
Qwen2.5-VL-72B-Instruct30.239.520.216.827.39.221.9
InternVL2.5-78B<u>31.1</u>38.720.914.125.514.823.2
Medical-domain
HuatuoGPT-Vision-34B-hf30.019.222.6<u>21.0</u>29.517.733.3
Lingshu-32B30.129.220.415.626.2<u>22.4</u>28.1
MedGemma-27B-it30.223.519.613.115.213.523.0
Statistical Baseline
Random10.014.214.29.17.38.48.3
Majority30.537.425.132.924.835.223.1

Table 10. SURGKNOWBENCH Stage I phase-recognition per-dataset breakdown. We report accuracy (%) across the seven phase-recognition datasets; the main-table value is the unweighted (macro-) average across these columns. Bold / underline mark the best / second-best per column among models; the Random and Majority rows are uninformed reference baselines (see main-table caption).

CVS assessment. For ordinal safety criteria, accuracy alone can obscure predictions concentrated on a frequent rating. Table 11 therefore pairs criterion-level accuracy with chance-corrected agreement, while MAE and exact dictionary accuracy describe rating errors and joint correctness across criteria.

ModelPer-key Acc. Acc.↑κ\kappa κ\kappa ↑MAE MAE↓Dict-Acc. Acc.↑
Proprietary
GPT-4o-mini15.60.91.000.0
Claude-Haiku-4.517.0-4.81.151.0
Open-source (<10B)
InternVL2.5-8B71.20.00.3245.8
Qwen2.5-VL-7B-Instruct<u>71.2</u><u>0.0</u><u>0.32</u><u>45.8</u>
Qwen3-VL-8B-Instruct31.65.40.8912.5
Qwen3-VL-8B-Thinking58.32.70.4213.5
Open-source (10–30B)
Pixtral-12B-240971.20.00.3245.8
InternVL3-14B24.0-0.40.781.0
InternVL2.5-26B34.4-0.70.784.2
Gemma-3-27B-it37.5-3.90.632.1
Open-source (>30B)
Qwen2.5-VL-32B-Instruct24.3-0.40.792.1
Qwen3-VL-32B-Instruct31.68.20.724.2
Qwen2.5-VL-72B-Instruct47.9-2.70.5312.5
InternVL2.5-78B25.7-0.40.762.1
Medical-domain
HuatuoGPT-Vision-34B-hf14.60.11.020.0
Lingshu-32B9.4-0.21.492.1
MedGemma-27B-it19.44.01.179.4
Statistical Baseline
Random32.81.70.924.2
Majority71.20.00.3245.8

Table 11. Tab. 11: SURGKNOWBENCH Stage I CVS-assessment breakdown on Cholec80-CVS (3-key dict, ordinal labels in {0,1,2}\{0, 1, 2\}). Per-key Acc. and κ\kappa (quadratic-weighted, multiplied by 100) are the main-table metrics; MAE (lower is better; range [0,2][0, 2]) and Dict-Acc. (strict whole-dictionary accuracy) are reported as diagnostics. Bold / <u>underline</u> mark the best / second-best per column among models; the Random and Majority rows are uninformed reference baselines (see main-table caption).

Skill assessment. Table 12 evaluates whether predicted skill ratings track the reference assessments. Reading MAE together with quadratic-weighted κ\kappa separates closeness to common ratings from agreement beyond chance; the majority baseline makes this distinction visible.

ModelMAE ↓\downarrowκ\kappa ↑\uparrow
Proprietary
GPT-4o-mini0.840.2
Claude-Haiku-4.50.99<u>10.9</u>
Open-source (<10B)
InternVL2.5-8B0.86-7.2
Qwen2.5-VL-7B-Instruct1.148.0
Qwen3-VL-8B-Instruct1.053.9
Qwen3-VL-8B-Thinking0.964.7
Open-source (10–30B)
Pixtral-12B-24091.21-0.5
InternVL3-14B0.994.0
InternVL2.5-26B0.640.0
Gemma-3-27B-it1.393.3
Open-source (>30B)
Qwen2.5-VL-32B-Instruct0.913.9
Qwen3-VL-32B-Instruct0.8114.3
Qwen2.5-VL-72B-Instruct1.087.2
InternVL2.5-78B0.920.1
Medical-domain
HuatuoGPT-Vision-34B-hf<u>0.77</u>1.7
Lingshu-32B0.800.4
MedGemma-27B-it0.921.6
Statistical Baseline
Random0.911.6
Majority0.630.0

Table 12. SURGKNOWBENCH Stage I surgical-skill assessment on HeiChole (GOALS-style 5-key rubric, 1–5 Likert). We report MAE (lower is better; range [0, 4]) and quadratic-weighted Cohen’s κ\kappa (multiplied by 100), each macro-averaged across the three sub-axes (holistic, Calot-phase, dissection-phase). Bold / <u>underline</u> mark the best / second-best per column among models; the Random and Majority rows are uninformed reference baselines (see main-table caption).

Tool localization. Identifying that an instrument is present is different from locating it accurately. Table 13 reports localization F1 alongside overlap, precision, and recall, allowing missed instruments and inaccurate or spurious boxes to be examined separately.

Model[email protected]mIoU[email protected][email protected]
Proprietary
GPT-4o-mini0.01.30.00.0
Claude-Haiku-4.51.65.81.71.6
Open-source (<10B)
InternVL2.5-8B11.313.111.711.0
Qwen2.5-VL-7B-Instruct11.412.011.811.0
Qwen3-VL-8B-Instruct13.814.814.213.4
Qwen3-VL-8B-Thinking16.316.617.015.8
Open-source (10–30B)
Pixtral-12B-24094.010.04.23.9
InternVL3-14B8.912.79.28.7
InternVL2.5-26B14.615.615.114.2
Gemma-3-27B-it0.05.10.00.0
Open-source (>30B)
Qwen2.5-VL-32B-Instruct6.59.16.76.3
Qwen3-VL-32B-Instruct14.613.915.014.2
Qwen2.5-VL-72B-Instruct11.412.711.911.0
InternVL2.5-78B22.723.323.322.1
Medical-domain
HuatuoGPT-Vision-34B-hf0.93.31.20.8
Lingshu-32B1.66.41.71.6
MedGemma-27B-it0.02.20.00.0
Statistical Baseline
Random0.00.60.00.0
Majority0.03.40.00.0

Table 13. Tab. 13: SurgKnowBench Stage I tool-localization breakdown on CholecT50. [email protected] is the main-table metric (Hungarian matching with same-class constraint, IoU⁡≥0.5\operatorname{IoU} \ge0.5); mIoU, [email protected], and [email protected] are reported as auxiliary diagnostics. All values are percentages. Bold / _underline_ mark the best / second-best per column among models; the Random and Majority rows are uninformed reference baselines (see main-table caption).

Action recognition. Table 14 complements class-balanced macro-F1 with pooled precision, recall, and micro-F1. Their comparison shows how performance depends on common versus less frequent actions, while set-exact accuracy measures whether the complete action label set is recovered.

Modelmacro-F1 macro-F1↑micro-F1 F1↑Precision Precision↑Recall Recall↑Set-Acc. Acc.↑
Proprietary
GPT-4o-mini27.151.141.965.424.9
Claude-Haiku-4.521.8<u>44.6</u><u>35.1</u><u>61.2</u>7.1
Open-source (<10B)
InternVL2.5-8B11.710.910.111.77.7
Qwen2.5-VL-7B-Instruct8.77.16.77.56.2
Qwen3-VL-8B-Instruct13.513.712.415.210.7
Qwen3-VL-8B-Thinking3.93.42.74.41.1
Open-source (10–30B)
Pixtral-12B-2409<u>23.8</u>38.229.952.89.6
InternVL3-14B14.423.821.127.46.1
InternVL2.5-26B9.810.910.011.84.5
Gemma-3-27B-it12.613.510.319.83.8
Open-source (>30B)
Qwen2.5-VL-32B-Instruct14.317.616.818.510.2
Qwen3-VL-32B-Instruct21.329.323.539.09.8
Qwen2.5-VL-72B-Instruct18.530.428.732.2<u>20.2</u>
InternVL2.5-78B11.89.98.811.48.8
Medical-domain
HuatuoGPT-Vision-34B-hf21.431.126.238.29.9
Lingshu-32B16.525.224.326.120.2
MedGemma-27B-it16.223.018.530.54.5
Statistical Baseline
Random13.523.120.426.514.9
Majority17.678.379.077.777.3

Table 14. SURGKNOWBENCH Stage I action-recognition breakdown on HeiChole (multi-label, 4 actions). The main-table value is macro-F1; we additionally report micro-F1 / precision / recall (set-level, sample-aggregated) and set-exact accuracy as diagnostics. All values are percentages. Bold / <u>underline</u> mark the best / second-best per column among models; the Random and Majority rows are uninformed reference baselines (see main-table caption).

Stage II: agentic evaluation

This section provides per-dataset and auxiliary-metric breakdowns for the Stage II Full-setting evaluation on SURGKNOWBENCH-Hard, complementing the aggregate results in Section 4. Tool detection with agent interaction. Tab. 15 breaks down instrument-recognition performance on the hard subset by dataset. These absolute scores show where the full agent setting succeeds or remains weak, beyond the aggregate across-stage changes in the main results.

ModelCholecT50PitVisCataractsAutoLaparoGraSP
Proprietary
GPT-4o-mini7.00.023.718.612.6
Claude-Haiku-4.55.30.94.44.519.8
Open-source (<10B)
InternVL2.5-8B8.54.6<u>11.7</u>9.80.0
Qwen2.5-VL-7B-Instruct3.50.03.113.00.0
Qwen3-VL-8B-Instruct4.20.04.117.411.5
Qwen3-VL-8B-Thinking1.30.91.118.724.7
Open-source (10–30B)
Pixtral-12B-2409<u>14.0</u>8.36.523.529.9
InternVL3-14B2.50.06.217.015.9
InternVL2.5-26B0.03.32.11.617.8
Gemma-3-27B-it7.90.00.021.713.2
Open-source (>30B)
Qwen2.5-VL-32B-Instruct5.72.23.117.426.2
Qwen3-VL-32B-Instruct1.32.711.25.014.8
Qwen2.5-VL-72B-Instruct1.20.78.817.62.7
InternVL2.5-78B11.40.90.09.211.1
Medical-domain
Lingshu-32B10.0<u>8.3</u>2.9<u>23.1</u>36.9
MedGemma-27B-it15.42.92.122.6<u>33.3</u>
Statistical Baseline
Random20.310.115.725.144.1
Majority14.75.923.717.141.9

Table 15. SURGKNOWBENCH-Hard Stage II tool-detection per-dataset breakdown. macro-F1 (%) across the five datasets. Bold / <u>underline</u> mark the best / second-best per column among models; the Random and Majority rows are uninformed reference baselines computed on the hard GT.

Triplet detection with agent interaction. Tab. 16 tests whether agents recover complete surgical interactions on hard episodes or only their individual components. Comparing the triplet score with instrument, verb, and target F1 helps locate the remaining compositional difficulty.

Phase recognition with agent interaction. Tab. 17 reports the full agent’s phase-recognition scores for each dataset in the hard subset. The dataset-specific baselines provide context for interpreting absolute accuracy across procedures with different phase distributions.

CVS assessment with agent interaction. Tab. 18 examines whether tool-mediated judgments agree with the reference CVS ratings on hard episodes. Criterion-level accuracy and κ\kappa are shown together so that frequent-rating predictions are not mistaken for reliable assessment; the auxiliary metrics describe the size and joint pattern of errors.

Skill assessment with agent interaction. Tab. 19 reports absolute skill-rating performance after agent interaction. MAE measures rating distance, whereas quadratic-weighted κ\kappa tests agreement beyond chance, providing complementary views of whether agents distinguish different levels of surgical skill.

Tool localization with agent interaction. Tab. 20 evaluates the spatial accuracy of the full agent on hard localization episodes. Precision and recall separate false or inaccurate proposals from missed instruments, while mIoU complements the thresholded localization F1.

ModelTriplet macro-F1↑Instrument macro-F1↑Verb macro-F1↑Target macro-F1↑
Proprietary
GPT-4o-mini0.016.511.40.0
Claude-Haiku-4.50.021.214.05.7
Open-source (<10B)
InternVL2.5-8B0.316.711.16.6
Qwen2.5-VL-7B-Instruct0.08.38.27.8
Qwen3-VL-8B-Instruct0.014.610.37.9
Qwen3-VL-8B-Thinking0.09.49.65.0
Open-source (10–30B)
Pixtral-12B-24090.019.414.012.7
InternVL3-14B2.18.25.17.9
InternVL2.5-26B1.512.24.36.3
Gemma-3-27B-it2.123.714.45.9
Open-source (>30B)
Qwen2.5-VL-32B-Instruct7.723.017.510.6
Qwen3-VL-32B-Instruct0.012.410.09.0
Qwen2.5-VL-72B-Instruct0.013.07.47.8
InternVL2.5-78B3.820.316.210.8
Medical-domain
Lingshu-32B0.019.97.87.6
MedGemma-27B-it1.010.75.97.8
Statistical Baseline
Random4.713.09.06.3
Majority3.711.87.96.6

Table 16. SURGKNOWBENCH-Hard Stage II triplet-detection breakdown on CholecT50. Macro-F1 (%) for the full triplet (the headline metric) and each component as a standalone label set. Bold / underline mark best / second-best per column among models; the Random and Majority rows are uninformed reference baselines.

ModelCataract-101 Acc.↑Cholec80 Acc.↑AutoLaparo Acc.↑GraSP Acc.↑HeiCo Acc.↑PitVis Acc.↑MultiBypass140 Acc.↑
Proprietary
GPT-4o-mini0.00.00.00.00.04.00.0
Claude-Haiku-4.56.08.02.00.00.020.08.0
Open-source (<10B)
InternVL2.5-8B0.00.06.00.00.00.00.0
Qwen2.5-VL-7B-Instruct4.04.04.00.04.00.06.0
Qwen3-VL-8B-Instruct8.02.012.00.00.04.00.0
Qwen3-VL-8B-Thinking0.02.04.00.02.04.00.0
Open-source (10–30B)
Pixtral-12B-24092.014.02.00.00.00.00.0
InternVL3-14B0.04.08.00.00.04.00.0
InternVL2.5-26B0.00.010.00.00.00.00.0
Gemma-3-27B-it0.02.06.00.06.00.08.0
Open-source (>30B)
Qwen2.5-VL-32B-Instruct4.022.012.00.04.02.08.0
Qwen3-VL-32B-Instruct4.06.010.06.00.00.010.0
Qwen2.5-VL-72B-Instruct8.04.08.00.02.00.012.0
InternVL2.5-78B6.00.02.00.02.00.00.0
Medical-domain
Lingshu-32B2.06.08.02.02.00.014.0
MedGemma-27B-it0.08.08.00.00.00.06.0
Statistical Baseline
Random19.225.617.235.612.015.615.6
Majority40.054.038.068.040.066.038.0

Table 17. SURGKNOWBENCH-Hard Stage II phase-recognition per-dataset breakdown. Accuracy (%) across the seven datasets. Bold / underline mark best / second-best per column among models; the Random and Majority rows are uninformed reference baselines.

ModelPer-key Acc. Acc.↑κ\kappa κ\kappa↑MAE MAE↓Dict-Acc. Acc.↑
Proprietary
GPT-4o-mini18.8-1.00.962.1
Claude-Haiku-4.523.32.20.872.1
Open-source (<10B)
InternVL2.5-8B36.5-4.80.863.1
Qwen2.5-VL-7B-Instruct40.6-4.30.8011.5
Qwen3-VL-8B-Instruct<u>45.8</u>3.60.6221.9
Qwen3-VL-8B-Thinking29.9-0.81.039.4
Open-source (10–30B)
Pixtral-12B-240922.9-1.60.894.2
InternVL3-14B25.70.60.752.1
InternVL2.5-26B22.90.70.972.1
Gemma-3-27B-it33.3-5.40.776.2
Open-source (>30B)
Qwen2.5-VL-32B-Instruct28.83.30.763.1
Qwen3-VL-32B-Instruct40.67.70.6113.5
Qwen2.5-VL-72B-Instruct49.7<u>4.8</u>0.5122.9
InternVL2.5-78B36.8-2.70.648.3
Medical-domain
Lingshu-32B8.01.11.411.0
MedGemma-27B-it22.6-0.21.167.3
Statistical Baseline
Random34.4-1.50.894.2
Majority71.20.00.3245.8

Table 18. SURGKNOWBENCH-Hard Stage II CVS-assessment breakdown on Cholec80-CVS (3-key dict, ordinal {0,1,2}\{0, 1, 2\}). Per-key Acc. and κ\kappa (quadratic-weighted, multiplied by 100) are the headline metrics; accuracy values are percentages; MAE (lower is better; range [0,2][0, 2]) and Dict-Acc. are diagnostics. Bold / <u>underline</u> mark best / second-best per column among models (highest for ↑, lowest for MAE); the Random and Majority rows are uninformed reference baselines.

Action recognition with agent interaction. Table 21 reports absolute action-recognition performance on the hard subset. Macro-F1 and the majority baseline contextualize the gains reported in the main text, while pooled metrics and exact-set accuracy reveal how those gains translate into complete multi-label predictions.

Modelmacro-F1 macro-F1↑micro-F1 F1↑Precision Precision↑Recall Recall↑Set-Acc. Acc.↑
Proprietary
GPT-4o-mini13.220.020.020.020.0
Claude-Haiku-4.522.826.721.236.00.0
Open-source (<10B)
InternVL2.5-8B18.030.928.334.0<u>32.0</u>
Qwen2.5-VL-7B-Instruct3.03.93.94.04.0
Qwen3-VL-8B-Instruct<u>27.3</u>50.048.152.048.0
Qwen3-VL-8B-Thinking13.5<u>10.8</u>9.812.08.0
Open-source (10–30B)
Pixtral-12B-240927.646.635.468.04.0
InternVL3-14B4.25.44.96.00.0
InternVL2.5-26B4.45.75.56.02.0
Gemma-3-27B-it12.815.312.320.02.0
Open-source (>30B)
Qwen2.5-VL-32B-Instruct25.129.324.736.08.0
Qwen3-VL-32B-Instruct26.522.118.228.02.0
Qwen2.5-VL-72B-Instruct25.645.536.660.010.0
InternVL2.5-78B9.212.411.114.06.0
Medical-domain
Lingshu-32B26.747.1<u>37.2</u>64.012.0
MedGemma-27B-it24.038.630.0<u>54.0</u>18.0
Statistical Baseline
Random25.436.433.140.421.6
Majority30.082.082.082.082.0

Table 21. SURGKNOWBENCH-Hard Stage II action-recognition breakdown on HeiChole (multi-label, 4 actions). macro-F1 is the headline metric; micro-F1 / precision / recall (set-level, sample-aggregated) and set-exact accuracy are diagnostics. All values are percentages. Bold / <u>underline</u> mark best / second-best per column among models; the Random and Majority rows are uninformed reference baselines.

Tab. 19: SURGKNOWBENCH-Hard Stage II surgical-skill assessment on HeiChole (GOALS-style 5-key rubric, 1–5 Likert). MAE (lower is better; range [0, 4]) and quadratic-weighted Cohen’s κ\kappa (multiplied by 100), each macro-averaged across the three sub-axes (holistic, Calot-phase, dissection-phase). Bold / _underline_ mark best / second-best per column among models (lowest for MAE, highest for κ\kappa); the _Random_ and _Majority_ rows are uninformed reference baselines.

ModelMAEκ\kappa
MAE↓\downarrowκ↑\kappa\uparrow
_Proprietary_
GPT-4o-mini0.681.4
Claude-Haiku-4.50.93_6.2_
_Open-source (<10B)_
InternVL2.5-8B0.965.8
Qwen2.5-VL-7B-Instruct0.88-4.7
Qwen3-VL-8B-Instruct1.071.8
Qwen3-VL-8B-Thinking0.7312.1
_Open-source (10–30B)_
Pixtral-12B-24090.796.0
InternVL3-14B0.844.8
InternVL2.5-26B_0.69_-0.9
Gemma-3-27B-it0.88-1.1
_Open-source (>30B)_
Qwen2.5-VL-32B-Instruct0.82-0.7
Qwen3-VL-32B-Instruct0.69-0.8
Qwen2.5-VL-72B-Instruct0.894.4
InternVL2.5-78B0.92-6.3
_Medical-domain_
Lingshu-32B0.75-2.1
MedGemma-27B-it0.94-8.9
_Statistical Baseline_
Random0.94-3.9
Majority0.630.0

Table 19. SURGKNOWBENCH-Hard Stage II surgical-skill assessment on HeiChole (GOALS-style 5-key rubric, 1–5 Likert). MAE (lower is better; range [0, 4]) and quadratic-weighted Cohen’s κ\kappa (multiplied by 100), each macro-averaged across the three sub-axes (holistic, Calot-phase, dissection-phase). Bold / _underline_ mark best / second-best per column among models (lowest for MAE, highest for κ\kappa); the _Random_ and _Majority_ rows are uninformed reference baselines.

Model[email protected]mIoU[email protected][email protected]
[email protected] ↑\uparrowmIoU↑\uparrow[email protected]↑\uparrow[email protected]↑\uparrow
Proprietary
GPT-4o-mini5.75.45.85.5
Claude-Haiku-4.51.66.11.71.6
Open-source (<10B)
InternVL2.5-8B0.74.30.70.8
Qwen2.5-VL-7B-Instruct10.511.410.810.2
Qwen3-VL-8B-Instruct<u>15.4</u>16.215.8<u>15.0</u>
Qwen3-VL-8B-Thinking10.512.410.810.2
Open-source (10–30B)
Pixtral-12B-24098.910.49.28.7
InternVL3-14B11.311.811.711.0
InternVL2.5-26B4.95.65.04.7
Gemma-3-27B-it5.77.25.85.5
Open-source (>30B)
Qwen2.5-VL-32B-Instruct6.59.96.76.3
Qwen3-VL-32B-Instruct17.8<u>15.7</u>18.317.3
Qwen2.5-VL-72B-Instruct11.411.311.811.0
InternVL2.5-78B3.23.13.33.1
Medical-domain
Lingshu-32B6.56.06.76.3
MedGemma-27B-it4.06.94.23.9
Statistical Baseline
Random0.21.00.20.2
Majority0.03.40.00.0

Table 20. Tab. 20: SURGKNOWBENCH-Hard Stage II tool-localization breakdown on CholecT50. [email protected] is the headline metric; mIoU, [email protected], [email protected] are diagnostics. All values are percentages. Bold / <u>underline</u> mark best / second-best per column among models; the Random and Majority rows are uninformed reference baselines.

Matched tool comparison

To separate the contribution of tool access from the change in the overall agent setting, we compare tools off and on under the same task instruction and two-sentence reasoning preamble. The tool-enabled condition adds a neutral tool-availability instruction. Both conditions use SURGKNOWBENCH-Hard with the full knowledge base and reasoning enabled. Tab. 3, Panel B, reports the matched differences for five backbones; Tab. 22 provides the corresponding absolute scores for InternVL2.5-8B, Lingshu-32B, and Gemma-3-27B.

ModelRegimeTool Det. F1↑Triplet Det. F1↑Tool Loc. [email protected] ↑Phase Rec. Acc.↑Action Rec. F1↑CVS Ass. κ\kappa ↑Skill Ass. κ\kappa ↑
InternVL2.5-8BTools off0.00.00.01.40.0−1.30.0
Tools on1.90.68.11.714.05.0−1.7
Δ\Delta+1.9+0.6+8.1+0.3+14.0+6.3−1.7
Lingshu-32BTools off0.42.22.40.021.7−1.23.8
Tools on10.10.00.02.338.2−0.3−1.9
Δ\Delta+9.7−2.2−2.4+2.3+16.5+0.9−5.7
Gemma-3-27BTools off5.20.00.00.922.9−1.62.1
Tools on9.33.04.92.629.50.96.6
Δ\Delta+4.1+3.0+4.9+1.7+6.6+2.5+4.5

Table 22. Absolute scores under a matched prompt on SURGKNOWBENCH-Hard (952 episodes, full knowledge base, reasoning enabled). The no-tool and tool conditions share the task instruction and reasoning preamble; only tool availability changes. Δ\Delta is the tool-enabled score minus the no-tool score. All scores and differences are multiplied by 100.

Tool performance matrix

We compare four representative open-source backbones—InternVL2.5-8B, Qwen3-VL-32B-Instruct, Qwen2.5-VL-72B-Instruct, and Lingshu-32B—spanning differences in parameter scale, architecture family, model generation, and medical specialization. In particular, Lingshu-32B serves as a medically aligned backbone, while the remaining models are general-purpose MLLMs. Full enables both perceive() and retrieve(), whereas w/o perceive() and w/o retrieve() disable the corresponding tool.

Findings. Each (model, config, dataset, task) combination corresponds to a Stage II evaluation with greedy decoding on SURGKNOWBENCH-Hard using the same decoding, frame, and interaction budgets as the main experiments. We report the headline metrics from Tab. 2, macro-averaged across constituent datasets. Three key observations emerge from Tab. 23.

ModelConfigTool Det. macro-F1↑Triplet Det. macro-F1↑Tool Loc. [email protected] ↑Phase Rec. Acc.↑Action Rec. macro-F1↑CVS Ass. Acc./κ\kappa ↑Skill Ass. MAE/κ\kappa ↑
Qwen3-VL-32B-Instruct [20]Full7.00.017.85.126.540.6/7.70.69/−0.8
w/o perceive()8.15.112.24.024.847.2/4.51.06/6.6
w/o retrieve()6.20.016.24.017.027.4/4.20.66/9.7
Qwen2.5-VL-72B-Instruct [19]Full6.20.011.44.925.649.7/4.80.89/4.4
w/o perceive()14.50.013.13.716.452.8/10.11.01/−16.8
w/o retrieve()7.30.014.69.720.236.8/−1.60.83/−0.1
Lingshu-32B [25]Full16.30.06.54.926.78.0/1.10.75/−2.1
w/o perceive()10.52.00.04.335.610.1/1.70.76/9.6
w/o retrieve()8.82.312.24.345.814.2/1.90.80/−1.3
InternVL2.5-8B [18]Full6.90.30.70.918.036.5/−4.80.96/5.8
w/o perceive()4.32.00.01.711.531.2/0.41.10/2.8
w/o retrieve()3.60.80.02.312.654.9/0.02.37/0.7

Table 23. Toolbox ablation on SURGKNOWBENCH-Hard under Stage II interaction. Full enables both perceive() and retrieve(), while w/o perceive() and w/o retrieve() disable the corresponding tool. F1 and accuracy are percentages, κ\kappa is multiplied by 100, and MAE is on the original ordinal scale.

(i) The optimal toolbox is backbone-dependent. No single tool configuration consistently dominates across models. While the Full setup achieves strictly the best performance on 4/7 evaluation axes for Qwen3-VL-32B-Instruct and InternVL2.5-8B, it wins only 2/7 axes for Qwen2.5-VL-72B-Instruct and Lingshu-32B, where ablated configurations outperform the full toolbox on most tasks. Counts use κ\kappa for workflow assessment and exclude ties. This suggests that the utility of external perception and retrieval depends strongly on the backbone’s internal representations and priors.

(ii) Retrieval can hurt medically specialized models. For Lingshu-32B, disabling RETRIEVE improves performance on five of seven axes, most notably on Action Recognition (26.7 → 45.8). Tool Localization (6.5 → 12.2) and Triplet Detection (0.0 → 2.3) exhibit similar trends. Disabling retrieval also improves three of seven axes for Qwen2.5-VL-72B-Instruct (tool detection, tool localization, and phase recognition), three for InternVL2.5-8B (triplet detection, phase recognition, and CVS assessment), and one for Qwen3-VL-32B-Instruct (skill assessment). These counts use κ\kappa for workflow-assessment axes.

(iii) Skill assessment is highly configuration-sensitive. Unlike perception-heavy tasks, skill assessment does not benefit consistently from any single toolbox configuration. Quadratic-weighted κ\kappa varies substantially across settings for every backbone, with maximum swings of 21.2, 11.7, 10.5, and 5.1 points for Qwen2.5-VL-72B-Instruct, Lingshu-32B, Qwen3-VL-32B-Instruct, and InternVL2.5-8B, respectively. The optimal configuration differs across models, indicating that higher-level workflow evaluation depends critically on how external evidence is integrated, rather than on the mere availability of tools.

Perception backbone, retrieval budget, and knowledge sources

Tab. 24 examines alternative perception backbones, retrieval budgets, and knowledge-base configurations for Qwen3-VL-8B and Qwen2.5-VL-7B. Entries are changes relative to Stage I direct answers. They describe the performance of each configuration; the matched tool-access comparison is reported separately in Appendix E.2.1.

SettingModelTool Det. Δ\DeltaF1↑Triplet Det. Δ\DeltaF1↑Tool Loc. Δ\Delta[email protected] ↑Phase Rec. Δ\DeltaAcc.↑Action Rec. Δ\DeltaF1↑CVS Ass. Δκ\Delta\kappa ↑Skill Ass. Δκ\Delta\kappa ↑
Main (budget = 2, full KB, SAM2)Qwen3-VL-8B+6.2—+1.6+3.7+27.3−1.8−2.1
Main (budget = 2, full KB, SAM2)Qwen2.5-VL-7B+3.9—−0.8+3.1+3.0−4.3−12.7
Main w/ SurgWIKI-OpenQwen3-VL-8B+6.4—−8.0+5.1+22.2−8.6−12.2
Main w/ SurgWIKI-OpenQwen2.5-VL-7B+3.4—0.0+3.1+4.5−1.4−10.8
Retrieve budget = 1Qwen3-VL-8B+14.6—0.0+2.9+20.3−3.0−6.5
Retrieve budget = 1Qwen2.5-VL-7B+13.6—0.0+2.3+6.9+1.3+0.4
Retrieve budget = 3Qwen3-VL-8B+15.6—−0.8+3.1+20.9−1.6−13.4
Retrieve budget = 3Qwen2.5-VL-7B+14.0—−0.8+2.6+5.7+2.5−12.0
w/o case_record sourceQwen3-VL-8B+14.7+0.7−0.8+3.4+25.1−4.0+16.2
w/o case_record sourceQwen2.5-VL-7B+15.0—+0.8+0.6+10.5+2.5+6.1
SurgSAM-2 perceive() backboneQwen3-VL-8B+14.8—−10.5+2.3+17.7−2.1+6.1
SurgSAM-2 perceive() backboneQwen2.5-VL-7B+14.8—−6.7+5.1+10.5+7.2−12.3

Table 24. Tool-configuration comparisons on SurgKnowBench-Hard. Entries are differences from Stage I direct answers, multiplied by 100. The action-recognition and skill-assessment ablations use reasoning enabled and a four-frame cap; the other ablation columns use reasoning disabled. These across-setting changes are distinct from the matched tool comparison in Tab. 3, Panel B.

Perception backbone. Replacing SAM 2 with the surgically adapted SurgSAM-2 retains task-dependent gains and regressions. Both backbones achieve a +14.8-point tool-detection gain relative to Stage I, but tool localization declines by 10.5 and 6.7 points for Qwen3-VL-8B and Qwen2.5-VL-7B, respectively. Thus, a medically adapted perception backend alone does not remove the localization bottleneck.

Retrieval budget. We vary the per-source retrieve() call budget k∈{1,2,3}k \in\{1, 2, 3\}, with k=2k = 2 in the default setting. Increasing the budget from one to three calls does not produce uniform gains: tool detection and phase recognition improve slightly for both backbones, while skill-assessment changes fall from −6.5 to −13.4 points for Qwen3-VL-8B and from +0.4 to −12.0 for Qwen2.5-VL-7B. Access to more retrieved evidence therefore does not ensure better rubric-based assessment.

Case-record retrieval. Removing the case-record source retains positive tool-detection and action-recognition changes relative to Stage I for both backbones. Workflow assessment remains model-dependent: without case records, the CVS-assessment change is −4.0 points for Qwen3-VL-8B and +2.5 for Qwen2.5-VL-7B. These results show that the utility of a knowledge source depends on how the agent uses its evidence.

SURGWIKI-Open. Table 25 compares Stage II minus Stage I changes for the full knowledge base and SURGWIKI-Open, which excludes commercial sources, across temporal reasoning and workflow assessment.

State-of-the-art baselines

To contextualize zero-shot generalist performance, Table 26 collects published results from task-specific supervised methods. Their training data and evaluation protocols differ, so these values serve as reference points rather than a matched ranking against our agents.

TaskDatasetReference MethodReported MetricScore
Spatial Recognition
Tool DetectionCholecT50UniSurg [44]AP-I91.55
Tool DetectionPitVisDKFZ Ensemble [13]Macro-F141.70
Tool DetectionCATARACTSOVFM [46]Micro-AUC98.50
Tool DetectionGraSPTAPIS [12]mAP94.33
Triplet DetectionCholecT50UniSurg [44]AP-IVT39.54
Tool LocalizationCholecT50MCIT-IG [77]5Instrument [email protected]60.10
Temporal Reasoning
Phase RecognitionCataract-101GL-MSTCN [78]Accuracy96.50
Phase RecognitionCholec80Surgformer + NFSM [79]Accuracy / Jaccard92.30 / 78.00
Phase RecognitionAutoLaparoUniSurg [44]Accuracy / Jaccard86.40 / 69.90
Phase RecognitionGraSPTAPIS [12]mAP / F176.72 / 63.42
Phase RecognitionHeiCoV-JEPA [43]6Accuracy64.00
Phase RecognitionPitVisUniSurg [44]Accuracy / Jaccard86.50 / 65.50
Phase RecognitionMultiBypass140ZEN [45]F1 / Accuracy / Jaccard89.42 / 72.73 / 62.36
Action RecognitionHeiCholeWintegral (ResNet50 ensemble) [2]Average F123.28
Workflow Assessment
CVS AssessmentCholec80-CVS—2——
Skill AssessmentHeiCholeCareSyntax (3D-ResNet18) [2]3MAE0.78

Table 26. Published supervised reference results for each task–dataset combination in SURGKNOWBENCH, organized using the same capability taxonomy as Table 2. These methods typically rely on task-specific architectures, dense supervision, or benchmark-specific optimization, in contrast to the zero-shot generalist MLLM setting evaluated in SURGKNOWBENCH. Results are intended as contextual reference points only and are not directly comparable to SURGKNOWBENCH Stage I evaluation due to differences in supervision, protocol, and metric definitions.

Analysis of agent tool-use behavior

To further inspect retrieval quality, we analyze the Jina reranker scores returned by retrieve() in the Full setting. We focus on the four trajectory-analysis models used in § 4.3. Each histogram in Figure 7 includes only successful retrieval calls with a returned reranker score; rejected or warning-triggering calls are excluded from the score distribution.

Four stacked histograms of reranker score distributions for InternVL2.5-8B, Lingshu-32B, Qwen2.5-VL-72B-Instruct, and Qwen3-VL-32B-Instruct. The x-axis ranges from 0.0 to 1.0 and is labeled Retrieved queries; the y-axis is labeled Retrieved queries. Legends show Mean 0.54, Mean 0.45, Mean 0.46, and Mean 0.54, respectively, with stacked categories Textbook, Guideline, Tutorial video, and Case record.

Figure 7. Reranker score distributions across four backbones on SURGKNOWBENCH-Hard in the Full setting. Bars are stacked by retrieval source type, and the red vertical line marks the mean Jina reranker score for each model.

The score distributions show that retrieval behavior differs substantially across backbones. Qwen3-VL-32B-Instruct and InternVL2.5-8B obtain higher mean reranker scores, with many calls concentrated around moderate-to-high relevance. Qwen2.5-VL-72B-Instruct and Lingshu-32B have broader distributions centered at lower scores. These differences suggest that retrieval quality is not determined only by access to the same knowledge base and backend; it also depends on how each agent formulates queries and chooses source types. Thus, the retrieval tool exposes a second bottleneck beyond final-answer integration: agents must first ask searchable, source-appropriate questions.

Retrieval behavior

Figure 8 separates retrieval calls by evidence source for each task, complementing the tool-invocation patterns in Figure 4.

Tool use and episode outcomes. Table 27 compares tool-use patterns between successful and failed episodes for the four trajectory-analysis models. The success-conditioned rates and call counts describe associations between agent behavior and outcomes; they do not isolate the causal effect of a tool call.

Computational resources

All open-source experiments were conducted on the NHR@FAU Alex cluster. Closed-source model evaluations were run through their respective public APIs and did not consume local GPU resources.

Hardware. We used two GPU partitions of Alex: (i) alex-a100: nodes with NVIDIA A100 80 GB SXM4 GPUs interconnected via NVLink, used for all VLM inference; (ii) alex-a40: nodes with NVIDIA A40 48 GB GPUs, used for lighter-weight evaluation and data preprocessing.

Software stack. PyTorch, HuggingFace Transformers, and vLLM for high-throughput inference, running under CUDA on Rocky Linux 8.

Total compute. Including preliminary, failed, and ablation runs, the total compute budget of this paper is approximately 4,766 GPU h on A100 80 GB and 703 GPU h on A40 48 GB, for a combined ~5,470 GPU h.

License

We provide the evaluation code and a data curation pipeline; we do not distribute data. Users obtain the source datasets directly from their original providers under the applicable access conditions and licenses. SURGKNOWBENCH harmonizes these datasets into a unified evaluation format (Appendix C.1). Our harmonization code is released under CC BY-NC 4.0; the source datasets remain subject to their original licenses, listed below with access links.

CholecT50 [1]: CC BY-NC-SA 4.0. https://github.com/CAMMA-public/cholect50/blob/master/LICENSE

Cholec80 [5]: CC BY-NC-SA 4.0. https://camma.unistra.fr/datasets/

Cholec80-CVS [8]: the CVS annotations are published under CC0; the underlying videos remain subject to the Cholec80 license. https://doi.org/10.6084/m9.figshare.22183885

HeiChole [2]: CC BY-NC-SA 4.0. https://www.synapse.org/Synapse:syn18824884

PitVis [13]: the provider’s current release (version 3) is licensed under CC BY-NC-SA 4.0. https://rdr.ucl.ac.uk/articles/dataset/PitVis_Challenge_Endoscopic_Pituitary_Surgery_videos/26531686

Cataract-101 [14]: CC BY-NC 4.0. https://ftp.itec.aau.at/datasets/ovid/cat-101/

CATARACTS [15]: research-use only. https://cataracts.grand-challenge.org/

GraSP [12]: MIT license (public repository); dataset access through the project repository. https://github.com/BCV-Uniandes/GraSP/blob/main/LICENSE

AutoLaparo [10]: CC BY-NC-SA 4.0; academic research only. https://autolaparo.github.io/

HeiCo [11]: CC BY-NC-SA 4.0. https://www.synapse.org/Synapse:syn21903917

MultiBypass140 [9]: CC BY-NC-SA 4.0. https://github.com/CAMMA-public/MultiBypass140/blob/main/LICENSE

References

  1. [1]Chinedu Innocent Nwoye, Tong Yu, Cristians Gonzalez, Barbara Seeliger, Pietro Mascagni, Didier Mutter, Jacques Marescaux, and Nicolas Padoy. Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos. Medical Image Analysis, 78: 102433, 2022. doi: 10.1016/j.media.2022.102433.DOI
  2. [2]Martin Wagner, Beat-Peter Müller-Stich, Anna Kisilenko, Duc Tran, Patrick Heger, Lars Mündermann, David M. Lubotsky, Benjamin Müller, Torkone Davitashvili, Manuela Capek, et al. Comparative validation of machine learning algorithms for surgical workflow and skill analysis with the HeiChole benchmark. Medical Image Analysis, 86:102770, 2023. doi: 10.1016/j.media.2023.102770.DOI
  3. [3]Natalia Valderrama, Paola Ruiz Puentes, Isabela Hernández, Nicolás Ayobi, Mathilde Verlyck, Jessica Santander, Juan Caicedo, Nicolás Fernández, and Pablo Arbeláez. Towards holistic surgical scene understanding. In International conference on medical image computing and computer-assisted intervention, pages 442–452. Springer, 2022.DOI
  4. [4]Zhitao Zeng, Zhu Zhuo, Xiaojun Jia, Erli Zhang, Junde Wu, Jiaan Zhang, Yuxuan Wang, Chang Han Low, Jian Jiang, Zilong Zheng, Xiaochun Cao, Yutong Ban, Qi Dou, Yang Liu, and Yueming Jin. SurgVLM: A large vision-language model and systematic evaluation benchmark for surgical intelligence. arXiv preprint arXiv:2506.02555, 2025. URL https://arxiv.org/abs/2506.02555.
  5. [5]Andru P. Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel de Mathelin, and Nicolas Padoy. EndoNet: A deep architecture for recognition tasks on laparoscopic videos. IEEE Transactions on Medical Imaging, 36(1):86–97, 2017. doi: 10.1109/TMI.2016.2593957.DOI
  6. [6]Matthias Carstens, Shubha Vasisht, Zheyuan Zhang, Iulia Barbur, Annika Reinke, Lena Maier-Hein, Daniel A Hashimoto, and Fiona R Kolbinger. Artificial intelligence for surgical scene understanding: a systematic review and reporting quality meta-analysis. npj Digital Medicine, 9 (1):59, 2026. doi: 10.1038/s41746-025-02227-4. URL https://www.nature.com/articles/s41746-025-02227-4.DOI
  7. [7]Lena Maier-Hein, Matthias Eisenmann, Duygu Sarikaya, Keno März, Toby Collins, Anand Malpani, Johannes Fallert, Hubertus Feussner, Stamatia Giannarou, Pietro Mascagni, et al. Surgical data science–from concepts toward clinical translation. Medical image analysis, 76: 102306, 2022.
  8. [8]Manuel Sebastián Ríos, María Alejandra Molina-Rodriguez, Daniella Londoño, Camilo Andrés Guillén, Sebastián Sierra, Felipe Zapata, and Luis Felipe Giraldo. Cholec80-CVS: An open dataset with an evaluation of Strasberg’s critical view of safety for AI. Scientific Data, 10(1): 194, 2023. doi: 10.1038/s41597-023-02073-7.DOI
  9. [9]Joel L. Lavanchy, Sanat Ramesh, Diego Dall’Alba, Cristians Gonzalez, Paolo Fiorini, Beat P. Müller-Stich, Philipp C. Nett, Jacques Marescaux, Didier Mutter, and Nicolas Padoy. Challenges in multi-centric generalization: phase and step recognition in Roux-en-Y gastric bypass surgery. International Journal of Computer Assisted Radiology and Surgery, 19:2249–2257, 2024. doi: 10.1007/s11548-024-03166-3.
  10. [10]Ziyi Wang, Bo Lu, Yonghao Long, Fangxun Zhong, Tak-Hong Cheung, Qi Dou, and Yunhui Liu. AutoLaparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hysterectomy. In Medical Image Computing and Computer Assisted Intervention (MICCAI), pages 486–496, 2022. doi: 10.1007/978-3-031-16449-1_46.DOI
  11. [11]Lena Maier-Hein, Martin Wagner, Tobias Ross, Annika Reinke, Sebastian Bodenstedt, Peter M. Full, Hellena Hempe, Diana Mindroc-Filimon, Patrick Scholz, Thuy Nuong Tran, et al. Heidelberg colorectal data set for surgical data science in the sensor operating room. Scientific Data, 8 (1):101, 2021. doi: 10.1038/s41597-021-00882-2.
  12. [12]Nicolás Ayobi, Santiago Rodríguez, Alejandra Pérez, Isabela Hernández, Nicolás Aparicio, Eugénie Dessvres, Sebastián Peña, Jessica Santander, Juan Ignacio Caicedo, Nicolás Fernández, and Pablo Arbeláez. Pixel-wise recognition for holistic surgical scene understanding. Medical Image Analysis, 106:103726, 2025. doi: 10.1016/j.media.2025.103726.DOI
  13. [13]Adrito Das, Danyal Z Khan, Dimitrios Psychogiyos, Yitong Zhang, John G Hanrahan, Francisco Vasconcelos, You Pang, Zhen Chen, Jinlin Wu, Xiaoyang Zou, Guoyan Zheng, Abdul Qayyum, Moona Mazher, Imran Razzak, Tianbin Li, Jin Ye, Junjun He, Szymon Plotka, Joanna Kaleta, Amine Yamlahi, Antoine Jund, Patrick Godau, Satoshi Kondo, Satoshi Kasai, Kousuke Hirasawa, Dominik Rivoir, Alejandra Pérez, Santiago Rodriguez, Pablo Arbeláez, Danail Stoyanov, Hani J Marcus, and Sophia Bano. PitVis-2023 challenge: Workflow recognition in videos of endoscopic pituitary surgery. Medical Image Analysis, page 103716, 2025.DOI
  14. [14]Klaus Schoeffmann, Mario Taschwer, Stephanie Sarny, Bernd Münzer, Manfred Jürgen Primus, and Doris Putzgruber-Adamitsch. Cataract-101: video dataset of 101 cataract surgeries. In Proceedings of the 9th ACM Multimedia Systems Conference (MMSys), pages 421–425, 2018. doi: 10.1145/3204949.3208137.DOI
  15. [15]Hassan Al Hajj, Mathieu Lamard, Pierre-Henri Conze, Soumali Roychowdhury, Xiaowei Hu, Gabija Maršalkaitė, Odysseas Zisimopoulos, Muneer Ahmad Dedmari, Fenqiang Zhao, Jonas Prellberg, et al. CATARACTS: Challenge on automatic tool annotation for catarACT surgery. Medical Image Analysis, 52:24–41, 2019. doi: 10.1016/j.media.2018.11.008.
  16. [16]OpenAI. GPT-4o mini: advancing cost-efficient intelligence, July 2024. URL https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/.
  17. [17]Anthropic. Claude Haiku 4.5. https://www.anthropic.com/news/claude-haiku-4-5, 2025. Model card; accessed 2026-05.
  18. [18]Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. InternVL 2.5: Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2024. URL https://arxiv.org/abs/2412.05271.
  19. [19]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL technical report, 2025. URL https://arxiv.org/abs/2502.13923.
  20. [20]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shitong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen You, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-VL technical report, 2025. URL https://arxiv.org/abs/2511.21631.
  21. [21]Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo, William Marshall, Louis Martin, Arthur Mensch, Pavankumar Muddireddy, Valera Nemchinnikova, Marie Pellat, Patrick Von Platen, Nikhil Raghuraman, Baptiste Rozière, Alexandre Sablayrolles, Lucile Saulnier, Romain Sauvestre, Wendy Shang, Roman Soletskyi, Lawrence Stewart, Pierre Stock, Joachim Studnia, Sandeep Subramanian, Sagar Vaze, Thomas Wang, and Sophia Yang. Pixtral 12B, 2024. URL https://arxiv.org/abs/2410.07073.
  22. [22]Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, weijie Su, Jie Shao, et al. InternVL3: Exploring advanced training and test-time recipes for open-source multimodal models, 2025. URL https://arxiv.org/abs/2504.10479.
  23. [23]Gemma Team. Gemma 3 technical report, 2025. URL https://arxiv.org/abs/2503.19786.
  24. [24]Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Hardy Chen, Xidong Wang, Zhenyang Cai, Ke Ji, Xiang Wan, et al. Towards injecting medical visual knowledge into multimodal LLMs at scale. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7346–7370, 2024.
  25. [25]LingshuTeam. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning, 2025. URL https://arxiv.org/abs/2506.07044.
  26. [26]Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cian Hughes, Charles Lau, Justin Chen, Fereshteh Mahvar, Liron Yang, et al. MedGemma technical report, 2025. URL https://arxiv.org/abs/2507.05201.
  27. [27]David R. Krathwohl. A revision of Bloom’s taxonomy: An overview. Theory Into Practice, 41(4):212–218, 2002. ISSN 00405841, 15430421. URL http://www.jstor.org/stable/1477405.DOI
  28. [28]FT Drake, KD Horvath, AB Goldin, and KW Gow. The general surgery chief resident operative experience: 23 years of national ACGME case logs. JAMA Surgery, 148(9):841–847, 2013. doi: 10.1001/jamasurg.2013.2919.DOI
  29. [29]Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking retrieval-augmented generation for medicine. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 6233–6251, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.372. URL https://aclanthology.org/2024.findings-acl.372/.DOI
  30. [30]Kaiwen Wei, Rui Shan, Dongsheng Zou, Jianzhong Yang, Bi Zhao, Junnan Zhu, and Jiang Zhong. MIRAGE: Scaling test-time inference with parallel graph-retrieval-augmented reasoning chains, 2026. URL https://arxiv.org/abs/2508.18260.
  31. [31]B Huo, N Marfo, P Sylla, E Calabrese, S Kumar, BJ Slater, DS Walsh, and W Vosburg. Clinical artificial intelligence: teaching a large language model to generate recommendations that align with guidelines for the surgical management of GERD. Surgical Endoscopy, 38(10):5668–5677, 2024. doi: 10.1007/s00464-024-11155-5.DOI
  32. [32]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. SAM 2: Segment anything in images and videos. In International Conference on Learning Representations, volume 2025, pages 28085–28128, 2025.arxiv.org/abs/2408.00714
  33. [33]Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-Mark prompting unleashes extraordinary visual grounding in GPT-4V. arXiv preprint arXiv:2310.11441, 2023.arxiv.org/abs/2310.11441
  34. [34]OpenAI. GPT-5 System Card. https://openai.com/index/gpt-5-system-card/, 2025.
  35. [35]Max Allan, Alex Shvets, Thomas Kurmann, Zichen Zhang, Rahul Duggal, Yun-Hsuan Su, Nicola Rieke, Iro Laina, Niveditha Kalavakonda, Sebastian Bodenstedt, Luis Herrera, Wenqi Li, Vladimir Iglovikov, Huoling Luo, Jian Yang, Danail Stoyanov, Lena Maier-Hein, Stefanie Speidel, and Mahdi Azizian. 2017 robotic instrument segmentation challenge, 2019. URL https://arxiv.org/abs/1902.06426.
  36. [36]Kun Yuan, Vinkle Srivastav, Nassir Navab, and Nicolas Padoy. HecVL: Hierarchical video-language pretraining for zero-shot surgical phase recognition. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2024, volume 15006 of Lecture Notes in Computer Science, pages 306–316. Springer Nature Switzerland, 2024. doi: 10.1007/978-3-031-72089-5_29.DOI
  37. [37]Jiajie Li, Garrett Skinner, Gene Yang, Brian R. Quaranto, Steven D. Schwaitzberg, Peter C. W. Kim, and Jinjun Xiong. LLaVA-Surg: Towards multimodal surgical assistant via structured surgical video learning. arXiv preprint arXiv:2408.07981, 2024. URL https://arxiv.org/abs/2408.07981.
  38. [38]Kun Yuan, Manasi Kattel, Joël L Lavanchy, Nassir Navab, Vinkle Srivastav, and Nicolas Padoy. Advancing surgical VQA with scene graph knowledge. International Journal of Computer Assisted Radiology and Surgery, 19(7):1409–1417, July 2024. doi: 10.1007/s11548-024-03141-y. URL https://doi.org/10.1007/s11548-024-03141-y.
  39. [39]Jianhui Wei, Zikai Xiao, Danyu Sun, Luqi Gong, Zongxin Yang, Zuozhu Liu, and Jian Wu. SurgBench: A unified large-scale benchmark for surgical video analysis, 2025. URL https://arxiv.org/abs/2506.07603.
  40. [40]Alejandra Perez, Anita Rau, Lee White, Busisiwe Mlambo, Chinedu Nwoye, Muhammad Abdullah Jamal, and Omid Mohareri. Sureon: A benchmark and vision-language-model for surgical reasoning. arXiv preprint arXiv:2603.06570, 2026.arxiv.org/abs/2603.06570
  41. [41]Yaoqian Li, Xikai Yang, Dunyuan Xu, Yang Yu, Litao Zhao, Xiaowei Hu, Jinpeng Li, and Pheng-Ann Heng. SurgPub-Video: A comprehensive surgical video framework for enhanced surgical intelligence in vision-language model. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2026). Underline Science Inc., 2026. doi: 10.48448/ppqb-tp30.DOI
  42. [42]Zhitao Zeng, Mengya Xu, Jian Jiang, Pengfei Guo, Yunqiu Xu, Zhu Zhuo, Chang Han Low, Yufan He, Dong Yang, Chenxi Lin, Yiming Gu, Jiaxin Guo, Yutong Ban, Daguang Xu, Qi Dou, and Yueming Jin. SurgZ: A spectrum of large-scale multimodal data and foundation models for surgical intelligence, 2026. URL https://arxiv.org/abs/2603.16822.
  43. [43]Simon Pezold, Jérôme A. Kurylec, Jan S. Liechti, Beat Peter Müller, and Joël L. Lavanchy. Leveraging generic foundation models for multimodal surgical data analysis. In ML-CDS Workshop, MICCAI, 2025. arXiv:2509.06831.arxiv.org/abs/2509.06831
  44. [44]Jinlin Wu, Felix Holm, Chuxi Chen, An Wang, Yaxin Hu, Xiaofan Ye, Zelin Zang, Miao Xu, Lihua Zhou, Huai Liao, Danny T. M. Chan, Ming Feng, Wai S. Poon, Hongliang Ren, Dong Yi, Nassir Navab, Gaofeng Meng, Jiebo Luo, Hongbin Liu, and Zhen Lei. UniSurg: A video-native foundation model for universal understanding of surgical videos. arXiv preprint arXiv:2602.05638v1, 2026. URL https://arxiv.org/abs/2602.05638v1.
  45. [45]Kangil Park, Yongjun Jeon, Soyoung Lim, Seonmin Park, Jongmin Shin, Jung Yong Kim, Sehyeon An, Jinsoo Rhu, Jongman Kim, Gyu-Seong Choi, Namke Oh, and Kyu-Hwan Jung. A generalizable foundation model for intraoperative understanding across surgical procedures. arXiv preprint arXiv:2602.13633, 2026. URL https://arxiv.org/abs/2602.13633.
  46. [46]Puxun Tu, Ce Zheng, Xiaoling Xie, Jiao Lv, Meng Xie, Jinming Guo, Shengjie Yin, Kunliang Qiu, Yue Wei, Chongyang Wang, Jingfeng Cai, Wei Mi, Yafu Wang, Xiao Zhang, Danba Jiachu, Kun Peng, Wanqi Zhang, Fengfeng Tong, Huiying Chu, Peiquan Zhao, Mingzhi Zhang, and Xiaojun Chen. An ophthalmic video foundation model for surgical recognition and navigation with wet-lab porcine eye validation. Nature Biomedical Engineering, 2026. doi: 10.1038/s41551-026-01622-w. Published online 03 March 2026.DOI
  47. [47]Samuel Schmidgall, Joseph Cho, Cyril Zakka, and William Hiesinger. GP-VLS: A general-purpose vision language model for surgery, 2024. URL https://arxiv.org/abs/2407.19305.
  48. [48]Samuel Schmidgall, Rojin Ziaei, Carl Harris, Ji Woong Kim, Eduardo Pontes Reis, Jeffrey Jopling, and Michael Moor. AgentClinic: a multimodal benchmark for tool-using clinical AI agents. npj Digital Medicine, 2026. ISSN 2398-6352. doi: 10.1038/s41746-026-02674-7. URL https://doi.org/10.1038/s41746-026-02674-7.
  49. [49]Yixing Jiang, Kameron C. Black, Gloria Geng, Danny Park, James Zou, Andrew Y. Ng, and Jonathan H. Chen. MedAgentBench: A virtual EHR environment to benchmark medical LLM agents. NEJM AI, 2(9):AIdbp2500144, 2025. doi: 10.1056/AIdbp2500144. URL https://ai.nejm.org/doi/full/10.1056/AIdbp2500144.DOI
  50. [50]Binxu Li, Tiankai Yan, Yuanting Pan, Jie Luo, Ruiyang Ji, Jiayuan Ding, Zhe Xu, Shilong Liu, Haoyu Dong, Zihao Lin, and Yixin Wang. MMedAgent: Learning to use medical tools with multi-modal agent. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8745–8760, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.510. URL https://aclanthology.org/2024.findings-emnlp.510/.DOI
  51. [51]Chang Han Low, Ziyue Wang, Tianyi Zhang, Zhu Zhuo, Zhitao Zeng, Evangelos B. Mazomenos, and Yueming Jin. SurgRAW: Multi-agent workflow with chain of thought reasoning for robotic surgical video analysis. IEEE Robotics and Automation Letters, 11(4):4857–4864, 2026. doi: 10.1109/LRA.2026.3665443.DOI
  52. [52]Chang Han Low, Zhu Zhuo, Ziyue Wang, Jialang Xu, Haofeng Liu, Nazir Sirajudeen, Matthew Boal, Philip J. Edwards, Danail Stoyanov, Nader Francis, Jiehui Zhong, Di Gu, Evangelos B. Mazomenos, and Yueming Jin. CARES: Collaborative agentic reasoning for error detection in surgery, 2025. URL https://arxiv.org/abs/2508.08764.
  53. [53]Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, et al. MinerU: An open-source solution for precise document content extraction. arXiv preprint arXiv:2409.18839, 2024.arxiv.org/abs/2409.18839
  54. [54]Michael Günther, Saba Sturua, Mohammad Kalim Akram, Isabelle Mohr, Andrei Ungureanu, Bo Wang, Sediqeh Eslami, Scott Martens, Maximilian Werk, Nan Wang, et al. jina-embeddings-v4: Universal embeddings for multimodal multilingual retrieval. In Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025), pages 531–550, 2025.
  55. [55]Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389, 2009.
  56. [56]Gordon V. Cormack, Charles L A Clarke, and Stefan Buettcher. Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. In SIGIR ’09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 758–759, New York, NY, USA, 2009. URL http://doi.acm.org/10.1145/1571941.1572114.DOI
  57. [57]Jina AI. jina-reranker-v2-base-multilingual. https://jina.ai/models/jina-reranker-v2-base-multilingual/, 2024. Model documentation. Released 2024-06-25. Accessed: 2026-05-06.
  58. [58]Norman Williams, P Ronan O’Connell, and Andrew McCaskie. Bailey & Love’s Short Practice of Surgery, 27th Edition. CRC Press, 2018.
  59. [59]Susan Standring. Gray’s Anatomy: The Anatomical Basis of Clinical Practice, 41st Edition. Elsevier Health Sciences, 2015.
  60. [60]Vanderbilt University Medical Center. Open Manual of Surgery in Resource-Limited Settings. https://www.vumc.org/global-surgical-atlas/about. Accessed: 2026-05-04.
  61. [61]Society of American Gastrointestinal and Endoscopic Surgeons (SAGES). SAGES guidelines for the clinical application of laparoscopic biliary tract surgery. https://www.sages.org/publications/guidelines/, 2023.
  62. [62]Nicola de’Angelis, Fausto Catena, Riccardo Memeo, Federico Coccolini, Aleix Martínez-Pérez, Oreste M Romeo, Belinda De Simone, Salomon Di Saverio, Raffaele Brustia, Rami Rhaiem, et al. 2020 WSES guidelines for the detection and management of bile duct injury during cholecystectomy. World journal of emergency surgery, 16(1):30, 2021.
  63. [63]Go Wakabayashi, Yukio Iwashita, Taizo Hibi, Tadahiro Takada, Steven M Strasberg, Horacio J Asbun, Itaru Endo, Akiko Umezawa, Koji Asai, Kenji Suzuki, et al. Tokyo Guidelines 2018: surgical management of acute cholecystitis: safe steps in laparoscopic cholecystectomy for acute cholecystitis (with videos). Journal of Hepato-biliary-pancreatic Sciences, 25(1):73–86, 2018.DOI
  64. [64]Shivali Patel, V Reddy, KR Shumway, and JF Araujo. StatPearls. Treasure Island (FL): StatPearls Publishing, 2025.
  65. [65]Alejandra Perez, Chinedu Nwoye, Ramtin Raji Kermani, Omid Mohareri, and Muhammad Abdullah Jamal. SurgLaVi: Large-scale hierarchical dataset for surgical vision-language representation learning. Medical Image Analysis, 110:103982, 2026.DOI
  66. [66]Zhitong Wang, Cheng Gao, Chaoyin Xiao, Yufei Huang, Shuzheng Si, Kangyang Luo, Yuzhuo Bai, Wenhao Li, Tangjian Duan, Chuanchen Lv, Guoshan Lu, Gang Chen, Fanchao Qi, and Maosong Sun. Document segmentation matters for retrieval-augmented generation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 8063–8075, Vienna, Austria, July 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.findings-acl.422. URL https://aclanthology.org/2025.findings-acl.422/.DOI
  67. [67]Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. C-Pack: Packed resources for general Chinese embeddings. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 641–649, New York, NY, USA, 2024. Association for Computing Machinery. doi: 10.1145/3626772.3657878. URL https://doi.org/10.1145/3626772.3657878.
  68. [68]Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training, 2022. URL https://arxiv.org/abs/2212.03533.
  69. [69]Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning, 2023. URL https://arxiv.org/abs/2308.03281.
  70. [70]Zach Nussbaum, John X. Morris, Brandon Duderstadt, and Andriy Mulyar. Nomic Embed: Training a reproducible long context text embedder, 2025. URL https://arxiv.org/abs/2402.01613.
  71. [71]Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1410. URL https://aclanthology.org/D19-1410/.DOI
  72. [72]Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MPNet: masked and permuted pre-training for language understanding. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA, 2020. Curran Associates Inc.arxiv.org/abs/2004.09297
  73. [73]Qiao Jin, Won Kim, Qingyu Chen, Donald C Comeau, Lana Yeganova, W John Wilbur, and Zhiyong Lu. MedCPT: Contrastive pre-trained transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval. Bioinformatics, 39(11):btad651, 11 2023. ISSN 1367-4811. doi: 10.1093/bioinformatics/btad651. URL https://doi.org/10.1093/bioinformatics/btad651.
  74. [74]Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukoumas, Nan Wang, and Han Xiao. jina-embeddings-v3: Multilingual embeddings with task LoRA, 2024. URL https://arxiv.org/abs/2409.10173.
  75. [75]OpenAI. GPT-4o system card, 2024. URL https://arxiv.org/abs/2410.21276.
  76. [76]Rodrigo Nogueira and Kyunghyun Cho. Passage Re-ranking with BERT. arXiv e-prints, art. arXiv:1901.04085, January 2019. doi: 10.48550/arXiv.1901.04085.DOI
  77. [77]Saurav Sharma, Chinedu Innocent Nwoye, Didier Mutter, and Nicolas Padoy. Surgical action triplet detection by mixed supervised learning of instrument-tissue interactions. In Medical Image Computing and Computer Assisted Intervention — MICCAI 2023, volume 14228 of Lecture Notes in Computer Science, pages 505–514. Springer, 2023. doi: 10.1007/978-3-031-43996-4_48.DOI
  78. [78]Lixin Fang, Lei Mou, Yuanyuan Gu, Yan Hu, Bang Chen, Xu Chen, Yang Wang, Jiang Liu, and Yitian Zhao. Global–local multi-stage temporal convolutional network for cataract surgery phase recognition. BioMedical Engineering OnLine, 21(1):82, 2022. doi: 10.1186/s12938-022-01048-w.DOI
  79. [79]Hao Ding, Zhongpai Gao, Benjamin Planche, Tianyu Luan, Abhishek Sharma, Meng Zheng, Ange Lou, Terrence Chen, Mathias Unberath, and Ziyan Wu. Neural finite-state machines for surgical phase recognition. arXiv preprint arXiv:2411.18018, 2025.arxiv.org/abs/2411.18018
  80. [80]Chinedu Innocent Nwoye, Deepak Alapatt, Tong Yu, Armine Vardazaryan, Fangfang Xia, Zixuan Zhao, Tong Xia, Fucang Jia, Yuxuan Yang, Hao Wang, Derong Yu, Guoyan Zheng, Xiaotian Duan, Neil Getty, Ricardo Sanchez-Matilla, Maria Robu, Li Zhang, Huabin Chen, Jiacheng Wang, Liansheng Wang, Bokai Zhang, Beerend Gerats, Sista Raviteja, Rachana Sathish, Rong Tao, Satoshi Kondo, Winnie Pang, Hongliang Ren, Julian Ronald AbbIng, Mohammad Hasan Sarhan, Sebastian Bodenstedt, Nithya Bhasker, Bruno Oliveira, Helena R. Torres, Li Ling, Finn Gaida, Tobias Czempiel, João L. Vilaça, Pedro Morais, Jaime Fonseca, Ruby Mae Egging, Inge Nicole Wijma, Chen Qian, Guibin Bian, Zhen Li, Velmurugan Balasubramanian, Debodoot Sheet, Imanol Luengo, Yuanbo Zhu, Shuai Ding, Jakob-Anton Aschenbrenner, Nicolas Elini van der Kar, Mengya Xu, Mobarakol Islam, Lalithkumar Seenivasan, Alexander Jenke, Danail Stoyanov, Didier Mutter, Pietro Mascagni, Barbara Seeliger, Cristians Gonzalez, and Nicolas Padoy. Cholectriplet2021: A benchmark challenge for surgical action triplet recognition. Medical Image Analysis, 86:102803, 2023. doi: 10.1016/j.media.2023.102803.DOI

Paper details

vlnc:W_4hxz62pj
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents