In this paper
AbstractIntroductionRelated workSurgical anatomy segmentation datasetsProcedural and phase context in surgical video understandingFoundation models for surgical and endoscopic visionVideo segmentation and temporal memoryMethodsATLAS-120k: Anatomy Segmentation DatasetData Sourcing and ProcessingDiversity and Class DistributionAnnotation Protocol and Quality AssuranceATLAS: Segmentation ModelTransformer InputProcedure Context QueriesPhase Context QueriesTemporal Query PropagationLoss functionPretrainingExperimental setupDatasetATLAS implementationCompared methodsEvaluationGeneralization experimentsData leakageResultsAnnotation qualityInter-observer variabilityAnnotation propagation performanceSegmentation performance and efficiencyComparison with existing methodsInference speedQualitative resultsPer-class resultsAblation studiesContribution of model componentsEffect of training clip lengthEffect of training data sizeGeneralizationZero-shot transferTransfer to surgical phase recognitionDiscussionMain findingsRole of procedural contextTemporal consistencyPretraining and architectureDataset and annotationGeneralizationLimitationsConclusionCRediT authorship contribution statementDeclaration of competing interestData availabilityAcknowledgmentsDeclaration of generative AI and AI-assisted technologies in the manuscript preparation processDataset splitsReferencesWorksComputer Science
ATLAS: A Multi-Procedure Benchmark and Context-Aware Model for Surgical Anatomy Segmentation
Abstract
Recognizing anatomical structures during minimally invasive surgery (MIS) could support intraoperative navigation and improve patient safety, but progress is limited by small, single-procedure datasets and by segmentation methods developed for natural scenes. This work addresses both limitations through a new dataset and model. First, we introduce ATLAS-120k, a clip-level semantic segmentation dataset of 121,018 annotated frames from 100 surgical videos, covering 14 laparoscopic and robot-assisted procedures and 41 classes. Annotations were created with a pipeline that combines manual labeling, tracker-based propagation, and review by surgeons. Second, we propose ATLAS (Anatomy Recognition with Context Learning using Foundation Representations), a video semantic segmentation model for surgical anatomy recognition. ATLAS builds on in-domain foundation-model embeddings and adds context queries for the procedure and surgical phase, together with query propagation between frames for short-term memory. ATLAS outperforms state-of-the-art natural-image and in-domain foundation models in both segmentation accuracy and temporal consistency, achieving a Dice score of 0.70 with ViT-L while running in real time at 76.2 FPS. Ablations show that the context queries improve accuracy and that query propagation improves temporal consistency. ATLAS also generalizes well, matching a dedicated model zero-shot on CholecSeg8k and improving phase recognition on AutoLaparo over SurgeNet initialization. The model weights, dataset annotations, and annotation tool are publicly available at https://github.com/TimJaspers0801/ATLAS.
Introduction
Intraoperative surgical computer vision holds potential to transform minimally invasive surgery (MIS) and robot-assisted surgery (RAS) by delivering real-time decision support, automated anatomical navigation, and enhanced patient safety. However, translating computer vision algorithms into dependable clinical guidance systems remains challenging. In the absence of haptic feedback and under constrained endoscopic fields of view, surgeons must continually interpret complex anatomical relationships while navigating adverse visual conditions such as tissue deformation, bleeding, and surgical smoke. Achieving robust clinical utility requires automated vision models capable of dense, real-time anatomical perception that generalizes reliably across diverse surgical environments [18, 3, 21].
Despite significant progress in surgical scene understanding, prior research has largely prioritized constrained sub-tasks, including surgical phase recognition Maier-Hein et al. [19]; Twinanda et al. [28], instrument detection and tracking Zia et al. [34], and tool segmentation Allan et al. [1]. While these tasks provide foundational capabilities for surgical workflow analysis, they do not offer the dense, semantic spatial reasoning required for anatomy-guided navigation. Existing anatomy segmentation benchmarks have taken initial steps toward bridging this gap, but suffer from key systemic limitations: they are typically restricted to a single surgical procedure (most notably laparoscopic cholecystectomy), encompass small patient cohorts with limited video counts, or offer only isolated, single-organ segmentations rather than dense, holistic scene annotations [5, 10, 20]. Consequently, models trained on current benchmarks struggle to generalize across varying anatomical topologies, surgical modalities, and procedural contexts.
Methodologically, existing video semantic segmentation frameworks adapted from natural image domains fall short in surgical settings because they treat video frames as generic visual sequences without exploiting intraoperative structural domain priors. In clinical practice, expert surgical cognition is inherently hierarchical and context-driven: a surgeon’s expectation of class presence, anatomical exposure, and structural boundary location is conditioned on the overarching surgical procedure type and the immediate surgical phase. Standard memory- or tracking-based segmentation networks prioritize localized frame-to-frame pixel correspondence while failing to model these global temporal and procedural priors.
To address these bottlenecks, we introduce a unified dataset and algorithmic framework designed to establish a scalable foundation for anatomy-aware surgical perception. Our main contributions are summarized as follows:
ATLAS-120k Dataset: We present a novel, large-scale surgical video dataset comprising 121,018 densely annotated frames across 502 video clips sourced from 100 surgical procedures. Spanning 14 distinct laparoscopic and robot-assisted MIS operations and 41 semantic classes, ATLAS-120k expands procedural, technological, and anatomical diversity compared to existing benchmarks.
ATLAS Framework: We propose ATLAS (Anatomy Recognition with Context Learning using Foundation Representations), a video semantic segmentation model specifically tailored to surgical scene understanding. ATLAS combines strong visual embeddings from a surgical foundation model (Jaspers et al., [11]) with explicit procedure- and phase-level context queries, enabling global context-conditioned feature learning alongside short-term query propagation for temporally stable predictions.
Open resources and insights: We publicly release the ATLAS-120k annotations and the ATLAS model weights. We also report on the dataset creation process, including inter-observer variability and how well tracker-based propagation reduces annotation effort, and analyze the model through ablations of its components, training clip length and amount of annotated data, per-class results, and visualizations of its predictions.
This work extends our MICCAI conference paper (de Jong et al., [13]). Compared with that version, we add a review of related work, a more detailed description and analysis of the dataset, including its feature-space diversity, an extended five-step annotation protocol with full-sequence validation, an inter-observer variability study, and an analysis of tracker-based annotation propagation. The ablation study is extended with leave-one-out experiments for each component and analyses of training clip length and the amount of annotated data. We further report per-class results and inference speed for all model sizes, and evaluate generalization through zero-shot transfer to CholecSeg8k and transfer to surgical phase recognition on AutoLaparo.
Related work
Surgical anatomy segmentation datasets
Several public datasets have advanced anatomy-oriented segmentation in minimally invasive surgery (MIS), but each remains constrained in scope. CholecSeg8k (Hong et al., [10]) provides 8,080 frames drawn from 17 laparoscopic cholecystectomy videos of the Cholec80 dataset, annotated with 12 classes covering anatomy and instruments. The Dresden Surgical Anatomy Dataset (DSAD) (Carstens et al., [5]) offers a larger pool of 13,195 frames across 32 videos with 11 anatomical classes; however, the majority of its frames depict isolated organ segmentations rather than dense, full-scene semantic labels, which limits its suitability for holistic scene understanding. Endoscapes-Seg50 (Mascagni et al., [20]) targets Critical-View-of-Safety assessment during cholecystectomy, providing 493 annotated frames drawn from 50 videos across 6 classes. Collectively, these resources share limitations that constrain the generalizability of models trained on them: each covers only a single surgical procedure, so representations learned from them may not generalize well to other procedural contexts; the number of source videos is modest (17-50); and anatomical coverage is narrow (6-12 classes). ATLAS-120k addresses these limitations by covering multiple procedures, both laparoscopic and robot-assisted, with dense full-scene annotations in a single taxonomy. Table 1 compares ATLAS-120k with existing datasets.
Table 1. Comparison of surgical video segmentation dataset characteristics. MIS indicates minimally invasive surgery; RA indicates robot-assisted procedures.
Procedural and phase context in surgical video understanding
A separate line of work has shown that information about the overall procedure provides useful context for interpreting surgical video. EndoNet (Twinanda et al., [28]) was among the first to show that tool presence and surgical phase could be learned jointly from a shared CNN backbone, indicating that these two aspects of surgical context are entangled and mutually informative. TeCNO (Czempiel et al., [8]) reformulated phase recognition as a temporal segmentation problem, using a multi-stage temporal convolutional network with causal, dilated convolutions over pre-extracted frame features to obtain a large receptive field while retaining online, real-time inference. Trans-SVNet (Gao et al., [9]) subsequently showed that fusing spatial (frame-level) and temporal (phase-level) embeddings through a lightweight cross-attention transformer improves phase recognition accuracy over purely temporal models, highlighting the complementarity of instantaneous visual evidence and longer-range procedural context. A common limitation of current work is that phase and procedure recognition are treated as separate tasks, rather than being used to inform downstream perception such as segmentation. ATLAS builds on the value of procedural context by incorporating procedure and phase information directly into the segmentation model through additional query tokens. These context queries are learned jointly with the segmentation task, allowing information about the current procedural context to influence the anatomical predictions.
Foundation models for surgical and endoscopic vision
Motivated by the success of large-scale self-supervised pretraining on natural images, such as DINOv2 (Oquab et al. [24]) and DINOv3 (Siméoni et al. [27]), a growing body of work adapts this paradigm to the surgical and endoscopic domain to compensate for the relative scarcity of densely annotated data. EndoViT (Batić et al. [2]) pretrains a masked-autoencoder vision transformer on a large corpus of endoscopic images, while EndoFM (Wang et al. [29]) instead performs spatio-temporal self-supervised pretraining directly on unlabeled endoscopic video. GastroNet5M (Jong et al. [12]) scales this approach further using a multi-center gastrointestinal endoscopy dataset. In the surgical domain specifically, SurgeNet and its large-scale successor SurgeNetXL (Jaspers et al. [11]) systematically scale self-supervised pretraining for surgical video, while GSViT (Schmidgall et al. [26]) and LEMON (Che et al. [6]) pursue related video-level pretraining strategies for general surgery and endoscopy, respectively.
general surgery and endoscopy, respectively. Collectively, these models demonstrate that domain-specific pretraining yields embeddings that transfer more effectively to downstream surgical perception tasks than their natural-image counterparts. ATLAS builds directly on this line of work: rather than training a new foundation encoder from scratch, it leverages strong embeddings from a surgical foundation model together with in-domain pretrained weights (de Jong et al. [14]), and instead focuses its contribution on the lightweight temporal and contextual reasoning required to exploit these embeddings for consistent, procedure-aware video segmentation.
Video segmentation and temporal memory
Most video segmentation models used in surgery address video object segmentation, where the targets are given in the first frame and tracked without class labels. Cutie (Cheng et al. [7]) and SAM2 (Ravi et al. [25]) store features and masks of past frames in an explicit memory bank. Surgical adaptations of SAM2 make this memory more robust to occlusion and smoke [33] or suppress predictions when the target is out of view [17]. These methods need a prompt per target and do not predict anatomical classes. SAM3 [4] replaces object prompts with concept prompts, but is still queried one concept at a time and was trained on general-domain concepts.
Our setting is video semantic segmentation, where every pixel is assigned one of a fixed set of classes. VidEoMT [23] addresses this task with a plain vision transformer that processes learned queries together with the image patches, following EoMT [15], so no separate decoder is needed. Each query predicts a class and a mask. The output queries of each frame are passed on to the next frame, so a query keeps representing the same structure over time at a constant cost per frame. What is carried forward, however, is limited to the tracked structures. VidEoMT has no representation of the procedure or of how far it has progressed. ATLAS builds on VidEoMT and adds this context (§ 3.2).
Methods
ATLAS-120k: Anatomy Segmentation Dataset
Data Sourcing and Processing
A total of 100 videos were sourced from the open-source GSViT dataset [26], from which 14 distinct procedures were selected for annotation. Rather than annotating full videos, clips of varying lengths were extracted to maximize the breadth of scenes. To standardize the dataset, all videos were downsampled to 15 fps, which corresponds to the lowest original frame rate observed across the collection. Spatial resolutions across the videos varied from to pixels. Figure 1 shows representative examples from the ATLAS-120k dataset, with at least one example per procedure.

Figure 1. Randomly selected frames from ATLAS-120k. Odd rows show the original frames and even rows the same frames with their annotations overlaid. Each of the 14 procedures and each semantic class is represented in at least one example.
Diversity and Class Distribution
A core contribution of ATLAS-120k is its procedural and anatomical diversity, as detailed across three dimensions in Figure 2. First, Figure 2a presents the distribution of clip durations. Most clips contain fewer than 400 frames (under 30 seconds), as we extracted many short clips per video to cover a wide range of scenes rather than annotating full videos. Second, Figure 2b illustrates the number of clips and frames per procedure. This distribution is uneven, reflecting the availability of each procedure in the GSViT dataset [26]. Third, Figure 2c highlights the clips and frames per semantic class, clearly demonstrating a long-tail class distribution. This inherent class imbalance reflects the reality of diverse intraoperative environments and introduces a critical benchmark challenge for models to reliably recognize rare but essential anatomical structures. To further illustrate this diversity, Figure 3 shows a t-SNE projection of frame embeddings extracted with a frozen SurgeNet backbone [11]. The left pane contrasts ATLAS-120k against related benchmarks (Endoscapes-Seg50, CholecSeg8k, and DSAD), illustrating its broader feature space. The right pane visualizes the internal diversity across the 14 distinct surgical procedures within ATLAS-120k, highlighting the substantial inter-procedural variations that models must be equipped to navigate.
Figure 2. ATLAS-120k dataset characteristics.

Figure 3. t-SNE projection of SurgeNet (Jaspers et al., [11]) frame embeddings; each point is one frame, and frames with similar features lie close together. Left: ATLAS-120k compared with Endoscapes-Seg50, CholecSeg8k and DSAD, colored by dataset. Right: ATLAS-120k colored by procedure.
Annotation Protocol and Quality Assurance
Annotations were produced by two surgical research fellows under the direct supervision of three experienced surgeons, each possessing over 10 years of clinical expertise. Although the primary objective was anatomical structure recognition, all visible surgical instruments were also annotated to enable additional applications such as surgical tool segmentation. The annotation workflow was conducted in five controlled steps to ensure data integrity:
Manual Initialization: The initial frame of each extracted clip was manually annotated with high precision.
Propagation and Refinement: A lightweight object-tracking model [7] was used to propagate these initial annotations across all subsequent frames. When propagation errors occurred, the affected frame was manually corrected and subsequently used as a new initialization point to resume the automated tracking. To improve tracking performance and efficiency during this phase, the model was fine-tuned on all clips annotated so far. This was repeated once approximately 10%, 25%, 50%, and 100% of the clips had been annotated, using the original training parameters of [7].
Expert Review: The first frame of every clip was independently reviewed by at least one experienced surgeon to verify the clinical correctness and completeness of the anatomical labels.
Comprehensive Sequence Validation: The full clip was comprehensively reviewed by an independent expert rater to guarantee high annotation quality and temporal consistency across the sequence.
Inter-rater agreement analysis: To assess annotation reliability, the test set (see § 4.1) was additionally annotated by a second, independent annotator.
The dataset annotations are released as open source to actively facilitate further research in this direction.
ATLAS: Segmentation Model
Many anatomical structures in surgical video are difficult to identify from appearance alone. Adjacent tissues often have a similar color and texture, and structures are partly covered by fat or connective tissue, deformed by retraction, or obscured by blood and smoke. Surgeons partly resolve this ambiguity with context. Knowing which procedure is being performed and how far it has progressed, they know which structures to expect and roughly where to find them. ATLAS is designed to give a segmentation model access to this type of context, while keeping its predictions consistent from frame to frame.
ATLAS builds on VidEoMT ([23]), a video segmentation model based on a plain vision transformer (ViT). A ViT divides each frame into patches of pixels and represents each patch as a token, the patch tokens. VidEoMT adds a set of learnable query tokens to this sequence, and the ViT processes the patch and query tokens together. At the output, each query predicts a class label and a mask for one structure in the frame. To handle video, the output queries of frame are used as input queries for frame . A query that segmented a structure in one frame therefore starts the next frame with what it learned about that structure, so the model can keep track of structures it has already identified, and its predictions stay consistent over time.
We use VidEoMT as a starting point for two reasons. First, because it represents what it segments as query tokens, context can be added as extra queries without changing the architecture. The segmentation queries can then attend to these context queries inside the same encoder. Second, each frame is processed only once and only the queries are carried forward, so the cost per frame stays constant and the model runs in real time.
The information VidEoMT carries between frames describes only the individual structures it tracks. It has no representation of which procedure is being performed or how far it has progressed. ATLAS adds this information through two types of context queries, a procedure query and phase queries. They are processed in the same encoder as the segmentation queries, propagated to the next frame in the same way, and trained with their own auxiliary losses. Figure 4 gives an overview of the model.
Figure 4. Overview of ATLAS. For each frame, a ViT encoder jointly processes the patch tokens and the query tokens , which consist of segmentation, procedure and phase queries. The output queries predict the masks and classes, the procedure label and a phase embedding . They are then propagated to the next frame as . Only past frames are used, so the model runs online. This schematic was created with the assistance of Claude (Anthropic).
Transformer Input
At frame , the encoder receives the token sequence
where denotes the visual patch tokens of the current frame and , and the segmentation, procedure and phase queries. The query tokens are built from two sources. The initial queries , and are learnable parameters shared across all frames and videos, and act as a learned prior for each query type. The propagated queries , and are the output queries of the previous frame and carry what the model has observed so far: the segmentation queries hold information about individual structures, and the context queries hold information about the procedure and its progression. The input queries at frame are obtained by mapping the propagated queries with a linear layer and adding the initial queries. All tokens are processed by a single transformer encoder. Because self-attention is global, each segmentation query can attend both to the visual evidence in and to the context queries within the same layer. The context therefore shapes the class and mask predictions directly, rather than being fused with them at a later stage.
Procedure Context Queries
The procedure query represents information that holds for an entire video: the type of operation being performed. At every frame, the output procedure query is mapped by a multilayer perceptron (MLP) to predict a distribution of scores over all possible procedure classes
which is compared with the ground-truth label using the cross-entropy loss
Procedure classification is not a goal in itself. The loss serves to make the query summarize the global cues that identify a procedure, such as the instruments in use and the organs in view. Since the segmentation queries attend to this summary, they can use it to judge which anatomical classes are plausible in the current scene.
Phase Context Queries
Within a procedure, the expected anatomy also changes over time, as dissection gradually exposes new structures. The phase queries are intended to capture this progression. Rather than supervising them with phase labels, we train them in a self-supervised manner. For each frame, the output phase queries are averaged and projected by an MLP to an -normalized embedding
These embeddings are optimized with a contrastive loss. Each training batch consists of short windows of consecutive frames, and the other frames in the same window as frame form its set of positives . Since each frame has several positives, we use the multi-positive form of the InfoNCE loss (Khosla et al., [16]):
where runs over all other frames in the batch and is a temperature parameter. The loss is averaged over all frames in the batch. Frames from other windows serve as negatives. The loss thus pulls together the embeddings of frames from the same window and pushes apart those of different windows, which typically come from different procedures or from different moments within a procedure. As a result, the phase queries learn a representation of the surgical state that varies smoothly within a clip and distinguishes between different stages of a procedure.
Temporal Query Propagation
Following VidEoMT (Norouzi et al., [23]), queries are passed from one frame to the next to provide short-term memory and temporal consistency. The input queries at frame are obtained from the output queries of the previous frame and the initial learnable queries :
where Linear is a learned linear layer and
and
The linear layer maps the output queries back to the input query space, and the added initial queries retain a learned prior in every query, so that the model can still detect structures that newly enter the field of view. Segmentation and context queries are propagated by the same mechanism. The segmentation queries thereby keep track of individual structures, while the context queries accumulate information about the procedure over time. Because each frame depends only on the queries of the previous frame, the model runs online, and its computational cost per frame does not grow with the length of the video.
Loss function
The segmentation loss is taken from EoMT (Kerssies et al., [15]): masks are supervised with a binary cross-entropy loss and a Dice loss , and class predictions with a cross-entropy loss . Adding the procedure loss and the phase loss gives the total loss
where the weights balance the terms.
Pretraining
The encoder is initialized with in-domain weights obtained by DINOv3 (Siméoni et al., [27]) pretraining on the SurgeNet dataset (Jaspers et al., [11]), following (de Jong et al., [14]). This adapts the encoder to surgical scenes before training on ATLAS-120k. Because the encoder already captures the appearance of surgical anatomy and instruments, the model starts from features that are better suited to the target domain than natural-image features.
Experimental setup
Dataset
The ATLAS-120k dataset was partitioned into patient-level training (70 videos), validation (10 videos), and test (20 videos) splits. All 14 procedures were represented in the training set, while the validation and test sets covered 6 and 9 procedures, respectively. Model training was restricted to 27 classes. Specific anatomical classes were mapped to broader parent classes within a hierarchical label structure: vena cava and azygos vein to vein, aorta to artery, cystic duct, common bile duct, and hepatic duct to bile duct, and omentum and mesentery to fat. Catheter was mapped to non-anatomical. Classes absent from the training or test set, or with fewer than 500 annotated training frames (esophagus), were relabeled as background. The size and class distribution of each split are given in Appendix § 14.
ATLAS implementation
ATLAS uses a DINOv3 ViT-S/16, ViT-B/16 or ViT-L/16 encoder at an input resolution of pixels, with the queries inserted before the last four encoder blocks, as in EoMT (Kerssies et al., [15]). We use 100 segmentation queries, one procedure query and four phase queries. The model is trained on windows of five consecutive frames with a batch size of 24 windows, for 10 epochs on a single NVIDIA H100 GPU, using AdamW (learning rate , weight decay 0.05, layer-wise decay 0.8), the warm-up and polynomial decay schedule of EoMT, and mixed precision. Augmentation consists of random flipping, scaling, cropping and color jitter, applied identically to all frames of a window. For the phase loss, temperature parameter was set to 0.07. The loss weights , , , , were set to 5.0, 5.0, 2.0, 1.0, and 0.1, respectively. The checkpoint with the highest validation Dice is used for testing. Ablations use ViT-L, and all hyperparameters are kept fixed unless otherwise stated.
Compared methods
Baselines are initialized from their public pretrained weights and fine-tuned with a pixel-wise cross-entropy loss. DINOv2, DINOv3 and GastroNet5M use a linear head, the SurgeNet models an FPN decoder, EndoViT a DPT decoder, and EndoFM a TransUNet decoder; these encoders are fine-tuned end-to-end. SAM2-UNet and SAM3-UNet keep their image encoders frozen and train lightweight adapters together with a U-Net decoder, following Xiong et al. [32].
Evaluation
Models are evaluated on the ATLAS-120k test set, predicting frame by frame in temporal order. Segmentation accuracy is measured with Dice and 95th percentile Hausdorff distance (HD95). Both are first computed per structure in each frame, then averaged within each clip, and finally averaged over clips. HD95 is normalized by the image diagonal, and a class that is present but not predicted receives a value of 1. The per-class results use a pixel-based Dice, computed from pixel counts summed over the entire test set. This gives a more stable estimate for classes that occur in only a few clips or are small in many frames. Temporal consistency is measured with mVC and mVC ([22]): the fraction of non-background pixels with a constant ground-truth label over a sliding window of 12 or 24 frames whose prediction also stays constant. As this measures stability rather than correctness, we interpret it together with Dice and HD95.
Generalization experiments
Generalization is assessed on CholecSeg8k ([10]) for segmentation and on AutoLaparo ([30]) for phase recognition. For CholecSeg8k, we use videos 12, 20, 48 and 55 for testing, 1 and 18 for validation, and the remaining 11 for training, and merge grasper and L-hook into one instrument class, giving eight classes. The fine-tuned reference uses SurgeNet initialization. For zero-shot evaluation, the predictions of the unmodified ATLAS model are mapped to these classes: small intestine, colon/rectum and stomach to gastrointestinal tract; hepatic ligament and cystic plate to connective tissue; tools/camera to instrument; matching classes directly; and all others to background.
For phase recognition, the ATLAS ViT-L encoder initializes a causal temporal convolutional network ([8]) (four residual blocks, dilations 1–8) applied to frame features sampled at 1 fps and resized to 256×256. The model is trained end-to-end for 20 epochs with cross-entropy and AdamW (learning rate for the head, for the encoder), on 10 training, 4 validation and 7 test videos. We report frame-wise accuracy, balanced accuracy, F1 score and Jaccard index.
Data leakage
To prevent data leakage, we used the variant of SurgeNet pretrained without YouTube data, ensuring that ATLAS-120k videos were not included in pretraining. The training, validation, and test splits of ATLAS-120k were made at the video level, ensuring that no video appears in more than one split. For CholecSeg8k, all videos in the respective test sets were excluded from the pretraining data. AutoLaparo was not part of the pretraining data. Thus, all evaluations were conducted on videos not seen during pretraining or model training.
Results
Annotation quality
Inter-observer variability
To assess the reliability of the annotations, the full test set was annotated independently by a second annotator.
Agreement was computed with the same metrics and per-clip averaging as the model evaluation, using the original annotation as reference and the second annotation as prediction. Table 2 shows the results. The annotators reached a Dice of 0.85 and an HD95 of 0.10. The annotations were also temporally stable, with an mVC of 0.96 and an mVC of 0.94. ATLAS ViT-L remains below this level on all metrics. The gap is largest for Dice and mVC (0.15 each), and smaller for mVC (0.11) and HD95 (0.07). The model therefore falls behind the annotators both in segmentation accuracy and in temporal consistency over longer horizons.
| Dice ↑ | HD95 ↓ | mVC12 ↑ | mVC24 ↑ | |
| Inter-observer | 0.85 ± 0.10 | 0.10 ± 0.08 | 0.96 ± 0.05 | 0.94 ± 0.08 |
| ATLAS ViT-L | 0.70 ± 0.13 | 0.17 ± 0.11 | 0.85 ± 0.10 | 0.79 ± 0.15 |
Table 2. Inter-observer variability compared with ATLAS ViT-L on the full test set. Values are the mean ± standard deviation over clips.
Annotation propagation performance
Figure 5 shows how accurately the tracker in the annotation tool propagates an annotation over a clip. For each test clip, the manually annotated first frame was propagated to all subsequent frames without any correction. The result was compared with the final annotations using the Dice score. Results are shown for the tracker with fine-tuning and after fine-tuning on 10%, 25%, 50%, and 100% of the training set. For all trackers, Dice decreases steadily with the distance to the first frame. Without fine-tuning, it drops to approximately 0.84 halfway through the clip and to 0.77 at the end. The propagated annotations thus never match the final annotations exactly, so manual correction was needed throughout each clip. Fine-tuning improves propagation at every point in the clip. With 10% of the training data, the final Dice increases to approximately 0.83. With 25% or more, it reaches approximately 0.85, and the curves for 25%, 50%, and 100% are almost identical. Most of the improvement from fine-tuning is thus obtained with 25% of the training data.
Figure 5. Dice score between annotations propagated from the first frame without manual correction and the final test-set annotations, as a function of the position in the clip. Results are shown for the tracker without fine-tuning and after fine-tuning on 10–100% of the training set.
Segmentation performance and efficiency
Comparison with existing methods
Table 3 compares ATLAS with natural-image foundation models and in-domain pretrained medical models on the ATLAS-120k test set. ATLAS ViT-L achieves the best results on all four metrics. ATLAS ViT-B, with 94M parameters, also outperforms all baselines on every metric. This includes SAM3-UNet, which has almost five times as many parameters (449M) and reaches a Dice of 0.60. Compared with the best baseline for each metric, ATLAS ViT-L improves Dice by 0.09, HD95 by 0.06, mVC12 by 0.07, and mVC24 by 0.09, so it improves both segmentation accuracy and temporal consistency.
| Model | Backbone | #PARAMS (M) ↓ | Dice ↑ | HD95 ↓ | mVC12 ↑ | mVC24 ↑ |
| DINOv2 ([24]) | ViT-B | 87 | 0.49 ± 0.14 | 0.32 ± 0.13 | 0.70 ± 0.13 | 0.59 ± 0.18 |
| DINOv2 ([24]) | ViT-L | 304 | 0.36 ± 0.12 | 0.38 ± 0.12 | 0.58 ± 0.15 | 0.46 ± 0.19 |
| DINOv3 ([27]) | ViT-B | 86 | 0.56 ± 0.14 | 0.26 ± 0.12 | 0.73 ± 0.14 | 0.64 ± 0.18 |
| DINOv3 ([27]) | ViT-L | 303 | 0.55 ± 0.14 | 0.26 ± 0.11 | 0.72 ± 0.14 | 0.63 ± 0.19 |
| SAM2-UNet ([32]) | Hiera-L | 217 | 0.54 ± 0.14 | 0.27 ± 0.11 | 0.66 ± 0.17 | 0.55 ± 0.21 |
| SAM3-UNet ([31]) | Hiera-L | 449 | 0.60 ± 0.14 | 0.24 ± 0.11 | 0.76 ± 0.14 | 0.67 ± 0.20 |
| EndoFM ([29]) | ViT-B | 141 | 0.37 ± 0.12 | 0.38 ± 0.13 | 0.52 ± 0.15 | 0.39 ± 0.18 |
| GastroNet5M ([12]) | ViT-B | 86 | 0.45 ± 0.14 | 0.33 ± 0.13 | 0.67 ± 0.15 | 0.55 ± 0.20 |
| EndoViT ([2]) | ViT-B | 112 | 0.57 ± 0.16 | 0.27 ± 0.12 | 0.73 ± 0.16 | 0.64 ± 0.21 |
| SurgeNet-pvt ([11]) | PVTv2 | 27 | 0.60 ± 0.14 | 0.23 ± 0.09 | 0.78 ± 0.14 | 0.70 ± 0.19 |
| SurgeNet-conv ([11]) | ConvNeXtv2 | 30 | 0.60 ± 0.14 | 0.24 ± 0.10 | 0.76 ± 0.14 | 0.67 ± 0.19 |
| SurgeNet-ca ([11]) | CaFormer-S18 | 26 | 0.61 ± 0.14 | 0.24 ± 0.11 | 0.77 ± 0.14 | 0.69 ± 0.19 |
| ATLAS (ours) | ViT-S | 24 | 0.57 ± 0.14 | 0.30 ± 0.12 | 0.73 ± 0.14 | 0.64 ± 0.18 |
| ATLAS (ours) | ViT-B | 94 | 0.67 ± 0.13 | 0.21 ± 0.11 | 0.83 ± 0.11 | 0.77 ± 0.16 |
| ATLAS (ours) | ViT-L | 318 | 0.70* ± 0.13 | 0.17* ± 0.11 | 0.85* ± 0.10 | 0.79* ± 0.15 |
Table 3. Quantitative comparison of foundation models for segmentation. Models are grouped by pretraining domain: natural-image foundation models (top), in-domain pretrained medical models (middle), and our proposed ATLAS models (bottom). Values are the mean ± standard deviation over test clips. For DINOv2 and DINOv3, we train a linear layer attached to the ViT. Cell shading is scaled linearly within each column from the worst (white) to the best (dark green) value. *Statistically significant improvement over the best baseline for that metric, i.e., the best-performing non-ATLAS model (, Wilcoxon signed-rank test on paired per-clip scores).
At a matched backbone size, the comparison isolates the effect of the ATLAS architecture and pretraining. Among ViT-B models, ATLAS reaches a Dice of 0.67, against 0.57 for EndoViT and 0.56 for DINOv3. ATLAS ViT-S (24M parameters) reaches a Dice of 0.57, which matches EndoViT with roughly a fifth of the parameters. However, it remains below the three SurgeNet models (0.60–0.61 Dice, 26–30M parameters) on all metrics. At the smallest scale, these hierarchical backbones therefore remain competitive, and the advantage of ATLAS emerges from ViT-B onwards.
Among the natural-image models, DINOv3 outperforms DINOv2 (0.56 vs. 0.49 Dice for ViT-B). For both models, scaling from ViT-B to ViT-L does not improve results, which suggests that a linear head limits how well these embeddings can be used for dense surgical segmentation. In-domain pretraining helps only in some cases. EndoViT and the three SurgeNet models are among the strongest baselines. In contrast, EndoFM and GastroNet5M fall below DINOv3, even though both were pretrained on endoscopic data. ATLAS is the only model that ranks first on both accuracy and consistency.
Inference speed
Table 4 reports the inference speed of ATLAS on a single NVIDIA H100 GPU. ATLAS ViT-S runs at 263.7 FPS (3.79 ms per frame), ATLAS ViT-B at 160.3 FPS (6.24 ms per frame), and ATLAS ViT-L at 76.2 FPS (13.16 ms per frame). Because the model processes each frame once and carries information forward only through the propagated queries, the cost per frame does not grow with the length of the video. All three variants therefore run well above the frame rate of the ATLAS-120k videos (15 FPS) and of typical endoscopic video (25–30 FPS), making online use during surgery feasible.
| Model | #Params (M) | Latency (ms) ↓ | FPS ↑ |
| ATLAS ViT-S | 23.8 | 3.79 | 263.7 |
| ATLAS ViT-B | 94.3 | 6.24 | 160.3 |
| ATLAS ViT-L | 318.3 | 13.16 | 76.2 |
Table 4. Inference speed of ATLAS on a single NVIDIA H100 GPU.
Qualitative results
Figure 6 shows predictions of ATLAS ViT-L on three test clips. In the cholecystectomy and RARP examples, the predictions closely follow the annotations. Instruments, the liver, gallbladder and abdominal wall, and the prostate and seminal vesicles are segmented with accurate boundaries, and their masks remain stable over the 8 seconds shown, including when instruments move across or partly occlude them. The remaining errors are mostly local: fat is segmented in fragments, the seminal vesicle boundary varies slightly between frames, and small false-positive regions appear. The hemicolectomy example shows a clear failure case. The instruments are segmented correctly, but the thin artery and vein are missed in all frames, and most of the fat is not recognized.
Figure 6. Qualitative segmentation results of ATLAS ViT-L on three test clips: robotic cholecystectomy (top), robot-assisted radical prostatectomy (RARP, middle) and robotic hemicolectomy (bottom). Each example shows five frames sampled every 2 s from the same clip.
Per-class results
Table 5 reports the results of ATLAS ViT-L for each class. Dice ranges from 0.94 for tools/camera to 0.03 for the hepatic ligament, and 13 of the 27 classes reach 0.70 or higher. The best-segmented classes are instruments and large organs with clear boundaries, such as the lung (0.91), liver (0.89), and gallbladder (0.86), which also have a low HD95 (0.03–0.11). The lowest Dice is found for thin or tubular structures, namely arteries (0.35), bile ducts (0.33), airways (0.22), and nerves (0.17), and for the spleen (0.17) and hepatic ligament (0.03). The latter two also have the highest HD95 (0.84 and 0.81), which indicates that they are often missed entirely. Temporal consistency generally follows segmentation accuracy: large organs such as the liver and uterus reach an mVC12 of 0.87 and 0.91, while arteries and nerves reach 0.52 and 0.55. There are several exceptions. The ovary and the non-anatomical class combine a relatively high Dice (0.74 and 0.83) with low consistency (0.44 and 0.57 mVC12), so their predictions are mostly correct but fluctuate over time. Conversely, the hepatic ligament and ligated plexus have higher consistency (0.69 and 0.90 mVC12) than their Dice would suggest. For the hepatic ligament, this reflects predictions that are stable but incorrect. Per-class consistency for rare classes is based on few test clips and should be interpreted with care.
| Class | Dice ↑ | HD95 ↓ | mVC12 ↑ | mVC24 ↑ |
| Tools/camera | 0.94 | 0.03 | 0.87 | 0.81 |
| Lung | 0.91 | 0.04 | 0.83 | 0.77 |
| Liver | 0.89 | 0.11 | 0.87 | 0.83 |
| Gallbladder | 0.86 | 0.07 | 0.84 | 0.79 |
| Non-anatomical | 0.83 | 0.18 | 0.57 | 0.47 |
| Ovary | 0.74 | 0.18 | 0.44 | 0.29 |
| Pericardium | 0.73 | 0.14 | 0.60 | 0.50 |
| Fat | 0.72 | 0.24 | 0.83 | 0.79 |
| Vein | 0.72 | 0.28 | 0.80 | 0.73 |
| Abdominal wall | 0.71 | 0.20 | 0.85 | 0.80 |
| Uterus | 0.71 | 0.29 | 0.91 | 0.87 |
| Seminal vesicle | 0.70 | 0.10 | 0.87 | 0.83 |
| Prostate | 0.70 | 0.18 | 0.82 | 0.72 |
| Cystic plate | 0.63 | 0.10 | 0.58 | 0.44 |
| Colon/rectum | 0.59 | 0.35 | 0.68 | 0.55 |
| Ligated plexus | 0.56 | 0.05 | 0.90 | 0.86 |
| Urethra | 0.46 | 0.19 | 0.55 | 0.42 |
| Small intestine | 0.43 | 0.40 | 0.66 | 0.54 |
| Fallopian tubes | 0.40 | 0.18 | 0.68 | 0.55 |
| Artery | 0.35 | 0.33 | 0.52 | 0.40 |
| Stomach | 0.34 | 0.43 | 0.60 | 0.55 |
| Bladder | 0.33 | 0.68 | 0.65 | 0.56 |
| Bile duct | 0.33 | 0.27 | 0.58 | 0.41 |
| Trachea/bronchus | 0.22 | 0.45 | 0.79 | 0.70 |
| Major nerve | 0.17 | 0.05 | 0.55 | 0.39 |
| Spleen | 0.17 | 0.84 | 0.58 | 0.47 |
| Hepatic ligament | 0.03 | 0.81 | 0.69 | 0.60 |
Table 5. Per-class results of ATLAS ViT-L on the ATLAS-120k test set, sorted by pooled Dice. Pooled Dice is computed over all test frames together; HD95 is averaged over frames; mVC12 and mVC24 are averaged over the test clips in which the class occurs. Cell shading is scaled linearly within each column from the worst (white) to the best (dark green) value.
Ablation studies
Contribution of model components
Table 6 shows the contribution of each ATLAS component. Removing in-domain pretraining causes the largest overall drop, lowering Dice from 0.70 to 0.67 and both consistency metrics by 0.03. A strong surgical initialization therefore benefits both accuracy and temporal stability. The other components each mainly affect a different metric. Without temporal propagation, Dice and HD95 change little, but mVC12 and mVC24 drop to the level of the model without in-domain pretraining. Query propagation is therefore the main source of temporal consistency. Removing the phase queries has the opposite effect: consistency is unchanged, but Dice drops to 0.67 and HD95 increases from 0.17 to 0.23, the largest boundary error of all variants. This suggests that phase context mainly helps the model decide which structures are present and where they are, rather than keeping predictions stable over time. Removing the procedure queries gives a smaller drop in accuracy (0.69 Dice, 0.19 HD95) with little effect on consistency. Because each component improves a different aspect of performance, only
| Method | Dice ↑ | HD95 ↓ | mVC12 ↑ | mVC24 ↑ |
| Full | ||||
| w/o in-domain pretr. | ||||
| w/o temporal prop. | ||||
| w/o procedure queries | ||||
| w/o phase queries |
Table 6. Ablation studies showing the effect of in-domain pretraining, temporal propagation, and procedure and phase queries on surgical video segmentation. Values are the mean ± standard deviation over test clips. Cell shading is scaled linearly within each column from the worst (white) to the best (dark green) value.
the full model reaches the best accuracy and consistency together.
Effect of training clip length
Table 7 shows how the clip length used during training affects performance. A clip length of 1 corresponds to training without temporal propagation. Increasing the clip length to 5 frames mainly improves temporal consistency, raising mVC12 from 0.82 to 0.85 and mVC24 from 0.76 to 0.79, with a small gain in Dice and HD95. Longer clips bring little further benefit. At 10 and 15 frames, consistency stays at 0.85 mVC12 and gains at most 0.01 mVC24. Over the same range, HD95 worsens from 0.17 to 0.19 and 0.21, and at 15 frames Dice falls back to 0.69. Longer clips may
| Clip length | Dice ↑ | HD95 ↓ | mVC12 ↑ | mVC24 ↑ |
| 1 (no temporal prop.) | ||||
| 5 (ATLAS) | ||||
| 10 | ||||
| 15 |
Table 7. Effect of the clip length used for temporal propagation on surgical video segmentation. Values are the mean ± standard deviation over test clips. Cell shading is scaled linearly within each column from the worst (white) to the best (dark green) value.
let errors in the propagated queries accumulate over more steps during training, which affects boundary accuracy more than consistency. A clip length of 5 gives the best balance between accuracy and temporal consistency, and we use it for all other experiments.
Effect of training data size
Figure 7 shows how performance changes with the fraction of each training clip used for training. With five annotated frames per clip, the model fails to learn useful segmentations: Dice is close to zero and HD95 close to its maximum. Its consistency scores are still moderate (0.63 mVC12), which again shows that consistency cannot be judged separately from accuracy. Performance rises quickly as more of each clip is used. With 10% of each clip, Dice reaches 0.50, and with 20% it reaches 0.65, with HD95 dropping from 0.36 to 0.21. From 40% onwards the curves level off. Dice stays between 0.66 and 0.70, HD95 between 0.17 and 0.21, and mVC24 between 0.76 and 0.80. Training on 40% of each clip gives results close to the full dataset. This suggests that, beyond this point, extra frames from the same clips add little new information, because consecutive frames are highly redundant.

Figure 7. Effect of training data size on segmentation accuracy (mDice, HD95) and temporal consistency (mVC12, mVC24). The x-axis shows the fraction of frames from each training clip used for training and "First 5 frames" corresponds to five annotated frames per clip.
Generalization
Zero-shot transfer
Table 8 compares ATLAS applied zero-shot to Cholec-Seg8k (Hong et al., [10]), without any training on Cholec-Seg8k data, with the same model trained from SurgeNet initialization on the CholecSeg8k training set. The zero-shot model reaches the same Dice (0.83) and a slightly lower
| Setup | Dice ↑ | HD95 ↓ | mVC12 ↑ | mVC24 ↑ |
| Zero-shot | 0.83 ± 0.08 | 0.12 ± 0.06 | 0.94 ± 0.03 | 0.92 ± 0.03 |
| SurgeNet | 0.83 ± 0.10 | 0.13 ± 0.08 | 0.96 ± 0.02 | 0.95 ± 0.02 |
Table 8. Zero-shot transfer of ATLAS to CholecSeg8k compared with training with SurgeNet initialization on CholecSeg8k.
HD95 (0.12 vs. 0.13). Temporal consistency is slightly lower than for the model trained on CholecSeg8k (0.94 vs. 0.96 mVC12 and 0.92 vs. 0.95 mVC24). A model trained on ATLAS-120k thus matches the segmentation accuracy of a dedicated model on an unseen dataset, at a small cost in temporal consistency.
Transfer to surgical phase recognition
Table 9 reports phase recognition on AutoLaparo when the model is initialized from SurgeNet or from ATLAS weights. ATLAS pretraining improves all metrics. Accuracy increases from 0.82 to 0.85, F1 by 0.03, and balanced accuracy and Jaccard by 0.04.
| Dataset | Init | Acc ↑ | Bal. Acc ↑ | F1 ↑ | Jaccard ↑ |
| AutoLaparo | SurgeNet | 0.82 ± 0.09 | 0.73 ± 0.07 | 0.73 ± 0.08 | 0.63 ± 0.09 |
| AutoLaparo | ATLAS | 0.85 ± 0.07 | 0.77 ± 0.08 | 0.76 ± 0.09 | 0.67 ± 0.10 |
Table 9. Surgical phase recognition on AutoLaparo when initializing from SurgeNet versus from ATLAS pretraining.
Discussion
Main findings
This work introduced ATLAS-120k, a multi-procedure dataset for surgical anatomy segmentation, and ATLAS, a video segmentation model that conditions its predictions on procedure and phase context. ATLAS outperformed all compared methods on both segmentation accuracy and temporal consistency. The gains did not depend on model size alone: ATLAS ViT-B outperformed SAM3-UNet with about a fifth of the parameters, and at matched backbone size ATLAS ViT-B improved Dice by 0.10 over EndoViT. The ablations show that each component addresses a different part of the problem. In-domain pretraining improved all metrics, query propagation mainly improved temporal consistency, and the phase queries mainly improved accuracy and boundary placement.
Role of procedural context
The phase queries had the largest effect on HD95 of all components, while leaving temporal consistency unchanged. This fits the motivation for the model. In a dataset with 14 procedures, which structures can appear in a frame depends strongly on what is being done and how far the operation has progressed. Context reduces this uncertainty, and the effect is most visible in HD95, which penalizes predictions of structures that are absent and failures to detect structures that are present. The phase queries are trained without phase labels, so this benefit does not require additional annotation and can be extended to procedures for which no phase definitions exist.
The procedure queries contributed less. One explanation is that the procedure is relatively easy to infer from the visual content, so the segmentation queries may already capture much of this information through the patch tokens. In clinical practice the procedure is known before surgery, so it could also be supplied as an input rather than predicted. We kept it as a learned query so that the model does not depend on external metadata, but providing it directly may be a simpler alternative in deployment.
Temporal consistency
Query propagation was the main source of temporal consistency, but its benefit was limited to short training clips. Training on more than five frames did not further improve consistency and worsened HD95, possibly because errors in the propagated queries accumulate over more steps during training. The comparison with the annotators points in the same direction: the gap was 0.11 for mVC12 but 0.15 for mVC24. Propagating queries from only the previous frame thus keeps predictions stable over short horizons, but not over longer ones. A memory that spans more frames, or that is updated selectively based on prediction confidence, may close part of this gap.
Pretraining and architecture
In-domain pretraining was beneficial but not sufficient. EndoFM and GastroNet5M, both pretrained on endoscopic data, performed worse than DINOv3, and EndoViT and the SurgeNet models were the strongest baselines. The pretraining data and objective therefore matter more than whether the data are endoscopic. For DINOv2 and DINOv3, scaling from ViT-B to ViT-L did not help, which suggests that a linear head limits how much of the encoder’s representation can be used for dense prediction. ATLAS avoids a separate decoder as well, but uses the encoder itself to produce masks through the segmentation queries.
At the smallest scale, the SurgeNet models outperformed ATLAS ViT-S. These models use hierarchical backbones with multi-scale features and a dedicated decoder, which may matter more when the encoder has little capacity. The per-class results point to a related limitation. The lowest Dice was found for thin structures such as nerves, arteries, ducts and airways. With an input resolution of 256 × 256 and a patch size of 16, each patch covers a region that can be wider than these structures. Higher input resolutions or multi-scale features may therefore be needed for the structures that matter most for surgical safety.
Dataset and annotation
The inter-observer study indicates both how reliable the annotations are and what level of performance can be expected on ATLAS-120k. The annotators agreed with a Dice of 0.85 and were highly consistent over time (0.96 mVC₁₂), so the annotations are reproducible, although not exactly. The remaining disagreement likely reflects structures whose extent is ambiguous in the endoscopic view, such as fat and connective tissue. ATLAS ViT-L remained 0.15 Dice below the annotators, leaving substantial room for improvement. The annotation process also gives practical guidance for future datasets. Fine-tuning the tracker on 25% of the data gave almost all of the improvement in propagation quality, and training ATLAS on 40% of each clip gave results close to using all frames. Because consecutive frames are highly redundant, annotating more clips sparsely is likely a more efficient use of annotation effort than annotating fewer clips densely.
The standard deviations in Table 3 were large for all models (0.12–0.16 Dice) and of similar size across models. They reflect differences between test clips rather than uncertainty in the mean. ATLAS-120k covers 14 procedures, and clips differ in which structures are visible and in how large and how clearly visible these structures are. Clips dominated by thin or rare structures are likely harder for every model than clips dominated by large organs, which is consistent with the per-class results. Mean scores on ATLAS-120k should therefore be read together with the per-class results, and differences between models are better judged with paired comparisons over the same clips than from the spread of each model alone.
Generalization
Applied zero-shot to CholecSeg8k, ATLAS matched the Dice of a model trained on that dataset. This suggests that a broad multi-procedure dataset can replace procedure-specific training for at least some procedures. Cholecystectomy is, however, the procedure with the most data in ATLAS-120k, so zero-shot performance on less represented procedures is likely lower. The improvement in phase recognition on AutoLaparo indicates that training on ATLAS-120k procedures features that also carry information about surgical workflow, not only about anatomy.
Limitations
This study has several limitations. All videos were taken from a single source, the GSViT dataset, and come with little information on patients, centers or recording equipment. Performance on other institutions and systems therefore remains to be confirmed. The class distribution is long-tailed, and several rare classes occur in only a few test clips, so their per-class results are uncertain. Some clinically distinct structures were merged into broader classes because they are hard to tell apart in the endoscopic view, which limits the level of anatomical detail the model provides. Most clips are shorter than 30 seconds, so temporal consistency was evaluated only over short horizons and not over full procedures. Finally, segmentation quality was evaluated with technical metrics only; whether the predictions are accurate enough to support surgeons during an operation was not assessed.
Conclusion
We introduced ATLAS-120k, a dataset of 121,018 densely annotated frames covering 14 laparoscopic and robot-assisted procedures, and ATLAS, a video segmentation model that combines procedure and phase context queries with query propagation between frames. ATLAS outperformed natural-image and in-domain foundation models on both segmentation accuracy and temporal consistency, matched a dedicated model zero-shot on CholecSeg8k, and runs well above real-time frame rates. It still falls 0.15 Dice short of inter-observer agreement, and thin structures such as nerves, arteries and ducts remain the hardest to segment. We release the ATLAS-120k annotations and the ATLAS model weights to support further work toward reliable anatomy recognition during surgery.
CRediT authorship contribution statement
Ronald L. P. D. de Jong: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Visualization, Writing – original draft, Writing – review & editing. Tim J. M. Jaspers: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Writing – review & editing. Raf A. H. Vervoort: Data curation, Writing – review & editing. Aron F. H. A. Bakker: Data curation, Writing – review & editing. Anna L. Dijk: Data curation, Writing – review & editing. Yiping Li: Methodology, Writing – review & editing. Jip L. Tolenaar: Data curation, Writing – review & editing. Jelle P. Ruurda: Data curation, Funding acquisition, Project administration, Writing – review & editing. Willem M. Brinkman: Data curation, Writing – review & editing. Daan de Geus: Methodology, Writing – review & editing. Fons van der Sommen: Funding acquisition, Supervision, Writing – review & editing. Josien P. W. Pluim: Funding acquisition, Project administration, Supervision, Writing – review & editing. Marcel Breeuwer: Project administration, Supervision, Writing – review & editing.
Declaration of competing interest
Ronald L. P. D. de Jong reports financial support was provided by Nederlandse Organisatie voor Wetenschappelijk Onderzoek Toegepaste en Technische wetenschappen. Yiping Li reports financial support was provided by Hanarth Fund Foundation. Jelle P. Ruurda reports financial support was provided by Intuitive Surgical Inc. Jelle P. Ruurda reports financial support was provided by Medtronic Inc. Jelle P. Ruurda reports financial support was provided by Olympus Corporation. Josien P. W. Pluim is one of the editors of the Medical Image Analysis Journal. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Data availability
The ATLAS-120k annotations, model weights and annotation tool are publicly available at https://github.com/TimJaspers0801/ATLAS. The source videos are part of the GSViT dataset [26].
Acknowledgments
This research was funded by Stichting Hanarth Fonds, study number 2022-13, and the Dutch Research Council (NWO), study number KICH1.ST03.21.019, grant ID https://doi.org/10.61686/NKMZR04566. It is part of the INTRA-SURGE (INTelligent computeR-Aided Surgical gUidance for Robot-assisted surGEry) project aimed at advancing the future of surgery. Furthermore, we thank SURF (www.surf.nl) for the support in using the Dutch National supercomputer Snellius.
Declaration of generative AI and AI-assisted technologies in the manuscript preparation process
During the preparation of this work the authors used Claude (Anthropic) to improve the readability and language of the manuscript and to assist in creating the schematic in Figure 4. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.
Dataset splits
Table 10 shows how the annotated classes are distributed over the training, validation and test splits. The splits contain 70, 10 and 20 videos, respectively, with 368, 39 and 95 clips and 84,113, 9,604 and 27,301 annotated frames. All 14 procedures occur in the training set, whereas the validation and test sets cover 6 and 9 procedures. As a result, the class distributions differ between splits. The most frequent classes, such as tools/camera, fat, abdominal wall and liver, occur in all splits. Classes related to cholecystectomy, such as gallbladder and cystic duct, are more common in the validation and test sets than in training. Most procedure-specific classes, for example those of prostatectomy, occur in only one or two test videos. All evaluated classes are present in the training set.
| Annotated class | Evaluated as | Train % frames | Train Videos | Val % frames | Val Videos | Test % frames | Test Videos |
| Tools/camera | 97.9 | 70 | 98.6 | 10 | 99.1 | 20 | |
| Omentum | Fat | 41.4 | 38 | 60.0 | 7 | 57.4 | 14 |
| Abdominal wall | 42.2 | 43 | 42.2 | 8 | 48.4 | 12 | |
| Liver | 31.3 | 29 | 56.8 | 7 | 50.8 | 12 | |
| Colon/rectum | 32.4 | 26 | 11.4 | 3 | 27.3 | 8 | |
| Non-anatomical | 28.8 | 35 | 16.6 | 6 | 18.7 | 14 | |
| Gallbladder | 17.1 | 24 | 46.1 | 5 | 44.1 | 10 | |
| Small intestine | 29.9 | 27 | 14.8 | 6 | 8.4 | 7 | |
| Stomach | 18.7 | 15 | 17.3 | 3 | 16.1 | 8 | |
| Mesentery | Fat | 20.1 | 14 | 7.1 | 2 | 6.6 | 2 |
| Artery | 16.0 | 19 | 14.6 | 3 | 14.8 | 12 | |
| Spleen | 6.4 | 4 | 15.2 | 1 | 6.3 | 1 | |
| Cystic duct | Bile duct | 3.2 | 10 | 21.7 | 4 | 13.6 | 8 |
| Vein | 9.2 | 23 | <0.1 | 1 | 1.0 | 5 | |
| Azygos vein | Vein | 5.5 | 8 | 11.5 | 1 | 5.6 | 2 |
| Lung | 4.8 | 8 | 10.3 | 1 | 5.8 | 2 | |
| Prostate | 3.6 | 6 | 2.1 | 1 | 8.8 | 2 | |
| Duodenum | – | 5.9 | 4 | – | – | – | – |
| Seminal vesicle | 3.1 | 4 | – | – | 4.6 | 1 | |
| Pericardium | 1.6 | 6 | 9.6 | 1 | 5.2 | 2 | |
| Hepatic ligament | 0.9 | 5 | 13.1 | 4 | 5.9 | 4 | |
| Mesocolon | – | 4.3 | 4 | <0.1 | 1 | – | – |
| Pancreas | – | 4.2 | 2 | – | – | – | – |
| Urethra | 2.1 | 3 | 4.8 | 1 | 3.2 | 2 | |
| Fallopian tubes | 2.0 | 3 | 1.4 | 1 | 2.9 | 1 | |
| Ligated plexus | 2.0 | 3 | 3.1 | 1 | 1.9 | 1 | |
| Ovary | 2.0 | 3 | – | – | 2.6 | 1 | |
| Major nerve | 2.1 | 6 | – | – | 1.5 | 1 | |
| Uterus | 1.6 | 3 | 1.4 | 1 | 2.5 | 1 | |
| Bladder | 1.3 | 4 | 3.8 | 1 | 2.5 | 2 | |
| Cystic plate | 1.1 | 5 | 3.1 | 3 | 3.3 | 4 | |
| Diaphragm | – | – | – | 7.0 | 4 | 5.2 | 5 |
| Trachea/bronchus | 1.5 | 5 | 1.6 | 1 | 1.9 | 1 | |
| Aorta | Artery | 1.8 | 4 | – | – | <0.1 | 1 |
| Vena cava | Vein | 1.5 | 3 | – | – | – | – |
| Catheter | Non-anatomical | 0.8 | 3 | 1.9 | 1 | 1.1 | 1 |
| Adrenal gland | – | 0.9 | 1 | – | – | – | – |
| Common bile duct | Bile duct | 0.7 | 2 | – | – | 0.5 | 1 |
| Kidney | – | 0.5 | 2 | – | – | – | – |
| Esophagus | – | 0.4 | 2 | – | – | 0.2 | 1 |
| Hepatic duct | Bile duct | 0.5 | 1 | – | – | – | – |
Table 10. Class distribution over the training, validation and test splits of ATLAS-120k, sorted by the total number of annotated frames. For each split, the percentage of annotated frames in which the class occurs and the number of videos that contain it are given. Classes that were merged into a parent class for training and evaluation list that parent class in the second column; classes marked – were relabeled as background.
References
- [1]Allan, M., Shvets, A., Kurmann, T., et al., 2019. 2017 robotic instrument segmentation challenge. URL: https://arxiv.org/abs/1902.06426, arXiv:1902.06426.
- [2]Batić, D., Holm, F., Özsoy, E., Czempiel, T., Navab, N., 2024. EndoViT: pretraining vision transformers on a large collection of endoscopic images. International Journal of Computer Assisted Radiology and Surgery 19, 1085–1091. URL: https://doi.org/10.1007/s11548-024-03091-5, doi:10.1007/s11548-024-03091-5.
- [3]den Boer, R.B., Jaspers, T.J.M., de Jong, C., et al., 2023. Deep learning-based recognition of key anatomical structures during robot-assisted minimally invasive esophagectomy. Surgical Endoscopy 37, 5164–5175. doi:10.1007/s00464-023-09990-z.DOI
- [4]Carion, N., Gustafson, L., Hu, Y.T., Debnath, S., Hu, R., Suris Coll-Vinent, D., Ryali, C., Alwala, K.V., Khedr, H., Huang, A., et al., 2026. Sam 3: Segment anything with concepts, in: International conference on learning representations, pp. 138846–138923.arxiv.org/abs/2511.16719
- [5]Carstens, M., Rimmer, F.M., Bodenstedt, S., et al., 2023. The dresden surgical anatomy dataset for abdominal organ segmentation in surgical data science. Scientific Data 10, 3. doi:10.1038/s41597-022-01719-2.DOI
- [6]Che, C., Wang, C., Vercauteren, T., Tsoka, S., Garcia-Peraza-Herrera, L.C., 2026. Lemon: A large endoscopic monocular dataset and foundation model for perception in surgical settings, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 42659–42669.arxiv.org/abs/2503.19740
- [7]Cheng, H.K., Oh, S.W., Price, B., Lee, J.Y., Schwing, A., 2024. Putting the object back into video object segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3151–3161.arxiv.org/abs/2310.12982
- [8]Czempiel, T., Paschali, M., Keicher, M., Simson, W., Feussner, H., Kim, S.T., Navab, N., 2020. Tecno: Surgical phase recognition with multi-stage temporal convolutional networks, in: Martel, A.L., Abolmaesumi, P., Stoyanov, D., Mateus, D., Zuluaga, M.A., Zhou, S.K., Racoceanu, D., Joskowicz, L. (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, Springer International Publishing, Cham. pp. 343–352.DOI
- [9]Gao, X., Jin, Y., Long, Y., Dou, Q., Heng, P.A., 2021. Trans-svnet: Accurate phase recognition from surgical videos via hybrid embedding aggregation transformer, in: de Bruijne, M., Cattin, P.C., Cotin, S., Padoy, N., Speidel, S., Zheng, Y., Essert, C. (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2021, Springer International Publishing, Cham. pp. 593–603.arxiv.org/abs/2103.09712
- [10]Hong, W.Y., Kao, C.L., Kuo, Y.H., Wang, J.R., Chang, W.L., Shih, C.S., 2020. Cholecseg8k: A semantic segmentation dataset for laparoscopic cholecystectomy based on cholec80. URL: https://arxiv.org/abs/2012.12453, arXiv:2012.12453.
- [11]Jaspers, T.J.M., de Jong, R.L.P.D., Li, Y., Kusters, C.H.J., Bakker, F.H.A., van Jaarsveld, R.C., Kuiper, G.M., van Hillegersberg, R., Ruurda, J.P., Brinkman, W.M., Pluim, J.P.W., de With, P.H.N., Breuweer, M., Al Khalil, Y., van der Sommen, F., 2026. Scaling up self-supervised learning for improved surgical foundation models. Medical Image Analysis 108, 103873. doi:10.1016/j.media.2025.103873.DOI
- [12]Jong, M.R., Boers, T.G.W., Fockens, K.N., Jukema, J.B., Kusters, C.H.J., Jaspers, T.J.M., van Eijck van Heslinga, R.A.H., Slooter, F.C., Struynenberg, M.R., Bisschops, R., et al., 2026. Gastronet-5m: a multicenter dataset for developing foundation models in gastrointestinal endoscopy. Gastroenterology 170, 174–187.DOI
- [13]de Jong, R.L.P.D., Jaspers, T.J.M., Vervoort, R.A.H., Bakker, A.F.H.A., Li, Y., Tolenaar, J.L., Ruurda, J.P., Brinkman, W.M., Pluim, J.P.W., Breuweer, M., de Geus, D., van der Sommen, F., 2026a. Surgical anatomy recognition with context learning using foundation representations, in: Medical Image Computing and Computer Assisted Intervention – MICCAI 2026, Springer Nature Switzerland, Cham. pp. 531–541.arxiv.org/abs/2606.22124
- [14]de Jong, R.L.P.D., Li, Y., Jaspers, T.J.M., van Jaarsveld, R.C., Kuiper, G.M., Badaloni, F., van Hillegersberg, R., Ruurda, J.P., van der Sommen, F., Pluim, J.P.W., Breuweer, M., 2026b. Towards effective surgical representation learning with dino models, in: Huo, Y., Gao, M., Kuo, C.F., Jin, Y., Deng, R. (Eds.), Proceedings of the 9th International Conference on Medical Imaging with Deep Learning, PMLR. pp. 1065–1080. URL: https://proceedings.mlr.press/v315/jong26a.html.
- [15]Kerssies, T., Cavagnaro, N., Herman, A., Norouzi, N., Garcia, G., Leibe, B., Dubbelman, G., de Geus, D., 2025. Your ViT is Secretly an Image Segmentation Model, in: 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 25303–25313. doi:10.1109/CVPR52734.2025.02356.DOI
- [16]Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., Krishnan, D., 2020. Supervised contrastive learning. Advances in neural information processing systems 33, 18661–18673.arxiv.org/abs/2004.11362
- [17]Liu, H., Wang, Z., Kong, A.Y.W., Qin, G., Xu, Y., Low, C.H., Gao, M., Chan, L.Y.L., Jin, Y., 2026. Usingsam: A unified promptable model for reliable surgical video segmentation. URL: https://arxiv.org/abs/2604.03645, arXiv:2604.03645.
- [18]Maier-Hein, L., Eisenmann, M., Sarikaya, D., et al., 2022. Surgical data science – from concepts toward clinical translation. Medical Image Analysis 76, 102306. doi:10.1016/j.media.2021.102306.
- [19]Maier-Hein, L., Wagner, M., Ross, T., Reinke, A., Bobdenstedt, S., Full, P.M., Hempe, H., Mindroc-Filimon, D., Scholz, P., Tran, T.N., et al., 2021. Heidelberg colorectal data set for surgical data science in the sensor operating room. Scientific data 8, 101.
- [20]Mascagni, P., Alapatt, D., Murali, A., Vardazaryan, A., Garcia, A., Okamoto, N., Costamagna, G., Mutter, D., Marseaux, J., Dallemagne, B., Padoy, N., 2025. Endoscapes, a critical view of safety and surgical scene segmentation dataset for laparoscopic cholecystectomy. Scientific Data 12, 331. doi:10.1038/s41597-025-04642-4.
- [21]Mascagni, P., Vardazaryan, A., Alapatt, D., et al., 2022. Artificial intelligence for surgical safety: Automatic assessment of the critical view of safety in laparoscopic cholecystectomy using deep learning. Annals of Surgery 275, 955–961. doi:doi:10.1097/SLA.00000000000004351.DOI
- [22]Miao, J., Wei, Y., Wu, Y., Liang, C., Li, G., Yang, Y., 2021. Vspw: A large-scale dataset for video scene parsing in the wild, in: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4131–4141. doi:10.1109/CVPR46437.2021.00412.
- [23]Norouzi, N., Zulfikar, I.E., Cavagnaro, N., Kerssies, T., Leibe, B., Dubbelman, G., de Geus, D., 2026. Videomt: Your vit is secretly also a video segmentation model, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 35177–35186.
- [24]Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.Y., Li, S.W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., Bojanowski, P., 2024. DINOV2: Learning robust visual features without supervision. Transactions on Machine Learning Research URL: https://openreview.net/forum?id=a6SUt6zFt. featured Certification.
- [25]Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rä dle, R., Rolland, C., Gustafson, L., et al., 2025. Sam 2: Segment anything in images and videos, in: International Conference on Learning Representations, pp. 28085–28128.
- [26]Schmidgall, S., Kim, J.W., Jopling, J., Krieger, A., 2024. General surgery vision transformer: A video pre-trained foundation model for general surgery. URL: https://arxiv.org/abs/2403.05949, arXiv:2403.05949.
- [27]Siméoni, O., Vo, H.V., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khalidov, V., Szafraniec, M., Yi, S., Ramamonjisoa, M., Massa, F., Haziza, D., Wehrstedt, L., Wang, J., Darcet, T., Moutakanni, T., Sentana, L., Roberts, C., Vedaldi, A., Tolan, J., Brandt, J., Couprie, C., Mairal, J., Jégou, H., Labatut, P., Bojanowski, P., 2025. Dinov3. URL: https://arxiv.org/abs/2508.10104, arXiv:2508.10104.
- [28]Twinanda, A.P., Shehata, S., et al., 2017. Endonet: A deep architecture for recognition tasks on laparoscopic videos. IEEE Transactions on Medical Imaging 36, 86–97. doi:10.1109/TMI.2016.2593957.
- [29]Wang, Z., Liu, C., Zhang, S., Dou, Q., 2023. Foundation model for endoscopy video analysis via large-scale self-supervised pre-train, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 101–111.
- [30]Wang, Z., Lu, B., Long, Y., Zhong, F., Cheung, T.H., Dou, Q., Liu, Y., 2022. Autolaparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hysterectomy, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 486–496.
- [31]Xiong, X., Wu, Z., Lu, L., Xia, Y., 2025. Sam3-unet: Simplified adaptation of segment anything model 3. URL: https://arxiv.org/abs/2512.01789, arXiv:2512.01789.
- [32]Xiong, X., Wu, Z., Tan, S., Li, W., Tang, F., Chen, Y., Li, S., Ma, J., Li, G., 2026. Sam2-unet: Segment anything 2 makes strong encoder for natural and medical image segmentation. Visual Intelligence 4, 2.
- [33]Yin, M., Wang, F., Ye, X., Meng, Y., Fu, Z., 2025. Memory-augmented sam2 for training-free surgical video segmentation. URL: https://arxiv.org/abs/2507.09577, arXiv:2507.09577.
- [34]Zia, A., Bhattacharyya, K., Liu, X., et al., 2023. Surgical tool classification and localization: results and methods from the miccai 2022 surgtooloc challenge. URL: https://arxiv.org/abs/2305.07152, arXiv:2305.07152.



