Annotation-Free Representation Learning for Polyp Retrieval in Colonoscopy Videos

Introduction

Colonoscopy is the gold standard for the prevention and early detection of colorectal cancer [1]. Deep learning increasingly supports it through computer-aided polyp detection [2] and characterization [3]. Polyp retrieval aims to recover additional views of a target lesion from colonoscopy videos, starting from one or a few observations. Aggregating information across these complementary views could support polyp characterization and facilitate lesion assessment and reporting during post-procedure review [4, 5]. However, learning representations of polyp tracklets—sequences of consecutive detections of the same polyp—is challenging: different polyps may look similar, while the same polyp may vary substantially in appearance over time [6]. Representations must therefore be robust to appearance changes while remaining discriminative across polyps.

Supervised contrastive learning on polyp entity annotations, i.e., labels linking observations of the same polyp, yields strong tracklet representations [7]. Yet, such labels are costly to collect, as they require expert clinicians. Temporal self-supervised methods reduce this cost by deriving positive pairs from temporal continuity: by splitting tracklets into temporal segments [8, 9], by integrating textual reports through momentum contrast [10], or by sampling pseudo-positives among temporally close tracklets of the same procedure [11].

These approaches nevertheless rely on ground-truth bounding boxes to construct polyp-centered training inputs, providing explicit spatial supervision during representation learning. Such annotations are costly and time-consuming to obtain. They also restrict training to annotated polyp observations, excluding the large majority of each procedure, namely frames without visible polyps, which still carry useful information about normal colonic appearance.

In this work, we learn polyp tracklet representations directly from raw colonoscopy videos, without bounding boxes or polyp identity labels. Since tracklets can no longer be polyp-centered, we redefine them as sequences of consecutive frames. However, our experiments show that global frame-level views underperform because of the lack of locality. Therefore, we adopt a temporally consistent random cropping strategy that restores locality without annotations. This enables temporal self-supervision on full-procedure videos and yields state-of-the-art results on REAL-Colon for tracklet retrieval and re-identification. We then introduce few-shot polyp retrieval, which retrieves the remaining tracklets of a lesion from a few examples, as in post-procedure review. With only two support tracklets, our method outperforms most baselines that use ten.

Our contributions are: (i) an annotation-free framework that achieves state-of-the-art results by learning directly from raw full-procedure videos; (ii) a novel few-shot polyp retrieval protocol; (iii) analyses of the trade-off between spatial supervision and data scale, and of cross-cohort robustness.

Annotation-Free Representation Learning Framework

We train a visual encoder by contrastive learning without the two sources of supervision of polyp-centric settings: ground-truth polyp detections and polyp entity annotations. Temporally consistent crops of raw frames recover the locality of polyp-centered views (§ 2.1), extending training to full-procedure videos, including frames without visible polyps (§ 2.2). Pseudo-positive associations derived from temporal proximity within each video replace polyp entity labels (§ 2.3), enabling annotation-free representation learning.

Annotation-Free Tracklet Views

Polyp-centric frameworks build each training sample from ground-truth polyp detections. Without them, we redefine a tracklet as a short sequence of consecutive frames from the same video, whether or not a polyp is visible.

The most direct view is then a sequence of full frames, which removes the spatial prior that focused the encoder on the polyp. However, this full-frame baseline loses 15% (relative) with respect to the crop-based state-of-the-art trained with ground-truth detections. Full frames shift the visual domain toward global endoscopic views with large background regions, instruments, specular highlights, and blur.

We therefore keep local views but sample them without detections: a random crop is drawn once per tracklet and applied to all its frames, preserving spatial coherence. This yields a 27% relative improvement over full frames, surpasses the crop-based state-of-the-art by 8%, and performs comparably to a synthetic box crop based on dataset-level bounding-box statistics, which, unlike ours, requires annotations (Table 2).

Training setup				Tracklet Retrieval		Tracklet ReID
Annot.	Loss	Views	Full	mAP	HR@1	AUROC	AUPR
ID+Box	SupCon	BBox	$\times$	57.52	84.24	94.33	43.81
Box	Noise-Aw.	BBox	$\checkmark$	61.49	91.17	93.89	51.78
Stats	Noise-Aw.	Synth.	$\checkmark$	65.60	91.71	95.14	57.72
–	Noise-Aw.	Frame	$\checkmark$	52.27	84.10	89.24	43.74
–	SupCon	Crop	$\checkmark$	64.31	89.54	94.97	49.83
–	Noise-Aw.	Crop	$\checkmark$	66.47	92.80	95.03	54.91

Full-Procedure Training

Since views no longer depend on detections, training is not confined to annotated polyp tracklets and extends to raw full-procedure videos, including the mucosal patterns, lumen views, and other colonoscopy-specific content that polyp-centric methods discard.

We build the training set from REAL-Colon, which contains 60 full-procedure videos from four acquisition cohorts. We follow the split of [11], but retain the full content of its 27 training videos with polyps and add the 14 videos without visible polyps that it omitted. The resulting 41-video training set, a superset of the previously used annotated data, contains 1,725,537 frames, of which only 171,957 (approximately 10%) show at least one polyp.

Temporal Self-Supervision

We optimize the tracklet encoder with the temporal self-supervised framework of [11], which derives pseudo-positive associations from the temporal structure of colonoscopy videos rather than from polyp identity labels. We leave the objective unchanged and modify only its input domain: tracklets are no longer required to contain a polyp, which allows training on full-procedure videos.

Formally, each training tracklet $x_i$ has a source video $v_i$ and a temporal position $p_i$. For an anchor $a_i$, the candidate pseudo-positives are the tracklets of the same video ($v_j = v_i$), ranked by temporal distance. We sample $\kappa$ of them without replacement from an exponential distribution over ranks with temperature $\tau$, obtaining a bag $B_i = \{b_{ik}\}_{k=1}^{\kappa}$; a curriculum increases $\tau$ in training, from close to broader context.

We minimize the noise-aware contrastive loss [11]:

where $y_i = f_\theta(a_i)$ and $z_{ik} = f_\theta(b_{ik})$ are the encoder representations of the anchor and of the $k$-th element of its bag, and $s(\cdot,\cdot)$ is a similarity function. Summing over the bag lets the model weight the most relevant pseudo-positives instead of forcing all temporal neighbors to match the anchor equally.

Experimental Results

Dataset and Downstream Setup

We use REAL-Colon [19], which contains 60 full-procedure colonoscopy videos from four acquisition cohorts recorded at six clinical centers on three continents with equipment from two manufacturers. We follow the setup of [11]. Training uses 41 videos without any annotation (§ 2.2); evaluation uses the 19-video test set (47 polyps), which contains only tracklets with visible polyps.

Tracklet retrieval and ReID Following [11], each test tracklet is used in turn as a query, and the remaining tracklets are ranked by the cosine similarity of their embeddings to it; we report mAP and HR@1, the fraction of queries whose top-ranked tracklet depicts the same polyp. For re-identification, the cosine similarity of each tracklet pair is thresholded to separate same-polyp from different-polyp pairs, and performance is summarized by AUROC and AUPR.

Few-shot polyp retrieval Tracklet retrieval treats each tracklet as an independent query, whereas in clinical review an expert may hold one or a few observations of a polyp and search for its other appearances. We therefore retrieve at the polyp level: for each target polyp, we randomly sample a support set $S = \{s_1, \ldots, s_K\}$ of $K \in\{1, 2, 3, 5, 10\}$ of its tracklets and rank each gallery tracklet $g \in\mathcal{G}$ by $\operatorname{sim}(g,S) = \max_{s_i \in S} \cos(f_\theta(g), f_\theta(s_i))$, keeping the support tracklets as separate exemplars. Random sampling models an average review scenario rather than a curated selection; we repeat it $R = 50$ times and report the mean mAP.

Method	Params (M)	FLOPs (G)	Tracklet Retrieval mAP (↑)	Tracklet Retrieval HR@1 (↑)	Tracklet ReID AUROC (↑)	Tracklet ReID AUPR (↑)	Few-Shot Polyp Retrieval $K=1$	Few-Shot Polyp Retrieval $K=2$	Few-Shot Polyp Retrieval $K=3$	Few-Shot Polyp Retrieval $K=5$	Few-Shot Polyp Retrieval $K=10$
General Purpose FM
DinoV2-S/14 [12]	22.1	44.2	47.31	89.81	80.07	29.77	47.91	57.04	62.73	65.64	72.57
DinoV2-g/14 [12]	1136	2331	48.29	90.35	78.23	27.04	48.08	57.20	62.40	66.46	72.13
DinoV3-S/16 [13]	21.6	34.7	48.95	88.99	80.47	31.77	47.60	57.36	62.95	66.80	76.59
V-JEPA2 [14]	326.0	272.1	29.01	70.79	69.84	17.14	22.29	28.46	32.70	33.38	39.93
Endoscopy FM
EndoFM [15]	121.3	190.1	45.75	87.64	79.71	31.91	45.12	56.28	61.80	65.46	73.03
EndoFM-LV [16]	121.3	190.1	28.21	63.18	67.56	19.28	22.54	28.22	28.92	30.95	38.22
EndoViT [17]	86.6	134.9	31.12	68.34	68.26	17.51	28.18	35.60	38.68	40.77	47.77
SurgeNet [18]	24.3	31.0	44.75	85.87	75.86	25.23	47.14	58.25	63.37	68.18	75.87
Fully supervised
Sup [7]	25.6	33.1	54.72	81.93	93.85	41.58	50.60	55.75	58.23	59.60	63.22
TA [7]	25.6	33.1	57.52	84.24	94.33	43.81	53.12	57.94	60.49	62.33	68.52
Weakly supervised
MVE [8]	25.6	33.1	42.05	81.25	79.73	27.47	48.41	56.25	58.47	60.58	65.89
TPC [9]	25.6	33.1	43.25	75.82	78.40	27.58	51.15	59.46	63.00	65.51	69.90
NTSSL [11]	25.6	33.1	61.49	91.17	93.89	51.78	59.24	67.64	71.36	74.69	81.34
Self-supervised
MVE [8]	25.6	33.1	46.22	80.43	89.74	31.93	44.69	51.67	54.03	54.62	59.30
TPC [9]	25.6	33.1	49.94	84.65	91.60	35.66	46.96	54.73	57.49	58.79	62.83
Ours	25.6	33.1	66.47	92.80	95.03	54.91	63.46	73.10	75.58	78.80	85.72

Implementation Details

The tracklet encoder follows prior work [8, 9, 7, 11]: an ImageNet-pretrained ResNet-50 [20] extracts frame features, which a three-layer transformer encoder (8 heads, $d_{\mathrm{model}} = 256$) aggregates and a two-layer MLP ($256 \to128$) projects for contrastive learning. We use $\kappa= 4$ and a batch size of 60. The curriculum schedules $\tau$ in the exponential distribution from $\tau_{\min} = 0.3$ to $\tau_{\max} = 12$, using a cosine scheduler [11]. Videos are temporally subsampled by a factor of four, and a tracklet spans $L = 8$ subsampled frames (about one second); views are random resized crops (scale range (0.08, 1), $232 \times232$ pixels) sampled once per tracklet, with no further augmentation. We train for 25 epochs with AdamW (learning rate $10^{-5}$, weight decay $10^{-4}$) on a single NVIDIA A100 GPU. At test time, as in [9, 7, 11], tracklets are built by linking ground-truth detections with an IoU of at least 0.1, each enlarged to $5\times$ the bounding-box diagonal.

Comparison with State-of-the-Art

Tab. 1 compares our approach with the state-of-the-art. Self-supervised baselines are re-trained on our training set, while weakly and fully supervised baselines use its annotated subset. Our framework outperforms previous self-supervised approaches by 33.10% in retrieval mAP and by 3.74% and 53.98% in ReID AUROC and AUPR (relative). MVE [8] and TPC [9] draw positive views from distinct segments of a tracklet, presuming a polyp that stays visible throughout, which rarely holds for generic frame sequences.

Our method also surpasses NTSSL [11], a weakly supervised method that shares our objective but relies on bounding boxes, and the fully supervised baselines on all metrics; against TA [7], our method slightly improves AUROC and improves AUPR by 25.34%, i.e., with fewer false positives. Foundation models give strong generic features but are not optimized for the temporal, low-semantics structure of colonoscopy; endoscopy-specific ones rarely beat general-purpose ones, consistent with their smaller scale and surgical training data.

Few-shot polyp retrieval All methods benefit from additional support observations, confirming that single views of a polyp are ambiguous. Moving from $K = 1$ to $K = 2$ raises our mAP from 63.46 to 73.10, already exceeds the $K = 10$ performance of nearly all baselines; the only exceptions are DinoV3-S/16, SurgeNet, and NTSSL, which uses ground-truth boxes during training. With $K = 10$, our method reaches 85.72 mAP.

Supervision vs. Scaling

We next ask how much unlabeled full-procedure video is needed to offset the removal of spatial supervision. We train on nested subsets of the 41-video training split (4, 8, 16, 24, 32, and 41 videos), with five independent subsets per size to average out the choice of videos. Fig. 1 compares the results with NTSSL [11], which uses ground-truth boxes on the full training set.

Performance grows steadily with the number of videos: the annotation-free model matches NTSSL at around 24 videos on retrieval and AUPR, and at 32 videos on AUROC, and keeps improving up to the full 41-video set. In line with scaling results on natural images [21], more unlabeled videos can thus be more effective than fewer spatially annotated polyp observations. This break-even point is specific to REAL-Colon and should not be read as a universal threshold.

Robustness to Cohort Removal

We assess robustness to unseen acquisition domains by re-training our model from scratch with each of the four REAL-Colon cohorts withheld in turn (leave-one-cohort-out, LOCO) and evaluating it on the unchanged test set. As a matched control, we use the 32-video random subsets of § 3.4, which remove a comparable amount of data drawn from all cohorts.

In Figure 2, LOCO averages 62.90 mAP, above NTSSL (61.49), which uses ground-truth boxes and every cohort, and close to the control (64.11). Re-identification is slightly more sensitive (92.32 against 93.95 AUROC for the control). Withholding a cohort thus costs about as much as removing the same amount of data, and in mAP it still surpasses NTSSL trained on all cohorts, suggesting resilience to cohort removal.

Ablation Study

Table 2 traces the path from polyp-centric supervised learning to our annotation-free setting. Replacing identity labels with temporal self-supervision and the noise-aware loss raises retrieval mAP from 57.52 to 61.49 and ReID AUPR from 43.81 to 51.78, showing that temporal associations are an effective supervisory signal when their noise is addressed. Switching to random crops on full-procedure videos with the same loss improves every metric further, reaching 66.47 mAP and 54.91

AUPR. A supervised contrastive loss over the same temporal bags remains competitive (64.31 mAP), so the temporal sampling itself provides useful structure, but it lowers HR@1 and AUPR. Full frames perform worst on every metric, while synthetic boxes, which rely on annotation statistics, trade lower retrieval scores for higher ReID AUPR.

Conclusion

We presented an annotation-free framework that learns polyp tracklet representations from raw full-procedure colonoscopy videos. Without bounding-box or identity annotations, it achieves state-of-the-art results on REAL-Colon for tracklet retrieval, re-identification, and few-shot polyp retrieval, using only a modest number of unlabeled procedures. Validation is currently confined to REAL-Colon; although it spans six centers on three continents and two manufacturers, it is the only public full-procedure dataset with persistent polyp identities, and we hope new datasets will enable broader validation.

Declarations

Ethics. This retrospective study uses open-access human subject data from REAL-Colon [19]; ethical approval was not required, as confirmed by the data license.

Competing interests. C.B. is affiliated with Cosmo Intelligent Medical Devices, developer of the GI Genius device.

Acknowledgments. We acknowledge ISCRA for awarding this project access to the LEONARDO supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CINECA (Italy).

AI disclosure. Claude (Anthropic) was used to improve the writing and to prepare the charts; the authors reviewed and take responsibility for all content.