betaFeedback

WorksComputer Science

Where is My Mug? Training-Free Framework for Personalized Image Segmentation

Abstract

Personalized Image Segmentation (PIS) aims to segment the specific object in a target image that matches a user-provided reference. Unlike category-based segmentation or few-shot segmentation, PIS must discriminate between multiple visually similar instances, a requirement central to deployed applications such as augmented reality, personalized content editing, robotics manipulation, inventory tracking, and fine-grained visual retrieval. We show that the strong numbers reported by existing PIS methods are largely an artifact of single-instance benchmarks: when evaluated on realistic scenes with several same-category look-alikes, prior point-prompt methods collapse, because the correct instance is frequently never surfaced as a prompt. We trace this to a recall bottleneck rather than a mask-quality one, and address it with a training-free, modular system. The pipeline first generates high-recall object proposals using an off-the-shelf class-agnostic detector, then matches each proposal to the reference using foundation-model features (e.g., BLIP-2), and finally refines the selected box with SAM. For robust matching we propose \textbf{top-KK Sinkhorn}, a patch-level distance that scores each candidate only on its most confidently matched patches; trimming the Sinkhorn dual potentials recovers the robustness of partial optimal transport at more than 100×100\times lower cost, keeping it inside the per-proposal candidate loop. To evaluate this regime, we introduce \datasetname, a realistic multi-instance benchmark. Across \datasetname, PerSeg, PerMIS, and DAVIS, our system surpasses PerSAM, P2SAM\mathrm{P}^2\mathrm{SAM}, and PDM by up to ∼ ⁣48\sim\!48 mIoU while matching their runtime and using fewer parameters, establishing a strong, deployable training-free baseline for real-world personalized segmentation.

Introduction

Personalized image segmentation (PIS) aims to segment the specific physical object in a target image that corresponds to a user-provided reference. This capability is increasingly important in deployed perception systems, including warehouse automation, assistive and in-home robotics, augmented reality, and large-scale product retrieval, where models must reliably distinguish one object among several visually similar instances. In these settings, incorrect instance localization directly leads to failure modes such as grasping the wrong item, retrieving the wrong product, or degrading the user experience.

Unlike semantic segmentation, which predicts category-level masks (e.g., “mug,” “chair”), PIS requires instance-identity recognition across changes in viewpoint, lighting, occlusion, and clutter. From a systems perspective, this task stresses the components most essential for real-world reliability: high-recall object proposals, robust instance-level matching, and stable mask generation. Prior works such as PerSAM [57] and PDM [42] introduced the concept, but existing benchmarks seldom include multiple same-class objects, limiting their ability to evaluate these critical system behaviors. As illustrated in Figure 1, practical deployments often require selecting a specific green ceramic mug from a group of nearly identical mugs, a task category-based or few-shot segmentation methods struggle to handle.

Figure 1. An example from our challenging PerMultiSeg dataset. Given a distinctive green ceramic mug in the reference image, the task is to accurately identify and segment the same mug among similar ones in the target image. While existing PIS methods such as PerSAM [57] fail, our method successfully distinguishes and segments the correct mug. In contrast, few-shot segmentation (FSS) methods would segment all mugs in the scene, without differentiating the specific instance.

Although PIS superficially resembles few-shot segmentation (FSS), the two differ fundamentally: FSS generalizes to new semantic classes from a few labeled examples [17, 34], whereas PIS must generalize to new instances of the same identity, relying on fine-grained visual cues rather than category-level semantics. This underpins use cases such as instance-specific retrieval [26, 35, 36], virtual try-on [11, 18, 20, 21, 23, 33, 51], and robotic manipulation in multi-object scenes [38, 39, 54], where FSS methods are not suitable replacements.

To meet these demands, we develop a training-free, modular system pipeline that emphasizes robustness and deployment efficiency. Our three-stage design consists of high-recall class-agnostic proposals, foundation-model-based reference–candidate matching, and a mask-generation module. First, class-agnostic detectors ensure that all plausible objects are surfaced; missing the correct instance at this stage is unrecoverable in downstream processing. Second, we perform matching using CLIP [40], DINOv2 [36], and other foundation-model features. Empirical analysis shows that naive feature averaging is brittle in cluttered scenes, and even patch-level matching with the standard Sinkhorn divergence [44] degrades when same-category look-alikes share most of their patch content: high-cost outlier patches (occluders, clutter, and non-matching regions) accumulate transport cost and blur the gap between the correct instance and its look-alikes. Robust optimal transport addresses exactly this outlier structure, but solving it directly is too slow to run inside a per-proposal candidate loop. We therefore propose top-KK Sinkhorn matching, which scores each candidate only on its most confidently matched patches by discarding the highest-cost patches before summing the transport cost. This recovers the robustness of partial optimal transport [4, 8, 15] at more than 100× lower cost (a single sort on top of the standard Sinkhorn pass), making inlier-restricted matching practical for deployment. Finally, the chosen proposal is refined into a segmentation mask. Because each component is modular and swappable without retraining, the system meets diverse hardware and latency budgets.

A major challenge is the lack of datasets that adequately evaluate system-level PIS performance. Existing datasets such as PerSeg [57], PerMIS [42], and DAVIS adaptations [37] typically feature only one same-class object per target image. While this leads to high reported mIoU (95–97%), these benchmarks do not stress the failure modes that dominate real-world performance: proposal recall, instance-level discrimination, and robustness to clutter. Thus, methods that perform well on these datasets often fail in realistic multi-object environments.

To more accurately assess system reliability, we introduce PerMultiSeg, a challenging multi-instance benchmark featuring scenes with 3–4 same-class objects on average, varied viewpoints, natural distractors, and occlusions, better reflecting the operational conditions of robotics, retail perception, and visual search systems.

We evaluate our system across PerSeg, DAVIS, PerMIS, and PerMultiSeg, measuring not only accuracy but also proposal recall, matching reliability, and efficiency trade-offs. Our approach achieves strong improvements, improving on the best prior method by 22.08 mIoU on PerMIS and by over 45 mIoU on PerMultiSeg, demonstrating that a recall-first system, paired with an efficient robust matcher, delivers reliable personalized segmentation in real environments.

Our contributions are:

  • A modular, training-free PIS system built around a recall-first design: high-recall class-agnostic proposals, foundation-model matching, and SAM refinement. Each module (detector, encoder, matcher) can be swapped independently to meet hardware or latency budgets without retraining, while outperforming prior PIS methods by up to ∼48 mIoU at comparable runtime.

  • Top-KK Sinkhorn matching, a robust patch-level dis-

tance that scores each candidate only on its most confidently matched patches via a single sort on the Sinkhorn dual potentials. To our knowledge it is the first use of inlier-restricted (partial) optimal transport as an instance-level discriminator in personalized segmentation; it recovers the robustness of partial optimal transport at more than 100× lower cost, making inlier-restricted matching practical inside the per-proposal candidate loop.

  • PerMultiSeg, a realistic multi-instance evaluation benchmark. Motivated by our finding that the high mIoU of existing methods is largely an artifact of single-instance benchmarks, where the dominant failure mode is proposal recall, not mask quality, PerMultiSeg contains 3–4 same-class objects per target image to stress-test instance-level discrimination that prior PIS benchmarks (PerSeg, PerMIS, DAVIS adaptations) do not capture.

Related Work

Vision Foundation Models. Recent advances in vision foundation models have shown remarkable generalization, making them highly adaptable for diverse tasks. For example, CLIP [40] uses contrastive learning on 400 million image-text pairs for zero-shot visual recognition. BLIP-2 [26] combines frozen image encoders and large language models (LLMs) with a lightweight transformer, achieving state-of-the-art results on vision-language tasks. DINOv2 [36] learns versatile visual representations via self-supervised learning at both image and patch levels, while SAM [24], pre-trained on one billion masks, enables promptable and highly accurate segmentation. Despite their versatility, these models struggle with personalized image segmentation, as they are not trained to segment the same object across images. In this paper, we propose a training-free framework that leverages off-the-shelf vision foundation models for one-shot personalized segmentation, enabling precise localization and segmentation of uniquely identified objects.

Image Segmentation. Image segmentation has progressed significantly across semantic, instance, panoptic, and few-shot segmentation tasks [6, 9, 27, 30, 34, 48], supported by advances in object proposal generation. However, most traditional approaches rely on extensive task-specific training and fine-tuning, limiting their ability to adapt to new scenarios without retraining. Recent class-agnostic segmentation models, such as SAM [24], achieve broad generalization through large-scale pretraining and prompt-based inference. Painter [52] frames vision tasks as image-to-image prediction using masked image modeling, while SegGPT [53] improves generalization via random colorization. SEEM [59] further expands generality by supporting spatial, visual, and text prompts within a single framework. Despite their versatility, these models are not designed for object-specific segmentation, where the objective is to identify and segment the exact same object instance across images. This fine-grained identity matching remains a key gap in existing foundation-model-based segmentation systems, one that our work directly addresses.

Personalized Image Segmentation. Prior works have explored personalized segmentation, but existing approaches struggle to robustly segment the same object instance with its unique identity. PerSAM [57] adapts SAM for personalized and video segmentation using either zero-shot inference or minimal fine-tuning, extracting features from reference and target images to produce confidence maps. P2^{2}SAM [58] improves robustness by filtering outlier prompts, while Matcher [29] combines class-agnostic segmentation with bidirectional matching for dense correspondence. PDM [42] instead leverages intermediate features from pre-trained text-to-image models to compute appearance and semantic similarity. These methods are largely evaluated on benchmarks with only a single same-category object per target image, which does not reflect real deployment settings.

Optimal Transport for Feature Matching. OT has been applied across vision tasks: DeepEMD [56] uses earth-mover’s distance on patch features for few-shot classification; OTAM [5] uses temporal alignment for few-shot video recognition; SuperGlue [43] and LoFTR [49] employ Sinkhorn and dual-softmax to obtain soft keypoint assignments. On the theoretical side, partial and robust OT relax marginal constraints to tolerate outliers [3, 4, 8, 15, 32]. Despite this literature, robust and partial OT have seen little use as an instance-level discriminator for personalized segmentation, where the matcher must run inside a per-proposal candidate loop. The closest work, Matcher [29], uses full balanced OT to score mask quality, not partial OT to discriminate a specific instance. Existing methods either aggregate over all pairs (DeepEMD, full Sinkhorn) or trim correspondences post-hoc (SuperGlue); Screenkhorn [1] screens dual variables before solving to accelerate exact OT. In contrast, our top-KK Sinkhorn trims the transport objective itself, recovering partial-OT-level robustness at a fraction of the cost. To our knowledge this is the first use of inlier-restricted OT as an instance-level discriminator in personalized segmentation.

Our PerMultiSeg Dataset

Existing PIS benchmarks are limited along two axes (Figure 2, Table 1): (i) they rarely contain multiple same-category instances in a single target: PerSeg [57] is essentially single-instance, while PerMIS [42] and DAVIS [37] are video-derived with minimal reference-to-target variation; (ii) they cover fewer than 100 categories at small scale. As stress tests for fine-grained instance discrimination, these benchmarks fall short.

Examples of existing PIS benchmarks

Figure 2. Examples of existing PIS benchmarks (left: reference, right: target). PerSeg has only one instance per target image; PerMIS [42] suffers from low-quality and near-duplicate frames (blue rectangle: targets too small to identify); DAVIS [37] exhibits minimal frame-to-frame variation. PerMultiSeg provides multiple same-category instances per target image (e.g., three mugs, three smartphones).

Dataset# Ref# Identities# Categories# Obj/Img
PerSeg [57]4040401.00
PerMIS [42]140140163.10
DAVIS [37]9191303.18
PerMultiSeg2,1471,377453.67

Table 1. Comparison of PerMultiSeg with prior PIS datasets.

To address these gaps, we introduce PerMultiSeg, a larger and more challenging PIS benchmark. Each target image contains multiple same-category objects (3.67 per image on average) paired with multiple references, and the dataset spans 45 categories collected across diverse environments, lighting conditions, and viewpoints. In total, PerMultiSeg provides 1,079 target images, 2,147 reference images, and 3,964 annotated instances. Each reference is manually matched to its counterpart in the target and annotated with a precise binary mask. Crucially, the distractors are selected to be the same product and same colour as the reference, that is, visually near-identical units that differ only in physical identity, pose, and wear; this removes category-and colour-level shortcuts and forces the matcher onto fine-grained localised evidence. Because the criterion needs near-identical units to co-occur naturally, categories skew toward portable manufactured goods and scenes toward indoor settings, which bounds external validity but not the comparisons, since all methods see identical pairs. Detailed dataset statistics, the collection and annotation protocol, selection criterion, and category-level breakdowns (Figure 5) are provided in the supp.

Classes distribution and instances per image for PerMultiSeg and COCO2017

Figure 5. Analysis of our PerMultiSeg dataset vs. MS COCO. (a) Class distribution (% of images per category). (b) Instances per image. PerMultiSeg shows a more balanced class distribution and a higher frequency of multi-object scenes than COCO.

Our Framework

Overall System

In multi-object scenes, failures in personalized segmentation arise from missed localization, not mask refinement: once the correct instance is absent from the proposal set, no downstream step recovers it. This is the bottleneck in point-prompt pipelines, whose SAM confidence maps miss the correct instance in clutter, so PerSAM [57] and PDM [42] degrade sharply on PerMultiSeg despite strong mask generators (low proposal recall, Table 2). We therefore adopt a recall-first, proposal-based framework [19] that decouples candidate generation from instance matching, in three stages:

PerSAM [57]PDM [42]P²SAM [58]Ours
Recall (%) ↑34.3330.6034.3389.55
Inf. time (s) ↓1.0726.291.441.02

Table 2. Proposal-stage recall (before matching) and end-to-end inference time on PerMultiSeg. Each method is evaluated under its native protocol (details in supp.). Our method achieves significantly higher recall, improving final segmentation.

  1. Class-agnostic object proposals: detect bounding boxes for all potential objects without using category labels (§ 4.2);

  2. Reference–candidate matching: identify the proposal that best corresponds to the reference object (§ 4.3);

  3. Mask refinement: prompt SAM with the selected box.

Decoupling these stages keeps each module replaceable under different latency or hardware budgets without retraining. The pipeline is illustrated in Figure 3; despite being training-free, it matches the runtime of PerSAM and P²SAM while substantially outperforming them (Table 2).

Figure 3. Given a reference and target image, we first use an object detector to identify all potential object candidates in the target image. These candidates, along with the reference image, are then passed through an image encoder to extract feature representations. Next, we determine the best-matching candidate by computing the distance between each candidate’s features and the reference features. Finally, the selected candidate’s bounding box is fed into SAM, which generates the final segmentation mask.

Class-agnostic Object Proposals

We capture a comprehensive set of object proposals without considering their classes, leveraging any object detector such as YOLO-World [10] or CoDet [31], both trained with the open vocabulary of Object365 [45] (Table 9). After this step, we obtain BB class-agnostic box proposals {bi}i=1B\{b_i\}_{i=1}^{B}, which achieve substantially higher proposal recall than prior PIS methods (Table 2).

MethodParamsPerMultiSegDAVIS2017
mIoU ↑\uparrowRecall ↑\uparrowmIoU ↑\uparrowRecall ↑\uparrow
SAM [24]641M95.5599.3880.8982.60
CODET [31]45M94.0998.5180.3783.79
YOLOW [10]47M85.9590.2490.0097.51
YOLOv11 [22]25M90.9796.2194.0998.51

Table 9. Ablation on object proposal methods. Recall is proposal-level (box IoU >0.5> 0.5) on the validation set.

Reference–Candidate Matching

Given the proposals, we extract features for each bib_i and the reference IrI_r using any vision foundation encoder [24, 25, 36, 40, 41] (compared in Table 6). When a [CLS] token is available we take it as the global feature v0∈Rd\mathbf{v}_0 \in\mathbb{R}^{d} and the remaining N=HWN = HW patch tokens as V∈RN×d\mathbf{V} \in\mathbb{R}^{N \times d}; the global distance dg(v0r,v0(i))d_g(\mathbf{v}_0^r,\mathbf{v}_0^{(i)}) is cosine distance.

EncoderCLS onlyMean poolTop-KKΔ\Delta vs CLS
BLIP-2 [26]83.8683.1687.97+4.11
SigLIP-2 [50]77.0284.5486.50+9.48
EVA-02 CLIP [13]71.6772.3275.69+4.02
DINOv2 [36]66.9670.9280.71+13.75
DINOv3 [47]65.2170.9478.26+13.05
CLIP [40]71.3574.7674.82+3.47

Table 6. Study on image encoder. The Top-KK column reports the best result per encoder from the KK sweep (Figure 7, supp.); Δ\Delta is vs. CLS-only.

Patch-level distance: limitations of full Sinkhorn. A natural patch-level distance is the entropic-regularised Sinkhorn divergence [44]

Sε(Vr,V(i))=OT⁡ε(Vr,V(i))−12OT⁡ε(Vr,Vr)−12OT⁡ε(V(i),V(i)),(1)\mathcal{S}_{\varepsilon}(\mathbf{V}^r,\mathbf{V}^{(i)}) = \operatorname{OT}_{\varepsilon}(\mathbf{V}^r,\mathbf{V}^{(i)}) - \frac{1}{2}\operatorname{OT}_{\varepsilon}(\mathbf{V}^r,\mathbf{V}^r) - \frac{1}{2}\operatorname{OT}_{\varepsilon}(\mathbf{V}^{(i)},\mathbf{V}^{(i)}), \tag*{(1)}

where OT⁡ε\operatorname{OT}_{\varepsilon} is the entropic OT cost [12] (full definition in supp.). In practice, we use only the cross term OT⁡ε(Vr,V(i))\operatorname{OT}_{\varepsilon}(\mathbf{V}^r,\mathbf{V}^{(i)}), since the de-biased self-terms add no discriminative signal for candidate ranking (verified empirically in supp., Table 11). However, even the cross term alone aggregates over all N×NN \times N patch pairs. In multi-instance PIS this is harmful: a large fraction of patches in Vr\mathbf{V}^r and V(i)\mathbf{V}^{(i)} correspond to background, partial occluders, or shared category structure common to all same-category candidates. These high-cost outlier patches accumulate transport cost roughly in proportion to their number, compressing the gap between the correct instance and its look-alikes. The result is that wrong candidates and the correct one yield comparable Sε\mathcal{S}_{\varepsilon} scores.

ConfigKKmIoUms/pair
Cross-only1.0085.048.3
Cross-only1/387.977.9
De-biased SϵS_{\epsilon}1.0085.0422.9
De-biased SϵS_{\epsilon}1/387.9722.7

Table 11. Self-term ablation: cross-only vs. de-biased SϵS_{\epsilon}.

Top-KK Sinkhorn matching. We exploit the inlier structure of the problem and score each candidate only on its most confidently matched patches. Let f⋆,g⋆∈RN\mathbf{f}^{\star},\mathbf{g}^{\star} \in\mathbb{R}^{N} be the Sinkhorn dual potentials of the cross term OT⁡ε(Vr,V(i))\operatorname{OT}_{\varepsilon}(\mathbf{V}^r,\mathbf{V}^{(i)}) and a,b\mathbf{a},\mathbf{b} the (uniform) marginal weights. By Sinkhorn duality [12],

OT⁡ε(Vr,V(i))=∑j=1Najfj⋆+∑k=1Nbkgk⋆,(2)\operatorname{OT}_{\varepsilon}(\mathbf{V}^r,\mathbf{V}^{(i)}) = \sum_{j=1}^{N} a_j f_j^{\star} + \sum_{k=1}^{N} b_k g_k^{\star}, \tag*{(2)}

so each per-patch term ajfj⋆a_j f_j^{\star} is the marginal cost of transporting source patch jj: it is small (often negative) for patches with a close match in V(i)\mathbf{V}^{(i)} and large for patches without, exactly the inlier/outlier structure our matcher relies on.

Given a keep-ratio K∈(0,1]K \in(0,1] (the fraction of patches we retain on each side, not an integer count, so that K=1/3K = 1/3 keeps the cleanest one-third), let M=⌊KN⌋M = \lfloor KN \rfloor and let (a⊙f⋆)(1)≤⋯≤(a⊙f⋆)(N)(\mathbf{a} \odot\mathbf{f}^{\star})_{(1)} \leq\cdots\leq(\mathbf{a} \odot\mathbf{f}^{\star})_{(N)} be the per-patch contributions sorted ascending (here (⋅)(m)(\cdot)_{(m)} denotes the mm-th order statistic, i.e. the mm-th smallest element). The top-KK Sinkhorn distance keeps only the smallest MM contributions on each side:

OT⁡^εK(Vr,V(i))=∑m=1M(a⊙f⋆)(m)+∑m=1M(b⊙g⋆)(m).(3)\widehat{\operatorname{OT}}_{\varepsilon}^{K}(\mathbf{V}^r,\mathbf{V}^{(i)}) = \sum_{m=1}^{M}(\mathbf{a}\odot\mathbf{f}^{\star})_{(m)} + \sum_{m=1}^{M}(\mathbf{b}\odot\mathbf{g}^{\star})_{(m)}. \tag*{(3)}

At K=1K = 1, Eq. (3) recovers full entropic OT; smaller KK progressively restricts the score to the cleanest inlier subset. This is motivated by partial optimal transport [4, 8, 15], which down-weights outlier mass to gain robustness; rather than solving the partial-OT program directly (which modifies the Sinkhorn iteration and is too slow for the per-proposal candidate loop), we trim the dual potentials of the full solve as a cheap surrogate, and verify empirically (Table 5) that it recovers partial-OT-level accuracy. We adopt

Table 5. Matcher ablation on the PerMultiSeg validation split (135 pairs). All rows use BLIP-2 + αp=0.75\alpha_{p} = 0.75 + the same frozen proposals (paper default). Top-KK Sinkhorn is indistinguishable from the partial-OT solver at this sample size, at a fraction of its latency.

MatchermIoU ↑Recall ↑bIoU ↑ms/pair ↓
Avg. feature map65.2966.4460.970.5
[CLS] token only83.8685.5878.770.07
Full Sinkhorn (K=1K=1)85.0486.8379.938
Sinkhorn + mutual-NN filter81.7082.9276.9210
Partial OT (m=1/31/3) [8]87.9789.8382.69848
Top-KK Sinkhorn (K=1/3K=1/3)87.9789.8382.698

Table 5.

dp(Vr,V(i))=OT~ϵK(Vr,V(i)),(4)d_{\mathrm{p}}(\mathbf{V}^{r}, \mathbf{V}^{(i)}) = \widetilde{\mathrm{OT}}_{\epsilon}^{K}\left(\mathbf{V}^{r}, \mathbf{V}^{(i)}\right), \tag*{(4)}

with K=1/3K = 1/3 throughout (trim-ratio sweep in supp., Table 8). Trimming is applied directly to the dual potentials returned by the Sinkhorn solver, so top-KK Sinkhorn requires only an O(Nlog⁡N)O(N \log N) sort per pair on top of the standard Sinkhorn forward pass; no modification of the iteration is needed. In practice this makes robust inlier-restricted matching more than 100× cheaper than a direct partial-OT solver (Table 5), cheap enough to remain inside the candidate loop at deployment.

KKmIoU ↑\uparrowRecall ↑\uparrowbIoU ↑\uparrow
0.1085.6087.3880.40
0.2086.2388.0581.03
0.2586.6788.5081.47
1/387.9789.8382.69
1/287.9789.8382.69
3/486.6788.5081.47
1 (full)85.0486.8379.93

Table 8. Study on the trim ratio KK in top-KK Sinkhorn (BLIP-2, αp=0.75\alpha_p = 0.75).

Candidate selection and segmentation. The best candidate combines patch- and global-level distances with weights αp,αg\alpha_{\mathrm{p}}, \alpha_{\mathrm{g}}:

i⋆=arg⁡min⁡i[αpdp(Vr,V(i))+αgdg(v0r,v0(i))].(5)i^{\star} = \arg\min_{i}\left[\alpha_{\mathrm{p}}d_{\mathrm{p}}\left(\mathbf{V}^{r}, \mathbf{V}^{(i)}\right) + \alpha_{\mathrm{g}}d_{\mathrm{g}}\left(\mathbf{v}_{0}^{r}, \mathbf{v}_{0}^{(i)}\right)\right]. \tag*{(5)}

The selected box bi⋆b_{i^{\star}} then prompts SAM [24] to produce the final segmentation mask.

Experiments

Experimental Setup

Datasets. We evaluate on PerSeg [57], DAVIS [37], PerMIS [42], and our proposed PerMultiSeg (§ 3). Component ablations use our validation set (135 reference–target pairs). For VOS, DAVIS uses the first frame as reference and subsequent frames as targets.

Metrics. Following [42, 57], we report mIoU and bIoU. On PerMultiSeg we additionally report Recall (detailed in supp.). For VOS we report J\mathcal{J}, F\mathcal{F}, and J&F\mathcal{J}\&\mathcal{F}.

Implementation details. All experiments use an A100 40GB GPU with PyTorch 2.6. Default configuration: BLIP-2 [26] (ViT-G/14 [13]) for feature extraction, YOLOv11 for detection (confidence 10−310^{-3}), SAM [24] (ViT-H) for mask generation. Weights: αp=0.75\alpha_{\mathrm{p}} = 0.75, αg=0.25\alpha_{\mathrm{g}} = 0.25. We solve the entropic OT via GeomLoss [14] and apply the top-KK Sinkhorn matching of § 4.3 with K=1/3K = 1/3. Following E5 (supp.), we use only the cross transport term, since the de-biased self-terms add no accuracy at 3×3\times the cost.

Baselines. PerSAM [57], PDM [42], P2^{2}SAM [58], all at default settings with the same SAM checkpoint. For P2^{2}SAM we report its official default configuration (supp., Sec. H.1).

Comparisons with Prior Approaches

Quantitative results. Table 3 shows our approach exceeds P2^{2}SAM by more than 45 mIoU on PerMultiSeg, confirming that existing methods fail in multi-instance scenes. Even the detector-free variant (SAM AMG) surpasses all baselines by a large margin. SAM2 and SAM3 target related but different tasks; augmenting SAM3 with Qwen2.5-VL caption closes much of the gap but still falls 10 mIoU short, confirming that PIS requires visual instance matching that text descriptions cannot replace. On PerMIS and DAVIS (Table 4), our method matches or exceeds all baselines.

Table 3. Comparison on the PerMultiSeg dataset. Recall is measured at the final-prediction stage: a method is counted as a hit if its output mask has IoU >0.5> 0.5 with the ground-truth mask (applied uniformly to all methods, regardless of prompt type). “Box Prompt” refers to using bounding box inputs for SAM, while “Point Prompt” denotes using point-based inputs.

MethodPromptBackbonemIoU ↑bIoU ↑Recall ↑Params ↓
PerSAM [57]PointSAM21.6514.8225.291.9B
PDM [42]PointSDXL-turbo23.6512.8243.912.6B
P2P^{2}SAM [58]PointSAM31.5827.9024.571.9B
SAM2‡ [41]1-shotSAM238.5233.1041.200.6B
SAM3‡ [7]ConceptSAM344.7340.9548.152.1B
SAM3 + Qwen2.5-VL‡ [2, 7]CaptionSAM3+Qwen70.1164.5771.099.1B
Ours (SAM detector)BoxSAM74.3368.3176.781.6B
Ours (YOLOv11 detector)BoxSAM80.0775.1582.291.6B

Table 3. ‡Address a different task (VOS/PCS: segment all matching instances, not one specific instance); shown for context, see text.

Table 4. Comparison on PerSeg, DAVIS, and PerMIS. PerSAM-F uses 10 s test-time fine-tuning per target.

MethodPerSegDAVIS2017PerMIS
mIoU ↑J&F ↑J ↑F ↑mIoU ↑
PerSAM [57]89.3066.9071.3075.10-
PerSAM-F† [57]95.3076.1074.9079.7016.50
Matcher [29]-73.5070.0077.50-
PDM [42]95.4075.8072.9080.1042.30
P2P^{2}SAM [58]95.70---43.57
Ours95.7578.6876.4480.9265.65

Table 4.

Qualitative results. Figure 4 shows representative results on PerMultiSeg. In successful cases (rows 1–4) the correct instance is precisely segmented despite multiple same-category distractors. The failure case (last row) occurs when the correct instance has its discriminative label hidden while a distractor displays a similar label facing the camera; any appearance-based matcher will prefer the more visible match. Supp. Figure 9 gives further examples; Tab. 17 reports a stage-wise error decomposition.

Qualitative comparison showing reference image, target image, PerSAM, PDM, P²SAM, Ours, and ground truth

Figure 4. Qualitative results on PerMultiSeg (rows 1–4: success; row 5: failure).

Hard cases on the PerMultiSeg test split

Figure 9. Hard cases on the PerMultiSeg test split. Each block shows reference, target, ground truth (green), and our prediction (red), labelled with category and per-pair IoU. Rows cover dense same-category clutter, partial occlusion, and near-duplicate instances.

On fair comparison. Our pipeline uses more pretrained components than the baselines, and uses box rather than point prompts; three controls show the gains are not solely explained by either. (i) At an identical prompt type, our pipeline with a center-point prompt (supp., Tab. 15) reaches 55.41 mIoU against 31.58 for P2P^{2}SAM, a +23.83 gap under the interface the baselines use, roughly half of the total +48.49. (ii) The Ours (SAM detector) row uses only SAM (same backbone as PerSAM, no external detector) and still achieves +42 mIoU over P2P^{2}SAM. (iii) With the encoder fixed, switching from a single global vector to patch-level OT yields large gains (e.g. +13 mIoU on DINOv2, CLS→patch-level OT), and top-KK Sinkhorn adds a further +2.9 mIoU over full Sinkhorn on BLIP-2 (Table 5).

Validating the Matcher Choice

All rows in Table 5 share the same frozen proposals, SAM checkpoint, and BLIP-2 encoder, differing only in the patch distance dpd_{p}, on the 135-pair validation split; the deployed last row scores 80.07 on test.

Robust matching at deployable cost. Top-KK Sinkhorn selects the same candidate as the principled partial-OT solver [8] on all 135 pairs, so their masks are identical, at more than 100× lower latency (8 vs. 848 ms per pair; supp., Sec. H.4). Against full Sinkhorn it gains +2.93 mIoU at the shared default αp=0.75\alpha_{p}=0.75 and +1.74 best-versus-best (supp., Table 12). Matching is only ∼1.5% of end-to-end latency (supp., Table 7), effectively free at deployment.

StageMean (ms)% of total
Object detection34.83.4%
Feature extraction749.273.7%
Matching (top-KK Sinkhorn)14.81.5%
Mask refinement218.121.4%
Total1016.9100%

Table 7. Per-stage latency breakdown of our pipeline on PerMultiSeg val.

αp\alpha_pFull SinkhornTop-KKΔ\Delta
0.0083.8683.860.00
0.2585.6085.600.00
0.5086.2386.230.00
0.7585.0487.97+2.93
0.8585.8387.00+1.17
1.0085.4586.80+1.35

Table 12. αp\alpha_p sweep: full Sinkhorn vs. top-KK Sinkhorn (mIoU).

Simpler matchers are insufficient. Single-vector matching is brittle: mean-pooling and even the [CLS] token discard the spatial structure this task needs. Full Sinkhorn is diluted by the ∼2/3 of patches that are background or shared-category content; trimming the correspondences post-hoc (mutual-NN, SuperGlue-style) is worse, breaking the transport coupling. Top-KK instead trims the per-patch dual contributions, keeping only confident matches; the trim is robust for any K∈[1/3,1/2]K \in[1/3, 1/2], and we use the most aggressive, K=1/3K=1/3.

Component Ablations

Object proposals. We compare class-agnostic detectors in the supp (Table 9) and sweep the YOLOv11 confidence threshold (Table 10). SAM’s Automatic Mask Generator achieves the highest recall on PerMultiSeg (99.38%) but at roughly 25×25 \times the latency of YOLOv11; YOLOv11 offers the best recall–accuracy–size trade-off across both datasets and is our default at τ=10−3\tau=10^{-3}, where lower thresholds monotonically improve final mIoU because top-KK matching absorbs the extra outlier proposals.

τ\tauAvg props/imgProp. Rec. ↑\uparrowMatch (ms)mIoU ↑\uparrowRecall ↑\uparrow
10−110^{-1}4.5174.6317.665.6067.25
10−210^{-2}9.9988.6017.382.7484.72
10−310^{-3}27.6991.4218.987.9789.83

Table 10. Detection threshold sensitivity (PerMultiSeg val, αp=0.75\alpha_p = 0.75, top-KK Sinkhorn at K=1/3K=1/3). Proposal recall: box IoU >0.5> 0.5 with GT box. Recall: output mask IoU >0.5> 0.5.

Image encoder. Table 6 compares six encoders under three matchers. Two findings: (1) patch-level OT massively outperforms CLS-only for self-supervised encoders (DINOv2: +13 mIoU); (2) top-KK Sinkhorn improves over full Sinkhorn for every encoder tested (per-encoder sweep in supp., Figure 7), confirming inlier-restricted scoring as a general matching primitive rather than a BLIP-2-specific trick.

Figure 7. Top-KK Sinkhorn KK sweep across encoders on PerMultiSeg val. ★ marks the per-encoder optimal KK. The dotted line at K=1K=1 (full Sinkhorn) is suboptimal for every encoder. Optimal KK varies across encoders (K∈[1/3,0.9]K \in[1/3, 0.9]) but trimming is universally beneficial.

BLIP-2 delivers the highest absolute accuracy (87.97) and is our default.

Conclusion

We showed that strong reported PIS numbers are largely a single-instance artifact; proposal recall is the real bottleneck. Our training-free, recall-first system introduces top-KK Sinkhorn, matching partial-OT accuracy at more than 100×100 \times lower cost. Across PerMultiSeg and three existing datasets, it surpasses prior methods by up to ∼48\sim48 mIoU at comparable runtime.

Optimal Transport Background

This section provides the formal definitions of entropic regularised optimal transport and Sinkhorn divergence used in § 4.3 of the main paper.

Entropic regularised OT. Let α=1N∑j=1NδVjr\alpha= \frac{1}{N}\sum_{j=1}^{N}\delta_{V^{r}_{j}} and β=1N∑k=1NδVk(i)\beta= \frac{1}{N}\sum_{k=1}^{N}\delta_{V^{(i)}_{k}} be the empirical measures over the reference and candidate patch grids, with uniform marginal weights a,b∈R+N\mathbf{a}, \mathbf{b} \in\mathbb{R}_{+}^{N}. Given a ground cost matrix C∈R+N×N\mathbf{C} \in\mathbb{R}_{+}^{N \times N} whose entry Cj,kC_{j,k} is the squared Euclidean distance between VjrV^{r}_{j} and Vk(i)V^{(i)}_{k}, the entropic regularised OT cost [12] is

OT⁡ϵ(α,β)=min⁡P∈U(a,b)∑j,kPj,kCj,k+ϵ∑j,kPj,klog⁡Pj,k,(6)\operatorname{OT}_{\epsilon}(\alpha,\beta) = \min_{\mathbf{P} \in U(\mathbf{a},\mathbf{b})} \sum_{j,k} P_{j,k} C_{j,k} + \epsilon\sum_{j,k} P_{j,k} \log P_{j,k}, \tag*{(6)}

where P∈[0,1]N×N\mathbf{P} \in[0,1]^{N \times N} is the transport plan, U(a,b)U(\mathbf{a},\mathbf{b}) is the polytope of joint distributions with marginals a,b\mathbf{a}, \mathbf{b}, and ϵ>0\epsilon> 0 is the entropic regularisation strength. We solve Eq. (6) with the Sinkhorn algorithm via the GeomLoss implementation [14], which returns the dual potentials f⋆,g⋆∈RN\mathbf{f}^{\star}, \mathbf{g}^{\star} \in\mathbb{R}^{N} satisfying OT⁡ϵ(α,β)=∑jajfj⋆+∑kbkgk⋆\operatorname{OT}_{\epsilon}(\alpha,\beta) = \sum_{j} a_{j}f_{j}^{\star} + \sum_{k} b_{k}g_{k}^{\star}.

Sinkhorn divergence. Equation (1) of the main paper defines the de-biased Sinkhorn divergence [44]

Sϵ(Vr,V(i))=OT⁡ϵ(Vr,V(i))−12OT⁡ϵ(Vr,Vr)−12OT⁡ϵ(V(i),V(i)),(7)\mathcal{S}_{\epsilon}(\mathbf{V}^{r}, \mathbf{V}^{(i)}) = \operatorname{OT}_{\epsilon}(\mathbf{V}^{r}, \mathbf{V}^{(i)}) - \frac{1}{2}\operatorname{OT}_{\epsilon}(\mathbf{V}^{r}, \mathbf{V}^{r}) - \frac{1}{2}\operatorname{OT}_{\epsilon}(\mathbf{V}^{(i)}, \mathbf{V}^{(i)}), \tag*{(7)}

which subtracts the two self-transport terms to remove the entropic bias of OT⁡ϵ\operatorname{OT}_{\epsilon}. To compute Sϵ\mathcal{S}_{\epsilon}, we solve three independent OT⁡ϵ\operatorname{OT}_{\epsilon} subproblems with cost matrices Cj,kr,i=∥Vjr−Vk(i)∥2C^{r,i}_{j,k} = \lVert V^{r}_{j} - V^{(i)}_{k} \rVert_{2}, Cj,kr,r=∥Vjr−Vkr∥2C^{r,r}_{j,k} = \lVert V^{r}_{j} - V^{r}_{k} \rVert_{2}, and Cj,ki,i=∥Vj(i)−Vk(i)∥2C^{i,i}_{j,k} = \lVert V^{(i)}_{j} - V^{(i)}_{k} \rVert_{2}, and combine them via Eq. (7).

Top-KK Sinkhorn and partial OT. Equation (3) of the main paper trims the dual sum of the cross term OT⁡ϵ(Vr,V(i))\operatorname{OT}_{\epsilon}(\mathbf{V}^{r}, \mathbf{V}^{(i)}) to its ⌊KN⌋\lfloor K N \rfloor smallest per-patch contributions on each side. This is motivated by the partial optimal transport problem [4, 8, 15], in which a fraction K∈(0,1]K \in(0,1] of mass on each side is allowed to be ignored:

inf⁡α′≤α, β′≤β∣α′∣=∣β′∣=KOT⁡ϵ(α′,β′).\inf_{\substack{\alpha' \le\alpha,\ \beta' \le\beta\\ |\alpha'| = |\beta'| = K}} \operatorname{OT}_{\epsilon}(\alpha',\beta').

We emphasise that top-KK Sinkhorn is a surrogate, not an exact solver: the dual potentials f⋆,g⋆\mathbf{f}^{\star}, \mathbf{g}^{\star} are obtained from the full OT problem and then trimmed post-hoc, whereas solving partial OT directly requires modifying the Sinkhorn iteration with a dummy mass. Our shortcut does not provably solve the partial program, but it targets the same inlier-restricted scoring at the cost of a single O(Nlog⁡N)O(N \log N) sort per pair, and Table 5 shows it achieves comparable accuracy to the principled partial-OT solver of Chapel et al. [8] (87.97 vs. 87.97 mIoU on our 135-pair validation set) at more than 100×\times lower latency (8 ms vs. 848 ms per pair). This keeps it inside the per-proposal candidate loop without breaking the runtime budget reported in Table 2.

Ground cost: cosine for global, squared-Euclidean for patch OT. The global distance dgd_{g} uses cosine distance to match the metric the vision–language encoders are trained under. The patch-level OT uses squared-Euclidean ground cost on ℓ2\ell_{2}-normalised patch tokens, the native cost of the p=2p=2 entropic Sinkhorn solver we use [12, 14]. This is consistent with cosine matching: after ℓ2\ell_{2} normalisation, the identity ∥u−v∥22=2(1−uTv)\lVert u-v\rVert_{2}^{2} = 2(1-u^{\mathsf{T}}v) shows that squared-Euclidean cost and cosine distance induce identical orderings on any pair of patches, so the choice does not change which patches the OT solver couples together. In pilot experiments using cosine on patches we observed no measurable accuracy difference, consistent with the orderings being identical.

Per-matcher latency (details). The per-pair latencies reported in Table 5 of the main paper are measured on an A100 GPU with N=256N=256 BLIP-2 patches, averaged over the 135 PerMultiSeg val pairs. The ordering follows directly from each matcher’s computational structure. Top-KK Sinkhorn shares the same Sinkhorn forward pass as Full Sinkhorn, so the only added work is an O(Nlog⁡N)O(N \log N) sort per side, empirically indistinguishable from the un-trimmed 8 ms cost. Partial OT, in contrast, requires a network-simplex linear-programming solve [8] which does not parallelise on the GPU and is two orders of magnitude slower (848 ms). Mutual-NN filtering adds an O(N2)O(N^{2}) pairwise argmax on top of Full Sinkhorn (10 ms), so it is slightly more expensive while also degrading accuracy. This is why top-KK Sinkhorn is the only matcher in Table 5 that attains partial-OT-level accuracy while remaining cheap enough for the per-proposal candidate loop.

End-to-end latency breakdown. Table 7 decomposes the end-to-end inference time of Table 2 into its constituent stages, measured on a single A100 40 GB GPU (same hardware as the rest of the paper) and averaged over the 135 PerMultiSeg val pairs, with B≈19B \approx19 proposals and $N \approx 1.6 references per image; mask refinement is run only on the chosen box, not on all BB proposals. The dominant cost is foundation-model feature extraction (BLIP-2, 74% of the budget), followed by SAM mask refinement (21%). The proposed top-KK Sinkhorn matcher contributes only 1.5% of total wall-clock time despite being the methodological centrepiece, confirming that the inlier-restricted matcher is essentially free at the system level. Conversely, this analysis identifies the encoder forward as the natural target for future deployment-side optimisation (e.g., distilled or smaller-resolution vision backbones), not the matcher.

Evaluation Metrics

We evaluate the performance of our method using Recall and Mean Intersection over Union (mIoU). These metrics assess both the quality of segmentation (PIS task) and the effectiveness of object proposals.

Metrics for PIS Task

Recall: For point-based methods such as PerSAM [57], PDM [42], and P2^{2}SAM [58], Recall is computed by checking whether at least one predicted point prompt falls within the ground truth mask. For bounding box-based methods like ours, a prediction is considered correct if its Intersection over Union (IoU) with the ground truth mask exceeds 0.5.

Mean Intersection over Union (mIoU): To compute the mIoU for the PIS task, we first define the IoU for each category cc as:

IoU⁡c=∣Pc∩Gc∣∣Pc∪Gc∣(8)\operatorname{IoU}_{c} = \frac{|P_{c} \cap G_{c}|}{|P_{c} \cup G_{c}|} \tag*{(8)}

The overall mIoU is then calculated as the average IoU across all CC categories:

mIoU⁡=1C∑c=1CIoU⁡c=1C∑c=1C∣Pc∩Gc∣∣Pc∪Gc∣(9)\operatorname{mIoU} = \frac{1}{C} \sum_{c=1}^{C} \operatorname{IoU}_{c} = \frac{1}{C} \sum_{c=1}^{C} \frac{|P_{c} \cap G_{c}|}{|P_{c} \cup G_{c}|} \tag*{(9)}

where:

  • PcP_c represents the set of predicted pixels for category cc,

  • GcG_c represents the set of ground truth pixels for category cc,

  • CC is the total number of categories.

This metric evaluates segmentation quality by measuring the overlap between predicted and ground truth regions, with higher values indicating better performance.

Object Proposal Evaluation (tab:analysis-results)

Tab. 2 of the main paper reports proposal-stage recall before any downstream matching. Because point-prompt and box-prompt methods produce different proposal types, each is scored under its native protocol:

Recall (point-prompt baselines). For PerSAM [57], PDM [42], and P2^{2}SAM [58], a prediction is counted as a match if the predicted point prompt falls inside the ground-truth mask.

Recall (our box-based proposals). For our method, a prediction is counted as a match if the proposal bounding box has IoU >0.5> 0.5 with the ground-truth bounding box. Recall is then the percentage of ground-truth objects that have at least one matching proposal.

Inference time. End-to-end runtime per target image, in milliseconds.

Mean IoU (mIoU) of matched proposals. For all matched proposals, we compute the mean IoU as:

mIoU⁡=1N∑i=1N∣Pi∩Gi∣∣Pi∪Gi∣(10)\operatorname{mIoU} = \frac{1}{N} \sum_{i=1}^{N} \frac{|P_i \cap G_i|}{|P_i \cup G_i|} \tag*{(10)}

where:

  • PiP_i is the predicted bounding box for object ii,

  • GiG_i is the ground truth bounding box for object ii,

  • NN is the number of matched proposals.

This ensures that the evaluation considers both whether the reference object is reachable from the proposal stage (Recall) and the geometric quality of the matched proposals (mIoU).

PerMultiSeg Statistics

Collection and annotation protocol. PerMultiSeg consists of target images collected from both online sources and self-captured photographs, covering 45 categories with an average of 24–25 multi-object images per category. The images span diverse environments (indoor and outdoor), lighting conditions, camera viewpoints, and object types. As shown in Figure 5, PerMultiSeg maintains a class distribution similar to MS COCO while introducing significantly richer multi-object compositions: most target images contain two to four same-category instances (Figure 5b), increasing candidate selection difficulty and placing higher demands on proposal recall and instance-level matching.

Overall, the dataset includes 1,079 target images and 2,147 reference images, with each target paired with roughly two references and 3,964 annotated instances in total (over three same-category objects per target on average). Each reference object is manually matched to its corresponding instance in the target, background-removed to eliminate noise, and annotated with a precise binary mask. This guarantees that every reference has a correctly labelled counterpart in the target, providing high-quality supervision for PIS evaluation.

Instance-selection criterion. The difficulty of a PIS benchmark is set by how hard the distractors are, so we select them deliberately rather than accepting whatever same-category objects happen to be in a scene. For each target image we require the co-occurring instances to be the same product and the same colour as the reference object, that is, visually near-identical units that differ only in physical identity, pose, wear, and placement. A distractor that is merely the same category (a different mug model, or the same mug in a different colour) is discriminable from a single global appearance vector, which is precisely the regime in which prior benchmarks saturate. Requiring same-product, same-colour distractors removes category- and colour-level shortcuts and forces the matcher to rely on fine-grained, spatially localised evidence such as labels, scuffs, contents, and surface detail. Each reference is annotated against exactly one instance in the target, so the ground truth remains unambiguous: the distractors share the reference’s product and colour but never its identity. Figure 9 shows representative cases.

Category and scene bias. Two forms of bias are worth stating explicitly. Category bias: the 45 categories are dominated by portable manufactured objects (mugs, bottles, bags, consumer electronics, footwear), because the same-product, same-colour criterion above requires multiple near-identical units to co-occur naturally, a condition mass-produced goods satisfy and organic or unique objects (animals, plants, artworks, faces) largely do not. Per-

MultiSeg therefore measures instance discrimination in the manufactured-object regime and should not be read as covering deformable or organic instances. The per-category counts are nonetheless close to uniform (Figure 5a), avoiding the long-tailed frequency skew of MS COCO. Scene bias: images are weighted toward indoor, human-scale scenes (desks, shelves, tables, retail displays), a consequence of where multiple identical products naturally appear together; outdoor and long-range scenes are present but less frequent. Neither bias affects the comparisons in this paper, since all methods are evaluated on identical image and reference pairs, but both bound the external validity of the absolute numbers.

The category-level histogram is shown in Figure 6. Each category includes diverse object scenarios (varying environments, lighting conditions, camera views), and samples feature a range of resolutions and aspect ratios, making them more suitable for real-world applications than fixed-resolution benchmarks.

Figure 6. The statistics of PerMultiSeg, with the figure showing the category histogram.

Additional Ablations

Trim ratio KK (wider sweep). Table 8 sweeps K∈{0.10,…,1.00}K \in\{0.10,\ldots,1.00\} on the default BLIP-2 configuration. Full Sinkhorn (K=1K=1) is not the best; any K∈[1/3,1/2]K \in[1/3,1/2] reaches the peak, and below K=0.20K=0.20 the signal degrades as too few patches survive. We adopt K=1/3K=1/3 in the main paper as the most aggressive trim at peak accuracy.

Per-encoder KK sweep. Figure 7 shows mIoU vs. KK for each encoder. Every encoder benefits from some degree of trimming over full Sinkhorn (K=1K=1): the gain ranges from +0.4 mIoU (CLIP-L) to +2.9 mIoU (BLIP-2), and trimming never hurts. The optimal trim ratio is mildly encoder-specific (K∗∈[1/3,0.9]K^* \in[1/3,0.9]), but the inlier-restricted scoring principle is universal. The strongest patch encoders (BLIP-2, the SigLIP-2 family) both benefit and peak at aggressive trims (K⋆=1/3K^\star= 1/3 for BLIP-2 and SigLIP-2 Giant), consistent with the outlier-rejection view of top-KK: discarding the highest-cost patches removes background and shared-category mass that otherwise dilutes the instance signal.

Object proposal methods. Table 9 compares class-agnostic detectors as the proposal stage, on PerMultiSeg and DAVIS2017. SAM’s Automatic Mask Generator gives the highest recall on PerMultiSeg but is far heavier and slower; YOLOv11 gives the best recall–accuracy–size trade-off across both datasets and is our default in the main paper.

Detection threshold sensitivity. Table 10 sweeps the YOLOv11 confidence threshold τ\tau on the validation set (68 target images, 135 reference–target pairs). Lower τ\tau yields more proposals, monotonically higher proposal recall, and monotonically higher final mIoU; the deployed τ=10−3\tau=10^{-3} wins on every metric, reaching the same 87.97 mIoU as the matcher ablation in Table 5. The extra proposals at low τ\tau

do not degrade matching: top-KK trims the high-cost outlier patches and per-pair matching latency is essentially flat across the sweep (17–19 ms) despite a 6× increase in proposal count, so the recall–noise trade-off the threshold could expose is absorbed by the matcher rather than passed to the final mask.

Self-term ablation (E5). The deployed top-KK Sinkhorn uses only the cross transport term Oϵ(Vr,V(i))O_{\epsilon}(\mathbf{V}^{r}, \mathbf{V}^{(i)}), omitting the two self-terms of the de-biased Sinkhorn divergence (Eq. (1)). Table 11 verifies this is lossless: adding self-terms yields identical mIoU at 3× higher latency (three Sinkhorn solves instead of one).

αp\alpha_p sweep (E3). Table 12 shows mIoU across blend weights αp\alpha_p for both full Sinkhorn and top-KK. Top-KK helps at every αp≥0.75\alpha_p \ge0.75 (up to +2.93 mIoU) and never hurts; the gain is not a hyperparameter accident. The advantage of top-KK over full Sinkhorn is largest where the patch distance carries the most weight: at the deployed default αp=0.75\alpha_p = 0.75 the gap is +2.93 mIoU (87.97 vs. 85.04), narrowing to +1.17 at αp=0.85\alpha_p = 0.85 as the global CLS term begins to dominate the score. For completeness we also state the best-versus-best comparison, which is the more conservative reading: full Sinkhorn peaks at 86.23 (αp=0.50\alpha_p = 0.50) while top-KK peaks at 87.97 (αp=0.75\alpha_p = 0.75), a gain of +1.74 mIoU. The larger +2.93 figure is measured at a shared αp\alpha_p where full Sinkhorn happens to dip below both of its neighbours, so +1.74 is the number we would defend as the matcher’s own contribution. We adopt αp=0.75\alpha_p = 0.75 as the paper default, used consistently across all ablations.

Entropic regularisation ϵ\epsilon. Table 13 shows that ϵ=0.5\epsilon= 0.5 yields the best balance; smaller values lead to unstable estimates while larger ones over-smooth the matching.

ϵ\epsilonmIoU ↑\uparrowRecall ↑\uparrowbIoU ↑\uparrow
0.186.7689.0581.35
0.587.3989.7281.93
0.987.0589.3681.68
1.086.7689.0581.35

Table 13. Study on ϵ\epsilon in entropic OT.

Challenging conditions. Table 14 stratifies by object count, occlusion, clutter, and illumination. Performance degrades under occlusion and many objects; clutter has modest effect.

[0–7][8+]OccNoOccClutNoClutoverexpnormal
mIoU78.8975.8270.7081.4579.7278.6480.4378.23
Recall80.8979.3472.5875.7181.8580.6774.5980.34

Table 14. Performance under challenging conditions.

SAM family as image encoder. Tab. 16 evaluates the vision encoders of SAM [24], SAM2 [41], and SAM3 [7] as feature extractors for matching (not as mask generators). Top-KK Sinkhorn improves over full Sinkhorn for all three SAM variants, confirming that the inlier-restricted scoring principle is not specific to vision-language encoders. SAM3’s stronger semantic features yield the highest absolute mIoU (80.25) among SAM-family encoders, but still fall short of BLIP-2 (87.97) for PIS, since SAM-family encoders are trained for mask prediction rather than instance discrimination.

Additional Quantitative Results

As few-shot segmentation (FSS) is closely related to personalized instance segmentation (PIS), we also evaluate FSS methods on the proposed PerMultiSeg dataset. As shown in Tab. 15, our method using a box prompt outperforms FSS methods by a significant margin. We also report results for our method using a center-point prompt, where the prompt is provided to SAM after candidate selection; this configuration still achieves better performance than prior methods.

Additional Qualitative Results

We provide additional qualitative examples on PerMultiSeg in Figure 8. Our segmentation results outperform prior methods even in challenging cases such as the books (third row) and the fruit (last row).

Additional qualitative examples showing reference images, target images, PerSAM, PDM, P²SAM, Ours, and GT

Figure 8. Additional qualitative examples on PerMultiSeg.

PIS vs. One-Shot Category Segmentation on LVIS

A natural question is why we do not evaluate on LVIS [16] or COCO [28], given that prior work such as PerSAM [57] reports one-shot LVIS numbers. We address this directly rather than dismiss it. LVIS is a valid one-shot benchmark, but it measures a different form of one-shot generalization: one-shot LVIS evaluates one-shot category generalization (given one example of a category, segment all its instances, scored by per-category AP), whereas PIS evaluates one-shot instance generalization (given a reference image of one specific object, segment that same object among same-category look-alikes). These are complementary properties, not competing measurements of the same one: a method that segments all mugs scores well on LVIS but, on PIS, returns a distractor for every mug except the reference. Evaluating PIS on LVIS is possible in principle, but LVIS provides no reference-to-target instance correspondence (no ground truth for “is this the same physical object as the reference?”), so it would require re-annotating correspondences and redefining the metric, yielding a new benchmark rather than a comparison on LVIS as published. We instead build PerMultiSeg with this correspondence infrastructure from the ground up, alongside the instance-level benchmarks PerSeg [57], PerMIS [42], and DAVIS [37]. That strong one-shot category performance does not transfer to one-shot instance discrimination is confirmed empirically: one-shot category-segmentation (FSS) methods reach only 23–38 mIoU on PerMultiSeg (Tab. 15), precisely because they cannot isolate the reference instance.

Design Discussion

Why training-free. None of the foundation models we compose (CLIP/BLIP-2 encoders, class-agnostic detectors, SAM) is trained on a personalized-segmentation objective, and building one is non-trivial: every new application domain (a specific user’s belongings, a specific product catalogue, a specific industrial part) would require its own labelled reference–target instance correspondences and a tuned fine-tuning recipe, breaking the very generalisation that makes foundation models attractive. We therefore compose off-the-shelf detectors, encoders, and SAM into a personalized-segmentation pipeline with no per-domain training step. This keeps each component independently upgradable, so future stronger detectors or encoders translate directly into stronger PIS without re-collecting instance correspondences. Unified trained alternatives such as SAM3’s Promptable Concept Segmentation [7] solve a related but different task (segment all matching instances of a referenced concept, not the single specific instance corre-

MethodTaskmIoU ↑\uparrowRecall ↑\uparrowbIoU ↑\uparrow
SegGPT [53]FSS29.3416.8621.23
Matcher [29]FSS23.2211.2918.53
DCAMA [46]FSS32.1517.1624.41
GF-SAM [55]FSS37.9922.1132.98
PerSAM [57]PIS21.6525.2914.82
PDM [42]PIS23.6543.9112.82
P2P^2SAM [58]PIS31.5824.5727.90
Ours (point)PIS55.4155.2048.66
Ours (box)PIS80.0782.2975.15

caption: Comparison on PerMultiSeg among few-shot semantic segmentation (FSS) methods and PIS methods.

:::

sponding to a reference image), as the SAM3 row of Table 3 shows empirically.

Limitations and reference protocol. Our evaluation uses background-removed reference crops with a guaranteed correct match in the target, which is cleaner than many deployment scenarios where the user provides a phone screenshot, a loose crop, or a manual box, and the reference may not even appear in the target. Two failure modes follow directly: (i) when no proposal covers the correct instance (detector recall, the bottleneck we already trade compute against in Table 9), and (ii) when the discriminative cue is occluded while a distractor displays a similar cue, as in the failure row of Figure 4 of the main paper. Since the pipeline is modular and training-free, both are inherited from off-the-shelf components rather than baked-in: stronger class-agnostic detectors directly raise proposal recall, and the matcher accepts noisier reference crops to the extent the underlying encoder’s CLS–patch fields remain instance-discriminative. Extending the benchmark to in-the-wild references (screenshots, loose crops, manual boxes) and adding a “no-match” rejection option are natural next steps.

EncoderCLSMeanFullTop-KK
SAM ViT-H [24]52.5452.5355.4457.46
SAM2 Hiera-L [41]61.1559.1957.9459.08
SAM3 [7]65.4966.5878.3080.25

caption: SAM family as image encoder for matching (PerMultiSeg val).

:::

Baseline Configuration

For P2P^2SAM [58] we report its official default configuration: part-aware clustering enabled (1≤nclusters≤51 \le n_{\mathrm{clusters}} \le5), guided attention, and the same SAM ViT-H checkpoint as all other methods. Running the baseline with a single cluster disables its part-aware selection and understates it, so we use the stronger setting throughout. Relative to the single-cluster setting, the official configuration raises mIoU and bIoU but lowers final-prediction Recall: it produces better masks where it fires, yet clears the IoU >0.5> 0.5 hit threshold less often.

Where Does the Error Come From?

We decompose end-to-end accuracy on the PerMultiSeg test split (1,971 pairs) into the two stages that can fail independently. A pair is recalled if some proposal box overlaps the ground-truth box with IoU >0.5> 0.5, and correct if the final mask reaches IoU >0.5> 0.5. Overall success approximately factorises as success ≈\approx ceiling ×\times accuracy ∣\mid recalled (91.58%×85.98%=78.74%91.58\% \times85.98\% = 78.74\% vs. 79.55%79.55\% measured; the small residual reflects mild positive correlation between proposal recall and matching accuracy across pairs).

QuantityValue
Proposal-recall ceiling91.58%
Matching accuracy given recall85.98%
Overall success79.55%
Unrecoverable (no proposal covers the object)8.42%
Recalled but mismatched12.84%
mIoU when recalled82.68
mIoU when the proposal is missed14.26

caption: Error decomposition on PerMultiSeg test (1,971 pairs).

:::

Two consequences are worth stating plainly. First, the pipeline has no recovery mechanism: when the proposal stage misses the object, mIoU falls from 82.68 to 14.26, so a detector miss is effectively terminal. Second, the residual error is not dominated by the early modules: 12.84% of pairs are recalled but mismatched against 8.42% unrecoverable, so more loss currently sits in matching than in proposal generation. This is why the matcher, rather than the detector, is the component we ablate most carefully.

Hard Cases: Clutter, Occlusion, and Near-Duplicates

Figure 9 shows twelve test-split examples selected to span the conditions that make PerMultiSeg difficult: dense same-category clutter (bags, cakes, mugs, instant noodles, kitchenware, lipsticks, watercolour sets), partial occlusion (a book behind a plant stack, dogs behind cage bars), and near-duplicate discrimination where instances differ only in a small region (traffic signs, toothbrushes, stuffed animals). In each row the reference fixes a single instance among visually similar candidates, and the prediction is shown against the ground truth.

Statistical Reading of the Matcher Ablation

Top-KK Sinkhorn and the partial-OT solver of Chapel et al. [8] report identical values in Table 5 (87.97 / 89.83 / 82.69). This is neither a coincidence nor a transcription error, and the underlying agreement is stronger than the table alone conveys: the two matchers select the same candidate box on all 135 validation pairs, so their predicted masks are identical and every downstream metric coincides exactly. Selection agreement, rather than a coincidence of rounded averages, is what makes top-KK a usable stand-in for the partial-OT solver it approximates at more than 100×100 \times lower cost. The same effect explains why neighbouring trim ratios share rows in Table 8: within the plateau K∈[1/3,1/2]K \in[1/3, 1/2] the trim does not change which candidate wins.

We deliberately describe this as agreement on selections rather than statistical equivalence. Because the prediction is decided by which candidate wins the argmin, mIoU on

this 135-pair split moves in discrete steps of roughly 0.7 per flipped selection, so the split is better suited to establishing that two matchers behave identically than to resolving small differences between them. For the same reason we report the conservative +1.74 best-versus-best gain over full Sinkhorn alongside the +2.93 measured at the shared default αp=0.75\alpha_{\mathrm{p}}=0.75 (Table 12).

Matched-Proposal Control: Giving Baselines Our Proposals

A natural objection to Table 3 is that our pipeline consumes a high-recall class-agnostic proposal set while the point-prompt baselines do not, so part of the margin could reflect the input representation rather than the quality of reference–candidate matching. We test this directly by giving P2P^{2}SAM [58] the same proposals our system uses: instead of selecting a point from its similarity map, it scores each of our candidate boxes with its own reference embedding (mean-pooled over the in-mask reference patches) and the highest-scoring box is refined by the same SAM checkpoint. Every other component is left untouched.

SplitVariantmIoURecallbIoU
valP2P^{2}SAM (official default)35.5228.7729.96
val+ our proposals (mean pool)33.5633.6930.22
val+ our proposals (max pool)32.0432.6929.28
testP2P^{2}SAM (official default)31.5824.5727.90
test+ our proposals (mean pool)22.1122.1520.04

Table 18. Matched-proposal control on PerMultiSeg. “Matched” replaces P2P^{2}SAM’s point selection with a choice among our proposals, holding the proposal set, SAM checkpoint, and its own features fixed.

Supplying the proposals does not improve the baseline: it costs 1.96 mIoU on val and 9.47 mIoU on test. High-recall proposals are therefore not a transferable advantage on their own. Once the correct instance is present among many same-category candidates, matching contributes beyond proposal recall: a single mean-pooled embedding cannot separate look-alikes even when the correct box is available to select. The prompt representation itself also matters (Tab. 15 shows a center-point variant of our system loses roughly half the gain), so the full margin reflects both effects, not matching alone. The matching contribution is corroborated from the opposite direction (Table 5), where replacing the patch-level distance with mean-pooling costs more than 22 mIoU with the proposal set held fixed. We note that this control favours the baseline on one axis, since it removes its point-selection stage entirely and hands it an oracle-free but already-filtered candidate list.

References

  1. [1]Mokhtar Z. Alaya, Maxime Bérar, Gilles Gasso, and Alain Rakotomamonjy. Screening sinkhorn algorithm for regularized optimal transport. In Advances in Neural Information Processing Systems, 2019. 3arxiv.org/abs/1906.08540
  2. [2]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Owen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025. 7arxiv.org/abs/2502.13923
  3. [3]Yogesh Balaji, Rama Chellappa, and Soheil Feizi. Robust optimal transport with applications in generative modeling and domain adaptation. In Advances in Neural Information Processing Systems, 2020. 3arxiv.org/abs/2010.05862
  4. [4]Luis A. Caffarelli and Robert J. McCann. Free boundaries in optimal transport and Monge–Ampère obstacle problems. Annals of Mathematics, 171(2):673–730, 2020. 2, 3, 6, 1
  5. [5]Kaidi Cao, Jiajun Ji, Zhangjie Cao, Chien-Yi Chang, and Juan Carlos Niebles. Few-shot video classification via temporal alignment. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 3arxiv.org/abs/1906.11415
  6. [6]Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213–229. Springer, 2020. 3arxiv.org/abs/2005.12872
  7. [7]Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. Sam 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719, 2025. 7, 4, 6
  8. [8]Laetitia Chapel, Mokhtar Z. Alaya, and Gilles Gasso. Partial optimal transport with applications on positive-unlabeled learning. In Advances in Neural Information Processing Systems, 2020. 2, 3, 6, 7, 1arxiv.org/abs/2002.08276
  9. [9]Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in neural information processing systems, 34:17864–17875, 2021. 3arxiv.org/abs/2107.06278
  10. [10]Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16901–16911, 2024. 5, 4arxiv.org/abs/2401.17270
  11. [11]Seungwhan Choi, Sunghyun Park, Minsoo Lee, and Jaegul Choo. Viton-hd: High-resolution virtual try-on via misalignment-aware normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14131–14140, 2021. 1arxiv.org/abs/2103.16874
  12. [12]Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems, 2013. 5, 1
  13. [13]Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19358–19369, 2023. 6, 8arxiv.org/abs/2211.07636
  14. [14]Jean Feydy, Thibault Séjourné, François-Xavier Vialard, Shun-ichi Amari, Alain Trouvé, and Gabriel Peyré. Interpolating between optimal transport and mmd using sinkhorn divergences. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2681–2690, 2019. 6, 1
  15. [15]Alessio Figalli. The optimal partial transport problem. Archive for Rational Mechanics and Analysis, 195:533–560, 2010. 2, 3, 6, 1DOI
  16. [16]Agrim Gupta, Piotr Dollár, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. 5arxiv.org/abs/1908.03195
  17. [17]Sina Haijimiri, Malik Boudiaf, Ismail Ben Ayed, and Jose Dolz. A strong baseline for generalized few-shot semantic segmentation. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11269–11278, 2023. 1
  18. [18]Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Yu, and Larry S Davis. Viton: An image-based virtual try-on network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7543–7552, 2018. 1arxiv.org/abs/1711.08447
  19. [19]Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017. 5
  20. [20]Jianbin Jiang, Tan Wang, He Yan, and Junhai Liu. Cloth-former: Taming video virtual try-on in all module. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10799–10808, 2022. 1
  21. [21]Johanna Karras, Yingwei Li, Nan Liu, Luyang Xin, Infarn Yoo, Andreas Lugmayr, Chris Lee, and Ira Kemelmacher-Shlizerman. Fashion-vdm: Video diffusion model for virtual try-on. In SIGGRAPH Asia 2024 Conference Papers, New York, NY, USA, 2024. Association for Computing Machinery. 1arxiv.org/abs/2411.00225
  22. [22]Rahima Khanam and Muhammad Hussain. Yolov11: An overview of the key architectural enhancements. 2024. 4arxiv.org/abs/2410.17725
  23. [23]Jeongho Kim, Guojung Gu, Minho Park, Sunghyun Park, and Jaegul Choo. Stableviton: Learning semantic correspondence with latent diffusion model for virtual try-on. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8176–8185, 2024. 1arxiv.org/abs/2312.01725
  24. [24]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023. 3, 5, 6, 4, 7arxiv.org/abs/2304.02643
  25. [25]Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022. 5arxiv.org/abs/2201.12086
  26. [26]Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023. 1, 3, 6, 8arxiv.org/abs/2301.12597
  27. [27]Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European conference on computer vision, pages 280–296. Springer, 2022. 3arxiv.org/abs/2203.16527
  28. [28]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision, 2014. 5arxiv.org/abs/1405.0312
  29. [29]Yang Liu, Muzhi Zhu, Hengtai Li, Hao Chen, Xinlong Wang, and Chunhua Shen. Matcher: Segment anything with one shot using all-purpose feature matching. arXiv preprint arXiv:2305.13310, 2023. 3, 7arxiv.org/abs/2305.13310
  30. [30]Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015. 3
  31. [31]Chuofan Ma, Yi Jiang, Xin Wen, Zehuan Yuan, and Xiaojuan Qi. Codet: Co-occurrence guided region-word alignment for open-vocabulary object detection. In Advances in Neural Information Processing Systems, 2023. 5, 4arxiv.org/abs/2310.16667
  32. [32]Debargha Mukherjee, Aritra Gуха, Justin Solomon, Yuekai Sun, and Mikhail Yurochkin. Outlier-robust optimal transport. In International Conference on Machine Learning, 2021. 3arxiv.org/abs/2012.07363
  33. [33]Hung Nguyen, Quang Qui-Vinh Nguyen, Khoi Nguyen, and Rang Nguyen. Swifttry: Fast and consistent video virtual try-on with diffusion models. arXiv preprint arXiv:2412.10178, 2024. 1arxiv.org/abs/2412.10178
  34. [34]Khoi Nguyen and Sinisa Todorovic. Feature weighting and boosting for few-shot segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 622–631, 2019. 1, 3arxiv.org/abs/1909.13140
  35. [35]Hyeonwoo Noh, Andre Araújo, Jack Sim, Tobias Weyand, and Bohyung Han. Large-scale image retrieval with attentive deep local features. In Proceedings of the IEEE international conference on computer vision, pages 3456–3465, 2017. 1arxiv.org/abs/1612.06321
  36. [36]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 1, 2, 3, 5, 8arxiv.org/abs/2304.07193
  37. [37]Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675, 2017. 2, 3, 4, 6, 5
  38. [38]Howard H. Qian, Yangxiao Lu, Kejia Ren, Gaotian Wang, Ninad Khargonekar, Yu Xiang, and Kaiyu Hang. Riseg: Robot interactive object segmentation via body frame-invariant features. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 13954–13960, 2024. 1arxiv.org/abs/2403.01731
  39. [39]Howard H Qian, Yiting Chen, Gaotian Wang, Podshara Chanrungmaneekul, and Kaiyu Hang. rt-riseg: Real-time model-free robot interactive segmentation for active instance-level object understanding. pages 11561–11568, 2025. 1arxiv.org/abs/2507.10776
  40. [40]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021. 2, 3, 5, 8arxiv.org/abs/2103.00020
  41. [41]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaithanya Ryali, Tengyu Ma, Huisheng Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 5, 7, 4arxiv.org/abs/2408.00714
  42. [42]Dvir Samuel, Raman Barı-Em, Matan Levy, Nir Darshan, and Gal Chechik. Where's waldo: Diffusion features for personalized segmentation and retrieval. 37:128160–128181, 2024. 1, 2, 3, 4, 5, 6, 7arxiv.org/abs/2405.18025
  43. [43]Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 3
  44. [44]Thibault S’ejourn’e, Jean Feydy, Francois-Xavier Vialard, Alain Trouv’e, and Gabriel Peyr’e. Sinkhorn divergences for unbalanced optimal transport. ArXiv, abs/1910.12958, 2019. 2, 5, 1
  45. [45]Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Jing Li, Xiangyu Zhang, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8425–8434, 2019. 5DOI
  46. [46]Xinyu Shi, Dong Wei, Yu Zhang, Donghuan Lu, Mунan Ning, Jiashun Chen, Kai Ma, and Yefeng Zheng. Dense cross-query-and-support attention weighted mask aggregation for few-shot segmentation. In European Conference on Computer Vision, pages 151–168. Springer, 2022. 7arxiv.org/abs/2207.08549
  47. [47]Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508.10104, 2025. 8arxiv.org/abs/2508.10104
  48. [48]Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7262–7272, 2021. 3arxiv.org/abs/2105.05633
  49. [49]Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. LoFTR: Detector-free local feature matching with transformers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. 3arxiv.org/abs/2104.00680
  50. [50]Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025. 8arxiv.org/abs/2502.14786
  51. [51]Bochao Wang, Huabin Zheng, Xiaodan Liang, Yimin Chen, Liang Lin, and Meng Yang. Toward characteristic-preserving image-based virtual try-on network. In Proceedings of the European conference on computer vision (ECCV), pages 589–604, 2018. 1arxiv.org/abs/1807.07688
  52. [52]Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6830–6839, 2023. 3arxiv.org/abs/2212.02499
  53. [53]Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304.03284, 2023. 3, 7arxiv.org/abs/2304.03284
  54. [54]Bichen Wu, Forrest Iandola, Peter H Jin, and Kurt Keutzer. Squeezedet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 129–137, 2017. 1arxiv.org/abs/1612.01051
  55. [55]Anqi Zhang, Guangyu Gao, Jianbo Jiao, Chi Harold Liu, and Yunchao Wei. Bridge the points: Graph-based few-shot segment anything semantically. 2024. 7arxiv.org/abs/2410.06964
  56. [56]Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable earth mover’s distance and structured classifiers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 3
  57. [57]Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Xianzheng Ma, Hao Dong, Peng Gao, and Hongsheng Li. Personalize segment anything model with one shot. arXiv preprint arXiv:2305.03048, 2023. 1, 2, 3, 4, 5, 6, 7arxiv.org/abs/2305.03048
  58. [58]Chenhui Zhao and Liyue Shen. Part-aware prompted segment anything model for adaptive segmentation. Transactions on Machine Learning Research. 3, 5, 6, 7, 2, 9arxiv.org/abs/2403.05433
  59. [59]Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in neural information processing systems, 36:19769–19782, 2023. 3arxiv.org/abs/2304.06718

Paper details

vlnc:W_g3u2t7k4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 11

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents