betaFeedback

WorksComputer Science

MESAM: Enhancing SAM2 for Few-Shot Segmentation through Semantic-Visual Memory Generation and Refinement

Abstract

This paper investigates how to leverage the powerful capabilities of the Segment Anything Model 2 (SAM2) to tackle few-shot segmentation (FSS) tasks. One major challenge lies in the substantial appearance differences between the support and query images, which may diminish the effectiveness of the memory-matching process in SAM2. Through systematic analysis, we find that the memory generated based on the categorical semantic information is robust to this appearance divergence and is more compatible with SAM2. Based on these observations, we propose a new Memory-Enhanced SAM2 (MESAM) method, which first introduces a Semantic-Visual Memory Generation (SVMG) module to separately construct a query semantic memory, a query foreground visual memory, and a support adaptive visual memory. Then, in our designed Interactive Semantic Memory Refinement (ISMR) module, we further enrich the obtained semantic memory with fine-grained visual details, and suppress its misatcivated background responses. Extensive experiments on the PASCAL-5i5^i, COCO-20i20^i, and LVIS-92i92^i datasets demonstrate that MESAM consistently outperforms existing FSS methods, achieving new state-of-the-art segmentation performance. Our source code is available at: \href{https://github.com/tianyu-zou/MESAM}{https://github.com/tianyu-zou/MESAM}.

CCS Concepts

  • Computing methodologies → Image segmentation.

ACM Reference Format:

Tianyu Zou, Shengwu Xiong, Rui Wu, Ruilin Yao, and Yi Rong. 2026. MESAM: Enhancing SAM2 for Few-Shot Segmentation through Semantic-Visual Memory Generation and Refinement. In Proceedings of Proceedings of the 34th ACM International Conference on Multimedia (MM ’26). ACM, New York, NY, USA, 17 pages. https://doi.org/10.1145/3767308.3835979

Introduction

Semantic segmentation [4, 15, 21] represents a fundamental task in the field of computer vision, having made significant progress in both academic and industrial communities. These achievements are primarily reliant upon the availability of numerous well-annotated training data. However, manually annotating pixel-level object masks for large-scale image samples is laborious and time-consuming. Moreover, traditional segmentation methods struggle to handle new object categories not encountered during training, thus limiting their generalization ability for segmenting arbitrary classes. To overcome these limitations, few-shot segmentation (FSS) [5, 31, 57] has been proposed. By emulating the human intelligence that can understand unseen concepts through observing only a few examples, FSS attempts to utilize transferable prior knowledge learned from some base classes, alongside useful information extracted from limited annotated support samples, to segment query images containing target objects belonging to novel categories. However, due to the absence of novel class data during training, most existing

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

MM ’26, Rio de Janeiro, Brazil

© 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM.

ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM

https://doi.org/10.1145/3767308.3835979

FSS models tend to overfit to base classes, thereby diminishing the transferability of their learned prior knowledge.

To address this problem, some works [22, 23, 53] have begun to explore the use of vision foundation models (e.g., DINOv2 [25] and SAM [14]) to leverage their fine-grained visual representations or prompt-based segmentation capability. These will provide more transferable prior knowledge or more effective adaptation procedures, thus enhancing the learning of FSS models. More recently, SAM2 [29] has demonstrated impressive inter-frame foreground-to-foreground (FG-FG) matching ability in tackling video segmentation tasks, which may also be beneficial for solving FSS tasks. However, unlike objects exhibiting high visual similarity across consecutive video frames, the targets in support and query images within an FSS task typically present significant variations in appearance. This will prevent the direct use of SAM2 in FSS, since a substantial misalignment will exist between the query samples and the SAM2 segmentation memory generated from support images, resulting in performance degradation (see the third row in Figure 1).

Figure 1. Illustration of the large support-query appearance divergence challenge. With only partial visual cues provided by the support image, existing methods segment only visually similar parts in the query, instead of the entire object. In contrast, with the refined semantic memory, our MESAM method yields better segmentation results. The orange and blue circles indicate foreground (FG) misses and background (BG) misactivations, respectively.

Despite emerging studies [1, 50, 51] attempting to reduce the visual difference between support and query images, such as by generating intermediate samples [35], predicting query-oriented prompts [44], or using feature adaptation [6], the large support-query appearance divergence remains an unsolved challenge when applying SAM2 to FSS. As illustrated in Figure 1, when the support sample presents only partial appearances of the target class (e.g., showing the hull of a boat without sail or only the shoulder straps of a backpack), even the state-of-the-art SAM2-based FSSAM method [44] can only segment the visually similar object parts in the query image, rather than the entire object. We analyze that it is because current methods construct SAM2 memory based solely on visual cues, but support images alone cannot provide the essential evidence required for accurate query segmentation, due to their visual discrepancies.

In this paper, we explore a more effective memory mechanism to enhance the capability of SAM2 to address the above challenge in FSS. We argue that utilizing the categorical textual descriptions is a better choice for memory construction, since they can introduce class-specific semantic information that is not affected by support-query appearance divergence. We conduct two toy experiments to show the superiority of the semantic memory and explain why it is more compatible with SAM2. Based on this analysis, we propose a Memory-Enhanced SAM2 (MESAM) method for FSS, which exploits semantic and visual information in a complementary manner. Specifically, our method first develops a Semantic-Visual Memory Generation (SVMG) module to produce a query semantic memory based on the GradCAM [30] of CLIP’s [28] text classifier, whilst simultaneously generating a query foreground (FG) visual memory and a support adaptive visual memory based on the DINOv2 image representations. Then, in the following Interactive Semantic Memory Refinement (ISMR) module, the query semantic memory is refined by interacting with the query FG memory to absorb its useful visual detail information. Additionally, the support adaptive memory is employed to further calibrate this refinement process, suppressing background (BG) responses and preventing excessive activation from visual cues. In this way, the obtained refined memory representations integrate both semantic information that is robust to support-query appearance divergence and fine-grained visual features that are beneficial to precise segmentation (see the bottom row in Figure 1). Finally, this refined semantic memory is passed through SAM2’s memory attention and mask decoder to generate the final segmentation mask.

Our main contributions are threefold: 1) We discover that the memory generated from categorical semantic information is more compatible with SAM2 and leads to better FSS performance, which motivates the design of a memory mechanism that integrates both robust semantic cues and visual details. 2) We propose a new SAM2-based MESAM method for FSS and design two modules, i.e., SVMG and ISMR, for constructing and refining memory representations, respectively. Based on our knowledge, MESAM is the first attempt to exploit semantic information to empower SAM2 for mitigating the challenge of large support-query appearance divergence in FSS. 3) Extensive experiments on the PASCAL-5i5^i, COCO-20i20^i, and LVIS-92i92^i datasets show that MESAM outperforms existing state-of-the-art methods, validating the effectiveness of our designed semantic-visual memory mechanism.

Related work

Traditional Few-Shot Segmentation. Early FSS methods are dominated by prototype learning [11, 18, 47], evolving from single-prototype models [52] to multi-part and query-related prototypes [8, 16, 27, 45]. To alleviate the detail loss of prototype compression, affinity learning [13, 36, 39, 49] constructs dense support-query correlations (e.g., cycle correlation [48], hyper correlation [24, 32]). Such affinity methods often suffer from FG-BG misalignment, motivating self-calibration and feature purification [42, 43, 56]. Despite these advances, many FSS models may still overfit to base classes, limiting the transferability of learned priors to unseen concepts.

Foundation Based Few-Shot Segmentation. Vision foundation models such as DINOv2 [25] and SAM [14] provide rich semantic priors and strong generalization, addressing limitations of traditional FSS. Training-free methods like Matcher [20] and GF-SAM [46] use DINOv2 features to prompt SAM, while VRP-SAM [33], MM-Promptt [41], and ProSAM [37] design prompt encoder for frozen SAM models. SAM2 [29] introduces a memory mechanism for same-object FG–FG matching, but large appearance differences between support and query images in FSS can cause memory misalignment and degraded performance [40, 58]. Recent adaptation strategies include FSSAM [44] introduces external visual guidance from DINOv2, SANSA [6] leverages the latent semantic knowledge in SAM2, and CAV-SAM [35] uses diffusion-generated intermediate frames. Despite these efforts, the large appearance differences between support and query images remains an unresolved challenge when applying SAM2 to FSS. Different from all existing methods, our proposed MESAM attempts to use semantic information to construct more effective and robust memory representations for SAM2 so as to address this challenge.

Methodology

The main goal of this work is to learn a segmentation model on a training set Dtrain\mathcal{D}_{\mathrm{train}} with sufficient data from base classes Cbase\mathcal{C}_{\mathrm{base}}, expecting that the learned model can generalize well on the FSS tasks sampled from a testing set Dtest\mathcal{D}_{\mathrm{test}} of novel classes Cnovel\mathcal{C}_{\mathrm{novel}}, which are not overlapped with Cbase\mathcal{C}_{\mathrm{base}}, e.g., Cbase∩Cnovel=∅\mathcal{C}_{\mathrm{base}} \cap\mathcal{C}_{\mathrm{novel}} = \varnothing. Following the standard setup, we describe the kk-shot segmentation task as an episode E={S,Q}\mathcal{E} = \{\mathcal{S}, \mathcal{Q}\} that consists of a support set S={(Ins,Mns)}n=1k\mathcal{S} = \{(I_n^s, M_n^s)\}_{n=1}^{k} with kk samples for model adaptation and a query set Q=(Iq,Mq)\mathcal{Q} = (I_q, M_q) for performance evaluation. Here, S\mathcal{S} and Q\mathcal{Q} are two disjoint sets drawn from the same specific object category, satisfying S∩Q=∅\mathcal{S} \cap\mathcal{Q} = \varnothing. I∈RH×W×3I \in\mathbb{R}^{H \times W \times3} and M∈{0,1}H×WM \in\{0, 1\}^{H \times W} denote an input image and its binary segmentation mask, where HH and WW indicate their height and width, respectively. To deal with testing FFS tasks, an episodic learning strategy is usually employed. It constructs a series of episodes from Dtrain\mathcal{D}_{\mathrm{train}} for model training to simulate the processing of the FFS tasks that will be encountered at the evaluation stage.

Analysis of Semantic Memory

In this subsection, we investigate two questions: 1) Whether generating the semantic memory for SAM2 based on the categorical texts yields better FSS segmentation results? 2) If so, why such semantic memory is more compatible with SAM2 when processing FSS tasks? Following FSSAM [44], we construct our visual memory based on the prior mask generated for the query image. Semantic memory is derived from the GradCAM of CLIP’s text classifier, which categorizes query images into FG and BG classes with and without the target object. The details of this construction process will be presented in the following subsections.

For the first question, we compare our semantic memory with the FG and DISC visual memories constructed by FSSAM to validate its effectiveness. We present their segmentation performance measured by mIoU in Table 1. Their segmented results and the prior masks used for generating these memories are also visualized in Figure 2. As can be seen, since the prior masks for the FG and DISC memories are calculated from the visual similarity between the support prototypes and query image pixels, these prototypes may incorrectly activate some visually similar BG areas in query samples (e.g., the dog and the horse in the first two rows of Figure 2), while overlooking FG objects with great appearance differences (e.g., the stop sign in the bottom row of Figure 2), thus leading to relatively lower segmentation accuracy. By contrast, the GradCAM of CLIP’s text classifier captures the semantic information of the target class, allowing the prior mask for constructing our semantic memory to be unaffected by the visual representations of support images. Consequently, this memory can produce more reliable predictions and obtain overall higher mIoU values (82.8% and 58.9% on the two datasets), verifying its superiority.

Priors and segmentation results of two visual memories and the semantic memory

Figure 2. The priors and the segmentation results of two visual memories (i.e., FG, DISC) and our semantic memory.

MemoryFGDISCSemantic
PASCAL-5i5^{i}81.178.182.8
COCO-20i20^{i}55.956.058.9

Table 1. Performance comparison between the visual memories constructed in FSSAM and our semantic memory.

Next, we attempt to answer the second question in order to explain the effectiveness of semantic memory. To do this, we first convert the prior masks for each memory on different query images into binary masks using a threshold of 0.5, and then report their average false positive (FP) and false negative (FN) rates in Table 2. We can find that the prior masks for visual memories typically have higher FP but lower FN rates, indicating that they typically cover more FG pixels and also incorrectly over-segment certain BG areas. In contrast, the prior masks for our semantic memory tend to only cover reliable pixels that are strongly identified as belonging to FG areas, thus reducing FP rates while increasing FN rates. Based on these statistics, we conduct a toy experiment, which applies dilation and erosion operations to the ground-truth segmentation masks of query samples to simulate the prior masks used for generating the visual and semantic memories, respectively. Note that, in addition to the standard erosion operation (i.e., vanilla-erosion), we also employ an enhanced erosion process that preserves the skeleton of target objects unchanged (i.e., skeleton-erosion).

MemoryPASCAL-5i5^{i} FPPASCAL-5i5^{i} FNCOCO-20i20^{i} FPCOCO-20i20^{i} FN
FG0.08650.04360.12710.0381
DISC0.01860.10560.03240.0820
Semantic0.01390.12070.01550.0915

Table 2. The false positive (FP) and false negative (FN) rates of the prior masks used for generating FG and DISC visual memories, and our semantic memory.

Figure 3 displays the segmentation results achieved by different memories constructed from these prior masks under varying dilation and erosion rates. We can observe that: 1) The skeleton-erosion memory consistently outperforms the dilation memory, suggesting that erroneously introducing BG noise like the visual memories may impose a more severe negative impact on SAM2 segmentation than the loss of some FG information. 2) The vanilla-erosion memory without retaining the target object’s skeleton results in performance degradation compared to the skeleton-erosion memory and even the dilation memory. This indicates that preserving key FG features is also critical for SAM2 to achieve precise segmentation. Based on these observations, we can illustrate the compatibility between our semantic memory and SAM2: Since the semantic information introduced by the CLIP classifier is classification-oriented global information shared across the entire target category, it exhibits high reliability and representativeness. It is less likely to introduce BG noise and will capture representative object features that are essential for classification. Therefore, with these properties, our semantic memory can work well with SAM2 for better FSS results.

Performance comparison between different memories obtained under varying dilation and erosion ratios

Figure 3. Performance comparison between different memories obtained under varying dilation and erosion ratios.

Our Proposed MESAM Method

Based on the above analysis, we build our MESAM framework primarily upon the semantic memory. However, since the global categorical semantic information cannot fully represent all FG features, utilizing only the semantic memory will result in insufficient segmentation of several target objects in some query images (see orange circles in Figure 2). To address this issue, we further integrate the visual information into our semantic memory, allowing it to be refined through the incorporation of additional FG details. Next, we will detail how we achieve this through the two modules we designed, namely SVMG and ISMR. For clarity, this introduction will be presented under the 1-shot (k=1k=1) setting, yet it can be easily extended to the scenarios of k>1k>1.

Semantic-Visual Memory Generation (SVMG). As illustrated in Figure 4(a), SVMG aims to construct three types of memories, including a query semantic memory, a query FG visual memory, and a support adaptive visual memory. To this end, SVMG first calculates corresponding prior masks to generate these memories as follows.

Figure 4. Illustration of our MESAM method. (a) The SVMG module generates three initial memories MemssMem_{s}^{s}, MemqvMem_{q}^{v}, and MemsvMem_{s}^{v}. (b) In the ISMR module, MemqsMem_{q}^{s} is enhanced by interacting with MemqvMem_{q}^{v} through the cross-attention operation, while MemsvMem_{s}^{v} is used to suppress the BG noise in the attention scores. Subsequently, the refined semantic memory Memqs′Mem_{q}^{s'} is processed by SAM2’s memory attention module and mask decoder to produce the final segmentation mask.

  1. Query Semantic Prior Mask: For each object category to be segmented, we define the texts describing its FG and BG concepts using the prompts “a photo of [target class]” and “a photo without [target class]”, respectively. The CLIP text encoder takes these descriptions as input and produces the weights FfgtF^{t}_{fg} and FbgtF^{t}_{bg} for the FG and BG classes of a binary text classifier. Given a query image IqI_q, we can get its feature maps Fqclip∈Rh×w×dF^{clip}_q \in\mathbb{R}^{h \times w \times d} through the CLIP image encoder, where dd denotes the number of channels, hh and ww represent the height and width, respectively. We then apply the representative average pooling on FqclipF^{clip}_q to obtain the global embedding fqclip∈Rdf^{clip}_q \in\mathbb{R}^{d} of IqI_q, and classify it into the FG and BG classes based on the following classification scores:

Si=exp⁡(τCos⁡(fqclip,Fit))∑j∈{fg,bg}exp⁡(τCos⁡(fqclip,Fjt)),i∈{fg,bg}.(1)S_i = \frac{\exp(\tau\operatorname{Cos}(f^{clip}_q,F^t_i))}{\sum_{j \in\{fg,bg\}}\exp(\tau\operatorname{Cos}(f^{clip}_q,F^t_j))}, i \in\{fg,bg\}. \tag*{(1)}

Here, Cos⁡(⋅,⋅)\operatorname{Cos}(\cdot,\cdot) denotes cosine similarity, τ\tau is a positive temperature coefficient. Following GradCAM, we calculate the gradient of the FG classification score SfgS_{fg} with respect to each feature map of FqclipF^{clip}_q:

wm=1hw∑i∑j∂Sfg∂Fq,mclip(i,j).(2)w_m = \frac{1}{hw}\sum_i\sum_j\frac{\partial S_{fg}}{\partial F^{clip}_{q,m}(i,j)}. \tag*{(2)}

where Fq,mclipF^{clip}_{q,m} denotes the mm-th feature map, and (i,j)(i,j) indexes the spatial location of each pixel. According to this definition, wmw_m represents the semantic relevance between the mm-th feature map and the FG class. By taking wmw_m as the weight, we aggregate all feature maps to obtain our query semantic prior mask PqsP^s_q as follows:

Pqs=ReLU⁡(∑mwmFq,mclip),(3)P^s_q = \operatorname{ReLU}\left(\sum_m w_m F^{clip}_{q,m}\right), \tag*{(3)}

where ReLU(⋅)(\cdot) filters out negative responses. In this way, PqsP^s_q highlights the image regions within IqI_q that make greater contributions towards classifying IqI_q into the FG class, which are highly relevant to the target object semantics.

  1. Query FG Visual Prior Mask: To refine the semantic memory with complementary FG details, we need to extract FG-related visual information from the query image. For this purpose, we first employ the DINOv2 image encoder to convert the support and query images into their visual features FsdinoF^{dino}_s and FqdinoF^{dino}_q, respectively. Then, based on the ground-truth mask MsM_s of the support image IsI_s, we perform the masked average pooling (MAP) operation on FsdinoF^{dino}_s to generate the support FG prototype ProsfgPro_s^{fg}. This prototype is subsequently utilized as an indicator for identifying the FG visual features in FqdinoF^{dino}_q, so that our query FG visual prior mask PqvP^v_q can be generated as follows:

Pqv=Norm⁡(Cos⁡(Fqdino,Prosfg)).(4)P^v_q = \operatorname{Norm}(\operatorname{Cos}(F^{dino}_q,Pro_s^{fg})). \tag*{(4)}

Here, Norm(⋅)(\cdot) denotes the min-max normalization. By doing so, PqvP^v_q emphasizes the query image regions that exhibit more consistent features with the support FG areas.

3) Support Adaptive Visual Prior Mask: We also exploit the visual features of the support image to enhance our semantic memory. Unlike previous works that directly treat MsM_s as the support prior mask, we attempt to leverage the query-oriented FG information in FsdinoF_{s}^{dino} to generate an adaptive prior mask, thereby further mitigating the support-query appearance divergence problem. Concretely, we first apply the MAP operation to FqdinoF_{q}^{dino} to compute the query FG prototype ProqfgProq^{fg} according to the semantic prior mask PqvP_{q}^{v} defined in Eq. (3). Then, by comparing ProqfgProq^{fg} with FsdinoF_{s}^{dino}, we obtain the support adaptive visual prior mask PsvP_{s}^{v} as follows:

Psv=Ms⊙Norm⁡(Cos⁡(Fsdino,Proqfg)).(5)P_{s}^{v}=M_s\odot\operatorname{Norm}(\operatorname{Cos}(F_{s}^{dino},Proq^{fg})). \tag*{(5)}

Here, ⊙\odot represents the element-wise multiplication, and MsM_s is used to filter out the responses located in the BG regions. With this definition, PsvP_{s}^{v} highlights partial FG areas within IsI_s that are more similar to the FG regions of IqI_q.

After obtaining the three prior masks PqsP_{q}^{s}, PqvP_{q}^{v}, and PsvP_{s}^{v}, we interpolate them to the size of input images H×WH\times W, and then feed them into the SAM2 memory encoder MemEnc⁡(⋅,⋅)\operatorname{MemEnc}(\cdot,\cdot) to construct the corresponding memories:

Memqs=MemEnc⁡(Fqsam,Pqs),Memqv=MemEnc⁡(Fqsam,Pqv),Memsv=MemEnc⁡(Fssam,Psv),(6)\begin{aligned} Mem_{q}^{s}&=\operatorname{MemEnc}(F_{q}^{sam},P_{q}^{s}),\\ Mem_{q}^{v}&=\operatorname{MemEnc}(F_{q}^{sam},P_{q}^{v}),\\ Mem_{s}^{v}&=\operatorname{MemEnc}(F_{s}^{sam},P_{s}^{v}), \tag*{(6)} \end{aligned}

where FssamF_{s}^{sam} and FqsamF_{q}^{sam} denote the support and query image features produced by the SAM2 image encoder. These three memories share the same size of hmwm×dmh_mw_m\times d_m. Here, hmwmh_mw_m denotes the spatial size and dmd_m is the number of dimensions.

Interactive Semantic Memory Refinement (ISMR). In this module, the query semantic memory MemqsMem_{q}^{s} is further refined by absorbing fine-grained visual information from MemqvMem_{q}^{v} through a cross-attention operation. However, since some BG areas in PqvP_{q}^{v} may be over-activated by the similar visual cues in ProsfgPro_{s}^{fg}, MemqvMem_{q}^{v} will inevitably introduce BG noise that affects such refinement process. To address this problem, considering that MsM_s has already removed the BG responses from PsvP_{s}^{v}, the resulting MemsvMem_{s}^{v} contains only pure FG information. We therefore use it to suppress BG noise in the cross-attention scores generated above.

Specifically, MemqsMem_{q}^{s}, MemqvMem_{q}^{v}, and MemsvMem_{s}^{v} are first projected into a dcd_c-dimensional common space:

Q=Linear⁡Q(Memqs),K=Linear⁡K(Memqv),Q=\operatorname{Linear}_{Q}(Mem_{q}^{s}),\quad K=\operatorname{Linear}_{K}(Mem_{q}^{v}),
V=Linear⁡V(Memqv),Ks=Linear⁡Ks(Memsv),(7)V=\operatorname{Linear}_{V}(Mem_{q}^{v}),\quad K_s=\operatorname{Linear}_{K_s}(Mem_{s}^{v}), \tag*{(7)}

where Linear⁡i(⋅)\operatorname{Linear}_{i}(\cdot) indicate the linear projectors with different parameters. The resulting representations {Q,K,V,Ks}∈RN×dc\{Q,K,V,K_s\}\in\mathbb{R}^{N\times d_c} (N=hmwmN=h_mw_m) will be used for subsequent semantic memory refinement. Following the cross-attention mechanism, we calculate the scaled dot-product attention scores Aqq∈RN×NA_{qq}\in\mathbb{R}^{N\times N} between two query memories QQ and KK:

Aqq=Q⋅KT/dc.(8)A_{qq}=Q\cdot K^{\mathsf{T}}/\sqrt{d_c}. \tag*{(8)}

To handle the BG noise in AqqA_{qq}, the MAP operation is first imposed on KsK_s and PsvP_{s}^{v} to produce a query-oriented support FG prototype Pros∈R1×NPro_s\in\mathbb{R}^{1\times N}. We then use it to scan each spatial location of QQ through the following cosine similarity, to identify potential BG regions in the query memories:

Asqi=Pros⋅QiT∥Pros∥⋅∥Qi∥,i=1,2,…,N(9)A_{sq}^{i}=\frac{Pro_s\cdot Q_i^{\mathsf{T}}}{\lVert Pro_s\rVert\cdot\lVert Q_i\rVert},\quad i=1,2,\ldots,N \tag*{(9)}

where QiQ_i denotes the ii-th feature vector in QQ. The entry Asqi∈[−1,1]A_{sq}^{i}\in[-1,1] with a smaller value indicates the pixel that is highly likely to be located within the BG regions, as it is less similar to ProsPro_s. Note that in Eq. (9), we use QQ rather than KK to calculate Asq∈[−1,1]1×NA_{sq}\in[-1,1]^{1\times N} because the semantic memory MemqsMem_{q}^{s} typically yields more reliable predictions than MemqvMem_{q}^{v}, which may benefit the identification of BG pixels. Next, by normalizing the values of AqqA_{qq} and AsqA_{sq} to [0,1][0,1], we obtain the following attention calibration scores:

Aqq′=min⁡[Norm⁡(Aqq)+(Norm⁡(Asq)−1),0].(10)A_{qq}'=\min[\operatorname{Norm}(A_{qq})+(\operatorname{Norm}(A_{sq})-1),0]. \tag*{(10)}

Here, min⁡[⋅,0]\min[\cdot,0] preserves the negative entries in Aqq′A_{qq}' to suppress the BG activations. Finally, we calibrate the attention scores AqqA_{qq} and get the refined semantic memory as follows:

Aqq′′=Aqq+αAqq′(11)A''_{q q} = A_{q q} + \alpha A'_{q q} \tag*{(11)}
Mems′′=Memsq+Linear⁡O(Softmax⁡(Aqq′′)⋅V)(12)Mem^{\prime\prime}_{s} = Mem^{q}_{s} + \operatorname{Linear}_{O}(\operatorname{Softmax}(A''_{q q}) \cdot V) \tag*{(12)}

where Softmax⁡(⋅)\operatorname{Softmax}(\cdot) denotes the softmax computation along the column. α\alpha is a scaling factor that is set to 10.

Segmentation Mask Prediction. Following the standard pipeline of SAM2, the refined semantic memory Mems′′Mem^{\prime\prime}_{s} is jointly input with FqsF^{s}_{q} into a memory attention module to obtain their matched query features. These features are subsequently decoded by SAM2’s mask decoder to predict the final segmentation mask M^q\hat{M}_{q}.

Training Loss. We adopt the Binary Cross-Entropy (BCE) loss between the predicted and ground-truth masks to optimize our MESAM model, which is formulated as:

LBCE=−[Mqlog⁡(M^q)+(1−Mq)log⁡(1−M^q)].(13)\mathcal{L}_{\mathrm{BCE}} = -[M_{q} \log(\hat{M}_{q}) + (1 - M_{q}) \log(1 - \hat{M}_{q})]. \tag*{(13)}

Extension from 1-shot to kk-shot. The 1-shot pipeline described in § 3.2 can be easily extended to the kk-shot setting by modifying the following operations: 1) Query FG visual prior mask: We first obtain kk different support FG prototypes ProsfgPro^{fg}_{s} by processing each support sample independently, and then use them to calculate kk query visual priors through Eq. (4). Subsequently, these priors are averaged to generate the final query visual prior mask PqvP^{v}_{q}. 2) Support adaptive visual prior mask: We repeat the 1-shot operation for each support sample, yielding kk support adaptive visual prior masks PqsP^{s}_{q}. 3) BG suppression in ISMR: Each support adaptive visual memory is matched with the QQ to compute a cosine similarity map. The kk similarity maps are averaged to obtain the final similarity score AsqA_{sq}.

Experiments

Experimental Settings

Dataset. In this section, we utilize three datasets with increasing levels of difficulty for our experiments. PASCAL-5i^{i} consists of 20 categories from PASCAL VOC 2012 [7] with additional annotations from SDS [11]. COCO-20i^{i} is built upon MSCOCO [17] and contains 80 classes, presenting more challenging due to multiple tiny objects and complex BG scenarios. LVIS-92i^{i} [20] is constructed based on LVIS [10], which contains 920 fine-grained object categories and significantly higher visual complexity. With these datasets, we evaluate the segmentation performance of our method under two different FSS settings: 1) For the Standard Setting, we evenly split the object categories of PASCAL-5i^{i} and COCO-20i^{i} into four disjoint folds, and conduct the experiments in a cross-validation manner (i.e., three folds are used for model training and the remaining one for testing with 1,000 episodes sampled). 2) For the Cross-Dataset Setting, we examine the generalization ability of MESAM when the training and testing samples are drawn from different data distributions. To this end, we directly evaluate the 1-shot model trained for COCO-20i^{i} (using the standard setting) on each LVIS-92i^{i} fold, where 2,300 episodes are sampled.

Evaluation Metrics. Consistent with standard FSS protocols [44, 56], two evaluation metrics are used: 1) mIoU measures the overall segmentation quality by averaging IoU across all foreground classes; 2) FB-IoU separately calculates FG and BG IoUs, then takes their average to evaluate class balance performance.

Implementation Details. Our MESAM model is trained on 4 NVIDIA RTX 4090 (24GB) GPUs and evaluated on a single 4090 GPU. We use the AdamW optimizer to train our ISMR module, as well as SAM2’s memory encoder, memory attention module, and mask decoder, while keeping all other components frozen. The learning rate is initialized to 0.001 and scheduled with a polynomial decay strategy [34]. During training, the same data augmentation techniques as in previous works [6, 44] are employed, the input images are randomly cropped into 512 × 512-sized patches. For evaluation, the testing predictions are resized back to the original resolution to compute the mIoU and FB-IoU scores. The batch size is set to 8. The number of training epochs is set to 50 for PASCAL-5i^{i} and 10 for COCO-20i^{i}.

Comparison with State-of-the-Art Methods

Quantitative Comparison under the Standard Setting. We compare the segmentation performance of our MESAM method with that achieved by recently proposed approaches. Three traditional FSS models built upon the ResNet50 backbone and several foundation-based methods are selected for comparison. The results for the PASCAL-5i^{i} and COCO-20i^{i} datasets under both 1-shot and 5-shot settings are presented in Table 3 and Table 4. We can get the following observations from these tables: 1) Foundation-based methods generally yield better results. Even the best traditional model (PAHNet) still performs worse than most approaches built upon SAM, DINOv2, and SAM2. We believe this is because these foundation models have already acquired more transferable prior knowledge from large-scale data during their pre-training process. This knowledge and the prompt-based segmentation capability of SAM/SAM2 can both alleviate the learning burden on base classes, thereby reducing the risk of model overfitting to these classes and ultimately benefiting FSS tasks. 2) By constructing the semantic memory, MESAM significantly outperforms other competing baselines, achieving new state-of-the-art results. Specifically, compared with other SAM2-based models that rely solely on visual cues, MESAM realizes average mIoU score improvements of 4.0%-11.9%, 1.6%-12.2% on PASCAL-5i^{i}, and 5.0%-12.0%, 0.9%-11.8% on COCO-20i^{i} under the two settings, respectively. As for the FB-IoU scores, the performance gains achieved by MESAM over FSSAM are 2.2% (1-shot) and 0.9% (5-shot) on PASCAL-5i^{i}, which can be more pronounced on COCO-20i^{i}, reaching 4.0% (1-shot) and 1.2% (5-shot). All these results show the superiority of our MESAM method and the effectiveness of introducing the semantic information to empower SAM2 in addressing FSS tasks.

MethodsBackboneVenue\1-shot 505^{0}515^{1}525^{2}535^{3}MeanFB-IoU\5-shot 505^{0}515^{1}525^{2}535^{3}MeanFB-IoU
ABCBNet [54]ResNet50CVPR’2472.976.069.564.070.6-74.478.073.968.373.6-
PAHNet [56]ResNet50ICCV’2572.176.471.066.671.582.475.878.676.973.276.185.7
OCNet [38]ResNet50ICCV’2573.575.971.164.971.482.275.977.174.170.974.584.7
Matcher [20]SAM-HICLR’2467.770.766.967.068.1-71.477.574.172.874.0-
VRP-SAM [33]SAM-HCVPR’2473.978.370.665.171.9-------
GF-SAM [46]SAM-HNIPS’2471.175.769.273.372.1-81.586.379.782.982.6-
FCP [26]SAM-HAAAI’2574.977.471.868.873.2-77.278.872.267.774.0-
FounFSS [2]DINOV2Arxiv’2476.581.372.177.476.8-79.584.875.882.580.7-
FS-DINO [55]DINOV2Arxiv’2575.680.372.273.475.4-78.183.173.676.377.8-
FS-SAM2 [9]SAM2-BICIP’2574.079.673.566.773.4-75.779.675.368.874.8-
FSSAM [44]SAM2-SICML’2581.684.981.676.081.089.484.188.583.885.085.491.9
SANSA [6]SAM2-LNIPS’2578.180.068.366.173.1-------
MESAM (Ours)SAM2-S–83.788.983.783.785.091.684.290.186.287.587.092.8

Table 3. Performance comparisons on PASCAL-5i^{i}. “5i5^{i}” shows the mIoU scores of 5 novel classes in fold ii, “Mean” denotes the averaged mIoU score across all four folds. The best and second-best results are highlighted in bold and underlined, respectively.

MethodsBackboneVenue\1-shot 20020^{0}20120^{1}20220^{2}20320^{3}MeanFB-IoU\5-shot 20020^{0}20120^{1}20220^{2}20320^{3}MeanFB-IoU
ABCBNet [54]ResNet50CVPR’2444.254.052.149.850.0-50.559.157.053.655.1-
PAHNet [56]ResNet50ICCV’2545.258.254.451.252.375.253.064.160.858.759.278.1
OCNet [38]ResNet50ICCV’2545.956.952.950.451.573.752.763.157.454.857.076.8
Matcher [20]SAM-HICLR’2452.753.552.652.152.7-60.162.760.959.260.7-
VRP-SAM [33]SAM-HCVPR’2456.061.357.958.858.5-61.469.465.964.965.4-
GF-SAM [46]SAM-HNIPS’2456.661.459.657.158.7-67.169.466.064.866.8-
FCP [26]SAM-HAAAI’2546.456.455.351.852.5-52.663.359.856.158.0-
FounFSS [2]DINOV2Arxiv’2456.861.064.259.760.4-------
FS-DINO [55]DINOV2Arxiv’2558.666.062.362.662.4-62.170.266.163.065.4-
FS-SAM2 [9]SAM2-BICIP’2549.357.658.356.155.3-55.160.658.759.358.4-
FSSAM [44]SAM2-SICML’2559.965.662.161.662.377.368.674.064.569.969.382.9
SANSA [6]SAM2-LNIPS’2557.761.762.858.560.2-----67.3-
MESAM (Ours)SAM2-S–63.970.466.268.667.381.367.274.166.073.370.284.1

Table 4. Performance comparisons on COCO-20i^{i}. “20i20^{i}” shows the mIoU scores of 20 novel classes in fold ii, “Mean” denotes the averaged mIoU score across all four folds. The best and second-best results are highlighted in bold and underlined, respectively.

Quantitative Comparison under the Cross-Dataset Setting. We further evaluate our method on the more challenging LVIS-92i^{i} [10], as shown in Table 5. MESAM consistently achieves superior performance across all folds, obtaining a mean mIoU of 41.9%, which surpasses Matcher (33.0%) and SINE (31.2%) by 8.9% and 10.7%, respectively. Compared with FSSAM, which already exhibits strong cross-dataset generalization, MESAM further improves the performance by 4.6% (37.3% → 41.9%). In addition, MESAM achieves the highest FB-IoU score of 70.9%. These results further validate the effectiveness of introducing semantic information and highlight the strong generalization capability of MESAM.

Qualitative Comparison. To better understand the effectiveness of our method, we visualize the segmentation results of FSSAM (which achieves the second-best performance in most cases) and MESAM in Figure 5 to make a qualitative comparison. It can be seen that, with large appearance differences between the support and query images, FSSAM often fails to segment the target object completely, and may also produce spurious BG activations. For instance, in the fourth column of the PASCAL-5i^{i} dataset, the support image displays the frontal view of a “bottle” (bottle body), while the query image presents the bottles from a top-down view (bottle opening), causing FSSAM to misidentify the target objects and segment irrelevant regions. Similarly, in the second column of the COCO-20i^{i} dataset, the support image only shows the handle of a pair of “scissors”, resulting in FSSAM missing the scissor blades in the query image. Likewise, in the second column of the LVIS-92i^{i} dataset, the significant appearance gap between the phone “camera lens” in the support image and the “camera lens” in the query image leads FSSAM to produce incomplete segmentation and mistakenly activate on the eyes of a cat. In contrast, our MESAM method can generate relatively accurate and complete predictions for the target objects in query images, demonstrating its capability to address the challenge of substantial support-query appearance divergence.

Qualitative comparisons between FSSAM and MESAM on PASCAL-5^i, COCO-20^i and LVIS-92^i

Figure 5. Qualitative comparisons between FSSAM and MESAM on PASCAL-5i^{i}, COCO-20i^{i} and LVIS-92i^{i} under the 1-shot setting.

Methods92092^{0}92192^{1}92292^{2}92392^{3}92492^{4}92592^{5}92692^{6}92792^{7}92892^{8}92992^{9}MeanFB-IoU
Matcher [20]31.430.933.738.130.532.535.934.233.029.733.066.2
SINE [19]28.331.031.934.630.031.932.233.730.627.831.263.5
FSSAM [44]34.737.837.241.133.938.140.638.936.933.837.368.4
MESAM (Ours)38.044.743.544.838.241.444.041.244.039.441.970.9

Table 5. Performance comparisons on LVIS-92i^{i}. “92i92^{i}” shows the mIoU scores of 92 novel classes in fold ii, “Mean” denotes the averaged mIoU score across all ten folds. The 1-shot results are reported, where the best results are highlighted in bold.

Ablation Studies

In this section, we investigate how different implementations of our MESAM method affect its segmentation performance. Note that unless otherwise specified, all experiments are conducted on the COCO-20i^{i} dataset using the 1-shot setup.

Effect of Different Memories. In our SVMG module, we construct three types of memories based on their corresponding prior masks.

To study the effect of each memory, we establish our MESAM models under four different settings: “A: Only MemqsMem^{s}_{q}” performs the segmentation based solely on the query semantic memory MemqsMem^{s}_{q}; “B: Memqs+MemqvMem^{s}_{q} + Mem^{v}_{q}” refines MemqsMem^{s}_{q} only through the cross-attention with MemqvMem^{v}_{q}; “C: Memqs+Memqv+Memsv(Ms)Mem^{s}_{q} + Mem^{v}_{q} + Mem^{v}_{s}(M_{s})” calibrates the above refinement process using the support memory Memsv(Ms)Mem^{v}_{s}(M_{s}), which directly takes MsM_{s} as the prior mask; “D: Memqs+Memqv+MemsvMem^{s}_{q} + Mem^{v}_{q} + Mem^{v}_{s}” is the full MESAM model. The results obtained by these variations are shown in Table 6. As can be seen, 1) Since the semantic memory alone cannot represent all the FG features in each query image, Setting A achieves the lowest mIoU score of 58.9%, showing the importance of our ISMR module. 2) Settings C and D both out-perform Setting B, suggesting that suppressing the BG noise in attention scores facilitates the memory refinement process. 3) Setting D surpasses Setting C by a large margin of 1.1%, indicating the effectiveness of our generated support visual prior mask PqsP^{s}_{q} in identifying the query-oriented FG features within support images.

Settings200^{0}201^{1}202^{2}203^{3}Mean
A: Only MemqsMem^{s}_{q}51.267.158.458.758.9
B: Memqs+MemqvMem^{s}_{q} + Mem^{v}_{q}62.168.965.166.965.8
C: Memqs+Memqv+Memsv(Ms)Mem^{s}_{q} + Mem^{v}_{q} + Mem^{v}_{s}(M_{s})61.969.964.768.166.2
D: Memqs+Memqv+MemsvMem^{s}_{q} + Mem^{v}_{q} + Mem^{v}_{s}63.970.466.268.667.3

Table 6. Ablation study on the effect of different memories.

Figure 6 visualizes the cross-attention weights for a selected FG pixel (see orange cross) and segmentation results obtained under Settings B and D, i.e., w/o and w/ the support calibration. We can find that using pure FG information in MemqvMem^{v}_{q} to calibrate attention scores can effectively reduce the influence of misactivated BG areas (see blue circles), thus improving the segmentation performance.

The cross-attention weights and the segmentation results obtained without and with the support calibration

Figure 6. The cross-attention weights and the segmentation results obtained without and with the support calibration.

Effect of Different ISMR Designs. We also explore the best design of our ISMR module. We modify some operations in ISMR and report the mIoU scores achieved by the modified modules in Table 7. When the computation of AsqA_{sq} in Eq. (9) is based on the cosine similarity over KK instead of QQ (“Cosine over KK”), the overall performance drops from 67.3% to 66.4%. This indicates that QQ is more effective than KK in identifying potential background regions because it derives from more reliable semantic memory. When replacing min⁡[.,0]\min[.,0] in Eq. (10) with max⁡[.,0]\max[.,0] to preserve the positive entries in Aqq′′A''_{qq} (“FG Enhancement”), the negative impact of BG noise remains unmitigated, leading to a 0.6% decrease in segmentation accuracy. Finally, swapping the roles of semantic and visual memories in the cross-attention (“Memory Swap”) results in a 0.5% performance degradation, showing the effectiveness of treating the semantic memory as a foundation and enhancing it with visual information.

200^{0}201^{1}202^{2}203^{3}Mean
Cosine over KK61.770.166.767.166.4
FG Enhancement62.069.867.267.766.7
Memory Swap61.170.567.867.966.8
ISMR in MESAM63.970.466.268.667.3

Table 7. Ablation study on the effect of different ISMR.

Conclusion

This work reveals the superiority of the semantic memory in handling support-query appearance divergences when applying SAM2 to FSS tasks, and explains why it exhibits stronger compatibility with SAM2. With this analysis, we develop our MESAM method primarily upon the semantic memory, which is generated based on the GradCAM of CLIP’s text classifier. We then utilize the visual memories derived from the DINOv2 image representations to enhance such semantic memory, enabling it to integrate more fine-grained visual details. Experimental results on PASCAL-5i^{i}, COCO-20i^{i}, and LVIS-92i^{i} show that MESAM consistently achieves new state-of-the-art performance, demonstrating its effectiveness.

Acknowledgments

This work was supported by the Young Scientists Fund of the National Natural Science Foundation of China (Grant No. 62306219) and the Project of Sanya Yazhou Bay Science and Technology City (Grant No. SKJC-JYRC-2024-65).

References

Mask Erosion and Dilation

Algorithmic Details

To analyze the impact of different prior coverage patterns on memory construction in SAM2-based FSS, we conduct a toy experiment that generates pseudo prior masks with varying spatial extents by applying the mask dilation, vanilla-erosion, and skeleton-erosion to the ground-truth segmentation masks of query images. These operations are designed to simulate different statistical behaviors observed in real visual and semantic prior masks.

Algorithm 1 Mask Dilation

Input: Query ground-truth mask MqM_q, target ratio rr, kernel size kk, padding parameter padpad

Output: Dilated mask MM

Stage 1: Initialization

M←MqM \leftarrow M_q;

kernel←ones⁡(k,k)kernel \leftarrow\operatorname{ones}(k,k);

initial_area←initial\_area \leftarrow sum of non-zero pixels in MM;

target_area←initial_area×(1+r)target\_area \leftarrow initial\_area \times(1+r);

current_area←initial_areacurrent\_area \leftarrow initial\_area;

step←0step \leftarrow0;

Stage 2: Iterative Dilation

while current_area<target_areacurrent\_area < target\_area do

  1. M←I(Conv2D⁡(M,kernel,pad)>0)M \leftarrow\mathbb{I}(\operatorname{Conv2D}(M,kernel,pad)>0);

  2. current_area←sum⁡(I(M>0))current\_area \leftarrow\operatorname{sum}(\mathbb{I}(M>0));

  3. step←step+1step \leftarrow step+1;

  4. if step>1000step>1000: then break;

M←I(M>0)M \leftarrow\mathbb{I}(M>0)

Algorithm 2 Mask Erosion

Input: Query ground-truth mask MqM_q, target ratio rr, kernel size kk, padding parameter padpad

Output: Eroded mask MM

Stage 1: Initialization

M←MqM \leftarrow M_q;

kernel←ones⁡(k,k)kernel \leftarrow\operatorname{ones}(k,k);

initial_area←initial\_area \leftarrow sum of non-zero pixels in MM;

target_area←initial_area×(1−r)target\_area \leftarrow initial\_area \times(1-r);

current_area←initial_areacurrent\_area \leftarrow initial\_area;

step←0step \leftarrow0;

S←Skeleton⁡(M)S \leftarrow\operatorname{Skeleton}(M); // Generating skeleton from mask

Stage 2: Iterative Erosion (Two implementations)

Option 2a: Vanilla-Erosion

while current_area>target_areacurrent\_area > target\_area do

  1. M←1−I(Conv2D⁡(1−M,kernel,pad)>0)M \leftarrow1-\mathbb{I}(\operatorname{Conv2D}(1-M,kernel,pad)>0);

  2. current_area←sum⁡(I(M>0))current\_area \leftarrow\operatorname{sum}(\mathbb{I}(M>0));

  3. step←step+1step \leftarrow step+1;

  4. if step>1000step>1000: then break;

M←I(M>0)M \leftarrow\mathbb{I}(M>0)

Option 2b: Skeleton-Erosion

while current_area>target_areacurrent\_area > target\_area do

  1. M←1−I(Conv2D⁡(1−M,kernel,pad)>0)M \leftarrow1-\mathbb{I}(\operatorname{Conv2D}(1-M,kernel,pad)>0);

  2. current_area←sum⁡(I(M+S>0))current\_area \leftarrow\operatorname{sum}(\mathbb{I}(M+S>0));

  3. step←step+1step \leftarrow step+1;

  4. if step>1000step>1000: then break;

M←I(M+S>0)M \leftarrow\mathbb{I}(M+S>0)

Dilation. As shown in Algorithm 1, starting from the query ground-truth mask MqM_q, the dilation operation repeatedly performs a 2D convolution (with a 3×33 \times3 all-ones kernel and a padding parameter padpad) on the mask MM obtained in the previous iteration, until the area of its FG regions becomes 1+r1+r times larger than that in MqM_q. This process typically introduces additional BG pixels, similar to the FG visual prior mask with higher FP rates.

Vanilla-Erosion. The only difference between the vanilla-erosion and dilation operations lies in Step 1 of Option 2a within Algorithm 2, where the 2D convolution is applied to 1−M1-M and the resulting mask is inverted to obtain the current MM. The iteration terminates when the FG area of MM is 1−r1-r times smaller than that in MqM_q. The mask generated by this operation will exhibit high reliability because the BG regions are filtered out. However, it will also result in the loss of certain important FG information, thus leading to higher FN rates, similar to the DISC visual prior mask.

Skeleton-Erosion. As shown in Option 2b of Algorithm 2, the FG regions obtained by the skeleton-erosion operation are defined as the union area of the eroded mask MM and the skeleton SS (see M+SM+S in Step 2). By doing so, this process further extends the vanilla-erosion by preserving the representative skeletal structure of target objects in MqM_q unchanged. Consequently, the resulting mask MM is similar to our semantic prior mask, which demonstrates both high reliability and representativeness.

Qualitative Analysis

In Figure 7, we visualize the prior masks generated by these three operations under different target ratios and their corresponding segmentation results. As can be seen, the dilation operation introduces BG noise into the prior mask, resulting in erroneous activations within BG regions (see blue circles). Meanwhile, the vanilla-erosion removes some FG details, leading to incomplete segmentation of targets or the omission of certain tiny objects (see orange circles). In contrast, by retaining the skeleton SS, the skeleton-erosion yields much better predictions than the other two operations, even though its prior mask is quite similar to that produced by the vanilla-erosion. These results indicate that generating prior masks with reliable and representative FG information is crucial for achieving precise FSS segmentation with SAM2. This also explains why our semantic memory is compatible with SAM2, as it introduces classification-oriented semantic information that demonstrates both reliability and representativeness for the target category.

Qualitative comparisons between dilation, vanilla-erosion, and skeleton-erosion operations with varying target ratios.

Figure 7. Qualitative comparisons between dilation, vanilla-erosion, and skeleton-erosion operations with varying target ratios.

As a supplement to the main text, we present more prior masks and segmentation results of two visual memories (i.e., FG and DISC) and our semantic memory in Figure 8 and Figure 9. This provides additional evidence to support our claim: The semantic memory outperforms visual memories in solving FSS tasks. But it cannot cover all FG details in query images by itself, so it is necessary to refine it with fine-grained visual information. Such insight motivates the design of our MESAM method.

Prior masks and segmentation results of visual memories and semantic memory on PASCAL-5ᵢ

Figure 8. The prior masks and the segmentation results of visual memories (FG, DISC) and semantic memory on PASCAL-5i^{i}.

Prior masks and segmentation results of visual memories and semantic memory on COCO-20ᵢ

Figure 9. The prior masks and the segmentation results of visual memories (FG, DISC) and semantic memory on COCO-20i^{i}.

Additional Experimental Results

Performance Stability

We repeat the testing procedure of our MESAM method five times using different random seeds {321\{321 (default), 0, 1, 2, 3}\} to investigate its performance stability. Note that these seeds are used only to determine the 1,000 testing episodes by selecting different support and query images from PASCAL-5i5^{i} and COCO-20i20^{i}. The training process is conducted under the default setting, where the seed value is fixed to 321. From Table 8 and Table 9, we can observe that: 1) On

Seed1-shot1-shot1-shot1-shot1-shot5-shot5-shot5-shot5-shot5-shot
505^{0}515^{1}525^{2}535^{3}Mean505^{0}515^{1}525^{2}535^{3}Mean
default83.7088.9383.7483.7085.0284.2190.0886.2187.5087.00
083.7589.2983.8883.3885.0883.6589.5786.4987.6986.85
183.4488.7583.4783.4684.7883.9589.9186.3887.4586.92
283.4389.1883.3883.3084.8283.9289.7786.0387.8786.90
383.3889.2383.8283.3384.9484.1189.6686.6287.1586.89
Average83.5489.0883.6683.4384.9383.9789.8086.3587.5386.91
Std0.170.230.220.160.130.210.200.230.270.06

Table 8. Performance of MESAM on PASCAL-5i5^{i} with five random seeds {321 (default), 0, 1, 2, 3}. 1,000 episodes are randomly sampled for evaluation. “Average” and “Std” denote the mean mIoU scores and the standard deviations across different random seeds.

Seed1-shot1-shot1-shot1-shot1-shot5-shot5-shot5-shot5-shot5-shot
20020^{0}20120^{1}20220^{2}20320^{3}Mean20020^{0}20120^{1}20220^{2}20320^{3}Mean
default63.9270.4166.1768.6367.2867.2274.1366.0173.3270.17
062.9770.5965.1367.4866.5465.5972.8468.1672.6469.81
163.5570.3365.4667.2166.6468.5274.4563.3172.7969.77
263.3870.1262.3367.5965.8668.1673.4566.9672.7070.32
362.6570.6064.9468.0766.5765.6173.6567.2571.8569.59
Average63.2970.4164.8167.8066.5867.0273.7066.3472.6669.93
Std0.500.201.460.560.511.380.621.860.530.30

Table 9. Performance of MESAM on COCO-20i20^{i} with five random seeds {321 (default), 0, 1, 2, 3}. 1,000 episodes are randomly sampled for evaluation. “Average” and “Std” denote the mean mIoU scores and the standard deviations across different random seeds.

PASCAL-5i^{i}, the mIoU scores averaged over five seeds are 84.93% ±\pm 0.13% (1-shot) and 86.91% ±\pm 0.06% (5-shot), showing high consistency with the results reported in the main text achieved under the default setting. The small standard deviations indicate that MESAM demonstrates strong robustness to the randomness in this dataset. 2) On COCO-20i^{i}, MESAM achieves results of 66.58% ±\pm 0.51% (1-shot) and 69.93% ±\pm 0.30% (5-shot), whose standard deviations are larger than those on PASCAL-5i^{i}, indicating that the performance on this dataset is more sensitive to episode sampling. It is mainly because the images in COCO-20i^{i} often contain multiple objects, small-sized targets, and complex background scenes. This makes the episodes sampled by different seeds vary significantly in segmentation difficulty, ultimately leading to performance fluctuations. Nevertheless,

MESAM with different random seeds still consistently outperforms other competing methods. As previously discussed, owing to the complexity of the COCO-20i20^{i} dataset, using only 1,000 episodes may lead to biased or unstable evaluation of model performance. Therefore, we vary the number of testing episodes within the range of {1,000 (default), 4,000, 10,000, 20,000} and summarize the results in Table 10 to analyze how MESAM performs on more episodes. It can be observed that the mIoU scores obtained with 1,000 episodes exhibit no significant difference compared to those averaged across 1,000, 4,000, 10,000, and 20,000 episodes. Specifically, under 1-shot and 5-shot settings, using 1,000 episodes for evaluation yields mIoU scores of 67.28% and 70.17%, respectively, while the average scores are 67.10% and 70.74%. The differences between these two are merely 0.18% and 0.57%, showing the stability of our MESAM method.

#Episodes1-shot1-shot1-shot1-shot1-shot1-shot5-shot5-shot5-shot5-shot5-shot5-shot
20020^{0}20120^{1}20220^{2}20320^{3}MeanTime (s)20020^{0}20120^{1}20220^{2}20320^{3}MeanTime (s)
100063.9270.4166.1768.6367.28143.167.2274.1366.0173.3270.17218.7
400065.3869.8668.6866.5767.62568.266.7573.8172.5272.3271.35881.4
1000063.6670.3567.2765.4366.681406.566.3873.7370.7371.8470.672183.2
2000063.8070.5567.4165.5466.832799.966.5773.8770.6971.8770.754425.7
Average64.1970.2967.3866.5467.101229.466.7373.8969.9972.3470.741927.3
Std0.800.301.031.480.43-0.30.172.790.690.48-

Table 10. Performance of MESAM on COCO-20i20^{i} evaluated with varying numbers of testing episodes {1,000, 4,000, 10,000, 20,000}. “Time” denotes the total inference time for the evaluation process. “Average” and “Std” denote the mean mIoU scores and the standard deviations across different numbers of testing episodes.

Inference Time Analysis

In Table 10, we also report the inference time (s) of MESAM under each episode configuration. The evaluations are conducted on a single NVIDIA RTX 4090 GPU with 24GB memory, and the SAM2-S model is used as our backbone. Although the deployment of large foundation models inevitably introduces additional computational overhead, the inference time of our method is still acceptable. Concretely, MESAM achieves processing speeds of around 7.1 FPS (1-shot) and 4.6 FPS (5-shot), which are comparable to FSSAM [44] (7.3 FPS and 4.6 FPS). Meanwhile, MESAM achieves significant performance gains of 5.0% and 0.9% over FSSAM under the two settings. This demonstrates that our method maintains a favorable trade-off between segmentation accuracy and model efficiency, whilst benefiting from strong foundation models.

Ablation Study on α\alpha

We further analyze the impact of the scaling factor α\alpha in the attention calibration (Eq. (11)). As shown in Figure 10, the performance first improves as α\alpha increases from 1 to 10, with mIoU rising from 61.99 to a peak of 63.90. This suggests that properly amplifying the negative components in Aqq′A^{\prime}_{q q} helps suppress BG activations and enhances

Figure 10. Ablation study of the scaling factor α\alpha on COCO-20i^{i}.

the quality of attention calibration. However, when α\alpha continues to increase beyond 10, the performance gradually declines (62.16 at α=15\alpha=15 and 61.81 at α=20\alpha=20), indicating that excessively large α\alpha may over-amplify negative responses, leading to biased attention scores and degraded segmentation quality. Overall, α=10\alpha=10 achieves the best trade-off between effective BG suppression and stable attention calibration, and is adopted in our final model.

Effect of Different SAM2 Sizes

We then investigate the effect of different SAM2 sizes on the performance of our method. We build different MESAM models upon four SAM2 versions (including tiny (T), small (S), base (B), and large (L)) and report their segmentation accuracy in Table 11. It can be found that SAM2-T and SAM2-S achieve comparable mIoU scores (both 67.3%), while SAM2-B performs slightly worse (66.5%). Moreover, although SAM2-L yields the best results, its improvements over other smaller variants are marginal, particularly considering its significantly larger parameter size (224.4 M). These results indicate that the effectiveness of our method does not rely on large SAM2 backbones and can achieve strong performance with a relatively lightweight SAM2 version (i.e., SAM2-S). This is desirable for practical deployment when computational resources are limited.

BackbonesSize (M)200^{0}201^{1}202^{2}203^{3}Mean
SAM2-T38.962.870.666.469.467.3
SAM2-S46.063.970.466.268.667.3
SAM2-B80.863.069.865.667.766.5
SAM2-L224.462.471.168.768.467.7

Table 11. Performance comparisons between different MESAM models built upon four SAM2 versions.

Cross-Dataset Generalization

In this subsection, following HSNet [24], we compare the cross-dataset segmentation performance of different methods under a COCO-20i^{i}→PASCAL-5i^{i} setting, where models are trained on COCO-20i^{i} and directly tested on PASCAL-5i^{i}. The object categories in the training and testing sets are disjoint. As shown in Table 12, MESAM achieves substantial performance gains across all four folds, outperforming SANSA by 13.2% and FSSAM by 3.9% on average, indicating its superior generalization capability under data distribution shifts. Moreover, compared to Table 3 in the main text, the performances of foundation-based approaches (i.e., GF-SAM, SANSA, FSSAM, and MESAM) are not significantly degraded, where FSSAM and MESAM even produce better results. It may be because the knowledge learned from the more complicated COCO-20i^{i} dataset can be effectively applied to the relatively easier PASCAL-5i^{i} dataset.

Methods50^{0}51^{1}52^{2}53^{3}Mean
HSNet [24]47.065.267.177.164.1
VAT [12]68.364.967.579.865.1
PMNet [3]71.072.366.663.868.4
GF-SAM [46]71.175.769.273.372.1
SANSA [6]67.170.973.578.672.5
FSSAM [44]79.585.785.476.581.8
MESAM (ours)83.089.888.281.985.7

Table 12. Performance comparisons under COCO-20i^{i}→PASCAL-5i^{i} cross-dataset setting. The 1-shot results are reported, where the best results are highlighted in bold.

Additional Qualitative Comparison

Figure 12 provides additional qualitative comparisons between FSSAM and MESAM. Consistent with the main text, FSSAM fails to handle the significant appearance differences between support and query images, leading to incomplete segmentation (e.g., the “Bus” and “Cup” in the first row) and erroneous BG activations (e.g., the human head and bags in the second row). Furthermore, we present additional cross-dataset qualitative results on the LVIS-92i92^{i} dataset. As shown in the last row of Figure 11, FSSAM fails to correctly segment the “birdcage” due to appearance discrepancy, leading to missing regions and erroneous activations. By contrast, the predictions yielded by MESAM are more complete and accurate, illustrating that introducing the semantic memory can effectively enhance MESAM’s robustness against support-query appearance divergence.

Additional qualitative comparisons showing Support, Query, FSSAM, and MESAM results

Figure 11. Additional qualitative comparisons between FSSAM and MESAM on LVIS-92i92^{i} under the 1-shot setting.

Additional qualitative comparisons between FSSAM and MESAM on PASCAL-$5^{i}$ and COCO-$20^{i}$ under the 1-shot setting

Figure 12. Additional qualitative comparisons between FSSAM and MESAM on PASCAL-5i5^{i} and COCO-20i20^{i} under the 1-shot setting.

Limitations and Future Works.

By incorporating the effective memory mechanism we proposed, MESAM realizes consistent improvement over existing FSS approaches. However, as described in the main text, we process multiple support samples by simply averaging their derived prior masks and cosine-similarity maps. These operations may fail to fully leverage the diverse visual information contained in different support images. As a result, our method achieves relatively less significant performance gains under the 5-shot setting. Therefore, in future work, we will explore more effective multi-support fusion mechanisms and adaptive memory selection strategies, enabling the model to better benefit from additional annotations. Moreover, MESAM still requires a training process to update its parameters before it can adapt to the target dataset, which can increase the computational overhead. This motivates us to design more efficient training-free or test-time adaptation strategies, which we will pursue as a future research direction.

References

  1. [1]Yunhao Bai, Boxiang Yun, Zeli Chen, Qinji Yu, Yingda Xia, and Yan Wang. 2024. RevSAM2: Prompt SAM2 for Medical Image Segmentation via Reverse-Propagation without Fine-tuning. arXiv preprint arXiv:2409.04298 (2024).arxiv.org/abs/2409.04298
  2. [2]Shijie Chang, Lihe Zhang, and Huichuan Lu. 2024. High-Performance Few-Shot Segmentation with Foundation Models: An Empirical Study. arXiv preprint arXiv:2409.06305 (2024).arxiv.org/abs/2409.06305
  3. [3]Hao Chen, Yonghan Dong, Zheming Lu, Yunlong Yu, and Jungong Han. 2024. Pixel matching network for cross-domain few-shot segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 978–987.DOI
  4. [4]Bowen Cheng, Ishani Misra, Alexander Schwing, Alexander Kirillov, and Rohit Girdhar. 2022. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1290–1299.
  5. [5]Chong Chung, Chunbo Lang, and Junwei Han. 2022. Holistic prototype activation for few-shot segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 4 (2022), 4650–4666.
  6. [6]Claudia Cuttano, Gabriele Trivigno, Giuseppe Averta, and Carlo Masone. 2025. SANSA: Unleashing the Hidden Semantics in SAM2 for Few-Shot Segmentation. In Advances in Neural Information Processing Systems.arxiv.org/abs/2505.21795
  7. [7]Mark Everingham, Luc Van Gool, Christopher K. Williams, John Winn, and Andrew Zisserman. 2010. The pascal visual object classes (voc) challenge. International Journal of Computer Vision 88 (2010), 303–338.DOI
  8. [8]Qi Fan, Wenjie Pei, Yu-Wing Tai, and Chi-Keung Tang. 2022. Self-support few-shot semantic segmentation. In European Conference on Computer Vision. Springer, 701–719.arxiv.org/abs/2207.11549
  9. [9]Bernardo Forni, Gabriele Lombardi, Federico Pozzi, and Mirco Planamente. 2025. FS-SAM2: Adapting Segment Anything Model 2 for Few-Shot Semantic Segmentation via Low-Rank Adaptation. In International Conference on Image Analysis and Processing. Springer, 67–78.arxiv.org/abs/2509.12105
  10. [10]Agrim Gupta, Piotr Dollár, and Ross Girshick. 2019. LVIS: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5356–5364.arxiv.org/abs/1908.03195
  11. [11]Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. 2011. Semantic contours from inverse detectors. In Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, 991–998.DOI
  12. [12]Sunghwan Hong, Seokju Cho, Jisu Nam, Stephen Lin, and Seungryong Kim. 2022. Cost aggregation with 4d convolutional swin transformer for few-shot segmentation. In European Conference on Computer Vision. Springer, 108–126.arxiv.org/abs/2207.10866
  13. [13]Tao Hu, Pengwan Yang, Chilang Zhang, Gang Yu, Yadong Mu, and Ces CM Snoek. 2019. Attention-based multi-context guiding for few-shot semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 8441–8448.DOI
  14. [14]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4015–4026.arxiv.org/abs/2304.02643
  15. [15]Dong-Hyun Lee. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, Vol. 3. Atlanta, 896.
  16. [16]Gen Li, Varun Jampani, Laura Sevilla-Lara, Deqing Sun, Jonghyun Kim, and Joongkyu Kim. 2021. Adaptive prototype learning and allocation for few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8334–8343.arxiv.org/abs/2104.01893
  17. [17]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European Conference on Computer Vision. Springer, 740–755.arxiv.org/abs/1405.0312
  18. [18]Jie Liu, Yanqi Bao, Guo-Sen Xie, Huan Xiong, Jan-Jakob Sonke, and Efstratios Gavves. 2022. Dynamic prototype convolution network for few-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11553–11562.arxiv.org/abs/2204.10638
  19. [19]Yang Liu, Chenchen Jing, Hengtao Li, Muzhi Zhu, Hao Chen, Xinlong Wang, and Chunhua Shen. 2024. A simple image segmentation framework via in-context examples. Advances in Neural Information Processing Systems 37 (2024), 25095–25119.arxiv.org/abs/2410.04842
  20. [20]Yang Liu, Muzhi Zhu, Hengtao Li, Hao Chen, Xinlong Wang, and Chunhua Shen. 2024. Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching. In International Conference on Learning Representations. 17904–17925.arxiv.org/abs/2305.13310
  21. [21]Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3431–3440.
  22. [22]Xinyu Mao, Xiaohan Xing, Fei Meng, Jianbang Liu, Fan Bai, Qiang Nie, and Max Meng. 2025. One Polyp Identifies All: One-Shot Polyp Segmentation with SAM via Cascaded Priors and Iterative Prompt Evolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 24182–24191.arxiv.org/abs/2507.16337
  23. [23]Lingchen Meng, Shiyi Lan, Hengduo Li, Jose M Alvarez, Zuxuan Wu, and Yu-Gang Jiang. 2024. Segic: Unleashing the emergent correspondence for in-context segmentation. In European Conference on Computer Vision. Springer, 203–220.arxiv.org/abs/2311.14671
  24. [24]Juhong Min, Dahyun Kang, and Minsu Cho. 2021. Hypercorrelation squeeze for few-shot segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 6941–6952.arxiv.org/abs/2104.01538
  25. [25]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaeldin El-Nouby, et al. 2024. DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research (2024).
  26. [26]Suho Park, SuBeen Lee, Hyun Seok Seong, Jaejoon Yoo, and Jae-il Heo. 2025. Foreground-covering prototype generation and matching for sam-aided few-shot segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 6425–6433.
  27. [27]Bohao Peng, Zhuotao Tian, Xiaoyang Wu, Chengyao Wang, Shu Liu, Jingyong Su, and Jiaya Jia. 2023. Hierarchical dense correlation distillation for few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 23641–23651.arxiv.org/abs/2303.14652
  28. [28]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. 8748–8763.arxiv.org/abs/2103.00020
  29. [29]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädlе, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. 2025. SAM 2: Segment Anything in Images and Videos. In International Conference on Learning Representations. 28085–28128.
  30. [30]Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 618–626.
  31. [31]Amirreza Shaban, Bryan Bansal, Zhen Liu, Irfan Essa, and Byron Boots. 2017. One-Shot Learning for Semantic Segmentation. In Proceedings of the British Machine Vision Conference (BMVC). BMVA Press, Article 167, 13 pages.arxiv.org/abs/1709.03410
  32. [32]Xinyu Shi, Dong Wei, Yu Zhang, Donghuan Lu, Muhan Ning, Jiashun Chen, Kai Ma, and Yefeng Zheng. 2022. Dense cross-query-and-support attention weighted mask aggregation for few-shot segmentation. In European Conference on Computer Vision. Springer, 151–168.arxiv.org/abs/2207.08549
  33. [33]Yanpeng Sun, Jiahui Chen, Shan Zhang, Xinyu Zhang, Qiang Chen, Gang Zhang, Errui Ding, Jingdong Wang, and Zechao Li. 2024. VRP-SAM: SAM with visual reference prompt. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 23565–23574.arxiv.org/abs/2402.17726
  34. [34]Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, and Jiaya Jia. 2020. Prior guided feature enrichment network for few-shot segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 2 (2020), 1050–1065.arxiv.org/abs/2008.01449
  35. [35]Haoran Wang, Zekun Li, Jian Zhang, Lei Qi, and Yinghuan Shi. 2025. Correspondence is Video: Test-Time Adaptation on SAM2 for Reference Segmentation in the Wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 8177–8186.arxiv.org/abs/2508.07759
  36. [36]Haochen Wang, Xudong Zhang, Yuato Hu, Yandan Yang, Xianbin Cao, and Xiangtong Shen. 2020. Few-shot semantic segmentation with decimatic attention networks. In European Conference on Computer Vision. Springer, 730–746.DOI
  37. [37]Xiaoqi Wang, Clint Sebastian, Wenbin He, and Liu Ren. 2025. ProSAM: Enhancing the Robustness of SAM-based Visual Reference Segmentation with Probabilistic Prompts. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 20487–20496.arxiv.org/abs/2506.21835
  38. [38]Chunlin Wen, Yu Zhang, Jie Fan, Hongyuan Zhu, Xiu-Shen Wei, Yijun Wang, Zhiqiang Kou, and Shuzhou Sun. 2025. Object-level Correlation for Few-Shot Segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 23689–23699.arxiv.org/abs/2509.07917
  39. [39]Guo-Sen Xie, Jie Liu, Huan Xiong, and Ling Shao. 2021. Scale-aware graph neural network for few-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5475–5484.DOI
  40. [40]Yang Xing, Jiong Wu, Yuheng Bu, and Kuang Gong. 2025. SAM2-SGP: Enhancing SAM2 for Medical Image Segmentation via Support-Set Guided Prompting. arXiv preprint arXiv:2506.19658 (2025).arxiv.org/abs/2506.19658
  41. [41]Hang Xiong, Runmin Cong, Jinping Chen, Chen Zhang, Feng Li, Huihui Bai, and Sam Kwong. 2025. MM-Prompt: Multi-modality and Multi-granularity Prompts for Few-Shot Segmentation. In Proceedings of the 33rd ACM International Conference on Multimedia. 3067–3075.
  42. [42]Qianxiong Xu, Guosheng Lin, Chen Change Loy, Cheng Long, Ziyue Li, and Rui Zhao. 2025. Eliminating feature ambiguity for few-shot segmentation. In European Conference on Computer Vision. Springer, 416–433.arxiv.org/abs/2407.09842
  43. [43]Qianxiong Xu, Wentang Zhao, Guosheng Lin, and Cheng Long. 2023. Self-calibrated cross attention network for few-shot segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 655–665.arxiv.org/abs/2308.09294
  44. [44]Qianxiong Xu, Lanyun Zhu, Xuanyi Liu, Guosheng Lin, Cheng Long, Ziyue Li, and Rui Zhao. 2025. Unlocking the Power of SAM 2 for Few-Shot Segmentation. In International Conference on Machine Learning.arxiv.org/abs/2505.14100
  45. [45]Buyu Yang, Chang Liu, Bohao Li, Jianbin Jiao, and Qixiang Ye. 2020. Prototype mixture models for few-shot semantic segmentation. In European Conference on Computer Vision. Springer, 763–778.
  46. [46]Anqi Zhang, Guangyu Gao, Jianbo Jiao, Chi Liu, and Yunchao Wei. 2024. Bridge the points: Graph-based few-shot segment anything semantically. Advances in Neural Information Processing Systems 37 (2024), 32332–33261.arxiv.org/abs/2410.06964
  47. [47]Bingfeng Zhang, Jimin Xiao, and Terry Qin. 2021. Self-guided and cross-guided learning for few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8312–8321.arxiv.org/abs/2103.16129
  48. [48]Gengwei Zhang, Guoliang Kang, Yi Yang, and Yunchao Wei. 2021. Few-shot segmentation via cycle-consistent transformer. Advances in Neural Information Processing Systems 34 (2021), 21984–21996.arxiv.org/abs/2106.02320
  49. [49]Gengwei Zhang, Shant Navasardyan, Ling Chen, Yao Zhao, Yunchao Wei, Humphrey Shi, et al. 2022. Mask matching transformer for few-shot segmentation. Advances in Neural Information Processing Systems 35 (2022), 823–836.arxiv.org/abs/2301.01208
  50. [50]Junyi Zhang, Charles Herrmann, Junhwa Hur, Eric Chen, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. 2024. Telling left from right: Identifying geometry-aware semantic correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3076–3085.arxiv.org/abs/2311.17034
  51. [51]Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. 2023. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. Advances in Neural Information Processing Systems 36 (2023), 45533–45547.arxiv.org/abs/2305.15347
  52. [52]Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S Huang. 2020. Sg-one: Similarity guidance network for one-shot semantic segmentation. IEEE Transactions on Cybernetics 50, 9 (2020), 3855–3865.DOI
  53. [53]Ziqin Zhou, Hai-Ming Xu, Yangyang Shu, and Lingqiao Liu. 2024. Unlocking the potential of pre-trained vision transformers for few-shot semantic segmentation through relationship descriptors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3817–3827.DOI
  54. [54]Lanyun Zhu, Tianrun Chen, Jianxiong Yin, Simon See, and Jun Liu. 2024. Addressing Background Context Bias in Few-Shot Segmentation through Iterative Modulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3370–3379.
  55. [55]Wei Zhuo, Zhiye Tang, Wufang Xue, Hao Ding, and Linlin Shen. 2025. DINOv2-powered Few-Shot Semantic Segmentation: A Unified Framework via Cross-Model Distillation and 4D Correlation Mining. arXiv preprint arXiv:2504.15669 (2025).
  56. [56]Tianyu Zou, Shengwu Xiong, Ruilin Yao, and Yi Rong. 2025. Balancing Conservatism and Aggressiveness: Prototype-Affinity Hybrid Network for Few-Shot Segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 20561–20571.arxiv.org/abs/2507.19140
  57. [57]Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. 2023. Segment everything everywhere all at once. Advances in Neural Information Processing Systems 36 (2023), 19769–19782.arxiv.org/abs/2304.06718
  58. [58]Haiyue Zu, Jun Ge, Heting Xiao, Jile Xie, Zhangzhe Zhou, Yifan Meng, Jiayi Ni, Junjie Niu, Linlin Zhang, Li Ni, et al. 2025. Rethinking Few-Shot Medical Image Segmentation by SAM2: A Training-Free Framework with Augmentative Prompting and Dynamic Matching. arXiv preprint arXiv:2503.04826 (2025).arxiv.org/abs/2503.04826

Paper details

vlnc:W_7db3vgpv
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 12

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents