Efficient SAM3 Adaptation for Multi-Class Semantic Segmentation via Dense Competitive Representations

Introduction

The evolution of Visual Foundation Models (VFMs) [10, 42, 49, 27, 46, 9] achieved a breakthrough with the Segment Anything Model 3 [9] (SAM3). As a pioneering architecture in Promptable Concept Segmentation (PCS), SAM3 [9] demonstrates exceptional capability in single-concept segmentation guided by short noun phrases (e.g., a red billboard). By incorporating deep cross-modal attention mechanisms, the model leverages textual prompts to dynamically condition visual representations, enabling precise mask generation.

While SAM3 [9] excels at single-concept segmentation, complex real-world scenarios [31, 33, 6] (e.g., autonomous driving and remote sensing) require simultaneous identification and segmentation of multiple concepts within a single scene. Conventional semantic segmentation methods [62, 11, 19, 25] achieve multi-class segmentation by assigning individual pixels to a set of predefined classes. However, these methods face limitations in complex environments. They typically rely on narrow, task-specific feature representations and lack broad semantic priors of large-scale foundation models. On the other hand, SAM3 [9] leverage robust visual-linguistic representations to achieve remarkable

Preprint. performance in single-concept segmentation. However, it faces two critical challenges when extended to multi-class scenarios: ❶ SAM3 typically fails to simultaneously process similar concepts, leading to semantic misclassifications and redundant mask predictions. ❷ As the number of target concepts increases, the original SAM3 inference paradigm incurs exceptionally high inference latency.

Recent works [31, 14] achieve multi-class segmentation by aggregating the individual masks independently predicted by SAM3 for multiple categories. However, this paradigm inherently treats the multi-class segmentation as a series of independent binary segmentation tasks. Consequently, there is a complete absence of competition among categories, which leaves the model without explicit constraints to differentiate semantically similar concepts. As shown in Figure 1, the model frequently produces “double-high” activations for identical pixel regions when processing similar concepts (e.g., tree vs. plant). This unconstrained behavior inevitably leads to semantic misclassifications and overlapping masks, severely degrading the reliability of the dense prediction results.

On the other hand, the standard architecture of SAM3 requires executing independent and computationally expensive cross-attention operations for each concept to acquire concept-specific dense visual features, followed by an independent decoding process. As shown in Figure 2, our inference latency scaling analysis reveals that on the ADE20K [70] dataset with 150 categories, the inference time scales linearly ($O(n)$) as the number of categories increases, ultimately resulting in severe inference latency.

To address these limitations, we propose the Dense Competitive Representations SAM (DCR-SAM) framework, which adapts SAM3 into an efficient architecture for multi-class segmentation. Specifically, the proposed framework resolves the aforementioned challenges through three key designs: ❶ To overcome the $O(n)$ inference latency, we decouple the deep cross-modal interactions by replacing class-specific text queries with a unified learnable prompt sequence. This approach enables the extraction of a globally contextualized visual representation shared across all concepts through a single forward pass, achieving a nearly 33× inference speedup compared to the native SAM3, as shown in Figure 2. ❷ To resolve semantic misclassification among similar concepts, we transition from isolated decoding to a dense competitive paradigm through the introduction of the Vision-Conditioned Text Refinement (VCTR) and Vision-Guided Text Pooling (VGTP) modules, coupled with a dense competition head. VCTR explicitly aligns textual sequence with the modified visual representation by injecting image-specific visual cues, whereas VGTP dynamically compresses this aligned sequence into highly discriminative global class anchors. By enforcing explicit inter-class competition, the proposed method effectively alleviates “double-high” activations and generates high-fidelity masks. Furthermore, to enhance adaptability across diverse unconstrained scenarios, we incorporate a dynamically learnable background token to absorb non-target objects adaptively.

The main contributions of our work are summarized as follows:

• We reveal how the independent inference paradigm of SAM3 induces severe semantic misclassification among conceptually similar categories and results in an $O(n)$ inference latency.

• We propose DCR-SAM, by decoupling cross-modal interactions to alleviate inference latency and dynamically aligning textual concepts with visual representation, the proposed method effectively suppresses overlapping predictions to generate high-fidelity masks.

• Extensive experimental evaluations demonstrate that our method achieves state-of-the-art (SOTA) performance across multiple domain benchmarks while exhibiting excellent inference efficiency.

Related Work

Promptable Segmentation Models The Segment Anything Model (SAM) [27, 46, 9] represents a significant milestone for promptable segmentation models, demonstrating remarkable cross-domain generalization across diverse downstream applications [31, 3, 54, 41, 6, 33]. Central to this success is the Promptable Visual Segmentation (PVS) paradigm introduced by SAM [27] and SAM 2 [46], where visual prompts (e.g., points, boxes, or masks) guide the segmentation of specific targets. Driven by large-scale, high-quality datasets, this paradigm establishes a robust baseline for generalized segmentation. Building upon this foundation, SAM3 [9] extends the architecture from PVS to Promptable Concept Segmentation (PCS), enabling the model to segment all instances of a given concept using short noun phrases (e.g., a red billboard). Recent studies [30, 44, 63, 64, 13] explore improvements to this architecture. For example, EfficientSAM3 [63, 64] enhances the overall inference efficiency of SAM3 through distillation, SAM3-Adapter [13] injects prior heuristic information for downstream tasks via a lightweight staged adapter. However, these works fundamentally retain the inherent serial inference mechanism of the native architecture. Consequently, they incur non-negligible inference latency and semantic misclassification when applied to real-world multi-class segmentation tasks. Meanwhile CoCo-SAM3 [14] proposes a training-free framework that separates the inference process into intra-class enhancement and inter-class competition modules to alleviate semantic misclassification. Nevertheless, this method relies heavily on complex score calibration and multi-prompt encoding. In this work, we alleviate these issues at the architectural level by decoupling cross-modal interactions. Furthermore, by dynamically aligning textual anchors with visual representations and introducing explicit inter-class competition, the proposed framework effectively mitigates semantic misclassification and generates high-fidelity masks.

Multi-Class Semantic Segmentation and Foundation Models Multi-class semantic segmentation is a fundamental task in computer vision, with the objective of assigning a specific semantic label to every pixel within an image. The field has witnessed remarkable progress, evolving from early CNN-based models, such as FCN [35] and U-Net [47], to modern Transformer-based architectures, such as SegFormer [61], Mask2Former [16] and EoMT [25]. While these methods excel at efficient dense prediction, they typically rely on narrow, task-specific feature representations, thereby lacking the broad semantic priors inherent in large-scale foundation models. Several vision-language models, such as CLIP [45], SigLIP [65], and ALIGN [24], emphasize global image-text alignment and inherently lack pixel-level granularity, rendering them unsuitable for direct application to downstream dense prediction tasks. Conversely, self-supervised vision foundation models, such as DINO [10, 42, 49], capture excellent fine-grained pixel-level representations on massive image corpora, yet they lack explicit cross-modal semantic guidance and require the training of task-specific decoder heads for multi-class segmentation. In this work, we fully leverage the powerful cross-modal priors of SAM3 [9] to enhance the potential of the model in dense prediction tasks.

Methodology

In this section, we first briefly review the Promptable Concept Segmentation (PCS) pipeline of SAM3 (§ 3.1). Subsequently, we elaborate on the overall architecture of our proposed DCR-SAM framework (§ 3.2). The overview of our method is illustrated in Figure 3. Finally, we introduce the joint optimization and the corresponding objective functions (§ 3.3).

Preliminary

Unlike dual-encoder architectures such as CLIP [45], SAM3 [9] employs an encoder-decoder architecture where the core modules comprise the vision encoder, the text encoder, the fusion encoder, and the semantic decoding head (for simplicity, we exclusively discuss the modules relevant to this work). Given an input image $I \in\mathbb{R}^{H \times W \times3}$ and a target concept set comprising $N$ categories $C = \{c_1, c_2, \ldots, c_N\}$, the network initially processes the image through a vision encoder to extract the unconditioned visual embeddings $\mathcal{V} \in\mathbb{R}^{hw \times d}$, where $d$ denotes the embedding dimension, $h$ and $w$ depend on the patch resolution of the vision encoder. Simultaneously, each specific concept $c_i$ is encoded into prompt tokens $p_i \in\mathbb{R}^{l \times d}$ via a text encoder, where $l$ denotes the sequence length. The complete set of prompt tokens is denoted as $\mathcal{P} = \{p_1, p_2, \ldots, p_N\}$. Subsequently, to achieve concept-specific localization, the unconditioned visual embeddings must be dynamically conditioned on the prompt tokens. Following the native architecture of SAM3 [9], a fusion encoder $E_{fusion}$ accepts the unconditioned visual embeddings $\mathcal{V}$ and conditions these embeddings on the prompt tokens $p_i$ using a stack of Transformer [55] blocks equipped with self-attention and cross-attention layers. The output of fusion encoder constitutes the conditioned visual embeddings:

Finally, these conditioned embeddings are fed into a semantic decoding head, which is adapted from the MaskFormer [15] architecture, to decode the binary semantic logits map $\mathcal{M}_i$ for the specific concept $c_i$:

In a dense multi-class scenario, to obtain the comprehensive multi-class prediction $\mathcal{M}$, the independently generated binary semantic logit maps must first be aggregated. Subsequently, a standard softmax activation is applied to normalize the probabilities across categories, followed by an argmax operation to derive the final segmentation mask:

Limitations This pipeline must execute the feature fusion and decoding processes independently for each category, which introduces a inference latency with a complexity of $O(n)$. Furthermore, the independent prediction paradigm for each category inherently lacks inter-class competition, thereby provoking severe semantic misclassifications and overlapping masks. In contrast, by decoupling the cross-modal fusion process and constructing a dense competition head, our method transforms the native serial binary segmenter into a highly efficient competitive multi-class architecture.

Architecture of DCR-SAM

Decoupled Visual Conditioning To alleviate the prohibitive $O(n)$ inference latency caused by the concept-specific fusion and decoding process, we design a decoupled visual conditioning strategy. Instead of repeatedly feeding the specific text prompt $p_i$ into the computationally intensive fusion encoder (cf. Eq. (1)), we introduce a unified learnable prompt sequence $p^* \in\mathbb{R}^{l \times d}$. By substituting the concept-specific tokens with $p^*$, We completely decouple the deep cross-modal feature fusion from specific prompt tokens. Consequently, the fusion encoder integrates the visual features with the unified prompt sequence through a single forward pass, yielding the joint representation $V^*$:

This modification adapts the native fusion encoder to extract high-quality, globally contextualized visual representations shared across all concepts. These representations serve as a universal visual foundation for all subsequent multi-class prediction tasks. Benefiting from this strictly category-agnostic feature fusion process, our approach fundamentally alleviates the inherent inference latency bottleneck of the native architecture.

Vision-Conditioned Text Refinement The native SAM3 [9] independent inference paradigm lacks inter-class competition, leading to severe semantic misclassification and overlapping masks among conceptually similar categories. To fundamentally resolve this issue, our DCR-SAM framework shifts from isolated decoding to a dense competitive paradigm, where textual features are explicitly aligned with globally contextualized visual representations to ensure robust cross-modal grounding. However, original textual features extracted by the text encoder are static and context-free, rendering direct alignment suboptimal. Therefore, we propose the Vision-Conditioned Text Refinement (VCTR) module, which achieves dynamic alignment by integrating image-specific visual cues into textual features. Specifically, we design a lightweight Cross-Attention (CA) module to refine the text representations. The text tokens $p_i$ for each category serve as the Query ($Q$), while the globally contextualized visual features $V^*$ act as the Key ($K$) and Value ($V$). Finally, we aggregate all category features to obtain the aligned textual feature set $\widetilde{P} = \{\widetilde{p}_1,\widetilde{p}_2,\ldots,\widetilde{p}_N\}$:

where MHA and LN denote the Multi-Head Attention and Layer Normalization, respectively. Notably, this lightweight module involves only simple matrix operations and a minimal number of additional parameters, incurring a negligible inference latency.

Vision-Guided Text Pooling For highly efficient dense matching, the aligned text sequence $\widetilde{P}$ must be compressed into a global feature embedding (anchor) for each category. An intuitive approach applies average pooling across the sequence. However, this introduces redundant noise from less relevant content or padding tokens, compromising the discriminative capability of semantic anchors. Instead, we propose the Vision-Guided Text Pooling (VGTP) module. First, we apply Global Average Pooling (GAP) and a linear projection layer $\mathcal{F}_{proj}$ to the globally contextualized visual representation $V^*$ to generate a global visual embedding $V_g$:

Next, we compute the attention scores matrix $\mathcal{A}$ between visual embedding and the aligned text sequence. These scores facilitate a weighted aggregation along the sequence length $l$ to generate the final global text anchors $T_{anchor}$ for each category:

Dynamic Background In multi-class segmentation tasks, pixels not belonging to predefined target categories are typically assigned to a background class. To adapt the SAM3 [9] to this formulation, a straightforward solution uses a generic text prompt, e.g., background. However, this concept is overly broad and lacks explicit visual grounding. Alternatively, hard-coding background categories (e.g., sky or cloud) severely degrades architectural universality, requires extensive dataset-specific engineering, and fails to exhaust all potential unknown distractors. To address these limitations, we introduce an independent, learnable background token $T_{bg} \in\mathbb{R}^{1 \times d}$ and concatenate it with the generated text anchors to form a complete semantic anchor set $\mathcal{T}$:

Dense Competitive Head Equipped with the complete set of text anchors $\mathcal{T}$ and the globally contextualized visual representation $\mathcal{V}^{*}$, we discard the isolated decoding paradigm in favor of a dense competitive head. To ensure stable optimization and prevent feature magnitude discrepancies from dominating the logit computation, we apply $L_{2}$ normalization along the channel dimension and introduce a temperature scaling parameter $\tau$ prior to the dot product. Specifically, we compute the scaled similarity between the normalized dense visual features and the text anchors to generate the dense logits map $\mathcal{M}^{*}$:

Joint Optimization for DCR-SAM

Global Presence Score While the dense competitive head suppresses semantic misclassification, relying solely on local dense features can occasionally lead to false positive predictions due to localized visual similarities. To provide a robust constraint, we follow the training strategy of SAM3 [9] and introduce a global presence score during the training phase. Specifically, we compute the dot product between the global visual embedding $\mathcal{V}_{g}$ and the set of target semantic anchors $\mathcal{T}_{anchor}$, followed by a sigmoid operation, to generate the image-level presence scores $S_{presence}$:

Joint Objective To holistically optimize the proposed model, we formulate a joint end-to-end training objective. We employ the cross-entropy loss and the dice loss to supervise the final mask, and utilize the binary cross-entropy loss to supervise the global presence score, and $\lambda_i$ denotes the weight to rescale the loss terms:

Experiments

Experimental Setup

Datasets and Evaluation Metric We evaluate our DCR-SAM on four highly challenging natural-scene benchmarks: ADE20K [70], which encompasses a dense label space of 150 classes, Cityscapes [17], which contains 19 classes tailored for urban driving scenarios, COCO-Stuff-164K [7] (COCO-Stuff), which features a vast and diverse label space of 171 classes, and PASCAL-Context59 [40] (Context59), which provides dense annotations for 59 prevalent categories. To demonstrate cross-domain robustness, we extend our evaluation to four representative remote sensing benchmarks: LoveDA [56] with 7 classes, ISPRS Potsdam [48] with 6 classes, ISPRS Vaihingen [39] with 6 classes, and ISAID [58] with 16 classes. Unless stated otherwise, all models are trained on their respective training sets, and the results are reported on the official validation sets. We employ the standard mean Intersection over Union (mIoU) as our primary metric.

Baselines We compare DCR-SAM with state-of-the-art semantic segmentation methods, including InternImage-H [57], EoMT [25], and SegMAN-L [19]. For SAM3 [9], we adopt its complete official architecture, retaining all core components, including the detector and presence prediction modules, and establish two baselines: ① Native-Binary (SAM3), which follows the official training strategy to independently predict binary masks for each category; and ➁ Native-Multi-Class (SAM3^‡), which concatenates category-wise predictions and is fine-tuned under a standard multi-class training paradigm. Following established SAM3 adaptation practices, semantic masks from both the semantic and detector heads are fused to obtain the final segmentation results. Notably, the latter configuration lacks explicit inter-class competition, potentially leading to semantic overlap among related categories.

Implementation Details The proposed DCR-SAM is implemented in the PyTorch framework [43]. All models are initialized with the official pre-trained weights of SAM3 [9]. For the concept set, we directly utilize the original category names provided by the respective datasets. The length of the unified learnable prompt sequence $p^{*}$ is set to 32. The scaling parameter $\tau$ is set to 0.07. The scaling weights $(\lambda_{1}, \lambda_{2}, \text{and } \lambda_{3})$ of the loss functions are set to 1 by default. The dynamic background strategy is optional, it can be omitted when all pixels in the evaluation dataset belong to the target categories. During the training phase, we freeze the original weights of SAM3 backbone and only fine-tune the newly introduced parameters: the learnable prompt sequence, the lightweight cross-modal attention module, and the learnable background token. The network is optimized end-to-end using the AdamW optimizer [36] with an initial learning rate of $1 \times10^{-4}$ and a weight decay of $5 \times10^{-4}$. The batch size is set to 8, and all models are trained for 80, 000 iterations. All experiments are conducted on 8 NVIDIA RTX 4090 GPUs. Further details are provided in the Appendix § 6.§ 1.

Overall Performance

Quantitative Comparisons As shown in Table 1, our method demonstrates exceptional performance across various benchmarks. In natural scene datasets, the proposed method significantly outperforms previous SOTA approaches, yielding substantial improvements ranging from 2.3% $\sim$ 8.4%. Even compared with InternImage-H [57], a vision foundation model built upon deformable convolutions, our method consistently achieves superior results. In cross-domain remote sensing scenarios, it continues to demonstrate substantial superiority, achieving improvements of approximately 2.4% ∼ 10.7%. Furthermore, it significantly surpasses specialized methods designed for remote sensing, such as LOGCAN++ [38] and SCSM [37], verifying its robust scene adaptability. Notably, the performance gains achieved in remote sensing are generally more pronounced than those in natural scenes. We attribute this phenomenon to the significantly smaller concept space of remote sensing imagery compared to natural scenes, enabling the dense competition head to establish more stable semantic anchors.

Methods	Pub. & Year	Backbone	ADE20K	Cityscapes	COCO-Stuff	Context59
DeepLabV3 [11]	ECCV’18	ResNet-101	42.6	77.1	37.9	52.0
DeepLabV3+ [12]	ECCV’18	ResNet-101	44.5	73.6	–	53.2
HiPoSeg [32]	ICLR’26	ResNet-101	49.0	84.0	–	–
HRNet [51]	arXiv’19	HRNetV2-W48	41.9	78.5	36.0	51.9
OCRNet [62]	ECCV’20	HRNetV2-W48	41.5	80.0	39.0	54.4
OCRNet [62]	ECCV’20	D-ResNet-101	45.3	81.8	–	54.8
SETR-PUP [68]	CVPR’21	ViT-L	50.1	82.2	–	55.3
EoMT [25]	CVPR’25	ViT-L	58.4	84.2	–	–
DTop [52]	ICCV’23	ViT-L	52.8	–	47.1	62.7
SegViT [66]	NeurIPS’22	ViT-L	53.3	–	47.4	63.0
ECENet [34]	ICCV’23	Swin-L	55.2	84.5	–	65.9
CPT [53]	TIP’25	Swin-L	58.5	84.6	–	65.6
SegMAN-L [19]	CVPR’25	LASS	53.2	84.2	–	–
SegNeXt-T [21]	NeurIPS’22	MSCAN-T	41.1	79.8	38.7	51.2
InternImage-H [57]	CVPR’23	DCNv3	60.3	86.1	–	70.3
SAM3† [9]	ICLR’26	SAM3 (PE-L+/14)	46.5	75.4	48.1	66.3
SAM3‡ [9]	ICLR’26	SAM3 (PE-L+/14)	41.2	72.6	45.4	64.3
DCR-SAM (Ours)	–	SAM3 (PE-L+/14)	68.7↑8.4	89.1↑3.0	50.8↑2.7	72.6↑2.3
Methods	Pub. & Year	Backbone	LoveDA	Potsdam	Vaihingen	iSAID
PSPNet [67]	CVPR’17	ResNet-50	48.3	82.0	76.8	60.2
DeepLabV3+ [12]	ECCV’18	ResNet-50	47.6	84.2	77.1	61.9
DANet [18]	CVPR’19	ResNet-50	50.2	81.4	77.3	60.0
SemanticFPN [26]	CVPR’19	ResNet-50	48.2	84.6	77.9	59.3
FarSeg [69]	CVPR’20	ResNet-50	48.2	84.3	79.1	63.7
LOGCAN++ [38]	TGRS’25	ResNet-50	53.4	87.6	83.9	65.0
SCSM [37]	ISPRS’25	ResNet-50	54.6	87.8	84.7	66.9
Segmenter [50]	ICCV’21	ViT-L	47.1	86.5	79.4	–
EfficientViT [8]	ICCV’23	EfficientViT-L2	47.1	84.2	80.5	–
DDP [23]	ICCV’23	Swin-T	52.5	87.4	82.6	–
SAM3† [9]	ICLR’26	SAM3 (PE-L+/14)	55.8	66.4	69.8	61.1
SAM3‡ [9]	ICLR’26	SAM3 (PE-L+/14)	50.4	61.3	65.7	55.4
DCR-SAM (Ours)	–	SAM3 (PE-L+/14)	66.5↑10.7	90.2↑2.4	89.2↑4.5	75.4↑8.5

Moreover, the comparison with the two SAM3 adaptation baselines demonstrates that both training paradigms fail to achieve optimal results without modifying the core network architecture. Notably, the performance of the Native-Multi-Class paradigm (SAM3^‡) is inferior to that of the original Native-Binary paradigm (SAM3^†). This performance degradation indicates that merely adjusting the training strategy fails to compensate for the inherent architectural defects of the model in multi-class segmentation tasks. More experimental analysis in the Appendix § 7.§ 1.

Qualitative Results We visualize the segmentation results across both natural scene and remote sensing benchmarks in Figure 4. SAM3 exhibits limitations such as semantic fragmentation, noisy artifacts (e.g., on the person in Context59), and regional misclassifications (e.g., omitting vegetation in LoveDA) in complex semantic contexts. Conversely, our DCR-SAM preserves the high-fidelity spatial delineation inherent to SAM3 while demonstrating significant improvements in semantic discriminability.

Further analysis

Component Ablation To validate the effectiveness of the proposed modules, we conduct an ablation analysis on the ADE20K and LoveDA benchmarks. Table 2 reports the per-image inference time (ms) and the mIoU on both benchmarks. The proposed method achieves a 33× inference speedup on the ADE20K benchmark, decreasing the inference time from 8038 ms to 244 ms alongside substantial performance gains, which demonstrates the necessity of decoupling the cross-modal feature fusion. Furthermore, the text refinement results indicate that incorporating the VCTR module yields significant improvements with only a marginal increase in inference latency, extending the inference time from 224 ms to 244 ms on the ADE20K dataset. Regarding the pooling strategy, the proposed VGTP module significantly outperforms conventional approaches such as first, last, and mean pooling, confirming that dynamic compression constructs highly discriminative textual anchors. For the background category, it can be observed that the proposed dynamic background strategy ($T_{bg}$) achieves optimal performance. Although expanding the number of hard-coding background categories can further improve the performance, this strategy requires extensive prompt engineering and manual adaptations specific to the dataset, thereby significantly limiting the general applicability of the model. Additional ablation analyses are detailed in Appendix § 7.§ 1.

Components	ADE20K		LoveDA
	Time.↓	mIoU	Time.↓	mIoU
baseline	8038	46.5	647	55.8
(a) text refinement
DCR-SAM w/o VCTR	224	61.4	179	60.9
DCR-SAM w/ VCTR	244	68.7	183	66.5
(b) pooling strategy
First	244	63.1	183	61.8
Last	244	62.8	183	60.2
Mean	244	66.5	183	63.2
VGTP.	244	68.7	183	66.5
(c) background strategy
generic text	–	–	183	59.5
hard-coding	–	–	193	64.1
dynamic background	–	–	183	66.5

Impact of Learnable Prompt Sequence Length We investigate the impact of the learnable prompt sequence length ($l \in\{4, 8, 16, 32, 64, 128\}$) on the final segmentation performance, as illustrated in Figure 5. The performance variations exhibit a highly consistent trend across all four evaluation benchmarks, uniformly achieving optimal results when $l$ is set to 32. When the sequence length is highly restricted ($l \le8$), the model lacks the representational capacity to capture complex semantic variations, resulting in suboptimal segmentation performance. Conversely, the performance degrades when the sequence length exceeds 32. This degradation is likely attributable to redundant parameters that introduce noise and increase the risk of overfitting. Therefore, we apply a unified sequence length across all benchmarks in this work.

Analysis of Objective Function We evaluate the individual contributions of the three loss components ($\mathcal{L}_{ce}$, $\mathcal{L}_{dice}$, and $\mathcal{L}_{bce}^{presence}$) in Table 3. The results indicate that $\mathcal{L}_{ce}$ and $\mathcal{L}_{dice}$ are highly complementary, and their joint application yields substantial improvements. Ultimately, the incorporation of the binary cross-entropy loss ($\mathcal{L}_{bce}^{presence}$) for the prediction of category presence achieves peak mIoU scores of 68.7% and 66.5% on the two benchmarks, respectively. This demonstrates the critical role of this component in mitigating false positives. In addition, Figure 6 visualizes the training trajectories, which illustrate a highly stable and synchronous convergence process across both domains, thereby validating the robustness of the joint optimization strategy.

$\mathcal{L}_{ce}$	$\mathcal{L}_{dice}$	$\mathcal{L}_{bce}^{presence}$	ADE20K	LoveDA
✓	✗	✗	66.7	64.0
✓	✓	✗	67.3	64.9
✓	✓	✓	68.7	66.5

Comparison with SAM 3-Based Methods.

We further compare DCR-SAM with four recent SAM 3-based methods, including SegEarth-OV3 [31], CoCo-SAM3 [14], EfficientSAM3 [63], and SAM3-Adapter [13]. As shown in Table 4, DCR-SAM consistently outperforms these methods, achieving 68.7% mIoU on ADE20K and 66.5% mIoU on LoveDA, surpassing the strongest baselines by 21.5% and 13.6%, respectively. While SegEarth-OV3 and CoCo-SAM3 adopt training-free strategies, EfficientSAM3 and

Method	ADE20K	LoveDA
SegEarth-OV3	36.6	47.4
CoCo-SAM3	38.3	–
EfficientSAM3	43.2	50.3
SAM3-Adapter	47.2	52.9
DCR-SAM	68.7	66.5

SAM3-Adapter primarily focus on efficient adaptation without explicitly addressing inter-class semantic conflicts. In contrast, DCR-SAM introduces a dense competitive head to enforce explicit inter-class competition, effectively mitigating semantic overlap and improving multi-class segmentation performance.

Comparison under a Unified Backbone To isolate the contribution of our segmentation architecture, we further compare DCR-SAM with representative segmentation heads, including UPerNet [60], Mask2Former [16], and EoMT-Style [25], using the same PE-L+/14 backbone [4]. As shown in Table 5, DCR-SAM achieves 68.7 mIoU on ADE20K, outperforming UPerNet, EoMT-Style, and Mask2Former by 9.1%, 16.6%, and 7.4%, respectively. These results demonstrate that the performance gains are not solely attributable to the backbone, but also benefit from our proposed dense competitive architecture. Notably, the PE backbone is specifically optimized for open-vocabulary tasks, making it less suitable for conventional closed-set segmentation heads. By preserving the native promptable architecture of SAM 3 and introducing explicit inter-class competition, DCR-SAM better exploits its pre-trained cross-modal representations for dense multi-class segmentation.

Method	Backbone	mIoU$\uparrow$
UPerNet	PE-L+/14	59.6
EoMT-Style	PE-L+/14	52.1
Mask2Former	PE-L+/14	61.3
DCR-SAM	PE-L+/14	68.7

Conclusion

In this paper, we propose the Dense Competitive Representations SAM (DCR-SAM) framework to mitigate the inference latency and inherent semantic misclassifications encountered during the adaptation of SAM3 to multi-class segmentation. The proposed method decouples the feature fusion process to ensure optimal inference efficiency. Furthermore, the alignment of textual and visual representations within a dense competitive paradigm effectively suppresses overlapping predictions among conceptually similar categories. Extensive experiments demonstrate that the framework achieves SOTA performance across diverse benchmarks while delivering a $33 \times$ speedup. We hope our work paves the way for adapting foundation models to complex, real-world multi-class segmentation tasks.

Limitations DCR-SAM reduces text prompt sensitivity and inference latency, facilitating its potential application in autonomous systems and medical imaging. However, its reliance on large-scale foundation models incurs substantial computational costs and may inherit biases from pre-training data. Moreover, inter-class competition may impair the recognition of unseen concepts. Future work will focus on improving computational efficiency and generalization in open-vocabulary segmentation.

Implementation and Architectural Details

Detailed Implementation Setup

Training and Evaluation Protocols During the training phase, we strictly adhere to the data processing protocols established for the SAM3 [9] model. To maintain batch consistency across varying image scales, all input images are randomly resized to dimensions within the range of 480 to 1008 pixels utilizing a stride of 32, and subsequently padded to a unified spatial resolution of $1008 \times1008$. In addition, standard data augmentation techniques, including random horizontal flipping and color jittering, are uniformly applied. During the inference phase, all images are processed using the standard $1008 \times1008$ input resolution natively required by the SAM3 architecture. Furthermore, to ensure fairness in the evaluation, all experimental results are derived directly from the raw outputs of the model without the application of any post-processing techniques (e.g., PAMR [1] or DenseCRF [29]).

Measurement of Inference Latency For a fair and rigorous evaluation of inference efficiency as reported in Figure 2 and Table 2, all inference latency measurements are conducted on a single NVIDIA RTX 4090 GPU. We report the per-image processing time configured with a batch size of 1. To eliminate hardware initialization and memory allocation overhead, we perform 50 warm-up iterations prior to measurement, and the final latency is calculated as the average execution time of the subsequent forward passes.

Configurations for SAM3 Baselines To thoroughly evaluate our method, we fine-tune the pre-trained SAM3 model under two distinct paradigms. As shown in Figure 7, for both baselines, the image encoder and the text encoder are strictly frozen to preserve broad generic priors, while only the fusion encoder and the segmentation heads are fully optimized. For foreground categories, we directly utilize the original class names provided by the dataset, whereas a generic text prompt, i.e., “background”, is employed to represent the background category. The detailed settings are as follows:

I) Native-Binary (SAM3{sup}†`): This baseline strictly adheres to the official strategy of SAM3. The fusion encoder processes each text prompt independently in a serial $\mathcal{O}(n)$ manner. The model is optimized using isolated binary prediction heads, supervised by a combination of the binary cross-entropy loss, dice loss, and presence loss for each individual concept. To construct the training data for this binary paradigm, we decouple the dense ground-truth annotations into independent binary masks based on the target categories present in the current image. Furthermore, we randomly introduce negative samples (i.e., unrepresented categories with completely empty ground-truth masks) during training to enhance the model’s semantic discriminability and effectively suppress false positive predictions.

II) Native-Multi-Class (SAM3{sup}‡`): To adapt the native SAM3 architecture for a conventional multi-class setting, we concatenate the independently generated logits along the category dimension. A standard softmax activation is then applied to explicitly calibrate probabilities across the $N$ classes, followed by optimization via the cross-entropy loss and dice loss. To prevent Out-of-Memory (OOM) errors caused by processing an excessive number of categories simultaneously, we employ a dynamic prompt sampling strategy during training: we sample all ground-truth categories present in the current scene alongside a dynamically varying number of negative categories, ensuring a stable memory footprint per forward pass.

Hard-Coding Background Strategy Details In the component ablation study presented in Table 2, we evaluated a hard-coding background strategy as a comparative baseline against our proposed dynamic background token ($T_{bg}$). To implement this strategy without extensive manual annotation, we employed large language models (LLMs) to automatically generate a comprehensive, dataset-specific list of negative concepts (e.g., “sky”, “road”, and “cloud”) that frequently appear in scenes but belong to the background class. Specifically, we utilized GPT-5 [5] to generate $K$ diverse background concepts tailored to the domain of each benchmark. To prevent excessive semantic overlap with existing foreground categories, we applied the filtering pipeline of previous work [59] to these background concepts. First, we discarded visually uninformative and abstract concepts, such as “image” and “view”. Subsequently, we used the CLIP [45] model to compute the semantic similarity between the generated background concepts and the predefined foreground categories. Any background concept exhibiting a similarity score exceeding a threshold of 0.8 was removed. The specific prompts used are detailed below:

You are an expert in computer vision and semantic segmentation. I need you to generate {K} background or negative concept words that typically appear within the scenes of {dataset name} dataset: {dataset scene type, foreground categories, and additional related information}. These concepts should encompass a wide range of background elements, environmental descriptors, or non-target objects to ensure high diversity. examples: “asphalt road”, “cloudy sky”, “dense forest” and “bare soil”. All generated terms must be in lowercase English. You are required to output exclusively a Python list containing {K} strings, formatted similarly to [“sky”, “road”, “water”], and to omit any additional explanations or code block markers.

Expanded Architectural Details of DCR-SAM

To complement the methodological overview (§ 3.2) in DCR-SAM, we detail the micro-architectural designs and precise mathematical formulations of our proposed modules, ensuring complete structural transparency.

Vision-Conditioned Text Refinement (VCTR) In practice, the Vision-Conditioned Text Refinement (VCTR) module is implemented using the standard nn.MultiheadAttention function provided by the PyTorch framework. Specifically, the text tokens $p_i \in\mathbb{R}^{l \times d}$ serves as the Query ($Q$), while the globally contextualized visual representation $\mathbf{V}^{*} \in\mathbb{R}^{hw \times d}$ acts simultaneously as both the Key ($K$) and the Value ($V$). To ensure optimization stability, this process is naturally accompanied by standard Layer Normalization [2] (LN) and residual connections [22]. The entire module relies exclusively on highly optimized matrix operations, thereby introducing minimal computational latency to the overall inference pipeline (please see Table 2 for more details).

Vision-Guided Text Pooling (VGTP) Regarding the dimensional broadcasting in Eq. (7) and Eq. (8), the global visual embedding $V_g \in\mathbb{R}^{1 \times d}$ is implicitly broadcasted along the category dimension $N$ to match the shape of the aligned text sequences. Crucially, the attention computation is strictly performed per class rather than being shared. The dot product between $V_g$ and $\widetilde{P}$ yields an attention matrix $A \in\mathbb{R}^{N \times l}$, where the softmax activation normalizes the weights strictly along the sequence length $l$ for each specific category. This ensures that the aggregated text anchors $T_{anchor}$ capture class-specific discriminative cues without inter-class sequence interference.

Dense Competitive Head After obtaining the dense logits map $M^{*} \in\mathbb{R}^{(N+1) \times hw}$, we subsequently reshape it back to the 2D spatial dimensions of $\mathbb{R}^{(N+1) \times h \times w}$. A bilinear upsampling algorithm [28] is then applied to align the resolution with that of the original input image. Following this, an argmax operation is performed along the category dimension to produce the final segmentation masks.

Additional Experimental Analysis and Discussion

Further Experimental Analysis

Impact of Concept Space Scale on Performance As observed in Table 1, DCR-SAM achieves more pronounced performance gains on remote sensing benchmarks compared to natural scene benchmarks. We hypothesize that this phenomenon is fundamentally driven by the scale of the concept space: the massive number of fine-grained categories in natural scenes exacerbates feature space crowding and semantic competition. To validate this hypothesis, we conducted a progressive subsetting experiment on the highly complex ADE20K [70] dataset ($N = 150$). Specifically, we establish a base set comprising the Top-30 most frequent categories and progressively expand the target category count $N \in\{30, 60, 90, 150\}$. For subsets where $N < 150$, all pixels belonging to unselected categories are uniformly absorbed by our dynamic background token ($T_{bg}$). Crucially, we isolate and track the performance of the identical Base-30 classes across all experimental configurations.

As shown in Table 6, As the number of semantic categories increases ($30 \rightarrow150$), the overall performance declines due to the introduction of difficult long-tail classes. Furthermore, the isolated Base-30 class also suffers a substantial degradation, dropping from 79.4% to 70.3%. This indicates that even the most well-represented foundational categories are negatively impacted by the severe feature competition and representational crowding triggered by a larger concept space. Consequently, in remote sensing scenarios where the concept space is typically constrained (e.g., $N \le10$), the dense competitive head can establish highly stable and non-overlapping semantic anchors. This effectively unlocks the full discriminative potential of the model architecture, thereby yielding superior performance gains.

Target Category Count	Semantic Space Composition	Time.↓	Overall mIoU (%)	Base-30 mIoU (%)
$N = 30$	Top-30 frequent classes only	236	79.4	79.4
$N = 60$	Base-30 + Next 30 frequent classes	238	75.6↓3.8	77.5↓1.9
$N = 90$	Base-30 + Next 60 frequent classes	241	73.5↓5.9	73.9↓5.5
$N = 150$	Full concept space	244	68.7↓10.7	70.3↓9.1

Impact of Concept Space Scale on Inference Efficiency We further investigate the impact of the concept space scale on inference latency. Specifically, we evaluate the DCR-SAM framework on the ADE20K dataset with 150 categories (A-150) alongside its full version comprising 847 categories (A-847), as well as the PASCAL Context dataset with 59 categories (PC-59) and its full version comprising 459 categories (PC-459). As illustrated in Figure 8, expanding the target categories from 150 to 847 on ADE20K introduces a manageable latency overhead of +369 ms (244 ms $\rightarrow$ 613 ms).

Similarly, scaling from 59 to 459 categories on PASCAL Context incurs a mere +217 ms increase (199 ms → 416 ms). Compared to the staggering 8038 ms required by the native SAM3 on the A-150 benchmark (please see Table 2), these fractional increments are practically negligible. While our core cross-modal feature fusion process remains strictly category-agnostic, the lightweight matrix operations within the VCTR module and the dense competitive head naturally introduce a sub-linear computational overhead as the number of target concepts $N$ grows. Nevertheless, achieving high-fidelity semantic segmentation for nearly a thousand categories in approximately 600 ms thoroughly demonstrates the extreme efficiency and scalability of our architecture, effectively bypassing the severe $O(N)$ complexity bottleneck inherent to the native SAM3 baseline.

Investigating Spatial-Preserving Pooling Strategies Our VGTP module employs Global Average Pooling (GAP) on the visual features $V^*$ to generate a global visual embedding $V_g$ (cf. Eq.6). A reasonable concern is whether this GAP operation collapses fine-grained spatial cues, which are often deemed critical for resolving small objects in dense segmentation tasks. To investigate this, we conducted an ablation study comparing our GAP-based VGTP against a Spatial-Preserving Attention (SPA) [20] variant. Specifically, we bypassed the GAP operation and flattened the spatial dimensions, computing a dense similarity matrix $S$ directly between the aligned text sequence $\widetilde{P} \in\mathbb{R}^{N \times l \times d}$ and the full spatial visual features $V^* \in\mathbb{R}^{hw \times d}$. The spatial-aware attention weights $A_{\mathrm{spa}} \in\mathbb{R}^{N \times l}$ are obtained by marginalizing this dense similarity matrix over the spatial dimensions:

The final semantic anchors are subsequently aggregated using these spatial-aware weights, formulated as $T_{\mathrm{anchor}}=\sum_{i=1}^{l}(A_{\mathrm{spa}})_{:,i}\odot\widetilde{P}_{:,i}$.

As presented in Table 7, the incorporation of the full spatial resolution in the SPA variant elevates the computational complexity of the attention mechanism. This introduces a slight inference latency and a concurrent performance decline (e.g., a 4.8% mIoU drop on the ADE20K dataset). We argue that the retention of the full spatial resolution during the anchor generation phase compels the model to overfit to localized, noisy visual artifacts. In contrast, the GAP operation serves as an effective information bottleneck. It distills robust global representations from the features to reliably guide the aggregation of the text sequence, while delegating the high-fidelity spatial delineation to the final Dense Competitive Head.

Pooling Strategy	ADE20K		Cityscapes		LoveDA		Potsdam
	Time.↓	mIoU	Time.↓	mIoU	Time.↓	mIoU	Time.↓	mIoU
SPA (Spatial-Preserving)	248	63.9	184	81.5	187	59.6	185	85.8
VGTP (GAP-based)	244	68.7	183	89.1	183	66.5	183	90.2

Impact of Hard-Coding Background Concept Number To thoroughly investigate the efficacy and potential limitations of the hard-coding strategy, we conducted an ablation study varying the number of generated background concepts, denoted as $K$. Using the filtered background concepts (please see Appendix § 6.§ 1), we reported the overall mIoU and inference time on the remote sensing benchmarks by progressively scaling the hard-coding concept number, i.e., $K \in\{10,20,30,40,50\}$. During the inference phase, we appended these filtered concepts to the target category list to explicitly absorb non-target pixels. As shown in Figure 9, due to the highly efficient, decoupled architecture of DCR-SAM, processing these additional concepts introduces only a marginal computational overhead rather than a severe latency bottleneck. However, the primary limitation emerges in the semantic performance, while scaling up $K$ initially improves background absorption, the performance gains rapidly saturate, even with a large set of explicitly designed negative concepts, the rigid textual anchors struggle to encompass all unconstrained background variations, resulting in suboptimal performance compared to our proposed dynamic background strategy ($T_{bg}$), which consistently achieves superior performance, demonstrating that it provides a more elegant and adaptable solution for robust semantic masking without the necessity of exhaustive prompt engineering.

Compared with batched inference mode To further evaluate inference efficiency, we compare DCR-SAM with native SAM 3 under different batch sizes on ADE20K, which contains 150 categories. As shown in Table 8, increasing the batch size from 1 to 50 reduces the inference latency of SAM 3 from 8038 ms to 5687 ms per image, but increases memory consumption from 9010 MB to 29080 MB. In contrast, DCR-SAM achieves a substantially lower latency of 244 ms with only 6318 MB of memory.

Inference Strategy	Backbone Passes	Time (ms)↓	Memory (MB)↓
SAM 3 (Sequential, bs=1)	150	8038	9010
SAM 3 (Batched, bs=15)	10	6392	12960
SAM 3 (Batched, bs=25)	6	6067	17488
SAM 3 (Batched, bs=30)	5	5966	19816
SAM 3 (Batched, bs=50)	3	5687	29080
DCR-SAM (Ours)	1	244	6318

The limited acceleration of batched inference stems from the architecture of SAM 3, which involves backbone inference, feature fusion, and mask decoding. Although batching reduces redundant backbone computations by sharing visual features across categories, feature fusion and mask decoding must still be performed separately for each category, resulting in increasing computational overhead as the number of categories grows. In contrast, DCR-SAM performs backbone inference and feature fusion only once, with negligible additional overhead from the lightweight matrix operations in the VCTR module. These results demonstrate the superior computational efficiency and memory utilization of DCR-SAM in large-scale multi-class segmentation.

Expanded Qualitative Analysis

Cross-Modal Activation Map Visualizations

To verify the effectiveness of our method in mitigating overlapping predictions, we conduct a visual analysis of the cross-modal activation maps. As illustrated in Figure 10, it can be observed that directly aggregating the masks obtained from the independent inference of the native SAM3 results in overlapping predictions within the same regions. In contrast, our method effectively alleviates this issue through explicit inter-class competition.

Extended Qualitative Comparisons

To further substantiate the qualitative superiority of our proposed framework, we present an extended set of visual comparisons across diverse natural scene and remote sensing benchmarks, as shown in Figure 11-Figure 18. These comprehensive visualizations demonstrate that the proposed method robustly delineates high-fidelity spatial boundaries while mitigating the severe semantic fragmentation prevalent in baseline models.