WorksComputer Science
Text-Guided Modality Correction for Open-Vocabulary Object Detection
Abstract
Open-vocabulary object detection (OVD) is essential for detecting objects beyond the categories seen during training. Establishing vision-language associations is crucial for recognizing unseen categories, while integrating complementary visual features across levels supports detection at different scales. However, existing methods often provide insufficient textual guidance for object regions, rely on limited cross-modal alignment strategies, and underuse complementary features across levels, limiting their ability to detect unseen categories and objects of varying scales. To address these limitations, we propose a Text-Guided Modality Correction Network (TMCNet) that incorporates textual guidance into region feature extraction, cross-modal alignment, and multi-scale decoding to strengthen region-category semantic correspondence and promote complementary feature fusion across levels. First, a text-guided modality encoder incorporates textual semantics into queries to guide regional feature aggregation, reducing semantic ambiguity in object representations. Second, a cross-modal alignment module strengthens the correspondence between region representations and category semantics through interactions between visual and textual features. Finally, a progressive text correction decoder combines a top-down pathway with lateral connections to progressively fuse text-guided deep semantic features with shallow visual features at the corresponding scales, preserving both semantic information and spatial detail for objects of different sizes. Experiments on two public datasets, COCO-OVD and LVIS-OVD, demonstrate that TMCNet achieves strong performance in open-vocabulary object detection using simple text queries as guidance. Extending TMCNet to open-vocabulary segmentation also yields performance gains, demonstrating its cross-task applicability and extensibility.
Introduction
Open-vocabulary object detection [1, 2] recognizes and localizes novel categories using textual descriptions, without requiring their detection annotations during training. Unlike conventional detectors trained on a fixed category set, it matches region representations with category semantics to extend recognition beyond base categories. Vision-language pretraining provides semantic priors for this purpose, but transferring image-level knowledge to localized object representations remains challenging.
Existing methods incorporate vision-language knowledge into several detection architectures. Two-stage methods (Figure 1(a)) classify region proposals through region-text matching and refine their bounding boxes. OVR-CNN [2] learns visual-semantic representations from image-caption pairs, while ViLD [1] distills pretrained knowledge into region features. PromptDet [3] adapts text representations through regional prompt learning, and BARON [4] exploits contextual relationships among regions for alignment. Single-stage methods (Figure 1(b)) integrate semantic knowledge into detection. Existing approaches use hierarchical knowledge distillation [5], vision-language path aggregation with region-text contrastive learning [6], or lightweight alignment with adaptive decision learning [7] to combine category generalization with efficient detection. Query-based Transformer detectors (Figure 1(c)) instead aggregate visual features through object queries [8]. OV-DETR [9] conditions detection on text or exemplar image embeddings. Despite these advances, image-level semantic alignment does not directly provide the spatial representations required for foreground-background discrimination and multiscale localization. MMC-Det [10] introduces localization guidance through cross-modal masked language modeling; and CLIPSelf [11] refines region representations through self-distillation. These approaches motivate integrating textual guidance into visual feature processing beyond category matching.
Figure 1. Comparison of representative open-vocabulary object detection architectures. (a) Two-stage detectors extract proposal features for region–text classification and bounding-box regression. (b) Single-stage detectors fuse visual and textual features for dense classification and localization. (c) Transformer-based detectors use object queries to aggregate semantically aligned visual features for instance-level prediction. (d) TMCNet incorporates textual priors into query representations and combines cross-modal alignment with progressive multiscale decoding to guide visual feature construction.
We propose the Text-Guided Modality Correction Network (TMCNet) to integrate category semantics into query construction, cross-modal alignment, and multiscale decoding (Figure 1(d)). The text-guided modality encoder combines cross-attention and masked max pooling to construct queries with textual priors. The cross-modal alignment pipeline uses the aligned queries to correct lateral visual features and fuses them with projected vision-text association features to form joint representations. The progressive text correction decoder then aggregates the corrected lateral features and scale-adapted joint representations with the current decoder features. This design combines category semantics with spatial detail across feature levels for open-vocabulary classification and localization.
The main contributions of this paper are as follows:
We propose a text-guided modality encoder that incorporates semantic priors into region feature extraction
through interactions between text tokens and object queries, strengthening textual guidance for object representation learning.
We design a cross-modal alignment module that combines modality projection with region-level vision–language interaction to strengthen region–category semantic correspondence and provide joint representations for subsequent decoding.
We develop a progressive text correction decoder that progressively fuses text-guided deep semantic features with multi-scale shallow visual features, combining semantic information and spatial detail across feature levels.
We evaluate detection performance on public benchmarks and extend the method to segmentation to assess its applicability across tasks.
Related Work
Open-Vocabulary Object Detection (OVD)
Open-vocabulary object detection uses vision-language knowledge to detect novel categories without their bounding box annotations during training. OVR-CNN [2] learns visual-semantic representations from image-caption pairs, while ViLD [1] transfers pretrained knowledge through region-level distillation. For query-based detection, OV-DETR [9] conditions predictions on text or exemplar image embeddings. Beyond semantic transfer, CAKE [12] combines knowledge distillation with category-generalizable proposals to reduce novel-object suppression. WeDetect [13] performs region-text retrieval with a fusion-free dual-tower architecture. These methods address semantic transfer, novel-object proposal generation, and matching efficiency. TMCNet focuses on incorporating category semantics into visual feature construction and multiscale decoding. Its text-guided modality encoder constructs queries with textual priors, which the cross-modal alignment pipeline uses for lateral feature correction and joint feature modeling. The progressive text correction decoder integrates these outputs across scales to provide features for object classification and localization.
Text-Guided Representation Learning and Cross-Modal Alignment
Text-guided representation learning introduces category semantics into visual recognition. MMC-Det [10] models regional context through cross-modal masked language modeling to guide novel-object localization, while SRA-Det [14] extracts and matches multiple semantic facets from token-level text features for fine-grained recognition. These methods exploit complementary category information, regional context, and textual attributes. Building on these directions, TMCNet incorporates textual priors into query representations to guide subsequent visual feature processing. Its text-guided modality encoder combines cross-attention between object queries and text tokens with masked max pooling to construct semantic queries for feature correction and aggregation.
Cross-modal alignment further connects visual representations with category semantics. PromptDet [3] adapts text representations through regional prompt learning, whereas BARON [4] exploits relationships among contextually related regions for vision-language alignment. CLIPSelf [11] aligns image-level and region-level visual representations through self-distillation. These methods improve alignment through text adaptation, contextual modeling, or visual representation refinement. TMCNet integrates alignment with multiscale feature decoding: its cross-modal alignment pipeline transforms TGME queries into semantic guidance for lateral feature correction and fuses them with projected vision-text association features. The resulting joint features support progressive aggregation in the decoder, extending textual guidance to multiple feature scales.
Multi-Scale Feature Fusion
Multi-scale feature fusion combines shallow spatial detail with deep semantic information for object recognition and localization. FPN [15] propagates semantic information through a top-down pathway and lateral connections, while PANet [16] adds a bottom-up pathway to strengthen localization cues at higher levels. For open-vocabulary detection, YOLO-World [6] integrates category semantics into hierarchical visual features through a re-parameterizable vision-language path aggregation network. Open-vocabulary segmentation also benefits from combining semantic correspondence with spatial detail. CAT-Seg [17] aggregates vision-language similarity costs for dense prediction, whereas SED [18] combines hierarchical encoding with progressive decoding. CLIPer [19] uses features and attention from early CLIP layers to improve spatial coherence and diffusion-model attention maps to refine local detail. TMCNet incorporates textual guidance into both lateral feature correction and progressive aggregation. Its progressive text correction decoder combines the current decoder features with semantic-query-corrected lateral features and scale-adapted joint features from the cross-modal alignment pipeline. This design integrates category semantics with spatial information across feature levels to support object classification and localization.
Methods
This section presents the architecture and components of the Text-Guided Modality Correction Network (TMCNet). § 3.1 provides an overview of the TMCNet and its processing pipeline. § 3.2–§ 3.4 then detail the text-guided modality encoder, cross-modal alignment pipeline, and progressive text correction decoder, respectively. Finally, § 3.5 describes the extension of TMCNet to open-vocabulary semantic segmentation.
Overall architecture
Existing open-vocabulary object detection methods rely on pretrained text representations for category matching but underuse category semantics in region feature extraction and across visual hierarchies. Vision–text similarity alone also inadequately captures correspondences between local visual cues and category semantics. Inspired by recent advances in open-vocabulary detection and segmentation [18, 6], we propose the TMCNet, which integrates text information into object representations through semantically guided queries, region-level cross-modal interactions, and progressive feature fusion. As shown in Figure 2, TMCNet comprises a hierarchical visual encoder, a pretrained text encoder, a Text-Guided Modality Encoder (TGME), a Cross-Modal Alignment Pipeline (CMAP), a Progressive Text Correction Decoder (PTCD), and a detection head, jointly supporting visual–text encoding, semantic guidance, cross-modal fusion, and multi-scale prediction.
Figure 2. Overview of TMCNet. A hierarchical visual encoder and a pretrained text encoder extract visual and category representations, respectively. The text-guided modality encoder (TGME) constructs queries with textual priors. The cross-modal alignment pipeline (CMAP) uses these queries to guide lateral feature correction and generate joint vision-language features. The progressive text correction decoder (PTCD) integrates both outputs through multiscale aggregation, producing features for object classification and localization.
Specifically, in the visual branch, ConvNeXt [20] extracts four feature levels () from an input image at , , , and resolution, respectively. Features , , and supply multi-scale spatial information to the decoder via lateral connections. An MLP projects into , matching the text embedding dimension for vision–text association, where , , and is the text embedding dimension. In the text branch, prompt templates [17, 21, 1] generate descriptions per category, such as “a photo of a class”. For category , these descriptions form . A pretrained text encoder produces embeddings , where is the number of categories. Cosine similarities between visual features at each spatial location and category text embeddings form a dense vision–text cost map .
Here, denotes a spatial location, while and index categories and prompt templates, respectively. The cost map encodes initial semantic associations between spatial locations and category descriptions. A convolutional embedding maps to the initial decoder features , where denotes the decoder feature dimension.
TGME incorporates category semantics into object queries to guide visual feature aggregation. The Text-Prior Object Query Module (TPOQ) uses multi-head cross-attention with padding masks, positional embeddings and to aggregate text features into learnable object queries. In parallel, masked max pooling extracts a text representation from valid tokens. The attention output, pooled representation, and initial queries are summed to obtain text-prior object queries , which are then combined with projected learnable object tokens to form for subsequent cross-modal feature fusion. Details of TGME are provided in § 3.2. CMAP aligns and fuses semantically guided queries with vision-text association features to support decoding through two complementary branches. First, the Alignment Layer Module (ALM) transforms into a representation compatible with the visual branch, while the image projection layer maps to . The aligned queries retain category semantics for lateral feature correction and are also combined with through the Fusion Operation (FO). The fused representation then passes through the Universal Projection Module (UPM) to produce joint vision-language features for decoder feature aggregation. These two branches incorporate text semantics into lateral feature correction and feature aggregation, respectively. Details of CMAP are provided in § 3.3. Finally, PTCD progressively fuses , , and through a top-down pathway with lateral connections. At each stage, the ALM representation is adapted to the spatial and channel dimensions of the corresponding visual features and added element-wise for semantic correction. The Progressive Feature Aggregation Module (PFAM) [18] then fuses these corrected features with the current decoder features and UPM joint features to restore spatial resolution. This process combines high-level category semantics with fine spatial details across feature levels. The resulting features are passed to a detection head for category prediction and bounding box regression, supervised by classification and regression losses. Details of PTCD are provided in § 3.4.
TGME: Text-Guided Modality Encoder
The TGME incorporates category semantics into learnable object queries through the TPOQ, providing semantic priors for subsequent visual feature aggregation. As shown in Figure 2, the encoder first aggregates text information using multi-head cross-attention and masked max pooling to generate text-prior object queries. These queries are then fused with projected learnable object tokens to form semantically guided representations.
Multi-modal Token Construction. The text encoder outputs embeddings , which are mapped to text representations . Here, , , and denote the number of categories, prompt templates per category, and text embedding dimension, respectively. The corresponding padding mask is derived from input tokenization and padding to identify valid tokens, where is the maximum text length. The text tokens are then linearly projected along the channel dimension to obtain , with denoting the number of tokens used in subsequent interactions and the projected channel dimension. A learnable object token is introduced for the query token branch. Meanwhile, the image projection layer in CMAP maps to using parameters independent of the text projection. The object, image, and text tokens are collectively denoted by Query, with their corresponding padding masks denoted by :
where , , and denote the padding masks corresponding to , , and , respectively.
Text-Prior Object Queries. The TPOQ incorporates input category semantics into learnable object queries to guide subsequent visual feature aggregation. As shown in Figure 2, it first projects the text features and adapts the corresponding padding mask:
Here, TP projects the text features to , while MP adapts the text padding mask to obtain , which identifies valid and padded positions in the text sequence. Using these inputs, multi-head cross-attention aggregates query-specific text information, with projected object queries as queries, position-encoded text features as keys, and as values. In parallel, masked max pooling extracts a global text representation from valid positions and broadcasts it across queries. Both outputs are added to to obtain text-prior object queries:
Here, denotes multi-head cross-attention with queries , keys , values , and a mask that excludes padded positions. QP and PEP are independent linear projections of the queries and text positional encodings , respectively. MMP computes channel-wise maxima over valid text positions, and expands the pooled representation to match the queries. The resulting combines query-specific text information with shared global semantics for subsequent cross-modal interactions.
Query Token Branch. This branch combines the learnable object token with the text-prior queries generated by TPOQ. Specifically, OP linearly projects into the query feature space, and the projected token is broadcast across queries and added element-wise to . An MLP then transforms the fused features into semantically guided representations:
Here, denotes TPOQ, and refers to the initial queries , where is the number of object queries. The resulting carries text-derived category semantics and is passed to CMAP to guide visual feature fusion.
CMAP: Cross-Modal Alignment Pipeline
TGME queries encode category semantics, while the initial decoder features capture associations between spatial locations and category descriptions. CMAP brings these representations together to guide visual feature aggregation. As shown in Figure 2, it comprises an Alignment Layer Module (ALM), a Fusion Operation (FO), and a Universal Projection Module (UPM). The ALM output supplies category semantics for lateral feature correction and also enters FO and UPM to form joint features for decoder aggregation.
Alignment Layer Module. To enable interaction between the two representations, ALM uses a two-layer MLP to map the TGME queries into a shared feature space:
Here, is the number of queries, is the shared channel dimension, and denotes the transformed queries. This mapping prepares the queries for cross-modal interaction; their correspondence with visual content is learned through subsequent attention and detection supervision.
Fusion Operation. FO uses the aligned queries to aggregate vision-text association features. First, the spatial and category dimensions of are flattened, and the image projection layer IP maps the resulting features to channels:
where and are the spatial dimensions, is the number of categories, and is the decoder feature dimension. The resulting tokens each represent a spatial location-category pair. Their original indices are retained for subsequent layout restoration. The aligned queries and modality tokens are then concatenated into a joint sequence:
Here, denotes sequence-wise concatenation, placing the query tokens before the modality tokens. To enrich the queries with relevant visual context, scaled dot-product attention uses as queries and as keys and values:
Softmax normalizes scores over the modality tokens, yielding query-specific weights for aggregation:
The resulting context features update only the query positions through residual addition, leaving the modality tokens unchanged:
where is a zero tensor indicating that the residual update applies only to the query positions in the joint sequence. retains the original vision–text association tokens alongside queries enriched with visual context.
Universal Projection Module. UPM applies Transformer encoder layers to the fused sequence, modeling token interactions through self-attention and token-wise transformations while preserving sequence length:
Joint self-attention allows modality tokens to incorporate query semantics and contextual information from other tokens. The last output tokens are then reshaped according to their original indices to recover . After spatial and channel adaptation, these joint features are supplied to the Progressive Feature Aggregation Module at each decoding stage.
Together, the two CMAP branches provide complementary decoder inputs: modulates the lateral features , , and through dimensional adaptation, while supplies cross-modal context for progressive aggregation. The decoder combines these inputs with its current features to restore spatial resolution, integrating high-level semantics with fine spatial details for category prediction and bounding box regression.
PTCD: Progressive Text Correction Decoder
As shown in Figure 2, PTCD progressively fuses hierarchical visual features through a top-down pathway with lateral connections. The two CMAP branches provide complementary guidance: ALM queries supply category semantics for lateral feature correction, while UPM joint features provide cross-modal context for progressive aggregation.
Semantically Guided Lateral Feature Correction. Since ALM queries have a sequence structure, dimensional adaptation first maps them to representations matching the spatial layout and channel dimension of each lateral feature. These representations are then added element-wise to , , and , respectively, incorporating category semantics before decoder aggregation. This mapping converts query representations into spatially compatible features rather than directly upsampling the query sequence.
Progressive Aggregation Guided by Joint Cross-Modal Features. The UPM branch complements lateral correction with joint features that encode spatial location–category associations. After their spatial layout is restored, these features are resampled and channel-projected to match each decoding stage, providing PFAM [18] with semantic and visual context at multiple scales.
Multi-Scale Feature Fusion and Prediction. Starting from , PTCD performs three aggregation stages corresponding to , , and . At each stage, the current decoder features are resized to match the lateral resolution. PFAM [18] then fuses them with the semantically corrected lateral features and the adapted UPM features. The output proceeds to the next stage to incorporate higher-resolution information.
This process propagates high-level semantics along the top-down pathway while adding spatial details through lateral connections. The final features are passed to the detection head for category prediction and bounding box regression.
Extension to Open-Vocabulary Semantic Segmentation
Open-vocabulary semantic segmentation [22, 23, 24] faces similar challenges: limited adaptability of pretrained text representations, underuse of textual semantics, and a mismatch between global image-text alignment and pixel-level prediction [18]. To address these challenges, we integrate TMCNet into the SED [18] and OVSeg [21] frameworks to enhance pixel-level representations through text-guided feature correction, cross-modal alignment, and multi-scale fusion. Experiments show improved segmentation accuracy in both frameworks, supporting the effectiveness of TMCNet for dense prediction and its applicability across tasks.
Experiments
We evaluate TMCNet on the COCO-OVD [2] and LVIS-OVD [1] datasets and examine its extension to open-vocabulary semantic segmentation. § 4.1 and § 4.2 describe the dataset, evaluation metrics, and implementation details. § 5 evaluates TGME, CMAP, and PTCD through ablation studies and comparisons with alternative designs. § 5.4 presents quantitative comparisons with recent state-of-the-art detection methods and qualitative visualizations. Finally, § 5.5 evaluates the segmentation extension using SED [18] and OVSeg [21] as baselines.
Datasets and Evaluation Metrics
Datasets. We evaluate TMCNet on open-vocabulary object detection and semantic segmentation using the following datasets and protocols.
Open-vocabulary object detection (OVD): Following prior work [2, 12, 1, 25], we use COCO-OVD and LVIS-OVD, built on MS-COCO [26] and LVIS v1.0 [27], respectively. We adopt the category splits and evaluation protocols of OVR-CNN [2] for COCO-OVD and ViLD [1] for LVIS-OVD.
Open-vocabulary semantic segmentation (OVS): Following prior methods [17, 28, 18, 29], we train on COCO-Stuff [30] and evaluate on ADE20K [31], PASCAL VOC [32], and PASCAL-Context [33]. Evaluation covers the 150-class and 847-class settings of ADE20K (A-150 and A-847), the 20-class setting of PASCAL VOC (PAS-20), and the 59-class and 459-class settings of PASCAL-Context (PC-59 and PC-459).
Evaluation Metrics. We assess prediction accuracy using task-specific metrics and report computational efficiency for detection.
Open-vocabulary object detection (OVD): On COCO-OVD, the primary metric is , the average precision for novel categories at an IoU threshold of 0.5. We also report and for base categories and the union of base and novel categories, respectively. On LVIS-OVD, for rare categories is the primary metric for novel-category detection, supplemented by for common categories, for frequent categories, and mAP across all categories. All LVIS-OVD AP metrics are averaged over IoU thresholds from 0.50 to 0.95 in increments of 0.05.
Open-vocabulary semantic segmentation (OVS): We report mean Intersection over Union (mIoU) on all evaluation datasets, computed by averaging the IoU between predicted and ground-truth regions across evaluated categories.
For detection, we additionally report inference time (Inf.time), floating-point operations (FLOPs), and parameter count (Params.) to measure inference latency, computational cost, and model size, respectively.
Implementation Details
Baselines. We build TMCNet on SED [18] and OVSeg [21], two open-vocabulary semantic segmentation frameworks. For detection, we adapt them into SED-OVD and OVSeg-OVD using the CLIP_Trans head for instance classification and bounding box prediction, as described in § 4.2. For segmentation, we retain their original prediction mechanisms. Unless required for module integration or task adaptation, other settings follow the corresponding baseline.
OVSeg-OVD: OVSeg [21] combines a MaskFormer-based segmentation model [34] with mask-adapted CLIP and ensembles their class predictions for masked regions. We use ResNet-101c as the segmentation backbone and CLIP with a ViT-B/16 [35] visual encoder. Mask prediction and class ensembling are retained for segmentation, while the detection variant uses the task-specific head.
SED-OVD: SED [18] combines a hierarchical visual encoder with a progressive fusion decoder to aggregate image-text cost features for pixel-level prediction. We use pretrained CLIP with a ConvNeXt-B [18] visual encoder and its corresponding text encoder. The text embedding dimension is , with prompt templates per category. The initial decoder features have 128 channels.
Visual Backbones and Encoders. The visual encoders provide spatial and semantic representations according to the underlying framework:
ResNet-101c: In the OVSeg-OVD framework, convolutional and residual stages extract spatial features for subsequent fusion and task-specific prediction.
ViT-B/16 [35]: The CLIP visual encoder embeds non-overlapping patches and processes them with a classification token and positional embeddings through Transformer layers. The classification token aggregates image-level information, while patch tokens retain spatial correspondence. Unlike ResNet-101c, standard ViT-B/16 does not directly produce a hierarchical feature pyramid.
ConvNeXt-B [20]: In the SED-OVD framework, four convolutional stages provide hierarchical features for image-text matching and progressive fusion. For an input of size , with both dimensions divisible by 32, the output strides are 4, 8, 16, and 32, with channel dimensions of 128, 256, 512, and 1024, respectively. In height-width-channel order, the feature shapes are , , , and . These backbone channel dimensions differ from the projected image-text embedding dimension .
Detection Head and Task Adaptation. The query-based CLIP_Trans head uses learnable instance queries enriched with TGME semantics to attend to channel-projected PTCD features with positional embeddings. Through query self-attention and cross-attention to these visual features, a Transformer produces instance representations for classification and localization. The classification branch projects them into the category text embedding space and computes cosine similarities with category embeddings, while a class-agnostic MLP predicts normalized bounding box coordinates. Training uses bounding boxes and class labels from base categories. At inference, the head predicts class labels, confidence scores, and bounding boxes over a vocabulary containing both base and novel categories. For segmentation, the proposed text-guided feature correction modules are integrated into SED and OVSeg while retaining their segmentation prediction mechanisms. The detection and segmentation models are trained with bounding box annotations and pixel-level semantic annotations, respectively.
Training Settings. Experiments run on two NVIDIA GeForce RTX 4090 GPUs, each with 24 GB of memory. The OVSeg-OVD and SED-OVD detectors model use the following configurations.
OVSeg-OVD: We follow the OVSeg training setup with an adjusted batch size, using random crops of 640 × 640 pixels, a global batch size of 8, and 120 000 iterations. AdamW [36] uses an initial learning rate of and weight decay of , with polynomial learning rate decay.
SED-OVD: Distributed data parallel training uses random crops of 768 × 768 pixels and a global batch size of 2, with one image per GPU and no gradient accumulation. AdamW [36] uses a base learning rate of and default weight decay of ; normalization and embedding layers are exempt from weight decay. We apply cosine learning rate decay without
warm-up and clip the global gradient norm to 0.01. The maximum training budget is 1 200 000 iterations, with checkpoints saved every 5 000 iterations and evaluation every 10 000 iterations.
Inference Settings. Inference preprocessing follows the configuration of each model. For SED-OVD, images are resized with a target short-side length of 640 pixels and a maximum long-side length of 2560 pixels while preserving the aspect ratio. Neither test-time augmentation nor sliding-window inference is used. Inference time (Inf. time) is measured on a single NVIDIA GeForce RTX 4090 GPU with 24 GB of memory, using FP16 precision and a batch size of 1.
Ablation Study
We conduct ablation studies on COCO-OVD and LVIS-OVD using detection-adapted OVSeg-OVD and SED-OVD as baselines. Table 1 and Table 2 evaluate module configurations on COCO-OVD and LVIS-OVD, respectively. Table 3 compares TGME with direct cosine similarity matching, feature fusion, and encoder-based interaction. With TGME enabled, Table 4 examines alternative cross-modal alignment strategies, complemented by the visualizations in Figure 4. Finally, Table 5 compares detection heads with TGME and CMAP enabled, including the CLIP+Transformer head.
| Detection-adapted Baseline | VLM | Feature backbone | TGME | CMAP | PTCD | COCO-OVD(%) | |||
| OVSeg-OVD* | ViT-B/16 | ResNet-101c | ✗ | ✗ | ✗ | 33.70 | 53.70 | 49.30 | |
| ViT-B/16 | ResNet-101c | ✓ | ✗ | ✗ | 37.90+4.20 | 58.00 | 52.80 | ||
| ViT-B/16 | ResNet-101c | ✗ | ✓ | ✗ | 35.10+1.40 | 55.20 | 51.40 | ||
| ViT-B/16 | ResNet-101c | ✓ | ✓ | ✗ | 38.50+4.80 | 58.20 | 53.00 | ||
| ViT-B/16 | ResNet-101c | ✓ | ✓ | ✓ | 39.20+5.50 | 58.50 | 53.30 | ||
| SED-OVD* | ConvNeXt-B | - | ✗ | ✗ | ✗ | 36.10 | 54.20 | 49.90 | |
| ConvNeXt-B | - | ✓ | ✗ | ✗ | 40.40+4.30 | 58.50 | 54.10 | ||
| ConvNeXt-B | - | ✗ | ✓ | ✗ | 37.70+1.60 | 55.90 | 51.40 | ||
| ConvNeXt-B | - | ✓ | ✓ | ✗ | 41.00+4.90 | 59.50 | 54.70 | ||
| ConvNeXt-B | - | ✓ | ✓ | ✓ | 41.60+5.50 | 60.90 | 55.40 |
Table 1. Ablation study of TGME, CMAP, and PTCD on two detection-adapted baselines on COCO-OVD. All results are reported in %. * indicates the detection-adapted variants of OVSeg and SED. ✓ and ✗ indicate the presence and absence of the corresponding module, respectively. Bold indicates the best result for each baseline.
| Detection-adapted Baseline | VLM | Feature backbone | TGME | CMAP | PTCD | LVIS-OVD(%) | mAP | |||
| OVSeg-OVD* | ViT-B/16 | ResNet-101c | ✗ | ✗ | ✗ | 22.40 | 28.30 | 36.00 | 32.40 | |
| ViT-B/16 | ResNet-101c | ✓ | ✗ | ✗ | 24.90+2.50 | 30.70 | 38.20 | 34.50 | ||
| ViT-B/16 | ResNet-101c | ✗ | ✓ | ✗ | 23.80+1.40 | 29.50 | 37.40 | 33.70 | ||
| ViT-B/16 | ResNet-101c | ✓ | ✓ | ✗ | 26.1+3.70 | 31.90 | 39.50 | 36.00 | ||
| ViT-B/16 | ResNet-101c | ✓ | ✓ | ✓ | 26.70+4.30 | 37.10 | 40.00 | 36.50 | ||
| SED-OVD* | ConvNeXt-B | - | ✗ | ✗ | ✗ | 24.80 | 34.50 | 38.00 | 34.00 | |
| ConvNeXt-B | - | ✓ | ✗ | ✗ | 25.90+1.10 | 35.80 | 39.20 | 35.30 | ||
| ConvNeXt-B | - | ✗ | ✓ | ✗ | 25.30+0.50 | 34.90 | 38.60 | 34.80 | ||
| ConvNeXt-B | - | ✓ | ✓ | ✗ | 27.40+2.60 | 36.70 | 40.40 | 36.60 | ||
| ConvNeXt-B | - | ✓ | ✓ | ✓ | 28.60+3.80 | 38.10 | 41.00 | 37.70 |
Table 2. Ablation results for different module configurations on LVIS-OVD. All results are reported in %. * indicates the detection-adapted variants of OVSeg and SED. ✓ and ✗ indicate the presence and absence of the corresponding module, respectively. Bold indicates the best result for each baseline.
| Methods | RTM | CAF | TGME | COCO-OVD(%) | LVIS-OVD(%) | |||||
| OVSeg-OVD* | AP | AP | AP | AP | AP | AP | mAP | |||
| OVSeg-OVD* | 33.70 | 53.70 | 49.30 | 22.40 | 28.30 | 36.00 | 32.40 | |||
| TMCNet | 37.00 | 56.50 | 51.00 | 23.60 | 33.70 | 36.90 | 33.40 | |||
| TMCNet | 37.70 | 57.00 | 51.50 | 23.90 | 34.10 | 37.30 | 33.80 | |||
| TMCNet | 39.20 | 58.50 | 53.30 | 26.70 | 37.10 | 40.00 | 36.50 | |||
| SED-OVD* | 36.10 | 54.20 | 49.90 | 24.80 | 34.50 | 38.00 | 34.00 | |||
| TMCNet | 38.70 | 57.90 | 53.00 | 25.00 | 35.10 | 38.20 | 34.60 | |||
| TMCNet | 39.80 | 59.20 | 53.10 | 25.50 | 35.30 | 38.70 | 34.80 | |||
| TMCNet | 41.60 | 60.90 | 55.40 | 28.60 | 38.10 | 41.00 | 37.70 |
Table 3. Comparison of text semantic modeling strategies on COCO-OVD and LVIS-OVD. RTM, CAF, and TGME denote Region–Text Matching, Cross-Attention Fusion, and the Text-Guided Modality Encoder, respectively. All results are reported in %. * marks the detection-adapted OVSeg and SED baselines. Bold indicates the best result within each framework; red subscripts report gains in percentage points over the corresponding baseline.
| Methods | CLIP | LP | CMAP | COCO-OVD(%) | LVIS-OVD(%) | |||||
| OVSeg-OVD* | CLIP | LP | CMAP | AP | AP | AP | AP | AP | AP | mAP |
| OVSeg-OVD* | 33.70 | 53.70 | 49.30 | 22.40 | 28.30 | 36.00 | 32.40 | |||
| TMCNet | 37.50 | 57.10 | 51.60 | 24.00 | 34.10 | 37.60 | 34.10 | |||
| TMCNet | 38.40 | 57.70 | 52.30 | 24.50 | 34.60 | 38.00 | 34.50 | |||
| TMCNet | 39.20 | 58.50 | 53.30 | 26.70 | 37.10 | 40.00 | 36.50 | |||
| SED-OVD* | 36.10 | 54.20 | 49.90 | 24.80 | 34.50 | 38.00 | 34.00 | |||
| TMCNet | 39.30 | 58.40 | 53.50 | 25.40 | 35.70 | 38.70 | 35.00 | |||
| TMCNet | 40.20 | 59.40 | 53.70 | 25.70 | 35.70 | 39.10 | 35.00 | |||
| TMCNet | 41.60 | 60.90 | 55.40 | 28.60 | 38.10 | 41.00 | 37.70 |
Table 4. Comparison of cross-modal alignment schemes on COCO-OVD and LVIS-OVD. CLIP, LP, and CMAP denote CLIP feature matching, linear projection, and the proposed Cross-Modal Alignment Pipeline, respectively. All results are reported in %. * marks the detection-adapted OVSeg and SED baselines. Bold indicates the best result within each framework; red subscripts report gains in percentage points over the corresponding baseline.
Comparison of detection configurations on COCO-OVD and LVIS-OVD, with TGME, CMAP, and PTCD incorporated in both frameworks. CLIP, Def_DETR, and CLIP_Trans denote the CLIP feature-matching configuration, the Deformable DETR-based configuration, and the adopted query-based CLIP–Transformer head, respectively. ✓ and ✗ indicate whether a configuration is used. All results are reported in %. Bold indicates the best result within each framework; red subscripts report gains in percentage points over the corresponding OVSeg-OVD or SED-OVD baseline.
| Detection-adapted Methods | CLIP | Def_DETR | CLIP_Trans | COCO-OVD(%) | LVIS-OVD(%) | |||||
| AP | AP | AP | AP | AP | AP | mAP | ||||
| TMCNet | ✓ | ✗ | ✗ | 38.00 | 57.30 | 52.00 | 24.00 | 34.00 | 37.30 | 33.50 |
| ✗ | ✓ | ✗ | 38.80 | 58.00 | 52.80 | 24.90 | 35.00 | 38.20 | 34.60 | |
| ✗ | ✗ | ✓ | 39.20 | 58.50 | 53.30 | 26.70 | 37.10 | 40.00 | 36.50 | |
| TMCNet | ✓ | ✗ | ✗ | 39.40 | 58.50 | 52.90 | 25.50 | 35.20 | 38.80 | 34.80 |
| ✗ | ✓ | ✗ | 40.70 | 59.90 | 54.30 | 26.00 | 35.90 | 39.40 | 35.50 | |
| ✗ | ✗ | ✓ | 41.60 | 60.90 | 55.40 | 28.60 | 38.10 | 41.00 | 37.70 |
Table 5.

Figure 4. Qualitative detection results for SED-OVD and its variants on COCO-OVD. From left to right: input images and predictions from SED-OVD, SED-OVD, and TMCNet. Red solid boxes in the input images highlight selected novel-category objects, while white solid boxes in the predictions highlight additional objects detected relative to SED-OVD. Colored bounding boxes and their labels indicate predicted locations, categories, and confidence scores.
Effectiveness Across Baselines
For each baseline, we evaluate TGME and CMAP alone, their combination, and the complete model with PTCD. This comparison assesses the individual contributions of TGME and CMAP and the additional benefit of PTCD when both modules are present.
COCO-OVD. Table 1 shows that TGME alone improves by 4.20 and 4.30 percentage points on the OVSeg-OVD and SED-OVD baselines, respectively, exceeding the gains of 1.40 and 1.60 points from CMAP alone. Combining TGME and CMAP raises novel-category AP to 38.50% and 41.00%. Adding PTCD further improves these scores by 0.70 and 0.60 points, yielding 39.20% and 41.60%, respectively. The complete model also achieves the highest base- and all-category AP, with improving over the baselines by 4.00 and 5.50 points.
LVIS-OVD. Table 2 shows consistent improvements across both baselines. TGME alone increases by 2.50 and 1.10 percentage points for OVSeg-OVD and SED-OVD, respectively, while CMAP alone yields gains of 1.40 and 0.50 points. Their combination reaches 26.10% and 27.40% rare-category AP. Adding PTCD provides further gains of 0.60 and 1.20 points, bringing to 26.70% and 28.60%. The complete model performs best across rare, common, and frequent categories, with overall mAP reaching 36.50% and 37.70%, respectively.
Across both benchmarks, TGME yields larger novel-category gains than CMAP when added individually, while their combination outperforms either module alone. PTCD provides further improvements through semantically guided lateral correction and progressive multi-scale aggregation. These results support the effectiveness of the complete design across both detection-adapted frameworks.
On LVIS-OVD, TGME reaches AP scores of 26.70% and 28.60%, outperforming RTM by 3.10 and 3.60 points and CAF by 2.80 and 3.10 points, respectively. Improvements also extend to common and frequent categories, yielding the highest overall mAP of 36.50% and 37.70%. These consistent gains support encoding textual priors into queries to guide subsequent visual feature processing. Compared with direct region-text matching and cross-attention fusion, the TGME configuration improves novel-category detection while maintaining stronger performance across the full evaluated vocabulary.
Effect of TGME
We compare three strategies for incorporating textual semantics: Region-Text Matching (RTM), Cross-Attention Fusion (CAF), and TGME. RTM computes similarities between region and category text features, while CAF directly interacts visual and textual features through cross-attention. TGME instead combines query-specific text information, masked max-pooled semantics, and initial queries, then fuses the resulting queries with projected object tokens. These semantic queries guide subsequent cross-modal alignment in CMAP and multi-scale decoding in PTCD.
Table 3 reports results for both detection-adapted frameworks, with their baselines included for reference. On COCO-OVD, TGME achieves AP scores of 39.20% and 41.60% for the OVSeg-OVD and SED-OVD models, respectively. These results exceed RTM by 2.20 and 2.90 percentage points and CAF by 1.50 and 1.80 points. TGME also achieves the highest base- and all-category AP in both frameworks.
Effect of CMAP
CMAP incorporates TGME queries into two complementary pathways: ALM supplies query semantics for lateral feature correction, while FO and UPM integrate the aligned queries with image-projected vision-text association features for progressive aggregation. With TGME incorporated, Table 4 compares CLIP feature matching, linear projection (LP), and CMAP. On COCO-OVD, CMAP achieves AP scores of 39.20% and 41.60% in the OVSeg-OVD and SED-OVD frameworks, respectively, exceeding CLIP by 1.70 and 2.30 percentage points and LP by 0.80 and 1.40 points. On LVIS-OVD, the corresponding AP scores reach 26.70% and 28.60%, surpassing CLIP by 2.70 and 3.20 points and LP by 2.20 and 2.90 points. CMAP also achieves highest scores on the remaining metrics in both frameworks, supporting its effectiveness in incorporating textual guidance into visual feature processing. Figure 3 further compares novel-category pseudo-label coverage during training on COCO-OVD. CMAP increases coverage from approximately 32% at 10 k iterations to 74% at 70 k, remaining near 73% at 90 k. It exceeds both alternatives at every reported checkpoint from 30 k onward, while CLIP and LP remain within approximately 41%-49%. This trend indicates broader coverage of novel-category objects and is consistent with the detection gains, although coverage alone does not measure pseudo-label precision.

Figure 3. Novel-category pseudo-label coverage on the COCO-OVD validation set across training iterations for different cross-modal alignment schemes. The horizontal axis shows training iterations in thousands, and the vertical axis shows coverage in %.
Effect of Detection Head
We compare three detection configurations within the OVSeg-OVD and SED-OVD frameworks, with TGME, CMAP, and PTCD incorporated: CLIP feature matching (CLIP), Deformable DETR (Def_DETR), and the proposed query-based CLIP_Trans head. As shown in Table 5, CLIP_Trans achieves AP scores of 39.20% and 41.60% on COCO-OVD in the OVSeg-OVD and SED-OVD frameworks, respectively. These scores exceed CLIP by 1.20 and 2.20 percentage points and Def_DETR by 0.40 and 0.90 points. On LVIS-OVD, the corresponding AP scores reach 26.70% and 28.60%, surpassing CLIP by 2.70 and 3.10 points and Def_DETR by 1.80 and 2.60 points. CLIP_Trans also achieves the highest scores on all remaining metrics in both frameworks. These results support its use for instance-level classification and localization with the text-guided features produced by TMCNet.
Comparisons with State-of-the-Art Methods
Table 6 and Table 7 compare TMCNet with representative open-vocabulary detectors on COCO-OVD and LVIS-OVD, respectively. Figure 4 provides qualitative comparisons.
Comparison of TMCNet with representative open-vocabulary object detection methods on COCO-OVD. † denotes results obtained by retraining and evaluating the models in this work. * indicates the detection-adapted variants of OVSeg and SED. Def_DETR denotes the Deformable DETR-based detection configuration, and CLIP_Trans denotes the query-based CLIP+Transformer detection head adopted in this work. − denotes unreported results.
| Methods | Publication | Train Setting | Backbone | detectors | COCO-OVD(%) | ||
| AP | AP | AP | |||||
| OVR-CNN [2]† | CVPR’21 | V-OVD | RN50-C4 | Faster R-CNN | 22.80 | 46.00 | 39.90 |
| BARON [4]† | CVPR’23 | V-OVD | RN50-C4 | Faster R-CNN | 33.10 | 54.80 | 49.10 |
| BIND [37]† | CVPR’24 | V-OVD | ViT-B/16 | Built-in detector | 36.30 | 54.70 | 50.20 |
| CAKE [12]† | AAAI’25 | V-OVD | RN50 | Faster R-CNN | 38.20 | 58.00 | 52.80 |
| RegionCLIP [38]† | CVPR’22 | C-OVD | RN50-C4 | CLIP | 26.80 | 54.80 | 47.50 |
| CoDet [39]† | NeurIPS’23 | C-OVD | RN50 | CLIP | 30.60 | 52.30 | 46.60 |
| CAKE [12]† | AAAI’25 | C-OVD | RN50 | Faster R-CNN | 41.30 | 60.20 | 55.30 |
| OV-DETR [9]† | ECCV’22 | G-OVD | RN50 | Def_DETR | 29.40 | 61.00 | 52.70 |
| CORA [40]† | CVPR’23 | G-OVD | RN50 | Def_DETR | 35.10 | 35.50 | 35.40 |
| CAKE [12]† | AAAI’25 | G-OVD | RN50 | Faster R-CNN | 39.10 | 58.10 | 53.10 |
| Detic [41]† | ECCV’22 | WS-OVD | Swin-B | Faster R-CNN | 28.40 | 53.80 | 47.20 |
| GOAT [42] | ICCV’23 | WS-OVD | RN50 | Faster R-CNN | 36.40 | 53.00 | 48.60 |
| SAS-Det [43]† | CVPR’24 | WS-OVD | RN50-C4 | Faster R-CNN | 37.40 | 58.50 | 53.00 |
| CAKE [12]† | AAAI’25 | WS-OVD | RN50 | Faster R-CNN | 41.80 | 60.60 | 55.70 |
| MSPL [44]† | ECCV’26 | Pseudo-labels | RN50-FPN | Faster R-CNN | 39.00 | 55.30 | − |
| BARON-BFDet [45]† | AAAI’26 | Pseudo-labels | RN50-C4 | Faster R-CNN | 37.80 | 58.20 | 52.00 |
| VLDet-BFDet [45]† | AAAI’26 | Pseudo-labels | RN50-C4 | Faster R-CNN | 29.90 | 62.10 | 53.70 |
| RegionCLIP-BFDet [45]† | AAAI’26 | Pseudo-labels | RN50-C4 | Faster R-CNN | 29.50 | 58.00 | 50.40 |
| VLDet-BFDet [45]† | AAAI’26 | Pseudo-labels | RN50-C4 | Faster R-CNN | 35.10 | 54.10 | 49.00 |
| OVSeg-OVD* | CVPR’23 | V-OVD | ViT-B/16 | CLIP_Trans | 33.70 | 53.70 | 49.30 |
| SED-OVD* | CVPR’24 | V-OVD | ConvNeXt-B | CLIP_Trans | 36.10 | 54.20 | 49.90 |
| TMCNet | None | V-OVD | ViT-B/16 | CLIP_Trans | 39.20 | 58.50 | 53.30 |
| TMCNet | None | V-OVD | ConvNeXt-B | CLIP_Trans | 41.60 | 60.90 | 55.40 |
Table 6.
Comparison of open-vocabulary object detection methods on LVIS-OVD. † denotes models retrained by the authors of this work. * indicates the detection-adapted variants of OVSeg and SED. Def_DETR denotes the Deformable DETR-based detection configuration, and CLIP_Trans denotes the query-based CLIP+Transformer detection head adopted in this work. − denotes an unreported result.
| Methods | Publication | Backbone | detectors | LVIS-OVD(%) | Inf. time (ms/image) | |||
| AP | AP | AP | mAP | |||||
| OV-DETR [9]† | ECCV’22 | RN50 | Def_DETR | 21.00 | 25.00 | 32.50 | 26.60 | 574.20 |
| BIND [37]† | CVPR’24 | ViT-B/16 | Built-in detector | 29.40 | 30.60 | 33.50 | 31.40 | 42.60 |
| SAS-Det [43]† | CVPR’24 | RN50-C4 | Faster R-CNN | 20.90 | 26.10 | 31.60 | 27.40 | 71.20 |
| CAKE [12]† | AAAI’25 | RN50 | Faster R-CNN | 25.00 | 34.80 | 38.40 | 34.90 | 35.80 |
| MSPL [44]† | ECCV’26 | RN50-FPN | Faster R-CNN | 25.00 | 33.30 | 37.20 | 33.50 | − |
| GenRep-DLM [46]† | ICLR’26 | LDM | − | 27.90 | 30.60 | 33.10 | 32.00 | − |
| GenRep-DiT [46]† | ICLR’26 | DiT | − | 28.00 | 30.10 | 35.80 | 34.60 | − |
| OVSeg-OVD* | CVPR’23 | ViT-B/16 | CLIP_Trans | 22.40 | 28.30 | 36.00 | 32.40 | 240.00 |
| SED-OVD* | CVPR’24 | ConvNeXt-B | CLIP_Trans | 24.80 | 34.50 | 38.00 | 34.00 | 64.00 |
| TMCNet | None | ViT-B/16 | CLIP_Trans | 26.70 | 37.10 | 40.00 | 36.50 | 212.70 |
| TMCNet | None | ConvNeXt-B | CLIP_Trans | 28.60 | 38.10 | 41.00 | 37.70 | 23.30 |
Table 7.
Detection Accuracy and Inference Efficiency
COCO-OVD. Table 6 shows that TMCNet and TMCNet achieve AP scores of 39.20% and 41.60%, respectively, both improving their baselines by 5.50 percentage points. TMCNet leads the listed V-OVD methods across all three metrics, exceeding CAKE [12] by 3.40 points on novel categories. These results show improved novel-category detection alongside gains on base categories across both frameworks.
LVIS-OVD. In Table 7, TMCNet and TMCNet achieve AP scores of 26.70% and 28.60%, exceeding their baselines by 4.30 and 3.80 points, respectively. TMCNet achieves the highest common-category AP, frequent-category AP, and overall mAP among the listed methods, with mAP reaching 37.70%, although BIND retains the highest rare-category AP. Under the reported configurations, the two TMCNet implementations reduce baseline latency by 11.38% and 63.59%, respectively. These results demonstrate gains across category frequency groups and lower inference latency relative to both baselines, with particularly strong overall accuracy for TMCNet.
Qualitative Results on COCO-OVD
Figure 4 presents four examples of detection results from SED-OVD, its TGME-augmented variant SED-OVD, and the full model TMCNet on COCO-OVD. These examples include scenes with multiple objects, occlusion, and densely arranged objects. In the first example, all three models detect the main zebra instances, while incorporating TGME and the full set of modules increases the prediction confidence for some instances. In the second example, an indoor scene, the proposed variants recover several book instances missed by SED-OVD. They also detect objects from novel categories, including refrigerators, microwaves, beds, and potted plants, and accurately localize occluded bottles. In the third example, all models detect the kite, while the proposed variants identify additional distant or partially occluded people. In the fourth example, the proposed variants detect more of the densely arranged books on the shelves. These visual comparisons illustrate fewer missed detections in several challenging scenes. The observations are consistent with the quantitative results and provide complementary evidence for the effectiveness of the proposed text-guided modality correction mechanism.
Extension to Open-Vocabulary Semantic Segmentation
To assess its applicability to dense prediction, we integrate TMCNet into OVSeg [21] and SED [18] while retaining their segmentation prediction mechanisms. Both models are trained on COCO-Stuff-171 with pixel-level annotations and evaluated using mIoU under five settings across ADE20K, PASCAL-Context, and PASCAL VOC.
Effectiveness of TMCNet. As shown in Table 8, TMCNet improves both reproduced baselines across all five settings, with gains of 2.00-6.10 percentage points for OVSeg and 1.60-10.10 points for SED. The largest gains occur on A-150 and PC-59, respectively. Improvements on A-847 and PC-459 also demonstrate benefits for pixel-level prediction with large category vocabularies.
| Methods | VLM | Feature backbone | Train Dataset | A-847 | PC-459 | A-150 | PC-59 | PAS-20 |
| OVSeg [21]† | ViT-B/16 | Swin-B | COCO-Stuff-171 | 7.10 | 11.00 | 24.80 | 53.30 | 92.60 |
| TMCNet | ViT-B/16 | Swin-B | COCO-Stuff-171 | 9.10{sub}+2.00` | 14.20{sub}+3.20` | 30.90{sub}+6.10` | 55.70{sub}+2.40` | 95.40{sub}+2.80` |
| SED [18]† | ConvNeXt-B | - | COCO-Stuff-171 | 8.90 | 12.40 | 31.10 | 47.30 | 93.70 |
| TMCNet | ConvNeXt-B | - | COCO-Stuff-171 | 11.40{sub}+2.50` | 16.70{sub}+4.30` | 32.80{sub}+1.60` | 57.40{sub}+10.10` | 95.30{sub}+1.60` |
Table 8. Effectiveness of TMCNet in different open-vocabulary semantic segmentation frameworks. † denotes results obtained by retraining and evaluating the models in this work. – indicates that no additional feature backbone is used. Boldface highlights our methods, and red subscripts indicate absolute gains in percentage points over the corresponding reproduced baseline.
Comparison with Representative Methods. Table 9 shows that TMCNet achieves the highest mIoU among the listed configurations on four settings, while TMCNet leads on PAS-20 with 95.40%. Across all five settings, TMCNet outperforms the reproduced CAT-Seg + DeCLIP [49] by 1.30-2.70 points and LoGoSeg [50] with the same ConvNeXt-B visual encoder by 0.40-2.30 points. These results support the applicability of text-guided modality correction to open-vocabulary semantic segmentation across both frameworks.
| Methods | Publication | VLM | Training Dataset | A-847 | PC-459 | A-150 | PC-59 | PAS-20 |
| ZegFormer [47] | CVPR’22 | ViT-B/16 | COCO-Stuff-156 | 4.90 | 9.10 | 16.90 | 42.80 | 86.20 |
| ZSseg [48] | ECCV’22 | ViT-B/16 | COCO-Stuff-171 | 7.00 | – | 20.50 | 47.70 | 88.40 |
| OVSeg [21]† | CVPR’23 | ViT-B/16 | COCO-Stuff-171 | 7.10 | 11.00 | 24.80 | 53.30 | 92.60 |
| SED [18]† | CVPR’24 | ConvNeXt-B | COCO-Stuff-171 | 8.90 | 12.40 | 31.20 | 47.30 | 93.70 |
| CAT-Seg + DeCLIP [49]† | CVPR’25 | ViT-B/16 | COCO-Stuff-171 | 9.50 | 15.40 | 31.30 | 55.60 | 92.60 |
| LoGoSeg [50]† | AAAI’26 | ConvNeXt-B | COCO-Stuff-171 | 10.20 | 16.30 | 32.00 | 55.80 | 93.00 |
| LoGoSeg [50]† | AAAI’26 | ViT-B/16 | COCO-Stuff-171 | 10.60 | 16.50 | 32.60 | 56.20 | 93.40 |
| TMCNet | Ours | ViT-B/16 | COCO-Stuff-171 | 9.10 | 14.20 | 30.90 | 55.70 | 95.40 |
| TMCNet | Ours | ConvNeXt-B | COCO-Stuff-171 | 11.40 | 16.70 | 32.80 | 57.40 | 95.30 |
Table 9. Comparison with representative open-vocabulary semantic segmentation methods. All results are reported in mIoU (%). † denotes results reproduced in this work; other baseline results are taken from the corresponding papers or official implementations. – denotes an unreported result.
Conclusions
In this work, we present TMCNet, a Text-Guided Modality Correction Network for open-vocabulary object detection that incorporates category semantics into query construction, cross-modal alignment, and multiscale decoding. The Text-Guided Modality Encoder uses multi-head cross-attention and masked max pooling to encode textual priors into initial queries, which are combined with projected object tokens to form semantic query representations. The Cross-Modal Alignment Pipeline maps these queries into a shared feature space and provides two complementary outputs: dimension-adapted queries for lateral visual feature correction, and joint features obtained by fusing aligned queries with projected vision–text association features. The Progressive Text Correction Decoder then integrates the corrected lateral features and scale-adapted joint features with the evolving decoder representation, combining high-level semantics with spatial details for object classification and bounding box regression. Experiments on COCO-OVD and LVIS-OVD show improvements in novel-category and overall detection performance across both detection-adapted frameworks. Incorporating the same mechanism into OVSeg and SED also improves segmentation accuracy across all five evaluation settings, supporting its applicability to open-vocabulary pixel-level prediction.
References
- [1]X. Gu, T.-Y. Lin, W. Kuo, Y. Cui, Open-vocabulary object detection via vision and language knowledge distillation, in: ICLR, 2022.arxiv.org/abs/2104.13921
- [2]A. Azareian, K. D. Rosa, D. H. Hu, S.-F. Chang, Open-vocabulary object detection using captions, in: ICCV, 2021, pp. 14393–14402.arxiv.org/abs/2011.10678
- [3]C. Feng, Y. Zhong, Z. Jie, X. Chu, H. Ren, X. Wei, W. Xie, L. Ma, Promptdet: Towards open-vocabulary object detection using uncurated images, in: ECCV, Vol. 13669, 2022, pp. 701–717.arxiv.org/abs/2203.16513
- [4]S. Wu, W. Zhang, S. Jin, W. Liu, C. C. Loy, Aligning bag of regions for open-vocabulary object detection, in: ICCV, 2023, pp. 15254–15264.arxiv.org/abs/2302.13996
- [5]Z. Ma, G. Luo, J. Gao, L. Li, Y. Chen, S. Wang, C. Zhang, W. Hu, Open-vocabulary one-stage detection with hierarchical visual-language knowledge distillation, in: CVPR, 2022, pp. 14074–14083.arxiv.org/abs/2203.10593
- [6]T. Cheng, L. Song, Y. Ge, W. Liu, X. Wang, Y. Shan, Yolo-World: Real-time open-vocabulary object detection, in: CVPR, 2024, pp. 16901–16911.arxiv.org/abs/2401.17270
- [7]L. Liu, J. Feng, H. Chen, A. Wang, L. Song, J. Han, G. Ding, Yolo-uniow: Efficient universal open-world object detection, arXiv (2024).arxiv.org/abs/2412.20645
- [8]N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, S. Zagoruyko, End-to-end object detection with transformers, in: ECCV, Vol. 12346, 2020, pp. 213–229.arxiv.org/abs/2005.12872
- [9]Y. Zang, W. Li, K. Zhou, C. Huang, C. C. Loy, Open-vocabulary detr with conditional matching, in: ECCV, Vol. 13669, 2022, pp. 106–122.DOI
- [10]Y. Xu, M. Zhang, X. Yang, C. Xu, Exploring multi-modal contextual knowledge for open-vocabulary object detection, IEEE Transactions on Image Processing 33 (2024) 6253–6267.arxiv.org/abs/2308.15846
- [11]S. Wu, W. Zhang, L. Xu, S. Jin, X. Li, W. Liu, C. C. Loy, Clipself: Vision transformer distills itself for open-vocabulary dense prediction, in: ICLR, 2024.arxiv.org/abs/2310.01403
- [12]S. Ma, D. Qian, K. Ye, S. Zhang, CAKE: Category aware knowledge extraction for open-vocabulary object detection, Proceedings of the AAAI Conference on Artificial Intelligence 39 (6) (2025) 5982–5990.DOI
- [13]S. Fu, Y. Su, F. Rao, J. Lyu, X. Xie, W.-S. Zheng, WeDetect: Fast open-vocabulary object detection as retrieval, in: CVPR, 2026, pp. 20377–20387.arxiv.org/abs/2512.12309
- [14]L. Yang, B. Cai, W. Liu, Y. Wang, C. Yuan, B. Li, W. Hu, SRA-Det: Learning omni-grained open-vocabulary detection beyond category names, in: CVPR, 2026, pp. 27611–27620.
- [15]T. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, S. Belongie, Feature pyramid networks for object detection, in: CVPR, 2017, pp. 2117–2125.arxiv.org/abs/1612.03144
- [16]S. Liu, L. Qi, H. Qin, J. Shi, J. Jia, Path aggregation network for instance segmentation, in: CVPR, 2018, pp. 8759–8768.arxiv.org/abs/1803.01534
- [17]S. Cho, H. Shin, S. Hong, A. Arnab, P. H. Seo, S. Kim, CAT-Seg: Cost aggregation for open-vocabulary semantic segmentation, in: CVPR, 2024, pp. 4113–4123.arxiv.org/abs/2303.11797
- [18]B. Xie, J. Cao, J. Xie, F. S. Khan, Y. Pang, Sed: A simple encoder-decoder for open-vocabulary semantic segmentation, in: CVPR, 2024, pp. 3426–3436.
- [19]L. Sun, J. Cao, J. Xie, X. Jiang, Y. Pang, CLIPer: Hierarchically improving spatial representation of CLIP for open-vocabulary semantic segmentation, in: ICCV, 2025, pp. 23199–23209.arxiv.org/abs/2411.13836
- [20]Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, S. Xie, A convnet for the 2020s, in: CVPR, 2022, pp. 11976–11986.arxiv.org/abs/2201.03545
- [21]F. Liang, B. Wu, X. Dai, K. Li, Y. Zhao, H. Zhang, P. Zhang, P. Vajda, D. Marculescu, Open-vocabulary semantic segmentation with mask-adapted clip, in: CVPR, 2023, pp. 7061–7070.arxiv.org/abs/2210.04150
- [22]C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, J. Gao, Llava-med: Training a large language-and-vision assistant for biomedicine in one day, in: NeurIPS, Vol. 36, 2023, pp. 28541–28564.arxiv.org/abs/2306.00890
- [23]H. Liu, C. Li, Q. Wu, Y. J. Lee, Visual instruction tuning, in: NeurIPS, Vol. 36, 2023, pp. 34892–34916.arxiv.org/abs/2304.08485
- [24]F. Liu, T. Zhu, X. Wu, B. Yang, C. You, C. Wang, L. Lu, Z. Liu, Y. Zheng, X. Sun, et al., A medical multimodal large language model for future pandemics, NPJ Digital Medicine 6 (1) (2023) 226.DOI
- [25]S. Lee, G. Lee, H. Park, B. Ham, Exploring hierarchical consistency and unbiased objectness for open-vocabulary object detection, in: CVPR, 2026, pp. 6819–6828.arxiv.org/abs/2604.23344
- [26]T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, C. L. Zitnick, Microsoft coco: Common objects in context, in: ECCV, Vol. 8693, 2014, pp. 740–755.arxiv.org/abs/1405.0312
- [27]A. Gupta, P. Dollar, R. Girshick, LVIS: A dataset for large vocabulary instance segmentation, in: CVPR, 2019, pp. 5356–5364.arxiv.org/abs/1908.03195
- [28]M. Xu, Z. Zhang, F. Wei, H. Hu, X. Bai, Side adapter network for open-vocabulary semantic segmentation, in: CVPR, 2023, pp. 2945–2954.arxiv.org/abs/2302.12242
- [29]L. Chen, Q. Yang, K. Ding, Z. Li, G. Shen, F. Li, Q. Cao, S. Xiang, Efficient redundancy reduction for open-vocabulary semantic segmentation, Neurocomputing 665 (2026) 132229.DOI
- [30]H. Caesar, J. Uijlings, V. Ferrari, Coco-stuff: Thing and stuff classes in context, in: CVPR, 2018, pp. 1209–1218.arxiv.org/abs/1612.03716
- [31]B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, A. Torralba, Semantic understanding of scenes through the ade20k dataset, International Journal of Computer Vision 127 (3) (2019) 302–321.arxiv.org/abs/1608.05442
- [32]M. Everingham, L. V. Gool, C. Williams, J. Winn, A. Zisserman, The pascal visual object classes (voc) challenge, International Journal of Computer Vision 88 (2) (2010) 303–338.
- [33]R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, A. Yuille, The role of context for object detection and semantic segmentation in the wild, in: CVPR, 2014, pp. 891–898.
- [34]B. Cheng, A. Schwing, A. Kirillov, Per-pixel classification is not all you need for semantic segmentation, in: NeurIPS, Vol. 34, 2021, pp. 17864–17875.arxiv.org/abs/2107.06278
- [35]A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., An image is worth 16x16 words: Transformers for image recognition at scale, in: ICLR, 2021.arxiv.org/abs/2010.11929
- [36]I. Loshchilov, F. Hutter, Decoupled weight decay regularization, in: ICLR, 2019.
- [37]H. Zhang, Q. Zhao, L. Zheng, H. Zeng, Z. Ge, T. Li, S. Xu, Exploring region-word alignment in built-in detector for open-vocabulary object detection, in: CVPR, 2024, pp. 16975–16984.DOI
- [38]Y. Zhong, J. Yang, P. Zhang, C. Li, N. Codella, L. H. Li, L. Zhou, X. Dai, L. Yuan, Y. Li, J. Gao, RegionCLIP: Region-based language-image pretraining, in: CVPR, 2022, pp. 16793–16803.arxiv.org/abs/2112.09106
- [39]C. Ma, Y. Jiang, X. Wen, Z. Yuan, X. Qi, CoDet: Co-occurrence guided region-word alignment for open-vocabulary object detection, in: NeurIPS, Vol. 36, 2023, pp. 71078–71094.arxiv.org/abs/2310.16667
- [40]X. Wu, F. Zhu, R. Zhao, H. Li, CORA: Adapting CLIP for open-vocabulary detection with region prompting and anchor pre-matching, in: CVPR, 2023, pp. 7031–7040.arxiv.org/abs/2303.13076
- [41]X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, I. Misra, Detecting twenty-thousand classes using image-level supervision, in: ECCV, Vol. 13669, 2022, pp. 350–368.arxiv.org/abs/2201.02605
- [42]J. Wang, Z. Zhang, H. Hong, X. Jin, Y. He, H. Xue, Z. Zhao, Open-vocabulary object detection with an open corpus, in: ICCV, 2023, pp. 6759–6769.DOI
- [43]S. Zhao, S. Schulter, L. Zhao, Z. Zhang, Vijay Kumar B G, Y. Suh, M. Chandraker, D. N. Metaxas, Taming self-training for open-vocabulary object detection, in: CVPR, 2024, pp. 13938–13947.arxiv.org/abs/2308.06412
- [44]H. Choi, Y. Lim, J. Shin, H. Shim, MSPL: Multi-step pseudo-labeling for open-vocabulary object detection, in: ECCV, Vol. 17037, 2026, pp. 501–520.arxiv.org/abs/2510.14792
- [45]Y. Li, J. Niu, N. Gu, T. Ren, From scene to object: Enhancing open-vocabulary object detection via foreground-background context reasoning, Proceedings of the AAAI Conference on Artificial Intelligence 40 (8) (2026) 6601–6609.
- [46]L. Li, Y. Yang, W. Wang, Reconciling visual perception and generation in diffusion models, in: ICLR, 2026.
- [47]J. Ding, N. Xue, G.-S. Xia, D. Dai, Decoupling zero-shot semantic segmentation, in: CVPR, 2022, pp. 11583–11592.arxiv.org/abs/2112.07910
- [48]M. Xu, Z. Zhang, F. Wei, W. Li, Y. Cao, H. Hu, X. Bai, A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model, in: ECCV, Vol. 13689, 2022, pp. 736–753.arxiv.org/abs/2112.14757
- [49]J. Wang, B. Chen, Y. Li, B. Kang, Y. Chen, Z. Tian, DeCLIP: Decoupled learning for open-vocabulary dense perception, in: CVPR, 2025, pp. 14824–14834.arxiv.org/abs/2505.04410
- [50]J. Chen, X. Lv, Z. Kou, X. Sheng, N. Xu, Y. Qiao, LoGoSeg: Integrating local and global features for open-vocabulary semantic segmentation, Proceedings of the AAAI Conference on Artificial Intelligence 40 (4) (2026) 2886–2894.arxiv.org/abs/2602.05578

