WorksComputer Science
See and Detect: Image-Driven Open-Vocabulary Multi-View 3D Object Detection
Abstract
Recent open-vocabulary multi-view indoor 3D detection methods typically follow a point-cloud-driven pipeline, where pseudo 3D boxes are generated from reconstructed or estimated geometry and semantic labels are inferred from projected regions. However, such a design makes box generation dependent on the completeness of point clouds and restricts semantic learning to local appearance, causing inferior 3D object localization and open-vocabulary classification. In this paper, we present See and Detect, an image-driven framework that uses multi-view images as the primary evidence for 3D box and semantic label generation. For box generation, we first discover object candidates from individual images, enabling objects missing in point clouds to be recovered from visual evidence. However, single-view predictions are often noisy and redundant. To address this, we propose multi-view consensus box consolidation to fuse spatially consistent predictions across views, and box quality assessment to reduce the impact of boxes with partial object coverage. For semantic label generation, we introduce context-aware semantic learning, where semantic labels are queried from full image context, enabling more accurate semantic prediction. Extensive experiments on ScanNet and ARKitScenes demonstrate that our method achieves state-of-the-art open-vocabulary multi-view 3D detection performance while running over 500 faster than the previous state-of-the-art method.
Introduction
Open-vocabulary 3D object detection [3, 8, 14, 23] is a fundamental task for open-world 3D scene understanding, aiming to localize and classify arbitrary object categories in 3D space. It is essential for embodied applications such as navigation, manipulation, and human–robot interaction. Although point-cloud-based methods [3, 14] have achieved impressive progress, they rely on depth sensors or LiDAR during deployment to capture geometric inputs, which are expensive, power-intensive, and often unavailable on consumer-grade platforms. Recently, detecting 3D objects from posed multi-view RGB images offers a more accessible setting, as such images can be readily captured by embodied agents when they navigate an environment.
Existing multi-view open-vocabulary 3D detection methods, such as OpenM3D [7] and Zoo3D [12], use posed RGB images as input, but construct training supervision through a point-cloud-driven pipeline. Specifically, they generate pseudo boxes for class-agnostic localization by lifting 2D image segments into 3D followed by point-cloud validation [7], or by directly grouping point-cloud segments [12]. The resulting 3D boxes are then projected back to 2D images, where the corresponding image crops are used to infer semantic labels for open-vocabulary classification. While promising, this design makes both 3D box and semantic label generation mainly determined by point-cloud-derived 3D regions. For box generation, objects with missing or sparse geometry are overlooked, even when they are consistently visible across views (a brown chair that is visible in multiple views is missed by the point-cloud-based method in Figure 1 (i)). For semantic learning, labels inferred from projected regions capture only local appearance and thus ignore the surrounding scene context, leading to wrong predictions for small objects (power socket misclassified to card in Figure 1 (ii)). These limitations indicate that existing methods involve images but do not let image evidence drive the key decisions. In contrast, cross-view visual consistency and scene context can serve as primary evidence for box generation and semantic prediction.

Figure 1. Comparison of training pipelines for open-vocabulary multi-view 3D object detection. (a) Existing methods [7, 12] propose point-cloud-driven approach for 3D box and semantic label generation, which leads to (i) missing object discovery and (ii) incorrect semantic label generation. (b) See and Detect (Ours) proposes a multi-view image-driven approach: objects are discovered through cross-view visual evidence, and semantic labels are generated by considering image contexts.
To this end, we propose See and Detect, an image-driven framework that uses multi-view images as the primary evidence for both 3D box and semantic label generation. Instead of changing pseudo-box generation to reconstructed or sensor-captured point clouds, we derive object proposals from individual RGB views, with depth maps serving as intermediate geometric cues. Then, to consolidate single-view predictions, we propose multi-view consensus box consolidation, where spatially overlapping detections across views are linked into a scene-level graph and nodes with low connectivity are pruned as spurious. We further propose box quality assessment to reduce the impact of pseudo boxes with partial object coverage by measuring their cross-view visibility consistency and reweighting the localization loss accordingly.
For semantic label generation, we propose context-aware semantic learning to incorporate scene context into semantic label generation. Specifically, we query object labels from full images with projected regions highlighted, allowing the prediction to consider global scene context and consequently generating more accurate semantic labels. We use an MLLM to perform this context-aware query and average its predictions across views as semantic learning targets. We further introduce semantic-aware negative sampling for effective contrastive learning, which selects informative negative labels based on semantic similarity in the text feature space. Our main contributions are summarized as follows:
We develop an image-driven pseudo box supervision pipeline that leverages existing single-view 3D proposals and introduces multi-view consensus box consolidation and box quality assessment to construct reliable localization supervision.
We propose context-aware semantic learning, where semantic labels are inferred from full images with projected object regions to incorporate both local appearance and global scene context. We further introduce semantic-aware negative class selection for effective contrastive text–3D alignment.
We achieve state-of-the-art performance on both ScanNet and ARKitScenes datasets. Notably, our method outperforms prior work by +4.3% mAP on ScanNet60 open-vocabulary detection while running over faster.
Related Work
Multi-View Indoor 3D Object Detection. Multi-view indoor 3D object detection [19, 21, 22, 25, 26, 27] aims to detect 3D bounding boxes for an indoor scene by taking images as input. Compared to point-cloud-based methods [13, 16, 17, 20, 24, 28] that rely on the expensive depth sensor to capture input data, multi-view based methods are more cost-effective and scalable. The pioneer work, ImVoxelNet [19], builds a 3D feature volume in the world space, where each voxel center aggregates corresponding 2D features at its projected pixel location. Then, ImVoxelNet refines the feature volume through a shallow 3D U-Net before predicting 3D bounding boxes from each voxel. However, it remains challenging to aggregate correct image features since ground-truth depth is not known. Therefore, ImGeoNet [22] predicts the surface probability for each voxel to weigh the original feature volume. It supervises the prediction using ground-truth point cloud. Another line of works [25, 26, 27] propose to predict geometry in a self-supervised way. NerfDet [25] queries an opacity score for each voxel from a Neural Radiance Field (NeRF) [15] which is jointly trained with the detection pipeline. Later, Zhang et al. [27] reduce the computation of the feature volume by adaptively refining voxels with high occupancy probability. However, above models remain closed-set, i.e. they are restricted to a predefined set of training categories and cannot generalize to unseen classes at inference time.
Open-Vocabulary 3D Object Detection. To adapt the 3D object detector in the open world with emerging new classes, open-vocabulary 3D object detection [3, 4, 7, 8, 14, 23] is introduced, which can detect new unseen classes in the testing stage given only class names. The open-vocabulary ability is achieved through class-agnostic object localization and open-vocabulary classification. OV3DET [14] generates class-agnostic boxes by unprojecting the predicted 2D boxes into 3D and refining using point cloud. For the semantic alignment, it introduces De-biased Triplet Cross-Modal Contrastive Learning (DTCC) to align the modality of point cloud, image and text. CoDA [3] proposes 3D Novel Object Discovery strategy to identify the bounding boxes of novel objects by jointly utilizing the geometry priors from base categories and 2D semantic priors from the CLIP model [18]. For semantic alignment, it introduces discovery-driven cross-modal alignment module to align the 3D box features with the corresponding image crop and text label features extracted by CLIP. However, they all require point cloud as input, which requires depth sensor and is expensive to capture. Therefore, multi-view open-vocabulary 3D object detection [7, 12] is introduced that does not require point cloud as input. Built upon ImGeoNet [22], OpenM3D generates 3D pseudo boxes by lifting and grouping the 2D image segments into 3D and then validating each cluster with point cloud, which are then used for training object localization. To achieve open-vocabulary classification, it learns to align the voxel features with the CLIP features of corresponding 2D segments. Zoo3D [12] shares the similar strategy, where pseudo boxes are generated from point cloud instance segmentation and open-vocabulary classification is achieved by comparing the text features with the extracted region-based CLIP features that encode the semantics of the 3D boxes.
Despite their promise, they underutilize images for both box generation and semantic learning. Using point cloud for checking and aggregating 3D boxes can overlook objects with missing geometry, although observable in images. Only using projected image crop to derive semantic label is unreliable as local appearance can be ambiguous and global context is necessary to aid reasoning. To this end, we propose an image-driven pipeline to fully leverage the multi-view information for effective open-vocabulary 3D detection.
We note that concurrent BoxFusion [10] also associates single-view Cubify Anything [11] proposals across views. However, it targets training-free online detection, sequentially fusing keyframe proposals with existing detections under causal processing. In contrast, our offline setting enables scene-level association over all proposals, allowing cross-view consensus to suppress isolated predictions (Multi-View Consensus Box Consolidation) and assess imperfect boxes (Box Quality Assessment) for reliable pseudo supervision.
Method
In open-vocabulary multi-view indoor 3D object detection (OV-MVI3Det), we are given multi-view images of a scene, denoted as , together with the corresponding camera parameters , where denotes the intrinsic matrix, and denotes the extrinsic parameters of the -th view. The goal of OV-MVI3Det is to localize and classify objects from only multi-view images in the testing stage.
Framework Overview
Figure 2 shows our framework. We adopt SGCDet [27] as the detection backbone, which is the state-of-the-art multi-view 3D object detection model for indoor scenes. However, SGCDet is a closed-set detector, where object categories must be predefined during training. To extend it to open-vocabulary 3D detection, we augment the detector with two heads: a class-agnostic box head for object localization and a 3D semantic feature head for open-vocabulary classification. In the first head, we train for class-agnostic 3D object localization with the pseudo boxes generated via image-driven approach (§ 3.2), which produces complete and reliable boxes. In the second head (§ 3.3), we aim to align the 3D with text feature space, where the context-aware semantic learning is introduced to generate reliable semantic labels by reasoning with scene context. We further apply semantic-aware negative class selection for effective text-3D contrastive loss.
Figure 2. Our framework. During training, it is optimized by two main objectives: (1) a Weighted Localization Loss , where generated pseudo 3D boxes are weighted by the quality scores (Eqn. (3)) based on their geometric consistency (details shown in Figure 3); and (2) a Text-3D Contrastive Loss , which aligns 3D features with semantic labels inferred from MLLM by jointly considering local appearance and global context. We apply Semantic-aware Negative Class Selection to generate informative negative classes.
Image-Driven Pseudo Box Generation
Figure 3 shows the proposed image-driven pseudo box generation pipeline. We use Cubify Anything [11] as a per-view image-driven proposal generator to predict initial class-agnostic 3D box candidates. The predicted candidates from the -th view are denoted as , where is the -th predicted box candidate from image , and is the corresponding depth map of to provide 3D information. These candidates provide visual evidence for object discovery, but are noisy and redundant when generated independently from different views. Therefore, we further perform multi-view consensus box consolidation and box quality assessment to convert the per-view candidates into reliable pseudo 3D boxes for training.
Figure 3. Image-Driven Pseudo Box Generation. We perform single-view 3D object detection for each image using Cubify Anything. A scene-level undirected graph is constructed where edges represent spatial overlap (Eqn. (1)). Multi-View Consensus Box Consolidation is applied to fuse consistent predictions and suppress spurious single-view detections. Finally, each box is assigned a quality score through Box Quality Assessment based on cross-view visibility and SAM-based object coverage check , which weights the localization loss during training.
Multi-View Consensus Box Consolidation. To associate cross-view predictions, we construct an undirected graph for each scene. Each node represents a predicted 3D box . An edge between and is established if they have sufficient spatial overlap in 3D:
where denotes the 3D IoU.
Then, the resulting graph is decomposed into connected components. Each component corresponds to a set of cross-view box predictions that agree on a shared 3D location. We observe that true objects are typically detected consistently across multiple views, forming components with multiple nodes. In contrast, spurious detections often appear only in a single view, resulting in singleton components. Therefore, we introduce a multi-view consensus filtering strategy to retain only components containing more than one node and discard singleton components. For each retained component, we fuse all boxes within the component to generate a consolidated pseudo 3D box. Importantly, during fusion, we also record the associated view indices of all boxes within the component. For the -th component, which contains boxes, we denote its supporting view set as , where is the view from which a box in the component is predicted.
Box Quality Assessment. However, the fused boxes may correspond to incomplete object coverage, especially when they are formed from limited local views. As illustrated in Figure 4 (a), only a partial sofa is detected since all the supporting views show part of the sofa. To reduce the influence of inferior pseudo boxes during training, we propose box quality assessment. Our key observation is that a high-quality 3D box should satisfy two geometric consistency properties: (1) it should remain visible when projected to nearby views; (2) its projection should tightly cover the object instance rather than only a partial region. Therefore, we evaluate each pseudo box by checking visibility and object coverage in the neighbouring views of its supporting view set.
Figure 4. Box quality assessment. (a) shows a pseudo box with incomplete object coverage due to detection from local image views. (b) shows a pseudo box with complete object coverage. We project the boxes to the nearby view for box quality assessment. The quality score can correctly reflect box quality.
For each pseudo box, we select the view with the highest prediction confidence as the anchor view, and project the box to its nearby views denoted as . Let denote the homogeneous coordinates of the 3D box corners. The projection to view is computed as , followed by perspective division to obtain 2D projected corners , where . A projected corner is considered visible if it lies within the image boundary, i.e., and . We define the visibility indicator for view as
For visible projection, we further check object coverage using SAM [9]. Our intuition is that a projection that tightly covers the instance is a good prompt for SAM, leading to high-confidence mask prediction. Therefore, we use the minimum and maximum coordinates of the projected corners as the prompt. We denote the SAM confidence as . The final box quality score is computed as
where is the number of nearby views and is empirically set to 3. During training, is used as the weight for object localization loss so that imperfect pseudo boxes contribute less to optimization.
Context-Aware Semantic Learning
To leverage the rich context information in the images, we use a multi-modal large language model (MLLM) to infer semantic label for each pseudo box. Specifically, we project each pseudo box onto its supporting views and overlay the projected 2D box on the image as a visual prompt for Qwen3-VL [1], enabling category prediction with both local object appearance and scene context. Let denote the predicted label from the -th supporting view, and is the extracted text feature by CLIP. The semantic label for the pseudo box is computed as follows:
where is the number of supporting views of the pseudo box.
Semantic-Aware Negative Class Selection. To perform effective text-3D contrastive learning, the quality of negative classes matters. Therefore, we construct semantic-aware negative classes by measuring the cosine similarity between the semantic label and the text features of a class superset . Classes are then sorted in descending order based on the similarity scores. Classes ranked at the top are highly similar and may correspond to the true class or near-duplicate categories, while classes ranked at the bottom are semantically unrelated and provide limited discriminative learning signal. Therefore, we select the negative classes from a middle range denoted as , where represents the index of class ranked at position .
Optimization. To enable open-vocabulary classification, we convert the classification head of SGCDet to predict a fixed-length feature vector that is aligned with the CLIP feature space. We denote the feature vector as . The text-3D contrastive loss is as follows:
where . is the temperature hyper-parameter and set to 0.1. The network is jointly trained with object localization loss [27] weighted by our proposed box quality assessment score and semantic learning objective to achieve open-vocabulary 3D object detection.
Experiment
Datasets
We evaluate on two challenging 3D indoor detection datasets: ScanNet [5] and ARKitScenes [2]. ScanNet has 1,201 training and 312 testing scenes with annotations for 200 object categories. We evaluate on two benchmarks, i.e. ScanNet60 with 60 classes [3] and ScanNet200 with 200 classes [7]. We report mean average precision (mAP) under IoU thresholds of 0.25 and 0.5. ARKitScenes consists of 4493 training and 549 testing scans. These scans contain RGB-D frames along with 3D object bounding box annotations of 17 categories. Following [7], we use ARKitScenes only for 3D pseudo box evaluation and report precision and recall under IoU thresholds of 0.25 and 0.5.
Implementation Details
Our detection backbone is SGCDet [27], which is the state-of-the-art multi-view indoor 3D object detector. We follow the training and testing configuration of SGCDet. Images are resized to as the input with 40 and 100 images for training and testing, respectively. The network is trained end-to-end under 50 epochs using AdamW optimizer. In the pseudo box generation, we evenly sample 60 views for each scene to detect bounding boxes. We perform Non-Maximum Suppression and size constraints after multi-view consensus box consolidation. is set to 0.4 in the edge construction. In the context-aware semantic learning, we follow CoDA [3] to adopt LVIS [6] categories as the class superset. and are set to 10 and 150 in the semantic-aware negative class selection. We use Qwen3-VL-30B-A3B-Instruct [1] as the MLLM for semantic label generation. During testing stage, we measure the cosine similarity between the predicted 3D box feature and CLIP text embeddings of text prompts to perform open-vocabulary classification. Following CoDA [3], the template for text prompt is set as “A photo of [category name] in the scene”. Importantly, Cubify Anything, SAM, and Qwen3-VL are used only for offline pseudo-label generation during training. At inference time, our method uses only the feed-forward multi-view 3D detector with RGB images and camera parameters as input.
Comparison with Baselines
Open-Vocabulary 3D Object Detection. Tab. 1 compares our method with prior open-vocabulary 3D object detection approaches on ScanNet60 and ScanNet200 benchmarks. The baselines include state-of-the-art point-cloud-based methods such as CoDA [3], OV-3DET [14] and OV-Uni3DETR [23], as well as multi-view-based approaches including OpenM3D [7] and Zoo3D [12]. We report the best results in the setting of posed images from Zoo3D [12], which learns a class-agnostic 3D object detection from the pseudo boxes generated from point cloud instance segmentation and then extracts the CLIP features of the projected 2D box crops for open-vocabulary classification. Despite taking only RGB images as input, our method largely outperforms point-cloud-based open-vocabulary detectors (e.g. improves over OV-Uni3DETR by +9.1 on ScanNet60). Compared with multi-view based methods, our approach also achieves state-of-the-art performance. In particular, we outperform Zoo3D by +4.6 mAP and +4.3 mAP on ScanNet60. It verifies the superiority of the proposed image-driven approach for pseudo box and semantic label generation, which provides more complete 3D boxes and more accurate semantic labels than point-cloud-driven approaches (i.e. Zoo3D and OpenM3D). Moreover, we run much faster than Zoo3D during inference, achieving a 500x speedup. The large speed gap stems from the fact that Zoo3D performs online 3D reconstruction and aligns 3D boxes to 2D image crops for open-vocabulary classification at inference time, while our method is a single-stage feed-forward 3D detector. Figure 5 shows qualitative comparison on ScanNet60, where we show more compact and accurate open-vocabulary 3D object detections.

Figure 5. Qualitative results of open-vocabulary multi-view 3D object detection on ScanNet60.
Class-Agnostic 3D Object Detection. Tab. 2 reports the class-agnostic 3D detection performance on ScanNet200. In this setting, the detector predicts foreground 3D bounding boxes without category supervision, thus directly evaluating the quality of pseudo boxes. Our method significantly outperforms prior approaches across all metrics. In particular, we achieve 52.7 mAP and 25.3 mAP, surpassing Zoo3D by +16.6 at mAP and +11.4 at mAP, respectively. Compared to OpenM3D, our method improves mAP by +25.8 (26.9 52.7). It verifies the superiority of the proposed image-driven pseudo box generation pipeline, which provides high-quality box training data for class-agnostic localization.
3D Pseudo Box Evaluation. Tab. 3 shows the evaluation of 3D pseudo boxes on ScanNet200 and ARKitScenes datasets. We achieve best recall and precision, which verifies the proposed image-driven pseudo box generation method with multi-view consensus box consolidation provides complete and accurate pseudo boxes.
Ablation Study
Effectiveness of Multi-View Consensus Box Consolidation. Tab. 4 shows the ablation study of multi-view consensus filtering during pseudo box generation on ScanNet200. “No Filtering” denotes no multi-view consensus filtering, i.e. keep all components. “Filtering (< 3 views)” refers to removing components that have fewer than 3 nodes (i.e. observed in less than 3 views). “No Filtering” shows the highest recall (76.5 / 61.2 at ) but very low precision (34.1 / 27.3), as many spurious single-view detections are preserved. By removing only singleton components, precision improves significantly (e.g., increases from 27.3 to 41.9) while recall at IoU 0.5 drops only slightly (61.2
Table 1. Open-vocabulary multi-view 3D object detection on ScanNet60 and ScanNet200 benchmarks. “Test Input” denotes the input at inference stage. Inference time is measured in seconds per scene. We report the best results of Zoo3D from their paper under the multi-view setting. Best results are marked bold.
Table 2. Class-agnostic 3D object detection on ScanNet200. Best results are marked bold.
→ 59.6), indicating that most false positives originate from single-view predictions and can be effectively suppressed without sacrificing valid objects. Further applying stricter filtering (“Filtering (< 3 views)”) leads to very marginal improvement on precision but at a sacrifice of substantial decrease in recall. Based on this precision–recall trade-off, we adopt the strategy of removing only singleton components in the multi-view consensus filtering.
Effectiveness of Box Quality Assessment. Table 5 shows the ablation study on the box quality assessment on the class-agnostic 3D object detection on ScanNet200. “No Asses.” denotes no box quality assessment and for all boxes in the training. “Visibility Asses.” denotes only checking visibility to compute . “No Asses.” and
| Method | mAP | mAP | mAR | mAR |
| No Asses. | 50.5 | 23.6 | 71.1 | 38.6 |
| Visibility Asses. | 51.9 | 24.1 | 71.5 | 38.9 |
| Ours | 52.7 | 25.3 | 72.2 | 38.9 |
Table 5. Ablation study on the box quality assessment on ScanNet200. “No Asses.” denotes no box quality assessment, while “Visibility Asses.” denotes only checking visibility.
Table 3. 3D pseudo box evaluation on ScanNet200 and ARKitScenes dataset. We report recall (R) and precision (P) at IoU thresholds of 0.25 and 0.50. Best results are marked bold.
| Method | R | P | R | P |
| No Filtering | 76.5 | 34.1 | 61.2 | 27.3 |
| Filtering (< 3 views) | 65.5 | 53.7 | 56.3 | 43.9 |
| Ours | 71.9 | 50.5 | 59.6 | 41.9 |
Table 4. Ablation on multi-view consensus box consolidation on ScanNet200. “No Filtering” and “Filtering (< 3 views)” denote no filtering and filtering components observed less than 3 views, respectively.
“Visibility Asses.” are worse than ours because they do not fully assess pseudo box quality.
Effectiveness of Context-Aware Semantic Learning. Table 6 shows the ablation study on context-aware semantic learning. We compare with the semantic alignment strategies that rely on image crops. “IMG Crop” is the class-agnostic distillation loss [3], which aligns 3D with image region features. “IMG Crop + Text” further incorporates the DTCC loss [14], which introduces additional text supervision derived from region-level CLIP similarities. However, both models show worse performance than ours, indicating that semantic alignment built upon local image crops alone is limited. In contrast, we leverage MLLM to jointly reason
| Method | ScanNet60 | ScanNet200 | ||
| mAP | mAP | mAP | mAP | |
| IMG Crop | 20.6 | 11.4 | 7.8 | 4.4 |
| IMG Crop+Text | 21.0 | 11.7 | 9.9 | 4.5 |
| Ours | 28.5 | 15.1 | 16.4 | 7.7 |
Table 6. Ablation study on context-aware semantic learning. “IMG Crop“ and “IMG Crop+Text” are two semantic learning methods derived from image crops.
the global scene context and local object region, achieving effective semantic alignment.
Analysis of selection range for negative class. Table 7 shows the experiments of negative class selection on the open-vocabulary detection on ScanNet60 and ScanNet200 benchmarks. Using the full range [2, 1203] (excludes the top-1 most similar class) leads to sub-optimal performance. This is because the selected negatives include near-duplicative categories semantically too close to the positive class and many trivial easy negatives far away in the embedding space, consequently diluting informative negatives. We then evaluate different middle ranges and observe that selecting negatives from the upper-middle band [10, 150] yields the best performance.
| Range | ScanNet60 | ScanNet200 | ||
| mAP | mAP | mAP | mAP | |
| [2, 1203] | 27.0 | 13.4 | 15.0 | 6.8 |
| [150, 400] | 28.1 | 14.0 | 15.5 | 7.0 |
| [400, 800] | 27.1 | 13.0 | 15.0 | 6.5 |
| Ours | 28.5 | 15.1 | 16.4 | 7.7 |
Table 7. Ablation study on the range for the semantic-aware negative class selection on ScanNet60 and ScanNet200.
Conclusion
In this paper, we present See and Detect, an image-driven framework for open-vocabulary multi-view indoor 3D object detection. Unlike prior point-cloud-driven pipelines, our method treats multi-view images as the primary evidence for both class-agnostic localization and open-vocabulary classification. For localization, we leverage existing single-view 3D proposals and introduce multi-view consensus box consolidation to suppress spurious predictions, together with box quality assessment to down-weight imperfect boxes. For semantic learning, we introduce context-aware semantic learning, where semantic labels are inferred from full images overlaid with projected object boxes, allowing both local object appearance and global scene context to be effectively exploited. In addition, semantic-aware negative class selection is designed to improve contrastive text–3D alignment. We achieve state-of-the-art performance on ScanNet and ARKitScenes.
References
- [1]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. 3D open-vocabulary technical report. arXiv preprint arXiv:2511.21631, 2025. 5, 6
- [2]Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, et al. Articscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data. arXiv preprint arXiv:2111.08897, 2021. 6arxiv.org/abs/2111.08897
- [3]Yang Cao, Zeng Yihan, Hang Xu, and Dan Xu. Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection. Advances in Neural Information Processing Systems, 36:71862–71873, 2023. 1, 3, 6, 7, 8
- [4]Yang Cao, Yihan Zeng, Hang Xu, and Dan Xu. Collaborative novel object discovery and box-guided cross-modal alignment for open-vocabulary 3d object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025. 3DOI
- [5]Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017. 6arxiv.org/abs/1702.04405
- [6]Agrim Gupta, Piotr Dollár, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356–5364, 2019. 6arxiv.org/abs/1908.03195
- [7]Peng-Hao Hsu, Ke Zhang, Fu-En Wang, Tao Tu, Ming-Feng Li, Yu-Lun Liu, Albert YC Chen, Min Sun, and Cheng-Hao Kuo. Openm3d: Open vocabulary multi-view indoor 3d object detection without human annotations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8688–8698, 2025. 1, 2, 3, 6, 7, 8arxiv.org/abs/2508.20063
- [8]Pengkun Jiao, Na Zhao, Jingjing Chen, and Yu-Gang Jiang. Unlocking textual and visual wisdom: Open-vocabulary 3d object detection enhanced by comprehensive guidance from text and image. In European Conference on Computer Vision, pages 376–392. Springer, 2024. 1, 3arxiv.org/abs/2407.05256
- [9]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023. 5arxiv.org/abs/2304.02643
- [10]Yuqing Lan, Chenyang Zhu, Zhirui Gao, Jiazhao Zhang, Yihan Cao, Renjiao Yi, Yijie Wang, and Kai Xu. Boxfusion: Reconstruction-free open-vocabulary 3d object detection via real-time multi-view box fusion. In Computer Graphics Forum, page e70254. Wiley Online Library, 2025. 3arxiv.org/abs/2506.15610
- [11]Justin Lazarow, David Griffiths, Gefen Kohavi, Francisco Crespo, and Afshin Dehghan. Cubify anything: Scaling indoor 3d object detection. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 22225–22233, 2025. 3, 4arxiv.org/abs/2412.04458
- [12]Andrey Lemeshko, Bulat Gabdullin, Nikita Drozdov, Anton Konushin, Danila Rukhovich, and Maksim Kolodiazhnyi. Zo3d: Zero-shot 3d object detection at scene level. arXiv preprint arXiv:2511.20253, 2025. 1, 2, 3, 6, 7, 8arxiv.org/abs/2511.20253
- [13]Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via transformers. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2929–2938. IEEE, 2021. 2arxiv.org/abs/2104.00678
- [14]Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. Open-vocabulary point-cloud object detection without 3d annotation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1190–1199, 2023. 1, 3, 6, 7, 8arxiv.org/abs/2304.00788
- [15]Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021. 2arxiv.org/abs/2003.08934
- [16]Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2906–2917, 2021. 2arxiv.org/abs/2109.08141
- [17]Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9277–9286, 2019. 2arxiv.org/abs/1904.09664
- [18]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021. 3arxiv.org/abs/2103.00020
- [19]Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Imvoxelnet: Image to voxels projection for monocular and multi-view general-purpose 3d object detection. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2397–2406, 2022. 2arxiv.org/abs/2106.01178
- [20]Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Tr3d: Towards real-time indoor 3d object detection. In 2023 IEEE International Conference on Image Processing (ICIP), pages 281–285. IEEE, 2023. 2
- [21]Guanlin Shen, Jingwei Huang, Zhihua Hu, and Bin Wang. Cn-rma: Combined network with ray marching aggregation for 3d indoor object detection from multi-view images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21326–21335, 2024. 2arxiv.org/abs/2403.04198
- [22]Tao Tu, Shun-Po Chuang, Yu-Lun Liu, Cheng Sun, Ke Zhang, Donna Roy, Cheng-Hao Kuo, and Min Sun. Imgeonet: Image-induced geometry-aware voxel representation for multi-view 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6996–7007, 2023. 2, 3arxiv.org/abs/2308.09098
- [23]Zhenyu Wang, Yali Li, Taichi Liu, Hengshuang Zhao, and Shengjin Wang. Ov-uni3detr: Towards unified open-vocabulary 3d object detection via cycle-modality propagation. In European Conference on Computer Vision, pages 73–89. Springer, 2024. 1, 3, 6, 7arxiv.org/abs/2403.19580
- [24]Qian Xie, Yu-Kun Lai, Jing Wu, Zhoutao Wang, Yiming Zhang, Kai Xu, and Jun Wang. Mlvnet: Multi-level context votenet for 3d object detection. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10444–10453. IEEE, 2020. 2arxiv.org/abs/2004.05679
- [25]Chenfeng Xu, Bichen Wu, Ji Hou, Sam Tsai, Ruilong Li, Jialiang Wang, Wei Zhan, Zijian He, Peter Vajda, Kurt Keutzer, et al. Nerf-det: Learning geometry-aware volumetric representation for multi-view 3d object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 23320–23330, 2023. 2arxiv.org/abs/2307.14620
- [26]Yating Xu, Chen Li, and Gim Hee Lee. Mvsdet: Multi-view indoor 3d object detection via efficient plane sweeps. Advances in Neural Information Processing Systems, 37: 132824–132842, 2024.arxiv.org/abs/2410.21566
- [27]Runmin Zhang, Zhu Yu, Si-Yuan Cao, Lingyu Zhu, Guangyi Zhang, Xiaokai Bai, and Hui-Liang Shen. Boosting multi-view indoor 3d object detection via adaptive 3d volume construction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5980–5989, 2025. 2, 4, 5, 6arxiv.org/abs/2507.18331
- [28]Zaiwei Zhang, Bo Sun, Haitao Yang, and Qixing Huang. H3dnet: 3d object detection using hybrid geometric primitives. In European conference on computer vision, pages 311–329. Springer, 2020. 2arxiv.org/abs/2006.05682


