betaFeedback

WorksComputer Science

Annotator-Free Object Detection via Foundation Model Collaboration

Abstract

Object detection research remains constrained by high annotation costs and the closed-set assumptions of supervised models, limiting scalability to novel categories. We present a target-dataset annotation-free object detection framework that eliminates the need for human labeling on the target dataset by leveraging the complementary strengths of pretrained foundation models, including segmentation and vision-language systems, combined with a self-refining training loop. To further reduce reliance on expensive LVLM queries, we introduce a landmark-based clustering mechanism that amortizes labeling across image clusters, improving label consistency while lowering computational cost. Our method autonomously generates segmentation proposals, assigns semantic labels through multimodal reasoning, and iteratively improves both localization and classification using a lightweight open-set detector. This modular pipeline enables adaptation to completely unseen object classes and domains without any human-provided target-dataset annotations, while maintaining compatibility with existing detection architectures. Evaluations on COCO, LVIS, ODinW-35, and Roboflow-100 demonstrate substantial performance gains over state-of-the-art open-vocabulary and long-tail detection methods, achieving 60.8 AP on COCO and 47.4 AP on LVIS under zero-shot settings. Beyond strong benchmark results, our approach provides a scalable foundation for real-world detection systems in unknown environments, where annotation is infeasible.

Introduction and Related Work

Object detection is a fundamental task in computer vision with applications in autonomous driving, industrial automation, and surveillance [3, 28, 41]. State-of-the-art detectors such as YOLO [20] and DETR [2] achieve strong performance but rely heavily on annotated datasets, requiring precise bounding boxes and labels for every instance. This annotation process can take 5–30 minutes per image for computation. plex datasets like COCO [14], making large-scale training prohibitively expensive. Furthermore, detectors trained on such datasets are constrained to predefined categories, limiting their capacity to recognize novel or long-tail objects [13, 24]. This challenge is illustrated in Figure 2.

Traditional object detection and annotator-free object detection pipelines

Figure 2. Traditional object detection requires annotated paired data and remains closed-set. Our annotator-free pipeline relies only on image scenes, generating proposals via foundational models and refining them iteratively to expand to novel categories.

Recent directions attempt to reduce annotation dependency. Novel object detection explores recognizing unseen classes [19, 38]. RNCDL [7] takes a self-supervised approach but cannot assign semantic labels without validation data. Generalized Class Discovery [24] and PromptCAL [35] apply contrastive learning to categorize novel classes, yet they neglect localization [22]. Other works [6] generate outliers to better separate known from unknown classes, but practical deployment remains limited.

Open-vocabulary object detection (OVD) leverages vision-language alignment to expand recognition beyond closed-set constraints. Methods such as GLIP [12], GroundingDINO [15], and YOLO World [4] integrate natural language to detect unseen categories. However, these approaches still depend on closed-set detectors pre-trained on human-annotated data, keeping annotation costs high. While VLMs like CLIP [18] bridge vision and text, their effectiveness relies on supervised training of the detector.

Unsupervised object detection and discovery methods aim to localize and, in some cases, segment objects without any human annotation, typically by exploiting the emergent structure of self-supervised representations. CutLER [26] discovers and segments multiple objects per image using self-supervised features combined with iterative self-training, without relying on any labels. FreeSOLO [25] produces coarse object masks in a fully unsupervised manner and refines them through self-training for instance segmentation. TokenCut [27] and MaskDistill [23] exploit normalized-cut and self-distillation signals from self-supervised vision transformers to discover salient object regions without supervision. Complementary to these purely visual approaches, RegionCLIP [39], OWLv2 [17], and Grounded-SAM [21] pair region-level vision-language pretraining or grounding modules with segmentation models such as SAM [11] to enable open-vocabulary localization. While these methods substantially reduce annotation requirements, they either lack the ability to assign open-ended semantic labels (CutLER, FreeSOLO, TokenCut, MaskDistill) or still depend on some form of pre-existing detection or grounding supervision, or curated region-text pairs (RegionCLIP, OWLv2, Grounded-SAM). Our approach differs by coordinating segmentation, vision-language grounding, and LVLM reasoning entirely on the target dataset at training/inference time, without requiring any target-dataset annotations or a fine-tuned detection backbone.

In this work, we propose a target-dataset annotation-free object detection pipeline that eliminates the need for human annotation on the target dataset by adapting pre-trained foundation models rather than removing human supervision altogether. Our approach integrates large vision-language models [37] and foundational models such as SAM [11], which are themselves trained on human-curated data, to autonomously generate object proposals and semantic labels for the target domain. As shown in Fig. 3, proposals are iteratively refined, enabling the detector to generalize to previously unseen object categories. Unlike OVD, which extends recognition but relies on annotated bases, our method directly adapts to novel environments without prior target-dataset labels. Benchmarks on LVIS and COCO validate that our pipeline reduces annotation costs and also has competitive performance in detecting novel categories.

Our contributions are:

  1. A novel target-dataset annotation-free detection pipeline integrating foundational and vision-language models, removing the need for human annotation on the target dataset.

  2. A self-refining mechanism that progressively enhances detection without labeled data.

  3. A modular design leveraging pre-trained models, enabling generalization to unseen categories.

  4. Extensive experiments showing superior results compared to state-of-the-art on challenging datasets.

Annotator-free object detection

Annotator-free object detection removes the reliance on human-annotated ground truths, enabling detectors to train without prior category knowledge. In this setting, models encounter both known and unknown objects during inference. The goal is to eliminate annotation costs while improving detection of novel and existing classes. Unlike closed-set detectors that yield zero AP on unseen categories, our self-annotator mechanism, built on YOLO with support from LLMs, CLIP, and SAM, achieves gains in both novel

Proposed annotator-free pipeline

caption: Proposed annotator-free pipeline. Stage 1: LVLM and SAM generate scene-level labels and object proposals. Stage 2: CLIP encoders assign semantic labels to proposals. Stage 3: an open-set detector trained on these proposals generates refined prompts for SAM, which are re-labeled by the LVLM.

and known AP. Our approach follows three stages: Self-labeling, where foundation and language models generate bounding boxes and labels without human input, amortized via SAM+CLIP landmark clustering (§ 2.1); Detection, where an open-set detector trained on these labels recognizes both seen and unseen classes; and Refinement, where SAM, guided by the trained detector, improves proposal quality and localization, as illustrated in Fig. 3.

Deconstructing Dataset Preparation

Landmark-based clustering before LVLM queries. To avoid querying an LVLM on every SAM proposal, we amortize labeling across images. For image IiI_i, we run SAM to obtain masks {mik}\{m_{ik}\} and form object crops by masking out the background (black by default). Each crop is encoded with CLIP to produce unit-normalized landmark features zik∈R768z_{ik} \in\mathbb{R}^{768}. We keep up to K=32K=32 largest valid masks per image, yielding the set Zi={zik}k=1KiZ_i = \{z_{ik}\}_{k=1}^{K_i} (Ki≤KK_i \le K).

Neighbor shortlist. To avoid all-pairs, we compute exact cosine kNN on per-image means zˉi=1Ki∑kzik\bar{z}_i = \frac{1}{K_i}\sum_k z_{ik} (then ℓ2\ell_2-normalize), keeping Kshort=75K_{\mathrm{short}}=75 candidate neighbors.

Image-to-image distance. Given two images i,ji,j with landmark sets ZiZ_i and ZjZ_j, we measure dissimilarity via a symmetric Chamfer over cosine distance:

d(i,j)=12(1Ki∑a∈Zimin⁡b∈Zj(1−aTb)+1Kj∑b∈Zjmin⁡a∈Zi(1−aTb)).(1)d(i,j) = \frac{1}{2}\left(\frac{1}{K_i}\sum_{a\in Z_i}\min_{b\in Z_j}(1-\mathbf{a}^{\mathsf T}\mathbf{b})+\frac{1}{K_j}\sum_{b\in Z_j}\min_{a\in Z_i}(1-\mathbf{a}^{\mathsf T}\mathbf{b})\right). \tag*{(1)}

Sparse landmark graph. For an image ii we sort its neighbors by d(i,j)d(i,j) and retain an edge (i,j)(i,j) iff all filters pass:

  1. kNN: jj lies among the top-kk nearest of ii by dd;

  2. Local percentile: d(i,j)≤τid(i,j) \le\tau_i, where τi\tau_i is the pp-th percentile of {d(i,ℓ)}\{d(i,\ell)\} over the shortlist of ii;

  3. Global cutoff: d(i,j)≤τgd(i,j) \le\tau_g, the global percentile (e.g., median) over all shortlisted distances (optionally tightened via a shared-neighborhood constraint ∣NN(i)∩NN(j)∣≥s|\mathrm{NN}(i) \cap\mathrm{NN}(j)| \ge s, where NN(⋅)\mathrm{NN}(\cdot) are the Stage-B kNN lists).

We symmetrize with a mutual requirement (keep (i,j)(i,j) only if both ii keeps jj and jj keeps ii). Connected components of this sparse graph define image clusters.

Medoids and label propagation. For each component CC we select a medoid image by maximizing degree and, as a tie-break, minimizing the sum of in-cluster distances:

i∗=arg⁡max⁡i∈C(deg⁡(i),−∑j∈C, j≠id(i,j)).i^* = \underset{i\in C}{\arg\max}\left(\deg(i),-\sum_{j\in C,\ j\ne i}d(i,j)\right).

We query the LVLM only on medoids to obtain scene/object labels and propagate those labels to all images within the same component, reducing LVLM usage from O(∑iKi)O(\sum_i K_i) per-dataset proposals to O(C)O(C) queries while improving intra-cluster label consistency. We optionally repeat this clustering-and-labeling procedure across iterations to improve coverage: after each round, we (i) prune noisy SAM masks and re-compute landmarks for uncertain images using the newly assigned labels, and (ii) optionally update the detector (YOLO) on the current pseudo-labels, before rebuilding the shortlist and graph.

Producing Object Labels. The initial step in annotator-free object detection is autonomously generating semantic labels for detected objects, removing reliance on manual annotation. We leverage large vision-language models (VLMs) such as GPT-4o and CogVLM, which act as general-purpose observers, autonomously recognizing and describing objects present in a scene without any prior exposure or explicit definition of existing classes. Given an input image region xrx_r, the visual encoder Fv(VLM)F_v^{(\mathrm{VLM})} and multimodal fusion module Gf(VLM)G_f^{(\mathrm{VLM})} directly produce text outputs:

T=Gf(VLM)(Fv(VLM)(xr)),\mathrm{T}=G_f^{(\mathrm{VLM})}\left(F_v^{(\mathrm{VLM})}\left(x_r\right)\right),

where T represents descriptive object labels or captions reflecting objects the VLM inherently recognizes, without predefined label sets – enabling flexible labeling and novel category discovery.

Producing Bounding Box Proposals. The second step autonomously generates bounding box proposals to localize objects within images, using a segmentation-based proposal generation module (FsegF_{\mathrm{seg}}) – specifically SAM [11] – that produces segmentation masks M={mi}i=1N\mathcal{M}=\{m_i\}_{i=1}^{N}, from which bounding boxes B={bi}i=1N\mathcal{B}=\{b_i\}_{i=1}^{N} are derived. Because SAM is trained on vast and diverse data, it generalizes across classes and domains without fine-tuning, yielding precise, reliable bounding-box proposals even in completely unknown environments.

Assigning Labels to Bounding Box Proposals. After independently generating semantic labels and bounding box proposals, the next critical step involves matching these semantic labels to the bounding box proposals. Since segmentation masks lack inherent semantic labels, this matching step requires aligning visual embeddings of bounding boxes with textual embeddings of semantic labels. A suitable model for this matching step must operate within a unified embedding space for visual and textual data. Given visual embeddings for each bounding box proposal and textual embeddings for candidates, the matching’s given by:

yi=arg⁡max⁡c∈C sim⁡(ϕb(bi),ϕl(c)).y_i=\underset{c\in C}{\arg\max}\ \operatorname{sim}\left(\phi_b(b_i),\phi_l(c)\right).

This matching strategy enables effective assignment of labels to proposals, balancing accuracy and inference. Here bib_i is the ii-th proposal; CC is the candidate label set including background; ϕb(⋅)∈Rd\phi_b(\cdot)\in\mathbb{R}^d is the image encoder producing a unit-normalized embedding; ϕℓ(⋅)∈Rd\phi_\ell(\cdot)\in\mathbb{R}^d is the text encoder applied to a prompt producing a unit-normalized embedding; sim⁡(u,v)=uTv\operatorname{sim}(\mathbf{u},\mathbf{v})=\mathbf{u}^{\mathsf{T}}\mathbf{v} denotes cosine similarity for unit vectors; and yiy_i is the assigned label for proposal bib_i.

Self-labeling for proposal generation

The self-labeling stage involves generating initial bounding boxes as illustrated in Fig. 3 and object labels for YOLO World. This stage combines several foundational models to complement each other’s abilities. SAM produces segmentation masks for a given scene, aiding in Region of Interest (ROI) or proposal generation. Large language models (LLMs) or large vision-language models (LVLMs) are capable of identifying objects present in the scene, helping to construct scene labels. CLIP provides strong zero-shot classification capabilities to ground object proposals from SAM with scene labels derived from LLMs.

SAM is a foundational segmentation model that can produce high-precision segmentation masks for objects in a scene. However, SAM requires input prompts to segment specific objects from the scene. Since there is no way to prompt SAM without a supervised object detector, SAM can be configured to produce all possible segmentation masks for the scene. This approach, however, results in unnecessary segmentation masks, including backgrounds, redundant masks, fine-grained parts, and noisy segmentations. To enable annotator-free object detection, these unnecessary segmentations should be removed to yield appropriate object proposals for detector training.

Large vision-language models are trained on vast datasets and have a large capacity for parameterized knowledge. This capacity enables LVLMs to perform zero-shot inference for objects present in a scene without prior exposure to specific training or testing distributions. However, due to their high capacity, LVLMs have significant inference times, which leads to high computational costs when filtering out unnecessary object proposals from SAM, making the pipeline impractical for real-time applications.

Since LVLMs have slow inference times for object labeling, we employ CLIP for pre-filtering. CLIP, a vision-language model, is smaller in size compared to both SAM and LVLMs. Given the input image proposals and text proposals, CLIP assigns labels from a predefined label set significantly faster than any LVLM. Additionally, due to CLIP’s architecture, it produces confidence scores for each object proposal’s labels, enabling benefits such as Non-Maximum Suppression for redundancy removal.

Implementing the detector

In the scene-level label generation, we begin by using the text prompt “Generate the list of general labels of the objects present in the scene.” However, this approach does not guarantee that the object labels generated for instances of the same object class will be consistent. For example, “television” and “tv” refer to the same object category but are linguistically different labels. This issue creates a run-off problem for object classes and leads to label redundancy. To mitigate this, we construct a dictionary to update the learned object categories. The dictionary is initialized without any prior knowledge of object classes, and it is updated as new categories are learned. Furthermore, for each object label produced by the LVLM, we search for similar labels or synonyms in the dictionary using the instruction “Replace the labels in the {label list} with similar labels present in the {dictionary}” to remove redundant labels. Because LVLM APIs (e.g., GPT-4o) are not guaranteed to produce deterministic outputs across repeated calls or successive passes over the dataset, we freeze the label dictionary D\mathcal{D} once it converges (i.e., no new categories are added over a full pass through the dataset) and use this fixed dictionary for all subsequent proposal generation and evaluation. This prevents label-space drift caused by API non-determinism from affecting the reported results.

In the object-level label generation, we use text embeddings from the CLIP text encoder for scene-level labels and additionally append the class tag “background” to filter out unnecessary segmentations. To assign a label, we consider each segmentation instance: the segmented area is cropped using bounding boxes from SAM’s postprocessing. Then, the cropped image is multiplied by the binary segmentation mask to capture the region of interest (ROI) and exclude unnecessary parts from the cropped image. This step helps prevent background segmentation masks from being mislabeled as object labels, as the bounding box corresponding to the segmentation mask might include objects from other categories. Besides preventing background mislabeling, this also reduces cross-labeling of objects.

Once the CLIP labels are obtained, objects labeled as “background” are removed from the object proposals. To further reduce redundancy among object proposals, we apply Non-Maximum Suppression using CLIP scores as the confidence values. After generating the final object proposals for the dataset, we train YOLO World to develop a supervised object detector capable of performing OWD.

Proposal refinement

Once the open-world object detector is trained, we use the scene-level labels to perform inference. However, this inference stage operates under a closed-set setting, meaning we only consider the categories included in the dictionary developed during the proposal generation phase. Next, we use the bounding boxes from YOLO World as box prompts for SAM to refine segmentations and eliminate unnecessary segmentations that were generated during the self-labeling process. The supervised detector significantly reduces the number of proposals generated for a scene while also improving the quality of SAM’s segmentation masks.

Due to the substantial reduction in object proposals, we employ an LVLM to generate refined labels for these proposals during the refinement stage. This step leverages the classification head of YOLO World, which uses the CLIP text encoder, making the classification head functionally equivalent to object-level label generation, as seen in the self-labeling stage. In this process, the LVLM acts as a validator for the labels generated by CLIP, yielding more accurate labels for object proposals. To ensure coherence between object-level labels and scene-level labels within the dictionary, we prompt the LVLM with the instruction, “Pick a label from the {label list} for the object,” which helps assign a consistent label. After refining the object proposals, we retrain the open-set object detector with the improved proposals, further enhancing detection performance. Algorithm 1 summarizes the complete self-labeling and training process.

Algorithm 1 Self-Labeling and Training Process

  1. Input: Image x\mathbf{x}

  2. Initialize: Label dictionary D\mathcal{D}

  3. Produce scene-level labels Lscene\mathcal{L}_{\mathrm{scene}} for the image x\mathbf{x} using LVLM

  4. Realign labels Lscene\mathcal{L}_{\mathrm{scene}} with the existing labels in the dictionary D\mathcal{D} and update D\mathcal{D} for new labels

  5. Generate segmentation masks M\mathcal{M} for the image x\mathbf{x} using SAM

  6. Classify the object proposals in M\mathcal{M} using CLIP with scene-level labels Lscene\mathcal{L}_{\mathrm{scene}}

  7. Train YOLO World using the proposals generated for the dataset

  8. Produce box prompts bprompt\mathbf{b}_{\mathrm{prompt}} using the trained detector for SAM

  9. Label the object proposals P\mathcal{P} using LVLM to improve self-labeling quality

  10. Retrain detector with the improved object proposals P\mathcal{P}

Novel object detection

Process for Detecting Novel Object Classes. To detect novel object classes, we follow a process similar to the refinement stage. First, we generate scene-level labels for the scene and provide these labels to the open-set detector. The detector then produces regions of interest (ROIs), labels, and confidence scores for each detected object in the scene. From the scene-level labels, we separate the known and novel categories present in the scene. This allows us to focus on distinguishing high-confidence known categories from potential novel classes. For known categories, CLIP confidence scores tend to be higher for familiar class labels than for novel ones (reflecting CLIP’s pretraining bias toward frequently-seen concepts, which yields tighter image-text alignment), aiding in differentiating known objects from unknown ones. For object proposals with low confidence scores, we extract the corresponding bounding boxes and follow the procedure from the refinement stage to label these as novel classes. This approach allows for accurate labeling of novel objects within the scene.

Experiments and Results

In § 3.1, we outline the implementation settings for our study. We then present our main findings in § 3.2, where we compare our approach to leading models, GDINO [15] and T-Rex2 [10]. Lastly, in § 3.4, we provide extensive ablation studies that highlight the practical advantages of our proposed approach.

Implementation Details

Datasets: We evaluate on five benchmarks: COCO [14], LVIS [9], ODinW-35 [12], Roboflow-100 [5], and a COCO Open-Vocabulary Detection (OVD) split. COCO (2017 val) covers 80 classes, while the OVD split partitions 48 known and 17 novel classes. LVIS provides 1,203 categories for large-vocabulary evaluation. ODinW-35 and Roboflow-100 further test adaptability across varied domains, using their full validation splits without additional training data.

LVLMs: We employ two models: (i) GPT-4o, via API, for scene-level labeling, and (ii) CogVLM, open-source, for object-level labeling and for realigning scene labels with the label dictionary.

YOLO World: Built on MMYOLO/MMDetection, trained with AdamW (lr=0.002, wd=0.05) for 100 epochs on 8 NVIDIA A5000 GPUs (batch size 64). Following prior work, we apply color, affine, flip, and mosaic augmentations

during pre-training.

Evaluation protocol: Unless otherwise noted, all quantitative results reported in Table 1 to Table 4 use the standard AP@[.50:.95]AP@[.50:.95] metric under strict, exact category-name matching against each benchmark’s ground-truth taxonomy; a prediction whose label does not exactly match a ground-truth category name is counted as a false positive even if it names a synonymous concept (e.g., “television” vs. “tv”). This differs from the qualitative teaser in Figure 1, whose caption explicitly credits a detection as correct under IoU ≥0.5\ge0.5 with a synonym-mapped class label.

Qualitative results on COCO 2017 val. Ground-truth annotations and predictions from the label-free detector.

Figure 1. Qualitative results on COCO 2017 val. Top row: ground-truth annotations. Bottom row: predictions from our label-free detector, obtained by coordinating foundation models without any detection-specific supervision. The proposed method (1) detects most object classes present in the ground truth, even when expressed with synonymous labels (e.g., television for tv); (2) recovers objects that are missing annotations; and (3) proposes novel classes with no close counterpart in the dataset taxonomy (e.g., dog bed, radiator, pen, cabinet). For evaluation, a detection is considered correct if it has IoU ≥0.5\ge0.5 with a ground-truth box and a matching class (accounting for synonym mapping).

ModelBackboneCOCO valLVIS
GLIP-L [12]Swin-L49.826.9
DINOv [34]Swin-L46.2-
GDINO-L [15]Swin-L48.427.4
T-Rex2 (Text) [10]Swin-L52.245.8
T-Rex2 (Visual) [10]Swin-L46.545.3
DetCLIPv3 [31]Swin-T47.247.0
DetCLIPv3 [31]Swin-L48.548.8
OursSwin-T57.246.2
OursYOLO60.847.4

Table 1. Annotator-free object detection performance comparison on COCO val and LVIS minival. We consider the zero-shot evaluation of state-of-the-art long tail object detectors and open vocabulary object detectors for comparison.

MethodBackboneNovelKnownOverall
GLIP-L [12]Swin-L35.112.624.9
T-Rex2 [10]Swin-L31.921.227.1
GDINO-L [15]Swin-L38.426.533.0
OursYOLO43.732.838.8

Table 4. Comparison for novel object detection under domain shift with other state-of-the-art methods. Performance is evaluated on novel, known, and overall classes.

Annotator-free object detection

The performance on COCO val2017 and LVIS minival datasets is compared with state-of-the-art object detectors in Table 1. For a fair comparison, we evaluate all methods under a zero-shot setting, ensuring no prior access to the test set’s ground truth data. Our proposed method achieves a substantial improvement, with a 16.5% gain on COCO compared to long-tail detectors such as T-Rex2 and a 25.6% improvement over open-vocabulary detectors like GDINO. On LVIS, our method further surpasses long-tail detectors by 5% and open-vocabulary detectors by 72%. The label distributions visualized in Figure 4 further confirm that our pseudo-labeling scheme preserves COCO’s class frequencies while introducing new categories. Qualitative examples are further illustrated in Figure 6. To verify that these gains are not an artifact of the YOLO backbone, we additionally trained our pipeline using a Swin-T backbone [16], achieving 57.2 AP on COCO and 46.2 AP on LVIS (Table 1), confirming that our pseudo-labeling pipeline provides substantial gains regardless of the detector architecture.

Word clouds of COCO ground-truth instances and pseudo-labeled annotations

Figure 4. Word clouds of COCO ground-truth instances (top) and our pseudo-labeled annotations (bottom). The proposed annotation scheme preserves relative class frequencies (e.g., person, car, table) while introducing additional categories (e.g., fence, building, window), thereby enriching label diversity. This demonstrates that our method not only captures the underlying distribution but also scales to novel concepts beyond the original dataset.

Qualitative results for the detection performance of the proposed annotator-free method

Figure 6. Qualitative results for the detection performance of the proposed annotator-free method. The predicted detection results are on the left column, and the ground truth results are on the right.

The performance of our method on ODinW-35 val and Roboflow-100 val datasets is presented in Table 2. These datasets are designed to evaluate in-the-wild object detection performance. Our proposed method achieves a 17% improvement over long-tail object detectors and a

ModelBackboneODinWRoboflow
GLIP-L [12]Swin-L23.48.6
DINOv [34]Swin-L15.7-
GDINO-L [15]Swin-L22.38.3
T-Rex2 (Text) [10]Swin-L22.010.5
T-Rex2 (Visual) [10]Swin-L27.818.5
OursSwin-T30.120.8
OursYOLO32.523.7

Table 2. Annotator-free object detection performance comparison on ODinW-35 val and Roboflow-100 val. We consider the zero-shot evaluation of state-of-the-art long tail object detectors and open vocabulary object detectors for comparison.

@references♀♀♀♀♀♀

45% improvement over open-vocabulary object detectors on ODinW. On Roboflow, we observe even greater gains, with improvements of 28% and 86% for long-tail and open-vocabulary detection schemes, respectively. The ability to perform object detection in the wild without human annotations highlights the value of our pipeline, as it enables the development of automatic target recognition systems for unknown environments and niche object categories. The same Swin-T backbone described above achieves 30.1 AP on ODinW and 20.8 AP on Roboflow (Table 2), consistent with the backbone-parity trend observed on COCO and LVIS.

Novel object detection

To evaluate the performance of our method on the COCO Open-Vocabulary Detection (OVD) split, we trained on 48 base COCO classes and tested it on 17 novel classes. To ensure a fair comparison with other methods, we trained the detector on the base classes using ground truth data for object detection, consistent with the approach taken by previous methods. Unlike existing OVD methods [15], our approach does not rely on any additional training datasets.

Despite this, our method outperforms state-of-the-art approaches by a significant margin, as shown in Table 3. Specifically, we observe a 1.2-point improvement in AP50AP_{50} on novel classes (55.9 vs. 54.7) over the strongest competing method, DetCLIPv3 [31], underscoring the effectiveness of our approach relative to prior OVD methods. The Swin-T backbone variant introduced in § 3.2 shows a similar trend on this split, achieving 56.3/62.7/58.3 AP50AP_{50} on novel/known/all classes, respectively (Table 3).

MethodBackboneNovelKnownAll
OVR-CNN [33]RN5022.846.039.9
ViLD [8]ViT-B/3227.659.551.3
Detic [40]RN5027.847.145.0
OV-DETR [32]ViT-B/3229.461.052.7
BARON [29]RN5034.060.453.5
CORA [30]RN50x441.744.543.8
CoFoMo [1]ResNet10150.349.849.9
SAS-Det [36]ResNet10137.458.553.0
DetCLIPv3 [31]Swin-T54.742.846.9
OursSwin-T56.362.758.3
OursYOLO55.964.659.1

Table 3. Comparison for novel object detection with other state-of-the-art methods. Performance is evaluated on a novel, known, and all classes on COCO OVD split following a generalized open-vocabulary detection setting.

Additionally, we analyzed the pipeline’s performance for novel classes using an object detector trained with our annotator-free detection approach. For the base classes, this detector achieved an AP50AP_{50} of 48.9, while for novel classes, we observed an AP50AP_{50} of 55.9. Unlike the results in Table 3, where the base-class detector is trained using ground-truth annotations for fair comparison with prior OVD methods, the detector analyzed here is trained entirely through our annotator-free pipeline, without any ground-truth base-class labels; this is why its base-class AP50AP_{50} (48.9) differs from, and is not directly comparable to, the ground-truth-trained variant reported in Table 3. These results nonetheless demonstrate that with a sufficiently trained object detector, the proposed pipeline can effectively perform open-vocabulary detection without any ground-truth supervision at all.

Ablation study

Performance gain at different stages: The object detection performance at different stages of the proposed method is shown in Figure 5. Results are provided for four datasets under an annotator-free detection scheme. For the ablation study, we evaluate the original object proposals generated by SAM, followed by proposals filtered using Non-Maximum Suppression (NMS) with CLIP scores. Next, we report detection performance from the trained object detector using the self-labeled dataset, and finally, we present the performance of the complete pipeline after refining the object proposal generation process.

Ablation of performance improvement across pipeline stages

Figure 5. Ablation of performance improvement across pipeline stages (proposal generation, inference, refinement).

As seen in Figure 5, the largest improvement is achieved at the original object proposal stage, demonstrating the effectiveness of foundation models in generating proposals for annotator-free object detection. Notably, this stage operates independently of any standard object detector and requires no hyperparameter tuning, making it a fully training-free process. Despite its simplicity, this stage yields performance on par with zero-shot evaluations and surpasses open-vocabulary detectors like GroundingDINO.

The performance gain observed in the NMS filtering stage is due to the removal of overlapping proposals generated in the prior stage. Subsequently, the detection performance after primary training on the object detector shows an incremental boost across all four datasets. This improvement is attributed to enhanced bounding box detection accuracy, indicating the generalization benefits of a trained detector over other training-free methods.

Robustness under domain shift: To test generalization beyond our four main benchmarks, we evaluate on a synthetic domain-shift dataset containing both known and novel categories unseen during training (Applied Intuition simulator; full setup and per-class results in the supplementary material, Table 4). As shown in Figure 7, GroundingDINO can localize novel objects but often assigns incorrect classes, whereas our approach produces more accurate and consistent recognition. Our pipeline achieves 43.7/32.8/38.8 AP50_{50} on novel/known/overall classes, outperforming GLIP-L, T-Rex2, and GDINO-L.

Open-world detectors for novel object categories under domain shift

Figure 7. Open-world detectors for novel object categories under domain shift. The detection performance for GroundingDINO (left column) are compared with the proposed novel object detection (right column).

Computational cost: Proposal generation is the main overhead, with LVLM queries costing roughly one second per scene. We avoid labeling all SAM segmentations directly, using CLIP as a pre-filter to cut labeling cost by 60×\times. Our landmark-based clustering (§ 2.1) queries the LVLM only on cluster medoids and propagates labels across the cluster, dropping per-image labeling time from 15–20 minutes to ∼\sim45 seconds (∼\sim20–27×\times).

Limitations

Despite strong results, our pipeline has notable limitations (Figure 6 and Figure 7) such as: LVLM mislabeling of ambiguous or occluded objects propagates into the pseudo-labels used for training. Localization quality is bounded by SAM’s segmentation, which degrades under low contrast, clutter, or fine-grained/overlapping parts. Labeling is also sensitive to LVLM prompt phrasing, only partially mitigated by the label dictionary (§ 2.1). Finally, LVLM queries remain the dominant computational cost; our medoid-based clustering substantially reduces this (15–20 min to ∼\sim45 s per image), but the label-quality trade-off remains uncharacterized.

Conclusion

We introduced a novel annotator-free object detection framework that leverages large language and foundational models to eliminate human labeling. By automatically generating proposals and refining them iteratively, our modular pipeline integrates seamlessly with existing open-set detectors, boosting performance on both known and novel classes. Beyond strong results in zero-shot and open-vocabulary settings, our approach demonstrates robustness under domain shifts, outperforming state-of-the-art methods such as GroundingDINO. These findings highlight the scalability and adaptability of our method, making it a practical solution for real-world detection in unstructured environments with minimal human intervention.

Additional Experimental Details

In our proposed work, we demonstrated the performance of the proposed pipeline on novel object detection under domain shift by compiling a dataset of simulated images using a graphic simulator.

In this experiment, we consider the detection performance of the proposed pipeline when changes are introduced to the environment and sensor. To simulate the change in environment, we use an outdoor environment in the simulator. The sensor changes are introduced by the color and texture gap between real and synthetic images. We refer to this cumulative gap between the real images and simulated images as the domain shift.

To study the performance under domain shift, we need to introduce the domain gap during inference from the proposed pipeline. For that, we train the object detector on the COCO dataset as explained in the main paper for the annotator-free object detection task. Then, we use this pre-trained object detector to perform inference on the simulated dataset.

The simulated dataset was constructed using the Applied Intuition simulator. Here, we used five known and novel classes. The known classes: car, airplane, bus, truck, and bicycle, already exist in the COCO categories. The novel classes: tank, tractor, helicopter, scooter, and excavator do not appear in the training distribution. Below we provide some sample images from the simulated dataset.

Sample images from the simulated dataset

Figure 8. Sample images from the simulated dataset.

Quantitative comparisons in Table 4 report results separately for known and novel categories, similar to open-vocabulary evaluations. Our pipeline achieves a clear advantage on novel classes while maintaining strong performance on known categories, driven by the LLM-based classification branch, which supplies a broader and more adaptive vocabulary than conventional open-vocabulary or open-set detectors. Note that relative rankings among baselines on this synthetic benchmark can differ from the main paper’s COCO/LVIS/ODinW/Roboflow results: for instance, GLIP-L outperforms T-Rex2 on novel classes here despite the opposite trend there, because this dataset’s scene distribution and novel-category composition (e.g., tanks, tractors) differ substantially from those benchmarks.

Annotation-Free Detection Pipeline: Clarifications

Pipeline Design and Novelty

Unlike DetCLIPv3, RegionCLIP, VLDet, or T-Rex2, which all depend on labeled datasets or pre-trained supervised components, our pipeline is target-dataset annotation-free. It is built upon:

  • SAM for class-agnostic proposal generation,

  • CLIP and LVLMs for label assignment,

  • Self-refinement via re-prompting with trained detectors.

This design enables detector training without any target-dataset annotations, using only unannotated images. Our modular approach integrates pseudo-labeling, open-world reasoning, and label refinement into a single end-to-end framework.

Handling Noisy SAM Proposals

SAM’s automatic mode may produce over- or under-segmented regions. We address this via:

  1. CLIP-based filtering (threshold = 0.5),

  2. NMS filtering (IoU threshold = 0.6),

  3. Self-refinement with learned detector.

Noise reduction steps for SAM proposals

Figure 9. Noise reduction steps for SAM proposals.

Efficiency and Foundational Model Usage

Foundational models (CLIP, GPT-4o, CogVLM) are only used during offline proposal labeling. The final deployed detector is a real-time YOLO-based model.

  • CLIP filtering occurs early to reduce load.

  • GPT-4o is used only once to generate the label dictionary.

  • LVLMs are used in refinement (Step 9), limited to fewer high-confidence regions.

Average LVLM runtime is 1–2s per image, occurring entirely offline.

Rare Class Generalization

We evaluated the model on the LVIS rare class subset. The results were:

  • APall=47.4\mathrm{AP}_{\mathrm{all}} = 47.4

  • APr=46.1\mathrm{AP}_{\mathrm{r}} = 46.1

This small drop suggests better robustness to class imbalance compared to models like T-Rex2.

Scalability of Label Dictionary

While the label dictionary theoretically incurs O(n2)O(n^2) complexity due to transformer attention, we use:

  • Hashmap-based caching of label outputs,

  • Amortized O(1)O(1) lookups for repeated scenes.

This allows the system to scale efficiently even with large vocabularies.

Parameter Summary

  • CLIP Score Threshold (Step 6): 0.5

  • NMS IoU Threshold: 0.6

  • LVLMs: Used for semantic refinement, no confidence scores

  • Detector: YOLO World with default configs

References

  1. [1]Rohit Bharadwaj, Muzammal Naseer, Salman Khan, and Fahad Shahbaz Khan. Enhancing novel object detec- tion via cooperative foundational models. arXiv preprint arXiv:2311.12068, 2023. 7
  2. [2]Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213–229. Springer, 2020. 1arxiv.org/abs/2005.12872
  3. [3]Chenyi Chen, Ari Seff, Alain Kornhauser, and Jianxiong Xiao. Deepdriving: Learning affordance for direct perception in autonomous driving. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 2722–2730, 2015. 1arxiv.org/abs/1505.00256
  4. [4]Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16901–16911, 2024. 2arxiv.org/abs/2401.17270
  5. [5]Floriana Ciaglia, Francesco Saverio Zuppichini, Paul Guerrie, Mark McQuade, and Jacob Solawetz. Roboflow 100: A rich, multi-domain object detection benchmark. ArXiv, abs/2211.13523, 2022. 6arxiv.org/abs/2211.13523
  6. [6]Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don’t know by virtual outlier synthesis, 2022. 2arxiv.org/abs/2202.01197
  7. [7]Vladimir Fomenko, Ismail Elezi, Deva Ramanan, Laura Leal-Taix’e, and Aljoša Ošep. Learning to discover and detect objects. In Advances in Neural Information Processing Systems, 2022. 2arxiv.org/abs/2210.10774
  8. [8]Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation, 2022. 7arxiv.org/abs/2104.13921
  9. [9]Agrim Gupta, Piotr Dollár, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation, 2019. 6arxiv.org/abs/1908.03195
  10. [10]Qing Jiang, Feng Li, Zhaoyang Zeng, Tianhe Ren, Shilong Liu, and Lei Zhang. T-rex2: Towards generic object detection via text-visual prompt synergy. In European Conference on Computer Vision, pages 38–57. Springer, 2025. 6, 7, 1arxiv.org/abs/2403.14610
  11. [11]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023. 2, 4arxiv.org/abs/2304.02643
  12. [12]Liunian Harold Li*, Pengchuan Zhang*, Haotian Zhang*, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In CVPR, 2022. 2, 6, 7, 1arxiv.org/abs/2112.03857
  13. [13]Yu Li, Tao Wang, Bingyi Kang, Sheng Tang, Chunfeng Wang, Jintao Li, and Jiashi Feng. Overcoming classifier imbalance for long-tail object detection with balanced group softmax. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10991–11000, 2020. 2arxiv.org/abs/2006.10408
  14. [14]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014. 2, 6arxiv.org/abs/1405.0312
  15. [15]Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pages 38–55. Springer, 2024. 2, 5, 6, 7, 1arxiv.org/abs/2303.05499
  16. [16]Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF international conference on computer vision (ICCV), pages 9992–10002. Ieee, 2021. 6arxiv.org/abs/2103.14030
  17. [17]Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. In Advances in Neural Information Processing Systems, 2023. 2arxiv.org/abs/2306.09683
  18. [18]Alec Radford, Jong Wook Kim, Chihallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 8748–8763. PMLR, 2021. 2arxiv.org/abs/2103.00020
  19. [19]Shafin Rahman, Salman Khan, and Fatih Porikli. Zero-shot object detection: Learning to simultaneously recognize and localize novel concepts. In Asian Conference on Computer Vision, pages 547–563. Springer, 2018. 2arxiv.org/abs/1803.06049
  20. [20]J Redmon. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016. 1arxiv.org/abs/1506.02640
  21. [21]Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159, 2024. 2arxiv.org/abs/2401.14159
  22. [22]Mamshad Nayeem Rizve, Navid Kardan, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. OpenIndl: Learning to discover novel classes for open-world semi-supervised learning. In European Conference on Computer Vision, pages 382–401. Springer, 2022. 2arxiv.org/abs/2207.02261
  23. [23]Wouter Van Gansbeke, Simon Vandenhende, and Luc Van Gool. Discovering object masks with transformers for unsupervised semantic segmentation. arXiv preprint arXiv:2206.06363, 2022. 2arxiv.org/abs/2206.06363
  24. [24]Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7492–7501, 2022. 2arxiv.org/abs/2201.02609
  25. [25]Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M. Alvarez. Freesolo: Learning to segment objects without annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14176–14186, 2022. 2arxiv.org/abs/2202.12181
  26. [26]Xudong Wang, Rohit Giridhar, Stella X. Yu, and Ishan Misra. Cutler: Cut and learn for unsupervised object detection and instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4313–4314, 2023. 2
  27. [27]Yangtao Wang, Xi Shen, Shell Xu Hu, Yuan Yuan, James L. Crowley, and Dominique Vaufreydaz. Self-supervised transformers for unsupervised object discovery using normalized cut. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14543–14553, 2022. 2arxiv.org/abs/2202.11539
  28. [28]Yingjie Wang, Qiuyu Mao, Hanqi Zhu, Jiajun Deng, Yu Zhang, Jianmin Ji, Houqing Li, and Yanyong Zhang. Multi-modal 3d object detection in autonomous driving: a survey. International Journal of Computer Vision, pages 1–31, 2023. 1arxiv.org/abs/2106.12735
  29. [29]Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of regions for open-vocabulary object detection, 2023. 7arxiv.org/abs/2302.13996
  30. [30]Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching, 2023. 7arxiv.org/abs/2303.13076
  31. [31]Lewei Yao, Renjie Pi, Jianhua Han, Xiaodan Liang, Hang Xu, Wei Zhang, Zhenguo Li, and Dan Xu. Detclipv3: Towards versatile generative open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27391–27401, 2024. 6, 7
  32. [32]Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary DETR with conditional matching. In Lecture Notes in Computer Science, pages 106–122. Springer Nature Switzerland, 2022. 7DOI
  33. [33]Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions, 2021. 7arxiv.org/abs/2011.10678
  34. [34]Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection, 2022. 6, 7
  35. [35]Sheng Zhang, Salman Khan, Zhiqiang Shen, Muzammal Naseer, Guangyi Chen, and Fahad Shahbaz Khan. Prompt-cal: Contrastive affinity learning via auxiliary prompts for generalized novel category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3479–3488, 2023. 2arxiv.org/abs/2212.05590
  36. [36]Shiyu Zhao, Samuel Schulter, Long Zhao, Zhixing Zhang, Yumin Suh, Manmohan Chandraker, Dimitris N Metaxas, et al. Taming self-training for open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13938–13947, 2024. 7arxiv.org/abs/2308.06412
  37. [37]Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023. 2DOI
  38. [38]Zhong-Qiu Zhao, Peng Zheng, Shou-tao Xu, and Xindong Wu. Object detection with deep learning: A review. IEEE transactions on neural networks and learning systems, 30 (11):3212–3232, 2019. 2DOI
  39. [39]Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuang Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793–16803, 2022. 2arxiv.org/abs/2112.09106
  40. [40]Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision, 2022. 7arxiv.org/abs/2201.02605
  41. [41]Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye. Object detection in 20 years: A survey. Proceedings of the IEEE, 111(3):257–276, 2023. 1

Paper details

vlnc:W_nydst8f7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents