Visual-Prompt Guided Wildlife Instance-Level Recognition

Introduction

Generic object detection models categorize wildlife at the species level. Conversely, instance-level wildlife re-identification requires the discrimination of fine-grained visual patterns. The model is tasked with learning from the unique stripe configurations of a zebra or the spot patterns of a cheetah. Current state-of-the-art approaches deploy an isolated detection and re-identification pipeline using two separate models [8]. An object detection model isolates individuals in a scene, crops the bounding box pixels, resizes them to a fixed resolution, and passes them to an independent re-identification feature extractor for similarity matching against a database [2]. Recent advancements in promptable vision foundation models and visual grounding present a compelling alternative [5]. We could instead view the database as a prompt to query the scene directly to locate a target. However, adapting single-stage visual grounding architectures for instance-level re-identification introduces a training mismatch. A single transformer decoder layer is forced to simultaneously learn the global scenery to extract geometric boundaries of an object and the local texture patterns of a specific identity. To resolve this, we propose an end-to-end visual prompting task.

A frontier foundation model extracts robust spatial features and a specialist re-identification model extracts fine-grained features. Preliminary findings depict a competitive mean average precision score (mAP) over a two-stage approach. Qualitative results also depict tight bounding boxes around target identities in dense herd, occluded environments and distant subjects.

Methodology

The proposed framework in Figure 1 consists of three primary components. First, we extract scene and re-identification features from a frozen encoder. Next, we enhance the object queries with prompt re-identification features to enrich our region proposals. Lastly, we use a cross-attention detector to search for the target identity.

Multi-Scale Scene Encoder We use DINOv2 [6] to process the full scene image. To handle extreme scale variations in imagery, we adopt a similar strategy used by Grounding DINO [5]. We extract intermediate feature maps from layers 3, 6 and 10 within the transformer. These features are projected into a feature pyramid using varying convolutional strides, spatially interpolated to a uniform resolution, concatenated, and subsequently compressed via a $1 \times1$ convolution.

Visual prompt The visual prompt is obtained through a pretrained fine-grained wildlife re-identification model, MegaDescriptor [2].

Feature Enhancer We initialize a set of learnable object queries as the region proposals. The feature enhancement layer ensures that the queries cross-attend directly with the visual prompt features. We use both self and cross attention to enrich our region proposals (queries) with identity-related information from our visual prompts.

Cross-Attention Decoder The enriched region proposals attend to the multi-scale scene features via Deformable Attention [9]. The decoder outputs a set of $N$ cross-attended queries. The cross-attended queries are passed through a feed-forward network to encapsulate the detected animal with a bounding box. An additional objectiveness score is also retrieved, indicating the presence of a valid foreground animal.

Loss Function This work proposes the optimization of four objectives, such that:

is the detection loss. $\mathcal{L}_{FL}$ is the Sigmoid Focal Loss over the objectness score obtained from the $N$ queries. $\mathcal{L}_{L1}$ and $\mathcal{L}_{GIoU}$ is the L1 loss and Generalized Intersection over Union of the predicted and ground truth bounding boxes. $\mathcal{L}_{div}$ encourages diversity of the embeddings of the queries produced by the feature enhancement layer.

Experimental Setup

Datasets We use a curated re-identification dataset from WildlifeDatasets [2]. We specifically utilize the GiraffeZebraID [7] and ATRW [4] datasets because they come with the full-scene image and the bounding box metadata.

Implmentation We employ a ViT-S/14 variant of DINOv2 for the multi-scale scene encoder and a MegaDescriptor-T variant for the prompt encoder. Scene images are resized to $518 \times518$ to align with the DINOv2 patch configuration, while prompt crops are resized to $224 \times224$. We selectively unfreeze the final transformer block and normalization layers of both encoders during training, allowing their distinct latent manifolds to synchronize. We use PyTorch Lightning for efficient training [3].

Hyperparameters: The network is optimized end-to-end using the AdamW optimizer with a base learning rate of $1 \times10^{-4}$ and a weight decay of $1 \times10^{-4}$. We employ a MultiStepLR learning rate scheduler, which decays the rate by a factor of $\gamma= 0.1$. The objectness, identity matching, bounding box L1 distance, and Generalized IoU is set to 2.0, 2.0, 5.0, and 2.0, respectively.

Preliminary Results

We provide qualitative preliminary findings in Figure 2. The proposed framework produces bounding boxes that encapsulate the target animal. Our model can handle herds, domain shifts, occlusion, distance, and camouflage. We present results on retrieval performance (ReID) in Figure 3 using our end-to-end latent grounding method and compared it to the two-stage approach that uses MegadetectorV6 for detection, crop and followed by MegaDescriptor to embed the cropped images to perform retrieval. The results show the potential of the proposed approach even though it has not yet achieved state-of-the-art performance. However, the proposed approach is much more efficient.

Conclusion

This paper proposes a framework that combines visual prompt along with global scene features to perform end-to-end object detection together with re-identification. The preliminary findings suggest that visual identity prompts can reduce the reliance on traditional two-stage detect and crop followed by ReID pipelines by allowing the model to localize and match a target directly within the full scene image. We believe continued work in this direction could lead to state-of-the-art performance.