betaFeedback

WorksComputer Science

Synthetic-to-Real Text-Based Person Anomaly Search via Multi-Model Retrieval Fusion and Cross-Encoder Reranking

Abstract

Text-based person anomaly search requires a model to retrieve individuals from a large image gallery using natural-language descriptions of both appearance and behavior. This task is challenging because model training relies on synthetic image-text pairs, whereas evaluation is performed on real surveillance images containing domain differences, near-duplicate frames, and many visually similar distractors. We present a multi-model retrieval and reranking framework designed to improve synthetic-to-real transfer. The framework first adapts Qwen3-VL-Embedding-8B to synthetic person descriptions using bidirectional contrastive learning, explicit negative sampling, false-negative masking, and domain randomization. It then combines the normalized scores of six complementary Qwen3-VL checkpoints with those of a frozen SigLIP2 model using four-view gallery encoding. The fused retrieval scores identify the strongest candidates, which are jointly evaluated with the query using Qwen3-VL-Reranker-8B. The cross-encoder and retrieval scores are subsequently blended to produce the final ranking. We also construct a distractor-matched validation protocol entirely from synthetic data to select checkpoint weights, fusion coefficients, candidate depths, and reranking parameters without accessing the official test images. The final system achieves 94.7815% mAP and 90.8493% R@1 under the official AI City Challenge 2026 Track 4 test setting, with a reported eighth-place finish on the public leaderboard.

Introduction

Text-based person anomaly search retrieves people from an image gallery using descriptions of both appearance and behavior [1, 2]. A useful match must satisfy clothing and physical-attribute constraints while also depicting the specified action, such as running, falling, or lying down. A visually similar person performing a different action is therefore a difficult distractor. The PAB benchmark makes this distinction explicit through synthetic training pairs and real-world evaluation images [1]. This setting raises a practical question: how can task-specific adaptation on synthetic data support reliable ranking in a real surveillance gallery?

Related transportation research provides several useful perspectives. Aboah’s vision-based traffic anomaly detector [3] and DeepSegmenter [4] study unusual events and driver actions, while QueryMotion combines attributes and motion reasoning for natural-language vehicle retrieval [5]. These tasks emphasize the importance of behavioral evidence alongside appearance, although their datasets and evaluation objectives differ from person anomaly search. In the present task, domain differences, near-duplicate frames, and large numbers of plausible distractors create additional ranking challenges.

Image not extracted

Figure 1. Synthetic-to-real text-based person anomaly retrieval setting. Given synthetic image-text pairs for training and a natural-language query at inference, the goal is to rank images from a real-world gallery according to their agreement with the described person and behavior. The examples show running and walking queries where multi-model retrieval fusion and cross-encoder reranking promote the best-matching real image to Rank 1 among visually similar distractors.

We address these challenges through a multi-stage framework. First, we adapt Qwen3-VL-Embedding-8B [6] to synthetic person descriptions using contrastive learning, explicit negative sampling, false-negative masking, and domain randomization. Second, we fuse standardized scores from six complementary Qwen3-VL checkpoints with a frozen SigLIP2 branch [7]. Third, Qwen3-VL-Reranker-8B jointly evaluates the query and the strongest image candidates, and its scores are blended with retrieval scores. A distractor-matched synthetic validation protocol selects the fusion and reranking settings.

The study makes three contributions:

  1. A retrieval framework that combines task-adapted Qwen3-VL checkpoints, frozen SigLIP2 representations, and selective cross-encoder refinement for synthetic-to-real person anomaly search.

  2. A synthetic validation protocol with a large gallery and identity-preserving, behavior-differing distractors, designed to expose errors obscured by nearly saturated conventional validation.

  3. An evaluation of retrieval, training, fusion, and reranking configurations under the AI City Challenge 2026 Track 4 setting [8]. The reported final result is 94.7815% mAP and 90.8493% R@1, placing SMART Lab eighth on the reported public leaderboard.

The contribution is the task-specific integration and evaluation of these components. The results support the value of complementary retrieval evidence and focused candidate refinement; they do not establish that frozen and adapted models are inherently complementary in every setting. Here, “synthetic-only training” refers to task-specific adaptation, rather than the pretraining data of the foundation models.

Related Work

Vision-Language Pretraining

CLIP [9] learns transferable image-text representations through natural-language supervision, while ALIGN [10] scales dual-encoder contrastive learning using noisy image-text pairs. These models establish a shared-space retrieval paradigm in which gallery images can be encoded independently of a query. SigLIP [11] replaces globally normalized contrastive training with a pairwise sigmoid objective, providing the foundation for the SigLIP2 branch used here. BLIP-2 [12] connects frozen image encoders and language models through a trainable querying transformer, illustrating a complementary approach to reusing pretrained components. These studies motivate separating general vision-language knowledge from task-specific adaptation, while leaving behavior-sensitive matching as an additional requirement.

Text-Based Person Retrieval and Anomaly Search

Li et al. [13] introduced person search with natural-language descriptions and the CUHK-PEDES benchmark. SSAN [14] improves part-aware image-text alignment, while DSSL [15] separates person information from surrounding visual content. IRRA [16] learns implicit relations between visual and textual tokens. UFineBench [17] extends evaluation to more detailed descriptions. Earlier cross-modal projection learning [18] uses matching and classification objectives to learn discriminative image-text embeddings. More recent approaches adapt the representation to subtle pedestrian details: CFine [19] transfers CLIP knowledge through fine-grained feature interaction, IVT [20] learns implicit alignment at sentence, phrase, and word levels, and PLIP [21] introduces person-specific pretraining tasks involving body regions, attributes, and identities. These methods highlight the need to preserve local semantic detail when transferring generic encoders to person search.

Training-pair reliability is another important consideration. RaSa [22] distinguishes strong and weak positive correspondences and learns sensitivity to replaced words. RDE [23] addresses noisy image-text correspondences through consensus-based clean-pair selection and a robust alignment loss. Their treatment of imperfect correspondence is relevant to descriptions that match only some attributes of an image. These works provide appearance-sensitive retrieval foundations. Beyond Walking [1] introduces the PAB benchmark and a cross-modal pose-aware framework for retrieving both routine and anomalous behaviors. AnomalyLMM [2] further studies behavioral saliency and knowledge-aware reranking on PAB. Our framework follows this behavior-sensitive task formulation and examines complementary embedding models and selective cross-encoder refinement.

Behavior Understanding and Multimodal Transportation Models

Behavior understanding has also been studied through temporal localization and multiple sensor views. DeepSegmenter [4] combines segmentation and classification in untrimmed driving videos; SigSegment [24] studies signal-based segmentation of anomalous driving behavior. Dontoh et al. [25] compare driver-facing and dual-view inputs for distraction detection. At the scene level, Dentel et al. [26] combine flow-guided attention with empirical mode decomposition for congestion classification. Together, these studies motivate distinguishing behavioral evidence from static appearance; temporal or multi-view evidence is not evaluated by our image-gallery protocol.

Language-based interpretations offer a complementary direction. Shoman et al. [27] study parallel dense video captioning for traffic-event analysis, and Kyem et al. [28] specialize separate vision-language models for traffic captioning and visual question answering. Muturi et al. [29] use structured prompts for spatial relation reasoning, while QueryMotion [5] combines attribute and motion cues for natural-language vehicle retrieval. These works support the relevance of explicit semantic and relational cues, but do not directly measure person anomaly retrieval on PAB.

Synthetic Data, Visual Robustness, and Domain Adaptation

Domain randomization [30] increases variability in simulated imagery to support real-world transfer. SimGAN [31] instead refines synthetic images adversarially using unlabeled real images while preserving their annotations. For person re-identification, CamStyle [32] uses camera-style translation to augment training images and smooth cross-camera differences. These studies represent different responses to appearance shift; the synthetic validation protocol in this work also addresses the separate problem of selecting configurations before real-gallery evaluation.

Synthetic pedestrian imagery and domain adaptation were explored by Bak et al. [33] and UnrealPerson [34]. UniPT [35] uses pseudo descriptions for task-specific person retrieval pretraining, and Cao et al. [36] systematically examine synthetic data for text-based person retrieval. MALS and its APTM framework [37] further combine diffusion-generated person images, attribute prompts, and text matching, demonstrating a concrete route from synthetic multimodal pretraining to real person-retrieval benchmarks. In a related infrastructure setting, Kyem et al. [38] use weather-adaptive synthetic data for power-line inspection. These studies motivate examining how generated data and task-specific adaptation transfer to real imagery.

Robust traffic perception also depends on image conditions and sampling. Tran et al. [39] combine low-light enhancement and ensemble learning for fisheye imagery; Owor et al. [40] develop an ensemble detection pipeline for distorted surveillance images. Aboah et al. [41] study few-shot data sampling for multi-class helmet-violation detection. Kyem et al. [42] investigate locally collected pavement imagery, and PaveSync [43] consolidates heterogeneous pavement datasets. These are related examples of data diversity and visual robustness, rather than baselines for our retrieval task.

Domain-Specific Adaptation, Fusion, and Candidate Refinement

PaveCap [44] connects dense descriptions with pavement condition estimation, and PaveGPT [45] studies domain-specific instruction tuning for infrastructure assessment. CrackGeoFM [46] combines a frozen visual backbone with task-specific adaptation for crack representation and topology preservation. Denteh et al. [47, 48] integrate demographic, travel-behavior, and satellite-image information for urban forecasting. Although these applications differ from person retrieval, they provide broader examples of adapting or combining representations to satisfy domain-specific semantic and structural constraints.

Retaining pretrained knowledge during adaptation is also studied by WiSE-FT [49], which interpolates pretrained and fine-tuned model weights to improve robustness under distribution shift. Model soups [50] average compatible fine-tuned weights to combine models without the inference overhead of a conventional ensemble. Both operate in parameter space, whereas our framework retains multiple encoders and combines their standardized similarity scores. This distinction is relevant to the accuracy–cost trade-off of the six-checkpoint retrieval ensemble.

Candidate refinement has a more direct retrieval precedent. Zhong et al. [51] use reciprocal-neighborhood information for person re-identification, and SA-Person [52] incorporates scene context through a second-stage reranker. Qwen3-VL-Embedding and Qwen3-VL-Reranker [6] provide complementary embedding and cross-encoder models. Reciprocal rank fusion [53] provides a rank-based way to combine retrieval lists without requiring comparable raw score scales. Our score standardization and weighted fusion use a different aggregation rule, followed by query-image cross-encoder refinement. The study evaluates how these model types can be combined with SigLIP2 and synthetic distractor-matched validation for behavior-sensitive person search.

Negative Sampling and Model Selection under Domain Shift

The quality of negative examples affects contrastive representation learning. Robinson et al. [54] study controllable hard-negative sampling, while debiased contrastive learning [55] corrects the contamination introduced when sampled negatives share the anchor’s semantic label. Together, these studies explain why challenging negatives and false-negative handling should be considered jointly. In person anomaly search, the issue is particularly nuanced because the same identity can be relevant for one behavioral description and irrelevant for another; the masking and validation design must therefore respect the full query semantics.

DomainBed [56] emphasizes that model-selection criteria are an integral part of evaluating domain generalization algorithms. This observation motivates specifying the validation domain and candidate construction alongside the training procedure. Our distractor-matched synthetic validation is designed to expose ranking errors before real-gallery evaluation, and its empirical value is assessed separately from the representation and reranking components.

Proposed Method

Problem Formulation

Given a natural-language query qq and an image gallery G={Ii}i=1N\mathcal{G} = \{I_i\}_{i=1}^{N}, text-based person anomaly search retrieves images that match both the person and the behavior described in the query, including clothing, physical attributes, routine actions, or anomalous activities like falling or lying down. The relevant images Gq+⊆G\mathcal{G}_{q}^{+} \subseteq\mathcal{G} must score higher than all distractors in G∖Gq+\mathcal{G} \setminus\mathcal{G}_{q}^{+}. The challenge lies in distinguishing complete matches from partial ones: distractors may show the correct person performing a different action, a different person with similar clothing, or a near-duplicate frame from adjacent video moments. The model must therefore jointly satisfy appearance and behavioral constraints, not just one or the other.

This objective is naturally formulated as embedding-space retrieval. An image encoder fI(⋅)f_I(\cdot) and text encoder fT(⋅)f_T(\cdot) project gallery images and the query into a shared space, producing normalized representations vi=fI(Ii)/∥fI(Ii)∥2\mathbf{v}_i = f_I(I_i)/\lVert f_I(I_i)\rVert_2 and uq=fT(q)/∥fT(q)∥2\mathbf{u}_q = f_T(q)/\lVert f_T(q)\rVert_2. Their correspondence is measured using cosine similarity:

s(Ii,q)=viTuq.(1)s(I_i, q) = \mathbf{v}_i^{\mathsf{T}}\mathbf{u}_q. \tag*{(1)}

The challenge in practice is that models are trained on synthetic images but must retrieve from real surveillance galleries, creating a substantial domain gap. Synthetic and real images differ in illumination, compression, background complexity, image quality, and visual artifacts. Standard retrieval approaches applied directly to this domain gap produce suboptimal results. We therefore propose a three-stage framework: first, adapt Qwen3-VL-Embedding-8B to the task using synthetic data; second, combine its checkpoints with a frozen SigLIP2 branch to fuse complementary representations; and third, refine the strongest candidates through joint query-image encoding via Qwen3-VL-Reranker-8B. Figure 2 provides an overview of this approach.

Image not extracted

Figure 2. Overview of the proposed synthetic-to-real text-based person anomaly retrieval framework. Synthetic image-text pairs are used to adapt Qwen3-VL-Embedding-8B through bidirectional contrastive learning, explicit negative sampling, false-negative masking, and domain randomization, producing six complementary checkpoints. During inference, their weighted similarity scores are fused with scores from a frozen SigLIP2 retrieval branch using four-view gallery encoding. The combined retrieval score retains the top-100 candidates, of which the top-50 are jointly processed by Qwen3-VL-Reranker-8B and blended with the retrieval score to produce the final ranking. Distractor-matched synthetic validation is used for model and parameter selection without accessing the official challenge test images.

Embedding Representation Learning

The first stage learns a shared representation of person appearance and behavior from synthetic PAB data [1], consisting of samples (In,tn,yn,bn)(I_n, t_n, y_n, b_n) where InI_n is a synthetic image, tnt_n is its text description, and yny_n and bnb_n denote the identity and behavior labels.

Qwen3-VL-Embedding-8B serves as the embedding backbone [6] and is adapted using LoRA [57] with rank 16 and scaling factor 32, applied to the eligible linear layers. The visual encoder remains frozen while the multimodal alignment layers remain trainable, preserving the general visual representation while adapting the shared embedding space to PAB descriptions.

Training uses a bidirectional contrastive objective with four explicit negatives per sample. The text-to-image loss is

Lt2i=−1B∑n=1Blog⁡exp⁡(s(In+,tn)/τ)exp⁡(s(In+,tn)/τ)+∑I∈Nnmn,Iexp⁡(s(I,tn)/τ)(2)\mathcal{L}_{t2i} = -\frac{1}{B}\sum_{n=1}^{B}\log\frac{\exp(s(I_n^{+}, t_n)/\tau)}{\exp(s(I_n^{+}, t_n)/\tau) + \sum_{I \in\mathcal{N}_n} m_{n,I}\exp(s(I,t_n)/\tau)} \tag*{(2)}

where BB is the batch size, τ\tau is the contrastive temperature, and mn,I∈{0,1}m_{n,I} \in\{0,1\} masks candidates identified as possible false negatives using a similarity margin of 0.02. The image-to-text loss Li2t\mathcal{L}_{i2t} follows the same formulation with the roles of image and text reversed, and the complete embedding loss is their average, Lemb=12(Lt2i+Li2t)\mathcal{L}_{emb} = \frac{1}{2}(\mathcal{L}_{t2i} + \mathcal{L}_{i2t}). This objective aligns matched image-text pairs while separating them from candidates with different identities or behaviors.

Domain randomization varies illumination, contrast, color, blur, noise, shadows, compression quality, crop scale, and horizontal orientation while preserving identity and behavior labels, encouraging the model to learn target semantics rather than synthetic visual patterns. Training and continuation runs under different augmentation settings and random seeds produce six complementary Qwen3-VL checkpoints. These checkpoints capture different perspectives on the synthetic-to-real transfer and are fused in the next stage via weighted ensemble.

Multi-Model Retrieval Fusion

The second stage fuses the six Qwen3-VL checkpoints with a frozen SigLIP2 [7] model to combine specialized behavior-aware retrieval with general-purpose visual-language understanding. SigLIP2 is kept frozen to preserve its pretrained representations and maintain complementarity with the task-adapted Qwen3-VL ensemble. Adapting both models on the same synthetic data could reduce their diversity; the reported fusion experiments evaluate the benefit of retaining a frozen branch without establishing representation redundancy directly.

Score Standardization:

Since checkpoints and model families produce scores on different scales, we standardize all scores to a common scale before fusion. For any score set {ai(q)}i=1N\{a_i(q)\}_{i=1}^{N} from query qq,

Zq(ai(q))=ai(q)−μa(q)σa(q)+ϵ(3)Z_q(a_i(q)) = \frac{a_i(q) - \mu_a(q)}{\sigma_a(q) + \epsilon} \tag*{(3)}

where μa(q)\mu_a(q) and σa(q)\sigma_a(q) are the gallery mean and standard deviation, and ϵ>0\epsilon> 0 prevents division by zero.

Qwen3-VL Checkpoint Ensemble:

Each of the six Qwen3-VL checkpoints encodes both the original gallery image and its horizontal reflection, which are averaged before scoring. The similarity scores are standardized and combined via weighted fusion:

sQ(Ii,q)=∑m=1MwmZq(smQ(Ii,q)),wm≥0,∑m=1Mwm=1,(4)s_Q(I_i,q) = \sum_{m=1}^{M} w_m Z_q(s^Q_m(I_i,q)), \qquad w_m \ge0,\qquad\sum_{m=1}^{M} w_m = 1, \tag*{(4)}

where wmw_m is the validation-selected weight for checkpoint mm and M=6M = 6.

Checkpoint Ensemble Strategy:

The six checkpoints are trained on the same PAB dataset but under different augmentation and random seeds, producing complementary representations that capture different aspects of the synthetic-to-real transfer. Some checkpoints may emphasize appearance details while others prioritize behavioral nuances. Their complementarity reduces overfitting to spurious synthetic patterns and provides more robust retrieval signals. The weighted fusion combines these complementary perspectives, with weights wmw_m selected via distractor-matched validation to balance their contributions.

SigLIP2 Four-View Encoding:

The frozen SigLIP2 branch encodes four image views: the full image, its horizontal flip, and upper and lower 60% crops. These embeddings are averaged and renormalized into a single gallery representation. The complete query is processed by the SigLIP2 text encoder without paraphrasing, yielding a similarity score sS(Ii,q)s_S(I_i,q) following Eq. (1).

Cross-Model Score Fusion:

The standardized Qwen3-VL ensemble score and the SigLIP2 score are combined as

sret(Ii,q)=(1−β)Zq(sQ(Ii,q))+βZq(sS(Ii,q)),(5)s_{\mathrm{ret}}(I_i,q) = (1-\beta) Z_q(s_Q(I_i,q)) + \beta Z_q(s_S(I_i,q)), \tag*{(5)}

where β∈[0,1]\beta\in[0,1] controls the SigLIP2 contribution. Distractor-matched validation selects β=0.30\beta= 0.30. The resulting retrieval score ranks the gallery and provides candidates for the reranking stage.

Candidate Reranking

The fused retrieval score efficiently ranks the gallery but may not fully separate candidates with similar appearance, behavior, or scene context. Detailed query-image comparison is therefore applied only to the strongest candidates, reducing computational cost while focusing refinement where it matters most.

For each query qq, the top 100 retrieval-scored images form a candidate pool Cq100C_q^{100}. Of these, the top 50 images Cq50C_q^{50} are processed by Qwen3-VL-Reranker-8B, while candidates ranked 51–100 retain their retrieval order. The cross-encoder jointly encodes the query and each candidate image, allowing direct visual-textual interaction that the dual encoders cannot capture. This produces a compatibility score sce(Ii,q)s_{\mathrm{ce}}(I_i, q) reflecting detailed agreement in appearance, action, body state, and scene context.

The retrieval and cross-encoder scores are independently standardized over Cq50\mathcal{C}_q^{50} and blended as

sfinal(Ii,q)=(1−α)sret(Ii,q)+αsce(Ii,q),Ii∈Cq50,(6)s_{\mathrm{final}}(I_i, q) = (1-\alpha)s_{\mathrm{ret}}(I_i, q) + \alpha s_{\mathrm{ce}}(I_i, q), \qquad I_i \in\mathcal{C}_q^{50}, \tag*{(6)}

where α∈[0,1]\alpha\in[0, 1] controls the cross-encoder weight. Distractor-matched validation selects α=0.70\alpha= 0.70. The top 50 candidates are reordered by sfinals_{\mathrm{final}}, while the remaining 51–100 retain their original retrieval ranking.

Distractor-Matched Validation

Model and parameter selection require validation on data that reflects the difficulty of the evaluation gallery. Conventional synthetic validation contains fewer challenging distractors and produces nearly saturated scores, making model selection less informative. We therefore propose a distractor-matched validation protocol designed to surface ranking errors that occur on real data.

The protocol uses 1,978 held-out queries and 34,795 synthetic distractors, forming a gallery of 36,773 images. Critically, 15% of the distractors share the query identity but depict different behaviors. These identity-preserving, behavior-differing negatives test whether the model distinguishes actions rather than relying solely on identity or clothing—a challenge that standard synthetic validation underrepresents.

This protocol selects all model and pipeline parameters: the Qwen3-VL checkpoint combination and weights wmw_m, the SigLIP2 fusion coefficient β\beta, the candidate pool depth (100) and reranking depth (50), and the cross-encoder blend weight α\alpha. Critically, the official challenge test images are never accessed during training or validation, so model selection is reported to use synthetic validation only.

Implementation Details

The Qwen3-VL embedding model is trained for 6,000 steps using AdamW with a cosine schedule, a warm-up ratio of 0.05, and weight decay of 0.1. The primary run uses a learning rate of 1×10−71 \times10^{-7} and continuation runs use 5×10−85 \times10^{-8}, with a contrastive temperature of 0.015. Text inputs are limited to 512 tokens and images use 32 to 64 visual tokens.

The SigLIP2 branch uses the pretrained google/siglip2-so400m-patch14-384 checkpoint without adaptation, combined with the Qwen3-VL ensemble using β=0.30\beta= 0.30. The fused retrieval model retains 100 candidates per query, of which the top 50 are sent to Qwen3-VL-Reranker-8B, with a final blend of α=0.70\alpha= 0.70. All checkpoint weights, fusion coefficients, candidate depths, and reranking parameters remain fixed after validation.

Results and Discussion

We evaluate the proposed framework under the official AI City Challenge 2026 Track 4 setting [8]. The analysis first presents the overall challenge results and then examines the retrieval representations, training strategies, reranking methods, and distractor-matched validation protocol. All ablations use stage-matched comparisons so that the effect of each component can be interpreted independently.

Official Challenge Results

Table 1 compares the proposed framework with transferred configurations of the Cross-Modal Pose-aware framework, denoted CMP [1]. CMP configurations A and B are independent implementations, while the CMP consensus combines their rankings. The remaining configurations show the main stages of the proposed framework.

The CMP consensus substantially improves upon the two individual CMP configurations, showing that their rankings contain complementary information. However, the Qwen3-VL retrieval ensemble achieves a further increase to 92.5510% mAP and 87.3610% R@1. This comparison applies only to the transferred CMP configurations evaluated in this study and does not represent CMP performance under its original evaluation protocol.

Cross-encoder refinement increases mAP by 1.9076 points and R@1 by 2.9828 points over the Qwen3-VL retrieval ensemble. The larger improvement at Rank 1 indicates that joint query-image reasoning is particularly useful for reordering visually or behaviorally similar candidates near the top of the ranking.

MethodmAPR@1R@5R@10
CMP transfer configuration A67.248558.190178.362082.7098
CMP transfer configuration B67.098357.937379.069882.0020
CMP consensus ensemble85.459676.188197.573398.6350
Qwen3-VL retrieval ensemble92.551087.361098.230598.9889
Qwen3-VL ensemble + cross-encoder94.458690.343899.039499.4439
Qwen3-VL + SigLIP2 fusion + cross-encoder94.781590.849399.039499.4944

Table 1. Performance under the official AI City Challenge 2026 Track 4 [8] test setting. All values are percentages.

RankTeam IDTeam NamemAPR@1R@5R@10
159Xiilab.AIpex99.302098.736199.9494100.0000
266Hallucination99.255198.786799.747299.7978
39hiensumi98.353597.219499.646199.8989
4137UIT-OpenCubee95.937092.416699.848399.8483
5103Safe AI95.925093.731098.584499.5450
653UWIPL95.304292.012198.887899.0394
727VGU AI LAB95.178292.467198.483398.6855
897SMART Lab94.781590.849399.039499.4944
9174Cascade94.648590.798898.887899.3933
10127Team KODE94.546590.546098.938399.3933

Table 2. Reported public leaderboard snapshot for AI City Challenge 2026 Track 4 [8].

Adding frozen SigLIP2 retrieval evidence produces the final result of 94.7815% mAP and 90.8493% R@1. This represents a further improvement of 0.3229 mAP points and 0.5055 R@1 points over Qwen3-VL reranking alone. The result shows that complementary retrieval evidence remains useful even after candidate-level refinement.

Table 2 presents the reported public leaderboard snapshot. The SMART Lab submission is listed eighth. The organizer’s published results summarize the leading five mAP entries [58].

The final R@10 of 99.4944% shows that relevant images are usually retained within the first ten results. The main remaining challenge is therefore the precise ordering of the strongest candidates.

Retrieval Representation and Fusion Analysis

We next isolate the retrieval stage to examine the effects of task adaptation, checkpoint ensembling, validation-based weighting, and cross-model fusion. Table 3 reports all configurations before cross-encoder reranking.

ConfigurationmAPR@1R@10
SigLIP2, frozen standalone90.887584.086098.5990
SigLIP2, task-adapted on PAB91.630185.542298.7991
Individual Qwen3-VL checkpoint91.091586.286598.5990
Qwen3-VL ensemble, unweighted92.092387.087298.8990
Qwen3-VL ensemble, validation-weighted92.551087.361098.9889
Qwen3-VL + frozen SigLIP2 fusion93.093388.088398.9991

Table 3. Retrieval configurations evaluated before cross-encoder reranking. All values are percentages.

Task adaptation improves standalone SigLIP2 by 0.7426 mAP points and 1.4562 R@1 points, confirming that the synthetic PAB data provide useful task-specific information. However, both SigLIP2 configurations remain below the Qwen3-VL ensemble.

Combining six Qwen3-VL checkpoints improves mAP from 91.0915% to 92.0923%. Validation-based weighting provides a further increase to 92.5510%. These results show that checkpoint diversity and validation-guided weighting both contribute to the retrieval stage.

The strongest retrieval-only result is obtained by combining the weighted Qwen3-VL ensemble with frozen SigLIP2. Fusion improves mAP by 0.5423 points and R@1 by 0.7273 points over the weighted Qwen3-VL ensemble. Although frozen SigLIP2 is weaker as a standalone model, it contributes complementary retrieval evidence when combined with the task-adapted Qwen3-VL representation.

Ablation Study

The following experiments examine embedding-stage training, reranked training, and candidate refinement. Each comparison keeps the remaining pipeline components fixed.

Embedding-Stage Training Strategies

Table 4 compares alternative training and augmentation strategies before reranking. The hard-negative fine-tuning configuration serves as the baseline.

ConfigurationmAPR@1Δ\Delta mAP
Hard-negative fine-tuning baseline92.551087.3610—
Query-paraphrase augmentation85.343277.8564-7.2078
Query-paraphrase ensemble93.190688.8777+0.6396
Iterative identity-based hard-negative mining89.187382.3054-3.3637
Watermark-style augmentation92.290887.1587-0.2602

Table 4. Embedding-stage strategies evaluated before reranking. Δ\DeltamAP is measured relative to the baseline.

Direct query-paraphrase augmentation produces the largest reduction, suggesting that some reformulations remove or alter behavior-related details. Combining the original and paraphrased queries avoids this loss and slightly exceeds the baseline, although the improvement remains modest.

Iterative identity-based hard-negative mining also reduces performance, indicating that excessive emphasis on synthetic identities can weaken transfer. Watermark-style augmentation has only a minor effect. The final embedding configuration therefore retains the original descriptions and a moderate hard-negative strategy.

Reranked Training Strategies

Table 5 compares training configurations after applying the same cross-encoder and score-blending procedure.

ConfigurationmAPR@1Δ\Delta mAP
Qwen3-VL ensemble + cross-encoder94.458690.3438—
Domain randomization + compliant real-data mixture94.329790.2932-0.1289
Identity-based hard-negative continuation94.219490.0910-0.2392

Table 5. Training strategies evaluated with fixed reranking and score-blending settings.

Neither alternative improves upon the reranked Qwen3-VL baseline. The real-data mixture remains close to the baseline but provides no measurable gain, while identity-based continuation produces a larger reduction. These results indicate that more complex training mixtures or additional identity specialization do not necessarily improve real-gallery ranking.

Candidate-Reranking Strategies

Table 6 compares reranking methods using the same Qwen3-VL retrieval candidate pool.

Reranking configurationmAPR@1Δ\Delta mAP
No reranking92.551087.3610—
Zero-shot Qwen3-VL cross-encoder94.458690.3438+1.9076
LoRA-adapted Qwen3-VL cross-encoder94.224090.0404+1.6730
30B mixture-of-experts reranker94.142289.9899+1.5912
Score blending, α≈0.75\alpha\approx0.7594.531690.5966+1.9806

Table 6. Candidate-reranking strategies using a fixed Qwen3-VL retrieval pool.

The zero-shot cross-encoder improves mAP by 1.9076 points over the fixed retrieval baseline. In contrast, LoRA adaptation and the larger 30B reranker perform slightly worse. This result shows that greater model capacity or additional synthetic adaptation does not automatically improve candidate ordering.

Increasing the cross-encoder weight to approximately α=0.75\alpha= 0.75 gives the strongest Qwen3-VL-only result. However, it remains 0.2499 mAP points below the complete Qwen3-VL–SigLIP2 system, confirming that the final gain cannot be attributed to reranker weighting alone.

Effect of Distractor-Matched Validation

The retrieval-fusion coefficient, cross-encoder blending weight, checkpoint weights, and candidate depths are selected using synthetic validation data without accessing the official challenge test images. Table 7 compares conventional and distractor-matched validation.

ProtocolQueriesGallerymAPR@1R@10
Conventional validation14,72214,72298.668697.697399.9185
Distractor-matched validation1,97836,77397.199595.348899.6461

Table 7. Diagnostic comparison of conventional and distractor-matched synthetic validation.

Conventional validation is nearly saturated. Increasing the gallery size and introducing more difficult distractors reduce mAP to 97.1995% and R@1 to 95.3488%. The smaller reduction in R@10 shows that the added distractors primarily affect top-k ranking rather than overall candidate retention.

The protocols use different query sets and gallery-construction procedures, so the numerical difference is interpreted only as a diagnostic comparison. The distractor-matched protocol provides a less saturated setting for model selection and more closely reproduces the large-gallery and high-distractor structure of the target task.

Qualitative Analysis

Figure 3 shows that the fused retrieval stage places the annotated image within the top three for all three examples. Cross-encoder refinement then promotes each match to Rank 1.

Image not extracted

Figure 3. Qualitative retrieval results under the official AI City Challenge 2026 Track 4 setting [8]. Each row shows the query, the initial top-five ranking from Qwen3-VL–SigLIP2 fusion, and the final ranking after cross-encoder refinement. Green checkmarks indicate the annotated match, while red crosses indicate distractors.

For Query 1, the description combines skateboarding with an interaction involving another person falling from a skateboard. The reranker promotes the candidate containing the complete behavioral context above images that mainly match the retail environment or individual appearance. Query 3 shows a similar improvement, as the final ranking gives greater weight to the cycling action together with the described clothing and scene.

Query 2 contains two nearly identical frames from the same skate-park sequence. The reranker selects the annotated frame, but the limited visual difference between the candidates illustrates the difficulty of exact frame-level retrieval. Overall, the examples show that the proposed task-specific combination of complementary retrieval representations and selective cross-encoder refinement improves appearance- and behavior-sensitive ordering while preserving a strong initial shortlist.

Limitations

The evaluation concerns one challenge setting, and the transferred CMP configurations are not a reproduction of CMP’s original benchmark protocol. The reported comparisons do not include uncertainty estimates, so small differences should not be interpreted as statistically established improvements. The six-checkpoint ensemble and cross-encoder introduce computation and storage costs that are not quantified here. Exact checkpoint weights, batch size, hardware, random seeds, and split identifiers are not specified, limiting exact reproduction. The conventional and distractor-matched validation sets also differ in both queries and gallery construction. Finally, image-level retrieval does not explicitly model temporal evidence, and near-duplicate frames remain difficult. Task-specific synthetic adaptation does not imply that the foundation backbones were pretrained exclusively on synthetic data.

Conclusion

We presented a multi-stage vision-language framework for synthetic-to-real text-based person anomaly search. The framework combines a Qwen3-VL retrieval ensemble with complementary SigLIP2 retrieval scores and applies cross-encoder reranking to the strongest gallery candidates. This design preserves efficient global retrieval while improving fine-grained agreement between the query and candidate images. The final system achieves 94.7815% mAP and 90.8493% R@1 under the official AI City Challenge 2026 Track 4 test setting, improving the Qwen3-VL retrieval baseline by 2.2305 and 3.4883 percentage points, respectively. The SMART Lab submission has a reported eighth-place finish on the public leaderboard. The results show that complementary retrieval representations and focused candidate reranking provide stronger transfer than increasing training complexity or reranker capacity alone. Future work will investigate temporal information from adjacent video frames and model distillation to improve near-duplicate discrimination while reducing inference cost.

References

  1. [17]Jialong Zuo, Hanyu Zhou, Ying Nie, Feng Zhang, Tianyu Guo, Nong Sang, Yunhe Wang, and Changxin Gao. UFineBench: Towards Text-based Person Retrieval with Ultra-fine Granularity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22010–22019, June 2024.arxiv.org/abs/2312.03441
  2. [18]Ying Zhang and Huchuan Lu. Deep Cross-Modal Projection Learning for Image-Text Matching. In Proceedings of the European Conference on Computer Vision (ECCV), 2018. Open-access proceedings.DOI
  3. [19]Shuanglin Yan, Neng Dong, Liyan Zhang, and Jinhui Tang. CLIP-Driven Fine-Grained Text-Image Person Re-Identification. IEEE Transactions on Image Processing, 32:6032–6046, 2023. 10.1109/TIP.2023.3327924.DOI
  4. [20]Xiujun Shu, Wei Wen, Haoqian Wu, Keyu Chen, Yiran Song, Ruizhi Qiao, Bo Ren, and Xiao Wang. See Finer, See More: Implicit Modality Alignment for Text-based Person Retrieval. arXiv preprint arXiv:2208.08608, 2022. 10.48550/arXiv.2208.08608.arxiv.org/abs/2208.08608
  5. [21]Jialong Zuo, Jiahao Hong, Feng Zhang, Changqian Yu, Hanyu Zhou, Changxin Gao, Nong Sang, and Jingdong Wang. PLIP: Language-Image Pre-training for Person Representation Learning. arXiv preprint arXiv:2305.08386, 2023. 10.48550/arXiv.2305.08386.arxiv.org/abs/2305.08386
  6. [22]Yang Bai, Min Cao, Daming Gao, Ziqiang Cao, Chen Chen, Zhenfeng Fan, Liqiang Nie, and Min Zhang. RaSa: Relation and Sensitivity Aware Representation Learning for Text-based Person Search. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 555–563, 2023. 10.24963/ijcai.2023/62.
  7. [23]Yang Qin, Yingke Chen, Dezhong Peng, Xi Peng, Joey Tianyi Zhou, and Peng Hu. Noisy-Correspondence Learning for Text-to-Image Person Re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27197–27206, 2024. Open-access proceedings.arxiv.org/abs/2308.09911
  8. [24]Kelvin Kwakye, Youngho Seong, Armstrong Aboah, and Sun Yi. SigSegment: A Signal-Based Segmentation Algorithm for Identifying Anomalous Driving Behaviours in Naturalistic Driving Videos. arXiv preprint arXiv:2304.09247, 2023. 10.48550/arXiv.2304.09247.arxiv.org/abs/2304.09247
  9. [25]Anthony Dontoh, Stephanie Ivey, and Armstrong Aboah. A Contextual Analysis of Driver-Facing and Dual-View Video Inputs for Distraction Detection in Naturalistic Driving Environments. arXiv preprint arXiv:2512.20025, 2025. 10.48550/arXiv.2512.20025.DOI
  10. [26]Eugene Kofi Okrah Denteh, Blessing Agyei Kyem, Joshua Kofi Asamoah, and Armstrong Aboah. Hybrid Congestion Classification Framework Using Flow-Guided Attention and Empirical Mode Decomposition. arXiv preprint arXiv:2605.04752, 2026. 10.48550/arXiv.2605.04752.arxiv.org/abs/2605.04752
  11. [27]Maged Shoman, Dongdong Wang, Armstrong Aboah, and Mohamed Abdel-Aty. Enhancing Traffic Safety with Parallel Dense Video Captioning for End-to-End Event Analysis. arXiv preprint arXiv:2404.08229, 2024. 10.48550/arXiv.2404.08229.arxiv.org/abs/2404.08229
  12. [28]Blessing Agyei Kyem, Neema J. Owor, Andrews Danyo, Joshua K. Asamoah, Eugene Denteh, Tanner Muturi, Anthony Dontoh, Yaw Adu-Gyamfi, and Armstrong Aboah. Task-Specific Dual-Model Framework for Comprehensive Traffic Safety Video Description and Analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 5384–5392, October 2025. Open-access proceedings.arxiv.org/abs/2510.11907
  13. [29]Tanner W. Muturi, Blessing Agyei Kyem, Joshua Kofi Asamoah, Neema Jakisa Owor, Richard Dzinyela, Andrews Danyo, Yaw Adu-Gyamfi, and Armstrong Aboah. Prompt-Guided Spatial Understanding with RGB-D Transformers for Fine-Grained Object Relation Reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 5339–5347, October 2025. Open-access proceedings.arxiv.org/abs/2510.11996
  14. [30]Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. arXiv preprint arXiv:1703.06907, 2017. 10.48550/arXiv.1703.06907.arxiv.org/abs/1703.06907
  15. [31]Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning From Simulated and Unsupervised Images Through Adversarial Training. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. Open-access proceedings.arxiv.org/abs/1612.07828
  16. [32]Zhun Zhong, Liang Zheng, Zhedong Zheng, Shaozi Li, and Yi Yang. Camera Style Adaptation for Person Re-Identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. Open-access proceedings.arxiv.org/abs/1711.10295
  17. [33]Slawomir Bak, Peter Carr, and Jean-Francois Lalonde. Domain Adaptation through Synthesis for Unsupervised Person Re-identification. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018. Open-access proceedings.arxiv.org/abs/1804.10094
  18. [34]Tianyu Zhang, Lingxi Xie, Longhui Wei, Zijie Zhuang, Yongfei Zhang, Bo Li, and Qi Tian. UnrealPerson: An Adaptive Pipeline Towards Costless Person Re-Identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11506–11515, June 2021. Open-access proceedings.arxiv.org/abs/2012.04268
  19. [35]Zhiyin Shao, Xinyu Zhang, Changxing Ding, Jian Wang, and Jingdong Wang. Unified Pre-Training with Pseudo Texts for Text-To-Image Person Re-Identification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11174–11184, October 2023. Open-access proceedings.arxiv.org/abs/2309.01420
  20. [36]Min Cao, Yuxin Lu, Ziyin Zeng, Dong Yi, Jinqiao Wang, and Mang Ye. An Empirical Study of Validating Synthetic Data for Text-Based Person Retrieval. arXiv preprint arXiv:2503.22171, 2025. 10.48550/arXiv.2503.22171.arxiv.org/abs/2503.22171
  21. [37]Shuyu Yang, Yinan Zhou, Zhedong Zheng, Yaxiong Wang, Li Zhu, and Yujiao Wu. Towards Unified Text-based Person Retrieval: A Large-scale Multi-Attribute and Language Search Benchmark. In Proceedings of the 31st ACM International Conference on Multimedia, pages 4492–4501, 2023. 10.1145/3581783.3611709.arxiv.org/abs/2306.02898
  22. [38]Blessing Agyei Kyem, Joshua Kofi Asamoah, Ying Huang, and Armstrong Aboah. Weather-Adaptive Synthetic Data Generation for Enhanced Power Line Inspection Using StarGAN. IEEE Access, 12:193882–193901, 2024. 10.1109/ACCESS.2024.3520120.
  23. [39]Dai Quoc Tran, Armstrong Aboah, Yuntae Jeon, Maged Shoman, Minsoo Park, and Seunghee Park. Low-Light Image Enhancement Framework for Improved Object Detection in Fisheye Lens Datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 7056–7065, June 2024. Open-access proceedings.
  24. [40]Neema Jakisa Owor, Joshua Kofi Asamoah, Tanner Wambui Muturi, Anneliese Jakisa Owor, Blessing Agyei Kyem, Andrews Danyo, Yaw Adu-Gyamfi, and Armstrong Aboah. A Unified Detection Pipeline for Robust Object Detection in Fisheye-Based Traffic Surveillance. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 5314–5321, October 2025. Open-access proceedings.arxiv.org/abs/2510.20016
  25. [41]Armstrong Aboah, Bin Wang, Ulas Bagci, and Yaw Adu-Gyamfi. Real-Time Multi-Class Helmet Violation Detection Using Few-Shot Data Sampling Technique and YOLOv8. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 5350–5358, June 2023. Open-access proceedings.arxiv.org/abs/2304.08256
  26. [42]Blessing Agyei Kyem, Eugene Kofi Okrah Denteh, Joshua Kofi Asamoah, Kenneth Adomako Tutu, and Armstrong Aboah. Advancing Pavement Distress Detection in Developing Countries: A Novel Deep Learning Approach with Locally-Collected Datasets. arXiv preprint arXiv:2408.05649, 2024. 10.48550/arXiv.2408.05649.arxiv.org/abs/2408.05649
  27. [43]Blessing Agyei Kyem, Joshua Kofi Asamoah, Anthony Dontoh, Andrews Danyo, Eugene Denteh, and Armstrong Aboah. PaveSync: A Unified and Comprehensive Vision Dataset for Pavement Distress Analysis and Classification. arXiv preprint arXiv:2512.20011, 2025. 10.48550/arXiv.2512.20011.DOI
  28. [44]Blessing Agyei Kyem, Eugene Denteh, Joshua Kofi Asamoah, Denver Tolliver, and Armstrong Aboah. PaveCap: A Multimodal Framework for Comprehensive Pavement Condition Assessment with Dense Captioning and PCI Estimation. Journal of Transportation Engineering, Part B: Pavements, 152(2), 2026. 10.1061/JPEODX.PVENG-1945.DOI
  29. [45]Blessing Agyei Kyem, Joshua Kofi Asamoah, Anthony Dontoh, and Armstrong Aboah. Vision-Language Foundation Models for Comprehensive Automated Pavement Condition Assessment. arXiv preprint arXiv:2604.08212, 2026. 10.48550/arXiv.2604.08212.arxiv.org/abs/2604.08212
  30. [46]Blessing Agyei Kyem, Joshua Kofi Asamoah, Eugene Denteh, and Armstrong Aboah. Multi-Task Crack Foundation Model for Engineering-Reliable Crack Representation and Topology Preservation in Civil Infrastructure. arXiv preprint arXiv:2606.05641, 2026. 10.48550/arXiv.2606.05641.arxiv.org/abs/2606.05641
  31. [47]Eugene Denteh, Andrews Danyo, Joshua Kofi Asamoah, Blessing Agyei Kyem, Twitchell Addai, and Armstrong Aboah. Integrating Travel Behavior Forecasting and Generative Modeling for Predicting Future Urban Mobility and Spatial Transformations. arXiv preprint arXiv:2503.21158, 2025. 10.48550/arXiv.2503.21158.arxiv.org/abs/2503.21158
  32. [48]Eugene Kofi Okrah Denteh, Andrews Danyo, Joshua Kofi Asamoah, Blessing Agyei Kyem, and Armstrong Aboah. Demographics-Informed Neural Network for Multi-Modal Spatiotemporal Forecasting of Urban Growth and Travel Patterns Using Satellite Imagery. arXiv preprint arXiv:2506.12456, 2025. 10.48550/arXiv.2506.12456.arxiv.org/abs/2506.12456
  33. [49]Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hanneke Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust Fine-Tuning of Zero-Shot Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7959–7971, 2022. Open-access proceedings.arxiv.org/abs/2109.01903

Paper details

vlnc:W_xsf995qu
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents