WorksComputer Science
StateBridge: Phase- and Track-Grounded Reasoning for Synthetic-to-Real Traffic Video Captioning and VQA
Abstract
Traffic safety videos provide rich evidence for understanding interactions between pedestrians and vehicles, but synthetic-to-real evaluation makes this setting challenging because models must learn from simulated scenes and generalize to real-world traffic. In this work, we present StateBridge, a traffic video-language framework designed to operate under this synthetic-only training protocol. Our approach uses phase-aware clip construction, pedestrian and vehicle track summaries, and structured prompts to ground vision-language models in stable traffic evidence rather than appearance alone. We fine-tune public vision-language models on synthetic captioning and visual question answering examples, then combine their outputs through fact-locked caption fusion and evidence-diverse VQA consensus to improve consistency across actor roles, behavioral phases, and question types. StateBridge achieves an official S2 score of 55.4679 on AI City Challenge 2026 Track 2 using only synthetic training data. These results demonstrate the effectiveness of structured evidence transfer for traffic safety captioning and visual question answering.
Abstract. Traffic safety videos provide rich evidence for understanding interactions between pedestrians and vehicles, but synthetic-to-real evaluation makes this setting challenging because models must learn from simulated scenes and generalize to real-world traffic. In this work, we present StateBridge, a traffic video-language framework designed to
Introduction
Traffic safety video understanding is an important step toward safer and more interpretable intelligent transportation systems [32]. Unlike generic video captioning, this task does not ask a model to produce a short scene description; it requires the model to explain a safety-critical pedestrian-vehicle interaction over time, including the actors’ locations, their attention, their motion, and the influence of the surrounding road environment on the event. Figure 1 illustrates this multi-view, phase-structured setting. AI City Challenge 2026 Track 2 increases the difficulty of this problem by restricting all adaptation to synthetic Digital Twin videos, while evaluation is conducted on real-world WTS traffic footage [8, 77].
Figure 1. Multi-view SynWTS sample across five behavioral phases: pre-recognition, recognition, judgment, action, and avoidance. The top row shows the overhead view, and the bottom row shows the vehicle view. Red bounding boxes indicate the pedestrian, while yellow bounding boxes indicate the vehicle. The figure also labels the phases as pre-recognition, recognition, judgment, action, and avoidance; the annotation categories are appearance, location, attention, behavior, and context attributes. Caption pedestrian: The pedestrian, a male in his 30s with a height of 170 cm, was wearing a black shirt and light blue jeans. He was walking across a multi-lane road with two-way traffic. The pedestrian’s body was oriented perpendicular to the vehicle, and he had a moderate relative distance from it. Despite the approaching vehicle, he seemed to be focused on crossing the road. He continued walking across the road without losing his balance or falling. The event took place on a clear day with bright lighting, dry road surface conditions, and light traffic volume. The road was level and made of asphalt. There were landscaped traffic islands and a pedestrian crossing near the pedestrian. Caption vehicle: The vehicle was approaching the pedestrian from the right and was at a moderate distance from him. Within the vehicle’s field of view, the pedestrian was visible. The vehicle took action and avoided a collision by slowing down as the pedestrian crossed the road. The environmental conditions indicated that the pedestrian was a male in his 30s with a height of 170 cm. He was wearing a black shirt as upper body clothing and light blue jeans as lower body clothing. The weather was clear with bright lighting. The road surface conditions were dry and level, with asphalt as the road surface type. The traffic volume was light on this multi-lane road, which had two-way traffic. Landscaped traffic islands and a pedestrian crossing were also present.
This synthetic-to-real setting introduces a structured but uneven domain gap because some aspects of the scene transfer more reliably than others. In particular, the Digital Twin data [62] preserve road layout, camera viewpoint, event timing, actor tracks, and the annotation schema used to describe each scenario. These elements define the task semantics and can transfer from simulation to real video. However, the visual cues that reveal fine-grained human behavior, such as pose, gaze, gait, contact dynamics, illumination, clothing detail, and small objects, are far harder to simulate faithfully, so a model trained on this data can learn the correct output style from synthetic supervision while relying on appearance evidence that becomes unreliable at test time.
The challenge is amplified by the structured outputs required by the task. Each event is divided into behavioral phases, and each phase requires separate pedestrian- and vehicle-centric captions organized around location, attention, behavior, and context, while the same scenario is also evaluated through multiple-choice questions about attributes, geometry, visibility, action, and environment. These outputs are different projections of the same traffic state, so generating them independently can produce fluent captions that contradict VQA answers, disagree across actor roles, or alter a correct distance, speed, or phase-specific action. This suggests that synthetic-to-real traffic understanding should transfer structured evidence rather than simulator appearance. Although prior traffic VLM systems have improved high-resolution perception, spatial prompting, phase decomposition, and caption refinement in real-data settings [57, 64, 65, 88], the synthetic-only protocol requires a more conservative strategy that exploits stable phase, role, geometry, and checklist semantics while limiting unsupported language changes caused by uncertain visual cues.
Building on this view, we propose StateBridge, a synthetic-only framework for phase- and track-grounded traffic safety captioning and VQA. StateBridge aligns each scenario by behavioral phase and actor role, summarizes pedestrian and vehicle tracks in a normalized coordinate system, and uses these signals as stable evidence for synthetic-trained video-language adapters. Caption hypotheses are fused through a fact-locking gate that accepts a role-phase edit only when it preserves supported attributes, relations, actions, and context. VQA hypotheses are combined through evidence-diverse consensus to reduce sensitivity to a single model, prompt, or grounding view. This design connects captioning and VQA through the same structured traffic evidence without retrieving target captions or adapting on real-world training videos.
Based on this formulation, this paper makes the following contributions:
We formulate Track 2 as asymmetric structured-state transfer, where geometry, phase metadata, actor roles, and output semantics are more stable across domains than appearance and fine-grained human dynamics.
We introduce a phase- and track-grounded evidence interface that aligns multi-view clips with pedestrian and vehicle trajectories and expresses actor motion through resolution-independent temporal box summaries.
We propose fact-locked caption fusion, a conservative role-phase selection strategy that accepts caption edits only when they preserve supported attributes, relations, context, and phase-specific behavior.
We develop an evidence-diverse VQA consensus over synthetic-trained adapters and provide a complete synthetic-only pipeline for dataset construction, model adaptation, inference, and official submission.
Related Work
Traffic Video-Language Understanding
Video-language understanding has evolved from generic event description and question answering to safety-critical reasoning in road scenes. Early benchmarks such as MSVD, MSR-VTT, ActivityNet Captions, MovieQA, TGIF-QA, TVQA, and ActivityNet-QA established video captioning, temporal localization, and VideoQA as core tasks [16, 25, 33, 41, 78, 90, 100], while recent VLMs and video-language models expanded these capabilities through multimodal instruction tuning, high-resolution perception, memory compression, token reduction, and long-context video reasoning [19, 43, 51, 67]. Driving-language datasets then brought these ideas closer to transportation by grounding commands, explanations, causal traffic reasoning, autonomous-driving VQA, graph reasoning, map-language understanding, and reliability analysis in road scenes [1, 2, 14, 20, 29, 31, 49, 66, 73, 76, 89, 92]. WTS and AI City Track 2 further specialize this direction to pedestrian-centered traffic safety with synchronized views, phase-level captions, actor boxes, and structured safety attributes [4, 11, 21, 32, 38, 57, 61, 64, 65, 75, 87, 88, 93]. These works show the value of decomposition, view selection, high-resolution inputs, box-guided prompting, task-specific adaptation, and caption refinement, but they primarily study real-data WTS settings. StateBridge instead targets the stricter synthetic-to-real protocol by treating phase, role, geometry, and checklist semantics as the transferable interface.
Grounded Multimodal Reasoning
Grounded multimodal reasoning connects language to specific objects, regions, relations, and temporal moments rather than treating an image or video as a single global feature. Object detectors and trackers provide the spatial and temporal substrate for this connection [5, 15, 36, 59, 70, 74, 79, 105, 108], while scene-graph and compositional VQA datasets show that visual understanding often depends on attributes and relations between entities [24, 34, 56, 84]. Referring-expression and phrase-grounding methods further align text with localized regions through modular reasoning, joint image-text pretraining, and language-conditioned detection [7, 27, 28, 45, 52, 53, 55, 99]. Recent multimodal models extend this grounding interface to coordinate-aware dialogue, region instruction tuning, visual marks, promptable segmentation, reasoning segmentation, pixel-level grounding, and open-ended visual task decoding [12, 17, 30, 35, 39, 58, 63, 69, 83, 95, 98, 103, 109]. In videos, temporal grounding methods localize query-relevant spans or spatio-temporal tubes [3, 40, 94, 101, 104], which is especially relevant for traffic scenes where the same actor changes position and intent across phases. StateBridge follows this grounded reasoning line, but it uses the boxes and phase annotations provided by the challenge as a synthetic-to-real evidence interface, summarizing tracks into normalized temporal boxes instead of relying on a target-domain detector.
Structured and Consistent Caption Generation
Structured caption generation aims to produce fluent language while preserving the semantic fields that should appear in the output. Early image and video captioning models learned this mapping with recurrent decoders, attention, temporal structure, dense region descriptions, object-aware attention, transformer decoders, and vision-language pretraining [10, 18, 19, 23, 26, 43, 44, 47, 81, 82, 91, 97, 102]. Since fluent captions can still contain unsupported objects or relations, later work studied metric-aligned training, semantic proposition evaluation, explicit detection, novel-object coverage, scene-graph guidance, hallucination analysis, and reference-free image-text scoring [6, 9, 13, 22, 48, 50, 54, 60, 71, 72, 80, 85, 86, 96]. Controllable captioning further separates content planning from realization through diverse generation, coarse-to-fine decoding, adversarial evaluators, retrieval support, and robustness-aware captioning [37, 42, 46]. This structure is central in WTS-style traffic description, where captions follow recurring appearance, location, environment, attention, and action fields and where prior AI City systems have used segment decomposition, rewrite modules, high-resolution VLMs, and fact checking [61, 64, 65, 93]. StateBridge follows this line but applies structure conservatively: it treats the phase-role checklist as a fact-preserving interface and accepts only caption fields that remain consistent with the shared traffic state.
Task and Problem Setup
Task Definition
AI City Challenge 2026 Track 2 evaluates traffic safety understanding under a synthetic-to-real protocol [8]. The system is trained on labeled SynWTS Digital Twin videos and tested on real WTS videos without real-world supervision. Test inputs include videos, phase metadata, questions, answer options, and challenge-provided visual annotations when available, but no target captions, target answers, or additional manual target annotations. The required outputs are phase-level pedestrian and vehicle captions, plus multiple-choice VQA answers. Captions describe location, attention, behavior, and context for each actor, while VQA covers attributes, environment, geometry, visibility, action, and vehicle state. The official S2 score combines caption metrics with VQA accuracy.
Synthetic-to-Real Structured-State Transfer
Let denote the labeled synthetic source domain. A scenario contains synchronized videos , actor tracks , phase boundaries , phase-level role captions , and VQA pairs . At test time, the target domain provides real videos , phase metadata, questions, answer options, and challenge-provided visual annotations when available, but no caption or answer supervision. For phase and role , we define the task-relevant traffic state
corresponding to location, attention, behavior, and shared context. Captioning verbalizes , while each VQA item queries one or more of its coordinates. Let and denote the source and target video distributions conditioned on the same traffic state, let denote the task outputs, and let and denote the source and target annotation schemas. The domain shift is asymmetric:
Equation (2) states that the rendering of a safety state changes across domains, but the meanings of phase, role, geometry, tracks, and checklist attributes remain stable. StateBridge therefore transfers structured evidence rather than simulator appearance: at inference time it estimates support for from the official target inputs and constrains captions and VQA by that evidence, without using target labels, target captions, target answers, or real WTS training videos.
Method
Framework Overview
StateBridge converts each traffic scenario into a phase- and role-grounded evidence representation before generating captions or VQA answers. Figure 2 summarizes the full pipeline from synthetic evidence construction to official caption and VQA outputs. This design follows the formulation in Eq. (1): instead of asking a VLM to transfer simulator appearance directly, we expose the stable elements of the event, namely phase, view, actor role, track geometry, and task schema. For a phase and request (a caption request or a VQA item), StateBridge constructs

Figure 2. Overview of the proposed StateBridge framework.
where are the phase-aligned video views, denotes normalized pedestrian and vehicle track summaries, and is the deterministic evidence-construction function. A set of synthetic-trained VLM adapters maps this evidence to candidate outputs. The final prediction is produced by a task-specific controller,
where denotes the controller for request type , and denotes facts supported by prompts, tracks, and model outputs. The controller is deliberately asymmetric: free-form captions are selected through fact-preserving gates, while multiple-choice VQA answers are aggregated through evidence-diverse consensus.
Phase-Aware Evidence Construction
The first stage aligns every training and test example with the temporal unit used by the challenge. For each event and normal_trimmed scenario, we build a manifest containing the split, scenario identifier, view, video path, phase interval, caption fields, VQA items, and available tracks. If phase spans timestamps and is the full video from view , each view is clipped as
where is the number of available views and denotes temporal clipping. Phase-specific captions and action or relation questions use these aligned clips, while environment questions retain broader scene context when the question is not tied to a short behavior interval. Each instruction states the scenario type, view, phase label, task, and expected output format. This construction makes the training instance match the evaluation instance and reduces leakage from neighboring behavioral phases.
Track-Grounded Prompting
Actor tracks provide a domain-stable cue because relative position and motion are less sensitive to rendering style than texture or illumination. Let be the pixel-coordinate box for a track summary position in a frame of width and height . We convert each pedestrian and vehicle box to a shared 0–1000 frame grid,
For each actor and phase, the prompt contains the first, middle, last, and mean boxes, together with the actor role, view, phase, and number of tracked frames. This compact summary exposes coarse displacement, scale change, and relative geometry without depending on video resolution. We also construct optional rendered overlays and pedestrian-vehicle interaction crops. These views can make small actors easier to inspect, but they can also shift the visual distribution, so StateBridge treats them as alternative evidence streams rather than replacing the raw phase clips.
Synthetic VLM Adaptation
We adapt public multimodal instruction models with LoRA using LLaMA-Factory [106]. Our main backbone is Qwen3-VL-8B-Instruct [12]. Each supervised row follows one of two explicit schemas. A caption row contains the phase-aligned videos, phase label, view name, track summaries, and an instruction to return a JSON object with caption_pedestrian and caption_vehicle. A VQA row contains the relevant videos, question, answer options, phase or environment scope, track summaries when available, and an instruction to return only the correct option letter.
We train three task mixtures from these rows. The caption-only adapter uses only caption rows, the VQA-only adapter uses only VQA rows, and the joint adapter mixes both tasks so that captioning and question answering share the same traffic vocabulary. We also train evidence variants with the same targets but different inputs: a text-box variant serializes normalized actor boxes in the prompt, an overlay variant renders colored actor tracks on the clips, a crop variant adds pedestrian-vehicle interaction crops, and a joint-visual variant combines raw clips with the derived grounding views. Each adapter minimizes the autoregressive supervised fine-tuning loss
where is the number of supervised examples, is the target token sequence for example , and is its evidence prompt. The parameters are frozen base-model weights, and is the trainable low-rank update. All examples in Eq. (7) come from the synthetic source domain; target videos are used only at inference.
Fact-Locked Caption Fusion
Caption candidates differ in useful but risky ways. One model may produce a better action phrase, while another may better preserve weather, road type, or distance. Because global rewriting can introduce unsupported attributes, we fuse captions at the role-phase field level. Let be the trusted field for phase and role , and let be alternative fields from other candidate systems. For each candidate, we compute
where measures token overlap, measures four-gram preservation, rewards agreement with supported facts, rewards compatible length, penalizes repetition, contradictions, artifacts, and incomplete endings, and are fixed nonnegative weights. A candidate can replace the trusted field only when it improves the proxy score and passes a fact lock:
where is the required score margin, is the minimum token-overlap threshold, and is a binary consistency test. In our implementation, is instantiated with checklist-term rewards for pedestrian, vehicle, weather, road, traffic, speed, and line-of-sight evidence; rejects incomplete endings, prompt artifacts, role contradictions, and impossible speed-collision combinations. The lock also protects identity, clothing, relative position, distance, obstacle, road, and weather terms when they are present. The final selected policy accepts fields whose length is between 0.84 and 1.22 times the trusted field and whose proxy score is no worse than 0.05 below the trusted field, so style can improve only when the edit remains supported by the shared traffic state.
Evidence-Diverse VQA Consensus
VQA has a different error profile from captioning because the output space is a small set of option letters. Some questions are easiest from scene context, such as weather or road type, while others depend on phase-local geometry, visibility, or actor motion. We therefore combine adapters that see complementary evidence rather than relying on one prompt or one visual view. For a question with option set , adapter predicts . The final answer is
where is the indicator function and is a fixed reliability weight for adapter . The candidate pool contains synthetic-trained streams from the joint Qwen3-VL adapter, VQA-specialized Qwen3-VL adapters, and grounding variants built from text-box summaries, rendered box overlays, and interaction crops. In the final synthetic-only submission, we set for every included stream and use the strongest joint Qwen3-VL adapter as the tie fallback. This rule lets an overlay or crop model influence questions where it agrees with other evidence, but prevents a weaker grounding view from changing a high-confidence answer by itself.
Experiments
Dataset and Protocol
We train only on the official SynWTS synthetic train and validation splits. The preprocessing pipeline indexes event and normal_trimmed scenarios, cuts phase-aligned clips, attaches available actor tracks, and exports instruction examples for captioning and VQA. This produces 14,198 supervised rows, including 2,465 phase-level caption examples and 11,733 VQA examples. At test time, the real WTS videos are used only for inference; no target labels, captions, answers, or manual annotations are added. We generate 798 caption rows and 19,624 VQA rows, then assemble them into the official scenario-level submission files.
We report the official server metrics: BLEU-4 [60], METEOR [13], ROUGE-L [50], CIDEr [80], VQA accuracy, and the aggregate S2 score. Target captions and answers are hidden, so all model choices, thresholds, and fusion rules are fixed without target-label feedback.
Implementation Details
We train the main video-language adapters from Qwen3-VL-8B-Instruct using the Qwen3-VL chat template. Each adapter is optimized for three epochs with LoRA rank 64, LoRA scale 128, dropout 0.05, bfloat16 precision, gradient checkpointing, maximum sequence length 8,192, per-device batch size one, gradient accumulation of eight, learning rate , cosine decay, and warmup ratio 0.03. The base model weights are frozen, and every prompt-target pair is generated automatically from SynWTS annotations.
We train three task mixtures. The caption-only adapter receives phase clips and predicts a JSON object with caption_pedestrian and caption_vehicle. The VQA-only adapter receives the question, answer options, relevant clips, and track evidence, and it is trained to output only the correct option letter. The joint adapter mixes both formats so that a single model learns the shared traffic vocabulary. We also train evidence variants that use the same targets but change the visual grounding: one uses only textual box summaries, one adds rendered track overlays, one adds pedestrian-vehicle interaction crops, and one combines raw clips with the visual grounding views.
For caption fusion, we assemble candidate captions from the Qwen3-VL adapters and additional synthetic-trained Qwen3.5-9B [68] and InternVL3-8B [107] adapters. All candidates must be valid JSON with the two required role fields. If a candidate ends with an incomplete phrase, we apply a deterministic completion repair; if it drops a required role, changes the phase structure, or contradicts locked facts, it is rejected. For VQA, each candidate system produces one option letter per question. The final answer is selected by the equal-vote consensus in Eq. (10), with the strongest joint Qwen3-VL adapter used only to break ties.
The systems compared in the tables differ only in how they combine these candidate outputs. The direct caption system uses one complete caption source without field replacement. Strict fact-locked fusion requires a larger score margin and length match, so it accepts fewer edits. Completion repair first removes incomplete final clauses before applying the same lock. Role-specific fusion applies separate pedestrian and vehicle edit budgets. Fixed-budget fusion accepts only the 45 or 48 highest-scoring locked fields. The final StateBridge system uses the selected fact-lock policy in § 4 without a fixed edit budget.
Official Challenge Results
Table ?? reports official challenge-server results for the main synthetic-only submissions. All rows use the same evidence-diverse VQA consensus, so the differences isolate caption fusion behavior. The final StateBridge system obtains the best S2 among these verified variants by changing only fields that remain consistent with the supported traffic facts. Our approach ranks 8th in the Track 2 of the AI City Competition as shown in Table 2.
| Rank | Team ID | Score |
| 1 | 47 | 60.0853 |
| 2 | 24 | 57.3307 |
| 3 | 266 | 56.7949 |
| 4 | 257 | 55.5901 |
| 5 | 71 | 55.5768 |
| 6 | 30 | 55.5728 |
| 7 | 122 | 55.5123 |
| 8 | 127 | 55.4679 |
| 9 | 176 | 55.4659 |
| 10 | 97 | 55.3330 |
Table 2. Official leaderboard ranking snapshot. Only rank, team ID, and aggregate score are shown.
Ablation Study
Table 3 summarizes verified submissions used to study the main components. The joint Qwen3-VL baseline uses one adapter for both captioning and VQA. The VQA consensus row keeps the direct caption source but replaces single-model VQA with the voting rule in Eq. (10). The remaining rows compare increasingly constrained caption fusion strategies. These comparisons show that small, fact-locked edits are more reliable than broad rewriting or aggressive field replacement.
Qualitative Results
Figure 3 shows StateBridge predictions on a real public-test sequence. The model produces role-specific captions and VQA answers across the behavioral phases while using only synthetic training data. Figure 4 illustrates the caption gate: an incomplete direct output is repaired by removing the dangling phrase while preserving supported facts. Figure 5 illustrates the VQA consensus rule on a validation question, where complementary evidence streams are combined before the final option is selected. Together, the examples show that StateBridge relies on phase, role, track geometry, and checklist semantics rather than unconstrained text generation.

Figure 3. Real public-test qualitative predictions from scenario 20230922_7_SN1_T1. The top row shows an overhead view and the bottom row shows the vehicle view across the five behavioral phases. The captions and VQA answers are StateBridge predictions produced at inference time.

Figure 4. Qualitative example of fact-locked caption refinement. A direct VLM output preserves most traffic facts but ends with an incomplete phrase, while StateBridge keeps the supported geometry, visibility, speed, pedestrian appearance, and road context and removes the dangling ending without freely rewriting the event.

Figure 5. Synthetic-validation illustration of evidence-diverse VQA consensus. The individual predictors are trained on synthetic data but receive different evidence constructions, including phase clips, textual track summaries, visual grounding views, and joint caption–VQA adaptation. Ground truth is shown only because this is a validation example.
| Variant | S2 | BLEU-4 | METEOR | ROUGE-L | CIDEr | Acc. |
| Joint Qwen3-VL baseline | 55.0643 | 0.2442 | 0.4169 | 0.4284 | 0.6590 | 81.2424 |
| VQA consensus only | 55.3330 | 0.2470 | 0.4199 | 0.4331 | 0.7502 | 81.2930 |
| Rewrite model with fact locking | 55.4235 | 0.2423 | 0.4235 | 0.4432 | 0.7321 | 81.2930 |
| Fact-locked fusion with completion repair | 55.4574 | 0.2436 | 0.4243 | 0.4446 | 0.7243 | 81.2930 |
| Fixed-budget fact-locked fusion, 45 fields | 55.4657 | 0.2438 | 0.4247 | 0.4446 | 0.7243 | 81.2930 |
| Role-specific fact-locked fusion | 55.4659 | 0.2438 | 0.4247 | 0.4446 | 0.7244 | 81.2930 |
| StateBridge: conservative fact-locked fusion | 55.4679 | 0.2438 | 0.4248 | 0.4446 | 0.7252 | 81.2930 |
Table 3. Ablation submissions for VQA consensus and caption fusion. Higher is better for all metrics.
Discussion
The results show that the main difficulty is not only generating fluent text, but deciding when a generated field should be trusted. In Table 1, fact-locked fusion improves METEOR and ROUGE-L over the direct caption source, although it can reduce CIDEr when an accepted edit changes frequent reference-style phrases. This tradeoff explains why StateBridge uses field-level replacement rather than full-caption rewriting. The VQA consensus is similarly conservative: it changes answers only when complementary synthetic-trained streams provide enough support. These observations support the central design choice of StateBridge, namely to transfer phase, role, geometry, and checklist structure across domains while limiting unsupported changes in the final submission.
| Method | S2 | BLEU-4 | METEOR | ROUGE-L | CIDEr | Acc. |
| Direct caption + VQA consensus | 55.3330 | 0.2470 | 0.4199 | 0.4331 | 0.7502 | 81.2930 |
| Strict fact-locked fusion | 55.4614 | 0.2436 | 0.4244 | 0.4446 | 0.7243 | 81.2930 |
| Fact-locked fusion with completion repair | 55.4574 | 0.2436 | 0.4243 | 0.4446 | 0.7243 | 81.2930 |
| Role-specific fact-locked fusion | 55.4659 | 0.2438 | 0.4247 | 0.4446 | 0.7244 | 81.2930 |
| Fixed-budget fact-locked fusion, 45 fields | 55.4657 | 0.2438 | 0.4247 | 0.4446 | 0.7243 | 81.2930 |
| Fixed-budget fact-locked fusion, 48 fields | 55.4657 | 0.2438 | 0.4247 | 0.4446 | 0.7244 | 81.2930 |
| Rewrite model with fact locking | 55.4235 | 0.2423 | 0.4235 | 0.4432 | 0.7321 | 81.2930 |
| StateBridge: conservative fact-locked fusion | 55.4679 | 0.2438 | 0.4248 | 0.4446 | 0.7252 | 81.2930 |
Table 1. Official challenge-server results for synthetic-only submissions. Higher is better for all metrics.
Conclusion
In this work, we present StateBridge, a synthetic-only framework designed to address the domain gap in traffic safety captioning and visual question answering, where models are trained on Digital Twin videos but evaluated on real-world traffic scenes. Our approach combines three core components that operate together: phase-aware evidence construction, track-grounded VLM adaptation, and conservative output fusion through fact-locked caption selection and evidence-diverse VQA consensus.
When evaluated on AI City Challenge 2026 Track 2, StateBridge achieved an official S2 score of 55.4679 under the synthetic-to-real protocol. These results show the value of transferring stable traffic structure, including phase, role, geometry, and checklist semantics, rather than relying only on appearance-level adaptation. Overall, StateBridge provides a practical foundation for traffic video-language understanding when real-world supervision is unavailable.
References cc / paper _ files / paper / 2023 / hash / 3ef61f7e4afacf9a2c5b71c726172b86 - Abstract-Conference.html
References
- [1]Aboah, A., Adu-Gyamfi, Y.: Smartphone-based pavement roughness estimation using deep learning with entity embedding. Advances in Data Science and Adaptive Analysis 12(03n04), 2050007 (2020). https://doi.org/10.1142/S2424922X20500072
- [2]Aboah, A., Adu-Gyamfi, Y., Gursoy, S.V., Merickel, J., Rizzo, M., Sharma, A.: Driver maneuver detection and analysis using time series segmentation and classification. Journal of Transportation Engineering, Part A: Systems 149(3), 04022157 (2023). https://doi.org/10.1061/JTEPBS.TEENG-7312, https://ascelibrary.org/doi/abs/10.1061/JTEPBS.TEENG-7312
- [3]Aboah, A., Bagci, U., Musah, A.R., Ofor, W.N.J., Adu-Gyamfi, Y.: Deepsegmenter: Temporal action localization for detecting anomalies in untrimmed naturalistic driving videos. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 5359–5365 (2023). https://doi.org/10.1109/CVPRW59228.2023.00565
- [4]Aboah, A., Shoman, M., Mandal, V., Davami, S., Adu-Gyamfi, Y., Sharma, A.: A vision-based system for traffic anomaly detection using deep learning and decision trees. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 4202–4207 (2021). https://doi.org/10.1109/CVPRW53098.2021.00475
- [5]Aboah, A., Wang, B., Bagci, U., Adu-Gyamfi, Y.: Real-time multi-class helmet violation detection using few-shot data sampling technique and yolov8. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 5350–5358 (2023). https://doi.org/10.1109/CVPRW59228.2023.00564
- [7]Agyei Kyem, B., Asamoah, J.K., Denteh, E., Danyo, A., Aboah, A.: Self-supervised multi-scale transformer with attention-guided fusion for efficient crack detection. Automation in Construction 181, 106591 (2026). https://doi.org/10.1016/j.autcon.2025.106591, https://www.sciencedirect.com/science/article/pii/S0926580525006314
- [8]AI City Challenge: AI City Challenge 2026 Track 2: Transportation safety understanding and captioning (sim2real). https://www.aicitychallenge.org/2026-track2/ (2026)
- [10]Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6077–6086 (2018), https://openaccess.thecvf.com/content_cvpr_2018/html/Anderson_Bottom-Up_and_Top-Down_CVPR_2018_paper.htmlarxiv.org/abs/1707.07998
- [11]Asamoah, J.K., Kyem, B.A., Aboah, A.: Querymotion: A modular framework for natural language-based vehicle retrieval via attribute and motion reasoning. IEEE Transactions on Intelligent Transportation Systems 27(8), 10069–10084 (2026). https://doi.org/10.1109/TITS.2026.3680396
- [12]Bai, S., Cai, Y., Chen, R., Chen, K., et al.: Qwen3-VL technical report. arXiv preprint arXiv:2511.21631 (2025)arxiv.org/abs/2511.21631
- [13]Banerjee, S., Lavie, A.: Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization (2005)
- [14]Cao, X., Zhou, T., Ma, Y., Ye, W., Cui, C., Tang, K., Cao, Z., Liang, K., Wang, Z., Rehg, J.M., Zheng, C.: MAPLM: A real-world large-scale vision-language benchmark for map and traffic scene understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21819–21830 (2024), https://openaccess.thecvf.com/content/CVPR2024/html/Cao_MAPLM_A_Real-World_Large-Scale_Vision-Language_Benchmark_for_Map_and_Traffic_CVPR_2024_paper.htmlDOI
- [15]Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: European Conference on Computer Vision. pp. 213–229 (2020), https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/832_ECCV_2020_paper.phparxiv.org/abs/2005.12872
- [16]Chen, D.L., Dolan, W.B.: Collecting highly parallel data for paraphrase evaluation. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. pp. 190–200 (2011), https://aclanthology.org/P11-1020/
- [17]Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., Zhao, R.: Shikra: Unleashing multimodal LLM’s referential dialogue magic. arXiv preprint arXiv:2306.15195 (2023), https://arxiv.org/abs/2306.15195
- [18]Cornia, M., Stefanini, M., Baraldi, L., Cucchiara, R.: Meshed-memory transformer for image captioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10578–10587 (2020), https://openaccess.thecvf.com/content_CVPR_2020/html/Cornia_Meshed-Memory_Transformer_for_Image_Captioning_CVPR_2020_paper.htmlarxiv.org/abs/1912.08226
- [19]Dai, W., Li, J., Li, D., Tiong, A.M.H., Zhao, J., Wang, W., Li, B., Fung, P., Hoi, S.C.H.: InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In: Advances in Neural Information Processing Systems. vol. 36 (2023), https://proceedings.neurips.cc/paper_files/paper/2023/hash/9a6a435e75419a836fe47ab6793623e6-Abstract-Conference.html
- [20]Deruyttere, T., Vandenhende, S., Grujicic, D., Van Gool, L., Moens, M.F.: Talk2Car: Taking control of your self-driving car. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. pp. 2088–2098 (2019), https://github.com/talk2car/Talk2CarDOI
- [21]Ha, V.T.D., Tran, T.H., Dong, G.T., Chu, N.C., Vu, H., Nguyen, T.C.: Domain-aware enhancements to vision-language models for urban traffic safety question answering. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. pp. 5425–5433 (2025), https://openaccess.thecvf.com/content/ICCV2025W/AICity/html/Ha_Domain-Aware_Enhancements_to_Vision-Language_Models_for_Urban_Traffic_Safety_Question_ICCVW_2025_paper.htmlDOI
- [23]Huang, L., Wang, W., Chen, J., Wei, X.Y.: Attention on attention for image captioning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4634–4643 (2019), https://openaccess.thecvf.com/content_ICCV_2019/html/Huang_Attention_on_Attention_for_Image_Captioning_ICCV_2019_paper.htmlarxiv.org/abs/1908.06954
- [24]Hudson, D.A., Manning, C.D.: GQA: A new dataset for real-world visual reasoning and compositional question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6700–6709 (2019). https://doi.org/10.1109/CVPR.2019.00686, https://m lanthology.org/cvpr/2019/hudson2019cvpr-gqa/
- [25]Jang, Y., Song, Y., Yu, Y., Kim, Y., Kim, G.: TGIF-QA: Toward spatio-temporal reasoning in visual question answering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1359–1367 (2017), https://openaccess.thecvf.com/content_cvpr_2017/html/Jang_TGIF-QA_Toward_Spatio-Temporal_CVPR_2017_paper.htmlarxiv.org/abs/1704.04497
- [26]Johnson, J., Karpathy, A., Fei-Fei, L.: Densecap: Fully convolutional localization networks for dense captioning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4565–4574 (2016), https://openaccess.thecvf.com/content_cvpr_2016/html/Johnson_DenseCap_Fully_Convolutional_CVPR_2016_paper.htmlarxiv.org/abs/1511.07571
- [27]Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., Carion, N.: MDETR: Modulated detection for end-to-end multi-modal understanding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1780–1790 (2021), https://openaccess.thecvf.com/content/ICCV2021/html/Kamath_MDETR_-_Modulated_Detection_for_End-to-End_Multi-Modal_Understanding_ICCV_2021_paper.htmlarxiv.org/abs/2104.12763
- [28]Kazemzadeh, S., Ordonez, V., Matten, M., Berg, T.L.: ReferItGame: Referring to objects in photographs of natural scenes. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. pp. 787–798 (2014), https://www.cs.rice.edu/~vo9/referit/DOI
- [29]Kim, J., Rohrbach, A., Darrell, T., Canny, J., Akata, Z.: Textual explanations for self-driving vehicles. In: Proceedings of the European Conference on Computer Vision (2018), https://github.com/JinkyuKimUCB/BDD-X-datasetarxiv.org/abs/1807.11546
- [30]Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollár, P., Girshick, R.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4015–4026 (2023), https://openaccess.thecvf.com/content/ICCV2023/html/Kirillov_Segment_Anything_ICCV_2023_paper.htmlarxiv.org/abs/2304.02643
- [31]Kong, F., Li, Y., Chen, W., Min, C., Li, Y., Gao, Z., Li, H., Guo, Z., Sun, H.: VLR-Driver: Large vision-language-reasoning models for embodied autonomous driving. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 26966–26976 (2025), https://openaccess.thecvf.com/content/ICCV2025/html/Kong_VLR-Driver_Large_Vision-Language-Reasoning_Models_for_Embodied_Autonomous_Driving_ICCV_2025_paper.htmlDOI
- [32]Kong, Q., Kawana, Y., Saini, R., Kumar, A., Pan, J., Gu, T., Ozao, Y., Opra, B., Anastasiu, D.C., Sato, Y., Kobori, N.: WTS: A pedestrian-centric traffic video dataset for fine-grained spatial-temporal understanding. In: European Conference on Computer Vision. pp. 1–18. Springer (2024)
- [33]Krishna, R., Hata, K., Ren, F., Fei-Fei, L., Niebles, J.C.: Dense-captioning events in videos. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 706–715 (2017), https://openaccess.thecvf.com/content_iccv_2017/html/Krishna_Dense-Captioning_Events_in_ICCV_2017_paper.htmlarxiv.org/abs/1705.00754
- [34]Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., Bernstein, M.S., Fei-Fei, L.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision 123, 32–73 (2017). https://doi.org/10.1007/s11263-016-0981-7, https://link.springer.com/article/10.1007/s11263-016-0981-7
- [35]Kyem, B.A., Asamoah, J.K., Aboah, A.: Context-CrackNet: A context-aware framework for precise segmentation of tiny cracks in pavement images. Construction and Building Materials 484, 141583 (2025). https://doi.org/10.1016/j.conbuildmat.2025.141583
- [36]Kyem, B.A., Asamoah, J.K., Huang, Y., Aboah, A.: Weather-adaptive synthetic data generation for enhanced power line inspection using StarGAN. IEEE Access 12, 193882–193901 (2024). https://doi.org/10.1109/ACCESS.2024.3520120
- [37]Kyem, B.A., Denteh, E., Asamoah, J.K., Tolliver, D., Aboah, A.: PaveCap: A multimodal framework for comprehensive pavement condition assessment with dense captioning and PCI estimation. Journal of Transportation Engineering, Part B: Pavements 152(2), 04026010 (2026). https://doi.org/10.1061/JPEODX.PVENG-1945
- [38]Kyem, B.A., Owor, N.J., Danyo, A., Asamoah, J.K., Denteh, E., Muturi, T., Dontoh, A., Adu-Gyamfi, Y., Aboah, A.: Task-specific dual-model framework for comprehensive traffic safety video description and analysis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. pp. 5384–5392 (2025), https://openaccess.thecvf.com/content/ICCV2025W/AICity/html/Kyem_Task-Specific_Dual-Model_Framework_for_Comprehensive_Traffic_Safety_Video_Description_and_ICCVW_2025_paper.htmlarxiv.org/abs/2510.11907
- [39]Lai, X., Tian, Z., Chen, Y., Li, Y., Yuan, Y., Liu, S., Jia, J.: LISA: Reasoning segmentation via large language model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9579–9589 (2024), https://openaccess.thecvf.com/content/CVPR2024/html/Lai_LISA_Reasoning_Segmentation_via_Large_Language_Model_CVPR_2024_paper.htmlarxiv.org/abs/2308.00692
- [40]Lei, J., Berg, T.L., Bansal, M.: Detecting moments and highlights in videos via natural language queries. In: Advances in Neural Information Processing Systems. vol. 34, pp. 11846–11858 (2021), https://proceedings.neurips.cc/paper_files/paper/2021/hash/62e0973455fd26eb03e91d5741a4a3bb-Abstract.html
- [41]Lei, J., Yu, L., Bansal, M., Berg, T.L.: TVQA: Localized, compositional video question answering. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 1369–1379 (2018). https://doi.org/10.18653/v1/D18-1167, https://aclanthology.org/D18-1167/
- [42]Li, J., Vo, D.M., Sugimoto, A., Nakayama, H.: EVCap: Retrieval-augmented image captioning with external visual-name memory for open-world comprehension. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13733–13742 (2024), https://openaccess.thecvf.com/ content/CVPR2024/html/Li_EVCap_Retrieval-Augmented_Image_Captioning_with_External_Visual-Linguistic_Memory_for_Open-World_CVPR_2024_paper.htmlarxiv.org/abs/2311.15879
- [43]Li, J., Li, D., Savarese, S., Hoi, S.C.H.: BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In: International Conference on Machine Learning. pp. 19730–19742 (2023), https://proceedings.mlr.press/v202/li23q.htmlarxiv.org/abs/2301.12597
- [44]Li, J., Li, D., Xiong, C., Hoi, S.: BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: Proceedings of the 39th International Conference on Machine Learning. pp. 12888–12900 (2022), https://proceedings.mlr.press/v162/li22n.htmlarxiv.org/abs/2201.12086
- [45]Li, L.H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.N., Chang, K.W., Gao, J.: Grounded language-image pre-training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10965–10975 (2022), https://github.com/microsoft/GLIParxiv.org/abs/2112.03857
- [46]Li, W., Li, J., Ramos, R., Tang, R., Elliott, D.: Understanding retrieval robustness for retrieval-augmented image captioning. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. pp. 9285–9299 (2024). https://doi.org/10.18653/v1/2024.acl-long.503, https://aclanthology.org/2024.acl-long.503
- [47]Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., Choi, Y., Gao, J.: Oscar: Object-semantics aligned pre-training for vision-language tasks. In: European Conference on Computer Vision. pp. 121–137 (2020), https://arxiv.org/abs/2004.06165
- [49]Li, Y., Tian, M., Lin, Z., Zhu, J., Zhu, D., Liu, H., Zhang, Y., Xiong, Z., Zhao, X.: Fine-grained evaluation of large vision-language models in autonomous driving. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9431–9442 (2025), https://openaccess.thecvf.com/content/ICCV2025/html/Li_Fine-Grained_Evaluation_of_Large_Vision-Language_Models_in_Autonomous_Driving_ICCV_2025_paper.htmlarxiv.org/abs/2503.21505
- [50]Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Text Summarization Branches Out (2004)
- [51]Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: Advances in Neural Information Processing Systems. vol. 36, pp. 34892–34916 (2023), https://proceedings.neurips.cc/paper_files/paper/2023/hash/6dcf277ea32ce3288914f369fe6de0-Abstract-Conference.html
- [52]Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., Zhu, J., Zhang, L.: Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 (2023), https://arxiv.org/abs/2303.05499
- [53]Lu, J., Batra, D., Parikh, D., Lee, S.: ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: Advances in Neural Information Processing Systems. vol. 32 (2019), https://proceedings.neurips.cc/paper_files/paper/2019/hash/c74d97b01eae257e44aa9d5bade97baf-Abstract.html
- [55]Minderer, M., Gritsenko, A., Stone, A., Neumann, M., Weissenborn, D., Dosovitskiy, A., Mahendran, A., Arnab, A., Dehghan, M., Shen, Z., Wang, X., Zhai, X., Kipf, T., Houlsby, N.: Simple open-vocabulary object detection with vision transformers. In: European Conference on Computer Vision. pp. 728–755 (2022), https://arxiv.org/abs/2205.06230
- [56]Muturi, T.W., Kyem, B.A., Asamoah, J.K., Owor, N.J., Dzinlyela, R., Danyo, A., Adu-Gyamfi, Y., Aboah, A.: Prompt-guided spatial understanding with rgb-d transformers for fine-grained object relation reasoning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops. pp. 5339–5347 (October 2025)arxiv.org/abs/2510.11996
- [57]Nguyen-Nhu, T.A., Minh, T.D.H., To-Thanh, D., Le-Gia, P., Vo-Lan, T., Nguyen, T.H.: STER-VLM: Spatio-temporal with enhanced reference vision-language models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. pp. 5516–5525 (2025)
- [58]Owor, N.J., Adu-Gyamfi, Y., Aboah, A., Amo-Boateng, M.: PaveSAM – segment anything for pavement distress. Road Materials and Pavement Design 26(3), 593–617 (2025). https://doi.org/10.1080/14680629.2024.2374863
- [59]Owor, N.J., Asamoah, J.K., Muturi, T.W., Owor, A.J., Kyem, B.A., Danyo, A., Adu-Gyamfi, Y., Aboah, A.: A unified detection pipeline for robust object detection in fisheye-based traffic surveillance. In: 2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). pp. 5314–5321 (2025). https://doi.org/10.1109/ICCVW69036.2025.00554
- [60]Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: A method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (2002)
- [61]Park, B., Yang, W., Yuan, S., Anwar, S.M., Marsic, I.: TrafficVILA: A multimodal framework for traffic safety description and analysis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. pp. 5460–5468 (2025), https://openaccess.thecvf.com/content/ICCV2025W/AICity/html/Park_TrafficVILA_A_Multimodal_Framework_for_Traffic_Safety_Description_and_Analysis_ICCVW_2025_paper.htmlDOI
- [62]Patil, D., Kachhadiya, R., Vattunone, A., Liang, H., Li, J., Wang, Y., Tang, Z., Kumar, A., Kong, Q., Anastasiu, D.C.: SynWTS: Multi-view digital-twin dataset for synthetic-to-real traffic safety understanding. In: ECCV Workshops. Malm"o, Sweden (9 2026)
- [63]Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., Wei, F.: Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824 (2023), https://arxiv.org/abs/2306.14824
- [64]Pervaiz, Z.B., Cha, S., Gulati, R., Jhuria, M., Praveen, V., Kornuta, T., Lu, Y., Murali, V.: TrafficVILA: Scaling vision-language models to high-resolution video understanding for traffic safety analysis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. pp. 5366–5373 (2025)
- [65]Phimsiri, S., Sunpawatr, S., Cherdchusakulchai, R., Kiawjak, P., Tosawadi, T., Tungjitnob, S., Trairattanapa, V., Vatathanavaro, S., Kudisthalert, W., Utintu, C., Saetan, W., Kongsawat, N., Borisutsawat, P., Mahakijdechachai, K., Su-Inn, N., Thamwawtthana, E., Sutchitcaya, V.: TrafficInternVL: Spatially-guided fine-tuning with caption refinement for fine-grained traffic safety captioning and visual question answering. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. pp. 5358–5365 (2025)
- [66]Qian, T., Chen, J., Zhuo, L., Jiao, Y., Jiang, Y.G.: NuScenes-QA: A multi-modal visual question answering benchmark for autonomous driving scenario. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 4542–4550 (2024). https://doi.org/10.1609/aaai.v38i5.28253, https://ojs.aaai.org/index.php/AAAI/article/view/28253
- [67]Qwen Team: Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923 (2025), https://arxiv.org/abs/2502.13923
- [68]Qwen Team: Qwen3.5-9B. https://huggingface.co/Qwen/Qwen3.5-9B (2026)
- [69]Rasheed, H., Maaz, M., Mulpillapy, S.S., Shaker, A., Khan, S., Cholakkal, H., Anwer, R.M., Xing, E., Yang, M.H., Khan, F.S.: GLaMM: Pixel grounding large multimodal model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13009–13018 (2024), https://dblp.org/rec/conf/cvpr/Rasheed0MSOCAX024.htmlarxiv.org/abs/2311.03356
- [70]Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems. vol. 28 (2015), https://proceedings.neurips.cc/paper/2015/hash/14bfa6bb14875e45bba028a21ed38046-Abstract.htmlarxiv.org/abs/1506.01497
- [73]Shoman, M., Aboah, A., Adu-Gyamfi, Y.: Deep learning framework for predicting bus delays on multiple routes using heterogenous datasets. Journal of Big Data Analytics in Transportation 2(3), 275–290 (2020). https://doi.org/10.1007/s42421-020-00031-y
- [74]Shoman, M., Aboah, A., Morehead, A., Duan, Y., Daud, A., Adu-Gyamfi, Y.: A region-based deep learning approach to automated retail checkout. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 3209–3214 (2022). https://doi.org/10.1109/CVPRW56347.2022.00362
- [75]Shoman, M., Wang, D., Aboah, A., Abdel-Aty, M.A.: Enhancing traffic safety with parallel dense video captioning for end-to-end event analysis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 7125–7133 (2024). https://doi.org/10.1109/CVPRW63382.2024.00707, https://mlanthology.org/cvprw/2024/shoman2024cvprw-enhancing/
- [76]Sima, C., Renz, K., Chitta, K., Chen, L., Zhang, H., Xie, C., Beisswenger, J., Luo, P., Geiger, A., Li, H.: DriveLMM: Driving with graph visual question answering. In: European Conference on Computer Vision. pp. 256–274 (2024). https://doi.org/10.1007/978-3-031-72943-0_15, https://eccv.ecva.net/virtual/2024/poster/130
- [77]Tang, Z., Wang, S., Anastasiu, D.C., Chang, M.C., et al.: The 10th AI City Challenge. In: ECCV Workshops. Malm"o, Sweden (2026)arxiv.org/abs/2608.17044
- [78]Tapaswi, M., Zhu, Y., Stiefelhagen, R., Torralba, A., Urtasun, R., Fidler, S.: MovieQA: Understanding stories in movies through question-answering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4631–4640 (2016), https://openaccess.thecvf.com/content_cvpr_2016/html/Tapaswi_MovieQA_Understanding_Stories_CVPR_2016_paper.htmlarxiv.org/abs/1512.02902
- [79]Tran, D.Q., Aboah, A., Jeon, Y., Shoman, M., Park, M., Park, S.: Low-light image enhancement framework for improved object detection in fisheye lens datasets. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. pp. 7056–7065 (June 2024)arxiv.org/abs/2404.10078
- [80]Vedantam, R., Lawrence Zitnick, C., Parikh, D.: Cider: Consensus-based image description evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015)arxiv.org/abs/1411.5726
- [81]Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T., Saenko, K.: Sequence to sequence: Video to text. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 4534–4542 (2015), https://openaccess.thecvf.com/content_iccv_2015/html/Venugopalan_Sequence_to_Sequence_ICCV_2015_paper.htmlarxiv.org/abs/1505.00487
- [82]Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3156–3164 (2015), https://openaccess.thecvf.com/content_cvpr_2015/html/Vinyals_Show_and_Tell_2015_CVPR_paper.html
- [83]Wang, B., Aboah, A., Zhang, Z., Pan, H., Bagci, U.: Gazesam: Interactive image segmentation with eye gaze and segment anything model. In: Madu Blessing, A., Wu, J., Zanca, D., Krupinski, E., Kashyap, S., Karargyris, A. (eds.) Proceedings of The 2nd Gaze Meets ML workshop. Proceedings of Machine Learning Research, vol. 226, pp. 254–265. PMLR (16 Dec 2024), https://proceedings.mlr.press/v226/wang24a.html
- [84]Wang, B., Pan, H., Aboah, A., Zhang, Z., Keles, E., Torigian, D., Turkbey, B., Krupinski, E., Udupa, J., Bagci, U.: GazeGnn: A gaze-guided graph neural network for chest x-ray classification. In: 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 2183–2192 (2024). https://doi.org/10.1109/WACV57701.2024.00219
- [87]Wang, S., Anastasiu, D.C., Tang, Z., Chang, M.C., Yao, Y., Zheng, L., Rahman, M.S., Arya, M.S., Sharma, A., Chakraborty, P., et al.: The 8th AI City Challenge. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 7261–7272 (2024), https://openaccess.thecvf.com/content/CVPR2024W/AICity/html/Wang_The_8th_AI_City_Challenge_CVPRW_2024_paper.htmlarxiv.org/abs/2404.09432
- [88]Wu, H.F., Yang, Y.T., Chen, Y.T., Chou, I.F.: TrafficInternVL: Understanding traffic scenarios with vision-language models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. pp. 5288–5295 (2025)DOI
- [89]Xie, S., Kong, L., Dong, Y., Sima, C., Zhang, W., Chen, Q.A., Liu, Z., Pan, L.: Are VLMs ready for autonomous driving? an empirical study from the reliability, data and metric perspectives. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 6585–6597 (2025), https://openaccess.thecvf.com/content/ICCV2025/html/Xie_Are_VLMs_Ready_for_Autonomous_Driving_An_Empirical_Study_from_ICCV_2025_paper.htmlarxiv.org/abs/2501.04003
- [90]Xu, J., Mei, T., Yao, T., Rui, Y.: MSR-VTT: A large video description dataset for bridging video and language. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5288–5296 (2016), https://openaccess.thecvf.com/content_cvpr_2016/html/Xu_MSR-VTT_A_Large_CVPR_2016_paper.html
- [91]Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., Bengio, Y.: Show, attend and tell: Neural image caption generation with visual attention. In: Proceedings of the 32nd International Conference on Machine Learning. pp. 2048–2057 (2015), https://proceedings.mlr.press/v37/xuc15.htmlarxiv.org/abs/1502.03044
- [92]Xu, L., Huang, H., Liu, J.: SUTD-TrafficQA: A question answering benchmark and an efficient network for video reasoning over traffic events. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9873–9883 (2021). https://doi.org/10.1109/CVPR46437.2021.00975, https://ieeexplore.ieee.org/document/9578653
- [93]Xuan, K.T., Nguyen, K.N., Ngo, B.H., Xuan, V.D., An, M.H., Dinh, Q.V.: Divide and conquer boosting for enhanced traffic safety description and analysis with large vision language model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 7046–7055 (2024)DOI
- [94]Yang, A., Miech, A., Sivic, J., Laptev, I., Schmid, C.: TubeDETR: Spatio-temporal video grounding with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16442–16453 (2022), https://openaccess.thecvf.com/content/CVPR2022/html/Yang_TubeDETR_Spatio-Temporal_Video_Grounding_With_Transformers_CVPR_2022_paper.htmlarxiv.org/abs/2203.16434
- [95]Yang, J., Zhang, H., Li, F., Zou, X., Li, C., Gao, J.: Set-of-mark prompting unleashes extraordinary visual grounding in GPT-4V. arXiv preprint arXiv:2310.11441 (2023), https://github.com/microsoft/SoMarxiv.org/abs/2310.11441
- [97]Yao, L., Torabi, A., Cho, K., Ballas, N., Pal, C., Larochelle, H., Courville, A.: Describing videos by exploiting temporal structure. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 4507–4515 (2015), https://openaccess.thecvf.com/content_iccv_2015/html/Yao_Describing_Videos_by_ICCV_2015_paper.htmlarxiv.org/abs/1502.08029
- [98]You, H., Zhang, H., Gan, Z., Du, X., Zhang, B., Wang, Z., Cao, L., Chang, S.F., Yang, Y.: Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704 (2023), https://machinelearning.apple.com/research/ferretarxiv.org/abs/2310.07704
- [99]Yu, L., Lin, Z., Shen, X., Yang, J., Lu, X., Bansal, M., Berg, T.L.: MAttNet: Modular attention network for referring expression comprehension. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1307–1315 (2018), https://openaccess.thecvf.com/content_cvpr_2018/html/Yu_MAtNet_Modular_Attention_CVPR_2018_paper.html
- [100]Yu, Z., Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y., Tao, D.: ActivityNet-QA: A dataset for understanding complex web videos via question answering. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 9127–9134 (2019). https://doi.org/10.1609/aaai.v33i01.33019127, https://ojs.aaai.org/index.php/AAAI/article/view/4946
- [101]Zhang, H., Sun, A., Jing, W., Zhou, J.T.: Span-based localizing network for natural language video localization. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 6543–6554 (2020). https://doi.org/10.18653/v1/2020.acl-main.585, https://aclanthology.org/2020.acl-main.585/
- [102]Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., Gao, J.: VinVL: Revisiting visual representations in vision-language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5579–5588 (2021), https://openaccess.thecvf.com/content/CVPR2021/html/Zhang_VinVL_Revisiting_Visual_Representations_in_Vision-Language_Models_CVPR_2021_paper.htmlarxiv.org/abs/2101.00529
- [103]Zhang, S., Sun, P., Chen, S., Xiao, M., Shao, W., Zhang, W., Lin, K., Luo, P.: GPT4RoI: Instruction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601 (2023), https://arxiv.org/abs/2307.03601
- [104]Zhang, S., Peng, H., Fu, J., Luo, J.: Learning 2d temporal adjacent networks for moment localization with natural language. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 12870–12877 (2020). https://doi.org/10.1609/aaai.v34i07.6984, https://ojs.aaai.org/index.php/AAAI/article/view/6984
- [105]Zhang, Y., Sun, P., Jiang, Y., Yu, D., Weng, F., Yuan, Z., Luo, P., Liu, W., Wang, X.: ByteTrack: Multi-object tracking by associating every detection box. In: European Conference on Computer Vision. pp. 1–21 (2022). https://doi.org/10.1007/978-3-031-20047-2_1, https://mlanthology.org/eccv/2022/zhang2022eccv-bytetrack/
- [106]Zheng, Y., Zhang, R., Zhang, J., Ye, Y., Luo, Z., Ma, Z., Ma, Y.: Llama-factory: Unified efficient fine-tuning of large language models. https://github.com/hiyouga/LLaMA-Factory (2024)
- [107]Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y., Su, W., Shao, J., Gao, Z., Cui, E., Wang, X., Cao, Y., Liu, Y., Wei, X., Zhang, H., Wang, H., Xu, W., Li, H., Wang, J., Deng, N., Li, S., He, Y., Jiang, T., Luo, J., Wang, Y., He, C., Shi, B., Zhang, X., Shao, W., He, J., Xiong, Y., Qu, W., Sun, P., Jiao, P., Lv, H., Wu, L., Zhang, K., Deng, H., Ge, J., Chen, K., Wang, L., Dou, M., Lu, L., Zhu, X., Lu, T., Lin, D., Qiao, Y., Dai, J., Wang, W.: InternVL3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479 (2025), https://arxiv.org/abs/2504.10479
- [108]Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable DETR: Deformable transformers for end-to-end object detection. In: International Conference on Learning Representations (2021), https://openreview.net/forum?id=gZ9hCDWe6kearxiv.org/abs/2010.04159
- [109]Zou, X., Yang, J., Zhang, H., Li, F., Li, L., Wang, J., Wang, L., Gao, J., Lee, Y.J.: Segment everything everywhere all at once. In: Advances in Neural Information Processing Systems. vol. 36 (2023), https://papers.neurips.arxiv.org/abs/2304.06718
