WorksComputer Science
Selective Cooperative Perception via Blind Spot Reasoning and Agentic Dialogue with Vision-Language Models
Abstract
Real-time cooperative perception (CP) in multi-agent traffic environments is highly resource-intensive, which posing scalability challenges for large-scale deployment. Efficiency can be significantly improved by selectively performing CP only in safety-critical scenarios and for key occluding objects. Vision-Language Models (VLMs) are well-suited for complex reasoning in dynamic and uncertain environments, making them a promising tool for this selective CP task. This study proposes a VLM-driven agentic framework for blind spot reasoning and dialogue-based CP. A mixed-model architecture is designed that integrates conventional object detection for stable perception, vision-language reasoning for identifying impactful occlusions, and a physics-based geometric module to align reasoning with real-world visibility constraints. The framework also incorporates a Chain-of-Thought (CoT) prompting mechanism to enhance interpretability and intent-aware decision-making. We collect real-world data from connected vehicles operating in urban traffic and compare the driving-critical blind spot identification and dialogue performance of four state-of-the-art VLMs. Experimental results demonstrate that our method achieves over 80% accuracy in driving-critical blind spot detection and over 90% success in dialogue-driven perception sharing. This work provides a scalable, interpretable, and bandwidth-efficient alternative to conventional CP pipelines, and offers new insights into agentic communication design for future connected traffic systems.
Introduction
Cooperative Perception (CP) plays a crucial role in enhancing vehicle situational awareness by enabling multiple agents to share perceptual information [3]. This capability allows vehicles to overcome limitations caused by occlusions, narrow sensor fields-of-view, or hardware failures. In complex traffic environments, especially those involving high-speed interactions or obstructed views, CP has the potential to significantly improve driving safety and decision-making accuracy by complementing ego perception with external observations [17].
However, in traffic scenarios with numerous connected agents, the cost of exchanging perceptual information can quickly become prohibitively high. Traditional fusion-based CP methods are commonly categorized into early, intermediate, and late fusion [3]. Early fusion aggregates raw sensor data such as point clouds or images, providing rich scene representations but requiring extremely high bandwidth and strict temporal synchronization [1]. Intermediate fusion exchanges processed features or object-level proposals, reducing bandwidth but remaining highly sensitive to calibration errors (Xu, Xiang, Tu, Xia, Yang and Ma, 2022). Late fusion transmits only final perception outputs, minimizing data size but offering limited robustness to occlusions, as missing information cannot be recovered. Although effective in controlled settings, these approaches face increasing limitations in real-world traffic, where many agents must continuously exchange high-frequency perception updates with multiple peers. The resulting bandwidth and computational burden hinders the scalability of conventional fusion-based CP. Recent works have highlighted these challenges and emphasized the need for more bandwidth-efficient CP alternatives [3, 2]. This motivates a shift toward selective CP. In most driving scenarios, vehicles do not require external information and can operate safely based on ego perception alone (Ma, Li, Long, Zhou, Liang, Li, Yu and Li, 2026). CP becomes essential only in certain contexts such as during occlusions, complex interactions, or ambiguous intent prediction [20, 11]. Therefore, if a system can reason about when CP is necessary, from whom, and what specific information is needed, it becomes possible to perform targeted CP with minimal cost. Prior studies have proposed early forms of this concept, including rule-based filters and context-aware triggers [2]. However, due to the dynamic and highly contextual nature of traffic, such rule-based approaches struggle to generalize across diverse real-world scenarios [19].
To enable reasonable judgment of when CP is necessary across diverse environments, the perceptual and reasoning capabilities of VLMs provide a promising solution. These VLMs can reason over complex logical relationships by integrating scene understanding, driving intentions, object-level dynamics, and commonsense priors [8] (Xu, Hu, Zhang, Meyer, Mustikovela, Srinivasa, Wolff and Huang 2024). Such reasoning capabilities are well-suited to determining when CP is required and formulating targeted queries for specific perceptual information. Recent work has explored using VLMs in agent communication, intention prediction, and modular perception–planning loops [19, 18, 4] (Du, Lyu, Shan, Qi, Zhang, Chen, Peng, Shu, Lee, Dariush et al. 2024). These studies demonstrate the potential of VLMs to perform decision-making with reduced communication overhead, but further investigation is needed to align these capabilities with safety-critical constraints.
Despite their promise, current VLM-based approaches face two major limitations that hinder their direct application to safety-critical perception. First, their visual understanding often suffers from perceptual hallucination, where the model misidentifies or fabricates objects due to weak grounding between visual tokens and real-world geometry [7]. This undermines reliability in tasks such as occlusion reasoning and obstacle detection, VLMs exhibit limited capability in spatial and numerical reasoning. They struggle to maintain geometric consistency and accurately infer object positions or trajectories in metric space, as their reasoning is primarily symbolic rather than physics-grounded. Consequently, purely VLM-driven pipelines (Du et al., 2024) remain prone to high spatial errors (often exceeding 50%) and hallucinations [4] in coordinates understanding or trajectories prediction tasks, posing unacceptable risks in real-world driving.
To address these limitations, this study proposes a hybrid framework that combines the robustness of conventional computer vision models, the reasoning and interaction capability of VLMs, and the consistency of physics-based geometric models. In this framework, a computer vision-based detector extracts object types and positions with stable numerical accuracy, avoiding hallucinations. VLMs are then used to identify which occlusions constitute driving-critical blind spots based on driving intention. Finally, an agentic dialogue mechanism is used to selectively query peer agents for supplementary perception in these blind spots. The contributions of this work are twofold: (1) a VLM-based blind-spot reasoning module that incorporates driving intent to identify high-impact occlusions, together with a physics-grounded geometric visibility verifier that aligns VLM outputs with real-world occlusion constraints; and (2) an agentic dialogue framework that enables selective CP with minimal communication cost while maintaining safety in critical scenarios.
The remainder of this paper is organized as follows. § 2 presents the proposed methodology, including the mixed-model architecture for blind-spot reasoning and dialogue-based selective CP, as well as its integration with geometric modeling. § 3 details the experimental validation using real-world data collected from connected vehicles operating in urban traffic scenarios. § 4 concludes the study and outlines future research directions.
Methodology
Framework Overview
Figure 1 illustrates the proposed blind spot reasoning and selective CP scenario. We consider a discrete-time mixed traffic environment over time steps , composed of both connected and non-connected agents (shown as colored and white participants in Figure 1, respectively). Let denote the set of all agents at time , partitioned into connected vehicles (CVs) and non-connected vehicles (NVs) , i.e., . Each connected agent is equipped with V2X communication capabilities, enabling information sharing and dialogue-based coordination with other CVs. For each CV , we define four key components of its internal state: (1) Static attribute , which includes appearance features (e.g., color), semantic class (e.g., light-duty vehicle, pedestrian), and physical dimensions (, ); (2) Real-time state , including the center position (, ), velocity , and heading angle ; (3) Driving intention , expressed in natural language (e.g., "turning left at the intersection" as in the case of CV in the figure); (4) Perception observation , referring to onboard sensor input. In this study, we focus specifically on the front-view image , which serves as the basis for blind spot analysis. This formulation naturally generalizes to other camera views. Under normal conditions, at each time step, each CV agent makes driving decisions based solely on its own onboard perception observation . Building on this baseline, this study introduces a supplementary module to assess whether the agent’s individual forward-facing perception is sufficient for safe and informed decision-making. If not, the system actively supplements the perception through communication with other connected agents.

Figure 1. Overview of the Agentic AI-based Selective Cooperative Perception Framework
Therefore, the goal of this study is for each CV to autonomously reason about driving-critical blind spots that may affect its driving safety and, when necessary, selectively acquire information from other CVs with minimal communication overhead. To balance perception completeness and communication efficiency, we propose an agentic framework for selective CP. From an information-sharing perspective, the static attribute and real-time state are lightweight and periodically broadcast to all CVs to maintain a common situational understanding. In contrast, the perception observation is high-volume and therefore unsuitable for continuous transmission under limited bandwidth.
Each CV independently determines whether CP is necessary. Step 1 involves driving-critical blind spot inference using its front-view image and driving intention . We adopt a Mix-Model Modular Blind Spot Reasoning architecture that integrates conventional computer vision accuracy with the reasoning capacity of VLMs. The reasoning process is structured as a CoT composed of scene understanding, object detection, intent interpretation, and blind spot analysis, allowing modular and robust inference of potential occlusions.
If no blind spot is identified or if the detected blind spot is deemed non-critical for the current driving task, the agent skips Step 2 and Step 3 and directly proceeds with its own perception output as sufficient. In most scenarios, this allows the system to operate without requiring additional communication or dialogue, thereby preserving bandwidth.
If a driving-critical blind spot is identified, the system proceeds to Step 2: Dialogue Target Matching. Using the shared and of all CVs in the detection area, the ego agent identifies which peer is most likely to observe the occluded region. To avoid hallucinations commonly seen in direct VLM-based spatial reasoning, the ego agent uses a physics-based geometric model that determine the most appropriate dialogue target based on relative positions and orientations.
Finally, Step 3: Perception Completion via Dialogue is initiated. The ego CV engages in targeted, context-aware dialogue with the selected peer rather than broadcasting its entire perception. This agentic communication framework enables CVs to cooperatively resolve perception gaps in a highly efficient manner, preserving bandwidth while maintaining safety and fluid traffic operations.
This targeted and conditional communication approach not only eliminates potential risks but also reduces the overhead compared to general CP frameworks by engaging in information exchange only when necessary.
Mix-Model Agentic Blind Spot Reasoning
This section introduces how a CV agent determines whether CP is needed. We propose a mixed-model structure that incorporates a designed CoT agentic mechanism, as illustrated in Figure 2. At each time step , the input to agent includes its front-view image captured by its onboard camera, and its driving intention , represented as a high-level context prompt. The model outputs include three key elements: whether a driving-critical blind spot is detected (if not, Steps 2 and 3 are skipped), the corresponding occluding object, and the reason for the driving-critical blind spot determination.

Figure 2. Modular Chain-of-Thought reasoning structure.
The CoT agentic mechanism includes three main steps: (1) scene understanding, (2) detection of blind spots within the scene (without considering driving intent), and (3) identification of driving-critical blind spots by incorporating the ego vehicle’s intention. All three steps are carried out using the VLMs. However, prior studies and our experiments consistently show that VLMs perform poorly in spatial and geometric reasoning when applied directly to raw images, due to their weak geometric grounding [14, 13]. To improve reliability, we first apply a conventional deep learning (DL) model to detect and classify objects in the scene, as these models offer more stable performance in object recognition. The detected objects are then provided to the VLMs to assist in higher-level reasoning and judgment. Since the DL model outputs precise numerical data (e.g., bounding boxes, categories), which are not ideal for direct VLM input, we introduce a semantic conversion process termed semanticization. This step transforms raw detection outputs into structured natural language descriptions that are more compatible with VLM input. The specific functions of each module are described below:
At first, the agent performs a Driving Scene Description step, extracting high-level contextual features from the input front-view image , such as road layout, intersection presence, lane boundaries, traffic signals, and weather conditions. These environmental cues provide essential context for interpreting visual content and anticipating traffic dynamics.
Next, the Object Detection module identifies all visible entities, including surrounding vehicles, pedestrians, and static infrastructure. While VLMs can be used for object-level semantic parsing, they often suffer from hallucinations or unreliable bounding box precision in complex traffic scenes. To ensure accurate localization, we first apply a DL–based object detector (e.g., YOLO [9]) to obtain structured bounding boxes for all objects in the front-view image . For each detected object , the model outputs its class label , a unique identifier, and a rectangular bounding region , where each corner point is defined in image-space proportion (normalized coordinates), as illustrated in Figure 2.
These bounding boxes are difficult for the VLMs to interpret directly in numerical form. To bridge this gap, we perform a semanticization step that transforms raw visual outputs into discrete textual descriptions. While object categories are already suitable for textual understanding, the object position and size require semantic transformation.
We divide the image space into six semantic regions based on the center location of each bounding box: Lateral-Left, Lateral-Front, Lateral-Right, Upper-Left, Upper-Center, and Upper-Right, as shown in Figure 3. This partitioning scheme is inspired by the explainability-oriented design in Hou, Wang, Li and Dong [6].

Figure 3. Semantic Region Mapping of Image Coordinates
Object size is also mapped to semantic categories based on the proportion of the image occupied by its bounding box. Specifically, objects whose bounding area exceeds 30% of the total image are labeled as large, those below 5% as small, and all others as medium. These semantic descriptors—category, position, and size—serve as inputs to the VLMs to support more stable and interpretable reasoning.
Based on the results of scene understanding and object recognition, the Blind Spot Analysis module evaluates occlusion and visibility uncertainty by reasoning over the spatial layout of detected objects. The primary goal of this module is to identify physical obstructions—such as large vehicles blocking adjacent lanes—that may conceal other agents or infrastructure behind them. At this stage, the analysis is limited to visual occlusions and does not consider the ego vehicle’s driving intention.
This reasoning process is handled by the VLMs, which operates on the semantically structured object list. For each object, the system infers whether it causes a blind spot and, if so, describes the occlusion using a fixed output format: #ObjectID #ObjectType has a blind spot on its #Direction, saliency: #High/Medium/Low, reason: #Explanation.
To avoid logical entanglement and unnecessary complexity in spatial reasoning, we adopt a looped querying structure in practice. That is, the system iterates over all recognized objects individually and queries the VLMs to assess whether each object may be causing a blind spot. This modular reasoning structure allows for stable and interpretable blind spot detection across varying scene layouts.
The previous blind spot analysis does not take driving intention into account. As a result, it may identify occlusions that are irrelevant to the current driving task and do not require perception completion. For instance, during highway driving in a straight lane, a vehicle in the adjacent left lane may block visibility of vehicles further left. While such occlusions would be flagged by the earlier spatial reasoning module, they do not impact the safety of a vehicle whose intention is to maintain a straight trajectory.
To address this, the Potential Risk Analysis module integrates the inferred blind spots with the ego vehicle’s driving intention to determine whether additional perception is necessary. Specifically, if any identified blind spot—regardless of its saliency level—intersects with the intended path of travel, the agent flags a potential perception gap. In such cases, the system prepares to initiate targeted cooperative communication with relevant neighboring agents.
This layered reasoning pipeline ensures that only driving-critical blind spots trigger CP requests. It allows the system to minimize unnecessary information exchange, while maintaining safety guarantees through intent-aware filtering. The result is a more efficient and interpretable blind spot reasoning process that avoids full sensor fusion and focuses only on what matters for the current driving decision.
Dialogue Target Matching
Once a potential driving-critical blind spot is identified in the image space, the agent must determine which peer agent is most suitable to provide the missing information. This process consists of two tightly coupled steps: matching the occluded object in the perception image to a real-world connected agent based on spatial and visual cues, and initiating a structured dialogue to retrieve targeted information. To achieve this, we design a Dialogue Target Matching Model that leverages geometric reasoning and field-of-view (FOV) projection, followed by a dialogue direction identification step that formalizes the query content.
Given the ego agent with state , driving intention , static dimensions , and front-view camera parameters including horizontal FOV and image width , we consider a set of surrounding connected agents . Each peer agent has known center position , heading angle , and dimensions . The driving-critical blind zone is identified in the image plane as a horizontal interval , along with its relative occlusion direction.
To identify the most relevant peer for dialogue, we project each agent into the ego agent’s image frame. The bearing angle from the ego agent to is computed as
If , the peer agent is within the FOV. The corresponding horizontal pixel coordinate is
The apparent width of the agent in pixels is approximated as
Hence, the agent occupies an image interval . The one with the largest overlap with the blind zone is selected as the matched target:
To determine the direction from which information is needed, we compute the relative bearing from agent to the matched agent as
and define the relative azimuth with respect to ’s heading as
The query direction is then discretized as:
The final output of this module is the matched dialogue target and the corresponding directional query label QueryDirection. This mechanism enables the agent to initiate dialogue with a specific peer and request perception only in the occluded direction, ensuring efficient, interpretable, and low-bandwidth CP. The complete procedure is summarized in Algorithm 1.
Algorithm 1 Dialogue Target Matching and Direction Identification
Require: Ego agent with state , camera FOV , image width , blind zone , peer agents , dimensions
Initialize BestMatch , MaxOverlap
for each do
if then
Projected range
if then
BestMatch , MaxOverlap
end if
end if
end for
BestMatch
if then
QueryDirection "front"
else if then
QueryDirection "left"
else if then
QueryDirection "right"
else
QueryDirection "rear"
end if
return Matched agent , direction QueryDirection
Perception Completion via Dialogue
Once the dialogue target agent and the query direction QueryDirection are determined, the ego agent initiates a spatially grounded query to obtain information about the occluded region. The core objective of this dialogue is not to perform open-ended language generation, but to request only the missing perceptual content necessary to supplement the ego agent’s current blind-spot awareness.
The dialogue mechanism follows a lightweight, interpretable structure to ensure consistency across agents. The ego agent issues a directional query anchored to the occluded zone, while the responding agent returns a structured description of objects, distances, and attributes relevant to the requested direction. After receiving the response, the ego agent fuses the reported object information with its local driving intention to update risk assessment and decision-making. If the returned object intersects with or poses risk to the future trajectory, the ego agent adjusts its behavior (e.g., slowing down, yielding, or maintaining caution). This targeted interaction enables perception completion with minimal bandwidth and high interpretability.
Dialogue Prompt Format. The actual prompts used for querying and responding follow the patterns below.
Ego Agent Query Prompt
You are a connected vehicle agent. Based on the driving scene, you have detected
a potential blind spot in the [QueryDirection] direction. Describe what you observe
in that direction using concise object-level information.
Responding Agent Output Template
There is a [ObjectType] at [RelativeDistance] and its motion state is [AdditionalAttributes].
Here, [ObjectType] denotes object category (e.g., vehicle, pedestrian), [RelativeDistance] uses discretized spatial terms (e.g., close, medium, far), and [AdditionalAttributes] may include color, heading, or whether the object is stationary or accelerating.
This structured design provides a lightweight and interpretable mechanism for perception sharing, enabling the ego agent to supplement missing information in occluded regions while maintaining minimal communication overhead.
Experiments
This section evaluates the proposed agentic perception framework across three core aspects: the computational performance of the complete perception–decision loop, the accuracy of blind spot reasoning, and the communication efficiency of dialogue-based cooperation.
To collect real-world CP data, we utilize two CVs retrofitted by our lab, both based on identical platforms—2016 Lincoln MKZs upgraded by the University of Wisconsin-Madison. Each vehicle is equipped with a front-facing Mobileye camera with a 120° horizontal FOV and a Novatel OEM7 GPS navigation unit for accurate localization. All perception and communication modules are integrated on an industrial-grade onboard computer (AutonomouStuff Spectra). The vehicles operate in various urban scenarios, including signalized intersections, straight road segments, and complex work zones, as shown in Figure 4.

Figure 4. Data collection platform and connected vehicle experiments.
Due to the current latency limitations of LLMs, particularly during visual-language grounding and dialogue reasoning, the proposed framework is not yet deployable for real-time operations. As such, we conduct all experiments using a replay-based offline simulation pipeline. Logged sensor data and ground-truth annotations from real-world traffic scenes are replayed frame by frame, preserving vehicle states, perception inputs, and synchronized occlusion ground truth, while allowing agentic reasoning and dialogue generation to proceed under controlled conditions.
Computation Time Analysis
We conducted offline evaluations using a desktop equipped with an Intel i9 processor, 64 GB RAM, and an NVIDIA RTX 4090 GPU. The analysis focuses on three key stages in our pipeline: object detection, blind spot reasoning, and dialogue-based communication between agents.
Object detection was performed using the YOLOv8 model [9]. For blind spot reasoning, we evaluated four VLMs: DeepSeek-VL-7B, Claude-3, GPT-4o, and LLaVA-1.5-7B. Note that model initialization and loading times are excluded from our timing measurements, as these occur only once at startup and are not representative of per-frame inference latency.
Across 100 front-view frames, the computation time results are shown in Figure 5. The YOLOv8 model completed object detection within 70 ms per frame, providing fast and reliable detection results suitable for real-time applications. For blind spot reasoning, the average response time varied significantly across models. LLaVA-1.5-7B was the fastest, typically requiring about 5 seconds, though its reasoning quality was relatively low (as discussed in the following section). DeepSeek-VL-7B achieved a balanced performance, with a reasoning time around 7 seconds per frame. GPT-4o and Claude-3 both took longer, averaging between 12 to 15 seconds, but exhibited stronger reasoning performance.

Figure 5. Computation time comparison for object detection, blind spot reasoning, and dialogue generation across different models.
Dialogue generation time also differed across models. LLaVA-1.5-7B and DeepSeek-VL-7B required between 20 to 35 seconds to handle multi-turn conversational inputs, largely due to their internal handling of long dialogue histories. In contrast, GPT-4o and Claude-3 responded more efficiently, demonstrating shorter response times and better suitability for agentic dialogue-driven CP tasks.
Case Studies
We conducted two types of case studies. The first focuses on analyzing driving-critical blind spot detection by evaluating how a single vehicle’s front-view perception performs in identifying blind spots (Step 1). The second case study involves a scenario with two vehicles operating simultaneously. By comparing their individual perception results and cooperative dialogue, we analyze the full end-to-end process of driving-critical blind spot detection and correction.
For the Step 1 case study, we present two real-world scenes involving significant visual occlusions Ma et al. (2026). These cases are analyzed using the DeepSeek-VL-7B model, following object detection performed by a YOLO-based deep learning pipeline. This setup allows us to evaluate the VLM’s reasoning ability in the presence of occluding objects, based on semantically enriched scene input.
As shown in Figure 6, case 1 occurs at a signalized intersection where the ego vehicle intends to make a left turn. A vehicle from the opposite direction is also waiting to turn left, positioned directly in front of the ego vehicle’s line of sight. This obstructs the ego vehicle’s view of potential oncoming through traffic, making it difficult to determine a safe left-turn gap.

Figure 6. Case studies of blind spot reasoning
Without CP, such scenarios are both common and hazardous, as the ego vehicle also blocks the opposing vehicle’s view, potentially leading to a mutual deadlock. If the ego vehicle initiates a left turn without adequate visibility, it risks colliding with fast-approaching through traffic. In our framework, after detecting surrounding objects, the VLM evaluates each for its blind spot saliency—categorized as High, Medium, or Low—based on their position and properties in the image. It then determines whether the resulting occlusion could impact driving decisions and provides reasoning when relevant. In this case, the left-turning vehicle from the opposing direction was correctly identified as causing a High blind spot with potential risk. Other vehicles in the scene did not significantly obstruct the FOV and were correctly filtered out. Upon identifying the driving-critical blind spot, the ego agent initiated a dialogue with the occluding vehicle to query the visibility of the obstructed region. After receiving confirmation that no vehicles were present in the blind zone, the ego vehicle proceeded safely with the left turn. A detailed demonstration of the dynamic process is available at: https://youtu.be/pgDqkrz4QT4.
The second case in Figure 6 involves a forward-driving scenario where the ego vehicle travels through a work zone area. A construction truck is parked on the roadside, partially obstructing the view of potential workers or equipment behind it. Additionally, another vehicle in front of the ego vehicle blocks a significant portion of the forward view, making it difficult to assess the full driving environment. Without CP, the ego vehicle must slow down and continuously observe the scene to ensure safety, which still leaves residual risks due to limited visibility. Using our proposed method, the system first identifies all visible objects in the scene and evaluates their potential to cause visual occlusions. Blind spot saliency is assessed, and for each significant occlusion, the model determines whether it could impact the driving task. In this case, the construction truck and the leading vehicle were successfully identified as major driving-critical blind spot sources that could interfere with safe forward movement. In contrast, other roadside vehicles and moving objects were correctly judged as not posing significant risk at the moment and were excluded from blind spot warnings. This result demonstrates the model’s capability to filter non-threatening occlusions and focus CP efforts where most needed.
The second case study focuses on the full CP process across multiple connected vehicles. As shown in Figure 7, we conduct end-to-end evaluation using two real-world scenes where both vehicles are simultaneously active and influence each other’s observations. In Figure 7(a), Case 1 presents a typical “hidden pedestrian” scenario. The center of the figure shows the bird’s-eye view (BEV) of the environment, with the left and right panels displaying the perception results of the red and black vehicles, respectively. Although our framework only uses the front-facing view for blind spot detection, additional information from six-directional surround-view cameras and LiDAR is used to help interpret the scene during post-analysis (see Li, Cao, Liang, Li, Oh, Chen, Liang, Zhou, Ma, Liu et al. (2025)) for sensor configuration details). In this scene, a pedestrian is walking in front of the black vehicle while the red vehicle attempts to pass on its side. However, the red vehicle’s view of the pedestrian is fully occluded by the black vehicle, creating a high-risk situation, especially when the pedestrian is accompanied by a child or pet. (Note: All tests were conducted in a protected environment.)

Figure 7. (a) Case 1: a hidden pedestrian scenario where the red vehicle infers a driving-critical blind spot caused by the black vehicle and receives confirmation. (b) Case 2: a mutual occlusion scenario at an unprotected left-turn intersection, where both vehicles identify and resolve each other’s blind spots through bidirectional communication.
Using the proposed Agentic CP model, the red vehicle successfully identifies the black vehicle as a driving-critical occluding object and initiates a query: whether any vulnerable road users are present in front of it. In response, the black vehicle confirms the presence of a pedestrian currently crossing. This dialogue-based exchange allows the red vehicle to receive a proactive hazard alert and take precautionary actions, demonstrating the effectiveness of our model in blind spot correction through cooperative dialogue. The multi-source sensor data used in this scenario—including surround-view cameras and LiDAR—are processed following Li et al. (2025). The fused perception outputs are converted into structured textual descriptions and provided as input to each agent, enabling them to reason over a semantically enriched representation of the environment. This allows agents to engage in informed, multi-party dialogue to resolve blind spots and collaboratively complete the perception task.
Figure 7(b) illustrates Case 2, which shares the same intersection geometry as Figure 6 Case 1. In this scenario, two connected vehicles approach the same intersection from opposite directions, each attempting an unprotected left turn. From the individual perception views shown in the figure, it is evident that the two vehicles mutually occlude each other’s forward FOV. This mutual occlusion makes it difficult for either vehicle to determine whether an oncoming vehicle is proceeding straight through the intersection, often resulting in a deadlock where both vehicles hesitate to proceed.
In this case, both vehicles correctly identify that their line of sight is blocked by the opposing vehicle and determine that a driving-critical blind spot exists behind the occluding vehicle. Each initiates a query to the other, requesting information about the occluded zone.
The black vehicle replies that there is no approaching vehicle within close range behind it, indicating it is safe for the red vehicle to complete the left turn. In contrast, the red vehicle reports that a fast-approaching vehicle is coming from behind the black vehicle. It warns that initiating a left turn at that moment would pose a collision risk. This successful bidirectional exchange of critical occlusion information demonstrates the model’s capability to resolve mutual blind zones through targeted dialogue, thus breaking the deadlock and improving intersection safety.
Accuracy of Blind Spot Reasoning
Since no existing study has explicitly addressed blind spot reasoning via interactive dialogue in CP, we compare our method against two baselines: (1) the no-fusion strategy, where each vehicle independently perceives the environment using its own onboard sensors, mimicking human drivers; and (2) an early fusion scheme, where vehicles share raw sensor data such as images or LiDAR point clouds for centralized or decentralized joint perception. The latter represents the upper bound of bandwidth consumption but provides full visibility coverage. To evaluate the driving-critical blind spot reasoning accuracy, we compare the inferred visibility and object descriptions against ground truth, with human driver occlusion judgment used as reference. For communication efficiency, we record the average data volume exchanged per decision cycle and compare our dialogue-based selective sharing mechanism with early fusion transmission.
In this section, we validate the effectiveness of the proposed method by evaluating the accuracy of driving-critical blind spot reasoning performed by the VLMs, as well as the success rate of the agent-to-agent dialogue process. All four VLMs introduced earlier were tested in this evaluation.
To establish ground truth, experienced human drivers reviewed the same front-view images and manually labeled key objects that caused driving-critical blind spots affecting driving decisions. These annotated objects serve as the reference for evaluating the VLMs’ blind spot identification accuracy.
The evaluation metrics for blind spot reasoning include True Positive Rate (TP Rate), False Positive Rate (FP Rate), and F1 Score. Note that we only assess the correctness of identifying which objects cause driving-critical blind spots; we do not evaluate the textual reasoning quality itself. This is because reasoning explanations, while important, are harder to quantify consistently and are not directly involved in selecting communication targets or initiating dialogue.
The dialogue success rate is evaluated based on whether the ego vehicle successfully acquires relevant information about the blind zone—specifically, whether any critical object affecting driving was confirmed to be present or absent through the communication process. The validation results are summarized in Table 1 and Figure 8.
| Model | TP Rate (%) | FP Rate (%) | F1 Score | Dialogue Success Rate (%) |
| DeepSeek-VL-7B | 63.3 | 8.1 | 0.727 | 88.0 |
| LLaVA-1.5-7B | 3.3 | 3.5 | 0.061 | 33.3 |
| GPT-4o | 80.0 | 4.7 | 0.862 | 95.1 |
| Claude-3 | 83.3 | 2.3 | 0.893 | 92.1 |
Table 1. Blind Spot Detection Accuracy and Dialogue Success Rates

Figure 8. Driving-Critical Blind Spot Detection Accuracy
The results show that, except for LLaVA-1.5-7B, all other models achieved reasonable performance in driving-critical blind spot identification and reasoning. Specifically, the true positive rate of blind spot recognition exceeded 60% for DeepSeek-VL-7B, and surpassed 80% for GPT-4o and Claude-3. The false positive rates were also relatively low, indicating a strong ability to filter out non-critical occlusions. LLaVA-1.5-7B showed weak sensitivity to this task, likely due to limitations in grounding and scene understanding. Dialogue success rates were also high across models. While DeepSeek-VL-7B exhibited slightly lower consistency in multi-turn dialogue compared to GPT-4o and Claude-3, it still performed reasonably well. Overall, the proposed approach effectively identifies driving-critical blind spot regions, significantly reducing the communication burden in CP.
Communication Cost Analysis
We compare the communication efficiency of the proposed agentic cooperative perception framework with conventional early fusion and late fusion approaches. Prior studies have extensively discussed the bandwidth burden imposed by early and late fusion schemes.
In early fusion methods, raw sensor data such as LiDAR point clouds or camera images are transmitted to neighboring agents or infrastructure nodes. This results in extremely high communication demands, often reaching the order of gigabits per second (Gbps) Ma et al. (2026). These methods typically require continuous high-bandwidth transmission, making them impractical under limited V2X bandwidth conditions.
Late fusion methods reduce the transmission load by only sharing high-level perception results such as detected object positions and categories. This significantly lowers the communication cost, often to the range of several kilobits per second (Kbps), depending on object count and update frequency. For example, sharing a fixed-length object list with 10–20 objects at 10 Hz typically results in data rates under 100 Kbps.
In contrast, our proposed method adopts an asynchronous, dialogue-triggered communication strategy. Vehicles broadcast only essential position and identity metadata during normal operation. Rich semantic messages are transmitted only when a driving-critical blind spot is detected and verified, which occurs infrequently—empirically less than 0.1% of operational frames in typical urban scenesMa et al. (2026). These text messages are compact, typically consisting of 1–3 short sentences (under 500 bytes), resulting in negligible bandwidth usage even when transmitted.
Assuming a frame rate of 10 Hz and one cooperative message every 1,000 frames, the average communication load of our method is approximately:
This is three orders of magnitude lower than late fusion methods, and over six orders lower than early fusion. Therefore, our method offers a scalable and communication-efficient alternative for selective, intent-driven cooperative perception under bandwidth-constrained conditions.
Conclusion
This study proposes an efficient CP framework based on VLMs that leverages blind spot reasoning and an agentic dialogue mechanism to selectively query perceptual information from other connected agents. The framework incorporates a mixed-model architecture that combines stable object detection, vision-language reasoning, and geometric consistency, along with a CoT prompt design tailored for driving intention and occlusion assessment. We collected real-world urban traffic data using front-facing cameras and conducted comparative evaluations across four VLMs, including DeepSeek-VL-7B, GPT-4o, Claude-3, and LLaVA-1.5-7B. Results show that the proposed approach achieves over 80% accuracy in identifying driving-critical blind spots and over 90% success in dialogue-based CP under safety-relevant scenarios.
Future work will focus on improving computational efficiency for real-time deployment, testing in more complex and diverse traffic environments, and designing enhanced models to handle multi-agent reasoning and coordination under dynamic conditions.
References
- [1]Bai, Z., Wu, G., Barth, M.J., Liu, Y., Sisbot, E.A., Oguchi, K., 2022. Pillargrid: Deep learning-based cooperative perception for 3d object detection from onboard roadside lidar, in: 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), IEEE. pp. 1743–1749.DOI
- [2]Bai, Z., Wu, G., Barth, M.J., Liu, Y., Sisbot, E.A., Oguchi, K., 2023. Coopreverse: A mobile-edge-cloud framework for universal cooperative perception with mixed connectivity and automation. arXiv preprint arXiv:2302.03128 .arxiv.org/abs/2302.03128
- [3]Bai, Z., Wu, G., Barth, M.J., Liu, Y., Sisbot, E.A., Oguchi, K., Huang, Z., 2024. A survey and framework of cooperative perception: From heterogeneous singleton to hierarchical cooperation. IEEE Transactions on Intelligent Transportation Systems .arxiv.org/abs/2208.10590
- [4]Chiu, H.k., Hachiuma, R., Wang, C.Y., Smith, S.F., Wang, Y.C.F., Chen, M.H., 2025. V2v-llm: Vehicle-to-vehicle cooperative autonomous driving with multi-modal large language models. arXiv preprint arXiv:2502.09980 .arxiv.org/abs/2502.09980
- [6]Hou, J., Wang, H., Li, T., Dong, E., 2025. An explainable end-to-end autonomous driving framework based on large language model and vision modality fusion: design and application of drivellm-v. Transportation Research Part C: Emerging Technologies 181, 105368.DOI
- [7]Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al., 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems 43, 1–55.DOI
- [8]Jiang, S., Huang, Z., Qian, K., Luo, Z., Zhu, T., Zhong, Y., Tang, Y., Kong, M., Wang, Y., Jiao, S., et al., 2025. A survey on vision-language-action models for autonomous driving. arXiv preprint arXiv:2506.24044 .arxiv.org/abs/2506.24044
- [9]Jocher, G., Chaurasia, A., Qiu, J., 2023. Yolo by ultralytics. https://github.com/ultralytics/ultralytics.
- [11]Liu, J., Ma, C., Zhou, H., Tang, W., Liang, S., Ding, H., Li, X., Ran, B., 2025. Sra-cp: Spontaneous risk-aware selective cooperative perception. URL: https://arxiv.org/abs/2511.17461, doi:10.48550/arXiv.2511.17461, arXiv:2511.17461.
- [13]Tang, Y., Qu, A., Wang, Z., Zhuang, D., Wu, Z., Ma, W., Wang, S., Zheng, Y., Zhao, Z., Zhao, J., 2024. Sparkle: Mastering basic spatial capabilities in vision language models elicits generalization to composite spatial reasoning .arxiv.org/abs/2410.16162
- [14]Tian, K., Mao, J., Zhang, Y., Jiang, J., Zhou, Y., Tu, Z., 2025. Nuscenes-spatialqa: A spatial understanding and reasoning benchmark for vision-language models in autonomous driving. arXiv preprint arXiv:2504.03164 .arxiv.org/abs/2504.03164
- [17]Yuan, Q., Shi, H., Xuan, A.T.Y., Gao, M., Xu, Q., Wang, J., 2023. Enhanced target tracking algorithm for autonomous driving based on visible and infrared image fusion. Journal of Intelligent and Connected Vehicles 6, 237–249.DOI
- [18]Zhang, C., Yang, K., Hu, S., Wang, Z., Li, G., Sun, Y., Zhang, C., Zhang, Z., Liu, A., Zhu, S.C., et al., 2024. Proagent: building proactive cooperative agents with large language models. in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 17591–17599.arxiv.org/abs/2308.11339
- [19]Zhang, H., Du, W., Shan, J., Zhou, Q., Du, Y., Tenenbaum, J.B., Shu, T., Gan, C., 2023. Building cooperative embodied agents modularly with large language models. arXiv preprint arXiv:2307.02485 .arxiv.org/abs/2307.02485
- [20]Zheng, X., Li, H., Zhang, Q., Liu, Y., Chen, X., Liu, H., Luo, T., Gao, J., Xia, L., 2024. Intelligent decision-making method for vehicles in emergency conditions based on artificial potential fields and finite state machines. Journal of Intelligent and Connected Vehicles 7, 19–29.