MirrorComm: A Communication-Efficient Cooperative Perception Framework via Mirrored World Models

Introduction

Accurate and reliable environmental perception is essential for autonomous agents to make informed decisions. However, single-agent perception is constrained by limited sensing range and occlusions. Cooperative perception (CP) mitigates these limitations by enabling agents to exchange sensing information over wireless channels. This paradigm has been widely applied in vehicular networks, where shared observations reveal objects and regions hidden from an agent’s own sensors or beyond its sensing range, enabling more complete and reliable environmental perception [1, 2, 3].

However, exchanging sensing information places substantial demands on limited communication bandwidth. Depending on the information exchanged, CP architectures can be categorized into raw-level, object-level, and feature-level approaches [4]. Raw-level methods, such as Cooper [5], preserve detailed sensor measurements but incur high communication costs. Object-level methods, such as OptiMatch [6], substantially reduce these costs by transmitting compact detection results, but their performance remains constrained by individual agents’ local detection capabilities. Feature-level methods, including F-Cooper [7], V2VNet [3], and V2X-ViT [8], balance these considerations by exchanging intermediate representations. More recent methods, including DiffCP [9], ERMVP [10], Where2comm [11], and CoST [12], further reduce bandwidth requirements through compact feature reconstruction or selective transmission. However, most existing compression methods follow a frame-wise paradigm, repeatedly transmitting predictable information that could be reconstructed from historical context.

Recent advances in world models offer a promising avenue for exploiting this temporal predictability. By maintaining internal environmental representations and learning state transitions, world models predict subsequent scene states from historical observations [13, 14, 15]. World4xV2X [16] demonstrates their potential for CP through scene evolution modeling and consistency-guided reconstruction, improving robustness to occlusions and degraded observations. World models have also been explored in semantic communication to predict video frames at the receiver and reduce frame-by-frame transmission [17]. This framework uses textual prompts to guide prediction and relies on depth-map feedback to determine when updates are needed, requiring a feedback link for transmission decisions. For communication-efficient CP, a key challenge is to enable the transmitter to identify receiver-side prediction errors and selectively transmit observation updates without prediction-quality feedback.

To address this challenge, we propose MirrorComm, a communication-efficient CP framework based on mirrored world models. At the ego receiving agent, a world model predicts the collaborative transmitting agent’s current observation from reconstructed historical observations. The model combines spatial context at multiple scales with causal temporal attention to capture both local details and scene dynamics for prediction. The transmitter maintains an identical model and synchronized history, allowing it to reproduce the receiver’s prediction and identify discrepancies from its current observation without receiver feedback. It selectively transmits RVQ-encoded observation features within the communication budget, which both ends use to update the corresponding spatial locations. The receiver then fuses the reconstructed collaborative observation with its own observation for downstream perception. This design not only avoids repeated transmission of predictable information but also allows agents to continue predicting scene evolution during communication interruptions, mitigating perception degradation caused by outdated collaborative information.

We summarize the contributions of this work as follows:

• We propose MirrorComm, to our knowledge, the first world model-based communication-efficient CP framework, which replaces part of the communication with prediction to reduce repeated transmissions. Moreover, it can be integrated into any existing mainstream collaborative algorithm to support diverse downstream tasks.

• We develop a world model that predicts current observations from historical context through multiscale causal spatiotemporal modeling. To preserve reconstruction fidelity with smaller codebooks, we encode the required observation updates using residual vector quantization (RVQ), which combines codewords across stages.

• Experiments on OPV2V for object detection and semantic segmentation show that MirrorComm reduces communication bandwidth requirements by up to approximately 66× while maintaining comparable performance to a state-of-the-art (SOTA) method. Additional experiments demonstrate its advantage over the frame-wise paradigm under intermittent communication. Ablation studies further confirm the effectiveness of the proposed modules.

Problem Formulation

We consider a CP system with $N$ agents that exchange information to support individual local perception. Each agent can act as a transmitter when sharing information and as a receiver when incorporating messages from its collaborators. For agent $i$, let $X_i$ denote its current sensory observation (e.g., from LiDAR or cameras) and $Y_i$ denote the ground-truth annotations used to evaluate its perception output. We use $M_{j\to i}$ to denote the message sent from agent $j$ to agent $i$. Our goal is to maximize aggregate perception performance under a total communication budget $C$. Specifically, the optimization problem can be formulated as follows:

where $\Phi_\theta$ is the perception model with parameters $\theta$, $h(\cdot,\cdot)$ measures task performance, and $c(\cdot)$ quantifies the communication cost of the messages. The challenge is to design messages $M_{j\to i}$ that both preserve task-relevant information and satisfy the communication budget.

Methodology

Overview of MirrorComm Architecture

MirrorComm reduces communication by allowing the receiver to predict the transmitter’s current observation and receive selected updates to correct prediction errors. As illustrated in Figure 1, each agent first encodes its observation into bird’s-eye view (BEV) features, and the receiver predicts the transmitter’s current features from four previous reconstructions. The transmitter maintains an identical world model and synchronized history, enabling it to reproduce this prediction and compare it with its current observation features. Within the communication budget, it selects the spatial locations with the largest discrepancies and transmits their RVQ-encoded features together with their positions. Both ends perform the same feature update by replacing the predictions at the selected positions with the decoded observation features and retaining the predictions elsewhere. Only these reconstructed transmitter features enter the shared histories for subsequent predictions. The receiver fuses the updated features with its own current features for downstream perception.

By reconstructing predictable information locally, MirrorComm avoids repeatedly transmitting it and concentrates the available bandwidth on observations that are not accurately predicted. The same predictive mechanism also allows the receiver to continue updating collaborative features during temporary communication outages, mitigating perception degradation caused by outdated collaborative information. Moreover, MirrorComm naturally supports multi-agent collaboration, with each agent reusing the same world model for receiver-side prediction and transmitter-side replication while maintaining a separate historical context for each communication link. We next describe MirrorComm using vehicular CP with LiDAR point clouds as an example.

BEV Feature Encoder

As a widely adopted representation for CP, BEV preserves the spatial structure of observations and facilitates feature alignment and fusion across agents. MirrorComm reconstructs the transmitter’s observations in this feature space, allowing the reconstructed features to be used by existing BEV-based CP fusion modules. Specifically, the LiDAR point cloud $X_{i,t}$ observed by agent $i$ at time $t$ is aligned to the receiver’s coordinate frame using relative poses, cropped to the region of interest, and discretized into a spatial grid. The encoder aggregates local point geometry within each grid cell and applies convolutional layers to extract spatial context, yielding the mapping from point clouds to BEV features

where $E_\phi$ denotes the encoder with parameters $\phi$, including preprocessing, and $H$, $W$, and $D$ denote the BEV feature height, width, and channel dimension, respectively.

World-Model-Based Predictive Communication

Mirrored World Model

1. Mirrored World Model: A world model learns how an environment evolves over time, enabling it to predict future states from historical observations. In MirrorComm, we use this capability to predict the transmitter’s current BEV features from the reconstructed features of the preceding four frames, expressed as

where $W_{\psi}$ denotes the world model with learnable parameters $\psi$, $\mathcal{H}_{t}$ is the shared four-frame historical context, and $\widetilde{F}_{t}$ denotes the predicted BEV features at time $t$. To capture spatial relationships and temporal evolution jointly, we implement $W_{\psi}$ using a multiscale causal spatiotemporal attention mechanism, as illustrated in Figure 1(b).

For a single communication link, we omit agent indices for brevity and denote observed and reconstructed BEV features by $F_{t}$ and $\widehat{F}_{t}$, respectively. To compensate for changes in the reference frame over time, we apply pose-based alignment $\mathcal{A}_{\tau\to t}$ to each historical reconstruction $\widehat{F}_{\tau}$, where $\tau\in\{t-4,\ldots,t-1\}$, bringing all four frames into the current coordinate system. The aligned features, together with their spatial validity masks and elapsed times, form the historical context $\mathcal{H}_{t}$ in chronological order. The masks indicate where the aligned historical features remain spatially valid, while the elapsed times specify the interval between each historical frame and the current frame.

The predictor first embeds the aligned BEV features of each historical frame together with its validity mask and elapsed time into a hidden representation. The resulting four-frame sequence then passes through three successive stages, each containing two spatiotemporal (ST) blocks. Between stages, the spatial resolution is halved while all four time steps are retained, yielding outputs $Z^{(1)}$, $Z^{(2)}$, and $Z^{(3)}$ at resolutions $H \times W$, $H/2 \times W/2$, and $H/4 \times W/4$, respectively. Fine-resolution features preserve local detail, while coarser features provide broader spatial context for predicting scene evolution.

Within each ST block, spatial and temporal dependencies are modeled sequentially through spatial attention (SA) and causal temporal attention (TA), followed by a feed-forward network (FFN). For an input feature sequence $Z_{\mathrm{in}}$, positional information is first introduced through a depthwise convolution, and the subsequent transformations are applied with separate residual connections as

where PE denotes the depthwise positional convolution, LN denotes layer normalization, $Z_{\mathrm{out}}$ is the block output, and subscripts PE, SA, and TA identify the corresponding intermediate stages. SA aggregates spatial information within each historical frame, using local windows at the two finer scales and global attention at the coarsest scale. TA then processes the spatially updated sequence along the time dimension at each aligned spatial location, with a causal mask $\mathbf{B}$ added to the attention scores before softmax, defined by

where $a$ and $b$ index the query and key time steps in chronological order, respectively, so that each historical step attends only to itself and earlier steps.

After spatiotemporal modeling, the decoder $f_d$ takes the last temporal output from each scale, denoted $Z_{4}^{(1)}$, $Z_{4}^{(2)}$, and $Z_{4}^{(3)}$, where the subscript 4 corresponds to time $t-1$. It progressively upsamples coarser features and combines them with finer features through skip connections to predict a feature residual $\Delta\mathbf{F}_t$. Adding this residual to the aligned reconstruction of the preceding frame produces the current prediction, expressed as

Together, the history embedding, three ST stages, and residual decoding constitute $W_\psi$, which predicts scene changes relative to the preceding reconstruction.

Prediction-Guided Communication and Feature Update

During initialization, the transmitter sends RVQ-encoded features at all spatial locations for four consecutive frames, after which both ends begin prediction from this shared history.

To correct prediction errors within the communication budget, the transmitter compares the predicted features with those extracted from its current observation and assigns each spatial position $p$ an L2 discrepancy score,

where $[p]$ extracts the feature vector at position $p$ and $\|\cdot\|_2$ denotes the Euclidean norm. After accounting for packet metadata, the transmitter selects the positions with the largest positive scores that fit the link budget, forming $S_t$, and transmits their positions and RVQ-encoded observation features as described in § 3-D. The budgets allocated to individual links sum to at most the total budget $C$. Both ends decode these features into $\mathbf{U}_t \in\mathbb{R}^{H\times W\times D}$, which contains the recovered features at $S_t$ and zeros elsewhere. Feature Update replaces predictions at selected positions with recovered features and retains predictions elsewhere, yielding

where $\mathbf{m}_t \in\{0,1\}^{H\times W}$ marks the selected positions and $\odot$ denotes element-wise multiplication with channel broadcasting. This deterministic operation introduces no learnable parameters. By applying the same feature update, both ends obtain identical reconstructed features $\widehat{\mathbf{F}}_t$, which are appended to their shared histories while the oldest frame is discarded, preserving a common four-frame context for subsequent predictions. When no update is transmitted, both ends use $\widehat{\mathbf{F}}_t = \widetilde{\mathbf{F}}_t$ to advance their histories through prediction alone. The receiver fuses $\widehat{\mathbf{F}}_t$ with its current local features for downstream perception, while retaining only the reconstructed transmitter features in the shared history.

Closed-Loop Training

We first train the perception backbone, then train the RVQ codec and adapt the receiver to quantized features as described in § 3-D. With the encoder and codebooks frozen, we jointly optimize the predictor, fusion module, and task head by unrolling the prediction–communication–update loop. Reconstructed features are fed back as history, exposing the predictor to errors accumulated during deployment. The feature-learning objective $\mathcal{L}_{\mathrm{WM}}$ combines three losses,

where $\lambda_p$, $\lambda_r$, and $\lambda_u$ are nonnegative weights. The three terms use the current observation features $\mathbf{F}_t$ as supervision and are defined as

where MSE and SmoothL1 denote the mean squared error and smooth $L_1$ loss averaged over feature elements, respectively. The latter uses a transition threshold of one. The mask subscript restricts the average to untransmitted positions, with the loss set to zero when none remain. These terms supervise prediction before communication, reconstruction after feature updates, and prediction at untransmitted positions, respectively. The joint objective adds the downstream task loss on the final perception output to $\mathcal{L}_{\mathrm{WM}}$. The feature-learning term supervises the predictor, while the task loss trains both the predictor and receiver through reconstructed features.

RVQ-Based Message Encoding

To communicate observation updates efficiently, we seek a compact encoding that preserves reconstruction fidelity with smaller codebooks. MirrorComm therefore uses four-stage RVQ, which combines codewords across stages by successively quantizing the remaining residual and transmits their indices (Figure 1(c)). Each stage $\ell\in\{1,2,3,4\}$ uses a codebook $\mathcal{C}_\ell= \{\mathbf{c}_{\ell,k}\}_{k=1}^{K} \subset\mathbb{R}^{D}$, where $K$ is the number of codewords and $\mathbf{c}_{\ell,k}$ is the $k$th codeword. The four codebooks provide up to $K^4$ index combinations with only $4K$ stored codewords.

For each selected position $p \in S_t$, the transmitter initializes $\mathbf{r}_0 = \mathbf{F}_t[p]$ and recursively quantizes the residual as

where $k_\ell$ is the selected index and $\mathbf{r}_\ell$ the remaining residual, with time and position indices omitted for brevity. The transmitter sends the four codeword indices, which both ends decode using identical preinstalled codebooks as

Unselected positions are set to zero to form $\mathbf{U}_t$ for the feature update in (8). For a link budget $C_{\mathrm{link}}$, the packet satisfies $b_h + b_r|S_t| \leq C_{\mathrm{link}}$, where $b_h$ and $b_r$ denote the metadata bytes and bytes per selected position, respectively.

The codec is trained using standard codebook-fitting and commitment losses [18], together with feature reconstruction and downstream task losses. Codewords are detached when forming subsequent residuals, while a straight-through estimator propagates gradients through quantization to the encoder.

Experimental Results

Experimental Setup

We conduct experiments on OPV2V [1], a simulated CP dataset generated with CARLA and OpenCDA. Our processed splits contain 6,374 training frames, 1,980 validation frames, and 2,170 test frames. We use its LiDAR sequences with a perception frequency of $f = 10$ Hz and process frames within each scene in chronological order. The shared histories are reset when switching to a new scene. The communication bandwidth is measured in Mbps as

where $B_{\mathrm{total}}$ is the total number of transmitted bytes, including metadata, over $N_f$ evaluated frames. For MirrorComm, this total also includes the cold-start transmissions used to initialize the shared history at the beginning of each scene.

Two downstream tasks are evaluated on this dataset, including object detection and BEV semantic segmentation. These tasks assess complementary aspects of reconstruction quality, as object detection requires accurate object locations and geometry, whereas semantic segmentation requires spatially dense semantic details. For object detection, we report BEV average precision (AP) at intersection-over-union (IoU) thresholds of 0.5 and 0.7, denoted AP50 and AP70, respectively. For semantic segmentation, we report mean IoU (mIoU) over the vehicle, road, and lane classes, excluding background.

We implement MirrorComm with the feature fusion module from CoMamba [19]. We evaluate per-link update budgets of 256, 512, 1,024, 2,048, and 4,096 bytes. We use AdamW with an initial learning rate of $10^{-4}$ and weight decay of $10^{-4}$, and apply truncated backpropagation through time over 20 steps of the closed-loop sequence. All implementations use PyTorch and run on NVIDIA RTX 4090 GPUs with 24 GB memory.

Trade-off between Performance and Bandwidth

We compare with representative feature-level information fusion methods for CP, including V2VNet [3], F-Cooper [7], V2X-ViT [8], and CoMamba [19], which fuse intermediate features from multiple agents. We also include methods targeting communication efficiency, namely DiffCP [9], ERMVP [10], Where2comm [11], and CoST [12]. For object detection, we additionally consider an object-level baseline that transmits local detection results without a bandwidth cap.

Figure 2 shows that MirrorComm substantially reduces communication bandwidth while preserving perception performance across both downstream tasks. At an update budget of 1,024 bytes, MirrorComm requires only 0.159 Mbps and achieves 96.84% AP50 and 92.15% AP70. Compared with the state-of-the-art method CoST, which achieves 95.78% AP50 and 92.24% AP70 at 10.497 Mbps, MirrorComm delivers higher AP50 and comparable AP70 while requiring approximately 66× less bandwidth. This advantage extends to semantic segmentation, where MirrorComm achieves 75.51% mIoU at the same bandwidth, surpassing CoST’s best evaluated mIoU of 74.62% while operating below its lowest evaluated bandwidth of 0.603 Mbps. MirrorComm also exceeds frame-wise CoMamba’s mIoU of 74.56%, which may stem from its use of historical context to complement current observations. Although DiffCP operates at lower bandwidths than MirrorComm, its perception performance is substantially lower on both tasks. Likewise, the object-level baseline requires only 0.043 Mbps but achieves an AP70 of 81.15%, revealing a substantial accuracy gap under the stricter localization criterion. Moreover, unlike most existing methods that operate with fixed data rates, MirrorComm can adapt to different communication budgets at inference time by varying the number of spatial positions selected for transmission, without retraining.

Ablation Studies

We conduct ablation studies by comparing four component configurations at a fixed communication update budget of 1,024 bytes, reporting AP50, AP70, and mIoU in Table 1. The configurations enable or disable RVQ and the world-model predictor (WM). Disabling RVQ replaces it with single-stage VQ. Disabling WM replaces prediction with the pose-aligned previous state, while retaining the same feature update at transmitted locations. Enabling both components gives the complete MirrorComm framework. Within each task and quantizer pair, both WM settings use the same receiver weights and communication budget. VQ uses one 1,024-entry codebook, whereas RVQ uses four 256-entry codebooks, keeping total codeword counts equal and accounting for index lengths in the communication cost.

Component	Performance (%)
RVQ	WM	AP50	AP70	mIoU
×	×	95.12	87.01	71.09
×	✓	96.18	89.11	72.30
✓	×	96.19	90.12	72.43
✓	✓	97.14	92.15	75.51

Update interval (frames)	CoMamba	MirrorComm
2	83.78	84.79
3	80.69	83.93
4	79.11	83.12
5	78.67	82.17
Random 2–5	78.01	83.54

WM improves both downstream tasks with either quantizer, increasing mIoU by 3.08 percentage points with RVQ. With WM enabled, replacing VQ with RVQ further improves all three metrics, including a 3.21-point mIoU gain. These results confirm that both components contribute to perception accuracy under the same communication budget.

Robustness to Intermittent Communication

To evaluate MirrorComm’s ability to mitigate information staleness relative to frame-wise perception, we test communication updates at fixed intervals of 2–5 frames and random intervals within this range. We choose CoMamba as the baseline because MirrorComm adopts its feature fusion architecture. CoMamba transmits complete collaborative features at each update and holds them between updates. Under the same schedule, MirrorComm sends 1,024-byte observation updates and predicts collaborative features between transmissions. Both methods are evaluated on object detection using AP70.

As shown in Table II, MirrorComm outperforms frame-wise CoMamba under every update schedule. As the fixed interval increases from 2 to 5 frames, its AP70 decreases by 2.62 percentage points, compared with 5.11 points for frame-wise CoMamba. Under random intervals, MirrorComm achieves 83.54% AP70, exceeding the baseline by 5.53 percentage points. These results show that predicting scene evolution mitigates degradation from stale collaborative information, even though frame-wise CoMamba receives complete collaborative features at every update.

Conclusion

This paper presented MirrorComm, a world-model-based communication-efficient CP framework. By maintaining mirrored world models at the transmitter and receiver, MirrorComm reconstructs predictable information locally and communicates compact observation updates to correct prediction errors. Experiments on OPV2V demonstrated substantial bandwidth savings while maintaining competitive object detection and semantic segmentation performance, with additional benefits under intermittent communication. These findings highlight the potential of shared predictive models to shift cooperative perception from transmitting observations frame by frame toward communicating only the information needed to update a shared reconstruction.