WorksComputer Science
SCALE: Real-Time Latent Communication for Cyber-Physical Agents
Abstract
Latent communication enables agents to collaborate efficiently by directly exchanging intermediate model representations rather than raw sensor data or text messages. Existing approaches, however, are primarily designed for LLM agents and do not consider the bandwidth constraints and heterogeneous task deadlines faced by cyber-physical agents. In this work, we propose SCALE, a real-time latent communication method tailored for cyber-physical agents. Motivated by the spatio-temporal continuity of the physical world, SCALE matches tokens across consecutive latent messages using similarity scores to identify reusable information and avoid redundant transmission. It then profiles the semantic gain and communication cost of transmitting different amounts of latent information. Based on these profiles, SCALE employs a real-time scheduler that jointly determines which pending messages to transmit and how many latent tokens to send, subject to available bandwidth and task deadlines. We evaluate SCALE across three cyber-physical benchmarks and a physical spider robot case study under diverse network conditions and heterogeneous task deadlines. Our results show that SCALE significantly improves both task performance and on-time delivery in bandwidth-constrained networks. Our code is available at: https://anonymous.4open.science/r/SCALE-4E80/.
Introduction
Communication is fundamental to cyber-physical agents. Individual agents often possess only partial observations or complementary capabilities, and therefore must exchange information with one another to accomplish diverse cyber-physical tasks [16, 6]. However, transmitting raw sensor observations incurs substantial communication overhead, while natural-language communication introduces additional computational overhead from generating text at the sender and interpreting it at the receiver. These limitations make conventional communication inefficient for real-time multi-agent systems.
Existing Literature and the Gap. Latent communication has emerged as a potential “AI language”, allowing agents to directly exchange hidden states or intermediate representations to improve efficiency [3, 19] (Zou et al. 2026; Ye et al. 2025). However, existing methods are primarily designed for LLM-based agents and focus on reducing model-side computation, with limited consideration of communication bandwidth and task deadlines. Meanwhile, communication-efficient collaborative-perception methods consider bandwidth and delay [14, 6, 8] (Yang et al. 2023), but primarily focus on task-specific perception rather than multiple concurrent CPS tasks sharing a bandwidth-constrained link. Thus, latent communication for CPS agents under practical network constraints and heterogeneous task deadlines remains largely underexplored.
Our Work. We propose SCALE, a real-time latent communication method tailored for cyber-physical agents. SCALE exploits the spatiotemporal continuity of the physical world to reuse latent information already available at the receiver, reducing redundant transmission. It employs a real-time scheduler that jointly determines which messages to transmit and how many latent tokens to send based on semantic gain, available bandwidth, and task deadlines, thereby improving both task utility and message on-time delivery. In addition, SCALE is lightweight and operates on latent tokens produced by existing modality encoders without modifying the underlying models, supporting different sensory modalities and downstream CPS tasks.
Designing SCALE addresses three technical challenges. **First, spatiotemporal variation makes it difficult to determine what information has actually changed. Physical observations often evolve continuously, but object, agent, or viewpoint motion can make fixed-position latent comparisons overestimate motion for semantic change. SCALE therefore performs semantic change evaluation by matching current latent tokens against the receiver’s acknowledged latent state and measuring change relative to their best correspondences. Second, the amount of latent information to transmit must adapt to each message. The semantic gain from transmitting additional latent tokens varies across messages, making a fixed transmission size inefficient. SCALE therefore performs utility–cost profiling for each message, ranking latent tokens by semantic change and characterizing the cumulative semantic gain and communication cost at different transmission sizes to guide real-time scheduling. Third, limited and time-varying bandwidth must be allocated across messages with heterogeneous deadlines. Allocating more latent tokens to one message increases its semantic gain but may delay other messages beyond their deadlines. SCALE therefore employs a real-time scheduler that greedily allocates latent tokens based on marginal semantic gain per communication cost, while maintaining deadline feasibility for all selected messages under earliest-deadline-first (EDF) ordering.
We implement and evaluate SCALE on CooperScene (Wu et al., 2026), OPV2V (Xu et al., 2022c), and LIBERO (Liu et al., [11]), covering autonomous driving and robotic manipulation benchmarks. We further conduct a physical spider robot case study. Our evaluation considers constrained bandwidth, time-varying link capacity, and background traffic, with heterogeneous task deadlines. We compare SCALE against full latent transmission, uniform compression, and four state-of-the-art methods (Xu et al., 2025, 2022b; Hu et al., [6]; Mukhopadhyay et al., [15]). Our results show that SCALE achieves the highest task performance in almost all settings while significantly improving on-time delivery rates (see Figure 1 for an example).
Figure 1. Comparison between SCALE and other latent communication baselines on the OPV2V benchmark at 15 Mbps. SCALE achieves the best mIoU scores and message on-time rates.
We make the following contributions:
We identify and formulate the problem of real-time latent communication for CPS agents, where latent messages from multiple downstream tasks with heterogeneous deadlines share a bandwidth-constrained communication link, a setting not adequately addressed by existing methods.
We propose SCALE, a real-time latent communication method that combines spatiotemporal latent reuse, utility–cost profiling, and deadline-aware scheduling to improve both task utility and message on-time delivery under bandwidth constraints.
We implement and evaluate SCALE across three CPS benchmarks and a physical spider robot case study under diverse network conditions, demonstrating its effectiveness in both benchmark evaluations and a physical robotic system.
Related Work
LLM-based latent communication. Recent LLM-based latent communication methods enable multi-agent systems to exchange latent representations, such as hidden states or KV caches, to reduce model inference and computation overhead ([3, 19, 4, 2] (Zou et al. 2026; Ye et al. 2025)). These approaches primarily focus on efficient latent interaction and reducing model-side computation, with limited consideration of the underlying communication network. SCALE instead studies latent communication over resource-constrained networks, a practical setting for cyber-physical agents.
Communication-efficient cyber-physical systems. Communication-efficient collaborative perception has been extensively studied in cyber-physical systems, where agents exchange intermediate features to improve shared perception (Wang et al., 2020; Li et al. [10]. Existing methods explore when, where, and how agents should communicate through communication triggering (Liu et al. [14]), spatial feature selection (Hu et al. [6]), and feature sparsification (Yang et al. (2023); Hu et al. [8]). Other work improves robustness to communication delay and asynchrony (Lei et al. [9]; Wei et al. (2023)) or exploits temporal redundancy to reduce repeated transmission (Mukhopadhyay et al. [15]). Recent methods further improve the perception–communication tradeoff through adaptive feature selection (Xu et al. (2025); Hu et al. [7]; Liu et al. [13]). In the semantic communication literature, existing task-oriented methods tailor transmitted representations according to downstream task utility (Xie et al. (2021); Gündüz et al. [5]), jointly allocate communication resources under bandwidth constraints (Zhang et al. (2023); Liu et al. [12]; Cai et al. [1]), and account for communication latency (Sagduyu and Erpek [17]). However, these methods are primarily designed for individual perception or communication tasks, rather than the broader CPS setting where multiple downstream tasks with heterogeneous deadlines coexist and share a bandwidth-constrained communication link.
Problem Formulation
System Model. Following common settings in prior works (Liu et al. [14]; Hu et al. [6]; Yang et al. (2023)), we consider cyber-physical agents that locally encode their observations and exchange latent representations over a communication link. For an observation , a local encoder produces a latent representation:
where each is a fixed-dimensional latent token, and is the token length. Encoders may be modality-specific, e.g., for image, audio, or text inputs. Within each agent, multiple downstream tasks may coexist and generate latent messages that share the same communication link.
Network and Timing Model. We denote the available rate of the communication link by , which may vary over time. Within each agent, multiple latent messages from different tasks compete for the same communication resources. We denote the set of messages pending transmission at time by and the deadline of message by . Let denote the transmission completion time of message . Following a common firm-deadline model in real-time systems, we assume that a message contributes to its downstream task only if the selected latent information arrives before its deadline, i.e., .
Scheduling Objective. At time , the scheduler considers all messages in the pending set . For each message , let denote the number of latent tokens selected for transmission, where indicates that the message is not selected. We denote the semantic gain and communication cost of transmitting tokens by and , respectively. Our goal is to determine which pending messages to transmit and how many tokens to transmit for each message, so as to maximize the total semantic gain delivered before their respective deadlines:
subject to available bandwidth constraints.
Approach
Overview
SCALE consists of three stages: semantic change evaluation, utility–cost profiling, and real-time scheduling. In the first two stages, it identifies latent information that is new relative to the receiver’s acknowledged state, and characterizes the semantic benefit and communication cost of different transmission lengths. The scheduler then combines these profiles with available bandwidth and task deadlines to determine which message to transmit and how many latent tokens to send. We demonstrate the system flow of SCALE in Figure 2.
Figure 2. SCALE system design. Semantic change evaluation and utility–cost profiling guide the real-time scheduler in jointly selecting messages and transmission sizes under bandwidth and deadline constraints.
Semantic Change Evaluation
Physical observations often evolve continuously, creating substantial redundancy between consecutive observations. In this stage, we aim to measure such semantic redundancy and identify the nonredundant information to transmit. A key challenge is that directly comparing two latent messages token by token can be unreliable because their corresponding observations may be misaligned due to object or viewpoint motion. SCALE therefore first establishes token-level correspondences between the current latent message and the receiver’s acknowledged latent state before quantifying their semantic changes.
Therefore, for each current token , we first find the most similar token in the receiver’s acknowledged latent state:
Let denote the corresponding maximum cosine similarity. We then quantify the semantic change of token as:
Consequently, highly similar tokens receive small change scores, while tokens with low similarity receive larger scores. We further use a similarity threshold to determine whether the matched receiver-side token can be reused: a match is reusable only when . Matching is performed against the receiver’s acknowledged state to ensure that only information confirmed to be available at the receiver is reused.
Utility–Cost Profiling
The semantic change scores obtained above capture the amount of new information carried by individual latent tokens. In this stage, we further characterize the trade-off between semantic gain and communication cost at different transmission sizes , where denotes the number of latent tokens transmitted for message .
Specifically, we prioritize tokens with larger semantic change scores by sorting them in descending order according to . We define the cumulative semantic gain and communication cost as:
where denotes the -th largest semantic change score, is the serialized size of one latent token, and accounts for the meta-data required to interpret the transmission, including token indices, correspondence mappings, and the reference-state version. The resulting pairs characterize the available transmission choices for message . As a result, a larger generally preserves more new semantic information but incurs greater communication cost.
Real-Time Scheduling
In the final stage, the scheduler takes as input the utility–cost profiles and task deadlines of all messages in , together with the current link rate . Based on this information, it determines the transmission size for each message and their transmission order.
SCALE first determines the transmission size for each pending message. Following the scheduling objective in § 3, we employ a greedy algorithm to maximize semantic gain under the deadline constraints. Specifically, we initialize for each pending message and iteratively allocate tokens one at a time. Suppose message has already been allocated tokens. The marginal semantic gain per communication cost of adding its next token is:
At each step, the scheduler considers the next token of every pending message and prioritizes the one with the largest , subject to the message deadlines. To check deadline feasibility, let denote the currently selected messages ordered by earliest deadline first (EDF). The completion time of message is estimated as:
An allocation is feasible only if for all . Among all feasible token increments, the scheduler selects the one with the largest , updates , and repeats this process until no additional token can be allocated without violating a deadline.
The resulting values determine the transmission size of each message, where indicates that the message is not selected. Messages with are transmitted in EDF order. In this way, SCALE prioritizes semantic gain relative to communication cost while using task deadlines as feasibility constraints on both message selection and transmission size. After transmission, the receiver updates its latent state with the received tokens and reuses matched tokens from its previous state according to the correspondences established in the semantic change evaluation stage. The receiver then acknowledges the updated state, allowing the sender to maintain an accurate view of the receiver’s state for subsequent communication. The detailed scheduling algorithm can be found in Appendix § 1.
Evaluation
Experimental Settings
CPS Benchmarks. We evaluate SCALE on three cyber-physical benchmarks: CooperScene (Wu et al., 2026), OPV2V (Xu et al., 2022c), and LIBERO (Liu et al., [11]). CooperScene is a real-world cooperative autonomous-driving benchmark with LiDAR and camera observations from vehicles and roadside infrastructure. OPV2V is a simulated cooperative-perception benchmark with LiDAR and camera observations from multiple vehicles. LIBERO provides language-conditioned robotic manipulation tasks with visual observations and robot states. These benchmarks cover different CPS use cases, and their detailed descriptions are provided in Appendix § 8.
Tasks and Models. On CooperScene, we evaluate concurrent 3D object detection and 5-sec trajectory prediction tasks, with deadlines of 100 ms and 200 ms. We use CoBEVT (Xu et al., 2022a) for detection and a frozen, pretrained Motion Transformer (MTR) (Shi et al., [18]) for prediction. On OPV2V, we evaluate concurrent 3D object detection and BEV semantic segmentation using the LiDAR-based detection and camera-based segmentation variants of CoBEVT, each with its own pretrained weights. Their deadlines are 100 ms and 200 ms, respectively. On LIBERO, we evaluate object manipulation using SmolVLA (Shukor et al., 2025), with the deadline determined by the time remaining before the robot exhausts its buffered actions, plus reserving 100 ms for inference and 10 ms for returning actions. Across all benchmarks, a sensing agent sends latent features to another agent or remote server for inference.
Network Settings. For each benchmark, we evaluate four fixed-bandwidth settings and one time-varying setting. The fixed rates correspond to different fractions (e.g., 20%, 50%) of the average bandwidth required for full latent transmission in each benchmark. In the time-varying setting, the link alternates between low and high rates every 2 seconds. In all settings, we account for both queueing and transmission time when computing communication delay. More detailed network settings, including the bandwidth values for each benchmark, are provided in Table 5 in the Appendix.
| Benchmark | Task | Test workload | Rate (Hz) | Deadline (ms) |
| CooperScene | 3D detection | 3,936 jobs | 10 | 100 |
| Trajectory prediction (5 s) | 18,693 targets | 5 | 200 | |
| OPV2V | 3D detection | 528 jobs | 10 | 100 |
| BEV segmentation | 265 jobs | 5 | 200 | |
| LIBERO | Object manipulation | 45 episodes | 20 | Variable |
| Benchmark | Full-send | Fixed rates | Fractions (%) | Varying rates |
| CooperScene | 118.5 | 15, 20, 25, 30 | 12.5, 16.7, 20.8, 25 | 30 ↔ 200 |
| OPV2V | 153.0 | 8, 15, 31, 61 | 5, 10, 20, 40 | 8 ↔ 61 |
| LIBERO | 13.46 | 3.4, 5.0, 6.7, 10.1 | 25, 37.5, 50, 75 | 3.4 ↔ 20 |
Table 5. Detailed experiment settings.
Baselines. We compare SCALE with Full-send, which transmits the complete latent message, and Uniform compression, which uniformly selects a fraction of latent features for transmission based on the available bandwidth. On CooperScene, we additionally compare with CoSDH (Xu et al., 2025) and V2X-ViT (Xu et al., 2022b). On OPV2V, we additionally compare with CooperTrim (Mukhopadhyay et al., [15]) and Where2comm (Hu et al., [6]). All baseline methods use earliest-deadline-first (EDF) transmission ordering.
Metrics. We evaluate SCALE in terms of task utility and message on-time rate. Task utility measures downstream task performance, while on-time rate is the percentage of messages received before their deadlines.
Benchmark Performance
CooperScene. We use task-specific performance metrics to evaluate the two tasks on CooperScene. For the 3D object detection task, we report average precision (AP) and the message on-time rate, where AP is computed using an intersection-over-union (IoU) threshold of 0.5 to match predicted boxes to ground-truth boxes. For the trajectory prediction task, we report task success and the message on-time rate, where task success is the percentage of prediction targets for which at least one of six predicted trajectories is within 2 m of the ground-truth position.
Table 1 presents the results. The results show that SCALE achieves the highest detection AP and prediction task success across all settings. In particular, for 3D object detection at 20 Mbps, SCALE achieves 51.53 AP, exceeding the strongest baseline, CoSDH, by 20.19 points. It also delivers 80.36% of detection messages on time, compared with 48.02% for CoSDH. For trajectory prediction under fixed bandwidth, SCALE maintains task success between 60.82% and 60.85%, while delivering at least 99.82% of prediction messages on time. Under varying bandwidth, SCALE achieves 56.09 detection AP and 60.81% prediction task success, with detection and prediction on-time rates of 94.52% and 99.71%, respectively.
| Method | 15 Mbps | 20 Mbps | 25 Mbps | 30 Mbps | Varying | |||||
| AP | OT | AP | OT | AP | OT | AP | OT | AP | OT | |
| Full-send | 22.77 | 0.00 | 22.77 | 0.00 | 22.77 | 0.00 | 22.77 | 0.00 | 40.26 | 49.97 |
| Uniform | 22.50 | 68.42 | 23.38 | 68.64 | 23.72 | 68.63 | 24.91 | 68.50 | 39.25 | 81.59 |
| CoSDH | 27.99 | 32.04 | 31.34 | 48.02 | 40.67 | 68.14 | 43.33 | 75.43 | 49.87 | 89.05 |
| V2X-ViT | 25.24 | 0.00 | 25.24 | 0.00 | 25.33 | 1.21 | 25.91 | 8.70 | 42.22 | 55.01 |
| SCALE | 45.62 | 64.95 | 51.53 | 80.36 | 54.87 | 88.13 | 56.28 | 92.84 | 56.09 | 94.52 |
| Method | 15 Mbps | 20 Mbps | 25 Mbps | 30 Mbps | Varying | |||||
| Succ. | OT | Succ. | OT | Succ. | OT | Succ. | OT | Succ. | OT | |
| Full-send | 43.72 | 0.26 | 44.73 | 6.54 | 49.57 | 35.32 | 48.50 | 25.05 | 54.88 | 64.89 |
| Uniform | 59.49 | 93.81 | 59.52 | 94.44 | 59.87 | 95.64 | 59.84 | 95.90 | 60.48 | 98.47 |
| CoSDH | 46.48 | 16.69 | 51.28 | 44.26 | 52.86 | 50.99 | 57.37 | 80.92 | 59.24 | 91.47 |
| V2X-ViT | 48.54 | 29.23 | 46.65 | 18.53 | 49.75 | 36.37 | 52.41 | 50.95 | 57.07 | 77.64 |
| SCALE | 60.82 | 99.82 | 60.85 | 100.00 | 60.85 | 100.00 | 60.85 | 100.00 | 60.81 | 99.71 |
Table 1. CooperScene results on 3,936 detection jobs and 18,693 prediction targets. AP denotes average precision, OT denotes the on-time rate (%), and Succ. denotes prediction success (%).
OPV2V. Similarly, we use task-specific performance metrics to evaluate the two tasks on OPV2V. For the 3D object detection task, we report AP and the message on-time rate. We evaluate BEV semantic segmentation using mean intersection-over-union (mIoU) and also report its message on-time rate, where mIoU averages the IoU between predicted and ground-truth regions across semantic classes.
Table 2 presents the results. The results show that SCALE achieves the highest segmentation mIoU and segmentation on-time rate across all settings, and the highest detection AP except at 8 Mbps. In this setting, SCALE achieves 83.41 AP, slightly below CooperTrim’s 84.25 AP, while delivering 37.74% of detection messages on time, compared with 0.73% for CooperTrim. For BEV semantic segmentation, the largest mIoU improvement over the strongest baseline occurs at 15 Mbps, where SCALE achieves 47.14 mIoU, exceeding Where2comm by 9.97 points. It also delivers 97.09% of segmentation messages on time, compared with 7.82% for Where2comm.
| Method | 8 Mbps | 15 Mbps | 31 Mbps | 61 Mbps | Varying | |||||
| AP | OT | AP | OT | AP | OT | AP | OT | AP | OT | |
| Full-send | 76.37 | 0.00 | 76.37 | 0.00 | 76.37 | 0.00 | 76.37 | 0.00 | 76.37 | 0.00 |
| Uniform | 78.29 | 46.49 | 79.87 | 44.21 | 77.80 | 41.20 | 81.90 | 39.93 | 79.65 | 42.30 |
| CooperTrim | 84.25 | 0.73 | 84.39 | 4.92 | 85.81 | 28.71 | 89.47 | 69.55 | 86.14 | 31.81 |
| Where2comm | 82.25 | 0.82 | 82.63 | 10.30 | 88.09 | 54.42 | 91.88 | 86.51 | 86.20 | 36.37 |
| SCALE | 83.41 | 37.74 | 86.42 | 50.23 | 90.30 | 75.39 | 92.52 | 91.70 | 86.44 | 56.43 |
| Method | 8 Mbps | 15 Mbps | 31 Mbps | 61 Mbps | Varying | |||||
| mIoU | OT | mIoU | OT | mIoU | OT | mIoU | OT | mIoU | OT | |
| Full-send | 27.02 | 0.00 | 27.14 | 0.73 | 28.80 | 10.36 | 31.77 | 27.82 | 28.74 | 11.09 |
| Uniform | 27.10 | 77.82 | 26.95 | 81.09 | 28.93 | 79.27 | 32.69 | 78.00 | 28.67 | 76.18 |
| CooperTrim | 33.08 | 12.36 | 35.52 | 32.36 | 40.87 | 67.64 | 43.31 | 92.73 | 37.75 | 48.55 |
| Where2comm | 37.14 | 0.18 | 37.17 | 7.82 | 37.76 | 52.00 | 38.32 | 93.64 | 37.56 | 42.55 |
| SCALE | 46.48 | 94.91 | 47.14 | 97.09 | 47.71 | 99.64 | 47.78 | 100.00 | 45.17 | 90.18 |
Table 2. OPV2V results on 528 detection jobs and 265 segmentation jobs. AP denotes average precision, mIoU denotes mean IoU score, and OT denotes on-time rate (%).
LIBERO. Our LIBERO evaluation focuses on a single language-conditioned robotic manipulation task. We report task success, defined as the percentage of episodes completed within 280 steps, and the communication on-time rate.
Table 3 presents the results. The results show that SCALE achieves the highest task success and on-time rate across all settings. At the lowest tested bandwidth of 3.4 Mbps, SCALE completes 44.44% of episodes, while both baselines fail all episodes. In particular, Uniform compression achieves a similar on-time rate (74.47% versus 77.76% for SCALE), showing that timely delivery alone does not ensure task success. The largest task success improvement over the strongest baseline occurs at 5.0 Mbps, where SCALE achieves 86.67% task success, compared with 24.44% for Full-send and
| Bandwidth | Method | Task success ↑ | On-time ↑ |
| 3.4 Mbps | Full-send | 0.00% | 0.00% |
| Uniform | 0.00% | 74.47% | |
| SCALE | 44.44% | 77.76% | |
| 5.0 Mbps | Full-send | 24.44% | 25.24% |
| Uniform | 0.00% | 74.47% | |
| SCALE | 86.67% | 84.57% | |
| 6.7 Mbps | Full-send | 60.00% | 25.98% |
| Uniform | 0.00% | 74.47% | |
| SCALE | 95.56% | 88.81% | |
| 10.1 Mbps | Full-send | 88.89% | 51.52% |
| Uniform | 42.22% | 74.85% | |
| SCALE | 97.78% | 94.21% | |
| Varying | Full-send | 88.89% | 65.87% |
| Uniform | 0.00% | 77.45% | |
| SCALE | 100.00% | 89.28% |
Table 3. LIBERO results over 45 episodes per method and bandwidth setting. We report the task success rate (%) and on-time rate (%).
0.00% for Uniform compression. Its communication on-time rate also reaches 84.57%, exceeding both baselines.
Real-Time Performance
We measure message lateness as , where and are the completion time and deadline of message , respectively. On-time messages have , while delayed messages have . To better illustrate real-time performance, Figure 3 shows the lateness distributions for CooperScene at 20 Mbps for both object detection and trajectory prediction tasks. In the figure, the cumulative percentage at zero gives the on-time rate, which is computed over all generated messages, including undelivered ones.
Figure 3. Message-lateness CDFs for CooperScene at 20 Mbps: (a) object detection and (b) trajectory prediction. The vertical dashed line marks each task’s deadline; the cumulative percentage at zero gives the on-time rate.
The results show that SCALE delivers 80.36% of detection messages and 100.00% of prediction messages on time, compared with 68.64% and 94.44% for Uniform compression. Uniform compression achieves the highest on-time rates among the baselines at this bandwidth, but its task utility is significantly lower than that of SCALE. We also observe a steep rise in the detection CDF just before the deadline for the object detection task, indicating that many messages complete shortly before their deadlines. This behavior is consistent with SCALE’s greedy allocation of available bandwidth to maximize semantic gain under deadline constraints. The complete distributions for all three benchmarks and bandwidth settings appear in Appendix § 9 (Figure 8, Figure 9, and Figure 10). Across these results, SCALE improves on-time delivery in every tested setting and achieves the highest on-time rate among all methods in most settings.
Figure 8. CooperScene message-lateness CDFs at the five selected bandwidth settings: detection (left) and prediction (right). Varying switches between 30 and 200 Mbps every 2 seconds. Zero marks each message’s deadline; the CDF at zero is its method’s on-time rate.
Figure 9. OPV2V message-lateness CDFs at the five selected bandwidth settings: detection (left) and segmentation (right). Varying switches between 8 and 61 Mbps every 2 seconds. Zero marks each message’s deadline; the CDF at zero is its method’s on-time rate.
Figure 10. LIBERO camera-update lateness CDFs at the five selected bandwidth settings, using all 45 episodes per method and setting. Varying switches between 3.4 and 20 Mbps every 2 seconds. Zero marks each message’s deadline; the CDF at zero is its method’s on-time rate.
Case Study
We conduct a hardware-in-the-loop case study using a Raspberry Pi-based spider robot, as shown in Figure 4. The robot is equipped with a camera and an ultrasonic sensor and performs two tasks: object detection and distance measurement on a fixed robot patrolling route. The patrolling route is a 55.4-second walk through a miniature city containing objects such as small road signs, pedestrian figures, and buildings. Among the two tasks, object detection is time-sensitive, with a communication deadline of 1,000 ms, while ultrasonic distance measurements generate background traffic over the shared link. We compare SCALE against two baselines: Raw RGB, which transmits uncompressed images, and Full-send, which transmits complete intermediate latent features. We evaluate the system in three bandwidth settings, including 1, 2, and 8 Mbps.

Figure 4. Spider robot patrolling in the mini-city with pedestrians, stop signs, and model cars. The spider patrols along an L-shaped route.
We report task success and message on-time rates. Table 4 presents the results. We observe
| Method | Metric | 1 | 2 | 8 |
| RGB | Success ↑ | 0.00% | 0.00% | 100.00% |
| On-time ↑ | 0.00% | 0.00% | 100.00% | |
| Latent | Success ↑ | 0.00% | 0.00% | 100.00% |
| On-time ↑ | 0.00% | 0.00% | 100.00% | |
| SCALE | Success ↑ | 77.62% | 91.67% | 100.00% |
| On-time ↑ | 100.00% | 96.43% | 100.00% |
Table 4. Spider robot mini-city case study results under different bandwidth constraints.
that SCALE substantially outperforms both baselines under constrained bandwidth. Specifically, at 1 and 2 Mbps, both baselines achieve zero detection success, whereas SCALE achieves success rates of 77.62% and 91.67%, respectively, with on-time rates of 100% and 96.43%. At 8 Mbps, all methods achieve 100% detection success and on-time delivery. These results demonstrate the effectiveness of SCALE in preserving timely detection performance on a physical robotic platform under limited bandwidth.
Conclusion
In this work, we present SCALE, a real-time latent communication method for cyber-physical agents. By exploiting spatiotemporal continuity, SCALE avoids retransmitting latent information already available at the receiver. To support timely communication, we design a real-time scheduler that jointly selects messages and transmission sizes based on semantic gain, available bandwidth, and task deadlines. We evaluate SCALE on three benchmarks and conduct a physical robot case study. The results demonstrate improved task performance and high message on-time rates under constrained and varying network conditions. These findings highlight SCALE as an effective and efficient method for meeting the communication and timing requirements of CPS agents.
Limitations
While SCALE significantly improves bandwidth efficiency and supports real-time scheduling for CPS agents, it has two limitations. First, the communication savings of SCALE depend on spatiotemporal redundancy in latent representations. However, observations in the physical world can change abruptly, which may reduce the amount of reusable information and increase the communication required to maintain task performance. Second, the scheduling algorithm assesses deadline feasibility using the current link rate. However, this design cannot handle unexpected bandwidth decreases that occur on timescales shorter than the minimum scheduling interval, which may affect message timeliness before the scheduler can adapt.
Scheduling Algorithm
Algorithm 1 Greedy Deadline-Feasible Semantic Allocation
Require: Pending messages ; utility–cost profiles ; deadlines ; link rate
Ensure: Transmission sizes and transmission order
,
repeat
for all such that do
;
if then
end if
end for
if then
break
end if
until no feasible token increment remains
messages in sorted by increasing deadline
return
function
messages in sorted by increasing deadline
for do
if then
return False
end if
end for
return True
end function
We provide the detailed scheduling procedure used by SCALE in Algorithm 1. We define for all messages . Our design separates semantic prioritization from deadline feasibility. At each iteration, it considers one additional token for every pending message and admits only increments for which all currently selected messages remain deadline-feasible under EDF ordering. Among these feasible increments, it selects the one with the largest marginal semantic gain per communication cost. The procedure terminates when no additional token can be admitted without causing a deadline violation.
Detailed Settings
CooperScene. CooperScene (Wu et al., 2026) is a real-world cooperative autonomy benchmark collected using three connected vehicles and one roadside unit. It provides synchronized LiDAR, camera, and GNSS/IMU observations, together with C-V2X communication measurements. The dataset contains approximately 59,000 LiDAR frames, 53,000 image frames, and 344,000 annotated 3D bounding boxes. Its scenes cover intersections, freeway ramps, local streets, and parking lots, supporting cooperative 3D object detection and motion prediction.
OPV2V. OPV2V (Xu et al., 2022c) is a simulated cooperative perception benchmark generated using OpenCDA and CARLA. It contains 73 scenes across eight CARLA towns and a digital replica of Culver City, with 11,464 frames and 232,913 annotated 3D vehicle bounding boxes. Each frame includes observations from two to seven connected vehicles, each equipped with a 64-channel LiDAR, four RGB cameras providing a combined 360-degree view, and GPS/IMU sensors. The dataset supports cooperative 3D object detection, BEV semantic segmentation, tracking, and prediction.
LIBERO. LIBERO ([11]) is a benchmark for knowledge transfer in language-conditioned robotic manipulation. It contains 130 tasks organized into four suites: LIBERO-Spatial, LIBERO-Object, and LIBERO-Goal, each containing 10 tasks, and LIBERO-100, containing 100 tasks. The suites vary spatial relationships, object identities, and manipulation goals to evaluate different forms of knowledge transfer. The benchmark provides language instructions, RGB observations from workspace and wrist cameras, and robot proprioceptive states.
Detailed experimental settings. Table 5 presents the detailed experimental settings for our evaluations on these benchmarks, including the evaluated tasks, test set sizes, task rates, deadlines, and bandwidth configurations.
Additional Results
Scene examples and CDFs. Figure 6 and Figure 7 compare the detection results of Full-send and SCALE against ground-truth annotations on CooperScene and OPV2V, respectively. Figure 5 illustrates a successful LIBERO manipulation episode with SCALE. Figure 8, Figure 9, and Figure 10 present the message lateness CDFs for CooperScene, OPV2V, and LIBERO, respectively, under fixed and varying bandwidth settings.

Figure 5. LIBERO task execution with SCALE at 10.1 Mbps: the robot picks up the soup can and places it in the basket. The sequence shows the start, midpoint, and successful completion of one test episode.
Figure 6. Vehicle detection on CooperScene at 25 Mbps. (a) Receiver camera. (b–c) Full-send and SCALE detections for the same frame, using the same CoBEVT model. Green boxes are detections; dashed blue boxes are ground truth. Both BEV panels use the same view.
Figure 7. Vehicle detection on OPV2V at 31 Mbps. (a) Receiver camera. (b–c) Full-send and SCALE detections for the same frame, using the same CoBEVT model. Green boxes are detections; dashed blue boxes are ground truth. Both BEV panels use the same view.
Runtime overhead. Table 6 reports the computational overhead of SCALE on CooperScene across five bandwidth settings. We measure runtime on an Intel Core i9-13900K using four CPU cores and four OpenBLAS threads. We report the overhead of the three design stages: semantic change evaluation, utility–cost profiling, and real-time scheduling, with the last stage also accounting for message preparation and state updates. Averaged over all generated detection and prediction messages, the total overhead remains below 1 ms per message across all settings, ranging from 0.813 to 0.884 ms. These results highlight the low computational overhead of SCALE.
| Bandwidth | Change evaluation | Utility–cost profiling | State update + scheduling | Total |
| 15 Mbps | 0.351 | 0.130 | 0.333 | 0.813 |
| 20 Mbps | 0.355 | 0.130 | 0.374 | 0.859 |
| 25 Mbps | 0.353 | 0.130 | 0.390 | 0.873 |
| 30 Mbps | 0.354 | 0.130 | 0.401 | 0.884 |
| Varying | 0.352 | 0.129 | 0.392 | 0.873 |
Table 6. Mean runtime per message of SCALE on CooperScene (ms).
AI Usage Statement
We used AI assistants to support the implementation and debugging of the experimental code described in Section 5. We also used AI assistants for grammar checking and revising the manuscript. The authors reviewed the AI-assisted code and text and tested the code. The authors take full responsibility for the experimental results and the final manuscript.
Full-send reference rates are estimated from validation or calibration data.
References
- [1]Qing Cai, Yiqing Zhou, Ling Liu, Hanxiao Yu, Yihao Wu, Ningzhe Shi, and Jinglin Shi. 2025. Query-aware semantic encoder-based resource allocation in task-oriented communications. IEEE Transactions on Mobile Computing, 24(7):6413–6429.DOI
- [2]Xiaoyu Chen, Fengge Wu, Junsuo Zhao, and Yun Fan. 2026. Condenseflow: Scalable latent space collaboration via semantic compression for multi-agent systems. In Findings of the Association for Computational Linguistics: ACL 2026, pages 13694–13712.DOI
- [3]Zhuoyun Du, Runze Wang, Huiyu Bai, Zouying Cao, Xiaoyang Zhu, Yu Cheng, Bo Zheng, Wei Chen, and Haochao Ying. 2026. Enabling agents to communicate entirely in latent space. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 27106–27129, San Diego, California, United States. Association for Computational Linguistics.
- [4]Tianyu Fu, Zihan Min, Hanling Zhang, Jichao Yan, Guohao Dai, Wanli Ouyang, and Yu Wang. 2026. Cache-to-cache: Direct semantic communication between large language models. In International Conference on Learning Representations, volume 2026, pages 43130–43158.arxiv.org/abs/2510.03215
- [5]Deniz Gündüz, Zhijin Qin, Inaki Estella Aguerri, Harpreet S Dhillon, Zhaohui Yang, Aylin Yener, Kai Kit Wong, and Chan-Byoung Chae. 2022. Beyond transmitting bits: Context, semantics, and task-oriented communications. IEEE Journal on Selected Areas in Communications, 41(1):5–41.arxiv.org/abs/2207.09353
- [6]Yue Hu, Shaoheng Fang, Zixing Lei, Yiqi Zhong, and Siheng Chen. 2022. Where2com: Communication-efficient collaborative perception via spatial confidence maps. In Advances in Neural Information Processing Systems, volume 35, pages 4874–4886. Curran Associates, Inc.arxiv.org/abs/2209.12836
- [7]Yue Hu, Xianghe Pang, Xiaoqi Qin, Yonina C Eldar, Siheng Chen, Ping Zhang, and Wenjun Zhang. 2026. Pragmatic communication in multi-agent collaborative perception. IEEE Transactions on Pattern Analysis and Machine Intelligence.DOI
- [8]Yue Hu, Juntong Peng, Sifei Liu, Junhao Ge, Si Liu, and Siheng Chen. 2024. Communication-efficient collaborative perception via information filling with codebook. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15481–15490. IEEE.arxiv.org/abs/2405.04966
- [9]Zixing Lei, Shunli Ren, Yue Hu, Wenjun Zhang, and Siheng Chen. 2022. Latency-aware collaborative perception. In European Conference on Computer Vision, pages 316–332. Springer.arxiv.org/abs/2207.08560
- [10]Yiming Li, Shunli Ren, Pengxiang Wu, Siheng Chen, Chen Feng, and Wenjun Zhang. 2021. Learning distilled collaboration graph for multi-agent perception. Advances in Neural Information Processing Systems, 34:29541–29552.arxiv.org/abs/2111.00643
- [11]Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. 2023a. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776–44791.arxiv.org/abs/2306.03310
- [12]Chuanhong Liu, Caili Guo, Yang Yang, and Nan Jiang. 2023b. Adaptable semantic compression and resource allocation for task-oriented communications. IEEE Transactions on Cognitive Communications and Networking, 10(3):769–782.arxiv.org/abs/2204.08910
- [13]Genjia Liu, Anning Hu, Yue Hu, Wenjun Zhang, and Siheng Chen. 2026. Rate-distortion optimized pragmatic communication for collaborative perception. In International Conference on Learning Representations, volume 2026, pages 119998–120019.arxiv.org/abs/2509.21994
- [14]Yen-Cheng Liu, Junjiao Tian, Nathaniel Glaser, and Zsolt Kira. 2020. When2com: Multi-agent perception via communication graph grouping. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4105–4114. IEEE.arxiv.org/abs/2006.00176
- [15]Shilpa Mukhopadhyay, Amit Roy-Chowdhury, and Hang Qiu. 2026. COOPERTRIM: Adaptive data selection for uncertainty-aware cooperative perception. In The Fourteenth International Conference on Learning Representations.arxiv.org/abs/2602.13287
- [16]Shivansh Patel, Saim Wani, Unnat Jain, Alexander G. Schwing, Svetlana Lazebnik, Manolis Savva, and Angel X. Chang. 2021. Interpretation of emergent communication in heterogeneous collaborative embodied agents. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15953–15963.arxiv.org/abs/2110.05769
- [17]Yalin E. Sagduyu and Tugba Erpek. 2026. When semantic communication meets queueing: Cross-layer latency and task fidelity optimization. Preprint, arXiv:2605.05514.arxiv.org/abs/2605.05514
- [18]Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. 2022. Motion transformer with global intention localization and local movement refinement. Advances in Neural Information Processing Systems, 35:6531–6543.arxiv.org/abs/2209.13508
- [19]Xiangyu Shi, Marco Chiesa, Gerald Q. Maguire, Jr., and Dejan Kostić. 2026. KVComm: Enabling efficient LLM communication through selective KV sharing. In The Fourteenth International Conference on Learning Representations.arxiv.org/abs/2510.03346







