betaFeedback

WorksComputer Science

SCALE: Real-Time Latent Communication for Cyber-Physical Agents

Abstract

Latent communication enables agents to collaborate efficiently by directly exchanging intermediate model representations rather than raw sensor data or text messages. Existing approaches, however, are primarily designed for LLM agents and do not consider the bandwidth constraints and heterogeneous task deadlines faced by cyber-physical agents. In this work, we propose SCALE, a real-time latent communication method tailored for cyber-physical agents. Motivated by the spatio-temporal continuity of the physical world, SCALE matches tokens across consecutive latent messages using similarity scores to identify reusable information and avoid redundant transmission. It then profiles the semantic gain and communication cost of transmitting different amounts of latent information. Based on these profiles, SCALE employs a real-time scheduler that jointly determines which pending messages to transmit and how many latent tokens to send, subject to available bandwidth and task deadlines. We evaluate SCALE across three cyber-physical benchmarks and a physical spider robot case study under diverse network conditions and heterogeneous task deadlines. Our results show that SCALE significantly improves both task performance and on-time delivery in bandwidth-constrained networks. Our code is available at: https://anonymous.4open.science/r/SCALE-4E80/.

Introduction

Communication is fundamental to cyber-physical agents. Individual agents often possess only partial observations or complementary capabilities, and therefore must exchange information with one another to accomplish diverse cyber-physical tasks [16, 6]. However, transmitting raw sensor observations incurs substantial communication overhead, while natural-language communication introduces additional computational overhead from generating text at the sender and interpreting it at the receiver. These limitations make conventional communication inefficient for real-time multi-agent systems.

Existing Literature and the Gap. Latent communication has emerged as a potential “AI language”, allowing agents to directly exchange hidden states or intermediate representations to improve efficiency [3, 19] (Zou et al. 2026; Ye et al. 2025). However, existing methods are primarily designed for LLM-based agents and focus on reducing model-side computation, with limited consideration of communication bandwidth and task deadlines. Meanwhile, communication-efficient collaborative-perception methods consider bandwidth and delay [14, 6, 8] (Yang et al. 2023), but primarily focus on task-specific perception rather than multiple concurrent CPS tasks sharing a bandwidth-constrained link. Thus, latent communication for CPS agents under practical network constraints and heterogeneous task deadlines remains largely underexplored.

Our Work. We propose SCALE, a real-time latent communication method tailored for cyber-physical agents. SCALE exploits the spatiotemporal continuity of the physical world to reuse latent information already available at the receiver, reducing redundant transmission. It employs a real-time scheduler that jointly determines which messages to transmit and how many latent tokens to send based on semantic gain, available bandwidth, and task deadlines, thereby improving both task utility and message on-time delivery. In addition, SCALE is lightweight and operates on latent tokens produced by existing modality encoders without modifying the underlying models, supporting different sensory modalities and downstream CPS tasks.

Designing SCALE addresses three technical challenges. **First, spatiotemporal variation makes it difficult to determine what information has actually changed. Physical observations often evolve continuously, but object, agent, or viewpoint motion can make fixed-position latent comparisons overestimate motion for semantic change. SCALE therefore performs semantic change evaluation by matching current latent tokens against the receiver’s acknowledged latent state and measuring change relative to their best correspondences. Second, the amount of latent information to transmit must adapt to each message. The semantic gain from transmitting additional latent tokens varies across messages, making a fixed transmission size inefficient. SCALE therefore performs utility–cost profiling for each message, ranking latent tokens by semantic change and characterizing the cumulative semantic gain and communication cost at different transmission sizes to guide real-time scheduling. Third, limited and time-varying bandwidth must be allocated across messages with heterogeneous deadlines. Allocating more latent tokens to one message increases its semantic gain but may delay other messages beyond their deadlines. SCALE therefore employs a real-time scheduler that greedily allocates latent tokens based on marginal semantic gain per communication cost, while maintaining deadline feasibility for all selected messages under earliest-deadline-first (EDF) ordering.

We implement and evaluate SCALE on CooperScene (Wu et al., 2026), OPV2V (Xu et al., 2022c), and LIBERO (Liu et al., [11]), covering autonomous driving and robotic manipulation benchmarks. We further conduct a physical spider robot case study. Our evaluation considers constrained bandwidth, time-varying link capacity, and background traffic, with heterogeneous task deadlines. We compare SCALE against full latent transmission, uniform compression, and four state-of-the-art methods (Xu et al., 2025, 2022b; Hu et al., [6]; Mukhopadhyay et al., [15]). Our results show that SCALE achieves the highest task performance in almost all settings while significantly improving on-time delivery rates (see Figure 1 for an example).

Figure 1. Comparison between SCALE and other latent communication baselines on the OPV2V benchmark at 15 Mbps. SCALE achieves the best mIoU scores and message on-time rates.

We make the following contributions:

  • We identify and formulate the problem of real-time latent communication for CPS agents, where latent messages from multiple downstream tasks with heterogeneous deadlines share a bandwidth-constrained communication link, a setting not adequately addressed by existing methods.

  • We propose SCALE, a real-time latent communication method that combines spatiotemporal latent reuse, utility–cost profiling, and deadline-aware scheduling to improve both task utility and message on-time delivery under bandwidth constraints.

  • We implement and evaluate SCALE across three CPS benchmarks and a physical spider robot case study under diverse network conditions, demonstrating its effectiveness in both benchmark evaluations and a physical robotic system.

Related Work

LLM-based latent communication. Recent LLM-based latent communication methods enable multi-agent systems to exchange latent representations, such as hidden states or KV caches, to reduce model inference and computation overhead ([3, 19, 4, 2] (Zou et al. 2026; Ye et al. 2025)). These approaches primarily focus on efficient latent interaction and reducing model-side computation, with limited consideration of the underlying communication network. SCALE instead studies latent communication over resource-constrained networks, a practical setting for cyber-physical agents.

Communication-efficient cyber-physical systems. Communication-efficient collaborative perception has been extensively studied in cyber-physical systems, where agents exchange intermediate features to improve shared perception (Wang et al., 2020; Li et al. [10]. Existing methods explore when, where, and how agents should communicate through communication triggering (Liu et al. [14]), spatial feature selection (Hu et al. [6]), and feature sparsification (Yang et al. (2023); Hu et al. [8]). Other work improves robustness to communication delay and asynchrony (Lei et al. [9]; Wei et al. (2023)) or exploits temporal redundancy to reduce repeated transmission (Mukhopadhyay et al. [15]). Recent methods further improve the perception–communication tradeoff through adaptive feature selection (Xu et al. (2025); Hu et al. [7]; Liu et al. [13]). In the semantic communication literature, existing task-oriented methods tailor transmitted representations according to downstream task utility (Xie et al. (2021); Gündüz et al. [5]), jointly allocate communication resources under bandwidth constraints (Zhang et al. (2023); Liu et al. [12]; Cai et al. [1]), and account for communication latency (Sagduyu and Erpek [17]). However, these methods are primarily designed for individual perception or communication tasks, rather than the broader CPS setting where multiple downstream tasks with heterogeneous deadlines coexist and share a bandwidth-constrained communication link.

Problem Formulation

System Model. Following common settings in prior works (Liu et al. [14]; Hu et al. [6]; Yang et al. (2023)), we consider cyber-physical agents that locally encode their observations and exchange latent representations over a communication link. For an observation xix_i, a local encoder produces a latent representation:

Hi=E(xi)={hi,1,…,hi,Ni},(1)H_i = E(x_i) = \{h_{i,1}, \ldots, h_{i,N_i}\}, \tag*{(1)}

where each hi,j∈Rdh_{i,j} \in\mathbb{R}^d is a fixed-dimensional latent token, and NiN_i is the token length. Encoders may be modality-specific, e.g., for image, audio, or text inputs. Within each agent, multiple downstream tasks may coexist and generate latent messages that share the same communication link.

Network and Timing Model. We denote the available rate of the communication link by R(t)R(t), which may vary over time. Within each agent, multiple latent messages from different tasks compete for the same communication resources. We denote the set of messages pending transmission at time tt by Q(t)\mathcal{Q}(t) and the deadline of message ii by did_i. Let CiC_i denote the transmission completion time of message ii. Following a common firm-deadline model in real-time systems, we assume that a message contributes to its downstream task only if the selected latent information arrives before its deadline, i.e., Ci≤diC_i \le d_i.

Scheduling Objective. At time tt, the scheduler considers all messages in the pending set Q(t)\mathcal{Q}(t). For each message i∈Q(t)i \in\mathcal{Q}(t), let Ki≤NiK_i \le N_i denote the number of latent tokens selected for transmission, where Ki=0K_i = 0 indicates that the message is not selected. We denote the semantic gain and communication cost of transmitting KiK_i tokens by Ui(Ki)U_i(K_i) and Bi(Ki)B_i(K_i), respectively. Our goal is to determine which pending messages to transmit and how many tokens to transmit for each message, so as to maximize the total semantic gain delivered before their respective deadlines:

max⁡{Ki}∑i∈Q(t)Ui(Ki)1[Ci≤di],(2)\max_{\{K_i\}} \sum_{i \in\mathcal{Q}(t)} U_i(K_i)\mathbf{1}[C_i \le d_i], \tag*{(2)}

subject to available bandwidth constraints.

Approach

Overview

SCALE consists of three stages: semantic change evaluation, utility–cost profiling, and real-time scheduling. In the first two stages, it identifies latent information that is new relative to the receiver’s acknowledged state, and characterizes the semantic benefit and communication cost of different transmission lengths. The scheduler then combines these profiles with available bandwidth and task deadlines to determine which message to transmit and how many latent tokens to send. We demonstrate the system flow of SCALE in Figure 2.

Figure 2. SCALE system design. Semantic change evaluation and utility–cost profiling guide the real-time scheduler in jointly selecting messages and transmission sizes under bandwidth and deadline constraints.

Semantic Change Evaluation

Physical observations often evolve continuously, creating substantial redundancy between consecutive observations. In this stage, we aim to measure such semantic redundancy and identify the nonredundant information to transmit. A key challenge is that directly comparing two latent messages token by token can be unreliable because their corresponding observations may be misaligned due to object or viewpoint motion. SCALE therefore first establishes token-level correspondences between the current latent message and the receiver’s acknowledged latent state before quantifying their semantic changes.

Therefore, for each current token hi,jh_{i,j}, we first find the most similar token in the receiver’s acknowledged latent state:

μi(j)=arg⁡max⁡kcos⁡(hi,j,h~i,krecv).(3)\mu_i(j) = \arg\max_k \cos\left(h_{i,j}, \tilde{h}^{\mathrm{recv}}_{i,k}\right). \tag*{(3)}

Let ci,jc_{i,j} denote the corresponding maximum cosine similarity. We then quantify the semantic change of token hi,jh_{i,j} as:

si,j=1−ci,j.(4)s_{i,j} = 1 - c_{i,j}. \tag*{(4)}

Consequently, highly similar tokens receive small change scores, while tokens with low similarity receive larger scores. We further use a similarity threshold τ\tau to determine whether the matched receiver-side token can be reused: a match is reusable only when ci,j≥τc_{i,j} \ge\tau. Matching is performed against the receiver’s acknowledged state to ensure that only information confirmed to be available at the receiver is reused.

Utility–Cost Profiling

The semantic change scores obtained above capture the amount of new information carried by individual latent tokens. In this stage, we further characterize the trade-off between semantic gain and communication cost at different transmission sizes KK, where K≤NiK \le N_i denotes the number of latent tokens transmitted for message ii.

Specifically, we prioritize tokens with larger semantic change scores by sorting them in descending order according to si,js_{i,j}. We define the cumulative semantic gain and communication cost as:

Ui(K)=∑k=1Ksi,(k),Bi(K)=Kbtok+Bimeta(K),(5)\begin{aligned} U_i(K) &= \sum_{k=1}^{K} s_{i,(k)},\\ B_i(K) &= Kb_{\mathrm{tok}} + B_i^{\mathrm{meta}}(K), \tag*{(5)} \end{aligned}

where si,(k)s_{i,(k)} denotes the kk-th largest semantic change score, btokb_{\mathrm{tok}} is the serialized size of one latent token, and Bimeta(K)B_i^{\mathrm{meta}}(K) accounts for the meta-data required to interpret the transmission, including token indices, correspondence mappings, and the reference-state version. The resulting pairs (Ui(K),Bi(K))(U_i(K), B_i(K)) characterize the available transmission choices for message ii. As a result, a larger KK generally preserves more new semantic information but incurs greater communication cost.

Real-Time Scheduling

In the final stage, the scheduler takes as input the utility–cost profiles {(Ui(K),Bi(K))}\{(U_i(K), B_i(K))\} and task deadlines of all messages in Q(t)\mathcal{Q}(t), together with the current link rate R(t)R(t). Based on this information, it determines the transmission size KiK_i for each message and their transmission order.

SCALE first determines the transmission size KiK_i for each pending message. Following the scheduling objective in § 3, we employ a greedy algorithm to maximize semantic gain under the deadline constraints. Specifically, we initialize Ki=0K_i = 0 for each pending message and iteratively allocate tokens one at a time. Suppose message ii has already been allocated KiK_i tokens. The marginal semantic gain per communication cost of adding its next token k=Ki+1k = K_i + 1 is:

ρi,k=Ui(k)−Ui(k−1)Bi(k)−Bi(k−1).(6)\rho_{i,k} = \frac{U_i(k) - U_i(k-1)}{B_i(k) - B_i(k-1)}. \tag*{(6)}

At each step, the scheduler considers the next token of every pending message and prioritizes the one with the largest ρi,k\rho_{i,k}, subject to the message deadlines. To check deadline feasibility, let π1,…,πm\pi_1,\ldots,\pi_m denote the currently selected messages ordered by earliest deadline first (EDF). The completion time of message πq\pi_q is estimated as:

Cπq=t+∑ℓ=1q8Bπℓ(Kπℓ)R(t).(7)C_{\pi_q}=t+\sum_{\ell=1}^{q}\frac{8B_{\pi_\ell}(K_{\pi_\ell})}{R(t)}. \tag*{(7)}

An allocation is feasible only if Cπq≤dπqC_{\pi_q} \le d_{\pi_q} for all q=1,…,mq = 1, \ldots, m. Among all feasible token increments, the scheduler selects the one with the largest ρi,k\rho_{i,k}, updates Ki←Ki+1K_i \leftarrow K_i + 1, and repeats this process until no additional token can be allocated without violating a deadline.

The resulting KiK_i values determine the transmission size of each message, where Ki=0K_i = 0 indicates that the message is not selected. Messages with Ki>0K_i > 0 are transmitted in EDF order. In this way, SCALE prioritizes semantic gain relative to communication cost while using task deadlines as feasibility constraints on both message selection and transmission size. After transmission, the receiver updates its latent state with the received tokens and reuses matched tokens from its previous state according to the correspondences established in the semantic change evaluation stage. The receiver then acknowledges the updated state, allowing the sender to maintain an accurate view of the receiver’s state for subsequent communication. The detailed scheduling algorithm can be found in Appendix § 1.

Evaluation

Experimental Settings

CPS Benchmarks. We evaluate SCALE on three cyber-physical benchmarks: CooperScene (Wu et al., 2026), OPV2V (Xu et al., 2022c), and LIBERO (Liu et al., [11]). CooperScene is a real-world cooperative autonomous-driving benchmark with LiDAR and camera observations from vehicles and roadside infrastructure. OPV2V is a simulated cooperative-perception benchmark with LiDAR and camera observations from multiple vehicles. LIBERO provides language-conditioned robotic manipulation tasks with visual observations and robot states. These benchmarks cover different CPS use cases, and their detailed descriptions are provided in Appendix § 8.

Tasks and Models. On CooperScene, we evaluate concurrent 3D object detection and 5-sec trajectory prediction tasks, with deadlines of 100 ms and 200 ms. We use CoBEVT (Xu et al., 2022a) for detection and a frozen, pretrained Motion Transformer (MTR) (Shi et al., [18]) for prediction. On OPV2V, we evaluate concurrent 3D object detection and BEV semantic segmentation using the LiDAR-based detection and camera-based segmentation variants of CoBEVT, each with its own pretrained weights. Their deadlines are 100 ms and 200 ms, respectively. On LIBERO, we evaluate object manipulation using SmolVLA (Shukor et al., 2025), with the deadline determined by the time remaining before the robot exhausts its buffered actions, plus reserving 100 ms for inference and 10 ms for returning actions. Across all benchmarks, a sensing agent sends latent features to another agent or remote server for inference.

Network Settings. For each benchmark, we evaluate four fixed-bandwidth settings and one time-varying setting. The fixed rates correspond to different fractions (e.g., 20%, 50%) of the average bandwidth required for full latent transmission in each benchmark. In the time-varying setting, the link alternates between low and high rates every 2 seconds. In all settings, we account for both queueing and transmission time when computing communication delay. More detailed network settings, including the bandwidth values for each benchmark, are provided in Table 5 in the Appendix.

BenchmarkTaskTest workloadRate (Hz)Deadline (ms)
CooperScene3D detection3,936 jobs10100
Trajectory prediction (5 s)18,693 targets5200
OPV2V3D detection528 jobs10100
BEV segmentation265 jobs5200
LIBEROObject manipulation45 episodes20Variable
BenchmarkFull-sendFixed ratesFractions (%)Varying rates
CooperScene118.515, 20, 25, 3012.5, 16.7, 20.8, 2530 ↔ 200
OPV2V153.08, 15, 31, 615, 10, 20, 408 ↔ 61
LIBERO13.463.4, 5.0, 6.7, 10.125, 37.5, 50, 753.4 ↔ 20

Table 5. Detailed experiment settings.

Baselines. We compare SCALE with Full-send, which transmits the complete latent message, and Uniform compression, which uniformly selects a fraction of latent features for transmission based on the available bandwidth. On CooperScene, we additionally compare with CoSDH (Xu et al., 2025) and V2X-ViT (Xu et al., 2022b). On OPV2V, we additionally compare with CooperTrim (Mukhopadhyay et al., [15]) and Where2comm (Hu et al., [6]). All baseline methods use earliest-deadline-first (EDF) transmission ordering.

Metrics. We evaluate SCALE in terms of task utility and message on-time rate. Task utility measures downstream task performance, while on-time rate is the percentage of messages received before their deadlines.

Benchmark Performance

CooperScene. We use task-specific performance metrics to evaluate the two tasks on CooperScene. For the 3D object detection task, we report average precision (AP) and the message on-time rate, where AP is computed using an intersection-over-union (IoU) threshold of 0.5 to match predicted boxes to ground-truth boxes. For the trajectory prediction task, we report task success and the message on-time rate, where task success is the percentage of prediction targets for which at least one of six predicted trajectories is within 2 m of the ground-truth position.

Table 1 presents the results. The results show that SCALE achieves the highest detection AP and prediction task success across all settings. In particular, for 3D object detection at 20 Mbps, SCALE achieves 51.53 AP, exceeding the strongest baseline, CoSDH, by 20.19 points. It also delivers 80.36% of detection messages on time, compared with 48.02% for CoSDH. For trajectory prediction under fixed bandwidth, SCALE maintains task success between 60.82% and 60.85%, while delivering at least 99.82% of prediction messages on time. Under varying bandwidth, SCALE achieves 56.09 detection AP and 60.81% prediction task success, with detection and prediction on-time rates of 94.52% and 99.71%, respectively.

Method15 Mbps20 Mbps25 Mbps30 MbpsVarying
AP ↑\uparrowOT ↑\uparrowAP ↑\uparrowOT ↑\uparrowAP ↑\uparrowOT ↑\uparrowAP ↑\uparrowOT ↑\uparrowAP ↑\uparrowOT ↑\uparrow
Full-send22.770.0022.770.0022.770.0022.770.0040.2649.97
Uniform22.5068.4223.3868.6423.7268.6324.9168.5039.2581.59
CoSDH27.9932.0431.3448.0240.6768.1443.3375.4349.8789.05
V2X-ViT25.240.0025.240.0025.331.2125.918.7042.2255.01
SCALE45.6264.9551.5380.3654.8788.1356.2892.8456.0994.52
Method15 Mbps20 Mbps25 Mbps30 MbpsVarying
Succ. ↑\uparrowOT ↑\uparrowSucc. ↑\uparrowOT ↑\uparrowSucc. ↑\uparrowOT ↑\uparrowSucc. ↑\uparrowOT ↑\uparrowSucc. ↑\uparrowOT ↑\uparrow
Full-send43.720.2644.736.5449.5735.3248.5025.0554.8864.89
Uniform59.4993.8159.5294.4459.8795.6459.8495.9060.4898.47
CoSDH46.4816.6951.2844.2652.8650.9957.3780.9259.2491.47
V2X-ViT48.5429.2346.6518.5349.7536.3752.4150.9557.0777.64
SCALE60.8299.8260.85100.0060.85100.0060.85100.0060.8199.71

Table 1. CooperScene results on 3,936 detection jobs and 18,693 prediction targets. AP denotes average precision, OT denotes the on-time rate (%), and Succ. denotes prediction success (%).

OPV2V. Similarly, we use task-specific performance metrics to evaluate the two tasks on OPV2V. For the 3D object detection task, we report AP and the message on-time rate. We evaluate BEV semantic segmentation using mean intersection-over-union (mIoU) and also report its message on-time rate, where mIoU averages the IoU between predicted and ground-truth regions across semantic classes.

Table 2 presents the results. The results show that SCALE achieves the highest segmentation mIoU and segmentation on-time rate across all settings, and the highest detection AP except at 8 Mbps. In this setting, SCALE achieves 83.41 AP, slightly below CooperTrim’s 84.25 AP, while delivering 37.74% of detection messages on time, compared with 0.73% for CooperTrim. For BEV semantic segmentation, the largest mIoU improvement over the strongest baseline occurs at 15 Mbps, where SCALE achieves 47.14 mIoU, exceeding Where2comm by 9.97 points. It also delivers 97.09% of segmentation messages on time, compared with 7.82% for Where2comm.

Method8 Mbps15 Mbps31 Mbps61 MbpsVarying
AP ↑\uparrowOT ↑\uparrowAP ↑\uparrowOT ↑\uparrowAP ↑\uparrowOT ↑\uparrowAP ↑\uparrowOT ↑\uparrowAP ↑\uparrowOT ↑\uparrow
Full-send76.370.0076.370.0076.370.0076.370.0076.370.00
Uniform78.2946.4979.8744.2177.8041.2081.9039.9379.6542.30
CooperTrim84.250.7384.394.9285.8128.7189.4769.5586.1431.81
Where2comm82.250.8282.6310.3088.0954.4291.8886.5186.2036.37
SCALE83.4137.7486.4250.2390.3075.3992.5291.7086.4456.43
Method8 Mbps15 Mbps31 Mbps61 MbpsVarying
mIoU ↑\uparrowOT ↑\uparrowmIoU ↑\uparrowOT ↑\uparrowmIoU ↑\uparrowOT ↑\uparrowmIoU ↑\uparrowOT ↑\uparrowmIoU ↑\uparrowOT ↑\uparrow
Full-send27.020.0027.140.7328.8010.3631.7727.8228.7411.09
Uniform27.1077.8226.9581.0928.9379.2732.6978.0028.6776.18
CooperTrim33.0812.3635.5232.3640.8767.6443.3192.7337.7548.55
Where2comm37.140.1837.177.8237.7652.0038.3293.6437.5642.55
SCALE46.4894.9147.1497.0947.7199.6447.78100.0045.1790.18

Table 2. OPV2V results on 528 detection jobs and 265 segmentation jobs. AP denotes average precision, mIoU denotes mean IoU score, and OT denotes on-time rate (%).

LIBERO. Our LIBERO evaluation focuses on a single language-conditioned robotic manipulation task. We report task success, defined as the percentage of episodes completed within 280 steps, and the communication on-time rate.

Table 3 presents the results. The results show that SCALE achieves the highest task success and on-time rate across all settings. At the lowest tested bandwidth of 3.4 Mbps, SCALE completes 44.44% of episodes, while both baselines fail all episodes. In particular, Uniform compression achieves a similar on-time rate (74.47% versus 77.76% for SCALE), showing that timely delivery alone does not ensure task success. The largest task success improvement over the strongest baseline occurs at 5.0 Mbps, where SCALE achieves 86.67% task success, compared with 24.44% for Full-send and

BandwidthMethodTask success ↑On-time ↑
3.4 MbpsFull-send0.00%0.00%
Uniform0.00%74.47%
SCALE44.44%77.76%
5.0 MbpsFull-send24.44%25.24%
Uniform0.00%74.47%
SCALE86.67%84.57%
6.7 MbpsFull-send60.00%25.98%
Uniform0.00%74.47%
SCALE95.56%88.81%
10.1 MbpsFull-send88.89%51.52%
Uniform42.22%74.85%
SCALE97.78%94.21%
VaryingFull-send88.89%65.87%
Uniform0.00%77.45%
SCALE100.00%89.28%

Table 3. LIBERO results over 45 episodes per method and bandwidth setting. We report the task success rate (%) and on-time rate (%).

0.00% for Uniform compression. Its communication on-time rate also reaches 84.57%, exceeding both baselines.

Real-Time Performance

We measure message lateness as Li=Ci−diL_i = C_i - d_i, where CiC_i and did_i are the completion time and deadline of message ii, respectively. On-time messages have Li≤0L_i \le0, while delayed messages have Li>0L_i > 0. To better illustrate real-time performance, Figure 3 shows the lateness distributions for CooperScene at 20 Mbps for both object detection and trajectory prediction tasks. In the figure, the cumulative percentage at zero gives the on-time rate, which is computed over all generated messages, including undelivered ones.

Figure 3. Message-lateness CDFs for CooperScene at 20 Mbps: (a) object detection and (b) trajectory prediction. The vertical dashed line marks each task’s deadline; the cumulative percentage at zero gives the on-time rate.

The results show that SCALE delivers 80.36% of detection messages and 100.00% of prediction messages on time, compared with 68.64% and 94.44% for Uniform compression. Uniform compression achieves the highest on-time rates among the baselines at this bandwidth, but its task utility is significantly lower than that of SCALE. We also observe a steep rise in the detection CDF just before the deadline for the object detection task, indicating that many messages complete shortly before their deadlines. This behavior is consistent with SCALE’s greedy allocation of available bandwidth to maximize semantic gain under deadline constraints. The complete distributions for all three benchmarks and bandwidth settings appear in Appendix § 9 (Figure 8, Figure 9, and Figure 10). Across these results, SCALE improves on-time delivery in every tested setting and achieves the highest on-time rate among all methods in most settings.

Figure 8. CooperScene message-lateness CDFs at the five selected bandwidth settings: detection (left) and prediction (right). Varying switches between 30 and 200 Mbps every 2 seconds. Zero marks each message’s deadline; the CDF at zero is its method’s on-time rate.

Figure 9. OPV2V message-lateness CDFs at the five selected bandwidth settings: detection (left) and segmentation (right). Varying switches between 8 and 61 Mbps every 2 seconds. Zero marks each message’s deadline; the CDF at zero is its method’s on-time rate.

Figure 10. LIBERO camera-update lateness CDFs at the five selected bandwidth settings, using all 45 episodes per method and setting. Varying switches between 3.4 and 20 Mbps every 2 seconds. Zero marks each message’s deadline; the CDF at zero is its method’s on-time rate.

Case Study

We conduct a hardware-in-the-loop case study using a Raspberry Pi-based spider robot, as shown in Figure 4. The robot is equipped with a camera and an ultrasonic sensor and performs two tasks: object detection and distance measurement on a fixed robot patrolling route. The patrolling route is a 55.4-second walk through a miniature city containing objects such as small road signs, pedestrian figures, and buildings. Among the two tasks, object detection is time-sensitive, with a communication deadline of 1,000 ms, while ultrasonic distance measurements generate background traffic over the shared link. We compare SCALE against two baselines: Raw RGB, which transmits uncompressed images, and Full-send, which transmits complete intermediate latent features. We evaluate the system in three bandwidth settings, including 1, 2, and 8 Mbps.

Spider robot patrolling in the mini-city with pedestrians, stop signs, and model cars

Figure 4. Spider robot patrolling in the mini-city with pedestrians, stop signs, and model cars. The spider patrols along an L-shaped route.

We report task success and message on-time rates. Table 4 presents the results. We observe

MethodMetric128
RGBSuccess ↑0.00%0.00%100.00%
On-time ↑0.00%0.00%100.00%
LatentSuccess ↑0.00%0.00%100.00%
On-time ↑0.00%0.00%100.00%
SCALESuccess ↑77.62%91.67%100.00%
On-time ↑100.00%96.43%100.00%

Table 4. Spider robot mini-city case study results under different bandwidth constraints.

that SCALE substantially outperforms both baselines under constrained bandwidth. Specifically, at 1 and 2 Mbps, both baselines achieve zero detection success, whereas SCALE achieves success rates of 77.62% and 91.67%, respectively, with on-time rates of 100% and 96.43%. At 8 Mbps, all methods achieve 100% detection success and on-time delivery. These results demonstrate the effectiveness of SCALE in preserving timely detection performance on a physical robotic platform under limited bandwidth.

Conclusion

In this work, we present SCALE, a real-time latent communication method for cyber-physical agents. By exploiting spatiotemporal continuity, SCALE avoids retransmitting latent information already available at the receiver. To support timely communication, we design a real-time scheduler that jointly selects messages and transmission sizes based on semantic gain, available bandwidth, and task deadlines. We evaluate SCALE on three benchmarks and conduct a physical robot case study. The results demonstrate improved task performance and high message on-time rates under constrained and varying network conditions. These findings highlight SCALE as an effective and efficient method for meeting the communication and timing requirements of CPS agents.

Limitations

While SCALE significantly improves bandwidth efficiency and supports real-time scheduling for CPS agents, it has two limitations. First, the communication savings of SCALE depend on spatiotemporal redundancy in latent representations. However, observations in the physical world can change abruptly, which may reduce the amount of reusable information and increase the communication required to maintain task performance. Second, the scheduling algorithm assesses deadline feasibility using the current link rate. However, this design cannot handle unexpected bandwidth decreases that occur on timescales shorter than the minimum scheduling interval, which may affect message timeliness before the scheduler can adapt.

Scheduling Algorithm

Algorithm 1 Greedy Deadline-Feasible Semantic Allocation

Require: Pending messages Q(t)Q(t); utility–cost profiles {Ui(K),Bi(K)}\{U_i(K), B_i(K)\}; deadlines {di}\{d_i\}; link rate R(t)R(t)

Ensure: Transmission sizes {Ki}\{K_i\} and transmission order π\pi

  1. Ki←0K_i \gets0, ∀i∈Q(t)\forall i \in Q(t)

  2. repeat

  3. F←∅\mathcal{F} \gets\varnothing

  4. for all i∈Q(t)i \in Q(t) such that Ki<NiK_i < N_i do

  5. k←Ki+1k \gets K_i + 1

  6. ρi,k←Ui(k)−Ui(k−1)Bi(k)−Bi(k−1)\rho_{i,k} \gets\frac{U_i(k)-U_i(k-1)}{B_i(k)-B_i(k-1)}

  7. K′←KK' \gets K; Ki′←Ki+1K'_i \gets K_i + 1

  8. if DEADLINEFEASIBLE⁡(K′,t)\operatorname{DEADLINEFEASIBLE}(K',t) then

  9. F←F∪{(i,ρi,k)}\mathcal{F} \gets\mathcal{F} \cup\{(i,\rho_{i,k})\}

  10. end if

  11. end for

  12. if F=∅\mathcal{F} = \varnothing then

  13. break

  14. end if

  15. i∗←arg⁡max⁡(i,ρ)∈Fρi^* \gets\arg\max_{(i,\rho)\in\mathcal{F}} \rho

  16. Ki∗←Ki∗+1K_{i^*} \gets K_{i^*}+1

  17. until no feasible token increment remains

  18. S←{i∈Q(t)∣Ki>0}\mathcal{S} \gets\{i \in Q(t)\mid K_i>0\}

  19. π←\pi\gets messages in S\mathcal{S} sorted by increasing deadline

  20. return {Ki},π\{K_i\},\pi

  21. function DEADLINEFEASIBLE⁡(K,t)\operatorname{DEADLINEFEASIBLE}(K,t)

  22. S←{i∈Q(t)∣Ki>0}\mathcal{S} \gets\{i \in Q(t)\mid K_i>0\}

  23. (π1,…,πm)←(\pi_1,\ldots,\pi_m) \gets messages in S\mathcal{S} sorted by increasing deadline

  24. for q=1,…,mq=1,\ldots,m do

  25. Cπq←t+∑ℓ=1qδBπℓ(Kπℓ)R(t)C_{\pi_q} \gets t+\sum_{\ell=1}^{q}\frac{\delta B_{\pi_\ell}(K_{\pi_\ell})}{R(t)}

  26. if Cπq>dπqC_{\pi_q}>d_{\pi_q} then

  27. return False

  28. end if

  29. end for

  30. return True

  31. end function

We provide the detailed scheduling procedure used by SCALE in Algorithm 1. We define Ui(0)=U_i(0)= Bi(0)=0B_i(0) = 0 for all messages ii. Our design separates semantic prioritization from deadline feasibility. At each iteration, it considers one additional token for every pending message and admits only increments for which all currently selected messages remain deadline-feasible under EDF ordering. Among these feasible increments, it selects the one with the largest marginal semantic gain per communication cost. The procedure terminates when no additional token can be admitted without causing a deadline violation.

Detailed Settings

CooperScene. CooperScene (Wu et al., 2026) is a real-world cooperative autonomy benchmark collected using three connected vehicles and one roadside unit. It provides synchronized LiDAR, camera, and GNSS/IMU observations, together with C-V2X communication measurements. The dataset contains approximately 59,000 LiDAR frames, 53,000 image frames, and 344,000 annotated 3D bounding boxes. Its scenes cover intersections, freeway ramps, local streets, and parking lots, supporting cooperative 3D object detection and motion prediction.

OPV2V. OPV2V (Xu et al., 2022c) is a simulated cooperative perception benchmark generated using OpenCDA and CARLA. It contains 73 scenes across eight CARLA towns and a digital replica of Culver City, with 11,464 frames and 232,913 annotated 3D vehicle bounding boxes. Each frame includes observations from two to seven connected vehicles, each equipped with a 64-channel LiDAR, four RGB cameras providing a combined 360-degree view, and GPS/IMU sensors. The dataset supports cooperative 3D object detection, BEV semantic segmentation, tracking, and prediction.

LIBERO. LIBERO ([11]) is a benchmark for knowledge transfer in language-conditioned robotic manipulation. It contains 130 tasks organized into four suites: LIBERO-Spatial, LIBERO-Object, and LIBERO-Goal, each containing 10 tasks, and LIBERO-100, containing 100 tasks. The suites vary spatial relationships, object identities, and manipulation goals to evaluate different forms of knowledge transfer. The benchmark provides language instructions, RGB observations from workspace and wrist cameras, and robot proprioceptive states.

Detailed experimental settings. Table 5 presents the detailed experimental settings for our evaluations on these benchmarks, including the evaluated tasks, test set sizes, task rates, deadlines, and bandwidth configurations.

Additional Results

Scene examples and CDFs. Figure 6 and Figure 7 compare the detection results of Full-send and SCALE against ground-truth annotations on CooperScene and OPV2V, respectively. Figure 5 illustrates a successful LIBERO manipulation episode with SCALE. Figure 8, Figure 9, and Figure 10 present the message lateness CDFs for CooperScene, OPV2V, and LIBERO, respectively, under fixed and varying bandwidth settings.

LIBERO task execution with SCALE at 10.1 Mbps

Figure 5. LIBERO task execution with SCALE at 10.1 Mbps: the robot picks up the soup can and places it in the basket. The sequence shows the start, midpoint, and successful completion of one test episode.

Figure 6. Vehicle detection on CooperScene at 25 Mbps. (a) Receiver camera. (b–c) Full-send and SCALE detections for the same frame, using the same CoBEVT model. Green boxes are detections; dashed blue boxes are ground truth. Both BEV panels use the same view.

Figure 7. Vehicle detection on OPV2V at 31 Mbps. (a) Receiver camera. (b–c) Full-send and SCALE detections for the same frame, using the same CoBEVT model. Green boxes are detections; dashed blue boxes are ground truth. Both BEV panels use the same view.

Runtime overhead. Table 6 reports the computational overhead of SCALE on CooperScene across five bandwidth settings. We measure runtime on an Intel Core i9-13900K using four CPU cores and four OpenBLAS threads. We report the overhead of the three design stages: semantic change evaluation, utility–cost profiling, and real-time scheduling, with the last stage also accounting for message preparation and state updates. Averaged over all generated detection and prediction messages, the total overhead remains below 1 ms per message across all settings, ranging from 0.813 to 0.884 ms. These results highlight the low computational overhead of SCALE.

BandwidthChange evaluationUtility–cost profilingState update + schedulingTotal
15 Mbps0.3510.1300.3330.813
20 Mbps0.3550.1300.3740.859
25 Mbps0.3530.1300.3900.873
30 Mbps0.3540.1300.4010.884
Varying0.3520.1290.3920.873

Table 6. Mean runtime per message of SCALE on CooperScene (ms).

AI Usage Statement

We used AI assistants to support the implementation and debugging of the experimental code described in Section 5. We also used AI assistants for grammar checking and revising the manuscript. The authors reviewed the AI-assisted code and text and tested the code. The authors take full responsibility for the experimental results and the final manuscript.

Full-send reference rates are estimated from validation or calibration data.

References

  1. [1]Qing Cai, Yiqing Zhou, Ling Liu, Hanxiao Yu, Yihao Wu, Ningzhe Shi, and Jinglin Shi. 2025. Query-aware semantic encoder-based resource allocation in task-oriented communications. IEEE Transactions on Mobile Computing, 24(7):6413–6429.DOI
  2. [2]Xiaoyu Chen, Fengge Wu, Junsuo Zhao, and Yun Fan. 2026. Condenseflow: Scalable latent space collaboration via semantic compression for multi-agent systems. In Findings of the Association for Computational Linguistics: ACL 2026, pages 13694–13712.DOI
  3. [3]Zhuoyun Du, Runze Wang, Huiyu Bai, Zouying Cao, Xiaoyang Zhu, Yu Cheng, Bo Zheng, Wei Chen, and Haochao Ying. 2026. Enabling agents to communicate entirely in latent space. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 27106–27129, San Diego, California, United States. Association for Computational Linguistics.
  4. [4]Tianyu Fu, Zihan Min, Hanling Zhang, Jichao Yan, Guohao Dai, Wanli Ouyang, and Yu Wang. 2026. Cache-to-cache: Direct semantic communication between large language models. In International Conference on Learning Representations, volume 2026, pages 43130–43158.arxiv.org/abs/2510.03215
  5. [5]Deniz Gündüz, Zhijin Qin, Inaki Estella Aguerri, Harpreet S Dhillon, Zhaohui Yang, Aylin Yener, Kai Kit Wong, and Chan-Byoung Chae. 2022. Beyond transmitting bits: Context, semantics, and task-oriented communications. IEEE Journal on Selected Areas in Communications, 41(1):5–41.arxiv.org/abs/2207.09353
  6. [6]Yue Hu, Shaoheng Fang, Zixing Lei, Yiqi Zhong, and Siheng Chen. 2022. Where2com: Communication-efficient collaborative perception via spatial confidence maps. In Advances in Neural Information Processing Systems, volume 35, pages 4874–4886. Curran Associates, Inc.arxiv.org/abs/2209.12836
  7. [7]Yue Hu, Xianghe Pang, Xiaoqi Qin, Yonina C Eldar, Siheng Chen, Ping Zhang, and Wenjun Zhang. 2026. Pragmatic communication in multi-agent collaborative perception. IEEE Transactions on Pattern Analysis and Machine Intelligence.DOI
  8. [8]Yue Hu, Juntong Peng, Sifei Liu, Junhao Ge, Si Liu, and Siheng Chen. 2024. Communication-efficient collaborative perception via information filling with codebook. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15481–15490. IEEE.arxiv.org/abs/2405.04966
  9. [9]Zixing Lei, Shunli Ren, Yue Hu, Wenjun Zhang, and Siheng Chen. 2022. Latency-aware collaborative perception. In European Conference on Computer Vision, pages 316–332. Springer.arxiv.org/abs/2207.08560
  10. [10]Yiming Li, Shunli Ren, Pengxiang Wu, Siheng Chen, Chen Feng, and Wenjun Zhang. 2021. Learning distilled collaboration graph for multi-agent perception. Advances in Neural Information Processing Systems, 34:29541–29552.arxiv.org/abs/2111.00643
  11. [11]Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. 2023a. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776–44791.arxiv.org/abs/2306.03310
  12. [12]Chuanhong Liu, Caili Guo, Yang Yang, and Nan Jiang. 2023b. Adaptable semantic compression and resource allocation for task-oriented communications. IEEE Transactions on Cognitive Communications and Networking, 10(3):769–782.arxiv.org/abs/2204.08910
  13. [13]Genjia Liu, Anning Hu, Yue Hu, Wenjun Zhang, and Siheng Chen. 2026. Rate-distortion optimized pragmatic communication for collaborative perception. In International Conference on Learning Representations, volume 2026, pages 119998–120019.arxiv.org/abs/2509.21994
  14. [14]Yen-Cheng Liu, Junjiao Tian, Nathaniel Glaser, and Zsolt Kira. 2020. When2com: Multi-agent perception via communication graph grouping. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4105–4114. IEEE.arxiv.org/abs/2006.00176
  15. [15]Shilpa Mukhopadhyay, Amit Roy-Chowdhury, and Hang Qiu. 2026. COOPERTRIM: Adaptive data selection for uncertainty-aware cooperative perception. In The Fourteenth International Conference on Learning Representations.arxiv.org/abs/2602.13287
  16. [16]Shivansh Patel, Saim Wani, Unnat Jain, Alexander G. Schwing, Svetlana Lazebnik, Manolis Savva, and Angel X. Chang. 2021. Interpretation of emergent communication in heterogeneous collaborative embodied agents. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15953–15963.arxiv.org/abs/2110.05769
  17. [17]Yalin E. Sagduyu and Tugba Erpek. 2026. When semantic communication meets queueing: Cross-layer latency and task fidelity optimization. Preprint, arXiv:2605.05514.arxiv.org/abs/2605.05514
  18. [18]Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. 2022. Motion transformer with global intention localization and local movement refinement. Advances in Neural Information Processing Systems, 35:6531–6543.arxiv.org/abs/2209.13508
  19. [19]Xiangyu Shi, Marco Chiesa, Gerald Q. Maguire, Jr., and Dejan Kostić. 2026. KVComm: Enabling efficient LLM communication through selective KV sharing. In The Fourteenth International Conference on Learning Representations.arxiv.org/abs/2510.03346

Paper details

vlnc:W_4u7qw2hs
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 19

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

AI assistants

View statement
Looking for related papers…
Contents