betaFeedback

WorksComputer Science

LatComm: Reusable Context Compression for Latent Communication between Large Language Models

Abstract

Systems that combine several large language models (LLMs) often use one context for many questions. When the context is passed between models as text, the receiving model processes all of its tokens again for every question. Avoiding this cost requires a context representation that is compact, reusable across questions, and readable by a different LLM. Existing compression and latent communication methods each provide only some of these properties. We introduce LatComm, a Latent Communication interface that provides all three by separating encoding from reading: the context is encoded once without the question, and the receiver selects what to read for each question. A frozen context encoder LLM compresses the context into a fixed-size packet and sends it to a different frozen receiver LLM, which reads the packet through cross-attention, with gates computed from its prompt. On 12 multiple-choice question-answering datasets, LatComm improves average accuracy over single-vector baselines by up to 10.2 points. On QuALITY, it reduces the receiver's input by 57x and answers the questions of an article 3x faster.

Introduction

Motivation. Communication between large language models (LLMs) enables models to contribute different roles to a shared task, such as processing documents and answering questions [8, 20, 19] (Wu et al. 2023; Hong et al. 2024; Wang et al. 2025). In context-dependent workflows [17] (Yao et al. 2023), the same context is often used many times: one document can support many questions, and one observation can inform later decisions. When the context is passed as text, the model that receives it, called the receiver, processes all of its tokens again for every question (Figure 1a). Shared contexts therefore lead to repeated computation across models and questions [13, 23 a].

Figure 1. Text passing vs. latent passing. (a) The receiver reads the context again for every question. (b) The context is encoded once and reused. (c, d) Measured cost on QuALITY.

For example, in the reading comprehension dataset QuALITY [31], each article comes with 18.1 questions on average. The receiver reads 5,915 tokens for each question, and 98% of them are the article (Figure 1d).

Latent passing. Latent passing avoids this repetition (Figure 1b). An encoder reads the context once and turns it into vectors, called latents. The latents are stored and reused, so the receiver reads only the question. To replace the text, the latents need three properties. They must be reusable: they are computed once, before any question is known. They must be compact: their size does not grow with the context length, so the cost of storing and reading them stays fixed. They must be readable by a different LLM, because systems often combine models of different sizes [18, 4] and from different families [5, 28] (Wang et al. 2025). These properties are hard to combine. Latents built without the question and with a fixed size cannot keep only what one question needs, so the receiver must select the relevant information when it reads.

Existing approaches. Existing methods provide only some of these properties. A receiver can store its key–value cache to avoid re-reading ([13, 23]), but this cache grows with the context length and can be used only by the model that produced it. Context compression methods shorten what a model reads ([30, 27, 6]). ICAE compresses a context into fixed memory slots without the question, but it adapts the LLM with LoRA, and the slots are read by the same LLM ([11]). xRAG keeps both models frozen and stores one vector per document ([1]). This vector is the output of a pretrained retrieval encoder, so the amount of context that it carries is fixed by that encoder. A larger representation would need a larger projector, and its trainable parameters would grow with the size of the representation. Latent communication methods pass representations between models in place of text ([34, 9] (Yang and Eshraghian 2025; Shi et al. 2026)). C2C transfers representations between different frozen LLMs, but both models read the same input, so the transfer depends on the question and grows with the input length ([10]). Table 1 compares representative methods on four properties, and only LatComm has all four.

MethodQuery-independentFixed-budget compressionCross-LLM communicationBridge-only training
LLMLingua-2 ([30])✓✓✗✗
AutoCompressor ([6])✓✗✗✗
ICAE ([11])✓✓✗✗
xRAG ([1])✓✓✗✓
CEPE ((Yen et al., 2024))✓✗✗✗
DroidSpeak ([22])✓✗✗✗
C2C ([10])✗✗✓✓
LatentMAS ((Zou et al., 2025))✗✗✗✗
LatComm (ours)✓✓✓✓

Table 1. Comparison of context compression and latent communication. Query-independent (reusable): the representation is produced without the question. Fixed-budget compression (compact): its size does not depend on the input length. Cross-LLM communication: latent representations pass from one LLM to a different LLM. Bridge-only training: only an interface between two frozen LLMs is trained.

Approach and results. We introduce LatComm, a latent communication interface that separates encoding from reading (Figure 2). In the encoding step, a frozen context encoder LLM reads the context without the question, and learned context compressors map its token states into a fixed number of low-dimensional slots. These slots form a compressed packet, which is computed once, stored, and sent to the receiver for every question. Because the compressors read all token states of the encoder through cross-attention, a larger packet needs only more slot queries rather than a larger projector. In the reading step, a frozen receiver LLM reads the packet through cross-attention, and gates computed from its prompt control the contribution of each slot, so that different questions can use different parts of the same packet. Both LLMs stay frozen throughout, and only the compressors, readers, and gates are trained. The interface can therefore connect the same model, models of different sizes, or models from different families. On 12 question-answering datasets, LatComm improves average accuracy over single-vector baselines by up to 10.2 points. On QuaLITY, it reduces the receiver’s input by 57×57 \times and answers all questions of an article 3×3 \times faster (Figure 1c, Figure 1d).

Figure 2. LatComm overview. (a) The encoder reads the context; the receiver reads a task prompt. LatComm connects selected layers and adds context information after each paired receiver block. Both LLMs remain frozen. (b) For one layer pair, a context compressor produces reusable slots ZiZ_i. The gates compare a summary of the receiver’s prompt states with the compressed slots. The reader combines the gated slot values and adds the result to the receiver state.

Contributions. Our contributions are as follows:

  • We formulate context communication between LLMs as two steps, encoding without the question and reading with the question, with a packet of fixed size between them.

  • We introduce LatComm, an interface between two frozen LLMs with context compressors, cross-attention readers, and per-layer, per-slot gates. It is trained in two stages, so that packets stored after the first stage remain valid.

  • We show that LatComm is more accurate than single-vector baselines, answers questions three times faster than text passing, and connects six LLM pairs of different sizes and families.

  • We release a PyTorch implementation with training and evaluation configurations.1

Related Work

Context compression. Context compression reduces the input that an LLM reads. LLMLingua-2 keeps a subset of the tokens [30], so its output is still text. Gist tokens, AutoCompressor, and ICAE train an LLM to encode a prompt or a context into a few vectors that the same LLM reads [27, 6, 11]. xRAG maps one retrieval embedding per document into a frozen LLM [1]. CEPE and Flamingo feed a separately encoded input to a frozen language model through inserted cross-attention layers [2] (Yen et al. 2024). CEPE trains a small encoder for its decoder and keeps one vector per context token, and the gates of Flamingo are learned constants. LatComm compresses the hidden states of one LLM into a fixed number of slots, which a different LLM reads through gates that change with its prompt.

Latent communication. Latent communication passes internal representations between models in place of text, such as activations, hidden states, translated vectors, or selected key–value (KV) pairs [34, 9] (Yang and Eshraghian 2025; Shi et al. 2026). DroidSpeak reuses KV caches across LLMs with the same architecture [22], and LatentMAS passes the KV caches of one agent to the next without training (Zou et al., 2025). C2C trains cache fusers between different frozen LLMs [10]. It transfers per-token states of an input that the two models both read, so the transfer grows with the input length. LatComm instead sends a packet with several slots per layer that is built without the question, and each receiver token reads it through gates computed from the prompt.

LatComm: Latent Communication

Consider a context xx, such as a document, and a prompt rr that contains a question about xx, the task instructions, and any answer choices. With text passing, the receiver LLM reads xx followed by rr and generates the answer yy, so it processes xx again for every prompt (Figure 1a). With LatComm, the receiver reads only rr (Figure 1b). A second LLM, the context encoder LLM, reads xx once and sends a compressed packet Z(x)\mathcal{Z}(x), a fixed number of learned vectors, and the receiver generates yy from rr and Z(x)\mathcal{Z}(x). The packet is the receiver’s only source of context information, and both LLMs stay frozen.

The packet passes between selected layers of the two models (Figure 2a). At each of these layer pairs, a compressor turns the encoder’s token states, the hidden states that the layer computes for each token, into a few slots. A reader with gates then adds the information that the current prompt needs to the receiver’s states (Figure 2b). Only the compressors, readers, and gates are trained, in two stages.

Encoding a Compressed Packet

We use PP connections, called layer pairs. Pair ii connects encoder layer aia_i to receiver layer bib_i. Its context compressor maps the encoder’s token states HaiE(x)∈RnE×dEH^{E}_{a_i}(x) \in\mathbb{R}^{n_E \times d_E} into SiS_i slots of dimension dd, where nEn_E is the number of context tokens and dEd_E is the encoder’s hidden dimension. With learned slot queries Ui∈RSi×dU_i \in\mathbb{R}^{S_i \times d}, compression is

Xi=HaiE(x)WiE,Ci=MHA⁡i(Ui,Xi,Xi),Bi=LN⁡iattn(Ui+Ci),Zi(x)=LN⁡iffn(Bi+FFN⁡i(Bi)).(1)\begin{aligned} X_i &= H^{E}_{a_i}(x)W^{E}_i,\\ C_i &= \operatorname{MHA}_i(U_i, X_i, X_i),\\ B_i &= \operatorname{LN}^{\mathrm{attn}}_i(U_i + C_i),\\ Z_i(x) &= \operatorname{LN}^{\mathrm{ffn}}_i\left(B_i + \operatorname{FFN}_i(B_i)\right). \tag*{(1)} \end{aligned}

The projection WiEW^{E}_i maps context states to width dd. Multi-head cross-attention (MHA) lets the slot queries collect information from these states through parallel weighted reads. Layer normalization (LN) normalizes features, and a feed-forward network (FFN) transforms the result. Each row of Zi(x)∈RSi×dZ_i(x) \in\mathbb{R}^{S_i \times d} is one slot. Each pair has its own compressor, and the packet Z(x)=(Z1(x),…,ZP(x))\mathcal{Z}(x) = (Z_1(x), \ldots, Z_P(x)) collects the slots of all pairs.

The compressors read hidden states rather than key–value (KV) caches. A hidden state is one vector per token, whereas a KV cache holds keys and values for every attention head and is tied to the token positions of its input, so methods that transfer caches align the tokens of the two models ([10]). Slots pooled from hidden states have no token positions, and the receiver can read them without reading the context.

Reading with Per-Layer, Per-Slot Gates

The encoder and the compressors see only the context, without the question or the answer choices, so the same context always produces the same packet, and the receiver can reuse it for different prompts. The packet stores d∑iSid \sum_i S_i numbers, which depends on the number of slots and their width but not on the number of context tokens. Because the slots come from learned queries, a larger packet needs only more queries, and the size of the compressor stays the same. The compressors are fixed after the first training stage, so each context is encoded once and its packet can be stored.

For each prompt, the receiver runs its usual layers, and after layer bib_i a trainable reader gives it access to the slots ZiZ_i. Let Hi∈RnT×dTH_i \in\mathbb{R}^{n_T \times d_T} be the receiver states after layer bib_i, where nTn_T is the number of receiver tokens and dTd_T is the receiver’s hidden dimension. The reader uses cross-attention, so each token assigns its own weights to the slots. These weights sum to one over the slots, so attention alone can shift weight between slots but cannot reduce the total contribution of the packet. A gate gi,s∈(0,1)g_{i,s} \in(0,1) therefore sets the strength of slot ss at pair ii. A learned pooling module, shared across pairs, summarizes the receiver states at the prompt positions into a vector pip_i, and the gate compares this summary with the slot zi,s=Zi(x)[s,:]z_{i,s} = Z_i(x)[s,:]:

gi,s=σ(⟨piWgQ,zi,sWgK⟩d+βi,s).(2)g_{i,s} = \sigma\left(\frac{\langle p_i W_g^Q, z_{i,s} W_g^K \rangle}{\sqrt{d}} + \beta_{i,s}\right). \tag*{(2)}

Here σ\sigma is the sigmoid, the projections WgQW_g^Q and WgKW_g^K are shared across pairs, and every layer–slot pair has its own bias βi,s\beta_{i,s}. At later layers, the prompt states already contain context information from earlier readers, so π\pi and the gates depend on both xx and rr.

The reader projects the normalized receiver states into queries QiQ_i and the normalized slots into keys KiK_i and values ViV_i. It then scales the values with the gates and adds the result to the receiver states. For one attention head,

Ai=softmax⁡(QiKi⊤/d),Hi′=Hi+AiGiViWiO.(3)A_i = \operatorname{softmax}\left(Q_i K_i^{\top}/\sqrt{d}\right), \qquad H_i^{\prime} = H_i + A_i G_i V_i W_i^{O}. \tag*{(3)}

where Gi=diag⁡(gi,1,…,gi,Si)G_i = \operatorname{diag}(g_{i,1},\ldots,g_{i,S_i}). The gates scale the values and leave the attention weights unchanged, and Hi′H_i^{\prime} continues through the rest of the receiver. We call this dynamic per-layer, per-slot value gating: every layer pair and slot has its own gate, and its value changes with the prompt. A gate is computed when the receiver processes the prompt, is shared by all receiver tokens, and is reused while the answer is generated. The pooling module, the multi-head form of the reader, and the initialization are specified in Appendix § 8.

Training

LatComm is trained in two stages (Figure 3). Stage 1, context reconstruction, teaches the interface to carry context information, and Stage 2, task supervision, teaches the receiver side to use it for a task. Let θ=(ϕ,ψ,η)\theta= (\phi,\psi,\eta) be the parameters of the compressors, readers, and gates. Both stages minimize the cross-entropy of the response tokens under teacher forcing. For a minibatch BB, let example jj have the prompt rjr_j, the context xjx_j, and the response yjy_j with supervised positions RjR_j, and let ℓj,t=−log⁡Pθ(yj,t∣rj,Zϕ(xj),yj,<t)\ell_{j,t} = -\log P_\theta(y_{j,t} \mid r_j, Z_\phi(x_j), y_{j,<t}) be the loss of response token tt. The minibatch loss is

Figure 3. Two-stage training of LatComm. (a) In Stage 1, the receiver gets an instruction to reproduce the context xx and can read it only from the packet Zϕ(x)\mathcal{Z}_{\phi}(x). The reconstruction loss Lrec\mathcal{L}_{\mathrm{rec}} trains the compressors, readers, and gates. (b) In Stage 2, the receiver gets the question with its answer choices, and the task loss Ltask\mathcal{L}_{\mathrm{task}} on the answer yy trains only the readers and gates. The compressors are fixed, so the packet of each training context is computed once and stored. Both LLMs stay frozen in both stages.

L(θ;B)=∑j∈B∑t∈Rjℓj,t∑j∈B∣Rj∣.(4)\mathcal{L}(\theta; B) = \frac{\sum_{j\in B}\sum_{t\in R_j}\ell_{j,t}} {\sum_{j\in B}|R_j|}. \tag*{(4)}

Context reconstruction. Stage 1 trains all of θ\theta. The receiver gets an instruction to reproduce the context, and the target is the context itself, tokenized by the receiver and followed by an end-of-sequence token. The receiver does not see the context in its prompt, so it can reproduce the context only from the packet. Predicting every context token trains the packet to keep information that different questions may need.

Task supervision. Stage 2 starts from the Stage 1 weights, keeps the compressors ϕ\phi fixed, and trains only the readers ψ\psi and the gates η\eta on the task data.

For multiple-choice questions, the target is the correct option letter. Because the compressors are fixed, the packet of each training context is computed once and stored, and the context encoder LLM is not run again in this stage. For the same reason, packets that were stored before task training remain valid after it.

Complexity

Consider a context of nEn_E tokens that is used for qq questions with prompts of nTn_T tokens, and let cEc_E and cTc_T be the cost per token of the context encoder and of the receiver. Reading the context as text for every question costs CtextC_{\mathrm{text}}, and encoding it once costs CpacketC_{\mathrm{packet}}:

Ctext=q(nE+nT)cT,Cpacket=nEcE+qnTcT.(5)C_{\mathrm{text}} = q(n_E+n_T)c_T, \qquad C_{\mathrm{packet}} = n_E c_E + qn_Tc_T. \tag*{(5)}

Encoding once therefore costs less when q>cE/cTq > c_E/c_T. Table 2 compares the growth rates with text passing and with a stored KV cache of the receiver. The receiver’s compute for each question grows quadratically with the context length for text and linearly for a KV cache, because each prompt token still attends to all nEn_E context positions. For LatComm it does not depend on nEn_E, because each prompt token attends to SS slots at each of the PP layer pairs. The readers and gates add O(nTdTd+nTSid)O(n_Td_Td+n_TS_id) work per layer pair, which Equation (5) omits. The stored size grows with nEn_E for the cache and is fixed for the packet: with 24 layer pairs, 32 slots per pair, d=256d=256, and 16-bit values, a packet takes 384 KiB.

MethodCompute per questionOne-time computeStored sizeCross-LLM
Text passingLT(nE+nT)2L_T(n_E+n_T)^2–nEn_E✓
KV cacheLTnT(nE+nT)L_Tn_T(n_E+n_T)LTnE2L_Tn_E^2LTnEdTL_Tn_Ed_T✗
LatCommLTnT2+PnTSL_Tn_T^2+Pn_TSLEnE2L_En_E^2PSdPSd✓

Table 2. Orders of growth for three ways to reuse a context of nEn_E tokens for questions with nTn_T-token prompts. LEL_E and LTL_T are the numbers of encoder and receiver layers. LatComm uses PP layer pairs with SS slots of width dd. The compute columns treat hidden sizes as constants and assume full self-attention in every layer.

Experiments

The experiments address three research questions.

RQ1: Can a context be encoded once, without the question, and reused for different questions?

RQ2: How much useful information does a packet keep under a fixed size budget?

RQ3: Can two different LLMs communicate through the packet, across model sizes and model families?

Setup

Datasets. Stage 1 uses 2,000,000 English passages from WikiText-103 (Merity et al., [24]) and FineWeb-Edu (Penedo et al., [32]). Stage 2 uses 12 multiple-choice datasets in which each question comes with a context: CLOTH (Xie et al., 2018), Cosmos QA (Huang et al., [14]), HellaSwag (Zellers et al., 2019), LogiQA (Liu et al., [21]), MCScript2.0 (Ostermann et al., [29]), MuTual (Cui et al., [7]), OpenBookQA (Mihaylov et al., [26]), QuALITY (Pang et al., [31]), ReClor (Yu et al., 2020), SciQ (Welbl et al., 2017), SocialIQA (Sap et al., 2019), and SWAG (Zellers et al., 2018). We also test three held-out datasets not used in training: RACE (Lai et al., [16]), DREAM (Sun et al., 2019), and QuAIL (Rogers et al., 2020) (sizes and splits in Appendix § 9). The prediction is the valid option letter with the highest logit at the first response position. We report the accuracy on each dataset and two unweighted means: Task Avg over the 12 task datasets and Held-out Avg over the held-out datasets.

Model pairs. Table 4 lists the six model pairs. The main receiver is Qwen3.5-0.8B (Qwen Team, 2026), with the same model or a larger Qwen3.5 model (4B or 9B) as the context encoder. Two more receivers cross model families: Gemma-3-1B-pt (Gemma Team, [12]) with Qwen3.5 encoders, and Llama-3.2-1B (Meta, [25]) with the Granite-4.1-8B base model (IBM Granite Team, 2026) as the encoder. Each reader is paired with the encoder layer at the same relative depth, and the default packet has 32 slots per pair of width d=256d = 256.

Context encoderReceiverPPSSddPacket
Same model
Qwen3.5-0.8BQwen3.5-0.8B2432256384 KiB
Larger encoder
Qwen3.5-4BQwen3.5-0.8B2432256384 KiB
Qwen3.5-9BQwen3.5-0.8B2432256384 KiB
Cross-family
Qwen3.5-0.8BGemma-3-1B-pt2432256384 KiB
Qwen3.5-9BGemma-3-1B-pt2632256416 KiB
Granite-4.1-8BLlama-3.2-1B1632256256 KiB

Table 4. Model pairs with the default packet. PP is the number of layer pairs, SS the number of slots per pair, and dd the slot width. The packet stores P×S×dP \times S \times d values in bfloat16, which takes 2PSd2PSd bytes.

Baselines. We group the compared methods by what the receiver gets. Full text: Raw-text gives the frozen receiver the original context before the prompt. It is not trained on the task data, so we report it as a reference. No context: Query-only gives the frozen receiver only the prompt, and Constant packet trains the same readers and gates as LatComm but gives every question the same packet, so it measures how much accuracy comes from task training alone. Compressed context: xRAG (Cheng et al., [1]) maps one context vector to one soft token, where the vector comes from SFR-Embedding-Mistral (Salesforce AI Research, 2024) or from the last-token state of Qwen3.5-4B, LatComm’s encoder. Embedding replaces the packet with one vector from a text embedding model, F2LLM-v2-14B (Zhang et al., 2026) or BGE-Multilingual-Gemma2 (Beijing Academy of Artificial Intelligence, [3]), which the same readers and gates read. Appendix § 10 describes each baseline. All trained methods share the task data, loss, batch size, epochs, and learning-rate schedule of Table 9.

SettingStage 1Stage 2
Data
Training examples2,000,000282,441
Epochs13
Updates5,2092,208
Batch size384384
Optimization
Trained modulesallreaders, gates
OptimizerAdamWAdamW
Peak learning rate10−410^{-4}10−410^{-4}
Warmup updates15745
Schedule after warmupconstanthalf-cosine to 0
Weight decay00
Gradient-norm clipping1.01.0
Random seed4242
Interface
Slots per layer pair SS32
Slot width dd256
Attention heads MM8
Pooling width dpd_p256
Initial gate bias βi,s\beta_{i,s}2
Precision
Frozen LLMsbfloat16
Interface modulesbfloat16

Table 9. Training hyperparameters of the default configuration. Stage 1 trains the compressors, readers, and gates, and Stage 2 trains only the readers and gates.

Main Results

Table 3 compares all methods with the Qwen3.5-0.8B receiver on the 12 task datasets and on the three held-out datasets. LatComm is the best compressed-context method on both averages. On the task datasets, it reaches 65.43 Task Avg, against 61.98 for the best baseline, and it has the highest accuracy among the compressed-context methods on 11 of the 12 datasets. On the held-out datasets, which are not used in training, it reaches 49.76 Held-out Avg, against 46.70 for the best baseline, and it has the highest accuracy on all three datasets. LatComm is also above the Constant packet by 5.2 points on Task Avg and by 6.4 points on Held-out Avg. The Constant packet uses the same trained readers and gates but carries no context information, so this gain shows that the packet gives the receiver useful information about the context.

MethodEncoderCLOTHCosmosHSwagLogiQAMCScr.MuTualOBQAQuAL.ReClorSciQSIQASWAGAvgRACEDREAMQuAILAvg
Full text
Raw-text45.340.837.031.082.348.068.044.938.697.548.235.651.4366.170.958.265.06
No context
Query-only25.044.828.926.062.632.353.632.227.877.137.727.339.6039.136.234.036.41
Constant packet44.263.268.937.575.671.968.839.541.888.157.665.860.2545.541.243.543.37
Compressed context
xRAGSFR-Mistral41.255.955.133.676.052.071.240.036.489.457.961.655.8747.448.843.946.70
xRAGQwen3.5-4B38.553.156.136.473.758.669.637.735.488.154.861.255.2744.844.844.644.71
EmbeddingF2LLM-v245.158.668.335.278.772.374.441.139.487.161.968.660.9044.244.936.141.72
EmbeddingBGE-Gemma246.258.470.536.178.671.773.243.740.489.864.970.261.9846.146.641.744.83
LatCommQwen3.5-4B52.161.973.538.680.974.079.243.149.892.466.573.165.4352.252.344.849.76

Table 3. Main results with the Qwen3.5-0.8B receiver: accuracy on each dataset. Raw-text and Query-only are the untrained receiver with and without the context in its prompt, and Constant packet trains the readers and gates with one fixed packet. Avg is Task Avg or Held-out Avg. Bold marks the best compressed-context method per column.

Raw-text is not trained on the task data. Every trained method is above Raw-text on the task datasets and below it on the held-out datasets (41.72–49.76 against 65.06 Held-out Avg), and this includes the Constant packet. The modules trained on the 12 task datasets therefore transfer only in part to new datasets. This holds for every trained method, and LatComm is the most accurate.

Cross-Family Receiver

Table 5 repeats the comparison with a receiver from another model family, Gemma-3-1B-pt, and with the Qwen3.5-9B context encoder. Without training, this receiver is close to chance level with or without the context (27.73 and 27.98 Task Avg). LatComm reaches 59.00 Task Avg, against 55.46 for the best

MethodEncoderCLOTHCosmosHSwagLogiQAMCScr.MuTualOBQAQuAL.ReClorSciQSIQASWAGAvgRACEDREAMQuAILAvg
Full text
Raw-text25.025.425.022.050.923.628.025.624.424.833.524.627.7325.934.725.228.63
No context
Query-only25.425.525.725.848.923.826.425.925.823.533.725.227.9826.434.625.228.74
Constant packet38.259.367.928.471.859.956.835.528.282.054.865.754.0439.041.643.341.30
Compressed context
EmbeddingF2LLM-v237.457.864.029.671.859.457.433.130.480.457.164.953.6238.840.537.738.98
EmbeddingBGE-Gemma237.860.168.430.072.158.560.836.533.682.058.866.955.4641.241.938.640.59
LatCommQwen3.5-9B46.358.970.132.178.864.168.431.836.485.662.573.059.0043.249.946.846.63

Table 5. Results with the Gemma-3-1B-pt receiver: accuracy on each dataset. Raw-text and Query-only are the untrained receiver with and without the context in its prompt, and both are near chance for this receiver. Avg is Task Avg or Held-out Avg. Bold marks the best compressed-context method per column.

baseline and 54.04 for the Constant packet. On the held-out datasets, LatComm reaches 46.63 Held-out Avg, against 40.59 for the best baseline and 41.30 for the Constant packet. It has the highest accuracy among the compressed-context methods on 10 of the 12 task datasets and on all three held-out datasets. The interface therefore also works when the context encoder and the receiver come from different model families.

The two Embedding baselines are below the Constant packet on Held-out Avg. This difference comes from QuAIL. Every QuAIL question has a “not enough information” option, which is correct for 306 of the 2,720 questions. The Embedding baselines choose it for 18.5% and 25.0% of the questions, against 7.4% for the Constant packet and 7.5% for LatComm. On RACE and DREAM, the Embedding baselines are within 2.3 points of the Constant packet.

Cross-LLM Communication

For RQ3, Table 6 compares LatComm with two trained references for six pairs of LLMs. In every pair, the receiver does not read the context, so its context information comes only from the packet that the context encoder LLM sends. With the Qwen3.5-0.8B receiver, Task Avg is 64.91 when the same model is the context encoder, 65.43 with the 4B encoder, and 65.80 with the 9B encoder. The three encoders are within 0.9 points of each other, so the interface works with a small encoder, and one model can serve as both encoder and receiver. Across model families, the Gemma-3-1B-pt receiver reaches 59.00–59.21 with Qwen3.5 encoders, and the Llama-3.2-1B receiver reaches 64.08 with the Granite-4.1-8B encoder. In all six pairs, LatComm is 3.5 to 5.6 points above the Constant packet and 2.7 to 3.8 points above the Embedding baseline, both trained with the same receiver. The packet thus passes context information between LLMs of different sizes and families.

EncoderReceiverConstant packetEmbeddingLatComm
Same model
Qwen3.5-0.8BQwen3.5-0.8B60.2561.9864.91
Larger encoder
Qwen3.5-4BQwen3.5-0.8B60.2561.9865.43
Qwen3.5-9BQwen3.5-0.8B60.2561.9865.80
Cross-family
Qwen3.5-0.8BGemma-3-1B-pt54.0455.4659.21
Qwen3.5-9BGemma-3-1B-pt54.0455.4659.00
Granite-4.1-8BLlama-3.2-1B60.6261.4264.08

Table 6. Task Avg (%) of six model pairs and two trained references with the same receiver. Embedding uses BGE-Multilingual-Gemma2. Each receiver’s Constant packet was trained with its 4B, 9B, or Granite pair.

The receiver’s answers depend on the packet it receives. With the Qwen3.5-4B encoder and the 0.8B receiver, the same readers and gates reach 65.4 Task Avg with the packet of the question’s own context, 45.6 with the packet of another context, and 39.6 without a packet (Table 11).

Packet given to the receiverCLOTHCosmosHSwagLogiQAMCScr.MuTualOBQAQuAL.ReClorSciQSIQASWAGAvgRACEDREAMQuAILAvg
Same trained readers and gates
Own packet52.261.873.437.980.873.979.643.249.892.366.573.265.3952.352.544.649.82
Packet of another context30.748.150.328.163.443.851.231.730.677.345.446.445.5937.838.532.536.28
No packet25.044.828.926.062.632.353.632.227.877.137.727.339.6039.136.234.036.41
Readers and gates trained with one fixed packet
Constant packet44.263.268.937.575.671.968.839.541.888.157.665.860.2545.541.243.543.37

Table 11. Packet controls for the default configuration: accuracy on each dataset. The first three rows use the same trained readers and gates and change only the packet. The packet of another context comes from a different test question. The three rows come from separate evaluation runs, so the first row differs slightly from Table 3. Columns and abbreviations follow Table 3.

Reuse Cost

For RQ1, one packet serves every question about its context. In the test set, one context has 18.1 questions on average in QuALITY, 14.2 in CLOTH, and 5.7 in MCScript2.0. Table 7 reports the measured cost on QuALITY. Encoding an article once takes 0.71 s. The receiver then answers a question in 131 ms with the packet, against 511 ms from the full text (Figure 1d). A Query-only pass takes 111 ms, so the readers and gates add only 20 ms per question. For one article with 18.1 questions, the packet path takes 3.09 s in total and the text path takes 9.26 s, so the packet path is 3.0 times faster (Figure 1c). Encoding once costs less from the second question.

QuantityRaw-textLatComm
Per question
Receiver input tokens5,915103
Receiver time (ms)511131
Receiver peak memory (GiB)0.870.06
Per context
Encoding time (s)–0.71
Total time, 18.1 questions (s)9.263.09

Table 7. Measured cost on the QuALITY test set (115 articles, 2,086 questions) with the Qwen3.5-4B encoder and the Qwen3.5-0.8B receiver. Times are measured on one H100 GPU at batch size 1. Peak memory is the GPU memory that one call adds to the loaded models.

The gain grows with the context length (Figure 7). With text passing, the time per question grows from 300 ms for the fifth of the questions with the shortest articles to 651 ms for the fifth with the longest. With a packet it stays near 131 ms, so the receiver is 2.3 to 4.9 times faster. Figure 4a compares the settings of LatComm with Raw-text and Query-only. Every setting takes between 121 and 133 ms per question. The slot count does not change the time, and with 6 layer pairs the readers and gates add 6.2 ms instead of 20.5 ms. Encoding takes 0.54 to 0.73 s with the three encoders (Figure 4b).

Figure 4. Reuse cost on QuALITY. (a) Receiver time per question for Raw-text, five LatComm settings, and Query-only. The hatched part of a bar is the time added by the readers and gates. (b) Encoding time per context for three context encoders.

Figure 7. Receiver time per question against the context length on QuALITY. Each point is one article.

For RQ2, the default packet takes 384 KiB per context. Task Avg stays between 65.0 and 65.8 when the slot count changes from 8 to 64, so a 96 KiB packet with 8 slots per pair keeps the accuracy of the default packet. A slot width of 512 raises Task Avg to 66.88. Every packet configuration is above the best single-vector baseline (61.98). Appendix § 11 reports all configurations.

Conclusion

We presented LatComm, a latent communication interface between frozen LLMs. A context encoder LLM compresses a context once into a fixed-size packet, and a receiver LLM, which can differ from the encoder in size and family, reads it for every question through cross-attention with gates computed from its prompt. LatComm improves average accuracy over single-vector baselines by up to 10.2 points, and on QuALITY it answers the questions of an article 3×3 \times faster than text passing. Future work will study packets as messages in multi-agent systems and as entries in agent memory.

Limitations

Our experiments cover multiple-choice question answering in English, where the answer is a single option letter. We do not evaluate free-form generation, and we do not test the multi-agent and agent-memory uses that motivate reusable packets. The receivers have about one billion parameters and the encoders have at most nine billion, so the behavior of the interface with larger receivers is not known. In addition, only Stage 2 of the default pair is repeated with three seeds, which gives a standard deviation of 0.23 points in Task Avg. Every other configuration uses one seed, so small differences between the configurations of the ablation study should be read with care.

LatComm is trained for one pair of models, so a new context encoder or receiver needs its own compressors, readers, and gates, and its own packets. The interface also reads and writes hidden states, and therefore it does not apply to models that are available only through a text interface. Finally, because the packet has a fixed size, a longer context is compressed more strongly. Stage 1 uses passages of at most 1,024 tokens, and the longest contexts in our evaluation are the QuALITY articles. We do not study how the packet size should grow with the context length.

References

Implementation Details

Layer pairing. Equations use row vectors and omit the batch dimension. Let nE,nTn_E, n_T be the context encoder and receiver token counts, and dE,dTd_E, d_T their hidden dimensions. We choose PP unique layer pairs (ai,bi)(a_i,b_i), with blocks selected uniformly across each model’s depth and paired in order. When P>1P > 1, the schedule includes the final blocks. Each pair has its own compressor and reader, and may have a different slot count SiS_i. In the six model pairs of Table 4, every receiver layer has a reader, with one exception: the Qwen3.5-0.8B encoder has 24 layers and the Gemma-3-1B-pt receiver has 26, so 24 receiver layers have a reader. The trainable parameters are θ=(ϕ,ψ,η)\theta= (\phi,\psi,\eta) for context compressors, readers, and gates, respectively. Both backbone parameter sets are fixed.

Context compressor. Equation 1 uses WE∈RdE×dW^E \in\mathbb{R}^{d_E \times d} and masks encoder padding positions in cross-attention. The learned-query residual is retained. The feed-forward network expands to 2d2d, applies SiLU, and projects back to dd:

FFN⁡i(B)=SiLU⁡(BWi,1+bi,1)Wi,2+bi,2.(6)\operatorname{FFN}_i(B)=\operatorname{SiLU}(BW_{i,1}+b_{i,1})W_{i,2}+b_{i,2}. \tag*{(6)}

Here Wi,1∈Rd×2dW_{i,1}\in\mathbb{R}^{d\times2d}, Wi,2∈R2d×dW_{i,2}\in\mathbb{R}^{2d\times d}, bi,1∈R1×2db_{i,1}\in\mathbb{R}^{1\times2d}, and bi,2∈R1×db_{i,2}\in\mathbb{R}^{1\times d}. Compressors and readers use the same attention-head count MM, which must divide dd. The implementation sets M=8M=8.

Pooling and gates. Let I(r)\mathcal{I}(r) be the unmasked prompt positions, and hi,th_{i,t} the receiver block output before the current reader. A shared attention pool summarizes these states:

hˉi,t=LN⁡g(hi,t),(7)\bar{h}_{i,t}=\operatorname{LN}_{g}(h_{i,t}), \tag*{(7)}
ei,t=tanh⁡(hˉi,tWp+bp)wp,(8)e_{i,t}=\tanh(\bar{h}_{i,t}W_p+b_p)w_p, \tag*{(8)}
αi,t=exp⁡(ei,t)∑v∈I(r)exp⁡(ei,v),(9)\alpha_{i,t}=\frac{\exp(e_{i,t})}{\sum_{v\in\mathcal{I}(r)}\exp(e_{i,v})}, \tag*{(9)}
pi(x,r)=∑t∈I(r)αi,thˉi,t.(10)p_i(x,r)=\sum_{t\in\mathcal{I}(r)}\alpha_{i,t}\bar{h}_{i,t}. \tag*{(10)}

Here Wp∈RdT×dpW_p\in\mathbb{R}^{d_T\times d_p}, bp∈R1×dpb_p\in\mathbb{R}^{1\times d_p}, and wp∈Rdp×1w_p\in\mathbb{R}^{d_p\times1}, where dpd_p is the hidden width of the pooling network. LayerNorm and pooling weights are trainable and shared across all pairs. Response and padding positions have zero pooling weight. The summary pools normalized contextual states rather than input embeddings. The gate projections are

ui(x,r)=pi(x,r)WgQ,(11)u_i(x,r)=p_i(x,r)W_g^Q, \tag*{(11)}
ki,s(x)=zi,s(x)WgK.(12)k_{i,s}(x)=z_{i,s}(x)W_g^K. \tag*{(12)}

Here pi(x,r)∈R1×dTp_i(x,r)\in\mathbb{R}^{1\times d_T}, zi,s(x)=Zi(x)[s,:]∈R1×dz_{i,s}(x)=Z_i(x)[s,:]\in\mathbb{R}^{1\times d}, WgQ∈RdT×dW_g^Q\in\mathbb{R}^{d_T\times d}, and WgK∈Rd×dW_g^K\in\mathbb{R}^{d\times d}. These bias-free projections are shared across all pairs and slots. These projections give the prompt–slot comparison in Equation (2). Gate keys ki,sk_{i,s} use a separate projection from reader attention keys KiK_i. Independent biases βi,s\beta_{i,s} are initialized to 2. Including pooling, the gate module has dTd+d2+∑iSi+dTdp+2dp+2dTd_Td+d^2+\sum_i S_i+d_Td_p+2d_p+2d_T parameters. The implementation sets dp=256d_p=256 and computes pooling and gate logits in bfloat16, without an additional LLM forward pass. Inference caches gates separately for each paired layer during prompt prefill; training retains the contextual gradient path, including earlier readers.

Reader. In Equation (3), WiQ∈RdT×dW_i^Q\in\mathbb{R}^{d_T\times d} and WiK,WiV∈Rd×dW_i^K,W_i^V\in\mathbb{R}^{d\times d}. All three projections are bias-free; receiver and latent LayerNorms are separate. For head mm, Qi(m)∈RnT×dhQ_i^{(m)}\in\mathbb{R}^{n_T\times d_h} and Ki(m),Vi(m)∈RSi×dhK_i^{(m)},V_i^{(m)}\in\mathbb{R}^{S_i\times d_h}. Attention has shape Ai(m)∈RnT×SiA_i^{(m)}\in\mathbb{R}^{n_T\times S_i}, and Gi∈RSi×SiG_i\in\mathbb{R}^{S_i\times S_i}. Thus the gated read in Equation (3) is well-defined and produces Ri(m)∈RnT×dhR_i^{(m)}\in\mathbb{R}^{n_T\times d_h}. Writing ai,t,s(m)=Ai(m)[t,s]a_{i,t,s}^{(m)}=A_i^{(m)}[t,s] and vi,s(m)=Vi(m)[s,:]v_{i,s}^{(m)}=V_i^{(m)}[s,:], the output for token tt is

ri,t(m)=∑s=1Siai,t,s(m)gi,svi,s(m).(13)r_{i,t}^{(m)}=\sum_{s=1}^{S_i}a_{i,t,s}^{(m)}g_{i,s}v_{i,s}^{(m)}. \tag*{(13)}

Attention coefficients sum to one across slots; their gated coefficients are not renormalized. The same GiG_i is used for all tokens and heads. It is notation for row-wise scaling, implemented by broadcasting rather than constructing a dense matrix. The output projection WiO∈Rd×dTW_i^O\in\mathbb{R}^{d\times d_T} is bias-free and initialized to zero, so the initial residual update is zero. Attention dropout is zero.

Supervision and payload. Reconstruction targets include the end-of-sequence token. Equation (4) averages over supervised response tokens, rather than giving equal weight to responses of different lengths. The context encoder LLM is evaluated without gradients. In Stage 1, gradients through the frozen receiver update the compressors, readers, and gates. In Stage 2, they update only the readers and gates. If a stored scalar occupies ww bytes, the packet’s payload is ∑iSiw\sum_i S_iw bytes, excluding metadata. Gate and reader parameters are stored at the receiver rather than transmitted in the packet.

Training protocol implementation. Stage 2 loads the complete Stage 1 interface, including the prompt-pooling and gate parameters, from the final Stage 1 checkpoint with strict key matching, and starts a new optimizer and learning-rate schedule. The model pair, layer pairing, slot count, and latent width are identical in both stages, so every interface tensor transfers without reshaping or reinitialization. The compressor parameters are then frozen. Packets for the training contexts are computed once with the frozen encoder and compressors; identical contexts share one stored packet. The packets are stored in bfloat16. Stage 2 training reads the packets and does not load the context encoder LLM.

Training settings. Table 9 lists the training hyperparameters. Stage 1 runs one epoch over the reconstruction passages. Stage 2 then fixes the compressors, stores one packet for each training context, and runs three epochs over the task training set. The readers and gates that it trains have 16.4M parameters for the Qwen3.5-0.8B receiver. Both stages use AdamW with the default momentum coefficients. In Stage 1, the learning rate rises linearly for 157 updates and then stays constant. In Stage 2, it rises linearly for 45 updates and then follows a half-cosine decay to zero. The trained baselines use the same task data, loss, batch size, number of epochs, and learning-rate schedule. Ev-

Every run uses random seed 42, and Stage 2 of the default pair is repeated with seeds 43 and 44.

Environment. LatComm runs use Python 3.12, PyTorch 2.11 or 2.13, and Hugging Face Transformers 5.15.1. The frozen models and the interface modules run in 16-bit bfloat16, and packets are stored in the same format. All experiments run on six machines, each with eight NVIDIA H100 GPUs (80 GB each). The trained interfaces are evaluated with Transformers. The Raw-text reference on the 12 task datasets is computed with vLLM.

Data Details

Reconstruction data. The Stage 1 training set contains 2,000,000 passages: 446,282 from WikiText-103 and 1,553,718 from FineWeb-Edu. They are a fixed selection of complete passages from the prepared corpus. Each passage contains 64–1,024 tokens under the Qwen3.5-0.8B tokenizer. The receiver is asked to reproduce the passage with one of 100 reconstruction instructions. Validation and test contain 15,043 and 14,983 passages. WikiText-103 keeps its official splits, and FineWeb-Edu is split by canonical URL before passage extraction. English filtering, passage deduplication, and document-level separation are applied.

Task data. Table 8 lists the 12 task datasets. After removing exact normalized context overlaps with the test set and invalid duplicate answer texts, the official training pool contains 284,477 questions. A grouped holdout from this pool gives 2,036 development questions, and the remaining 282,441 questions are used for training. The test set contains 55,890 labeled questions from public benchmark splits. These questions are not used for training or model selection. The original option text, option count, and option order are kept. SciQ uses a fixed deterministic option order because its source does not specify one. MuTual uses its labeled de-

DatasetTrain examplesTrain context tokensAvg. contextEval
Stage 1: context reconstruction
WikiText-103446,282100,278,346224.71,057
FineWeb-Edu1,553,7181,033,473,862665.213,926
Stage 1 total2,000,0001,133,752,208566.914,983
Stage 2: task supervision
CLOTH76,66034,320,807447.711,516
Cosmos QA25,0671,861,20074.22,985
HellaSwag31,3081,472,10147.010,042
LogiQA7,202578,04180.3651
MCScript2.014,0172,372,120169.23,610
MuTual6,901685,10699.3886
OpenBookQA3,20533,16010.3500
QuALITY2,39314,215,2515,940.32,086
ReClor4,422351,54679.5500
SciQ10,210992,36997.2884
SocialIQA33,232554,72516.72,224
SWAG67,8241,118,26516.520,006
Stage 2 total282,44158,554,691207.355,890
Held-out: test only
RACE––343.64,934
DREAM––156.42,041
QuAIL––419.62,720

Table 8. Data for both training stages and the held-out datasets. Train excludes the development holdout. Tokens count untruncated context text once per training example with the pinned Qwen3.5-0.8B tokenizer, excluding instructions, choices, targets, and special tokens. Reconstruction supervision contains the same text plus one EOS per passage (1,135,752,208 target tokens). Avg. context is the mean training context length in tokens, rounded to one decimal; total-row means are weighted by example count. For the held-out datasets, it is the mean context length of the test questions. Eval is the reconstruction test set, the fixed public multiple-choice reporting suite, or the held-out test set.

velopment split for testing because its official test labels are not public. We also compute Micro, the accuracy over all 55,890 test questions.

Held-out datasets. RACE, DREAM, and QuAIL are used only for testing. RACE and DREAM use their official test sets. QuAIL uses its validation and challenge sets. No held-out passage is an exact match of a Stage 2 training context. Some RACE passages share text with CLOTH training passages, and some DREAM dialogues share text with MuTual training dialogues, because these datasets draw on similar exam material. Removing the affected questions (294 in RACE and 349 in DREAM) changes the accuracy of every method by less than one point.

Token counts. Table 8 counts context tokens with the Qwen3.5-0.8B tokenizer, before truncation and without instructions, answer choices, responses, or special tokens. A context shared by several questions is counted once for each question. The reconstruction training set contains 1,133,752,208 context tokens, and the task training set contains 58,554,691. The cross-family pairs use the same texts with their own tokenizers.

Baseline Details

Constant packet. This control uses the receiver, readers, gates, Stage 1 initialization, data, and training settings of the corresponding LatComm pair: Qwen3.5-4B for the Qwen3.5-0.8B receiver, Qwen3.5-9B for the Gemma-3-1B-pt receiver, and Granite-4.1-8B for the Llama-3.2-1B receiver. The only change is the packet. We encode 2,048 randomly chosen training contexts with the pair’s Stage 1 compressors and average their packets. Every training and test question receives this mean packet, so the packet carries no information about the question’s context.

Embedding. This baseline compresses the context into one vector with a frozen text embedding model. We use F2LLM-v2-14B (Zhang et al., 2026) and BGE-Multilingual-Gemma2 (9B) [3], each paired with the frozen Qwen3.5-0.8B and Gemma-3-1B-pt receivers. The encoder processes the context as one input, without the question and without chunk pooling. Its last-token representation is L2-normalized, which gives a 5,120-dimensional vector for F2LLM and a 3,584-dimensional vector for BGE. A trainable shared projector maps this vector to one slot. The readers and gates are the same as in LatComm, and they read this slot at every receiver layer: 24 layers for Qwen3.5-0.8B and 26 for Gemma-3-1B-pt. The stored 32-bit vectors take 20 KiB and 14 KiB. Each interface is trained from a fresh initialization on the task data, without a reconstruction stage. F2LLM uses a 32,768-token input limit, which truncates no test context. BGE truncates its input to 8,192 tokens. This affects 93 of the 55,890 test questions, which share five documents. For Table 6, BGE is paired with the Llama-3.2-1B receiver (16 layers). In this run, the BGE vectors are computed in bfloat16 and stored, instead of in 32-bit precision during training. On 15 documents of different lengths, the two computations give a cosine similarity above 0.999.

xRAG. We adapt xRAG [1]. A frozen SFR-Embedding-Mistral encoder (Salesforce AI Research, 2024) reads the complete context, and a two-layer MLP maps its 4,096-dimensional last-token representation to one soft token placed before the receiver’s prompt. Only the MLP is trained, first on reconstruction and then on the task data. Our adaptation differs from the original in three ways: there is no retrieval or document selection, the receiver is Qwen3.5-0.8B, and task training uses the gold option letter without a distillation term. The second xRAG variant replaces the embedding model with the last-token state of Qwen3.5-4B, which has 2,560 dimensions.

Ablation Study

Packet settings. Table 10 changes the context encoder or one packet setting at a time. The packet settings use the Qwen3.5-4B encoder and change the number of slots (8, 32, 64), the slot width (128, 256, 512), and the number of layer pairs (24, 12, 6). Reconstruction improves with every increase in capacity: a larger encoder, more slots, wider slots, and more layer pairs all lower the Stage 1 loss and raise the token accuracy. The widest slots give the best reconstruction, with 79.0% token accuracy, and the highest Task Avg, 66.88. Task Avg changes less than reconstruction. It stays between 63.84 and 66.88 over all configurations, and every configuration is above the best single-vector baseline (61.98). With 8 slots per pair, the packet takes 96 KiB, one quarter of the default size, and Task Avg is 65.81, against 65.43 for the default packet. With 12 and 6 layer pairs, Task Avg is 64.76 and 63.84. At 96 KiB, a packet with all layer pairs and 8 slots is therefore more accurate than a packet with 6 layer pairs and 32 slots.

SettingPacketSizeReconstructionTask Avg
LossAcc.
Encoder
Qwen3.5-0.8B24×32×25624 \times32 \times256384 KiB1.8356.364.91
Qwen3.5-4B (default)24×32×25624 \times32 \times256384 KiB1.2468.565.43
Qwen3.5-9B24×32×25624 \times32 \times256384 KiB0.9575.165.80
Slots per pair
824×8×25624 \times8 \times25696 KiB1.5861.365.81
32 (default)24×32×25624 \times32 \times256384 KiB1.2468.565.43
6424×64×25624 \times64 \times256768 KiB1.1171.565.04
Slot width
12824×32×12824 \times32 \times128192 KiB1.8257.165.01
256 (default)24×32×25624 \times32 \times256384 KiB1.2468.565.43
51224×32×51224 \times32 \times512768 KiB0.7879.066.88
Layer pairs
66×32×2566 \times32 \times25696 KiB1.7757.963.84
1212×32×25612 \times32 \times256192 KiB1.4763.564.76
24 (default)24×32×25624 \times32 \times256384 KiB1.2468.565.43

Table 10. Ablation of the context encoder and the packet settings with the Qwen3.5-0.8B receiver. Each group changes one setting of the default configuration. Packet gives P×S×dP \times S \times d: layer pairs, slots per pair, and slot width. Size is the stored size. Reconstruction gives the loss and the token accuracy (%) on 256 held-out passages at the end of Stage 1. Task Avg (%) is measured after Stage 2.

Training loss. Figure 5 shows the training loss of both stages for the same configurations. In Stage 1, the loss decreases throughout training, and the configurations keep the order of Table 10: more capacity gives a lower reconstruction loss. In Stage 2, the curves are close to each other for the three encoders and for the three slot counts, and the loss is lower with wider slots and with more layer pairs.

Figure 5. Training loss of (a) Stage 1 and (b) Stage 2 against the number of context tokens processed, for the configurations of Table 10. Each column changes one setting, and the solid orange line is the default configuration. The token count is the number of processed examples times the mean context length of the training set. The curves are running means over 50 updates in Stage 1 and 100 updates in Stage 2. Dotted lines mark the ends of the first and second epochs of Stage 2.

Packet controls. Table 11 tests whether the receiver uses the content of the packet. The same trained readers and gates reach 65.39 Task Avg with the packet of the question’s own context, 45.59 with the packet of another context, and 39.60 without a packet. The own packet is the most accurate of the three on all 12 task datasets, and the packet of another context lowers the accuracy by 9.8 to 30.1 points. On the held-out datasets, the own packet reaches 49.82 Held-out Avg, against 36.28 with the packet of another context and 36.41 without a packet. A packet from another context therefore gives no gain on new datasets, and the gain comes from the content of the own packet. The Constant packet control, whose readers and gates are trained with one fixed packet, reaches 60.25 Task Avg and 43.37 Held-out Avg.

Additional Analysis

Seed variation. We repeat Stage 2 of the default pair with three seeds. The three runs share the compressors and the packets. Task Avg is 65.43, 65.72, and 65.88, with a mean of 65.68 and a standard deviation of 0.23. The main tables report the first run, which is the lowest of the three, and the gain over the best baseline (61.98) is much larger than the variation between seeds.

Gain source. We split the test questions of the 12 task datasets by whether the Qwen3.5-4B encoder answers them correctly when it reads the full text, which holds for 76% of the questions (Figure 6). On these questions, LatComm reaches 72.4% accuracy, against 68.7% for the Embedding baseline and 64.5% for the Constant packet. On the other questions, the three methods are close, with 50.5%, 48.0%, and 50.1%. The gain of LatComm is therefore on the questions that the encoder itself can answer from the context.

Figure 6. Accuracy on the test questions that the Qwen3.5-4B encoder answers correctly or wrongly from the full text. Embedding uses BGE-Gemma2.

Context length. Figure 7 plots the receiver time per question for each of the 115 QuALITY articles. With text passing, the time grows linearly with the context length, from 265 ms for the shortest article to 707 ms for the longest. With a packet, it stays between 130 and 135 ms for every article, because the receiver always reads the same number of slots.

Gate values. We record the gates of the default pair on a sample of 5,386 test questions (Figure 8). The mean gate is 0.40, and it differs across layer pairs, from 0.05 at the last pair to 0.59 at pair 18. The slots of one layer pair also differ: 23.4% of the layer–slot cells have a mean gate below 0.05, and 4.3% have a mean gate above 0.95. The receiver therefore weights the slots differently at each layer pair, which one gate per layer could not express.

Figure 8. Gate values of the default pair on 5,386 test questions. (a) Mean over the 32 slots of each layer pair. The band covers the 10th to 90th percentile of the questions. (b) Mean gate of each slot. The slots of a layer pair are sorted from the highest mean gate to the lowest.

Packet similarity. Figure 9 compares the packets of the contexts in the same sample. The slots of one layer pair are nearly orthogonal at the first layer pairs, with a mean cosine similarity of 0.04, and more similar at the last ones, with 0.65. The packets of two contexts from different datasets have a mean cosine similarity of 0.72, so a part of the packet is shared by all contexts, which agrees with the accuracy of the Constant packet control. The other part separates the contexts. Packets from the same dataset are more similar than packets from different datasets at every layer pair. A nearest-mean classifier on the mean slot of a layer pair, fitted on half of the contexts and tested on the rest, identifies the source dataset among 12 with 30% to 92% balanced accuracy, against 8.3% by chance.

Figure 9. Packets of the default pair at each layer pair. (a) Mean cosine similarity between two slots of one packet. (b) Mean cosine similarity between the packets of two contexts from the same dataset or from different datasets. (c) Balanced accuracy of identifying the source dataset of a context from its packet.

References

  1. [1]Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, and Dongyan Zhao. 2024. xrag: Extreme context compression for retrieval-augmented generation with one token. In Advances in Neural Information Processing Systems, NeurIPS, Vancouver, BC, Canada.arxiv.org/abs/2405.13792
  2. [2]Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hassan, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitiao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, and 8 others. 2022. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems, NeurIPS, New Orleans, LA.arxiv.org/abs/2204.14198
  3. [3]Beijing Academy of Artificial Intelligence. 2024. BGE-Multilingual-Gemma2: Model card.
  4. [4]Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Murali­dh­aran, Yingyan Celine Lin, and Pavlo Molchanov. 2025. Small language models are the future of agentic AI. CoRR, abs/2506.02153.arxiv.org/abs/2506.02153
  5. [5]Lingjiao Chen, Matei Zaharia, and James Zou. 2024. FrugalGPT: How to use large language models while reducing cost and improving performance. Trans. Mach. Learn. Res., 2024.arxiv.org/abs/2305.05176
  6. [6]Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. 2023. Adapting language models to compress contexts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 3829–3846, Singapore. Association for Computational Linguistics.arxiv.org/abs/2305.14788
  7. [7]Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. 2020. Mutual: A dataset for multi-turn dialogue reasoning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1406–1416.arxiv.org/abs/2004.04494
  8. [8]Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2024. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, ICML, volume 235 of Proceedings of Machine Learning Research, pages 11733–11763, Vienna, Austria.arxiv.org/abs/2305.14325
  9. [9]Zhuoyun Du, Runze Wang, Huiyu Bai, Zouying Cao, Xiaoyong Zhu, Yu Cheng, Bo Zheng, Wei Chen, and Haochao Ying. 2026. Enabling agents to communicate entirely in latent space. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, ACL, pages 27106–27129, San Diego, California.
  10. [10]Tianyu Fu, Zihan Min, Hanling Zhang, Jichao Yan, Guohao Dai, Wanli Ouyang, and Yu Wang. 2026. Cache-to-cache: Direct semantic communication between large language models. In The Fourteenth International Conference on Learning Representations, ICLR, Rio de Janeiro, Brazil.arxiv.org/abs/2510.03215
  11. [11]Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. 2024. In-context autoencoder for context compression in a large language model. In The Twelfth International Conference on Learning Representations, ICLR, Vienna, Austria.arxiv.org/abs/2307.06945
  12. [12]Gemma Team. 2025. Gemma-3-1B-pt: Model card.
  13. [13]In Gim, Guojun Chen, Seung-Seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. 2024. Prompt cache: Modular attention reuse for low-latency inference. In Proceedings of the Seventh Annual Conference on Machine Learning and Systems, MLSys, Santa Clara, CA.arxiv.org/abs/2311.04934
  14. [14]Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP, pages 2391–2401, Hong Kong, China.arxiv.org/abs/1909.00277
  15. [16]Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. 2017. RACE: large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 785–794, Copenhagen, Denmark.arxiv.org/abs/1704.04683
  16. [17]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Nandan Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, NeurIPS, Virtual.arxiv.org/abs/2005.11401
  17. [18]Senyao Li, Haozhao Wang, Wenchao Xu, Rui Zhang, Song Guo, Jingling Yuan, Xian Zhong, Tianwei Zhang, and Ruixuan Li. 2025. Collaborative inference and learning between edge llms and cloud llms: A survey of algorithms, execution, and open challenges. CoRR, abs/2507.16731.
  18. [19]Huawei Lin, Peng Li, Jie Song, Fuxin Jiang, and Tieying Zhang. 2026. MUSE-Autoskill: Self-evolving agents via skill creation, memory, management, and evaluation. CoRR, abs/2605.27366.arxiv.org/abs/2605.27366
  19. [20]Huawei Lin, Yunzhi Shi, Tong Geng, Weijie Zhao, Wei Wang, and Ravinder Pal Singh. 2025. Agent-omni: Test-time multimodal reasoning via model coordination for understanding anything. CoRR, abs/2511.02834.arxiv.org/abs/2511.02834
  20. [21]Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI, pages 3622–3628.arxiv.org/abs/2007.08124
  21. [22]Yuhan Liu, Yuyang Huang, Jiayi Yao, Shaoting Feng, Zhuohan Gu, Kuntai Du, Hanchen Li, Yihua Cheng, Juncheng Jiang, Shan Lu, et al. 2024a. Droidspeak: Kv cache sharing for cross-llm communication and multi-llm serving. arXiv preprint arXiv:2411.02820.arxiv.org/abs/2411.02820
  22. [23]Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. 2024b. Cachegen: KV cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference, ACM SIGCOMM, pages 38–56, Sydney, NSW, Australia.arxiv.org/abs/2310.07240
  23. [24]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR, Toulon, France.arxiv.org/abs/1609.07843
  24. [25]Meta. 2024. Llama-3.2-1B: Model card.
  25. [26]Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 2381–2391, Brussels, Belgium.
  26. [27]Jesse Mu, Xiang Li, and Noah D. Goodman. 2023. Learning to compress prompts with gist tokens. In Advances in Neural Information Processing Systems, NeurIPS, New Orleans, LA.arxiv.org/abs/2304.08467
  27. [28]Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. 2025. Routellm: Learning to route llms from preference data. In The Thirteenth International Conference on Learning Representations, ICLR, Singapore.arxiv.org/abs/2406.18665
  28. [29]Simon Ostermann, Michael Roth, and Manfred Pinkal. 2019. Mscript2.0: A machine comprehension corpus focused on script events and participants. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics, pages 103–117, Minneapolis, MN.arxiv.org/abs/1905.09531
  29. [30]Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. 2024. Lm lingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In Findings of the Association for Computational Linguistics, ACL, volume ACL 2024 of Findings of ACL, pages 963–981, Bangkok, Thailand.arxiv.org/abs/2403.12968
  30. [31]Richard Yuan zhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel R. Bowman. 2022. Quality: Question answering with long input texts, yes! In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL, pages 5336–5358, Seattle, WA.arxiv.org/abs/2112.08608
  31. [32]Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin A. Raffel, Leandro von Werra, and Thomas Wolf. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems, NeurIPS, Vancouver, BC, Canada.arxiv.org/abs/2406.17557
  32. [34]Vignav Ramesh and Kenneth Li. 2025. Communicating activations between language model agents. In Forty-second International Conference on Machine Learning, ICML, volume 267 of Proceedings of Machine Learning Research, Vancouver, BC.arxiv.org/abs/2501.14082

Paper details

vlnc:W_729nrff3
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents