SpecCompact: Accelerating LLM Agent Context Compaction with Slot-Parallel Semantic Speculative Verification

SpecCompact

Introduction

Large language model (LLM) agents now handle a growing range of tasks, especially personal work and coding ([17, 20, 1, 19]). While longer context windows have enabled better long-horizon performance, they also allow agents to accumulate larger histories of user requests, tool outputs, and intermediate decisions. Agent systems typically use context compaction to shorten these histories while retaining information needed for later actions ([21, 18]). However, compaction itself requires processing a long context and decoding a summary token by token. The resulting delay can significantly degrade the user experience. Compaction therefore requires balancing information retention against the latency of producing the summary.

Token-level speculative decoding, a general method for accelerating decoding, can speed up this compaction process. Specifically, a smaller model with lower inference costs can accelerate generation while preserving the target model’s output distribution by target model verification ([12, 3]). However, valid summaries can differ in wording yet have the same effect on later actions. Thus, although speculative decoding preserves lossless compaction behavior, it may not fully exploit this flexibility. Recent semantic speculation methods use target-model judgments to accept sentence-level drafts ([23, 8]), making them better suited to compaction. However, applying these methods to a single summary still requires sequential generation, which can be slow for long contexts. Moreover, asynchronous compaction overlaps summarization with continued agent execution, while parallel compaction summarizes input blocks concurrently ([4, 5]). These approaches raise a further question: *can the structure of the summary itself guide parallel generation and target-model verification?

Existing compaction templates provide a useful starting point. Agent frameworks such as OpenClaw and Hermes (OpenClaw, [21]; Nous Research, [18]) organize summaries into fields such as goals, constraints, progress, decisions, and next steps. We call these fields slots. Each slot describes one aspect of the task state and can be generated directly from the shared history. In our preliminary study, slots generated without using other slots’ outputs still preserve the necessary information, showing the potential for independent slot generation. Furthermore, reversing the slot generation order has little effect on the retained information. These findings suggest that slots can serve as units for independent and parallel generation, with verification and repair performed within each slot.

We introduce SpecCompact, a framework for accelerating agent context compaction through slot-parallel semantic speculation. A fine-tuned small model drafts each slot from the shared history and the content already accepted in that slot. The target model checks whether each proposed line is supported by the history and consistent with the accepted content. It accepts the line or generates a local replacement. These draft–verify–repair loops run in parallel, and their outputs are combined into a structured summary. We also introduce system-level optimizations that reduce execution overhead and further accelerate compaction.

Our contributions are as follows:

• We show that summary slots can retain relevant information without conditioning on other slots’ outputs, supporting their use as units of parallel compaction.

• We introduce SpecCompact, which combines training, parallel drafting, and line-wise semantic verification with local repair. We further optimize its execution to reduce repeated context processing and model calls.

• We demonstrate 2.04–5.63× compaction speedups over Hermes-style summarization, alongside downstream task evaluations and a native Hermes user study.

Background and Motivation

Background

Speculative decoding uses a small draft model to propose several tokens, which a larger target model verifies in a single forward pass (Leviathan et al., [12]; Chen et al., [3]). The accepted prefix is kept, and the first rejected token is replaced before drafting resumes. This allows each target-model call to produce multiple tokens while remaining lossless relative to target-only decoding.

Semantic speculation relaxes the lossless requirement by evaluating draft content based on its meaning. Related methods such as SpecReasons (Pan et al., [23]) and DREAM-R (Hu et al., [8]) apply this idea to reasoning steps for a lossy but much faster speculation inference. Like speculative decoding, these methods use a small model to draft a token sequence. Instead of verifying individual tokens, the target model evaluates the sequence as a complete reasoning step and outputs an accept-or-reject decision. In context compaction, summaries with different wording can preserve the same information needed for later actions. We therefore adopt semantic speculation for compaction, judging drafts by whether they preserve the information needed for later actions.

Motivation

Agent frameworks already use structured templates for context compaction. As shown in Figure 2(a), OpenClaw and Hermes organize summaries into fields [21, 18]. In our context, we call these fields slots. Each slot describes one aspect of the task state, such as the goal or current progress. Slots provide a natural unit for generation, but generating all slots as one sequence still requires later slots to wait for earlier ones. We therefore investigate whether this sequential dependency is necessary.

Vector (PDF)

We compare monolithic generation with independent slot generation on 100 trajectories. Monolithic generation produces all six slots in a single output, while independent generation produces each slot from the same history without access to other slots’ outputs. We use an LLM judge to evaluate how well each slot preserves relevant information from the full history, with a score from 0 to 5 obtained by summing five binary criteria. As shown in Figure 2(b), independent generation achieves a mean score of 4.73, compared with 4.44 for monolithic generation. The fraction of slots scoring at least 4 is 93% and 85%, respectively. Reversing the slot order for monolithic generation or the invocation order for independent generation changes the mean score by at most 0.03. These results suggest that the information needed for a slot can often be recovered from the history without conditioning on other slots. This motivates generating slots in parallel from the shared history.

SpecCompact

Figure 3 provides an overview of SpecCompact. We first fine tune a small drafter on target-generated slot content. During compaction, the drafter generates slots in parallel from the shared history. The target model verifies each proposed line against the history and the accepted slot prefix, repairing rejected lines. Each slot continues from its updated prefix, and the slots are assembled into the compacted context once all are complete.

Vector (PDF)

Slot-Conditioned Drafting

Based on our observation, we use slots as units of parallel generation. We follow OpenClaw’s six-slot summary schema [21]: Goal, Constraints and Preferences, Progress, Key Decisions, Next Steps, and Critical Context. Each slot has an instruction specifying its content and output format. Each request includes the shared conversation history and the full schema, but asks the drafter to generate only the requested slot. Figure 5 provides the prompt template and slot contracts.

![Drafter Prompt

You are a context summarization assistant. Your task is to read a conversation between a user and an AI coding assistant, then produce structured summary content following the requested format.

Do NOT continue the conversation. Do NOT respond to any questions in the conversation. ONLY output the requested summary content.

The messages below are a conversation to summarize. The next request will ask for one section of a structured context checkpoint summary that another LLM will use to continue the work.

The complete OpenClaw-style summary schema contains these sections: {slot_lines}

Conversation: <conversation> {trajectory} </conversation>

Keep the requested section concise and consistent with how it would appear inside the complete OpenClaw-style summary. Preserve exact file paths, function names, and error messages.

Slot request: Generate only the {slot} slot as one section of the complete OpenClaw-style summary. Slot title: {slot_title} Slot contract: {slot_contract}

Use this exact slot-body format: {slot_format}

Return only the slot content, not JSON. Do not include the top-level slot heading. Nested headings required by the slot contract are allowed.

Slot Contracts

goal: What is the user trying to accomplish? Can be multiple items if the session covers different tasks.

constraints_preferences: Any constraints, preferences, or requirements mentioned by the user. Return (none) if none were mentioned.

progress: Use the OpenClaw Progress substructure: Done contains completed tasks/changes; In Progress contains current work; Blocked contains issues preventing progress, if any.

key_decisions: Decisions with brief rationale, using the pattern Decision: rationale.

next_steps: Ordered list of what should happen next.

critical_context: Any data, examples, or references needed to continue. Return (none) if not applicable. Preserve exact file paths, function names, and error messages.](media/fig5.png) bbox=0.12,0.19,0.88,0.77

With the slot structure defined, we next fine-tune the drafter for slot-conditioned generation. We first use the target model with the generation prompt to produce supervised training data. We choose an independent small model as the drafter, using Qwen3-0.6B (Yang et al., 2025) in our experiments. Unlike vanilla speculative decoding, semantic speculation does not require the drafter and target to share a vocabulary. We align the drafter with the target’s slot generation through supervised fine-tuning (SFT). For each training example, we minimize the cross-entropy loss:

where $q_\theta$ is the drafter, $H$ is the history, $c_s$ is the slot instruction, and $y_s$ is the target-generated slot content represented as $L_s$ tokens under the drafter’s tokenizer. Other objectives, such as KL divergence between aligned output distributions, could also be used. We use text-only supervision to avoid storing target probability distributions. We construct one training example for each history-slot pair, so a dataset of $T$ histories yields $6T$ SFT examples.

During compaction, we run a separate drafting stream for each slot. All streams use the same fine-tuned drafter and shared history, but maintain their own slot instructions and accepted prefixes. While SFT uses complete slot outputs, online drafting proceeds through line-level proposals. The drafter generates tokens autoregressively and submits each proposed line to the target model for verification.

For slot $s$, let $A_{s,t-1}$ denote the accepted prefix before step $t$, with $A_{s,0}$ initially empty. This prefix contains the lines already accepted or generated as replacements by the target model. Given the history $H$ and slot instruction $c_s$, the drafter $q_\theta$ proposes the next line:

The proposed line is added to the prefix only after verification. If accepted, it is appended unchanged. Otherwise, the target-generated replacement is appended instead. The updated prefix then conditions subsequent drafting, so the drafter continues from the content retained in the summary, including any target repairs. The history and slot instruction remain fixed throughout this process. Each slot depends only on its own prefix, allowing the six drafting streams to run in parallel.

Semantic Verification

As in DREAM-R [8], we use binary semantic verification. For each proposed line $\widetilde{\ell}_{s,t}$, the target model checks whether its content is supported by the history $H$ and consistent with the accepted slot prefix $A_{s,t-1}$. The verification prompt includes the history, slot instruction, accepted prefix, and proposed line. It asks the target to return positive when both conditions hold, and negative otherwise. Figure 6 provides the verification prompt. We reduce the cost of repeatedly processing the history through shared-prefix caching, as discussed in the § 3.3.

The target generates one judgment token, from which we obtain the top-$k$ token probabilities. Let $\hat{p}^{+}_{s,t}$ and $\hat{p}^{-}_{s,t}$ denote the probability estimates for positive and negative. We compute the normalized acceptance score as

and accept the line when $r_{s,t} \ge\eta$. Although we define only two labels, positive and negative, each may have multiple token forms in the target tokenizer. For example, positive may start with a capital letter or include a leading space. For each label, we use the highest probability among its returned token variants. If a label is missing from the returned top-$k$ tokens, we first compute the remaining probability mass as $1-\sum_{i=1}^{k}p_i$, where $p_i$ is the probability of the $i$-th returned token. We then compare this value with the smallest returned token probability and use the smaller value as the substitute probability.

When a line is rejected, the target generates a replacement conditioned on the same history, slot instruction, and accepted prefix. The rejected line is excluded from this input. Given an accepted draft line or a valid target replacement, we update the slot prefix as

where $\ell^{T}_{s,t}$ is the target-generated replacement and $\oplus$ denotes concatenation. Drafting then continues from the updated prefix without regenerating previously accepted content.

Efficient Slot-Parallel Execution

Generating slots in parallel does not by itself make line-wise speculation efficient. Each slot repeatedly invokes the drafter and target, and these calls include the same long history. Without reuse, even a one-token verification response requires another expensive prefill. Requesting each draft line separately also adds model-call overhead, while repair generation may decode additional lines that are never used. We address these costs through the following optimizations.

Independent slot scheduling. We run the six slot streams concurrently, each with its own accepted prefix and sequential proposal, verification, and repair loop. While one slot waits for target verification or repair, other slots can continue drafting or submit their own verification requests. This allows work on the drafter and target to overlap and enables server-side batching across slots, reducing serial waiting. After all streams finish, we assemble their outputs in schema order.

Vector (PDF)

Shared-prefix caching. Each verification or repair call includes the full conversation history. Repeatedly prefilling this long input can be expensive even when the target generates only a short response. We then organize target requests with a common prefix containing the history and all six slot contracts. Slot instructions and the accepted prefix follow this shared portion, with the proposed line also included for verification. This keeps the leading tokens unchanged across slots and steps.

Draft buffering. Calling the drafter separately for every line adds request overhead. We therefore let the drafter generate multiple lines in a single call. We send the first line to the target for verification and store the remaining complete lines in a buffer. If a line is accepted unchanged, the next buffered line can be checked without another draft call. Reuse requires the prefix used to generate that line to exactly match the current accepted prefix $A_{s,t-1}$. If a target repair changes the prefix, we discard the remaining buffered lines and generate a new continuation from the repaired prefix. Each buffered proposal undergoes the same line-wise checks before acceptance.

Line-aware stopping. Each repair operation after rejection replaces one rejected line. However, the target may continue generating additional lines that are not used. We configure the target server to stop generation at the next line break, subject to the token limit, avoiding computation on these unused lines. To detect line breaks, we include all tokens ending in a newline in the stop set. Some of these tokens also contain text, so we retain that text when stopping. We preserve line and paragraph breaks when constructing the next input and keep any final content generated before end-of-sequence. The tokenizer and stop-token set are reused across calls to avoid repeated setup.

Checks and termination. In practice, the drafter sometimes produces repeated or abnormally long lines. We check whether a proposed line duplicates an accepted line or exceeds a configured character limit. Such lines go directly to target repair, avoiding an additional semantic verification call before replacement. We also cap the tokens generated per request and the lines and tokens retained per slot, preventing repetitive generation from continuing indefinitely. A slot stops when its budget is reached or a proposed addition would exceed it. Target replacements have the same checks and budget validation without further semantic verification. If a replacement is empty or fails these checks, we stop the slot and retain its accepted prefix. Structured summaries may produce standalone headings. After the length and duplicate checks, we directly accept lines beginning with # and recognized empty-slot markers such as (none), avoiding semantic verification calls for these standard output forms. Blank lines are retained as formatting. After assembly, we check the total summary token budget and raise an error if it is exceeded.

Experiments

We evaluate SpecCompact with Gemma4-31B-IT (Team et al., [27]) and Qwen3.5-122B-A10B (Qwen Team, [24]) as target models, each paired with a separately fine-tuned Qwen3-0.6B drafter (Yang et al., 2025). We use App-World (Trivedi et al., 2024}) and $\tau^3$ (Shi et al., 2026}) to evaluate whether compacted histories preserve the information needed for subsequent agent actions. AppWorld tests task completion through code and application APIs, while $\tau^{3}$ tests task completion through user interaction and tool use. We evaluate on the full AppWorld Test-Normal and Test-Challenge sets, and run three trials per task on an 80-task random selected subset of $\tau^{3}$ for each target model. We report task goal completion (TGC) and scenario goal completion (SGC) on AppWorld, and task success rate on $\tau^{3}$. Beyond agent tasks, we also evaluate long-context compaction on LongBench (Bai et al., [2]) and RULER (Hsieh et al., [6]). For LongBench, we use the GovReport and QMSum subsets and report ROUGE-L. For RULER, we evaluate full set of five task types at context lengths of 8k, and 16k using the official task-specific matching metrics. To measure compaction latency and throughput, we construct smaller subsets from AppWorld, $\tau^{3}$, and LongBench instead of using the complete sets.

We construct about 50K training data from AppWorld, $\tau^{3}$, and LiveClawBench (Long et al., [14]), using each target model to generate slot-level supervision. Training and validation examples are separated by trajectory ID. We fine-tune each drafter for one epoch with LoRA (Hu et al., [7]) applied to all linear layers, using rank 16, scaling factor 32, and zero dropout. Training uses AdamW (Loshchilov and Hutter, [15]), with a learning rate of $10^{-4}$, cosine decay, a warmup ratio of 0.05, zero weight decay, and gradient clipping at 1.0. We use a per-device batch size of 2 with 16 gradient accumulation steps, BF16 precision, a maximum sequence length of 16,384 tokens, and random seed 42. The cross-entropy loss is computed only over the target slot content.

During agent evaluation, we compact the interaction history every eight agent model calls, retaining system and setup messages and the current input. New messages accumulate until the next compaction. For latency experiments, we serve the models with vLLM (Kwon et al., [11]) on NVIDIA RTX PRO 6000 Blackwell GPUs. Gemma4 runs in BF16 on one GPU, while Qwen3.5 runs in FP8 on two GPUs. SpecCompact uses one additional GPU for the drafter and runs six slots concurrently with an acceptance threshold of $\eta= 0.7$. Drafting and repair allow up to 64 new tokens per request, while verification uses one token. Each slot is limited to 12 lines and 320 tokens. The summary budget is 2,048 tokens for SpecCompact and the baselines.

Agent Task Performance

Table 1 reports downstream task performance on AppWorld and $\tau^{3}$. The effect of SpecCompact varies across target models. Compared with Hermes, SpecCompact reduces Gemma4’s TGC by 2.98 and 4.07 percentage points on AppWorld Normal and Challenge, respectively, and reduces $\tau^{3}$ success by 2.50 points. For Qwen3.5, SpecCompact increases TGC by 4.76 and 1.68 points on the two AppWorld splits and increases $\tau^{3}$ success by 0.83 points, although Challenge SGC decreases by 0.72 points. SpecCompact therefore offers faster compaction with a model-dependent effect on downstream performance.

Models	Methods	AppWorld Normal		AppWorld Challenge		$\tau^{3}$
		TGC $\uparrow$	SGC $\uparrow$	TGC $\uparrow$	SGC $\uparrow$	Success $\uparrow$
Gemma4-31B-IT	Full History	62.50[51.79,72.62]	46.43[33.93,58.93]	53.24[45.80,60.43]	41.73[33.81,49.64]	52.08[45.00,58.75]
	OpenClaw	58.93[47.62,69.64]	44.64[32.14,57.14]	46.52[39.09,53.72]	33.09[25.18,41.01]	47.50[39.17,55.42]
	Hermes	57.74[47.02,67.86]	39.29[26.79,51.79]	47.24[40.05,54.44]	32.37[25.18,40.29]	50.00[42.08,57.92]
	SpecCompact	54.76[44.05,65.48]	37.50[25.00,50.00]	43.17[35.97,50.12]	28.06[20.86,35.25]	47.50[40.00,55.00]
Qwen3.5-122B-A10B	Full History	40.48[29.76,51.19]	26.79[16.07,37.50]	20.62[15.35,26.14]	8.63[4.32,13.67]	67.92[63.33,72.08]
	OpenClaw	28.57[19.05,38.69]	14.29[5.36,23.21]	17.75[12.71,23.26]	8.63[4.32,13.67]	65.83[60.83,70.83]
	Hermes	28.57[19.05,38.69]	14.29[5.36,23.21]	14.39[9.59,19.42]	7.19[2.88,11.51]	63.75[58.33,68.75]
	SpecCompact	33.33[23.21,44.05]	21.43[10.71,32.14]	16.07[11.27,21.10]	6.47[2.88,10.79]	64.58[59.17,69.58]

Long-Context Performance

Table 2 evaluates compaction on LongBench and RULER at 8k and 16k. Target-self uses the target model to select a draft slot directly, without a drafter or semantic verification. On Qwen3.5, SpecCompact scores within 0.6 points of Target-self on both LongBench datasets and achieves higher scores at both RULER lengths. On Gemma4, Spec-

Models	Methods	LongBench		RULER
Gemma4-31B-IT	Full History	GovReport $\uparrow$ 28.36 [27.73,29.02]	QMSum $\uparrow$ 24.34 [23.28,25.41]	8k $\uparrow$ 91.72 [88.77,94.62]	16k $\uparrow$ 92.12 [89.22,95.02]
	Target-self	25.05 [24.51,25.59]	18.35 [17.61,19.10]	71.00 [67.68,74.67]	71.02 [67.72,74.70]
	SpecCompact	22.28 [21.10,23.37]	17.52 [16.51,18.52]	66.50 [63.67,69.70]	68.55 [65.32,71.87]
Qwen3.5-122B-A10B	Full History	31.16 [30.53,31.82]	22.98 [22.03,23.96]	90.92 [87.63,93.82]	90.92 [87.63,94.17]
	Target-self	28.25 [27.71,28.79]	18.54 [17.86,19.25]	72.98 [69.33,76.67]	72.15 [68.87,75.80]
	SpecCompact	27.66 [27.05,28.26]	18.05 [17.24,18.87]	74.23 [70.55,77.93]	72.60 [68.97,76.27]

Compact scores 0.83–4.50 points below Target-self across the four settings.

Compaction Efficiency

Table 3 reports compaction latency on matched inputs. Relative to Hermes, SpecCompact achieves 4.28–5.63$\times$ speedups on the agent workloads with Gemma4 and 2.04–2.41$\times$ with Qwen3.5. On LongBench inputs of approximately 16k tokens, the speedups are 2.49$\times$ and 1.36$\times$, respectively. Three factors help explain the smaller speedups with Qwen3.5. First, Qwen3.5’s baseline summarization is already faster. Second, the larger model-size gap makes it harder for the 0.6B drafter to match Qwen3.5’s behavior, with a draft acceptance rate 19.6 percentage points lower than with Gemma4. Third, our vLLM deployment provides limited prefix-cache reuse for Qwen3.5’s hybrid attention. Standard attention can reuse cached prefix KV states, whereas Qwen3.5’s linear-attention layers also need to restore recurrent states from checkpoints saved at suitable positions. Matching text prefixes alone therefore do not guarantee cache reuse. SpecCompact also outperforms OpenClaw in seven of the eight settings. The exception is Qwen3.5 on $\tau^{3}$, where SpecCompact takes 8.71 seconds compared with OpenClaw’s 7.68 seconds.

Table 3: Compaction latency and speedup on 20 matched inputs per workload. Speedup is relative to Hermes. Bold marks the best result within each target. SpecCompact uses an additional drafter GPU.

Models	Methods	AppWorld				$\tau^{3}$		LongBench 16k
		Normal		Challenge
		Time (s) $\downarrow$	Speedup $\uparrow$	Time (s) $\downarrow$	Speedup $\uparrow$	Time (s) $\downarrow$	Speedup $\uparrow$	Time (s) $\downarrow$	Speedup $\uparrow$
Gemma4-31B-IT	Hermes	38.89	1.00×	39.12	1.00×	36.21	1.00×	74.13	1.00×
	OpenClaw	16.44	2.37×	18.70	2.09×	13.19	2.75×	63.55	1.17×
	SpecCompact	8.26	4.71×	9.14	4.28×	6.43	5.63×	29.72	2.49×
Qwen3.5-122B-A10B	Hermes	18.03	1.00×	16.98	1.00×	17.81	1.00×	29.43	1.00×
	OpenClaw	7.87	2.29×	8.69	1.95×	7.68	2.32×	24.62	1.20×
	SpecCompact	7.49	2.41×	7.55	2.25×	8.71	2.04×	21.67	1.36×

Effect of System Optimizations

Table 4 compares SpecCompact before and after the system optimizations. Both implementations already generate slots concurrently, allowing us to examine the additional gains from optimizing their execution. The optimizations reduce compaction latency by 1.88–2.44$\times$ for Gemma4 and 1.19–1.51$\times$ for Qwen3.5. For example, Gemma4’s latency on AppWorld Normal falls from 20.17 to 8.26 seconds. These gains highlight the value of reducing repeated context processing, model calls, and unnecessary decoding beyond slot parallelism alone. The table measures the combined effect of the optimizations rather than the contribution of each component. The full task evaluation uses the original implementation and has not been repeated with all system optimizations.

Table 4: Combined effect of system optimizations on 20 matched inputs per workload. Both versions use slot-parallel execution. Times are in seconds.

Benchmark	Before $\downarrow$	After $\downarrow$	Speedup $\uparrow$
Gemma4-31B-IT
AppWorld Normal	20.17	8.26	2.44×
AppWorld Challenge	21.56	9.14	2.36×
$\tau^{3}$	12.07	6.43	1.88×
Qwen3.5-122B-A10B
AppWorld Normal	9.55	7.49	1.28×
AppWorld Challenge	11.43	7.55	1.51×
$\tau^{3}$	10.38	8.71	1.19×

User Study

To evaluate user experience in a real agent setting, we integrate SpecCompact into the Hermes codebase and deploy the system with Gemma4 on four NVIDIA RTX PRO 6000 GPUs. We construct three scenarios from our own workflows, covering coding, math, and multitask question answering. For each scenario, we run Hermes in advance and pause at a fixed checkpoint to save a context snapshot. Users trigger compaction from the same snapshot with each method and select which feels faster without knowing the method identities. We also record the actual compaction time in the background. After compaction, we ask the agent to produce a summary and collect a separate blind preference between the two outputs.

According to the collected compaction data, SpecCompact reduces median compaction latency from 63.74 to 7.28 seconds, with a median paired speedup of 8.68$\times$. These measurements show that the compaction speedups also carry over to the native agent deployment. Among 28 summary-preference responses, 11 (39.3%) favor ours, 8 (28.6%) favor Hermes, 8 (28.6%) rate them similarly, and 1 (3.6%) cannot judge.

Related Work

Agent Context Management

LLM agents manage growing histories through external memory and context compression. Memory systems store and retrieve past information (Packer et al., 2024; Xu et al., 2025), while prompt and KV-cache compression reduce prompt length or cache size ([9, 13] (Xu et al. 2023)). For long-running agents, context compaction summarizes interaction histories into a shorter working context. Existing methods improve compression guidelines ([10]), periodically summarize trajectories (Wu et al., 2026), or train agents to manage their context ([16, 26] (Zhou et al. 2025; Ye et al. 2025)). Other work improves how compaction is executed. Slipstream runs compaction alongside continued agent execution and uses subsequent steps to validate summaries ([4]). Parallel compaction summarizes input blocks concurrently ([5]). SpecCompact accelerates the generation and verification of structured summaries through slot-parallel semantic speculation.

Speculative Decoding and Reasoning

Speculative decoding uses a small model to draft tokens and a target model to verify them in parallel, preserving the target model’s output distribution ([12, 3]). Semantic speculation relaxes this requirement. SpecReason uses target-model judgments to verify complete reasoning steps ([23]). DREAM-R further aligns the drafter through reinforcement learning and parallelizes drafting and verification ([8]). This approach fits context compaction, where different summaries can preserve the information needed for later actions. SpecCompact adapts semantic speculation to compaction through parallel slot generation and line-wise verification.

Discussion and Limitations

Limitations. SpecCompact focuses on the trade-off between accuracy and compaction speed, with an emphasis on the framework and its system implementation. It does not offer a consistent accuracy advantage over target-model summarization. We believe better drafter–target alignment, such as the reinforcement learning approach used in DREAM-R ([8]), could improve accuracy and provide a better trade-off.

Discussion. We also find that slots differ in where they obtain information from the history. Progress and Decisions use evidence spread throughout the context, while Goal, Constraints, and Next Steps mainly rely on beginning or end. This suggests slot-conditioned sliding-window attention as a way to further accelerate drafting. We leave this direction to future work due to limited time and resources.

Conclusion

We presented SpecCompact for accelerating context compaction in LLM agents. SpecCompact combines slot-parallel drafting, line-wise semantic verification, and system optimizations to reduce compaction latency. Experiments across agent and long-context benchmarks demonstrate substantial speedups with accuracy trade-offs, while a native Hermes user study confirms the latency benefits.

Prompt Template

Drafter Prompt

Figure 5 presents the drafter prompt and the six slot contracts. Each request includes the shared conversation history and the complete summary schema, together with the requested slot’s title, contract, and output format. The drafter generates only the requested slot’s content.

Target Prompt

Figure 6 shows the target verification prompt and its message structure. The full conversation history and all six slot contracts form a shared prefix, followed by the verification request for a proposed line. The target returns positive only if the line is supported by the history and consistent with the accepted summary lines.