betaFeedback

WorksComputer Science

Template Gap Reconstruction for Shell Output Simulation with Small Language Models

Abstract

Small language models offer a compact basis for shell simulation without executing submitted commands, but full-output decoding requires them to predict recurring utility structure alongside content that varies across executions. We introduce template gap reconstruction (TGR), which supplies reusable output literals interspersed with sentinels to be interleaved with autoregressively generated gaps. A template selector chooses which template to be reconstructed, with the empty template retaining full-output decoding. We construct a template bank from 18.7 million outputs collected across heterogeneous execution environments and train a template selector and five reconstruction models initialized from various small language model checkpoints on a target environment excluded from bank construction. Evaluation demonstrates that TGR improves output edit similarity (O-ES) over full-output decoding (FOD) when the optimal template covers more than 25% of the template, while performing similarly for lower-coverage outputs. Additionally, TGR outperforms FOD for shorter gap sequences, with O-ES values of 0.964 and 0.812, respectively, while maintaining similar performance for longer sequences. TGR reduces the number of tokens generated by 26.3–38.3% across used tokenizer families. These results support using a reusable output structure as a practical design choice for small shell simulators, improving response fidelity and shortening generation targets while keeping template selection and residual prediction central to response quality.

Introduction

Small language models generating the output of provided shell inputs offer a promising basis for command-line interaction under limited inference resources. Such an execution-free interface supports compact learned honeypots that interact with untrusted users [21]. Prior work uses in-context learning with API-accessible large language models [21] and fine-tuned locally deployed large language models to generate shell output [15, 25, 23]. Ragsdale & Boppana [20] demonstrate that this task can be extended to smaller language models (<<1B parameters) when trained on sufficient quantities of shell input-output pairs. The success of these methods depends on the similarity of the generated output to that of a real system [3]. This paper is motivated in the improvement of shell output modeling accuracy using small language models.

Inference depends on the text processed and generated, with autoregressive decoding producing each output token in sequence [17]. Full-output decoding (FOD) requires a shell output model to predict filenames, values, diagnostics, punctuation, and internal whitespace alongside recurring headings, separators, and wording. Supplying this recurring structure as context offers a way to reduce the prediction burden.

We investigate environment-invariant output templates: ordered literal spans shared by groups of recorded executions. Greedy alignment extracts spans whose empirical invariance is scoped to the construction group. Template gap reconstruction (TGR) supplies a template alongside the input, generates sentinel-delimited template gaps, and interleaves them with copied literals. Retained text thus provides context without remaining in the generation target Figure 1.

Full-output decoding (FOD) and template gap reconstruction (TGR) producing the same complete shell response

Figure 1. Full-output decoding (FOD) and template gap reconstruction (TGR) producing the same complete shell response. Blue marks copied template literals, and orange marks generated text. Sentinel markers delimit the generated gaps.

Before the output is known, an input-only selector with a shared Transformer encoder and command-specific heads chooses from a template bank containing the empty template T0T_0 for each command. A nonempty selection invokes TGR, while selecting T0T_0 invokes ordinary FOD through the same reconstruction checkpoint. The constructed template bank contains templates extracted from 18.7 million executions across heterogeneous environments.

We evaluate output similarity across five models fine-tuned for shell modeling and TGR spanning approximately 60–600 million parameters: CodeT5-small and CodeT5-base, ByT5-small and ByT5-base, and Qwen3-0.6B [24, 27, 28]. A target environment ronment supplies 1 million examples for selector and reconstruction training. Oracle assignment uses the reference output to maximize admissible coverage and defines the coverage and reference-gap length used to group all methods.

Selected TGR improves mean complete-response fidelity across all five models. Oracle TGR improves mean output edit similarity (O-ES) over FOD in every reported coverage group above 25%, with learned selection yielding consistent gains above 50%. Selected TGR achieves O-ES of 0.879 compared with 0.742 for FOD in the (50, 75]% coverage group. For responses with 0–64 reference-gap characters, equally weighting the three reported length groups gives mean O-ES of 0.964 for oracle TGR compared with 0.812 for FOD. The advantage diminishes for longer gaps, with closer scores of 0.772 and 0.790, respectively, in the longest reference-gap group. TGR also shortens mean tokenized reference targets by 26.3–38.3% across tokenizer families, including sentinel overhead.

We make the following contributions:

  1. We formalize reusable templates for complete shell responses and provide a greedy construction procedure that preserves admissibility within each group of recorded executions.

  1. We combine an input-only, command-specific selector with a reconstruction checkpoint jointly trained for sentinel-delimited gap generation and ordinary full-output fallback.

  1. We evaluate this interface for small language models at fixed reconstruction weights, distinguishing complete-response fidelity, generated-gap fidelity, and reference-target length across five backbones.

Background and Related Work

Small language models for shell simulation. Execution-derived specialization learns response behavior for a target environment ([20]). The evaluated backbones include encoder–decoder CodeT5-small/base (approximately 60M/220M parameters) ([24]), byte-level encoder–decoder ByT5-small/base (300M/580M) ([27]), and decoder-only Qwen3-0.6B (600M) ([28]). At fixed numerical precision, smaller models require less weight storage, with their inference time and working memory depending on architecture, sequence lengths, and hardware ([17]). Ordinary autoregressive decoding invokes the model for successive output tokens, making target length another relevant design choice.

Shell responses depend on filesystem contents, permissions, variable bindings, and available programs ([8] (Ope 2024)). Honeypots motivate shell-response generation without executing attacker-supplied commands on the host ([15]). Cowrie’s emulation mode combines command handling with a fake filesystem and session logging ([13]), while shellLM and advancedShellLM generate context-aware responses with language models ([21, 22]). SBASH examines system-prompt conditioning and retrieval augmentation for local Llama and Gemma models ([1]).

Fine-tuning also supports local shell-response models. Otal and Canbaz adapt Llama 3 8B using command–response supervision from Otal & Canbaz [15]. MANTIS fine-tunes a Llama-3.2-3B-Instruct response agent on recorded Linux command–output pairs and combines it with command routing, filesystem management, and interaction history Wei et al. [25]. Sladic et al. compare supervised fine-tuning and prompting for seven local open-weight large language models, finding that adaptation benefits depend on training-data coverage Sladić et al. [23]. ShIOEnv synthesizes shell inputs and records observed execution outputs at scale Ragsdale & Boppana [20]. These records support both specialization and the identification of recurring response fragments across environments.

Template induction. Log parsers similarly separate recurring message structure from variable fields. Spell incrementally compares token sequences using a longest common subsequence and updates the template associated with a log group Du & Li [6]. Drain uses a fixed-depth tree to narrow candidate groups before matching messages and updating their templates He et al. (2017). Sequence alignment also supports pattern induction for generation: Barzilay and Lee align sentences from comparable corpora to construct paraphrasing patterns represented by word lattices Barzilay & Lee [2].

The presented procedure applies alignment to complete recorded shell responses, including multiline output. Canonicalized inputs and exact observed exit codes define groups before induction. Each template retains ordered, contiguous character spans whose boundaries identify positions for text. The construction guarantee concerns their occurrence in every output of the observed group. Using that structure for a new execution requires an input-conditioned decision about which template is appropriate.

Span infilling and template-based generation. T5’s span-corruption objective replaces sampled spans with unique sentinels and trains the decoder to predict the removed text Raffel et al. [19]. Donahue et al. train language models to fill arbitrary blanks using sequences that concatenate masked context and missing text Donahue et al. [5]. These interfaces expose retained text to the model while restricting the prediction target to missing spans.

Template-based generation makes output organization explicit. Wiseman et al. learn latent discrete templates jointly with generation through a hidden semi-Markov decoder Wiseman et al. [26]. TempLM constructs an explicit template-based generator from a pretrained language model by replacing data values in generated outputs with field slots and refining the templates Zhang et al. [29]. TGR obtains its literals from repeated executions and uses their positions to similarly define sentinel-delimited targets. Its gaps contain all text outside the retained literals, including changing values and stable text that induction did not preserve. Gap scoring separately evaluates this generated text, whose length and content depend on the template.

Editing and constrained generation. Editing systems reuse text while predicting additions needed to complete an output. EditT5 combines keep/delete tagging and token reordering with autoregressive insertion of missing spans Mallisnon et al. (2022). POINTER preserves supplied tokens while progressively inserting new text between them Zhang et al. [30]. Lexically constrained decoding incorporates required words or phrases into a complete hypothesis during search Post & Vilar [18]. TGR combines execution-derived literals with input-only selection from a command-specific bank. The empty template invokes ordinary FOD in the same jointly trained checkpoint. Admissible templates can leave gaps with different lengths, segmentation, and predictability, making their utility depend on both compatibility and residual prediction.

Environment-Invariant Output Templates

Template gap reconstruction (TGR) predicts a shell response by combining copied template literals with generated residual spans. The method comprises template construction, template-conditioned gap generation, and input-conditioned template selection. Let xx denote a submitted shell input and yy its recorded output.

Template Construction

Let Σ\Sigma be the character alphabet of the recorded output, including retained whitespace and control characters. A template is an ordered sequence of non-empty text segments,

T=(τ1,…,τk),τi∈Σ+.T = (\tau_1,\ldots,\tau_k), \qquad\tau_i \in\Sigma^+.

A template overlays y∈Σ∗y \in\Sigma^*, written T⪯yT \preceq y, if there exist gaps s0,…,sk∈Σ∗s_0,\ldots,s_k \in\Sigma^* such that

y=s0τ1s1τ2⋯τksk.(1)y = s_0\tau_1s_1\tau_2\cdots\tau_ks_k. \tag*{(1)}

Each template segment occurs in order and without overlap. The gaps collect all remaining characters, including any leading or trailing text, and may be empty. Template identity includes the literal text and its segment boundaries: each boundary determines a separate position for a gap. For example, (ab,cd)(ab,cd) overlays abXcd, whereas (abcd) does not, although both templates retain four characters. The empty template T0=()T_0=() overlays every output, with s0=ys_0=y. Admissibility is checked by matching each literal at its earliest occurrence after the preceding match. An earlier match leaves at least as much suffix for later literals, so greedy failure implies that no overlay exists.

We group execution records by their canonicalized input and observed exit code. Canonicalization abstracts environment-specific resource arguments while preserving the command and its argument structure. For example, ls -la inputs with different directory arguments share the canonical form ls -la [Path]. A template is admissible if it overlays every output in its group.

Algorithm 1 Greedy admissibility-preserving template refinement
Require: Construction group $(y_1,\ldots,y_n)$
Ensure: Template $T \preceq y$ for all $y \in (y_1,\ldots,y_n)$
1: if $n = 0$ or $y_1 = \epsilon$ then
2:     return $()$
3: end if
4: $T \gets (y_1)$
5: for $i \gets 2$ to $n$ do
6:     $y \gets y_i$; $T' \gets ()$; $p \gets 1$
7:     for $j \gets 1$ to $|T|$ do
8:         if $p \le |y|$ then
9:             $B \gets \operatorname{LCSBlocks}(T_j,y_p \cdots y_{|y|})$
10:            for $h \gets 1$ to $|B|$ do
11:                $\beta \gets B_h$
12:                $q \gets \operatorname{FirstOccurrence}(y,\beta,p)$
13:                append$(T',\beta)$
14:                $p \gets q + |\beta|$
15:            end for
16:        end if
17:    end for
18:    $T \gets T'$
19: end for
20: return $T$

Coverage and refinement. Literal coverage measures the number of output characters retained by a template:

cov⁡(T)=∑i=1k∣τi∣.(2)\operatorname{cov}(T) = \sum_{i=1}^{k} |\tau_i|. \tag*{(2)}

Concatenating admissible literals gives a common subsequence of the construction outputs. Maximum admissible coverage therefore corresponds to a longest common subsequence (LCS) across the outputs. The decision version for an arbitrary number of sequences is NP-complete [11], motivating the greedy procedure in Algorithm 1.

The first output initializes a single-segment template. For each subsequent output, the algorithm traverses the literals in order, aligning each against the suffix beginning at the current cursor. Each literal is aligned against the suffix beginning at that cursor. LCSBlocks⁡(a,b)\operatorname{LCSBlocks}(a,b) partitions the pairwise LCS alignment into maximal runs whose matched indices are consecutive in both strings. These contiguous blocks replace the old literal in alignment order. An empty alignment leaves the cursor unchanged. Otherwise, the cursor advances past each block’s earliest available occurrence in the new output.

FirstOccurrence⁡(y,β,p)\operatorname{FirstOccurrence}(y,\beta,p) returns the earliest index q≥pq \ge p at which the substring β\beta begins in yy. Figure 2 illustrates this refinement process wherein the first two outputs retain .txt, while the third retains only the shared period at that position.

Greedy refinement of three illustrative outputs

Figure 2. Greedy refinement of three illustrative outputs. Blue blocks are retained literals. Refinement with the third output removes the previously shared txt while retaining the period. The final \n denotes one newline character.

Lemma 1. The template returned by Algorithm 1 overlays every output in its construction group.

Proof. The claim is vacuous for an empty collection. If its first output is empty, the algorithm returns the empty template, which overlays every output. Otherwise, T(1)=(y1)T^{(1)} = (y_1) overlays y1y_1.

Assume T(i−1)=(τ1,…,τk)T^{(i-1)} = (\tau_1,\ldots,\tau_k) overlays all previously processed outputs. Each old segment τj\tau_j is replaced by a sequence BjB_j of ordered, non-overlapping substrings of τj\tau_j, or by the empty sequence. Replacing an occurrence of τj\tau_j in any previous output by the corresponding occurrences of its retained substrings preserves their order and non-overlap. Thus the concatenated blocks T(i)=B1∥⋯∥BkT^{(i)} = B_1 \mathbin\Vert\cdots\mathbin\Vert B_k still overlays every previously processed output.

In the new output yiy_i, the LCS alignment supplies ordered, non-overlapping occurrences of each BjB_j in the suffix available when TjT_j is processed. Choosing the earliest available occurrence of a block places its endpoint no later than the endpoint supplied by that alignment. Inductively, each subsequent aligned occurrence remains available. The cursor is carried between old segments, so the blocks also occur in global append order without overlap. Consequently, T(i)T^{(i)} also overlays yiy_i, completing the induction. □\square

With output lengths bounded by LL, the current literals have total length at most LL. Quadratic dynamic programming therefore spends O(L2)O(L^2) alignment time per refinement, giving O(nL2)O(nL^2) alignment time per group.

Inventories and assignment. For command identifier cc, the bank Tc={T0,T1,…,Tkc}\mathcal{T}^c = \{T_0,T_1,\ldots,T_{k_c}\} contains the non-empty templates constructed from its input groups and the empty template. For a reference output yy, the admissible indices are

Ac(y)={j∈{0,…,kc}:Tj⪯y}.\mathcal{A}^c(y) = \{j \in\{0,\ldots,k_c\}: T_j \preceq y\}.

We assign the template with maximum literal coverage:

j⋆=arg max⁡j∈Ac(y) cov⁡(Tj),T⋆=Tj⋆.(3)j^\star= \underset{j \in\mathcal{A}^c(y)}{\operatorname{arg\,max}}\ \operatorname{cov}(T_j), \qquad T^\star= T_{j^\star}. \tag*{(3)}

Maximum-coverage assignment therefore minimizes the remaining output among admissible inventory templates. This single assignment supplies both the template used to construct each TGR training target and the corresponding label for input-conditioned template selection.

Template Gap Reconstruction

For a training pair (x,y)(x, y), we use T=T∗T = T^{*} from Equation (3). When repeated literals permit multiple decompositions, the earliest-match rule in § 3.1 defines consistent gaps s0,…,sks_0, \ldots, s_k from the unmatched prefix, intervening text, and suffix. Each generated span must retain its insertion position, including when the span is empty. We associate each gap with a sentinel and represent the generation target as

T~=[S0]s0[S1]s1⋯[Sk]sk.(4)\widetilde{T} = [\mathrm{S}_0]s_0[\mathrm{S}_1]s_1 \cdots[\mathrm{S}_k]s_k. \tag*{(4)}

The final sentinel begins the trailing gap sks_k, after which ordinary response termination ends generation.

The reconstruction model receives the input xx and template TT, with the same sentinels inserted at the corresponding gap positions in the template. This identifies each span’s position relative to the surrounding literals, while xx supplies the command’s options and resource arguments. TGR aims to maximize the conditional likelihood of the gap target under the reconstruction model θ\theta:

Pθ(T~∣x,T)=∏t=1mPθ(T~t∣x,T,T~<t),(5)P_{\theta}(\widetilde{T} \mid x, T) = \prod_{t=1}^{m} P_{\theta}(\widetilde{T}_t \mid x, T, \widetilde{T}_{<t}), \tag*{(5)}

where mm is the tokenized target length, T~t\widetilde{T}_t is its tt-th token, and T~<t\widetilde{T}_{<t} contains the preceding reference target tokens.

During generation, the sentinels delimit the predicted gaps s^0,…,s^k\widehat{s}_0, \ldots, \widehat{s}_k. For a complete gap sequence, reconstruction interleaves these predictions with the supplied template literals:

y^=s^0τ1s^1τ2⋯τks^k.(6)\widehat{y} = \widehat{s}_0\tau_1\widehat{s}_1\tau_2 \cdots\tau_k\widehat{s}_k. \tag*{(6)}

The literals are copied rather than predicted, leaving their intervening text to the model. Substituting the reference gaps recovers yy by Equation (1).

Worked example. Consider the shell input

x = \texttt{stat --printf="size=\%s\n" ./a.txt}

with recorded output y = \texttt{size=128\n}, where \n denotes one newline character. For T = (\texttt{size=}, \texttt{\n}), the reference gaps are s0=s2=ϵs_0 = s_2 = \epsilon and s1=128s_1 = 128. The template is supplied as

[S0]size=[S1]\n[S2]

and the generation target is

T~=[S0][S1]128[S2].\widetilde{T} = [\mathrm{S}_0][\mathrm{S}_1]128[\mathrm{S}_2].

The adjacent first two sentinels preserve the empty leading gap, while the final sentinel begins the empty trailing gap. Generating 128 for the middle gap and leaving the boundary gaps empty reconstructs the recorded output by copying size= and the newline around the predicted value.

Template Selection

The assignment in Equation (3) depends on the reference output, which is unavailable when choosing a template for a submitted input. We therefore train a selector to predict the assigned index j∗j^{*} from xx. Using this assignment aligns selector supervision with the maximum-coverage admissible template used to construct each reconstruction-training target.

The command identifier cc determines the candidate inventory TcT^{c}, while the submitted options and arguments can distinguish response forms within that inventory. We therefore encode the complete input xx with a Transformer encoder, obtaining a pooled representation hx∈Rdh_x \in\mathbb{R}^{d}. The encoder is shared across commands, and a command-specific linear head scores the entries in TcT^{c}. Sharing the encoder allows input features to be learned across commands, while the separate heads restrict each decision to the corresponding template classes. Figure 3 illustrates this architecture.

Input-conditioned template selection architecture

Figure 3. Input-conditioned template selection. A shared encoder represents the submitted input, and a command-specific head scores its template inventory, including the empty-template FOD class.

Template selection aims to maximize the conditional likelihood of the assigned index under the selector ϕ\phi:

Pϕ(j⋆∣x,c)=softmax⁡(Wchx+bc)j⋆,(7)P_{\phi}(j^{\star} \mid x,c)=\operatorname{softmax}(W_{c}h_{x}+b_{c})_{j^{\star}}, \tag*{(7)}

where Wc∈R(kc+1)×dW_{c} \in\mathbb{R}^{(k_{c}+1)\times d} and bc∈Rkc+1b_{c} \in\mathbb{R}^{k_{c}+1} parameterize the command-specific head. The selector parameters ϕ\phi comprise the shared encoder and these heads and are distinct from the reconstruction parameters θ\theta. When no non-empty template overlays the reference, the assigned index is j⋆=0j^{\star}=0. Including the empty-template class therefore provides supervision for full-output fallback within the same classification task, rather than requiring a non-empty template for every input.

At inference, the selector chooses the template with the highest conditional probability:

j^=arg⁡max⁡0≤j≤kc Pϕ(j∣x,c),T^=Tj^,(8)\hat{j}=\underset{0\leq j\leq k_{c}}{\arg\max}\ P_{\phi}(j\mid x,c),\qquad\hat{T}=T_{\hat{j}}, \tag*{(8)}

with ties resolved by the fixed inventory order.

For j^>0\hat{j}>0, the reconstruction model receives xx and T^\hat{T}, serialized with the same sentinel convention used during training. The predicted template replaces the reference assignment; gap generation and assembly follow Equation (6). For j^=0\hat{j}=0, the same jointly trained reconstruction checkpoint instead receives xx alone and directly generates the complete response through its ordinary FOD interface. This retains full-output prediction for inputs routed to the empty-template class without requiring a separate reconstruction model.

Evaluation

We evaluate whether reusable output structure improves response fidelity at fixed reconstruction weights and where that benefit is concentrated. Complete-response and gap-only analyses separate assembled-response quality from the gap prediction needed for TGR across template coverage and gap length. Tokenized reference-target lengths quantify the text assigned to generation.

Experimental Setup

Data and training. We construct the template bank from 18.7 million shell outputs gathered across execution contexts. The configurations in Table 1 span 25 filesystem images with different users, environment variables, and contents. The construction procedure groups canonicalized inputs by the observed exit code and requires at least 32 execution records per group. The resulting bank is fixed before model training and evaluation. A separate one-million-record corpus from a single target environment supplies reconstruction and selection training examples. That environment and its synthesized executions are excluded from bank construction, although individual features may coincide across environments.

UserN CPUsKernelMemoryDisk
root15.15.2200.5G0.5G
non-root26.1.1281G1G
6.8.02G2G

Table 1. MicroVM configuration features for each execution environment across 25 filesystems.

We evaluate Qwen3-0.6B [28], ByT5-small and ByT5-base [27], and CodeT5-small and CodeT5-base [24], covering decoder-only and encoder-decoder architectures with subword or byte-level representations. Each backbone uses a single joint checkpoint trained on 62,500 minibatches of 32 samples, with a balanced mixture of FOD and TGR examples. The selector uses a shared CodeBERT encoder [7] and 86 command-specific heads. Training uses Adam, an initial learning rate of 10−410^{-4}, a 0.05 warm-up fraction, and linear decay. TGR targets use maximum-coverage admissible assignment from Equation (3). GPT-4o-mini receives an environment-description prompt, while Cowrie mirrors the target’s visible kernel, user, environment variables, and filesystem [13]. Both provide contextual baselines. Section A documents the baseline prompt and filesystem.

Output generation methods

Output generation methods. FOD generates the complete response from the submitted input. Selected TGR uses an input-conditioned selector in which a nonempty selection supplies the corresponding template; selecting T0T_0 invokes FOD. Oracle TGR instead uses the reference to select the maximum-coverage admissible template, retaining the same generation and fallback procedures; this gives an upper bound on TGR performance. All three methods use the same joint checkpoint for each backbone and hold the reconstruction weights fixed. Consequently, the comparison evaluates alternative inference interfaces available to that checkpoint.

Metrics

Metrics. Each output-generation method is evaluated on 1,000 test inputs using the outputs produced in the target environment. Output edit similarity (O-ES) is computed per record and then averaged across records; by definition, the O-ES score equals 1 when both strings are empty.

\operatorname{O\mathchar`-ES}(y,\hat{y}) = 1 - \frac{d_{\mathrm{Lev}}(y,\hat{y})}{\max(\lvert y\rvert,\lvert\hat{y}\rvert,1)}. \tag*{(9)}

This edit-similarity measure provides a direct quantitative comparison to the output observed from the targeted system. In addition, we report n-gram-based metrics to assess sequential fidelity. Mean BLEU-4 is computed using case-sensitive 13a tokenization, exponential smoothing, and effective n-gram orders up to four [16, 4]. All BLEU scores are obtained via this record-level computation. ROUGE-L F1 [10] is calculated over the same token sequence and then averaged per record. For both token-based metrics, the score is set to 1 when comparing two empty sequences and to 0 when exactly one sequence is empty. O-ES is reported on a 0–1 scale, whereas BLEU-4 and ROUGE-L F1 are reported on a 0–100 scale, with larger values indicating better performance. For each metric, the overall comparison aggregates results over all 1,000 inputs. Reported means across models are obtained by averaging five independently computed scores.

Template selection

Template selection. The CodeBERT selector obtains a template hit rate (THR) of 94.2% and a false template rate (FTR) of 4.0%. THR credits an overlaying nonempty template, or T0T_0 when no nonempty template is admissible. FTR counts incompatible nonempty selections over all inputs,

identifying copied literals that already conflict with the reference. A compatible selection need not be the maximum-coverage template, and the predicted gaps still determine the assembled response. THR therefore measures compatibility rather than exact agreement with the training label or the fraction of correct final responses. The selected–oracle comparison measures the consequence of changing the assignment policy.

Full Output Evaluation

Overall means. Table 2 reports mean O-ES of 0.789, 0.808, and 0.824 for FOD, selected TGR, and oracle TGR. The corresponding mean BLEU-4 scores are 76.43, 78.21, and 79.93, and ROUGE-L scores are 79.67, 81.82, and 83.31. Selected TGR thus improves the mean by 0.019 O-ES, 1.78 BLEU-4 points, and 2.15 ROUGE-L points over FOD. The agreement across character and token metrics supports a benefit to the complete response under multiple notions of textual similarity. Oracle assignment adds an additional 0.016 O-ES, 1.72 BLEU-4 points, and 1.49 ROUGE-L points over the selected TGR, indicating that the choice of supplied structure remains consequential.

MethodO-ESBLEU-4ROUGE-L F1
Cowrie0.35429.1235.52
GPT-4o-mini0.48842.6350.07
FOD0.78976.4379.67
Selected TGR0.80878.2181.82
Oracle TGR0.82479.9383.31

Table 2. Mean complete-response fidelity, averaged across five models fine-tuned for FOD/TGR.

All five models show improvement, with varying magnitudes (Table 7). Selected O-ES gains range from 0.012 for CodeT5-base to 0.036 for Qwen3-0.6B. Qwen3-0.6B also has the largest BLEU-4 and ROUGE-L gains, at 2.98 and 3.71 points, while CodeT5-base retains the highest absolute scores under all three generation conditions. Thus, the model that benefits most from the interface is not necessarily the one with the strongest final response fidelity. These within-checkpoint improvements support the usefulness of supplied structure across the evaluated architectures without implying a uniform gain or a model-size scaling relation.

ModelConditionO-ESBLEU-4ROUGE-L
CowrieFull response0.35429.1235.52
GPT-4o-miniFull response0.48842.6350.07
Qwen3-0.6BFOD0.77575.5078.41
Qwen3-0.6BSelected TGR0.81178.4882.12
Qwen3-0.6BOracle TGR0.82880.6683.78
ByT5-baseFOD0.78475.6279.22
ByT5-baseSelected TGR0.79876.8880.85
ByT5-baseOracle TGR0.81378.3882.20
ByT5-smallFOD0.77574.4278.44
ByT5-smallSelected TGR0.79476.2880.42
ByT5-smallOracle TGR0.81077.7581.93
CodeT5-baseFOD0.82380.3382.71
CodeT5-baseSelected TGR0.83581.5784.33
CodeT5-baseOracle TGR0.85183.2585.82
CodeT5-smallFOD0.79176.2679.57
CodeT5-smallSelected TGR0.80477.8581.36
CodeT5-smallOracle TGR0.82079.6182.79
MeanFOD0.78976.4379.67
MeanSelected TGR0.80878.2181.82
MeanOracle TGR0.82479.9383.31

Table 7. Complete-response fidelity.

Cowrie and GPT obtain O-ES of 0.354 and 0.488, respectively, on the same 1,000 inputs, well below the trained-model means under all three metrics. Because these baselines differ in training and conditioning, the within-checkpoint comparisons provide more direct evidence for the reconstruction interface and are provided to contextualize the trained model performance.

By template coverage. For a nonempty reference, available template coverage is given as

ρ(y)=100cov⁡(T∗)∣y∣,(10)\rho(y) = 100 \frac{\operatorname{cov}(T^*)}{|y|}, \tag*{(10)}

where T∗T^* is the extracted template with the highest coverage for that output. Every method is grouped by this same reference-informed quantity, including FOD and the contextual baselines. Coverage therefore describes the structure available for that output rather than that which was utilized. Figure 4 reports accuracy for records bucketed by their template coverage.

Mean complete-response fidelity by reference-informed template coverage

Figure 4. Mean complete-response fidelity by reference-informed template coverage. Trained curves average five models fine-tuned for FOD/TGR.

The most pronounced advantages of the selected-TGR approach are observed at coverage levels exceeding 50%. In particular, selected O-ES attains mean values of 0.879 and 0.875 in the (50, 75]% and (75, 100)% strata, respectively, compared with 0.742 and 0.790 for FOD. The resulting absolute improvements (0.137 and 0.085) are greater than the overall mean gain, and both token-level metrics likewise increase within these strata (Table 8). Collectively, these strata indicate substantial performance benefits at higher coverage levels.

The lower-coverage groups distinguish two limitations. At (25, 50]%, oracle O-ES improves over FOD from 0.859 to 0.909, but selected TGR scores 0.848. An admissible reference assignment is useful on average in this group, yet learned selection does not realize that benefit. At (0, 25]%, both selected and oracle TGR score 0.673, slightly below FOD’s 0.684. Here, supplying the maximum-coverage admissible template is itself insufficient to improve the group means, as correcting incompatible selections alone would not explain or remove the regression.

At the endpoints, oracle TGR and FOD agree at 0.691 O-ES when no nonempty template is available, consistent with their shared T0T_0 fallback. The fully templated group reaches 0.998 under oracle assignment and 0.900 under learned selection, versus 0.799 for FOD. Because its reference gaps contain no characters, this group primarily tests selection and reconstruction of supplied structure, including completion of the sentinel interface.

By reference-gap length. Because coverage can conceal a large absolute prediction problem in a long response, we also group complete-response scores by total reference-gap length,

ℓgap(y)=∑i=0k∣si∣=∣y∣−cov⁡(T∗).(11)\ell_{\mathrm{gap}}(y) = \sum_{i=0}^{k} |s_i| = |y| - \operatorname{cov}(T^*). \tag*{(11)}

This is the number of reference characters outside the assigned literals, summed across all gap slots.

Table 3 shows that TGR’s advantage over FOD is concentrated on gaps whose total length is < 64. For the shorter-gap buckets, selected and oracle TGR achieve mean O-ES of 0.903 and 0.964, respectively, compared with 0.812 for FOD. Selected TGR also improves BLEU-4 and ROUGE-L by 8.88 and 9.25 points, respectively. For the longer-gap buckets, the methods perform more similarly: mean O-ES is 0.804 for selected TGR, 0.822 for oracle TGR, and 0.809 for FOD, while selected-TGR differences narrow to −0.79 BLEU-4 points and +1.05 ROUGE-L points (Table 9). These results indicate that reusable output structure provides substantial gains when relatively little gap text remains to be generated, while TGR and FOD achieve comparable fidelity for longer gap sequences.

Condition01–1617–6465–256257+
O-ES
FOD0.7990.8550.7830.8270.790
Selected TGR0.9000.9380.8700.8430.765
Oracle TGR0.9980.9580.9350.8720.772
BLEU-4
FOD77.6382.7775.3977.5376.98
Selected TGR89.2589.4483.7479.5073.43
Oracle TGR99.7392.8191.2481.6974.43
ROUGE-L F1
FOD80.3586.1678.2784.0280.69
Selected TGR91.0793.6087.8787.0779.74
Oracle TGR99.8195.8793.9688.5880.64

Table 3. Mean full output similarity by reference-gap length, averaged across five models fine-tuned for FOD/TGR.

Gap Prediction Evaluation

We score generated gaps without credit for supplied literals, evaluating text outside the maximum-coverage admissible template. Oracle TGR fixes the same reference-informed template decomposition for all five models, separating gap prediction from learned template selection. For corresponding gaps sis_i and s^i\hat{s}_i,

\operatorname{Gap\mathchar`-ES} = 1 - \frac{\sum_i d_{\mathrm{Lev}}(s_i,\hat{s}_i)}{\max(1,\sum_i \max(\lvert s_i\rvert,\lvert\hat{s}_i\rvert))} \tag*{(12)}

All expected gaps remain in the reference, including missing trailing predictions. Gap BLEU-4 and ROUGE-L use case-sensitive 13a tokens and within-slot counts, preventing matches across gap boundaries (Section D.1).

By template coverage. Table 4 uses the four partial-coverage groups shared across models. The (25, 50]% group has the highest mean under all three gap metrics. Mean Gap ES increases from 0.472 in (0, 25]% to 0.865 in (25, 50]%, then decreases to 0.828 and 0.797 in the higher-coverage groups. Greater available coverage is therefore not accompanied by uniformly higher gap fidelity across these inputs. In particular, oracle complete-response O-ES rises from 0.909 in (25, 50]% to 0.949 in (75, 100)% while Gap ES falls from 0.865 to 0.797. The assembled response can benefit from more supplied correct text even when the remaining text is harder to predict accurately. The 0% group reduces to FOD and the 100% group has no nonempty reference gap, resulting in there being no difference from the full output evaluation in Table 3.

Coverage (%)Gap ESBLEU-4ROUGE-L F1
(0, 25]0.47244.9852.02
(25, 50]0.86583.9787.16
(50, 75]0.82873.2980.32
(75, 100)0.79773.9078.92

Table 4. Mean oracle-TGR gap fidelity by reference-informed template coverage, averaged across five models fine-tuned for FOD/TGR.

By reference-gap length. Table 5 instead groups gap-only scores by total reference-gap characters. Fully templated responses (those with zero gap characters) are absent because they require no nonempty gap prediction. Mean Gap ES decreases from 0.860 for 1–16 characters to 0.510 for 257–1,024, and gap ROUGE-L decreases from 85.34 to 55.07. Mean BLEU-4 likewise decreases from 83.31 to 50.61 across these groups. Unlike the coverage stratification, all three backbone-mean gap metrics decline at every reported length boundary. The character-level decline is also present for each individual backbone (Table 12), although individual token metrics have exceptions. Gap length thus provides a more consistent descriptive ordering of gap difficulty than coverage.

Gap charactersGap ESBLEU-4ROUGE-L F1
1–160.86083.3185.34
17–640.77474.8977.69
65–2560.69960.1470.33
257+0.51050.6155.07

Table 5. Mean oracle-TGR gap fidelity by reference-gap length, averaged across five models fine-tuned for FOD/TGR.

Model1–1617–6465–256257–1,024
Reference-gap length (characters)
Gap ES (0–1)
Qwen3-0.6B0.8290.7080.6600.436
ByT5-base0.8630.7690.6250.547
ByT5-small0.8780.7470.6870.472
CodeT5-base0.8560.7900.7600.589
CodeT5-small0.8760.8570.7650.505
Mean0.8600.7740.6990.510
BLEU-4 (0–100)
Qwen3-0.6B80.4568.0556.3443.71
ByT5-base84.1975.1251.6553.95
ByT5-small86.0771.5958.2945.47
CodeT5-base84.0176.7169.2060.00
CodeT5-small81.8383.0065.2049.94
Mean83.3174.8960.1450.61
ROUGE-L F1 (0–100)
Qwen3-0.6B82.8571.8166.5147.59
ByT5-base86.1277.1462.0858.31
ByT5-small87.7674.2469.3252.20
CodeT5-base85.8279.4077.2263.12
CodeT5-small84.1685.8776.5554.15
Mean85.3477.6970.3355.07

Table 12. Oracle-TGR gap-only fidelity by reference-gap length. Mean rows average separately calculated model scores.

Target Generation Length

The one-million-record training corpus supplies tokenized FOD references and their oracle TGR gap targets. We compare each tokenizer against its own FOD references, since byte-level and subword positions represent different amounts of output text. Table 6 reports 26.3–38.3% shorter reference gap targets, with the largest reduction for byte-level ByT5. The absolute reductions are 11.44 positions for Qwen3, 49.82 for ByT5, and 12.67 for CodeT5 per record on average. That all three remain positive after counting sentinels shows that delimiter overhead does not erase the mean reduction in this corpus. The percentage difference across tokenizer families also shows why a character-coverage objective cannot be treated as a tokenizer-independent estimate of generation cost. These measurements concern reference targets under reference-informed assignment where shorter targets provide a concrete efficiency opportunity for small shell models.

TokenizerFODTGRReduction %
Qwen342.8931.4526.7
ByT5129.9880.1638.3
CodeT548.1635.4926.3

Table 6. Mean lengths of the gap sequence for the optimal template. TGR counts include sentinel delimiters. Reduction is 100(1−TGR/FOD)100(1 - \mathrm{TGR}/\mathrm{FOD}).

Discussion

The results support output representation as a design choice for small language models used in shell simulation. Selected TGR raises mean O-ES from 0.789 to 0.808 and improves all three aggregate metrics for every evaluated backbone (Table 2 and Table 7). With reconstruction weights fixed, these gains reflect changes to the information provided and the text generated within the jointly trained model.

Representation and reconstruction capacity. ByT5-small with selected TGR exceeds ByT5-base with FOD under all three overall metrics, including O-ES of 0.794 versus 0.784. This within-family finding indicates that the choice of output representation can mitigate the empirically observed similarity gap across reconstruction model sizes, although it does not establish a scaling law nor constitute a general replacement for increased parameterization. The selector module introduces additional parameters and inference-time overhead. Deployment decisions should evaluate the joint configuration of the reconstruction model, selection mechanism, and output representation with respect to output similarity and available computational and memory budgets.

Complete-response scores credit copied template literals, while gap scores evaluate predictions without bias from template selection. Mean oracle Gap ES falls from 0.865 in the (25, 50]% coverage group to 0.797 in (75, 100)%, even as complete-response O-ES increases across those groups. A better assembled response can therefore coexist with less accurate gap prediction. Mean gap similarity also declines across reference-gap-length groups, identifying gap generation as a source of error.

Template compatibility permits error-free copying but does not identify the best reconstruction option. These conditions are asymmetrical in impact: an unnecessary T0T_0 choice forgoes possible reuse, while an incompatible nonempty template commits the response to literals that gap prediction cannot revise. Minimizing the false template rate is thus imperative to bridge the gap between selected and oracle TGR. Selection based on predicted reconstruction quality for a given backbone could keep FOD as a candidate even when a nonempty template is admissible.

Target length and deployment cost. The 26.3–38.3% shorter mean reference targets include sentinels and quantify fewer generation positions on the training corpus. Maximum character coverage need not minimize tokenized length because segmentation introduces delimiters and tokenizers encode literals and gaps differently. Actual generated lengths can differ from reference lengths, while learned selection changes the templates used and the selector and template context add computation.

The bank excludes the target environment, but both learned components receive target-environment training. As such, these results concern a specialized simulator. Similarly, this study examines strategies for enhancing output generation under single-input conditions and does not consider modifications to the execution environment that could influence subsequent interactions in multi-input sessions. Nevertheless, the observed performance gains complement approaches that condition upon and update the execution environment and offer a mechanism that can be integrated alongside such methods for output generation. MANTIS evaluates consistency across command sequences using explicit filesystem state and interaction-history management ([25]). Session-level evaluation with explicit state conditioning and checks of consequential fields would connect response similarity to stateful honeypot behavior.

Conclusion

We introduced template gap reconstruction (TGR) to improve output fidelity and reduce the text predicted by small language models for execution-free shell simulation. Templates extracted from 18.7 million executions across heterogeneous environments supply recurring literal spans, while the model generates the gaps needed to complete each response. An input-only selector chooses the template, with the empty template preserving full-output decoding (FOD) through the same jointly trained reconstruction checkpoint.

For each of five backbones spanning approximately 60–600 million parameters, selected TGR improves mean complete-response fidelity over FOD at fixed reconstruction weights. The benefits are concentrated in responses with substantial reusable structure and short gaps. Oracle TGR improves O-ES in every reported coverage group above 25%, while learned selection yields consistent gains above 50%. For responses with 0–64 reference-gap characters, oracle TGR achieves mean O-ES of 0.964 compared with 0.812 for FOD, with closer performance for longer gaps.

TGR also shortens mean tokenized reference targets by 26.3–38.3% across tokenizer families, including sentinel overhead. Together, these findings support reusable output structure as a practical design choice for compact shell simulators: supplied literals improve response fidelity while reducing the text assigned to autoregressive generation. Template selection and gap prediction remain central to the quality of the assembled response.

Acknowledgements

The research was sponsored by the Army Research Laboratory, accomplished under Cooperative Agreement Number W911NF-24-2-0180, and conducted in the Systems and Networks (SyN) Lab at The University of Texas at San Antonio using the Cyber Deception Testbed, funded by the Army Research Office under grant number W911NF-21-1-0188 and the National Security Agency through the awards H98230-21-1-0171 and H98230-20-1-0392. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or the U.S. Government. Parts of this research were facilitated by the use of experimental hardware via the Nvidia Launchpad program.

Supplementary Material

The contextual baselines and trained reconstruction models have different sources of information about the target environment. GPT-4o-mini receives an explicit environment description and the submitted input, while the reconstruction backbones are specialized using recorded executions from that environment (§ 4.1). Figure 5 documents the prompted interface, and Figure 6 shows the target working directory. These details help interpret the baseline comparison in that matching visible environment information does not make the training or conditioning procedures equivalent.

Instruction template provided for GPT-4o-mini baseline
You are emulating a Linux command-line interface for a user to
interact with. Features of this simulated environment are as follows:
{
’arch’: ’x86_64’,
’kernel’: ’6.8’,
’cpus’: ’1’,
’mem’: ’216M’,
’os_family’: ’ubuntu-24’,
’cwd’: ’/home/ubuntu’,
...
’env’: {’HOME’: ’/’,
            ...
        },
’fs’: {’.’: {
            ’group’: ’ubuntu’,
            ’owner’: ’ubuntu’,
            ’perms’: ’drwxrwxr-x’,
            ’size’: 4096,
            ’time’: ’19:43’,
            ...},
        ...
    },
...
}
Respond with the output containing exactly the text that would
be printed to stdout or stderr.

Figure 5. Instruction template provided for GPT-4o-mini baseline

Working directory of execution environment

Figure 6. Working directory of execution environment

Scoring and Evaluation Populations

The appendices separate three questions: how faithfully the complete response is reproduced, where that fidelity changes across response structures, and how accurately the model predicts the residual text. Complete-response scores evaluate the assembled output, including copied literals. Gap-only scores evaluate corresponding generated spans with their boundaries retained, so correct literals cannot compensate for residual errors. O-ES and Gap ES are character-level scores on a 0–1 scale. BLEU-4 and ROUGE-L F1 use case-sensitive 13a tokens and are displayed on a 0–100 scale. All three measure textual agreement with the recorded response, not semantic equivalence or consistency of shell state.

We report the mean of smoothed record-level BLEU-4 scores between the case-sensitive WMT 13a tokenized prediction and reference [16, 4]. For each system, the overall and stratified BLEU results average these record scores, and the model means average the separately computed model means. A pair with no matching unigrams scores zero. Orders without candidate n-grams are omitted, allowing short matching responses to receive credit without a nonexistent fourth-order denominator. We explicitly extend the score to one when both token sequences are empty and zero when only one is empty. The reference is never cropped, and the brevity penalty is applied within every nonempty record. These same sentence-level scores use exponential smoothing and effective order as documented for sentence BLEU [4]. ROUGE-L uses the same token sequence and averages record-level token-LCS F1. Two empty token sequences score one, and a pair with only one empty sequence scores zero. The same empty-agreement convention applies to BLEU-4.

Matched Populations and Aggregation

The overall complete-response comparison includes the same 1,000 inputs for all metrics and systems, including empty references. Available coverage and reference-gap length are determined by the reference-informed assignment T⋆T^\star. Thus, a coverage group describes the structure available for that input even for FOD and the contextual baselines, which do not receive a template. Reference-gap length is the total number of residual characters, ∑i∣si∣\sum_i |s_i|, across all reference gaps. In the following tables, bold values identify the highest displayed trained-system score in each metric column, with a separate comparison among model-mean conditions where multiple conditions are reported.

/home/ubuntu
├── logs_archive
│   ├── auth.log
│   ├── error_logs.log
│   ├── log_file.txt
│   ├── logs_compression.sh
│   ├── server_archive.log
│   ├── server_log.log
│   ├── syslog
│   ├── syslog.1
│   └── system_logs.csv
├── scripts
│   ├── automation_tools
│   │   ├── auto_backup.sh
│   │   ├── auto_cleanup.sh
│   │   ├── auto_deployment.sh
│   │   └── auto_repair.sh
│   └── performance_tuning
│       ├── cpu_load_reduction.sh
│       ├── disk_usage_monitor.sh
│       ├── memory_tuning.sh
│       └── memory_tuning.sh.gz
│
├── system_optimization.sh
├── auto.tar
├── auto.tar.gz
├── backup.sh
├── backup_script.py
├── cleanup_log.log
├── daily_cleanup.sh
├── hourly_sync.sh
├── hourly_sync.sh.gz
├── monthly_backup.sh
├── restore_database.sh
├── system_reboot.sh
├── weekly_maintenance.sh
├── temp_files
│   ├── temp_session_data.txt
│   ├── temporary_data.txt
│   ├── temporary_data.txt.gz
│   └── temporary_log.log
├── .bash_logout
├── .bashrc
├── .profile
└── config_settings.json

├── contact.txt
├── data.csv
├── database_backup.sql
├── documents.tar
├── environment_variables.sh
├── logs_archive.tar
├── logs_archive.tar.gz
├── maintenance_script.sh
├── network_setup.sh
├── projects.html
├── server_config.py
├── system_info.txt
├── system_status.sh
├── system_status_report.txt
├── update_notes.txt
├── update_notes.txt.1
├── update_notes.txt.2
├── user_config.json
├── user_data.csv
├── user_data.csv.gz
├── user_data_backup.sh
└── web.html

Complete-Response Fidelity

Overall Scores

Table 7 shows that the aggregate ordering is shared by every model: selected TGR improves on FOD, and reference-informed oracle TGR improves on selected TGR under all three metrics. Selected-minus-FOD O-ES gains range from 0.012 for CodeT5-base to 0.036 for Qwen3-0.6B. The largest gain therefore occurs for a model that is not the best absolute performer. CodeT5-base retains the highest complete-response score in each generation condition. This distinguishes benefit from the interface from the ranking of the underlying reconstruction models.

The selected-to-oracle O-ES differences are much more similar across models, ranging from 0.015 to 0.017. These gaps describe the cost of using the learned selection policy relative to the maximum-coverage reference assignment, including differences between compatible templates and fallback choices. They cannot be attributed solely to incompatible selections. Both contextual baselines score below every trained model even under FOD, so their separation from TGR reflects differences in training and conditioning as well as the reconstruction interface.

By Template Coverage

Table 8 locates the overall improvements within reference-defined coverage groups. For O-ES, every model improves with selected TGR in the (50, 75]%, (75, 100)%, and fully templated groups. The mean selected-minus-FOD gains in those groups are 0.137, 0.085, and 0.101, respectively. This consistency supports the usefulness of reusable structure when it accounts for a substantial part of the response. The low-coverage comparisons also expose the limits of that benefit: selected mean O-ES is 0.011 below FOD in each of the (0, 25]% and (25, 50]% groups.

ModelCondition0(0, 25](25, 50](50, 75](75, 100)100
O-ES
CowrieFull response0.3290.1730.5050.4750.4410.520
GPT-4o-miniFull response0.4030.2800.4510.3600.3430.309
Qwen3-0.6BFOD0.6820.6230.8180.7060.7430.743
Selected TGR0.6800.6500.8440.8770.8540.910
Oracle TGR0.6820.6540.9130.9210.9311.000
ByT5-baseFOD0.7280.6650.8500.8020.8300.820
Selected TGR0.7270.6740.8240.8720.8840.900
Oracle TGR0.7280.6730.9070.9080.9401.000
ByT5-smallFOD0.6440.6370.8610.6730.7680.809
Selected TGR0.6410.5910.8610.8790.8820.911
Oracle TGR0.6440.5970.8840.9060.9691.000
CodeT5-baseFOD0.7360.7740.8830.7630.8310.798
Selected TGR0.7310.7540.8390.8720.9040.890
Oracle TGR0.7360.7560.9190.9280.9450.990
CodeT5-smallFOD0.6640.7220.8820.7680.7810.824
Selected TGR0.6600.6980.8750.8980.8530.889
Oracle TGR0.6640.6870.9210.9340.9591.000
MeanFOD0.6910.6840.8590.7420.7900.799
Selected TGR0.6880.6730.8480.8790.8750.900
Oracle TGR0.6910.6730.9090.9190.9490.998

Table 8. Complete-response fidelity by available template coverage.

ModelCondition0(0, 25](25, 50](50, 75](75, 100)100
Template coverage (%)
BLEU-4
CowrieFull response24.5010.1034.0436.8928.7741.98
GPT-4o-miniFull response32.6116.0841.1329.1124.6720.94
Qwen3-0.6BFOD65.8759.2379.9767.1370.4871.48
Selected TGR65.8360.4179.6580.7580.3390.66
Oracle TGR65.8761.3388.1187.0390.02100.00
ByT5-baseFOD69.2861.5983.1174.1578.9080.56
Selected TGR69.2861.9079.6680.5083.5889.87
Oracle TGR69.2861.2087.6385.8589.71100.00
ByT5-smallFOD59.7459.0484.9160.5171.6779.35
Selected TGR59.7053.5284.1182.2882.7690.77
Oracle TGR59.7452.9985.5985.7392.79100.00
CodeT5-baseFOD70.9676.2486.8969.2478.5276.82
Selected TGR70.7672.6581.9583.1786.1587.88
Oracle TGR70.9672.7188.7589.7291.2798.65
CodeT5-smallFOD62.2369.7187.0568.1872.3479.96
Selected TGR62.0767.4586.1584.2479.5887.07
Oracle TGR62.2365.8889.8688.7091.71100.00
MeanFOD65.6265.1684.3967.8474.3877.63
Selected TGR65.5363.1882.3082.1982.4889.25
Oracle TGR65.6262.8287.9987.4191.1099.73

Table 8 (PDF p. 17). Complete-response fidelity by available template coverage (continued).

ModelCondition0(0, 25](25, 50](50, 75]75,10075, 100100
ROUGE-L F1
CowrieFull response32.0819.8748.8046.8041.2753.09
GPT-4o-miniFull response40.6531.1847.2138.5435.1032.67
Qwen3-0.6BFOD68.9365.2683.0570.8274.9775.97
Selected TGR68.7168.1784.1487.9887.5692.21
Oracle TGR68.9369.0092.1291.7694.30100.00
ByT5-baseFOD71.3671.2585.1379.4484.0882.78
Selected TGR71.2772.4582.5387.6289.4091.61
Oracle TGR71.3672.3690.9391.3294.36100.00
ByT5-smallFOD63.3467.9187.0767.0478.4682.11
Selected TGR62.9563.8886.6287.8089.2492.24
Oracle TGR63.3465.0489.3589.8796.48100.00
CodeT5-baseFOD73.8079.1287.5376.5284.3978.42
Selected TGR72.9777.9184.8889.2591.3690.06
Oracle TGR73.8078.2393.1694.0994.3099.05
CodeT5-smallFOD65.6774.8387.5976.4078.5382.48
Selected TGR65.1474.1887.6689.9286.7189.21
Oracle TGR65.6772.7592.8293.6295.33100.00
MeanFOD68.6271.6786.0774.0480.0980.35
Selected TGR68.2171.3285.1788.5288.8591.07
Oracle TGR68.6271.4891.6892.1394.9599.81

Table 8 (PDF p. 18). Complete-response fidelity by available template coverage (continued).

At zero available coverage, oracle TGR and FOD agree for every model, as expected from their shared empty-template fallback. Selected scores are slightly lower because selection can still route such an input to a nonempty template. At 100% coverage, oracle O-ES is 1.000 for four models and 0.990 for CodeT5-base, while selected scores range from 0.889 to 0.911. These cases require no nonempty reference gaps, so they principally test access to the appropriate literals and correct completion of the reconstruction interface. Less than perfect oracle scores in this group result from adding text to the beginnding and end gaps that exist from a fully templated output.

The token metrics broadly corroborate the coverage pattern, but they do not induce identical local rankings. For example, Qwen3-0.6B improves O-ES from 0.818 to 0.844 in the (25, 50]% group while its BLEU-4 decreases from 79.97 to 79.65. Reporting both character and token metrics retains these distinctions instead of treating any one measure as a complete account of response fidelity.

By Reference-Gap Length

Table 9 groups complete responses by the absolute amount of text outside the reference-informed template. Selected TGR improves mean O-ES over FOD by 0.083 for 1–16 residual characters and 0.087 for 17–64, but only by 0.016 for 65–256. In the longest reported group for outputs with gaps greater than 256 characters, FOD exceeds both selected and oracle TGR under all three model-mean metrics. Thus the favorable aggregate result does not extend to every residual-length regime, even when an admissible template is supplied.

ModelCondition01–1617–6465–256257+
O-ES
CowrieFull response0.5200.6930.4030.1940.149
GPT-4o-miniFull response0.3090.5200.3360.3070.245
Qwen3-0.6BFOD0.7430.8600.6800.7630.700
Selected TGR0.9100.9270.8460.8360.686
Oracle TGR1.0000.9410.9250.8630.694
ByT5-baseFOD0.8200.8850.8200.8580.806
Selected TGR0.9000.9570.8870.8270.840
Oracle TGR1.0000.9510.9540.8550.837
ByT5-smallFOD0.8090.8390.7750.7740.785
Selected TGR0.9110.9360.8730.8510.702
Oracle TGR1.0000.9910.9250.8600.695
CodeT5-baseFOD0.7980.8670.8180.9020.853
Selected TGR0.8900.9550.8660.8590.842
Oracle TGR0.9900.9540.9350.8950.866
CodeT5-smallFOD0.8240.8240.8230.8390.809
Selected TGR0.8890.9140.8790.8430.757
Oracle TGR1.0000.9520.9340.8850.770
MeanFOD0.7990.8550.7830.8270.790
Selected TGR0.9000.9380.8700.8430.765
Oracle TGR0.9980.9580.9350.8720.772

Table 9. Complete-response fidelity by reference-gap length. Each metric uses the same reference-defined bins and fixed scoring protocol.

ModelCondition01–1617–6465–256257+
BLEU-4
CowrieFull response41.9852.0927.7511.0912.95
GPT-4o-miniFull response20.9445.0423.2121.3616.60
Qwen3-0.6BFOD71.4884.0965.5372.5167.26
Selected TGR90.6688.5779.6878.0764.08
Oracle TGR100.0091.6290.0480.4666.22
ByT5-baseFOD80.5685.8779.2581.7878.97
Selected TGR89.8791.5785.5778.2780.89
Oracle TGR100.0091.9893.9779.3580.51
ByT5-smallFOD79.3582.1874.1371.1976.17
Selected TGR90.7788.7184.7379.3565.82
Oracle TGR100.0096.5989.9479.1464.78
CodeT5-baseFOD76.8282.7678.8285.8183.98
Selected TGR87.8892.7883.9782.1781.74
Oracle TGR98.6593.3191.5185.7785.03
CodeT5-smallFOD79.9678.9479.2176.3478.54
Selected TGR87.0785.5884.7379.6374.61
Oracle TGR100.0090.5590.7583.7575.60
MeanFOD77.6382.7775.3977.5376.98
Selected TGR89.2589.4483.7479.5073.43
Oracle TGR99.7392.8191.2481.6974.43

Table 9 (PDF p. 20). Complete-response fidelity by reference-gap length (continued).

ModelCondition01–1617–6465–256257+
ROUGE-L F1
CowrieFull response53.0965.7542.3919.2318.31
GPT-4o-miniFull response32.6753.5034.6933.1229.33
Qwen3-0.6BFOD75.9786.4968.2477.1473.31
Selected TGR92.2192.8885.8985.3571.69
Oracle TGR100.0095.0393.8286.9372.77
ByT5-baseFOD82.7889.3781.7887.4682.09
Selected TGR91.6195.0388.8786.0586.36
Oracle TGR100.0095.5495.4487.2886.50
ByT5-smallFOD82.1185.4377.6478.9780.09
Selected TGR92.2493.3888.3587.2373.91
Oracle TGR100.0098.4292.6787.0473.99
CodeT5-baseFOD78.4286.9081.8291.5985.61
Selected TGR90.0695.8587.8988.9886.22
Oracle TGR99.0595.7294.4091.4488.39
CodeT5-smallFOD82.4882.6381.8684.9682.36
Selected TGR89.2190.8688.3487.7480.52
Oracle TGR100.0094.6493.4990.2181.55
MeanFOD80.3586.1678.2784.0280.69
Selected TGR91.0793.6087.8787.0779.74
Oracle TGR99.8195.8793.9688.5880.64

Table 9 (PDF p. 21). Complete-response fidelity by reference-gap length (continued).

Complete-Response Length Curves

Output edit similarity

(a) (a) Output edit similarity

Mean BLEU-4

(b) (b) Mean BLEU-4

ROUGE-L F1

(c) (c) ROUGE-L F1

Figure 7. Complete-response fidelity for five-models means and both baselines by reference-gap length.

Gap Prediction Fidelity

Gap-Scoring Protocol

Reference-informed assignment and earliest successive literal matches fix the reference gaps, with predictions being parsed from the saved sentinel sequence. Template literals and sentinels are not scored, and all expected gaps remain, including missing trailing predictions.

Gap-ES⁡(s,s^)=1−∑idLev(si,s^i)max⁡(1,∑imax⁡(∣si∣,∣s^i∣)).(13)\operatorname{Gap\text{-}ES}(s,\hat{s}) = 1 - \frac{\sum_i d_{\mathrm{Lev}}(s_i,\hat{s}_i)}{\max\left(1,\sum_i \max\left(\lvert s_i\rvert,\lvert\hat{s}_i\rvert\right)\right)}. \tag*{(13)}

BLEU-4 accumulates counts and lengths within corresponding gap slots of one record before scoring and averaging. Counts are scored independently within each slot, and neither matches nor candidate n-grams cross a gap boundary. Empty gap slots receive no separate scores or weights. Gap ROUGE-L sums the within-gap subsequence lengths and token totals before calculating each record’s F1. Those record scores are then averaged. No n-gram or subsequence match crosses a gap boundary, and no perfect scores are added for individual empty slots.

Overall Scores

ModelGap ESBLEU-4ROUGE-L F1
Qwen3-0.6B0.59755.9161.67
ByT5-base0.61456.1163.61
ByT5-small0.60154.8462.57
CodeT5-base0.68465.7670.56
CodeT5-small0.68363.5469.34
Mean0.63659.2365.55

Table 10. Overall oracle-TGR gap fidelity.

By Template Coverage

Model(0, 25](25, 50](50, 75](75, 100)
Gap ES (0–1)
Qwen3-0.6B0.4330.8750.8040.728
ByT5-base0.4430.8690.8080.781
ByT5-small0.4090.8380.8240.806
CodeT5-base0.5460.8570.8480.827
CodeT5-small0.5310.8850.8570.841
Mean0.4720.8650.8280.797
BLEU-4 (0–100)
Qwen3-0.6B42.0184.4570.8867.34
ByT5-base40.4184.2470.6872.76
ByT5-small36.7380.4272.7075.83
CodeT5-base53.9783.5378.6277.83
CodeT5-small51.7987.2073.5875.76
Mean44.9883.9773.2973.90
ROUGE-L F1 (0–100)
Qwen3-0.6B47.5988.0777.5073.63
ByT5-base49.9286.6677.4877.64
ByT5-small46.5684.3279.4980.41
CodeT5-base58.8887.0384.4182.00
CodeT5-small57.1689.7082.7480.93
Mean52.0287.1680.3278.92

Table 11. Oracle-TGR gap fidelity by reference-informed template coverage. Mean rows average separately calculated model scores.

By Reference-Gap Length

References

  1. [1]Adebimpe, A., Neukirchen, H., and Welsh, T. Shash: a framework for designing and evaluating rag vs. prompt-tuned llm honeypots. In 2025 3rd International Conference on Foundation and Large Language Models (FLLM), pp. 851–856. IEEE, 2025.DOI
  2. [2]Barzilay, R. and Lee, L. Learning to paraphrase: An unsupervised approach using multiple-sequence alignment. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pp. 16–23, 2003.
  3. [3]Bridges, R. A., Mitchell, T. R., Muñoz, M., and Henriksson, T. Sok: Honeypots & llms, more than the sum of their parts? In 2026 IEEE 11th European Symposium on Security and Privacy (EuroS&P), pp. 910–928. IEEE, 2026.arxiv.org/abs/2510.25939
  4. [4]Chen, B. and Cherry, C. A systematic comparison of smoothing techniques for sentence-level bleu. In Proceedings of the ninth workshop on statistical machine translation, pp. 362–367, 2014.DOI
  5. [5]Donahue, C., Lee, M., and Liang, P. Enabling language models to fill in the blanks. In Proceedings of the 58th annual meeting of the association for computational linguistics, pp. 2492–2501, 2020.arxiv.org/abs/2005.05339
  6. [6]Du, M. and Li, F. Spell: Streaming parsing of system event logs. In 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 859–864. IEEE, 2016.DOI
  7. [7]Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. Codebert: A pre-trained model for programming and natural languages. In Findings of the association for computational linguistics: EMNLP 2020, pp. 1536–1547, 2020.arxiv.org/abs/2002.08155
  8. [8]Greenberg, M. and Blatt, A. J. Executable formal semantics for the posix shell. Proceedings of the ACM on Programming Languages, 4(POPL):1–30, 2019.arxiv.org/abs/1907.05308
  9. [10]Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74–81, 2004.
  10. [11]Maier, D. The complexity of some problems on subsequences and supersequences. Journal of the ACM (JACM), 25(2):322–336, 1978.DOI
  11. [13]Oosterhof, M. cowrie/cowrie: Cowrie ssh/telnet honeypot, 2014. URL https://github.com/cowrie/cowrie.
  12. [15]Otal, H. T. and Canbaz, M. A. Lim honeypot: Leveraging large language models as advanced interactive honeypot systems. In 2024 IEEE Conference on Communications and Network Security (CNS), pp. 1–6. IEEE, 2024.DOI
  13. [16]Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311–318, 2002.
  14. [17]Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficient scaling transformer inference. Proceedings of machine learning and systems, 5:606–624, 2023.arxiv.org/abs/2211.05102
  15. [18]Post, M. and Vilar, D. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1314–1324, 2018.arxiv.org/abs/1804.06609
  16. [19]Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):1–67, 2020.arxiv.org/abs/1910.10683
  17. [20]Ragsdale, J. and Boppana, R. Shioenv: An environment for synthesizing bash shell inputs with execution artifacts. arXiv preprint arXiv:2505.18374, 2025.
  18. [21]Sladić, M., Valeros, V., Catania, C., and Garcia, S. In the shell: Generative honeypots. In 2024 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), pp. 430–435. IEEE, 2024.DOI
  19. [22]Sladić, M., Alibalić, E., Valeros, V., Catania, C., and Garcia, S. Advancedshell: A stateful multi-agent llm honeypot for ssh deception. arXiv preprint arXiv:2606.27990, 2026a.arxiv.org/abs/2606.27990
  20. [23]Sladić, M., Valeros, V., Alibalić, E., and Garcia, S. Improving llm-based ssh honeypots through prompting and fine-tuning. arXiv preprint arXiv:2608.18686, 2026b.arxiv.org/abs/2608.18686
  21. [24]Wang, Y., Wang, W., Joty, S., and Hoi, S. C. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 conference on empirical methods in natural language processing, pp. 8696–8708, 2021.arxiv.org/abs/2109.00859
  22. [25]Wei, K., Cui, Y., Shen, K., Wei, H., and Wang, G. A multi-agent framework for high-interaction terminal simulation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 32828–32845, 2026.
  23. [26]Wiseman, S., Shieber, S. M., and Rush, A. M. Learning neural templates for text generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3174–3187, 2018.arxiv.org/abs/1808.10122
  24. [27]Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., Kale, M., Roberts, A., and Raffel, C. Byt5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306, 2022.
  25. [28]Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
  26. [29]Zhang, T., Lee, M., Li, X. L., Shen, E., and Hashimoto, T. B. Templ: Distilling language models into template-based generators. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 1970–1994, 2023.arxiv.org/abs/2205.11055
  27. [30]Zhang, Y., Wang, G., Li, C., Gan, Z., Brockett, C., and Dolan, W. B. Pointer: Constrained progressive text generation via insertion-based generative pre-training. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp. 8649–8670, 2020.

Paper details

vlnc:W_m2jxvxxm
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 7

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents