WorksComputer Science
ReconPotBench: Evaluating LLM Honeypots in Multi-Round Linux Reconnaissance
Abstract
Large language models (LLMs) are increasingly used to build interactive honeypots, yet existing evaluations mainly assess isolated command outputs and cannot determine whether a system remains believable when later commands depend on earlier results. We present ReconPotBench, a multi-round benchmark for Linux reconnaissance honeypots. It contains 1{,}032 three-round reconnaissance interactions across eight reconnaissance categories, with reference outputs collected from a real Linux environment. Under a unified testing protocol, we compare Cowrie, open-source LLMs, and closed-source LLMs along three dimensions: round-level authenticity, cross-round contextual consistency, and hacker-oriented reconnaissance value. Results show that multi-round evaluation exposes failures hidden by single-round testing, as authenticity declines with interaction depth and both open-source and closed-source LLMs struggle to maintain coherent state across rounds. LLM-based honeypots nevertheless outperform Cowrie in authenticity and reconnaissance value, while Cowrie remains more consistent because of its rigid templated behavior. Category-level analysis further shows that tasks requiring systems to preserve network relationships across rounds are harder than tasks centered on explicit entities such as ports. ReconPotBench enables reproducible evaluation of LLM honeypots across authenticity, entity consistency, and hacker-perspective utility in dependent multi-round interactions. The dataset is available at: https://github.com/qianli-wang/ReconPotBench
Figure 1. Motivation of our benchmark: existing Linux command-output datasets mainly focus on isolated single-round interactions, while realistic reconnaissance involves multi-round command sequences with contextual dependencies across rounds.
Introduction
Reconnaissance is the critical first phase of the cyber attack lifecycle, and it forms the basis for all subsequent attack activities. As specified in the MITRE ATT&CK Framework (MITRE ATT&CK, 2025a), reconnaissance is the initial tactic adopted
by adversaries to collect information for targeting operations. Studies have shown that attackers spend most of their time on this phase (Kazimierczak et al., [5]). Using large language models (LLMs) as an interactive deception mechanism to counter reconnaissance activities is a promising paradigm in active defense. Evaluating the effectiveness of these LLM-based defense methods is critical, since effective detection and analysis in the reconnaissance phase allow defenders to identify and neutralize threats before they develop into active exploitation ([14, 4, 12, 6, 15]).
However, existing evaluations of LLM-based honeypots are not designed for multi-round Linux reconnaissance. Prior work mainly falls into three lines: feasibility studies showing that LLMs can simulate shell interactions (McKee and Noever, 2023); per-response realism evaluations based on expert judgment, template-based references, or surface-level similarity metrics (Sladić et al., [14]; Otal and Canbaz, [12]; Malhotra, [6]); and longer interactive evaluations, attacker engagement, or session-level deception quality (Guan et al., [4]; Wang et al., [15]; Yang et al., [17]). These studies demonstrate the promise of LLM honeypots, but their evaluations miss the dimensions attackers would actually notice during Linux reconnaissance. In realistic Linux reconnaissance, attackers assess not only whether each output looks authentic, but also whether entities such as IP addresses remain consistent across rounds, and whether the interaction continues to support meaningful follow-up probing. Failures in authenticity, contextual consistency, or hacker-perspective utility can reveal the deception, yet prior evaluations do not measure these dimensions jointly in sequentially dependent command-output interactions.
Figure 1 highlights this mismatch between existing single-round evaluation settings and the context-dependent nature of realistic reconnaissance behavior, motivating the need for a benchmark that can evaluate both round-level realism and cross-round contextual coherence. We introduce ReconPotBench, a benchmark of fixed three-round Linux reconnaissance interactions that jointly evaluates round-level authenticity, cross-round consistency, and attacker-perspective utility. Our approach stems from the observation that real-world attackers do not execute commands in isolation. They instead carry out extended interaction sequences, where each command relies on information gathered from prior interactions. This establishes multi-round testing as a necessary component rather than an optional step. Single-round evaluation only verifies whether an individual response appears authentic. It cannot determine whether the system preserves a coherent host state across turns, nor whether the accumulated outputs provide valuable information for follow-up reconnaissance. This distinction is important because attackers judge a system not only by local output realism, but also by whether entities remain consistent, and whether the interaction continues to support meaningful probing.
To evaluate these properties, we build a dataset with 1,032 three-round interaction test cases across eight reconnaissance scenarios. Each test case follows a Network Software Hardware hierarchy, with ground-truth outputs collected from a physical Linux system.
The commands are constructed from standard Linux command options (Shotts, [13]) to ensure valid reconnaissance intent. This design enables ReconPotBench to measure three properties: round-level authenticity, cross-round contextual consistency, and hacker-perspective utility. Together, they capture not only whether each output looks realistic, but also whether the full multi-round interaction remains coherent and useful from an attacker’s perspective. We summarize our main contributions as follows:
We introduce ReconPotBench, a multi-round benchmark for evaluating LLM-based honeypots in Linux reconnaissance. Instead of checking isolated command responses, ReconPotBench turns reconnaissance into 1,032 three-round NetworkSoftwareHardware interaction chains with real-system reference outputs.
We develop a testing protocol around three hacker-relevant dimensions: round-level authenticity, contextual consistency, and hacker-perspective utility. Applying it to Cowrie, closed-source LLMs, and open-source LLMs shows why single-round evaluation is not enough: LLM honeypots can produce more realistic and more valuable reconnaissance outputs than Cowrie, but their authenticity drops as the interaction deepens and their cross-round entity consistency remains weak.
We conduct a category-level evaluation over eight reconnaissance categories. The results show that scenarios with explicit entities, such as ports or service names, are easier to maintain, while tasks involving implicit relations, such as routing logic, DNS configuration, and interface-address bindings, expose more state-tracking failures. This points to a practical bottleneck for LLM honeypots: plausible terminal text alone is not enough; the system also needs stable host-state memory across rounds.
Background and Related Work
Recent studies have begun to use LLMs as response generators for shell or terminal honeypots. The evaluation question has therefore shifted from whether a model can answer a command to whether the honeypot remains believable as an attacker keeps interacting with it. Existing work approaches this problem from three angles: feasibility, single-response realism, and longer interactive behavior. Early work first establishes feasibility. McKee and Noever [8] shows that chatbot-based systems can emulate terminal interactions across operating systems and common application and network tools, suggesting that LLMs can serve as honeypot interfaces. This line of work mainly answers whether such interfaces can be built, rather than how closely each generated command output follows real shell behavior.
A second line evaluates command-level realism. Sladić et al. [14] asks experts to judge generated responses against standard Linux shell behavior on a per-command basis. Otal and Canbaz [12] compares generated outputs with Cowrie references and further validates the system through deployment as an interactive SSH honeypot. Malhotra [6] tests 138 representative Linux commands against real shell outputs, reporting automatic quality metrics together with latency and memory cost. However, their limitation is also clear: each command is treated as a standalone case, so contradictions that only appear after several dependent rounds are largely outside the scope.
A third line studies longer sessions. Guan et al. [4] moves beyond single-command realism and identifies accuracy and consistency as key challenges for natural shell emulation, using prompt engineering to support longer interactions. Wang et al. [15] compares a ChatGPT-powered terminal honeypot with traditional honeypots and real systems, and also reports a three-month deployment alongside Cowrie to observe attacker engagement in practice. Yang et al. [17] targets multi-round inconsistency directly through interaction history pruning and dynamic error simulation. These studies show that interaction-level behavior matters, but their evaluations are still centered on output similarity or coherence. They do not directly evaluate whether a dependent multi-round reconnaissance chain preserves hacker-relevant state and remains useful for follow-up probing.
ReconPotBench is designed as a direct response to the evaluation problems left by these three lines of work. For feasibility-oriented studies, the question is no longer whether an LLM can imitate a shell at all, but whether such a shell can be tested under realistic Linux reconnaissance; we therefore fix the benchmark around concrete terminal commands and real host outputs. For single-command realism studies, the missing piece is that a locally plausible response may still fail once later rounds depend on earlier rounds; we therefore organize each case as a controlled three-round Network Software Hardware reconnaissance chain and evaluate round-level authenticity, cross-round consistency, and hacker-perspective utility together. For longer-session studies, interaction quality or coherence alone still does not show whether the session is valuable from an attacker’s viewpoint; we therefore add hacker-perspective utility to measure whether the accumulated outputs support meaningful follow-up probing. This design makes ReconPotBench more than another shell-emulation test: it measures output authenticity, cross-round entity consistency, and the value of outputs for supporting hackers’ follow-up probing across dependent reconnaissance rounds.
Methodology
Benchmark Architecture
Figure 2 presents the overall design of ReconPotBench. Each instance is a three-round Linux reconnaissance chain that keeps a single reconnaissance objective across rounds, allowing us to test whether a system can remain believable as later commands depend on earlier outputs. We construct 1,032 multi-round interactions from a real Linux environment and organize them into eight representative reconnaissance scenarios. Under the unified protocol, the evaluated system receives only the current command in Round 1, while Rounds 2 and 3 additionally include the previous commands and generated outputs from the same interaction. The real-system reference outputs are never exposed during generation and are used only for evaluation. ReconPotBench compares traditional honeypots, closed-source LLMs, and open-source LLMs along three dimensions: round-level authenticity, cross-round consistency, and hacker-perspective utility. This design enables controlled comparison of whether a system can generate realistic terminal outputs, preserve coherent host state, and provide valuable reconnaissance information throughout a dependent multi-round interaction.
Figure 2. Overview of ReconPotBench. The benchmark targets multi-round Linux reconnaissance interactions, in which later commands depend on earlier context, and enables unified evaluation of systems along three dimensions: round-level authenticity, cross-round consistency, and hacker-perspective utility.
Dataset Construction
Three-Round Command Hierarchy
Our dataset follows a Network Layer Software Layer Hardware Layer hierarchy to model dependent Linux reconnaissance. Inspired by MITRE ATT&CK TA0043 (MITRE ATT&CK, [9]) and common attack workflows [16], this design reflects how attackers first gather network-level clues and then use them to inspect related services, software, and host resources. Round 1 probes IPs, routes, ports, and interfaces; Round 2 examines services, packages, configurations, and versions; and Round 3 probes hardware or host-resource context. This structure turns each instance into a connected reconnaissance chain, enabling ReconPotBench to evaluate authenticity, cross-round contextual consistency, and hacker-relevant reconnaissance value within the same interaction. The hierarchy provides a controlled sequence for examining related information within a case; it does not prescribe a universal order for real reconnaissance. For example, an interface-centered case can examine its addresses, kernel network configuration, and driver and bus location. MITRE reconnaissance and discovery behaviors inform task selection, while the fixed three-round structure is a benchmark design choice.
Eight-Category Command Taxonomy
The eight categories distinguish the objects inspected and the information returned, whereas the three-layer structure specifies how inspection proceeds within each case. For example, listening-port inspection asks which services are listening, connection-state inspection asks which communications are active, and connection-to-process mapping asks which processes are responsible for those communications. These tasks require different information and entity relationships, so we evaluate them separately. The categories cover the common inspection tasks selected for this benchmark and are not an exhaustive taxonomy of reconnaissance. We include 129 cases per category to provide balanced coverage, rather than to represent their frequency in real attacks. Table 5 summarizes the categories, and Appendix § 14 provides the reconnaissance references and representative commands for all eight categories.
| ID | Round 1 Network-layer | Round 2 Software-layer | Round 3 Hardware-layer |
| 1 | Network Interfaces & IP | Interface count, address type, and bound services | interface-related hardware |
| 2 | Open Ports & Listening Services | Port-to-service mapping and version clues | Service-specific hardware needs |
| 3 | Port Connection States | Client/server software behind active connections | Load-related hardware bottlenecks |
| 4 | Routing Table | Gateway/subnet clues for routing or firewall software | Routing and NIC hardware support |
| 5 | Service Versions & Components | Package, component, and vulnerability-related software clues | Deployment-related hardware support |
| 6 | DNS Configuration | DNS software stack | Local network role |
| 7 | ARP Cache & MAC Addresses | ARP-related system utilities | LAN device identity and hardware details |
| 8 | Connection-to-Process Mapping | Process paths and software dependencies | Host resource and hardware context |
Table 5. Eight reconnaissance categories and their three-round dependency chain.
Data Generation Methodology
First-Round Command. We manually design 1,032 first-round commands, with 129 commands per category. This design is important because the first command determines how the three-round dependency chain begins. It defines the initial reconnaissance intent, controls category coverage, provides the basis for meaningful follow-up commands, and helps ensure that subsequent evaluations are valid. Each command is derived from the eight reconnaissance scenarios in § 3.2.2, instantiated using standard Linux command options from authoritative documentation [13] and penetration-testing practice, and validated on a Linux system before inclusion.
Second-Round & Third-Round Commands. We generate the next two rounds of commands using GPT-5.4 with a prompt described in Appendix § 6. Because the first-round commands are manually designed and category-aligned, the LLM is used only to extend a controlled reconnaissance chain rather than to define the task from scratch. Given a first-round command, the prompt identifies its scenario and generates one software-layer command and one hardware-layer command under the Network Layer → Software Layer → Hardware Layer progression [16]. Both follow-up commands must remain within the same reconnaissance scenario and preserve logical dependence on prior rounds. We manually review all generated commands for technical correctness and contextual suitability.
Ground Truth Collection and Experimental Validation
We use a physical Linux host as a controlled, target-like reference environment for collecting real terminal outputs. All commands are executed on this host, and the resulting outputs are used as reference labels. Real-system labels are necessary because authenticity cannot be reliably judged against simulated or template-based outputs. We deploy standard penetration-testing tools and services on the host to support realistic Linux reconnaissance, preserve formatting and special characters in plaintext outputs, and reset system state when needed to avoid interference across multi-round command chains. We manually inspect the collected outputs to ensure that all commands execute correctly and that the collected references are free from execution or collection errors.
Dataset Statistics
The final dataset includes eight categories of reconnaissance scenarios as shown in Table 5. Each category includes 129 multi-round interaction groups, and each group comprises three rounds of command-output pairs. The dataset contains 1,032 test cases in total.
Testing Protocol
For single-round evaluation, we use the first command of each case without prior interaction history. For multi-round evaluation, we issue all three commands in order. During LLM response generation, Rounds 2 and 3 receive the preceding commands and generated outputs from the same conversation. Reference outputs are never included in this history. All three commands are constructed before evaluation and remain fixed across systems. Later responses therefore depend on the interaction history, but later commands are not rewritten from the generated outputs. An entity named in a follow-up command may have been specified during dataset construction. Appendix § 11 presents a concrete three-round Linux reconnaissance interaction, including the commands and reference outputs.
Evaluation Metrics
Overview. All three metrics in ReconPotBench are evaluated with an LLM-as-a-judge protocol. This choice is important because traditional metrics, such as exact matching or surface-level similarity, are too brittle for terminal-output evaluation: two outputs may differ in formatting while conveying the same system state, and a response may look locally plausible while still breaking entity continuity or offering little value for follow-up probing. Following our experimental pipeline, we use GPT-5.4-mini as a unified evaluator, and the full evaluation prompt is provided in Appendix § 6. This unified protocol ensures that Cowrie, closed-source LLMs, and open-source LLMs are assessed under the same standard. We evaluate each test subject from three perspectives: authenticity, cross-round consistency, and hacker-perspective utility, all computed on complete three-round reconnaissance interactions. These metrics are jointly necessary because they capture different properties of a LLM honeypot system. Authenticity measures realism with respect to real-system outputs, consistency measures whether entities are preserved, and utility measures whether the generated content remains valuable from the hacker’s perspective. A system may perform well on one of these dimensions while performing poorly on another, so evaluating only a single metric would provide an incomplete benchmark.
Let denote the number of evaluated three-round reconnaissance sequences, and let denote sequence . We use the reference sequence in Appendix § 11 to illustrate the scoring rules. For this illustration, the reference outputs themselves are treated as the evaluated outputs. The illustrative labels are not additional experimental results for any evaluated system.
Authenticity Score. Each command-output interaction receives one binary authenticity score. For round , let , , and denote the command, evaluated output, and corresponding reference output, respectively. The evaluator judges agreement in command semantics, the type of information returned, and terminal-output form:
A score of 1 indicates agreement with the reference under these criteria, and 0 indicates disagreement. This judgment does not calculate string overlap.
The mean authenticity across the three rounds is
For the example in Appendix § 11, we illustrate the calculation by treating the reference outputs themselves as the evaluated outputs. Under the scoring rules, the illustrative labels and their three-round mean are
These values illustrate the scoring rules and are not additional experimental results for any evaluated system.
Cross-Round Consistency Score. Consistency is evaluated over each complete three-round sequence. The evaluator assigns a label by examining entities and their relationships across rounds. A label of 1 indicates consistency, whereas 0 indicates a contradiction or a mismatch in these relationships. Sequences without explicit entities are labeled NA.
Consistency checks whether entities are used consistently across rounds. Different rounds may reveal different attributes of the same entity, but its identity, previously reported attribute values, and relationships with other entities must not change without justification.
In the example in Appendix § 11, all three rounds examine the same interface, wlan0. Round 1 reports its addresses, Round 2 reports its network configuration, and Round 3 reports its associated device information. The interface identity is preserved across rounds, with no conflicting attribute values or mismatched entity references, giving the illustrative label
Across the dataset, the consistency score is
| Metric | Matched / Total | Agreement |
| Authenticity | 1,134 / 1,248 | 90.87% |
| Consistency | 405 / 416 | 97.36% |
| Utility | 380 / 416 | 91.35% |
Table 1. Agreement between manual annotation and GPT-5.4-mini evaluator outputs on the sampled validation set.
where and are the numbers of complete three-round sequences labeled consistent and inconsistent, respectively. Three-round sequences labeled NA are excluded from the denominator.
Attacker-Perspective Utility Score. Utility is evaluated over each complete three-round sequence. The evaluator assigns an integer score based on the outputs’ relevance to the commands, technical detail, and value for subsequent reconnaissance. This score is assigned directly and is not calculated from the authenticity or consistency scores.
Across the dataset, utility is aggregated as
where is the number of sequences assigned a utility score of , and is the total number of evaluated sequences. Thus, is the observed proportion of sequences with score , and is the arithmetic mean of the sequence-level utility scores.
Three-round sequences labeled NA are also included in utility evaluation.
Human Sampling Validation. To validate the reliability of the evaluator, we manually inspect a balanced sample of 416 multi-round reconnaissance interactions, covering 40.31% of all conversations. The sample is drawn from the Claude Sonnet-4.6 results, with 52 interactions selected from each of the eight reconnaissance categories. The validation was conducted by 2 graduate researchers, including 1 master’s students and 1 Ph.D. students, and we compare manual judgments with GPT-5.4-mini outputs using the agreement-check template in Appendix § 10. Table 1 summarizes the agreement rates: 90.87% for authenticity, 97.36% for consistency, and 91.35% for hacker-perspective utility. These results support agreement between human judgments and GPT-5.4-mini on the sampled Claude Sonnet-4.6 outputs.
Evaluation
We organize the evaluation around three research questions:
RQ1: What does jointly evaluating authenticity, cross-round consistency, and attacker-perspective utility reveal beyond the assessment of individual command outputs?
RQ2: What trade-offs emerge among Cowrie, closed-source LLMs, and open-source LLMs across the three evaluation metrics?
RQ3: How does system performance vary across reconnaissance categories in terms of authenticity, cross-round consistency, and attacker-perspective utility?
Experimental Setup
Benchmark construction, response generation, and scoring. We evaluate Cowrie [11], closed-source LLMs, and open-source LLMs on a multi-round Linux reconnaissance benchmark containing 1,032 interactions. The evaluation has three separate stages. During dataset construction, GPT-5.4 extends the manually designed first-round commands into three-round interactions, which are reviewed and paired with real-system reference outputs. Once the command set is fixed, running the benchmark does not require calling GPT-5.4 again. During response generation, the evaluated systems receive the fixed command sequences; later LLM responses also receive the preceding commands and generated outputs. Finally, GPT-5.4-mini scores the saved responses using the prompts in Appendix § 6. Reference outputs are used for authenticity scoring and are never supplied during response generation. Regenerating responses or rescoring them may still be affected by changes to commercial model services.
Prompt control. All evaluated LLMs receive the same system prompt, described in Appendix § 12. This holds the prompt text constant across models. The results characterize these models under the shared prompting setup and do not establish their best achievable performance under other prompts or state-management architectures.
Test subjects and runtime. We compare Cowrie with representative closed-source and open-source LLMs under the same benchmark. Closed-source LLMs are accessed through an OpenAI-compatible API pipeline, whereas open-source LLMs are executed locally. The closed-source LLMs evaluated in this work include Moonshot-v1-128k (Moonshot-v1), Doubao-1-5-lite-32k-250115 (Doubao-1.5), Gemini-2.5-flash-lite (Gemini-2.5), GPT-5-nano, Claude-sonnet-4-6 (Sonnet-4.6), GPT-5.1, Gemini-3-Flash, Qwen3-Max, DeepSeek-V3.2, and DeepSeek-R1. The open-source LLMs include Qwen2.5-7B-Instruct (Qwen2.5-7B), Llama-3.1-8B-Instruct (Llama3.1-8B), Mistral-7B-Instruct (Mistral-7B), Hunyuan-7B, Qwen3-0.6B/8B/14B with and without thinking mode, and SecGPT-1.5B/7B/14B. Detailed runtime environments and inference configurations are reported in Appendix § 12.
Automatic evaluation. We adopt GPT-5.4-mini to evaluate the multi-round outputs of each test subject from three perspectives. Authenticity for each round is determined by comparing the output of the test subject with the corresponding label output. Cross-round consistency is marked at the conversation level with values in { }, based on whether key entities including IP addresses, ports, interface names, service names, process identifiers, routes, and paths stay consistent across rounds. The utility evaluated from a hacker perspective is scored per interaction as an integer from 0 to 5, which reflects how informative and valuable the generated content is for actual hackers’ reconnaissance efforts.
RQ1: Value of Joint Evaluation
Table 2 shows why individual-output authenticity is insufficient to characterize a reconnaissance interaction. GPT-5.1 and Gemini-3-Flash share the highest average authenticity score, at 0.30, but their consistency scores are only 0.15 and 0.16. Sonnet-4.6 achieves the highest attacker utility, at 3.24, while its consistency is 0.11. These system-level results distinguish the realism and information value of generated outputs from their compatibility across a conversation.
| System | R1 | R2 | R3 | Avg. | Consistency Score | Hacker Utility Score | Rank |
| Traditional Honeypot | |||||||
| Cowrie | 0.21 | 0.15 | 0.15 | 0.17 | 0.45 | 0.66 | 17 |
| Closed-source LLM | |||||||
| Sonnet-4.6 | 0.36 | 0.33 | 0.19 | 0.29 | 0.11 | 3.24 | 1 |
| Doubao-1.5 | 0.26 | 0.21 | 0.09 | 0.19 | 0.08 | 2.70 | 13 |
| Gemini-2.5 | 0.31 | 0.24 | 0.09 | 0.21 | 0.10 | 2.88 | 8 |
| GPT-5-nano | 0.33 | 0.27 | 0.13 | 0.24 | 0.12 | 3.04 | 7 |
| Moonshot-v1 | 0.30 | 0.19 | 0.10 | 0.20 | 0.12 | 2.79 | 9 |
| GPT-5.1 | 0.35 | 0.33 | 0.22 | 0.30 | 0.15 | 2.95 | 3 |
| Gemini-3-Flash | 0.33 | 0.35 | 0.22 | 0.30 | 0.16 | 2.94 | 1 |
| Qwen3-Max | 0.34 | 0.32 | 0.16 | 0.27 | 0.14 | 2.93 | 5 |
| DeepSeek-V3.2 | 0.33 | 0.27 | 0.16 | 0.25 | 0.12 | 3.02 | 6 |
| DeepSeek-R1 | 0.35 | 0.27 | 0.17 | 0.26 | 0.13 | 3.12 | 4 |
| Open-source LLM | |||||||
| Hunyuan-7B | 0.01 | 0.05 | 0.03 | 0.03 | 0.12 | 1.18 | 22 |
| Llama3.1-8B | 0.20 | 0.15 | 0.05 | 0.13 | 0.09 | 2.83 | 15 |
| Mistral-7B | 0.18 | 0.13 | 0.06 | 0.12 | 0.07 | 2.56 | 19 |
| Qwen2.5-7B | 0.28 | 0.25 | 0.10 | 0.21 | 0.12 | 2.38 | 14 |
| Qwen3-0.6B (unthinking) | 0.04 | 0.05 | 0.03 | 0.04 | 0.19 | 0.72 | 23 |
| Qwen3-0.6B (thinking) | 0.04 | 0.04 | 0.01 | 0.03 | 0.10 | 0.77 | 24 |
| Qwen3-8B (unthinking) | 0.27 | 0.19 | 0.09 | 0.18 | 0.12 | 2.28 | 16 |
| Qwen3-8B (thinking) | 0.21 | 0.16 | 0.08 | 0.15 | 0.13 | 2.35 | 18 |
| Qwen3-14B (unthinking) | 0.26 | 0.23 | 0.10 | 0.20 | 0.13 | 2.64 | 10 |
| Qwen3-14B (thinking) | 0.27 | 0.25 | 0.12 | 0.21 | 0.13 | 2.51 | 11 |
| SecGPT-1.5B | 0.15 | 0.14 | 0.07 | 0.12 | 0.21 | 1.20 | 21 |
| SecGPT-7B | 0.19 | 0.15 | 0.08 | 0.14 | 0.13 | 1.90 | 20 |
| SecGPT-14B | 0.25 | 0.21 | 0.10 | 0.19 | 0.13 | 2.58 | 12 |
Table 2. Comprehensive evaluation results for Cowrie and LLM honeypots in multi-round Linux reconnaissance. Authenticity is reported for each round, with “Avg.” denoting the mean across three rounds. Consistency is the fraction of conversations scored as fully consistent. Hacker utility is the weighted average of conversation-level scores from 0 to 5. Rank is computed by first converting hacker utility from the 0 to 5 scale to the 0 to 1 scale by dividing it by 5, and then averaging the converted hacker utility score with authenticity average and consistency. Higher values indicate better performance; all metric values are rounded to two decimals.
Authenticity is generally lower in later rounds, but the pattern is not universal. Sonnet-4.6 decreases from 0.36 to 0.33 to 0.19, whereas Hunyuan-7B increases from 0.01 to 0.05 before decreasing to 0.03.
Finding 1. Joint evaluation reveals weaknesses in cross-round consistency that individual-output authenticity does not capture. Systems with the highest authenticity or utility scores can still perform poorly on consistency.
RQ2: Comparison Across Systems
Cowrie achieves the highest consistency score, at 0.45, compared with 0.07 to 0.21 across the evaluated LLM configurations. Cowrie’s constrained command support and responses may contribute to this difference.
The authenticity comparison varies by model. All evaluated closed-source models exceed Cowrie’s average authenticity of 0.17. Open-source results are more mixed: Qwen2.5-7B scores 0.21, while Llama3.1-8B and Hunyuan-7B score 0.13 and 0.03, respectively. Therefore, the results do not support a general claim that open-source LLMs outperform Cowrie on authenticity.
Every evaluated LLM configuration exceeds Cowrie on attacker utility. Cowrie scores 0.66, compared with mean scores of 1.99 and 2.96 across the evaluated open-source and closed-source configurations. This comparison concerns the reconnaissance information supplied by the outputs.
The paired Qwen3 results also show no consistent benefit from enabling thinking mode. Depending on model size, improvements in one metric accompany decreases or little change in another. Appendix § 13 reports these comparisons in detail.
We test all LLMs with the same prompt and do not include a baseline that explicitly stores and updates the simulated host’s state. The results therefore describe performance under this setup. They do not establish the best performance these models could achieve with different prompts or ways of maintaining that state.
Finding 2. The evaluated LLMs all provide higher reconnaissance utility than Cowrie, but authenticity gains depend on the model, and none matches Cowrie’s consistency. No evaluated system leads across all three metrics.
RQ3: Category-Level Performance
We examine category-level performance using the Claude Sonnet-4.6 results in Table 3, comparing its authenticity, cross-round consistency, and attacker-perspective utility across the eight reconnaissance categories. Consistency is highest for Connection-to-Process Mapping (0.30), followed by Port Connection States (0.20) and Open Ports & Listening Services (0.18). It is lower for Network Interfaces & IP (0.04), Routing Table (0.03), DNS Configuration (0.02), and ARP Cache & MAC Addresses (0.01).
| Category | R1 | R2 | R3 | Auth. Avg. | Consistency | Utility Avg. |
| Network Interfaces & IP | 0.32 | 0.26 | 0.16 | 0.25 | 0.04 | 3.17 |
| Open Ports & Listening Services | 0.49 | 0.36 | 0.22 | 0.36 | 0.18 | 3.40 |
| Port Connection States | 0.26 | 0.28 | 0.35 | 0.30 | 0.20 | 3.64 |
| Routing Table | 0.14 | 0.34 | 0.05 | 0.18 | 0.03 | 2.93 |
| Service Versions & Components | 0.60 | 0.32 | 0.16 | 0.36 | 0.08 | 3.35 |
| DNS Configuration | 0.31 | 0.19 | 0.12 | 0.21 | 0.02 | 2.98 |
| ARP Cache & MAC Addresses | 0.24 | 0.64 | 0.16 | 0.35 | 0.01 | 2.84 |
| Connection-to-Process Mapping | 0.55 | 0.26 | 0.26 | 0.36 | 0.30 | 3.57 |
Table 3. Category-level evaluation scores for Claude Sonnet-4.6. The 1,032 interactions are ordered as eight blocks of 129 interactions following Table 5. R1 to R3 report round-wise authenticity scores, “Auth. Avg.” reports the average authenticity across the three rounds, “Consistency” reports the fraction of conversations with consistency score 1, and “Utility Avg.” reports the weighted average hacker-utility score. All reported values are rounded to two decimal places.
Similar authenticity scores can accompany different consistency scores. Connection-to-Process Mapping, Open Ports & Listening Services, and Service Versions & Components all have average authenticity 0.36, but their consistency scores are 0.30, 0.18, and 0.08. Port Connection States has the highest utility, at 3.64, while Connection-to-Process Mapping has the highest consistency. ARP Cache & MAC Addresses further illustrates the separation between metrics, with average authenticity 0.35 and consistency 0.01.
Round-level authenticity is also nonmonotonic in several categories. Port Connection States increases from 0.26 to 0.28 to 0.35, while ARP Cache & MAC Addresses increases from 0.24 to 0.64 before falling to 0.16. These results identify differences among the selected tasks.
Finding 3. For Claude Sonnet-4.6, performance varies across reconnaissance categories and across the three metrics. Categories with similar authenticity can have substantially different cross-round consistency, and the category with the highest attacker utility is not the most consistent.
Conclusion
We present ReconPotBench, a benchmark for evaluating LLM honeypots in multi-round Linux reconnaissance. It contains 1,032 three-round interactions across eight command categories, with real-system reference outputs, and evaluates systems along authenticity, cross-round consistency, and hacker-perspective utility. ReconPotBench reveals weaknesses in cross-round consistency that individual-output authenticity does not capture. Across systems, every evaluated LLM configuration provides higher reconnaissance utility than Cowrie, but authenticity gains depend on the model and Cowrie achieves the highest consistency. Within Claude Sonnet-4.6, similar authenticity scores can accompany substantially different consistency across categories.
Limitations
ReconPotBench is designed to evaluate LLM honeypots in multi-round Linux reconnaissance. Its three-round structure allows us to collect reliable real-system references and compare different systems under a reproducible setting. However, the current benchmark focuses on single-goal reconnaissance chains within Linux terminal environments, and does not yet cover cross-goal reconnaissance behavior or other operating systems and shell environments. Future work can extend ReconPotBench to broader platforms and more complex multi-stage reconnaissance workflows.
Ethical Considerations
All data in ReconPotBench is collected in a controlled local environment. Commands are executed only on our own physical Linux host, and we do not probe third-party websites, domains, or servers. Thus, our data collection does not affect the security or privacy of external systems or individuals. ReconPotBench may have dual-use implications, since methods for evaluating honeypots could be misused by attackers to better recognize or avoid honeypots. We release the benchmark only for academic and defensive research, with the goal of supporting reproducible evaluation and improving the robustness of LLM-based honeypots.
Acknowledgments
This work was supported in part by the Science and Technology Development Fund of Macau SAR (Grant Nos. FDCT/0007/2024/AKP, EF2024-00185-FST), the UM and UMDF (Grant Nos. MYRG-GRG2025-00236-FST, MYRG-GRG2026-00018-FIC, MYRG-CRG2026-00007-FIC), the Dr. Stanley Ho Medical Development Foundation (Grant No. SHMDF-AI/2026/001), and the National Natural Science Foundation of China (Grant No. 62266013).
Prompt Paradigm
Prompt for Authenticity Judge.

Figure 1 (PDF p. 10).
For each evaluation instance, consider only the current round and determine whether the LLM-generated output is similar to the corresponding real system output for the same Linux reconnaissance command.
Input. input_cmd: the Linux reconnaissance command at the current round. test_output: the output generated by the LLM for this command. system_output: the real system output for the same command, used as the reference output.
Evaluation Criterion. Judge whether test_output aligns with system_output in terms of the semantics, the type of returned information, and the overall form of the terminal output for input_cmd.
Scoring. Output 1 if test_output aligns with system_output. Output 0 if test_output does not align with system_output.
Output Requirement. Return exactly one character: 1 or 0. Do not output anything else.
Prompt for Hacker-Perspective Judge.
Prompt for Hacker Utility Evaluation
Role. You are an evaluator of multi-round Linux reconnaissance conversations from a hacker-perspective utility standpoint.
Task. Your task is to determine whether the conversation outputs, taken together, provide reconnaissance information that is useful and meaningful from a hacker’s perspective.
Input. The input unit is one complete “conversation” from XXX file, which contains 3 rounds. For each round, represent the conversation as a command-output pair in the following format:
contents: the concatenated string of all prior rounds up to the current command, ending with the current command
output: the corresponding output for that current round
Specifically, the 3 rounds should be interpreted as:
Round 1.
contents = Round 1 command
output = Round 1 output
Round 2.
contents = Round 1 command + Round 1 output + Round 2 command
output = Round 2 output
Round 3.
contents = Round 1 command + Round 1 output + Round 2 command + Round 2 output + Round 3 command
output = Round 3 output
You must evaluate the full 3-round conversation as one unit, based on the progression across these command-output pairs.
Evaluation Criteria. Assign an integer score from 0 to 5 according to the following standard:
0: The outputs are entirely useless from a hacker’s perspective. They are empty, irrelevant, obviously mismatched to the commands, or provide no meaningful reconnaissance value.
1: The outputs contain only minimal or fragmentary reconnaissance value. Most information is vague, generic, poorly matched to the commands, or not useful for guiding further investigation.
2: The outputs provide limited reconnaissance value. Some command-related information is present, but it is shallow, incomplete, weakly informative, or insufficient to meaningfully support further reconnaissance.
3: The outputs provide moderate reconnaissance value. They are generally relevant to the commands and contain some useful technical details, but the information is not consistently specific, well-connected, or strong enough to form a highly useful reconnaissance chain.
4: The outputs provide strong reconnaissance value. They are relevant to the commands, contain substantial attacker-relevant technical details, and show a mostly meaningful progression across the three rounds. However, at least one noticeable limitation remains, such as incomplete specificity, weaker progression between some rounds, partial genericity, or missing details that would prevent the conversation from being maximally useful for guiding the next reconnaissance step.
5: The outputs provide very strong reconnaissance value. They are consistently relevant to the commands, rich in concrete attacker-relevant technical details, and form a clear and coherent reconnaissance chain across all three rounds. The outputs not only reveal useful information, but also provide enough specific, well-connected, and operationally actionable detail to directly support plausible next-step attacker decision-making, with no major weakness in relevance, specificity, or progression.
Output Requirement. Return only one integer: 0, 1, 2, 3, 4, or 5.
Prompt for Consistency Judge.
Prompt for Consistency Evaluation
Role. You are an evaluator of multi-round Linux reconnaissance dialogues from a cross-round consistency standpoint.
Task. Your task is to determine whether the conversation maintains full contextual consistency across rounds when entities are present.
Input. The input unit is one complete “conversation” from XXX file, which contains 3 rounds. For each round, represent the conversation as a command-output pair in the following format:
contents: the concatenated string of all prior rounds up to the current command, ending with the current command
output: the corresponding output for that current round
Specifically, the 3 rounds should be interpreted as:
Round 1.
contents = Round 1 command
output = Round 1 output
Round 2.
contents = Round 1 command + Round 1 output + Round 2 command
output = Round 2 output
Round 3.
contents = Round 1 command + Round 1 output + Round 2 command + Round 2 output + Round 3 command
output = Round 3 output
You must evaluate the full 3-round conversation as one unit, based on the progression across these command-output pairs.
Evaluation Criteria. Judge the dialogue according to the following criteria:
First determine whether the multi-round interaction contains explicit entities, such as IP addresses, ports, interface names, service names, process identifiers, routes, paths, or device names.
If no such entities appear anywhere in the conversation, do not score this conversation for consistency.
If entities do appear, check whether they are used consistently across all rounds.
Entity consistency means that the same entity, once introduced, should not be changed, contradicted, mismatched, or replaced without justification in later rounds.
Cross-round context must remain fully aligned between commands and outputs. Any inconsistency in entity identity, value, reference, or contextual linkage counts as inconsistency.
Judge only contextual consistency across rounds.
Scoring.
Output 1 if the conversation contains entities and the context is fully consistent across rounds.
Output 0 if the conversation contains entities but the context is inconsistent across rounds.
Output NA if the conversation does not contain any entities and therefore should be excluded from the average consistency calculation.
Output Requirement. Return only a single value: 1, 0, or NA.
Prompt for Round 2/3 Command Generation.
| Round 2/3 Command Prompt |
| Role: You are a Linux reconnaissance command generation assistant. |
| Task: Given a user-provided Round 1 command for network-layer reconnaissance, generate exactly two follow-up commands: one for Round 2 software-layer reconnaissance and one for Round 3 hardware-layer reconnaissance. The three rounds must strictly follow the progression Network → Software → Hardware. |
Table 1 (PDF p. 12).
the progression Network → Software → Hardware.
Rules: First infer which of the following eight reconnaissance scenarios the Round 1 command belongs to: Network Interfaces and IP, Open Ports and Listening Services, Network Connections and Port States, Routing Table, Service Versions and Components, DNS Configuration, ARP Cache and MAC Addresses, or Connection-to-Process Mapping. Then generate Round 2 and Round 3 so that both remain within the same scenario and preserve logical dependence across rounds. The Round 2 command must target software information motivated by Round 1, and the Round 3 command must target hardware information motivated by Round 2. The generated commands must be valid, commonly used, non-destructive Linux commands. When applicable, context-dependent entities such as IP addresses, ports, interface names, service names, process identifiers, routes, paths, and device names should remain consistent across rounds. If the input does not contain a concrete entity value, generate a general but still scenario-consistent follow-up command. Do not modify or repeat the user-provided Round 1 command.
Input: first_command: {user_provided_round1_command}
Output: Exactly two lines only. Line 1 is the Round 2 command. Line 2 is the Round 3 command. Do not output any explanation, label, numbering, or additional text.
Examples:
Example 1:
Input: the command does inspect network interfaces, IP addresses, and host-side addressing configuration.
Output: Software layer: the command does inspect software or configuration responsible for interface management and host network settings.
Hardware layer: the command does inspect the network interface hardware that supports those interfaces and addresses.
Example 2:
Input: the command does inspect open ports and listening services exposed by the host.
Output: Software layer: the command does inspect the service software or service configuration corresponding to those listening ports.
Hardware layer: the command does inspect the hardware resources or network devices supporting those exposed services.
Example 3:
Input: the command does inspect active connections, connection states, and current communication status.
Output: Software layer: the command does inspect the software processes or service-side software state associated with those active connections.
Hardware layer: the command does inspect the hardware resources supporting the current connection workload.
Example 4:
Input: the command does inspect routing-table information and forwarding paths used for network reachability.
Output: Software layer: the command does inspect routing-related or firewall-related software and configuration. Hardware layer: the command does inspect the network hardware supporting packet forwarding or route-dependent communication.
Example 5:
Input: the command does inspect service versions and exposed service components.
Output: Software layer: the command does inspect installed packages and configuration associated with the identified software version. Hardware layer: the command does inspect platform or hardware characteristics that constrain software compatibility or deployment.
Example 6:
Input: the command does inspect DNS configuration and resolver-related network settings.
Output: Software layer: the command does inspect DNS-related software, resolver configuration, or name-service components. Hardware layer: the command does inspect the hardware resources supporting DNS-related operation.
Example 7:
Input: the command does inspect ARP cache entries, MAC addresses, and neighbor information at the link layer.
Output: Software layer: the command does inspect software or configuration related to neighbor discovery or local network interface management. Hardware layer: the command does inspect the network interface hardware involved in link-layer communication.
Example 8:
Input: the command does inspect the mapping between network connections and processes.
Output: Software layer: the command does inspect the binaries, libraries, configuration, or process-level software components associated with the identified process or service. Hardware layer: the command does inspect the hardware resources supporting that process, such as CPU, memory, or relevant devices.
Human Agreement Check
Template for Human Agreement Check
Task. For each sampled validation item, independently assign a human judgment using the same scoring rule as the corresponding evaluator prompt. Then compare the human judgment with the GPT-5.4-mini judgment.
Input. metric_type: authenticity, consistency, or hacker-perspective utility.
sample_content: the complete three-round interaction to be judged.
reference_output: the real-system output, used only for authenticity judgments.
gpt_judge_output: the score produced by GPT-5.4-mini.
Agreement Rule.
For authenticity, mark agreement if the human label and GPT-5.4-mini label are both 1 or both 0.
For consistency, mark agreement if the human label and GPT-5.4-mini label are both 1, both 0, or both NA.
For hacker utility, mark agreement if the human score and GPT-5.4-mini score are the same integer from 0 to 5.
Concrete Command Example
Example: A fixed three-round reconnaissance interaction.
This case from the released dataset examines the same network interface, wlan0, through its network state, software configuration, and underlying device information. The outputs below are collected from the reference host.
Round 1: Network state and addresses.
Command:
ip -br addr show dev wlan0Reference output:
wlan0 UP 192.168.0.100/24 fe80::ca35:41ed:8341:c564/64The output identifies wlan0 as an active interface with an IPv4 address and an IPv6 link-local address. It establishes the interface and its network state for this case.
Round 2: Software configuration of the same interface.
Command:
sysctl -a 2>/dev/null | grep -E '^net\.(ipv4|ipv6)\.(conf|neigh)\.wlan0(\.|$)' | sed -n '1,5p'Reference output:
net.ipv4.conf.wlan0.accept_local = 0
net.ipv4.conf.wlan0.accept_redirects = 1
net.ipv4.conf.wlan0.accept_source_route = 0
net.ipv4.conf.wlan0.arp_accept = 0
net.ipv4.conf.wlan0.arp_announce = 0This round adds kernel network settings for the interface. Every returned entry belongs to net.ipv4.conf.wlan0, so the configuration must refer to the same interface examined in Round 1. These values provide configuration information; they do not, by themselves, establish a vulnerability.
Round 3: Device context of the interface.
Command:
ethtool -i wlan0 2>/dev/null | sed -n '1,5p'Reference output:
driver: iwlwifi
version: 6.12.25-amd64
firmware-version: 89.4d42c933.0 so-a0-hr-b0-89.uc
expansion-rom-version:
bus-info: 0000:00:14.3The final round adds the driver, firmware, and bus location, connecting the logical interface to its underlying device context. This is the hardware-related information examined in this case.
Experimental Details
All LLM honeypots are evaluated with the same system prompt. The prompt instructs the LLM to act as a high-interaction honeypot agent operating as a realistic Kali Linux system. Given a reconnaissance command, the LLM is required to return terminal-style output that follows standard Linux conventions, reflects a believable Kali runtime environment, and provides valuable reconnaissance information. The prompt also requires the LLM to preserve details such as host configuration, network information, running services, file-system state, and active processes, and other relevant system artifacts across interaction rounds, so that the generated outputs support a continuous and informative reconnaissance workflow.
Table 4 reports the runtime environment and key inference configurations for open-source LLM generation.
| Configuration | Value |
| Inference framework | transformers + PyTorch |
| Sampling strategy | do_sample=False |
| Maximum generation length | max_new_tokens=1024 |
| GPU | NVIDIA A100-PCIE-40GB |
| CPU | 10 Xeon vCPUs |
| Operating system | Ubuntu 22.04 |
| Python version | Python 3.12 |
| PyTorch version | PyTorch 2.5.1 |
| CUDA version | CUDA 12.4 |
Table 4. Runtime environment and inference settings for open-source LLM generation.
Thinking Mode Analysis
We further examine whether enabling thinking mode improves multi-round honeypot performance within the Qwen3 family. The paired results show no consistent advantage. For Qwen3-0.6B, thinking mode slightly improves hacker-perspective utility from 0.72 to 0.77, but average authenticity decreases from 0.04 to 0.03, and consistency drops more clearly from 0.19 to 0.10. For Qwen3-8B, thinking mode reduces average authenticity from 0.18 to 0.15, while consistency increases only marginally from 0.12 to 0.13 and hacker-perspective utility from 2.28 to 2.35. For Qwen3-14B, thinking mode brings only a small authenticity gain, from 0.20 to 0.21, leaves consistency unchanged at 0.13, and lowers hacker-perspective utility from 2.64 to 2.51.
These results show no consistent benefit from enabling thinking mode under the tested settings. They do not isolate whether reasoning behavior, state management, or another factor accounts for the observed differences.
Analysis of Reconnaissance Categories
All eight scenarios are grounded in relevant reconnaissance references. Specifically, Network Interfaces & IP is supported by [1, 2]; Open Ports & Listening Services is supported by [1] (Marceiano 2024); Port Connection States corresponds to network connections and port-state reconnaissance in [1]; Routing Table is supported by [2]; Service Versions corresponds to port-service version reconnaissance in [2] (Marceiano 2024); DNS Configuration corresponds to DNS reconnaissance in [2]; ARP Cache & MAC Addresses corresponds to ARP reconnaissance in [3]; and Connection-to-Process Mapping corresponds to network connections and their associated processes in [10].
Table 5 lists all eight categories with representative commands. Table 3 reports the category-level scores of Claude Sonnet-4.6 across the eight reconnaissance command families, including round-wise authenticity, average authenticity, cross-round consistency, and hacker-perspective utility.
References
- [1]Andy. 2020. Reconnaissance for beginners (2). Accessed: 2026-02-09.
- [2]Justice Kwame Appati. 2025. Network reconnaissance and packet manipulation manual. Accessed: 2026-02-09.
- [3]Ar-Jay Ballaco. 2025. Beginner’s guide to active reconnaissance (part 3.2): Arp scan techniques on fire-walled hosts. Accessed: 2026-02-09.
- [4]Chongqi Guan, Guohong Cao, and Sencun Zhu. 2024. Honeymll: Enabling shell honeypots with large language models. In 2024 IEEE Conference on Communications and Network Security (CNS), pages 1–9.DOI
- [5]Mateusz Kazimierczak, Nuzaira Habib, Jonathan H. Chan, and Thanyathorn Thanapatheerakul. 2024. Impact of ai on the cyber kill chain: A systematic review. Heliyon, 10(24):e40699.DOI
- [6]Pranjay Malhotra. 2025. Llmhoney: A real-time ssh honeypot with large language model-driven dynamic response generation. Preprint, arXiv:2509.01463.arxiv.org/abs/2509.01463
- [8]Forrest McKee and David Noever. 2023. Chatbots in a honeypot world. Preprint, arXiv:2301.03771.arxiv.org/abs/2301.03771
- [9]MITRE ATT&CK. 2025a. Reconnaissance. Accessed: 2026-02-09.
- [10]MITRE ATT&CK. 2025b. System network connections discovery. Accessed: 2026-02-09.
- [11]Michel Oosterhof and Cowrie contributors. 2026. cowrie/cowrie: Cowrie ssh/telnet honeypot. https://github.com/cowrie/cowrie. GitHub repository, accessed March 4, 2026.
- [12]Hakan T. Otal and M. Abdullah Canbaz. 2024. Llm honeypot: Leveraging large language models as advanced interactive honeypot systems. In 2024 IEEE Conference on Communications and Network Security (CNS), page 1–6. IEEE.DOI
- [13]William Shotts. 2019. The Linux Command Line, 2nd Edition: A Complete Introduction, 2nd edition. No Starch Press, USA.
- [14]Muris Sladić, Veronica Valeros, Carlos Catania, and Sebastian Garcia. 2024. Llm in the shell: Generative honeypots. In 2024 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), pages 430–435.DOI
- [15]Ziyang Wang, Jianzhou You, Haining Wang, Tianwei Yuan, Shichao Lv, Yang Wang, and Limin Sun. 2025. Honeypgt: Breaking the trilemma in terminal honeypots with large language model. Preprint, arXiv:2406.01882.arxiv.org/abs/2406.01882
- [16]Senesh Wijayarathne. 2024. The stages of a cyber attack (cyber kill chain). Accessed: 2026-02-09.
- [17]Guan Yang, Zhengzheng Sun, and Yu Wang. 2025. Shellbox: Adversarially enhanced llm-interactive honeypot framework. IEEE Access, 13:143618–143630.DOI

