WorksComputer Science
Detecting and Mitigating Query-Based System Prompt Leakage Attacks
Abstract
Large Language Models (LLMs) rely on proprietary system prompts to enforce task directives, safety policies, and access controls. These prompts are vulnerable to query-based leakage attacks, where black-box adversaries extract confidential instructions through adaptive, multi-turn interactions. Existing defenses often assess queries independently, missing malicious behavior that emerges across interactions. We present AdvGuard, a model-agnostic framework for detecting and mitigating system prompt leakage. AdvGuard models cross-query relationships using a bounded semantic interaction graph, statistically filters candidate components, and converts them into line graphs for graph convolutional network (GCN) classification. This approach captures multi-hop patterns in query similarity that distinguish coordinated extraction from benign workflows. A lightweight prompt-level detector provides protection before sufficient interaction history accumulates, and the framework supports black-box LLM deployments. In mixed-query evaluations, AdvGuard improves recall by up to 36% over the strongest baseline while maintaining a false positive rate of at most 3.4%. In multi-turn workloads, it achieves 97.23%-99.1% F1 with 100% recall. Leave-one-attack-family-out tests yield 99.98% mean recall and 99.99% mean F1, with no observed false positives. Integrated into the adaptive PLeak and LeakAgent attack loops, AdvGuard limits exact-match leakage to 0.00-0.01 and substring-match leakage to 0.01-0.09 across evaluated victim models. These results demonstrate robust detection of unseen attack families and substantially reduced prompt reconstruction.
Introduction
Large Language Models (LLMs) are increasingly deployed in applications ranging from customer support and code generation to decision-support tools and autonomous agents [1, 3] [2]. In API-based and black-box deployments, developers often rely on hidden system prompts to encode application logic, enforce safety policies, and define tool-use protocols [4, 5].
Because these prompts can contain proprietary instructions and security-sensitive control logic, their exposure—known as prompt leakage—can reveal implementation details, weaken safety controls, and enable targeted follow-on attacks.
Attackers can elicit system-prompt content through explicit requests or more elaborate extraction strategies. Existing attacks use iterative probing [6, 7], adaptive reformulation [8, 9], encoding tricks [10, 11, 12], and multi-turn manipulation [13]. Although some attempts are overt and detectable from a single prompt, others distribute the extraction objective across multiple semantically related queries. In these cases, no individual query provides decisive evidence, while the accumulated interaction reveals repeated probing, reformulation, or contextual scaffolding.
Many existing defenses make decisions on individual inputs or outputs [14, 15, 16]. Such defenses can identify explicit extraction attempts, but they have limited visibility into attacks whose evidence emerges only across related interactions. Our key observation is not that every prompt-leakage attack requires multiple turns, but that iterative extraction can leave a distinctive cross-query relational structure. Repeated semantic reuse and coordinated reformulation may therefore provide a complementary signal when individual-query evidence is insufficient.
To capture this signal, we present AdvGuard, a lightweight, model-agnostic framework that operates at the input boundary of a black-box LLM service. AdvGuard first applies a prompt-level detector to identify explicit extraction attempts before interaction history is available. It then maintains a bounded semantic interaction graph that records similarity relationships among queries. Components with unusually high accumulated similarity are selected for further analysis and transformed into line graphs, allowing a graph convolutional network to classify their relational topology and edge strengths. This multi-stage design combines per-query and interaction-level evidence without requiring access to the protected system prompt, generated responses, or model internals.
We evaluate AdvGuard under controlled mixed-query traffic, benign multi-turn conversations, and response-adaptive attack loops. Across four prompt-leakage attack families and five benign workloads, AdvGuard achieves 91.14–100% recall while maintaining FPR at or below 3.4%. Under benign multi-turn conversational workloads, AdvGuard achieves 100% recall with FPR between 1.34% and 3.42%. When inserted into the response-adaptive PLeak and LeakAgent attack loops, AdvGuard limits exact-match leakage to 0.00–0.01 and substring-match leakage to 0.01–0.09 across the evaluated victim models. In our runtime evaluation, AdvGuard processes queries with an average latency of 32.96 ms.
In summary, this paper makes the following contributions:
We formulate iterative prompt leakage as an interaction-level detection problem and study accumulated cross-query structure as a signal that complements individual-prompt analysis.
We design AdvGuard, a lightweight black-box-compatible framework that combines prompt-level detection, bounded semantic interaction graphs, component-level candidate filtering, and line-graph classification.
We evaluate AdvGuard across multiple attack families, benign workloads, victim models, and response-adaptive attack loops, measuring detection effectiveness, leakage mitigation, generalization, and runtime overhead.
Background and Related Work
System Prompts and Prompt Leakage
LLM applications assemble model context from system instructions, conversation history, user messages, retrieved content, and tool outputs. The system prompt contains developer instructions that define the model’s role, safety policies, output format, and tool-use constraints. These instructions are typically hidden and may encode proprietary or security-sensitive application logic [4]. We treat their protected portions as sensitive assets. We use system-prompt leakage to mean unauthorized reconstruction of protected system instructions from model outputs. Leakage may be verbatim, near-verbatim, or semantically equivalent, and may follow one query or several interactions [13], [17].
Threat terminology. Prompt injection, jailbreaking, system-prompt extraction, and privacy leakage are related but distinct objectives. Prompt injection attempts to override application instructions, whereas jailbreaking attempts to bypass behavioral or safety restrictions. System-prompt extraction specifically targets hidden system instructions, while broader privacy leakage may target training data or personally identifiable information. Injection or jailbreak techniques can facilitate extraction, but success on one objective does not establish success on another.
System-Prompt Leakage Attacks
System-prompt extraction ranges from handcrafted requests to automated search. Perez and Ribeiro study prompt leaking as an objective of prompt injection [11], while Zhang et al. systematically evaluate whether outputs recover protected prompts [17]. PLeak searches for transferable extraction queries using shadow models [7], whereas RepeatLeakage exploits repetition behavior to elicit hidden context [18]. Raccoon benchmarks 14 categories of extraction strategies [19].
LeakAgent trains a reinforcement-learning red-teaming agent for privacy-leakage objectives that include system-prompt extraction [9]. Multi-turn settings additionally allow attackers to interpret prior outcomes and revise later queries [13]. Such interactions can accumulate evidence that is weak in isolation, although direct attacks may succeed in one turn and iterative attacks need not remain semantically similar.
Adjacent methods target different objectives: AutoDAN generates stealthy jailbreak prompts [20], while PromptFuzz uses coverage-guided mutation to test prompt injection [21]. Their mechanisms are relevant to detector evaluation, but neither method is specifically a system-prompt-reconstruction attack.
Defenses and Remaining Gap
Defenses differ in their signals and access requirements. Input-side defenses, including PromptGuard and LLM Guard, classify individual requests for injection, jailbreaks, or malicious content [14], [16]. Proxy Barrier blocks requests when a proxy LLM fails to reproduce the input as instructed [22]. These methods avoid modifying target-model weights, but primarily decide from the current request.
Output- and generation-aware defenses use evidence closer to disclosure. Output filters suppress responses overlapping protected instructions [23]. PromptKeeper tests response likelihoods and regenerates flagged outputs using a dummy prompt [24]. These methods require some combination of response access, protected content, generation statistics, or control over generation.
Prompt-hardening defenses alter the protected representation. Sandwich Defense reinforces trusted instructions around user content [25], Prompt Obfuscation transforms protected prompts [26], and ProxyPrompt substitutes an optimized proxy [27]. SysVec encodes instructions as internal vectors and therefore requires model-level access [28]. These methods directly reduce extractability but make different access and utility tradeoffs.
Recent work reasons over interaction history, primarily for jailbreak and privacy-risk detection. LeakScaler derives topical usage maps from historical interactions to identify jailbreak or PII-leakage anomalies [29]. G-Guard applies a GNN to an entity graph over multi-turn queries, harmful keywords, and a retrieved labeled query [30]. TRACE trains a model to reason over conversation trajectories and choose among allow, caution, and decline actions [31]. These systems establish the value of interaction context, but target harmful intent or general privacy risk rather than reconstruction of hidden system instructions.
Most system-prompt-leakage defenses therefore act on individual inputs, outputs, or protected representations. Interaction-aware defenses address adjacent jailbreak problems, leaving bounded, input-only monitoring of coordinated prompt extraction comparatively underexplored. AdvGuard targets this setting without access to the protected prompt, target-model internals, or generated responses.
Threat Model
Protected Asset and Security Objective. We consider an LLM application whose system prompt contains confidential instructions, policies, or proprietary control logic. The attacker’s primary objective is to recover all or part of this prompt from the model’s textual responses. A disclosure may occur in one response or be reconstructed by combining information obtained across multiple interactions. We therefore evaluate both direct overlap with the protected prompt and approximate reconstruction, as defined by the leakage metrics in § 10. The defender’s objective is to identify and block prompt-extraction behavior before a risky query reaches the target model while preserving access for benign users. Our claims are empirical: reducing measured leakage does not establish formal confidentiality or prove that no prompt information was disclosed.
Attacker Capabilities. We consider an adversary as a remote client with black-box query-and-response access to the deployed LLM application. The attacker may submit arbitrary syntactically valid queries, choose their content and timing, issue multiple queries within one logical session, and interleave extraction attempts with benign requests. We consider an adaptive attacker who may refine subsequent queries using the outcomes of previous interactions with the defended target, including model responses, denial messages, and unsuccessful extraction attempts. The attacker has no access to the target model’s architecture, weights, gradients, hidden states, training data, system prompts, or privileged side channels. The attacker also cannot modify the runtime or bypass the application gateway.
Defender Visibility and Capabilities. AdvGuard is deployed at the application boundary and observes each user query before it is forwarded to the target model. It receives the ordered query stream and a session identifier, and it may maintain bounded state for that session. Its detection decisions use query-side information only. In particular, AdvGuard does not inspect model responses, the protected system prompt, model internals, or training data. When a query or an evolving session is judged risky, the enforcement point may block the request and return a standardized denial message. Model responses are available to the evaluator only after an experiment so that leakage can be measured. Because the runtime detector does not observe those responses or the protected prompt, it detects behavior associated with extraction rather than determining whether leakage has already occurred.
Trust boundary and assumptions. The trusted computing base comprises the application gateway, session attribution and ordering, AdvGuard’s code and state, the enforcement path, and system-prompt storage and delivery. We assume that every in-scope request passes through AdvGuard before inference, queries from one interaction remain in the same
ordered session, and the attacker cannot bypass or tamper with the monitor. The target LLM is treated as potentially leakage-prone and is not relied upon to enforce prompt confidentiality. Because AdvGuard maintains finite per-session state, its behavioral evidence is limited to the configured observation window.
We exclude compromised infrastructure, white-box and training-time attacks, leakage through model internals or side channels, and collusion across accounts or independently identified sessions. We also exclude attacks delivered only through retrieved documents, tools, or other channels not visible in the monitored query stream. AdvGuard targets system-prompt extraction, not general jailbreak detection, harmful-content moderation, or arbitrary policy enforcement.
System Design
Design Goals
AdvGuard is a query-side defense for black-box LLM applications. Its design has three goals as follows.
Early-stage intervention. The system must detect explicit extraction attempts early before substantial leakage occurs. This requires a prompt-level risk signal for each incoming query that captures direct requests to reveal system instructions, instruction overrides, role confusion, repetition traps, and other leakage-oriented probes without relying solely on keyword matching.
Adaptation to mixed, multi-turn interactions. The system must remain effective when adversarial requests are interleaved with benign user activity. Since prompt leakage often unfolds across multiple turns, the system must model cross-turn relationships rather than treating each request in isolation. This enables detection of coordinated extraction patterns that may appear benign at the single-query level.
Low false-positive detection. The system must distinguish adversarial extraction chains from benign iterative
interactions, such as clarification, rewriting, or topic-focused follow-up questions. Since simple similarity aggregation can over-flag legitimate multi-turn behavior, the system must use structure-aware classification to preserve precision while maintaining strong recall against coordinated leakage attempts.
To meet these goals, AdvGuard combines three complementary detection stages. Figure 1 presents the overall design. First, the prompt-level detector evaluates each incoming query in isolation and can block explicit extraction attempts immediately. Second, queries that pass this stage update a session-level similarity graph, where statistical filtering identifies components exhibiting unusually strong accumulated similarity. Third, candidate components are transformed into line graphs and passed to the GCN classifier, which makes the final structural classification. The following subsections describe each stage, while § 4-E explains how they are orchestrated during online detection and how AdvGuard maintains bounded state.

Figure 1. AdvGuard’s three-stage per-query detection pipeline. The prompt-level detector can block an individual query immediately. Queries that pass this stage update the session graph. Components with sufficient history that pass the statistical filter are classified by the GCN. All allow-or-block decisions are made before model inference.
Prompt-level Attack Detection
To detect prompt extraction attempts at early stages, AdvGuard introduces a prompt-level attack detector that evaluates each query in isolation. This stage provides a fast and lightweight detection and is expected to be effective for naive prompt extraction attempts.
Prompt as a Signal. Based on our preliminary findings, prompt leakage attacks usually begin with short, targeted probes that explicitly or implicitly attempt to extract system instructions or privileged behaviors. Figure 2 shows examples of explicit and implicit prompt-level extraction attempts. Explicit extraction attempts are usually direct and use phrases that ask for the system prompt, rules, or internal instructions. They use phrases (e.g., “print / show / reveal / output” the system prompt / instructions / full context / hidden rules) and commands (e.g., “ignore previous instructions” or “enter developer mode”).
Figure 2. Explicit and Implicit Prompt Extraction Attempt Examples.
Implicit extraction attempts try to extract through role confusion, repetition traps, or templates.
Such attempts force repetition of a prompt-containing template (e.g., [Here is my system prompt: ”<insert full system prompt here>”]), and ask to “complete” or “echo” the instructions. Their extraction may also contain meta-requests to explain rules/behavior in ways that require quoting internal guidance. Both explicit and implicit extraction attempts show that even a single prompt can act as a strong standalone indicator of extraction behavior, particularly in the early stages of an attack.
Prompt-based Detector. With a prompt as a signal, AdvGuard’s early-stage detection employs a prompt leakage classifier for each incoming query to check whether the query exhibits prompt extraction behavior. The classifier is based on a fine-tuned, off-the-shelf open source model microsoft/deberta-v3-base [32], which has been trained on diverse prompt leakage datasets and captures a broad range of known leakage behaviors, including instruction-extraction attempts, system role confusion, meta-level probing, and variants of leakage templates. The model outputs a probability that is interpreted as the risk score , representing the likelihood that the query exhibits prompt extraction behavior. Higher scores indicate greater extraction risk.
This approach is efficient and effective in detecting naive extraction attempts in a single turn. However, it has limitations against adaptive multi-turn attacks, which we address next.
Structural Attack Detection
Prompt extraction can be adaptive and multi-turn, with each individual query appearing low-risk in isolation. For instance, asking about “guidelines” is not inherently malicious, and requesting “precise wording” can be interpreted as a benign clarification, as illustrated in Figure 1. However, when viewed relationally across turns, the query sequence reveals a persistent and coherent objective: to extract internal instructions verbatim. We refer to this class of behavior as a structural extraction attack.
This motivates AdvGuard to capture this cumulative structural extraction, enabling detection of stealthy, low-rate attacks that evade prompt-level detectors.
Similarity Graph. To capture cross-query relationships, AdvGuard models user queries as a similarity graph that captures semantic relationships between prompts over time. At time step , the interaction history is represented as a weighted, undirected graph , where denotes the set of queries observed up to time , represents the set of edges connecting semantically related queries, and stores the corresponding edge weights that quantify semantic similarity. For each query , we compute an embedding using a lightweight sentence encoder such as all-MiniLM-L6-v2. The semantic similarity between two queries and is then measured using cosine similarity:
Higher similarity scores indicate stronger semantic overlap, whereas lower scores correspond to semantically unrelated prompts.
Upon arrival of a new query , the system computes its embedding and identifies the most semantically similar historical query from .
We use a nearest-neighbor construction rather than top- connectivity to avoid spurious dense components and reduce latency. Appendix B-A validates this design choice.
An undirected edge is added if the similarity score exceeds a predefined threshold , which is calibrated offline, Appendix C-B reports choice. The corresponding edge weight is assigned using the computed similarity score . Otherwise, the query remains isolated in the graph. This incremental construction process enables AdvGuard to capture semantic continuity and progressively refined probing behavior across multi-turn interactions.
Baseline Initialization and Threshold Calibration. Starting graph construction from an empty state can cause early benign queries to form spurious connections due to incidental semantic overlap. To mitigate this effect, AdvGuard performs a one-time baseline initialization prior to processing live query streams.
To initialize the benign baseline, a fixed set of benign queries is inserted into the graph as isolated nodes. These queries are selected to be semantically diverse and serve as a reference corpus for modeling normal interaction behavior. We assume that benign queries dominate typical user queries, so we use this corpus to calibrate the similarity threshold defined earlier.
Specifically, for each benign query , we compute and rank its cosine similarity to all other benign queries , . Let
denote the similarity to its second nearest neighbor. We use the second nearest neighbor, rather than the first, to avoid self-matching effects and to capture a representative notion of local semantic distribution. Appendix B-A studies the sensitivity of detection performance to this threshold choice.
The threshold is then derived from the distribution of these second-nearest-neighbor similarities, ensuring that benign queries typically remain weakly connected or isolated in the graph [33].
As a result, benign interactions typically produce isolated nodes. In contrast, iterative prompt-extraction attacks repeatedly reformulate. As a result, these queries are more likely to exceed the similarity threshold and connect to previous attack queries. This produces a progressively growing, tree-structured component with consistently high edge-similarity weights. AdvGuard therefore distinguishes the two behaviors using component growth and accumulated semantic similarity.
This calibration establishes a clear separation: benign interactions are expected to yield sparse, weakly connected structures, while coordinated extraction behavior produces an evolving graph component. As such, the similarity graph provides a basic foundation for detecting progressive and coordinated prompt leakage.
Structural Score as Signal. Building on the calibrated similarity graph, AdvGuard computes structural scores over non-isolated connected components of query nodes. A connected component is a maximal set of query nodes connected through similarity edges. Intuitively, the structural score aims to differentiate benign queries from adversarial interactions, which form clusters of semantically related queries through repeated refinement. Let denote the set of connected components in the graph , where each component contains at least one edge. For each component , AdvGuard computes a structural score as the sum of edge weights within that component. That is, the score aggregates the semantic similarity values of all query pairs connected inside the component:
This score captures both the size of the component (i.e., how many queries are connected) and the strength of their relationships (i.e., how similar they are). As a result, it reflects the extent to which a group of queries exhibits consistent semantic overlap. Since benign queries either remain isolated or form small, weakly connected components, they tend to lead to lower structural scores. Prompt extraction queries, especially those based on iterative paraphrasing or progressive probing, form larger and denser components, and lead to higher structural score.
Therefore, the structural score serves as a compact indicator of sustained and coordinated query refinement, enabling AdvGuard to distinguish normal interaction patterns from progressive prompt extraction behavior.
Statistical Filtering. While higher structural scores indicate stronger evidence of coordinated prompt extraction, using a fixed threshold for detection is insufficient in practice. The scale of structural scores depends on factors such as query volume and similarity levels, which vary across deployments and over time [34], [35]. As a result, a static threshold is difficult to calibrate and may either miss subtle attacks or generate excessive false positives.
To address this, AdvGuard uses an outlier-based statistical filter over the component-level structural scores. Under benign workloads, these scores are expected to remain concentrated within a narrow range, reflecting sparse and weakly connected components. In contrast, coordinated extraction attacks produce rapidly accumulating edge weights and connectivity, resulting in extreme upper-tail deviations.
We identify unusually large structural scores using a Grubbs’ test [36]. The test evaluates the largest component score relative to the mean and standard deviation of the current score distribution. Components identified as outliers are forwarded to the graph-based classifier for further analysis. This stage serves as a lightweight candidate-selection mechanism. It concentrates subsequent computation on components exhibiting unusually strong accumulated similarity. We empirically compare this filter against fixed-threshold alternatives in Appendix B-B.
GCN Classification
The statistical filter selects components with unusually large structural scores. However, statistical extremeness alone does not establish malicious intent. Legitimate workflows, such as iterative text editing or code refinement, can also produce large components with substantial semantic similarity. Moreover, the structural score summarizes only the magnitude of accumulated similarity, not its internal topology.
For example, a chain and a star with four equally weighted edges have identical structural scores, although the former distributes similarity along successive query pairs and the latter concentrates it around a common query anchor. This distinction is invisible to the scalar score. Neither topology is inherently malicious; rather, their difference motivates examining whether relational structure provides additional discriminative information among statistically selected components.
To investigate this distinction, AdvGuard applies a graph-based classifier to candidate components. The classifier complements statistical filtering by evaluating the arrangement of similarity relationships, with the aim of distinguishing extraction-related activity from benign iterative workflows.
Line-Graph Construction. To explicitly capture these overlap patterns, AdvGuard transforms each flagged component into its corresponding line graph.
Let denote a candidate component of the query similarity graph. Under the nearest-neighbor attachment rule, each query connects to at most one earlier query, so the original graph is a forest. Its components can nevertheless exhibit different branching patterns and degrees of attachment to shared query anchors.
We represent these relationships using the line graph . Each edge becomes a node . Two distinct line-graph nodes are adjacent when their corresponding query edges share an endpoint:
For example, query edges and become adjacent line-graph nodes because both involve .
This transformation makes shared-anchor relationships explicit for classification. A chain of query similarities remains chain-shaped in the line graph, whereas edges incident to a common query induce a clique of line-graph nodes. Thus, local connectivity in represents how similarity relationships are concentrated around, or distributed across, queries. The transformation re-expresses the original component structure; it does not add unobserved semantic relationships.
Each line-graph node is initialized with the cosine-similarity weight of its corresponding query edge:
The classifier therefore receives both the strength of each retained similarity relationship and its connectivity to other relationships. Although these weights are derived from query embeddings, they do not retain the full query semantics.
GCN Training Design
We use a GCN to aggregate information over neighboring similarity relationships in . The model consists of three GCN layers followed by global mean pooling and a fully connected classification layer. Each layer combines a node’s representation with those of its neighbors, allowing successive layers to incorporate information from progressively larger neighborhoods of the line graph.
The classifier is trained offline on labeled components derived from benign and adversarial interaction traces, which remain independent from our evaluation (detailed in § 10-E), and then deployed for online detection. We use the GCN as a lightweight architecture for learning from edge weights and their relational structure, rather than relying on a fixed threshold over a single scalar feature.
GCN Final Detection. During online operation, components selected by the statistical filter are forwarded to the GCN. For each candidate component, the classifier produces a binary prediction . If , AdvGuard flags the component and blocks subsequent queries in the same session; otherwise, processing continues normally.
The classifier serves as a final-stage decision mechanism for statistically selected candidates. Its intended benefit is to reduce false positives arising from benign components with large aggregate scores while retaining detection of extraction attempts.
Online Detection & Bounded-State Management
AdvGuard is designed to operate on per-session interaction streams, where adversarial behavior unfolds over a sequence of queries from a single user. During runtime, each incoming query incrementally updates the interaction graph . The system computes the prompt-level risk score (§IV-B) and updates the structural score (§IV-C) for the connected component containing .
Detection proceeds progressively. In early stages, when interaction history is limited (), decisions rely on prompt-level signals, where queries with may trigger immediate intervention. As interaction context accumulates (), the full detection pipeline is activated, components exhibiting anomalous scores are flagged by the statistical filter and forwarded to the GCN classifier (§IV-D) for structure-aware analysis.
We define the time-to-detect (TTD) as the number of queries observed before our classifier-based defense action is triggered. When the GCN predicts , AdvGuard blocks the corresponding component and denies subsequent queries from the same interaction session (e.g., “Request denied due to safety policy”). AdvGuard’s detection occurs entirely at the system boundary, so the underlying LLM remains unchanged, preserving black-box compatibility.
To ensure scalable real-time deployment, AdvGuard maintains a bounded interaction state. Only the top-20 highest-risk components (ranked by ) are retained in memory, while lower-risk components are evicted using an LRU policy.
The interaction graph is periodically reset after a fixed interval (e.g., 24 hours). While resets may introduce a limited window for slow-rate attacks, adversarial interactions must re-accumulate structural signals within each cycle, which are captured by both prompt-level and structural detection stages. This mechanism ensures bounded memory usage and prevents unbounded growth of interaction state. Concrete parameter choices (e.g., , , TTD) are described in § 7.
Implementation and Experimental Setup
This section describes the implementation environment, detection configuration, baselines, attack setup, datasets, workload construction, and evaluation metrics used to evaluate AdvGuard.
Hardware and Software Environment
Core development and detector evaluation use three NVIDIA H100 GPUs (80 GB VRAM each), while eight A100 GPUs (40 GB VRAM each) support prolonged end-to-end experiments. We implement AdvGuard in Python 3.11 using PyTorch 2.9.0 [37] and HuggingFace Transformers [38, 39], with CUDA acceleration [40] and Bitsandbytes [41] for mixed-precision inference.
We evaluate AdvGuard on four open-source LLMs: GPT-2 [42], Falcon3 [43], LLaMA 3.2 [3], and Qwen 2.5 [44]. These models span diverse architectures and deployment settings commonly used in open LLM evaluations [45, 46]. Unless stated otherwise, all defenses use identical inference settings and public checkpoints.
Detection Configuration and Hyperparameters
For each user session, AdvGuard initializes the graph with 10,000 isolated nodes, one for each query to calibrate normal semantic similarity. These reference queries are excluded from component-size counts, structural scores, and GCN classification. The similarity threshold is set to the 80th percentile of second nearest-neighbor similarities computed over this corpus. We validate the threshold choice in Appendix C-B and the diversity of the initialization corpus in Appendix C-A.
When a component contains fewer than queries, detection relies on prompt-level signals. We set and trigger early intervention when the prompt-level risk score satisfies , with . Once a component reaches , we apply the Grubbs-based statistical filter to component-level structural scores at a significance level of . Components flagged by the statistical filter are forwarded to the GCN classifier. A defense action is triggered after a time-to-detect window of 10 queries. Parameter sensitivity is analyzed in Appendix C.
The GCN is trained as a binary graph classifier on labeled components using a 70:20:10 train/validation/test split. We use Adam with learning rate , batch size 32, cross-entropy loss, and train for 20 epochs. The impact of graph architecture and model depth is evaluated in § 10-E.
To ensure scalable online detection, AdvGuard maintains bounded interaction state. Only the top-20 highest-risk components, ranked by , are retained in memory, while lower-risk components are evicted using an LRU policy. This bounds memory and computation while preserving high-risk traces. We also evaluated full-history similarity, but found it prohibitively expensive. Therefore, we adopt bounded graph state for predictable latency and memory usage.
Detection Baselines and Attack Setup
We compare AdvGuard against two representative input-side defenses: PromptGuard2 [15] and LLMGuard [16]. Both are publicly available, operate under black-box assumptions, and have been used in prior LLM security evaluations [7, 47, 48]. We evaluate each baseline using its released configuration without tuning or retraining. All attacks are evaluated within single-user sessions, with AdvGuard constructing interaction graphs solely from queries within each session.
Prompt leakage attacks. We evaluate against four state-of-the-art prompt leakage attacks aligned with our threat model: Zhang et al. [8], PLeak [7], LeakAgent [9], and PromptFuzz [21]. These attacks cover iterative paraphrasing, adversarial optimization, agentic red-teaming, and fuzzing-based mutation. For each attack, we use the authors’ released implementations and prompt sets under their default configurations. For attacks using a shadow model, all generated candidate queries are collected and submitted to the defended target.
Evaluation settings. We evaluate AdvGuard under two settings: (i) offline detection on mixed benign-adversarial query workloads, and (ii) online in-loop defense evaluation against adaptive query-response attacks.
(i) Detection under mixed-query workloads. We construct mixed-query workloads by combining adversarial query traces from each attack with benign query streams. This setting evaluates the ability of AdvGuard to detect prompt leakage queries when adversarial behavior is interleaved with realistic user traffic.
(ii) In-loop defense evaluation. For the two strongest adaptive attacks (PLeak and LeakAgent), we place AdvGuard directly in the attack loop as a real-time pre-inference middleware. In this setting, each adversarial query must pass through AdvGuard before reaching the LLM, enabling measurement of both detection and end-to-end leakage mitigation under adaptive query-response attack behavior.
Prompt Leakage Mitigation Strategies
We compare AdvGuard against four mitigation baselines. Filter-based defense [7] removes leaked prompt content from model outputs. Sandwich Defense [49] places protective instructions before and after user queries. Prompt Obfuscation [26] optimizes alternative system-prompt representations to reduce extractability. PromptKeeper [24] uses statistical testing to detect leakage and regenerates flagged responses with a dummy system prompt. These baselines span input reinforcement, prompt protection, output filtering, and response regeneration, covering distinct intervention points and access requirements.
Datasets and Workload Construction
Prompt sets. The evaluation of AdvGuard is conducted using randomly shuffled query streams comprising benign and adversarial prompt behaviors. We construct these streams from three sources: general single-turn benign prompts, multi-turn conversational dialogues, and adversarial prompts designed for prompt leakage attacks.
Benign prompts. Following prior prompt-leakage evaluations [7], [8], we draw benign prompts from GSM8K [50], SST-2 [51], SQuAD v2.0 [52], Dolly-15k [53], and the benign subset of WildJailbreak [54]. These datasets cover mathematical reasoning, sentiment classification, question answering, and instruction-following tasks, providing diverse semantic patterns representative of real-world LLM usage.
Multi-turn conversational data. To evaluate robustness under realistic dialogue patterns, we use OASST1 [55], ChatAlpaca [56], and WildChat [57], which contain an average of 8.58, 4.32, and 12.48 turns per conversation, respectively. These datasets contain multi-turn conversations with natural semantic continuity, allowing us to test whether AdvGuard can distinguish benign conversational coherence from adversarial extraction behavior.
Adversarial prompts. Adversarial prompts are drawn from the default prompt sets released with Zhang et al. [8], PromptFuzz [21], LeakAgent [9], and PLeak [7]. We additionally include prompt extraction corpora from WildJailbreak [58], AdvPrompter [59], Salesforce-Leak [13], AI-curated prompts [1], and human-crafted prompts [21], [60]. These sources introduce broader attack strategies and prompt formulations beyond method-specific datasets. All prompts used in our evaluation are in English.
Mixed query workload. We construct mixed-query workloads by combining a benign query set with an adversarial query set . All adversarial queries are retained to preserve complete attack behavior and are mixed with benign queries to obscure extraction intent. We set , corresponding to a 1:9 adversarial-to-benign ratio. The combined stream is shuffled using a fixed random seed. Results are averaged over multiple benign samples to reduce sampling bias.
Evaluation Metrics
Detection effectiveness. We measure AdvGuard detection performance using standard classification metrics. Specifically, we report: (i) the false positive rate (FPR), which measures the fraction of benign queries incorrectly flagged as attacks; (ii) recall, which measures the fraction of adversarial queries that are successfully detected; and (iii) the F1 score, which captures the trade-off between false positives and false negatives.
Prompt leakage success. To measure how much sensitive information is extracted, we integrate AdvGuard into the attack frameworks and compare reconstructed outputs against ground-truth system prompts. Following prior work, we use prompts from awesome-chatgpt-prompts [61] as controlled secret prompts and evaluate leakage success for both PLeak [7] and LeakAgent [9]. We report four metrics: substring match (SM), which flags success when the true prompt appears as a contiguous substring after normalization; exact match (EM), which requires identical reconstruction; semantic similarity (SS), computed as cosine similarity between sentence embeddings of the true and reconstructed prompts; and ROUGE [62], which measures n-gram overlap.
Latency overhead. We report per-query latency at p50, p95, and p99 to capture median and tail latency introduced by AdvGuard under online deployment.
Evaluation
Using the experimental setup in § 7, we address four research questions and then analyze the contribution of individual components.
RQ1: Detection effectiveness. Can AdvGuard accurately detect extraction queries in benign-heavy and benign multi-turn workloads while maintaining a low false-positive rate?
RQ2: Adaptive leakage mitigation. How much prompt leakage remains when response-adaptive attackers interact with the defended pipeline?
RQ3: Prompt-only detection. Can prompt-level signals detect extraction attempts before structural patterns emerge?
RQ4: Runtime overhead. What latency and throughput overhead does AdvGuard incur in the evaluated online pipeline?
AdvGuard’s Detection Effectiveness
We address RQ1 by evaluating AdvGuard’s ability to detect prompt-leakage queries under realistic mixed-traffic, multi-turn conversational, and heterogeneous attack workloads.
Detection Under Mixed Benign-Adversarial Query Workloads
We construct a mixed-query workload by injecting queries from four state-of-the-art prompt-leakage attacks (Zhang et al., PromptFuzz, LeakAgent, and PLeak) into five benign workloads (GSM8K, SST-2, SQuAD_v2, Dolly15k, and WildJailbreak) following the mixed-query construction policy in § 7. This setting models attacks embedded within benign user activity. Table I reports FPR, recall, and F1 score.
| Benign Stream | AdvQueries | PromptGuard | LLMGuard | AdvGuard | ||||||
| FPR(%)↓ | Recall(%)↑ | F1(%)↑ | FPR(%)↓ | Recall(%)↑ | F1(%)↑ | FPR(%)↓ | Recall(%)↑ | F1(%)↑ | ||
| Attack: Zhang (COLM’24) | ||||||||||
| GSM8K | 11234 | 10.75 | 96.90 | 82.91 | 2.37 | 92.75 | 92.73 | 3.40 | 99.95 | 98.11 |
| SST2 | 12346 | 8.85 | 95.75 | 83.22 | 0.68 | 96.40 | 96.93 | 0.00 | 100.00 | 100.00 |
| SQuAD_v2 | 12824 | 5.73 | 94.50 | 86.88 | 2.12 | 92.25 | 92.09 | 0.95 | 92.13 | 95.88 |
| Dolly15k | 9397 | 3.92 | 96.50 | 94.22 | 4.47 | 92.60 | 91.43 | 0.00 | 100.00 | 100.00 |
| Wildjailbreak | 12055 | 10.10 | 96.30 | 82.75 | 3.21 | 92.00 | 90.60 | 0.00 | 91.14 | 95.12 |
| Attack: PromptFuzz | ||||||||||
| GSM8K | 11234 | 10.56 | 94.66 | 83.24 | 5.81 | 94.52 | 88.82 | 3.40 | 99.91 | 98.23 |
| SST2 | 12346 | 7.40 | 90.99 | 83.25 | 3.29 | 93.85 | 91.41 | 0.00 | 99.02 | 99.51 |
| SQuAD_v2 | 12824 | 5.00 | 94.28 | 88.19 | 1.46 | 92.80 | 93.63 | 0.89 | 94.18 | 96.92 |
| Dolly15k | 9397 | 11.00 | 93.66 | 86.05 | 6.63 | 93.66 | 90.12 | 0.00 | 99.08 | 99.54 |
| Wildjailbreak | 12055 | 10.69 | 91.47 | 79.63 | 7.01 | 92.76 | 85.16 | 0.08 | 93.27 | 96.08 |
| Attack: LeakAgent | ||||||||||
| GSM8K | 11234 | 4.15 | 35.48 | 50.41 | 2.18 | 71.34 | 81.98 | 2.75 | 97.02 | 97.06 |
| SST2 | 12346 | 7.97 | 34.92 | 47.46 | 5.52 | 72.42 | 80.07 | 0.00 | 98.73 | 99.23 |
| SQuAD_v2 | 12824 | 11.07 | 35.34 | 46.05 | 3.34 | 71.36 | 80.78 | 0.66 | 94.09 | 97.02 |
| Dolly15k | 9397 | 7.14 | 35.26 | 49.66 | 3.78 | 71.92 | 82.03 | 0.02 | 99.51 | 99.08 |
| Wildjailbreak | 12055 | 11.67 | 35.10 | 46.07 | 2.42 | 71.62 | 81.88 | 0.17 | 94.21 | 96.89 |
| Attack: PLeak | ||||||||||
| GSM8K | 11234 | 4.85 | 57.40 | 70.19 | 4.22 | 78.36 | 85.29 | 2.75 | 98.52 | 96.67 |
| SST2 | 12346 | 3.35 | 56.72 | 70.13 | 4.37 | 78.14 | 84.62 | 0.00 | 98.73 | 99.23 |
| SQuAD_v2 | 12824 | 3.71 | 56.82 | 69.88 | 4.59 | 77.88 | 84.09 | 0.25 | 94.39 | 94.83 |
| Dolly15k | 9397 | 12.51 | 56.84 | 67.35 | 5.48 | 77.32 | 84.86 | 0.11 | 97.69 | 98.37 |
| Wildjailbreak | 12055 | 6.33 | 57.16 | 68.76 | 4.27 | 78.48 | 84.95 | 0.06 | 98.15 | 99.07 |
Table I. Detection performance of AdvGuard and baseline defenses under mixed-query stream (benign + adversarial).
Results. AdvGuard consistently outperforms PromptGuard and LLMGuard in reported F1 across all workloads and attack families, achieving F1 scores of 94.83–100.00%. AdvGuard FPR remains at or below 3.40% and is less than 1% in most settings, while recall ranges from 91.14–100.00%. The largest performance gap appears under LeakAgent, where PromptGuard recall drops to approximately 35%, and LLM-Guard reaches approximately 71%, while AdvGuard maintains 94.09–99.51%. Performance is also stable across benign workloads, with F1 standard deviation at most 2.04 percentage points for every attack. Appendix A-A evaluates unseen-family transfer.
Key Insight. These results show that AdvGuard reliably detects prompt-leakage attempts hidden within benign traffic. By modeling interaction-level query structure rather than isolated prompts, AdvGuard achieves high detection accuracy with low false positives, making it suitable for real-world LLM deployment. Next, we evaluate robustness under realistic interaction patterns. We stress-test AdvGuard under multi-turn conversational streams.
b) Detection Under Multi-Turn Conversational Workloads: Real-world LLM interactions are inherently multi-turn and context-dependent. Benign follow-up queries can resemble coordinated probing because both exhibit semantic continuity. To test this setting, we mixed the adversarial queries into three multi-turn conversational datasets: OASST1 [55], ChatAlpaca [56], and WildChat [57].
Results. As shown in Table II, AdvGuard maintains strong detection performance across all conversational workloads. AdvGuard consistently outperforms PromptGuard and LLMGuard across all conversational workloads, achieving 97.23–99.12% F1 and 100% recall. FPR remains low at 1.34–3.42%, compared with up to 10.67% for PromptGuard, while both baselines achieve only 85–90% recall. Appendix A-B evaluates mixed attack families.
| Dataset | Queries | PromptGuard | LLMGuard | AdvGuard | ||||||
| FPR | Recall | F1 | FPR | Recall | F1 | FPR | Recall | F1 | ||
| OASST1 | 8246 | 4.74 | 86.45 | 91.92 | 1.47 | 89.85 | 94.41 | 1.34 | 100.00 | 99.12 |
| ChatAlpaca | 9276 | 3.77 | 87.68 | 92.41 | 3.98 | 89.52 | 93.38 | 1.76 | 100.00 | 98.12 |
| WildChat | 8054 | 10.67 | 85.92 | 90.64 | 3.63 | 87.75 | 92.88 | 3.42 | 100.00 | 97.23 |
Table II. Detection performance under multi-turn conversational streams. Metrics are reported in %.
Key Insight. Overall, AdvGuard effectively distinguishes benign conversational coherence from adversarial extraction patterns, maintaining robust detection in multi-turn settings.
Adaptive leakage mitigation
Building on the detection performance results in § 10-A, we evaluate whether AdvGuard can prevent system prompt reconstruction when deployed directly within the online inference pipeline, addressing RQ2.
We integrate AdvGuard into two adaptive prompt-leakage attacks, PLEAK and LeakAgent, which refine subsequent queries based on prior interactions. Suspicious queries receive a standardized denial response (“Request denied due to safety policy”), limiting further prompt exposure. This setting evaluates response-adaptive attacks rather than static attack templates.
We compare AdvGuard against filter-based defense [23], Sandwich Defense [25], Prompt Obfuscation [26], and Prompt-Keeper [24]. Unlike defenses requiring privileged information, AdvGuard uses incoming queries without access to the system prompt, model internals, or generated responses. Table III reports leakage across GPT-2 [42], Falcon3 [43], LLaMA 3.2 [3], and Qwen2.5 [44].
| Attack | Defense | GPT-2 SM↓ | GPT-2 EM↓ | GPT-2 SS↓ | GPT-2 ROUGE-L↓ | Falcon3 SM↓ | Falcon3 EM↓ | Falcon3 SS↓ | Falcon3 ROUGE-L↓ | LLaMA3.2 SM↓ | LLaMA3.2 EM↓ | LLaMA3.2 SS↓ | LLaMA3.2 ROUGE-L↓ | Qwen2.5 SM↓ | Qwen2.5 EM↓ | Qwen2.5 SS↓ | Qwen2.5 ROUGE-L↓ |
| PLEAK | No Defense | 0.74 | 0.73 | 0.92 | 0.95 | 0.78 | 0.73 | 0.81 | 0.91 | 0.92 | 0.92 | 0.99 | 0.98 | 0.97 | 0.95 | 0.99 | 0.97 |
| PLEAK | Sandwich-based | 0.48 | 0.28 | 0.82 | 0.71 | 0.64 | 0.55 | 0.81 | 0.83 | 0.61 | 0.52 | 0.82 | 0.71 | 0.91 | 0.90 | 0.94 | 0.92 |
| PLEAK | Filter-based | 0.09 | 0.01 | 0.41 | 0.47 | 0.20 | 0.04 | 0.53 | 0.64 | 0.14 | 0.03 | 0.48 | 0.54 | 0.29 | 0.11 | 0.61 | 0.66 |
| PLEAK | Prompt Obfuscation | 0.01 | 0.01 | 0.09 | 0.13 | 0.01 | 0.01 | 0.09 | 0.19 | 0.02 | 0.01 | 0.17 | 0.31 | 0.01 | 0.00 | 0.22 | 0.29 |
| PLEAK | PromptKeeper | 0.00 | 0.00 | 0.04 | 0.12 | 0.00 | 0.00 | 0.07 | 0.18 | 0.01 | 0.00 | 0.14 | 0.28 | 0.01 | 0.00 | 0.22 | 0.26 |
| PLEAK | AdvGuard | 0.01 | 0.00 | 0.05 | 0.12 | 0.01 | 0.00 | 0.12 | 0.30 | 0.02 | 0.00 | 0.16 | 0.21 | 0.02 | 0.01 | 0.21 | 0.23 |
| LeakAgent | No Defense | 0.90 | 0.89 | 0.90 | 0.93 | 0.85 | 0.84 | 0.95 | 0.95 | 0.94 | 0.89 | 0.97 | 0.95 | 0.98 | 0.95 | 0.99 | 0.94 |
| LeakAgent | Sandwich-based | 0.37 | 0.22 | 0.64 | 0.68 | 0.72 | 0.63 | 0.75 | 0.91 | 0.81 | 0.75 | 0.85 | 0.89 | 0.95 | 0.93 | 0.99 | 0.91 |
| LeakAgent | Filter-based | 0.09 | 0.00 | 0.33 | 0.34 | 0.16 | 0.11 | 0.56 | 0.60 | 0.12 | 0.09 | 0.38 | 0.44 | 0.33 | 0.19 | 0.63 | 0.66 |
| LeakAgent | Prompt Obfuscation | 0.01 | 0.01 | 0.17 | 0.35 | 0.02 | 0.02 | 0.26 | 0.40 | 0.03 | 0.01 | 0.14 | 0.25 | 0.14 | 0.04 | 0.30 | 0.61 |
| LeakAgent | PromptKeeper | 0.00 | 0.00 | 0.01 | 0.14 | 0.00 | 0.00 | 0.06 | 0.19 | 0.02 | 0.01 | 0.22 | 0.37 | 0.03 | 0.00 | 0.21 | 0.34 |
| LeakAgent | AdvGuard | 0.01 | 0.00 | 0.02 | 0.14 | 0.01 | 0.00 | 0.15 | 0.15 | 0.01 | 0.00 | 0.16 | 0.17 | 0.09 | 0.01 | 0.25 | 0.52 |
Table III. Prompt leakage scores for PLEAK and LeakAgent adaptive querying attacks across victim models and defenses. Lower values indicate stronger leakage mitigation.
Results. Table III shows that AdvGuard limits exact match (EM) to 0.00–0.01 and substring match (SM) to 0.01–0.09 across both adaptive attacks and all four victim models, compared with 0.73–0.95 EM and 0.74–0.98 SM without defense. Semantic similarity remains at 0.02–0.25, and both semantic similarity and ROUGE-L are lower than those of sandwich- and filter-based defenses in every setting. Despite operating solely on incoming user queries, AdvGuard also surpasses PromptKeeper on ROUGE-L under LeakAgent on Falcon3 (0.15 vs. 0.19) and LLaMA 3.2 (0.17 vs. 0.37), although PromptKeeper retains an advantage on Qwen 2.5 (0.34 vs. 0.52). These results demonstrate substantial leakage mitigation without access to system prompts, model internals, or generated responses.
Key Insight. AdvGuard effectively limits adaptive prompt reconstruction by detecting the progressive query patterns of extraction attacks. The principal advantage of AdvGuard is its combination of limited access requirements and consistently lower leakage.
Detection of Prompt Leakage in Sparse Adversarial Regimes
Prompt-leakage attacks may begin with a small number of engineered probes before sufficient interaction history emerges for structural analysis [63, 64]. RQ3 evaluates whether AdvGuard can detect such early-stage attempts using only prompt-level signals. We disable graph-based analysis and evaluate each query independently.
We construct five settings by combining benign [50, 51, 52, 53, 58] and adversarial query sets [13, 54, 58, 59, 60] drawn from the sources described in Section V, at a 1:9 adversarial-to-benign ratio. These include benchmark attacks and human- and model-crafted prompts targeting system instructions (e.g., instruction override and prompt disclosure queries). We classify queries as adversarial when their prompt-level risk score exceeds a threshold while benign queries typically fall within the range .
Results. Table IV shows that AdvGuard achieves a high recall of over 95% and F1 scores up to 99.35% across all settings. In the meantime, AdvGuard maintains low false-positive rates at no more than 1.20%. Performance remains stable across diverse adversarial prompt sources, including both benchmark datasets and curated attack prompts.
| Benign | Adversarial | #AdvQry | FPR(%)↓ | Recall(%)↑ | F1(%)↑ |
| GSM8K | WildJailbreak | 543 | 0.08 | 96.93 | 97.99 |
| SST-2 | AdvPrompter | 2004 | 1.20 | 95.00 | 97.00 |
| SQuAD_v2 | SalesforceLeak | 841 | 0.09 | 96.40 | 97.94 |
| Dolly15k | AI-curated | 958 | 0.00 | 99.67 | 99.35 |
| JailbreakBench | humanAI-Craft | 482 | 0.00 | 99.64 | 99.35 |
Table IV. Early-stage prompt-level detection performance of AdvGuard when graph-based structural analysis is disabled.
Key Insight. This demonstrates that AdvGuard can detect early-stage prompt-leakage attempts, preventing attackers from conducting slow and stealthy prompt-leakage attempts before larger structural patterns emerge.
Runtime overhead
RQ4 evaluates the runtime overhead of AdvGuard in online LLM inference. We compare AdvGuard with PromptGuard and LLMGuard on LLaMA3.2 with sequence length 512 over 1,000 consecutive queries, reporting average and tail latency (P50, P95, P99) and throughput (queries/s).
Overall performance. As shown in Figure 3, AdvGuard introduces moderate overhead compared with stateless baselines. Average latency is 32.96 ms/query, with 54.7 ms P95 and 98.9 ms P99 latency, while throughput decreases from 35–39 to 21.9 queries/s. This overhead primarily stems from interaction-aware processing, including structural analysis. A detailed component-level latency breakdown is provided in Appendix D-A. Despite this overhead, the absolute latency remains practical for real-time deployment.
Figure 3. Latency and throughput comparison between AdvGuard, PromptGuard, and LLMGuard.
Stability under long sessions. Despite its stateful architecture, Figure 4 shows that AdvGuard maintains stable latency across 1,000 queries, demonstrating bounded runtime overhead during long-running sessions. This stability is achieved through bounded graph state management, selective caching, and eviction policies, which collectively ensure predictable runtime behavior during long-running sessions.

Figure 4. Latency stability of AdvGuard compared with PROMPTGUARD and LLMGUARD. Each point represents the average latency of 200 queries.
Key Insight. While stateless defenses achieve lower latency, they provide substantially weaker protection against adaptive and coordinated attacks (as demonstrated in § 10-A–VI-B). AdvGuard intentionally trades a modest, well-bounded runtime overhead for significantly stronger security guarantees, making it a practical and deployable defense for production-scale LLM systems.
Ablation Study
We ablate AdvGuard to assess three hypotheses: (i) interaction-level modeling is necessary beyond per-query features, (ii) no detection stage is sufficient in isolation, and (iii) structural learning is required to capture coordinated extraction behavior. We systematically remove or simplify pipeline components and measure their impact on detection performance.
Sequence-level modeling necessary for detecting prompt leakage? We test whether reliable prompt-leakage detection requires modeling interaction structure beyond individual queries. Figure 5 shows aggregated similarity scores over a mixed-query stream. Benign scores remain unstructured, with 88% of values below 0.5 and no temporal trend. Adversarial scores increase monotonically as semantic reuse accumulates across turns. Such signals are not observable at the level of individual queries — each appears benign in isolation, but separation emerges only when similarities are aggregated over the interaction history .
Figure 5. Weighted sum distribution for benign and adversarial queries over time.
To further analyze separability, we project sequence-level graph embeddings using t-SNE (Figure 6). Benign interactions form diffuse, overlapping clusters, whereas adversarial sequences concentrate into tight, well-separated regions. This separation does not arise from individual query embeddings, but from the structural relationships encoded in the interaction graph, indicating that sequence-level representations provide significantly stronger discriminative power.

Figure 6. t-SNE projection of benign and adversarial query embeddings.
Key Insight. Prompt leakage is a structural phenomenon: adversarial intent spans semantically related queries, making per-query detection insufficient. AdvGuard’s interaction graph reveals this coordination, motivating the stronger structural reasoning analyzed next.
Is a Multi-Stage Detection Pipeline Necessary? Table V evaluates the contribution of each detection stage in AdvGuard.
| Dataset | Method | FPR(%)↓ | Recall(%)↑ | F1(%)↑ | |||
| OASST1 | Prompt Signal | 11.50 | — | 69.00 | — | 77.00 | — |
| OASST1 | Outlier Filter | 0.00 | -11.50 | 100.00 | +31.00 | 100.00 | +23.00 |
| OASST1 | Graph Classifier | 0.00 | -11.50 | 100.00 | +31.00 | 100.00 | +23.00 |
| WildChat | Prompt Signal | 18.30 | — | 69.00 | — | 74.00 | — |
| WildChat | Outlier Filter | 6.70 | -11.60 | 100.00 | +31.00 | 97.00 | +23.00 |
| WildChat | Graph Classifier | 3.18 | -15.12 | 100.00 | +31.00 | 98.00 | +24.00 |
| ChatAlpaca | Prompt Signal | 1.88 | — | 69.00 | — | 81.00 | — |
| ChatAlpaca | Outlier Filter | 3.46 | +1.58 | 100.00 | +31.00 | 98.00 | +17.00 |
| ChatAlpaca | Graph Classifier | 0.00 | -1.88 | 100.00 | +31.00 | 100.00 | +19.00 |
Table V. Component-wise ablation of AdvGuard. denotes the relative change from the prompt-only baseline.
a) Prompt-level detection is insufficient: Using only the prompt signal (Stage 1) yields moderate F1 scores (74–81%) but consistently low recall (69%) across all datasets, indicating limited detection of attacks distributed across queries.
b) Statistical filtering reduces false positives but is insufficient alone: Adding structural aggregation raises recall to 100% and F1 from 74–81% to 97–100%, showing the benefit of interaction-level modeling. However, false positives remain higher on conversational workloads such as WildChat due to ambiguity between adversarial sequences and semantically coherent benign interactions.
c) Graph-based reasoning reduces residual structural ambiguity: The graph classifier further improves robustness, preserving 100% recall while reducing or matching the filter’s FPR across datasets. On WildChat, for example, FPR decreases from 6.70% to 3.18%, while F1 improves from 97% to 98%. These results suggest that graph-based reasoning effectively distinguishes coordinated adversarial interaction patterns from benign but topically coherent conversations.
| Dataset | Size | SimMLP(%) | 1-GCN(%) | 3-GCN(%) |
| WildJailbreak | 100k | 30.13 | 94.05 | 99.99 |
| AdvPrompter | 15,119 | 42.61 | 96.09 | 100.00 |
| Salesforce-Leak | 7,780 | 43.25 | 90.34 | 100.00 |
| JailbreakBench | 5,855 | 45.94 | 94.90 | 100.00 |
| Average | – | 40.48 | 93.85 | 100.00 |
Table VI. Impact of GNN depth on adversarial query detection accuracy over 20 training epochs. All values are reported in percentages.
Key Insight. These results demonstrate that no single stage is sufficient: prompt-level analysis misses distributed attacks, while structural aggregation remains susceptible to benign conversational overlap. Combining prompt, interaction-level, and graph-based signals enables AdvGuard to achieve high recall with low false positives.
Is Structural Learning Necessary Beyond Scalar Aggregation. We test whether scalar aggregation of similarity features suffices for adversarial classification or whether graph-based relational modeling is needed. We construct interaction graphs from four adversarial datasets by incrementally inserting queries connected via semantic similarity, and compare three backends: SimMLP (scalar features, no connectivity), 1-GCN, and 3-GCN, trained for 20 epochs on a 70:20:10 split. Table VI shows that SimMLP fails substantially across all datasets, achieving only 40.48% average accuracy, compared with 93.85% for 1-GCN and nearly 100% for 3-GCN. These results support multi-hop relational modeling for capturing coordination that scalar aggregation misses.
Key Insight. Prompt-leakage structure is not linearly separable at the scalar level: relational graph modeling is necessary, not optional, for detecting coordinated multi-turn extraction.
Discussion
AdvGuard’s design leverages historical interaction patterns to achieve strong detection performance under adaptive, multi-turn prompt-leakage attacks.
Limitations. Because AdvGuard relies on interaction structure, meaningful graph structure requires sufficient query history. Very short sessions or extremely sparse attack patterns may therefore evade early detection. The system also depends on semantic embeddings to establish query similarity, which introduces an inductive bias that can reduce sensitivity to highly obfuscated, stylistically diverse queries. Finally, the system requires additional storage for graph structures.
Evasion considerations. An adaptive adversary might attempt to circumvent detection by distributing queries across multiple identities or systematically paraphrasing them. However, such strategies inherently inflate the adversary’s query budget and reduce the efficiency of information aggregation. Because successful prompt leakage typically requires iterative refinement and cross-query synthesis, eliminating all detectable structural patterns while still making meaningful progress remains difficult in practice.
Generalization. While our evaluation focuses on prompt leakage, the core principle of detecting malicious behavior via structured query provenance may naturally extend to a broader class of black-box attacks, including model extraction, jailbreak discovery, and data-exfiltration attempts that rely on iterative querying, Appendix E-A. Because AdvGuard is model-agnostic and requires no internal model access, it can be deployed across diverse LLM architectures and updated systems without retraining.
Future work. Several directions remain promising for improving robustness and scalability. These include exploring hybrid features that combine semantic embeddings with lightweight syntactic or statistical signals to better handle stylistically diverse attacks, investigating longer-horizon or session-linked memory mechanisms, evaluating performance against emerging multi-modal and agentic attack patterns [65], and studying federated or privacy-preserving provenance aggregation across distributed deployments [27]. Overall, AdvGuard does not claim to eliminate prompt leakage. Rather, it shifts the defense boundary from brittle prompt filtering to the significantly harder-to-evade dynamics of interaction structure.
Conclusion
This work demonstrates system prompt leakage as a progressive interaction-level threat rather than a single-query anomaly. Unlike existing defenses that analyze prompts in isolation, AdvGuard detects coordinated extraction behavior through a multi-stage pipeline combining prompt-level risk estimation, interaction-graph construction, statistical filtering, and graph-based classification.
Our evaluation shows that this design improves both detection and mitigation. AdvGuard achieves high recall with low false positives under mixed-query and multi-turn workloads, reduced leakage under adaptive attacks, effective early-stage detection, and bounded runtime overhead. Ablations further show that combining prompt-level, structural, and graph-based signals is necessary for robust detection.
Overall, AdvGuard establishes interaction-level monitoring as a practical and more robust alternative to prompt-only defenses, making system prompt extraction harder by detecting the coordinated query patterns that successful leakage requires.
Additional Robustness Evaluations
Leave-One-Attack-Family-Out Evaluation
We evaluated cross-family generalization using a leave-one-attack-family-out protocol. In each fold, one attack family was excluded from training and validation and reserved exclusively for testing. Benign prompts were split into disjoint training, validation, and test sets using a 70/20/10 split, while the remaining attack families were divided into 80% training and 20% validation sets. Each fold contained approximately 8,000 training graphs, 1,000 validation graphs, and 2,000 balanced test graphs. On unseen families, Table VII shows AdvGuard achieves 99.98% mean recall and 99.99% mean F1 with no false positives.
| Held-out family | FPR(%) | Rec(%) | F1(%) |
| LeakAgent | 0.00 | 100.00 | 100.00 |
| PLeak | 0.00 | 99.90 | 99.95 |
| PromptFuzz | 0.00 | 100.00 | 100.00 |
| Zhang et al. | 0.00 | 100.00 | 100.00 |
| Mean | 0.00 | 99.98 | 99.99 |
Table VII. LEAVE-ONE-ATTACK-FAMILY-OUT TEST RESULTS.
Detection Under Heterogeneous Attacks Mixed With Benign Multi-Turn Conversations
We evaluate AdvGuard under heterogeneous attacks mixed with benign multi-turn conversations. We combine multiple prompt-leakage attacks [7, 9, 17, 21], each contributing 25% of adversarial queries, and randomly mix them with a benign multi-turn conversational dataset drawn from ChatAlpaca [56], OASST1 [55], and WildChat [57].
Table VIII reports recall of 98.00–99.11% across the three streams. FPR is 1.19% on ChatAlpaca, 0.10% on OASST1, and 3.40% on WildChat. Despite high recall, WildChat has a lower F1 score of 85.74%, indicating greater false-positive sensitivity on the more diverse stream. Overall, AdvGuard remains robust under mixed heterogeneous attack families, maintaining high detection accuracy.
| Benign Stream | Queries | FPR(%)↓ | Recall(%)↑ | F1(%)↑ |
| ChatAlpaca | 9276 | 1.19 | 99.00 | 94.42 |
| OASST1 | 8152 | 0.10 | 99.11 | 99.55 |
| WildChat | 8056 | 3.40 | 98.00 | 85.74 |
Table VIII. Detection performance of AdvGuard under heterogeneous attack families randomly mixed with benign multi-turn conversations.
PLeak Leakage Analysis Across Its Evaluation Datasets
We evaluate PLeak extraction across domains and system prompts using its original setup [7], with LLaMA2 [66] generating adversarial queries, where PLeak achieves its highest success rates. Due to PLeak’s roughly two-hour runtime per dataset, we restrict this evaluation to LLaMA2 [66] only. We report the same four leakage metrics used in PLeak—Substring Match (SM), Exact Match (EM), Semantic Similarity (SS), and ROUGE-L scores.
Table IX shows AdvGuard consistently maintains low leakage. SM and EM remain at 0.00–0.02, while SS and ROUGE-L range from 0.08–0.33 across datasets.
| Dataset | SM↓ | EM↓ | SS↓ | ROUGE-L↓ |
| Roles | 0.00 | 0.00 | 0.12 | 0.14 |
| SIQA | 0.01 | 0.00 | 0.33 | 0.27 |
| Financial | 0.02 | 0.02 | 0.15 | 0.11 |
| SQuAD2 | 0.00 | 0.00 | 0.09 | 0.15 |
| GSM8K | 0.01 | 0.00 | 0.12 | 0.16 |
| ChatGPT-Roles | 0.02 | 0.00 | 0.15 | 0.17 |
| Rotten Tomatoes | 0.00 | 0.00 | 0.08 | 0.13 |
Table IX. PLeak’s performance with AdvGuard using default target system prompts across its evaluation datasets.
Additional Design Studies
Nearest-Neighbor vs. Top- Connectivity
AdvGuard relies on a similarity graph to capture structural relationships among queries over time. A key design question is how to construct this graph: whether each query should connect to only its closest neighbor, or to multiple similar queries (top-). We compared nearest-neighbor connectivity with top- () graph construction (Table X). Top- raises GSM8K FPR from 3.40% to 16.35% and Dolly-15k false negatives from 10 to 462, reducing recall from 99.52% to 78.00%. Additional similarity computations and graph updates also increase latency by approximately . We therefore adopt second nearest-neighbor connectivity for its stronger overall detection–latency trade-off.
| Data | Method | FPR(%)↓ | Rec(%)↑ | F1(%)↑ | Lat↓ |
| GSM8K | Top- | 16.35 | 100.00 | 93.00 | 92.48 |
| GSM8K | NN | 3.40 | 99.24 | 98.00 | 30.55 |
| SST-2 | Top- | 0.00 | 99.48 | 100.00 | 105.48 |
| SST-2 | NN | 0.00 | 99.48 | 100.00 | 32.96 |
| Dolly | Top- | 0.00 | 78.00 | 88.00 | 86.48 |
| Dolly | NN | 0.00 | 99.52 | 100.00 | 28.45 |
Table X. Design study for similarity graph connectivity (NN vs. top-). Latency (Lat) is reported in milliseconds.
Choice of Filter for Structural Anomaly Detection
We compare the Grubbs outlier filter with three fixed-threshold alternatives, (i) structural density (), (ii) weighted structural aggregation (), and (iii) size-gated aggregation ( and ).
Table XI shows that fixed thresholds are unstable across conversational datasets, yielding high FPR or reduced detection. In contrast, the Grubbs filter maintains high recall with substantially fewer false positives. The final setting uses the Grubbs outlier [36] test to select anomalous components.
| Filter | FPR(%)↓ | Rec(%)↑ | F1(%)↑ |
| Dataset: OASST1 | |||
| 34.90 | 0.47 | 1.00 | |
| 29.50 | 99.91 | 88.00 | |
| & | 5.58 | 100.00 | 98.00 |
| Outlier Test | 0.00 | 99.91 | 100.00 |
| Dataset: WildChat | |||
| 25.80 | 0.47 | 1.00 | |
| 49.60 | 99.91 | 81.00 | |
| & | 9.50 | 100.00 | 97.00 |
| Outlier Test | 6.70 | 99.91 | 97.00 |
| Dataset: ChatAlpaca | |||
| 46.28 | 0.38 | 1.00 | |
| 68.59 | 99.91 | 76.00 | |
| & | 17.69 | 99.91 | 65.00 |
| Outlier Test | 3.46 | 99.72 | 98.00 |
Table XI. Comparison between statistical filter and threshold-based filter
| Category | Avg Similarity↓ | Diversity↑ |
| Summarization | 0.0821 | 0.9179 |
| Sentiment | 0.3067 | 0.6933 |
| Conversation | 0.2531 | 0.7469 |
| Reasoning | 0.1105 | 0.8895 |
| QA | 0.0718 | 0.9282 |
| Instruction | 0.0426 | 0.9574 |
| Overall | 0.0513 | 0.9487 |
Table XII. Semantic diversity of baseline prompt initialization.
Baseline Diversity and Hyperparameter Sensitivity
Semantic diversity of baseline
We evaluate the semantic diversity of the baseline prompt initialization using average pairwise cosine similarity between prompt embeddings, with diversity defined as one minus similarity. Prompts are deduplicated prior to analysis.
Table XII shows high baseline diversity, with an overall score of 0.9487 and low average similarity. This confirms that the baseline is not dominated by redundant patterns, enabling reliable similarity threshold estimation with minimal bias.
Similarity Threshold Selection
We analyze the impact of the similarity threshold used for graph construction, which controls edge formation between queries. Table XIII shows a clear trade-off between detection sensitivity and robustness. Higher thresholds (e.g., 90–95%) result in conservative graphs with fewer edges, reducing recall as semantically related adversarial queries fail to connect. In contrast, lower thresholds (e.g., 60–70%) increase connectivity, preserving recall but introducing noise that degrades F1. Performance remains stable in the range of 75–85%, where high recall is maintained without significant loss in F1. We select as it achieves near-optimal recall and the highest F1, providing a robust operating point for adversarial query detection.
| Threshold (%) | Recall(%)↑ | F1(%)↑ |
| 95 | 81.00 (-18.2%) | 90.00 (-10.0%) |
| 90 | 94.00 (-5.1%) | 97.00 (-3.0%) |
| 85 | 99.00 (-0.7%) | 99.00 (-1.0%) |
| 80 | 99.00 (—) | 100.00 (—) |
| 75 | 99.00 (±0.0%) | 100.00 (±0.0%) |
| 70 | 100.00 (+1.0%) | 98.00 (-2.0%) |
| 60 | 100.00 (+1.0%) | 96.00 (-4.0%) |
Table XIII. Impact of similarity threshold on adversarial query detection performance. Relative change is computed with respect to the 80% threshold (baseline).
Hyperparameter Sensitivity
We evaluate the sensitivity of AdvGuard to key hyperparameters, by varying minimum graph size () and the detection trigger threshold (TTD), reporting mean precision, recall, and F1 over five independent trials. Figure 7 averages results across datasets and seeds, and Table XIV provides a representative case.
Figure 7. Impact of minimum input graph size on detection performance.
Figure 8. Effect of varying Time-to-Detect (TTD) on detection performance, evaluated at the default minimum graph size of 30 nodes.
a) Minimum interaction history (): Figure 7 shows detection performance as a function of the minimum interaction history required before classification. When is small (e.g., 10), interaction structure is insufficiently formed, leading to unstable recall and reduced F1. Increasing to 30 stabilizes detection across all datasets, yielding near-optimal recall and F1 while maintaining perfect precision. Larger values provide no consistent benefit and introduce unnecessary delay. We therefore use .
b) Time-to-detect (TTD): Figure 8 evaluates the effect of the TTD threshold, defined as the number of incoming queries between successive graph-based detection checks after a component reaches . A check triggers a defense action only when the statistical filter selects the component and the GCN classifies it as malicious. Smaller values (e.g., TTD=5) enable faster detection but introduce noise, increasing false negatives. Larger values delay detection and reduce recall. Both TTD=10 and TTD=20 achieve near-perfect performance; we select TTD=10 as it provides earlier detection without sacrificing accuracy.
Runtime Overhead
Per-Component Latency Overhead Breakdown
Table XV reports the per-component latency of AdvGuard. Measurements are independent and not strictly additive due to caching and overlapping execution.
Structural connectivity contributes the largest latency, while embedding and similarity search incur substantially lower overhead. Tail latencies follow a similar pattern, indicating predictable component-level runtime behavior.
| Graph Size | TP | FP | FN | FPR(%)↓ | Rec(%)↑ | F1(%)↑ |
| 5 | 600 | 25 | 726 | 4.00 | 45.20 | 61.51 |
| 10 | 820 | 18 | 506 | 2.80 | 61.80 | 75.79 |
| 15 | 950 | 12 | 376 | 1.90 | 71.60 | 83.04 |
| 20 | 1286 | 2 | 40 | 0.40 | 96.98 | 98.39 |
| 30 | 1300 | 0 | 26 | 0.00 | 98.04 | 99.00 |
Table XIV. Impact of minimum interaction history on detection performance. All performance metrics are reported in percentages.
| Component | P50 | P95 | P99 |
| Embedding | 7.50 | 11.98 | 14.31 |
| Similarity (NN) | 0.93 | 3.81 | 4.52 |
| Early-leakage | 7.99 | 12.90 | 17.50 |
| Structural connectivity | 26.78 | 42.99 | 46.63 |
Table XV. Per-prompt latency breakdown of AdvGuard (ms).
Beyond Prompt Leakage
Generalization to Jailbreaking and Prompt Injection
We evaluate whether AdvGuard generalizes beyond prompt leakage to jailbreaking and prompt injection, using 1,050 queries from JailbreakingLLMs [67] and 1,094 from Open Prompt Injection [68].
Table XVI shows that AdvGuard remains effective across both attacks. It achieves the lowest jailbreaking FPR (10.11%) with 91.50% F1, and remains competitive on prompt injection with 95.00% F1, matching or outperforming LLMGuard while substantially outperforming PromptGuard.
| Defense | FPR(%) | Recall(%) | F1(%) |
| Jailbreaking Attacks | |||
| PromptGuard | 99.11 | 11.25 | 1.66 |
| LLMGuard | 12.70 | 93.65 | 89.30 |
| AdvGuard | 10.11 | 93.65 | 91.50 |
| Prompt-Injection Attacks | |||
| PromptGuard | 94.25 | 8.54 | 6.87 |
| LLMGuard | 4.31 | 94.31 | 95.58 |
| AdvGuard | 4.42 | 94.31 | 95.00 |
Table XVI. Detection performance against jailbreaking and prompt-injection-style queries.
References
- [1]OpenAI, “ChatGPT,” 2025. [Online]. Available: https://chat.openai.com/ [2] Anthropic, “Claude AI,” 2025. [Online]. Available: https://claude.ai/
- [3]Meta, “Llama-3.1-8b-instruct,” https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct, 2024, hugging Face model repository.DOI
- [4]OWASP Gen AI Security Project, “Llm0702:2025 system prompt leakage,” https://genai.owasp.org/llmrisk/llm07025-system-prompt-leakage/, OWASP Gen AI Security Project, 2025. [Online]. Available: https://genai.owasp.org/llmrisk/llm07025-system-prompt-leakage/
- [5]Snyk Learn, “System prompt leakage in llms,” https://learn.snyk.io/lesson/llm-system-prompt-leakage/?ecosystem=aiml, Snyk Limited, 2025. [Online]. Available: https://learn.snyk.io/lesson/llm-system-prompt-leakage/?ecosystem=aiml
- [6]Z. Sha and Y. Zhang, “Prompt stealing attacks against large language models,” *ArXiv*, vol. abs/2402.12959, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:267760338arxiv.org/abs/2402.12959
- [7]B. Hui, H. Yuan, N. Gong, P. Burlina, and Y. Cao, “PLeak: Prompt leaking attacks against large language model applications,” in *Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security*. Salt Lake City UT USA: ACM, Dec. 2024, pp. 3600–3614. [Online]. Available: https://dl.acm.org/doi/10.1145/3658644.3670370DOI
- [8]Y. Zhang, N. Carlini, and D. Ippolito, “Effective prompt extraction from language models,” 2024. [Online]. Available: https://arxiv.org/abs/2307.06865
- [9]Y. Nie, Z. Wang, Y. Xu, X. Wu, X. Zhao, N. D. Bastian, W. Guo, and D. Song, “LeakAgent: RL-based red-teaming agent for LLM privacy leakage,” in *Second Conference on Language Modeling*, 2025, arXiv:2412.05734. [Online]. Available: https://openreview.net/forum?id= a rX?arxiv.org/abs/2412.05734
- [10]N. Pfister, V. Volhejn, M. Knott, S. Arias, J. Bazińska, M. Bichurin, A. Commike, J. Darling, P. Dienes, M. Fiedler *et al.*, “Gandalf the red: Adaptive security for llms,” *arXiv preprint arXiv:2501.07927*, 2025.arxiv.org/abs/2501.07927
- [11]F. Perez and I. Ribeiro, “Ignore previous prompt: Attack techniques for language models,” 2022. [Online]. Available: https://arxiv.org/abs/2211.09527
- [12]X. Liu, Z. Yu, Y. Zhang, N. Zhang, and C. Xiao, “Automatic and universal prompt injection attacks against large language models,” 2024.arxiv.org/abs/2403.04957
- [13]D. Agarwal, A. Fabbri, B. Risher, P. Laban, S. Joty, and C.-S. Wu, “Prompt Leakage effect and mitigation strategies for multi-turn LLM Applications,” in *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track*, F. Dernoncourt, D. Preoțiuc-Pietro, and A. Shimorina, Eds. Miami, Florida, US: Association for Computational Linguistics, Nov. 2024, pp. 1255–1275. [Online]. Available: https://aclanthology.org/2024.emnlp-industry.94/arxiv.org/abs/2404.16251
- [14]meta-llama, “Prompt-guard-86m,” https://huggingface.co/meta-llama/Prompt-Guard-86M, 2024, hugging Face model repository.
- [15]meta-llama, “Llama prompt guard 2 86m,” https://huggingface.com/meta-llama/Llama-Prompt-Guard-2-86M, 2025, hugging Face model repository.
- [16]Protect AI, “Llm guard: The security toolkit for llm interactions,” https://github.com/protectai/llm-guard, 2025, gitHub repository, open-source toolkit for securing LLM applications.
- [17]Y. Zhang, N. Carlini, and D. Ippolito, “Effective Prompt Extraction from Language Models,” Aug. 2024, arXiv:2307.06865 [cs]. [Online]. Available: http://arxiv.org/abs/2307.06865
- [18]Y. Peng, L. Zhang, P. Lv, and K. Chen, “RepeatLeak: Leak Prompts from Repeating as Large Language Model Is a Good Repeater,” *Proceedings of the AAAI Conference on Artificial Intelligence*, vol. 39, no. 25, pp. 26 335–26 343, Apr. 2025. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/34832DOI
- [19]J. Wang, T. Yang, R. Xie, and B. Dhingra, “Raccoon: Prompt Extraction Benchmark of LLM-Integrated Applications,” in *Findings of the Association for Computational Linguistics: ACL 2024*, L.-W. Ku, A. Martins, and V. Srikumar, Eds. Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 13 349–13 365. [Online]. Available: https://aclanthology.org/2024.findings-acl.791/DOI
- [20]X. Liu, N. Xu, M. Chen, and C. Xiao, “AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models,” Mar. 2024, arXiv:2310.04451 [cs]. [Online]. Available: http://arxiv.org/abs/2310.04451
- [21]J. Yu, Y. Shao, H. Miao, and J. Shi, “PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMs,” Apr. 2025, arXiv:2409.14729 [cs]. [Online]. Available: http://arxiv.org/abs/2409.14729
- [22]P. S. F. B. Ribeiro, I. A. Brito, R. T. Sousa, F. B. Färber, J. S. Dollis, and A. R. Galvão Filho, “Proxy Barrier: A Hidden Repeater Layer Defense Against System Prompt Leakage and Jailbreaking,” in *Findings of the Association for Computational Linguistics: EMNLP 2025*, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng, Eds. Suzhou, China: Association for Computational Linguistics, Nov. 2025, pp. 9960–9975. [Online]. Available: https://aclanthology.org/2025.findings-emnlp.528/
- [23]Learn Prompting, “Filtering,” 2026. [Online]. Available: https://learnprompting.org/docs/prompt_hacking/defensive_measures/filtering
- [24]Z. Jiang, Z. Jin, and G. He, “PromptKeeper: Safeguarding System Prompts for LLMs,” in *Findings of the Association for Computational Linguistics: EMNLP 2025*, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng, Eds. Suzhou, China: Association for Computational Linguistics, Nov. 2025, pp. 2712–2728. [Online]. Available: https://aclanthology.org/2025.findings-emnlp.147/arxiv.org/abs/2412.13426
- [25]Learn Prompting, “Sandwich Defense,” 2026. [Online]. Available: https://learnprompting.org/docs/prompt_hacking/defensive_measures/sandwich_defense
- [26]D. Pape, S. Mavli, T. Eisenhofer, and L. Schönherr, “Prompt obfuscation for large language models,” in *34th USENIX Security Symposium (USENIX Security 25)*. Seattle, WA: USENIX Association, Aug. 2025, pp. 2323–2342. [Online]. Available: https://www.usenix.org/conference/usenixsecurity25/presentation/papearxiv.org/abs/2409.11026
- [27]Z. Zhuang, M-I. Nicolae, H-P. Wang, and M. Fritz, “ProxyPrompt: Securing System Prompts against Prompt Extraction Attacks,” May 2025, arXiv:2505.11459 [cs]. [Online]. Available: http://arxiv.org/abs/2505.11459
- [28]B. Cao, C. Li, Y. Cao, Y. Ge, T. Wang, and J. Chen, “You Can’t Steal Nothing: Mitigating Prompt Leakages in LLMs via System Vectors,” in *Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security*. New York, NY, USA: Association for Computing Machinery, Sep. 2025, pp. 4423–4437. [Online]. Available: https://doi.org/10.1145/3719027.3765124
- [29]F. Panebianco, S. Bonfanti, F. Trovò, and M. Carminati, “LeakSealer: A Semisupervised Defense for LLMs Against Prompt Injection and Leakage Attacks,” Aug. 2025, arXiv:2508.00602 [cs]. [Online]. Available: http://arxiv.org/abs/2508.00602
- [30]Z. Huang, K. Huang, L. Yin, B. He, H. Zhen, M. Yuan, and Z. Shao, “Attention-aware GNN-based input defense against multi-turn LLM jailbreak,” 2025. [Online]. Available: https://arxiv.org/abs/2507.07146
- [31]M. M. M. Miah, A. Anika, Z. Yu, and R. Huang, “TRACE: Trajectory-aware reasoning for multi-turn adversarial conversation evaluation,” 2026. [Online]. Available: https://arxiv.org/abs/2608.15594
- [32]P. He, X. Liu, J. Gao, and W. Chen, “Deberta: Decoding-enhanced bert with disentangled attention,” in *International Conference on Learning Representations*, 2021. [Online]. Available: https://openreview.net/forum?id=XPIAotutsD
- [33]L. Bergman, N. Cohen, and Y. Hosh en, “Deep nearest neighbor anomaly detection,” 2020. [Online]. Available: https://arxiv.org/abs/2002.10445
- [34]V. Chandola, A. Banerjee, and V. Kumar, “Anomaly detection: A survey,” *ACM computing surveys (CSUR)*, vol. 41, no. 3, p. 1–58, 2009.
- [35]A. Mahara and N. Rishe, “Methods and trends in detecting ai-generated images: A comprehensive review,” *Computer Science Review*, vol. 60, p. 100908, 2026. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1574013726000171arxiv.org/abs/2502.15176
- [36]National Institute of Standards and Technology (NIST), (2003) 1.3.5.17.1. grubbs’ test for outliers. National Institute of Standards and Technology. Gaithersburg, MD, USA. NIST/SEMATECH e-Handbook of Statistical Methods, accessed 14 January 2026. [Online]. Available: https://www.itl.nist.gov/div898/handbook/eda/section3/eda35h1.htm
- [37]A. Paszke, S. Gross, F. Massa, A. Lehrer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, C. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” 2019. [Online]. Available: https://arxiv.org/abs/1912.01703
- [38]H. Face, “Hugging face,” 2025. [Online]. Available: https://huggingface.com/
- [39]T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Ra ult, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. W. Tu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush, “Transformers: State-of-the-art natural language processing,” in *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Online: Association for Computational Linguistics, Oct. 2020, pp. 38–45. [Online]. Available: https://www.aclweb.org/anthology/2020.emnlp-demos.6
- [40]NVIDIA, P. Vingelmann, and F. H. Fitzek, “Cuda, release: 10.2.89,” 2020. [Online]. Available: https://developer.nvidia.com/cuda-toolkit
- [41]T. Dettmers, M. Lewis, S. Shleifer, and L. Zettlemoyer, “8-bit optimizers via block-wise quantization,” 9th International Conference on Learning Representations, ICLR, 2022.arxiv.org/abs/2110.02861
- [42]A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019.
- [43]T. Team, “The falcon 3 family of open models,” December 2024.
- [44]Q. Team, “Qwen2.5: A party of foundation models,” September 2024. [Online]. Available: https://qwenlm.github.io/blog/qwen2.5/
- [45]B. C. Das, M. H. Amini, and Y. Wu, “Security and privacy challenges of large language models: a survey,” ACM Computing Surveys, vol. 57, no. 6, Feb. 2025, number of pages: 39 Place: New York, NY, USA Publisher: Association for Computing Machinery tex.articleno: 152 tex.issue_date: June 2025. [Online]. Available: https://doi.org/10.1145/3712001
- [46]K. Zhao, L. Ki, K. Ding, N. Z. Gong, Y. Zhao, and Y. Dong, “A survey on model extraction attacks and defenses for large language models,” in Proceedings of the 31st ACM SIGKDD conference on knowledge discovery and data mining V.2, ser. Kdd ’25. Toronto ON, Canada: Association for Computing Machinery, 2025, pp. 6227–6236, number of pages: 10 tex.address: New York, NY, USA. [Online]. Available: https://doi.org/10.1145/3711896.3736573
- [47]V. Gkhan and H. Bahsi, “Enhancing security in llm applications: A performance evaluation of early detection systems,” 2025. [Online]. Available: https://arxiv.org/abs/2506.19109
- [48]nicolas. (2024, Mar) Language models at risk: Safeguarding ai with llm guard. Medium blog post discussing LLM Guard and security considerations for large language models. [Online]. Available: https://medium.com/@dataenthusiast.io/language-models-at-risk-safeguarding-ai-with-llm-guard-11a3e7923af5
- [49]LearnPrompting.org, “Sandwich defense,” 2026. [Online]. Available: https://chat-test.learnprompting.org/docs/prompt_hacking/defensive_measures/sandwich_defense
- [50]K. Kobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168, 2021.
- [51]R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts, “Stanford sentiment treebank v2 (SST-2),” 2013. [Online]. Available: https://huggingface.co/datasets/stanfordnlp/sst2
- [52]P. Rajpurkar, R. Jia, and P. Liang, “Know what you don’t know: Unanswerable questions for squad,” 2018. [Online]. Available: https://arxiv.org/abs/1806.03822
- [53]M. Conover, M. Hayes, A. Mathur, J. Xie, J. Wan, S. Shah, A. Ghodsi, P. Wendell, M. Zaharia, and R. Xin. (2023) Free dolly: Introducing the world’s first truly open instruction-tuned llm. [Online]. Available: https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm
- [54]L. Jiang, K. Rao, S. Han, A. Ettinger, F. Brahman, S. Kumar, N. Mireshghallah, X. Lu, M. Sap, Y. Choi, and N. Dziri, “Wildteaming at scale: From the in-the-wild jailbreaks to (adversarially) safer language models,” 2024. [Online]. Available: https://arxiv.org/abs/2406.18510
- [55]A. Köpf, Y. Kilcher, D. von Rütte, S. Anagnostidis, Z.-R. Tam, K. Stevens, A. Barhoum, N. M. Duc, O. Stanley, R. Nagyfi, S. ES, S. Suri, D. Glushkov, A. Dantuluri, A. Maguire, C. Schuhmann, H. Nguyen, and A. Mattick, “Openassistant conversations – democratizing large language model alignment,” 2023. [Online]. Available: https://arxiv.org/abs/2304.07327
- [56]N. Bian, H. Lin, Y. Lu, X. Han, L. Sun, and B. He, “Chatpalaca: A multi-turn dialogue corpus based on alpaca instructions,” https://github.com/cascip/ChatAlpaca, 2023.
- [57]W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng, “Wildchat: 1m chatGPT interaction logs in the wild,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=B18u7ZRlbMarxiv.org/abs/2405.01470
- [58]L. Jiang, K. Rao, S. Han, A. Ettinger, F. Brahman, S. Kumar, N. Mireshghallah, X. Lu, M. Sap, Y. Choi, and N. Dziri, “Wildteaming at scale: From the in-the-wild jailbreaks to (adversarially) safer language models,” 2024. [Online]. Available: https://arxiv.org/abs/2406.18510
- [59]A. Paulus, A. Zharnagambetov, C. Guo, B. Amos, and Y. Tian, “Advprompter: Fast adaptive adversarial prompting for llms,” 2025. [Online]. Available: https://arxiv.org/abs/2404.16873
- [60]Z. Lin, Z. Wang, Y. Tong, Y. Wang, Y. Guo, Y. Wang, and J. Shang, “Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation,” 2023.arxiv.org/abs/2310.17389
- [61]f, “awesome-chatgpt-prompts,” https://github.com/f/awesome-chatgpt-prompts, 2023, github repository, CC0-1.0 License, accessed January 9, 2026.
- [62]K. Ganesan, “ROUGE 2.0: Updated and improved measures for evaluation of summarization tasks,” 2018, arXiv: 1803.01937 [Online]. Available: https://arxiv.org/abs/1803.01937
- [63]M. Alizadeh, Z. Samei, D. Tsetenko, and F. Gilardi, “Simple prompt injection attacks can leak personal data observed by llm agents during task execution,” 2025. [Online]. Available: https://arxiv.org/abs/2506.01055
- [64]E. Saravia, “Prompt Engineering Guide,” https://github.com/dair-ai/Prompt-Engineering-Guide, 12 2022.
- [65]T. Sternak, D. Runje, D. Granoša, and C. Wang, “Automating prompt leakage attacks on large language models using agentic approach,” in 2025 MIPRO 48th ICT and Electronics Convention, 2025, pp. 99–104.arxiv.org/abs/2502.12630
- [66]H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Gosswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kouramann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, J. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom, “Llama 2: Open foundation and fine-tuned chat models,” 2023, arXiv: 2307.09288 [cs.CL]. [Online]. Available: https://arxiv.org/abs/2307.09288
- [67]P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, “Jailbreaking black box large language models in twenty queries,” 2023.arxiv.org/abs/2310.08419
- [68]Y. Liu, Y. Jia, R. Geng, J. Jia, and N. Z. Gong, “Formalizing and benchmarking prompt injection attacks and defenses,” in USENIX Security Symposium, 2024.arxiv.org/abs/2310.12815




