WorksComputer Science
Jailbreak Detection on Large Language Models via Contrastive Prototype Structures
Abstract
Jailbreak attempts and harmful prompts remain a challenge for the safe deployment of large language models. We study whether contrastively learned representations can support safety detection without training an additional downstream classifier. We propose Jailbreak Detection with Prototype Inference (JDPI), which uses binary safety labels to construct harmful and benign prototypes and makes predictions from relative prototype distances. JDPI uses only binary safety labels for encoder adaptation, prototype construction, and validation. In our setup, it uses fewer labeled examples than the classifier baseline (Method A) by avoiding the separate split used to train the downstream classifier. Across four benchmarks covering different input formats, contrastive fine-tuning improves JDPI's detection accuracy by 9.1-18.0 percentage points. Using the same encoder, JDPI achieves slightly higher ROC-AUC on one benchmark and stays within 0.017 of the baseline across all four. With dataset-specific adaptation, JDPI exceeds the released Llama Guard 3-8B baseline by 0.173-0.310 in ROC-AUC across the four benchmarks. In multi-turn replay on TomGibbs, JDPI uses accumulated user history as the conversation unfolds, achieving higher conversation-level accuracy than when it uses the current turn alone. Across 12 directed transfers, JDPI outperforms the classifier by an average of 1.59 percentage points in accuracy, demonstrating stronger cross-dataset robustness.
Introduction
For cybersecurity and safety, AI agents are typically deployed with policy constraints and safety guardrails. However, the increasing use of external tools and environments also introduces emerging security risks, including indirect prompt injection (Zhan et al., 2024), tool-integrated agent security [6], and broader agent attack and defense scenarios (Zhang et al., 2025a). Despite these safeguards, security vulnerabilities and failure modes remain [29, 3, 28, 9] and so-called "jailbreaking" techniques are also commonly used to bypass safety alignment (Wei et al., 2023; Zou et al., 2023), which can pose risks to current AI-agent applications, starting from LLM use cases.
Many safety guards have been constructed at different levels. One commonly used tool is a classifier, which usually formulates safety detection as a classification problem [17, 32], and the workflow can be: first define or construct a series of policy-violating cases, and then train the classifier to detect whether an input or output is intended to trigger harmful content.
The problem arises from this setup itself: first, the taxonomies are usually designed according to predefined safety policies or risk categories [28, 27] and may not fully cover all cases. Second, attention may be directed toward distinguishing detailed categories rather than the underlying intent of the context itself. Third, the boundaries between detailed categories can be fuzzy and may not correspond directly to the structure of the representation space [17, 32, 8]. This motivates the question of whether harmful and benign inputs also form exploitable geometric structures in the learned representation space. If such structure exists, it raises two related questions: whether representation-aware structural features can improve conventional classifiers, and whether the classification process can be replaced by direct geometric inference based on the learned representation geometry.
We propose Jailbreak Detection with Prototype Inference (JDPI). We use contrastive learning to improve the separation between harmful and benign inputs in the representation space [11, 24, 4]. Using binary safety labels, JDPI constructs harmful and benign prototypes and classifies inputs by their relative distances to these prototypes. This design eliminates the need for a separate labeled split to train a downstream classifier.
Methods B–D test whether adding explicit clustering-derived features improves classifier-based detection. Because the encoder is adapted to labeled data from a particular source, we test cross-dataset robustness by transferring the detector to other datasets without target-dataset adaptation. Our four benchmarks cover prompt-level, question–answer, and multi-turn inputs and include adversarial examples. We assess whether retaining user history improves multi-turn detection on TomGibbs.
Related Work
A wide range of defense mechanisms has been proposed to improve LLM safety. Some methods enhance safety by prioritizing higher-level instructions over lower-level instructions during inference (Wallace et al., 2024), while others train models to reason about explicit safety specifications before generating responses [13]. Robustness has also been improved through adversarial training and the incorporation of adversarial objectives [34]. Additional approaches intervene directly at the representation or activation level through mechanisms such as representation rerouting (Zou et al., 2024). Traditional defense strategies, including input detection, pre-processing, and adversarial training, have also been systematically evaluated [18]. Input perturbation methods have also been explored, such as SmoothLLM [31]. Other work investigates internal hidden-layer representations and performs layer-wise safety analysis followed by binary classification [23]. Classifier-based detection is also widely used in safety guard systems [17, 32]. Embedding-based jailbreak detectors have combined pretrained sentence representations with conventional classifiers [8]. Despite these advances, many widely used guard models are constructed around predefined safety policies, risk categories, or labeled safety datasets. However, generalization to emerging attack strategies and previously unseen harmful behaviors remains a challenge.
Furthermore, the opposite failure mode, namely over-refusal of benign inputs, has received increasing attention and has been studied explicitly in recent work [5].
To determine whether latent representations contain meaningful safety signals, several studies have investigated safety-related information encoded within internal model states [2, 22]. JBShield identifies activated safety-related concepts in internal representations and leverages them for jailbreak detection and mitigation (Zhang et al., 2025b). SIREN further exploits internal safety-relevant features distributed across model layers for harmful-content detection [21]. RCS similarly performs jailbreak detection through representational contrastive scoring in latent space [15]. Collectively, these studies suggest that internal representations contain rich geometric and semantic information that can be effectively leveraged for safety assessment.
A growing number of benchmarks evaluate jailbreak and safety-related behavior using large collections of adversarial and red-teaming-derived inputs, covering both harmful and benign requests. Many of these benchmarks provide comprehensive collections of both harmful and benign prompts, Shen et al. [33] collect and analyze in-the-wild jailbreak prompts, characterizing their attack strategies and evaluating their effectiveness against LLMs and external safeguards. WildJailbreak and WildGuardMix contain vanilla and adversarial prompts covering harmful and benign requests, including synthetic adversarial prompts generated using tactics mined from in-the-wild interactions [20, 14]. BeaverTails provides question–answer pairs derived from red-teaming-related prompts designed to elicit unsafe responses, with safety annotations defined over the complete QA pair [19]. Gibbs et al. (2024) introduce paired single-turn and multi-turn cipher-based jailbreak inputs, alongside benign controls, and examine how prompting structure and conversational context affect guardrail performance. Evaluation frameworks such as StrongREJECT further emphasize that jailbreak assessment should account for whether harmful behavior is actually elicited rather than only whether an attack prompt is present [37].
Contrastive learning [4] improves representation structure by increasing the separation between semantically different samples. This principle has been applied to sentence representation learning [11] and supervised class-aware representation learning [24]. Sentence-transformer models provide fixed-dimensional sentence representations that are suitable for efficient similarity and distance computation [30], extending pretrained Transformer encoders such as BERT [7]. In our work, an MPNet-based sentence-transformer encoder is used to obtain semantic representations for subsequent structure analysis [36].
Prototypical Networks introduced prototype-based few-shot classification, where samples are classified according to distances from representative class prototypes in a learned metric space [35]. Prototype-oriented Conditional Transport (Tanwisuth et al., 2021) addresses source–target alignment by aligning target features with class prototypes. Recent safety detection methods have similarly explored geometric signals in representation space. RCS distinguishes benign and malicious inputs through contrastive scoring over internal model representations [15], while T3 models the distribution of acceptable prompts in semantic space and formulates safety detection as out-of-distribution detection without requiring harmful training examples [10]. The Semantic Codebooks approach compares multilingual query embeddings against a fixed codebook of jailbreak prompts, providing a training-free external guardrail based on semantic similarity [1].
Different from these approaches, our focus is not to construct a stronger embedding model or to model only a predefined safe or malicious reference distribution. Instead, we investigate whether geometric structure learned under binary harmful benign supervision can reduce the dependence on an additional downstream classifier. Using the same contrastively adapted encoder, we compare embedding-based classification and variants augmented with clustering-derived features against direct inference from harmful and benign prototype sets, without requiring fine-grained risk annotations. But how finer cluster structure and its refinement affect binary harmful–benign detection is still examined.
Method
The proposed framework consists of two main stages. First, contrastive representation learning is employed to increase the separation between harmful and benign inputs in the embedding space, thereby inducing a safety-aware geometric structure. Second, the resulting representation space is exploited for downstream detection through a series of increasingly structure-aware approaches, ranging from conventional classifier-based methods to direct prototype-based inference. Figure 1 provides an overview of the framework.

Figure 1. Overview of the proposed framework. Contrastive fine-tuning first learns safety-aware representations that increase the separation between harmful and benign inputs. Method A serves as the embedding-based classifier baseline. Methods B–D progressively incorporate richer structural information, whereas JDPI performs direct prototype-based inference without relying on an additional downstream classifier.
Problem formulation
Given a labeled dataset with , where denotes benign, we encode each input as [36, 30].
Contrastive Representation Learning
We fine-tune the sentence encoder through contrastive learning to improve harmful–benign separation [24, 11, 25] (Tunstall et al. 2022). For each training pair , we compute the cosine similarity . In the first phase, harmful–harmful and benign–benign pairs are treated as positive pairs and assigned label 1, while harmful–benign pairs are treated as negative pairs and assigned label 0 [24] (Tunstall et al. 2022). The encoder is optimized using CoSENTLoss [16]:
where and denote the sets of positive and negative training pairs in a mini-batch, respectively, and controls the scale of pairwise similarity differences.
We also evaluate an iterative refinement stage for harmful subclusters, described in Appendix § 12.
Structure-Aware detection method
Methods A–D examine whether clustering-derived structural features improve supervised detection beyond using the adapted embeddings alone.
Method A is the embedding-based baseline [8]. It predicts harmfulness directly from the embedding: . The decision threshold is selected on the validation split.
Method B augments the embedding with distances to harmful cluster centroids. We apply -means to harmful examples in the structure split [26], obtaining centroids . The augmented representation is
This comparison tests whether harmful-cluster geometry adds predictive information to the embedding.
Method C adds features describing regions with elevated false-negative rates. For each harmful cluster , is estimated from Method B predictions on the structure split. Clusters whose false-negative rate exceeds the overall harmful-class rate and whose size is at least form the high-FN set . If no cluster satisfies the high-FN criterion, we use the two clusters with the highest false-negative rates as . The remaining harmful clusters form . If this set is empty, is also used as the reference set for the low-FN features.
For an input representation, and denote its minimum distances to these two sets, while and are their aggregated distance-based soft-membership scores. The indicator records whether the nearest harmful centroid belongs to . These quantities form
Method C uses .
Method D incorporates benign structure and cross-class relationships through three feature groups. The first, , contains distance and soft-membership statistics for high-FN harmful regions, all harmful clusters, and benign clusters, together with their differences and ratios. For the second group, , each high-FN harmful centroid is paired with its nearest benign centroid in Euclidean distance. Features include the input’s distances to both centroids, their difference and ratio, and the difference between distance-based similarity scores computed using exponential decay.
The third group, , contains nine features derived from Methods A–C: each method’s harmful probability , distance from the decision midpoint , and uncertainty score . Training features use five-fold out-of-fold predictions; validation and test features use the corresponding models fitted on the training split. The final representation is
Jailbreak Detection with Prototype Inference
We refer to prototype-based inference (Method E) as JDPI. JDPI examines whether the geometry of the same representation space can instead support detection directly. It makes the harmful–benign decision from relative distances to harmful and benign prototypes, without training an additional downstream classifier.
Following prototype-based metric learning [35], we independently cluster harmful and benign examples in the structure split to obtain and . Both sets are constructed using binary safety labels.
For an input representation , its distances to the two prototype sets are
where denotes cosine distance. A two-class softmax converts these distances into a harmfulness score:
The temperature controls the sharpness of the score. An input is classified as harmful when . Both and are selected on the validation split.
Figure 2 illustrates how contrastive fine-tuning shapes the embedding space for direct prototype inference.

Figure 2. Overview of JDPI. Contrastive fine-tuning improves harmful–benign separation. JDPI constructs prototypes from each class in the structure split and scores new inputs using the cosine distances and to the nearest prototypes of the two classes. The point clouds are schematic.
Experiments
Datasets and Splits
We evaluate safety detection on four benchmarks using individual prompts, question–answer pairs, and multi-turn conversations. The evaluation includes adversarial prompts from WildGuardMix and WildJailbreak and multi-turn jailbreak attempts from TomGibbs. Class proportions below refer to our experimental test splits. We abbreviate WildGuardMix, WildJailbreak, BeaverTailsQA, and TomGibbs as WGM, WJB, BTQA, and TG, respectively. We derive the binary safety labels from the released annotations or categories of each benchmark and match the unit of evaluation to the unit those labels describe.
WildGuardMix. WildGuardMix (Han et al., [14]) contains vanilla and adversarial prompts with harmful and benign labels. We use the prompt text and its prompt-harmfulness annotation. The test split contains 53.10% harmful and 46.90% benign prompts.
WildJailbreak. WildJailbreak (Jiang et al., [20]) contains vanilla and adversarial versions of harmful and benign requests, including benign prompts that resemble attacks but have no harmful intent. Our classifier-training split combines 30,000 vanilla and 6,000 adversarial examples; the structure, validation, and test splits contain adversarial examples. The test split contains 51.77% harmful and 48.23% benign prompts.
BeaverTailsQA. BeaverTails (Ji et al., [19]) provides a safety label for each question–answer pair. The label depends on the answer in the context of the question, so the same question can receive different labels when paired with different answers. We therefore concatenate the user question and the assistant response as a single input. Unsafe pairs are mapped to the harmful class and safe pairs to the benign class, accounting for 55.42% and 44.58% of the test split, respectively.
TomGibbs. We use the input-only Multi-Turn Jailbreak Attack Datasets of Gibbs et al. [12]. Each example provides a complete single-turn prompt and a corresponding sequence of user turns. The attacks use word-substitution ciphers and distribute their instructions across turns.
The dataset includes harmful, completely benign, and semi-benign examples. Semi-benign examples contain harmful or toxic vocabulary but pursue benign goals; both benign subsets are assigned label 0. These three groups account for 63.39%, 18.75%, and 17.86% of our test split, respectively. For the separate static evaluation, we use the corresponding complete prompts from the same held-out examples.
Data splits. We divide each dataset into four splits. The structure split is used to fine-tune the encoder and construct clusters and class prototypes. The training split supplies the labeled examples for fitting the downstream classifiers. We use the validation split to select detector settings, including decision thresholds. The test split is held out throughout model development and used only for final evaluation.
For WildGuardMix and BeaverTailsQA, we pool the released training and test partitions, sample 60,000 examples stratified by the binary label, and construct new experimental splits. For TomGibbs, we assign entire goal groups to splits and downsample the training and structure splits to balance the binary labels. Table 1 summarizes the split sizes.
| Dataset | Train | Struct. | Val. | Test |
| WildGuardMix | 36,000 | 7,919 | 7,927 | 8,154 |
| WildJailbreak | 36,000 | 6,000 | 5,994 | 12,006 |
| BeaverTailsQA | 36,000 | 7,919 | 7,927 | 8,154 |
| TomGibbs | 2,880 | 624 | 864 | 896 |
Table 1. Experimental split sizes. Struct. denotes the structure split used for representation adaptation and prototype construction; Val. denotes validation.
Baselines and Evaluation Metrics
We compare JDPI with an embedding-based classifier (Method A) using the same encoder. Methods B–D provide additional comparisons using structure-derived features. Llama Guard 3-8B serves as an external baseline without task-specific fine-tuning. We report accuracy and ROC-AUC, supplemented by F1, false-negative rate, and false-positive rate. We also report silhouette scores to characterize harmful–benign separation and clustering structure within the harmful class.
By avoiding the separate classifier-training split, JDPI uses 65.9–75.0% fewer labeled examples for model development than classifier A across the four benchmarks (Table 8). In cross-dataset evaluation, the encoder, classifier or prototypes, and selected parameters are obtained from the source dataset and applied to the target test split without target-dataset adaptation.
| Dataset | A | JDPI | Saved | Reduction |
| WildGuardMix | 51,846 | 15,846 | 36,000 | 69.4% |
| WildJailbreak | 47,994 | 11,994 | 36,000 | 75.0% |
| BeaverTailsQA | 51,846 | 15,846 | 36,000 | 69.4% |
| TomGibbs | 4,368 | 1,488 | 2,880 | 65.9% |
Table 8. Labeled examples used for model development. Both methods use the structure and validation splits. Classifier A also uses a separate classifier-training split. Counts are based on dataset rows and exclude the test split. Reduction is relative to classifier A.
For fixed multi-turn replay on TomGibbs, we compare scoring the current user turn alone with scoring the accumulated user history up to and including that turn. Each method uses the detector and threshold selected in the complete-prompt setting throughout replay. Following the any-turn blocking criterion used by Gibbs et al. [12], we count a conversation as blocked if the detector flags the input at any evaluated user turn.
Implementation Details
We use a random seed of 42. The sentence encoder is all-mpnet-base-v2 from Sentence Transformers, with an encoding batch size of 64 for downstream evaluation. Contrastive fine-tuning uses CoSENTLoss. Phase 1 uses five epochs and 200 warm-up steps. Phase 2 runs for at most five iterations, with two epochs and 50 warm-up steps per iteration. Appendix C reports the encoder training configurations and contrastive pair counts.
For clustering, is selected from using the cosine-distance silhouette score. K-means uses 10 initializations. The maximum iteration counts for classifiers A–D are 3,000, 4,000, 5,000, and 6,000, respectively. Methods A–C use balanced class weights. Method D constructs its training prediction features through five-fold out-of-fold prediction.
For Methods A–C and JDPI, decision thresholds are selected on the validation split from . JDPI additionally selects its temperature from . Threshold and temperature selection for JDPI maximizes validation accuracy. Methods B–D use Euclidean cluster-distance features, while JDPI uses cosine distances to prototypes.
For the external baseline, we use Llama-Guard-3-8B with greedy decoding and at most 10 generated tokens. Outputs beginning with unsafe are mapped to the harmful class; remaining outputs are mapped to the benign class. Continuous unsafe scores are used for ROC-AUC calculation, separately from these binary decisions.
Results and Analysis
Main Detection Results
Table 2 compares direct embedding classification (A), classifiers augmented with structural features (B–D), and JDPI. Methods B–D provide modest and dataset-dependent improvements over A, with no single structural variant consistently performing best. These results suggest that much of the discriminative information is already accessible to the classifier through the adapted embeddings. JDPI examines a complementary question: whether this information can support detection through prototype distances without training an additional downstream classifier.
| Metric | A | B | C | D | JDPI |
| WildGuardMix | |||||
| Acc. | 94.43 | 94.55 | 94.49 | 94.69 | 94.13 |
| AUC | .9874 | .9876 | .9877 | .9878 | .9860 |
| WildJailbreak | |||||
| Acc. | 95.54 | 95.49 | 95.52 | 95.36 | 95.24 |
| AUC | .9906 | .9909 | .9907 | .9908 | .9900 |
| BeaverTails QA | |||||
| Acc. | 82.24 | 82.02 | 82.06 | 82.23 | 80.48 |
| AUC | .8938 | .8943 | .8949 | .8954 | .8772 |
Table 2. In-domain results of Methods A–E using the same fine-tuned encoder within each dataset. Accuracy is reported as a percentage. Bold indicates the highest value in each row.
JDPI closely matches the baseline on WildGuardMix and WildJailbreak. Compared with classifier A, JDPI’s accuracy is 0.30 percentage points lower on both WildGuardMix and WildJailbreak. Its ROC-AUC is lower by 0.0014 on WildGuardMix and 0.0006 on WildJailbreak. After contrastive fine-tuning improves the separation between harmful and benign examples in the embedding space, JDPI uses prototype distances to achieve performance close to that of classifier A on both benchmarks.
On BeaverTailsQA, classifier A exceeds JDPI by 1.76 percentage points in accuracy and by 0.0166 in ROC-AUC. We report the TomGibbs conversation-level results separately under the any-turn rule. On TomGibbs, classifier A achieves higher accuracy in the complete-prompt evaluation reported in the appendix. In the any-turn replay, JDPI achieves higher conversation-level accuracy when both methods use accumulated user history.
The central finding is that an additional downstream classifier is not always necessary to obtain competitive detection performance after contrastive adaptation. The remaining gaps also identify settings in which a learned decision function captures useful information beyond nearest-prototype distances. Figure 4 in the appendix shows which test examples JDPI and classifier A classify correctly.
Figure 4. Error composition of Methods A and E across four benchmarks, separated by the true benign and harmful classes. Bars show the percentages of examples for which only A is correct, only E is correct, or both methods are wrong. Percentages are computed within each true class; examples correctly classified by both methods are omitted.
Fixed Multi-Turn Replay Evaluation
We also evaluate detection as multi-turn conversations unfold. In TomGibbs, we compare scoring each user turn in isolation with scoring it using the accumulated user history (Table 3).
| Input | Method | Acc.↑ | FNR↓ | FPR↓ |
| Current | A | 63.39 | 0.00 | 100.00 |
| JDPI | 77.23 | 6.34 | 51.22 | |
| History | A | 63.39 | 0.00 | 100.00 |
| JDPI | 89.29 | 4.23 | 21.95 |
Table 3. Any-turn detection on TomGibbs (%).
Under the any-turn rule, JDPI improves conversation-level accuracy over classifier A by 13.8 percentage points with current-turn inputs and by 25.9 points with accumulated user history. With accumulated history, JDPI lowers the false-positive rate by 78.05 percentage points while raising the false-negative rate by 4.23 points. At its fixed threshold, classifier A flags every conversation in both input settings, yielding zero FNR but 100% FPR. The reduction in false alarms more than offsets the increase in missed harmful conversations.
For JDPI, using accumulated user history increases conversation-level accuracy by 12.06 percentage points and reduces FPR and FNR by 29.27 and 2.11 percentage points, respectively, compared with using only the current turn. Under this fixed replay protocol, accumulated context therefore improves both harmful-conversation detection and benign-conversation acceptance for JDPI.
Appendix A.1 reports the effect of contrastive fine-tuning.
Comparison with External Safety-Guard
We compare JDPI with Llama Guard 3-8B. JDPI uses an encoder and prototypes adapted to each benchmark, whereas Llama Guard uses its released weights without target-dataset fine-tuning. Both methods are evaluated on the same test splits, with ROC-AUC computed from continuous scores.
As shown in Table 4, JDPI achieves higher accuracy and ROC-AUC on all four benchmarks. Accuracy gains range from 9.40 to 24.00 percentage points, while ROC-AUC gains range from 0.1729 to 0.3097. The largest accuracy gain occurs on WildJailbreak, where accuracy increases from 71.24% to 95.24%.
| Dataset | Accuracy (%) | ROC-AUC | ||
| LG3 | JDPI | LG3 | JDPI | |
| WildGuardMix | 84.04 | 94.13 | 0.6781 | 0.9860 |
| WildJailbreak | 71.24 | 95.24 | 0.7284 | 0.9900 |
| BeaverTailsQA | 71.08 | 80.48 | 0.7043 | 0.8772 |
| TomGibbs | 77.90 | 89.73 | 0.6540 | 0.9637 |
Table 4. Comparison with Llama Guard 3-8B (LG3).
This comparison demonstrates the performance obtainable with dataset-adapted representations and direct prototype inference relative to the released guard. It complements the A/JDPI comparison, which holds the encoder fixed while comparing classifier-based and prototype-based decisions.
Cross-Dataset Generalization
We evaluate cross-dataset transfer to examine whether source-adapted detectors generalize to other datasets. Figure 3 compares Methods A and JDPI across all 12 directed transfers. Rows represent source datasets and columns represent target datasets. The sign convention is chosen so that positive values favor JDPI in every panel.
Figure 3. Cross-dataset differences between Methods A and JDPI. Accuracy and ROC-AUC show JDPI minus A; FNR and FPR show A minus JDPI. Positive values favor JDPI in every panel. Rows denote sources and columns denote targets. ROC-AUC differences are multiplied by 100.
JDPI achieves higher accuracy in eight of the 12 transfer directions. Across all 12 directions, weighted equally, JDPI exceeds classifier A by 1.59 percentage points in mean accuracy and by 0.0055 in mean ROC-AUC. These results show that prototype inference can retain competitive cross-dataset performance without training an additional downstream classifier.
The gains depend on the source–target pair. With TomGibbs as the source, JDPI exceeds classifier A in accuracy by 9.59 percentage points on WildGuardMix and 5.74 points on BeaverTailsQA. JDPI reduces mean FPR from 47.31% to 32.63%, while mean FNR increases from 16.74% to 26.05%. Appendix § 10 reports the full results.
Conclusion
This paper investigates how the structure of learned representations can support jailbreak and harmful-input detection. We examine two related questions: whether explicit structural information improves a conventional embedding-based classifier, and whether the learned geometry can support detection without training an additional downstream classifier.
Contrastive fine-tuning improves the separation between harmful and benign examples in the embedding space, providing a clearer structure for constructing class prototypes. It also improves JDPI’s detection accuracy across all four benchmarks. The clustering-derived features in Methods B–D yield small, dataset-dependent gains in the in-domain experiments. This suggests that classifier A already captures much of the signal needed for detection directly from the fine-tuned embeddings. On WildGuardMix, WildJailbreak, and BeaverTailsQA, JDPI achieves accuracy close to that of classifier A while saving the labeled data and computation required to train a downstream classifier. With dataset-specific adaptation, JDPI exceeds the released Llama Guard 3-8B baseline in accuracy and ROC-AUC on all four benchmarks. Together, these results support representation adaptation as a basis for effective detection with a simple downstream decision rule.
Across the 12 directed transfers, JDPI achieves higher mean accuracy than classifier A, suggesting greater robustness to dataset shift on this metric. In multi-turn replay on TomGibbs, retaining user history improves JDPI’s conversation-level accuracy, which exceeds that of classifier A under the same input setting. Together, these results show that contrastively adapted representations can support competitive prototype-based detection. JDPI uses binary safety labels for encoder adaptation, prototype construction, and validation. JDPI avoids the separate labeled split used to train classifier A and shows greater robustness to dataset shift as measured by mean accuracy across the evaluated transfers.
Limitations
Although JDPI removes the need for a trained downstream classifier, binary labels are still required for contrastive representation learning, class-specific prototype construction, and validation-based parameter selection. In our experimental setup, it avoids the additional labeled training split used by the classifier, but does not eliminate dependence on labeled data.
How larger fine-tuning datasets or greater encoder training budgets affect the relative performance of JDPI and the classifier baseline remains an open question. Although the decision threshold and temperature of JDPI are selected on the validation split, the search covers a finite set of candidate values. Sensitivity to these parameters, training duration, clustering configurations, and distance metrics has not been fully characterized.
Cross-dataset experiments show that transfer performance varies across source–target pairs and involves different false-positive and false-negative trade-offs.
Acknowledgments
References
Additional In-Domain Metrics
Table 5 reports the harmful-class F1, false-negative rate (FNR), and false-positive rate (FPR) for the in-domain experiments.
| Metric | A | B | C | D | JDPI |
| WildGuardMix | |||||
| F1 | 94.71 | 94.85 | 94.75 | 95.01 | 94.41 |
| FNR | 6.07 | 5.57 | 6.44 | 4.78 | 6.63 |
| FPR | 4.99 | 5.31 | 4.45 | 5.91 | 5.02 |
| WildJailbreak | |||||
| F1 | 95.68 | 95.67 | 95.69 | 95.56 | 95.39 |
| FNR | 4.51 | 3.75 | 3.88 | 3.65 | 4.68 |
| FPR | 4.42 | 5.34 | 5.13 | 5.70 | 4.85 |
| BeaverTailsQA | |||||
| F1 | 83.56 | 83.67 | 83.71 | 83.81 | 81.37 |
| FNR | 18.57 | 16.88 | 16.80 | 16.99 | 23.08 |
| FPR | 16.75 | 19.34 | 19.37 | 18.73 | 15.10 |
| TomGibbs | |||||
| F1 | 96.39 | 96.56 | 96.38 | 96.49 | 91.76 |
| FNR | 1.23 | 1.23 | 1.58 | 0.70 | 9.86 |
| FPR | 10.67 | 10.06 | 10.06 | 11.28 | 10.98 |
Table 5. Additional in-domain metrics for Methods A–E. All values are percentages.
Figure 4 further shows that the two methods make different errors. Examples correctly classified only by JDPI demonstrate that prototype inference does not simply reproduce the classifier’s predictions. Conversely, examples correctly classified only by A show where the learned decision function remains useful.
Effect of Contrastive Fine-Tuning
We compare the original encoder with the fine-tuned encoder used for each benchmark’s main results to assess the effect of contrastive adaptation. Appendix § 9 reports additional in-domain metrics. The subsequent analyses focus on Method A and JDPI to assess how much detection performance direct prototype inference retains without a trained downstream classifier.
The results are summarized in Figure 5. For TomGibbs, the comparison uses the final Phase 2 checkpoint and therefore reflects the combined effect of both training phases.
Figure 5. Effect of contrastive fine-tuning on Methods A and JDPI. Open and filled markers denote the original and fine-tuned encoders, respectively.
As shown in Figure 5, the fine-tuned embedding model benefits not only prototype-based detection but also produces a clear improvement for the baseline classifier. More importantly, after contrastive learning, Methods A and E achieve comparable accuracy, particularly on WildJailbreak and WildGuardMix.
Contrastive fine-tuning improves JDPI across all four benchmarks. Its accuracy increases by 14.81 percentage points on WildGuardMix, 16.54 on WildJailbreak, 9.14 on BeaverTailsQA, and 17.97 on TomGibbs. Classifier A also improves, with accuracy gains ranging from 3.75 to 12.59 percentage points.
The larger gains for JDPI suggest that contrastive adaptation makes the representation geometry more useful for direct prototype-based decisions. In particular, E approaches A on WildGuardMix and WildJailbreak after fine-tuning. The resulting performance therefore depends on the combination of representation learning and prototype inference, rather than on prototype construction alone.
Complete Cross-Dataset Results
Tables 6 and 7 report results for classifier A, variants B–D, and JDPI across all 12 directed transfers. WGM, WJB, BTQA, and TG denote WildGuardMix, WildJailbreak, BeaverTailsQA, and TomGibbs, respectively. Each mean gives equal weight to each transfer direction.
The two transfer directions between WildGuardMix and WildJailbreak contain shared prompt texts between source development data and target test data. Excluding both directions, JDPI achieves higher accuracy than classifier A in 7 of the remaining 10 directions, with a mean accuracy advantage of 1.92 percentage points across all 10 directions.
Implementation Details
Encoder adaptation. Table 9 reports the settings of the saved encoders. The training batch size was 8 in all runs (per device for WildJailbreak). WGM, WJB, BTQA, and TG denote WildGuardMix, WildJailbreak, BeaverTailsQA, and TomGibbs, respectively.
| Data | Run | Pairs | Epochs | Warm-up | LR |
| WGM | Phase 1 | ∼16,000 | 5 | 200 | |
| BTQA | Phase 1 | ∼16,000 | 5 | 200 | |
| TG | Phase 1 | 16,000 | 5 | 200 | |
| WJB | Phase 1 | 16,000 | 5 | 200 | |
| WGM | Phase 2 | ∼6,000 | 2 | 50 | |
| BTQA | Phase 2 | ∼6,000 | 2 | 50 | |
| TG | Phase 2 | 5,000 | 2 | 50 |
Table 9. Encoder training settings. Phase 2 values are per refinement iteration. Warm-up is measured in optimizer steps.
Method D configuration. We fixed the benign class weight at 1.0 and tested harmful class weights of 1.0, 1.2, and 1.5. For each weight, we searched thresholds in on the validation split. Among thresholds with precision of at least 0.70 and an FPR no more than one percentage point above Method B’s reference FPR, we selected the threshold with the highest recall. If no threshold met both constraints, we selected by recall over the full grid. Recall ties were broken by F1, balanced accuracy, and accuracy, in that order.
We computed Method B’s reference FPR and ranked the candidate class weights on the validation split. The ranking prioritized fewer false negatives, followed by lower FPR.
Ablation Study: Harmful-Subcluster Refinement
In the second phase, we further refined the internal structure of the harmful representations through an iterative clustering procedure. At each iteration, harmful samples are re-encoded using the current encoder and partitioned using -means clustering. The number of clusters is selected from by maximizing the silhouette score computed using cosine distance. The resulting cluster assignments are used as pseudo-labels to construct new contrastive pairs. Harmful samples belonging to the same cluster form positive pairs, while harmful samples from different clusters form negative pairs. Harmful–benign pairs are additionally retained as negative pairs to preserve the global separation between the two classes.
Using these newly constructed pairs, the encoder is further optimized with the same CoSENTLoss. The updated encoder is then used to re-encode harmful samples, and the clustering process is repeated for up to a predefined maximum number of iterations. By progressively refining substructures within the harmful class, this stage allows us to examine whether finer cluster separation improves binary harmful–benign detection.
| Source → Target | Accuracy (%) A | Accuracy (%) B | Accuracy (%) C | Accuracy (%) D | Accuracy (%) JDPI | ROC-AUC A | ROC-AUC B | ROC-AUC C | ROC-AUC D | ROC-AUC JDPI |
| WGM → WJB | 92.48 | 92.69 | 92.07 | 93.04 | 91.97 | 0.9812 | 0.9813 | 0.9816 | 0.9820 | 0.9797 |
| WGM → BTQA | 68.26 | 67.88 | 67.98 | 67.82 | 68.54 | 0.7406 | 0.7407 | 0.7393 | 0.7394 | 0.7439 |
| WGM → TG | 77.57 | 78.79 | 78.35 | 78.68 | 77.46 | 0.8409 | 0.8461 | 0.8422 | 0.8422 | 0.8307 |
| WJB → WGM | 90.58 | 90.43 | 90.79 | 89.88 | 90.91 | 0.9699 | 0.9708 | 0.9705 | 0.9690 | 0.9685 |
| WJB → BTQA | 58.30 | 58.57 | 58.61 | 57.89 | 61.09 | 0.5952 | 0.5982 | 0.6063 | 0.5950 | 0.6345 |
| WJB → TG | 72.88 | 72.10 | 72.43 | 70.76 | 74.00 | 0.7689 | 0.7669 | 0.7660 | 0.7605 | 0.7784 |
| BTQA → WGM | 74.27 | 73.68 | 74.86 | 71.31 | 73.34 | 0.8030 | 0.8180 | 0.8185 | 0.8030 | 0.8138 |
| BTQA → WJB | 68.38 | 69.80 | 69.81 | 69.69 | 63.65 | 0.7620 | 0.7642 | 0.7647 | 0.7657 | 0.7276 |
| BTQA → TG | 66.63 | 66.07 | 66.74 | 65.62 | 70.42 | 0.6888 | 0.6910 | 0.6937 | 0.6919 | 0.7227 |
| TG → WGM | 57.26 | 57.65 | 58.83 | 57.59 | 66.85 | 0.6896 | 0.6911 | 0.7011 | 0.6787 | 0.7135 |
| TG → WJB | 59.65 | 59.67 | 60.21 | 59.31 | 61.33 | 0.6660 | 0.6618 | 0.6631 | 0.6612 | 0.6541 |
| TG → BTQA | 57.75 | 57.89 | 58.67 | 57.86 | 63.49 | 0.6604 | 0.6606 | 0.6628 | 0.6544 | 0.6655 |
| Mean | 70.33 | 70.44 | 70.78 | 69.96 | 71.92 | 0.7639 | 0.7659 | 0.7675 | 0.7619 | 0.7694 |
Table 6. Cross-dataset accuracy and ROC-AUC for all five methods.
| Source → Target | FNR (%) A | FNR (%) B | FNR (%) C | FNR (%) D | FNR (%) JDPI | FPR (%) A | FPR (%) B | FPR (%) C | FPR (%) D | FPR (%) JDPI |
| WGM → WJB | 11.07 | 10.27 | 12.05 | 9.03 | 11.73 | 3.71 | 4.14 | 3.51 | 4.75 | 4.06 |
| WGM → BTQA | 20.40 | 19.30 | 21.46 | 18.57 | 22.33 | 45.83 | 48.06 | 45.14 | 49.11 | 42.81 |
| WGM → TG | 11.44 | 8.45 | 11.27 | 7.04 | 11.27 | 41.46 | 43.29 | 39.63 | 46.04 | 42.07 |
| WJB → WGM | 6.49 | 5.80 | 6.19 | 5.17 | 5.77 | 12.74 | 13.83 | 12.63 | 15.72 | 12.84 |
| WJB → BTQA | 37.20 | 33.97 | 37.09 | 31.87 | 34.94 | 47.29 | 50.70 | 46.74 | 54.86 | 43.85 |
| WJB → TG | 15.32 | 13.38 | 12.85 | 13.56 | 17.78 | 47.56 | 53.05 | 53.05 | 56.40 | 40.24 |
| BTQA → WGM | 28.22 | 18.04 | 22.24 | 16.51 | 38.96 | 22.91 | 35.70 | 28.43 | 42.47 | 12.74 |
| BTQA → WJB | 44.04 | 33.90 | 32.61 | 32.68 | 58.34 | 18.29 | 26.23 | 27.59 | 27.77 | 12.74 |
| BTQA → TG | 4.75 | 3.52 | 4.05 | 4.58 | 9.86 | 82.93 | 86.59 | 83.84 | 85.98 | 63.72 |
| TG → WGM | 8.15 | 8.66 | 10.46 | 8.27 | 31.85 | 81.90 | 80.49 | 75.94 | 81.07 | 34.62 |
| TG → WJB | 10.84 | 11.62 | 13.29 | 10.25 | 37.67 | 72.03 | 71.14 | 68.23 | 73.36 | 39.75 |
| TG → BTQA | 2.94 | 3.41 | 4.49 | 3.43 | 32.04 | 91.11 | 90.23 | 87.13 | 90.26 | 42.06 |
| Mean | 16.74 | 14.19 | 15.67 | 13.41 | 26.05 | 47.31 | 50.29 | 47.65 | 52.31 | 32.63 |
Table 7. Cross-dataset false-negative and false-positive rates for all five methods.
WildGuardMix and BeaverTailsQA. Table 10 reports the silhouette and detection results across refinement checkpoints for WildGuardMix and BeaverTailsQA. This analysis focuses on the performance gap between JDPI and the classifier-based baselines during iterative representation refinement. The refinement analysis is conducted on WildGuardMix and BeaverTails QA. The checkpoint results are shown in Figure 6.
Figure 6. Accuracy and ROC-AUC across refinement checkpoints for Methods A and JDPI.
| Stage | Silhouette | Acc. (%) | ROC-AUC | |||
| Global | Harm. | A | E | A | JDPI | |
| WildGuardMix | ||||||
| P1 | 0.9065 | 0.2209 | 94.43 | 94.13 | 0.9874 | 0.9860 |
| I1 | 0.6173 | 0.7029 | 94.22 | 93.57 | 0.9852 | 0.9815 |
| I2 | 0.5389 | 0.8176 | 93.62 | 92.38 | 0.9826 | 0.9742 |
| I3 | 0.5217 | 0.8627 | 93.30 | 92.02 | 0.9803 | 0.9615 |
| I4 | 0.5147 | 0.8762 | 93.06 | 91.01 | 0.9798 | 0.9584 |
| I5 | 0.4691 | 0.8875 | 92.79 | 91.24 | 0.9786 | 0.9608 |
| BeaverTailsQA | ||||||
| P1 | 0.8030 | 0.1221 | 82.24 | 80.48 | 0.8938 | 0.8772 |
| I1 | 0.5993 | 0.3132 | 81.87 | 80.87 | 0.8923 | 0.8747 |
| I2 | 0.5654 | 0.5454 | 80.93 | 80.03 | 0.8871 | 0.8574 |
| I3 | 0.5227 | 0.6395 | 81.57 | 78.56 | 0.8881 | 0.8519 |
| I4 | 0.4420 | 0.6816 | 81.04 | 79.31 | 0.8857 | 0.8361 |
| I5 | 0.4507 | 0.7247 | 81.30 | 79.30 | 0.8845 | 0.8330 |
Table 10. Refinement checkpoint results. Global denotes the harmful–benign silhouette and Harm. denotes the silhouette among harmful subclusters. P1 is the Phase 1 checkpoint and I1–I5 are the Phase 2 iterations.
Refinement increases harmful-cluster separation while reducing global harmful–benign separation on both benchmarks. On WildGuardMix, the harmful-cluster silhouette rises from 0.2209 to 0.8875, whereas the global silhouette falls from 0.9065 to 0.4691. On BeaverTailsQA, the corresponding changes are 0.1221 to 0.7247 and 0.8030 to 0.4507.
Over the same checkpoints, JDPI’s ROC-AUC decreases from 0.9860 to 0.9608 on WildGuardMix and from 0.8772 to 0.8330 on BeaverTailsQA. A is less affected, with ROC-AUC decreasing from 0.9874 to 0.9786 and from 0.8938 to 0.8845, respectively. Thus, improved separation among harmful clusters does not translate into improved binary detection in these experiments.
These trends are consistent with a mismatch between the refinement and detection objectives.
Phase 1 explicitly separates harmful and benign examples, whereas Phase 2 additionally separates subgroups within the harmful class. This can alter the geometry used by nearest-prototype inference without improving the harmful–benign decision boundary. A learned classifier may accommodate these changes more effectively by fitting a decision direction across embedding dimensions. The results therefore favor evaluating refinement through downstream detection performance, rather than harmful-cluster silhouette alone.
Future work could jointly optimize harmful–benign separation and harmful-subcluster separation, with their relative weights selected on validation data for the intended detection protocol.
TomGibbs. We evaluate Phase 1 and Iterations 1–5 on the same 896 held-out conversations. For multi-turn replay, we use each checkpoint’s thresholds from complete-prompt validation.
JDPI’s complete-prompt accuracy is highest at Phase 1, and its ROC-AUC peaks at Iteration 4 (Table 11). With accumulated user history, its any-turn accuracy improves after the first refinement iteration and varies across subsequent checkpoints, reaching 89.29% at Iteration 5 (Table 12). Method A leads JDPI at Iteration 4 under this replay setting. From Phase 1 to Iteration 5, JDPI’s validation-selected threshold changes from 0.25 to 0.60. With accumulated user history, Its replay false-positive rate falls from 100.00% to 21.95%, and its false-negative rate increases from 0.00% to 4.23%.
| Checkpoint | A | B | C | D | JDPI |
| Accuracy (%) | |||||
| Phase 1 | 92.41 | 92.08 | 92.52 | 91.63 | 93.42 |
| Iter. 1 | 92.75 | 93.75 | 93.86 | 94.20 | 89.84 |
| Iter. 2 | 91.85 | 92.19 | 93.53 | 93.97 | 88.62 |
| Iter. 3 | 94.98 | 95.42 | 95.31 | 95.42 | 89.84 |
| Iter. 4 | 93.86 | 93.08 | 94.08 | 93.08 | 90.51 |
| Iter. 5 | 95.31 | 95.54 | 95.31 | 95.42 | 89.73 |
| ROC-AUC | |||||
| Phase 1 | .9798 | .9760 | .9661 | .9671 | .9589 |
| Iter. 1 | .9719 | .9702 | .9706 | .9744 | .9576 |
| Iter. 2 | .9552 | .9557 | .9592 | .9674 | .9557 |
| Iter. 3 | .9585 | .9575 | .9559 | .9564 | .9644 |
| Iter. 4 | .9581 | .9586 | .9565 | .9584 | .9666 |
| Iter. 5 | .9581 | .9586 | .9566 | .9587 | .9637 |
Table 11. Complete-prompt detection across TomGibbs checkpoints.
| Checkpoint | Method A Acc. | Method A FNR | Method A FPR | JDPI Acc. | JDPI FNR | JDPI FPR |
| Current turn | ||||||
| Phase 1 | 63.39 | 0.00 | 100.00 | 63.39 | 0.00 | 100.00 |
| Iter. 1 | 63.39 | 0.00 | 100.00 | 76.67 | 2.82 | 58.84 |
| Iter. 2 | 63.39 | 0.00 | 100.00 | 78.79 | 17.25 | 28.05 |
| Iter. 3 | 63.39 | 0.00 | 100.00 | 75.00 | 6.34 | 57.32 |
| Iter. 4 | 70.54 | 0.00 | 80.49 | 74.55 | 1.41 | 67.07 |
| Iter. 5 | 63.39 | 0.00 | 100.00 | 77.23 | 6.34 | 51.22 |
| Accumulated user history | ||||||
| Phase 1 | 63.39 | 0.00 | 100.00 | 63.39 | 0.00 | 100.00 |
| Iter. 1 | 63.39 | 0.00 | 100.00 | 86.94 | 4.93 | 27.13 |
| Iter. 2 | 63.39 | 0.00 | 100.00 | 87.72 | 8.80 | 18.29 |
| Iter. 3 | 63.39 | 0.00 | 100.00 | 87.61 | 5.46 | 24.39 |
| Iter. 4 | 87.61 | 1.94 | 30.49 | 83.48 | 4.23 | 37.80 |
| Iter. 5 | 63.39 | 0.00 | 100.00 | 89.29 | 4.23 | 21.95 |
Table 12. Conversation-level any-turn results across TomGibbs checkpoints. All values are percentages.
| Benign subgroup | Input | A | JDPI |
| Completely benign () | Current | 100.00 | 28.57 |
| History | 100.00 | 0.00 | |
| Semi-benign () | Current | 100.00 | 75.00 |
| History | 100.00 | 45.00 |
Table 13. Any-turn false-positive rates on TomGibbs (%).
Additional Multi-Turn Analysis
We report conversation-level false-positive rates separately for completely benign and semi-benign conversations on TomGibbs under the any-turn rule.
References
- [1]Shirin Alanova, Bogdan Minko, Sabrina Sadiekh, and Evgeniy Kokuykin. 2026. Cross-lingual jailbreak detection via semantic codebooks. Preprint, arXiv:2604.25716.arxiv.org/abs/2604.25716
- [2]Sarah Ball, Frauke Kreuter, and Nina Panickssery. 2024. Understanding jailbreak success: A study of latent space dynamics in large language models. Preprint, arXiv:2406.09289.arxiv.org/abs/2406.09289
- [3]Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, and Eric Wong. 2024. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. Preprint, arXiv:2404.01318.arxiv.org/abs/2404.01318
- [4]Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. Preprint, arXiv:2002.05709.arxiv.org/abs/2002.05709
- [5]Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. 2025. Or-bench: An over-refusal benchmark for large language models. Preprint, arXiv:2405.20947.arxiv.org/abs/2405.20947
- [6]Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents. In Advances in Neural Information Processing Systems, volume 37, pages 82895–82920. Curran Associates, Inc.arxiv.org/abs/2406.13352
- [7]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. Preprint, arXiv:1810.04805.arxiv.org/abs/1810.04805
- [8]Erick Galinkin and Martin Sablotny. 2024. Improved large language model jailbreak detection via pre-trained embeddings. Preprint, arXiv:2412.01547.arxiv.org/abs/2412.01547
- [9]Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, and 17 others. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. Preprint, arXiv:2209.07858.arxiv.org/abs/2209.07858
- [10]Debargha Ganguly, Sreehari Sankar, Biyoa Zhang, Vikash Singh, Kanan Gupta, Harshini Kavuru, Alan Luo, Weicong Chen, Warren Morningstar, Raghu Machiraju, and Vipin Chaudhary. 2026. Trust the typical. Preprint, arXiv:2602.04581.arxiv.org/abs/2602.04581
- [11]Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2022. Simcse: Simple contrastive learning of sentence embeddings. Preprint, arXiv:2104.08821.arxiv.org/abs/2104.08821
- [12]Tom Gibbs, Ethan Kosak-Hine, George Ingebretsen, Jason Zhang, Julius Broomfield, Sara Pieri, Reihaneh Irmanmesh, Reihaneh Rabbany, and Kellin Perline. 2024. Emerging vulnerabilities in frontier models: Multi-turn jailbreak attacks. Preprint, arXiv:2409.00137.arxiv.org/abs/2409.00137
- [13]Melody Y. Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, Hyung Won Chung, Sam Toyer, Johannes Heidecke, Alex Beutel, and Amelia Glæse. 2025. Deliberative alignment: Reasoning enables safer language models. Preprint, arXiv:2412.16339.arxiv.org/abs/2412.16339
- [14]Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. 2024. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. Preprint, arXiv:2406.18495.arxiv.org/abs/2406.18495
- [15]Peichun Hua, Hao Li, Shanghao Shi, Zhiyuan Yu, and Ning Zhang. 2026. Rethinking jailbreak detection of large vision language models with representational contrastive scoring. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 21748–21785, San Diego, California, United States. Association for Computational Linguistics.
- [16]Xiang Huang, Hao Peng, Dongcheng Zou, Zhiwei Liu, Jianxin Li, Kay Liu, Jia Wu, Jianlin Su, and Philip S. Yu. 2024. Cosent: Consistent sentence embedding via similarity ranking. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:2800–2813.
- [17]Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. Preprint, arXiv:2312.06674.arxiv.org/abs/2312.06674
- [18]Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somanpalli, John Kirchenbauer, Ping yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline defenses for adversarial attacks against aligned language models. Preprint, arXiv:2309.00614.arxiv.org/abs/2309.00614
- [19]Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Preprint, arXiv:2307.04657.arxiv.org/abs/2307.04657
- [20]Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeez Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, and Nouha Dziri. 2024. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models. Preprint, arXiv:2406.18510.arxiv.org/abs/2406.18510
- [21]Difan Jiao, Yilun Liu, Ye Yuan, Zhenwei Tang, Linfeng Du, Haolun Wu, and Ashton Anderson. 2026. LLM safety from within: Detecting harmful content with internal representations. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 39711–39727, San Diego, California, United States. Association for Computational Linguistics.
- [22]Zhihua Jin, Shiyi Liu, Haotian Li, Xun Zhao, and Huamin Qu. 2024. Jailbreakhunter: A visual analytics approach for jailbreak prompts discovery from large-scale human-llm conversational datasets. Preprint, arXiv:2407.03045.arxiv.org/abs/2407.03045
- [23]Sri Durga Sai Sowmya Kadali and Evangelos E. Papalexakis. 2026. Jailbreaking leaves a trace: Understanding and detecting jailbreak attacks from internal representations of large language models. Preprint, arXiv:2602.11495.arxiv.org/abs/2602.11495
- [24]Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2021. Supervised contrastive learning. Preprint, arXiv:2004.11362.arxiv.org/abs/2004.11362
- [25]Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. Preprint, arXiv:2011.05864.arxiv.org/abs/2011.05864
- [26]S. Lloyd. 2006. Least squares quantization in pcm. IEEE Trans. Inf. Theor., 28(2):129–137.
- [27]Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, and Lilian Weng. 2023. A holistic approach to undesired content detection in the real world. Preprint, arXiv:2208.03274.arxiv.org/abs/2208.03274
- [28]Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. Preprint, arXiv:2402.04249.arxiv.org/abs/2402.04249
- [29]Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red teaming language models with language models. Preprint, arXiv:2202.03286.arxiv.org/abs/2202.03286
- [30]Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. Preprint, arXiv:1908.10084.arxiv.org/abs/1908.10084
- [31]Alexander Robey, Eric Wong, Hamed Hassani, and George J. Pappas. 2024. Smoothllm: Defending large language models against jailbreaking attacks. Preprint, arXiv:2310.03684.arxiv.org/abs/2310.03684
- [32]Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, Amanda Askell, Nathan B. Bailey, Joe Benton, Emma Bluemke, Samuel R. Bowman, Eric Christiansen, Hoagy Cunningham, Andy Dau, Anjali Gopal, and 24 others. 2025. Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming. Preprint, arXiv:2501.18837.arxiv.org/abs/2501.18837
- [33]Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. Preprint, arXiv:2308.03825.arxiv.org/abs/2308.03825
- [34]Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, and Stephen Casper. 2025. Latent adversarial training improves robustness to persistent harmful behaviors in llms. Preprint, arXiv:2407.15549.arxiv.org/abs/2407.15549
- [35]Jake Snell, Kevin Swersky, and Richard S. Zemel. 2017. Prototypical networks for few-shot learning. Preprint, arXiv:1703.05175.arxiv.org/abs/1703.05175
- [36]Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted pre-training for language understanding. Preprint, arXiv:2004.09297.arxiv.org/abs/2004.09297
- [37]Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. 2024. A strongrefect for empty jailbreaks. Preprint, arXiv:2402.10260.arxiv.org/abs/2402.10260



