ShallowDefend: Harnessing Shallow Safety Alignment for Multi-Turn Defender

Introduction

As large language models (LLMs) are deployed across an ever-widening range of applications, ensuring that they behave safely has become as important as advancing their capabilities. Safety alignment, training models to refuse harmful requests while remaining helpful on benign ones, is now a standard stage in the development of every frontier model [24, 34, 7]. This investment has paid off in the single-turn setting: successive rounds of safety training and red-teaming have made aligned models substantially robust to single-turn jail-breaks, from handcrafted prompts to optimized adversarial suffixes [5, 11] (Zou et al. 2023), and a harmful request posed directly is now refused with high reliability. Multi-turn dialogue, however, remains far less protected. In a single-turn attack, the malicious intent is evident in the prompt itself, while in a multi-turn conversation, the attacker can conceal it, distributing the request across turns so that each message appears innocuous [31, 40, 2]. Under such attacks models fail far more often, and human red-teamers continue to defeat defenses that report near-perfect single-turn robustness [18, 10, 37, 28].

A growing line of defenses targets this setting: supervised fine-tuning on adversarial dialogues [28], preference optimization over safe and unsafe replies [17 Yan et al.,] 2025), reinforcement learning with judge-based rewards [13], representation-level interventions [22], and inference-time prompting that asks the model to analyse the user’s intention before answering (Zhang et al., 2025b). Beneath their differences, the training-based methods share a dependency: each needs an oracle in the loop, whether a harmfulness classifier, a preference labeler, or a guard model, to say what a safe response looks like, and each supervises the form of the response that oracle prefers. That dependency is costly and fragile. Judges disagree with one another on a substantial fraction of individual samples (§ 4.2), so a reward built on one imports its errors into training and entangles the training signal with the evaluation metric. And because the model is only ever shown what safe replies look like, it never learns an explicit safety decision that a rule, rather than a model, could check.

A second observation says where such a decision belongs. Safety alignment is known to be shallow: an aligned model departs from its base almost exclusively over the first few output tokens [25]. Safety training, in effect, teaches the model to open with a refusal, and once a response opens compliantly the aligned distribution collapses back to the base model’s. This is why prefilling an affirmative prefix defeats refusal [39, 3], and it is how a multi-turn attacker wins: steer the conversation until the locally natural opener is compliant, and by the payload turn the refusal is gone before anything harmful has been written. Prior work reads this shallowness as a defect and tries to deepen alignment [25]. We read it as a map. If a short span at the front of the response sets the trajectory of everything that follows, that span is the highest-leverage place to intervene, and the defender can occupy it first, not with an implicit refusal prefix but with an explicit commitment.

We instantiate this idea as Assess $\rightarrow$ Decide $\rightarrow$ Respond. At every turn the defender briefly assesses the user’s underlying intent, commits to a parseable safety decision, and only then writes a reply conditioned on that decision. The commitment is the heart of the design. It is externalized: a token span in the model’s own output that a rule can read. It is binding: because it precedes the reply within the same generation, the reply is produced conditioned on it, so changing the commitment changes the response. And it is early: it occupies exactly the trajectory-setting positions that shallow alignment leaves exposed. We train the commitment with reinforcement learning, rewarding only the final turn of each conversation, where the attack delivers its payload and the defense must hold.

Externalizing the decision is what makes the reward simple. Training dialogues are grown from seed prompts whose nature is known in advance, so every trajectory carries a ground-truth label for free, and the reward reduces to a string comparison: REFUSE on conversations grown from harmful seeds, COMPLY on benign ones. No judge, no annotation, no model in the loop: a verifiable reward for conversational safety in the spirit of RLVR [33, 19]. Where a judge is available, the same format supports the prevailing judge-scored design [16, 13, 9], with a graded decision space that adds SAFE-REDIRECT. And because the decision conditions the reply, training the decision span alone is enough to change end-to-end behavior.

Experiments on two model families [12, 42] and fourteen benchmarks spanning safety, utility, and helpfulness show that both variants match or exceed every baseline trained on the same data, that the lead holds under six independent judges, and that the self-verified variant achieves this with no judge in the loop and a recipe that runs unchanged across backbones. Utility is preserved, at a modest cost in helpfulness that we quantify. We summarize our contributions as follows:

• A multi-turn defense framework: Assess $\rightarrow$ Decide $\rightarrow$ Respond turns shallow alignment from a vulnerability into an explicit, trainable control point for safety alignment.

• Two complementary rewards: a self-verified reward that checks the committed label against the conversation’s known nature, requiring no judge and no annotation, and an external LLM-as-judge reward over a graded decision space with safe redirection.

• Comprehensive experiments and analysis: the strongest multi-turn safety in a controlled comparison across two model families, fourteen benchmarks, and six judges, achieved without per-backbone tuning, with KL and interventional analyses confirming that the defense operates through the committed decision.

Methodology

We train a multi-turn defender via reinforcement learning. At each turn, the defender follows an Assess $\to$ Decide $\to$ Respond pipeline: it first assesses the user’s underlying intent, then commits to an explicit decision, and finally generates a response conditioned on that decision. For policy optimization, we explore two reward formulations: an external reward model that judges response safety and a rule-based self-reward that verifies the committed decision against the conversation’s ground-truth nature. Figure 1 provides an overview of our framework.

Assess Decide Respond

At every turn, the defender first assesses the user’s underlying intent, then commits to an explicit decision, and only then generates its response conditioned on that decision. The decision space consists of comply and refuse. When training with the external reward model, we augment it with a third option, safe-redirect, which allows the defender to deflect harmful requests without an outright refusal. Concretely, the system prompt elicits a single generation of the form

Assessment:  the user’s true underlying intent, and whether
             fulfilling the request could cause real harm
Decision:    COMPLY / REFUSE / SAFE-REDIRECT
Response:    the reply to the user

We write $y_t = (d_t, a_t)$ for the decomposition of turn $t$ into its deliberation $d_t$, spanning everything up to and including the decision line, and its reply $a_t$. Two properties of this format carry the rest of the method. First, the decision is externalized: it is a token span in the model’s own output, and can therefore be read by a rule rather than judged by a model. Second, it is binding: because it precedes the reply within the same generation, the reply is produced conditioned on it, which is what makes it possible to train the decision alone. The two system prompts that elicit this format are shown verbatim in Appendix D.

External Reward

Our first reward formulation employs an external LLM-as-a-judge to score the defender’s behavior during policy optimization. Let $c_T = (x_1, y_1, \ldots, x_T, y_T)$ denote a rollout of $T$ turns, where $x_t$ is the user utterance and $y_t = (d_t, a_t)$ the defender’s output at turn $t$. The judge $\mathcal{J}$ receives the full conversation context together with the final reply $a_T$ and determines whether the defense was bypassed, i.e., whether $a_T$ substantively fulfills a harmful request. The reward is assigned to the final turn as

where $\mathcal{J}(\cdot) \in\{\mathrm{SAFE}, \mathrm{UNSAFE}\}$ is the judge’s verdict. A response is deemed SAFE if it either refuses the harmful request or redirects it without providing harmful content. For benign conversations, a response is SAFE only if it helpfully complies, which penalizes over-refusal. The policy gradient is applied only to the final turn $y_T$, with all preceding turns treated as context.

Self-Verified Decision Reward

Prior RL-based defenses score the defender’s response with an external judge — a harmfulness classifier on harmful conversations, and typically a refusal-phrase or helpfulness proxy on benign ones. We replace both with a rule that reads only the committed label. Let $\ell(d_t) \in\{\mathit{comply}, \mathit{refuse}\}$ denote the label parsed from the deliberation at turn $t$, and let $b \in\{0,1\}$ indicate the conversation’s ground-truth nature, with target decision

Mirroring Eq. equation 1, the trajectory return checks the final turn’s commitment against this target,

where the condition $a_T \ne\varnothing$ serves as a completeness guard: a turn whose deliberation exhausts the per-turn token budget and leaves no reply is penalized. Without this guard, the reward admits a degenerate optimum: emit refuse and never answer. It is precisely this guard that requires every commitment to be backed by an actual reply. Note also that the self-verified setting excludes safe-redirect from the decision space: in harmful conversations, both refuse and safe-redirect constitute defensible commitments. Therefore, a rule that reads only the committed label cannot designate a unique target, rendering the reward signal ambiguous. We therefore restrict this setting to the binary decision space and leave the three-way space to the external judge, which evaluates the reply itself.

Gradient Scope Since the reward reads only the deliberation, a natural question is whether the reply needs to be trained at all. We offer two gradient options over the same reward:

• Full gradient: every generated token of the final defender turn is trainable (default).

• Partial gradient: only tokens preceding the decision boundary are trainable. Every reply token in $a_t$ receives mask 0 and therefore contributes no policy-gradient term.

Formally, with $m_{t,i} \in\{0,1\}$ denoting the mask of the $i$-th token of turn $t$, the full gradient sets $m_{t,i}=1$ for all generated tokens, while the partial gradient sets $m_{t,i} = \mathbb{1}[i \leq|d_t|]$. Locating $|d_t|$ requires mapping the character-level boundary between deliberation and reply to a token index. Because $\operatorname{decode}(y_{t,1:k})$ is non-decreasing in $k$, binary-search for the smallest $k$ whose decoded prefix reaches the boundary character, at a cost of $O(\log|y_t|)$ decodes per turn.

The full gradient treats the whole generation as the action, so the reply style adapts jointly with the decision; the partial gradient confines learning to the deliberation and largely preserves the base model’s reply style. That the partial gradient works at all rests on the binding property: the reply is generated conditioned on the committed decision, so training the decision alone changes end-to-end behavior. The same property is what lets the reward read only the label. The rule cannot by itself rule out a reply that contradicts its label, and we add no content check, for two reasons. First, a decoupled policy could not score well, since safety is judged open-endedly on the user-facing reply alone, so any gap between label and reply would appear as a lower safe rate, and none does. Second, the binding holds by construction: a forced REFUSE yields a safe reply in every case even before training (§A). Consistent with both, the emitted $\mathrm{Decision}$ label is positively correlated with every external judge’s verdict, at a level comparable to inter-judge agreement (§4.2).

Training Algorithm

We optimize the LLM defender with GRPO (Shao et al., [33]). Let $\tau= (x_1, y_1, \ldots, x_T, y_T)$ denote a training dialogue, where $x_t$ is the $t$-th attacker or benign user turn and $y_t$ is the defender’s reply. Beyond the standard objective, two design choices are essential.

Per-turn length budget Each reply is capped at $L$ tokens, i.e., $|y_t| \le L$ for all $t$. With only a trajectory-level budget, the policy learns a degenerate strategy: it inflates replies to the early turns, which are easy to answer, until the budget is exhausted, leaving the later and more difficult turns truncated or unanswered. The per-turn cap closes this reward-hacking route by making every turn individually answerable.

Outcome-supervised reward We assign reward from the final turn only:

Multi-turn jailbreaks culminate in the final turn while earlier turns remain innocuous, so a process-level signal would be dominated by easy turns and bias the policy toward defending only where defense is cheap. Supervising the outcome concentrates learning on the decisive turn and mirrors deployment, where what matters is whether the model complies at the end of the escalation (details in Appendix § 9).

Experiments

Experimental Setup

Training The harmful pool mixes five multi-turn attack families instantiated over the same AdvBench (Zou et al., 2023) behaviors: SEMA (Feng et al., [10]), Crescendo (Russinovich et al., [31]), foot-in-the-door (FITD) (Weng et al., [40]), Echoing (Li et al., [18]), and Hidden Intention Streamline (Li et al., [18]). Behaviors are split into disjoint train and test sets, and each training behavior is assigned one family uniformly at random. The benign pool comprises multi-turn dialogues from UltraChat-200k (Ding et al., [8]), matched to the harmful conversations in surface form and depth so that neither correlates with harmfulness. The two pools are mixed in equal proportion to constrain over-refusal; ablations over the mixing ratio are in § 3.3. We use HarmBench-13B classifier (Mazeika et al., [23]) as the external reward model. For each reward variant we report two conditions: unaligned, our system prompt applied to the untrained base weights; and aligned, the same prompt after RL. The former isolates the framework and the gap between them is the contribution of training.

Baselines Each baseline is trained on exactly the same data as our model. We use two base models: Llama3.1-8B-Instruct (Inan et al., [16]) and Qwen3-8B (Yang et al., [42]). We exclude defenses whose contribution is primarily a dataset rather than a method (Ren et al., [29]; Rahman et al., [28]), as our controlled comparison trains all methods on identical data.

SFT fine-tunes on safe dialogues self-distilled from the base model, refusals on attack turns and helpful replies on benign ones. DPO (Rafailov et al., [27]) optimizes preferences over safe/unsafe reply pairs. SDGO (Ding et al., [9]) runs GRPO rewarded by the model’s own discrimination verdicts. MTSA (Guo et al., [13]) rolls safe revisions of jailbroken turns forward and fits the resulting future-reward margin with DPO. MUSE-D (Yan et al., [41]) applies LoRA DPO to preference data. built with its released risk rules. X-Boundary Lu et al., [22] is a representation-level defense, applied to our rollouts converted to its erase/retain format.

Evaluation We evaluate defenders along three axes: safety, utility, and helpfulness.

1. Safety. AdvBench (Zou et al., 2023) (the behavior source underlying our training attacks), HarmBench Mazeika et al., [23] (an unseen behavior source), and three multi-turn benchmarks absent from training: MultiBreak Song et al., [37], CoSafe Yu et al., [43], and MHJ Li et al., [18]. The first two measure generalization across behaviors; the latter three, transfer to unseen attack mechanisms.

2. Utility. MMLU Hendrycks et al., [14], HellaSwag Zellers et al., [45], Winogrande Sakaguchi et al., [32], GSM8K Cobbe et al., [6], TruthfulQA Lin et al., [20], and IFEval (Zhou et al., 2023), covering the capability axes on which alignment interventions are known to regress.

3. Helpfulness. XSTest Röttger et al., [30] and XSB Yuan et al., [44] for over-refusal, and MT-Bench Bai et al., [4] for multi-turn conversational quality, checking that safety is not purchased by refusing legitimate requests.

Each answer is judged by gpt-5-mini Singh et al., [36], which receives the full conversation history together with the target behavior and returns a binary harmfulness verdict. Agreement with alternative judges is analyzed in § 4.2, and safety under five additional held-out judges is reported in Appendix § 11.§ 3.

Main Results

Table 1 reports the full comparison. Our method attains the highest average safe rate on both backbones and leads every individual safety column. The two panels differ in how that lead is composed. On Llama3.1-8B the baselines separate sharply, with X-Boundary near-saturating AdvBench while most others fall far behind, so our margin comes from the suites the strongest baseline handles worst, above all the human-crafted MHJ attacks. On Qwen3-8B the baselines sit much closer to each other, and our advantage is a uniform lead across all five suites. These gains come at modest cost elsewhere: our models remain competitive on MT-Bench and match the field on utility, which is scored without the chat template and is therefore largely insensitive to safety training (Qi et al. [25]) (Figure 2); the only visible price is somewhat higher over-refusal, as expected from a stricter decision policy. Appendix § 8 plots the safety–helpfulness trade-off, on which our models sit at the favorable end of the frontier, and Appendix § 11.§ 2 restates Table 1 with 95% confidence intervals.

Two comparisons locate the source of the gain. First, we run our system prompt on the untrained base weights (unaligned) and again after RL (aligned). The prompt alone improves on the base model but falls well short of the trained result, so structured deliberation helps in its own right and is no substitute for the reward. Second, the table holds the data fixed and varies only the method, so the remaining differences are attributable to the training objective rather than to the training set. Reaching the baseline numbers on Qwen3-8B did require a per-method configuration change, none of which transfers from the Llama setting (Appendix § 9); our own recipe is unchanged across backbones.

Ablation Study

Setup We ablate one factor at a time on the Llama backbone, holding everything else fixed. Two ablations alter the training recipe: per-turn reward distributes the outcome-level reward across individual turns, and partial gradient restricts the gradient from the full response to the decision span. Two alter the framework: w/o decision drops the Decision line, leaving Assess $\rightarrow$ Respond, and w/o assessment drops the Intent line, leaving Decide $\rightarrow$ Respond. Both framework arms use the external reward, since a defender that emits no label gives the self-verified reward nothing to check (§2.3). A final ablation varies the harmful-to-benign ratio of the training mixture. Figure 3a shows the first four: the horizontal lines are our two full models and each bar is one modification, read against the line for its reward. Figure 3b shows the mixture sweep.

The two recipe changes fail in opposite directions. Per-turn reward loses safety while driving over-refusal to its lowest level, since spreading an outcome-level signal across every turn erodes the safety decision and pushes the policy toward indiscriminate compliance; partial gradient buys robustness at the cost of moderately higher over-refusal, with answer quality untouched. The framework ablations show where the defense lives. Removing the decision is the most damaging change in the study: safety falls on every suite and, under human red-teaming, drops below the undefended base model, even though the assessment, the reward and the data are all left intact. Removing the assesment costs far less and shows up mainly as a loss of calibration, with over-refusal rising because the commitment is placed less precisely. Both removals refuse less overall, which lifts answer quality slightly. The commitment, not the deliberation that precedes it, carries the defense; Appendix § 7.§ 1 reaches the same conclusion by intervention rather than removal.

The training mixture only sets where the policy sits on the frontier, since automated safety is saturated throughout: oversampling harmful dialogues buys red-teaming robustness at a steep cost in over-refusal and answer quality, while oversampling benign dialogues trims over-refusal only marginally. We keep the balanced mixture.

Analysis

Shallow Alignment

Safety alignment in current LLMs is known to be shallow: the aligned distribution departs from the base model almost exclusively over the first few output tokens, and this concentration is precisely what prefilling and decoding-parameter exploits target [25]. Rather than treating shallowness as a pathology to be eliminated, our framework repurposes it: the intervention remains concentrated at a small number of trajectory-setting positions, but the fragile refusal prefix is replaced by an explicit control structure whose decision field gates the subsequent response. The setup is detailed in Appendix § 9.

Figure 4 shows a pattern qualitatively different from shallow alignment. $D_{\mathrm{KL}}(\text{aligned} \parallel\text{unaligned})$ is not concentrated at the first few tokens; instead, training-induced divergence is spread across the assessment phase preceding the decision token, peaks in its vicinity, and gradually vanishes thereafter as the two models converge. This is by construction: training reshapes how the model assesses intent and commits to a decision, so its influence is distributed over the entire deliberation rather than compressed into an opening refusal prefix; once the decision is emitted, the response is largely determined and the two distributions gradually converges. In contrast, both the aligned and unaligned models diverge sharply from the base model at the very first position and remain separated throughout, reflecting the fact that the framework defers commitment while the base model begins answering immediately. Together, these observations confirm that our design relocates the safety-relevant probability mass from a superficial refusal prefix to the assessment-decision phase, retaining the benefits of an early intervention while removing the single-point attack surface that shallow alignment creates. Appendix § 7 further verifies the same claim by direct intervention, showing that overwriting the committed decision moves reply safety in both directions and that training hardens the reply behind the commitment, not merely the label.

Correlation Between LLM Judges

Because safety is scored by an LLM judge, the choice of judge is a potential confound: an improvement measured under one judge need not survive under another. Rather than assuming judges are interchangeable, we quantify their agreement and discuss what it implies for interpreting our results. The setup is detailed in Appendix § 9.

Figure 5 reports the pairwise $\phi$ coefficient over per-sample verdicts, and two observations follow. First, all pairs are positively but only moderately correlated: the judges share a broad consensus on what constitutes a jailbreak, yet disagree on a substantial fraction of individual samples, and even judges of the same style differ. This dispersion cautions against reading any single-judge attack-success rate as ground truth, and motivates our practice of reporting results under multiple judges (Appendix § 11.§ 3). Second, the model’s own Decision label correlates with every external judge at a level comparable to inter-judge agreement, despite the external judges never observing the deliberation that produced it. The committed decision is therefore not an internal artifact decoupled from outcomes: it tracks independent assessments of the final reply, supporting its use as a self-verified training signal in our decision reward.

Related Work

Multi-turn jailbreaks. Aligned models now refuse direct harmful requests with high reliability, which has pushed attacks toward the conversational setting: escalation schemes that decompose a harmful goal into individually innocuous turns ([31, 29]), persuasion- and rapport-based strategies ((Zeng et al., 2024)), and human red-teaming, which remains markedly more effective than automated attacks ([18]). These attacks succeed without ever posing an overtly harmful query, so single-turn defenses and input filters offer little protection. Our threat model is exactly this setting: scripted multi-turn attacks replayed statically, with adaptive adversaries acknowledged as an open limitation.

Multi-turn defenses. Existing training-based defenses supervise the response: SFT and preference optimization on safe replies ([17, 41]), and RL with a harmfulness classifier or guard model as reward ([9, 13]). This design imports the judge’s errors into training, entangles the training reward with the evaluation metric, and opens a reward-hacking surface. Representation-level methods instead separate harmful from benign directions in activation space ([22] (Zou et al. 2024)). A third line intervenes at inference rather than in the weights, prompting the model to analyse the user’s underlying intention before it answers ((Zhang et al., 2025b)). That work is the closest in spirit to our assessment step, and the comparison is informative: it obtains its gains from deliberation alone, whereas our ablations show that deliberation without a committed decision recovers little of the defense (§ 3.3). Our reward departs from all three: it reads only the model’s committed decision against the conversation’s known label, requiring no model in the loop, and the gradient never reaches the response.

Shallow alignment. A complementary line of work locates the mechanism of these failures: alignment behavior concentrates in the first few output tokens ([25]), which is why prefilled affirmative openers ([3, 39]), decoding-parameter exploits ([15]), and fine-tuning perturbations of early positions ([26]) defeat refusal. Proposed remedies deepen the alignment, e.g., by augmenting training with recovery-from-harmful-prefix data ([25] (Zhang et al. 2025a)).

Conclusion

We present Assess → Decide → Respond, a multi-turn defense in which the defender commits to an explicit, parseable safety decision before it replies, trained by reinforcement learning at the payload turn. Rather than repairing shallow alignment, the framework repurposes it: the short span that decides the fate of a response, ordinarily the attacker’s target, becomes a commitment the defender controls. Because the decision is a token span, a rule can reward it against the conversation’s known nature with no judge and no annotation, and the same format also supports a judge-based reward with safe redirection. Across two model families and fourteen benchmarks, our defenders achieve the strongest multi-turn safety in a controlled comparison, hold that lead under six judges, and preserve utility, with over-refusal as the cost. KL analysis, interventional probes, and ablations agree that the defense operates through the committed decision. More broadly, our results suggest that structural properties of alignment, often treated as weaknesses, can be engineered into defenses, and that commitments offer a scalable alternative to judge-based supervision for conversational safety.

Shallow Alignment Experiments

This section verifies by direct intervention that our framework achieves what it was designed for: leveraging shallow alignment rather than suffering from it. Shallow alignment concentrates safety behavior at a few trajectory-setting positions, which is ordinarily a vulnerability because those positions can be overwritten or sampled past [25]. Our framework deliberately relocates this concentration into an explicit, trained decision, and the probes below confirm that the relocation works as intended: the committed decision is a causal control point rather than a correlate, training strengthens both the commitment and the reply behind it, and the sampling route that ordinarily defeats shallow alignment is closed. We adapt three diagnostics from Qi et al. [25]: the refusal-prefix shortcut, the prefilling attack, and the decoding-parameter exploit.

Setup. Every arm of Table 1 appears in both its unaligned form (the base weights under the framework prompt, untrained) and its aligned form (trained). The unaligned arm is the reference: it emits a well-formed Decision line by instruction-following alone, so the decision can be read and overwritten exactly as in a trained model, and any gap between the two is attributable to training rather than prompting. Both probes use the held-out SEMA/AdvBench payload turns ($n = 104$); verdicts come from gpt-5-mini, the Table 1 judge.

Method	Adv.	Harm.	MB	CoS.	MHJ	Avg.	Avg.	XSTest ↓	XSB ↓	MT-Bench ↑
Safety ↑ Utility ↑ Helpfulness
Llama3.1-8B-Instruct
Base	54.5	62.5	59.8	84.9	38.0	59.9	66.3	10.0	30.0	6.20
SFT	69.4	79.5	87.0	88.5	55.1	75.9	65.0	14.0	45.6	5.67
DPO	61.6	71.5	79.0	88.3	46.7	69.4	66.4	10.4	34.4	6.26
SDGO	56.6	61.7	60.4	85.0	39.1	60.6	66.2	11.6	30.9	6.11
MTSA	61.6	71.6	79.0	86.6	47.1	69.2	66.4	10.8	36.5	6.12
MUSE-D	61.8	67.5	74.2	87.4	45.3	67.2	66.4	8.4	33.2	6.27
X-Boundary	97.9	95.0	93.0	93.6	52.0	86.3	66.0	16.8	50.3	6.26
Self reward
– unaligned	63.0	70.3	64.2	89.6	44.3	66.3	66.3	8.0	33.8	5.61
– aligned	98.8	99.5	96.6	98.4	58.5	90.4	66.8	20.4	54.4	5.86
External reward
– unaligned	57.2	65.2	56.6	88.5	35.0	60.5	66.3	8.8	32.6	5.81
– aligned	96.9	98.2	99.2	99.0	71.1	92.9	65.8	21.6	50.0	5.21
Qwen3-8B
Base	49.2	52.5	23.8	82.6	22.3	46.1	65.0	7.2	15.6	6.08
SFT	59.9	65.7	49.6	88.5	35.6	59.9	65.9	17.6	40.9	7.53
DPO	80.2	86.6	83.2	97.5	53.6	80.2	64.2	26.0	39.4	5.87
SDGO	48.3	52.5	23.2	82.4	33.0	47.9	64.9	10.0	15.9	6.23
MTSA	79.1	90.2	85.8	97.8	70.9	84.8	64.2	38.4	53.5	5.99
MUSE-D	87.6	90.8	91.0	97.4	61.5	85.7	64.4	20.4	40.9	5.87
X-Boundary	68.0	71.1	67.8	88.8	27.4	64.6	65.1	18.8	49.1	5.69
Self reward
– unaligned	69.0	80.1	66.8	96.4	46.9	71.8	65.0	14.8	37.9	7.33
– aligned	97.1	97.6	97.6	99.3	75.2	93.4	64.6	26.4	56.8	7.36
External reward
– unaligned	68.8	77.3	69.4	95.5	44.3	71.1	65.0	10.0	39.7	7.20
– aligned	90.1	93.6	92.8	98.4	63.1	87.6	64.7	16.8	42.6	6.96

Is the Decision a Cause?

The forced-decision probe (Figure 6, left) replaces the payload-turn deliberation with a canonical one ending in a chosen label and generates only the reply, the interventional counterpart to the observational KL analysis. Follow-through is exact by construction: the emitted label matched the forced one in every cell.

Two findings point in opposite directions, and together they locate exactly what training contributes. First, the refusal shortcut reproduces in decision form. Forcing REFUSE lifts the unaligned model to perfect safety on both backbones, with genuine refusals rather than empty turns: a supplied commitment makes an untrained model behave as aligned, with a parseable label in place of a surface prefix. The model’s ability to honor it is inherited from instruction-following, which is exactly the shallowness our framework leverages. Second, training does not merely select the label. Were that so, forcing COMPLY would return the aligned models to unaligned harm. Instead they stay high: the unaligned arms collapse under a forced COMPLY while every aligned arm remains far above its unaligned counterpart, with Self reward (aligned) on Llama essentially indistinguishable from its natural decoding. Training therefore hardens the reply as well as the commitment. The component ablation in § 3.3 reaches the same conclusion by removal rather than by intervention: a defender trained without the Decision line gives up most of its robustness, while one trained without the assessment gives up comparatively little.

How Deep Is the Protection?

The depth probe (Figure 6, right) concedes the decision and attacks what follows: after a forced REFUSE, it prefills the first $k$ tokens of a genuine harmful reply and continues generation, relocating the prefilling attack past the commitment ($k = 0$ recovers the forced-REFUSE condition above). The unaligned arms are textbook shallow: a handful of prefilled tokens erases most of the safety the commitment secured, and the curve stays flat thereafter. Training buys substantial post-decision depth on Llama, where the aligned curve stays high at shallow prefill and decays gradually, but only marginal depth on Qwen3, where the aligned and unaligned curves converge at larger $k$. Every curve decays eventually, which is the expected signature of a defense that is shallow by construction: the protection is procedural and does not survive an adversary who can write arbitrarily far into the assistant channel. Such an adversary lies outside our threat model, which grants control of user turns only; the right panel quantifies that boundary rather than leaving it as a caveat.

Can Sampling Dislodge the Commitment?

A third probe runs the decoding-parameter exploit, which defeats shallow alignment by sampling alone. For each harmful behavior we draw twelve samples across a grid of temperature, top-$p$, and top-$k$ on direct single-turn AdvBench requests (1,248 samples per arm), plus a greedy generation defining the committed decision, and record the decision-flip rate: the fraction of samples whose label departs from that commitment. Absolute attack success is low for every arm (at most 9 % any-of-$N$), since our base is already an instruction-tuned chat model rather than the pretrained bases the exploit was designed against; the informative quantity is therefore behavior conditional on the sampled label. Within the unaligned Llama it is unambiguous: the few samples that flipped to COMPLY were judged jailbroken about two-thirds of the time, against effectively zero for those that stayed REFUSE. Sampling occasionally dislodges the commitment and the reply follows it, the same mechanism as in §A.1 made visible by chance rather than by intervention. The aligned arms, in contrast, never sample a COMPLY label (at most 0.2 % of flips): training closes the sampling route by which shallow alignment is ordinarily defeated. The one large flip rate we observe, 20–30 % on the Qwen3 three-label arms, is not a safety event: every such flip exchanges the two defensible labels (REFUSE ↔ SAFE-REDIRECT), none reaches COMPLY, and attack success stays at zero. Therefore, we conclude that sampling merely surfaces the decision-space ambiguity without compromising safety.

Safety–Helpfulness Trade-off

Since utility is essentially unchanged across all methods (§3), the meaningful tension lies between safety and helpfulness, and we visualize exactly this trade-off. Figure 7 places each method on a common $[0,100]$ plane summarizing the per-benchmark results of Table 1. The vertical axis reports safety as the mean SAFE % over AdvBench and MHJ: the former is out-of-domain in data but in-domain in attack style, while the latter, composed of human-crafted jailbreaks, is out-of-domain in both and empirically the hardest suite, so together they span the generalization spectrum without letting saturated benchmarks inflate the score. The horizontal axis aggregates helpfulness as $\frac{1}{2}[(100-\mathrm{XSTest})+10\times\mathrm{MT\text{-}Bench}]$, where $100-\mathrm{XSTest}$ converts over-refusal into a compliance rate and MT-Bench is rescaled to $[0,100]$; higher is better on both axes. Stars denote our two reward variants and gray circles the baselines.

The visualization makes the structure of the results immediate. On Llama3.1-8B the baselines cluster in a narrow band of the plane: they occupy similar helpfulness levels yet differ little in safety, indicating that existing defenses move along the helpfulness axis without meaningfully climbing the safety axis. Our two variants separate cleanly from this cluster, occupying the upper region at helpfulness comparable to the baselines rather than trading one axis for the other. On Qwen3-8B the retuned baselines spread out along the safety axis instead of clustering, and the strongest of them approach our operating region; what distinguishes our variants there is that they reach it without the per-method corrections catalogued in § 3, and they do so at lower over-refusal than MTSA, the closest baseline on the safety axis. The two variants do not, however, relate to each other consistently across backbones. On Qwen3-8B the self-verified reward is the safer and less helpful of the two, as one would expect from a reward that checks a committed label; on Llama3.1-8B the ordering reverses on both axes, with the external reward safer (92.9 versus 90.4 average SAFE %) and less helpful (MT-Bench 5.21 versus 5.86). The two therefore span a frontier the baselines do not reach, but which of them occupies which end is backbone-dependent, and our experiments do not isolate why: the rewards differ both in what they read and in whether a judge is in the loop, and the two backbones differ in whether a native reasoning channel precedes the decision. We flag this as an open question rather than an operating-point choice a practitioner could make a priori.

Implementation Details

All defenders are trained with GRPO in verl ([35]), driven by a custom multi-turn agent loop registered through verl’s agent-loop interface. We train two backbones independently, Llama-3.1-8B-Instruct and Qwen3-8B; Qwen3 is trained with its native reasoning channel enabled, so each turn contains the model’s own <think> block in addition to the scaffold’s output, while Llama-3.1 has no such channel. The policy is a LoRA adapter rather than a full fine-tune (rank 32, $\alpha= 32$, all linear modules), optimized with Adam at learning rate $10^{-4}$ under FSDP2 with gradient checkpointing. We apply a KL penalty toward the frozen reference policy with coefficient 0.001 using the low-variance estimator, and set the entropy bonus to zero. Rollouts are generated by vLLM in asynchronous mode with tensor parallelism 1 and 60% of GPU memory reserved for the rollout engine; the adapter is hot-swapped into the engine between updates, so no weight merge is needed during training.

Each training row is a complete, pre-generated multi-turn attack dialogue, so the attacker is fixed and only the defender is learned. A rollout replays the scripted attacker turns one at a time, and after each the policy generates one full turn capped at 2048 tokens. Generated tokens are appended to the running context and marked trainable; injected attacker turns are appended with mask 0, so no gradient flows through text the policy did not produce. Rollouts run to at most 7 defender turns, with a 1024-token prompt budget and a 12,288-token response budget inside a 14,336-token model window. We draw $n = 16$ rollouts per prompt, a size dictated by the binary reward: GRPO normalizes advantages within each prompt group, so a group whose rollouts all emit the same label has zero variance and contributes no gradient, and a large group is what maintains within-group label diversity.

Baseline reproduction on Qwen3-8B. Each baseline is trained on our rollouts with its published objective and its authors’ hyperparameters. On Llama-3.1 those settings transfer directly. On Qwen3-8B each of the four weight-trained baselines required one change before it improved on the base model at all, and in every case the default produced a defender whose scores were indistinguishable from the untrained backbone. DPO’s published learning rate and epoch count leave the LoRA update far too small on this backbone; we raise the learning rate to $10^{-4}$ and train for eight epochs. MUSE-D’s data builder emits two preference streams, and the mixed stream is only 32% refusals in its chosen field; we train on the risk stream, which is 89% refusals. MSTA fits the implicit reward margin to a target derived from the future-reward gap, and normalizing that gap—as we initially did—drives the target to roughly one nat, which the optimizer reaches while leaving behaviour unchanged; restoring the unnormalized scale of the authors’ formulation puts the target near 350 nats, comparable to the margin plain DPO reaches on the same pairs. X-Boundary needs two: its multi-turn boundary set must be supplied, since without it only the erase branch of the loss is active, and its retain term must be made scale-free. That term is a raw representation distance while the erase term is a cosine, so their balance depends on the backbone’s residual-stream norm, which we measure at 182 on Qwen3 against 10 on Llama-3.1; with the raw form the retain term dominates and the erase objective does not move. None of these is a property of Qwen3 as a model. Each is a setting that carries over silently from the Llama configuration, and each is reported here so the comparison in Table 1 is reproducible.

KL-divergence setup. We measure per-position KL divergence over a fixed, teacher-forced continuation, so all models are scored on the same tokens and any difference reflects the model rather than its sampling. For each held-out attack dialogue, the continuation $y$ is the base model’s greedy final-turn response, generated once and reused verbatim; position $i$ indexes the $i$-th token of $y$, and at each position we compute the exact next-token $D_{\mathrm{KL}}$ over the full vocabulary, averaged across the held-out set. We compare the aligned model (fine-tuned under our Assess $\rightarrow$ Decide $\rightarrow$ Respond framework), the unaligned model (framework prompt, no fine-tuning), and the base model (neither): the aligned–unaligned pair isolates the effect of training, while comparisons against the base isolate the framework itself. All three share identical dialogue content, differing only in system prompt and weights; for reasoning models we disable native thinking so $y$ is a direct continuation. The shaded band marks the decision-token position on the model’s own greedy generations.

Judges correlation setup. We compare seven verdict sources on our Qwen3-8B defender (two-label scaffold, self-verified decision reward): gpt-5-mini ([36]), gpt-oss-20b [1], StrongREJECT [38], Llama-Guard-3-8B (Llama Team, 2024), Qwen3Guard (Zhao et al., 2025), HarmBench-13B [23], and the model’s own committed Decision label. The pool covers all thirteen held-out multi-turn safety cells: five attack families (SEMA, Crescendo, FITD, Echo, HIS) on both AdvBench and HarmBench, plus MHJ, CoSafe and MultiBreak, in total 3738 dialogues. Each source reduces a dialogue to a single binary jailbroken verdict on the final turn. Critically, the six external judges see only the user-facing reply, with the assessment/Decision deliberation stripped.

Prompts

Figure 8 shows the two Assess $\to$ Decide $\to$ Respond system prompts verbatim. Both instruct the defender to precede every reply with an open, untagged deliberation of exactly two lines, an Intent line assessing the user’s underlying goal and its potential for harm, and a Decision line committing to a label, followed by a reply consistent with that decision. The prompts differ only in the label menu and the matching clause of the closing instruction. The two-label variant accompanies the self-verified reward: because that reward checks the emitted label against the row’s known ground truth, every admissible label must have one, and safe-redirect does not. The three-label variant is therefore usable only with an external reward that scores the reply itself, where a hedging option is meaningful. Neither prompt wraps the deliberation in tags; on a backbone with a native reasoning channel, this keeps the <think> block unambiguously the model’s own and the framework cleanly separable from it.

More Experimental Results

Utility

Table 2 breaks the utility average of Table 1 down by benchmark. The pattern is uniform: on both backbones, every method, including ours, stays within a narrow band of its base model on every task, and no benchmark shows a systematic degradation attributable to safety training. What variation exists is concentrated in the instruction-following suite, the one task most sensitive to chat behavior, while the knowledge- and reasoning-centric benchmarks are essentially untouched. This is the expected signature of the evaluation protocol discussed in § 3: these suites are scored as plain-text completion without the chat template, so they bypass both the scaffold and the shallow, format-concentrated component of safety training, and confirm that our intervention leaves the underlying capabilities of the model intact.

Method	MMLU	HellaSwag	Winogrande	GSM8K	TruthfulQA	IFEval	Avg.
Llama3.1-8B-Instruct
Base	68.34	80.49	78.69	69.60	54.55	46.40	66.34
SFT	67.36	79.09	75.37	69.60	53.95	44.55	64.99
DPO	68.28	80.35	78.37	70.13	54.71	46.77	66.44
SDGO	68.31	80.55	77.98	70.28	54.51	45.29	66.15
MTSA	68.30	80.34	78.14	69.52	54.64	47.50	66.41
MUSE-D	68.31	80.45	78.14	69.14	54.72	47.69	66.41
X-Boundary	68.04	80.51	77.66	68.08	54.41	47.50	66.04
Self reward
– unaligned	68.34	80.49	78.69	69.60	54.55	46.40	66.34
– aligned	68.14	80.17	78.14	72.10	54.51	47.69	66.79
External reward
– unaligned	68.34	80.49	78.69	69.60	54.55	46.40	66.34
– aligned	68.10	80.45	78.30	69.22	54.82	43.81	65.78
Qwen3-8B
Base	74.86	76.46	70.40	87.95	54.45	25.69	64.97
SFT	75.03	75.93	70.40	88.02	56.14	29.57	65.85
DPO	74.61	75.80	70.40	87.34	54.40	22.55	64.18
SDGO	74.89	76.44	70.80	87.49	54.35	25.51	64.91
MTSA	74.44	75.39	70.01	87.64	54.25	23.29	64.17
MUSE-D	74.75	75.65	70.24	88.10	54.13	23.48	64.39
X-Boundary	74.85	76.51	70.56	88.02	54.57	25.88	65.07
Self reward
– unaligned	74.86	76.46	70.40	87.95	54.45	25.69	64.97
– aligned	74.95	76.45	69.69	87.41	54.65	24.21	64.56
External reward
– unaligned	74.86	76.46	70.40	87.95	54.45	25.69	64.97
– aligned	74.98	76.54	70.40	87.19	54.29	24.58	64.66

Statistical Significance

We report the precision of Table 1 against the sampling noise of the evaluation: Table 3 restates every cell of Table 1 with a 95% confidence interval. Each safety cell is a SAFE rate over $n$ held-out dialogues ($n = 516$, 785, 500, 1400, and 537 for Adv., Harm., MB, CoS., and MHJ respectively; 3,738 in total), with one exception: the Llama3.1-8B Base MHJ cell is scored over 300 dialogues, and its wider interval reflects that, so its sampling error is that of a binomial proportion; we report the 95% Wilson interval as a half-width, and a behavior-clustered bootstrap interval for the pooled Avg. Over-refusal (XSTest, XSB) carries Wilson intervals ($n = 250$, 340), and MT-Bench a 95% $t$-interval over its 160 per-turn scores. Utility is the mean of six lm-eval tasks; since lm-eval reports an independent standard error per task, the mean carries a combined half-width about one point.

Table 1 with 95% confidence intervals, as value $\pm$half-width. Safety cells and over-refusal (XSTest, XSB) use Wilson intervals; the safety Avg. a behavior-clustered bootstrap; Utility a combined lm-eval standard error; MT-Bench a $t$-interval. Judge: gpt-5-mini.

Method	Adv.	Harm.	MB	CoS.	MHJ	Avg.	Avg.	XSTest $\downarrow$	XSB $\downarrow$	MT-Bench $\uparrow$
Llama3.1-8B-Instruct
Base	54.5 $\pm$4.3	62.5 $\pm$3.4	59.8 $\pm$4.3	84.9 $\pm$1.8	38.0 $\pm$5.4	59.9 $\pm$1.8	66.3 $\pm$1.0	10.0 $\pm$3.7	30.0 $\pm$4.8	6.20 $\pm$0.41
SFT	69.4 $\pm$4.0	79.5 $\pm$2.9	87.0 $\pm$3.0	88.5 $\pm$1.7	55.1 $\pm$4.2	75.9 $\pm$1.4	65.0 $\pm$1.1	14.0 $\pm$4.3	45.6 $\pm$5.3	5.67 $\pm$0.39
DPO	61.6 $\pm$4.2	71.5 $\pm$3.1	79.0 $\pm$3.5	88.3 $\pm$1.7	46.7 $\pm$4.2	69.4 $\pm$1.5	66.4 $\pm$1.0	10.4 $\pm$3.8	34.4 $\pm$5.0	6.26 $\pm$0.42
SDGO	56.6 $\pm$4.2	61.7 $\pm$3.4	60.4 $\pm$4.3	85.0 $\pm$1.9	39.1 $\pm$4.1	60.6 $\pm$1.7	66.2 $\pm$1.0	11.6 $\pm$4.0	30.9 $\pm$4.9	6.11 $\pm$0.42
MTSA	61.6 $\pm$4.2	71.6 $\pm$3.1	79.0 $\pm$3.5	86.6 $\pm$1.8	47.1 $\pm$4.2	69.2 $\pm$1.5	66.4 $\pm$1.0	10.8 $\pm$3.9	36.5 $\pm$5.1	6.12 $\pm$0.41
MUSE-D	61.8 $\pm$4.2	67.5 $\pm$3.2	74.2 $\pm$3.8	87.4 $\pm$1.8	45.3 $\pm$4.2	67.2 $\pm$1.6	66.4 $\pm$1.0	8.4 $\pm$3.5	33.2 $\pm$5.0	6.27 $\pm$0.42
X-Boundary	97.9 $\pm$1.3	95.0 $\pm$1.6	93.0 $\pm$3.0	93.6 $\pm$1.3	52.0 $\pm$4.2	86.3 $\pm$1.2	66.0 $\pm$1.0	16.8 $\pm$4.6	50.3 $\pm$5.3	6.26 $\pm$0.42
Self reward
– unaligned	63.0 $\pm$4.1	70.3 $\pm$3.2	64.2 $\pm$4.2	89.6 $\pm$1.6	44.3 $\pm$4.1	66.3 $\pm$1.7	66.3 $\pm$1.0	8.0 $\pm$3.4	33.8 $\pm$5.0	5.61 $\pm$0.40
– aligned	98.8 $\pm$1.0	99.5 $\pm$0.5	96.6 $\pm$1.7	98.4 $\pm$0.7	58.5 $\pm$4.2	90.4 $\pm$1.0	66.8 $\pm$1.0	20.4 $\pm$5.0	54.4 $\pm$5.3	5.86 $\pm$0.39
External reward
– unaligned	57.2 $\pm$4.2	65.2 $\pm$3.4	56.6 $\pm$4.3	88.5 $\pm$1.7	35.0 $\pm$4.0	60.5 $\pm$1.6	66.3 $\pm$1.0	8.8 $\pm$3.5	32.6 $\pm$5.0	5.81 $\pm$0.41
– aligned	96.9 $\pm$1.5	98.2 $\pm$1.0	99.2 $\pm$0.9	99.0 $\pm$0.6	71.1 $\pm$3.8	92.9 $\pm$0.9	65.8 $\pm$1.0	21.6 $\pm$5.1	50.0 $\pm$5.3	5.21 $\pm$0.39
Qwen3-8B
Base	49.2 $\pm$4.3	52.5 $\pm$3.5	23.8 $\pm$3.7	82.6 $\pm$2.0	22.3 $\pm$3.6	46.1 $\pm$1.6	65.0 $\pm$1.0	7.2 $\pm$3.2	15.6 $\pm$3.9	6.08 $\pm$0.52
SFT	59.9 $\pm$4.2	65.7 $\pm$3.4	49.6 $\pm$4.4	88.5 $\pm$1.7	35.6 $\pm$4.1	59.9 $\pm$1.7	65.9 $\pm$1.0	17.6 $\pm$4.7	40.9 $\pm$5.2	7.53 $\pm$0.33
DPO	80.2 $\pm$3.4	86.6 $\pm$2.4	83.2 $\pm$3.3	97.5 $\pm$0.8	53.6 $\pm$4.2	80.2 $\pm$1.4	64.2 $\pm$1.0	26.0 $\pm$5.4	39.4 $\pm$5.2	5.87 $\pm$0.52
SDGO	48.3 $\pm$4.3	52.5 $\pm$3.5	23.2 $\pm$3.7	82.4 $\pm$2.0	33.0 $\pm$3.9	47.9 $\pm$1.7	64.9 $\pm$1.0	10.0 $\pm$3.7	15.9 $\pm$3.9	6.23 $\pm$0.52
MTSA	79.1 $\pm$3.5	90.2 $\pm$2.1	85.8 $\pm$3.1	97.8 $\pm$0.8	70.9 $\pm$3.8	84.8 $\pm$1.3	64.2 $\pm$1.0	38.4 $\pm$6.0	53.5 $\pm$5.3	5.99 $\pm$0.52
MUSE-D	87.6 $\pm$2.8	90.8 $\pm$2.0	91.0 $\pm$2.5	97.4 $\pm$0.8	61.5 $\pm$4.1	85.7 $\pm$1.2	64.4 $\pm$1.0	20.4 $\pm$5.0	40.9 $\pm$5.2	5.87 $\pm$0.52
X-Boundary	68.0 $\pm$4.0	71.1 $\pm$3.2	67.8 $\pm$4.1	88.8 $\pm$1.7	27.4 $\pm$3.8	64.6 $\pm$1.5	65.1 $\pm$1.0	18.8 $\pm$4.8	49.1 $\pm$5.3	5.69 $\pm$0.52
Self reward
– unaligned	69.0 $\pm$3.9	80.1 $\pm$2.8	66.8 $\pm$4.1	96.4 $\pm$1.0	46.9 $\pm$4.2	71.8 $\pm$1.6	65.0 $\pm$1.0	14.8 $\pm$4.4	37.9 $\pm$5.1	7.33 $\pm$0.36
– aligned	97.1 $\pm$1.5	97.6 $\pm$1.1	97.6 $\pm$1.3	99.3 $\pm$0.4	75.2 $\pm$3.6	93.4 $\pm$0.9	64.6 $\pm$1.0	26.4 $\pm$5.4	56.8 $\pm$5.2	7.36 $\pm$0.34
External reward
– unaligned	68.8 $\pm$4.0	77.3 $\pm$2.9	69.4 $\pm$4.0	95.5 $\pm$1.1	44.3 $\pm$4.2	71.1 $\pm$1.5	65.0 $\pm$1.0	10.0 $\pm$3.7	39.7 $\pm$5.2	7.20 $\pm$0.35
– aligned	90.1 $\pm$2.6	93.6 $\pm$1.7	92.8 $\pm$2.3	98.4 $\pm$0.7	63.1 $\pm$4.0	87.6 $\pm$1.1	64.7 $\pm$1.0	16.8 $\pm$4.6	42.6 $\pm$5.2	6.96 $\pm$0.41

Three trends follow from the intervals. First, the safety conclusions are resolved unevenly, and the intervals are worth reading before the point estimates. The clearest case is the self-verified reward on Qwen3-8B: its average exceeds MUSE-D, the strongest baseline by average, by nearly nine half-widths, and its interval is disjoint from MUSE-D’s on all five suites. That is a statement about MUSE-D specifically. On MHJ — the hardest suite and the one most worth scrutinising — the self-verified interval (75.2 $\pm$ 3.6) overlaps MTSA’s (70.9 $\pm$ 3.8), so on that suite the two are not separated even though the point estimates differ by four points. The external reward on the same backbone is a different matter — its average lead is under two half-widths and its per-suite intervals overlap the strongest baseline’s on every suite, so on that backbone the two should be read as comparable rather than ordered. On Llama3.1-8B both variants clear the strongest baseline on average, but the per-suite intervals overlap X-Boundary’s on AdvBench for both, and on MultiBreak and MHJ for the self-verified variant. What the intervals support throughout is the separation between our variants and the field of baselines, not a resolved ordering against the single strongest one on every suite. Second, the Utility column is indistinguishable across methods: every entry lies within roughly one combined interval of its base model, confirming that no method, ours included, meaningfully changes capability. Third, the helpfulness intervals counsel asymmetric reading: they are wide relative to the small method-to-method differences on XSTest and MT-Bench, which should therefore not be over-interpreted, whereas the elevated over-refusal of the aligned defenders relative to their base models clears the intervals comfortably and is a real cost rather than noise.

Multiple Judges

Table 4 reports safety under five additional judges, all held out from training. The two rewards behave differently here, and the distinction matters. The self-verified reward places first or second in all 25 judge-by-benchmark cells on Qwen3-8B, and is the strict maximum in 22 of them. The three exceptions are CoSafe under Llama-Guard-3, where all four of our rows tie at 100.0; CoSafe under HarmBench-13B, where our external variant is marginally ahead (96.9 against 96.4); and MHJ under Qwen3Guard, the one cell in the panel where a baseline leads it (MTSA, 81.8 against 78.8). The external reward is not comparably robust: it falls outside the top two in 9 of the 25 cells, ranking third or fourth on MHJ under every judge and third or fourth on AdvBench and HarmBench under Qwen3Guard. Claims about judge-independence should therefore be made for the self-verified reward specifically. The baselines also reorder among themselves — MUSE-D leads MTSA under gpt-oss-20b, StrongREJECT and HarmBench-13B, and the two swap under Llama-Guard-3 and Qwen3Guard — so what the judges agree on is the position of the self-verified reward relative to the field, not a total ordering. The lead is also narrower than the ranking implies: averaged over the five judges the self-verified reward exceeds the strongest baseline by +7.1 on MultiBreak but only +2.6 on MHJ and +3.2 on CoSafe, and on CoSafe under Llama-Guard-3 the margin is +0.1 with both our variants at 100.0. The judges do differ in absolute calibration, with some scoring systematically stricter or more lenient than others, which is consistent with the moderate inter-judge agreement quantified in § 4.2; crucially, however, these shifts move all methods together and leave the ranking essentially unchanged. Since our reward never consults any of these judges, this consistency indicates that the measured gains reflect genuinely safer behavior rather than overfitting to the idiosyncrasies of a particular evaluator.

Method	Llama3.1-8B-Instruct Adv.	Llama3.1-8B-Instruct Harm.	Llama3.1-8B-Instruct MB	Llama3.1-8B-Instruct CoS.	Llama3.1-8B-Instruct MHJ	Qwen3-8B Adv.	Qwen3-8B Harm.	Qwen3-8B MB	Qwen3-8B CoS.	Qwen3-8B MHJ
Base	85.3	71.4	63.9	69.6	48.5	62.0	61.0	27.1	60.9	38.2
SFT	77.3	83.9	87.4	74.4	66.0	73.8	75.5	60.4	71.2	48.0
DPO	72.5	78.3	80.1	73.0	58.2	88.0	89.4	82.8	83.0	64.2
SDGO	66.0	70.2	62.7	68.5	52.1	62.2	61.5	28.2	61.9	44.7
MTSA	72.0	78.2	80.4	73.1	56.6	87.8	93.1	85.8	83.9	80.1
MUSE-D	70.1	75.4	75.0	71.7	56.7	92.6	94.4	90.6	85.4	71.5
X-Boundary	97.7	96.1	94.6	81.5	62.2	77.5	77.5	70.8	69.3	41.2
Self reward
– unaligned	72.4	76.9	66.2	75.7	55.8	77.7	83.7	70.0	86.3	60.3
– aligned	99.6	99.9	96.0	86.1	69.3	98.6	98.7	96.2	89.2	82.0
External reward
– unaligned	69.2	73.0	59.3	71.0	49.6	77.9	82.9	71.8	87.1	55.6
– aligned	97.9	99.2	98.2	90.9	79.9	94.8	96.3	92.4	86.0	72.6
Judge: StrongREJECT
Base	85.0	72.3	57.3	42.4	50.9	77.1	67.4	27.2	31.5	39.7
SFT	81.1	87.1	83.8	51.0	67.7	86.6	79.6	62.5	40.3	47.5
DPO	74.5	79.5	74.2	50.7	58.2	93.0	91.7	78.6	62.8	63.5
SDGO	70.4	72.9	57.9	44.3	53.2	77.5	66.6	27.0	31.4	45.7
MTSA	74.1	80.0	77.2	50.7	59.1	93.0	90.3	77.8	66.1	75.0
MUSE-D	74.4	76.9	72.1	48.2	58.1	94.8	93.6	85.0	66.2	71.1
X-Boundary	99.4	98.3	91.3	68.3	67.7	86.4	81.7	66.8	41.9	41.2
Self reward
– unaligned	77.3	79.2	62.6	52.9	60.2	85.0	86.2	69.4	58.6	62.4
– aligned	99.4	99.5	93.0	73.6	74.6	98.8	98.2	92.4	74.0	82.5
External reward
– unaligned	74.6	77.7	58.4	45.0	56.4	90.9	86.8	73.9	55.7	62.3
– aligned	99.6	99.4	96.8	77.4	84.9	98.6	96.6	87.6	67.0	74.6
Judge: Llama-Guard-3
Base	91.6	81.9	58.2	97.4	60.3	83.7	73.6	22.8	97.3	53.1
SFT	92.8	92.5	84.0	98.7	75.8	93.8	87.8	63.2	98.8	67.4
DPO	90.1	88.0	76.6	98.4	70.0	94.8	93.5	77.8	99.4	71.9
SDGO	84.7	80.3	58.8	97.3	66.3	85.1	75.8	20.8	97.4	52.9
MTSA	89.0	88.4	77.8	98.4	70.9	98.1	98.0	81.6	99.9	84.7
MUSE-D	88.4	84.3	72.0	97.9	68.9	98.3	96.6	86.2	99.4	76.0
X-Boundary	80.4	82.1	72.3	93.5	64.6	84.9	80.6	40.4	97.7	59.0
Self reward
– unaligned	89.0	83.8	62.0	97.9	69.1	94.2	91.8	69.8	100.0	73.6
– aligned	99.8	100.0	96.6	100.0	77.8	99.8	99.9	95.4	100.0	88.5
External reward
– unaligned	87.6	82.5	57.8	97.9	62.9	95.2	92.7	74.8	100.0	74.5
– aligned	99.8	99.5	99.8	100.0	85.3	99.4	99.5	93.8	100.0	82.5
Judge: Qwen3Guard
Base	65.3	46.2	61.0	90.1	47.3	42.4	40.8	26.0	93.1	44.3
SFT	52.9	61.9	88.0	91.7	66.3	52.5	55.5	61.4	96.5	53.1
DPO	50.8	58.5	79.2	93.2	58.7	69.0	75.9	84.4	98.1	60.9
SDGO	42.4	46.1	62.0	89.4	53.1	41.7	39.6	26.6	93.2	43.2
MTSA	50.0	58.6	80.0	92.9	58.5	74.8	83.8	87.4	99.3	81.8
MUSE-D	48.3	53.4	75.2	91.6	58.1	79.1	81.8	91.8	98.8	67.6
X-Boundary	36.8	53.2	73.3	82.4	52.3	43.8	47.0	42.8	93.5	48.4
Self reward
– unaligned	49.0	53.4	64.4	90.0	53.3	47.9	58.1	67.6	97.1	56.1
– aligned	98.4	98.6	98.6	99.1	69.6	90.1	91.1	97.8	99.9	78.8
External reward
– unaligned	44.4	51.1	59.0	94.3	49.3	50.8	58.1	66.2	99.3	52.0
– aligned	94.0	96.8	98.8	99.4	78.8	74.4	81.9	92.6	99.5	70.0
Judge: HarmBench-13B
Base	83.8	66.1	59.0	73.1	49.0	52.7	53.0	22.0	71.8	42.8
SFT	74.4	80.5	87.8	77.1	66.7	66.9	70.2	56.0	70.8	52.0
DPO	69.8	72.2	77.8	76.5	59.6	87.2	91.3	84.6	91.4	68.5
SDGO	65.1	65.0	61.0	73.5	52.5	49.4	54.9	20.8	71.6	49.7
MTSA	68.4	73.1	80.2	77.6	58.7	82.2	89.0	82.8	92.5	81.0
MUSE-D	69.6	70.6	75.4	75.9	57.4	94.2	95.4	90.4	92.5	76.4
X-Boundary	95.2	92.8	90.7	84.4	62.6	71.7	75.3	67.0	77.5	48.6
Self reward
– unaligned	70.7	71.7	65.2	82.9	59.4	80.2	84.3	72.0	93.6	65.9
– aligned	99.8	99.6	98.2	93.6	73.0	98.8	98.6	97.8	96.4	84.0
External reward
– unaligned	67.4	69.0	57.0	84.3	52.7	77.3	84.1	75.6	95.0	63.5
– aligned	98.6	98.7	99.6	99.0	82.5	97.9	97.3	96.2	96.9	74.7