betaFeedback

WorksComputer Science

WRDO: Weighted Refusal Direction Optimization in Aligned Language Models

Abstract

Aligned large language models are trained to comply with benign requests while refusing harmful ones. However, recent work shows that this refusal behavior is fragile and can often be disrupted by simple activation- or weight-space interventions. In particular, refusal has been shown to be largely mediated by a single residual-stream direction, typically computed as a difference-of-means vector between harmful and harmless prompts. However, this standard construction aggregates heterogeneous harmful prompts into one uniform average, potentially conflating distinct refusal-related phenomena. In this work, we study whether refusal interventions can be strengthened by learning how to combine prompt-specific refusal directions. We propose Weighted Refusal Direction Optimization (WRDO), a procedure that constructs a composite refusal direction from a bank of per-prompt directions using learned aggregation weights. This approach allows different harmful prompts to contribute unequally to the final intervention direction. We evaluate WRDO against prior single-direction methods on three open-weight language models using three guard judges and HarmBench as an external evaluator. We further measure the effect of the intervention on general model capability through standard utility benchmarks. Finally, we investigate a defense-oriented setting in which the same optimization machinery is used in reverse to restore refusal behavior in an abliterated model checkpoint. Our results show that weighted aggregation provides a more effective and flexible mechanism for manipulating refusal representations.

Introduction

Large language models deployed in user-facing settings are aligned to follow benign instructions and to refuse harmful ones, with this behavior shaped by a combination of supervised fine-tuning, instruction tuning and preference optimization [1, 2]. The resulting refusal mechanism is, in practice, brittle under both prompt-level adversarial attacks [3, 4] and direct interventions on the model itself, which has motivated a mechanistic line of work that asks how refusal is represented internally and which components of the model implement it. A central finding of this line is that refusal is, to a first approximation, mediated by a single direction in the residual stream of the transformer [5]: ablating one difference-of-means vector, computed between activations on harmful and harmless prompts at a chosen layer, is sufficient to disable refusal in a broad set of open-weight aligned models. The same work establishes an equivalence between activation ablation and weight-space orthogonalization of the attention output and feed-forward down projections, which makes the intervention a one-shot edit of the model with no inference-time overhead, and public toolkits such as Heretic [6] operationalize this recipe end-to-end. The standard procedure averages every harmful prompt into a single axis. A disinformation prompt, a malware prompt and a privacy-violation prompt all enter the same mean, and the resulting direction is diffuse in the sense that it conflates phenomena that need not be aligned in the residual stream. A natural alternative is to retain a per-prompt direction rir_i for each training prompt and to combine them with a coefficient vector ww. The simplest combination is the uniform average, r=normalize⁡(∑iri/N)r = \operatorname{normalize}(\sum_i r_i/N), but there is no a priori reason for every prompt to contribute equally. Performance-aware weighted aggregation is known to beat uniform averaging whenever the aggregated components differ in quality, from ensembling ranked classifiers [7] to federated averaging [8] and adapter selection [9], and per-prompt refusal directions are heterogeneous in exactly that way. In this work, we research the question of whether computing refusal direction using the aggregation weights yields a stronger intervention than the single-direction recipe.

Our Contributions:

  • We propose Weighted Refusal Direction Optimization (WRDO), a procedure that obtains a refusal direction by learning aggregation weights ww over a bank of per-prompt directions, together with two concrete instantiations of WRDO based on separable CMA-ES and a proposed GRPO variant with importance sampling (GRPO-IS).

  • We show empirically that the proposed WRDO instantiations outperform all considered prior single-direction baselines across three open-weight models and three guard judges, and that this superiority transfers to HarmBench ruling out judge-specific overfitting.

  • We obtain that WRD-GRPO-IS preserves the general capability of the intervened model substantially better than every other considered method on standard utility benchmarks.

  • We additionally study a defense setup that reuses the same machinery in reverse to restore refusal on an abliterated checkpoint, illustrating an alternative use of the proposed approach.

The remainder of the paper is organized as follows. § 2 formalizes the WRDO problem. § 3 presents the two WRDO instantiations introduced in this work, namely an adaptation of separable CMA-ES and a proposed GRPO variant with importance sampling, together with the three single-direction baselines against which they are compared. § 4 specifies the experimental setup, and § 5–§ 6 report the empirical results, ablations and discussion.

Problem Formulation

An aligned language model is denoted as fθf_\theta, a harmful training set of prompts Dtrainharm={xi}i=1N\mathcal{D}_{\mathrm{train}}^{\mathrm{harm}} = \{x_i\}_{i=1}^{N}, a held-out harmful validation set Dvalharm\mathcal{D}_{\mathrm{val}}^{\mathrm{harm}} and a harmless baseline corpus Dbenign\mathcal{D}_{\mathrm{benign}}. The hidden dimensionality of the residual stream is dd, and LL denotes the number of transformer blocks.

Candidate bank construction

Let h(l)(x)∈Rdh^{(l)}(x) \in\mathbb{R}^{d} denote the residual-stream activation of fθf_\theta on prompt xx at layer ll, evaluated at the last token position. Activations are extracted at every transformer block, including the embedding output, so that each prompt is summarized by a tensor in R(L+1)×d\mathbb{R}^{(L+1)\times d}. The mean benign activation μbenign(l)=Ex∼Dbenign[h(l)(x)]\mu_{\mathrm{benign}}^{(l)} = \mathbb{E}_{x\sim\mathcal{D}_{\mathrm{benign}}}[h^{(l)}(x)] is computed at every layer.

For each training prompt xix_i we define a per-prompt layerwise direction by taking the difference h(l)(xi)−μbenign(l)h^{(l)}(x_i)-\mu_{\mathrm{benign}}^{(l)} and applying L2L_2 normalization at every layer independently,

ri(l)=h(l)(xi)−μbenign(l)∥h(l)(xi)−μbenign(l)∥2,(1)r_i^{(l)} = \frac{h^{(l)}(x_i)-\mu_{\mathrm{benign}}^{(l)}}{\left\|h^{(l)}(x_i)-\mu_{\mathrm{benign}}^{(l)}\right\|_2}, \tag*{(1)}

and stack the per-layer vectors into a layerwise tensor ri∈R(L+1)×dr_i \in\mathbb{R}^{(L+1)\times d}. The candidate bank R={ri}i=1N\mathcal{R} = \{r_i\}_{i=1}^{N} contains one such tensor per harmful training prompt, with N=∣Dtrainharm∣N = \lvert\mathcal{D}_{\mathrm{train}}^{\mathrm{harm}}\rvert. The bank is computed once per model and is shared across all optimization methods.

Weighted direction

A coefficient vector w∈RNw \in\mathbb{R}^{N} aggregates the bank of § 2.1 by a scalar weighting that is shared across layers and across hidden coordinates, followed by a per-layer L2L_{2} renormalization. Concretely, the composite direction at layer ll is

r(l)(w)=∑i=1Nwiri(l)∥∑i=1Nwiri(l)∥2.(2)r^{(l)}(w)=\frac{\sum_{i=1}^{N}w_i r_i^{(l)}}{\left\lVert\sum_{i=1}^{N}w_i r_i^{(l)}\right\rVert_2}. \tag*{(2)}

The per-layer normalization enforces ∥r(l)(w)∥2=1\left\lVert r^{(l)}(w)\right\rVert_2=1 at every layer and removes the overall positive scale of ww, so that only the orientation of the composite at each layer is varied. Intervention intensity is controlled separately through the layerwise schedule described in § 2.3. The single-direction baseline of [5] is recovered, in this notation, by replacing the per-prompt rir_i by a single difference-of-means tensor (computed between the mean harmful and mean benign activations at every layer) and setting w1=1w_1=1 with N=1N=1.

Intervention operator

The primary intervention is weight-space orthogonalization. We adopt the intervention operator of the Heretic toolkit [6]. Given the layerwise composite direction r(l)(w)r^{(l)}(w) of Equation (2), every attention output projection WO(l)W_{O}^{(l)} and every feed-forward down-projection Wdown(l)W_{\mathrm{down}}^{(l)} at block ll is replaced by

W(l)′=W(l)−α(l)r(l)(w)r(l)(w)TW(l),(3)W^{(l)\prime}=W^{(l)}-\alpha^{(l)}r^{(l)}(w)r^{(l)}(w)^{\mathsf{T}}W^{(l)}, \tag*{(3)}

where α(l)\alpha^{(l)} is the fixed layerwise intensity schedule. The full form of α(l)\alpha^{(l)}, together with the values of the auxiliary parameters that we hold fixed throughout the paper, is given in Appendix § 11. The only departure from the Heretic recipe is the choice of r(l)(w)r^{(l)}(w): Heretic plugs in the single direction of [5], whereas WRDO substitutes the per-prompt composite of Equation (2).

The composition of fθf_{\theta} with the operator of Equation (3), evaluated at the layerwise direction r(w)={r(l)(w)}lr(w)=\{r^{(l)}(w)\}_{l} and a given α\alpha, is denoted Tr(w)(θ)T_{r(w)}(\theta). We write πTr(w)(θ)(⋅∣x)\pi_{T_{r(w)}(\theta)}(\cdot\mid x) for the conditional distribution over generations produced by the intervened model under greedy decoding.

Harmfulness metric

For each prompt xx and response yy, every guard model used in this paper outputs a binary safe/unsafe decision J(x,y)∈{0,1}J(x,y)\in\{0,1\}, with J=1J=1 denoting an unsafe response. The attack success rate (ASR) of the intervened model is the average of this per-prompt binary signal over the validation distribution,

ASR⁡J(w)=Ex∼Dvalharm[J(x,πTr(w)(θ)(⋅∣x))].(4)\operatorname{ASR}_{J}(w)=\mathbb{E}_{x\sim\mathcal{D}_{\mathrm{val}}^{\mathrm{harm}}}\left[J\left(x,\pi_{T_{r(w)}(\theta)}(\cdot\mid x)\right)\right]. \tag*{(4)}

In practice, ASR⁡J\operatorname{ASR}_{J} is estimated on a validation batch as the empirical mean of the per-prompt decisions. The optimization objective is the negative attack success rate,

LJ(w)=−ASR⁡J(w),(5)\mathcal{L}_{J}(w)=-\operatorname{ASR}_{J}(w), \tag*{(5)}

which is shared by every WRDO method. Some optimizers may augment this objective with method-specific regularizers when needed (see WRD-GRPO-IS in § 3.2 for a KL-regularized variant). The attack setup minimizes LJ\mathcal{L}_{J},

w⋆=arg⁡min⁡w∈W LJ(w),(6)w^{\star}=\underset{w\in\mathcal{W}}{\arg\min}\ \mathcal{L}_{J}(w), \tag*{(6)}

where the feasible set W⊂RN\mathcal{W}\subset\mathbb{R}^{N} is a per-coordinate box on ww, with bounds and initialization detailed in Appendix § 10.

Methods

Baselines

Basic RD. We refer to the single-direction refusal-ablation procedure of [5] as Basic RD throughout the paper. A single layerwise direction is computed as the per-layer L2L_2-normalized difference of means

rBasic(l)=μharm(l)−μbenign(l)∥μharm(l)−μbenign(l)∥2(7)r_{\mathrm{Basic}}^{(l)} = \frac{\mu_{\mathrm{harm}}^{(l)} - \mu_{\mathrm{benign}}^{(l)}}{\left\lVert\mu_{\mathrm{harm}}^{(l)} - \mu_{\mathrm{benign}}^{(l)} \right\rVert_2} \tag*{(7)}

on the training split, where μharm(l)\mu_{\mathrm{harm}}^{(l)} is the mean activation of the harmful training prompts at layer ll and μbenign(l)\mu_{\mathrm{benign}}^{(l)} is the mean activation of the benign baseline corpus. The resulting layerwise direction is applied through the operator of Equation (3). There is no candidate bank and no optimization. This is the reference against which any benefit of per-prompt aggregation must be measured.

Heretic. We additionally include the public Heretic toolkit [6] as a further baseline. Heretic applies the same weight-space orthogonalization of Equation (3) to the single Basic RD direction, but tunes the layerwise intensity schedule with its own internal Bayesian search rather than using the fixed schedule of Appendix § 11. There is no per-prompt candidate bank and no learned ww; the baseline isolates the contribution of schedule tuning alone, independent of the per-prompt aggregation introduced by WRDO. We run the released pipeline out of the box, with default hyperparameters.

RDO. We further include the gradient-based refusal-direction baseline of [10], denoted RDO. Instead of taking a difference of means as in Basic RD, RDO parameterizes a single refusal direction per layer and optimizes it directly with gradient descent against a refusal-suppression objective on the harmful training split, exploiting differentiability through the model’s hidden activations. We use the released implementation of Wollschläger et al. [10] unchanged: the comparison against Basic RD isolates the contribution of learning the direction itself. As with Basic RD and Heretic, there is no per-prompt candidate bank and no learned aggregation ww.

RDO-CONE. Wollschläger et al. [10] also argue that refusal is mediated by a cone of directions rather than by a single line, and release a concept-cone construction alongside the single-direction RDO objective. This is the prior construction closest to the motivation of WRDO, since it likewise rejects the assumption that one difference-of-means axis captures refusal, so we include it as a separate baseline denoted RDO-CONE. It differs from WRDO in where the multiplicity is placed: RDO-CONE learns a set of mutually constrained directions that span a cone and ablates the whole cone, whereas WRDO keeps a fixed bank of per-prompt directions and learns only how to aggregate them into the composite of Equation (2). We again use the released implementation of Wollschläger et al. [10] unchanged, with the authors’ default cone dimensionality.

WRD optimizers

For optimizing the refusal-direction weights in the WRDO pipeline, we propose two distinct procedures. The first is a WRDO-tailored adaptation of separable CMA-ES, a well-known derivative-free optimizer suited to our low-dimensional setting. The second is a novel scheme based on group-relative policy optimization with importance sampling, designed specifically for the WRDO objective.

WRD-CMAES. CMA-ES [11, 12] is a standard derivative-free optimizer for continuous black-box objectives, well suited to the noisy and gradient-free nature of the WRDO loss. In our setting we apply it to LJ\mathcal{L}_J of Equation (5): the search variables are the aggregation coefficients ww of the candidate bank, the search space is described in Appendix § 10, and each black-box query corresponds to one full WRDO evaluation, i.e. building the composite direction r(w)r(w), intervening the model through the operator of § 2.3, and measuring the resulting ASR on the validation batch. The concrete sampler choice, hyperparameter values and search-space bounds are listed in Appendix § 15.

Relation to evolution strategies. Both instantiations belong to the evolution-strategy family [11, 13, 14]: a distribution over ww is maintained, perturbed samples are scored by the black-box objective, and the distribution is moved towards the better-scoring ones. WRD-CMAES is its canonical member, and WRD-GRPO-IS reads as an ES with importance mixing, with the Gaussian rollout of Equation (11) as the search distribution and the weights v~m\tilde{v}_m as the mixing correction, so running both compares ES against RL under one objective and one budget. The analogy stops at the reward: NES-style updates see only the scalar fitness of each perturbation, whereas WRD-GRPO-IS routes it through token-level policy ratios and can therefore carry the KL term of Equation (8).

WRD-GRPO-IS. The target loss function described in 2.4 is a non-differentiable function, so it is natural to consider a reinforcement learning approach to its optimization. To keep the intervened model close to the clean base model on benign prompts, WRD-GRPO-IS augments the bare WRDO objective of Equation (5) with a KL-divergence penalty against the policy πfθ\pi_{f_\theta} evaluated on a random batch from DbenignD_{\mathrm{benign}}, yielding the method-specific objective

L~J(w)=LJ(w)+βKLw,(8)\widetilde{\mathcal{L}}_J(w) = \mathcal{L}_J(w) + \beta\mathrm{KL}_w, \tag*{(8)}

with β≥0\beta\ge0 and KLw=Ex∼Dbenign[KL(πTr(w)(⋅∣x)∥πfθ(⋅∣x))]\mathrm{KL}_w = \mathbb{E}_{x \sim D_{\mathrm{benign}}}\left[\mathrm{KL}\left(\pi_{T_r(w)}(\cdot\mid x) \mathbin\Vert\pi_{f_\theta}(\cdot\mid x)\right)\right] the KL divergence of the intervened policy from the clean policy on the benign batch. The KL term is specific to WRD-GRPO-IS: the WRD-CMAES sampler consumes the ASR-only signal of Equation (5) without any regularization.

Group Relative Policy Optimization. Let us denote the language model parameterized by θ\theta as a policy πθ\pi_\theta. Given a query xx drawn from a query set DD, it induces a distribution over responses yy, whose likelihood factorizes as

πθ(y∣x)=∏t=1∣y∣πθ(yt∣x,y<t),\pi_\theta(y \mid x) = \prod_{t=1}^{\lvert y\rvert} \pi_\theta(y_t \mid x,y_{<t}),

where ∣y∣\lvert y\rvert denotes the number of tokens in yy. A query-response pair (x,y)(x,y) is scored by a reward model SS, yielding a per-prompt reward S(x,y)∈{0,1}S(x,y) \in\{0,1\}; in our setting S(x,y)=J(x,y)S(x,y) = J(x,y) is the binary judge decision of Equation (4). The standard GRPO algorithm computes a relative advantage for each response within a group of GG responses generated for the same query from the old policy πθold\pi_{\theta_{\mathrm{old}}}. Specifically, GRPO optimizes

JGRPO(θ)=Ex,{yi}∼πθold(⋅∣x)[1G∑i=1G1∣yi∣∑t=1∣yi∣min⁡(ρi,t(θ)A^i,clip⁡(ρi,t(θ),1−ϵ,1+ϵ)A^i)],(9)\mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}_{x,\{y_i\}\sim\pi_{\theta_{\mathrm{old}}}(\cdot\mid x)}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{\lvert y_i\rvert}\sum_{t=1}^{\lvert y_i\rvert}\min\left(\rho_{i,t}(\theta)\widehat{A}_i,\operatorname{clip}\left(\rho_{i,t}(\theta),1-\epsilon,1+\epsilon\right)\widehat{A}_i\right)\right], \tag*{(9)}

where the token-level importance ratio is

ρi,t(θ)=πθ(yi,t∣x,yi,<t)πθold(yi,t∣x,yi,<t),\rho_{i,t}(\theta) = \frac{\pi_\theta(y_{i,t}\mid x,y_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(y_{i,t}\mid x,y_{i,<t})},

We write importance ratios as ρ\rho throughout this section to keep them distinct from the aggregation coefficients w∈RNw \in\mathbb{R}^{N} of § 2.2, which remain the only trainable quantity of WRDO. and all tokens in a response share the same group-relative advantage

A^i=S(x,yi)−mean⁡({S(x,yj)}j=1G)std⁡({S(x,yj)}j=1G).\widehat{A}_i = \frac{S(x,y_i)-\operatorname{mean}(\{S(x,y_j)\}_{j=1}^{G})}{\operatorname{std}(\{S(x,y_j)\}_{j=1}^{G})}.

Group Relative Policy Optimization with Importance Sampling. In the standard GRPO formulation (9), the group of responses for each query is sampled from the old policy πθold\pi_{\theta_{\mathrm{old}}}, and the clipped objective uses the token-level importance ratio between the current and old policies. Here, we consider a stratified data-collection scheme. For each query xx, we sample one response from each of MM behavior policies {μm}m=1M\{\mu_m\}_{m=1}^{M} that are obtained by shifting the original model weights with formula (3), i.e., ym∼μm(⋅∣x)y_m \sim\mu_m(\cdot\mid x). As the resulting group is off-policy with respect to πθold\pi_{\theta_{\mathrm{old}}}, directly applying (9) would generally yield a biased estimate of the GRPO objective. We introduce sequence-level importance sampling weights to correct this mismatch while retaining the original ratio against πθold\pi_{\theta_{\mathrm{old}}}. Specifically, for each response ymy_m sampled from μm\mu_m, we define the sequence-level importance weight

vm=πθold(ym∣x)μm(ym∣x).v_m = \frac{\pi_{\theta_{\mathrm{old}}}(y_m \mid x)}{\mu_m(y_m \mid x)}.

In practice, we stabilize the weights {vm}m=1M\{v_m\}_{m=1}^{M} by applying clipping, denoted as v~m\tilde{v}_m, and treat them as constants with respect to θ\theta during optimization. This reweighting ensures that the overall objective remains consistent with the GRPO surrogate.

Analogous to GRPO, we assign a shared advantage to all tokens in a response. However, to preserve the control-variate property under off-policy sampling, we compute the group baseline using the same sequence-level weights. Given the group {ym}m=1M\{y_m\}_{m=1}^{M} for query xx, we first form the importance-weighted baseline

b(x)=∑m=1Mv~mS(x,ym)∑m=1Mv~m,b(x) = \frac{\sum_{m=1}^{M}\tilde{v}_m S(x,y_m)}{\sum_{m=1}^{M}\tilde{v}_m},

and define the normalized group-relative advantage as

A^m=S(x,ym)−b(x)std⁡({S(x,yj)}j=1M),\widehat{A}_m = \frac{S(x,y_m)-b(x)}{\operatorname{std}\left(\{S(x,y_j)\}_{j=1}^{M}\right)},

where all tokens in ymy_m share the same advantage, i.e., A^m,t=A^m\widehat{A}_{m,t}=\widehat{A}_m for all tt. Finally, our GRPO objective with importance sampling is

JGRPO−IS(θ)=Ex,ym∼μm(⋅∣x)[1M∑m=1Mv~m1∣ym∣∑t=1∣ym∣min⁡(ρm,t(θ)A^m,clip⁡(ρm,t(θ),1−ϵ,1+ϵ)A^m)].(10)\mathcal{J}_{\mathrm{GRPO-IS}}(\theta)=\mathbb{E}_{x,y_m\sim\mu_m(\cdot\mid x)}\left[\frac{1}{M}\sum_{m=1}^{M}\tilde{v}_m\frac{1}{\lvert y_m\rvert}\sum_{t=1}^{\lvert y_m\rvert}\min\left(\rho_{m,t}(\theta)\widehat{A}_m,\operatorname{clip}\left(\rho_{m,t}(\theta),1-\epsilon,1+\epsilon\right)\widehat{A}_m\right)\right]. \tag*{(10)}

where the token-level importance ratio is defined as

ρm,t(θ)=πθ(ym,t∣x,ym,<t)πθold(ym,t∣x,ym,<t).\rho_{m,t}(\theta)=\frac{\pi_{\theta}(y_{m,t}\mid x,y_{m,<t})}{\pi_{\theta_{\mathrm{old}}}(y_{m,t}\mid x,y_{m,<t})}.

Compared with standard GRPO, our objective (10) differs in that the group is collected by stratified sampling from multiple behavior policies {μm}\{\mu_m\}, and a sequence-level importance weight v~m\tilde{v}_m is used to correct the off-policy sampling back to πθold\pi_{\theta_{\mathrm{old}}}.

In our setting, the trainable parameter is the coefficient vector w∈RNw\in\mathbb{R}^{N} of § 2.2 and the base policy πθold\pi_{\theta_{\mathrm{old}}} is the clean model fθf_{\theta}. At each training step, the MM behavior policies {μm}m=1M\{\mu_m\}_{m=1}^{M} are constructed by perturbing the current ww with Gaussian noise,

wm=w+ϵm,ϵm∼N(0,σ2IN),(11)w_m=w+\epsilon_m,\qquad\epsilon_m\sim\mathcal{N}(0,\sigma^2 I_N), \tag*{(11)}

with ϵ0=0\epsilon_0=0. Each wmw_m defines a layerwise composite direction r(l)(wm)r^{(l)}(w_m) via Equation (2) and an intervened model μm=Tr(wm)(θold)\mu_m=T_{r(w_m)}(\theta_{\mathrm{old}}) via the operator of § 2.3. Only ww is updated by the surrogate of Equation (10), while the candidate bank R\mathcal{R}, the base parameters θold\theta_{\mathrm{old}} and the schedule α(l)\alpha^{(l)} are held fixed throughout training.

Experimental Setup

Data

The harmful pool is the union of six public benchmarks, namely AdvBench [3], MaliciousInstruct [15], the TDC2023 [16], HarmBench [17], JailbreakBench [18] and StrongREJECT [19], deduplicated by exact prompt-string match and treated as a single distribution. The deduplicated pool is pre-partitioned into a training set, used only to compute the candidate bank, a validation set, used as the optimization objective in Equation (4), and a test set, used only for the final evaluation. Concrete sampling seeds and the source of the prepared splits are documented in Appendix § 12. The harmless baseline Dbenign\mathcal{D}_{\mathrm{benign}} is a fixed-size sub-sample of Alpaca [20] used at bank-construction time.

Models

We use three open-weight models: Qwen/Qwen3-8B-Base [21], allenai/OLMo-3-1025-7B [22] and Olmo-3-7B-Instruct [22]. All evaluations use greedy decoding (do_sample=False) with a maximum of 100 new tokens per completion, one completion per prompt, the default system prompt of each model and no few-shot context.

Judges

We use three guard models, namely allenai/wildguard [23], Qwen/Qwen3Guard-Gen-8B [24] and meta-llama/Llama-Guard-3-8B [25]. Each guard exposes its own decoding pipeline that maps a (prompt, response) pair to a binary safe/unsafe label, which we use directly as J(x,y)∈{0,1}J(x,y) \in\{0,1\} in Equation (4). The per-judge ASRJ∈[0,1]\mathrm{ASR}_J \in[0,1] is then obtained by averaging these per-prompt binary decisions over the test set and is reported separately for every judge. HarmBench [17] with the official classifier cais/HarmBench-Llama-2-13b-cls provides an additional external evaluator that is never used during optimization.

Metrics

The primary metric is the ASRJ\mathrm{ASR}_J of Equation (4), reported separately for each of the three judges JJ on the held-out test set. The secondary metrics are the utility benchmarks of § 4.5, the HarmBench ASR of § 5.4.

Utility benchmarks

Utility preservation is evaluated through the Language Model Evaluation Harness [26] on arc_easy and arc_challenge [27], mmlu [28], gsm8k and word-level perplexity on wikitext. Accuracy and perplexity are reported separately, with the direction of improvement made explicit in the tables.

Results

Attack setup

Table 1 reports the test-set ASR of every method on the three base models, evaluated separately against the three guard judges, at the two extremes of the peak intensity grid α∈{1.0,3.0}\alpha\in\{1.0, 3.0\} of the layerwise schedule introduced in § 2.3 (Equation (3); full form in Appendix § 11). The intermediate intensities α∈{1.5,2.0}\alpha\in\{1.5, 2.0\} are deferred to Table 5 in Appendix D, where the same conclusions hold. The no-intervention rows provide the absence-of-attack baselines on the same test set. The Heretic entries are duplicated across the α\alpha columns because Heretic does not consume this hyperparameter from the outside: it selects its own shift coefficient internally through the toolkit’s Bayesian search, so a single value is reported per model and judge and repeated across the α\alpha grid for layout consistency. The RDO rows are duplicated for the same reason: RDO does not expose α\alpha as a hyperparameter, the intensity is baked into the gradient-learned direction itself, and the resulting per-model, per-judge value is repeated across the α\alpha grid. To check whether the apparent benefit of WRDO is driven by the per-prompt candidate bank alone or by the learned aggregation on top of it, we introduce the Uniform-avg WRD sanity-check baseline in this experiment. Uniform-avg WRD uses the same candidate bank of § 2.1 as the learned WRDO instantiations, but fixes the coefficients to the uniform vector wi=1/Nw_i = 1/N without any optimization.

ModelMethodα=1.0\alpha= 1.0, WGα=1.0\alpha= 1.0, Q3Gα=1.0\alpha= 1.0, LG3α=3.0\alpha= 3.0, WGα=3.0\alpha= 3.0, Q3Gα=3.0\alpha= 3.0, LG3
Qwen3-8B-BaseNo intervention0.020.020.020.020.020.02
Qwen3-8B-BaseBasic RD0.22 ±0.020.11 ±0.030.44 ±0.020.81 ±0.030.80 ±0.030.91 ±0.04
Qwen3-8B-BaseHeretic0.77 ±0.040.83 ±0.050.81 ±0.050.77 ±0.040.83 ±0.050.81 ±0.05
Qwen3-8B-BaseRDO0.34 ±0.050.41 ±0.040.48 ±0.050.34 ±0.050.41 ±0.040.48 ±0.05
Qwen3-8B-BaseUniform-avg WRD0.08 ±0.0010.25 ±0.010.32 ±0.020.002 ±0.00030.89 ±0.030.92 ±0.02
Qwen3-8B-BaseWRD-CMAES0.88 ±0.050.83 ±0.020.89 ±0.020.91 ±0.050.99 ±0.0020.98 ±0.002
Qwen3-8B-BaseWRD-GRPO-IS0.91 ±0.070.94 ±0.020.87 ±0.020.93 ±0.090.95 ±0.020.96 ±0.01
OLMo-3-1025-7BNo intervention0.080.090.070.080.090.07
OLMo-3-1025-7BBasic RD0.41 ±0.020.51 ±0.030.43 ±0.030.76 ±0.020.72 ±0.030.73 ±0.02
OLMo-3-1025-7BHeretic0.76 ±0.030.73 ±0.040.70 ±0.030.76 ±0.030.73 ±0.040.70 ±0.03
OLMo-3-1025-7BRDO0.61 ±0.040.67 ±0.030.73 ±0.030.61 ±0.040.67 ±0.030.73 ±0.03
OLMo-3-1025-7BUniform-avg WRD0.64 ±0.020.66 ±0.010.68 ±0.020.77 ±0.030.76 ±0.020.79 ±0.01
OLMo-3-1025-7BWRD-CMAES0.87 ±0.020.86 ±0.030.82 ±0.020.88 ±0.010.88 ±0.020.85 ±0.03
OLMo-3-1025-7BWRD-GRPO-IS0.82 ±0.020.86 ±0.030.79 ±0.020.92 ±0.020.97 ±0.030.97 ±0.02
Olmo-3-7B-InstructNo intervention0.0030.0030.0040.0030.0030.004
Olmo-3-7B-InstructBasic RD0.40 ±0.020.42 ±0.010.43 ±0.030.53 ±0.020.61 ±0.030.59 ±0.01
Olmo-3-7B-InstructHeretic0.62 ±0.030.58 ±0.040.69 ±0.040.62 ±0.030.58 ±0.040.69 ±0.04
Olmo-3-7B-InstructRDO0.40 ±0.050.52 ±0.040.53 ±0.040.40 ±0.050.52 ±0.040.53 ±0.04
Olmo-3-7B-InstructUniform-avg WRD0.49 ±0.020.42 ±0.010.45 ±0.030.62 ±0.020.67 ±0.010.67 ±0.01
Olmo-3-7B-InstructWRD-CMAES0.92 ±0.020.89 ±0.010.93 ±0.010.89 ±0.020.93 ±0.020.86 ±0.01
Olmo-3-7B-InstructWRD-GRPO-IS0.92 ±0.010.88 ±0.020.93 ±0.020.93 ±0.010.90 ±0.020.88 ±0.01

Table 1. Attack setup. Test ASR (in [0, 1]) on held-out harmful prompts, reported at the two extremes of the intensity grid, α∈{1.0,3.0}\alpha\in\{1.0, 3.0\}. Judges: WG = WildGuard, Q3G = Qwen3Guard-Gen-8B, LG3 = Llama-Guard-3-8B. Rows for α\alpha-independent methods are repeated across columns for layout consistency. Bold marks the best result in each column within each model block. Higher is a stronger attack. The intermediate intensities α∈{1.5,2.0}\alpha\in\{1.5, 2.0\} are reported in Table 5 of Appendix D. Every intervened cell is the mean over three independent repetitions of the full pipeline, with the standard deviation over those repetitions. Cells that would round to 0.00 are reported at three decimals; none is an exact zero.

ModelMethodα=1.5\alpha= 1.5, WGQ3GLG3α=2.0\alpha= 2.0, WGQ3GLG3
Qwen3-8B-BaseNo intervention0.020.020.020.020.020.02
Basic RD0.33 ±\pm0.010.47 ±\pm0.010.67 ±\pm0.020.39 ±\pm0.030.87 ±\pm0.020.95 ±\pm0.03
Heretic0.77 ±\pm0.040.83 ±\pm0.050.81 ±\pm0.050.77 ±\pm0.040.83 ±\pm0.050.81 ±\pm0.05
RDO0.34 ±\pm0.050.41 ±\pm0.040.48 ±\pm0.050.34 ±\pm0.050.41 ±\pm0.040.48 ±\pm0.05
Uniform-avg WRD0.003 ±\pm0.00020.83 ±\pm0.020.81 ±\pm0.010.002 ±\pm0.00020.89 ±\pm0.010.85 ±\pm0.03
WRD-CMAES0.91 ±\pm0.050.98 ±\pm0.020.93 ±\pm0.020.93 ±\pm0.040.99 ±\pm0.010.98 ±\pm0.02
WRD-GRPO-IS0.92 ±\pm0.080.96 ±\pm0.020.93 ±\pm0.020.94 ±\pm0.090.95 ±\pm0.020.93 ±\pm0.03
OLMo-3-1025-7BNo intervention0.080.090.070.080.090.07
Basic RD0.58 ±\pm0.020.56 ±\pm0.030.48 ±\pm0.030.61 ±\pm0.020.62 ±\pm0.020.69 ±\pm0.03
Heretic0.76 ±\pm0.030.73 ±\pm0.040.70 ±\pm0.030.76 ±\pm0.030.73 ±\pm0.040.70 ±\pm0.03
RDO0.61 ±\pm0.040.67 ±\pm0.030.73 ±\pm0.030.61 ±\pm0.040.67 ±\pm0.030.73 ±\pm0.03
Uniform-avg WRD0.69 ±\pm0.020.52 ±\pm0.020.59 ±\pm0.030.73 ±\pm0.020.73 ±\pm0.030.75 ±\pm0.01
WRD-CMAES0.84 ±\pm0.020.78 ±\pm0.030.81 ±\pm0.020.74 ±\pm0.020.84 ±\pm0.030.81 ±\pm0.02
WRD-GRPO-IS0.81 ±\pm0.020.85 ±\pm0.030.86 ±\pm0.030.89 ±\pm0.020.91 ±\pm0.020.96 ±\pm0.01
Olmo-3-7B-InstructNo intervention0.0030.0030.0040.0030.0030.004
Basic RD0.49 ±\pm0.020.69 ±\pm0.010.52 ±\pm0.030.72 ±\pm0.020.87 ±\pm0.030.75 ±\pm0.02
Heretic0.62 ±\pm0.030.58 ±\pm0.040.69 ±\pm0.040.62 ±\pm0.030.58 ±\pm0.040.69 ±\pm0.04
RDO0.40 ±\pm0.050.52 ±\pm0.040.53 ±\pm0.040.40 ±\pm0.050.52 ±\pm0.040.53 ±\pm0.04
Uniform-avg WRD0.57 ±\pm0.020.53 ±\pm0.020.58 ±\pm0.030.54 ±\pm0.030.69 ±\pm0.020.61 ±\pm0.01
WRD-CMAES0.98 ±\pm0.020.96 ±\pm0.020.92 ±\pm0.010.92 ±\pm0.020.88 ±\pm0.030.93 ±\pm0.01
WRD-GRPO-IS0.93 ±\pm0.020.96 ±\pm0.020.95 ±\pm0.020.93 ±\pm0.010.93 ±\pm0.020.96 ±\pm0.01

Table 5. Attack setup at the intermediate intensities α∈{1.5,2.0}\alpha\in\{1.5, 2.0\}, complementing the main Table 1. Judges: WG = WildGuard, Q3G = Qwen3Guard-Gen-8B, LG3 = Llama-Guard-3-8B. Every intervened cell is a three-seed mean with the standard deviation over those repetitions; the Heretic and RDO rows repeat the values of Table 1 because neither method consumes α\alpha. Bold marks the best result in each column within each model block. Higher is a stronger attack.

Discussion. First, in nearly every configuration the two learned WRDO instantiations (WRD-CMAES and WRD-GRPO-IS) outperform every prior single-direction baseline by a clear margin, establishing the superiority of the weighted-aggregation approach over the difference-of-means recipe, the Heretic toolkit and the gradient-learned RDO baseline. Second, the Uniform-avg WRD baseline is consistently dominated by the learned WRDO instantiations across the α\alpha grid, confirming that learning the aggregation coefficients ww is what drives the gain, not the per-prompt bank alone.

Concept cones and scale

Two comparisons fall outside the grid of Table 1 and are reported in Appendix § 18. Against the concept cone of Wollschläger et al. [10], which treats refusal as a subspace rather than a line, the two are close: RDO-CONE leads by 3.3 ASR points on Qwen3-8B-Base, WRD-GRPO-IS by 5.5 and 8.5 on the two OLMo checkpoints. Repeating the attack setup on the 21B openai/gpt-oss-20b [29] reproduces the ranking with wider margins.

Utility preservation

Table 2 reports utility benchmarks for the same w∗w^* that produced Table 1, per base model, at the two extremes of the intensity grid α∈{1.0,3.0}\alpha\in\{1.0, 3.0\}. Intermediate values α∈{1.5,2.0}\alpha\in\{1.5, 2.0\} are deferred to Table 6 in Appendix E. In this experiment we restrict attention to the WildGuard-optimized runs only: w∗w^* in every cell below is the coefficient vector obtained by optimizing against WildGuard. Accuracy is reported for ARC-easy (ARC-e), ARC-challenge (ARC-c), MMLU and GSM8K; word-level perplexity is reported for WikiText (PPL). As in § 5.1, the Heretic entries are duplicated across the α\alpha columns because Heretic selects its own shift coefficient internally rather than consuming α\alpha from the outside, so a single value per model and metric is repeated across the α\alpha grid for layout consistency.

ModelMethodα=1.0\alpha= 1.0α=3.0\alpha= 3.0
ARC-eARC-cMMLUGSM8KPPLARC-eARC-cMMLUGSM8KPPL
Qwen3-8B-BaseNo intervention0.63 ± 0.010.38 ± 0.010.67 ± 0.0040.84 ± 0.0111.110.63 ± 0.010.38 ± 0.010.67 ± 0.0040.84 ± 0.0111.11
Basic RD0.59 ± 0.010.36 ± 0.010.64 ± 0.0040.67 ± 0.0113.810.26 ± 0.010.23 ± 0.010.23 ± 0.0040.00 ± 0.00> 101210^{12}
Heretic0.51 ± 0.010.30 ± 0.010.27 ± 0.0040.19 ± 0.0113.440.51 ± 0.010.30 ± 0.010.27 ± 0.0040.19 ± 0.0113.44
RDO0.63 ± 0.010.37 ± 0.010.67 ± 0.0040.82 ± 0.0111.160.63 ± 0.010.37 ± 0.010.67 ± 0.0040.82 ± 0.0111.16
WRD-CMAES0.53 ± 0.010.34 ± 0.010.67 ± 0.0030.01 ± 0.003161.180.37 ± 0.010.29 ± 0.010.51 ± 0.0040.004 ± 0.002> 10410^4
WRD-GRPO-IS0.63 ± 0.010.38 ± 0.010.65 ± 0.0040.83 ± 0.0111.150.64 ± 0.010.38 ± 0.010.62 ± 0.0040.83 ± 0.0111.35
OLMo-3-1025-7BNo intervention0.79 ± 0.010.48 ± 0.0150.62 ± 0.0040.53 ± 0.01462.50.79 ± 0.010.48 ± 0.0150.62 ± 0.0040.53 ± 0.01462.5
Basic RD0.80 ± 0.010.48 ± 0.010.61 ± 0.0040.53 ± 0.0162.710.42 ± 0.010.26 ± 0.010.32 ± 0.0040.0008 ± 0.0008432.02
Heretic0.79 ± 0.010.48 ± 0.010.62 ± 0.0040.52 ± 0.0162.610.79 ± 0.010.48 ± 0.010.62 ± 0.0040.52 ± 0.0162.61
RDO0.80 ± 0.010.48 ± 0.010.62 ± 0.0040.56 ± 0.0163.320.80 ± 0.010.48 ± 0.010.62 ± 0.0040.56 ± 0.0163.32
WRD-CMAES0.79 ± 0.010.48 ± 0.010.62 ± 0.0040.52 ± 0.0162.400.79 ± 0.010.48 ± 0.010.617 ± 0.0040.51 ± 0.0161.70
WRD-GRPO-IS0.79 ± 0.010.48 ± 0.0150.62 ± 0.0040.53 ± 0.01462.50.78 ± 0.010.49 ± 0.0150.61 ± 0.0040.55 ± 0.01462.21
Olmo-3-7B-InstructNo intervention0.76 ± 0.010.48 ± 0.010.53 ± 0.0040.67 ± 0.01100.830.76 ± 0.010.48 ± 0.010.53 ± 0.0040.67 ± 0.01100.83
Basic RD0.77 ± 0.010.48 ± 0.010.54 ± 0.0040.67 ± 0.01100.810.42 ± 0.010.31 ± 0.010.17 ± 0.0040.01 ± 0.01> 10510^5
Heretic0.76 ± 0.010.48 ± 0.010.54 ± 0.0040.67 ± 0.01101.270.76 ± 0.010.48 ± 0.010.54 ± 0.0040.67 ± 0.01101.27
RDO0.76 ± 0.010.48 ± 0.010.53 ± 0.0040.65 ± 0.01101.340.76 ± 0.010.48 ± 0.010.53 ± 0.0040.65 ± 0.01101.34
WRD-CMAES0.76 ± 0.010.48 ± 0.010.53 ± 0.0040.68 ± 0.01101.190.74 ± 0.010.46 ± 0.010.50 ± 0.0040.64 ± 0.01118.19
WRD-GRPO-IS0.76 ± 0.010.48 ± 0.010.53 ± 0.0040.67 ± 0.01100.810.76 ± 0.010.47 ± 0.010.53 ± 0.0040.67 ± 0.01100.49

Table 2. Table 2: Utility preservation per base model, reported at α∈{1.0,3.0}\alpha\in\{1.0, 3.0\}. Higher is better for ARC-e, ARC-c, MMLU and GSM8K (accuracy as a fraction in [0,1][0, 1]); lower is better for PPL (WikiText word-level perplexity). Rows for α\alpha-independent methods are repeated across columns for layout consistency. Intermediate values α∈{1.5,2.0}\alpha\in\{1.5, 2.0\} are reported in Table 6 of Appendix E.

ModelMethodα=1.5\alpha= 1.5 ARC-eARC-cMMLUGSM8KPPLα=2.0\alpha= 2.0 ARC-eARC-cMMLUGSM8KPPL
Qwen3-8B-BaseNo intervention0.63 ± 0.010.38 ± 0.010.67 ± 0.0040.84 ± 0.0111.110.63 ± 0.010.38 ± 0.010.67 ± 0.0040.84 ± 0.0111.11
Qwen3-8B-BaseBasic RD0.42 ± 0.010.26 ± 0.010.23 ± 0.0040.008 ± 0.002113.980.29 ± 0.010.22 ± 0.010.23 ± 0.0040.00 ± 0.00> 10710^{7}
Qwen3-8B-BaseHeretic0.51 ± 0.010.30 ± 0.010.27 ± 0.0040.19 ± 0.0113.440.51 ± 0.010.30 ± 0.010.27 ± 0.0040.19 ± 0.0113.44
Qwen3-8B-BaseRDO0.63 ± 0.010.37 ± 0.010.67 ± 0.0040.82 ± 0.0111.160.63 ± 0.010.37 ± 0.010.67 ± 0.0040.82 ± 0.0111.16
Qwen3-8B-BaseWRD-CMAES0.63 ± 0.010.38 ± 0.010.64 ± 0.0040.84 ± 0.0111.250.51 ± 0.010.30 ± 0.010.47 ± 0.0040.04 ± 0.00564.84
Qwen3-8B-BaseWRD-GRPO-IS0.63 ± 0.010.38 ± 0.010.64 ± 0.0040.84 ± 0.0111.170.64 ± 0.010.38 ± 0.010.65 ± 0.0040.85 ± 0.0111.27
OLMo-3-1025-7BNo intervention0.79 ± 0.010.48 ± 0.010.62 ± 0.0040.53 ± 0.0162.50.79 ± 0.010.48 ± 0.010.62 ± 0.0040.53 ± 0.0162.5
OLMo-3-1025-7BBasic RD0.79 ± 0.010.48 ± 0.010.62 ± 0.0040.55 ± 0.0161.810.79 ± 0.010.47 ± 0.010.61 ± 0.0040.50 ± 0.0172.75
OLMo-3-1025-7BHeretic0.79 ± 0.010.48 ± 0.010.62 ± 0.0040.52 ± 0.0162.610.79 ± 0.010.48 ± 0.010.62 ± 0.0040.52 ± 0.0162.61
OLMo-3-1025-7BRDO0.80 ± 0.010.48 ± 0.010.62 ± 0.0040.56 ± 0.0163.320.80 ± 0.010.48 ± 0.010.62 ± 0.0040.56 ± 0.0163.32
OLMo-3-1025-7BWRD-CMAES0.79 ± 0.010.47 ± 0.010.62 ± 0.0040.52 ± 0.0162.390.79 ± 0.010.48 ± 0.010.62 ± 0.0040.51 ± 0.0162.57
OLMo-3-1025-7BWRD-GRPO-IS0.79 ± 0.010.48 ± 0.010.62 ± 0.0040.52 ± 0.0162.880.79 ± 0.010.48 ± 0.010.62 ± 0.0040.53 ± 0.0162.31
Olmo-3-7B-InstructNo intervention0.76 ± 0.010.48 ± 0.010.53 ± 0.0040.67 ± 0.01100.830.76 ± 0.010.48 ± 0.010.53 ± 0.0040.67 ± 0.01100.83
Olmo-3-7B-InstructBasic RD0.75 ± 0.010.47 ± 0.010.53 ± 0.0040.65 ± 0.01102.960.59 ± 0.010.31 ± 0.010.42 ± 0.0040.61 ± 0.01> 10410^{4}
Olmo-3-7B-InstructHeretic0.76 ± 0.010.48 ± 0.010.54 ± 0.0040.67 ± 0.01101.270.76 ± 0.010.48 ± 0.010.54 ± 0.0040.67 ± 0.01101.27
Olmo-3-7B-InstructRDO0.76 ± 0.010.48 ± 0.010.53 ± 0.040.65 ± 0.01101.340.76 ± 0.010.48 ± 0.010.53 ± 0.040.65 ± 0.01101.34
Olmo-3-7B-InstructWRD-CMAES0.77 ± 0.010.47 ± 0.010.53 ± 0.0040.67 ± 0.01101.190.76 ± 0.010.47 ± 0.010.53 ± 0.0040.66 ± 0.01103.42
Olmo-3-7B-InstructWRD-GRPO-IS0.76 ± 0.0080.48 ± 0.010.53 ± 0.0040.67 ± 0.01100.830.76 ± 0.0080.49 ± 0.010.52 ± 0.0040.67 ± 0.01100.22

Table 6. Utility preservation per base model at the intermediate intensities α∈{1.5,2.0}\alpha\in\{1.5, 2.0\}, complementing the main Table 2. Higher is better for ARC-e, ARC-c, MMLU and GSM8K; lower is better for PPL.

Discussion. First, the prior single-direction baselines degrade general capabilities substantially at the larger values of α\alpha: accuracy on ARC and GSM8K collapses and WikiText perplexity grows by orders of magnitude. Heretic is partially protected, since its Bayesian search minimizes a KL divergence to the unintervened model, and it does hold perplexity near the clean value throughout. It is not uniformly utility-preserving, however: on Qwen3-8B-Base it costs 40 MMLU and 65 GSM8K points, so its KL target constrains the benign output distribution without protecting downstream benchmarks. Second, among the learned WRDO instantiations WRD-CMAES exhibits the same degradation pattern as the prior baselines: the Optuna target consumed by the sampler is the bare ASR-only signal without a KL term, so the optimizer is free to find directions that succeed at suppressing refusal but disrupt unrelated capabilities. Third, WRD-GRPO-IS preserves utility best because its surrogate of Equation (10) optimizes the KL-regularized objective of Equation (8), keeping the edit close to the clean policy even at high α\alpha.

HarmBench

We additionally re-evaluate the considered methods through an external benchmark that was not used during optimization, in order to corroborate the significance of the attack-setup results with a training-independent evaluator and to control for potential overfitting to the guard judges of § 4.3. Because HarmBench is designed primarily as an evaluator of instruction-tuned models, in this setup we restrict attention to the instruct variants of our model pool, additionally including Qwen/Qwen3-8B [21].

Table 3 reports HarmBench ASR. We adopt the same HarmBench evaluation setup as [5]: we take the prompt set from [17] and score the responses generated by the intervened model with the official HarmBench classifier cais/HarmBench-Llama-2-13b-cls. As in the utility experiment of § 5.3, we restrict this evaluation to the WildGuard-optimized runs: for every method the intervened model evaluated by HarmBench is the one whose coefficient vector was selected against WildGuard during training. HarmBench itself is never used during optimization and serves as an external evaluator that controls for judge-specific overfitting.

MethodQwen/Qwen3-8B 1.0Qwen/Qwen3-8B 1.5Qwen/Qwen3-8B 2.0Qwen/Qwen3-8B 3.0Olmo-3-7B-Instruct 1.0Olmo-3-7B-Instruct 1.5Olmo-3-7B-Instruct 2.0Olmo-3-7B-Instruct 3.0
No intervention0.070.070.070.070.000.000.000.00
Basic RD0.220.310.460.480.510.770.680.66
Heretic0.870.870.870.870.770.770.770.77
RDO0.390.390.390.390.200.200.200.20
WRD-CMAES0.870.920.970.820.810.850.880.89
WRD-GRPO-IS0.950.940.910.970.830.930.860.92

Table 3. HarmBench Standard ASR (in [0, 1]) computed with cais/HarmBench-Llama-2-13b-cls, varying α\alpha and model. Bold marks the best result in each column within each model block. Higher is a stronger attack.

Discussion. The HarmBench evaluation confirms the superiority of the WRDO approach observed in § 5.1: the learned WRDO instantiations attain the highest HarmBench ASR across the α\alpha grid. The agreement between the guard-judge metric and a classifier that was never used during optimization confirms that the WRDO advantage over the prior single-direction recipes is a real effect rather than an artifact of the training signal.

Defense setup

We additionally consider a defense setup that reverses the role of WRDO: instead of unlocking harmful behavior on an aligned model, we attempt to suppress it on a model whose refusal has already been broken. Notably, we operate on a third-party abliterated checkpoint whose editing procedure is unknown to us, so we work exclusively from its weights without any knowledge of how the refusal removal was performed.

Table 4 reports the defense setup. As the abliterated checkpoint we take the publicly released Orenguteng/Llama-3.1-8B-Lexi-Uncensored-V2, an uncensored derivative of Llama-3.1-8B-Instruct whose refusal behavior has already been removed by a third-party procedure. We apply the same WRDO pipeline to this checkpoint with the same α\alpha grid as in the attack setup, only with reversed sign. For Basic RD this coincides exactly with the defense recipe of § 3.2 of [5]. The Heretic toolkit and RDO do not expose a defense mode and are therefore excluded from this experiment.

Methodα=−1.0\alpha=-1.0 WGα=−1.0\alpha=-1.0 Q3Gα=−1.0\alpha=-1.0 LG3α=−1.5\alpha=-1.5 WGα=−1.5\alpha=-1.5 Q3Gα=−1.5\alpha=-1.5 LG3α=−2.0\alpha=-2.0 WGα=−2.0\alpha=-2.0 Q3Gα=−2.0\alpha=-2.0 LG3α=−3.0\alpha=-3.0 WGα=−3.0\alpha=-3.0 Q3Gα=−3.0\alpha=-3.0 LG3
Abliterated (no defense)0.850.830.840.850.830.830.850.830.840.850.830.84
Basic RD0.84±0.020.84 \pm0.020.85±0.030.85 \pm0.030.82±0.020.82 \pm0.020.79±0.030.79 \pm0.030.84±0.020.84 \pm0.020.79±0.030.79 \pm0.030.80±0.030.80 \pm0.030.87±0.020.87 \pm0.020.87±0.030.87 \pm0.030.63±0.020.63 \pm0.020.68±0.030.68 \pm0.030.74±0.020.74 \pm0.02
WRD-CMAES0.47±0.05\mathbf{0.47 \pm0.05}0.56±0.040.56 \pm0.040.53±0.060.53 \pm0.060.011±0.0040.011 \pm0.0040.023±0.0050.023 \pm0.0050.073±0.0040.073 \pm0.0040.021±0.0050.021 \pm0.0050.053±0.0060.053 \pm0.0060.042±0.0040.042 \pm0.0040.014±0.0050.014 \pm0.0050.086±0.0060.086 \pm0.0060.079±0.0080.079 \pm0.008
WRD-GRPO-IS0.50±0.080.50 \pm0.080.52±0.07\mathbf{0.52 \pm0.07}0.47±0.09\mathbf{0.47 \pm0.09}0.014±0.0070.014 \pm0.0070.034±0.0080.034 \pm0.0080.047±0.006\mathbf{0.047 \pm0.006}0.015±0.007\mathbf{0.015 \pm0.007}0.014±0.003\mathbf{0.014 \pm0.003}0.020±0.005\mathbf{0.020 \pm0.005}0.019±0.0090.019 \pm0.0090.015±0.003\mathbf{0.015 \pm0.003}0.022±0.008\mathbf{0.022 \pm0.008}

Table 4. Defense setup. Test ASR (in [0, 1]) on the abliterated checkpoint, across the sign-reversed intensity α\alpha and the three judges. Lower is a stronger defense. The three intervened rows are means over three independently optimized w∗w^*, with the standard deviation over those runs; the Abliterated row receives no intervention and is reported from a single evaluation. Bold marks the strongest defense per column.

Discussion. The two learned WRDO instantiations clearly win the defense setup, driving the abliterated checkpoint’s ASR far below the no-defense baseline, while Basic RD fails to restore refusal. This shows that the proposed methods adapt to the alternative setup through a simple sign flip, with no change to the optimizer or the candidate bank.

Cross-judge transfer

A cross-judge transfer analysis (Appendix § 17) identifies WildGuard as the most reliable optimization signal among the considered guards: interventions optimized against WildGuard transfer well to the other two judges, while the reverse direction is less stable.

Conclusion

We introduced Weighted Refusal Direction Optimization (WRDO), a procedure that constructs a refusal direction by learning aggregation coefficients ww over a bank of per-prompt directions, and we presented two derivative-free instantiations of the resulting black-box problem: a WRDO-tailored adaptation of separable CMA-ES and a proposed GRPO variant with importance sampling. Across three open-weight models, three guard judges and the HarmBench classifier, the two WRDO instantiations exceed all prior single-direction baselines and transfer cleanly to the defense setup. Among them, WRD-GRPO-IS is the most reliable: it preserves the general capabilities of the intervened model substantially better than every other method considered, and the cross-judge analysis singles out WildGuard as the most reliable optimization signal, so we recommend WRD-GRPO-IS optimized against WildGuard as the default configuration for both auditing and defense.

Limitations

The full protocol of § 5.1–§ 5.5 covers three open-weight checkpoints of comparable size, in the 7–8B parameter range.

A second limitation concerns the WRD-CMAES instantiation: its current implementation consumes the bare ASR-only objective of Equation (5) and does not yet support the KL regularizer of Equation (8). Extending the Optuna interface to expose a KL-aware sampling target is under active development, and we expect to incorporate this support in a future revision of the codebase.

Acknowledgments

The work of G. Evseev, D. Bylinkin, D. Medyakov, V. Aletov, A. Beznosikov was supported by the Ministry of Economic Development of the Russian Federation (agreement No. 139-15-2025-013, dated June 20, 2025, IGK 000000C313925P4B0002).

Ethical Considerations

This work is inherently dual-use: methods for identifying and modifying refusal directions can help analyze and improve model safety, but they can also be used to weaken safeguards in aligned models. In particular, interventions that suppress refusal may increase the risk of harmful compliance in domains such as cyber abuse, privacy violations, disinformation, or other unsafe requests.

We therefore view WRDO primarily as a tool for controlled mechanistic analysis and safety evaluation. Our experiments are conducted offline on open-weight models and use automated guard judges, HarmBench-style evaluation, and utility benchmarks to study both safety and capability effects. The goal is to better understand refusal representations, not to encourage unsafe deployment or misuse.

The broader implication of our findings is that refusal behavior may be concentrated in low-dimensional internal features and can therefore be brittle under targeted interventions. This suggests that refusal alone should not be treated as a complete safety mechanism; robust deployment requires layered defenses, including safer training, adversarial evaluation, monitoring, external classifiers, and system-level policy enforcement.

At the same time, the defense experiment shows a constructive use of the same machinery: WRDO-like methods may help diagnose and partially restore refusal behavior in modified or ablIterated checkpoints. Overall, we hope this work supports more robust alignment methods and encourages careful, transparent evaluation of both the risks and defensive uses of refusal-direction interventions.

Coefficient-Vector Constraints

The coefficients wiw_i in Equation (2) are signed real numbers, with no further normalization across coordinates beyond the per-layer L2L_2 renormalization already built into Equation (2). The feasible set W\mathcal{W} of Equation (6) is the per-coordinate box

W=[wmin⁡,wmax⁡]N(12)\mathcal{W} = [w_{\min}, w_{\max}]^N \tag*{(12)}

with bounds wmin⁡w_{\min} and wmax⁡w_{\max} fixed as hyperparameters of the search and shared across all derivative-free methods of § 3. We do not impose a simplex constraint, a unit-norm constraint or a top-kk sparsity constraint: the box of Equation (12) is the only restriction on ww. All derivative-free methods are initialized at the uniform-average point wi=1/Nw_i = 1/N, which coincides with the Uniform-avg WRD sanity-check baseline of § 5.1.

Layerwise Intensity Schedule

The fixed layerwise schedule α(l)\alpha^{(l)} of Equation (3) is adopted verbatim from the Heretic toolkit [6]. Let ℓ⋆=max_weight_position⋅(L−1)\ell^\star= \texttt{max\_weight\_position} \cdot(L - 1) denote the absolute peak position in the stack, Δ=min_weight_distance⋅(L−1)\Delta= \texttt{min\_weight\_distance} \cdot(L - 1) the half-width of the active window, αmin⁡=min_weight\alpha_{\min} = \texttt{min\_weight} the boundary value of the schedule, and dl=∣l−ℓ⋆∣/Δd_l = \lvert l - \ell^\star\rvert/\Delta the normalized distance of layer ll from the peak. The schedule is

α(l)={α+dl(αmin⁡−α),dl≤1,0,otherwise,(13)\alpha^{(l)} = \begin{cases} \alpha+ d_l(\alpha_{\min} - \alpha), & d_l \le1, \\ 0, & \text{otherwise}, \end{cases} \tag*{(13)}

so that the intensity decays linearly from the peak value α=max_weight\alpha= \texttt{max\_weight} at ℓ⋆\ell^\star to αmin⁡\alpha_{\min} at the window boundary, and no intervention is applied to layers further than Δ\Delta from the peak. We follow the parameter naming of [6] verbatim. Throughout the paper we use max_weight_position=0.7\texttt{max\_weight\_position} = 0.7, min_weight_distance=0.3\texttt{min\_weight\_distance} = 0.3 and αmin⁡=0\alpha_{\min} = 0, leaving α=max_weight∈{1.0,1.5,2.0,3.0}\alpha= \texttt{max\_weight} \in\{1.0, 1.5, 2.0, 3.0\} as the only varied parameter.

Data Splits and Sampling Seeds

Source of the splits. All harmful-prompt splits are loaded from the prepared dataset erinya/dataset_splits on the Hugging Face Hub, with the JSON files splits/harmful_{train,val,test}.json merged from the six public benchmarks listed in § 4.1 after exact-prompt-string deduplication.

Harmful train sample for the bank. At bank-construction time we sample a fixed-size subset of the harmful train split to populate the candidate bank R\mathcal{R}. The number of sampled prompts is N=128N = 128 (ALL_CATEGORIES_HARMFUL_PROMPT_COUNT in the released code), and the sampling seed is 42 (ALL_CATEGORIES_HARMFUL_PROMPT_SEED). The same seed is reused across models and methods so that the candidate bank is built from the same N=128N = 128 prompts in every run.

Harmful validation and test splits. Optimization uses the harmful validation split as the source of Dvalharm\mathcal{D}^{\mathrm{harm}}_{\mathrm{val}} in Equation (4), and the harmful test split as Dtestharm\mathcal{D}^{\mathrm{harm}}_{\mathrm{test}} for the held-out evaluations reported in § 5. Both splits are consumed in order, with a model batch size of 32 per evaluation step (BATCH_SIZE in the released code).

Benign baseline. The harmless baseline Dbenign\mathcal{D}_{\mathrm{benign}} used to compute μbenign(l)\mu_{\mathrm{benign}}^{(l)} in Equation (1) is the first 400 prompts of the train split of mlabonne/harmless_alpaca. A separate 50-prompt subset of the test split of the same dataset is used as the benign batch for the KL penalty in Equation (8).

Optimizer seeds. The Optuna sampler seed is fixed at 42 (OPTUNA_SAMPLER_SEED). For WRD-GRPO-IS, behavior-policy perturbations ϵm∼N(0,σ2IN)\epsilon_m \sim\mathcal{N}(0, \sigma^2 I_N) are drawn from the active PyTorch RNG at training time, with σ\sigma controlled by GRPO_NOISE_SCALE.

Attack Setup at Intermediate α\alpha

Table 5 complements Table 1 by reporting the attack setup of Section 5.1 at the intermediate intensities α∈{1.5,2.0}\alpha\in\{1.5, 2.0\} omitted from the main text. Column conventions, the bold-per-column rule and the three-seed reporting follow Table 1 unchanged; because Heretic and RDO do not consume α\alpha, their rows repeat the values of Table 1. The reading of Section 5.1 carries over without exception: the two learned WRDO instantiations hold the best cell in all model-judge columns of this table, and Uniform-avg WRD remains below them everywhere, so neither conclusion of Section 5.1 depends on the two intensities kept in the main text.

Utility Preservation at Intermediate α\alpha

Table 6 complements Table 2 by reporting the utility benchmarks at the intermediate intensity values α∈{1.5,2.0}\alpha\in\{1.5, 2.0\} omitted from the main text. Column conventions, the WildGuard-only restriction and the Heretic-duplication remark of Section 5.3 apply unchanged.

Training Hyperparameters

This section documents the training-time hyperparameters of all six methods used in the main experiments. All values are reproduced in Table 7; the paragraphs below give a short narrative for each method. For the prior baselines (Basic RD, Heretic, RDO) we use the hyperparameter values from their original implementations without any additional tuning.

MethodHyperparameterValue
Basic RDkl_threshold0.1
induce_refusal_threshold0.0
prune_layer_percentage0.2
Hereticn_trials / n_startup_trials200 / 60
kl_divergence_scale / _target1.0 / 0.01
refusal_threshold_count0
max_weight range[0.8, 1.5]
min_weight range[0.0, 1.0]
seed42
RDOoptimizerAdamW
lr10−210^{-2}
batch_size / effective_batch_size1 / 16
epochs / patience / n_lr_reduce1 / 5 / 2
(λabl,λadd,λret)(\lambda_{\mathrm{abl}}, \lambda_{\mathrm{add}}, \lambda_{\mathrm{ret}})(1.0, 0.2, 1.0)
num_target_tokens8
target_generation_batch_size64
seed42
WRD-CMAESsamplerCmaEsSampler (separable)
n_trials / sampler_seed50 / 42
(wmin⁡,wmax⁡)(w_{\min}, w_{\max})(−2.0, +2.0)
init / modeuniform-avg / scalar
WRD-GRPO-ISoptimizerAdamW
learning_rate10−310^{-3}
n_groups (MM)8
n_epochs50
σ\sigma (noise_scale)1
clip_ratio / is_clip_ratio0.2 / 5.0
β\beta (KL weight)1000.0
ref_alpha1.0

Table 7. Table 7: Training hyperparameters for the six methods of § 3. Values are read verbatim from the configuration files in the released codebase. Hyperparameters shared across methods (sampling seed 42, batch size 16 or 32, maximum response length, system prompt) are listed in Appendix § 12.

Basic RD. Basic RD performs no learned optimization. The only tunable knobs are the dataset filters used to construct the harmful/harmless splits and to select the seed difference-of-means direction: harmful prompts that the unintervened model already refuses are discarded if their refusal-induction score is below induce_refusal_threshold, candidate directions whose KL to the clean policy on the benign batch exceeds kl_threshold are dropped, and the bottom prune_layer_percentage of layers is excluded from the layer-search. All three thresholds are taken verbatim from the reference implementation of Arditi et al. [5].

Heretic. Heretic runs an Optuna-based Bayesian search over the layerwise schedule of Equation (3). The search budget is n_trials with n_startup_trials random-initialization trials, and uses a KL constraint with scale kl_divergence_scale and target kl_divergence_target. Trial selection on the Pareto front prefers the minimum-KL trial whose refusal count is at most refusal_threshold_count. The search ranges over max_weight ∈[0.8,1.5]\in[0.8, 1.5], max_weight_position relative to depth in [0.6,1.0][0.6, 1.0], min_weight ∈[0.0,1.0]\in[0.0, 1.0], min_weight_distance relative to depth in [0.05,0.6][0.05, 0.6], the direction-index fraction in [0.4,0.9][0.4, 0.9], and the schedule scope ∈{global,per-layer}\in\{\text{global}, \text{per-layer}\}. We use the released Heretic toolkit out of the box, with the defaults of Weidmann [6] for every hyperparameter listed above.

RDO. RDO trains a single per-layer direction by gradient descent with AdamW. The training loop runs for epochs steps with patience-based early stopping and at most n_lr_reduce learning-rate reductions, at learning rate lr and effective batch size effective_batch_size accumulated from micro-batches of size batch_size. The composite loss is a weighted sum of an ablation term (weight_ablation_lambda), an addition term (addition_lambda) and a retain term (retain_lambda). Target sequences for the loss are generated greedily up to num_target_tokens tokens in batches of target_generation_batch_size. All values match the reference implementation released by Wollschläger et al. [10].

WRD-CMAES. WRD-CMAES uses the Optuna [30] implementation of separable CMA-ES through the CmaEsSampler interface. The search variables are the coefficients w∈RNw \in\mathbb{R}^{N} of § 2.2, constrained to the box [wmin⁡,wmax⁡]N[w_{\min},w_{\max}]^{N} of Appendix § 10. The total query budget is n_trials, the sampler seed is sampler_seed, the per-coordinate bounds are wmin⁡=−2.0w_{\min}=-2.0 and wmax⁡=2.0w_{\max}=2.0, and the parameterization is the scalar mode initialized at the uniform-average point (wi=1/Nw_i=1/N). All other sampler hyperparameters are left at library defaults.

WRD-GRPO-IS. WRD-GRPO-IS optimizes the surrogate of Equation (10) with AdamW at learning rate learning_rate. Each training step samples a group of n_groups behaviour policies {μm}\{\mu_m\} from the Gaussian rollout of Equation (11) with noise scale σ\sigma = noise_scale, and a fresh group is drawn every epoch over n_epochs total epochs. The PPO/GRPO clip ratio is clip_ratio and the importance-sampling clip ratio is is_clip_ratio; loss is aggregated with the loss_agg_mode reduction. The KL penalty of Equation (8) is realized with coefficient β\beta = beta. The reference-policy mixture weight is ref_alpha. The reward is the harmfulness score with the sign of Equation (6), and ww is parameterized in the scalar mode initialized at the uniform-average point.

Compute Resources

Every single run reported in this paper was conducted on a single NVIDIA H100 GPU with 80 GB of memory. Table 8 reports the approximate wall-clock runtime per method for the attack-setup experiments of § 5.1.

MethodApprox. runtime, hours
Basic RD∼2\sim2
Heretic∼7\sim7
RDO∼3\sim3
Uniform-avg WRD∼2\sim2
WRD-CMAES∼4\sim4
WRD-GRPO-IS∼11\sim11

Table 8. Approximate wall-clock runtime per method on the attack-setup experiments of § 5.1, on a single NVIDIA H100 80 GB GPU.

Cross-Judge Transfer

The learned WRDO instantiations consume a specific guard judge as their training signal, which raises the possibility that the resulting coefficient vector w∗w^* exploits idiosyncrasies of that judge rather than capturing a transferable refusal-suppression direction. We probe this potential judge overfitting through a cross-judge transfer analysis: w∗w^* is optimized against each guard in turn and then re-evaluated against the other two, with the magnitude of the off-diagonal drop quantifying the extent to which performance is judge-specific.

Table 9 reports the cross-judge transfer matrix obtained by optimizing w∗w^* against one guard judge and then evaluating the resulting intervention against each of the three guard judges, for every value of α∈{1.0,1.5,2.0,3.0}\alpha\in\{1.0, 1.5, 2.0, 3.0\}. Large off-diagonal drops indicate that the optimizer has fit features specific to the training judge rather than a transferable refusal-suppression signal. The matrix is reported for the three judge-optimized methods; Basic RD, Heretic and RDO have no judge dependence and are omitted.

ModelMethodOpt. judgeα=1.0\alpha= 1.0 WGα=1.0\alpha= 1.0 Q3Gα=1.0\alpha= 1.0 LG3α=1.5\alpha= 1.5 WGα=1.5\alpha= 1.5 Q3Gα=1.5\alpha= 1.5 LG3α=2.0\alpha= 2.0 WGα=2.0\alpha= 2.0 Q3Gα=2.0\alpha= 2.0 LG3α=3.0\alpha= 3.0 WGα=3.0\alpha= 3.0 Q3Gα=3.0\alpha= 3.0 LG3
Qwen3-8B-BaseWRD-CMAESWG0.890.890.920.910.920.950.950.960.960.900.930.92
Q3G0.000.840.970.000.980.960.001.000.970.001.000.97
LG30.000.730.880.020.940.930.160.960.980.010.980.98
WRD-GRPO-ISWG0.910.940.900.930.950.920.950.960.940.950.960.93
Q3G0.910.940.900.940.950.920.010.960.980.940.950.93
LG30.870.920.870.940.960.920.360.420.390.910.950.97
OLMo-3-1025-7BWRD-CMAESWG0.860.940.500.830.750.200.740.700.350.890.890.49
Q3G0.850.860.550.770.790.530.820.830.380.940.870.46
LG30.650.690.820.630.650.800.690.740.800.910.910.85
WRD-GRPO-ISWG0.830.760.420.800.730.400.890.790.500.910.890.43
Q3G0.620.860.300.650.840.320.850.910.450.920.970.50
LG30.600.650.790.650.700.850.800.850.950.880.920.96
Olmo-3-7B-InstructWRD-CMAESWG0.910.910.950.980.960.970.920.950.930.880.830.76
Q3G0.170.890.820.290.950.960.620.870.860.760.930.88
LG30.170.960.920.220.970.910.590.940.920.790.880.85
WRD-GRPO-ISWG0.900.950.210.930.900.470.930.980.580.930.970.72
Q3G0.230.880.780.350.960.800.580.920.900.780.910.80
LG30.220.300.910.300.420.940.450.580.950.800.850.87

Table 9. Cross-judge transfer. Test ASR (in [0,1][0, 1]) when w⋆w^{\star} is optimized against the row judge and evaluated against the column judge, reported per base model, for every α∈{1.0,1.5,2.0,3.0}\alpha\in\{1.0, 1.5, 2.0, 3.0\}. Within each α\alpha block, diagonal cells coincide with the corresponding entries of Table 1 or, for α∈{1.5,2.0}\alpha\in\{1.5, 2.0\}, of Table 5.

Discussion. The cross-judge analysis shows that learned WRDO interventions do not always transfer equally across guard judges. In several cases, optimizing against one judge yields high ASR under that same judge but noticeably weaker performance under another, indicating that part of the learned direction may depend on judge-specific decision boundaries rather than on a fully transferable refusal-suppression signal.

The most pronounced asymmetry is observed for WildGuard: interventions that perform well under WildGuard tend to transfer relatively well to the other judges, while interventions optimized for other judges can fail to preserve high WildGuard ASR. This suggests that WildGuard provides the most reliable and demanding optimization signal among the considered guards.

By contrast, Llama-Guard-3 often reports lower ASR than the other judges. We interpret this cautiously: rather than necessarily indicating weaker attacks, this may reflect lower sensitivity of Llama-Guard-3 to harmful behavior in the evaluated generations, leading to more false negatives or inconsistent harmfulness detection. As a result, lower Llama-Guard-3 scores should not be treated as definitive evidence that the intervention is less effective.

Concept Cones and Scale

Two comparisons fall outside the grid of Table 1. First, the closest prior work argues that refusal occupies a subspace rather than a line [10], so we evaluate its concept-cone construction RDO-CONE (§ 3.1) on the same three models and test split. Second, all three checkpoints sit in the 7–8B range, so we repeat the attack setup on openai/gpt-oss-20b [29], a 21B mixture-of-experts model with 3.6B active parameters, leaving the pipeline of § 5.1 unchanged. Both are scored by WildGuard and reported in Table 12.

ModelMethodASR
Qwen3-8B-BaseRDO-CONE0.953
Qwen3-8B-BaseWRD-GRPO-IS0.93±0.090.93 \pm0.09
OLMo-3-1025-7BRDO-CONE0.865
OLMo-3-1025-7BWRD-GRPO-IS0.92±0.020.92 \pm0.02
Olmo-3-7B-InstructRDO-CONE0.845
Olmo-3-7B-InstructWRD-GRPO-IS0.93±0.010.93 \pm0.01
gpt-oss-20bNo intervention0.001
gpt-oss-20bBasic RD0.308
gpt-oss-20bHeretic0.772
gpt-oss-20bWRD-CMAES0.923
gpt-oss-20bWRD-GRPO-IS0.931

Table 12. Concept-cone comparison. Test ASR under WildGuard; WRDO rows are taken at α=3.0\alpha= 3.0. The WRD-GRPO-IS rows of the upper blocks repeat the corresponding cells of Table 1 and carry its three-seed spread; the RDO-CONE and gpt-oss-20b rows are single runs. Higher is a stronger attack.

Discussion. The two constructions perform comparably, with the ordering depending on the checkpoint rather than favouring either consistently. Learned weighted aggregation is therefore competitive with an explicitly multi-dimensional cone while keeping the operator of Equation (3) rank-one per layer. We read this as evidence that the gain of WRDO comes from where the harmful prompts are allowed to disagree rather than from the dimensionality of the ablated subspace.

External Evaluation of the Defense Setup

Table 11 re-scores the defended checkpoints of § 5.5 with the external HarmBench classifier, which plays no part in the optimization, and therefore complements the three guard judges of Table 4. At α≤−1.5\alpha\le-1.5 both learned instantiations hold HarmBench ASR at or below 0.03 against 0.94 for the undefended checkpoint, whereas Basic

Method−1.0-1.0−1.5-1.5−2.0-2.0−3.0-3.0
Abliterated (no defense)0.940.940.940.94
Basic RD0.940.920.910.31
WRD-CMAES0.470.030.000.03
WRD-GRPO-IS0.470.020.000.00

Table 11. Defense setup evaluated externally. HarmBench Standard ASR computed with cais/HarmBench-Llama-2-13b-cls on the abliterated checkpoint, across the sign-reversed intensity α\alpha. HarmBench is never used during optimization. Lower is a stronger defense.

RD stays above 0.90 until α=−3.0\alpha= -3.0, an intensity at which its utility has already collapsed in the attack direction (Table 2).

Sensitivity to the KL Coefficient β\beta

The KL coefficient β\beta of Equation (8) is the single hyperparameter that sets where WRD-GRPO-IS sits on the attack-versus-utility trade-off, so the value reported throughout the paper is not an arbitrary default. We select it by sweeping β∈{1,10,102,103,104}\beta\in\{1, 10, 10^{2}, 10^{3}, 10^{4}\} on Qwen3-8B-Base at α=1.0\alpha= 1.0, with everything else held at the values of Table 7, and reading ASR and utility off the same run. Table 10 and Table 13 report both sides of the sweep. The selection is made once, on this model, and the resulting value is reused unchanged for every other model and judge in the paper; we did not tune β\beta per model.

β\beta11010210^{2}10310^{3}10410^{4}
ASR (WG)0.950.920.910.910.78

Table 10. Attack side of the β\beta sweep. Test ASR under WildGuard for WRD-GRPO-IS on Qwen3-8B-Base at α=1.0\alpha= 1.0. Higher is a stronger attack.

β\betaARC-eARC-cMMLUGSM8KPPL
10.520.340.250.38261.32
100.510.360.340.43260.13
10210^{2}0.600.360.590.67150.72
10310^{3}0.630.380.650.8311.15
10410^{4}0.630.380.650.8411.12

Table 13. Utility side of the β\beta sweep, same runs as Table 10. Higher is better for the four accuracies, lower is better for WikiText perplexity. Standard errors are 0.01 for ARC-e, ARC-c and GSM8K and 0.004 for MMLU. Bold marks the best value per column.

The two sides of the sweep move in opposite directions, but not at the same rate. ASR falls gradually as β\beta grows, staying almost flat over the middle of the range before dropping off at the largest value. Utility does not track it: it stays poor while the penalty is weak, with MMLU and GSM8K far below the clean model and perplexity an order of magnitude above it, and then recovers abruptly in a single step to the unintervened level. The two regimes overlap in one place, and that is where we operate: β=103\beta= 10^{3} is the only setting that combines an ASR at the top of the flat region with utility indistinguishable from the clean checkpoint, and it is the value used throughout the paper.

References

  1. [1]Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.arxiv.org/abs/2203.02155
  2. [2]Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.arxiv.org/abs/2204.05862
  3. [3]Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023.arxiv.org/abs/2307.15043
  4. [4]Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail?, 2023.arxiv.org/abs/2307.02483
  5. [5]Andy Arditi, Oscar Obeso, Aaquiib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction, 2024. URL https://arxiv.org/abs/2406.11717.
  6. [6]Philipp Emanuel Weidmann. Heretic: Fully automatic censorship removal for language models. https://github.com/p-e-w/heretic, 2025.
  7. [7]Mehmet Eren Ahsen, Robert M. Vogel, and Gustavo A. Stolovitzky. Unsupervised evaluation and weighted aggregation of ranked classification predictions. Journal of Machine Learning Research, 20(166):1–40, 2019.
  8. [8]Abdullah Abdul Sattar Shaikh, M. S. Bhargavi, and Pavan Kumar C. FedWAPR: Bridging theory and practice in probability-driven weighted aggregation for federated learning. Information Sciences, 723:122697, 2026.DOI
  9. [9]Andrey Veprikov, Vladimir Solodkin, Alexander Zyl, Andrey Savchenko, and Aleksandr Beznosikov. Weight-LoRA: Keep only necessary adapters. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL), pages 12419–12437, 2026.
  10. [10]Tom Wollschläger, Jannes Elstner, Simon Geisler, Vincent Cohen-Addad, Stephan Günnemann, and Johannes Gasteiger. The geometry of refusal in large language models: Concept cones and representational independence. arXiv preprint arXiv:2502.17420, 2025.arxiv.org/abs/2502.17420
  11. [11]Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation, 9(2):159–195, 2001.pubmed.ncbi.nlm.nih.gov/11382355
  12. [12]Nikolaus Hansen. The cma evolution strategy: A tutorial, 2023. URL https://arxiv.org/abs/1604.00772.
  13. [13]Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies, 2014.
  14. [14]Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning, 2017.arxiv.org/abs/1703.03864
  15. [15]Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-source LLMs via exploiting generation, 2023.arxiv.org/abs/2310.06987
  16. [16]Mantas Mazeika, Andy Zou, Norman Mu, Long Phan, Zifan Wang, Chunru Yu, Adam Khoja, Fengqing Jiang, Aidan O’Gara, Zhen Xiang, Arezoo Rajabi, Dan Hendrycks, Radha Poovendran, Bo Li, and David Forsyth. TDC 2023 (LLM Edition): The trojan detection challenge. NeurIPS 2023 Competition Track, 2023. URL https://neurips.cc/virtual/2023/competition/66583.
  17. [17]Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. HarmBench: A standardized evaluation framework for automated red teaming and robust refusal, 2024.arxiv.org/abs/2402.04249
  18. [18]Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Seh w ag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong. JailbreakBench: An open robustness benchmark for jailbreaking large language models, 2024.arxiv.org/abs/2404.01318
  19. [19]Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. A StrongREJECT for empty jailbreaks, 2024.arxiv.org/abs/2402.10260
  20. [20]Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpaca: A strong, replicable instruction-following model, 2023.
  21. [21]Qwen Team. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.
  22. [22]OLMo Team. Olmo 3, 2025. arXiv:2512.13961.arxiv.org/abs/2512.13961
  23. [23]Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. WildGuard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of LLMs, 2024.arxiv.org/abs/2406.18495
  24. [24]Qwen Team. Qwen3Guard technical report, 2025. arXiv:2510.14276.
  25. [25]Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. LlamaGuard: LLM-based input-output safeguard for human-AI conversations, 2023.arxiv.org/abs/2312.06674
  26. [26]Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL https://zenodo.org/records/12608602.
  27. [27]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge, 2018.arxiv.org/abs/1803.05457
  28. [28]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2020.
  29. [29]OpenAI. gpt-oss-120b and gpt-oss-20b model card, 2025.arxiv.org/abs/2508.10925
  30. [30]Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In KDD, 2019.arxiv.org/abs/1907.10902

Paper details

vlnc:W_wuaut2tz
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents