betaFeedback

WorksComputer Science

Attacker Diversity Reduction in Adaptive LLM Red Teaming

Abstract

Reinforcement learning can make automated red-teaming more successful while narrowing the attacks it produces. Co-training an attacker and a defender with group-relative policy optimization (GRPO) has been reported to reduce attacker diversity. This experiment reproduces attack diversity reduction using three runs with a Qwen2.5-14B attacker and a Phi-4-mini defender trained by direct preference optimization (DPO). The attacker receives a success-only reward and uses a length-unbiased loss, bounded updates, warm-up and ratio clipping. The attacker is evaluated on 26 AdvBench behaviors. Training ended after 12 rounds, following a decision based on development checkpoints, so the reported intervals are nominal. Attack Diversity Retention (ADR) compares the within-behavior of the trained attacker spread with the untrained attacker. In order for an attack diversity to be consider as having been reduced, a criterion requires the ADR to be below a 0.75 target with a 95% confidence interval together with continued attacker effectiveness. Therefore, if an attacker narrows its attack vector by 25% consistently its considered less diverse. After 24 attacker and 12 defender LORA-based updates, ADR for the three runs were 0.63, 0.61 and 0.63, respectively. Each run met the criterion at round 12; the earlier checks at rounds 4 and 8 fell short. Despite attacks being less diverse, attack success continued to improve as the attacker was updated after each training run.

Introduction

Red teaming probes how generative AI systems fail in use. Lessons from red teaming more than 100 products emphasize human judgment as an essential part of this work [2]. PyRIT is used for recurring tasks [3], and the “break-fix” cycle connects red teaming to safety post-training [4]. Training a model to generate attacks extends beyond hand-written test cases [5], [6], and reinforcement learning lets that model adapt to the defender it faces.

Bullwinkel et al. AdvGRPO study [1] reports diversity reduction during attacker–defender co-training with group-relative policy optimization (GRPO) [7]. This experiment aims to reproduce the attack diversity reduction with specific models, stabilized updates and pre-specified diversity reduction criterion. An attacker that repeatedly applies similar attacks gives the defender a narrower range of failures to learn from and successfully defend. In order to make AI models that defend against diverse attack vectors, attack diversity needs to be maintained throughout a red teaming exercise. In this experiment the attack narrowing phenomena is tracked with open-weight models and public benchmarks.

A reproduction needs an attacker that remains usable throughout training. Per-completion loss normalization gives tokens in long responses less weight and can bias GRPO toward longer responses [8]. Uniform-reward groups have zero advantage, leaving a sparse set of rows to drive an update [9]. Many steps with a newly initialized optimizer can also move the policy far from the one that generated the data. The design of the experiment addresses these artifacts with four stabilization techniques and a pre-specified rule for stopping a degenerate run.

This experiment attempts to answer three questions: does attacker diversity reduce under this schedule and update rules, does attack diversity deteriorate over training updates, and how does attack diversity relate to attack success? The experiment has one success-reward training arm and focuses on reproduction. The experiment follows three runs, measuring diversity after each attacker update and checking for reduction after four co-training rounds.

Contributions.

  1. A specified open-weight co-training procedure: a Qwen2.5-14B attacker, a Phi-4-mini defender, public benchmarks, four optimizer stabilization techniques and a degenerate-run rule.

  2. Attack Diversity Retention (ADR): paired, within-attack behavior diversity relative to the untrained attacker, with confidence intervals and an exploratory length-matched comparison.

  3. A training trajectory: measurements of diversity, attack success and strategy composition are captured throughout three runs.

Pre-Specified Design

Models, Data and Selection Firewall

Table I lists the models and settings. 520 AdvBench behaviors [6] is split into 286 training, 26 development and 208 test behaviors using the master seed. The second test set contains 172 of HarmBench standard behaviors [14], after excluding the chemical and biological category. Benign prompts drawn from the Alpaca instructions [17] are split into 200 retention, 56 development and 100 test prompts. A count or hash mismatch blocks a run. All splits are prepared, hashed and stored before optimization.

Each run uses its own adapters, seeded training draws and attack banks from these established public benchmarks. Frozen benign attacker answers are shared. AdvBench training supplies GRPO behaviors, and benign training provides retention targets for the defender. Evaluation during these runs uses the AdvBench and benign development splits.

The attacker prompt supplies the behavior and the context of an authorized safety evaluation. Strategy descriptions and @sreferences

SettingValue
AttackerQwen2.5-14B-Instruct [10], low-rank adaptation (LoRA) [11] rank 16, α=32\alpha= 32
DefenderPhi-4-mini-instruct [12], LoRA rank 16, α=32\alpha= 32, direct preference optimization (DPO) [13]
JudgeHarmBench-Llama-2-13b classifier [14]
Embedderbge-large-en-v1.5 [15]
Attacker optimizerAdamW [16], no weight decay, learning rate 2×10−42 \times10^{-4}, gradient-norm clip 1.0, eight rows per step
Kullback–Leibler (KL) weight0.005 to the base model
Group size, temperature, tokens16, 1.0, at most 300 new tokens
Training panelone seeded 96-behavior panel per run, reused
Jobs per round2, each from a fresh rollout
Stabilization Techniquesconstant loss normalizer 300; at most 512 rows, skip below 64; four-step warm-up; clip ϵ=0.2\epsilon= 0.2
Defender DPOonce per round if at least 4 successful pairs; learning rate 2×10−52 \times10^{-5}, 1 epoch, β=0.1\beta= 0.1
Pairs per roundat most 64 successful-attack and 32 benign retention pairs
Defender decodingtemperature 0, at most 600 new tokens
Diversity reduction checksevery 4 rounds
Development checkpointsat the untrained attacker and after every actual attacker job
Evaluation decoding8 samples per behavior, temperature 1.0, same sample seeds at every checkpoint
Bootstrap2,000 paired behavior draws
Computeone NVIDIA A100 80 GB GPU per cloud job, at most 4 runs in parallel

examples are excluded from that prompt. This leaves strategy choice to the attacker.

Co-Training and the Stabilized Attacker Update

Each run starts from the base models, with the untrained attacker performing adversial attacks against the frozen defender. Figure 1 shows the five components and their interactions.

Co-training and measurement diagram

Figure 1. Co-training and measurement. Benchmark behaviors prompt the attacker; the defender answers its attacks, and the judge scores success. That score supplies the attacker reward. Successful attacks also yield preference pairs for defender training. The diversity meter is an evaluation component: it embeds valid attacks and measures within-behavior spread. Node colors identify the model and data roles; edge colors distinguish their interactions.

A round has two attacker training jobs followed by one defender training job. For each attacker job, the current attacker writes G=16G = 16 attacks per behavior on a seeded panel of 96 AdvBench training behaviors. The panel stays fixed for the run. The current defender answers, the HarmBench classifier judges success, and the attacker takes one pass of a group-relative policy-gradient update. The second job uses a new rollout from the updated attacker.

The defender is updated after both attacker jobs when at least four successful pairs are available. Each successful attack produces a preference pair favoring a short refusal over the harmful answer. Benign retention pairs favor the answer from the frozen defender over a refusal to encourage continued responses to harmless requests.

For the GG attacks written for one behavior, the reward and group-relative advantage are

ri=1[successi],r_i = \mathbf{1}[\mathrm{success}_i],
{#eq:1}

A_i = \frac{r_i - \bar{r}}{\sigma_r + \eta},

{#eq:2}

where rˉ\bar{r} and σr\sigma_r are the group mean and standard deviation of rewards, and η=10−6\eta= 10^{-6} provides numerical stability. Uniform-reward groups have zero advantage. The reward is binary attack success; quality gates validity (§ 3). A signal row is a prompt xx paired with an attack yy of at most 300 tokens and a nonzero advantage AA.

Let πθ\pi_\theta be the attacker being trained, π0\pi_0 the base model with its adapter disabled, and πold\pi_{\mathrm{old}} the attacker at the start of the job. πold\pi_{\mathrm{old}} is evaluated once, before optimization, and define

ρt=πθ(yt∣x,y<t)πold(yt∣x,y<t),δt=log⁡π0(yt∣x,y<t)πθ(yt∣x,y<t).(1)\rho_t = \frac{\pi_\theta(y_t \mid x,y_{<t})}{\pi_{\mathrm{old}}(y_t \mid x,y_{<t})}, \qquad \delta_t = \log\frac{\pi_0(y_t \mid x,y_{<t})}{\pi_\theta(y_t \mid x,y_{<t})}. \tag*{(1)}

Each signal row minimizes

L=1C∑t=1∣y∣[−min⁡(ρtA,clip⁡(ρt,1−ϵ,1+ϵ)A)+β(eδt−δt−1)],(2)\mathcal{L} = \frac{1}{C}\sum_{t=1}^{|y|}\left[-\min\left(\rho_t A,\operatorname{clip}(\rho_t,1-\epsilon,1+\epsilon)A\right) + \beta\left(e^{\delta_t}-\delta_t-1\right)\right], \tag*{(2)}

with normalizer C=300C = 300, the generation limit, clipping width ϵ=0.2\epsilon= 0.2 and KL weight β=0.005\beta= 0.005. The second term is the unbiased KL estimator used in GRPO [7]. The reward stays success-only throughout. Table II explains the four optimizer stabilizers.

An update is a completed attacker training job; an optimizer step is one minibatch operation within that job. An attempt is a rollout followed by either training or a skip. The experiment records attempts, completed jobs, optimizer steps and skips separately, alongside KL, signal fraction, gradient norms and the share of tokens whose gradient the ratio clip removes.

Diversity Reduction Checks and the Degenerate-Run Rule

The diversity reduction criterion and the degenerate-run rule were fixed before training. Diversity reduction checks use the AdvBench development panel to track attacker performance and the benign development panel to guard defender over-refusal.

Diversity Reduction checks. Every four completed rounds, attacks are drawn from an additional development attack bank and measure the benign over-refusal of the co-trained defender. A run meets the diversity reduction criterion when ADR and its 95% confidence interval are below 0.75 and all five effectiveness guards hold:

  • first-sample attack success rate (ASR@1) against the frozen defender has risen by at least ten percentage points over the untrained attacker;

  • attacks cover at least 90% of behaviors;

ChangeWhat it doesWhy
Length-unbiased lossEach completion token loss is divided by the constant C=300C = 300 (Eq. (2)). The group-mean baseline and standard-deviation scaling is retained. Liu et al. [8] also remove the scaling and updates to the model keeps it.A per-completion mean gives tokens in long responses less weight and biases GRPO toward longer responses, especially incorrect ones. A constant normalizer removes this weighting bias [8].
Bounded updateA job with fewer than 64 signal rows is recorded as skipped. A larger job uses a seeded uniform subsample of at most 512 rows, visited once in groups of eight: at most 64 optimizer steps.Uniform-reward groups have zero advantage [9], leaving few rows to carry an update. The floor avoids training on a handful of rows; the cap limits optimization per rollout.
Warm-upThe learning rate rises linearly over the first four optimizer steps of a job and then stays at 2×10−42 \times10^{-4}. A job initializes its own AdamW state [16]. Warm-up reduces the variance of early adaptive steps [18].
Ratio clippingThe surrogate clips the probability ratio of each token against the starting policy for the job to [0.8,1.2][0.8, 1.2]. The KL term uses the base model (Eq. (2)).The surrogate limits incentives for large probability changes across minibatch steps [7], [19]. It is applied throughout the job sequence of up to 64 steps.

Table 2. Four optimizer stabilization techniques used with the success-only reward.

  • at most 5% of attacks are invalid;

  • benign over-refusal has risen by at most five percentage points;

  • the defender has completed at least two updates.

The degenerate-run rule. This rule detects gross validity, attack-success and learning failures. Passing the screen does not rule out optimizer-driven narrowing in a functioning attacker. Three stopping conditions are checked after each attempt:

  1. the training-panel invalid-attack rate exceeds 20% in each of the last three attempts;

  2. development success within eight samples (ASR@8) previously reached 0.50 and is at most 0.10 at each of the last four checkpoints;

  3. each of the last six attempts was skipped for lack of a learning signal.

A condition stops the run as soon as it holds. Stopped runs count as failing the diversity reduction criterion and are retained without replacement or rollback. The stored state records the triggering condition and its evidence.

Measurement

Attack Diversity and Attack Diversity Retention

An attack is valid if it has at least 20 and fewer than 1,500 characters and passes an opening-text refusal filter. The filter rejects attacker messages that themselves begin with a stock refusal. An incomplete or unscored evaluation, or a zero denominator for the untrained-attacker, is reported as a measurement failure. For behavior bb at checkpoint uu, let db(u)d_b(u) be the mean cosine distance between all pairs of valid attacks, using embedding from a sentence encoder. For the set B\mathcal{B} of behaviors with at least two valid attacks in both banks,

ADR illustrated with Run 1 measured values

Figure 2. ADR illustrated with Run 1 measured values. Dot positions are schematic; they illustrate the metric rather than actual embedding coordinates. The three clouds represent the untrained attacker and its checkpoints after updates 9 and 24. Eight dots represent attacks for one behavior; connecting lines represent pairwise distances. The printed mean distance averages within-behavior diversity over 25 comparable development behaviors. Each cloud is scaled by its ADR relative to the first. Update 9 is Run 1 first checkpoint below 1 (0.98); update 24 is its last (0.64).

ADR⁡(u)=∑b∈Bdb(u)∑b∈Bdb(0),(3)\operatorname{ADR}(u) = \frac{\sum_{b \in\mathcal{B}} d_b(u)}{\sum_{b \in\mathcal{B}} d_b(0)}, \tag*{(3)}

where db(0)d_b(0) comes from the same untrained attacker seed under the same behaviors, decoding and sample seeds. This primary ADR measures how much within-behavior spread the trained attacker retains (Figure 2). A value of 1 preserves the initial spread; 0.75 corresponds to a quarter lost; values above 1 indicate greater spread. The diversity reduction criterion combines this relative diversity reading with continued attack success.

The intervals describe behavior sampling conditional on the trained model and its sampled completions. The bootstrap treats a behavior and its eight completions as one sampling unit. 2,000 paired draws [20] are made, stratified by benchmark, applying the same resampled behavior indices to the trained and untrained banks. The 2.5th and 97.5th percentiles give the confidence interval (CI), whose upper end ADR+^{+} enters the diversity reduction criterion. ASR@1 uses the first completion for each behavior; ASR@8 counts success in any of the eight.

Development Checkpoints and Diversity Reduction Checks

After each completed attacker job, the attacker writes eight attacks for each of the 26 AdvBench development behaviors. These evaluation banks are kept from training. Checkpoint 0 is the untrained attacker, with ADR = 1 by definition. Behaviors and sample seeds stay fixed throughout the trajectory.

ADR compares the spread of the attack with the untrained model; attack success is scored against the frozen defender. This separates attacker learning from changes in the guardrails of the co-trained defender [21]. The checkpoint index counts completed jobs, so skipped attempts leave the index unchanged. Diversity reduction checks draw an additional bank from the attacker at the end of the round.

Secondary Diagnostics and the Length-Matched Measure

Strategy labels are automated descriptions of successful attacks. Each attack receives the nearest of seven short descriptions in embedding space when cosine similarity is at least 0.25; lower similarities receive the label “Other.” Section VI-E gives the descriptions. The share of each attack family and their entropy is reported, normalized by ln⁡8\ln8 for the seven families plus “Other.”

Additional diagnostics include within-behavior near-duplicates, defined by cosine similarity above 0.95, and the mean and spread of attack length in characters. Training diagnostics record reward signal, KL, gradient norms, the clipped share of tokens, completed and skipped updates, and benign refusal. The classifier may shorten the generated answer to fit its 2,048-token context; clipping counts are logged per job.

An exploratory length-matched ADR was added after completing the pre-specified design. It compares distances within bands defined by thirds of the untrained attacker’s pair lengths, measured as the mean character length of two attacks. The bands are under 184, 184 to under 244, and 244 or more characters. Within each band, distances are averaged over pairs from the shared behaviors. A band is eligible when both attackers have at least 15 pairs in it. Eligible bands are weighted by the untrained attacker share of the pairs in each band, normalized over those bands. This additional measure uses 1,000 behavior-bootstrap draws for its intervals.

Onset Analyses

The onset analyses describe the trajectory and are separate from the diversity reduction criterion. The persistent clear onset is the earliest checkpoint beginning a sequence of three successive trained checkpoints, each with ADR < 0.75, ADR+^{+} < 0.75 and the validity plus support guards satisfied. This definition was fixed before training. It was applied retrospectively, alongside correlations of ADR with development ASR@1 and update number.

Implementation, Provenance and Containment

Implementation. Sampling can vary when inference is repeated. This experiment aims to prevent variation by having the pipeline use a pinned code version and configuration, with atomic checkpoints for saved training state. Its co-training, development measurements, optimizer diagnostics, degenerate-run rule and paired intervals passed an offline regression suite before launch.

Provenance and availability. The design and configuration were stored in version control before training. Retained metadata include the configuration and dataset digests, package versions, model identifiers, container image, code digest and numerical run summaries.

Containment. The study excludes production targets, functional payloads and external model APIs. Raw benchmark text, attacks, answers, preference examples and adapters remain private. Analyses use numerical summaries. Research jobs use an outbound allowlist, with dataset download hosts enabled during preparation; a disable switch stops scripts at their next checkpoint.

Results

What Ran

Runs 1 to 3 each completed 24 attacker updates and 12 defender updates, totaling 72 attacker and 36 defender updates. All recorded attempts produced an update, and the degenerate-run rule remained untriggered. At trained checkpoints, invalid attacks accounted for at most 2.9% of a development bank; group coverage was 1.000.

Each run generated 36,864 training attacks, or 110,592 in total. Their success rates against the current defender were 33.5%, 17.3% and 25.2%. Each run produced 28 development banks of 208 attacks, 24 checkpoints and three diversity reduction checks. The median training success rate rose from 0.042 over each of the first three updates to 0.401 over its last three. The share of training groups carrying a learning signal rose from 0.39 to 0.89.

Stopping. This was a data-dependent stop. Budget considerations informed the decision to end training after the diversity reduction checks at round 12. The decision followed the update-22 checkpoints, whose intervals were below 0.75 in each run, and preceded the drawing of the final diversity-reduction banks. Each run then completed those checks.

Diversity Reduction Checks

Table III reports all nine checks. The six checks at rounds 4 and 8 fell short of the criterion. At round 4, ADR was 1.08, 0.84 and 0.79; Run 2 success gain of 0.04 also fell short of the required 0.10. At round 8, Run 1 point estimate was 0.67 and its interval reached 0.754, exceeding the threshold by 0.004. Runs 2 and 3 had point estimates of 0.77 and 0.79.

The final checks met the criterion in each run. ADR ranged from 0.608 to 0.631, with upper bounds of 0.704, 0.682 and 0.697. Their margins below the threshold were 0.046, 0.068 and 0.053. Success gains were 0.81, 0.58 and 0.77, group coverage was 1.000, and paired coverage was 0.962. All attacks were valid and each defender had completed 12 updates. Over-refusal rose by 0.036, 0.000 and 0.018, within the 0.05 limit.

The across-run summary is post hoc and nominal. Its tt tests describe variation among three seed values conditional on the shared behavior panel; behavior sampling and the data-dependent stop are outside these tests. Mean ADR was 0.623, with standard deviation 0.013. One-sample tt tests place the mean below 1 (t(2)=−48.9t(2) = -48.9, p=0.0004p = 0.0004) and below 0.75 (t(2)=−16.4t(2) = -16.4, p=0.004p = 0.004).

Trajectories of Diversity and Attack Success

ADR initially rose in two runs (Figure 3). Run 1 reached 1.41 after one update and remained above 1 for eight updates; Run 2 reached 1.66 after its second. Run 3 began near 1, at 1.01, before falling.

Figure 3. (a) ADR with 95% interval; stars are diversity reduction checks. (b) attack success: ASR@8 solid, ASR@1 dashed. Development trajectories after each attacker update. (a) ADR with its 95% bootstrap interval. The dashed line marks the diversity reduction threshold of 0.75; stars mark the additional diversity-reduction banks at rounds 4, 8 and 12. (b) Attack success against the frozen defender: ASR@1 uses the first sample, and ASR@8 uses any of eight samples.

The three-checkpoint onset is bounded by the observation window: with 24 checkpoints, the latest possible onset is update 22. Two onset estimates fall at this boundary. Point estimates first crossed below 0.75 at updates 12, 10 and 11 for Runs 1 to 3. The entire interval was below 0.75 at 4, 3 and 4 of the 24 checkpoints, and all after update 15. Run 1 qualifying checkpoints were 16 and 22 to 24. Run 2 qualifying checkpoints were 22 to 24, and Run 3 qualifying checkpoints were 20 to 22 and 24. Persistent clear onset occurred at updates 22, 22 and 20.

The trajectories include reversals: Run 1 recovered from 0.66 at update 16 to 0.83 at update 18 before falling again. Median changes between consecutive checkpoints were 0.04, 0.06 and 0.04, and average interval widths were 0.15 to 0.19. These fluctuations make the sustained trajectory more informative than a single checkpoint.

The correlations describe joint trends with training time; they do not establish which variable leads or a causal effect. Across the 24 checkpoints, the Spearman correlation was ρ\rho between ADR and ASR@1 was -0.91, -0.84 and -0.86; with update number it was -0.89, -0.90 and -0.90. At the last checkpoint, ASR@1 was 0.73, 0.58 and 0.81, and ASR@8 reached 1.00 in each run. ASR@1 retains resolution after ASR@8 reaches its ceiling.

Attack Length and the Length-Matched Measure

Embedding distance covaries with attack length even before training. In the untrained banks, mean within-behavior distance was 0.239 to 0.261 for the shortest third of pairs and 0.196 to 0.208 for the longest. Pair distance correlated with length at -0.18 to -0.28. By the last checkpoint, mean attack lengths had grown to 1.89, 1.16 and 2.07 times their initial values. They began at 217, 222 and 209 characters and ended at 410, 258 and 431.

Figure 4 compares the two diversity measures. At update 24, length-matched ADR was 0.72, 0.67 and 0.71, compared with primary ADR of 0.64, 0.60 and 0.62. At the final diversity reduction checks, the exploratory comparison was 0.71, 0.68 and 0.72. Matching raised these final estimates by 0.07 to 0.09.

Figure 4. Primary and length-matched ADR, with attack length. Top: primary ADR with a shaded 95% interval and the comparison with interval bars. Hollow markers identify thin estimates using one or two of the three length bands. Stars mark diversity reduction checks; the horizontal line marks 0.75. Bottom: mean attack length relative to the untrained attacker.

The longest band has no upper edge, leaving residual length differences within it. For Runs 1 and 3, at least 92% of trained pairs fall in that band from update 11 onward. The exploratory comparison use only one band. Each length-checked comparison interval at the final checks extended above 0.75. One of the 72 checkpoint intervals lie wholly below it: Run 2 at update 24 (0.67, interval 0.54 to 0.74). All six final comparison intervals, from the last checkpoints and diversity reduction checks, lie below 1. At the checks their upper bounds were 0.843, 0.751 and 0.800.

Strategy Families, Behaviors and the Defender

The composition of successful attacks changed during training. Figure 5 follows the family shares by update, and Table IV pools them into windows.

Figure 5. Automated family labels among successful training attacks, with one pair of panels per run. Each upper bar gives the shares for one rollout, stacked in table order. Lower panels give the number of successes out of 1,536 attacks. Labels identify the nearest description in embedding space.

A family label records similarity to a description, and a message combining techniques still receives one label. A roll-out contains 1,536 attacks, with 40 to 889 judged successful across the observed updates. Colored segments divide those successes into shares totaling 100%. The seven descriptions are:

RoundRunADR (95% CI)Length-matched ADR (95% CI)ASR@1: untrained → checkOver-refusal increaseShows reduction
411.08 (0.99–1.18)1.29 (1.17–1.39)0.04 → 0.42+0.000no: ADR above 0.75
420.84 (0.77–0.93)0.86 (0.75–0.97)0.00 → 0.04+0.000no: ADR above 0.75; ASR gain 0.04
430.79 (0.71–0.87)0.99 (0.89–1.10)0.00 → 0.35+0.000no: ADR above 0.75
810.67 (0.59–0.754)0.755 (0.65–0.87)0.04 → 0.92+0.018no: upper bound 0.754
820.77 (0.69–0.88)0.89 (0.80–1.01)0.00 → 0.38+0.036no: ADR above 0.75
830.79 (0.70–0.89)0.99 (0.86–1.12)0.00 → 0.58+0.000no: ADR above 0.75
1210.63 (0.57–0.70)0.71 (0.61–0.84)0.04 → 0.85+0.036yes
1220.61 (0.54–0.68)0.68 (0.54–0.751)0.00 → 0.58+0.000yes
1230.63 (0.57–0.70)0.72 (0.65–0.80)0.00 → 0.77+0.018yes

Table 3. Diversity reduction checks under the pre-specified criterion. ADR is shown with its 95% bootstrap interval; the length-matched comparison is exploratory. ASR@1 is the share of development behaviors whose first attack succeeded. Over-refusal is the increase in refusals on 56 benign development prompts; one prompt is 0.018. A “yes” requires both the diversity threshold and the effectiveness guards in Section III.

Family (%)UntrainedFirst 3Latest 3
successes (nn)205415,616
hypothetical framing0.014.463.2
format transformation65.053.215.7
instruction override30.016.87.7
contextual manipulation0.010.411.1
role play0.03.92.2
obfuscation5.00.60.1
authority claims0.00.70.0

Table 4. Family shares among labeled successes (nn), pooled over runs. The untrained column uses the development banks of the untrained attacker; the early and late columns use the first and last three training rollouts. All successes received one of the seven family labels.

  • hypothetical framing: the request framed as fiction, a hypothetical, a thought experiment or a story;

  • format transformation: the content asked for in another form, such as a translation, summary, list or piece of code;

  • instruction override: an attempt to override, reset or ignore earlier instructions and rules;

  • contextual manipulation: a benign-looking scenario or backstory built around the request;

  • role play: a persona or character that behaves differently from the usual behavior of the assistant;

  • obfuscation: the request disguised by encoding, euphemism, misspelling or splitting it into pieces;

  • authority claims: claimed official permission, professional credentials or a duty that supposedly justifies the request.

Format transformation accounted for 54%, 70% and 59% of successes at update 1; hypothetical framing accounted for 18%, 11% and 12%. Hypothetical framing later led at 19 of 24 updates in Run 1 and at 8 and 10 updates in Runs 2 and 3. Its uninterrupted leads began at updates 7, 17 and 18. At update 24, its shares were 79%, 63% and 32%, and format transformation had fallen to 9%, 25% and 19%.

The ending mixes differed: Run 1 concentrated 79% of successes in one family, with each other family at 9% or less. Run 3 spread successes across hypothetical framing (32%), contextual manipulation (30%), format transformation (19%) and instruction override (17%). Role play, obfuscation and authority claims reached maxima success of 9.3%, 2.9% and 2.8%.

Run 1 family entropy was 0.47 to 0.74 over its first 20 updates and 0.29 to 0.38 over its last four. Hypothetical framing then accounted for 79% to 85% of successes. Runs 2 and 3 stayed within entropy ranges of 0.47 to 0.65 and 0.60 to 0.71.

The earliest shares are coarse: the first three rollouts per run had 40 to 82 successes, compared with later maxima of 889, 478 and 640. Pooled format-transformation shares fell from 65.0% of the 20 successes of untrained attacker to 53.2% in the early window and 15.7% in the late window. Hypothetical framing rose from 0% through 14.4% to 63.2%; instruction override fell from 30.0% through 16.8% to 7.7%. Across all updates, hypothetical framing represented 64.6%, 44.1% and 32.7% of successes in Runs 1 to 3.

The exploratory family-by-length comparison uses the same encoder as ADR, so it supplies descriptive rather than independent strategy evidence. Its short bands are sparse: 111 and 491 of the latest 5,616 successes. Late hypothetical-framing successes averaged 383 characters, close to format transformation at 370 and the other families at 397. Its shares rose within the short, middle and long bands from 11%, 14% and 16% to 66%, 70% and 62%. Reweighting the late successes to the early length mix gave a share of 64.9%, compared with 63.2% observed.

The cross-run comparison points in the same direction. Run 2 attack length stayed closest to its reference, at 1.16 times the starting length, and 65% of late successes were hypothetical framing. Run 3 length grew most, to 2.07 times its reference, and its late hypothetical-framing share was 38%. The family gained share within comparable length ranges as well as across the pooled rollouts.

One development behavior lacks two valid untrained attacks, leaving 25 comparable behaviors. At the last checkpoint, within-behavior spread was lower than the untrained attacker for 22, 24 and 24 behaviors in Runs 1 to 3. Ratios were below 0.75 for 16, 19 and 20 behaviors, with medians of 0.60, 0.58 and 0.64. Near-duplicate shares rose from 0.045, 0.030 and 0.000 to 0.212, 0.192 and 0.130.

The training rollouts provide a larger view of 96 training behaviors against the moving defender (Table V). Mean diversity over the last three rollouts fell to 0.69, 0.66 and 0.66 of first rollout of each run. At all nine diversity reduction checks, benign over-refusal rose by at most two of the 56 prompts, or 0.036.

Discussion

The three runs share one development panel, so their agreement makes sense for these behaviors. The criterion operationalizes diversity reduction as a loss of embedding spread while attack effectiveness continues. Under that definition, the runs provide a reproduction of the diversity loss reported for GRPO co-training [1]. The decline appears in the per-behavior comparisons and in the larger training rollouts as well as in aggregate ADR.

QuantityRun 1Run 2Run 3
ADR at the last checkpoint (95% CI)0.64 (0.57–0.72)0.60 (0.54–0.68)0.62 (0.56–0.69)
Checkpoints with ADR below 0.75: value / whole interval (of 24)10 / 48 / 310 / 4
Behaviors less diverse than untrained / with ratio below 0.75 (of 25)22 / 1624 / 1924 / 20
Diversity, untrained → last (mean cosine distance)0.221 → 0.1430.228 → 0.1370.235 → 0.143
ASR@1, untrained → last0.04 → 0.730.00 → 0.580.00 → 0.81
ASR@8, untrained → last0.15 → 1.000.23 → 1.000.19 → 1.00
Mean attack length in characters, untrained → last217 → 410 (1.89×)222 → 258 (1.16×)209 → 431 (2.07×)
Training-rollout diversity, last three updates vs. the first rollout0.690.660.66
Successful attacks in the training rollouts33.5%17.3%25.2%

Table 5. The three runs at their last checkpoint (update 24). Diversity is the mean cosine distance between the attacks written for one behavior.

The 0.75 threshold is a convention for losing a quarter of the initial spread. Near-duplicate shares also depend on a chosen cutoff, 0.95, and paraphrases can implement the same template while falling below it. Template counts and human strategy coverage therefore remain unmeasured. At the last checkpoint, near-duplicate shares were 0.13 to 0.21: most samples stayed outside the cutoff even as overall spread declined.

Length grew during training even with constant loss normalization. That normalizer addresses token weighting, while output length remains a property of the learned policy. Banding and pair weighting make the comparison evaluation a sensitivity analysis; residual within-band differences prevent an exact attribution of the decline to length. Primary ADR records the total change in embedding spread, and the length-check comparison shows how the reading changes under those length bands.

Stopping selection can make the intervals and pp values optimistic. The final banks were drawn after that decision, scoring the same selected attackers and behaviors; they share 47% to 74% of their texts with the preceding checkpoints. They provide within-run consistency evidence, with ADR differing by at most 0.03 between the two banks.

Limitations

  1. Sampling. Joint uncertainty over seeds, behaviors and completion draws remains unestimated. The behavior bootstrap conditions on a seed and its completions; the across-run tt test conditions on the shared panel. The test splits remained unscored. This experiment reports each run individually, following recommendations for small-sample reinforcement learning studies [22, 23].

  1. Stopping. The data-dependent stop departs from the pre-specified design. Further rounds, additional seeds, the planned budget check and held-out evaluation were left unrun. Intervals and pp values are uncorrected for stopping and the three diversity reduction checks per run; their role here is descriptive.

  1. Measurement. Embedding geometry, wording and attack length affect the diversity reading. Classifier preferences could also contribute to the observed strategy shift. Model judges have known biases [24]. A separate agent-training study found judge-based rewards more susceptible to reward hacking than an embedding-based reward [25]. Human strategy labels and a human audit were not collected. Family labels and lexical over-refusal checks provide automated diagnostics.

  1. Attribution. A single training arm leaves the effects of co-training and optimizer stabilization unresolved. Testing those effects would require frozen-defender and unstabilized comparison arms; prevention would require an intervention arm. The present result characterizes one co-training schedule with all four stabilization techniques applied together.

  1. Scope. Multi-turn strategies such as Crescendo [26], deployed systems and adaptive human attackers [2] lie outside this evaluation. It covers single-turn attacks between two open-weight models, using one implementation of the update and fixed public benchmarks.

  1. Clipped answers. A clipped answer supplies partial evidence to the classifier. Its 2,048-token context can require shortening the generation section. The scoring procedure preserves the rubric and answer cue.

  1. Pre-specification. The design was dated and version-controlled before launch. Three analyses were added afterward: length-matched ADR, the across-run tt test and the family-by-length comparison. The diversity reduction criterion continues to use primary ADR.

Related Work

Language-model red teaming includes model-generated test cases [5] and optimized adversarial suffixes [6], with HarmBench standardizing automated attack and refusal evaluation [14]. Several methods directly reward diversity: curiosity-driven red teaming adds a novelty objective [27], Rainbow Teaming uses quality-diversity search [28], and Beutel et al. reward attacks that differ from past attempts [29]. Diversity loss appears in reinforcement learning from human feedback [30], and repeated learning from narrowed outputs can erode distribution tails [31]. These findings motivate tracking the distribution of attacks alongside their success rate.

For reasoning models, Liu et al. identify the per-completion length bias of GRPO [8]; Yu et al. identify change loss aggregation and resample uniform-reward groups [9]. Their findings motivate the use of constant normalizer and signal-row bounds. Ratio clipping [19] and learning-rate warm-up [18] address update stability. These safeguards were intended to reduce length-weighting bias, sparse-update noise and excessive policy changes within a rollout, making the diversity trajectory easier to interpret.

Small-sample reinforcement learning results benefit from individual runs and interval estimates [22, 23]. That reporting approach was followed. The across-run mean supplements the individual measurements as a post hoc summary.

Conclusion

This experiment uses a data-dependent stopping time and nominal intervals. The three runs met the diversity reduction criterion at round 12, with a 37% to 39% loss of within-behavior embedding spread and attack success above its starting level. Length matching gives a smaller decline, and the strategy mix changes toward hypothetical framing. Held-out evaluation and comparison arms would test generalization and cause. The runs demonstrated effective attack generation alongside a narrower within-behavior distribution.

References

  1. [1]B. Bullwinkel, E. Kim, A. Minnich, and M. Russinovich, “Learning to attack and defend: Adaptive red teaming of language models via GRPO,” arXiv preprint arXiv:2606.09701, 2026.arxiv.org/abs/2606.09701
  2. [2]B. Bullwinkel, A. Minnich, S. Chawla, G. Lopez, M. Pouliot, W. Maxwell, J. de Gruyter, K. Pratt, S. Qi, N. Chikanov et al., “Lessons from red teaming 100 generative AI products,” arXiv preprint arXiv:2501.07238, 2025.arxiv.org/abs/2501.07238
  3. [3]G. D. Lopez Munoz, A. J. Minnich, R. Lutz, R. Lundeen, R. S. R. Dheekonda, N. Chikanov, B.-E. Jagdadgordj, M. Pouliot, S. Chawla, W. Maxwell et al., “PyRIT: A framework for security risk identification and red teaming in generative AI system,” arXiv preprint arXiv:2410.02828, 2024.
  4. [4]E. Haider, D. Perez-Becker, T. Portet, P. Madan, A. Garg, A. Ashfaq, D. Majercak, W. Wen, D. Kim, Z. Yang et al., “Phi-3 safety post-training: Aligning language models with a “break-fix” cycle,” arXiv preprint arXiv:2407.13833, 2024.arxiv.org/abs/2407.13833
  5. [5]E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving, “Red teaming language models with language models,” in Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022, arXiv:2202.03286.arxiv.org/abs/2202.03286
  6. [6]A. Zou, Z. Wang, N. Kolter, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” arXiv preprint arXiv:2307.15043, 2023.
  7. [7]Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo, “DeepSeekMath: Pushing the limits of mathematical reasoning in open language models,” arXiv preprint arXiv:2402.03300, 2024.
  8. [8]Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin, “Understanding R1-Zero-Like training: A critical perspective,” arXiv preprint arXiv:2503.20783, 2025.arxiv.org/abs/2503.20783
  9. [9]Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue et al., “DAPO: An open-source LLM reinforcement learning system at scale,” arXiv preprint arXiv:2503.14476, 2025.arxiv.org/abs/2503.14476
  10. [10]Qwen Team, “Qwen2.5 technical report,” arXiv preprint arXiv:2412.15115, 2024.
  11. [11]E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations (ICLR), 2022, arXiv:2106.09685.arxiv.org/abs/2106.09685
  12. [12]Microsoft, “Phi-4-Mini technical report: Compact yet powerful multimodal language models via mixture-of-LoRAs,” arXiv preprint arXiv:2503.01743, 2025.
  13. [13]R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” in Advances in Neural Information Processing Systems (NeurIPS), 2023, arXiv:2305.18290.arxiv.org/abs/2305.18290
  14. [14]M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, D. Forsyth, and D. Hendrycks, “HarmBench: A standardized evaluation framework for automated red teaming and robust refusal,” in International Conference on Machine Learning (ICML), 2024, arXiv:2402.04249.arxiv.org/abs/2402.04249
  15. [15]S. Xiao, Z. Liu, P. Zhang, N. Muennighoff, D. Lian, and J.-Y. Nie, “C-Pack: Packed resources for general Chinese embeddings,” in ACM SIGIR Conference on Research and Development in Information Retrieval, 2024, arXiv:2309.07597.arxiv.org/abs/2309.07597
  16. [16]I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations (ICLR), 2019, arXiv:1711.05101.arxiv.org/abs/1711.05101
  17. [17]R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Stanford Alpaca: An instruction-following LLaMA model,” https://github.com/tatsu-lab/stanford_alpaca, 2023.
  18. [18]L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han, “On the variance of the adaptive learning rate and beyond,” in International Conference on Learning Representations (ICLR), 2020, arXiv:1908.03265.arxiv.org/abs/1908.03265
  19. [19]J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
  20. [20]B. Efron, “Bootstrap methods: Another look at the jackknife,” The Annals of Statistics, vol. 7, no. 1, pp. 1–26, 1979.
  21. [21]L. Hsiung, T. Pang, Y.-C. Tang, L. Song, T.-Y. Ho, P.-Y. Chen, and Y. Yang, “Why LLM safety guardrails collapse after fine-tuning: A similarity analysis between alignment and fine-tuning datasets,” in Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). San Diego, California, United States: Association for Computational Linguistics, 2026, pp. 16 601–16 619.
  22. [22]P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meجر, “Deep reinforcement learning that matters,” in AAAI Conference on Artificial Intelligence, 2018, arXiv:1709.06560.
  23. [23]R. Agarwal, M. Schwarzer, P. S. Castro, A. Courville, and M. G. Bellemare, “Deep reinforcement learning at the edge of the statistical precipice,” in Advances in Neural Information Processing Systems (NeurIPS), 2021, arXiv:2108.13264.arxiv.org/abs/2108.13264
  24. [24]L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and L. Stoica, “Judging LLM-as-a-Judge with MT-Bench and chatbot arena,” in Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023, arXiv:2306.05685.arxiv.org/abs/2306.05685
  25. [25]X. Yu, B. Peng, R. Xu, Y. Shen, P. He, S. Nath, N. Singh, J. Gao, and Z. Yu, “Reinforcement world model learning for LLM-based agents,” arXiv preprint arXiv:2602.05842, 2026.arxiv.org/abs/2602.05842
  26. [26]M. Russinovich, A. Salem, and R. Eldan, “Great, now write an article about that: The Crescendo multi-turn LLM jailbreak attack,” arXiv preprint arXiv:2404.01833, 2024.arxiv.org/abs/2404.01833
  27. [27]Z.-W. Hong, I. Shenfeld, T.-H. Wang, Y.-S. Chuang, A. Pareja, J. Glass, A. Srivastava, and P. Agrawal, “Curiosity-driven red-teaming for large language models,” in International Conference on Learning Representations (ICLR), 2024, arXiv:2402.19464.arxiv.org/abs/2402.19464
  28. [28]M. Sambyal, S. C. Raparthy, A. Lupu, E. Hambro, A. H. Markosyan, M. Bhatt, Y. Mao, M. Jiang, J. Parker-Holder, J. Foerster, T. Rocktäschel, and R. Raileanu, “Rainbow teaming: Open-ended generation of diverse adversarial prompts,” arXiv preprint arXiv:2402.16822, 2024.
  29. [29]A. Beutel, K. Xiao, J. Heidecke, and L. Weng, “Diverse and effective red teaming with auto-generated rewards and multi-step reinforcement learning,” arXiv preprint arXiv:2412.18693, 2024.arxiv.org/abs/2412.18693
  30. [30]R. Kirk, I. Mediratta, C. Nalmpantis, J. Luketina, E. Hambro, E. Grefenstette, and R. Raileanu, “Understanding the effects of RLHF on LLM generalisation and diversity,” arXiv preprint arXiv:2310.06452, 2023.arxiv.org/abs/2310.06452
  31. [31]I. Shumailov, Z. Shumaylov, Y. Zhao, N. Papernot, R. Anderson, and Y. Gal, “AI models collapse when trained on recursively generated data,” Nature, vol. 631, no. 8022, pp. 755–759, 2024.DOI

Paper details

vlnc:W_ysfk3v7u
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents