betaFeedback

WorksComputer Science

Hardening Agentic Web Populations Through Multi-Criterion Competitive Co-Evolution: Lessons from Wargame Strategy Optimization

Abstract

Agents on the open web will face adversaries that keep adapting, so a defense that is only tested once will eventually be found and broken. We propose treating jailbreak attacks and safety guardrails as two populations that evolve against each other, rather than as a fixed attacker probing a fixed defense. This idea comes directly from wargame strategy optimization, where two-population competitive co-evolution with alternating evolution cycles and cross-play evaluation has already been shown to produce strategies that are robust to a moving opponent, and where a companion line of work has shown that the same machinery can measure how much competitive pressure a population must absorb. However, real-world applications can never rely on a single criterion as a key performance indicator, rather must use multiple conflicting criteria, such as, cost and efficiency. We describe how this mechanism transfers to language model safety, argue why the resulting multi-criteria, competitive and coevolutionary arms race is a genuinely networked, population-level problem rather than a two-agent toy setting, and lay out the open problems that stand between the wargame version of this idea and an agentic web version of it.

Introduction

Most evaluations of language model safety still follow a single-shot pattern: a fixed set of jailbreak prompts is run against a fixed model, once. This misses the central fact about the agentic web: attackers and defenders are not static. An attacker who fails against today’s guardrail will keep adapting until a new prompt works, and a defender who only ever sees yesterday’s attacks will be blind to tomorrow’s. Static red-teaming benchmarks cannot capture this back-and-forth, because they freeze one side of the interaction.

We propose a different setup, illustrated in Figure 1: run the attacker and the defender as two co-evolving populations, and let them adapt to each other over many rounds instead of one. This is not a new idea in coevolutionary computation. It is the same mechanism used to optimize offense and defense strategies in wargame simulations [5, 6 b], and it is the same mechanism a companion line of work has used to measure competitive balance directly on a simulated game [9 b]. Our contribution here is a position, not a new algorithm: we argue that this two-population, alternating, cross-play mechanism is a natural fit for hardening agentic web systems against external manipulation, and we describe what has to change to move it from a wargame simulator to a population of language model agents interacting at web scale.

Competitive co-evolution mechanism for wargame strategy optimization

Figure 1. Our proposed mechanism, adapted from multi-criterion competitive co-evolution in wargame strategy optimization. A jailbreaker population and a defender population are scored against each other in a cross-play matrix, fitness vectors are aggregated across all opponents to avoid overfitting to a single rival, and the two populations take turns evolving while the other is held fixed. Each side keeps its own Pareto-optimal set of solutions resulting from multiple conflicting criteria, and the intended outcome is that both sides become resistant to manipulation by a co-evolved adversary rather than by a fixed one.

Background: Competitive co-evolution in wargame strategy optimization

The mechanism in Figure 1 was developed for a different domain: optimizing offense and defense strategies in a military simulation. Guha et al. [ [5] formulated this as a two-population, multi-criterion problem. An offense population and a defense population are each made of candidate strategies, and instead of evaluating a strategy against a single opponent, every strategy is evaluated against every member of the opposing population. This is the cross-play evaluation matrix in Figure 1. Formally, if xx is an offense strategy and yy is a defense strategy, a single match produces an objective vector

f(x,y)=[f1off(x,y),f2off(x,y),f1def(x,y),f2def(x,y)]T,(1)\mathbf{f}(x,y) = [f^{\mathrm{off}}_{1}(x,y), f^{\mathrm{off}}_{2}(x,y), f^{\mathrm{def}}_{1}(x,y), f^{\mathrm{def}}_{2}(x,y)]^{\mathrm{T}}, \tag*{(1)}

and the full set of matches between an offense population of size NoffN_{\mathrm{off}} and a defense population of size NdefN_{\mathrm{def}} produces an evaluation tensor Ft∈RNoff×Ndef×4\mathbf{F}^{t} \in\mathbb{R}^{N_{\mathrm{off}} \times N_{\mathrm{def}} \times4} at generation tt. Each strategy’s fitness is obtained by aggregating its row or column of this tensor across all opponents it faced, rather than by its result against any one opponent. This aggregation step is what prevents a strategy from winning simply by overfitting to a single rival, which is a well known failure mode of competitive co-evolution [1].

The two populations do not evolve at the same time. Instead, following an alternating cycle [τoff,τdef][\tau_{\mathrm{off}}, \tau_{\mathrm{def}}], one population evolves for τoff\tau_{\mathrm{off}} generations using NSGA-II [4] while the other is held fixed, then the roles switch. This alternation, together with the cross-play aggregation, is what produces populations whose Pareto-optimal fronts represent strategies robust to an adapting opponent rather than to a single frozen one. Follow-up work extended this framework with progressive shrinking of the decision space for more realistic sequential decision-making [6], and with a regularity-based procedure for selecting one actionable strategy off the Pareto front [7].

A companion line of work, Raj et al. [ [9] and Raj et al. [ [10], applies the same two-population, alternating, cross-play mechanism to a different question: not just optimizing strategy, but measuring balance. That work introduces a movement metric that compares a hypervolume While et al. [ [11], Bader et al. [ [2] achieved under a cooperative upper bound (a strategy that assumes a fully cooperative opponent) against the hypervolume actually achieved at co-evolutionary equilibrium. The gap between these two,

Movement=HVcoop−HVcoev,(2)\mathrm{Movement} = \mathrm{HV}_{\mathrm{coop}} - \mathrm{HV}_{\mathrm{coev}}, \tag*{(2)}

quantifies how much a side is forced to sacrifice by competitive pressure. When both sides have approximately equal movement, the interaction is judged balanced; when one side’s movement is much larger than the other’s, that side is systematically disadvantaged. This matters for our proposal because it shows that the same co-evolutionary machinery used to harden strategies can also be turned into a diagnostic: it does not just produce robust attackers and defenders, it can tell us how much pressure each side is actually under.

Mapping the wargame mechanism to jailbreak and defense coevolutionary framework

The translation from wargame agents to language model agents is direct. The offense population becomes a population of jailbreaker strategies: candidate prompts, prompt templates, or prompting policies whose objective is to elicit a disallowed response while remaining hard to detect. The defense population becomes a population of guardrail strategies: system prompts, classifiers, or refusal policies whose objective is to block disallowed responses while keeping the false refusal rate low. Both sides face a genuine multi-criterion trade-off, which is why the multi-criterion machinery in Section 2 carries over rather than a simple win or lose signal: an attacker that always succeeds but is trivially detectable is not actually strong, and a defender that never fails but refuses half of all benign requests is not actually useful. The existing studies either do not consider multiple objectives for each agent in coevolutionary studies or combine multiple criteria into a single one by a scalarization approach which are always subjective and non-generic.

The part of the mechanism we think matters most is the combination of cross-play aggregation and alternation. Existing adversarial robustness work for language models typically evaluates a fixed set of jailbreak techniques against a fixed model [8, 12, 3], which finds real weaknesses but only tells us about that particular snapshot of attacker and defender. If a defender is instead only trained against the current best-known attack, it can overfit to that attack and remain fragile against a different one, and the reverse is true for an attacker trained against a single defense. Cross-play evaluation directly addresses this: a candidate jailbreaker is scored against the entire defender population, not the single best defender, and its fitness is an aggregate measure of all of them. Alternation between the two phases then makes sure that when the defender population is updated, the jailbreaker population has to keep up in the next phase rather than the two populations drifting independently. Importantly, the multiple Pareto strategies provide alternative options to each agent to analyze and choose a single preferred one. With a single criterion based coevolutionary approach will always produce a single strategy for each agent at each step, without the flexibility of choosing among a few by the human or machine-based agents, forcing agents to "take" it or "leave" it.

We can also carry over the movement metric idea in Section 2 as a diagnostic for safety, not just as a robustness training procedure. A cooperative upper bound for the jailbreaker side would be its attack success rate against an undefended model; a cooperative upper bound for the defender side would be its refusal accuracy against a fully compliant, non-adversarial user. The gap between each of these upper bounds and what each side actually achieves at co-evolutionary equilibrium is a direct, measurable answer to a question that is currently answered mostly by intuition: how much is a given guardrail actually costing an attacker, and how much is a given attacker actually costing a defender’s usability, providing a transparent trade-off information at every step to both agents.

Why the proposed mapping is an agentic web problem?

The setup above is still a two-population picture, which on its own looks like an adversarial training method rather than something specific to the agentic web. We think the extension to that setting is what makes this workshop the right venue for this study, for three reasons.

First, at internet scale the attacker and defender sides are not each a single strategy space, they are themselves heterogeneous populations of independently operated agents that discover each other rather than being paired by a benchmark designer. A discovery and pairing mechanism, not just an evaluation loop, becomes part of the problem: which jailbreaker agents get matched against which defender agents, and how that matching itself evolves, is closely related to the capability discovery question this workshop already identifies as open.

Second, credit assignment becomes harder once there are many agents on each side rather than two populations. In the wargame version, aggregating a strategy’s fitness across every opponent it faced is tractable because both populations are of a fixed, moderate size and every match is a cheap simulation. At web scale, a defender agent may be probed by thousands of independently evolving attacker lineages, and a single simulation is not a cheap local computation but an actual language model call. Whether alternating, cross-play co-evolution still converges rather than destabilizes when the population sizes and the number of simultaneous lineages grow is an open question, and it is exactly the kind of learning-dynamics question we think this workshop is targeting to answer.

Third, the outcome we want is not just two robust populations in isolation, it is a form of trust that can propagate across a network: an agent that has proven resistant to a wide, co-evolved range of manipulation should be able to carry that reputation into interactions with agents it has never met. This connects the mechanism directly to the manipulation-resistant reputation problem this workshop names as a core open direction, rather than leaving it as a self-contained adversarial training result.

Open problems and a note on dual use

Moving this mechanism from a wargame simulator to language model agents raises problems that did not exist in the original setting. The cross-play evaluation matrix requires Noff×NdefN_{\mathrm{off}} \times N_{\mathrm{def}} evaluations per generation; when each evaluation is a language model call rather than a fast simulator step, this cost grows quickly and some form of surrogate modeling or opponent sub-sampling, similar in spirit to the surrogate models used in the earliest wargame version of this framework [5], will likely be necessary. A more recent version of this framework removes surrogates entirely and operates directly on the game dynamics to avoid surrogate inaccuracy [10], which is only possible because a wargame simulator step is cheap; a language model call is not, so this trade-off between surrogate accuracy and evaluation cost will need to be revisited for agentic web settings. The two-population framework also makes a simplification: a more realistic agentic web setting has many attacker and defender lineages rather than just two agents, and it is not yet known whether the alternating cycle that keeps two populations from drifting apart still works once there are many populations updating on different schedules. In the presence of multiple agents, how would the sequence of agents affect the final optimal strategies must be researched. Moreover, the effect of cooperation and/or conflict among multiple jailbreaker agents on defender’s optimal strategy would be another interesting direction to pursue. Our multi-criterion coevolutionary strategy is generic enough to study all these possibilities, a gamut of which we plan to embed in an agentic web application.

Finally, we want to be direct about their dual use. A method for co-evolving strong jailbreak strategies is, by construction, also a method for producing strong jailbreak strategies, and any implementation of this idea should keep the co-evolved attacker population access-controlled and should report defender robustness rather than attacker prompts.

Conclusion

Competitive co-evolution with alternating cycles and cross-play evaluation in the original study was built to optimize and to measure balance between two adapting sides in a wargame setting We have argued here that the same mechanism is a natural way to harden jailbreaker and defender agents against each other. The ability to consider multiple conflicting decision criteria for each agent is an unique hallmark feature of our approach. The multi-criterion coevolutionary approach allows each agent to analyze multiple trade-off Pareto strategies before choosing the most preferred and appropriate strategy in every move, thereby providing the confidence and comprehensiveness of chosen decisions. Extending it to many heterogeneous, discoverable agents at web scale, rather than two fixed populations, is what turns it from an adversarial training trick into a genuine agentic web research problem.

References

  1. [1]Luis Miguel Antonio and Carlos A Coello Coello. Coevolutionary multiobjective evolutionary algorithms: Survey of the state-of-the-art. IEEE Transactions on Evolutionary Computation, 22(6): 851–865, 2017.DOI
  2. [2]J. Bader, K. Deb, and E. Zitzler. Faster hypervolume-based search using Monte Carlo sampling. In Proceedings of Multiple Criteria Decision Making (MCDM 2008), pages 313–326. Heidelberg: Springer, 2010. LNEMS-634.DOI
  3. [3]Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 23–42. IEEE, 2025.arxiv.org/abs/2310.08419
  4. [4]Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and T. Meyarivan. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE transactions on evolutionary computation, 6(2):182–197, 2002.DOI
  5. [5]Ritam Guha, Ryan Mckendrick, Bradley Feest, and Kalyanmoy Deb. Attacker-defender strategy optimization using multi-objective competitive co-evolution. In International Conference on Parallel Problem Solving from Nature, pages 351–366. Springer, 2024.DOI
  6. [6]Ritam Guha, Ryan McKendrick, Bradley Feest, and Kalyanmoy Deb. A multi-objective competitive co-evolutionary framework with progressive shrinking for wargame scenarios. In International Conference on Evolutionary Multi-Criterion Optimization, pages 288–302. Springer, 2025a.DOI
  7. [7]Ritam Guha, Ryan McKendrick, Bradley Feest, and Kalyanmoy Deb. Multiobjective competitive co-evolutionary optimization and regularity-based decision-making for two-agent wargame strategy optimization. IEEE Transactions on Evolutionary Computation, 30(2):652–666, 2025b.DOI
  8. [8]Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. pages 3419–3448, 2022.arxiv.org/abs/2202.03286
  9. [9]Shashank Raj, Auden Garrard, Ryan McKendrick, Bradley Feest, and Kalyanmoy Deb. To balance competitive games using a multi-objective coevolutionary approach. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1191–1199, 2026a.DOI
  10. [10]Shashank Raj, Ryan McKendrick, Bradley Feest, and Kalyanmoy Deb. Towards balanced wargames: A multi-objective co-evolutionary adversarial approach for full-game and meta-game balance. Technical Report 2026008, COIN Report, 2026b. URL https://kdeb.us/papers/c2026008.pdf. Accessed August 23, 2026.
  11. [11]L. While, P. Hingston, L. Barone, and S. Huband. A faster algorithm for calculating hypervolume. IEEE Transactions on Evolutionary Computation, 10(1):29–38, 2006.DOI
  12. [12]Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023.arxiv.org/abs/2307.15043

Paper details

vlnc:W_znj3drsh
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents