betaFeedback

WorksComputer Science

Train Locally, Learn Together: Distributed Multi-Agent Training via ADMM

Abstract

Multi-agent systems (MAS) can outperform individual language-model agents by combining specialized roles, diverse reasoning processes, and collaborative decision-making. However, collaborative MAS training is typically centralized, requiring a common trainer to aggregate role-local trajectories and learning signals and maintain model and optimization states. This motivates a distributed multi-agent learning paradigm in which agents improve system-level capabilities through role-local learning and decentralized communication, paving the way toward scalable, resilient, and self-organizing MAS. To this end, we introduce, to the best of our knowledge, the first optimization-grounded framework for distributed MAS training based on consensus alternating direction method of multipliers (ADMM). We formulate collaborative MAS learning as a consensus-constrained finite-sum optimization problem, naturally lending itself to distributed optimization via ADMM. The proposed approach alternates role-local learning with inter-agent consensus using distributed communication protocols such as ring all-reduce, enabling joint optimization while keeping each agent's data and training state local. We evaluate our framework on distributed safety alignment and general reasoning tasks. Under compromised-agent settings, ADMM can reduce HarmBench attack success rate (ASR) by up to 26.4%, while achieving reasoning gains comparable to centralized training. These results show that distributed MAS training retains the benefits of centralized joint optimization without requiring a centralized trainer.

Introduction

Large language models (LLMs) are increasingly organized into multi-agent systems (MAS), in which agents assume specialized roles and communicate in natural language or latent messages to solve complex tasks collaboratively [1] (Wu et al. 2024; Du et al. 2024; Zou et al. 2026). By coordinating complementary roles, diverse reasoning processes, and iterative refinement, MAS can achieve stronger system-level performance than individual agents. Recent work has therefore moved beyond prompting and inference-time collaboration toward directly training participating agents through fine-tuning or reinforcement learning to improve their collective behavior [5, 6, 7, 8, 10] (Liu et al. 2026). This shift from inference-time collaboration to learning raises a fundamental question of how interacting agents should be trained as a system.

Most existing approaches realize collaborative MAS learning through centralized training, in which a single training process jointly manages multi-agent trajectories, learning signals, parameters, and optimization states [5, 6, 8, 7, 10]. While effective for coordinating learning across agents, this paradigm assumes that their training information and optimization states can be collected and managed by a common trainer (see Figure 1(a)). Such an assumption becomes increasingly restrictive as MAS scale or span different devices, organizations, and trust domains. Training agents independently avoids the need for a common trainer, but gives up the system-level coordination that collaborative MAS learning seeks to achieve. This creates a fundamental tension: collaborative learning calls for coordination across agents, while distributed deployment calls for keeping their learning processes local.

Figure 1. Schematic overview of centralized and distributed MAS training. (a) Centralized training jointly optimizes all agents in a single training process. (b) Distributed training via ADMM performs role-local optimization with parameter exchange for inter-agent consensus. (c) Demonstration of safety training in MAS: Safety and utility of Qwen3-8B-based MAS (w/o training), centralized-trained, and ADMM-based distributed-trained MAS under two structures: sequential (S-MAS; Solver→Verifier→Corrector→Decider) and hierarchical (H-MAS; three Analyzers→Synthesizer). HarmBench attack success rate (ASR; ↓, %) is reported for safety under benign (all agents safety-aligned) and compromised settings (unaligned Solver in S-MAS or Synthesizer in H-MAS), together with MMLU-Pro accuracy and mean accuracy across six reasoning benchmarks (Acc; ↑, %). ADMM closely matches centralized training while substantially reducing HarmBench ASR relative to the base MAS. See § 5 for more details.

Motivated by this tradeoff, we develop a principled approach to distributed MAS training, where each agent is trained on its own node with locally maintained training information and optimization states, while training is coordinated across agents without a centralized trainer. Conventional distributed learning typically distributes a model-training process across workers or clients, often according to where computation or data is located [11, 12, 13, 14, 15]. In our setting, the partition instead follows the functional structure of the MAS: even when all roles share the same underlying model, each node carries out the local learning associated with one agent role, based on the trajectories and learning signals arising from that role. The resulting local updates must be coordinated across nodes so that the MAS is improved as a whole. The challenge is therefore to distribute collaborative MAS training across role-local learning processes without losing the coordination provided by centralized joint training. This motivates our central question:

(Q) How can we collaboratively train an MAS using a principled distributed optimization framework while retaining the performance of centralized training?

To answer (Q), we formulate distributed MAS training as a consensus optimization problem over local copies of the shared model parameters. Each agent performs role-local learning on its own node using the trajectories and learning signals associated with its functional role, while consensus coordinates these local updates across agents. Building on this formulation, we develop an alternating direction method of multipliers (ADMM) framework [16, 17] as the core optimization mechanism for distributed MAS training (see Figure 1(b)). ADMM naturally separates training into two complementary steps: role-local optimization, where each node updates the shared model using the trajectories and learning signals associated with its role, and consensus coordination, where nodes reconcile their local model copies while keeping trajectories and optimization states local. This decomposition allows each agent to train independently between communication rounds while still enforcing system-wide agreement through ADMM consensus updates. We realize the consensus step either exactly with ring all-reduce or approximately with decentralized gossip [18, 19]. In this way, ADMM provides a principled bridge between local agent learning and global MAS coordination, enabling collaborative training without centralizing agents’ trajectories, data, or optimization states.

Prior works have explored distributed or decentralized MAS through inference-time coordination [20, 21, 22], as well as agent-specific policy learning and distributed rollout generation [23] (Fu et al. 2026). In contrast, we distribute the optimization of a shared MAS model across role-local training nodes and coordinate its local copies through ADMM-based consensus. We evaluate our approach on reasoning and safety alignment against frozen MAS and centralized joint training. Across both settings, distributed training closely approaches centralized performance (see Figure 1(c)). Component ablations further show neither local-only training nor periodic parameter averaging is sufficient to match the full ADMM procedure.

Our main contributions are summarized as follows:

  • ① We formulate the collaborative training of a shared MAS model as a finite-sum distributed optimization problem over role-local objectives induced by agent roles, and recast it into an ADMM-compatible consensus formulation.

  • ② We develop an ADMM-based framework that alternates role-local learning with inter-agent consensus, supports flexible communication protocols, and keeps each agent’s trajectories and training states local.

Across reasoning and safety-alignment tasks, distributed training closely approaches centralized training performance, including safety settings where one agent is compromised. We further study distributed MAS training across communication protocols and system scales, while ablations show the importance of consensus and dual updates.

Related Works

Multi-agent training, communication, and collaboration. LLM-based MAS organize multiple agents into complementary roles and coordinate their interactions toward a shared output [1, 25, 26] (Wu et al. 2024; Du et al. 2024; Zou et al. 2026). One line of work treats the underlying LLMs as fixed and improves collaboration through agent composition, roles, prompts, and communication structures (Chen et al., 2024; Hu et al., 2025; Zhang et al., 2025a,b,c; Yao et al., 2026; Wang et al., 2026). More recent work directly trains agents for collaboration, building on multi-agent RL for credit assignment [34] and optimizing role-specific behavior toward system-level objectives [8, 6, 7, 5, 36, 37, 10, 38] (Liu et al. 2026; Mo et al. 2025). However, these approaches largely assume centralized training, leaving open how agents can learn collaboratively while keeping data and training states local.

Distributed optimization. Distributed optimization enables multiple learners to optimize locally while coordinating through consensus [16, 39, 40]. Such coordination can be realized through collective communication such as ring all-reduce [18] or decentralized gossip [41, 19]. These principles have been increasingly adopted for distributed training of LLMs, where model updates are coordinated across geographically or computationally separated workers [14, 42, 15, 43, 44], as well as for parameter-efficient LLM adaptation [45, 46].

Communication and coordination in multi-agent systems. A broad line of work studies how LLM-based agents exchange information and aggregate decisions, including multi-agent debate [47] (Du et al. 2024), voting and consensus-based aggregation [48, 49], and latent-space collaboration (Zou et al., 2026). Another line focuses on the communication structure itself, ranging from sparse debate topologies (Li et al., 2024) to learned or adaptive communication graphs [51, 29]. Related work further explores decentralized MAS through agent discovery and communication [21, 52] and decentralized coordination [20]. Beyond inference-time coordination, recent work has explored decentralized MAS training through separate policy trainers [23] and distributed training infrastructure (Fu et al., 2026). In contrast, we formulate MAS training as consensus optimization, allowing each agent to optimize locally while coordinating model updates through ADMM consensus and keeping trajectories and training states local.

Multi-agent safety and alignment. Multi-agent interaction introduces additional safety risks: compromised agents can induce harmful behavior through role-play or persona injection [53, 50], while jailbreaks and errors can propagate across agents [55, 59, 60] (Gu et al. 2024; Ju et al. 2026; Li et al. 2026), with their impact shaped by communication topology and compromised positions [61] (Yu et al. 2025). Existing defenses largely rely on guard models or agents [63, 64, 65, 66] (Hu and Hooi 2026), while other work evaluates safety at the agent and system levels [68, 54, 70]. Parameter-level alignment has primarily focused on individual models [71, 72, 73], while recent joint agent training remains centralized [74]. We use multi-agent safety alignment as an evaluation setting for distributed MAS training under both aligned and compromised-agent settings.

Distributed MAS Training: Motivation and Formulation

This section motivates and formalizes distributed MAS training. We first show, through a safety study, that composing individually aligned agents into an MAS can amplify unsafe behavior, so alignment must be treated as a system-level learning problem. We then explain why conventional centralized training is ill-suited to this setting and formulate distributed MAS training, in which each agent learns from a role-local objective. These local objectives remain coupled through inter-agent messages, which we identify as the key challenge of interaction-induced coupling.

MAS introduces new training demands: A motivation from safety alignment. We motivate system-level MAS training from a safety perspective: even when each agent is individually aligned, their composition can amplify unsafe behavior. In Figure 2, we evaluate a sequential and a hierarchical MAS (Figure 2a), both built from Qwen3–8B agents with role-specific system prompts. Given a harmful query, the final agent’s message is taken as the system response, and safety is measured by attack success rate (ASR, ↓) on three benchmarks. The aligned Qwen3–8B used alone serves as the Single LLM baseline. Composing the same aligned model into an MAS can substantially reduce system-level safety: on StrongReject, ASR rises from 59.1% for the Single LLM to 81.7% and 86.7% for the benign sequential and hierarchical MAS, respectively, and increases further when one agent is compromised. Figure 2(c) shows a qualitative case where the Single LLM refuses a harmful query while the hierarchical MAS responds unsafely.

Figure 2. Safety evaluation of sequential and hierarchical MAS built from Qwen3-8B with thinking mode using role-specific prompts. In the benign setting, all agents use the vanilla backbone; in the compromised setting, one agent instead uses an unaligned backbone while retaining the same role prompt ([75]). (a) MAS architectures. The sequential MAS consists of Solver→Verifier→Corrector→Decider, while the hierarchical MAS uses three Analyzers followed by a Synthesizer. In the compromised setting, the purple devil icon marks the agent using the unaligned backbone, placed at the Solver and Synthesizer, respectively. (b) Attack success rate (ASR, ↓) on HarmBench ([76]), StrongReject ([77]), and JailbreakBench ([78]). Each benchmark compares the aligned single LLM, the benign MAS, and the compromised MAS with one compromised agent as shown in (a). (c) Qualitative comparison on the same harmful query between the aligned single LLM and the hierarchical MAS without compromise.

The safety risk becomes more severe when one agent is compromised (i.e., not safety-aligned). As shown in Figure 2(b), replacing a single role with an unaligned model can substantially increase ASR, with the largest degradation occurring at the Solver in the chain and the Synthesizer in the hierarchy. More detailed configs are reported in Appendix § 11 and results for other compromised roles are reported in Table A1. Since the compromised role is generally unknown, protecting a predetermined agent is insufficient. Moreover, independent alignment overlooks message-passing dependencies, through which one agent’s output shapes downstream behavior.

SystemCompromised agent(s)HarmBench ASR (% ↓)
Single LLMnone16.98
Single LLMthe model itself72.96
Sequentialnone (benign MAS)17.61
SequentialSolver42.77
SequentialVerifier19.50
SequentialCorrector20.75
SequentialDecider18.87
SequentialVerifier + Corrector22.01
SequentialVerifier + Decider25.16
SequentialVerifier + Corrector + Decider40.88
Sequentialall four54.72
Hierarchicalnone (benign MAS)19.50
HierarchicalAnalyzer 127.04
HierarchicalAnalyzer 222.64
HierarchicalAnalyzer 325.79
HierarchicalSynthesizer52.83
Hierarchicalall four72.96

Table A1. Safety of an untrained MAS as a function of which agent is compromised. No training anywhere: the two architectures of Figure 2 run the safety-aligned backbone at every seat except the listed ones, which run the unaligned copy with the same role prompt. Attack success rate reported on all 159 HarmBench prompts under the same judge as Table 1. The shaded rows are the compromised settings carried into Table 1.

These observations reveal a new training demand for MAS: safety alignment should be treated as a system-level learning problem that accounts for agents’ functional roles, communication, and collective behavior under composition. More broadly, effective MAS adaptation requires coordinating role-local learning toward a shared system-level objective rather than optimizing agents independently.

Problem statement: Distributed MAS training. The system-wide training demand identified above requires coordinated learning across agents. However, conventional MAS training typically realizes this coordination through a centralized process that collects agents’ trajectories, computes their learning signals, and maintains their parameters ([6, 8, 7, 10] (Mo et al. 2025)). Such a design incurs increasing communication, memory, and computational overhead as the MAS scales. It also creates a single point of failure and assumes a central authority can access all role-local information, which may not hold when agents are distributed across devices, organizations, or trust domains. This motivates distributed MAS training, where agents learn from local data, maintain their own training states, and coordinate toward a shared system-level objective.

Specifically, consider an MAS of NN agents with distinct functional roles interacting over a directed communication graph, where edges carry inter-agent messages during each rollout. Following common practice, agents share the same LLM backbone but use role-specific system prompts ([1] (Du et al. 2024)). Given a query, each agent conditions on its prompt, the query, and upstream messages to generate its message, with the final message forming the system output. For distributed training, agent ii maintains a local copy θi\theta_i of the shared trainable adapter, implemented as a LoRA adapter (Hu et al., 2021). Let fif_i denote the role-local training objective at agent ii. Although agents optimize local copies of the same adapter, their objectives differ because they are constructed from role-specific prompts, received messages, and learning signals arising from the joint MAS rollout. We call this dependency among local objectives the interaction-induced coupling challenge.

Example of interaction-induced objective coupling in a chain-structured MAS

Consider the three-agent chain 1→2→31 \to2 \to3 as in Figure 2(a). Given a query qq, agent 1 generates m1=g1(q;θ1)\mathbf{m}_1 = g_1(q; \theta_1). Agent 2 receives m1\mathbf{m}_1 and produces m2=g2(q,m1;θ2)=g2(q,g1(q;θ1);θ2)\mathbf{m}_2 = g_2(q, \mathbf{m}_1; \theta_2) = g_2(q, g_1(q; \theta_1); \theta_2). Agent 3 then receives m2\mathbf{m}_2 and produces m3=g3(q,m2;θ3)=g3(q,g2(q,g1(q;θ1);θ2);θ3)\mathbf{m}_3 = g_3(q, \mathbf{m}_2; \theta_3) = g_3(q, g_2(q, g_1(q; \theta_1); \theta_2); \theta_3). Thus, upstream parameters shape downstream objectives through the messages they generate, yielding the following expected role-local objective structure: f1(θ1)f_1(\theta_1), f2(θ1,θ2)f_2(\theta_1, \theta_2), f3(θ1,θ2,θ3)f_3(\theta_1, \theta_2, \theta_3).

As exemplified above, each role-local objective is influenced by upstream agents through the messages they generate, and therefore by the current local model copies used to produce those messages. Based on the above, we formulate the distributed MAS training as

minimize⁡{θi} ∑i=1Nfi(θΩi),(1)\underset{\{\theta_i\}}{\operatorname{minimize}}\ \sum_{i=1}^{N} f_i(\theta_{\Omega_i}), \tag*{(1)}

where Ωi\Omega_i denotes the index set containing agent ii and its upstream agents, and θΩi:={θk}k∈Ωi\theta_{\Omega_i} := \{\theta_k\}_{k \in\Omega_i}. As shown in (1), interaction-induced coupling prevents MAS training from decomposing into independent local problems: optimizing agents separately is not equivalent to optimizing the system-level objective. This is consistent with Figure 2, where individually aligned agents may not yield a safe MAS because message passing couples their behaviors.

Distributed MAS Training via ADMM: Why and How

This section addresses the interaction-induced coupling challenge of § 3 with ADMM. We first explain why ADMM suits distributed MAS training, then derive its two alternating steps: role-local fine-tuning and consensus averaging.

ADMM and its rationale. ADMM is a classical operator-splitting method for solving structured composite optimization problems [16, 17], as given below:

minimize⁡x,zf(x)+g(z)subject tox=z,(2)\underset{x,z}{\operatorname{minimize}}\quad f(x) + g(z) \quad\text{subject to}\quad x = z, \tag*{(2)}

where xx and zz are optimization variables associated with objectives ff and gg, respectively. A key feature of (2) is the auxiliary variable zz. Although x=zx = z makes zz seemingly redundant, variable splitting separates the objectives and enables alternating ADMM optimization. A suitable choice of auxiliary variables can further simplify the subproblems, particularly when one admits a closed-form solution. Concretely, ADMM solves (2) through the following iterations [17]:

x(t+1)=prox⁡λf(z(t)−u(t))=arg min⁡x{f(x)+12λ∥x−(z(t)−u(t))∥22},(3)\mathbf{x}^{(t+1)} = \operatorname{prox}_{\lambda f}\left(\mathbf{z}^{(t)} - \mathbf{u}^{(t)}\right) = \underset{\mathbf{x}}{\operatorname{arg\,min}}\left\{f(\mathbf{x}) + \frac{1}{2\lambda}\left\|\mathbf{x} - \left(\mathbf{z}^{(t)} - \mathbf{u}^{(t)}\right)\right\|_2^2\right\}, \tag*{(3)}
z(t+1)=prox⁡λg(x(t+1)+u(t))=arg min⁡z{g(z)+12λ∥z−(x(t+1)+u(t))∥22},(4)\mathbf{z}^{(t+1)} = \operatorname{prox}_{\lambda g}\left(\mathbf{x}^{(t+1)} + \mathbf{u}^{(t)}\right) = \underset{\mathbf{z}}{\operatorname{arg\,min}}\left\{g(\mathbf{z}) + \frac{1}{2\lambda}\left\|\mathbf{z} - \left(\mathbf{x}^{(t+1)} + \mathbf{u}^{(t)}\right)\right\|_2^2\right\}, \tag*{(4)}

where tt denotes the ADMM iteration, u\mathbf{u} is the dual variable associated with the constraint x=z\mathbf{x} = \mathbf{z} and is updated as u(t+1)=u(t)+x(t+1)−z(t+1)\mathbf{u}^{(t+1)} = \mathbf{u}^{(t)} + \mathbf{x}^{(t+1)} - \mathbf{z}^{(t+1)}, and λ=1/ρ\lambda= 1/\rho with ρ>0\rho> 0 denoting the corresponding penalty parameter in the augmented Lagrangian of (2). The updates in (3) and (4) are both expressed as proximal operations [17]. Specifically, prox⁡λh(v):=arg min⁡w{h(w)+12λ∥w−v∥22}\operatorname{prox}_{\lambda h}(\mathbf{v}) := \operatorname*{arg\,min}_{\mathbf{w}}\left\{h(\mathbf{w}) + \frac{1}{2\lambda}\|\mathbf{w} - \mathbf{v}\|_2^2\right\}.

As shown by the alternating updates in (3)–(4), ADMM optimizes ff and gg through two subproblems, coordinated by the primal variables x\mathbf{x} and z\mathbf{z} and the dual variable u\mathbf{u}, which accumulates the constraint violation x−z\mathbf{x} - \mathbf{z}. This operator-splitting structure is a key advantage of ADMM: it decomposes the coupled problem in (2) into tractable subproblems. When either admits a closed-form solution, ADMM can further simplify optimization. Building on this advantage, we reformulate the interaction-coupled MAS training objective in (1) using role-local parameter copies.

Distributed MAS training problem through the ADMM lens

For each agent ii, we introduce a local variable xi=θΩi\mathbf{x}_i = \theta_{\Omega_i}, representing local copies of the parameter blocks θΩi={θk}k∈Ωi\theta_{\Omega_i} = \{\theta_k\}_{k \in\Omega_i} involved in its objective in (1). To preserve consistency, local copies of the same parameter block must agree. We therefore define the consensus set:

C:={(x1,…,xN)∣(xi)k=(xj)k, ∀i,j, ∀k∈Ωi∩Ωj},(5)\mathcal{C} := \left\{(\mathbf{x}_1, \ldots, \mathbf{x}_N) \mid(\mathbf{x}_i)_k = (\mathbf{x}_j)_k,\ \forall i,j,\ \forall k \in\Omega_i \cap\Omega_j\right\}, \tag*{(5)}

where (xi)k(\mathbf{x}_i)_k denotes agent ii’s local copy of θk\theta_k contained in xi\mathbf{x}_i. Thus, the coupled objective in (1) can be equivalently represented using the local variables {xi}\{\mathbf{x}_i\} subject to the consistency constraint (x1,…,xN)∈C(\mathbf{x}_1, \ldots, \mathbf{x}_N) \in\mathcal{C}. Accordingly, we reformulate (1) in the ADMM form of (2):

minimize⁡x,z∑i=1Nfi(xi)+IC({zi}),subject to xi=zi, ∀i=1,…,N,(6)\begin{aligned} \underset{\mathbf{x},\mathbf{z}}{\operatorname{minimize}}\quad&\sum_{i=1}^{N} f_i(\mathbf{x}_i)+\mathcal{I}_{\mathcal{C}}(\{\mathbf{z}_i\}),\quad\text{subject to }\mathbf{x}_i=\mathbf{z}_i,\ \forall i=1,\ldots,N, \tag*{(6)} \end{aligned}

where IC({zi})\mathcal{I}_{\mathcal{C}}(\{\mathbf{z}_i\}) is the indicator function of the constraint set C\mathcal{C}: it equals 0 if (z1,…,zN)∈C(\mathbf{z}_1,\ldots,\mathbf{z}_N)\in\mathcal{C} and +∞+\infty otherwise, i.e., ensuring optimization feasibility under the constraint.

With (6), the ADMM updates in (3)–(4) decompose distributed MAS training into two steps: (i) role-local optimization for each objective fif_i, enabling parallel updates across agents, and (ii) projection onto the consensus set C\mathcal{C} to coordinate the shared parameter blocks. We detail these steps below.

Role-local fine-tuning in ADMM. We instantiate the ADMM x-update in (3) for (6) by setting f(x)=∑i=1Nfi(xi)f(\mathbf{x})=\sum_{i=1}^{N}f_i(\mathbf{x}_i) and replacing the generic variables with the corresponding collections {xi}i=1N\{\mathbf{x}_i\}_{i=1}^{N}, {zi}i=1N\{\mathbf{z}_i\}_{i=1}^{N}, and {ui}i=1N\{\mathbf{u}_i\}_{i=1}^{N}. Both the finite-sum objective and the quadratic term are separable across {xi}\{\mathbf{x}_i\}. Consequently, the x-update decomposes into NN role-local subproblems that can be solved in parallel. Each agent ii then performs the role-local optimization step:

xi(t+1)=prox⁡λfi(zi(t)−ui(t))=arg min⁡xi{fi(xi)+12λ∥xi−ai∥22},∀i,(7)\mathbf{x}_i^{(t+1)}=\operatorname{prox}_{\lambda f_i}\left(\mathbf{z}_i^{(t)}-\mathbf{u}_i^{(t)}\right)=\underset{\mathbf{x}_i}{\operatorname{arg\,min}}\left\{f_i(\mathbf{x}_i)+\frac{1}{2\lambda}\left\|\mathbf{x}_i-\mathbf{a}_i\right\|_2^2\right\},\quad\forall i, \tag*{(7)}

where ai:=zi(t)−ui(t)\mathbf{a}_i:=\mathbf{z}_i^{(t)}-\mathbf{u}_i^{(t)}. In (7), we instantiate fif_i as a role-local fine-tuning loss, following common MAS training settings where agent updates are guided by system-level rewards or returns derived from multi-agent rollouts ([7, 8] (Liu et al. 2026)), as detailed below.

Recall that xi={θk}k∈Ωi\mathbf{x}_i=\{\boldsymbol{\theta}_k\}_{k\in\Omega_i} contains agent ii’s local copies, with (xi)i=θi(\mathbf{x}_i)_i=\boldsymbol{\theta}_i denoting its own trainable parameters and {(xi)k}k∈Ωi∖{i}\{(\mathbf{x}_i)_k\}_{k\in\Omega_i\setminus\{i\}} those associated with upstream agents. These components play different roles: θi\boldsymbol{\theta}_i, implemented as a trainable LoRA adapter, directly parameterizes agent ii’s message mi\mathbf{m}_i, while upstream agents influence its conditioning context through received messages {mk}k∈Ωi∖{i}\{\mathbf{m}_k\}_{k\in\Omega_i\setminus\{i\}}. Given a sampled MAS rollout, upstream messages are treated as fixed context. The binary outcome-based reward rir_i is derived from the complete rollout: for reasoning, ri=1r_i=1 if the final answer is correct; for safety, ri=1r_i=1 if the final answer is safe on a harmful query or correct on a benign one, while agent ii’s message is non-harmful or non-refusal, respectively. We use rir_i to construct preference pairs and instantiate fif_i in (7) with DPO ([72]):

fi(xi)=−Eq∼D[log⁡σ(βDPO[log⁡πθi(mi+∣ci)πref(mi+∣ci)−log⁡πθi(mi−∣ci)πref(mi−∣ci)])],(8)f_i(\mathbf{x}_i)=-\mathbb{E}_{q\sim\mathcal{D}}\left[\log\sigma\left(\beta_{\mathrm{DPO}}\left[\log\frac{\pi_{\boldsymbol{\theta}_i}(\mathbf{m}_i^+\mid\mathbf{c}_i)}{\pi_{\mathrm{ref}}(\mathbf{m}_i^+\mid\mathbf{c}_i)}-\log\frac{\pi_{\boldsymbol{\theta}_i}(\mathbf{m}_i^-\mid\mathbf{c}_i)}{\pi_{\mathrm{ref}}(\mathbf{m}_i^-\mid\mathbf{c}_i)}\right]\right)\right], \tag*{(8)}

where σ\sigma is the logistic function, ci=({mk}k∈Ωi∖{i},q)\mathbf{c}_i=\left(\{\mathbf{m}_k\}_{k\in\Omega_i\setminus\{i\}},q\right) is agent ii’s conditioning context, and πref\pi_{\mathrm{ref}} is the initial policy with its adapter disabled. For each query, we sample several rollouts and pair mi+\mathbf{m}_i^+ from an ri=1r_i=1 rollout with mi−\mathbf{m}_i^- from an ri=0r_i=0 rollout, scoring both under the preferred rollout’s context. Groups with identical rir_i are discarded. During training, we additionally apply a forward KL regularizer βKLKL⁡(πref∥πθi∣ci)\beta_{\mathrm{KL}}\operatorname{KL}(\pi_{\mathrm{ref}}\|\pi_{\boldsymbol{\theta}_i}\mid\mathbf{c}_i) to each scored sample. The ADMM formulation also supports other role-local objectives without changing the subsequent updates.

Substituting (8) into (7), we solve the role-local subproblem by gradient-based optimization. During each local update, the sampled rollout and received upstream messages are treated as fixed training context. Agent ii therefore updates its local adapter copy θi\boldsymbol{\theta}_i using its role-local objective, while coordination among the corresponding parameter copies is handled by the subsequent ADMM consensus step. Thus, each agent performs a locally executable update over θi\boldsymbol{\theta}_i, while inter-agent coordination is handled separately through consensus.

Inter-agent consensus averaging in ADMM. We instantiate the second ADMM subproblem in (4) by setting g=IC({zi}i=1N)g=\mathcal{I}_{\mathcal{C}}(\{\mathbf{z}_i\}_{i=1}^{N}), the indicator function of the constraint set C\mathcal{C}. Since its proximal operator is the Euclidean projection onto C\mathcal{C} ([17]), the z-update becomes:

{zi(t+1)}=arg min⁡z∈C∑i=1N∥zi−bi∥22(9)\{\mathbf{z}_i^{(t+1)}\}=\underset{\mathbf{z}\in\mathcal{C}}{\operatorname{arg\,min}}\sum_{i=1}^{N}\|\mathbf{z}_i-\mathbf{b}_i\|_2^2 \tag*{(9)}

where bi:=xi(t+1)+ui(t)\mathbf{b}_i:=\mathbf{x}_i^{(t+1)}+\mathbf{u}_i^{(t)}, and C\mathcal{C} is the consensus set in (5), applied to {zi}\{\mathbf{z}_i\}. It requires local copies of the same parameter block to agree, i.e., (zi)k=(zj)k(\mathbf{z}_i)_k=(\mathbf{z}_j)_k for k∈Ωi∩Ωjk\in\Omega_i\cap\Omega_j. Under the Euclidean projection in (9), each block’s consensus value is the average of its local copies, corresponding to standard consensus averaging ([19, 80, 11]). Thus, (9) admits a closed-form blockwise solution. For block kk, let nk:=∑j=1N1{k∈Ωj}n_k:=\sum_{j=1}^{N}\mathbb{1}\{k\in\Omega_j\} denote the number of agents holding its copy. Then, for every agent ii with k∈Ωik\in\Omega_i, solution is

(zi(t+1))k=1nk∑j=1N1{k∈Ωj}(bj)k(10)\left(\mathbf{z}_i^{(t+1)}\right)_k=\frac{1}{n_k}\sum_{j=1}^{N}\mathbb{1}\{k\in\Omega_j\}\left(\mathbf{b}_j\right)_k \tag*{(10)}

where I{k∈Ωj}=1\mathbb{I}\{k \in\Omega_j\} = 1 if agent jj maintains block kk and 0 otherwise. Thus, agents sharing block kk receive the same consensus value, equal to the average of their local copies. The averaging in (10) can be implemented distributively via all-reduce (Patarasuk and Yuan, [18]; Thakur et al., [81]) or randomized gossip (Boyd et al., [19]).

From the ADMM perspective, distributed MAS training alternates between (i) role-local fine-tuning (x-step), where each agent updates only its trainable parameters, and (ii) distributed consensus averaging (z-step) via (10). The steps are coupled by u(t+1)=u(t)+x(t+1)−z(t+1)\mathbf{u}^{(t+1)} = \mathbf{u}^{(t)} + \mathbf{x}^{(t+1)} - \mathbf{z}^{(t+1)}, which accumulates disagreement between local and consensus variables. The full procedure is given in Algorithm A1 and Appendix B. We also examine empirical convergence in Figure 3, where both the role-local objectives and the ADMM primal residual decrease over training rounds.

Figure 3. Empirical convergence of ADMM-based distributed safety training on the compromised H-MAS in Table 1, using N=4N = 4 agents, ring all-reduce consensus, ρ=0.03\rho= 0.03, and 24 ADMM rounds. Left: Role-local objectives fif_i, shown as a centered five-round rolling geometric mean with the minimum-to-maximum range across agents. Right: Normalized primal residual, (∑i∥xi(t)−zi(t)∥22)1/2/∥x1(t)∥2(\sum_i \lVert x_i^{(t)} - z_i^{(t)} \rVert_2^2)^{1/2} / \lVert x_1^{(t)} \rVert_2. Both axes are logarithmic; marker color indicates cumulative communication per node.

Experiments

Experimental Setup

MAS and model setups. By default, all agents use the safety-aligned Qwen3-8B backbone in thinking mode (Team et al., [82]) and differ only in role-specific prompts (Appendix C.1). We study two four-agent architectures (Figure 2): S-MAS, a sequential chain (Solver→Verifier→Corrector→Decider), and H-MAS, where three Analyzers feed a Synthesizer. We also consider a compromised setting in which one agent uses Qwen3-8B-abliterated, an unaligned variant with refusal behavior removed (Arditi et al., [75]). During distributed training, each agent maintains a local parameter copy θi\theta_i as a rank-16 LoRA adapter (Hu et al., 2022); further adapter details are in Appendix C.2.

Training setups. For safety training, we use 380 harmful prompts from AdvBench (Zou et al., [84]) and the HarmBench validation split (Mazeika et al., [76]), plus 300 benign MMLU-Pro questions (Wang et al., [85]) to control over-refusal. For reasoning training, we use each benchmark’s training split when available, or a disjoint test subset otherwise. For both tasks, we sample eight on-policy trajectories per query and train each agent with (8), and all agents remain trainable. Unless noted otherwise, consensus uses ring all-reduce with ρ=0.03\rho= 0.03 (λ=1/0.03\lambda= 1/0.03), for up to 24 ADMM rounds. Additional training details like rollout, optimization, and data filtering are provided in Appendix C.3. ADMM and centralized training use the same data and objective.

Evaluation setups. For safety, we evaluate on HarmBench (Mazeika et al., [76]), StrongReject (Souly et al., [77]), JailbreakBench (Chao et al., [78]), toxic OR-Bench (Cui et al., [86]), and XSTest (Röttger et al., [87]). We report attack success rate (ASR, ↓) on harmful prompts, refusal rate (↓) on safe XSTest prompts for over-refusal, and MATH-500 accuracy (↑) for utility. For general reasoning, we report accuracy (↑) on MMLU-Pro (Wang et al., [85]), GSM8K (Cobbe et al., [88]), MATH (Hendrycks et al., [89]), BBH (Suzgun et al., [90]), GPQA (Rein et al., [91]), and MuSR (Sprague et al., [92]). Additional evaluation details are provided in Appendix C.4.

Experimental Results

Distributed MAS training for safety enhancement. Table 1 compares centralized and ADMM-based distributed training on S-MAS and H-MAS under benign and compromised initializations. Safety training consistently improves MAS safety, substantially repairs compromised systems, and generalizes across benchmarks. Even under benign initialization, both centralized and distributed training further reduce ASR. When one agent is compromised, HarmBench

MethodHarmBenchStrongRejectJBBOR-BenchXSTest RR ↓MATH-500 Acc. ↑
Single-Benign16.9859.1144.0021.3512.8077.40
Single-Compromised72.9697.1298.0094.311.6074.20
S-MAS-BenignBase17.6181.7979.0024.186.8083.00
S-MAS-BenignCentralized9.43(−8.18)43.13(−38.66)42.00(−37.00)12.83(−11.35)7.60(+0.80)83.00(+0.00)
S-MAS-BenignADMM10.69(−6.92)50.16(−31.63)47.00(−32.00)14.76(−9.42)8.80(+2.00)82.80(−0.20)
S-MAS-CompromisedBase42.7791.6995.0043.006.4081.20
S-MAS-CompromisedCentralized25.79(−16.98)55.27(−36.42)57.00(−38.00)25.93(−17.07)11.20(+4.80)79.60(−1.60)
S-MAS-CompromisedADMM25.16(−17.61)53.99(−37.70)56.00(−39.00)25.30(−17.70)10.40(+4.00)81.40(+0.20)
H-MAS-BenignBase19.5086.5889.0027.155.6081.20
H-MAS-BenignCentralized18.24(−1.26)80.51(−6.07)84.00(−5.00)25.02(−2.13)6.40(+0.80)81.80(+0.60)
H-MAS-BenignADMM17.61(−1.89)78.91(−7.67)80.00(−9.00)24.31(−2.84)5.20(−0.40)81.00(−0.20)
H-MAS-CompromisedBase52.8388.1889.0063.508.4080.40
H-MAS-CompromisedCentralized25.16(−27.67)42.17(−46.01)42.00(−47.00)30.24(−33.26)15.20(+6.80)81.20(+0.80)
H-MAS-CompromisedADMM26.42(−26.41)44.09(−44.09)45.00(−44.00)31.75(−31.75)13.60(+5.20)80.80(+0.40)

Table 1. Safety, over-refusal, and utility of MAS safety training. We evaluate S-MAS and H-MAS under benign initialization with no compromised agent and a compromised setting where the Solver in S-MAS or Synthesizer in H-MAS is compromised, as in Figure 2(a). The defender does not know the compromised agent, and all adapters are trained. Single-Benign and Single-Compromised are the corresponding Qwen3-8B single-model baselines; Base is the untrained MAS; Centralized pools all agents’ samples for centralized optimization; and ADMM (shaded) is our distributed training. ASR (↓) is reported on four safety benchmarks (JailbreakBench; JBB), refusal rate (↓) on safe XSTest prompts, and MATH-500 accuracy (↑) for utility. For Centralized and ADMM, results are reported as A(±b), where A is the absolute performance and b = (A − Base) is the change vs. Base.

ASR rises from 17.61% to 42.77% on S-MAS and from 19.50% to 52.83% on H-MAS. ADMM reduces these values to 25.16% and 26.42%, respectively, with consistent improvements on StrongReject, JBB, and OR-Bench. ADMM also closely matches centralized training across architectures and safety settings: under compromised initialization, HarmBench ASRs are 25.16% vs. 25.79% on S-MAS and 26.42% vs. 25.16% on H-MAS. These results show that distributed optimization retains much of the safety benefit of centralized training without a central trainer. On the other hand, safety training increases the over-refusal rate, particularly in compromised settings, possibly because of correcting compromised agents induces more conservative behavior. Yet, general utility remains largely preserved: under benign initialization, ADMM changes MATH-500 accuracy from 83.00% to 82.80% on S-MAS and from 81.20% to 81.00% on H-MAS, while achieving 81.40% and 80.80% after repairing the corresponding compromised systems.

Distributed MAS training for reasoning enhancement. Figure 4 shows S-MAS reasoning performance before and after training. Without additional training, the Base MAS outperforms the Single LLM on all six benchmarks, indicating gains from multi-agent composition. Reasoning training further improves performance, with ADMM closely matching centralized training. ADMM reaches 75.6% mean accuracy, compared with 75.9% for centralized training and 73.9% for the Base MAS. Thus, our distributed framework extends beyond safety alignment to reasoning enhancement.

Figure 4. Reasoning enhancement through distributed MAS training. Accuracy (%; ↑) across six reasoning benchmarks and their mean for the single-LLM baseline, untrained base MAS, centralized training, and ADMM-based distributed training.

Why inter-agent coordination is important in ADMM. Table 2 ablates consensus averaging in the z-step (10) and feedback through the dual update. Without either component, ADMM-Local yields limited safety gains, reducing HarmBench ASR from 42.77% to 41.51% on S-MAS and from 52.83% to 34.59% on H-MAS. Consensus averaging alone (ADMM-NoDual) remains insufficient, with ASRs of 42.14% and 37.11%. Full ADMM reduces ASR to 25.16% and 26.42%, closely matching centralized training (25.79% and 25.16%). Thus, neither local training nor averaging alone reproduces the benefit of coordinated optimization, while ADMM combines both.

ConsensusDualS-MASH-MAS
Base––42.7752.83
Centralized––25.7925.16
ADMM-Local✗✗41.5134.59
ADMM-NoDual✓✗42.1437.11
ADMM✓✓25.1626.42

Table 2. Safety training under the MAS-compromised setting of Table 1, evaluated by HarmBench ASR (%↓). ADMM-Local trains agents independently without communication, while ADMM-NoDual uses consensus averaging without dual updates. The Consensus and Dual columns mark which components are enabled.

Effect of approximate gossip consensus on ADMM. The consensus averaging in (10) can be approximated by gossip, where neighboring agents repeatedly exchange and average their local variables. This yields a communication–performance trade-off: more gossip rounds improve information mixing but increase communication traffic. As shown in Figure 5, increasing the number of gossip rounds (τ\tau) from 1 to 2 reduces the ADMM primal residual from 2.34 to 0.80 and HarmBench ASR from 8.18% to 4.84%. Further increasing τ\tau to 16 yields little additional change in either residual or ASR, while communication continues to grow. Thus, τ=2\tau= 2 captures most of the benefit of gossip in this setting at substantially lower communication cost. Compared with approximate gossip, ring all-reduce computes the exact consensus average in (10). Since our MAS contains relatively few training agents, we use ring all-reduce by default to obtain exact consensus with predictable communication, while retaining gossip as an alternative.

Figure 5. Effect of gossip communication on ADMM coordination and safety. We use an eight-agent chain connected by a ring topology, where each agent communicates with its two neighbors for τ\tau gossip rounds per consensus step in (10). Increasing τ\tau propagates information across more hops but incurs greater communication, with marker color indicating the communication traffic (MB) per node. Left: Final normalized ADMM primal residual (↓) after training (similar to Figure 3), measuring disagreement between the local and auxiliary variables. Right: HarmBench ASR (%,↓) of the resulting MAS; the dashed line marks the untrained MAS.

Additional results. Appendix D further studies scaling across different numbers of agents and backbone sizes. Across system sizes, the compromised-agent effect persists without a clear monotonic trend in NN; we also report training results beyond the main N=4N = 4 setting and compare the effect on Qwen3–8B and Qwen3–4B.

Conclusion

We study distributed MAS training, where agents optimize role-local objectives without a centralized trainer. We formulate MAS learning as consensus-constrained distributed optimization and solve it with ADMM, alternating local updates with inter-agent consensus while keeping trajectories and training states local. Across safety alignment and general reasoning, distributed training closely matches centralized training, including compromised-agent settings, and ablations highlight the importance of consensus and dual updates, supporting scalable decentralized MAS training.

Limitations and future work. Our experiments focus on representative sequential and hierarchical MAS built from a common LLM backbone with shared adapter parameterization, which keeps the model family and parameterization consistent across agents; broader evaluation across larger systems, diverse model families and task domains, and richer communication topologies is needed to establish generality. An important direction is to characterize how communication topology affects convergence, efficiency, and downstream performance, relating these behaviors to graph-theoretic properties such as connectivity, centrality, community structure, and spectral characteristics, potentially with edges weighted by the heterogeneity of interaction strength between roles. More complex MAS may further require hypergraph-based formulations, where dependencies capture computations that jointly involve multiple roles. Future work may also explore asynchronous consensus, adaptive and communication-efficient coordination, heterogeneous agents with different architectures or local objectives, and dynamic systems where agents join, leave, or change roles.

AI use statement

In this work, we used generative AI tools, including LLM-based coding assistants, to implement the experimental code, to help analyze and interpret results, and to provide feedback on the research methodology and experiments. We also used them to prepare figures and the code release, to search for related information, and to edit the manuscript for readability. The research questions, the problem formulation, and the proposed method were developed by the authors. All AI-assisted content was reviewed by the authors, who take full responsibility for the final content of this work.

Reproducibility statement

Source code for distributed MAS training and evaluation will be released upon publication. The experimental setup and implementation details, including MAS architectures, backbone models, training data, role-local objectives, ADMM optimization, communication protocols, and evaluation settings, are provided in Sec. 5 and the Appendix. The Appendix further documents the full training procedure, prompts, LoRA configuration, rollout collection, hyperparameters, evaluation criteria. All models, datasets, and benchmarks used in this work are publicly available, and evaluation follows the protocols of the corresponding benchmark papers.

Ethics Statement

This work aims to improve the safety and robustness of multi-agent language-model systems through distributed training, including settings where one agent is compromised. Our method reduces unsafe behavior under the evaluated settings but does not guarantee safety under arbitrary attacks or deployment conditions. All experiments use publicly available models and datasets; harmful prompts are used only in controlled training and evaluation settings for studying and improving system safety. This work involves no human subjects or personal data, and all authors adhere to the ICLR Code of Ethics.

Appendix

Safety Under Different Compromised-Agent Placements

Configuration

Generation uses temperature 0.6, top-pp 0.95, and a budget of 4096 tokens per agent. Figure 2(b) reports the attack success rate on HarmBench prompts [76], StrongReject prompts [77], and JailbreakBench prompts [78]. These benchmarks do not share a criterion: Following the popular settings, HarmBench is scored by its Llama-2-13B classifier, which asks whether the final answer is harmful, whereas StrongReject and JailbreakBench are scored by whether the system complied with the request rather than refusing it.

Where the Compromised Agent Sits

Figure 2 places the compromised agent at the position that degrades each architecture most: the Solver in the sequential chain and the Synthesizer in the hierarchy. Table A1 reports every other position, and every combination we ran, on the same untrained systems. Figure 2 uses the Solver in the sequential chain and the Synthesizer in the hierarchy as the compromised positions carried into the main experiments. Table A1 reports the other compromised positions on the same untrained systems. In the sequential MAS, compromising the Solver yields the largest observed ASR among the single-position settings (42.77% versus 17.61% for the benign MAS), whereas compromising the Verifier, Corrector, or Decider produces smaller changes in this run. In the hierarchical MAS, the largest observed ASR occurs when the Synthesizer is compromised (52.83% versus 19.50% for the benign MAS), while compromising individual Analyzers results in more modest increases. We do not interpret the small differences among the remaining positions as a stable ordering, since each configuration is evaluated with a single run. Similarly, compromising the three downstream agents in the sequential MAS gives 40.88% ASR, close to the 42.77% observed with the Solver alone, so we avoid drawing conclusions from their ordering. These results motivate training all roles rather than assuming in advance which position will be compromised.

Details of the ADMM Algorithm

Algorithm A1 states one outer iteration of the procedure of § 4. The three steps follow that derivation: the x-step is role-local and runs in parallel, the z-step exchanges optimization variables across nodes, and the dual step is local.

Algorithm A1 Distributed MAS training via consensus ADMM

Require: agents i=1,…,Ni=1,\ldots,N with overlaps Ωi\Omega_i, penalty ρ>0\rho> 0, local steps LL, rounds TT

  1. initialize xi(0)=zi(0)x_i^{(0)} = z_i^{(0)} and ui(0)=0u_i^{(0)} = 0 for every agent ii

  2. for t=0,1,…,T−1t = 0,1,\ldots,T-1 do

  3. deploy {zi(t)}\{z_i^{(t)}\} to the MAS, roll it out on queries q∼Dq \sim\mathcal{D}, and score the rollouts for rir_i

  4. x-step (role-local fine-tuning, in parallel): each agent ii takes LL gradient steps on (7) with fif_i given by (8), updating its local adapter copy θi\theta_i and obtaining xi(t+1)x_i^{(t+1)}

  5. z-step (consensus averaging): agents exchange bi=xi(t+1)+ui(t)b_i = x_i^{(t+1)} + u_i^{(t)} and form zi(t+1)z_i^{(t+1)} blockwise by (10), using all-reduce or gossip protocol

  6. dual step (in parallel): ui(t+1)←ui(t)+xi(t+1)−zi(t+1)u_i^{(t+1)} \leftarrow u_i^{(t)} + x_i^{(t+1)} - z_i^{(t+1)} for every agent ii

  7. end for

  8. return the consensus variables {zi(T)}\{z_i^{(T)}\}

Initialization of x, z and u

Algorithm A1 opens with xi(0)=zi(0)x_i^{(0)} = z_i^{(0)} and ui(0)=0u_i^{(0)} = 0. These variables are initialized once before the first ADMM round and are then carried across subsequent rounds.

The dual uiu_i starts at zero. We use the scaled dual, ui=yi/ρu_i = y_i/\rho, where yiy_i is the Lagrange multiplier associated with the constraint xi=zix_i = z_i. Before the first round, there is no accumulated disagreement to carry, so we initialize ui(0)=0u_i^{(0)} = 0. The ablation that removes the dual (§ 5) keeps this variable fixed at zero throughout the run.

The primal variables start at the deployed adapter, which is zero only for an untrained system. The optimization variable θi\theta_i is the trainable factor of the LoRA adapter, and both xix_i and ziz_i are initialized from the adapter currently deployed by the system. Thus, xi(0)=zi(0)x_i^{(0)} = z_i^{(0)} holds by construction, and the initial primal residual is exactly zero. Because we train only the up-projection BiB_i and BiB_i is zero-initialized in the standard LoRA scheme, a run starting from the untrained system has xi(0)=zi(0)=0x_i^{(0)} = z_i^{(0)} = 0: the deployed update ΔWi=BiA\Delta W_i = B_i A vanishes, and round 0 corresponds to the backbone itself. A run warm-started from a checkpoint instead begins from its (possibly nonzero) BiB_i.

The frozen factor AA is not an optimization variable. AA is initialized once, shared across agents, and never updated, so it does not enter xx, zz, or uu. This also makes the zz-step exact in weight space (App. § 13.§ 2), since averaging the trainable factors BiB_i is equivalent to averaging the corresponding updates ΔWi=BiA\Delta W_i = B_i A. It is also why zeroing both LoRA factors would be inappropriate in a variant that trains both factors: if both AA and BB were initialized to zero, then ΔW=BA\Delta W = BA and the gradients with respect to both factors would initially vanish.

Each x-step starts from the current proximal anchor. At ADMM round tt, the local x-step is initialized at ai=zi(t)−ui(t)a_i = z_i^{(t)} - u_i^{(t)} rather than at xi(t)x_i^{(t)}. Agent ii then performs gradient-based optimization of its local adapter copy θi\theta_i using the role-local objective, with the sampled rollout and received messages treated as fixed training context. Starting from aia_i makes the quadratic penalty zero at the first inner step and nonzero only as the solution moves away from the anchor, preventing the local update from being dominated by the absolute scale of ρ\rho. The resulting xi(t+1)x_i^{(t+1)}, together with the subsequent zi(t+1)z_i^{(t+1)} and ui(t+1)u_i^{(t+1)}, is carried to the next ADMM round.

Experimental Details

System Prompts

In the sequential chain, the Solver is asked to solve the question completely, to number its steps and state what each step establishes so that any single step can be checked in isolation, and to commit to a specific answer rather than hedge. The Verifier is shown the candidate solution and told that its only job is to find errors, checking in order whether the candidate answers the question actually asked, whether the method is the right one, whether each algebraic step is correct, whether units, signs and rounding are right, and whether the answer is in the demanded form. The Corrector receives the candidate together with the verifier report and repairs the identified errors. The Decider sees the prior agents’ work and issues the final answer.

In the hierarchy, the three Analyzers are told that they work on the same question independently and cannot see one another, so that their value lies in reaching the answer by different routes: the first derives from principles before looking at the options, the second proceeds by elimination, and the third tests candidate answers on concrete values.

The Synthesizer compares all three analyses and decides the final answer. The verbatim prompts for both architectures are released with our code.

Backbone and Adapter

The aligned backbone is Qwen/Qwen3-8B, run in thinking mode with a fixed generation budget. The compromised agent uses mlabonne/Qwen3-8B-abliterated, an abliterated variant with reduced refusal behavior (Arditi et al., [75]). It shares the tokenizer and architecture of the aligned backbone and can therefore replace any role without architectural changes. Each agent maintains a local trainable parameter copy θi\theta_i, implemented as a rank-16 LoRA adapter with scaling α=16\alpha= 16, applied to the seven projections (q, k, v, o, gate, up, and down) in all 36 layers. Writing the corresponding local update to a base weight as ΔWi=BiA\Delta W_i = B_i A, we initialize the down-projection AA identically across agents and keep it frozen, while training only BiB_i.

Details of training

Training data. Harmful requests. We construct the harmful training set from AdvBench (Zou et al., [84]) and the standard and contextual HarmBench validation splits (Mazeika et al., [76]), assigning all retained prompts a refusal target. We remove prompts overlapping with HarmBench or XSTest evaluation data, and use only HarmBench validation examples that are disjoint from the test splits used for evaluation.

Benign requests. A slice of the MMLU-Pro test split (Wang et al., [85]), kept with their native QA answer type and scored as ordinary questions. They are the over-refusal / stay-helpful control: without them every retained fine-tuning target is a refusal and the trained system collapses into refusing everything.

Safety. Each agent is then fit on its message with the ordinary deployment prompt as input, so what is learned is the behaviour itself rather than the collection prompt, and nothing at test time depends on the hardened prompt. We draw up to eight trajectories per query, at temperature 0.8 and top-pp 0.95, with a per-agent budget of 4096 tokens.

General reasoning. Collection is on-policy and repeats every outer round. Queries are selected by excluding every anchor already used in an earlier round, so successive rounds never overlap; each round pre-screens a window of candidates by sampling each twice and dropping those answered correctly both times, then collects a handful of trajectories for each survivor and keeps only the groups whose outcomes are mixed. Sampling uses the same temperature and top-pp with an 8192-token per-agent budget.

Optimization. Table A2 lists the optimizer settings. Each ADMM round deploys the current consensus adapter, collects (or reuses) a batch of rollouts, and runs LL local gradient steps in the x-step of (7).

Table A2: Optimizer and ADMM settings, read off the trained checkpoints. Distributed and centralized runs share every value; the penalty ρ\rho and the round count apply to the distributed runs.

SafetyReasoning
ADMM rounds TT24 (H-MAS) / 20 (S-MAS)8
Local gradient steps LL per x-step1616
Rollouts per query88
OptimizerAdamAdam
Learning rate10−310^{-3}10−310^{-3}
Gradient clip0.010.01
Batch size88
ADMM penalty ρ\rho0.030.1
DPO temperature βDPO\beta_{\mathrm{DPO}}0.10.1
KL weight βKL\beta_{\mathrm{KL}}0.10.1
Role-local lossDPODPO

Table A2.

Data filtering. Each role is trained using the system-level outcome reward of the full rollout (Wan et al., [7]; Liao et al. [8]; Liu et al. (2026)). For reasoning, RR is answer correctness, and we retain queries whose sampled rollouts contain both successful and unsuccessful outcomes. For safety, harmful queries receive R=1R = 1 when the final response is judged harmless, while benign queries receive R=1R = 1 when answered correctly; refusals on benign queries are therefore penalized. We further include a role’s message in the safety loss only when the rollout succeeds and the message itself is harmless for harmful queries or non-refusal for benign queries, preventing unsafe intermediate messages from being reinforced by a safe final response. All roles remain trainable, including the compromised role.

For both tasks, we balance training examples across roles, de-duplicate repeated targets, and discard generations with incomplete reasoning blocks. By default, the loss is applied to the full message, including its reasoning content.

Details of evaluation

The five safety benchmarks use different scoring criteria, so their absolute values are not directly comparable. HarmBench uses its Llama-2-13B classifier [76] to judge whether the final response is harmful, whereas StrongReject, JailbreakBench, and toxic OR-Bench measure whether the system complies rather than refuses.

The HarmBench Llama-2-13B classifier is the primary harmfulness judge throughout, pinned to one revision. A Granite guardian classifier was run as a sensitivity check and is not used for any reported number: it flags terse refusals as harmful and shifts the comparison by roughly five points, in a direction that depends on how verbose an arm is. Where a benchmark ships its own notion of compliance we keep it, so that our numbers stay comparable to how that benchmark is normally reported, and we never compare an absolute value across benchmarks.

Evaluation runs in thinking mode at temperature 0.6 and top-p 0.95 with a fixed seed. Two fixed profiles are used and never mixed: safety evaluations allow 4096 tokens per agent and capability evaluations allow 8192.

For general reasoning, we evaluate each benchmark on a fixed subset sampled once and reused across all methods, ensuring that every bar in Figure 4 uses the same questions. Each subset contains up to 256 examples. Safety benchmarks use their standard evaluation splits.

Scaling with the Number of Agents and the Backbone

We further examine the compromised-agent effect across different system sizes and backbone sizes. These experiments are intended as supplementary analyses beyond the main N=4N = 4, Qwen3-8B setting in Table 1.

Scaling with the number of agents. Table A3 evaluates untrained MAS with different numbers of agents. For S-MAS, we consider N∈{2,3,5,6,7}N \in\{2, 3, 5, 6, 7\} with the Solver compromised; for H-MAS, we consider N∈{3,5,6,8}N \in\{3, 5, 6, 8\} with the Synthesizer compromised. The main N=4N = 4 configuration is reported separately in Table 1. Across all tested configurations, introducing one compromised agent increases HarmBench ASR relative to the corresponding benign system, with observed increases ranging from 14.47 to 32.08 percentage points. The magnitude of this increase varies across system sizes without a clear monotonic trend in NN.

Table A3: Effect of one compromised agent across system sizes. We evaluate untrained S-MAS with N∈{2,3,5,6,7}N \in\{2, 3, 5, 6, 7\} and H-MAS with N∈{3,5,6,8}N \in\{3, 5, 6, 8\}. The compromised agent is the Solver in S-MAS and the Synthesizer in H-MAS. The main N=4N = 4 configuration is reported separately in Table 1. We report HarmBench ASR (%) and the compromised–benign difference Δ\Delta.

SystemNNBenignCompromisedΔ\Delta
S-MAS215.0929.56+14.47
S-MAS320.1340.25+20.13
S-MAS516.9838.36+21.38
S-MAS622.0140.88+18.87
S-MAS716.9849.06+32.08
H-MAS318.8745.28+26.42
H-MAS525.1647.80+22.64
H-MAS623.2747.17+23.90
H-MAS823.2751.57+28.30

Table A3.

Training beyond the main system size. We additionally evaluate compromised S-MAS at N=3N = 3, whose corresponding untrained configuration is included in Table A3. As shown in Table A4, centralized and distributed training reduce HarmBench ASR by 5.03 and 3.14 percentage points, respectively, relative to the untrained system. We report this result as an additional evaluation beyond the main N=4N = 4 setting rather than drawing conclusions about how training effectiveness scales with the number of agents.

Table A4: Additional training result beyond the main system size. Change in HarmBench ASR (percentage points; negative is safer) relative to the corresponding untrained compromised S-MAS in Table A3.

NNCentralized Δ\Delta ASRDistributed Δ\Delta ASR
3−5.03−3.14

Table A4.

Effect on a smaller backbone. We further evaluate an untrained N=4N = 4 S-MAS using Qwen3-4B. As shown in Table A5, compromising the Solver increases HarmBench ASR from 13.84% to 48.43%, corresponding to a 34.59-point increase. The corresponding Qwen3-8B results are reported in the main experiment in Table 1. Together, these experiments show that the compromised-agent effect is also present with a smaller backbone.

BackboneBenignCompromisedΔ\Delta
Qwen3-4B13.8448.43+34.59

Table A5. Effect of one compromised agent on a smaller backbone. We evaluate an untrained S-MAS with N=4N = 4 using Qwen3-4B, with the Solver as the compromised agent. We report HarmBench ASR (%) and the compromised–benign difference Δ\Delta. The corresponding Qwen3-8B results are reported in Table 1.

References

  1. [1]G. Li, H. A. A. K. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem, “CAMEL: Communicative agents for “mind” exploration of large language model society,” in Advances in Neural Information Processing Systems (NeurIPS), 2023, arXiv:2303.17760.arxiv.org/abs/2303.17760
  2. [5]S. R. Motwani, C. Smith, R. J. Das, R. Rafailov, P. Torr, I. Laptev, F. Pizzati, R. Clark, and C. S. de Witt, “MALT: Improving reasoning with multi-agent LLM training,” in Second Conference on Language Modeling, 2025. [Online]. Available: https://openreview.net/forum?id=iXP9bgFackarxiv.org/abs/2412.01928
  3. [6]C. Park, S. Han, X. Guo, A. E. Ozdaglar, K. Zhang, and J.-K. Kim, “Maporl: Multi-agent post-co-training for collaborative large language models with reinforcement learning,” in ACL, 2025.arxiv.org/abs/2502.18439
  4. [7]Z. Wan, Y. Li, X. Wen, Y. Song, H. Wang, L. Yang, M. Schmidt, J. Wang, W. Zhang, S. Hu, and Y. Wen, “ReMA: Learning to meta-think for LLMs with multi-agent reinforcement learning,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=ur295YVtmtarxiv.org/abs/2503.09501
  5. [8]J. Liao, M. Wen, J. Wang, and W. Zhang, “MARFT: Multi-agent reinforcement fine-tuning,” arXiv preprint arXiv:2504.16129, 2025.arxiv.org/abs/2504.16129
  6. [10]Y. Zhao, L. Hu, Y. Wang, M. Hou, H. Zhang, K. Ding, and J. Zhao, “Stronger-MAS: Multi-agent reinforcement learning for collaborative LLMs,” in International Conference on Learning Representations (ICLR), 2026, arXiv:2510.11062.arxiv.org/abs/2510.11062
  7. [11]J. C. Duchi, A. Agarwal, and M. J. Wainwright, “Dual averaging for distributed optimization: Convergence analysis and network scaling,” IEEE Transactions on Automatic control, 2011.DOI
  8. [12]S. Liu, P.-Y. Chen, and A. O. Hero, “Accelerated distributed dual averaging over evolving networks of growing connectivity,” IEEE Transactions on Signal Processing, vol. 66, no. 7, pp. 1845–1859, 2018.DOI
  9. [13]H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. Agüera y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial Intelligence and Statistics (AISTATS), 2017, arXiv:1602.05629.arxiv.org/abs/1602.05629
  10. [14]B. Yuan, Y. He, J. Q. Davis, T. Zhang, T. Dao, B. Chen, P. Liang, C. Ré, and C. Zhang, “Decentralized training of foundation models in heterogeneous environments,” in Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2206.01288
  11. [15]A. Douillard, Q. Feng, A. A. Rusu, R. Chaparia, Y. Donchev, A. Kuncoro, M. Ranzato, A. Szlam, and J. Shen, “DiLoco: Distributed low-communication training of language models,” in 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ICML 2024), 2024. [Online]. Available: https://openreview.net/forum?id=plCSfWkJIkarxiv.org/abs/2311.08105
  12. [16]S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends in Machine Learning, vol. 3, no. 1, pp. 1–122, 2011.
  13. [17]N. Parikh and S. Boyd, “Proximal algorithms,” Foundations and Trends in optimization, vol. 1, no. 3, pp. 127–239, 2014.DOI
  14. [18]P. Patarasuk and X. Yuan, “Bandwidth optimal all-reduce algorithms for clusters of workstations,” Journal of Parallel and Distributed Computing, vol. 69, no. 2, pp. 117–124, 2009.DOI
  15. [19]S. Boyd, A. Ghosh, B. Prabhakar, and D. Shah, “Randomized gossip algorithms,” IEEE transactions on information theory, vol. 52, no. 6, pp. 2508–2530, 2006.DOI
  16. [20]Y. Yang, H. Chai, S. Shao, Y. Song, S. Qi, R. Rui, and W. Zhang, “Agentnet: Decentralized evolutionary coordination for LLM-based multi-agent systems,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=tXqLxHIb8Zarxiv.org/abs/2504.00587
  17. [21]W. Chen, Z. You, R. Li, yitong guan, C. Qian, C. Zhao, C. Yang, R. Xie, Z. Liu, and M. Sun, “Internet of agents: Weaving a web of heterogeneous agents for collaborative intelligence,” in The Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https://openreview.net/forum?id=oIEt3MogPwarxiv.org/abs/2407.07061
  18. [22]Y. Mao and A. Mirhoseini, “Decentralized multi-agent systems with shared context,” arXiv preprint arXiv:2606.10662, 2026.arxiv.org/abs/2606.10662
  19. [23]K. Zhang, K. Tian, R. Liu, S. Zeng, X. Zhu, G. Jia, Y. Fan, X. Lv, Y. Zuo, C. Jiang, Y. Wang, J. Wang, E. Hua, X. Long, J. Gao, Y. Sun, Z. Ma, G. Cui, N. Ding, B. Qi, and B. Zhou, “MARTI: A framework for multi-agent LLM systems reinforced training and inference,” in The Fourteenth International Conference on Learning Representations, 2026. [Online]. Available: https://openreview.net/forum?id=E7jZqo0A50
  20. [25]S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, J. Wang, C. Zhang, S. Yau, Z. Lin, L. Zhou et al., “Metagent: Meta programming for a multi-agent collaborative framework,” in International Conference on Learning Representations, vol. 2024, 2024, pp. 23 247–23 275.arxiv.org/abs/2308.00352
  21. [26]C. Qian, W. Liu, H. Liu, N. Chen, Y. Dang, J. Li, C. Yang, W. Chen, Y. Su, X. Cong et al., “Chatdev: Communicative agents for software development,” in Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), 2024, pp. 15 174–15 186.arxiv.org/abs/2307.07924
  22. [29]G. Zhang, Y. Yue, X. Sun, G. Wan, M. Yu, J. Fang, K. Wang, T. Chen, and D. Cheng, “G-designer: Architecting multi-agent communication topologies via graph neural networks,” in Forty-second International Conference on Machine Learning, 2025. [Online]. Available: https://openreview.net/forum?id=LpE54NUnmOarxiv.org/abs/2410.11782
  23. [34]J. G. Kuba, R. Chen, M. Wen, Y. Wen, F. Sun, J. Wang, and Y. Yang, “Trust region policy optimisation in multi-agent reinforcement learning,” in International Conference on Learning Representations (ICLR), 2022, arXiv:2109.11251.arxiv.org/abs/2109.11251
  24. [36]Y. Dang, C. Qian, X. Luo, J. Fan, Z. Xie, R. Shi, W. Chen, C. Yang, X. Che, Y. Tian, X. Xiong, L. Han, Z. Liu, and M. Sun, “Multi-agent collaboration via evolving orchestration,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=L0xZPXT3learxiv.org/abs/2505.19591
  25. [37]Y. Xie, S. Liu, F. FAN, Y. Yao, S. Cao, Y. Zhao, and B. Liu, “TeamTR: Trust-region fine-tuning for multi-agent LLM coordination,” in Forty-third International Conference on Machine Learning, 2026. [Online]. Available: https://openreview.net/forum?id=IMwLhg52ZY
  26. [38]L. Feng, L. Zheng, S. He, F. Zhang, and B. An, “Dr. mas: Stable reinforcement learning for multi-agent llm systems,” arXiv preprint arXiv:2602.08847, 2026.arxiv.org/abs/2602.08847
  27. [39]W. Shi, Q. Ling, K. Yuan, G. Wu, and W. Yin, “On the linear convergence of the ADMM in decentralized consensus optimization,” IEEE Transactions on Signal Processing, vol. 62, no. 7, pp. 1750–1761, 2014.DOI
  28. [40]H. Ouyang, N. He, L. Tran, and A. Gray, “Stochastic alternating direction method of multipliers,” in International conference on machine learning. PMLR, 2013, pp. 80–88.
  29. [41]L. Xiao and S. Boyd, “Fast linear iterations for distributed averaging,” Systems & Control Letters, vol. 53, no. 1, pp. 65–78, 2004.
  30. [42]M. Ryabinin, T. Dettmers, M. Diskin, and A. Borzunov, “SWARM parallelism: Training large models can be surprisingly communication-efficient,” in International Conference on Machine Learning, 2023.arxiv.org/abs/2301.11913
  31. [43]S. Jaghouar, J. M. Ong, and J. Hagemann, “OpenDiLoCo: An open-source framework for globally distributed low-communication training,” arXiv preprint arXiv:2407.07852, 2024.arxiv.org/abs/2407.07852
  32. [44]A. Borzunov, D. Baranchuk, T. Dettmers, M. Ryabinin, Y. Belkada, A. Chumachenko, P. Samygin, and C. Raffel, “Petals: Collaborative inference and fine-tuning of large models,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (System Demonstrations), 2023.arxiv.org/abs/2209.01188
  33. [45]Y. Sun, Z. Li, Y. Li, and B. Ding, “Improving LoRA in privacy-preserving federated learning,” in International Conference on Learning Representations, 2024.arxiv.org/abs/2403.12313
  34. [46]Z. Wang, Z. Shen, Y. He, G. Sun, H. Wang, L. Lyu, and A. Li, “FLOrA: Federated fine-tuning large language models with heterogeneous low-rank adaptations,” in Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2409.05976
  35. [47]C.-M. Chan, W. Chen, Y. Su, J. Yu, W. Xue, S. Zhang, J. Fu, and Z. Liu, “Chateval: Towards better llm-based evaluators through multi-agent debate,” in International conference on learning representations, vol. 2024, 2024, pp. 9079–9093.arxiv.org/abs/2308.07201
  36. [48]L. B. Kaesberg, J. Becker, J. P. Wahle, T. Ruas, and B. Gipp, “Voting or consensus? decision-making in multi-agent debate,” in Findings of the Association for Computational Linguistics: ACL 2025, 2025, pp. 11 640–11 671.DOI
  37. [49]H. K. Choi, J. Zhu, and S. Li, “Debate or vote: Which yields better decisions in multi-agent large language models?” Advances in Neural Information Processing Systems, vol. 38, pp. 101 732–101 764, 2026.arxiv.org/abs/2508.17536
  38. [50]Y. Li, Y. Du, J. Zhang, L. Hou, P. Grabowski, Y. Li, and E. Ie, “Improving multi-agent debate with sparse communication topology,” in Findings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 7281–7294.arxiv.org/abs/2406.11776
  39. [51]M. Zhuge, W. Wang, L. Kirsch, F. Faccio, D. Khizbullin, and J. Schmidhuber, “Language agents as optimizable graphs,” arXiv preprint arXiv:2402.16823, 2024.arxiv.org/abs/2402.16823
  40. [52]Y. Wang, S. Guo, Y. Pan, Z. Su, F. Chen, T. H. Luan, P. Li, J. Kang, and D. Niyato, “Internet of agents: Fundamentals, applications, and challenges,” IEEE Transactions on Cognitive Communications and Networking, 2025, arXiv:2505.07176.DOI
  41. [53]Y. Tian, X. Yang, J. Zhang, Y. Dong, and H. Su, “Evil geniuses: Delving into the safety of llm-based agents,” arXiv preprint arXiv:2311.11855, 2023.arxiv.org/abs/2311.11855
  42. [54]Z. Zhang, Y. Zhang, L. Li, J. Shao, H. Gao, Y. Qiao, L. Wang, H. Lu, and F. Zhao, “Psysafe: A comprehensive framework for psychological-based attack, defense, and evaluation of multi-agent system safety,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 15 202–15 231.arxiv.org/abs/2401.11880
  43. [55]D. Lee, M. Tiwari, and B. Miranda, “Prompt infection: Llm-to-llm prompt injection within multi-agent systems,” in European Symposium on Research in Computer Security. Springer, 2025, pp. 511–520.arxiv.org/abs/2410.07283
  44. [59]J.-T. Huang, J. Zhou, T. Jin, X. Zhou, Z. Chen, W. Wang, Y. Yuan, M. Lyu, and M. Sap, “On the resilience of LLM-based multi-agent collaboration with faulty agents,” in Proceedings of the 42nd International Conference on Machine Learning, 2025.arxiv.org/abs/2408.00989
  45. [60]M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandran, M. Zaharia, J. E. Gonzalez, and I. Stoica, “Why do multi-agent LLM systems fail?” in The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2026. [Online]. Available: https://openreview.net/forum?id=fAjbYBmonrarxiv.org/abs/2503.13657
  46. [61]M. Yu, S. Wang, G. Zhang, J. Mao, C. Yin, Q. Liu, K. Wang, Q. Wen, and Y. Wang, “NetSafe: Exploring the topological safety of multi-agent system,” in Findings of the Association for Computational Linguistics: ACL 2025, 2025.arxiv.org/abs/2410.15686
  47. [63]H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine et al., “Llama guard: Llm-based input-output safeguard for human-ai conversations,” arXiv preprint arXiv:2312.06674, 2023.arxiv.org/abs/2312.06674
  48. [64]S. Han, K. Rao, A. Ettinger, L. Jiang, B. Y. Lin, N. Lambert, Y. Choi, and N. Dzirri, “Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms,” Advances in neural information processing systems, vol. 37, pp. 8093–8131, 2024.
  49. [65]Z. Xiang, L. Zheng, Y. Li, J. Hong, Q. Li, H. Xie, J. Zhang, Z. Xiong, C. Xie, C. Yang, D. Song, and B. Li, “GuardAgent: Safeguard LLM agents by a guard agent via knowledge-enabled reasoning,” in International Conference on Machine Learning (ICML), 2025, arXiv:2406.09187.
  50. [66]Y. Zeng, Y. Wu, X. Zhang, H. Wang, and Q. Wu, “Autodefense: Multi-agent LLM defense against jailbreak attacks,” in NeurIPS Safe Generative AI Workshop 2024, 2024. [Online]. Available: https://openreview.net/forum?id=WMwoSLAENSarxiv.org/abs/2403.04783
  51. [68]M. Andriushchenko, A. Souly, M. Dziemian, D. Duenas, M. Lin, J. Wang, D. Hendrycks, A. Zou, Z. Kolter, M. Fredrikson et al., “Agentharm: A benchmark for measuring harmfulness of llm agents,” in International Conference on Learning Representations, vol. 2025, 2025, pp. 79 185–79 220.arxiv.org/abs/2410.09024
  52. [70]V. I. Naik, C. Xu, D. Dong, H. Hassan, A. Pradhan, O. Mendelevitch, T. Shafat, and H. Irshad, “Guardianagentbench: Where agents fail and how to guard them,” arXiv preprint arXiv:2607.20982, 2026.
  53. [71]L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, A. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,” in NeurIPS, 2022, arXiv:2203.02155.arxiv.org/abs/2203.02155
  54. [72]R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” in Advances in Neural Information Processing Systems (NeurIPS), 2023, arXiv:2305.18290.arxiv.org/abs/2305.18290
  55. [73]X. Q. Loye, Q. Su, Z. Zhang, S. Cui, Q. Zhu, F. Mi, H. Wang, and M. Huang, “Rubas: Rubric-based reinforcement learning for agent safety,” arXiv preprint arXiv:2606.04051, 2026.arxiv.org/abs/2606.04051
  56. [74]J. J. Zhang, H. Wang, E. M. Smith, S. Wang, A. Sharaf, M. Pasupuleti, B. Van Durme, D. Khashabi, J. E. Weston, and H. Zhan, “The alignment waltz: Jointly training agents to collaborate for safety,” in International Conference on Learning Representations, vol. 2026, 2026, pp. 92 751–92 776.
  57. [75]A. Arditi, O. B. Obeso, A. Syed, D. Paleka, N. Rimsky, W. Gurnee, and N. Nanda, “Refusal in language models is mediated by a single direction,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [Online]. Available: https://openreview.net/forum?id=pH3XAQME6c
  58. [76]M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang et al., “HarmBench: A standardized evaluation framework for automated red teaming and robust refusal,” in ICML, 2024.
  59. [77]A. Souly, Q. Lu, D. Bowen, T. Trinh, E. Hsieh, S. Pandey, P. Abbeel, J. Svegliato, S. Emmons, O. Watkins et al., “A strongreject for empty jailbreaks,” Advances in Neural Information Processing Systems, vol. 37, pp. 125 416–125 440, 2024.arxiv.org/abs/2402.10260
  60. [78]P. Chao, E. Debenedetti, A. Robey, M. Andriushchenko, F. Croce, V. Sehwag, E. Dobriban, N. Flammarion, G. J. Pappas, F. Tramer et al., “Jailbreakbench: An open robustness benchmark for jailbreaking large language models,” Advances in Neural Information Processing Systems, vol. 37, pp. 55 005–55 029, 2024.
  61. [80]F. Bénézit, A. G. Dimakis, P. Thiran, and M. Vetterli, “Order-optimal consensus through randomized path averaging,” IEEE Transactions on Information Theory, pp. 5150–5167, 2010.arxiv.org/abs/0802.2587
  62. [81]R. Thakur, R. Rabenseifner, and W. Gropp, “Optimization of collective communication operations in mpich,” The International Journal of High Performance Computing Applications, vol. 19, no. 1, pp. 49–66, 2005.DOI
  63. [82]Q. Team et al., “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025.
  64. [84]A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” arXiv preprint arXiv:2307.15043, 2023.arxiv.org/abs/2307.15043
  65. [85]Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang et al., “MMLU-Pro: A more robust and challenging multi-task language understanding benchmark,” in Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2024.arxiv.org/abs/2406.01574
  66. [86]J. Cui, W.-L. Chiang, I. Stoica, and C.-J. Hsieh, “OR-bench: An over-refusal benchmark for large language models,” in Forty-second International Conference on Machine Learning, 2025. [Online]. Available: https://openreview.net/forum?id=CdFnEuO JZVarxiv.org/abs/2405.20947
  67. [87]P. Röttger, H. R. Kirk, B. Vidgen, G. Attanasio, F. Bianchi, and D. Hovy, “XSTest: A test suite for identifying exaggerated safety behaviours in large language models,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2024.arxiv.org/abs/2308.01263
  68. [88]K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
  69. [89]D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, “Measuring mathematical problem solving with the MATH dataset,” in Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2021.
  70. [90]M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, and J. Wei, “Challenging BIG-Bench tasks and whether chain-of-thought can solve them,” in Findings of the Association for Computational Linguistics (ACL), 2023.arxiv.org/abs/2210.09261
  71. [91]D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman, “GPQA: A graduate-level google-proof q&a benchmark,” in Conference on Language Modeling (COLM), 2024.arxiv.org/abs/2311.12022
  72. [92]Z. Sprague, X. Ye, K. Bostrom, S. Chaudhuri, and G. Durrett, “MuSR: Testing the limits of chain-of-thought with multistep soft reasoning,” in International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.16049

Paper details

vlnc:W_caztsupg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

LLM-based coding assistants

View statement
Looking for related papers…
Contents