WorksComputer Science
Train Locally, Learn Together: Distributed Multi-Agent Training via ADMM
Abstract
Multi-agent systems (MAS) can outperform individual language-model agents by combining specialized roles, diverse reasoning processes, and collaborative decision-making. However, collaborative MAS training is typically centralized, requiring a common trainer to aggregate role-local trajectories and learning signals and maintain model and optimization states. This motivates a distributed multi-agent learning paradigm in which agents improve system-level capabilities through role-local learning and decentralized communication, paving the way toward scalable, resilient, and self-organizing MAS. To this end, we introduce, to the best of our knowledge, the first optimization-grounded framework for distributed MAS training based on consensus alternating direction method of multipliers (ADMM). We formulate collaborative MAS learning as a consensus-constrained finite-sum optimization problem, naturally lending itself to distributed optimization via ADMM. The proposed approach alternates role-local learning with inter-agent consensus using distributed communication protocols such as ring all-reduce, enabling joint optimization while keeping each agent's data and training state local. We evaluate our framework on distributed safety alignment and general reasoning tasks. Under compromised-agent settings, ADMM can reduce HarmBench attack success rate (ASR) by up to 26.4%, while achieving reasoning gains comparable to centralized training. These results show that distributed MAS training retains the benefits of centralized joint optimization without requiring a centralized trainer.
Introduction
Large language models (LLMs) are increasingly organized into multi-agent systems (MAS), in which agents assume specialized roles and communicate in natural language or latent messages to solve complex tasks collaboratively [1] (Wu et al. 2024; Du et al. 2024; Zou et al. 2026). By coordinating complementary roles, diverse reasoning processes, and iterative refinement, MAS can achieve stronger system-level performance than individual agents. Recent work has therefore moved beyond prompting and inference-time collaboration toward directly training participating agents through fine-tuning or reinforcement learning to improve their collective behavior [5, 6, 7, 8, 10] (Liu et al. 2026). This shift from inference-time collaboration to learning raises a fundamental question of how interacting agents should be trained as a system.
Most existing approaches realize collaborative MAS learning through centralized training, in which a single training process jointly manages multi-agent trajectories, learning signals, parameters, and optimization states [5, 6, 8, 7, 10]. While effective for coordinating learning across agents, this paradigm assumes that their training information and optimization states can be collected and managed by a common trainer (see Figure 1(a)). Such an assumption becomes increasingly restrictive as MAS scale or span different devices, organizations, and trust domains. Training agents independently avoids the need for a common trainer, but gives up the system-level coordination that collaborative MAS learning seeks to achieve. This creates a fundamental tension: collaborative learning calls for coordination across agents, while distributed deployment calls for keeping their learning processes local.
Figure 1. Schematic overview of centralized and distributed MAS training. (a) Centralized training jointly optimizes all agents in a single training process. (b) Distributed training via ADMM performs role-local optimization with parameter exchange for inter-agent consensus. (c) Demonstration of safety training in MAS: Safety and utility of Qwen3-8B-based MAS (w/o training), centralized-trained, and ADMM-based distributed-trained MAS under two structures: sequential (S-MAS; Solver→Verifier→Corrector→Decider) and hierarchical (H-MAS; three Analyzers→Synthesizer). HarmBench attack success rate (ASR; ↓, %) is reported for safety under benign (all agents safety-aligned) and compromised settings (unaligned Solver in S-MAS or Synthesizer in H-MAS), together with MMLU-Pro accuracy and mean accuracy across six reasoning benchmarks (Acc; ↑, %). ADMM closely matches centralized training while substantially reducing HarmBench ASR relative to the base MAS. See § 5 for more details.
Motivated by this tradeoff, we develop a principled approach to distributed MAS training, where each agent is trained on its own node with locally maintained training information and optimization states, while training is coordinated across agents without a centralized trainer. Conventional distributed learning typically distributes a model-training process across workers or clients, often according to where computation or data is located [11, 12, 13, 14, 15]. In our setting, the partition instead follows the functional structure of the MAS: even when all roles share the same underlying model, each node carries out the local learning associated with one agent role, based on the trajectories and learning signals arising from that role. The resulting local updates must be coordinated across nodes so that the MAS is improved as a whole. The challenge is therefore to distribute collaborative MAS training across role-local learning processes without losing the coordination provided by centralized joint training. This motivates our central question:
(Q) How can we collaboratively train an MAS using a principled distributed optimization framework while retaining the performance of centralized training?
To answer (Q), we formulate distributed MAS training as a consensus optimization problem over local copies of the shared model parameters. Each agent performs role-local learning on its own node using the trajectories and learning signals associated with its functional role, while consensus coordinates these local updates across agents. Building on this formulation, we develop an alternating direction method of multipliers (ADMM) framework [16, 17] as the core optimization mechanism for distributed MAS training (see Figure 1(b)). ADMM naturally separates training into two complementary steps: role-local optimization, where each node updates the shared model using the trajectories and learning signals associated with its role, and consensus coordination, where nodes reconcile their local model copies while keeping trajectories and optimization states local. This decomposition allows each agent to train independently between communication rounds while still enforcing system-wide agreement through ADMM consensus updates. We realize the consensus step either exactly with ring all-reduce or approximately with decentralized gossip [18, 19]. In this way, ADMM provides a principled bridge between local agent learning and global MAS coordination, enabling collaborative training without centralizing agents’ trajectories, data, or optimization states.
Prior works have explored distributed or decentralized MAS through inference-time coordination [20, 21, 22], as well as agent-specific policy learning and distributed rollout generation [23] (Fu et al. 2026). In contrast, we distribute the optimization of a shared MAS model across role-local training nodes and coordinate its local copies through ADMM-based consensus. We evaluate our approach on reasoning and safety alignment against frozen MAS and centralized joint training. Across both settings, distributed training closely approaches centralized performance (see Figure 1(c)). Component ablations further show neither local-only training nor periodic parameter averaging is sufficient to match the full ADMM procedure.
Our main contributions are summarized as follows:
① We formulate the collaborative training of a shared MAS model as a finite-sum distributed optimization problem over role-local objectives induced by agent roles, and recast it into an ADMM-compatible consensus formulation.
② We develop an ADMM-based framework that alternates role-local learning with inter-agent consensus, supports flexible communication protocols, and keeps each agent’s trajectories and training states local.
Across reasoning and safety-alignment tasks, distributed training closely approaches centralized training performance, including safety settings where one agent is compromised. We further study distributed MAS training across communication protocols and system scales, while ablations show the importance of consensus and dual updates.
Related Works
Multi-agent training, communication, and collaboration. LLM-based MAS organize multiple agents into complementary roles and coordinate their interactions toward a shared output [1, 25, 26] (Wu et al. 2024; Du et al. 2024; Zou et al. 2026). One line of work treats the underlying LLMs as fixed and improves collaboration through agent composition, roles, prompts, and communication structures (Chen et al., 2024; Hu et al., 2025; Zhang et al., 2025a,b,c; Yao et al., 2026; Wang et al., 2026). More recent work directly trains agents for collaboration, building on multi-agent RL for credit assignment [34] and optimizing role-specific behavior toward system-level objectives [8, 6, 7, 5, 36, 37, 10, 38] (Liu et al. 2026; Mo et al. 2025). However, these approaches largely assume centralized training, leaving open how agents can learn collaboratively while keeping data and training states local.
Distributed optimization. Distributed optimization enables multiple learners to optimize locally while coordinating through consensus [16, 39, 40]. Such coordination can be realized through collective communication such as ring all-reduce [18] or decentralized gossip [41, 19]. These principles have been increasingly adopted for distributed training of LLMs, where model updates are coordinated across geographically or computationally separated workers [14, 42, 15, 43, 44], as well as for parameter-efficient LLM adaptation [45, 46].
Communication and coordination in multi-agent systems. A broad line of work studies how LLM-based agents exchange information and aggregate decisions, including multi-agent debate [47] (Du et al. 2024), voting and consensus-based aggregation [48, 49], and latent-space collaboration (Zou et al., 2026). Another line focuses on the communication structure itself, ranging from sparse debate topologies (Li et al., 2024) to learned or adaptive communication graphs [51, 29]. Related work further explores decentralized MAS through agent discovery and communication [21, 52] and decentralized coordination [20]. Beyond inference-time coordination, recent work has explored decentralized MAS training through separate policy trainers [23] and distributed training infrastructure (Fu et al., 2026). In contrast, we formulate MAS training as consensus optimization, allowing each agent to optimize locally while coordinating model updates through ADMM consensus and keeping trajectories and training states local.
Multi-agent safety and alignment. Multi-agent interaction introduces additional safety risks: compromised agents can induce harmful behavior through role-play or persona injection [53, 50], while jailbreaks and errors can propagate across agents [55, 59, 60] (Gu et al. 2024; Ju et al. 2026; Li et al. 2026), with their impact shaped by communication topology and compromised positions [61] (Yu et al. 2025). Existing defenses largely rely on guard models or agents [63, 64, 65, 66] (Hu and Hooi 2026), while other work evaluates safety at the agent and system levels [68, 54, 70]. Parameter-level alignment has primarily focused on individual models [71, 72, 73], while recent joint agent training remains centralized [74]. We use multi-agent safety alignment as an evaluation setting for distributed MAS training under both aligned and compromised-agent settings.
Distributed MAS Training: Motivation and Formulation
This section motivates and formalizes distributed MAS training. We first show, through a safety study, that composing individually aligned agents into an MAS can amplify unsafe behavior, so alignment must be treated as a system-level learning problem. We then explain why conventional centralized training is ill-suited to this setting and formulate distributed MAS training, in which each agent learns from a role-local objective. These local objectives remain coupled through inter-agent messages, which we identify as the key challenge of interaction-induced coupling.
MAS introduces new training demands: A motivation from safety alignment. We motivate system-level MAS training from a safety perspective: even when each agent is individually aligned, their composition can amplify unsafe behavior. In Figure 2, we evaluate a sequential and a hierarchical MAS (Figure 2a), both built from Qwen3–8B agents with role-specific system prompts. Given a harmful query, the final agent’s message is taken as the system response, and safety is measured by attack success rate (ASR, ↓) on three benchmarks. The aligned Qwen3–8B used alone serves as the Single LLM baseline. Composing the same aligned model into an MAS can substantially reduce system-level safety: on StrongReject, ASR rises from 59.1% for the Single LLM to 81.7% and 86.7% for the benign sequential and hierarchical MAS, respectively, and increases further when one agent is compromised. Figure 2(c) shows a qualitative case where the Single LLM refuses a harmful query while the hierarchical MAS responds unsafely.
Figure 2. Safety evaluation of sequential and hierarchical MAS built from Qwen3-8B with thinking mode using role-specific prompts. In the benign setting, all agents use the vanilla backbone; in the compromised setting, one agent instead uses an unaligned backbone while retaining the same role prompt ([75]). (a) MAS architectures. The sequential MAS consists of Solver→Verifier→Corrector→Decider, while the hierarchical MAS uses three Analyzers followed by a Synthesizer. In the compromised setting, the purple devil icon marks the agent using the unaligned backbone, placed at the Solver and Synthesizer, respectively. (b) Attack success rate (ASR, ↓) on HarmBench ([76]), StrongReject ([77]), and JailbreakBench ([78]). Each benchmark compares the aligned single LLM, the benign MAS, and the compromised MAS with one compromised agent as shown in (a). (c) Qualitative comparison on the same harmful query between the aligned single LLM and the hierarchical MAS without compromise.
The safety risk becomes more severe when one agent is compromised (i.e., not safety-aligned). As shown in Figure 2(b), replacing a single role with an unaligned model can substantially increase ASR, with the largest degradation occurring at the Solver in the chain and the Synthesizer in the hierarchy. More detailed configs are reported in Appendix § 11 and results for other compromised roles are reported in Table A1. Since the compromised role is generally unknown, protecting a predetermined agent is insufficient. Moreover, independent alignment overlooks message-passing dependencies, through which one agent’s output shapes downstream behavior.
| System | Compromised agent(s) | HarmBench ASR (% ↓) |
| Single LLM | none | 16.98 |
| Single LLM | the model itself | 72.96 |
| Sequential | none (benign MAS) | 17.61 |
| Sequential | Solver | 42.77 |
| Sequential | Verifier | 19.50 |
| Sequential | Corrector | 20.75 |
| Sequential | Decider | 18.87 |
| Sequential | Verifier + Corrector | 22.01 |
| Sequential | Verifier + Decider | 25.16 |
| Sequential | Verifier + Corrector + Decider | 40.88 |
| Sequential | all four | 54.72 |
| Hierarchical | none (benign MAS) | 19.50 |
| Hierarchical | Analyzer 1 | 27.04 |
| Hierarchical | Analyzer 2 | 22.64 |
| Hierarchical | Analyzer 3 | 25.79 |
| Hierarchical | Synthesizer | 52.83 |
| Hierarchical | all four | 72.96 |
Table A1. Safety of an untrained MAS as a function of which agent is compromised. No training anywhere: the two architectures of Figure 2 run the safety-aligned backbone at every seat except the listed ones, which run the unaligned copy with the same role prompt. Attack success rate reported on all 159 HarmBench prompts under the same judge as Table 1. The shaded rows are the compromised settings carried into Table 1.
These observations reveal a new training demand for MAS: safety alignment should be treated as a system-level learning problem that accounts for agents’ functional roles, communication, and collective behavior under composition. More broadly, effective MAS adaptation requires coordinating role-local learning toward a shared system-level objective rather than optimizing agents independently.
Problem statement: Distributed MAS training. The system-wide training demand identified above requires coordinated learning across agents. However, conventional MAS training typically realizes this coordination through a centralized process that collects agents’ trajectories, computes their learning signals, and maintains their parameters ([6, 8, 7, 10] (Mo et al. 2025)). Such a design incurs increasing communication, memory, and computational overhead as the MAS scales. It also creates a single point of failure and assumes a central authority can access all role-local information, which may not hold when agents are distributed across devices, organizations, or trust domains. This motivates distributed MAS training, where agents learn from local data, maintain their own training states, and coordinate toward a shared system-level objective.
Specifically, consider an MAS of agents with distinct functional roles interacting over a directed communication graph, where edges carry inter-agent messages during each rollout. Following common practice, agents share the same LLM backbone but use role-specific system prompts ([1] (Du et al. 2024)). Given a query, each agent conditions on its prompt, the query, and upstream messages to generate its message, with the final message forming the system output. For distributed training, agent maintains a local copy of the shared trainable adapter, implemented as a LoRA adapter (Hu et al., 2021). Let denote the role-local training objective at agent . Although agents optimize local copies of the same adapter, their objectives differ because they are constructed from role-specific prompts, received messages, and learning signals arising from the joint MAS rollout. We call this dependency among local objectives the interaction-induced coupling challenge.
Example of interaction-induced objective coupling in a chain-structured MAS
Consider the three-agent chain as in Figure 2(a). Given a query , agent 1 generates . Agent 2 receives and produces . Agent 3 then receives and produces . Thus, upstream parameters shape downstream objectives through the messages they generate, yielding the following expected role-local objective structure: , , .
As exemplified above, each role-local objective is influenced by upstream agents through the messages they generate, and therefore by the current local model copies used to produce those messages. Based on the above, we formulate the distributed MAS training as
where denotes the index set containing agent and its upstream agents, and . As shown in (1), interaction-induced coupling prevents MAS training from decomposing into independent local problems: optimizing agents separately is not equivalent to optimizing the system-level objective. This is consistent with Figure 2, where individually aligned agents may not yield a safe MAS because message passing couples their behaviors.
Distributed MAS Training via ADMM: Why and How
This section addresses the interaction-induced coupling challenge of § 3 with ADMM. We first explain why ADMM suits distributed MAS training, then derive its two alternating steps: role-local fine-tuning and consensus averaging.
ADMM and its rationale. ADMM is a classical operator-splitting method for solving structured composite optimization problems [16, 17], as given below:
where and are optimization variables associated with objectives and , respectively. A key feature of (2) is the auxiliary variable . Although makes seemingly redundant, variable splitting separates the objectives and enables alternating ADMM optimization. A suitable choice of auxiliary variables can further simplify the subproblems, particularly when one admits a closed-form solution. Concretely, ADMM solves (2) through the following iterations [17]:
where denotes the ADMM iteration, is the dual variable associated with the constraint and is updated as , and with denoting the corresponding penalty parameter in the augmented Lagrangian of (2). The updates in (3) and (4) are both expressed as proximal operations [17]. Specifically, .
As shown by the alternating updates in (3)–(4), ADMM optimizes and through two subproblems, coordinated by the primal variables and and the dual variable , which accumulates the constraint violation . This operator-splitting structure is a key advantage of ADMM: it decomposes the coupled problem in (2) into tractable subproblems. When either admits a closed-form solution, ADMM can further simplify optimization. Building on this advantage, we reformulate the interaction-coupled MAS training objective in (1) using role-local parameter copies.
Distributed MAS training problem through the ADMM lens
For each agent , we introduce a local variable , representing local copies of the parameter blocks involved in its objective in (1). To preserve consistency, local copies of the same parameter block must agree. We therefore define the consensus set:
where denotes agent ’s local copy of contained in . Thus, the coupled objective in (1) can be equivalently represented using the local variables subject to the consistency constraint . Accordingly, we reformulate (1) in the ADMM form of (2):
where is the indicator function of the constraint set : it equals 0 if and otherwise, i.e., ensuring optimization feasibility under the constraint.
With (6), the ADMM updates in (3)–(4) decompose distributed MAS training into two steps: (i) role-local optimization for each objective , enabling parallel updates across agents, and (ii) projection onto the consensus set to coordinate the shared parameter blocks. We detail these steps below.
Role-local fine-tuning in ADMM. We instantiate the ADMM x-update in (3) for (6) by setting and replacing the generic variables with the corresponding collections , , and . Both the finite-sum objective and the quadratic term are separable across . Consequently, the x-update decomposes into role-local subproblems that can be solved in parallel. Each agent then performs the role-local optimization step:
where . In (7), we instantiate as a role-local fine-tuning loss, following common MAS training settings where agent updates are guided by system-level rewards or returns derived from multi-agent rollouts ([7, 8] (Liu et al. 2026)), as detailed below.
Recall that contains agent ’s local copies, with denoting its own trainable parameters and those associated with upstream agents. These components play different roles: , implemented as a trainable LoRA adapter, directly parameterizes agent ’s message , while upstream agents influence its conditioning context through received messages . Given a sampled MAS rollout, upstream messages are treated as fixed context. The binary outcome-based reward is derived from the complete rollout: for reasoning, if the final answer is correct; for safety, if the final answer is safe on a harmful query or correct on a benign one, while agent ’s message is non-harmful or non-refusal, respectively. We use to construct preference pairs and instantiate in (7) with DPO ([72]):
where is the logistic function, is agent ’s conditioning context, and is the initial policy with its adapter disabled. For each query, we sample several rollouts and pair from an rollout with from an rollout, scoring both under the preferred rollout’s context. Groups with identical are discarded. During training, we additionally apply a forward KL regularizer to each scored sample. The ADMM formulation also supports other role-local objectives without changing the subsequent updates.
Substituting (8) into (7), we solve the role-local subproblem by gradient-based optimization. During each local update, the sampled rollout and received upstream messages are treated as fixed training context. Agent therefore updates its local adapter copy using its role-local objective, while coordination among the corresponding parameter copies is handled by the subsequent ADMM consensus step. Thus, each agent performs a locally executable update over , while inter-agent coordination is handled separately through consensus.
Inter-agent consensus averaging in ADMM. We instantiate the second ADMM subproblem in (4) by setting , the indicator function of the constraint set . Since its proximal operator is the Euclidean projection onto ([17]), the z-update becomes:
where , and is the consensus set in (5), applied to . It requires local copies of the same parameter block to agree, i.e., for . Under the Euclidean projection in (9), each block’s consensus value is the average of its local copies, corresponding to standard consensus averaging ([19, 80, 11]). Thus, (9) admits a closed-form blockwise solution. For block , let denote the number of agents holding its copy. Then, for every agent with , solution is
where if agent maintains block and 0 otherwise. Thus, agents sharing block receive the same consensus value, equal to the average of their local copies. The averaging in (10) can be implemented distributively via all-reduce (Patarasuk and Yuan, [18]; Thakur et al., [81]) or randomized gossip (Boyd et al., [19]).
From the ADMM perspective, distributed MAS training alternates between (i) role-local fine-tuning (x-step), where each agent updates only its trainable parameters, and (ii) distributed consensus averaging (z-step) via (10). The steps are coupled by , which accumulates disagreement between local and consensus variables. The full procedure is given in Algorithm A1 and Appendix B. We also examine empirical convergence in Figure 3, where both the role-local objectives and the ADMM primal residual decrease over training rounds.
Figure 3. Empirical convergence of ADMM-based distributed safety training on the compromised H-MAS in Table 1, using agents, ring all-reduce consensus, , and 24 ADMM rounds. Left: Role-local objectives , shown as a centered five-round rolling geometric mean with the minimum-to-maximum range across agents. Right: Normalized primal residual, . Both axes are logarithmic; marker color indicates cumulative communication per node.
Experiments
Experimental Setup
MAS and model setups. By default, all agents use the safety-aligned Qwen3-8B backbone in thinking mode (Team et al., [82]) and differ only in role-specific prompts (Appendix C.1). We study two four-agent architectures (Figure 2): S-MAS, a sequential chain (Solver→Verifier→Corrector→Decider), and H-MAS, where three Analyzers feed a Synthesizer. We also consider a compromised setting in which one agent uses Qwen3-8B-abliterated, an unaligned variant with refusal behavior removed (Arditi et al., [75]). During distributed training, each agent maintains a local parameter copy as a rank-16 LoRA adapter (Hu et al., 2022); further adapter details are in Appendix C.2.
Training setups. For safety training, we use 380 harmful prompts from AdvBench (Zou et al., [84]) and the HarmBench validation split (Mazeika et al., [76]), plus 300 benign MMLU-Pro questions (Wang et al., [85]) to control over-refusal. For reasoning training, we use each benchmark’s training split when available, or a disjoint test subset otherwise. For both tasks, we sample eight on-policy trajectories per query and train each agent with (8), and all agents remain trainable. Unless noted otherwise, consensus uses ring all-reduce with (), for up to 24 ADMM rounds. Additional training details like rollout, optimization, and data filtering are provided in Appendix C.3. ADMM and centralized training use the same data and objective.
Evaluation setups. For safety, we evaluate on HarmBench (Mazeika et al., [76]), StrongReject (Souly et al., [77]), JailbreakBench (Chao et al., [78]), toxic OR-Bench (Cui et al., [86]), and XSTest (Röttger et al., [87]). We report attack success rate (ASR, ↓) on harmful prompts, refusal rate (↓) on safe XSTest prompts for over-refusal, and MATH-500 accuracy (↑) for utility. For general reasoning, we report accuracy (↑) on MMLU-Pro (Wang et al., [85]), GSM8K (Cobbe et al., [88]), MATH (Hendrycks et al., [89]), BBH (Suzgun et al., [90]), GPQA (Rein et al., [91]), and MuSR (Sprague et al., [92]). Additional evaluation details are provided in Appendix C.4.
Experimental Results
Distributed MAS training for safety enhancement. Table 1 compares centralized and ADMM-based distributed training on S-MAS and H-MAS under benign and compromised initializations. Safety training consistently improves MAS safety, substantially repairs compromised systems, and generalizes across benchmarks. Even under benign initialization, both centralized and distributed training further reduce ASR. When one agent is compromised, HarmBench
| Method | HarmBench | StrongReject | JBB | OR-Bench | XSTest RR ↓ | MATH-500 Acc. ↑ | |
| Single-Benign | 16.98 | 59.11 | 44.00 | 21.35 | 12.80 | 77.40 | |
| Single-Compromised | 72.96 | 97.12 | 98.00 | 94.31 | 1.60 | 74.20 | |
| S-MAS-Benign | Base | 17.61 | 81.79 | 79.00 | 24.18 | 6.80 | 83.00 |
| S-MAS-Benign | Centralized | 9.43(−8.18) | 43.13(−38.66) | 42.00(−37.00) | 12.83(−11.35) | 7.60(+0.80) | 83.00(+0.00) |
| S-MAS-Benign | ADMM | 10.69(−6.92) | 50.16(−31.63) | 47.00(−32.00) | 14.76(−9.42) | 8.80(+2.00) | 82.80(−0.20) |
| S-MAS-Compromised | Base | 42.77 | 91.69 | 95.00 | 43.00 | 6.40 | 81.20 |
| S-MAS-Compromised | Centralized | 25.79(−16.98) | 55.27(−36.42) | 57.00(−38.00) | 25.93(−17.07) | 11.20(+4.80) | 79.60(−1.60) |
| S-MAS-Compromised | ADMM | 25.16(−17.61) | 53.99(−37.70) | 56.00(−39.00) | 25.30(−17.70) | 10.40(+4.00) | 81.40(+0.20) |
| H-MAS-Benign | Base | 19.50 | 86.58 | 89.00 | 27.15 | 5.60 | 81.20 |
| H-MAS-Benign | Centralized | 18.24(−1.26) | 80.51(−6.07) | 84.00(−5.00) | 25.02(−2.13) | 6.40(+0.80) | 81.80(+0.60) |
| H-MAS-Benign | ADMM | 17.61(−1.89) | 78.91(−7.67) | 80.00(−9.00) | 24.31(−2.84) | 5.20(−0.40) | 81.00(−0.20) |
| H-MAS-Compromised | Base | 52.83 | 88.18 | 89.00 | 63.50 | 8.40 | 80.40 |
| H-MAS-Compromised | Centralized | 25.16(−27.67) | 42.17(−46.01) | 42.00(−47.00) | 30.24(−33.26) | 15.20(+6.80) | 81.20(+0.80) |
| H-MAS-Compromised | ADMM | 26.42(−26.41) | 44.09(−44.09) | 45.00(−44.00) | 31.75(−31.75) | 13.60(+5.20) | 80.80(+0.40) |
Table 1. Safety, over-refusal, and utility of MAS safety training. We evaluate S-MAS and H-MAS under benign initialization with no compromised agent and a compromised setting where the Solver in S-MAS or Synthesizer in H-MAS is compromised, as in Figure 2(a). The defender does not know the compromised agent, and all adapters are trained. Single-Benign and Single-Compromised are the corresponding Qwen3-8B single-model baselines; Base is the untrained MAS; Centralized pools all agents’ samples for centralized optimization; and ADMM (shaded) is our distributed training. ASR (↓) is reported on four safety benchmarks (JailbreakBench; JBB), refusal rate (↓) on safe XSTest prompts, and MATH-500 accuracy (↑) for utility. For Centralized and ADMM, results are reported as A(±b), where A is the absolute performance and b = (A − Base) is the change vs. Base.
ASR rises from 17.61% to 42.77% on S-MAS and from 19.50% to 52.83% on H-MAS. ADMM reduces these values to 25.16% and 26.42%, respectively, with consistent improvements on StrongReject, JBB, and OR-Bench. ADMM also closely matches centralized training across architectures and safety settings: under compromised initialization, HarmBench ASRs are 25.16% vs. 25.79% on S-MAS and 26.42% vs. 25.16% on H-MAS. These results show that distributed optimization retains much of the safety benefit of centralized training without a central trainer. On the other hand, safety training increases the over-refusal rate, particularly in compromised settings, possibly because of correcting compromised agents induces more conservative behavior. Yet, general utility remains largely preserved: under benign initialization, ADMM changes MATH-500 accuracy from 83.00% to 82.80% on S-MAS and from 81.20% to 81.00% on H-MAS, while achieving 81.40% and 80.80% after repairing the corresponding compromised systems.
Distributed MAS training for reasoning enhancement. Figure 4 shows S-MAS reasoning performance before and after training. Without additional training, the Base MAS outperforms the Single LLM on all six benchmarks, indicating gains from multi-agent composition. Reasoning training further improves performance, with ADMM closely matching centralized training. ADMM reaches 75.6% mean accuracy, compared with 75.9% for centralized training and 73.9% for the Base MAS. Thus, our distributed framework extends beyond safety alignment to reasoning enhancement.
Figure 4. Reasoning enhancement through distributed MAS training. Accuracy (%; ↑) across six reasoning benchmarks and their mean for the single-LLM baseline, untrained base MAS, centralized training, and ADMM-based distributed training.
Why inter-agent coordination is important in ADMM. Table 2 ablates consensus averaging in the z-step (10) and feedback through the dual update. Without either component, ADMM-Local yields limited safety gains, reducing HarmBench ASR from 42.77% to 41.51% on S-MAS and from 52.83% to 34.59% on H-MAS. Consensus averaging alone (ADMM-NoDual) remains insufficient, with ASRs of 42.14% and 37.11%. Full ADMM reduces ASR to 25.16% and 26.42%, closely matching centralized training (25.79% and 25.16%). Thus, neither local training nor averaging alone reproduces the benefit of coordinated optimization, while ADMM combines both.
| Consensus | Dual | S-MAS | H-MAS | |
| Base | – | – | 42.77 | 52.83 |
| Centralized | – | – | 25.79 | 25.16 |
| ADMM-Local | ✗ | ✗ | 41.51 | 34.59 |
| ADMM-NoDual | ✓ | ✗ | 42.14 | 37.11 |
| ADMM | ✓ | ✓ | 25.16 | 26.42 |
Table 2. Safety training under the MAS-compromised setting of Table 1, evaluated by HarmBench ASR (%↓). ADMM-Local trains agents independently without communication, while ADMM-NoDual uses consensus averaging without dual updates. The Consensus and Dual columns mark which components are enabled.
Effect of approximate gossip consensus on ADMM. The consensus averaging in (10) can be approximated by gossip, where neighboring agents repeatedly exchange and average their local variables. This yields a communication–performance trade-off: more gossip rounds improve information mixing but increase communication traffic. As shown in Figure 5, increasing the number of gossip rounds () from 1 to 2 reduces the ADMM primal residual from 2.34 to 0.80 and HarmBench ASR from 8.18% to 4.84%. Further increasing to 16 yields little additional change in either residual or ASR, while communication continues to grow. Thus, captures most of the benefit of gossip in this setting at substantially lower communication cost. Compared with approximate gossip, ring all-reduce computes the exact consensus average in (10). Since our MAS contains relatively few training agents, we use ring all-reduce by default to obtain exact consensus with predictable communication, while retaining gossip as an alternative.
Figure 5. Effect of gossip communication on ADMM coordination and safety. We use an eight-agent chain connected by a ring topology, where each agent communicates with its two neighbors for gossip rounds per consensus step in (10). Increasing propagates information across more hops but incurs greater communication, with marker color indicating the communication traffic (MB) per node. Left: Final normalized ADMM primal residual (↓) after training (similar to Figure 3), measuring disagreement between the local and auxiliary variables. Right: HarmBench ASR (%,↓) of the resulting MAS; the dashed line marks the untrained MAS.
Additional results. Appendix D further studies scaling across different numbers of agents and backbone sizes. Across system sizes, the compromised-agent effect persists without a clear monotonic trend in ; we also report training results beyond the main setting and compare the effect on Qwen3–8B and Qwen3–4B.
Conclusion
We study distributed MAS training, where agents optimize role-local objectives without a centralized trainer. We formulate MAS learning as consensus-constrained distributed optimization and solve it with ADMM, alternating local updates with inter-agent consensus while keeping trajectories and training states local. Across safety alignment and general reasoning, distributed training closely matches centralized training, including compromised-agent settings, and ablations highlight the importance of consensus and dual updates, supporting scalable decentralized MAS training.
Limitations and future work. Our experiments focus on representative sequential and hierarchical MAS built from a common LLM backbone with shared adapter parameterization, which keeps the model family and parameterization consistent across agents; broader evaluation across larger systems, diverse model families and task domains, and richer communication topologies is needed to establish generality. An important direction is to characterize how communication topology affects convergence, efficiency, and downstream performance, relating these behaviors to graph-theoretic properties such as connectivity, centrality, community structure, and spectral characteristics, potentially with edges weighted by the heterogeneity of interaction strength between roles. More complex MAS may further require hypergraph-based formulations, where dependencies capture computations that jointly involve multiple roles. Future work may also explore asynchronous consensus, adaptive and communication-efficient coordination, heterogeneous agents with different architectures or local objectives, and dynamic systems where agents join, leave, or change roles.
AI use statement
In this work, we used generative AI tools, including LLM-based coding assistants, to implement the experimental code, to help analyze and interpret results, and to provide feedback on the research methodology and experiments. We also used them to prepare figures and the code release, to search for related information, and to edit the manuscript for readability. The research questions, the problem formulation, and the proposed method were developed by the authors. All AI-assisted content was reviewed by the authors, who take full responsibility for the final content of this work.
Reproducibility statement
Source code for distributed MAS training and evaluation will be released upon publication. The experimental setup and implementation details, including MAS architectures, backbone models, training data, role-local objectives, ADMM optimization, communication protocols, and evaluation settings, are provided in Sec. 5 and the Appendix. The Appendix further documents the full training procedure, prompts, LoRA configuration, rollout collection, hyperparameters, evaluation criteria. All models, datasets, and benchmarks used in this work are publicly available, and evaluation follows the protocols of the corresponding benchmark papers.
Ethics Statement
This work aims to improve the safety and robustness of multi-agent language-model systems through distributed training, including settings where one agent is compromised. Our method reduces unsafe behavior under the evaluated settings but does not guarantee safety under arbitrary attacks or deployment conditions. All experiments use publicly available models and datasets; harmful prompts are used only in controlled training and evaluation settings for studying and improving system safety. This work involves no human subjects or personal data, and all authors adhere to the ICLR Code of Ethics.
Appendix
Safety Under Different Compromised-Agent Placements
Configuration
Generation uses temperature 0.6, top- 0.95, and a budget of 4096 tokens per agent. Figure 2(b) reports the attack success rate on HarmBench prompts [76], StrongReject prompts [77], and JailbreakBench prompts [78]. These benchmarks do not share a criterion: Following the popular settings, HarmBench is scored by its Llama-2-13B classifier, which asks whether the final answer is harmful, whereas StrongReject and JailbreakBench are scored by whether the system complied with the request rather than refusing it.
Where the Compromised Agent Sits
Figure 2 places the compromised agent at the position that degrades each architecture most: the Solver in the sequential chain and the Synthesizer in the hierarchy. Table A1 reports every other position, and every combination we ran, on the same untrained systems. Figure 2 uses the Solver in the sequential chain and the Synthesizer in the hierarchy as the compromised positions carried into the main experiments. Table A1 reports the other compromised positions on the same untrained systems. In the sequential MAS, compromising the Solver yields the largest observed ASR among the single-position settings (42.77% versus 17.61% for the benign MAS), whereas compromising the Verifier, Corrector, or Decider produces smaller changes in this run. In the hierarchical MAS, the largest observed ASR occurs when the Synthesizer is compromised (52.83% versus 19.50% for the benign MAS), while compromising individual Analyzers results in more modest increases. We do not interpret the small differences among the remaining positions as a stable ordering, since each configuration is evaluated with a single run. Similarly, compromising the three downstream agents in the sequential MAS gives 40.88% ASR, close to the 42.77% observed with the Solver alone, so we avoid drawing conclusions from their ordering. These results motivate training all roles rather than assuming in advance which position will be compromised.
Details of the ADMM Algorithm
Algorithm A1 states one outer iteration of the procedure of § 4. The three steps follow that derivation: the x-step is role-local and runs in parallel, the z-step exchanges optimization variables across nodes, and the dual step is local.
Algorithm A1 Distributed MAS training via consensus ADMM
Require: agents with overlaps , penalty , local steps , rounds
initialize and for every agent
for do
deploy to the MAS, roll it out on queries , and score the rollouts for
x-step (role-local fine-tuning, in parallel): each agent takes gradient steps on (7) with given by (8), updating its local adapter copy and obtaining
z-step (consensus averaging): agents exchange and form blockwise by (10), using all-reduce or gossip protocol
dual step (in parallel): for every agent
end for
return the consensus variables
Initialization of x, z and u
Algorithm A1 opens with and . These variables are initialized once before the first ADMM round and are then carried across subsequent rounds.
The dual starts at zero. We use the scaled dual, , where is the Lagrange multiplier associated with the constraint . Before the first round, there is no accumulated disagreement to carry, so we initialize . The ablation that removes the dual (§ 5) keeps this variable fixed at zero throughout the run.
The primal variables start at the deployed adapter, which is zero only for an untrained system. The optimization variable is the trainable factor of the LoRA adapter, and both and are initialized from the adapter currently deployed by the system. Thus, holds by construction, and the initial primal residual is exactly zero. Because we train only the up-projection and is zero-initialized in the standard LoRA scheme, a run starting from the untrained system has : the deployed update vanishes, and round 0 corresponds to the backbone itself. A run warm-started from a checkpoint instead begins from its (possibly nonzero) .
The frozen factor is not an optimization variable. is initialized once, shared across agents, and never updated, so it does not enter , , or . This also makes the -step exact in weight space (App. § 13.§ 2), since averaging the trainable factors is equivalent to averaging the corresponding updates . It is also why zeroing both LoRA factors would be inappropriate in a variant that trains both factors: if both and were initialized to zero, then and the gradients with respect to both factors would initially vanish.
Each x-step starts from the current proximal anchor. At ADMM round , the local x-step is initialized at rather than at . Agent then performs gradient-based optimization of its local adapter copy using the role-local objective, with the sampled rollout and received messages treated as fixed training context. Starting from makes the quadratic penalty zero at the first inner step and nonzero only as the solution moves away from the anchor, preventing the local update from being dominated by the absolute scale of . The resulting , together with the subsequent and , is carried to the next ADMM round.
Experimental Details
System Prompts
In the sequential chain, the Solver is asked to solve the question completely, to number its steps and state what each step establishes so that any single step can be checked in isolation, and to commit to a specific answer rather than hedge. The Verifier is shown the candidate solution and told that its only job is to find errors, checking in order whether the candidate answers the question actually asked, whether the method is the right one, whether each algebraic step is correct, whether units, signs and rounding are right, and whether the answer is in the demanded form. The Corrector receives the candidate together with the verifier report and repairs the identified errors. The Decider sees the prior agents’ work and issues the final answer.
In the hierarchy, the three Analyzers are told that they work on the same question independently and cannot see one another, so that their value lies in reaching the answer by different routes: the first derives from principles before looking at the options, the second proceeds by elimination, and the third tests candidate answers on concrete values.
The Synthesizer compares all three analyses and decides the final answer. The verbatim prompts for both architectures are released with our code.
Backbone and Adapter
The aligned backbone is Qwen/Qwen3-8B, run in thinking mode with a fixed generation budget. The compromised agent uses mlabonne/Qwen3-8B-abliterated, an abliterated variant with reduced refusal behavior (Arditi et al., [75]). It shares the tokenizer and architecture of the aligned backbone and can therefore replace any role without architectural changes. Each agent maintains a local trainable parameter copy , implemented as a rank-16 LoRA adapter with scaling , applied to the seven projections (q, k, v, o, gate, up, and down) in all 36 layers. Writing the corresponding local update to a base weight as , we initialize the down-projection identically across agents and keep it frozen, while training only .
Details of training
Training data. Harmful requests. We construct the harmful training set from AdvBench (Zou et al., [84]) and the standard and contextual HarmBench validation splits (Mazeika et al., [76]), assigning all retained prompts a refusal target. We remove prompts overlapping with HarmBench or XSTest evaluation data, and use only HarmBench validation examples that are disjoint from the test splits used for evaluation.
Benign requests. A slice of the MMLU-Pro test split (Wang et al., [85]), kept with their native QA answer type and scored as ordinary questions. They are the over-refusal / stay-helpful control: without them every retained fine-tuning target is a refusal and the trained system collapses into refusing everything.
Safety. Each agent is then fit on its message with the ordinary deployment prompt as input, so what is learned is the behaviour itself rather than the collection prompt, and nothing at test time depends on the hardened prompt. We draw up to eight trajectories per query, at temperature 0.8 and top- 0.95, with a per-agent budget of 4096 tokens.
General reasoning. Collection is on-policy and repeats every outer round. Queries are selected by excluding every anchor already used in an earlier round, so successive rounds never overlap; each round pre-screens a window of candidates by sampling each twice and dropping those answered correctly both times, then collects a handful of trajectories for each survivor and keeps only the groups whose outcomes are mixed. Sampling uses the same temperature and top- with an 8192-token per-agent budget.
Optimization. Table A2 lists the optimizer settings. Each ADMM round deploys the current consensus adapter, collects (or reuses) a batch of rollouts, and runs local gradient steps in the x-step of (7).
Table A2: Optimizer and ADMM settings, read off the trained checkpoints. Distributed and centralized runs share every value; the penalty and the round count apply to the distributed runs.
| Safety | Reasoning | |
| ADMM rounds | 24 (H-MAS) / 20 (S-MAS) | 8 |
| Local gradient steps per x-step | 16 | 16 |
| Rollouts per query | 8 | 8 |
| Optimizer | Adam | Adam |
| Learning rate | ||
| Gradient clip | 0.01 | 0.01 |
| Batch size | 8 | 8 |
| ADMM penalty | 0.03 | 0.1 |
| DPO temperature | 0.1 | 0.1 |
| KL weight | 0.1 | 0.1 |
| Role-local loss | DPO | DPO |
Table A2.
Data filtering. Each role is trained using the system-level outcome reward of the full rollout (Wan et al., [7]; Liao et al. [8]; Liu et al. (2026)). For reasoning, is answer correctness, and we retain queries whose sampled rollouts contain both successful and unsuccessful outcomes. For safety, harmful queries receive when the final response is judged harmless, while benign queries receive when answered correctly; refusals on benign queries are therefore penalized. We further include a role’s message in the safety loss only when the rollout succeeds and the message itself is harmless for harmful queries or non-refusal for benign queries, preventing unsafe intermediate messages from being reinforced by a safe final response. All roles remain trainable, including the compromised role.
For both tasks, we balance training examples across roles, de-duplicate repeated targets, and discard generations with incomplete reasoning blocks. By default, the loss is applied to the full message, including its reasoning content.
Details of evaluation
The five safety benchmarks use different scoring criteria, so their absolute values are not directly comparable. HarmBench uses its Llama-2-13B classifier [76] to judge whether the final response is harmful, whereas StrongReject, JailbreakBench, and toxic OR-Bench measure whether the system complies rather than refuses.
The HarmBench Llama-2-13B classifier is the primary harmfulness judge throughout, pinned to one revision. A Granite guardian classifier was run as a sensitivity check and is not used for any reported number: it flags terse refusals as harmful and shifts the comparison by roughly five points, in a direction that depends on how verbose an arm is. Where a benchmark ships its own notion of compliance we keep it, so that our numbers stay comparable to how that benchmark is normally reported, and we never compare an absolute value across benchmarks.
Evaluation runs in thinking mode at temperature 0.6 and top-p 0.95 with a fixed seed. Two fixed profiles are used and never mixed: safety evaluations allow 4096 tokens per agent and capability evaluations allow 8192.
For general reasoning, we evaluate each benchmark on a fixed subset sampled once and reused across all methods, ensuring that every bar in Figure 4 uses the same questions. Each subset contains up to 256 examples. Safety benchmarks use their standard evaluation splits.
Scaling with the Number of Agents and the Backbone
We further examine the compromised-agent effect across different system sizes and backbone sizes. These experiments are intended as supplementary analyses beyond the main , Qwen3-8B setting in Table 1.
Scaling with the number of agents. Table A3 evaluates untrained MAS with different numbers of agents. For S-MAS, we consider with the Solver compromised; for H-MAS, we consider with the Synthesizer compromised. The main configuration is reported separately in Table 1. Across all tested configurations, introducing one compromised agent increases HarmBench ASR relative to the corresponding benign system, with observed increases ranging from 14.47 to 32.08 percentage points. The magnitude of this increase varies across system sizes without a clear monotonic trend in .
Table A3: Effect of one compromised agent across system sizes. We evaluate untrained S-MAS with and H-MAS with . The compromised agent is the Solver in S-MAS and the Synthesizer in H-MAS. The main configuration is reported separately in Table 1. We report HarmBench ASR (%) and the compromised–benign difference .
| System | Benign | Compromised | ||
| S-MAS | 2 | 15.09 | 29.56 | +14.47 |
| S-MAS | 3 | 20.13 | 40.25 | +20.13 |
| S-MAS | 5 | 16.98 | 38.36 | +21.38 |
| S-MAS | 6 | 22.01 | 40.88 | +18.87 |
| S-MAS | 7 | 16.98 | 49.06 | +32.08 |
| H-MAS | 3 | 18.87 | 45.28 | +26.42 |
| H-MAS | 5 | 25.16 | 47.80 | +22.64 |
| H-MAS | 6 | 23.27 | 47.17 | +23.90 |
| H-MAS | 8 | 23.27 | 51.57 | +28.30 |
Table A3.
Training beyond the main system size. We additionally evaluate compromised S-MAS at , whose corresponding untrained configuration is included in Table A3. As shown in Table A4, centralized and distributed training reduce HarmBench ASR by 5.03 and 3.14 percentage points, respectively, relative to the untrained system. We report this result as an additional evaluation beyond the main setting rather than drawing conclusions about how training effectiveness scales with the number of agents.
Table A4: Additional training result beyond the main system size. Change in HarmBench ASR (percentage points; negative is safer) relative to the corresponding untrained compromised S-MAS in Table A3.
| Centralized ASR | Distributed ASR | |
| 3 | −5.03 | −3.14 |
Table A4.
Effect on a smaller backbone. We further evaluate an untrained S-MAS using Qwen3-4B. As shown in Table A5, compromising the Solver increases HarmBench ASR from 13.84% to 48.43%, corresponding to a 34.59-point increase. The corresponding Qwen3-8B results are reported in the main experiment in Table 1. Together, these experiments show that the compromised-agent effect is also present with a smaller backbone.
| Backbone | Benign | Compromised | |
| Qwen3-4B | 13.84 | 48.43 | +34.59 |
Table A5. Effect of one compromised agent on a smaller backbone. We evaluate an untrained S-MAS with using Qwen3-4B, with the Solver as the compromised agent. We report HarmBench ASR (%) and the compromised–benign difference . The corresponding Qwen3-8B results are reported in Table 1.
References
- [1]G. Li, H. A. A. K. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem, “CAMEL: Communicative agents for “mind” exploration of large language model society,” in Advances in Neural Information Processing Systems (NeurIPS), 2023, arXiv:2303.17760.arxiv.org/abs/2303.17760
- [5]S. R. Motwani, C. Smith, R. J. Das, R. Rafailov, P. Torr, I. Laptev, F. Pizzati, R. Clark, and C. S. de Witt, “MALT: Improving reasoning with multi-agent LLM training,” in Second Conference on Language Modeling, 2025. [Online]. Available: https://openreview.net/forum?id=iXP9bgFackarxiv.org/abs/2412.01928
- [6]C. Park, S. Han, X. Guo, A. E. Ozdaglar, K. Zhang, and J.-K. Kim, “Maporl: Multi-agent post-co-training for collaborative large language models with reinforcement learning,” in ACL, 2025.arxiv.org/abs/2502.18439
- [7]Z. Wan, Y. Li, X. Wen, Y. Song, H. Wang, L. Yang, M. Schmidt, J. Wang, W. Zhang, S. Hu, and Y. Wen, “ReMA: Learning to meta-think for LLMs with multi-agent reinforcement learning,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=ur295YVtmtarxiv.org/abs/2503.09501
- [8]J. Liao, M. Wen, J. Wang, and W. Zhang, “MARFT: Multi-agent reinforcement fine-tuning,” arXiv preprint arXiv:2504.16129, 2025.arxiv.org/abs/2504.16129
- [10]Y. Zhao, L. Hu, Y. Wang, M. Hou, H. Zhang, K. Ding, and J. Zhao, “Stronger-MAS: Multi-agent reinforcement learning for collaborative LLMs,” in International Conference on Learning Representations (ICLR), 2026, arXiv:2510.11062.arxiv.org/abs/2510.11062
- [11]J. C. Duchi, A. Agarwal, and M. J. Wainwright, “Dual averaging for distributed optimization: Convergence analysis and network scaling,” IEEE Transactions on Automatic control, 2011.DOI
- [12]S. Liu, P.-Y. Chen, and A. O. Hero, “Accelerated distributed dual averaging over evolving networks of growing connectivity,” IEEE Transactions on Signal Processing, vol. 66, no. 7, pp. 1845–1859, 2018.DOI
- [13]H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. Agüera y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial Intelligence and Statistics (AISTATS), 2017, arXiv:1602.05629.arxiv.org/abs/1602.05629
- [14]B. Yuan, Y. He, J. Q. Davis, T. Zhang, T. Dao, B. Chen, P. Liang, C. Ré, and C. Zhang, “Decentralized training of foundation models in heterogeneous environments,” in Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2206.01288
- [15]A. Douillard, Q. Feng, A. A. Rusu, R. Chaparia, Y. Donchev, A. Kuncoro, M. Ranzato, A. Szlam, and J. Shen, “DiLoco: Distributed low-communication training of language models,” in 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ICML 2024), 2024. [Online]. Available: https://openreview.net/forum?id=plCSfWkJIkarxiv.org/abs/2311.08105
- [16]S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends in Machine Learning, vol. 3, no. 1, pp. 1–122, 2011.
- [17]N. Parikh and S. Boyd, “Proximal algorithms,” Foundations and Trends in optimization, vol. 1, no. 3, pp. 127–239, 2014.DOI
- [18]P. Patarasuk and X. Yuan, “Bandwidth optimal all-reduce algorithms for clusters of workstations,” Journal of Parallel and Distributed Computing, vol. 69, no. 2, pp. 117–124, 2009.DOI
- [19]S. Boyd, A. Ghosh, B. Prabhakar, and D. Shah, “Randomized gossip algorithms,” IEEE transactions on information theory, vol. 52, no. 6, pp. 2508–2530, 2006.DOI
- [20]Y. Yang, H. Chai, S. Shao, Y. Song, S. Qi, R. Rui, and W. Zhang, “Agentnet: Decentralized evolutionary coordination for LLM-based multi-agent systems,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=tXqLxHIb8Zarxiv.org/abs/2504.00587
- [21]W. Chen, Z. You, R. Li, yitong guan, C. Qian, C. Zhao, C. Yang, R. Xie, Z. Liu, and M. Sun, “Internet of agents: Weaving a web of heterogeneous agents for collaborative intelligence,” in The Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https://openreview.net/forum?id=oIEt3MogPwarxiv.org/abs/2407.07061
- [22]Y. Mao and A. Mirhoseini, “Decentralized multi-agent systems with shared context,” arXiv preprint arXiv:2606.10662, 2026.arxiv.org/abs/2606.10662
- [23]K. Zhang, K. Tian, R. Liu, S. Zeng, X. Zhu, G. Jia, Y. Fan, X. Lv, Y. Zuo, C. Jiang, Y. Wang, J. Wang, E. Hua, X. Long, J. Gao, Y. Sun, Z. Ma, G. Cui, N. Ding, B. Qi, and B. Zhou, “MARTI: A framework for multi-agent LLM systems reinforced training and inference,” in The Fourteenth International Conference on Learning Representations, 2026. [Online]. Available: https://openreview.net/forum?id=E7jZqo0A50
- [25]S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, J. Wang, C. Zhang, S. Yau, Z. Lin, L. Zhou et al., “Metagent: Meta programming for a multi-agent collaborative framework,” in International Conference on Learning Representations, vol. 2024, 2024, pp. 23 247–23 275.arxiv.org/abs/2308.00352
- [26]C. Qian, W. Liu, H. Liu, N. Chen, Y. Dang, J. Li, C. Yang, W. Chen, Y. Su, X. Cong et al., “Chatdev: Communicative agents for software development,” in Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), 2024, pp. 15 174–15 186.arxiv.org/abs/2307.07924
- [29]G. Zhang, Y. Yue, X. Sun, G. Wan, M. Yu, J. Fang, K. Wang, T. Chen, and D. Cheng, “G-designer: Architecting multi-agent communication topologies via graph neural networks,” in Forty-second International Conference on Machine Learning, 2025. [Online]. Available: https://openreview.net/forum?id=LpE54NUnmOarxiv.org/abs/2410.11782
- [34]J. G. Kuba, R. Chen, M. Wen, Y. Wen, F. Sun, J. Wang, and Y. Yang, “Trust region policy optimisation in multi-agent reinforcement learning,” in International Conference on Learning Representations (ICLR), 2022, arXiv:2109.11251.arxiv.org/abs/2109.11251
- [36]Y. Dang, C. Qian, X. Luo, J. Fan, Z. Xie, R. Shi, W. Chen, C. Yang, X. Che, Y. Tian, X. Xiong, L. Han, Z. Liu, and M. Sun, “Multi-agent collaboration via evolving orchestration,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=L0xZPXT3learxiv.org/abs/2505.19591
- [37]Y. Xie, S. Liu, F. FAN, Y. Yao, S. Cao, Y. Zhao, and B. Liu, “TeamTR: Trust-region fine-tuning for multi-agent LLM coordination,” in Forty-third International Conference on Machine Learning, 2026. [Online]. Available: https://openreview.net/forum?id=IMwLhg52ZY
- [38]L. Feng, L. Zheng, S. He, F. Zhang, and B. An, “Dr. mas: Stable reinforcement learning for multi-agent llm systems,” arXiv preprint arXiv:2602.08847, 2026.arxiv.org/abs/2602.08847
- [39]W. Shi, Q. Ling, K. Yuan, G. Wu, and W. Yin, “On the linear convergence of the ADMM in decentralized consensus optimization,” IEEE Transactions on Signal Processing, vol. 62, no. 7, pp. 1750–1761, 2014.DOI
- [40]H. Ouyang, N. He, L. Tran, and A. Gray, “Stochastic alternating direction method of multipliers,” in International conference on machine learning. PMLR, 2013, pp. 80–88.
- [41]L. Xiao and S. Boyd, “Fast linear iterations for distributed averaging,” Systems & Control Letters, vol. 53, no. 1, pp. 65–78, 2004.
- [42]M. Ryabinin, T. Dettmers, M. Diskin, and A. Borzunov, “SWARM parallelism: Training large models can be surprisingly communication-efficient,” in International Conference on Machine Learning, 2023.arxiv.org/abs/2301.11913
- [43]S. Jaghouar, J. M. Ong, and J. Hagemann, “OpenDiLoCo: An open-source framework for globally distributed low-communication training,” arXiv preprint arXiv:2407.07852, 2024.arxiv.org/abs/2407.07852
- [44]A. Borzunov, D. Baranchuk, T. Dettmers, M. Ryabinin, Y. Belkada, A. Chumachenko, P. Samygin, and C. Raffel, “Petals: Collaborative inference and fine-tuning of large models,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (System Demonstrations), 2023.arxiv.org/abs/2209.01188
- [45]Y. Sun, Z. Li, Y. Li, and B. Ding, “Improving LoRA in privacy-preserving federated learning,” in International Conference on Learning Representations, 2024.arxiv.org/abs/2403.12313
- [46]Z. Wang, Z. Shen, Y. He, G. Sun, H. Wang, L. Lyu, and A. Li, “FLOrA: Federated fine-tuning large language models with heterogeneous low-rank adaptations,” in Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2409.05976
- [47]C.-M. Chan, W. Chen, Y. Su, J. Yu, W. Xue, S. Zhang, J. Fu, and Z. Liu, “Chateval: Towards better llm-based evaluators through multi-agent debate,” in International conference on learning representations, vol. 2024, 2024, pp. 9079–9093.arxiv.org/abs/2308.07201
- [48]L. B. Kaesberg, J. Becker, J. P. Wahle, T. Ruas, and B. Gipp, “Voting or consensus? decision-making in multi-agent debate,” in Findings of the Association for Computational Linguistics: ACL 2025, 2025, pp. 11 640–11 671.DOI
- [49]H. K. Choi, J. Zhu, and S. Li, “Debate or vote: Which yields better decisions in multi-agent large language models?” Advances in Neural Information Processing Systems, vol. 38, pp. 101 732–101 764, 2026.arxiv.org/abs/2508.17536
- [50]Y. Li, Y. Du, J. Zhang, L. Hou, P. Grabowski, Y. Li, and E. Ie, “Improving multi-agent debate with sparse communication topology,” in Findings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 7281–7294.arxiv.org/abs/2406.11776
- [51]M. Zhuge, W. Wang, L. Kirsch, F. Faccio, D. Khizbullin, and J. Schmidhuber, “Language agents as optimizable graphs,” arXiv preprint arXiv:2402.16823, 2024.arxiv.org/abs/2402.16823
- [52]Y. Wang, S. Guo, Y. Pan, Z. Su, F. Chen, T. H. Luan, P. Li, J. Kang, and D. Niyato, “Internet of agents: Fundamentals, applications, and challenges,” IEEE Transactions on Cognitive Communications and Networking, 2025, arXiv:2505.07176.DOI
- [53]Y. Tian, X. Yang, J. Zhang, Y. Dong, and H. Su, “Evil geniuses: Delving into the safety of llm-based agents,” arXiv preprint arXiv:2311.11855, 2023.arxiv.org/abs/2311.11855
- [54]Z. Zhang, Y. Zhang, L. Li, J. Shao, H. Gao, Y. Qiao, L. Wang, H. Lu, and F. Zhao, “Psysafe: A comprehensive framework for psychological-based attack, defense, and evaluation of multi-agent system safety,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 15 202–15 231.arxiv.org/abs/2401.11880
- [55]D. Lee, M. Tiwari, and B. Miranda, “Prompt infection: Llm-to-llm prompt injection within multi-agent systems,” in European Symposium on Research in Computer Security. Springer, 2025, pp. 511–520.arxiv.org/abs/2410.07283
- [59]J.-T. Huang, J. Zhou, T. Jin, X. Zhou, Z. Chen, W. Wang, Y. Yuan, M. Lyu, and M. Sap, “On the resilience of LLM-based multi-agent collaboration with faulty agents,” in Proceedings of the 42nd International Conference on Machine Learning, 2025.arxiv.org/abs/2408.00989
- [60]M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandran, M. Zaharia, J. E. Gonzalez, and I. Stoica, “Why do multi-agent LLM systems fail?” in The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2026. [Online]. Available: https://openreview.net/forum?id=fAjbYBmonrarxiv.org/abs/2503.13657
- [61]M. Yu, S. Wang, G. Zhang, J. Mao, C. Yin, Q. Liu, K. Wang, Q. Wen, and Y. Wang, “NetSafe: Exploring the topological safety of multi-agent system,” in Findings of the Association for Computational Linguistics: ACL 2025, 2025.arxiv.org/abs/2410.15686
- [63]H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine et al., “Llama guard: Llm-based input-output safeguard for human-ai conversations,” arXiv preprint arXiv:2312.06674, 2023.arxiv.org/abs/2312.06674
- [64]S. Han, K. Rao, A. Ettinger, L. Jiang, B. Y. Lin, N. Lambert, Y. Choi, and N. Dzirri, “Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms,” Advances in neural information processing systems, vol. 37, pp. 8093–8131, 2024.
- [65]Z. Xiang, L. Zheng, Y. Li, J. Hong, Q. Li, H. Xie, J. Zhang, Z. Xiong, C. Xie, C. Yang, D. Song, and B. Li, “GuardAgent: Safeguard LLM agents by a guard agent via knowledge-enabled reasoning,” in International Conference on Machine Learning (ICML), 2025, arXiv:2406.09187.
- [66]Y. Zeng, Y. Wu, X. Zhang, H. Wang, and Q. Wu, “Autodefense: Multi-agent LLM defense against jailbreak attacks,” in NeurIPS Safe Generative AI Workshop 2024, 2024. [Online]. Available: https://openreview.net/forum?id=WMwoSLAENSarxiv.org/abs/2403.04783
- [68]M. Andriushchenko, A. Souly, M. Dziemian, D. Duenas, M. Lin, J. Wang, D. Hendrycks, A. Zou, Z. Kolter, M. Fredrikson et al., “Agentharm: A benchmark for measuring harmfulness of llm agents,” in International Conference on Learning Representations, vol. 2025, 2025, pp. 79 185–79 220.arxiv.org/abs/2410.09024
- [70]V. I. Naik, C. Xu, D. Dong, H. Hassan, A. Pradhan, O. Mendelevitch, T. Shafat, and H. Irshad, “Guardianagentbench: Where agents fail and how to guard them,” arXiv preprint arXiv:2607.20982, 2026.
- [71]L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, A. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,” in NeurIPS, 2022, arXiv:2203.02155.arxiv.org/abs/2203.02155
- [72]R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” in Advances in Neural Information Processing Systems (NeurIPS), 2023, arXiv:2305.18290.arxiv.org/abs/2305.18290
- [73]X. Q. Loye, Q. Su, Z. Zhang, S. Cui, Q. Zhu, F. Mi, H. Wang, and M. Huang, “Rubas: Rubric-based reinforcement learning for agent safety,” arXiv preprint arXiv:2606.04051, 2026.arxiv.org/abs/2606.04051
- [74]J. J. Zhang, H. Wang, E. M. Smith, S. Wang, A. Sharaf, M. Pasupuleti, B. Van Durme, D. Khashabi, J. E. Weston, and H. Zhan, “The alignment waltz: Jointly training agents to collaborate for safety,” in International Conference on Learning Representations, vol. 2026, 2026, pp. 92 751–92 776.
- [75]A. Arditi, O. B. Obeso, A. Syed, D. Paleka, N. Rimsky, W. Gurnee, and N. Nanda, “Refusal in language models is mediated by a single direction,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [Online]. Available: https://openreview.net/forum?id=pH3XAQME6c
- [76]M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang et al., “HarmBench: A standardized evaluation framework for automated red teaming and robust refusal,” in ICML, 2024.
- [77]A. Souly, Q. Lu, D. Bowen, T. Trinh, E. Hsieh, S. Pandey, P. Abbeel, J. Svegliato, S. Emmons, O. Watkins et al., “A strongreject for empty jailbreaks,” Advances in Neural Information Processing Systems, vol. 37, pp. 125 416–125 440, 2024.arxiv.org/abs/2402.10260
- [78]P. Chao, E. Debenedetti, A. Robey, M. Andriushchenko, F. Croce, V. Sehwag, E. Dobriban, N. Flammarion, G. J. Pappas, F. Tramer et al., “Jailbreakbench: An open robustness benchmark for jailbreaking large language models,” Advances in Neural Information Processing Systems, vol. 37, pp. 55 005–55 029, 2024.
- [80]F. Bénézit, A. G. Dimakis, P. Thiran, and M. Vetterli, “Order-optimal consensus through randomized path averaging,” IEEE Transactions on Information Theory, pp. 5150–5167, 2010.arxiv.org/abs/0802.2587
- [81]R. Thakur, R. Rabenseifner, and W. Gropp, “Optimization of collective communication operations in mpich,” The International Journal of High Performance Computing Applications, vol. 19, no. 1, pp. 49–66, 2005.DOI
- [82]Q. Team et al., “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025.
- [84]A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” arXiv preprint arXiv:2307.15043, 2023.arxiv.org/abs/2307.15043
- [85]Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang et al., “MMLU-Pro: A more robust and challenging multi-task language understanding benchmark,” in Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2024.arxiv.org/abs/2406.01574
- [86]J. Cui, W.-L. Chiang, I. Stoica, and C.-J. Hsieh, “OR-bench: An over-refusal benchmark for large language models,” in Forty-second International Conference on Machine Learning, 2025. [Online]. Available: https://openreview.net/forum?id=CdFnEuO JZVarxiv.org/abs/2405.20947
- [87]P. Röttger, H. R. Kirk, B. Vidgen, G. Attanasio, F. Bianchi, and D. Hovy, “XSTest: A test suite for identifying exaggerated safety behaviours in large language models,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2024.arxiv.org/abs/2308.01263
- [88]K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
- [89]D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, “Measuring mathematical problem solving with the MATH dataset,” in Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2021.
- [90]M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, and J. Wei, “Challenging BIG-Bench tasks and whether chain-of-thought can solve them,” in Findings of the Association for Computational Linguistics (ACL), 2023.arxiv.org/abs/2210.09261
- [91]D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman, “GPQA: A graduate-level google-proof q&a benchmark,” in Conference on Language Modeling (COLM), 2024.arxiv.org/abs/2311.12022
- [92]Z. Sprague, X. Ye, K. Bostrom, S. Chaudhuri, and G. Durrett, “MuSR: Testing the limits of chain-of-thought with multistep soft reasoning,” in International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.16049




