WorksComputer Science
Addressing Exogenous Variability in Cooperative Multi-Agent Reinforcement Learning
Abstract
Cooperative multi-agent reinforcement learning (MARL) often fails under exogenous variability such as shifts in opponents or environment regimes that cannot be controlled by the team but alter the transition dynamics. We formalize this challenge as Exogenous Dec-POMDP (ED-POMDP), in which an episode-level exogenous regime, selected independently of the team, indexes the dynamics, and the global state is partitioned into components that are directly responsive to the current team action and components that are not directly action-responsive at the current step. This formulation exposes mediated influence: while team actions do not directly update the latter at the current step, they can shape their later evolution through induced changes in the former. Based on this, we propose LEICA, a CTDE-compatible algorithm that learns history-conditioned context representations for both components and shapes policy updates using agent-wise, influence-weighted counterfactual intrinsic rewards. Across SMAX benchmarks with opponent strategy shifts, LEICA consistently improves both training performance and generalization to unseen opponents' strategies over existing baselines, supporting the usefulness of influence-weighted shaping under exogenous train-test regime shifts.
Introduction
Multi-Agent Reinforcement Learning (MARL) has achieved success across a wide range of domains including games [1, 2], autonomous robotics, and large-scale decision-making. In particular, cooperative benchmarks such as the StarCraft Multi-Agent Challenge (SMAC) [3] designed to model battles against opponent teams, have spurred the development of various MARL algorithms including value decomposition methods [4, 5] and actor-critic methods [6, 7].
Despite such progress, current cooperative MARL methods remain brittle under task variations, such as changes in opponent strategies, scenario parameters, or latent environment regimes. Policies trained on a narrow distribution often overfit to specific training regimes and collapse at test time [8]. Crucially, these variations are typically exogenous: they are not controlled by the agent team, but they alter the environment dynamics [9]. While previous MARL methods treat these variations as ordinary task changes, we explicitly leverage their exogenous nature to improve both training performance and generalization.
Our idea is to separate the state components that the current team action updates directly from those that are updated mainly by the opponent or the environment, and to learn which directly induced changes influence the later evolution of the latter under the current regime. We formalize this view as Exogenous Dec-POMDPs (ED-POMDPs): the term exogenous refers to the episode-level regime, which is selected independently of the team, while the global state is partitioned into directly action-responsive components and components that are not directly action-responsive at the current step. The latter are not independent of the team over a trajectory; team actions can affect them later through the evolving state, for example when ally movement changes how a fixed but state-reactive opponent moves and attacks. Based on ED-POMDP, we propose Learning Exogenous Influence for Coordination and Adaptation (LEICA), a CTDE-compatible method whose influence-weighted intrinsic rewards concentrate policy updates on directly induced state changes that can shape the later response of the opponent or environment.
We evaluated LEICA on SMAX [10] with train/test shifts in opponent strategies and initial conditions. Across multiple settings, LEICA improves performance over existing CTDE, context-inference, and robust MARL baselines that do not exploit this structure, and additional experiments on MABrax and MPE-Tag provide complementary evidence beyond SMAX.
Contributions
We formalize ED-POMDP for cooperative MARL under exogenous variability, which separates the exogenous regime from an approximate one-step action-responsiveness partition of the state and highlights influence mediated by directly action-responsive transitions.
We propose LEICA, a CTDE method that shapes policy updates with influence-weighted, policy-relative changes of directly action-responsive state. Its multi-agent contribution is an agent-wise marginal transition predictor and counterfactual credit signal that attribute team-level effects to individual policies without enumerating joint actions, while execution remains decentralized.
We present a SMAX evaluation protocol with train/test shifts in opponent strategies and initial conditions as our main evaluation, where LEICA improves performance over existing MARL baselines, complemented by MABrax and MPE-Tag experiments.
Background
A cooperative multi-agent environment is formulated as a Decentralized Partially-Observable Markov Decision Process (Dec-POMDP) [11, 12], defined by the tuple , where is the set of agents, the state space, the action space of agent , transition dynamics, the reward function, the observation space of agent , and the discount factor. At each timestep , the environment is in global state . Each agent receives a local observation and selects an action . The joint action induces a transition and produces a team reward .
Each agent has an action-observation history , and follows a stochastic policy , where denotes the set of histories and denotes the set of probability distributions on . The objective is to maximize the expected discounted return . We adopt centralized training with decentralized execution (CTDE): global state information is accessible in training time, whereas in test time agent policies rely on local observations.
Related Works
MARL under CTDE CTDE is the standard paradigm for cooperative MARL with partial observability. Major actor-critic variants include MADDPG and MAA2C [13, 14], on-policy MAPPO and IPPO [6, 15], and theory-based trust-region methods HATRPO and HAPPO with heterogeneous-agent extensions [16, 7]. For value-based methods, VDN and QMIX implement additive and monotonic mixing based on Individual-Global-Maximum principle [4, 5], and many subsequent studies investigate the value decomposition structure of MARL [17, 18, 19, 20, 21, 22].
Influence-based coordination Influence-based methods use how much an agent affects other agents or the environment as an intrinsic signal for coordination, exploration, or opponent-aware behavior. Prior work has studied influence through communication, where messages shape other agents’ behavior [23, 24], and has proposed metrics for causal influence among agents [25, 26]. Other methods define influence from agents’ actions and mainly use it to promote exploration [27– 29]. Opponent modeling infers other agents’ policies or types [30], and opponent-learning-aware methods explicitly model or anticipate another learner’s policy updates [31, 32, 33, 34]. LEICA differs from these lines in that it does not use a separate communication channel or model concurrently learning opponents; it measures the influence of directly induced state changes on the later response of fixed but state-reactive opponents and uses it to support both adaptation to regime variation and team coordination.
Context-based adaptation and generalization in RL Several neighboring areas study generalization under distributional or strategic variation. Robust MARL seeks policies that remain effective under train-to-test shifts, for example through minimax optimization against altered opponents [35] or robust value factorization under environmental uncertainty [36]. Meta-RL adapts to task variation by encoding interaction history into recurrent states or latent task variables, as in , PEARL, and VariBAD [37, 38, 39]; a few extensions to multi-agent settings are available in decentralized and game-theoretic formulations [40, 41]. LEICA shares the idea that history can reveal the current regime, and our MAPPO+ctx baseline follows this spirit by adding a VariBAD-style context encoder. The key difference is that LEICA does not rely on context inference alone but further exploits state decomposition for better generalization. Zero-shot coordination focuses on compatibility with novel cooperative partners [42, 43], and ad hoc teamwork studies cooperation with previously unseen teammates, often without prior joint training [44, 45]. LEICA instead assumes a jointly trained team and studies shifts in opponent or environment regimes; these settings are complementary and not subsumed by our formulation.
Structured state representations Factored MDPs exploit conditional independence among state variables for planning [46], and Block MDPs and exogenous-information MDPs recover latent controllable states or discard exogenous noise for sample efficiency [47, 48]. Rather than removing the not directly action-responsive component for single-agent efficiency, LEICA estimates how directly induced changes steer its later evolution and attributes this influence to individual agents.
Motivation and Problem Setup
We first show that cooperative MARL policies can overfit to training regimes and collapse under exogenous shifts, and that multi-regime training with context inference is still insufficient; we then formally define the problem setting.
Why Exogenous Variation Breaks Coordination
In cooperative MARL, policy updates are driven by the interactions experienced during training. This process reinforces coordination patterns tailored to the training opponents and environment dynamics. While such policies may exhibit high performance in training, they fail when encountering shifts in factors outside the team’s control at test time. We refer to such externally driven shifts as exogenous variability in multi-agent systems.
Figure 1a illustrates this failure situation, and Figure 1b shows an empirical example in SMAX [10]. MAPPO and QMIX achieve near-perfect win rates against the seen opponent, but their performance collapses under modified opponent strategies. Thus, high training-regime performance does not imply robust coordination under exogenous shifts.

Figure 1. Motivation. (a) An episode-level exogenous regime (e.g., opponent strategy and initial distance) changes how a fixed but state-reactive opponent responds. Ally actions directly change ally state (e.g., positions), which then affects the opponent’s later movement and attacks. (b) Policies trained against the default SMAX opponent achieve high performance but collapse under unseen opponent strategies. (c) Multi-regime training with VariBAD-style context inference [39] (MAPPO+ctx) still suffers under exogenous variation.
A natural response to this collapse is to train on multiple regimes and infer the current regime from history. However, this context inference helps recognize which regime the team is in, but does not specify which directly induced state changes are useful for adaptation. Figure 1c shows this gap: MAPPO and QMIX reach about 60% win rate on seen regimes but drop to roughly 20% on unseen regimes, and MAPPO with VariBAD-style context inference [39] (MAPPO+ctx) only slightly narrows the gap. Therefore, the key challenge lies not only in identifying the exogenous regime, but also in learning how the team should intervene through directly action-responsive state variables. What is missing is an influence signal from directly induced state changes to the later response of the opponent or environment, which our method learns and uses to reward team actions that steer this response.
Problem Setup
We formalize exogenous variability as episode-to-episode changes in the environment that the team cannot control, but that alter the interaction dynamics. Formally, we consider a family of cooperative Dec-POMDPs indexed by a variation index that specifies the exogenous regime. Across different , the state, action, and observation spaces and the discount factor are fixed, while the transition kernel, reward function, and initial-state distribution may vary. We write these components as , , and .
The goal is to learn a joint policy that performs well under training regimes while remaining effective under possibly shifted regimes at evaluation. Let denote a regime-conditioned disturbance or opponent response. Under this conditional-exogeneity model, the regime-specific transition kernel is the marginal
If denotes the episode-level regime itself, is a point mass at . If it denotes a reactive opponent response, its realization may vary with the state even though the opponent policy parameters remain fixed within the episode. Our main setting uses such fixed-but-reactive opponents; concurrently learning opponents, best-response dynamics, and equilibrium adaptation are outside the main scope.
Training and evaluation During training, is sampled from , and during evaluation from . With the expected discounted team return under joint policy , the training objective is , while generalization under exogenous shift is assessed on , which may contain unseen regimes.
Exogenous Dec-POMDP
We formalize exogenous variability with an Exogenous Dec-POMDP (ED-POMDP). Throughout, exogenous refers to the external selection of the episode-level regime . A state-dependent response need not be independent of the team’s actions over a trajectory. In ED-POMDP, the global state is written as , where denotes directly action-responsive components, whose next-step values are directly updated by the current team action, and denotes components that are not directly action-responsive at the current step; they are primarily updated by the regime-dependent opponent or environment, and may still be influenced by earlier team actions through the evolving state. Here, the partition can be given by domain knowledge or estimated as shown in § 6.
Definition 5.1 (Approximate one-step action-responsiveness). Let be a partition map and define and , where returns the complementary coordinates. The partition satisfies the approximate one-step action-responsiveness condition with constants if
where denotes conditional mutual information.
These conditions are conditional and one-step: given the current state and regime, the current team action carries little additional information about the next , and given the current state and action, the regime carries little additional information about the next . They do not imply that is independent of the team’s actions over a trajectory.
Lemma 5.2 (Marginal invariance and conditional factorization). Let . (a) The one-step marginals satisfy
(b) If, in addition, , the transition factorizes as
Part (a) follows directly from Definition 5.1; part (b) requires the additional conditional-independence assumption (proof in Appendix § 14). LEICA does not require or enforce the exact factorization in (b): its predictors and influence signal are learned directly from rollouts and rely only on approximate one-step action-responsiveness. In SMAX, the domain-informed split is -approximate, since contacts and sub-step interactions can couple the components.
Under Definition 5.1 with , the conditional marginal distribution of is invariant to the current action on the support of the data distribution. As shown in Figure 2, however, this invariance does not rule out delayed dependence on the action. The action can first update the directly action-responsive state , which may subsequently influence . Thus, represents one possible mediated influence chain. In SMAX, for example, ally movement changes ally positions, which can subsequently alter the movement, targeting, or attacks of the fixed but state-reactive opponent.
Figure 2. Illustration of delayed mediated influence. can influence a later response through .
Consequently, the actionable adaptation problem is to learn which directly induced changes can steer the subsequent evolution of via this chain, and to concentrate policy updates on directly action-responsive coordinates with high mediated influence. LEICA implements this principle with a scalable agent-wise influence signal in the next section.
Proposed Method
We present Learning Exogenous Influence for Coordination and Adaptation (LEICA), a CTDE-compatible algorithm for ED-POMDPs. As illustrated in Figure 3, LEICA consists of two components: (1) an ED-POMDP module that infers history-conditioned contexts for predicting the directly action-responsive and the not directly action-responsive state components, and (2) an influence reward that highlights directly induced changes predictive of the later response of the opponent or environment.

Figure 3. Overview of LEICA.
Context Inference for Direct and Response Dynamics
State partition In an ED-POMDP, the global state decomposes as . Here, a state coordinate is one scalar feature of the centralized training state. With domain knowledge, coordinates whose next-step update is directly affected by the current team action are assigned to and those primarily updated by the opponent or environment to . Otherwise, LEICA constructs the split automatically: an auxiliary one-step transition predictor is trained, and counterfactual action perturbations replace one agent’s action with another valid action while holding the state and other agents’ actions fixed. A coordinate’s action-sensitivity score is its normalized predicted next-step change under these perturbations, and two-cluster splitting assigns high- and low-sensitivity coordinates to and . The partition is estimated during the first 10% of training and then fixed, assuming that the action-responsiveness structure stays sufficiently stable.
Latent context inference Given the resulting partition, each agent infers two latent contexts from its local trajectory history : a direct context for action-conditioned transitions of , and a regime context for action-free transitions of . Then, the direct and response dynamics for agent are modeled as
where and are implemented with function approximators, and denotes the set of steps associated with the context time . The direct predictor conditions on the individual action , while the response predictor does not take actions, reflecting the one-step condition in Definition 5.1. During centralized training, both predictors receive the centralized state, which provides the current physical configuration, and a context inferred from agent ’s local history, which supplies regime-relevant information; no predictor infers the global transition from local history alone. The direct predictor is an agent-wise marginal model under the empirical on-policy behavior of other agents rather than a joint-action model, which avoids enumerating joint actions. The implementation details and training objectives for the context extraction and the dynamics models are provided in Appendix C.
Influence Scores and Weight Construction
We use the learned predictors and to construct rewards that favor directly induced changes that influence the later response (adaptation) and are sensitive to team value (coordination). This follows the delayed influence link in Figure 2.
Mediated influence scores At timestep , agent ’s response predictor outputs . All agent-specific predictors share the same centralized next-state target but may infer different contexts . We compute the Jacobian with respect to the directly action-responsive state:
(5) is a local mediated-influence estimator of the path under context ; recomputed along trajectories, it can also highlight intermediate changes in longer chains when informative intermediate responses exist. For each coordinate of , we define its influence score by aggregating the agent-wise Jacobian column norms
where is a training batch and is the dimension of . (6) aggregates sensitivities rather than averaging predicted states, so it uses multiple local histories without requiring their contexts to be identical.
Value sensitivity Note that influence alone can emphasize coordinates that only affect opponent or environment responses but are not useful for improving task performance. We therefore compute a value-sensitivity score, the absolute centralized-critic gradient, for each coordinate of .
Weight construction We then standardize in (6) and in (7) across coordinates, yielding . We then form a combined score vector
where controls the trade-off between adaptation-relevant influence and value-sensitive coordination. We decay over training updates, prioritizing mediated influence early when the critic is less reliable and shifting toward value sensitivity later. Finally, is converted into a nonnegative coordinate influence weight vector , which carries the essence of the score vector . For details, please refer to Appendix § 14.§ 3.
Influence-weighted Intrinsic Reward
We now quantify how agent ’s action changes the directly action-responsive coordinates relative to the expectation under the current policy . Using the learned direct predictor in Eq. (4), we compute
where is element-wise absolute value, and is a model prediction, not an unrealized simulator transition. For each recorded transition, we keep and fixed and compare the realized action with valid alternatives under the agent’s current recurrent policy. In SMAX, the expectation is computed exactly over valid discrete actions, with invalid actions masked. Since does not take as input, other agents’ behavior is marginalized through the on-policy training distribution rather than conditioned on their realized actions. We then combine and the influence weight vector in § 6.2, to construct the intrinsic reward, given as the weighted direct change:
Let denote the shared extrinsic team reward from the environment. Our shaped reward is given by
where controls the intrinsic reward scale.
Actors are trained with the MAPPO clipped surrogate objective [6] using individual shaped rewards , while the centralized critic regresses returns from the team-aggregated shaped reward . Actor-critic gradients are not backpropagated into the predictors; the context encoders and predictors are updated only by their predictive losses. Implementation details and the full algorithm are provided in Appendix § 14.
Experiments
We compare LEICA with strong CTDE, intrinsic-reward, context-inference, and robust MARL baselines, inspect the learned influence weights, and ablate the main components. SMAX serves as our main evaluation; experiments on continuous physical perturbations and a mixed setting provide complementary evidence on whether the mediated-influence principle extends beyond SMAX-style opponent shifts.
Setup
We evaluate LEICA on SMAX [10], a JAX-based implementation of SMAC [3]. Its GPU-native implementation makes it straightforward to alter the policies of opponent units, which is essential for studying exogenous variability.
We use five cooperative scenarios: 5m_vs_6m, 10m_vs_11m, 3s5z_vs_3s6z, 6h_vs_9z, and 3s_vs_6z. The first three are standard super-hard SMAX tasks, while the last two are extreme-hard variants obtained by increasing enemy sizes.
Exogenous configurations We define a discrete opponent strategy space with five strategic axes, yielding 63 opponent strategies, and independently vary the initial distance between teams. This gives 189 exogenous configurations in total. We group them into three scales: small (train 3 / test 3), medium (train 10 / test 5), and large (train 30 / test 10). Training uses only configurations in the training set, while evaluation reports performance separately on seen training configurations and unseen held-out configurations. Details are provided in Appendix § 13.
Training and evaluation protocol All methods are trained for 10M environment steps. The evaluation unit is an independently trained run, scored by its final win rate on the seen or unseen configurations; we report mean SEM over five tasks eight seeds. For Table 1, we also use unpaired stratified permutation tests with Holm correction (Appendix § 17.§ 2); other conclusions rest on mean SEM. Policies are checkpointed every 0.5M steps and evaluated on multiple episodes per checkpoint. While hyperparameters and compute details are reported in Appendix § 15, we note that LEICA’s additional Jacobian and predictor computations are training-time only; at execution, agents use only local histories and the learned context encoder, yielding less than 5% runtime overhead.
| Method | Small Seen | Small Unseen | Medium Seen | Medium Unseen | Large Seen | Large Unseen | Overall Seen | Overall Unseen |
| MAPPO | ||||||||
| MAPPO+ctx | ||||||||
| QMIX | ||||||||
| LAIES | ||||||||
| SHAQ | ||||||||
| M3DDPG | ||||||||
| M3PPO | ||||||||
| LEICA | ||||||||
| LEICA (manual) |
Table 1. Scale-wise SMAX results under opponent strategy variation. Win rate (%, mean SEM) over five tasks and eight training seeds per task and scale, evaluated on training (Seen) and held-out (Unseen) configurations. Bold indicates the highest mean in each column. Full per-task results are provided in Appendix § 17.
Baselines
We compare against standard CTDE baselines MAPPO [6], QMIX [5], and SHAQ [49]. We include LAIES [29], a partition-informed intrinsic-reward method, and MAPPO+ctx, a VariBAD-style context-inference baseline [39], to test whether context recognition alone is sufficient. We also include robust MARL baselines M3DDPG [35] and M3PPO, our PPO-based variant, to evaluate whether generic max-min training preserves generalization under exogenous variability. For M3DDPG and M3PPO, we instantiate the uncertainty set through action perturbations and tune the adversarial rate under the same protocol as other hyperparameters.
We report the results of the default variant of LEICA, which employs the automatic partitioning module. We additionally provide the results of LEICA (manual), which instead uses a domain-informed action-responsiveness partition when such knowledge is available. Both partition procedures are detailed in Appendix § 13.§ 2 and Appendix § 13.§ 3.
Performance Comparison
Table 1 summarizes performance across small, medium, and large scales. LEICA outperforms all non-LEICA baselines in both seen and unseen regimes at every scale, with statistically significant differences in all columns, improving the overall score over MAPPO from 61.0 to 78.2 in seen regimes and from 34.5 to 59.5 in unseen regimes. MAPPO+ctx improves over vanilla MAPPO in some settings, but remains far below LEICA. This suggests that recognizing the current regime is not enough; the policy also needs a learning signal that identifies which directly induced state changes can affect the later opponent response. Indeed, robust max-min training is insufficient in this setting, as M3DDPG performs poorly overall and M3PPO remains below MAPPO.
Notably, the default LEICA variant with automatic partitioning performs comparably to LEICA (manual) overall (no significant difference in the Overall columns), indicating that LEICA can recover effective action-responsiveness partitions without prior domain knowledge of the state structure. The domain-informed partition remains significantly better at the large scale, whereas the automatic one is better on small unseen configurations. Figure 4 complements the final performance table by showing that LEICA improves both training dynamics and unseen-regime performance.
Figure 4. Aggregated learning curves on SMAX. Win rates are averaged across five tasks for each scale and reported on seen and unseen exogenous configurations.
Learned Influence Weights
To illustrate what LEICA emphasizes during training, we visualize the coordinate weights on 3s_vs_6z in Figure 5 (X/Y: spatial coordinates, CD: weapon cooldown, HP: health point).

Figure 5. Influence weights over training on 3s_vs_6z.
Early in training, the weights are focused on directly action-responsive movement coordinates such as ally0_X, ally1_X, and ally2_X, which are predictive of subsequent opponent responses. As training progresses, the emphasis shifts towards task-relevant outcome coordinates such as enemy HP, consistent with the decaying schedule on : LEICA first prioritizes coordinates with high mediated influence and later shifts toward value-sensitive coordinates as the critic stabilizes.
Component Evaluation and Ablation
We ablate LEICA to isolate which components drive generalization under exogenous variability. All results in Table 2 are reported in the medium scale setting, and the base algorithm is LEICA (manual).
| Category | Variant | Seen | Unseen |
| Partition mode | LEICA | 76.1±3.7 | 55.1±5.7 |
| Partition mode | LEICA (manual) | 76.0±3.1 | 56.5±4.6 |
| Partition mode | full-state reconstruction | 69.3±3.5 | 43.6±5.3 |
| Component evaluation | w.o. intrinsic reward | 59.4±3.5 | 28.1±4.8 |
| Component evaluation | w.o. counterfactual design | 29.2±3.4 | 17.0±3.7 |
| Component evaluation | vanilla MAPPO | 53.7±2.9 | 19.7±4.5 |
| scheduling | (influence-only) | 67.6±3.2 | 40.2±5.1 |
| scheduling | (value-only) | 65.7±3.5 | 37.1±5.6 |
| scheduling | (constant) | 70.4±3.3 | 46.6±5.5 |
| scheduling | linear decaying | 71.5±3.5 | 49.6±5.7 |
| Partition robustness | swapping 10% | 65.4±3.5 | 39.1±5.6 |
| Partition robustness | swapping 30% | 67.2±3.9 | 37.0±5.8 |
| Partition robustness | swapping 50% | 55.6±3.7 | 22.7±5.2 |
Table 2. Ablations on the medium scale. Bold: highest mean.
Partition mode LEICA performs strongly without a hand-specified partition, while LEICA (manual) gives the better unseen generalization when a reliable domain partition is available. The full-state reconstruction variant also improves over MAPPO, indicating that influence-weighted shaping is useful even without an explicit split, but partitioned variants provide cleaner signals.
Component evaluation We test whether context inference alone is sufficient and whether counterfactual endogenous change is necessary. Removing intrinsic reward keeps the context modules but removes the shaping signal, yielding a context-only variant. Table 2 shows that this variant remains above vanilla MAPPO but is far below full LEICA. Removing the counterfactual design causes a sharper collapse, indicating that policy-relative endogenous change is crucial for stable shaping.
scheduling We compare variants of the influence coefficient , which balances mediated influence and value sensitivity when forming . Linear decay and a moderate constant trade-off perform well, while using only one signal consistently degrades performance. This supports our design: early training benefits from influence and later training benefits from value sensitive coordinates.
Partition robustness Randomly swapping coordinates between and degrades performance gradually, approaching MAPPO at 50%. This ablation measures robustness to misassigned coordinates, not whether a partition is causally correct: better alignment with one-step action dependence improves the influence signal, but a perfect partition is not required.
Additional Validation Beyond SMAX
We evaluate LEICA on MABrax [10, 50], multi-agent continuous-control locomotion tasks where agents jointly control different parts of a robot. The details of regime design and implementation are provided in Appendix § 17.§ 4. As shown in Table 3, LEICA improves over MAPPO on both seen and unseen regimes across all three tasks. These results are complementary to and less extensive than the SMAX study. We also report a mixed-setting validation on MPE-Tag [51] in Appendix § 17.§ 5. While predators and prey are trained by concurrent self-play, cross-play shows that LEICA-trained predators and prey each improve their own returns against MAPPO, which we regard as preliminary evidence beyond fixed opponents (Appendix).
| Environment | MAPPO | LEICA |
| Walker2d 6x1 | 677±71 / 544±31 | 987±62 / 706±18 |
| Ant 8x1 | 785±83 / 514±24 | 1084±52 / 677±26 |
| HalfCheetah 6x1 | 1799±31 / 665±88 | 2280±131 / 1704±91 |
Table 3. MABrax validation: returns on seen / unseen regimes.
Conclusion and Limitations
We addressed cooperative MARL under exogenous variability by introducing ED-POMDP, a Dec-POMDP formalism that separates an episode-level exogenous regime from an approximate one-step partition of the state. Building on this view, we proposed LEICA, which shapes policy updates with agent-wise counterfactual changes in the directly action-responsive state weighted by their influence on the later response of the opponent or environment. Experiments on SMAX with opponent strategy variation show that LEICA improves performance on unseen regimes and reduces train-test gaps relative to strong baselines, with ablations confirming the contribution of each component.
Limitations LEICA relies on identifying action-dependent state structure from training rollouts. Its effectiveness may decrease with noisy transitions, hidden variables, changing dynamics, or limited data. It remains effective under mild and moderate action corruption, but its advantage disappears under severe corruption. LEICA also does not guarantee disentanglement of entangled state inputs. Since Eq. (5) measures local influence, it may miss effects that emerge only after several steps without observable intermediate changes. Our main experiments assume fixed but state-reactive opponents. Learning opponents and equilibrium adaptation remain outside our scope. Finally, the automatic partition remains fixed after early estimation and may require updating when the environment changes. SMAX is also our only comprehensive benchmark.
Acknowledgments and Disclosure of Funding
This work was supported in part by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) (No. RS-2024-00457882, National AI Research Lab Project), by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT) (No. RS-206-25510069), and by the “Advanced GPU Utilization Support Program” funded by the Government of the Republic of Korea (Ministry of Science and ICT) (No. 02-26-01-0110). The authors declare no competing interests.
References
Broader Impact
This work is primarily methodological and aims to improve robustness in multi-agent reinforcement learning under changing external conditions. Potential applications include coordinated control, logistics, and resource allocation systems where agents must adapt to regime shifts. As with other reinforcement-learning methods, deployment in real-world settings should be accompanied by task-specific validation and monitoring.
Experiment Details
We provide a complete specification of our experimental setup to support reproducibility and to clarify the evaluation protocol. We first describe the platform, scenarios, and the SMAX interfaces in Sec. B.1. We then describe two partitioning choices: the automatic action-sensitivity partition used by default in LEICA, and the domain-informed SMAX partition used for LEICA (manual), in Secs. B.2–B.3. We finally define the scripted opponent configuration space and our train-test split construction in Sec. B.4.
Environment and Evaluation Protocol
We run all experiments in SMAX using JAXMARL [10], which provides a fully vectorized JAX implementation of SMAC-style micromanagement tasks. Table 4 lists the five scenarios used in the main paper together with their episode horizons.
| Name | Ally Team | Enemy Team | Max Steps |
| 5m_vs_6m | 5 Marines | 6 Marines | 70 |
| 10m_vs_11m | 10 Marines | 11 Marines | 150 |
| 3s5z_vs_3s6z | 3 Stalkers, 5 Zealots | 3 Stalkers, 6 Zealots | 170 |
| 6h_vs_9z | 6 Hydralisks | 9 Zealots | 150 |
| 3s_vs_6z | 3 Stalkers | 6 Zealots | 250 |
Table 4. SMAX scenarios used in the experiments.
Each agent receives a local observation defined by SMAX. The observation contains: (i) relative position, (ii) current health, (iii) last_movement_x, last_movement_y, (iv) last_targeted, and (v) weapon cooldown, for observable entities within sight range. We use the default SMAX visibility and featurization provided by JAXMARL.
The discrete action set consists of 4 move actions (up, down, left, right), stop, and attack actions for each attackable enemy unit. As in SMAC/SMAX, invalid actions are masked. The environment reward matches the SMAX default: (i) damage shaping proportional to total enemy damage dealt over the episode, and (ii) a win bonus upon victory.
Automatic Action-Sensitivity Partition
The default LEICA does not use manually assigned state partition. Instead, it constructs the partition from training rollouts using action sensitivity. The principle is that coordinates whose next-step values change substantially under local counterfactual action perturbations are treated as directly action-responsive, while coordinates with weak action sensitivity are treated as not directly action-responsive at the current step.
Terminology A state coordinate is one scalar feature of the centralized training state . A counterfactual action perturbation replaces one agent’s action with another valid action while holding the state and the other agents’ actions fixed. The action-sensitivity score below is the normalized predicted next-step change of coordinate under such perturbations. The value-sensitivity score in (7) is the absolute gradient of the centralized critic with respect to coordinate of . We say that a partition is aligned with one-step action dependence when the coordinates assigned to are those whose next-step values depend on the current team action given the state and regime.
Concretely, we first fit a lightweight one-step transition predictor on training rollouts to predict the next global state. For each state coordinate , we compute an action-sensitivity score
where denotes valid counterfactual action perturbations and is the empirical standard deviation of coordinate in the training rollouts.
For discrete SMAX actions, is formed by replacing one agent’s action with another valid action, and averaging over sampled agents and valid alternatives. In our implementation, we normalize the scores and apply a one-dimensional two-cluster split on a logarithmic scale. Coordinates in the higher-sensitivity cluster are assigned to , while the remaining coordinates are assigned to . The partition is estimated using rollouts collected during the first of training and is then kept fixed for the remaining policy optimization. Until the partition is fixed, both predictors are trained on all coordinates and no intrinsic reward is applied.
Stationarity assumption This procedure does not assume that the full dynamics are stationary, since the regime and the policies change during training. It does assume that the action-responsiveness structure, i.e., which coordinates respond directly to the current team action, remains sufficiently stable after the estimation window. Settings in which this structure changes substantially during training would require online repartitioning, which is outside the scope of this work.
Domain-Informed Partition and SMAX Example
For LEICA (manual), we use a domain-informed partition.
Procedure We provide (i) the list of centralized-state coordinates and (ii) domain knowledge about which current team actions can directly affect each coordinate’s next-step update. Each coordinate is then assigned by the following rule:
if the team action directly updates the coordinate within one decision step, it is assigned to ;
if the coordinate is primarily updated by the opponent or the environment, it is assigned to ;
if both mechanisms act on the coordinate, it is assigned according to the dominant one-step mechanism, so that the resulting partition is approximate.
The resulting split corresponds to the approximate one-step action-responsiveness condition in Definition 5.1 with small but nonzero , rather than to an exact factorization of the simulator.
In SMAX, ally moves immediately change ally positions, and ally attacks (when valid) immediately update ally weapon cooldown and reduce enemy health. Accordingly, we define as the union of: (i) ally positions , (ii) ally weapon cooldowns, and (iii) enemy health values.
Not directly action-responsive coordinates Enemy positions and enemy cooldowns evolve according to the opponent micro-policy, and ally health changes as a consequence of enemy attacks. Thus, we define as: (i) enemy positions , (ii) enemy weapon cooldowns, and (iii) ally health values.
This partition captures the dominant one-step update mechanism. Ally actions can influence the response coordinates through later interaction, for example by changing ally positions and thereby changing subsequent opponent behavior. It does not assert exact absence of within-step coupling. This matches the ED-POMDP interpretation used in the main paper. Because contacts between units and sub-step interactions within a decision step can couple the two groups, this SMAX split is an -approximate controllability partition that captures the dominant one-step mechanism rather than an exact simulator factorization.
Opponent Strategy Space and Train-Test Regimes
Scripted Opponent Strategy Axes. To evaluate generalization under exogenous regime shifts, we vary the built-in scripted opponent configuration along the following discrete axes:
When , attack-specific axes are inactive; the remaining behavior is governed by movement/targeting logic. Counting unique opponent scripts yields 63 configurations over the non-distance axes; crossing with distance produces total configurations.
Spawn proximity (distance) We parameterize the initial ally-enemy separation by (abbreviated as 4,6,8). With map width , the nominal team-center separation is , where +2 enforces a minimum separation after per-unit jitter. All ally (enemy) units are placed by adding i.i.d. noise of around their team center, so realized separations concentrate around . A configuration is sampled at episode start and fixed for the entire episode.
Strategy encoding We encode each configuration with a 6-character token [S] [A] [P] [M] [T] [D]: S for ; A for ; P for persistency; M for ; T for target_mode = ; D for distance. When an axis is inactive (e.g., some attack-related axes under ), we use a placeholder symbol (e.g., X) for that position in the token.
Train-test splits We construct train-test regimes by selecting disjoint sets of strategy tokens. For each scale, training and test sets contain disjoint configuration tokens. A held-out token can differ from a training token in only one axis, so this protocol tests generalization to unseen configurations within the defined configuration space rather than enforcing a minimum separation beyond token disjointness.
Small (3 train / 3 test) Train={TCTCC6, TWTCW6, TRFAR6}; Test={TCTCR6, TWFAC6, TRTCW6}.
Medium (10 train / 5 test) Train={TCTCR6, TCFAR8, TCFRW4, TWTRC6, TWTRR4, TWFAC8, TRTCC8, TRTCW6, TRFAW4, FXXAR6}; Test={TCTCC4, TCTAW6, TWTCW8, TWTAC4, TRFCC4}.
Large (30 train / 10 test) Train={TCTCW4, TCTCR6, TCTAC6, TCTRC6, TCTRR4, TCFCW6, TCFAC8, TCFAR8, TCFRW4, TWTCR4, TWTRC6, TWTRR4, TWTRR6, TWFCC8, TWFAC8, TWFAR6, TWFRC8, TWFRC8, TRTC C8, TRTCW6, TRTAW8, TRTRW6, TRFCW8, TRFAW4, TRFRC8, TRFRR6, FXXAW4, FXXCR4, FXXCC8, FXXRW8, FXXAR6}; Test={TCTCC4, TCTAW4, TCTAW6, T WTCW4, TWTCW8, TWTAC4, TW IAC6, TWTRW6, TRTCR4, TRFCC4}.
Standard SMAX (single-opponent) For completeness, we also report results on the default SMAX setting without opponent variation (Appendix F), using the built-in heuristic configuration TCTCR6.
Relation to Other Benchmarks
Our evaluation protocol is complementary to recent SMAC-derived benchmarks that were introduced to stress-test generalization beyond the default single-opponent setting.
SMAC-HARD [8] is closely aligned with our motivation. It explicitly targets opponent diversity by enabling customizable opponent strategy scripts, randomization over adversarial behaviors, and an explicit black-box testing protocol. Our SMAX-based protocol shares the same motivation, but differs in emphasis. We construct multi-axis exogenous regime shifts by composing multiple explicit opponent-behavior knobs, rather than focusing on edited scripts and self-play interfaces.
SMACv2 [52] addresses a different but related generalization gap by procedurally generating scenarios and requiring agents to generalize to previously unseen settings sampled from the same distribution at evaluation time, with additional variants designed to enforce meaningful partial observability. In contrast, our protocol keeps the underlying scenario fixed and focuses on exogenous regime shifts driven by opponent micro-policy variations, isolating robustness to opponent/environment changes from robustness to scenario generation. SMACv2 primarily tests distributional variation in initial conditions and scenario structure, while our benchmark tests behavioral variation of the opponent script under a fixed task.
Method Details
Theoretical Results
Proof of Lemma 5.2 Throughout, let and fix . We use the fact that, for random variables , holds if and only if , i.e., for every in the support.
Part (a): marginal invariance. The condition in Definition 5.1 gives , i.e., . Hence
Similarly, gives , i.e., , and therefore
These two identities concern only the one-step marginals of and ; they do not constrain the dependence between the two components.
Part (b): factorization. Since , the chain rule gives
Without further assumptions, the second factor may depend on and, through it, on ; Definition 5.1 alone therefore does not imply a product form. Under the additional assumption ,
Substituting this identity and the two marginal identities from part (a) into the chain-rule expansion yields
The exact factorization in part (b) should be understood as an idealized case that requires the additional conditional-independence assumption. In practical environments, the partition is approximate, and weak coupling between the two components may remain. LEICA neither enforces nor requires this factorization. The predictors and influence signal are learned directly from rollouts and rely only on approximate one-step action responsiveness.
Context Inference and Predictor Training
For each agent and time , we infer two stochastic contexts with reparameterization,
where the history encoder for contexts is a shared GRU [53] over the agent’s trajectory , and are sampled via the standard reparameterization trick. We use filtered priors in the style of variBAD [39]:
GRU is not shared with the actor or critic encoders. The two contexts share the same GRU and branch into separate heads for each and . With , we train two predictors on multiple indices sampled from the same episode:
Here includes both components for inputs to both predictors. Each agent-specific predictor receives the centralized state together with the context inferred from agent ’s local history. To respect the condition in Definition 5.1, the response predictor does not take actions as input. The direct predictor conditions on the individual action .
We sample a fixed number of valid indices. For a rollout window of length , predictor evaluation therefore costs per batch element. The current same-episode index sampler additionally constructs time-pair masks, so the linear claim applies to predictor evaluation rather than the entire sampling pipeline. Decoder training uses dropout with . We use in the reported configurations.
Predictive Objective and KL Regularization
For each anchor we optimize two separate ELBOs. One for the direct context and one for the regime context .
Direct (for and ):
Response (for and ):
with filtered priors from Eq. (13).
In practice, we replace the Gaussian log-likelihood terms with reconstruction MSE losses scaled by , and we use a single reparameterized sample from each posterior to estimate the expectations. The total objective maximized is
Context encoders and predictors are updated only by above. When intrinsic rewards are formed, gradients are stopped through the influence weights and through the counterfactual direct change . Thus, RL losses do not update the representation modules through the reward path.
Details of Weight Construction
This subsection complements § 6.2 by specifying the practical details.
Let denote the set of decision-time indices in a PPO minibatch. From each transition in , we compute (i) influence scores from Eq. (6), and (ii) value-sensitivity scores from Eq. (7). We aggregate them across the minibatch by averaging:
We standardize each vector across coordinates:
where and denote the mean and standard deviation over , is the ones vector, and is a small constant for numerical stability.
Given the combined score vector from Eq. (8), we map it to a positive weight vector via a temperature-scaled softmax:
where controls the sharpness of the distribution.
To reduce noise from Jacobians and value gradients, we apply an exponential moving average (EMA) to the weights:
Here is maintained across policy updates and initialized as the uniform vector at the start of training. This smoothing yields a temporally more consistent weighting signal and prevents abrupt changes of the intrinsic reward when the critic or predictors are still adapting.
Schedule of We implement the exponentially decaying trade-off using the update index (number of PPO updates):
with chosen so that remains large in early training and decreases gradually thereafter. We use , where is the total number of PPO updates.
Details regarding the influence-based reward
Counterfactual Baseline Counterfactual baselines are widely used to reduce variance and to provide local credit assignment in policy gradient methods. In the single-agent case, adding a state-dependent baseline leaves the gradient unbiased since
so one may replace returns with advantages without bias [54].
In cooperative multi-agent settings under CTDE, this counterfactual idea is used to isolate the marginal effect of a single agent while marginalizing over its own action. Practically, this choice provides variance reduction and agent-level credit signals, and has been widely adopted in related multi-agent policy gradient methods [55, 56, 57]. The canonical example is COMA [55], which defines the agent-wise counterfactual advantage
holding other agents fixed at .
In LEICA, the same principle is applied not to values but to predicted transitions of the directly action-responsive state, inspired by recent study [58]. The baseline for agent is defined as a policy expectation at time ,
the expected magnitude of the direct change under the agent’s own policy, computed exactly over the valid actions with the policy renormalized over them. Contrasting this baseline with the realized action gives in Eq. (9). The comparison keeps the recorded state and agent-specific context fixed. Unlike COMA, the predictor does not condition on the other agents’ realized actions and the teammate behavior is marginalized through the on-policy training distribution. This substitution preserves the spirit of counterfactual baselines, marginalizing the agent’s own action under its current policy, while tailoring the signal to LEICA’s objective of weighting direct changes by their mediated influence on the later response and by value sensitivity.
We summarize the full training pipeline of LEICA with Algorithm 1.
Implementation Details
All experiments use JAX/Flax with JaxMARL and vectorized environments, collecting fixed-length rollouts from multiple parallel environments and training on minibatches on GPU [10]. We apply parameter sharing for homogeneous agents and action masking at the policy head. For MAPPO [6], QMIX [5], and SHAQ [49], we follow the JaxMARL implementation. For LAIES [29], as there was no code implemented in a version suitable for SMAX, we re-implement on top of QMIX by adding the original intrinsic influence term from the authors’ code and paper,2 keeping the QMIX
Algorithm 1 LEICA
1: Initialize parameters $\Theta$, partition predictor $\hat{f}_{\omega}$, and normalization statistics.
2: $w \leftarrow 0$ (auto) or $1/d_{d}$ (manual).
3: for $x = 0, \ldots, X - 1$ do
4: (A) Rollout and targets
5: $\pi_{\mathrm{old}} \leftarrow \pi_{\theta}$; $\mathcal{D} \leftarrow \operatorname{Rollout}(\pi_{\mathrm{old}})$.
6: $\Delta^{i}_{t} \leftarrow \operatorname{CounterfactualChange}(S_{t}, a^{i}_{t}, z^{i}_{t}; \pi_{\mathrm{old}})$ ({xref}`Eq. 9`)
7: $r^{\mathrm{int},i}_{t} \leftarrow \langle w, \Delta^{i}_{t}\rangle$
8: $\tilde{r}^{i}_{t} \leftarrow r^{\mathrm{ext}}_{t} + \lambda r^{\mathrm{int},i}_{t}$; $\tilde{r}_{t} \leftarrow N^{-1}\sum_{i=1}^{N}\tilde{r}^{i}_{t}$.
9: $\hat{A}^{i} \leftarrow \operatorname{GAE}(\tilde{r}^{i}, V_{\phi})$; $\hat{R} \leftarrow V_{\phi} + \operatorname{GAE}(\tilde{r}, V_{\phi})$.
10: (B) Joint optimization
11: for each PPO epoch do
12: $z^{i}_{t} \sim q_{\phi^{i}_{t}}$, $y^{i}_{t} \sim q_{\phi^{i}_{t}}$; $K_{t} \leftarrow \operatorname{SampleSameEpisode}(\mathcal{D})$ ({xref}`Appendix C.2`).
13: $\mathcal{L}_{\mathrm{total}} \leftarrow \mathcal{L}_{\mathrm{actor}} + \mathcal{L}_{\mathrm{critic}} + \mathcal{L}^{i,t}_{\mathrm{ELBO},z} + \mathcal{L}^{i,t}_{\mathrm{ELBO},y}$.
14: Update $\Theta$ with $\mathcal{L}_{\mathrm{total}}$.
15: end for
16: (C) Weights for the next rollout
17: if automatic partition is unresolved and warm-up has ended then
18: $\chi \leftarrow \operatorname{ActionSensitivity}(\hat{f}_{\omega}, \mathcal{D})$.
19: $(S^{d}, S^{r}) \leftarrow \operatorname{TwoMeans}\left(\log\left(\chi/(\max_{j}\chi_{j}+10^{-8})+0.01\right)\right)$.
20: Freeze the partition; $w \leftarrow 1/d_{d}$.
21: end if
22: if the partition is available then
23: Compute $c^{r}, c^{v}$ by {xref}`Eqs. 6 and 7`.
24: $\bar{c}^{r} \leftarrow \operatorname{Std}(c^{r})$; $\bar{c}^{v} \leftarrow \operatorname{Std}(c^{v})$.
25: $\alpha \leftarrow e^{-kx}$; $c \leftarrow \alpha\bar{c}^{r} + (1-\alpha)\bar{c}^{v}$.
26: $w \leftarrow \operatorname{sg}[(1-\beta)w+\beta\operatorname{softmax}(c/\tau)]$.
27: end if
28: end for
mixing and target update structure [29]. We implement M3DDPG [35] by following EPyMARL [14] and the official M3DDPG codebase3, reusing only the buffer and sampling structure of QMIX implementation while re-implementing the algorithmic components. For M3PPO, we modify MAPPO to learn a state-action value instead of a state value, and apply the same max-min update scheme used in M3DDPG.
For M3DDPG and M3PPO, the adversarial component is trained only on the training opponent-regime set and never observes held-out test regimes. The adversarial rate controls the strength of the max-min update and is selected from the search space in Table 8 and Table 9. In M3PPO, we use the same PPO rollout data as MAPPO but replace the state-value critic with a state-action critic so that the inner minimization can evaluate adversarial action perturbations. This makes M3PPO the closest PPO-style analogue of M3DDPG under our CTDE implementation.
| Hyperparameter | Value |
| Entropy Coefficient | |
| Learning Rate | |
| Maximum gradient norm | |
| PPO clipping | |
| PPO epochs | |
| M3PPO specific | |
| Adversarial rate | |
| MAPPO+ctx specific | |
| latent size | |
| Reconstruction weight | |
| KL regularization coef | |
| LEICA specific | |
| latent size | |
| Sampling number | |
| Reconstruction weight | |
| KL regularization |
Table 8. Hyperparameter search spaces for MAPPO, MAPPO+ctx, M3PPO, and LEICA.
| Hyperparameter | Value |
| Learning Rate | |
| Rollout length | |
| Replay buffer size | |
| Target update interval | |
| SHAQ specific | |
| Sample size | |
| M3DDPG specific | |
| Adversarial rate |
Table 9. Hyperparameter search spaces for QMIX, LAIES, SHAQ, and M3DDPG.
Training and Evaluation Protocol
At evaluation time, each episode samples a configuration uniformly from the union of the training and test sets and holds it fixed for the full episode. Win rates are reported separately for the seen and unseen configurations. Each evaluation runs 256 parallel environments for 256 steps with greedy actions, and all reported results are averaged over 8 random seeds. Our primary metric is average win rate.
Computational cost Experiments were run on systems equipped with NVIDIA RTX 3090 (24GB) or RTX 4090 (24GB) GPUs. We report both parameter overhead and runtime overhead relative to MAPPO in Table 5. LEICA introduces additional train-time computation from context inference, transition predictors, and influence-weight updates. These components are used for training-time shaping only. At execution time, the policy does not use the predictors or Jacobian scores.
| Method | Train-time parameters | Execution-time parameters | Runtime overhead |
| MAPPO | 1.00× | 1.00× | 1.00× |
| LEICA | 1.45× | 1.06× | 1.13× train / < 1.05× execution |
Table 5. Computational overhead relative to MAPPO. Auxiliary predictors and Jacobian computations are training-time components.
Hyperparameters
Hyperparameters were tuned with Bayesian optimization over a fixed budget [59], and search spaces are detailed below.
Default parameters
| Hyperparameter | Value |
| Discount factor | 0.99 |
| GAE parameter | 0.95 |
| Value loss coefficient | 0.5 |
| Number of minibatches | 1 |
| Critic Network | [128, 128] |
| Actor Network | [128, 128] |
| Recurrent Network | GRU |
| Recurrent Hidden | 128 |
| Activation | tanh |
| Learning-rate schedule | linear decay |
| Value-loss clipping | PPO clip |
| LEICA specific | |
| Predictor network | [64] |
| exponentially decaying | |
| Reward weight | 0.03 |
| EMA ratio | 0.2 |
| Softmax | 0.5 |
Table 6. Default hyperparameters for MAPPO, MAPPO+ctx, M3PPO, and LEICA.
| Hyperparameter | Value |
| Discount factor | 0.99 |
| Buffer batch size | 64 |
| Network size | [128, 128] |
| Mixer embedding size | 32 |
| Mixer hypernet size | 64 |
| Recurrent Network | GRU |
| Recurrent Hidden | 128 |
| Activation | ReLU |
| Maximum gradient norm | 10 |
| Minimum exploration coef | 0.05 |
| Epochs | 4 |
Table 7. Default hyperparameters for QMIX, LAIES, SHAQ, and M3DDPG.
Hyperparameter search spaces
Guidelines for Hyperparameter Selection
The softmax temperature controls how selectively the weights focus on influential directly action-responsive coordinates. We find empirically that values in the range 0.1-1.0 work well. The EMA coefficient smooths weight updates and improves stability. Its effective behavior is correlated with the update interval. When updates occur frequently with small batches, a lower is preferable, whereas larger batches or slower update intervals allow for higher . The intrinsic reward should guide adaptation without overwhelming the task reward. work reliably in SMAX, but this coefficient is generally a sensitive parameter and should be tuned per environment.
LLM Usage
We used LLMs for polishing text. All algorithmic design, mathematics and experiments were written by the authors.
Full Experimental Results
Component Evaluation and Ablation
This appendix complements Table 2 by visualizing the full learning dynamics of LEICA ablations on the medium-scale benchmark. Each figure reports win-rate learning curves evaluated separately on seen and unseen configurations. Curves are averaged across the five SMAX scenarios and eight random seeds, and shaded regions indicate standard error of the mean.
Figure 6. Ablation on reward design. Win-rate learning curves on the medium-scale benchmark. We compare LEICA (default) against two variants: no intrinsic reward (contexts retained but intrinsic reward removed) and no counterfactual design (intrinsic reward retained but the counterfactual baseline in Eq. (9) removed).
Figure 7. Ablation on full-state reconstruction. LEICA() removes the state partition and trains predictors by reconstructing the full state. We additionally report LEICA(, ) (value-only weighting), LEICA(, ) (influence-only weighting), and LEICA(, no reward).
Figure 8. Ablation on scheduling. We compare the default exponentially decaying schedule (training update index ) against linear decay, and constant choices .
Figure 9. Robustness to imperfect partitions. We randomly swap a fraction of coordinates between and (swapping rate ) and compare against the default LEICA ().
Statistical Analysis of the Main Results
Evaluation units and procedure Each independently trained run is scored by its final win rate on seen or unseen configurations, with runs grouped by the five SMAX tasks. Since shared seeds do not ensure paired runs across methods, we use unpaired comparisons. Differences are block-balanced mean differences (percentage points). We apply within-task stratified permutation tests, a two-stage cluster bootstrap (tasks, then runs) for 95% confidence intervals, and Holm-Bonferroni correction across baselines within each scale and metric. The probability of improvement (PoI) is the block-balanced fraction of run pairs favoring LEICA, with ties counted as . Overall pools all 15 task-scale blocks.
Table 10 shows that LEICA significantly outperforms all non-LEICA baselines in every column. Compared with LEICA (manual), it performs significantly better on small unseen configurations but worse at large scale, with no significant difference in Overall.
| Scale | LEICA vs. | Seen [95% CI] | Seen PoI [95% CI] | Seen | Unseen [95% CI] | Unseen PoI [95% CI] | Unseen |
| Small | LEICA (manual) | 2.72 [−3.45, 8.56] | 0.614 [0.345, 0.866] | 0.053 | 5.35 [0.54, 10.35] | 0.681 [0.463, 0.891] | 0.001 |
| Small | MAPPO | 19.25 [6.22, 34.40] | 0.853 [0.681, 0.984] | <0.001 | 22.85 [10.20, 36.21] | 0.994 [0.969, 1.000] | <0.001 |
| Small | MAPPO+ctx | 11.69 [4.43, 18.82] | 0.847 [0.675, 0.981] | <0.001 | 14.21 [8.39, 19.59] | 0.903 [0.772, 0.991] | <0.001 |
| Small | QMIX | 31.21 [21.99, 40.44] | 0.997 [0.981, 1.000] | <0.001 | 27.89 [20.54, 35.37] | 1.000 [1.000, 1.000] | <0.001 |
| Small | LAIES | 50.23 [35.13, 68.62] | 0.997 [0.981, 1.000] | <0.001 | 48.50 [32.95, 67.77] | 0.970 [0.894, 1.000] | <0.001 |
| Small | SHAQ | 92.47 [87.90, 96.80] | 1.000 [1.000, 1.000] | <0.001 | 87.58 [82.51, 94.03] | 1.000 [1.000, 1.000] | <0.001 |
| Small | M3DDPG | 87.99 [74.93, 96.75] | 1.000 [1.000, 1.000] | <0.001 | 82.64 [69.01, 93.87] | 1.000 [1.000, 1.000] | <0.001 |
| Small | M3PPO | 52.85 [35.27, 70.01] | 0.972 [0.900, 1.000] | <0.001 | 54.84 [35.74, 72.73] | 0.986 [0.938, 1.000] | <0.001 |
| Medium | LEICA (manual) | 0.11 [−7.29, 6.90] | 0.562 [0.362, 0.753] | 0.971 | −1.39 [−12.08, 7.16] | 0.541 [0.330, 0.725] | 0.709 |
| Medium | MAPPO | 22.45 [11.93, 32.45] | 0.806 [0.631, 0.947] | <0.001 | 35.39 [18.49, 52.83] | 0.891 [0.794, 0.972] | <0.001 |
| Medium | MAPPO+ctx | 19.74 [6.38, 32.41] | 0.791 [0.600, 0.944] | <0.001 | 28.09 [6.84, 47.63] | 0.853 [0.698, 0.963] | <0.001 |
| Medium | QMIX | 17.85 [−0.31, 36.64] | 0.825 [0.550, 1.000] | <0.001 | 38.29 [18.07, 58.02] | 0.819 [0.550, 1.000] | <0.001 |
| Medium | LAIES | 16.37 [−5.71, 38.75] | 0.637 [0.306, 0.950] | <0.001 | 45.35 [15.72, 74.43] | 0.802 [0.520, 1.000] | <0.001 |
| Medium | SHAQ | 54.18 [39.33, 68.19] | 0.975 [0.900, 1.000] | <0.001 | 40.16 [13.74, 67.26] | 0.753 [0.459, 0.975] | <0.001 |
| Medium | M3DDPG | 60.56 [43.61, 77.72] | 0.988 [0.941, 1.000] | <0.001 | 54.86 [27.18, 81.56] | 0.981 [0.938, 1.000] | <0.001 |
| Medium | M3PPO | 44.42 [28.69, 60.90] | 0.959 [0.850, 1.000] | <0.001 | 53.31 [26.94, 78.42] | 0.963 [0.867, 1.000] | <0.001 |
| Large | LEICA (manual) | −6.22 [−12.84, −0.39] | 0.378 [0.188, 0.569] | 0.025 | −12.03 [−22.98, −2.63] | 0.339 [0.203, 0.484] | 0.015 |
| Large | MAPPO | 9.97 [1.61, 20.97] | 0.706 [0.525, 0.866] | 0.002 | 16.61 [2.79, 35.69] | 0.775 [0.597, 0.927] | 0.002 |
| Large | MAPPO+ctx | 9.90 [−0.80, 25.80] | 0.675 [0.472, 0.875] | 0.002 | 15.64 [−1.41, 41.59] | 0.697 [0.491, 0.894] | 0.007 |
| Large | QMIX | 11.27 [−1.31, 23.90] | 0.781 [0.547, 0.981] | <0.001 | 10.43 [−4.61, 24.70] | 0.759 [0.463, 0.991] | 0.015 |
| Large | LAIES | 6.28 [−7.56, 22.07] | 0.569 [0.269, 0.869] | 0.034 | 19.87 [−0.23, 44.88] | 0.700 [0.425, 0.944] | <0.001 |
| Large | SHAQ | 30.14 [18.87, 41.45] | 1.000 [1.000, 1.000] | <0.001 | 9.23 [−10.43, 31.93] | 0.512 [0.169, 0.853] | 0.029 |
| Large | M3DDPG | 48.34 [40.19, 59.44] | 1.000 [1.000, 1.000] | <0.001 | 32.88 [16.75, 54.29] | 1.000 [1.000, 1.000] | <0.001 |
| Large | M3PPO | 16.94 [6.83, 28.48] | 0.869 [0.731, 0.975] | <0.001 | 24.37 [7.85, 45.68] | 0.887 [0.694, 1.000] | <0.001 |
| Overall | LEICA (manual) | −1.13 [−5.43, 2.99] | 0.518 [0.381, 0.656] | 0.404 | −2.69 [−9.21, 3.09] | 0.570 [0.392, 0.653] | 0.454 |
| Overall | MAPPO | 17.22 [10.25, 24.87] | 0.789 [0.685, 0.883] | <0.001 | 24.95 [15.32, 35.36] | 0.886 [0.801, 0.956] | <0.001 |
| Overall | MAPPO+ctx | 13.78 [6.89, 21.29] | 0.771 [0.656, 0.874] | <0.001 | 19.31 [9.22, 30.54] | 0.818 [0.713, 0.909] | <0.001 |
| Overall | QMIX | 20.11 [10.95, 29.13] | 0.868 [0.739, 0.973] | <0.001 | 25.54 [15.15, 36.11] | 0.859 [0.715, 0.974] | <0.001 |
| Overall | LAIES | 24.29 [10.28, 38.60] | 0.734 [0.549, 0.900] | <0.001 | 37.91 [22.81, 53.16] | 0.824 [0.681, 0.946] | <0.001 |
| Overall | SHAQ | 58.93 [44.47, 73.19] | 0.992 [0.967, 1.000] | <0.001 | 45.66 [25.38, 65.44] | 0.755 [0.565, 0.919] | <0.001 |
| Overall | M3DDPG | 65.63 [54.33, 77.03] | 0.996 [0.979, 1.000] | <0.001 | 56.79 [40.62, 72.38] | 0.994 [0.978, 1.000] | <0.001 |
| Overall | M3PPO | 38.07 [26.79, 49.67] | 0.933 [0.867, 0.983] | <0.001 | 44.17 [29.80, 58.64] | 0.945 [0.867, 0.996] | <0.001 |
Table 10. Unpaired statistical comparisons of LEICA against each method in Table 1: block-balanced difference (percentage points) with 95% cluster-bootstrap CI, PoI with 95% CI, and Holm-adjusted stratified permutation -value.
Sensitivity to Stochasticity
To examine how LEICA behaves when its relative predictive signal is perturbed by stochastic dynamics, we conduct two sensitivity experiments on the five medium-scale tasks with eight seeds per task, using the same statistical procedure as in Appendix F.2. In the first, each opponent action is replaced by a randomly sampled valid action with probability . In the second, we use an action-attack protocol common in robust MARL: each cooperative agent’s executed action is replaced by a randomly sampled valid action with probability . Table 11 reports the final win rates (as fractions).
| Perturbation | Regime | LEICA | MAPPO | Difference [95% CI] | PoI [95% CI] | -value | |
| Opponent random action | Seen | 0.1 | 0.713 | 0.278 | 0.435 [0.306, 0.563] | 0.795 [0.719, 0.867] | <0.001 |
| Seen | 0.2 | 0.837 | 0.640 | 0.196 [0.124, 0.274] | 0.709 [0.644, 0.783] | <0.001 | |
| Seen | 0.4 | 0.964 | 0.956 | 0.008 [-0.004, 0.016] | 0.486 [0.378, 0.606] | 0.270 | |
| Unseen | 0.1 | 0.681 | 0.241 | 0.441 [0.318, 0.605] | 0.777 [0.697, 0.861] | <0.001 | |
| Unseen | 0.2 | 0.819 | 0.554 | 0.265 [0.175, 0.357] | 0.784 [0.716, 0.853] | <0.001 | |
| Unseen | 0.4 | 0.962 | 0.931 | 0.031 [0.017, 0.040] | 0.720 [0.633, 0.809] | 0.0014 | |
| Ally action attack | Seen | 0.1 | 0.604 | 0.522 | 0.082 [0.044, 0.121] | 0.691 [0.578, 0.800] | 0.0049 |
| Seen | 0.2 | 0.500 | 0.396 | 0.104 [0.083, 0.123] | 0.747 [0.647, 0.856] | 0.0003 | |
| Seen | 0.4 | 0.301 | 0.294 | 0.007 [-0.007, 0.021] | 0.545 [0.416, 0.694] | 0.371 | |
| Unseen | 0.1 | 0.320 | 0.184 | 0.136 [0.078, 0.194] | 0.645 [0.534, 0.756] | 0.0047 | |
| Unseen | 0.2 | 0.120 | 0.027 | 0.094 [0.062, 0.120] | 0.694 [0.634, 0.750] | 0.0070 | |
| Unseen | 0.4 | 0.000 | 0.000 | 0.000 [0.000, 0.000] | 0.487 [0.463, 0.500] | 1.000 |
Table 11. Sensitivity to stochasticity on the medium scale: block-balanced win rates, difference with 95% cluster-bootstrap CI, PoI with 95% CI, and Holm-adjusted stratified permutation -value.
Opponent stochasticity LEICA remains significantly better than MAPPO at all tested levels except the seen regime at , where both methods approach saturated performance. Random opponent actions also weaken the opponent, as reflected by the increasing win rates of both methods. These results show maintained task performance under the tested opponent-action stochasticity.
Ally action attacks Absolute performance decreases as the attack becomes stronger, but LEICA maintains a statistically supported advantage over MAPPO under mild and moderate attacks () in both seen and unseen regimes. Under the strongest attack (), task performance is severely degraded and LEICA no longer differs significantly from MAPPO. This experiment characterizes performance under corruption of the cooperative agents’ execution channel; attributing the degradation specifically to partition recovery requires a separate quantitative analysis.
MABrax Continuous-Control Validation
We additionally evaluate a latent predictive variant of LEICA on MABrax, a multi-agent continuous-control benchmark. Below are the details.
Exogenous context construction Each episode is associated with an exogenous context
where is an actuator-scale multiplier, is the target forward velocity, is the momentum coefficient of a reactive wind perturbation, and controls the strength with which forward velocity drives the wind. At timestep , the action sent to the underlying MABrax environment is , where is a scalar wind state shared across action dimensions. The wind evolves according to , where denotes the current forward velocity. The wind update uses the pre-step velocity . Therefore, the current action cannot directly determine at the same decision step. Instead, it can affect the next robot velocity and thereby influence and later wind states. The task reward is further shaped by the context-specific target velocity, , with in our experiments.
Train/test context split During training, each episode samples one context uniformly from the four training contexts in Table 12. Evaluation is performed on both the seen training contexts and three held-out contexts. The held-out contexts simultaneously change actuation strength, target velocity, and wind dynamics, providing continuous physical exogenous shifts rather than opponent-script shifts.
Latent-space LEICA implementation Let denote the recurrent latent state produced by agent ’s encoder from its local observation history. The MABrax implementation uses two stochastic latent heads, and . The direct decoder predicts the next latent state using the current latent state and the action,
whereas the response decoder predicts the next latent state without the action,
Table 12: MABrax exogenous contexts. Each context is , corresponding to actuator scale, target velocity, wind momentum, and wind gain.
| Split | ||||
| Train 1 | 0.90 | 0.80 | 0.85 | 0.05 |
| Train 2 | 0.95 | 1.00 | 0.80 | 0.10 |
| Train 3 | 1.05 | 1.20 | 0.75 | 0.08 |
| Train 4 | 1.10 | 1.00 | 0.90 | 0.12 |
| Test 1 | 0.85 | 0.60 | 0.70 | 0.15 |
| Test 2 | 1.00 | 1.40 | 0.95 | 0.18 |
| Test 3 | 1.15 | 1.10 | 0.65 | 0.06 |
Table 12.
This mirrors the SMAX design in latent space: the direct branch captures action-sensitive transitions, while the response branch captures action-free latent transition structure. The intrinsic reward compares the predicted latent change under the sampled action with the predicted change under the policy mean action,
where denotes the policy mean action. The coordinate weights are updated from the Jacobian norm of the response decoder with respect to the latent state with EMA smoothing. As in the main experiments, the auxiliary decoders and Jacobian computations are used during training only.
MPE-Tag Mixed-Setting Validation
We also evaluate a mixed cooperative-competitive extension on MPE-Tag [51]. MPE-Tag contains asymmetric predator and prey roles with opposing objectives. We train role-specific policies with either MAPPO or LEICA and evaluate whether replacing one role’s MAPPO policy with a LEICA policy improves that role’s own return while holding the counterpart fixed. Table 13 reports role-wise comparisons.
Table 13: MPE-Tag role-wise cross-play validation.
| Evaluated role | Fixed counterpart | MAPPO role policy | LEICA role policy | |
| Predator | MAPPO prey | 162.7 | 168.6 | +5.9 |
| Predator | LEICA prey | 159.2 | 165.2 | +6.0 |
| Prey | MAPPO predator | -224.3 | -215.6 | +8.7 |
| Prey | LEICA predator | -232.1 | -220.2 | +11.9 |
Table 13.
The cross-play results isolate the role-wise effect of LEICA. Positive means that replacing MAPPO with LEICA improves the return of the replaced role. In all cases, replacing MAPPO policy with a LEICA policy improves the performance. These results are intended as an additional role-wise validation rather than a full mixed-game benchmark. They suggest that the mediated-influence mechanism is not restricted to purely cooperative SMAX settings, while a broader study of mixed and competitive games, including self-play dynamics and opponent-learning baselines, remains future work.
Single-Task Verification on Standard SMAX
On the original single-strategy SMAX benchmarks, LEICA maintains competitive or superior performance relative to all baselines, as shown in Figure 10. This confirms that the proposed design is not specialized solely for distribution shifts but also remains effective in fixed-opponent settings.
Figure 10. Single-task SMAX. Learning curves on standard SMAX without exogenous variation. LEICA matches or exceeds baselines, indicating that the proposed method does not degrade conventional performance.
Additional Results per All Task-Regime Pairs
We provide all task results below. Figures 11- 13 present the results for Small, Medium, Large scale respectively.
Figure 11. Average win rate for all task-regime pairs in the small-scale benchmark.
Figure 12. Average win rate for all task-regime pairs in the medium-scale benchmark.
Figure 13. Average win rate for all task-regime pairs in the large-scale benchmark.
References
- [1]D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershlevam, M. Lanctot et al., “Mastering the game of go with deep neural networks and tree search,” nature, vol. 529, no. 7587, pp. 484–489, 2016.DOI
- [2]O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev et al., “Grandmaster level in starcraft ii using multi-agent reinforcement learning,” nature, vol. 575, no. 7782, pp. 350–354, 2019.DOI
- [3]M. Samvelyan, T. Rashid, C. S. De Witt, G. Farquhar, N. Nardelli, T. G. Rudner, C.-M. Hung, P. H. Torr, J. Foerster, and S. Whiteson, “The starcraft multi-agent challenge,” arXiv preprint arXiv:1902.04043, 2019.arxiv.org/abs/1902.04043
- [4]P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls et al., “Value-decomposition networks for cooperative multi-agent learning,” arXiv preprint arXiv:1706.05296, 2017.arxiv.org/abs/1706.05296
- [5]T. Rashid, M. Samvelyan, C. Schroeder, G. Farquhar, J. Foerster, and S. Whiteson, “Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning,” in International Conference on Machine Learning. PMLR, 2018, pp. 4295–4304.
- [6]C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu, “The surprising effectiveness of ppo in cooperative multi-agent games,” Advances in neural information processing systems, vol. 35, pp. 24 611–24 624, 2022.arxiv.org/abs/2103.01955
- [7]Y. Zhong, J. G. Kuba, X. Feng, S. Hu, J. Ji, and Y. Yang, “Heterogeneous-agent reinforcement learning,” Journal of Machine Learning Research, vol. 25, no. 32, pp. 1–67, 2024.arxiv.org/abs/2304.09870
- [8]Y. Deng, Y. Yu, W. Ma, Z. Wang, W. Zhu, J. Zhao, and Y. Zhang, “Smac-hard: Enabling mixed opponent strategy script and self-play on smac,” arXiv preprint arXiv:2412.17707, 2024.arxiv.org/abs/2412.17707
- [9]A. Mahajan, M. Samvelyan, T. Gupta, B. Ellis, M. Sun, T. Rocktäschel, and S. Whiteson, “Generalization in cooperative multi-agent systems,” arXiv preprint arXiv:2202.00104, 2022.arxiv.org/abs/2202.00104
- [10]A. Rutherford, B. Ellis, M. Gallici, J. Cook, A. Lupu, G. Ingvarsson Juto, T. Willi, R. Hammond, A. Khan, C. Schroeder de Witt et al., “Jaxmarl: Multi-agent rl environments and algorithms in jax,” Advances in Neural Information Processing Systems, vol. 37, pp. 50 925–50 951, 2024.
- [11]D. S. Bernstein, R. Givan, N. Immerman, and S. Zilberstein, “The complexity of decentralized control of markov decision processes,” Mathematics of operations research, vol. 27, no. 4, pp. 819–840, 2002.
- [12]F. A. Oliehoek, C. Amato et al., A concise introduction to decentralized POMDPs. Springer, 2016, vol. 1.DOI
- [13]R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” Advances in neural information processing systems, vol. 30, 2017.arxiv.org/abs/1706.02275
- [14]G. Papoudakis, F. Christianos, L. Schäfer, and S. V. Albrecht, “Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks,” arXiv preprint arXiv:2006.07869, 2020.arxiv.org/abs/2006.07869
- [15]C. S. De Witt, T. Gupta, D. Makovichchuk, V. Makoviychuk, P. H. Torr, M. Sun, and S. Whiteson, “Is independent learning all you need in the starcraft multi-agent challenge?” arXiv preprint arXiv:2011.09533, 2020.arxiv.org/abs/2011.09533
- [16]J. G. Kuba, R. Chen, M. Wen, Y. Wen, F. Sun, J. Wang, and Y. Yang, “Trust region policy optimisation in multi-agent reinforcement learning,” arXiv preprint arXiv:2109.11251, 2021.arxiv.org/abs/2109.11251
- [17]J. Wang, Z. Ren, T. Liu, Y. Yu, and C. Zhang, “Qplex: Duplex dueling multi-agent q-learning,” arXiv preprint arXiv:2008.01062, 2020.arxiv.org/abs/2008.01062
- [18]K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y. Yi, “Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning,” in International conference on machine learning. PMLR, 2019, pp. 5887–5896.arxiv.org/abs/1905.05408
- [19]T. Rashid, G. Farquhar, B. Peng, and S. Whiteson, “Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning,” Advances in neural information processing systems, vol. 33, pp. 10 199–10 210, 2020.
- [20]Y. Yang, J. Hao, B. Liao, K. Shao, G. Chen, W. Liu, and H. Tang, “Qatten: A general framework for cooperative multiagent reinforcement learning,” arXiv preprint arXiv:2002.03939, 2020.arxiv.org/abs/2002.03939
- [21]L. Zheng, J. Chen, J. Wang, J. He, Y. Hu, Y. Chen, C. Fan, Y. Gao, and C. Zhang, “Episodic multi-agent reinforcement learning with curiosity-driven exploration,” Advances in Neural Information Processing Systems, vol. 34, pp. 3757–3769, 2021.arxiv.org/abs/2111.11032
- [22]K. Son, Y. Yi, J. Kim, J. Shin, S. Ahn, and R. D. Reyes, “Disentangling sources of risk for distributional multi-agent reinforcement learning,” in The 39th International Conference on Machine Learning, ICML 2022. International Conference on Machine Learning, 2022.
- [23]D. Strouse, M. Kleiman-Weiner, J. Tenenbaum, M. Botvinick, and D. J. Schwab, “Learning to share and hide intentions using information regularization,” Advances in neural information processing systems, vol. 31, 2018.arxiv.org/abs/1808.02093
- [24]J. Foerster, I. A. Assael, N. De Freitas, and S. Whiteson, “Learning to communicate with deep multi-agent reinforcement learning,” Advances in neural information processing systems, vol. 29, 2016.arxiv.org/abs/1605.06676
- [25]R. Lowe, J. Foerster, Y.-L. Boureau, J. Pineau, and Y. Dauphin, “On the pitfalls of measuring emergent communication,” arXiv preprint arXiv:1903.05168, 2019.arxiv.org/abs/1903.05168
- [26]W. Kim, W. Jung, M. Cho, and Y. Sung, “A variational approach to mutual information-based coordination for multi-agent reinforcement learning,” in Proceedings of the 22nd International Conference on Autonomous Agents and Multiagent Systems (AAMAS), May 2023.arxiv.org/abs/2303.00451
- [29]B. Liu, Z. Pu, Y. Pan, J. Yi, Y. Liang, and D. Zhang, “Lazy agents: A new perspective on solving sparse reward problem in multi-agent reinforcement learning,” in Proceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 202. PMLR, 2023, pp. 21 937–21 950.
- [30]H. He, J. Boyd-Graber, K. Kwok, and H. Daumé III, “Opponent modeling in deep reinforcement learning,” in International conference on machine learning. PMLR, 2016, pp. 1804–1813.arxiv.org/abs/1609.05559
- [31]J. N. Foerster, R. Y. Chen, M. Al-Shedivat, S. Whiteson, P. Abbeel, and I. Mordatch, “Learning with opponent-learning awareness,” arXiv preprint arXiv:1709.04326, 2017.arxiv.org/abs/1709.04326
- [32]S. Zhao, C. Lu, R. B. Grosse, and J. Foerster, “Proximal learning with opponent-learning awareness,” Advances in Neural Information Processing Systems, vol. 35, pp. 26 324–26 336, 2022.arxiv.org/abs/2210.10125
- [33]M. Aghajohari, J. A. Duque, T. Cooijmans, and A. Courville, “Loqa: Learning with opponent q-learning awareness,” arXiv preprint arXiv:2405.01035, 2024.arxiv.org/abs/2405.01035
- [34]J. A. Duque, M. Aghajohari, T. Cooijmans, R. Ciuca, T. Zhang, G. Gidel, and A. Courville, “Advantage alignment algorithms,” arXiv preprint arXiv:2406.14662, 2024.arxiv.org/abs/2406.14662
- [35]S. Li, Y. Wu, X. Cui, H. Dong, F. Fang, and S. Russell, “Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient,” in Proceedings of the AAAI conference on artificial intelligence, vol. 33, no. 01, 2019, pp. 4213–4220.DOI
- [36]C. Qu, C. Yeh, K. Panaganti, E. Mazumdar, and A. Wierman, “Distributionally robust cooperative multi-agent reinforcement learning with value factorization,” in The Fourteenth International Conference on Learning Representations, 2026. [Online]. Available: https://openreview.net/forum?id=2T3LOpqIOOarxiv.org/abs/2602.11437
- [37]Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel, “RI²: Fast reinforcement learning via slow reinforcement learning,” arXiv preprint arXiv:1611.02779, 2016.arxiv.org/abs/1611.02779
- [38]K. Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen, “Efficient off-policy meta-reinforcement learning via probabilistic context variables,” in International conference on machine learning. PMLR, 2019, pp. 5331–5340.arxiv.org/abs/1903.08254
- [39]L. Zintgraf, K. Shiarlis, M. Igl, S. Schulze, Y. Gal, K. Hofmann, and S. Whiteson, “Varibad: A very good method for bayes-adaptive deep rl via meta-learning,” arXiv preprint arXiv:1910.08348, 2019.arxiv.org/abs/1910.08348
- [40]M. Kayaalp, S. Vlaski, and A. H. Sayed, “Dif-maml: Decentralized multi-agent meta-learning,” IEEE Open Journal of Signal Processing, vol. 3, pp. 71–93, 2022.DOI
- [41]W. Mao, H. Qiu, C. Wang, H. Franke, Z. Kalbarczyk, R. Iyer, and T. Basar, “Multi-agent meta-reinforcement learning: Sharper convergence rates with task similarity,” Advances in Neural Information Processing Systems, vol. 36, pp. 66 556–66 570, 2023.DOI
- [42]H. Hu, A. Lerer, A. Peysakhovich, and J. Foerster, ““other-play” for zero-shot coordination,” in International conference on machine learning. PMLR, 2020, pp. 4399–4410.arxiv.org/abs/2003.02979
- [43]D. Strouse, K. R. McKee, M. Botvinick, E. Hughes, and R. Everett, “Collaborating with humans without human data,” in Advances in Neural Information Processing Systems, vol. 34, 2021, pp. 14 502–14 515.arxiv.org/abs/2110.08176
- [44]P. Stone, G. Kaminka, S. Kraus, and J. Rosenschein, “Ad hoc autonomous agent teams: Collaboration without pre-coordination,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 24, no. 1, 2010, pp. 1504–1509.DOI
- [45]M. A. Rahman, N. Hopner, F. Christianos, and S. V. Albrecht, “Towards open ad hoc teamwork using graph-based policy learning,” in Proceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 18–24 Jul 2021, pp. 8776–8786.arxiv.org/abs/2006.10412
- [46]C. Boutilier, R. Dearden, and M. Goldszmidt, “Stochastic dynamic programming with factored representations,” Artif. Intell., vol. 121, no. 1–2, p. 49–107, 2000.DOI
- [47]Y. Efroni, D. J. Foster, D. Misra, A. Krishnamurthy, and J. Langford, “Sample-efficient reinforcement learning in the presence of exogenous information,” in Conference on Learning Theory. PMLR, 2022, pp. 5062–5127.arxiv.org/abs/2206.04282
- [48]S. R. Sinclair, F. V. Frujeri, C.-A. Cheng, I. Marshall, H. D. O. Barbalho, J. Li, J. Neville, I. Menache, and A. Swaminathan, “Hindsight learning for mdps with exogenous inputs,” in International Conference on Machine Learning. PMLR, 2023, pp. 31 877–31 914.arxiv.org/abs/2207.06272
- [49]J. Wang, Y. Zhang, Y. Gu, and T.-K. Kim, “Shaq: Incorporating shapley value theory into multi-agent q-learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 5941–5954, 2022.arxiv.org/abs/2105.15013
- [50]C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem, “Brax - a differentiable physics engine for large scale rigid body simulation,” in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.arxiv.org/abs/2106.13281
- [51]I. Mordatch and P. Abbeel, “Emergence of grounded compositional language in multi-agent populations,” arXiv preprint arXiv:1703.04908, 2017.arxiv.org/abs/1703.04908
- [52]B. Ellis, J. Cook, S. Moalla, M. Samvelyan, M. Sun, A. Mahajan, J. Foerster, and S. Whiteson, “Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning,” Advances in Neural Information Processing Systems, vol. 36, pp. 37567–37593, 2023.
- [53]J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” arXiv preprint arXiv:1412.3555, 2014.arxiv.org/abs/1412.3555
- [54]R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine learning, vol. 8, no. 3, pp. 229–256, 1992.
- [55]J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, “Counterfactual multi-agent policy gradients,” in Proceedings of the AAAI conference on artificial intelligence, vol. 32, 2018.arxiv.org/abs/1705.08926
- [56]J. Shao, Y. Qu, C. Chen, H. Zhang, and X. Ji, “Counterfactual conservative q learning for offline multi-agent reinforcement learning,” Advances in Neural Information Processing Systems, vol. 36, pp. 77 290–77 312, 2023.arxiv.org/abs/2309.12696
- [57]S. Kim, G. Park, W. Kim, J. Jeon, S. Han, and Y. Sung, “Generalized per-agent advantage estimation for multi-agent policy optimization,” in Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2026.
- [58]Y. Park, S. Lee, and S. Han, “Center of gravity-guided focusing influence mechanism for multi-agent reinforcement learning,” arXiv preprint arXiv:2506.19417, 2025.arxiv.org/abs/2506.19417
- [59]J. Snoek, H. Larochelle, and R. P. Adams, “Practical bayesian optimization of machine learning algorithms,” Advances in neural information processing systems, vol. 25, 2012.arxiv.org/abs/1206.2944









