WorksComputer Science
Discovering Learning Rules that Shape Others
Abstract
In general-sum games, self-interested learners can reach outcomes that are poor for everyone, which has motivated methods that shape the opponent's learning. Methods like Learning with Opponent-Learning Awareness (LOLA), which differentiates through the opponent's learning process, and Model-Free Opponent Shaping (M-FOS), which uses model-free optimisation methods for long-horizon opponent shaping, have enabled mutual cooperation and exploitation of learners. However, these methods require access to the opponent's parameters, policy, rewards or learning rule during deployment. In single-agent reinforcement learning, update rules discovered by meta-learning have achieved state-of-the-art results (e.g. DiscoRL). These rules are separate from the policy they train and read only the agent's own experience. We hereby extend this framework to general-sum settings by introducing Discovered Update Rule for Opponent Shaping (DUOS). To the best of our knowledge, this is the first method to discover an update rule for an agent in general-sum games. DUOS meta-trains the rule to maximise the agent's expected return over its lifetime, and updates an inner network using the agent's own experience. We show that our discovered rule exhibits behaviour consistent with opponent shaping without reading opponent policies or parameters, and ranks 1st in normalised mean score and 1st in self-play among tournament entrants.
Introduction
Real-world multi-agent environments, from self-driving cars [55] to multi-agent large language model (LLM) systems [3, 50], can be framed as games ranging from zero-sum to fully cooperative. General-sum games cover the mixed-motive settings in between, such as social dilemmas [13, 36] or buyer-seller negotiations [39]. Naive self-interested learners within general-sum games often converge towards outcomes that are poor for everyone [20, 36]. For example, in Iterated Prisoner’s Dilemma (IPD) [5, 51], they often converge to mutual defection rather than mutual cooperation. Opponent shaping [20, 42] addresses this problem by influencing how others learn to achieve mutually beneficial outcomes.
Existing shaping methods leave a gap. Hand-derived rules such as LOLA [20] and POLA [67] differentiate through the opponent’s update, and require their parameters, payoffs or learning rules. Meta-learned shapers such as M-FOS [42], SHAPER [29] and COALA-PG [47] do not require a model of the opponent’s learning rule as SHAPER and COALA-PG act from their own history, but their learning algorithm is inside a black-box network, retrained each game. In single-agent reinforcement learning (RL), where the environment is stationary, rules discovered by meta-learning [48, 49] have achieved state-of-the-art results. Such rules also have practical advantages over a black-box learner: a discovered rule is not limited to updates derived by hand [49], is separate from the policy it trains, can transfer to different games without retraining, and its outputs can be read and interpreted directly. We therefore ask: can an update rule that learns from an agent’s own experience be discovered for general-sum games? However, learning update rules is difficult because learning is coupled, meta-gradients explode through co-learning, and opponents adapt, making the setting non-stationary. We hereby introduce Discovered Update Rule for Opponent Shaping (DUOS), a meta-reinforcement learning algorithm that discovers its own update rule by interacting with opponents, without needing their policies or parameters.
In this paper, we design the meta-objective to maximise the agent’s return over its lifetime. The rule reads the agent’s own experience and outputs policy updates, hence at deployment it does not require any privileged information: the agent never sees its opponents’ parameters, policies, learning rules or rewards. To avoid meta-gradients that explode through co-learning, the rule is trained with a likelihood-ratio meta-gradient, without differentiating through any learning. We then examine whether opponent shaping emerges from the rule without specifying it explicitly.
Although DUOS is trained to maximise lifetime return, this objective alone does not establish robustness to opponents trained specifically against its learned update rule. We therefore add a stage inspired by Policy Space Response Oracles (PSRO) [35] to meta-training: we train opponents to maximise their returns against the current rule, add them to the training population, and continue optimising the rule against the expanded population. We evaluate whether PSRO improves resistance to exploitation for DUOS and how it affects tournament results.
We evaluate DUOS in IPD, Iterated Matching Pennies (IMP), Chicken, Stag Hunt and Coin Game [37] in round-robin tournaments against established opponent-shaping methods, including methods that observe their opponents’ policies or parameters. Each Preprint. Under review. baseline is reproduced from its original settings, checked against results where possible, and flagged if it fails. We also observe DUOS self-play and zero-shot transfer of the IPD-trained rule to Chicken and Stag Hunt. Despite using only its own experience and learning an update rule from scratch, DUOS ranks 1st in normalised mean score across all games tested, and 1st in normalised mean score for self-play among tournament entrants.
Related Work
Opponent shaping: LOLA introduced the idea of opponent shaping by differentiating through the opponent’s learning step [20], which DiCE estimates from sampled trajectories [21]. Stable Opponent Shaping [38], COLA [61] and POLA [67] then further refined the idea, each requiring the opponent’s parameters (or in POLA’s opponent-modelling variant, a model of its policy fitted to its actions), as well as its payoffs and learning rule. LOQA [2] and Advantage Alignment [16] instead shape through estimates of the opponent’s values, while Reciprocator [68] and LIO [66] shape through rewards tied to the opponent’s returns. Best Response Shaping trains against an approximate best response to the agent instead [1]. All of these methods need opponent information each time they update their policy, whereas DUOS only needs it during meta-training.
One level above, meta-game methods learn to shape over many episodes. M-FOS trains a meta-policy where the action is the agent’s next policy [42]. For matrix games, M-FOS observes both players’ policies, and in Coin Game, it only observes the game state and both players’ rewards. SHAPER [29], COALA-PG [47], and in-context co-player inference [60] use recurrent or sequence-model agents that act with their own history. For all of these methods, the opponent-shaping behaviour and the learning algorithm are held in a black-box network. Meta-MAPG [30] and FURTHER [31] also take into account other agents’ learning, through meta-gradients and by influencing their long-run behaviour, respectively. The Good Shepherd [6] optimises a fixed policy that shapes learners without learning at deployment. DUOS updates a separate policy, but unlike M-FOS, it reads only its own experience and payoffs.
Learned update functions for games: COLA represents both players’ update functions with a neural network, and fits them to a consistency equation derived from LOLA [61]. Meta-Value Learning (MeVa) learns a value over the joint policy, and the gradient of the value leads to each player’s update [11]. MeVa extorts naive learners in IPD but reads both players’ parameters and is fitted to each game.
Update rules for centralised solvers have been discovered by meta-gradients or LLM-driven search [17, 40, 63] for games but not for agents. Other work instead learns optimisers or regret minimisers that converge towards equilibrium [8, 25, 56, 57, 58]. These target consistency per game or equilibrium computation, whereas DUOS is meta-trained for an agent’s own lifetime return against a population of opponents, reading only its experience.
Discovered reinforcement learning algorithms: In single-agent reinforcement learning, Xu et al. Xu et al. [65] use meta-gradients to tune hyperparameters such as the discount factor, and later to discover the objective itself [64]. LPG goes further and meta-learns an entire update rule across many lifetimes, generalising from toy environments to Atari [49]. DiscoRL [48] scales this approach to a large number of environments and discovers a rule that outperforms state-of-the-art hand-designed algorithms. Related work discovers objectives, drift functions and optimisers [10, 22, 27, 28, 34, 41]. Other work analyses the meta-gradients [18, 19, 59] that these algorithms utilise. Black-box meta-learners such as RL [15] keep the learning algorithm within the agent’s network, whereas these methods separate the rule from the agent it trains [33]. These methods form the many-shot branch of meta-reinforcement learning [7], where a learning algorithm trains an agent over a long lifetime. DUOS belongs to this branch and is applied to general-sum games, where opponents also learn.
Evolutionary game theory has asked which learning rules survive selection in repeated games, selecting from hand-designed learning rules or their parameters [12, 14, 44]. To our knowledge, DUOS is the first update rule discovered for an agent in general-sum games.
Background
We formalise the underlying game as a partially observable stochastic game (POSG) [24] that represents general-sum games and all games in this paper, written as a tuple of agents , states , joint actions , joint observations , an observation function , a transition function , rewards and a discount . In round , each agent receives an observation according to , samples an action , and receives a reward , where is the joint action and denotes the opponent.
In two-player iterated matrix games, actions and rewards . We assign a starting state to the first round of each match. Thereafter, the state records the joint action as (agent ’s action, opponent’s action), providing the state space:
A memory-one policy is conditioned on the previous joint action, with parameters , and plays action 0 given state with probability , where is the logistic function and represents the log-odds of action 0.
To assess the performance of a fixed pair of policies, we define an agent’s value as the discounted value function at the starting state and evaluate it as:
When the opponent also uses a memory-one policy, this value can be computed exactly [20]. This value provides the performance measure for our meta-training objective (Section 4.3). This value is only provided to DUOS during training, and DUOS adapts from sampled matches without receiving this value during deployment.
Update Rules
A lifetime is defined as a sequence of matches . Each match starts in , and the agent’s policy parameters do not reset in between. Unless stated otherwise, learners update between matches using the policies from the completed match. We define agent ’s policy in match as and their value as .
A naive learner (NL) [20] treats the opponent’s policy as fixed when computing its own update. In our experiments, an NL playing as agent , with logits and step size , takes one exact gradient step after each match,
using agent ’s policy before its own update. Opponent shaping instead chooses the agent’s own updates to steer the opponent’s learning, as LOLA and M-FOS do.
Meta-Learning
M-FOS casts opponent shaping as a meta-game: a meta-step is one match, the meta-state holds both players’ policies, the meta-reward is the agent’s return in the match, and the meta-action is its next policy. In matrix games, its meta-policy samples the agent’s next parameters given both players’ current policy parameters, .1
A related approach is to meta-learn an update rule. LPG [49] separates a rule with meta-parameters from the agent’s parameters and meta-trains to maximise the expected return of the policy obtained after a lifetime of updates , across a distribution of environments.
When meta-learning an update rule, meta-gradients through unrolled updates [65] can explode over long unrolls because the learning dynamics amplify small perturbations [46]. In multi-agent
settings, the problem worsens as the unrolls also pass through the opponent’s learning. Instead, we use the likelihood-ratio gradient [62], which does not differentiate through the game, the opponent’s learning or the agents’ parameter updates.
Discovered Update Rules for Opponent Shaping
In iterated matrix games, after every match DUOS updates the log-odds in every state via its parameters (Figure 1). Our key idea is to meta-learn a rule that reads only the agent’s own experience and payoff matrix. In this section, we will first describe the architecture of the meta-network and its outputs. Then, we will analyse how the inner network takes in the meta-network output to adjust its weights accordingly. Lastly, we will talk about how meta-training is optimised, and introduce robustness training against exploiters.
Figure 1. Overview of DUOS. After every match, the meta-network reads the agent’s experience and outputs log-odds changes , to which the inner network is fitted. The opponent’s policy, parameters, learning rule and payoffs are never observed. Meta-training perturbs with Gaussian noise and updates with a likelihood-ratio gradient whose baseline comes from lifetimes against the same opponent (§ 4).
Meta-Network
The update rule is learned by training the meta-network across lifetimes. The meta-network consists of a gated recurrent unit (GRU) [9] and a multi-layer perceptron (MLP) [4], whose weights are denoted by meta-parameters (§ 4.3). During deployment, the meta-network is frozen and the weights are fixed while DUOS uses the rule to adjust the parameters of its inner MLP after each match.
At the end of match , let denote the trajectory of states, own actions, opponent actions and own rewards observed by DUOS. For each state , the meta-network takes in the feature vector
where encodes the state, represents the fraction of rounds spent in that state, estimates the opponent’s smoothed frequency of action 0 during match , is the agent’s current probability of action 0, and is the agent’s normalised mean reward in that state. The GRU reads all five , the mean reward and the agent’s own payoff matrix after each match, and the MLP maps and the GRU’s memory to output a scalar that represents the change to the policy’s log-odds in state . The rule is also applied with the two actions relabelled and the outputs averaged, so the update does not depend on what the actions are called.
Inner Network
The inner network is an MLP with parameters that represents DUOS’s policy. With log-odds changes from the meta-network, we add them to the current log-odds and clip the results to obtain
To translate into changes in , we use damped Gauss-Newton steps [43]. A small change in the parameter shifts the log-odds by , where is the Jacobian
Hence, we require , where and . We take three steps of the approximate
where is a damping term that makes the matrix invertible, and and are recomputed each time. The best candidate is then kept as (Appendix D).
Optimising Meta-Training
Objective: We train the meta-network to look for meta-parameters whose rule produces a high value throughout a lifetime. A training lifetime has matches with an update after each match. We denote as the credited value (defined below) of the agent’s policy after updates, computed with Equation (1), where represents the initial policy. Meta-training then maximises the weighted average
where the opponent is sampled from the training mixture (Equation (4)), represents a lifetime sampled with training noise, and
Thus, averages the mean value of the policies after 0 to updates over the four horizons , such that the rule adapts quickly and does not fall off later.
Credit and opponents: The credited value is the agent’s value except in self-play, where both agents are credited with the mean value . The mean value is zero in IMP, so self-play does not add any gradient to the IMP rule. To train the base rule , we weight lifetimes against each opponent by:
Self is self-play, NL denotes a naive learner, and AllC and AllD are training-only opponents that almost always play action 0 and action 1, respectively (cooperation and defection in IPD).
Algorithm 1 Meta-training of DUOS
Require: Base mixture , lifetime length , PSRO rounds
Initialise meta-parameters
for do
if then
Train exploiter against
Update the meta-game
Compute exploiter mixture
Form training mixture
for each meta-training step do
for all training lifetimes, opponent do
Initialise policies and recurrent memory
for do
Evaluate (Eq. (1))
Play match
Sample
Update using (§ 4.2)
Update the opponent (NL: Eq. (2))
Evaluate
Compute returns and baselines
Estimate from (Eq. (5))
Update with Adam
Store
return
Estimator: Inspired by Gaussian perturbations used in evolution strategies [41, 53], we add Gaussian noise () to the log-odds changes during meta-training, and use the perturbed changes to update the inner network (§ 4.2). We train the meta-network using a likelihood-ratio gradient estimator [62], where each perturbed change contributes
Here, is the distribution of the perturbed changes given history , which consists of the agent’s payoff matrix , its policy , and trajectory in each match . The weighted value of the later policies is .
We sample four lifetimes from the same initial policies. For each lifetime, is the average of in the other three. This reinforces changes that beat the baseline and discourages worse ones.
We sum over each lifetime, including both agents in self-play, and average across training lifetimes using the opponent mixture . Adam [32] is then used to update the meta-network’s weights .
Training Against Exploiters
Inspired by PSRO [35] and the double oracle algorithm [45], each round we fix the meta-network and train an exploiter to maximise its own lifetime value against DUOS. The exploiter is able to observe DUOS’s current policy table and selects its policy after DUOS has updated.
We evaluate the earlier versions of the rule against all exploiters and use the resulting meta-game matrix to compute an exploiter mixture . This mixture minimises the highest value attained by any previous rule against it, i.e. each exploiter is scored by DUOS’s loss, which equals the exploiter’s own gain in zero-sum IMP but not in IPD. We continue maximising (Equation (3)) against a mixture of self-play, naive learners, the exploiter mixture and fixed strategies (Appendix § 13, Supplementary Material). Each round trains the rule for 200 meta-training steps. After rounds, we finalise and deploy the rule (Algorithm 1).
Experiments
Games. We evaluate on four iterated matrix games: Iterated Prisoner’s Dilemma (IPD), Iterated Matching Pennies (IMP), Chicken and Stag Hunt, and additionally on Coin Game. Table 1 shows the payoff matrices for the games.
| IPD | IMP | ||
| Cooperate | Defect | Heads | Tails |
| Chicken | Stag Hunt | ||
| Swerve | Straight | Stag | Hare |
Table 1. Payoff matrix for each game.
Coin Game. Coin Game [37] is a sequential social dilemma game that extends IPD to a grid world, and the rules are given in Appendix § 10 (Supplementary Material). Because Coin Game has too many states for per-state log-odds changes, we train a variant of DUOS for it: after each episode, its rule instead outputs a multiplier for each pickup scenario (which coin each player last collected), and the inner network takes one Adam step along , where is an advantage estimate.
Adam’s second moment uses the gradient with , so the step scales with . Hence, where , it is exactly like REINFORCE [62] (Table S13, Supplementary Material).
| Hyperparameter | Value |
| Steps per episode | 16 |
| Episodes per lifetime | 16 |
| Parallel games | 512 |
| Inner network hidden units | 32, 32 |
| Adam second moment | from the gradient |
| Advantage | return minus ridge baseline |
| Inner learning rate (Adam) | 0.05 |
| Return discount | 0.96 |
| Rule GRU units | 32 |
| Rule MLP hidden units | 64, 64 |
| Pickup scenarios | 9 |
| Noise on multipliers | 0.3 |
| Lifetimes per meta-training step | 256 |
| Opponent mixture (Self, learner, greedy, own-coin) | 0.4, 0.4, 0.1, 0.1 |
| Meta learning rate (Adam) | |
| Base meta-training steps | 500 |
| PSRO rounds meta-training steps | |
| PSRO mixture (Self, , greedy, own-coin, exploiters) | 0.28, 0.28, 0.07, 0.07, 0.30 |
| Exploiter GRU units (ES) | 16 |
| Exploiter MLP hidden units (PPO) | 64, 64 |
| Exploiter iterations (ES, PPO) | 150, 150 |
| learner’s learning-rate factor (training) | [0.5, 2] |
| Deployed multipliers (measured) | 0.99–1.00 |
| Training seeds | 3 |
Table S13. Hyperparameters of DUOS in Coin Game.
Deployment. At deployment, DUOS’s meta-network is frozen and no training noise is added. In each match, DUOS plays against its opponent for 20 rounds using its current policy and updates its inner network for the next match. The inner-network parameters and recurrent memory persist across matches and reset at the start of each lifetime. Unlike meta-training, tournaments score undiscounted 20-round matches over 50-match lifetimes and use a naive learner that learns faster than any seen in training.
Baselines. We run rigorous reproduction tests before using the baselines for comparison against our method. Each baseline uses its authors’ settings and is checked against their results (where comparable) before entering the tournament (Appendix § 12, Supplementary Material). We flag SHAPER [29], which falls slightly short in IPD. M-FOS passes and performs better in Coin Game than reported. Reciprocator passes with its released code but not with the paper’s published formula, so we use the released code, and its Coin Game check is also flagged. Advantage Alignment [16] and LOQA [2] did not pass and are excluded. All hyperparameters and protocols are in Appendix § 14 (Supplementary Material).
Evaluation. One tournament is held per game. DUOS, M-FOS [42], SHAPER [29] and NL participate in all five games. LOLA-OM [20] and POLA-OM* participate in the four matrix games, where POLA-OM* builds on Outer-POLA [67], which computes POLA’s proximal update from exact values for tabular policies. Like POLA’s opponent-modelling variant (POLA-OM), POLA-OM* replaces the opponent’s policy with a model fitted to its observed actions. IPD and IMP additionally include the history-based M-FOS variant (M-FOS-H) [42]. COALA-PG*, our pure-shaping reconstruction of COALA-PG [47], is included in IPD. Reciprocator [68] is included in IPD, Chicken, Stag Hunt and Coin Game.
In the matrix games, every pairing plays 128 lifetimes 50 matches 20 rounds. We evaluate five training seeds for DUOS, M-FOS, M-FOS-H and COALA-PG*, three for SHAPER, and none for LOLA-OM, POLA-OM*, Reciprocator and NL because they learn online and do not require training.
Information access differs between methods: DUOS uses only its own payoff matrix and observed match trajectories, whereas M-FOS, NL and Reciprocator receive the opponent’s policy, with memory-one approximations for recurrent opponents. Reciprocator also uses the opponent’s payoff matrix and 2,048 unscored practice matches of 20 rounds per update against the opponent. M-FOS-H reads its own policy and the 20 joint actions of the last match through a memory that resets after each match. LOLA-OM and POLA-OM* use observed actions and both payoff matrices and assume an NL update. SHAPER and COALA-PG* only use their own history.
Scoring. A method’s score is defined as its undiscounted mean reward per round (per step for Coin Game), averaged equally over its opponents, excluding self-play. To compare across varying payoff scales, we normalise the score of each game:
where is method ’s score and is the set of entrants in game . The normalised mean score averages over all games equally. In IMP, weights both player roles equally.
Zero-shot transfer. To observe effects of generalisation, DUOS plays Chicken and Stag Hunt zero-shot with its IPD rule as well as in self-play.
Results
Head-to-Head
Figure 3 and Table 2 compare the tournament scores across all five games. The full matchup results are given in Supplementary Material (Appendix B). DUOS ranks 1st by normalised mean score, both over the four matrix games and over all five games, where it plays zero-shot in Chicken and Stag Hunt, trained only in IPD. We only rank methods that have participated in all of the five games. Individually, DUOS ranks 1st in IPD, 2nd in IMP, 3rd in Chicken, 5th in Stag Hunt and 3rd in Coin Game.
Figure 3. Tournament scores after 50 matches (reward per round) or 16 episodes for Coin Game (reward per step). Self-play is excluded. Top row: DUOS trained on each game. Bottom row: DUOS trained on IPD playing zero-shot.
| Method | IPD | IMP | Chicken | Stag Hunt | Coin | Matrix mean | Five-game mean | Rank |
| DUOS (ours) | 1.000 | 0.841 | 0.829 | 0.727 | 0.680 | 0.849 | 0.815 [0.78, 0.85] | 1 |
| SHAPER | 0.631 | 1.000 | 0.032 | 0.902 | 1.000 | 0.641 | 0.713 [0.64, 0.78] | 2 |
| M-FOS | 0.583 | 0.646 | 0.000 | 1.000 | 0.844 | 0.557 | 0.614 [0.57, 0.66] | 3 |
| NL | 0.150 | 0.000 | 0.868 | 0.637 | 0.302 | 0.414 | 0.391 [0.35, 0.43] | 4 |
| LOLA-OM | 0.659 | 0.350 | 0.690 | 0.868 | – | 0.642 | – | – |
| POLA-OM* | 0.386 | 0.708 | 0.005 | 0.925 | – | 0.506 | – | – |
| M-FOS-H | 0.292 | 0.751 | – | – | – | – | – | – |
| Reciprocator | 0.159 | – | 1.000 | 0.000 | 0.000 | – | – | – |
| COALA-PG* | 0.000 | – | – | – | – | – | – | – |
Table 2. Normalised tournament scores. Each game’s scores are normalised in accordance with the scoring criteria (1 = best, 0 = worst). Matrix mean: four 2 × 2 matrix games. Five-game mean: all five games for the methods that play all five (95% seed-bootstrap intervals). Rank: by five-game mean. NL in Coin Game is a PPO learner [54]. Bold indicates the best result in each column.
Opponent shaping: Figure 2 shows that in IPD, both players are initially cooperating in about half of the rounds. By the end, NL has cooperated in 92.9% of the rounds and DUOS only in 16.4%. As such, DUOS’s mean reward per round during its final rounds has doubled compared to its initial rounds, and vice versa for NL. Separately, we replay each leftist with DUOS’s inner network frozen at its final policy from the first match, and show that NL cooperates in only 14.1% of the rounds (Table S27, Supplementary Material). Therefore, NL’s cooperation is produced by how DUOS’s policy changes over time, not by its final strategy alone, which is consistent with opponent shaping behaviour.
Figure 2. IPD learning trajectories over 50 matches. Left: rewards of DUOS against a naive learner (NL). Centre: joint-action frequencies against NL (DUOS’s action first, C: cooperation, D: defection). Right: mutual cooperation in self-play, within and across training seeds. Curves average five DUOS seeds. Shaded bands show 95% -intervals.
| Measure | Actual | Frozen | [95% CI] |
| NL cooperation, last match | 0.929 | 0.141 | [, ] |
| DUOS reward per round, last match | 4.387 | 1.371 | [, ] |
| DUOS reward per round, lifetime | 3.961 | 1.421 | [, ] |
| NL reward per round, lifetime | 0.900 | 1.145 | [, ] |
Table S27. Frozen-table control in IPD against NL, after 50 matches (5 seeds 128 lifetimes). Actual: DUOS’s recorded lifetimes. Frozen: the same NL, from the same first-match policy, against DUOS’s final (match-50) table held fixed from the first match. Exact expectations over 20-round matches. Mean s.e. over seeds. : frozen minus actual, with a paired 95% -interval (all 5 seeds have the same sign).
IPD: DUOS obtains the highest mean score in IPD despite losing most head-to-head matchups against opponents (6 out of 8, Table S26, Supplementary Material). Head-to-head matchups do not fully translate to final rankings in general-sum games because the scores count the reward each method obtains, not its margin over the opponent. For example, winning 2.10 to 1.30 scores less than mutual cooperation at 3.0 each. This reflects the meta-objective (Equation (3)), which maximises expected return rather than head-to-head margins. This finding mirrors Axelrod’s tournaments, where tit-for-tat won on total score without outscoring any single opponent [5].
| Opponent | IPD | IMP | Chicken | Stag Hunt | Coin |
| M-FOS | −0.268 (0/5) | +0.124 (4/5) | −0.195 (0/5) | +0.141 (5/5) | −0.216 (0/3) |
| M-FOS-H | −0.260 (0/5) | +0.026 (3/5) | – | – | – |
| SHAPER | −0.201 (0/5) | +0.019 (4/5) | −0.206 (0/5) | +0.120 (5/5) | −0.500 (0/3) |
| COAL-PG* | −0.017 (2/5) | – | – | – | – |
| LOLA-OM | −0.182 (0/5) | +0.313 (5/5) | −0.102 (0/5) | +0.092 (5/5) | – |
| POLA-OM* | −0.197 (0/5) | +0.004 (4/5) | −0.219 (0/5) | +0.065 (5/5) | – |
| NL (Coin: PPO NL) | +3.068 (5/5) | +0.385 (5/5) | +1.769 (5/5) | +0.869 (5/5) | +0.446 (3/3) |
| Reciprocator | +0.710 (5/5) | – | +0.647 (5/5) | +0.757 (5/5) | +0.935 (3/3) |
| Pairings won | 2/8 | 6/6 | 2/6 | 6/6 | 2/4 |
Table S26. Mean head-to-head margin of DUOS (its reward minus the opponent’s, per round, or per step in Coin Game) and, in parentheses, the seeds with a positive margin. Matrix games after 50 matches, Coin Game after 16 episodes. –: not in that game.
IMP: DUOS wins against every opponent in IMP, and is the only method that SHAPER does not beat (Table S2, Supplementary Material). This can be partly credited to PSRO training, which lowers DUOS’s exploitability from 0.376 to 0.144 and reduces how much M-FOS and SHAPER, which specialise in exploiting learners, gain against it (Table S24, Supplementary Material).
Table S2: IMP tournament after 50 matches. Each entry is the row method’s mean reward per round against the column method, averaged over the two seats (so the matrix is antisymmetric). Self-play is excluded from the mean score. DUOS’s diagonal entry pools same-seed and cross-seed pairings of its training seeds. DUOS’s row is in bold.
| Row vs. column | M-FOS | M-FOS-H | SHAPER | LOLA-OM | POLA-OM* | NL | DUOS | Mean score | Rank |
| M-FOS | – | −0.04 | −0.35 | 0.16 | −0.12 | 0.47 | −0.06 | 0.01 | 5 |
| M-FOS-H | 0.04 | – | −0.15 | 0.04 | −0.05 | 0.40 | −0.01 | 0.04 | 3 |
| SHAPER | 0.35 | 0.15 | – | 0.11 | 0.01 | 0.13 | −0.01 | 0.12 | 1 |
| LOLA-OM | −0.16 | −0.04 | −0.11 | – | −0.02 | −0.02 | −0.16 | −0.08 | 6 |
| POLA-OM* | 0.12 | 0.05 | −0.01 | 0.02 | – | 0.00 | 0.00 | 0.03 | 4 |
| NL | −0.47 | −0.40 | −0.13 | 0.02 | 0.00 | – | −0.19 | −0.20 | 7 |
| DUOS (ours) | 0.06 | 0.01 | 0.01 | 0.16 | 0.00 | 0.19 | 0.00 | 0.07 | 2 |
Table S2.
Table S24. What PSRO adds: the deployed rule against its pre-PSRO base (control) on the same 5 seeds. : after minus before, with a paired 95% -interval. (a) IMP head-to-head margin after 50 matches. (b) Exploitability: a fresh exploiter’s value against the rule minus the value that a reference strategy (IPD: tit-for-tat, IMP: the Nash player) obtains against the rule. Lower is better. IMP has separate rules for the two seats, so it has 10 runs (2 seats × 5 seeds). Its interval pairs the 5 seeds, each averaged over its two seats.
Chicken: Playing Chicken zero-shot, DUOS ranks 3rd with 1.52 per round, and 1st without SHAPER participating (1.81, Table S7, Supplementary Material). SHAPER and M-FOS play Straight in Chicken and DUOS does not yield to them, so DUOS ends up mostly crashing, which lowers its score. This might be due to SHAPER and M-FOS training only against naive learners, and thus learning mainly to exploit. Consistent with this, SHAPER and M-FOS have the two lowest Chicken self-play scores (Table 3). The Chicken result suggests that learned rules transfer across different settings, as also observed in single-agent reinforcement learning [48], although a rule trained on Chicken itself still scores higher (2.105 vs. 1.810, Table S25, Supplementary Material).
| Method | IPD | Chicken | Stag Hunt | Coin | Matrix mean | Four-game mean | Rank |
| DUOS (ours) | 1.000 | 1.000 | 0.883 | 1.000 | 0.961 | 0.971 [0.969, 0.973] | 1 |
| Reciprocator | 0.621 | 0.804 | 0.000 | 0.954 | 0.475 | 0.595 [0.590, 0.601] | 2 |
| NL | 0.031 | 0.848 | 0.101 | 0.900 | 0.327 | 0.470 [0.465, 0.476] | 3 |
| M-FOS | 0.005 | 0.073 | 0.970 | 0.774 | 0.349 | 0.456 [0.396, 0.499] | 4 |
| SHAPER | 0.327 | 0.000 | 1.000 | 0.000 | 0.442 | 0.332 [0.274, 0.364] | 5 |
| LOLA-OM | 0.428 | 0.516 | 0.554 | – | 0.499 | – | – |
| POLA-OM* | 0.058 | 0.079 | 0.782 | – | 0.306 | – | – |
| COALA-PG* | 0.091 | – | – | – | – | – | – |
| M-FOS-H | 0.000 | – | – | – | – | – | – |
| M-FOS (meta-self-play)‡ | 1.028 | – | – | 1.248 | – | – | – |
Table 3. Normalised self-play scores: each method’s mean own reward against a same-seed copy of itself per round, averaged over a lifetime of 50 matches (per step over 16 episodes in Coin Game). Each game’s scores are normalised in accordance with the scoring criteria. IMP is omitted because it is zero-sum. ‡Outside the tournament (two learners of each run play each other).
| Method | Chicken without | Chicken with | Stag Hunt without | Stag Hunt with |
| M-FOS | 1.60 (6) | 1.34 (7) | 3.61 (1) | 3.67 (1) |
| SHAPER | – | 1.35 (5) | – | 3.58 (3) |
| LOLA-OM | 1.78 (2) | 1.49 (4) | 3.52 (3) | 3.55 (4) |
| POLA-OM* | 1.60 (5) | 1.34 (6) | 3.54 (2) | 3.60 (2) |
| NL | 1.64 (4) | 1.53 (2) | 3.24 (5) | 3.34 (6) |
| Reciprocator | 1.70 (3) | 1.56 (1) | 2.68 (6) | 2.76 (7) |
| DUOS (ours) | 1.81 (1) | 1.52 (3) | 3.42 (4) | 3.42 (5) |
Table S7. Chicken and Stag Hunt scores (rank) after 50 matches, without SHAPER (original field) and with SHAPER.
Table S25. Zero-shot transfer: the IPD rule played zero-shot compared with rules trained in Chicken and Stag Hunt, each entered in place of it in the same field, after 50 matches (5 seeds, in the field without SHAPER of Table S7, with the trained rules using the IPD recipe, not retuned). : trained minus zero-shot, with a paired 95% -interval. Seeds: seeds on which the trained rule is better. Self-play return: same-seed pairs from the same tournament runs. Exploitability is as in Table S24, with tit-for-tat as the reference. A higher score can come with a lower rank because the other entrants’ scores also change: the Chicken-trained rule yields to Straight, which raises M-FOS’s score from 1.60 to 2.31 and POLA-OM*’s from 1.60 to 2.20.
Stag Hunt: DUOS plays Stag Hunt zero-shot and ranks 5th in the tournament with a score of 3.42, only 0.25 behind M-FOS in 1st. Yet, DUOS wins every head-to-head matchup here, the opposite of its IPD result. In Stag Hunt, a player out-earns its opponent only by playing Hare (3) against Stag (0), giving up mutual Stag (4), on which the other methods coordinate. Most of the gap is the cost of transferring from IPD: in the tournament without SHAPER, when trained in Stag Hunt itself, DUOS scores 0.25 more and ranks 2nd instead of 4th (Table S25, Supplementary Material).
Coin Game: DUOS ranks 3rd of five with 0.048 per step (Table S5, Supplementary Material). Head to head, DUOS out-earns NL and Reciprocator but loses to M-FOS and SHAPER. DUOS learns a rule that is akin to REINFORCE [62]: its multiplier is always around 1.00, and thus DUOS learns like a plain () learner rather than shaping its opponent. This is likely a limit of the rule form, which can only rescale the REINFORCE step by a multiplier in .
Table S5: Coin Game tournament after 16 episodes: the row method’s mean reward per step against each column method. DUOS’s self-play (diagonal) entry pools same-seed and cross-seed pairings of its training seeds and is excluded from the score. NL is M-FOS’s PPO naive learner. M-FOS (stronger than in its paper) and Reciprocator are flagged (Appendix § 12).
| Row vs. column | M-FOS | SHAPER | NL (PPO) | Reciprocator | DUOS | Mean score | Rank |
| M-FOS | – | −0.298 | 0.411 | 0.567 | 0.097 | 0.194 | 2 |
| SHAPER | 0.032 | – | 0.534 | 0.643 | 0.127 | 0.334 | 1 |
| NL (PPO) | −0.447 | −0.754 | – | 0.259 | −0.216 | −0.289 | 4 |
| Reciprocator | −0.618 | −0.865 | −0.272 | – | −0.482 | −0.559 | 5 |
| DUOS (ours) | −0.119 | −0.373 | 0.230 | 0.454 | 0.014 | 0.048 | 3 |
Table S5.
Self-Play
A widely adopted algorithm will often face copies of itself. M-FOS reaches cooperation in self-play through a meta-self-play curriculum [42]. DUOS is trained against a population that includes 40% self-play. In IPD, DUOS reaches mutual cooperation (Figure 2, right), with both players cooperating in 93.4% of rounds, eventually reaching 99.8% during the final rounds. We pair different training seeds with each other, and also observe similar trajectories, showing that cooperation extends beyond exact same copies. DUOS therefore behaves differently against NL (Figure 2) than against another copy of itself: its changing policy makes NL more cooperative, which it then exploits, but it reciprocates mutual cooperation in self-play. These trajectories are consistent with opponent shaping.
Tournaments: DUOS clearly outperforms all baselines in self-play score (Table 3) and ranks 1st in 3 out of 4 games against methods tested in all four (deployed zero-shot in Chicken and Stag Hunt). Outside of the tournaments, M-FOS (meta-self-play) scores higher than DUOS, which we suspect stems from the different training mixtures. Hence, we train an exploratory self-play variant of DUOS with one rule shared by both copies and compare it with M-FOS (meta-self-play) in Figure 4 (Appendix § 15).
Figure 4. Learning curves of the variants trained for self-play. (a) IPD: mean over 5 seeds (s.e. ≤ 0.009). (b) Coin Game (3 seeds), reward per step. (c) DUOS (self-play) in the three matrix games with normalised reward: reward per round divided by the game’s mutual-cooperation payoff (3, 3 and 4), s.e. ≤ 0.007.
DUOS (self-play) vs. M-FOS (meta-self-play): The two methods are trained differently. DUOS (self-play) is trained only in self-play from the start with no naive learners, fixed strategies or PSRO. M-FOS (meta-self-play) instead follows its meta-self-play curriculum, where two separately initialised learners play each other with increasing probability, and otherwise play against naive learners. DUOS (self-play) is trained on 256,000 self-play lifetimes in IPD and 243,200 in Coin Game, whereas M-FOS (meta-self-play) is trained on around 2.8 million and around 900 lifetimes, respectively. In IPD, M-FOS (meta-self-play) hears mutual cooperation from the first match, and is fully cooperative with sampling noise off. DUOS (self-play) starts from a fresh random policy (with a trained meta-network) and fully converges by match 9–10. In Coin Game, the reward of M-FOS (meta-self-play) stays roughly constant, while DUOS (self-play) stays near zero for the first few episodes and then increases, overtaking M-FOS (meta-self-play) after episode 10. Over the lifetime, both methods are level (0.075 vs. 0.073 per step). Trained in each game, DUOS (self-play) also converges to mutual cooperation in Chicken and Stag Hunt, akin to its behaviour in IPD.
Conclusion
In single-agent reinforcement learning, discovered algorithms [22, 23, 28] have achieved state-of-the-art results [48]. We introduce DUOS, to our knowledge the first method to discover an update rule for an agent in general-sum games. Without reading the opponent’s policies or parameters, DUOS ranks 1st in normalised mean score across all games, and 1st for self-play among tournament entrants. In IPD, it exploits a naive learner while still cooperating with a copy of itself, consistent with opponent shaping. The IPD-trained rule also transfers zero-shot to two other matrix games, Chicken and Stag Hunt, with competitive scores, as discovered rules also do in single-agent settings [48]. Meta-training against exploiters (PSRO) makes the rule harder to exploit, lowering its exploitability in IMP by more than half. In Coin Game, DUOS’s learned rule reduces to REINFORCE () and ranks 3rd of 5 entrants. In self-play, DUOS (self-play) achieves full mutual cooperation in IPD, Chicken and Stag Hunt within 12 matches from a fresh policy, and eventually obtains more reward per step than M-FOS (meta-self-play) after 10 episodes in Coin Game. Our work has limitations. Each rule is trained per game, and in Chicken, a rule trained on
Chicken specifically still scores higher than the IPD rule played zero-shot. In Coin Game, shaping is likely limited by the rule form, which can only rescale the REINFORCE step. Future work could investigate meta-training one rule across several games, as DiscoRL’s discovered rule improved with more compute and more diverse training environments [48]. We could also investigate different rule forms and training settings that shape opponents and generalise to different sequential social dilemma games [36] such as Harvest [26, 52] and Cleanup [26].
References
Supplementary Material
Coin Game rules
We use the two-coin version of Coin Game [37] implemented in pax by SHAPER [29]. There are two players, Red and Blue, playing on a grid whose edges wrap around (a torus). The grid holds one red and one blue coin, owned by Red and Blue respectively. The players and coins can share the same cells. At each step, both players move simultaneously and choose one of the five actions: right, left, up, down or stay. A player collects a coin by moving to or staying on the cell the coin is occupying. Collecting any coin gives the collector . If the coin belongs to the other player, the other player also receives . If both players collect the same coin, both receive and its owner also receives (net ). A collected coin reappears on a uniformly random cell. An episode starts with the four positions drawn independently and uniformly over nine cells. The episode lasts 16 steps, of which 15 are paid: the 16th step applies the moves and resets the board, giving both players zero reward. Throughout the paper, a Coin Game reward per step is an episode’s total reward divided by its 15 paid steps.
Tournament results
Tables S1, S2, S3, S4 and S5 give every head-to-head result of the tournaments in Table 2. SHAPER was trained for Chicken and Stag Hunt and added to those fields after the first scores had been computed. Table S6 gives the normalised scores without it, and Table S7 compares the fields without and with it. Table S8 replaces DUOS with its pre-PSRO base rule in the same fields, with the same random draws.
Table S1: IPD tournament after 50 matches. Each entry is the row method’s mean reward per round against the column method. Self-play is excluded from the mean score. DUOS’s diagonal entry pools same-seed and cross-seed pairings of its training seeds. DUOS’s row is in bold.
| Row vs. column | M-FOS | M-FOS-H | SHAPER | COALA-PG* | LOLA-OM | POLA-OM* | NL | Reciprocator | DUOS | Mean score | Rank |
| M-FOS | – | 1.05 | 1.42 | 1.25 | 1.66 | 1.43 | 3.88 | 2.61 | 2.00 | 1.91 | 4 |
| M-FOS-H | 1.04 | – | 1.38 | 1.26 | 1.36 | 1.11 | 3.13 | 2.45 | 2.09 | 1.73 | 6 |
| SHAPER | 1.35 | 1.29 | – | 1.34 | 1.92 | 1.61 | 3.53 | 2.39 | 2.13 | 1.94 | 3 |
| COALA-PG* | 1.18 | 1.13 | 1.36 | – | 1.46 | 1.36 | 2.02 | 2.18 | 1.62 | 1.54 | 9 |
| LOLA-OM | 1.53 | 1.31 | 1.77 | 1.37 | – | 1.34 | 2.90 | 2.72 | 2.76 | 1.96 | 2 |
| POLA-OM* | 1.36 | 1.09 | 1.59 | 1.36 | 1.39 | – | 3.75 | 2.28 | 1.48 | 1.79 | 5 |
| NL | 1.08 | 1.83 | 1.57 | 1.93 | 2.60 | 1.26 | – | 1.93 | 0.90 | 1.64 | 8 |
| Reciprocator | 1.55 | 1.58 | 1.83 | 1.55 | 2.47 | 1.24 | 1.08 | – | 1.82 | 1.64 | 7 |
| DUOS (ours) | 1.73 | 1.83 | 1.93 | 1.61 | 2.58 | 1.28 | 3.96 | 2.53 | 2.93 | 2.18 | 1 |
Table S1.
Table S3: Chicken tournament after 50 matches, with SHAPER: the row method’s mean reward per round against each column method. DUOS plays its IPD rule zero-shot. Its self-play (diagonal) entry pools same-seed and cross-seed pairings of its training seeds and is excluded from the score.
| Row vs. column | M-FOS | SHAPER | LOLA-OM | POLA-OM* | NL | Reciprocator | DUOS | Mean score | Rank |
| M-FOS | – | 0.06 | 0.38 | 0.15 | 3.91 | 3.23 | 0.30 | 1.34 | 7 |
| SHAPER | 0.26 | – | 0.23 | 0.11 | 3.92 | 3.29 | 0.27 | 1.35 | 5 |
| LOLA-OM | 0.11 | 0.06 | – | 0.40 | 3.18 | 2.65 | 2.54 | 1.49 | 4 |
| POLA-OM* | 0.09 | 0.03 | 0.51 | – | 3.87 | 2.83 | 0.73 | 1.34 | 6 |
| NL | 0.98 | 0.98 | 2.34 | 1.04 | – | 2.08 | 1.74 | 1.53 | 2 |
| Reciprocator | 0.82 | 0.82 | 2.36 | 0.81 | 2.69 | – | 1.83 | 1.56 | 1 |
| DUOS (ours) | 0.11 | 0.07 | 2.44 | 0.51 | 3.51 | 2.48 | 2.92 | 1.52 | 3 |
Table S3.
Table S4: Stag Hunt tournament after 50 matches, with SHAPER: the row method’s mean reward per round against each column method. DUOS plays its IPD rule zero-shot. Its self-play (diagonal) entry pools same-seed and cross-seed pairings of its training seeds and is excluded from the score.
| Row vs. column | M-FOS | SHAPER | LOLA-OM | POLA-OM* | NL | Reciprocator | DUOS | Mean score | Rank |
| M-FOS | – | 3.97 | 3.77 | 3.89 | 3.93 | 2.91 | 3.57 | 3.67 | 1 |
| SHAPER | 3.97 | – | 3.63 | 3.91 | 3.82 | 2.82 | 3.34 | 3.58 | 3 |
| LOLA-OM | 3.83 | 3.68 | – | 3.51 | 3.74 | 3.07 | 3.47 | 3.55 | 4 |
| POLA-OM* | 3.92 | 3.93 | 3.53 | – | 3.76 | 2.91 | 3.57 | 3.60 | 2 |
| NL | 3.94 | 3.85 | 3.73 | 3.75 | – | 2.26 | 2.49 | 3.34 | 6 |
| Reciprocator | 3.33 | 3.12 | 3.15 | 2.95 | 1.92 | – | 2.06 | 2.76 | 7 |
| DUOS (ours) | 3.71 | 3.46 | 3.56 | 3.63 | 3.36 | 2.81 | 3.79 | 3.42 | 5 |
Table S4.
Table S6: Normalised tournament scores in which SHAPER plays only IPD, IMP and Coin Game. Columns are as in Table 2, without the rank column. SHAPER was only added to Chicken and Stag Hunt after these scores were computed.
| Method | IPD | IMP | Chicken | Stag Hunt | Coin | Matrix mean | Five-game mean |
| DUOS (ours) | 1.000 | 0.841 | 1.000 | 0.787 | 0.680 | 0.907 | 0.862 [0.84, 0.88] |
| M-FOS | 0.583 | 0.646 | 0.000 | 1.000 | 0.844 | 0.557 | 0.614 [0.57, 0.66] |
| NL | 0.150 | 0.000 | 0.196 | 0.596 | 0.302 | 0.236 | 0.249 [0.21, 0.29] |
| LOLA-OM | 0.659 | 0.350 | 0.842 | 0.905 | – | 0.689 | – |
| POLA-OM* | 0.386 | 0.708 | 0.040 | 0.920 | – | 0.514 | – |
| SHAPER | 0.631 | 1.000 | – | – | 1.000 | – | – |
| M-FOS-H | 0.292 | 0.751 | – | – | – | – | – |
| Reciprocator | 0.159 | – | 0.502 | 0.000 | 0.000 | – | – |
| COALA-PG* | 0.000 | – | – | – | – | – | – |
Table S6.
| Game | With PSRO | Pre-PSRO | Difference | |
| IPD | 50 | 2.182 (1) | 2.162 (1) | +0.020 [−0.151, +0.191] |
| 10 | 1.945 (1) | 1.922 (3) | +0.023 [−0.024, +0.069] | |
| IMP | 50 | 0.073 (2) | 0.029 (4) | +0.043 [+0.022, +0.065] |
| 10 | 0.054 (2) | 0.031 (2) | +0.023 [+0.009, +0.037] | |
| Chicken | 50 | 1.520 (3) | 1.531 (2) | −0.011 [−0.055, +0.034] |
| 10 | 1.362 (2) | 1.367 (2) | −0.005 [−0.032, +0.023] | |
| Stag Hunt | 50 | 3.422 (5) | 3.244 (6) | +0.178 [−0.037, +0.392] |
| 10 | 3.051 (6) | 2.915 (6) | +0.136 [+0.060, +0.213] |
Table S8. Mean score (rank in its own field) of DUOS with and without PSRO after matches, and their difference (with PSRO minus pre-PSRO) paired by seed with a 95% -interval. Bold: the interval excludes 0. Chicken and Stag Hunt: fields with SHAPER.
Baseline reproduction
Each baseline uses its authors’ published settings and is adapted to our settings where necessary. Appendix § 14 lists every setting, and † marks each setting that differs from the paper’s for a reason other than our tournament format. Before a baseline entered our tournaments, we checked our implementation against its own paper by reproducing it with different seeds against a reported result where the paper has one in a comparable setting, and otherwise against the paper’s settings. Each check and its pass rule were written down before the checks were run, except the post hoc COALA-PG* diagnostic described below. Table S9, Table S10 and Table S11 give every check, its target, our result and the outcome. Below, we report the flagged and excluded baselines.
Table S9: Baseline fidelity in the matrix games (I): reproduction checks against each baseline’s paper. Target: the target or bar set before the check, and n/r means no target.
| Baseline | Check | Target | Ours | Outcome | Tournament |
| M-FOS, IPD | Table 8 (authors’ payoffs), 2 seeds | M-FOS −0.51 ± 0.1, NL −2.14 ± 0.2 | M-FOS −0.476, −0.474, NL −2.174, −2.176 | pass | entrant |
| M-FOS, IMP | Table 5, authors’ code (B) vs ours at lr 0.002 (C) | 0.20, B ≥ 0.18, | B 0.194, C 0.194 | pass | entrant (lr 0.002) |
| M-FOS, Chicken | Table 12 | M-FOS above NL | M-FOS 3.94, NL 0.99 (5/5 seeds) | pass | entrant |
| M-FOS, Stag Hunt | no paper result | n/r | M-FOS 3.93, NL 3.95 | – | entrant |
| M-FOS, meta-self-play | self-play, | n/r | IPD 2.988, IMP 0.012 | – | self-play only |
| SHAPER, IPD | recovery, 3 seeds | 3.15 | 2.967 | fail | entrant (flagged) |
| SHAPER, IMP | recovery, both seats | 0.34, 0.52 | 0.719, 0.757 | pass | entrant |
| SHAPER, Chicken and Stag Hunt | training health | n/r | 6/6 runs healthy | pass | entrant |
Table S9.
Table S10: Baseline fidelity in the matrix games (II), as in Table S9. Counts are seeds meeting the condition, and is the payoff-range factor.
| Baseline | Check | Target | Ours | Outcome | Tournament |
| POLA-OM* | POLA’s steps in our payoff units | , , | – | entrant | |
| LOLA-OM | LOLA’s Table 3 (exact LOLA, authors’ payoffs) | n/r (paper −1.06, −1.98) | LOLA −1.04, NL −1.98 | match | entrant |
| POLA-OM* | Outer POLA vs authors’ code, Figure 3 and Table 3 | n/r | within , both reproduced | match | entrant |
| Reciprocator, IPD | shaping gate, 8 seeds | ≥ 5/8 | paper formula 0/8, released code 8/8 | pass (code) | entrant (code) |
| COALA-PG*, pure shaping | Fig. 5A | return ≥ 0.40 | 1.376, 1.401 | pass | entrant (IPD) |
| COALA-PG*, mixed pool | Fig. 5C | recovery ≥ 0.5 | 0.035 | fail | – |
| LOQA | Figure 1 policy, 3 seeds | ≥ 2/3 | paper 1/3, code 0/3 | fail | excluded |
| Advantage Alignment | Figure 1(b) policy, 3 seeds | P(C | state) bars | LOLA payoffs 0/3, shifted 0/3 | fail | excluded |
Table S10.
Table S11: Baseline fidelity in Coin Game, as in Table S9. M-FOS returns are in the authors’ units, summed over 240 paid steps (M-FOS / NL).
| Baseline | Check | Target | Ours | Outcome | Tournament |
| M-FOS, authors’ code | updates 951–1000, 3 seeds | M-FOS , NL in seeds | 70.89 / −75.00, 2/3 seeds | pass (stronger) | – |
| M-FOS, pax | same, not gated | n/r | M-FOS 113.55, 105.21, 75.94 | – | entrant (flagged) |
| M-FOS, meta-self-play | authors’ code, self-play, 3 seeds | in seeds | 8.74, 14.44, 22.42 | fail | – |
| M-FOS, meta-self-play pax | self-play, 3 seeds, not gated | n/r | 29.95, 25.98, −3.48 | – | self-play only |
| SHAPER | paper’s ES budget | n/r | 3/3 seeds completed | – | entrant |
| Reciprocator | NL own-coin rate, 4 seeds | in seeds, mean , controls | 0.617 (4/4), controls 0.473, 0.531 | uninformative | entrant (flagged) |
| PPO NL | M-FOS’s tbl:20 | n/r | settings match | – | entrant |
Table S11.
Flagged baselines. SHAPER passes its checks in IMP but falls slightly short in IPD (three-seed mean 2.97 against a bar of 3.15, and one of three seeds passed). The paper reports SHAPER only in its own format, so this check asks whether training in our format recovers at least half of the reward that a model trained in the paper’s format loses in ours: the bar is the midpoint of that model’s reward in our format (2.67) and in its own (3.63). The IMP bars are set in the same way. We enter the model trained with the budget of the authors’ configuration files (Table S16) and flag its IPD result. M-FOS passes its Coin Game check on two of three seeds and its returns exceed the paper’s, and is flagged as stronger than in its paper. The check used the authors’ four-action game. The entrant, trained in the same way in our five-action game and not gated, returns more still. Reciprocator fails our checks with its paper’s formula but passes with its released code, which we enter. Its Coin Game check is uninformative, because in one of the two control runs, two naive learners without a Reciprocator also exceeded the bar. Its shaping also needs more episodes than our 16-episode lifetimes (in the paper the effect appears after about 25 episodes), so it is flagged in Coin Game. We also scored Coin Game without it, and the order of the other four entrants is unchanged (SHAPER 0.231, M-FOS 0.070, DUOS −0.087, PPO naive learner −0.472).
| Hyperparameter | Value |
| GRU units | 16† |
| ES population | 1,000 |
| Generations | 1,000† |
| ES learning rate | 0.1† |
| Perturbation scale | 0.04 |
| Opponents per candidate | 10 |
| Matches per lifetime (trial) | 50 |
| Rounds per match | 20 |
| NL learning rate | 1 |
| NL epochs minibatches | |
| NL discount | 0.96 |
| Training seeds | 3 |
Table S16. Hyperparameters of SHAPER in the matrix games. †The paper’s Table 11 (IPD) lists a GRU of 25 units, 5,000 generations and an ES learning rate of 0.01. We use the authors’ configuration files (whose GRU size matches the parameter count in the paper’s Table 20), as in Coin Game (Table S17).
| Hyperparameter | Value |
| Linear and GRU units | 16, 16 |
| ES population | 4,000 |
| Generations | 3,000 |
| ES learning rate | 0.1{sup}†` |
| Perturbation scale | 0.04 |
| L2 penalty | 0.1 |
| Parallel games | 512 |
| Episodes per lifetime (trial) | 16 |
| NL learning rate | 0.005 |
| NL epochs minibatches | 2 8 |
| Training seeds | 3 |
Table S17. Hyperparameters of SHAPER in Coin Game. {sup}†`The paper’s table lists 0.01. We use the authors’ configuration file.
COALA-PG⁎. Our reconstruction of COALA-PG passes the check on the pure-shaping experiment of its paper (Fig. 5A), learning faster than the paper’s curve. The entrants use one optimiser step per iteration, which a speed check found to match the paper’s pace. It does not reproduce the mixed-pool training (Fig. 5C) either in the check or in the post hoc diagnostic with the paper’s optimiser step count. COALA-PG⁎ therefore plays only IPD, trained in the pure-shaping setting. The asterisk marks it as our reconstruction. There is no released code for the method.
Excluded baselines. Advantage Alignment has no released code for its IPD experiment: the loss in the authors’ released code is written for other environments, and a part of it did not reproduce the paper’s IPD result. We rebuilt the method on the code base of LOQA, with the loaded-aa loss and the paper’s IPD settings, and tried to reproduce Figure 1(b) from the paper on 3 seeds with the IPD payoffs used by LOLA, POLA and LOQA, since the paper does not print its own, and on 3 seeds with every payoff shifted by +1.5. None of the seeds passed (0 of 6). LOQA reproduced Figure 1 on 1 of 3 seeds with the settings in its paper and on 0 of 3 with those in its released code. Both methods are therefore excluded from the tournaments.
Other variants. M-FOS-H, our matrix-game version of the history-based M-FOS of the paper’s Coin Game experiments, has no paper result to check against. The M-FOS meta-self-play curriculum fails its Coin Game check with the authors’ code (1 of 3 seeds). The Coin Game version in the self-play comparison (Table 3 and Figure 4) was trained in the same way on our five-action game and was not gated (Appendix § 15).
Implementation details of DUOS
This section gives the details of DUOS that Section 4 leaves out. Tables S12 and S13 list its settings.
Rule inputs. After match , the features of state are computed from the agent’s own record of the match. The feature is a one-hot code of . The visit share is , where is the number of rounds played in . The opponent’s smoothed frequency of action 0 is , where is the number of those rounds in which the opponent played action 0. The agent’s probability of action 0 before the update is . Finally, is the agent’s mean reward in divided by , the largest absolute payoff in the agent’s payoff matrix ( in IPD, 1 in IMP and 4 in Chicken and Stag Hunt). A state that was not visited has , and , and still receives a change.
Meta-network. The GRU has 32 units and one memory shared by all states. After each match, it reads the five feature vectors , the match’s mean reward divided by and the agent’s payoff matrix (50 inputs). For each state, one shared MLP with two hidden layers of 64 units maps and the GRU’s memory to a change in . All its units, including the output, are tanh. The meta-network has 14,881 parameters. Its MLP weights and biases are drawn from and its GRU weights from . The GRU biases start at 0.
Action symmetry. The rule is applied twice to every match: once as observed, and once with the two actions relabelled. Relabelling exchanges the states and and the states and , swaps the rows and columns of the payoff matrix and negates the log-odds. Each view has its own GRU memory, and the change from the relabelled view is mapped back (exchanging the states and negating) and averaged with the other, which gives . The update is therefore the same whichever action is called 0.
Inner network. The inner network has two hidden layers of 32 tanh units and 1,314 parameters. Its input codes the state with five indicators (the start state, and one indicator per action for the agent’s and the opponent’s last actions), and its two outputs give the log-odds . At the start of each lifetime, the network is fitted to initial log-odds drawn independently from for each state.
Fitting the targets. After match , the targets are (in training, with the perturbed change in place of ). The network is fitted to them with damped Gauss–Newton steps (Section 4.2). Each step is , where , the row of for state is , and . Equivalently, minimises . The weight is the number of rounds played in state $s, or 1 if was not visited. The damping is tiny: $10^{-6}$ times the median of the five states. So is close to the smallest parameter change that matches the targets to first order. Two more steps recompute , and at the new parameters. The fit then keeps as whichever of five candidates has the smallest : no change, the parameters after each of the three steps, or the midpoint between and the parameters after the third step.
Meta-training. Each meta-training step draws 32 pairs of initial policies for every opponent type of the mixture and plays each pair four times with independent match and noise draws, giving 128 lifetimes per type, whose average gradients are weighted by . The base rule is trained for 800 meta-training steps, and the gradient’s global norm is clipped to 1 before each Adam step (learning rate ). In IMP, the matching and mismatching seats have separate rules. In IMP self-play, a copy of the seat’s rule plays the other seat, and the credited mean value is zero.
Gradient estimate. The four replays share the initial policies and the opponent (in PSRO, also the drawn naive learner, exploiter or fixed strategy). This makes the leave-one-out baseline a close one. Because their match and noise draws are independent, does not depend on the lifetime’s own noise, so the estimator stays unbiased. The noise in , with and , is added to the changes of all five states before the clip. Hence , where holds the agent’s payoff matrix and its policy and trajectory in each match . The values , , are computed exactly before each match and after the last update. The gradient passes through both GRU memories over the whole lifetime, but not through the game, the opponent’s update or the fit.
Naive learners. The naive learners in training take one exact gradient step (Equation (2)) on their value of Equation (1) after each match and read DUOS’s exact table. The step size is 3 in the base stage and is drawn log-uniformly from for each opponent in PSRO. The tournament’s naive learner steps on the unnormalised value with step size 1, which is on , so it learns faster than any naive learner seen in training. AllC and AllD have log-odds 8 and in every state.
Training against exploiters. Each PSRO round trains a fresh exploiter against the latest rule without noise. The exploiter is an MLP with two hidden layers of 32 tanh units (1,605 parameters). Before each match , it reads DUOS’s current table (privileged information), its own previous table and , and outputs its own table for the match. Its inputs are centred at 1/2 and scaled by 4, and its outputs are scaled by 4. Its first previous table is with . It is trained by antithetic evolution strategies to maximise its own horizon-weighted value : 150 iterations of 64 directions, each evaluated on 32 lifetimes with common random numbers, with perturbation scale 0.1 and Adam (learning rate 0.03).
Meta-game and training mixture. The meta-game matrix holds DUOS’s horizon-weighted value without noise, estimated on 512 common lifetimes, for each earlier rule against each exploiter . The exploiter mixture , which minimises the largest value of any earlier rule against it, is found by linear programming in both IPD and IMP. The rule is then trained for 200 meta-training steps against the mixture of self-play (0.30), naive learners (0.25), exploiters drawn from (0.30) and fixed strategies (0.15). The fixed strategies are drawn uniformly from tit-for-tat, generous tit-for-tat (forgiving with probability 1/3), win-stay-lose-shift (staying after a payoff above the mean of the payoff matrix), always action 0, always action 1, and a strategy that plays a random near-deterministic table (log-odds ) for 1 to 10 matches and then learns naively. Adam’s state is reset at the first round and then carried over. After six rounds, the last rule is deployed. The rule mixture of the meta-game is not used.
Exploitability. For Table S24 and S25, a fresh exploiter is trained for 300 iterations with its own random seed, shared by the rules being compared, and evaluated on 1,024 held-out lifetimes of matches with the horizon weights . Exploitability is its value minus the value that the reference strategy (the uniform Nash strategy in IMP, tit-for-tat in the other games) obtains against the same rule.
Coin Game rule. A pickup scenario is the pair of the two players’ last pickups in the current episode (none, its own coin or the other player’s coin). This gives nine scenarios. The agent tracks its scenario from its own observations, and the scenario resets at every episode. The policy network reads the agent’s observation and its current scenario. After each episode, the rule computes for each scenario a code of its identity, the share of the episode’s paid steps spent in it, the agent’s mean reward divided by 2, four pickup rates per paid step (each player taking its own or the other’s coin), the mean normalised advantage (clipped to ) and the mean probability of the agent’s chosen actions. A GRU with 32 units reads these statistics, without the identity codes, for all nine scenarios and the episode’s mean reward (73 inputs). An MLP with two hidden layers of 64 tanh units maps each scenario’s statistics and the GRU’s memory to . In training, noise with is added to after the tanh.
Coin Game inner update. The advantage is , where is the discounted return to the end of the episode (). The coefficients are fitted by ridge regression (penalty 1) on the previous episode (on the current one at the first update), with 47 features: the cells of the opponent and both coins, the scenario, the remaining time, its product with the scenario and a constant. The advantages are normalised over the batch. The inner update is one Adam step (learning rate 0.05). Its first moment uses the weighted gradient , and its second moment uses the same gradient with . The step is therefore linear in the multipliers of this and earlier updates. With at every update, it is the plain REINFORCE step. Adam’s state persists over the 15 updates of a lifetime, and the policy network is re-initialised for each lifetime.
Coin Game meta-training. The training opponents are self-play, the same learner with (a REINFORCE learner, not the tournament’s PPO), a scripted greedy agent that moves to the nearest coin, and a scripted agent that moves only to its own coin. Meta-training differs from the matrix games’ in four ways. (i) is the mean reward per paid step in episode over the 512 games (a Monte Carlo estimate, not (1)), is credited as in the matrix games, and the 16 episodes are weighted equally (, with updates). (ii) Each meta-training step plays 128 antithetic pairs of lifetimes that share every random draw except the sign of the noise on . Here is the mean of over the other pairs against the same opponent type, and only scenarios visited in the episode enter . (iii) The learner’s learning rate is multiplied by a factor drawn log-uniformly from for each lifetime. (iv) After 500 base meta-training steps, three PSRO rounds of 150 meta-training steps follow. Each round trains two fresh exploiters against the latest rule: a GRU policy trained by evolution strategies that also reads DUOS’s action probabilities on 16 fixed probe observations (privileged), and a memoryless policy trained by PPO. The meta-game entry is DUOS’s return minus the exploiter’s. Exploiters drawn from its minimax mixture make up 30% of training, and the other opponents keep their proportions in the remaining 70%. The last rule is deployed.
Hyperparameters and information access
Table S12 and S13 give the hyperparameters of DUOS. Those of the baselines are in Table S14 and S15 (M-FOS), S16 and S17 (SHAPER), S18 (LOLA-OM and POLA-OM*), S19 (COALA-PG*), S20 (Reciprocator) and S21 (naive learners). Table S22 gives the tournament protocol, and Table S23 the information each method uses at deployment.
| Hyperparameter | Value |
| Rounds per match | 20 |
| Matches per training lifetime | 100 |
| Objective horizons | 10, 25, 50, 100 |
| Discount | 0.96 |
| Meta-network GRU units | 32 |
| Meta-network MLP hidden units | 64, 64 |
| Inner network hidden units | 32, 32 |
| Noise on log-odds changes | 0.3 |
| Lifetimes per opponent type per meta-training step | 128 |
| Opponent mixture (Self, NL, AllC, AllD) | 0.4, 0.4, 0.1, 0.1 |
| NL step size on of Eq. (1) (base, PSRO) | 3, log-uniform [1, 10] |
| Meta learning rate (Adam) | |
| Base meta-training steps | 800 |
| PSRO rounds meta-training steps | |
| PSRO mixture (Self, NL, exploiters, fixed) | 0.30, 0.25, 0.30, 0.15 |
| Exploiter evolution-strategy (ES) iterations, directions | 150, 64 |
| Training seeds | 5 |
Table S12. Hyperparameters of DUOS in the matrix games.
| Hyperparameter | Value |
| Actor and critic hidden units | 256 |
| M-FOS-H GRU units | 16 |
| Learning rate | |
| Learning rate, IMP | † |
| PPO epochs | 4 |
| PPO clip | 0.2 |
| Entropy coefficient | 0.01 |
| Value-loss coefficient | 0.5 |
| Meta discount | 0.99 |
| Meta-updates | 1,024 |
| Lifetimes per update | 4,096 |
| Meta-steps per lifetime | 100 |
| Inner discount | 0.96 |
| NL step size on (25 on ) | 1 |
| NL step size on , IMP | 0.1 |
| Training seeds | 5 |
Table S14. Hyperparameters of M-FOS and M-FOS-H in the matrix games. †The paper’s Table 14 gives , at which our IMP runs reach 0.134 (paper 0.20). The authors’ naive-learner script uses , which reaches 0.194 (Table S9). The meta-self-play curriculum keeps .
| Setting | Value |
| Matrix games | |
| Matches per lifetime | 50 |
| Matches per lifetime , secondary | 10 |
| Rounds per match | 20 |
| Lifetimes per pair of training seeds (IMP: per seat order) | 128 |
| Trained seeds (DUOS, M-FOS, M-FOS-H, COALA-PG*) | 5 |
| Trained seeds (SHAPER) | 3 |
| Coin Game | |
| Steps per episode | 16 |
| Episodes per lifetime | 16 |
| Parallel games | 512 |
| Lifetimes per pair of training seeds and seat | 8 |
| Trained seeds | 3 |
| Protocol seed | 23962026 |
Table S22. Tournament protocol. Intervals for normalised scores are 95% percentile bootstrap intervals over 2,000 resamples (10,000 for Table 3) of each learned method’s training seeds within each game, recomputing scores, ranks and the min–max normalisation in every resample. Lifetimes are not resampled.
| Method | Own trajectory | Opponent’s actions | Opponent’s policy | Opponent’s payoffs | Assumed learning rule | Extra |
| Matrix games | ||||||
| DUOS | last match2 | observed | no | no | no | own payoffs, own table |
| M-FOS | no | no | table3 | no | no | own table |
| M-FOS-H | last match4 | observed | no | no | no | own table |
| SHAPER | every round5 | observed | no | no | no | none |
| COALA-PG* | every round5 | observed | no | no | no | none |
| LOLA-OM | last match | observed6 | fitted6 | matrix | NL step7 | exact values |
| POLA-OM* | last match | observed6 | fitted6 | matrix | NL step7 | exact values |
| Reciprocator | practice only8 | in practice8 | table3 | matrix | no | practice matches8 |
| NL | no | no | table3 | no | no | exact values |
| Coin Game | ||||||
| DUOS | last episode | no9 | no | no | no | none |
| M-FOS | every step | no | no | rewards10 | no | none |
| SHAPER | every step5 | no | no | no | no | none |
| Reciprocator | every step | given11 | no | rewards11 | no | none |
| PPO NL | last episode | no | no | no | no | none |
Table S23. Information each method uses at deployment.
| Hyperparameter | Value |
| Convolution channels | 16, 16 |
| Linear and GRU units | 16, 16 |
| Learning rate | |
| PPO epochs | 16 |
| PPO clip | 0.2 |
| Entropy coefficient | 0.01 |
| Value-loss coefficient | 0.5 |
| Meta discount | 0.99 |
| Meta-updates | 1,000 |
| Parallel games | 512 |
| Episodes per lifetime (meta-episode) | 16 |
| Training seeds | 3 |
Table S15. Hyperparameters of M-FOS in Coin Game.
| Hyperparameter | Value |
| LOLA-OM step size | 1 |
| LOLA-OM look-ahead step | 1 |
| LOLA-OM discount | 0.96, 0.9, 0.96, 0.96 |
| POLA-OM* inner step | 1, 2.5, 1.25, 1.25 |
| POLA-OM* outer step | 0.06, 0.15, 0.075, 0.075 |
| POLA-OM* KL weight | 0.5, 0.2, 0.4, 0.4 |
| POLA-OM* proximal steps (maximum) | 5,000 |
| POLA-OM* discount | 0.96 |
| Opponent-model step size | 0.05 |
| Opponent-model steps (first match, later) | 50, 1 |
Table S18. Hyperparameters of LOLA-OM and POLA-OM*. Lists give IPD, IMP, Chicken and Stag Hunt. Here is the payoff-range factor. Step sizes are on the unnormalised value , as in Table S21.
| Hyperparameter | Value |
| Matrix games (NL) | |
| Step size on (25 on of Eq. (1)) | 1 |
| Discount | 0.96 |
| Coin Game (PPO NL) | |
| Convolution channels | 16, 16 |
| Linear units | 16 |
| Learning rate | 0.005 |
| PPO epochs, clip | 80, 0.2 |
| Entropy, value-loss coefficients | 0.01, 0.5 |
| Discount | 0.96 |
Table S21. Hyperparameters of the naive learners.
| Hyperparameter | Value |
| Hawk width, heads | 32, 2 |
| PPO iterations | 3,000 |
| Meta-batch | 128 |
| Inner batch | 16 |
| Matches rounds per lifetime (meta-episode) | 20 10 |
| Optimiser steps per iteration | 1{sup}†` |
| Learning rate | |
| PPO clip | 0.2 |
| Value coefficient | 0.5 |
| NL learning rate | 0.005 |
| NL pool size | 10 |
| Training seeds | 5 |
Table S19. Hyperparameters of COALA-PG* in IPD. {sup}†The paper’s Table 4 gives 4 epochs $\times$ 2 minibatches. A speed check found one step per iteration to match the pace of its {xref}Fig. 5A`.
| Hyperparameter | Value |
| Matrix games | |
| Reciprocal reward weight | 5.0 |
| Hidden units | 2 |
| Learning rate | 0.005 |
| Discount | 0.96 |
| PPO epochs, clip, entropy | 10, 0.1, 0.02 |
| Practice matches per update | 2,048 |
| Rounds per practice match | 20 |
| Coin Game | |
| Reciprocal reward weight | 1.0 |
| Convolution channels, GRU units | 4, 16 |
| Learning rate | 0.005 |
| PPO epochs, clip, entropy | 40, 0.15, 0.01 |
| Influence-model learning rate | 0.01 |
| Parallel games | 512 |
Table S20. Hyperparameters of Reciprocator.
Additional results
Self-play variants. DUOS (self-play) in Figure 4 is exploratory and post hoc. It was designed after Table 3 had been seen, and it is not a tournament entrant. It uses the entrant’s rule form, code and settings, but trains only in self-play from the first meta-training step, with no naive learners, fixed strategies or PSRO. Both seats run the same rule, each with its own inner network, and both copies are credited with their mean value. As the games are symmetric, this gives an unbiased estimate of the gradient, with respect to the shared rule, of one copy’s expected return, a common-interest objective.
It is trained separately in IPD, Chicken and Stag Hunt for 2,000 meta-training steps of 128 lifetimes of matches, and in Coin Game for 950 meta-training steps of 256 lifetimes of 16 episodes in 512 games (the entrant’s totals, 800 + 6 × 200 and 500 + 3 × 150), with 5 and 3 seeds. IMP is not trained, since the mean value, and hence the gradient, is zero there.
M-FOS curriculum. M-FOS (meta-self-play) instead follows its authors’ curriculum [42]: two separately initialised learners, each trained on its own reward, play each other with probability , and otherwise each plays its own batch of naive learners. The paper describes the schedule only qualitatively. In the authors’ code, which we follow, starts at 1 and falls by 0.0015 per update in IPD (1,024 updates of 4,096 lifetimes of 100 matches) and by 0.005 in Coin Game (1,000 updates of one lifetime of 16 episodes in 512 games). It reaches 0 at update 667 and 200 respectively, after which training is pure self-play. In the runs shown, 66–68% (IPD) and 89–91% (Coin Game) of updates were self-play, and the last naive-learner update came at update 619–650 and 169–195.
Comparison. Counted in self-play lifetimes, M-FOS (meta-self-play) trains on about 11 times as many as DUOS (self-play) in IPD (about 2.8 million vs. 256,000), while in Coin Game DUOS (self-play) trains on about 270 times as many (243,200 vs. about 900). In IPD, M-FOS observes both players’ tables, sets its policy directly and samples its table from its learned Gaussian at every match, as in the authors’ code. In both games, its self-play scores pair the two co-trained learners of one run. On the scale of Table 3, DUOS (self-play) scores 1.019 in IPD and 1.257 in Coin Game, against 1.028 and 1.248 for M-FOS (meta-self-play). Figure 4 therefore compares two methods, each trained for self-play with its own recipe and budget, not identical training regimes.
| Opponent | Control | PSRO | [95% CI] | Seeds PSRO better |
| M-FOS | −0.020 | +0.124 | +0.144 [+0.057, +0.231] | 5/5 |
| M-FOS-H | −0.199 | +0.026 | +0.225 [+0.145, +0.305] | 5/5 |
| SHAPER | −0.092 | +0.019 | +0.111 [+0.065, +0.158] | 5/5 |
| Pairings won (all 6 IMP opponents) | 3/6 | 6/6 |
Table S24a. (a) IMP head-to-head margin,
| Game | Runs | Control | PSRO | [95% CI] | Runs lower |
| IMP | 10 | 0.376 | 0.144 | −0.232 [−0.292, −0.171] | 10/10 |
| IPD | 5 | 0.106 | 0.078 | −0.029 [−0.114, +0.057] | 3/5 |
Table S24b. (b) Exploitability
| Game | Measure | Trained | Zero-shot | [95% CI] | Seeds |
| Chicken | Tournament score | 2.105 | 1.810 | +0.295 [+0.218, +0.371] | 5/5 |
| Rank (of 6) | 3 | 1 | – | – | |
| Own return vs M-FOS | 0.979 | 0.108 | +0.871 [+0.857, +0.886] | 5/5 | |
| Margin vs M-FOS | −2.893 | −0.195 | −2.697 [−2.753, −2.642] | 0/5 | |
| Self-play return | 2.903 | 2.920 | −0.017 [−0.033, −0.001] | 0/5 | |
| Exploitability | 0.885 | 0.020 | +0.865 [+0.849, +0.880] | 0/5 | |
| Stag Hunt | Tournament score | 3.667 | 3.415 | +0.252 [+0.190, +0.314] | 5/5 |
| Rank (of 6) | 2 | 4 | – | – | |
| Own return vs M-FOS | 3.930 | 3.706 | +0.224 [+0.105, +0.343] | 5/5 | |
| Margin vs M-FOS | 0.024 | 0.141 | −0.117 [−0.158, −0.076] | 0/5 | |
| Self-play return | 3.905 | 3.803 | +0.103 [+0.060, +0.146] | 5/5 | |
| Exploitability | 0.022 | 0.035 | −0.013 [−0.027, +0.001] | 5/5 |
Table S25a.
References
- [1]Milad Aghahari, Tim Cooijmans, Juan Agustin Duque, Shunichi Akatsuka, and Aaron Courville. 2024. Best Response Shaping. Reinforcement Learning Journal 2 (2024), 798–818.
- [2]Milad Aghahari, Juan Agustin Duque, Tim Cooijmans, and Aaron Courville. 2024. LOQA: Learning with Opponent Q-Learning Awareness. In International Conference on Learning Representations.
- [3]Eli Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. 2025. Playing repeated games with large language models. Nature Human Behaviour 9, 7 (July 2025), 1380–1390. doi:10.1038/s41562-025-02172-yDOI
- [4]Stefano V. Albrecht, Filippos Christianos, and Lukas Schäfer. 2024. Multi-agent reinforcement learning: foundations and modern approaches. The MIT Press, Cambridge, Massachusetts.
- [5]Robert M. Axelrod. 2006. The evolution of cooperation (revised ed.). Basic Books, New York, NY.
- [6]Jan Balaguer, Raphael Koster, Christopher Summerfield, and Andrea Tachetti. 2022. The Good Shepherd: An Oracle Agent for Mechanism Design. https://arxiv.org/abs/2202.10135
- [7]Jacob Beck, Risto Vuorinen, Evan Zheran Liu, Zheng Xiong, Luisa Zingraf, Chelsea Finn, and Shimon Whiteston. 2025. A Tutorial on Meta-Reinforcement Learning. Foundations and Trends in Machine Learning 18, 2-3 (April 2025), 224–384. doi:10.1561/2200000080
- [8]Xuxi Chen, Nelson Vadori, Tianlong Chen, and Zhangyang Wang. 2023. Learning to Optimize Differentiable Games. In Proceedings of the 40th International Conference on Machine Learning. PMLR, 5036–5051. https://proceedings.mlr.press/v202/chen23ab.html
- [9]Kyunghyun Cho, Bart van Merriënboer, Çağlar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, 1724–1734. doi:10.3115/v1/D14-1179DOI
- [10]John D. Co-Reyes, Yingjie Miao, Dayiyi Peng, Esteban Real, Sergey Levine, Quoc V. Le, Honglak Lee, and Aleksandra Faust. 2021. Evolving Reinforcement Learning Algorithms. In International Conference on Learning Representations.
- [11]Tim Cooijmans, Milad Alghahari, and Aaron Courville. 2023. Meta-Value Learning: a General Framework for Learning with Learning Awareness. https://arxiv.org/abs/2307.08863
- [12]Marta C. Couto, Fernando P. Santos, and Christian Hilbe. 2026. Evolution of noisy learning in games. Proceedings of the National Academy of Sciences 123, 20 (May 2026), e2529959123. doi:10.1073/pnas.2529959123DOI
- [13]R. M. Dawes. 1980. Social Dilemmas. Annual Review of Psychology 31, 1 (Jan. 1980), 169–193. doi:10.1146/annurev.ps.31.020180.001125DOI
- [14]Silmane Dridi and Erol Akçay. 2018. Learning to Cooperate: The Evolution of Social Rewards in Repeated Interactions. The American Naturalist 191, 1 (Jan. 2018), 58–73. doi:10.1086/694822DOI
- [15]Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. 2016. RL²: Fast Reinforcement Learning via Slow Reinforcement Learning. https://arxiv.org/abs/1611.02779
- [16]Juan Agustin Duque, Milad Aghajohari, Tim Cooijmans, Razvan Ciuca, Tianyu Zhang, Gauthier Gidel, and Aaron Courville. 2025. Advantage Alignment Algorithms. In International Conference on Learning Representations.arxiv.org/abs/2406.14662
- [17]Jidong Feng, Oliver Slumbers, Ziyu Wan, Bo Liu, Stephen McAleer, Ying Wen, Jun Wang, and Yaodong Yang. 2021. Neural Auto-Curricula in Two-Player Zero-Sum Games. In Advances in Neural Information Processing Systems, Vol. 34. 3504–3517.
- [18]Sebastian Flennerhag, Yannick Schroecker, Tom Zahavy, Hado van Hasselt, David Silver, and Satinder Singh. 2022. Bootstrapped Meta-Learning. In International Conference on Learning Representations.arxiv.org/abs/2109.04504
- [19]Sebastian Flennerhag, Tom Zahavy, Brendan O’Donoghue, Hado van Hasselt, Andrés György, and Satinder Singh. 2023. Optimist Meta-Gradients. In Advances in Neural Information Processing Systems, Vol. 36. 57852–57862. doi:10.52202/075280-2521
- [20]Jakob Foerster, Richard Y. Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. 2018. Learning with Opponent-Learning Awareness. In Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2018). International Foundation for Autonomous Agents and Multiagent Systems, Stockholm, Sweden, 122–130. doi:10.65109/hgwa8807DOI
- [21]Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric P. Xing, and Shimon Whiteson. 2018. DiCE: The Infinitely Differentiable Monte Carlo Estimator. In International Conference on Machine Learning, Vol. 80. 1529–1538.arxiv.org/abs/1802.05098
- [22]Alexander D. Goldie, Chris Lu, Matthew T. Jackson, Shimon Whiteson, and Jakob N. Foerster. 2024. Can Learned Optimization Make Reinforcement Learning Less Difficult?. In Advances in Neural Information Processing Systems, Vol. 37. 5454–5497. doi:10.52202/079017-0177DOI
- [23]Jess Daffin, Jaron Cohen, Jakob Nicolaus Foerster, and Shimon Whiteson. 2025. How Should We Meta-Learn Reinforcement Learning Algorithms? arXiv:2507.17668 [cs.LG], doi:10.48550/arXiv.2507.17668
- [24]Eric A. Hansen, Daniel S. Bernstein, and Shlomo Zilberstein. 2004. Dynamic Programming for Partially Observable Stochastic Games. In AAAI Conference on Artificial Intelligence. 709–715.
- [25]Keegan Harris, Ioannis Anagnostides, Gabriele Farina, Mikhail Khodak, Zhi-wei Steven Wu, and Tuomas Sandholm. 2023. Meta-Learning in Games. In International Conference on Learning Representations.arxiv.org/abs/2209.14110
- [26]Edward Hughes, Joel Z. Leibo, Matthew Phillips, Karl Tuyls, Edgar Duéñez-Guzman, Antonio García Castañeda, Iain Dunning, Tina Zhu, Kevin McKee, Raphael Koster, Heather Roff, and Thore Graepel. 2018. Inequity Aversion Improves Cooperation in Intertemporal Social Dilemmas. In Advances in Neural Information Processing Systems, Vol. 31. 3326–3336.arxiv.org/abs/1803.08884
- [27]Matthew T. Jackson, Minqi Jiang, Jack Parker-Holder, Risto Vuorio, Chris Lu, Gregory Farquhar, Shimon Whiteson, and Jakob N. Foerster. 2023. Discovering General Reinforcement Learning Algorithms with Adversarial Environment Design. In Advances in Neural Information Processing Systems, Vol. 36. 79980–79998. doi:10.52202/075280-3503DOI
- [28]Matthew T. Jackson, Chris Lu, Louis Kirsch, Robert T. Lange, Shimon Whiteson, and Jakob N. Foerster. 2024. Discovering Temporally-Aware Reinforcementarxiv.org/abs/2402.05828
- [29]Akbir Khan, Timon Willi, Newton Kwan, Andrea Tacchetti, Chris Lu, Edward Grefenstette, Tim Rocktäschel, and Jakob Foerster. 2024. Scaling Opponent Shaping to High Dimensional Games. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2024). International Foundation for Autonomous Agents and Multiagent Systems, Auckland, New Zealand, 1001–1010. doi:10.65109/tmpy7673arxiv.org/abs/2312.12568
- [30]Dong-Ki Kim, Miao Liu, Matthew Riemer, Chuangchuang Sun, Marwa Abdulhai, Golnaz Habibi, Sebastian Lopez-Cot, Gerald Tesauro, and Jonathan P. How. 2021. A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning. In International Conference on Machine Learning, Vol. 139. 5541–5550.arxiv.org/abs/2011.00382
- [31]Dong-Ki Kim, Matthew Riemer, Miao Liu, Jakob N. Foerster, Michael Everett, Chuangchuang Sun, Gerald Tesauro, and Jonathan P. How. 2022. Influencing Long-Term Behavior in Multiagent Reinforcement Learning. In Advances in Neural Information Processing Systems, Vol. 35. 18808–18821. doi:10.52202/068431-1366DOI
- [32]Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations.arxiv.org/abs/1412.6980
- [33]Louis Kirsch, Sebastian Flennerhag, Hado van Hasselt, Abram L. Friesen, Junhyuk Oh, and Yutian Chen. 2022. Introducing Symmetries to Black Box Meta Reinforcement Learning. In AAAI Conference on Artificial Intelligence. 7202–7210. doi:10.1609/aaai.v36i7.20681DOI
- [34]Louis Kirsch, Sjoerd van Steenkiste, and Jürgen Schmidhuber. 2020. Improving Generalization in Meta Reinforcement Learning Using Learned Objectives. In International Conference on Learning Representations.arxiv.org/abs/1910.04098
- [35]Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. 2017. A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning. In Advances in Neural Information Processing Systems, Vol. 30. 4190–4203.arxiv.org/abs/1711.00832
- [36]Joel Z. Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. 2017. Multi-agent Reinforcement Learning in Sequential Social Dilemmas. In Proceedings of the 16th International Conference on Autonomous Agents and Multi-agent Systems (AAMAS 2017). International Foundation for Autonomous Agents and Multiagent Systems, São Paulo, Brazil, 464–473. doi:10.65109/qmsu8421DOI
- [37]Adam Lerer and Alexander Peysakhovich. 2018. Maintaining cooperation in complex social dilemmas using deep reinforcement learning. https://arxiv.org/abs/1707.01068
- [38]Alistair Letcher, Jakob Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson. 2019. Stable Opponent Shaping in Differentiable Games. In International Conference on Learning Representations.arxiv.org/abs/1811.08469
- [39]Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhrub Batra. 2017. Deal or No Deal? End-to-End Learning of Negotiation Dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, 2443–2453. doi:10.18653/v1/D17-1259DOI
- [40]Zun Li, John Schultz, Daniel Hennes, and Marc Lanctot. 2026. Discovering Multiagent Learning Algorithms with Large Language Models. https://arxiv.org/abs/2602.16928
- [41]Chris Lu, Jakub Grudzien Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. 2022. Discovered Policy Optimisation. In Advances in Neural Information Processing Systems, Vol. 35. 16455–16468. doi:10.52202/068431-1197DOI
- [42]Chris Lu, Timon Willi, Christian Schroeder de Witt, and Jakob Foerster. 2022. Model-Free Opponent Shaping. In International Conference on Machine Learning, Vol. 162. 14398–14411.arxiv.org/abs/2205.01447
- [43]James Martens. 2020. New Insights and Perspectives on the Natural Gradient Method. Journal of Machine Learning Research 21, 146 (2020), 1–76.
- [44]Alex McAvoy, Julian Kates-Harbeck, Krishnendu Chatterjee, and Christian Hilbe. 2022. Evolutionary instability of selfish learning in repeated games. PNAS Nexus 1, 4 (Sept. 2022), pgac141. doi:10.1093/pnasnexus/pgac141
- [45]H. Brendan McMahan, Geoffrey J. Gordon, and Avrim Blum. 2003. Planning in the Presence of Cost Functions Controlled by an Adversary. In International Conference on Machine Learning. 536–543.
- [46]Luke Metz, Niru Mewawatharan, Jeremy Nixon, C. Daniel Freeman, and Jascha Sohl-Dickstein. 2019. Understanding and Correcting Pathologies in the Training of Learned Optimizers. In International Conference on Machine Learning, Vol. 97. 4556–4565.arxiv.org/abs/1810.10180
- [47]Alexander Meulemans, Seijin Kobayashi, Johannes von Oswald, Nino Scherrer, Eric Elmoznino, Blake A. Richards, Guillaume Lajoie, Blaise Agüera y Arcas, and João Sacramento. 2025. Multi-Agent Cooperation Through Learning-Aware Policy Gradients. In International Conference on Learning Representations.arxiv.org/abs/2410.18636
- [48]Junhyuk Oh, Gregory Farquhar, Iurii Kemaev, Dan A. Calian, Matteo Hessel, Luisa Zintgraf, Satinder Singh, Hado van Hasselt, and David Silver. 2025. Discovering state-of-the-art reinforcement learning algorithms. Nature 648, 8093 (Dec. 2025), 312–319. doi:10.1038/s41586-025-09761-xDOI
- [49]Junhyuk Oh, Matteo Hessel, Wojciech M. Czarnecki, Zhongwen Xu, Hado van Hasselt, Satinder Singh, and David Silver. 2020. Discovering Reinforcement Learning Algorithms. In Advances in Neural Information Processing Systems, Vol. 33. 1060–1070.
- [50]Joong Sun Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23). ACM, 1–22. doi:10.1145/3586183.3606763DOI
- [51]William H. Press and Freeman J. Dyson. 2012. Iterated Prisoner’s Dilemma contains strategies that dominate any evolutionary opponent. Proceedings of the National Academy of Sciences 109, 26 (June 2012), 10409–10413. doi:10.1073/pnas.1206569109DOI
- [52]Julien Pérólat, Joel Z. Leibo, Vinicius Zambaldi, Charles Beattie, Karl Tuyls, and Thore Graepel. 2017. A Multi-Agent Reinforcement Learning Model of Common-Pool Resource Appropriation. In Advances in Neural Information Processing Systems, Vol. 30. 3643–3652.arxiv.org/abs/1707.06600
- [53]Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. 2017. Evolution Strategies as a Scalable Alternative to Reinforcement Learning. https://arxiv.org/abs/1703.03864
- [54]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs.LG]. doi:10.48550/arXiv.1707.06347arxiv.org/abs/1707.06347
- [55]Wilko Schwarting, Alyssa Pierson, Javier Alonso-Mora, Sertac Karaman, and Daniela Rus. 2019. Social behavior for autonomous vehicles. Proceedings of the National Academy of Sciences 116, 50 (Dec. 2019), 24972–24978. doi:10.1073/pnas.1820676116
- [56]David Sychrovský, Christopher Solinas, Revan MacQueen, Kevin Wang, James R. Wright, Nathan R. Sturtevant, and Michael Bowling. 2026. Approximating Nash Equilibria in General-Sum Games via Meta-Learning. In Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026). International Foundation for Autonomous Agents and Multiagent Systems, Paphos, Cyprus, 2374–2382. doi:10.65109/sfsz6668arxiv.org/abs/2504.18868
- [57]David Sychrovský, Michal Šustr, Elnaz Davoodi, Michael Bowling, Marc Lanctot, and Martin Schmid. 2024. Learning Not to Regret. In AAAI Conference on Artificial Intelligence. 15202–15210. doi:10.1609/aaai.v38i14.29443DOI
- [58]Nelson Vadori, Rahul Savani, Thomas Spooner, and Sumitra Ganesh. 2022. Consensus Multiplicative Weights Update: Learning to Learn Using Projector-Based Game Signatures. In International Conference on Machine Learning, Vol. 162. 21901–21926.arxiv.org/abs/2106.02615
- [59]Risto Vuorio, Jacob Beck, Shimon Whiteson, Jakob Foerster, and Gregory Farquhar. 2022. An Investigation of the Bias-Variance Tradeoff in Meta-Gradients. https://arxiv.org/abs/2209.11303
- [60]Marissa A. Weis, Maciej Wołczyk, Rajai Nasser, Rif A. Saroukos, Blaise Agüera y Arcas, Joao Sacramento, and Alexander Meulemans. 2026. Multi-agent cooperation through in-context co-player inference. https://arxiv.org/abs/2602.16301
- [61]Timon Willi, Alistair Letcher, Johannes Treutlein, and Jakob Foerster. 2022. COLA: Consistent Learning with Opponent-Learning Awareness. In International Conference on Machine Learning, Vol. 162. 23804–23831.
- [62]Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8, 3-4 (May 1992), 229–256. doi:10.1007/BF00992696DOI
- [63]Hang Xu, Kai Li, Haobo Fu, Qiang Fu, and Junliang Xing. 2022. AutoCFR: Learning to Design Counterfactual Regret Minimization Algorithms. In AAAI Conference on Artificial Intelligence. 5244–5251. doi:10.1609/aaai.v36i5.20460DOI
- [64]Zhongwen Xu, Hado van Hasselt, Matteo Hessel, Junhyuk Oh, Satinder Singh, and David Silver. 2020. Meta-Gradient Reinforcement Learning with an Objective Discovered Online. In Advances in Neural Information Processing Systems, Vol. 33. 15254–15264.arxiv.org/abs/2007.08433
- [65]Zhongwen Xu, Hado van Hasselt, and David Silver. 2018. Meta-Gradient Reinforcement Learning. In Advances in Neural Information Processing Systems, Vol. 31. 2396–2407.arxiv.org/abs/1805.09801
- [66]Jiachen Yang, Ang Li, Mehrdad Farajtabar, Peter Sunehag, Edward Hughes, and Hongyun Zha. 2020. Learning to Incentivize Other Learning Agents. In Advances in Neural Information Processing Systems, Vol. 33. 15208–15219. https://proceedings.neurips.cc/paper_files/paper/2020/file/ad7ed54d7b9baceb12045a929e7e2f66-Paper.pdfarxiv.org/abs/2006.06051
- [67]Stephen Zhou, Chris Lu, Roger B. Grosse, and Jakob Foerster. 2022. Proximal Learning with Opponent-Learning Awareness. In Advances in Neural Information Processing Systems, Vol. 35. 26324–26336. doi:10.52202/068431-1909
- [68]John L. Zhou, Weizhe Hong, and Jonathan C. Kao. 2024. Reciprocal Reward Influence Encourages Cooperation from Self-Interested Agents. In Advances in Neural Information Processing Systems, Vol. 37. 59491–59512. doi:10.52202/079017-1900DOI



