betaFeedback

Reading v1View latest version

WorksComputer Science

Discovering Learning Rules that Shape Others

Abstract

In general-sum games, self-interested learners can reach outcomes that are poor for everyone, which has motivated methods that shape the opponent's learning. Methods like Learning with Opponent-Learning Awareness (LOLA), which differentiates through the opponent's learning process, and Model-Free Opponent Shaping (M-FOS), which uses model-free optimisation methods for long-horizon opponent shaping, have enabled mutual cooperation and exploitation of learners. However, these methods require access to the opponent's parameters, policy, rewards or learning rule during deployment. In single-agent reinforcement learning, update rules discovered by meta-learning have achieved state-of-the-art results (e.g. DiscoRL). These rules are separate from the policy they train and read only the agent's own experience. We hereby extend this framework to general-sum settings by introducing Discovered Update Rule for Opponent Shaping (DUOS). To the best of our knowledge, this is the first method to discover an update rule for an agent in general-sum games. DUOS meta-trains the rule to maximise the agent's expected return over its lifetime, and updates an inner network using the agent's own experience. We show that our discovered rule exhibits behaviour consistent with opponent shaping without reading opponent policies or parameters, and ranks 1st in normalised mean score and 1st in self-play among tournament entrants.

Introduction

Real-world multi-agent environments, from self-driving cars [55] to multi-agent large language model (LLM) systems [3, 50], can be framed as games ranging from zero-sum to fully cooperative. General-sum games cover the mixed-motive settings in between, such as social dilemmas [13, 36] or buyer-seller negotiations [39]. Naive self-interested learners within general-sum games often converge towards outcomes that are poor for everyone [20, 36]. For example, in Iterated Prisoner’s Dilemma (IPD) [5, 51], they often converge to mutual defection rather than mutual cooperation. Opponent shaping [20, 42] addresses this problem by influencing how others learn to achieve mutually beneficial outcomes.

Existing shaping methods leave a gap. Hand-derived rules such as LOLA [20] and POLA [67] differentiate through the opponent’s update, and require their parameters, payoffs or learning rules. Meta-learned shapers such as M-FOS [42], SHAPER [29] and COALA-PG [47] do not require a model of the opponent’s learning rule as SHAPER and COALA-PG act from their own history, but their learning algorithm is inside a black-box network, retrained each game. In single-agent reinforcement learning (RL), where the environment is stationary, rules discovered by meta-learning [48, 49] have achieved state-of-the-art results. Such rules also have practical advantages over a black-box learner: a discovered rule is not limited to updates derived by hand [49], is separate from the policy it trains, can transfer to different games without retraining, and its outputs can be read and interpreted directly. We therefore ask: can an update rule that learns from an agent’s own experience be discovered for general-sum games? However, learning update rules is difficult because learning is coupled, meta-gradients explode through co-learning, and opponents adapt, making the setting non-stationary. We hereby introduce Discovered Update Rule for Opponent Shaping (DUOS), a meta-reinforcement learning algorithm that discovers its own update rule by interacting with opponents, without needing their policies or parameters.

In this paper, we design the meta-objective to maximise the agent’s return over its lifetime. The rule reads the agent’s own experience and outputs policy updates, hence at deployment it does not require any privileged information: the agent never sees its opponents’ parameters, policies, learning rules or rewards. To avoid meta-gradients that explode through co-learning, the rule is trained with a likelihood-ratio meta-gradient, without differentiating through any learning. We then examine whether opponent shaping emerges from the rule without specifying it explicitly.

Although DUOS is trained to maximise lifetime return, this objective alone does not establish robustness to opponents trained specifically against its learned update rule. We therefore add a stage inspired by Policy Space Response Oracles (PSRO) [35] to meta-training: we train opponents to maximise their returns against the current rule, add them to the training population, and continue optimising the rule against the expanded population. We evaluate whether PSRO improves resistance to exploitation for DUOS and how it affects tournament results.

We evaluate DUOS in IPD, Iterated Matching Pennies (IMP), Chicken, Stag Hunt and Coin Game [37] in round-robin tournaments against established opponent-shaping methods, including methods that observe their opponents’ policies or parameters. Each Preprint. Under review. baseline is reproduced from its original settings, checked against results where possible, and flagged if it fails. We also observe DUOS self-play and zero-shot transfer of the IPD-trained rule to Chicken and Stag Hunt. Despite using only its own experience and learning an update rule from scratch, DUOS ranks 1st in normalised mean score across all games tested, and 1st in normalised mean score for self-play among tournament entrants.

Related Work

Opponent shaping: LOLA introduced the idea of opponent shaping by differentiating through the opponent’s learning step [20], which DiCE estimates from sampled trajectories [21]. Stable Opponent Shaping [38], COLA [61] and POLA [67] then further refined the idea, each requiring the opponent’s parameters (or in POLA’s opponent-modelling variant, a model of its policy fitted to its actions), as well as its payoffs and learning rule. LOQA [2] and Advantage Alignment [16] instead shape through estimates of the opponent’s values, while Reciprocator [68] and LIO [66] shape through rewards tied to the opponent’s returns. Best Response Shaping trains against an approximate best response to the agent instead [1]. All of these methods need opponent information each time they update their policy, whereas DUOS only needs it during meta-training.

One level above, meta-game methods learn to shape over many episodes. M-FOS trains a meta-policy where the action is the agent’s next policy [42]. For matrix games, M-FOS observes both players’ policies, and in Coin Game, it only observes the game state and both players’ rewards. SHAPER [29], COALA-PG [47], and in-context co-player inference [60] use recurrent or sequence-model agents that act with their own history. For all of these methods, the opponent-shaping behaviour and the learning algorithm are held in a black-box network. Meta-MAPG [30] and FURTHER [31] also take into account other agents’ learning, through meta-gradients and by influencing their long-run behaviour, respectively. The Good Shepherd [6] optimises a fixed policy that shapes learners without learning at deployment. DUOS updates a separate policy, but unlike M-FOS, it reads only its own experience and payoffs.

Learned update functions for games: COLA represents both players’ update functions with a neural network, and fits them to a consistency equation derived from LOLA [61]. Meta-Value Learning (MeVa) learns a value over the joint policy, and the gradient of the value leads to each player’s update [11]. MeVa extorts naive learners in IPD but reads both players’ parameters and is fitted to each game.

Update rules for centralised solvers have been discovered by meta-gradients or LLM-driven search [17, 40, 63] for games but not for agents. Other work instead learns optimisers or regret minimisers that converge towards equilibrium [8, 25, 56, 57, 58]. These target consistency per game or equilibrium computation, whereas DUOS is meta-trained for an agent’s own lifetime return against a population of opponents, reading only its experience.

Discovered reinforcement learning algorithms: In single-agent reinforcement learning, Xu et al. Xu et al. [65] use meta-gradients to tune hyperparameters such as the discount factor, and later to discover the objective itself [64]. LPG goes further and meta-learns an entire update rule across many lifetimes, generalising from toy environments to Atari [49]. DiscoRL [48] scales this approach to a large number of environments and discovers a rule that outperforms state-of-the-art hand-designed algorithms. Related work discovers objectives, drift functions and optimisers [10, 22, 27, 28, 34, 41]. Other work analyses the meta-gradients [18, 19, 59] that these algorithms utilise. Black-box meta-learners such as RL2^{2} [15] keep the learning algorithm within the agent’s network, whereas these methods separate the rule from the agent it trains [33]. These methods form the many-shot branch of meta-reinforcement learning [7], where a learning algorithm trains an agent over a long lifetime. DUOS belongs to this branch and is applied to general-sum games, where opponents also learn.

Evolutionary game theory has asked which learning rules survive selection in repeated games, selecting from hand-designed learning rules or their parameters [12, 14, 44]. To our knowledge, DUOS is the first update rule discovered for an agent in general-sum games.

Background

We formalise the underlying game as a partially observable stochastic game (POSG) [24] that represents general-sum games and all games in this paper, written as a tuple ⟨I,S,A,O,P,R,γ⟩\langle\mathcal{I}, \mathcal{S}, \mathcal{A}, \mathcal{O}, \mathcal{P}, \mathcal{R}, \gamma\rangle of agents I\mathcal{I}, states S\mathcal{S}, joint actions A=×i∈IAi\mathcal{A} = \times_{i \in\mathcal{I}} \mathcal{A}^{i}, joint observations O=×i∈IOi\mathcal{O} = \times_{i \in\mathcal{I}} \mathcal{O}^{i}, an observation function O\mathcal{O}, a transition function P\mathcal{P}, rewards R=×i∈IRi\mathcal{R} = \times_{i \in\mathcal{I}} \mathcal{R}^{i} and a discount γ∈[0,1)\gamma\in[0, 1). In round tt, each agent ii receives an observation otio^{i}_{t} according to O\mathcal{O}, samples an action ati∼πi(⋅∣oti)a^{i}_{t} \sim\pi^{i}(\cdot\mid o^{i}_{t}), and receives a reward rti=Ri(st,at)r^{i}_{t} = \mathcal{R}^{i}(s_{t}, a_{t}), where at=(ati,at−i)a_{t} = (a^{i}_{t}, a^{-i}_{t}) is the joint action and −i-i denotes the opponent.

In two-player 2×22 \times2 iterated matrix games, actions Ai={0,1}\mathcal{A}^{i} = \{0, 1\} and rewards rti=Mi[ati,at−i]r^{i}_{t} = M^{i}[a^{i}_{t}, a^{-i}_{t}]. We assign a starting state s∅s_{\varnothing} to the first round of each match. Thereafter, the state records the joint action as (agent ii’s action, opponent’s action), providing the state space:

S={s∅,(0,0),(0,1),(1,0),(1,1)}.\mathcal{S} = \{s_{\varnothing}, (0, 0), (0, 1), (1, 0), (1, 1)\}.

A memory-one policy is conditioned on the previous joint action, with parameters θ\theta, and plays action 0 given state ss with probability πθ(0∣s)=σ(dθ(s))\pi_{\theta}(0 \mid s) = \sigma(d_{\theta}(s)), where σ\sigma is the logistic function and dθ(s)=log⁡(πθ(0∣s)πθ(1∣s))d_{\theta}(s) = \log\left(\frac{\pi_{\theta}(0 \mid s)}{\pi_{\theta}(1 \mid s)}\right) represents the log-odds of action 0.

To assess the performance of a fixed pair of policies, we define an agent’s value as the discounted value function at the starting state and evaluate it as:

Vi(πi,π−i)=(1−γ)E[∑t=0∞γtrti].(1)V^{i}(\pi^{i}, \pi^{-i}) = (1 - \gamma)\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}r^{i}_{t}\right]. \tag*{(1)}

When the opponent also uses a memory-one policy, this value can be computed exactly [20]. This value provides the performance measure for our meta-training objective (Section 4.3). This value is only provided to DUOS during training, and DUOS adapts from sampled matches without receiving this value during deployment.

Update Rules

A lifetime is defined as a sequence of matches k=0,1,…k = 0, 1, \ldots. Each match starts in s∅s_{\varnothing}, and the agent’s policy parameters do not reset in between. Unless stated otherwise, learners update between matches using the policies from the completed match. We define agent ii’s policy in match kk as πki\pi^{i}_{k} and their value as Vki=Vi(πki,πk−i)V^{i}_{k} = V^{i}(\pi^{i}_{k}, \pi^{-i}_{k}).

A naive learner (NL) [20] treats the opponent’s policy as fixed when computing its own update. In our experiments, an NL playing as agent −i-i, with logits ϕk\phi_k and step size α\alpha, takes one exact gradient step after each match,

ϕk+1=ϕk+α∇ϕV−i(πk−i,σ(ϕ))∣ϕ=ϕk(2)\phi_{k+1} = \phi_k + \alpha\nabla_{\phi} V^{-i}\left(\pi_k^{-i}, \sigma(\phi)\right)\big|_{\phi=\phi_k} \tag*{(2)}

using agent ii’s policy before its own update. Opponent shaping instead chooses the agent’s own updates to steer the opponent’s learning, as LOLA and M-FOS do.

Meta-Learning

M-FOS casts opponent shaping as a meta-game: a meta-step is one match, the meta-state holds both players’ policies, the meta-reward is the agent’s return in the match, and the meta-action is its next policy. In matrix games, its meta-policy πmeta\pi^{\mathrm{meta}} samples the agent’s next parameters given both players’ current policy parameters, θk+1i∼πmeta(⋅∣θki,θk−i)\theta_{k+1}^{i} \sim\pi^{\mathrm{meta}}\left(\cdot\mid\theta_{k}^{i}, \theta_{k}^{-i}\right).1

A related approach is to meta-learn an update rule. LPG [49] separates a rule with meta-parameters η\eta from the agent’s parameters θ\theta and meta-trains η\eta to maximise the expected return of the policy obtained after a lifetime of updates θ0→θ1→⋯→θN\theta_0 \to\theta_1 \to\cdots\to\theta_N, across a distribution of environments.

When meta-learning an update rule, meta-gradients through unrolled updates [65] can explode over long unrolls because the learning dynamics amplify small perturbations [46]. In multi-agent

settings, the problem worsens as the unrolls also pass through the opponent’s learning. Instead, we use the likelihood-ratio gradient [62], which does not differentiate through the game, the opponent’s learning or the agents’ parameter updates.

Discovered Update Rules for Opponent Shaping

In iterated matrix games, after every match DUOS updates the log-odds in every state via its parameters (Figure 1). Our key idea is to meta-learn a rule that reads only the agent’s own experience and payoff matrix. In this section, we will first describe the architecture of the meta-network and its outputs. Then, we will analyse how the inner network takes in the meta-network output to adjust its weights accordingly. Lastly, we will talk about how meta-training is optimised, and introduce robustness training against exploiters.

Figure 1. Overview of DUOS. After every match, the meta-network η\eta reads the agent’s experience Dki\mathcal{D}^{i}_{k} and outputs log-odds changes yky_k, to which the inner network θ\theta is fitted. The opponent’s policy, parameters, learning rule and payoffs are never observed. Meta-training perturbs yky_k with Gaussian noise and updates η\eta with a likelihood-ratio gradient whose baseline bkb_k comes from lifetimes against the same opponent (§ 4).

Meta-Network

The update rule is learned by training the meta-network across lifetimes. The meta-network consists of a gated recurrent unit (GRU) [9] and a multi-layer perceptron (MLP) [4], whose weights are denoted by meta-parameters η\eta (§ 4.3). During deployment, the meta-network is frozen and the weights are fixed while DUOS uses the rule to adjust the parameters θ\theta of its inner MLP after each match.

At the end of match kk, let Dki=(sti,ati,at−i,rti)t<T\mathcal{D}^{i}_{k} = (s^{i}_{t}, a^{i}_{t}, a^{-i}_{t}, r^{i}_{t})_{t<T} denote the trajectory of states, own actions, opponent actions and own rewards observed by DUOS. For each state s∈Ss \in\mathcal{S}, the meta-network takes in the feature vector

xk,s=[es,vs,q^s,ps,rˉs]∈R9,x_{k,s} = [e_s, v_s, \hat{q}_s, p_s, \bar{r}_s] \in\mathbb{R}^{9},

where es∈R5e_s \in\mathbb{R}^{5} encodes the state, vsv_s represents the fraction of rounds spent in that state, q^s\hat{q}_s estimates the opponent’s smoothed frequency of action 0 during match kk, psp_s is the agent’s current probability of action 0, and rˉs\bar{r}_s is the agent’s normalised mean reward in that state. The GRU reads all five xk,sx_{k,s}, the mean reward and the agent’s own payoff matrix after each match, and the MLP maps xk,sx_{k,s} and the GRU’s memory to output a scalar that represents the change to the policy’s log-odds in state ss. The rule is also applied with the two actions relabelled and the outputs averaged, so the update does not depend on what the actions are called.

Inner Network

The inner network is an MLP with parameters θ\theta that represents DUOS’s policy. With log-odds changes yk,sy_{k,s} from the meta-network, we add them to the current log-odds and clip the results to obtain

dsclip=clip⁡(dθk(s)+yk,s,−12,12).d_s^{\mathrm{clip}} = \operatorname{clip}(d_{\theta_k}(s) + y_{k,s}, -12, 12).

To translate dsclipd_s^{\mathrm{clip}} into changes in θ\theta, we use damped Gauss-Newton steps [43]. A small change δ\delta in the parameter shifts the log-odds by JδJ\delta, where JJ is the Jacobian

J=∂dθ∂θ.J = \frac{\partial d_\theta}{\partial\theta}.

Hence, we require Jδ=uJ\delta= u, where us=dsclip−dθ(s)u_s = d_s^{\mathrm{clip}} - d_\theta(s) and Jδ=uJ\delta= u. We take three steps of the approximate

δ=J⊤(JJ⊤+Λ)−1u,\delta= J^\top(JJ^\top+ \Lambda)^{-1}u,

where Λ\Lambda is a damping term that makes the matrix invertible, and JJ and uu are recomputed each time. The best candidate is then kept as θk+1\theta_{k+1} (Appendix D).

Optimising Meta-Training

Objective: We train the meta-network to look for meta-parameters η\eta whose rule produces a high value throughout a lifetime. A training lifetime has N=100N = 100 matches with an update after each match. We denote VkcrV_k^{\mathrm{cr}} as the credited value (defined below) of the agent’s policy after kk updates, computed with Equation (1), where k=0k = 0 represents the initial policy. Meta-training then maximises the weighted average

J(η)=Ec∼ρ,τ[∑k=0NwkVkcr],(3)\mathcal{J}(\eta) = \mathbb{E}_{c \sim\rho, \tau}\left[\sum_{k=0}^{N} w_k V_k^{\mathrm{cr}}\right], \tag*{(3)}

where the opponent cc is sampled from the training mixture ρ\rho (Equation (4)), τ\tau represents a lifetime sampled with training noise, and

wk=14∑h≥k1h+1,h∈{10,25,50,100}.w_k = \frac{1}{4}\sum_{h \ge k}\frac{1}{h+1}, \qquad h \in\{10,25,50,100\}.

Thus, J(η)\mathcal{J}(\eta) averages the mean value of the policies after 0 to hh updates over the four horizons hh, such that the rule adapts quickly and does not fall off later.

Credit and opponents: The credited value is the agent’s value Vkcr=VkiV_k^{\mathrm{cr}} = V_k^i except in self-play, where both agents are credited with the mean value 12(Vki+Vk−i)\frac{1}{2}(V_k^i + V_k^{-i}). The mean value is zero in IMP, so self-play does not add any gradient to the IMP rule. To train the base rule η(0)\eta^{(0)}, we weight lifetimes against each opponent by:

ρ0(Self)=0.4,ρ0(NL)=0.4,\rho_0(\mathrm{Self}) = 0.4,\qquad\rho_0(\mathrm{NL}) = 0.4,
ρ0(AllC)=0.1,ρ0(AllD)=0.1.(4)\rho_0(\mathrm{AllC}) = 0.1,\qquad\rho_0(\mathrm{AllD}) = 0.1. \tag*{(4)}

Self is self-play, NL denotes a naive learner, and AllC and AllD are training-only opponents that almost always play action 0 and action 1, respectively (cooperation and defection in IPD).

Algorithm 1 Meta-training of DUOS

Require: Base mixture ρ0\rho_0, lifetime length NN, PSRO rounds LL

  1. Initialise meta-parameters η\eta

  2. for ℓ=0,…,L\ell= 0,\ldots,L do

  3. if ℓ>0\ell> 0 then

  4. Train exploiter ψ(ℓ)\psi^{(\ell)} against η\eta

  5. Update the meta-game U(ℓ)U^{(\ell)}

  6. Compute exploiter mixture μ∗\mu^*

  7. Form training mixture ρℓ\rho_\ell

  8. for each meta-training step do

  9. for all training lifetimes, opponent c∼ρℓc \sim\rho_\ell do

  10. Initialise policies and recurrent memory

  11. for k=0,…,N−1k = 0,\ldots,N-1 do

  12. Evaluate VkcrV_k^{\mathrm{cr}} (Eq. (1))

  13. Play match kk

  14. Sample y~k∼π~η(⋅∣Hk)\tilde{y}_k \sim\tilde{\pi}_\eta(\cdot\mid\mathcal{H}_k)

  15. Update θ\theta using y~k\tilde{y}_k (§ 4.2)

  16. Update the opponent (NL: Eq. (2))

  17. Evaluate VNcrV_N^{\mathrm{cr}}

  18. Compute returns GkG_k and baselines bkb_k

  19. Estimate ∇ηJ\nabla_\eta\mathcal{J} from g^k\hat{g}_k (Eq. (5))

  20. Update η\eta with Adam

  21. Store η(ℓ)←η\eta^{(\ell)} \leftarrow\eta

  22. return η(L)\eta^{(L)}

Estimator: Inspired by Gaussian perturbations used in evolution strategies [41, 53], we add Gaussian noise (σ=0.3\sigma= 0.3) to the log-odds changes during meta-training, and use the perturbed changes y~k\tilde{y}_k to update the inner network (§ 4.2). We train the meta-network using a likelihood-ratio gradient estimator [62], where each perturbed change contributes

g^k=(Gk−bk)∇ηlog⁡π~η(y~k∣Hk).(5)\hat{g}_k = (G_k - b_k)\nabla_\eta\log\tilde{\pi}_\eta(\tilde{y}_k \mid\mathcal{H}_k). \tag*{(5)}

Here, π~η(⋅∣Hk)=N(yk,σ2I)\tilde{\pi}_\eta(\cdot\mid\mathcal{H}_k) = \mathcal{N}(y_k,\sigma^2 I) is the distribution of the perturbed changes given history Hk=(Mki,πki,Dki)\mathcal{H}_k = (M_k^i,\pi_k^i,\mathcal{D}_k^i), which consists of the agent’s payoff matrix MiM^i, its policy πji\pi_j^i, and trajectory Dji\mathcal{D}_j^i in each match j≤kj \le k. The weighted value of the later policies θk+1,…,θN\theta_{k+1},\ldots,\theta_N is Gk=∑j>kwjVjcrG_k = \sum_{j>k} w_j V_j^{\mathrm{cr}}.

We sample four lifetimes from the same initial policies. For each lifetime, bkb_k is the average of GkG_k in the other three. This reinforces changes that beat the baseline and discourages worse ones.

We sum g^k\hat{g}_k over each lifetime, including both agents in self-play, and average across training lifetimes using the opponent mixture ρ\rho. Adam [32] is then used to update the meta-network’s weights η\eta.

Training Against Exploiters

Inspired by PSRO [35] and the double oracle algorithm [45], each round we fix the meta-network and train an exploiter ψ(ℓ)\psi^{(\ell)} to maximise its own lifetime value against DUOS. The exploiter is able to observe DUOS’s current policy table and selects its policy after DUOS has updated.

We evaluate the earlier versions of the rule against all exploiters and use the resulting meta-game matrix UU to compute an exploiter mixture μ⋆\mu^\star. This mixture minimises the highest value attained by any previous rule against it, i.e. each exploiter is scored by DUOS’s loss, which equals the exploiter’s own gain in zero-sum IMP but not in IPD. We continue maximising J(η)J(\eta) (Equation (3)) against a mixture ρℓ\rho_\ell of self-play, naive learners, the exploiter mixture and fixed strategies (Appendix § 13, Supplementary Material). Each round trains the rule for 200 meta-training steps. After L=6L = 6 rounds, we finalise and deploy the rule (Algorithm 1).

Experiments

Games. We evaluate on four iterated matrix games: Iterated Prisoner’s Dilemma (IPD), Iterated Matching Pennies (IMP), Chicken and Stag Hunt, and additionally on Coin Game. Table 1 shows the payoff matrices for the 2×22 \times2 games.

IPDIMP
CooperateDefectHeadsTails
(3,3)(3, 3)(0,5)(0, 5)(1,−1)(1, -1)(−1,1)(-1, 1)
(5,0)(5, 0)(1,1)(1, 1)(−1,1)(-1, 1)(1,−1)(1, -1)
ChickenStag Hunt
SwerveStraightStagHare
(3,3)(3, 3)(1,4)(1, 4)(4,4)(4, 4)(0,3)(0, 3)
(4,1)(4, 1)(0,0)(0, 0)(3,0)(3, 0)(2,2)(2, 2)

Table 1. Payoff matrix for each 2×22 \times2 game.

Coin Game. Coin Game [37] is a sequential social dilemma game that extends IPD to a grid world, and the rules are given in Appendix § 10 (Supplementary Material). Because Coin Game has too many states for per-state log-odds changes, we train a variant of DUOS for it: after each episode, its rule instead outputs a multiplier αc∈(−1,1)\alpha_c \in(-1, 1) for each pickup scenario cc (which coin each player last collected), and the inner network takes one Adam step along ∑tαctA^t∇θlog⁡πθ(at∣ot,ct)\sum_t \alpha_{c_t}\hat{A}_t\nabla_\theta\log\pi_\theta(a_t \mid o_t, c_t), where A^t\hat{A}_t is an advantage estimate.

Adam’s second moment uses the gradient with α≡1\alpha\equiv1, so the step scales with α\alpha. Hence, where α≡1\alpha\equiv1, it is exactly like REINFORCE [62] (Table S13, Supplementary Material).

HyperparameterValue
Steps per episode16
Episodes per lifetime16
Parallel games512
Inner network hidden units32, 32
Adam second momentfrom the α≡1\alpha\equiv1 gradient
Advantage A^t\hat{A}_treturn minus ridge baseline
Inner learning rate (Adam)0.05
Return discount γ\gamma0.96
Rule GRU units32
Rule MLP hidden units64, 64
Pickup scenarios9
Noise σ\sigma on multipliers0.3
Lifetimes per meta-training step256
Opponent mixture (Self, α≡1\alpha\equiv1 learner, greedy, own-coin)0.4, 0.4, 0.1, 0.1
Meta learning rate (Adam)10−310^{-3}
Base meta-training steps500
PSRO rounds ×\times meta-training steps3×1503 \times150
PSRO mixture (Self, α≡1\alpha\equiv1, greedy, own-coin, exploiters)0.28, 0.28, 0.07, 0.07, 0.30
Exploiter GRU units (ES)16
Exploiter MLP hidden units (PPO)64, 64
Exploiter iterations (ES, PPO)150, 150
α≡1\alpha\equiv1 learner’s learning-rate factor (training)[0.5, 2]
Deployed multipliers αc\alpha_c (measured)0.99–1.00
Training seeds3

Table S13. Hyperparameters of DUOS in Coin Game.

Deployment. At deployment, DUOS’s meta-network is frozen and no training noise is added. In each match, DUOS plays against its opponent for 20 rounds using its current policy and updates its inner network for the next match. The inner-network parameters and recurrent memory persist across matches and reset at the start of each lifetime. Unlike meta-training, tournaments score undiscounted 20-round matches over 50-match lifetimes and use a naive learner that learns faster than any seen in training.

Baselines. We run rigorous reproduction tests before using the baselines for comparison against our method. Each baseline uses its authors’ settings and is checked against their results (where comparable) before entering the tournament (Appendix § 12, Supplementary Material). We flag SHAPER [29], which falls slightly short in IPD. M-FOS passes and performs better in Coin Game than reported. Reciprocator passes with its released code but not with the paper’s published formula, so we use the released code, and its Coin Game check is also flagged. Advantage Alignment [16] and LOQA [2] did not pass and are excluded. All hyperparameters and protocols are in Appendix § 14 (Supplementary Material).

Evaluation. One tournament is held per game. DUOS, M-FOS [42], SHAPER [29] and NL participate in all five games. LOLA-OM [20] and POLA-OM* participate in the four matrix games, where POLA-OM* builds on Outer-POLA [67], which computes POLA’s proximal update from exact values for tabular policies. Like POLA’s opponent-modelling variant (POLA-OM), POLA-OM* replaces the opponent’s policy with a model fitted to its observed actions. IPD and IMP additionally include the history-based M-FOS variant (M-FOS-H) [42]. COALA-PG*, our pure-shaping reconstruction of COALA-PG [47], is included in IPD. Reciprocator [68] is included in IPD, Chicken, Stag Hunt and Coin Game.

In the matrix games, every pairing plays 128 lifetimes ×\times 50 matches ×\times 20 rounds. We evaluate five training seeds for DUOS, M-FOS, M-FOS-H and COALA-PG*, three for SHAPER, and none for LOLA-OM, POLA-OM*, Reciprocator and NL because they learn online and do not require training.

Information access differs between methods: DUOS uses only its own payoff matrix and observed match trajectories, whereas M-FOS, NL and Reciprocator receive the opponent’s policy, with memory-one approximations for recurrent opponents. Reciprocator also uses the opponent’s payoff matrix and 2,048 unscored practice matches of 20 rounds per update against the opponent. M-FOS-H reads its own policy and the 20 joint actions of the last match through a memory that resets after each match. LOLA-OM and POLA-OM* use observed actions and both payoff matrices and assume an NL update. SHAPER and COALA-PG* only use their own history.

Scoring. A method’s score is defined as its undiscounted mean reward per round (per step for Coin Game), averaged equally over its opponents, excluding self-play. To compare across varying payoff scales, we normalise the score of each game:

ng(a)=sg(a)−min⁡b∈Egsg(b)max⁡b∈Egsg(b)−min⁡b∈Egsg(b)n_g(a) = \frac{s_g(a) - \min_{b \in E_g} s_g(b)}{\max_{b \in E_g} s_g(b) - \min_{b \in E_g} s_g(b)}

where sg(a)s_g(a) is method aa’s score and EgE_g is the set of entrants in game gg. The normalised mean score averages ng(a)n_g(a) over all games equally. In IMP, sgs_g weights both player roles equally.

Zero-shot transfer. To observe effects of generalisation, DUOS plays Chicken and Stag Hunt zero-shot with its IPD rule as well as in self-play.

Results

Head-to-Head

Figure 3 and Table 2 compare the tournament scores across all five games. The full matchup results are given in Supplementary Material (Appendix B). DUOS ranks 1st by normalised mean score, both over the four 2×22 \times2 matrix games and over all five games, where it plays zero-shot in Chicken and Stag Hunt, trained only in IPD. We only rank methods that have participated in all of the five games. Individually, DUOS ranks 1st in IPD, 2nd in IMP, 3rd in Chicken, 5th in Stag Hunt and 3rd in Coin Game.

Figure 3. Tournament scores after 50 matches (reward per round) or 16 episodes for Coin Game (reward per step). Self-play is excluded. Top row: DUOS trained on each game. Bottom row: DUOS trained on IPD playing zero-shot.

MethodIPDIMPChickenStag HuntCoinMatrix meanFive-game meanRank
DUOS (ours)1.0000.8410.8290.7270.6800.8490.815 [0.78, 0.85]1
SHAPER0.6311.0000.0320.9021.0000.6410.713 [0.64, 0.78]2
M-FOS0.5830.6460.0001.0000.8440.5570.614 [0.57, 0.66]3
NL0.1500.0000.8680.6370.3020.4140.391 [0.35, 0.43]4
LOLA-OM0.6590.3500.6900.868–0.642––
POLA-OM*0.3860.7080.0050.925–0.506––
M-FOS-H0.2920.751––––––
Reciprocator0.159–1.0000.0000.000–––
COALA-PG*0.000–––––––

Table 2. Normalised tournament scores. Each game’s scores are normalised in accordance with the scoring criteria (1 = best, 0 = worst). Matrix mean: four 2 × 2 matrix games. Five-game mean: all five games for the methods that play all five (95% seed-bootstrap intervals). Rank: by five-game mean. NL in Coin Game is a PPO learner [54]. Bold indicates the best result in each column.

Opponent shaping: Figure 2 shows that in IPD, both players are initially cooperating in about half of the rounds. By the end, NL has cooperated in 92.9% of the rounds and DUOS only in 16.4%. As such, DUOS’s mean reward per round during its final rounds has doubled compared to its initial rounds, and vice versa for NL. Separately, we replay each leftist with DUOS’s inner network frozen at its final policy from the first match, and show that NL cooperates in only 14.1% of the rounds (Table S27, Supplementary Material). Therefore, NL’s cooperation is produced by how DUOS’s policy changes over time, not by its final strategy alone, which is consistent with opponent shaping behaviour.

Figure 2. IPD learning trajectories over 50 matches. Left: rewards of DUOS against a naive learner (NL). Centre: joint-action frequencies against NL (DUOS’s action first, C: cooperation, D: defection). Right: mutual cooperation in self-play, within and across training seeds. Curves average five DUOS seeds. Shaded bands show 95% tt-intervals.

MeasureActualFrozendd [95% CI]
NL cooperation, last match0.929 ±0.010\pm0.0100.141 ±0.019\pm0.019−0.788-0.788 [−0.823-0.823, −0.754-0.754]
DUOS reward per round, last match4.387 ±0.017\pm0.0171.371 ±0.051\pm0.051−3.016-3.016 [−3.172-3.172, −2.861-2.861]
DUOS reward per round, lifetime3.961 ±0.024\pm0.0241.421 ±0.046\pm0.046−2.540-2.540 [−2.721-2.721, −2.358-2.358]
NL reward per round, lifetime0.900 ±0.030\pm0.0301.145 ±0.023\pm0.023+0.245+0.245 [+0.209+0.209, +0.281+0.281]

Table S27. Frozen-table control in IPD against NL, after 50 matches (5 seeds ×\times 128 lifetimes). Actual: DUOS’s recorded lifetimes. Frozen: the same NL, from the same first-match policy, against DUOS’s final (match-50) table held fixed from the first match. Exact expectations over 20-round matches. Mean ±\pm s.e. over seeds. dd: frozen minus actual, with a paired 95% tt-interval (all 5 seeds have the same sign).

IPD: DUOS obtains the highest mean score in IPD despite losing most head-to-head matchups against opponents (6 out of 8, Table S26, Supplementary Material). Head-to-head matchups do not fully translate to final rankings in general-sum games because the scores count the reward each method obtains, not its margin over the opponent. For example, winning 2.10 to 1.30 scores less than mutual cooperation at 3.0 each. This reflects the meta-objective (Equation (3)), which maximises expected return rather than head-to-head margins. This finding mirrors Axelrod’s tournaments, where tit-for-tat won on total score without outscoring any single opponent [5].

OpponentIPDIMPChickenStag HuntCoin
M-FOS−0.268 (0/5)+0.124 (4/5)−0.195 (0/5)+0.141 (5/5)−0.216 (0/3)
M-FOS-H−0.260 (0/5)+0.026 (3/5)–––
SHAPER−0.201 (0/5)+0.019 (4/5)−0.206 (0/5)+0.120 (5/5)−0.500 (0/3)
COAL-PG*−0.017 (2/5)––––
LOLA-OM−0.182 (0/5)+0.313 (5/5)−0.102 (0/5)+0.092 (5/5)–
POLA-OM*−0.197 (0/5)+0.004 (4/5)−0.219 (0/5)+0.065 (5/5)–
NL (Coin: PPO NL)+3.068 (5/5)+0.385 (5/5)+1.769 (5/5)+0.869 (5/5)+0.446 (3/3)
Reciprocator+0.710 (5/5)–+0.647 (5/5)+0.757 (5/5)+0.935 (3/3)
Pairings won2/86/62/66/62/4

Table S26. Mean head-to-head margin of DUOS (its reward minus the opponent’s, per round, or per step in Coin Game) and, in parentheses, the seeds with a positive margin. Matrix games after 50 matches, Coin Game after 16 episodes. –: not in that game.

IMP: DUOS wins against every opponent in IMP, and is the only method that SHAPER does not beat (Table S2, Supplementary Material). This can be partly credited to PSRO training, which lowers DUOS’s exploitability from 0.376 to 0.144 and reduces how much M-FOS and SHAPER, which specialise in exploiting learners, gain against it (Table S24, Supplementary Material).

Table S2: IMP tournament after 50 matches. Each entry is the row method’s mean reward per round against the column method, averaged over the two seats (so the matrix is antisymmetric). Self-play is excluded from the mean score. DUOS’s diagonal entry pools same-seed and cross-seed pairings of its training seeds. DUOS’s row is in bold.

Row vs. columnM-FOSM-FOS-HSHAPERLOLA-OMPOLA-OM*NLDUOSMean scoreRank
M-FOS–−0.04−0.350.16−0.120.47−0.060.015
M-FOS-H0.04–−0.150.04−0.050.40−0.010.043
SHAPER0.350.15–0.110.010.13−0.010.121
LOLA-OM−0.16−0.04−0.11–−0.02−0.02−0.16−0.086
POLA-OM*0.120.05−0.010.02–0.000.000.034
NL−0.47−0.40−0.130.020.00–−0.19−0.207
DUOS (ours)0.060.010.010.160.000.190.000.072

Table S2.

Table S24. What PSRO adds: the deployed rule against its pre-PSRO base (control) on the same 5 seeds. dd: after minus before, with a paired 95% tt-interval. (a) IMP head-to-head margin after 50 matches. (b) Exploitability: a fresh exploiter’s value against the rule minus the value that a reference strategy (IPD: tit-for-tat, IMP: the Nash player) obtains against the rule. Lower is better. IMP has separate rules for the two seats, so it has 10 runs (2 seats × 5 seeds). Its interval pairs the 5 seeds, each averaged over its two seats.

Chicken: Playing Chicken zero-shot, DUOS ranks 3rd with 1.52 per round, and 1st without SHAPER participating (1.81, Table S7, Supplementary Material). SHAPER and M-FOS play Straight in Chicken and DUOS does not yield to them, so DUOS ends up mostly crashing, which lowers its score. This might be due to SHAPER and M-FOS training only against naive learners, and thus learning mainly to exploit. Consistent with this, SHAPER and M-FOS have the two lowest Chicken self-play scores (Table 3). The Chicken result suggests that learned rules transfer across different settings, as also observed in single-agent reinforcement learning [48], although a rule trained on Chicken itself still scores higher (2.105 vs. 1.810, Table S25, Supplementary Material).

MethodIPDChickenStag HuntCoinMatrix meanFour-game meanRank
DUOS (ours)1.0001.0000.8831.0000.9610.971 [0.969, 0.973]1
Reciprocator0.6210.8040.0000.9540.4750.595 [0.590, 0.601]2
NL0.0310.8480.1010.9000.3270.470 [0.465, 0.476]3
M-FOS0.0050.0730.9700.7740.3490.456 [0.396, 0.499]4
SHAPER0.3270.0001.0000.0000.4420.332 [0.274, 0.364]5
LOLA-OM0.4280.5160.554–0.499––
POLA-OM*0.0580.0790.782–0.306––
COALA-PG*0.091––––––
M-FOS-H0.000––––––
M-FOS (meta-self-play)‡1.028––1.248–––

Table 3. Normalised self-play scores: each method’s mean own reward against a same-seed copy of itself per round, averaged over a lifetime of 50 matches (per step over 16 episodes in Coin Game). Each game’s scores are normalised in accordance with the scoring criteria. IMP is omitted because it is zero-sum. ‡Outside the tournament (two learners of each run play each other).

MethodChicken withoutChicken withStag Hunt withoutStag Hunt with
M-FOS1.60 (6)1.34 (7)3.61 (1)3.67 (1)
SHAPER–1.35 (5)–3.58 (3)
LOLA-OM1.78 (2)1.49 (4)3.52 (3)3.55 (4)
POLA-OM*1.60 (5)1.34 (6)3.54 (2)3.60 (2)
NL1.64 (4)1.53 (2)3.24 (5)3.34 (6)
Reciprocator1.70 (3)1.56 (1)2.68 (6)2.76 (7)
DUOS (ours)1.81 (1)1.52 (3)3.42 (4)3.42 (5)

Table S7. Chicken and Stag Hunt scores (rank) after 50 matches, without SHAPER (original field) and with SHAPER.

Table S25. Zero-shot transfer: the IPD rule played zero-shot compared with rules trained in Chicken and Stag Hunt, each entered in place of it in the same field, after 50 matches (5 seeds, in the field without SHAPER of Table S7, with the trained rules using the IPD recipe, not retuned). dd: trained minus zero-shot, with a paired 95% tt-interval. Seeds: seeds on which the trained rule is better. Self-play return: same-seed pairs from the same tournament runs. Exploitability is as in Table S24, with tit-for-tat as the reference. A higher score can come with a lower rank because the other entrants’ scores also change: the Chicken-trained rule yields to Straight, which raises M-FOS’s score from 1.60 to 2.31 and POLA-OM*’s from 1.60 to 2.20.

Stag Hunt: DUOS plays Stag Hunt zero-shot and ranks 5th in the tournament with a score of 3.42, only 0.25 behind M-FOS in 1st. Yet, DUOS wins every head-to-head matchup here, the opposite of its IPD result. In Stag Hunt, a player out-earns its opponent only by playing Hare (3) against Stag (0), giving up mutual Stag (4), on which the other methods coordinate. Most of the gap is the cost of transferring from IPD: in the tournament without SHAPER, when trained in Stag Hunt itself, DUOS scores 0.25 more and ranks 2nd instead of 4th (Table S25, Supplementary Material).

Coin Game: DUOS ranks 3rd of five with 0.048 per step (Table S5, Supplementary Material). Head to head, DUOS out-earns NL and Reciprocator but loses to M-FOS and SHAPER. DUOS learns a rule that is akin to REINFORCE [62]: its multiplier α\alpha is always around 1.00, and thus DUOS learns like a plain (α=1\alpha= 1) learner rather than shaping its opponent. This is likely a limit of the rule form, which can only rescale the REINFORCE step by a multiplier in (−1,1)(-1, 1).

Table S5: Coin Game tournament after 16 episodes: the row method’s mean reward per step against each column method. DUOS’s self-play (diagonal) entry pools same-seed and cross-seed pairings of its training seeds and is excluded from the score. NL is M-FOS’s PPO naive learner. M-FOS (stronger than in its paper) and Reciprocator are flagged (Appendix § 12).

Row vs. columnM-FOSSHAPERNL (PPO)ReciprocatorDUOSMean scoreRank
M-FOS–−0.2980.4110.5670.0970.1942
SHAPER0.032–0.5340.6430.1270.3341
NL (PPO)−0.447−0.754–0.259−0.216−0.2894
Reciprocator−0.618−0.865−0.272–−0.482−0.5595
DUOS (ours)−0.119−0.3730.2300.4540.0140.0483

Table S5.

Self-Play

A widely adopted algorithm will often face copies of itself. M-FOS reaches cooperation in self-play through a meta-self-play curriculum [42]. DUOS is trained against a population that includes 40% self-play. In IPD, DUOS reaches mutual cooperation (Figure 2, right), with both players cooperating in 93.4% of rounds, eventually reaching 99.8% during the final rounds. We pair different training seeds with each other, and also observe similar trajectories, showing that cooperation extends beyond exact same copies. DUOS therefore behaves differently against NL (Figure 2) than against another copy of itself: its changing policy makes NL more cooperative, which it then exploits, but it reciprocates mutual cooperation in self-play. These trajectories are consistent with opponent shaping.

Tournaments: DUOS clearly outperforms all baselines in self-play score (Table 3) and ranks 1st in 3 out of 4 games against methods tested in all four (deployed zero-shot in Chicken and Stag Hunt). Outside of the tournaments, M-FOS (meta-self-play) scores higher than DUOS, which we suspect stems from the different training mixtures. Hence, we train an exploratory self-play variant of DUOS with one rule shared by both copies and compare it with M-FOS (meta-self-play) in Figure 4 (Appendix § 15).

Figure 4. Learning curves of the variants trained for self-play. (a) IPD: mean over 5 seeds (s.e. ≤ 0.009). (b) Coin Game (3 seeds), reward per step. (c) DUOS (self-play) in the three matrix games with normalised reward: reward per round divided by the game’s mutual-cooperation payoff (3, 3 and 4), s.e. ≤ 0.007.

DUOS (self-play) vs. M-FOS (meta-self-play): The two methods are trained differently. DUOS (self-play) is trained only in self-play from the start with no naive learners, fixed strategies or PSRO. M-FOS (meta-self-play) instead follows its meta-self-play curriculum, where two separately initialised learners play each other with increasing probability, and otherwise play against naive learners. DUOS (self-play) is trained on 256,000 self-play lifetimes in IPD and 243,200 in Coin Game, whereas M-FOS (meta-self-play) is trained on around 2.8 million and around 900 lifetimes, respectively. In IPD, M-FOS (meta-self-play) hears mutual cooperation from the first match, and is fully cooperative with sampling noise off. DUOS (self-play) starts from a fresh random policy (with a trained meta-network) and fully converges by match 9–10. In Coin Game, the reward of M-FOS (meta-self-play) stays roughly constant, while DUOS (self-play) stays near zero for the first few episodes and then increases, overtaking M-FOS (meta-self-play) after episode 10. Over the lifetime, both methods are level (0.075 vs. 0.073 per step). Trained in each game, DUOS (self-play) also converges to mutual cooperation in Chicken and Stag Hunt, akin to its behaviour in IPD.

Conclusion

In single-agent reinforcement learning, discovered algorithms [22, 23, 28] have achieved state-of-the-art results [48]. We introduce DUOS, to our knowledge the first method to discover an update rule for an agent in general-sum games. Without reading the opponent’s policies or parameters, DUOS ranks 1st in normalised mean score across all games, and 1st for self-play among tournament entrants. In IPD, it exploits a naive learner while still cooperating with a copy of itself, consistent with opponent shaping. The IPD-trained rule also transfers zero-shot to two other 2×22 \times2 matrix games, Chicken and Stag Hunt, with competitive scores, as discovered rules also do in single-agent settings [48]. Meta-training against exploiters (PSRO) makes the rule harder to exploit, lowering its exploitability in IMP by more than half. In Coin Game, DUOS’s learned rule reduces to REINFORCE (α≈1\alpha\approx1) and ranks 3rd of 5 entrants. In self-play, DUOS (self-play) achieves full mutual cooperation in IPD, Chicken and Stag Hunt within 12 matches from a fresh policy, and eventually obtains more reward per step than M-FOS (meta-self-play) after 10 episodes in Coin Game. Our work has limitations. Each rule is trained per game, and in Chicken, a rule trained on

Chicken specifically still scores higher than the IPD rule played zero-shot. In Coin Game, shaping is likely limited by the rule form, which can only rescale the REINFORCE step. Future work could investigate meta-training one rule across several games, as DiscoRL’s discovered rule improved with more compute and more diverse training environments [48]. We could also investigate different rule forms and training settings that shape opponents and generalise to different sequential social dilemma games [36] such as Harvest [26, 52] and Cleanup [26].

References

Supplementary Material

Coin Game rules

We use the two-coin version of Coin Game [37] implemented in pax by SHAPER [29]. There are two players, Red and Blue, playing on a 3×33 \times3 grid whose edges wrap around (a torus). The grid holds one red and one blue coin, owned by Red and Blue respectively. The players and coins can share the same cells. At each step, both players move simultaneously and choose one of the five actions: right, left, up, down or stay. A player collects a coin by moving to or staying on the cell the coin is occupying. Collecting any coin gives the collector +1+1. If the coin belongs to the other player, the other player also receives −2-2. If both players collect the same coin, both receive +1+1 and its owner also receives −2-2 (net −1-1). A collected coin reappears on a uniformly random cell. An episode starts with the four positions drawn independently and uniformly over nine cells. The episode lasts 16 steps, of which 15 are paid: the 16th step applies the moves and resets the board, giving both players zero reward. Throughout the paper, a Coin Game reward per step is an episode’s total reward divided by its 15 paid steps.

Tournament results

Tables S1, S2, S3, S4 and S5 give every head-to-head result of the tournaments in Table 2. SHAPER was trained for Chicken and Stag Hunt and added to those fields after the first scores had been computed. Table S6 gives the normalised scores without it, and Table S7 compares the fields without and with it. Table S8 replaces DUOS with its pre-PSRO base rule in the same fields, with the same random draws.

Table S1: IPD tournament after 50 matches. Each entry is the row method’s mean reward per round against the column method. Self-play is excluded from the mean score. DUOS’s diagonal entry pools same-seed and cross-seed pairings of its training seeds. DUOS’s row is in bold.

Row vs. columnM-FOSM-FOS-HSHAPERCOALA-PG*LOLA-OMPOLA-OM*NLReciprocatorDUOSMean scoreRank
M-FOS–1.051.421.251.661.433.882.612.001.914
M-FOS-H1.04–1.381.261.361.113.132.452.091.736
SHAPER1.351.29–1.341.921.613.532.392.131.943
COALA-PG*1.181.131.36–1.461.362.022.181.621.549
LOLA-OM1.531.311.771.37–1.342.902.722.761.962
POLA-OM*1.361.091.591.361.39–3.752.281.481.795
NL1.081.831.571.932.601.26–1.930.901.648
Reciprocator1.551.581.831.552.471.241.08–1.821.647
DUOS (ours)1.731.831.931.612.581.283.962.532.932.181

Table S1.

Table S3: Chicken tournament after 50 matches, with SHAPER: the row method’s mean reward per round against each column method. DUOS plays its IPD rule zero-shot. Its self-play (diagonal) entry pools same-seed and cross-seed pairings of its training seeds and is excluded from the score.

Row vs. columnM-FOSSHAPERLOLA-OMPOLA-OM*NLReciprocatorDUOSMean scoreRank
M-FOS–0.060.380.153.913.230.301.347
SHAPER0.26–0.230.113.923.290.271.355
LOLA-OM0.110.06–0.403.182.652.541.494
POLA-OM*0.090.030.51–3.872.830.731.346
NL0.980.982.341.04–2.081.741.532
Reciprocator0.820.822.360.812.69–1.831.561
DUOS (ours)0.110.072.440.513.512.482.921.523

Table S3.

Table S4: Stag Hunt tournament after 50 matches, with SHAPER: the row method’s mean reward per round against each column method. DUOS plays its IPD rule zero-shot. Its self-play (diagonal) entry pools same-seed and cross-seed pairings of its training seeds and is excluded from the score.

Row vs. columnM-FOSSHAPERLOLA-OMPOLA-OM*NLReciprocatorDUOSMean scoreRank
M-FOS–3.973.773.893.932.913.573.671
SHAPER3.97–3.633.913.822.823.343.583
LOLA-OM3.833.68–3.513.743.073.473.554
POLA-OM*3.923.933.53–3.762.913.573.602
NL3.943.853.733.75–2.262.493.346
Reciprocator3.333.123.152.951.92–2.062.767
DUOS (ours)3.713.463.563.633.362.813.793.425

Table S4.

Table S6: Normalised tournament scores in which SHAPER plays only IPD, IMP and Coin Game. Columns are as in Table 2, without the rank column. SHAPER was only added to Chicken and Stag Hunt after these scores were computed.

MethodIPDIMPChickenStag HuntCoinMatrix meanFive-game mean
DUOS (ours)1.0000.8411.0000.7870.6800.9070.862 [0.84, 0.88]
M-FOS0.5830.6460.0001.0000.8440.5570.614 [0.57, 0.66]
NL0.1500.0000.1960.5960.3020.2360.249 [0.21, 0.29]
LOLA-OM0.6590.3500.8420.905–0.689–
POLA-OM*0.3860.7080.0400.920–0.514–
SHAPER0.6311.000––1.000––
M-FOS-H0.2920.751–––––
Reciprocator0.159–0.5020.0000.000––
COALA-PG*0.000––––––

Table S6.

GameKKWith PSROPre-PSRODifference
IPD502.182 (1)2.162 (1)+0.020 [−0.151, +0.191]
101.945 (1)1.922 (3)+0.023 [−0.024, +0.069]
IMP500.073 (2)0.029 (4)+0.043 [+0.022, +0.065]
100.054 (2)0.031 (2)+0.023 [+0.009, +0.037]
Chicken501.520 (3)1.531 (2)−0.011 [−0.055, +0.034]
101.362 (2)1.367 (2)−0.005 [−0.032, +0.023]
Stag Hunt503.422 (5)3.244 (6)+0.178 [−0.037, +0.392]
103.051 (6)2.915 (6)+0.136 [+0.060, +0.213]

Table S8. Mean score (rank in its own field) of DUOS with and without PSRO after KK matches, and their difference (with PSRO minus pre-PSRO) paired by seed with a 95% tt-interval. Bold: the interval excludes 0. Chicken and Stag Hunt: fields with SHAPER.

Baseline reproduction

Each baseline uses its authors’ published settings and is adapted to our settings where necessary. Appendix § 14 lists every setting, and † marks each setting that differs from the paper’s for a reason other than our tournament format. Before a baseline entered our tournaments, we checked our implementation against its own paper by reproducing it with different seeds against a reported result where the paper has one in a comparable setting, and otherwise against the paper’s settings. Each check and its pass rule were written down before the checks were run, except the post hoc COALA-PG* diagnostic described below. Table S9, Table S10 and Table S11 give every check, its target, our result and the outcome. Below, we report the flagged and excluded baselines.

Table S9: Baseline fidelity in the matrix games (I): reproduction checks against each baseline’s paper. Target: the target or bar set before the check, and n/r means no target.

BaselineCheckTargetOursOutcomeTournament
M-FOS, IPDTable 8 (authors’ payoffs), 2 seedsM-FOS −0.51 ± 0.1, NL −2.14 ± 0.2M-FOS −0.476, −0.474, NL −2.174, −2.176passentrant
M-FOS, IMPTable 5, authors’ code (B) vs ours at lr 0.002 (C)0.20, B ≥ 0.18, ∣C−B∣≤0.03\lvert C - B\rvert\le0.03B 0.194, C 0.194passentrant (lr 0.002)
M-FOS, ChickenTable 12M-FOS above NLM-FOS 3.94, NL 0.99 (5/5 seeds)passentrant
M-FOS, Stag Huntno paper resultn/rM-FOS 3.93, NL 3.95–entrant
M-FOS, meta-self-playself-play, K=50K = 50n/rIPD 2.988, IMP 0.012–self-play only
SHAPER, IPDrecovery, 3 seeds3.152.967failentrant (flagged)
SHAPER, IMPrecovery, both seats0.34, 0.520.719, 0.757passentrant
SHAPER, Chicken and Stag Hunttraining healthn/r6/6 runs healthypassentrant

Table S9.

Table S10: Baseline fidelity in the matrix games (II), as in Table S9. Counts are seeds meeting the condition, and cc is the payoff-range factor.

BaselineCheckTargetOursOutcomeTournament
POLA-OM*POLA’s steps in our payoff units5/c5/c, 0.3/c0.3/c, 0.1c0.1cc=5,2,4,4c = 5, 2, 4, 4–entrant
LOLA-OMLOLA’s Table 3 (exact LOLA, authors’ payoffs)n/r (paper −1.06, −1.98)LOLA −1.04, NL −1.98matchentrant
POLA-OM*Outer POLA vs authors’ code, Figure 3 and Table 3n/rwithin 10−910^{-9}, both reproducedmatchentrant
Reciprocator, IPDshaping gate, 8 seeds≥ 5/8paper formula 0/8, released code 8/8pass (code)entrant (code)
COALA-PG*, pure shapingFig. 5Areturn ≥ 0.401.376, 1.401passentrant (IPD)
COALA-PG*, mixed poolFig. 5Crecovery ≥ 0.50.035fail–
LOQAFigure 1 policy, 3 seeds≥ 2/3paper 1/3, code 0/3failexcluded
Advantage AlignmentFigure 1(b) policy, 3 seedsP(C | state) barsLOLA payoffs 0/3, shifted 0/3failexcluded

Table S10.

Table S11: Baseline fidelity in Coin Game, as in Table S9. M-FOS returns are in the authors’ units, summed over 240 paid steps (M-FOS / NL).

BaselineCheckTargetOursOutcomeTournament
M-FOS, authors’ codeupdates 951–1000, 3 seedsM-FOS ≥33.2\ge33.2, NL ≤−18.5\le-18.5 in ≥2/3\ge2/3 seeds70.89 / −75.00, 2/3 seedspass (stronger)–
M-FOS, paxsame, not gatedn/rM-FOS 113.55, 105.21, 75.94–entrant (flagged)
M-FOS, meta-self-playauthors’ code, self-play, 3 seeds≥15.4\ge15.4 in ≥2/3\ge2/3 seeds8.74, 14.44, 22.42fail–
M-FOS, meta-self-play paxself-play, 3 seeds, not gatedn/r29.95, 25.98, −3.48–self-play only
SHAPERpaper’s ES budgetn/r3/3 seeds completed–entrant
ReciprocatorNL own-coin rate, 4 seeds≥0.52\ge0.52 in ≥3/4\ge3/4 seeds, mean ≥0.56\ge0.56, controls ≤0.52\le0.520.617 (4/4), controls 0.473, 0.531uninformativeentrant (flagged)
PPO NLM-FOS’s tbl:20n/rsettings match–entrant

Table S11.

Flagged baselines. SHAPER passes its checks in IMP but falls slightly short in IPD (three-seed mean 2.97 against a bar of 3.15, and one of three seeds passed). The paper reports SHAPER only in its own 100×100100 \times100 format, so this check asks whether training in our 50×2050 \times20 format recovers at least half of the reward that a model trained in the paper’s format loses in ours: the bar is the midpoint of that model’s reward in our format (2.67) and in its own (3.63). The IMP bars are set in the same way. We enter the model trained with the budget of the authors’ configuration files (Table S16) and flag its IPD result. M-FOS passes its Coin Game check on two of three seeds and its returns exceed the paper’s, and is flagged as stronger than in its paper. The check used the authors’ four-action game. The entrant, trained in the same way in our five-action game and not gated, returns more still. Reciprocator fails our checks with its paper’s formula but passes with its released code, which we enter. Its Coin Game check is uninformative, because in one of the two control runs, two naive learners without a Reciprocator also exceeded the bar. Its shaping also needs more episodes than our 16-episode lifetimes (in the paper the effect appears after about 25 episodes), so it is flagged in Coin Game. We also scored Coin Game without it, and the order of the other four entrants is unchanged (SHAPER 0.231, M-FOS 0.070, DUOS −0.087, PPO naive learner −0.472).

HyperparameterValue
GRU units16†
ES population1,000
Generations1,000†
ES learning rate0.1†
Perturbation scale0.04
Opponents per candidate10
Matches per lifetime (trial)50
Rounds per match20
NL learning rate1
NL epochs ×\times minibatches2×42 \times4
NL discount γ\gamma0.96
Training seeds3

Table S16. Hyperparameters of SHAPER in the matrix games. †The paper’s Table 11 (IPD) lists a GRU of 25 units, 5,000 generations and an ES learning rate of 0.01. We use the authors’ configuration files (whose GRU size matches the parameter count in the paper’s Table 20), as in Coin Game (Table S17).

HyperparameterValue
Linear and GRU units16, 16
ES population4,000
Generations3,000
ES learning rate0.1{sup}†`
Perturbation scale0.04
L2 penalty0.1
Parallel games512
Episodes per lifetime (trial)16
NL learning rate0.005
NL epochs ×\times minibatches2 ×\times 8
Training seeds3

Table S17. Hyperparameters of SHAPER in Coin Game. {sup}†`The paper’s table lists 0.01. We use the authors’ configuration file.

COALA-PG⁎. Our reconstruction of COALA-PG passes the check on the pure-shaping experiment of its paper (Fig. 5A), learning faster than the paper’s curve. The entrants use one optimiser step per iteration, which a speed check found to match the paper’s pace. It does not reproduce the mixed-pool training (Fig. 5C) either in the check or in the post hoc diagnostic with the paper’s optimiser step count. COALA-PG⁎ therefore plays only IPD, trained in the pure-shaping setting. The asterisk marks it as our reconstruction. There is no released code for the method.

Excluded baselines. Advantage Alignment has no released code for its IPD experiment: the loss in the authors’ released code is written for other environments, and a part of it did not reproduce the paper’s IPD result. We rebuilt the method on the code base of LOQA, with the loaded-aa loss and the paper’s IPD settings, and tried to reproduce Figure 1(b) from the paper on 3 seeds with the IPD payoffs used by LOLA, POLA and LOQA, since the paper does not print its own, and on 3 seeds with every payoff shifted by +1.5. None of the seeds passed (0 of 6). LOQA reproduced Figure 1 on 1 of 3 seeds with the settings in its paper and on 0 of 3 with those in its released code. Both methods are therefore excluded from the tournaments.

Other variants. M-FOS-H, our matrix-game version of the history-based M-FOS of the paper’s Coin Game experiments, has no paper result to check against. The M-FOS meta-self-play curriculum fails its Coin Game check with the authors’ code (1 of 3 seeds). The Coin Game version in the self-play comparison (Table 3 and Figure 4) was trained in the same way on our five-action game and was not gated (Appendix § 15).

Implementation details of DUOS

This section gives the details of DUOS that Section 4 leaves out. Tables S12 and S13 list its settings.

Rule inputs. After match kk, the features xk,sx_{k,s} of state ss are computed from the agent’s own record DikD_i^k of the match. The feature ese_s is a one-hot code of ss. The visit share is vs=ns/Tv_s = n_s/T, where nsn_s is the number of rounds played in ss. The opponent’s smoothed frequency of action 0 is q^s=(nso+1)/(ns+2)\hat{q}_s = (n_s^o + 1)/(n_s + 2), where nson_s^o is the number of those rounds in which the opponent played action 0. The agent’s probability of action 0 before the update is ps=πθk(0∣s)=σ(dθk(s))p_s = \pi_{\theta_k}(0 \mid s) = \sigma(d_{\theta_k}(s)). Finally, rˉs\bar{r}_s is the agent’s mean reward in ss divided by κ\kappa, the largest absolute payoff in the agent’s payoff matrix MiM^i (κ=5\kappa= 5 in IPD, 1 in IMP and 4 in Chicken and Stag Hunt). A state that was not visited has vs=0v_s = 0, q^s=1/2\hat{q}_s = 1/2 and rˉs=0\bar{r}_s = 0, and still receives a change.

Meta-network. The GRU has 32 units and one memory shared by all states. After each match, it reads the five feature vectors xk,sx_{k,s}, the match’s mean reward divided by κ\kappa and the agent’s payoff matrix Mi/κM^i/\kappa (50 inputs). For each state, one shared MLP with two hidden layers of 64 units maps xk,sx_{k,s} and the GRU’s memory to a change in (−1,1)(-1, 1). All its units, including the output, are tanh. The meta-network has 14,881 parameters. Its MLP weights and biases are drawn from U(±1/fan-in)\mathcal{U}(\pm1/\sqrt{\text{fan-in}}) and its GRU weights from U(±1/32)\mathcal{U}(\pm1/\sqrt{32}). The GRU biases start at 0.

Action symmetry. The rule is applied twice to every match: once as observed, and once with the two actions relabelled. Relabelling exchanges the states (0,0)(0, 0) and (1,1)(1, 1) and the states (0,1)(0, 1) and (1,0)(1, 0), swaps the rows and columns of the payoff matrix and negates the log-odds. Each view has its own GRU memory, and the change from the relabelled view is mapped back (exchanging the states and negating) and averaged with the other, which gives yk,sy_{k,s}. The update is therefore the same whichever action is called 0.

Inner network. The inner network has two hidden layers of 32 tanh units and 1,314 parameters. Its input codes the state with five indicators (the start state, and one indicator per action for the agent’s and the opponent’s last actions), and its two outputs z0,z1z_0, z_1 give the log-odds dθ(s)=z0−z1d_\theta(s) = z_0 - z_1. At the start of each lifetime, the network is fitted to initial log-odds drawn independently from N(0,1)\mathcal{N}(0,1) for each state.

Fitting the targets. After match kk, the targets are dsclip=clip⁡(dθk(s)+yk,s,−12,12)d_s^{\mathrm{clip}} = \operatorname{clip}(d_{\theta_k}(s) + y_{k,s}, -12, 12) (in training, with the perturbed change y~k,s\tilde{y}_{k,s} in place of yk,sy_{k,s}). The network is fitted to them with damped Gauss–Newton steps (Section 4.2). Each step is δ=JT(JJT+Λ)−1u\delta= J^\mathsf{T}(JJ^\mathsf{T} + \Lambda)^{-1}u, where us=dsclip−dθ(s)u_s = d_s^{\mathrm{clip}} - d_\theta(s), the row of JJ for state ss is Js=∂dθ(s)/∂θJ_s = \partial d_\theta(s)/\partial\theta, and Λ=λdiag⁡(m)−1\Lambda= \lambda\operatorname{diag}(m)^{-1}. Equivalently, δ\delta minimises ∑sms(Jsδ−us)2+λ∥δ∥2\sum_s m_s(J_s\delta-u_s)^2 + \lambda\lVert\delta\rVert^2. The weight msm_s is the number of rounds played in state $s, or 1 if ss was not visited. The damping λ\lambda is tiny: $10^{-6}$ times the median of the five states. So δ\delta is close to the smallest parameter change that matches the targets to first order. Two more steps recompute JJ, λ\lambda and uu at the new parameters. The fit then keeps as θk+1\theta_{k+1} whichever of five candidates has the smallest max⁡s∣dθ(s)−dsclip∣\max_s |d_\theta(s) - d_s^{\mathrm{clip}}|: no change, the parameters after each of the three steps, or the midpoint between θk\theta_k and the parameters after the third step.

Meta-training. Each meta-training step draws 32 pairs of initial policies for every opponent type of the mixture and plays each pair four times with independent match and noise draws, giving 128 lifetimes per type, whose average gradients are weighted by ρ\rho. The base rule η(0)\eta^{(0)} is trained for 800 meta-training steps, and the gradient’s global norm is clipped to 1 before each Adam step (learning rate 10−310^{-3}). In IMP, the matching and mismatching seats have separate rules. In IMP self-play, a copy of the seat’s rule plays the other seat, and the credited mean value is zero.

Gradient estimate. The four replays share the initial policies and the opponent (in PSRO, also the drawn naive learner, exploiter or fixed strategy). This makes the leave-one-out baseline bkb_k a close one. Because their match and noise draws are independent, bkb_k does not depend on the lifetime’s own noise, so the estimator stays unbiased. The noise σξk\sigma\xi_k in y~k=yk+σξk\tilde{y}_k = y_k + \sigma\xi_k, with ξk∼N(0,I)\xi_k \sim\mathcal{N}(0,I) and σ=0.3\sigma= 0.3, is added to the changes of all five states before the clip. Hence πˉη(⋅∣Hk)=N(yk,σ2I)\bar{\pi}_\eta(\cdot\mid\mathcal{H}_k) = \mathcal{N}(y_k,\sigma^2I), where Hk=(Mi,πji,Dji)\mathcal{H}_k = (M^i,\pi_j^i,D_j^i) holds the agent’s payoff matrix and its policy and trajectory in each match j≤kj \le k. The values VkcrV_k^{\mathrm{cr}}, k=0,…,Nk = 0, \ldots, N, are computed exactly before each match and after the last update. The gradient passes through both GRU memories over the whole lifetime, but not through the game, the opponent’s update or the fit.

Naive learners. The naive learners in training take one exact gradient step (Equation (2)) on their value VV of Equation (1) after each match and read DUOS’s exact table. The step size α\alpha is 3 in the base stage and is drawn log-uniformly from [1,10][1, 10] for each opponent in PSRO. The tournament’s naive learner steps on the unnormalised value V/(1−γ)V/(1-\gamma) with step size 1, which is α=25\alpha= 25 on VV, so it learns faster than any naive learner seen in training. AllC and AllD have log-odds 8 and −8-8 in every state.

Training against exploiters. Each PSRO round ℓ\ell trains a fresh exploiter ψ(ℓ)\psi^{(\ell)} against the latest rule without noise. The exploiter is an MLP with two hidden layers of 32 tanh units (1,605 parameters). Before each match kk, it reads DUOS’s current table (privileged information), its own previous table and k/Nk/N, and outputs its own table for the match. Its inputs are centred at 1/2 and scaled by 4, and its outputs are scaled by 4. Its first previous table is σ(ϕ)\sigma(\phi) with ϕ∼N(0,I)\phi\sim\mathcal{N}(0, I). It is trained by antithetic evolution strategies to maximise its own horizon-weighted value ∑k=0NwkVki\sum_{k=0}^{N} w_k V_k^{\mathrm{i}}: 150 iterations of 64 directions, each evaluated on 32 lifetimes with common random numbers, with perturbation scale 0.1 and Adam (learning rate 0.03).

Meta-game and training mixture. The meta-game matrix U(ℓ)U^{(\ell)} holds DUOS’s horizon-weighted value ∑k=0NwkVki\sum_{k=0}^{N} w_k V_k^{\mathrm{i}} without noise, estimated on 512 common lifetimes, for each earlier rule η(0),…,η(ℓ−1)\eta^{(0)}, \ldots, \eta^{(\ell-1)} against each exploiter ψ(1),…,ψ(ℓ)\psi^{(1)}, \ldots, \psi^{(\ell)}. The exploiter mixture μ⋆\mu^\star, which minimises the largest value of any earlier rule against it, is found by linear programming in both IPD and IMP. The rule is then trained for 200 meta-training steps against the mixture ρℓ\rho_\ell of self-play (0.30), naive learners (0.25), exploiters drawn from μ⋆\mu^\star (0.30) and fixed strategies (0.15). The fixed strategies are drawn uniformly from tit-for-tat, generous tit-for-tat (forgiving with probability 1/3), win-stay-lose-shift (staying after a payoff above the mean of the payoff matrix), always action 0, always action 1, and a strategy that plays a random near-deterministic table (log-odds ±3\pm3) for 1 to 10 matches and then learns naively. Adam’s state is reset at the first round and then carried over. After six rounds, the last rule is deployed. The rule mixture of the meta-game is not used.

Exploitability. For Table S24 and S25, a fresh exploiter is trained for 300 iterations with its own random seed, shared by the rules being compared, and evaluated on 1,024 held-out lifetimes of N=100N = 100 matches with the horizon weights wkw_k. Exploitability is its value minus the value that the reference strategy (the uniform Nash strategy in IMP, tit-for-tat in the other games) obtains against the same rule.

Coin Game rule. A pickup scenario cc is the pair of the two players’ last pickups in the current episode (none, its own coin or the other player’s coin). This gives nine scenarios. The agent tracks its scenario from its own observations, and the scenario resets at every episode. The policy network reads the agent’s observation and its current scenario. After each episode, the rule computes for each scenario a code of its identity, the share of the episode’s paid steps spent in it, the agent’s mean reward divided by 2, four pickup rates per paid step (each player taking its own or the other’s coin), the mean normalised advantage (clipped to [−5,5][-5, 5]) and the mean probability of the agent’s chosen actions. A GRU with 32 units reads these statistics, without the identity codes, for all nine scenarios and the episode’s mean reward (73 inputs). An MLP with two hidden layers of 64 tanh units maps each scenario’s statistics and the GRU’s memory to αc=tanh⁡(⋅)∈(−1,1)\alpha_c = \tanh(\cdot) \in(-1, 1). In training, noise with σ=0.3\sigma= 0.3 is added to αc\alpha_c after the tanh.

Coin Game inner update. The advantage is A^t=Rt−ftTβ\hat{A}_t = R_t - f_t^{\mathsf{T}}\beta, where RtR_t is the discounted return to the end of the episode (γ=0.96\gamma= 0.96). The coefficients β\beta are fitted by ridge regression (penalty 1) on the previous episode (on the current one at the first update), with 47 features: the cells of the opponent and both coins, the scenario, the remaining time, its product with the scenario and a constant. The advantages are normalised over the batch. The inner update is one Adam step (learning rate 0.05). Its first moment uses the weighted gradient ∑tαctA^t∇θlog⁡πθ(at∣ot,ct)\sum_t \alpha_{c_t}\hat{A}_t\nabla_\theta\log\pi_\theta(a_t \mid o_t, c_t), and its second moment uses the same gradient with α=1\alpha= 1. The step is therefore linear in the multipliers of this and earlier updates. With α=1\alpha= 1 at every update, it is the plain REINFORCE step. Adam’s state persists over the 15 updates of a lifetime, and the policy network is re-initialised for each lifetime.

Coin Game meta-training. The training opponents are self-play, the same learner with α=1\alpha= 1 (a REINFORCE learner, not the tournament’s PPO), a scripted greedy agent that moves to the nearest coin, and a scripted agent that moves only to its own coin. Meta-training differs from the matrix games’ in four ways. (i) VkiV_k^{\mathrm{i}} is the mean reward per paid step in episode kk over the 512 games (a Monte Carlo estimate, not (1)), VkcrV_k^{\mathrm{cr}} is credited as in the matrix games, and the 16 episodes are weighted equally (wk=1/16w_k = 1/16, with N=15N = 15 updates). (ii) Each meta-training step plays 128 antithetic pairs of lifetimes that share every random draw except the sign of the noise on α\alpha. Here bkb_k is the mean of GkG_k over the other pairs against the same opponent type, and only scenarios visited in the episode enter log⁡πˉn\log\bar{\pi}_n. (iii) The α=1\alpha= 1 learner’s learning rate is multiplied by a factor drawn log-uniformly from [0.5,2][0.5, 2] for each lifetime. (iv) After 500 base meta-training steps, three PSRO rounds of 150 meta-training steps follow. Each round trains two fresh exploiters against the latest rule: a GRU policy trained by evolution strategies that also reads DUOS’s action probabilities on 16 fixed probe observations (privileged), and a memoryless policy trained by PPO. The meta-game entry is DUOS’s return minus the exploiter’s. Exploiters drawn from its minimax mixture make up 30% of training, and the other opponents keep their proportions in the remaining 70%. The last rule is deployed.

Hyperparameters and information access

Table S12 and S13 give the hyperparameters of DUOS. Those of the baselines are in Table S14 and S15 (M-FOS), S16 and S17 (SHAPER), S18 (LOLA-OM and POLA-OM*), S19 (COALA-PG*), S20 (Reciprocator) and S21 (naive learners). Table S22 gives the tournament protocol, and Table S23 the information each method uses at deployment.

HyperparameterValue
Rounds per match TT20
Matches per training lifetime NN100
Objective horizons hh10, 25, 50, 100
Discount γ\gamma0.96
Meta-network GRU units32
Meta-network MLP hidden units64, 64
Inner network hidden units32, 32
Noise σ\sigma on log-odds changes0.3
Lifetimes per opponent type per meta-training step128
Opponent mixture ρ0\rho_0 (Self, NL, AllC, AllD)0.4, 0.4, 0.1, 0.1
NL step size on VV of Eq. (1) (base, PSRO)3, log-uniform [1, 10]
Meta learning rate (Adam)10−310^{-3}
Base meta-training steps800
PSRO rounds ×\times meta-training steps6×2006 \times200
PSRO mixture (Self, NL, exploiters, fixed)0.30, 0.25, 0.30, 0.15
Exploiter evolution-strategy (ES) iterations, directions150, 64
Training seeds5

Table S12. Hyperparameters of DUOS in the matrix games.

HyperparameterValue
Actor and critic hidden units256
M-FOS-H GRU units16
Learning rate2×10−42 \times10^{-4}
Learning rate, IMP2×10−32 \times10^{-3}†
PPO epochs4
PPO clip0.2
Entropy coefficient0.01
Value-loss coefficient0.5
Meta discount0.99
Meta-updates1,024
Lifetimes per update4,096
Meta-steps per lifetime100
Inner discount γ\gamma0.96
NL step size on V/(1−γ)V/(1-\gamma) (25 on VV)1
NL step size on V/(1−γ)V/(1-\gamma), IMP0.1
Training seeds5

Table S14. Hyperparameters of M-FOS and M-FOS-H in the matrix games. †The paper’s Table 14 gives 2×10−42 \times10^{-4}, at which our IMP runs reach 0.134 (paper 0.20). The authors’ naive-learner script uses 2×10−32 \times10^{-3}, which reaches 0.194 (Table S9). The meta-self-play curriculum keeps 2×10−42 \times10^{-4}.

SettingValue
Matrix games
Matches per lifetime KK50
Matches per lifetime KK, secondary10
Rounds per match20
Lifetimes per pair of training seeds (IMP: per seat order)128
Trained seeds (DUOS, M-FOS, M-FOS-H, COALA-PG*)5
Trained seeds (SHAPER)3
Coin Game
Steps per episode16
Episodes per lifetime16
Parallel games512
Lifetimes per pair of training seeds and seat8
Trained seeds3
Protocol seed23962026

Table S22. Tournament protocol. Intervals for normalised scores are 95% percentile bootstrap intervals over 2,000 resamples (10,000 for Table 3) of each learned method’s training seeds within each game, recomputing scores, ranks and the min–max normalisation in every resample. Lifetimes are not resampled.

MethodOwn trajectoryOpponent’s actionsOpponent’s policyOpponent’s payoffsAssumed learning ruleExtra
Matrix games
DUOSlast match2observednononoown payoffs, own table
M-FOSnonotable3nonoown table
M-FOS-Hlast match4observednononoown table
SHAPERevery round5observednonononone
COALA-PG*every round5observednonononone
LOLA-OMlast matchobserved6fitted6matrixNL step7exact values
POLA-OM*last matchobserved6fitted6matrixNL step7exact values
Reciprocatorpractice only8in practice8table3matrixnopractice matches8
NLnonotable3nonoexact values
Coin Game
DUOSlast episodeno9nonononone
M-FOSevery stepnonorewards10nonone
SHAPERevery step5nononononone
Reciprocatorevery stepgiven11norewards11nonone
PPO NLlast episodenononononone

Table S23. Information each method uses at deployment.

HyperparameterValue
Convolution channels16, 16
Linear and GRU units16, 16
Learning rate2×10−42 \times10^{-4}
PPO epochs16
PPO clip0.2
Entropy coefficient0.01
Value-loss coefficient0.5
Meta discount0.99
Meta-updates1,000
Parallel games512
Episodes per lifetime (meta-episode)16
Training seeds3

Table S15. Hyperparameters of M-FOS in Coin Game.

HyperparameterValue
LOLA-OM step size1
LOLA-OM look-ahead step1
LOLA-OM discount γ\gamma0.96, 0.9, 0.96, 0.96
POLA-OM* inner step (5/c)(5/c)1, 2.5, 1.25, 1.25
POLA-OM* outer step (0.3/c)(0.3/c)0.06, 0.15, 0.075, 0.075
POLA-OM* KL weight (0.1c)(0.1c)0.5, 0.2, 0.4, 0.4
POLA-OM* proximal steps (maximum)5,000
POLA-OM* discount γ\gamma0.96
Opponent-model step size0.05
Opponent-model steps (first match, later)50, 1

Table S18. Hyperparameters of LOLA-OM and POLA-OM*. Lists give IPD, IMP, Chicken and Stag Hunt. Here cc is the payoff-range factor. Step sizes are on the unnormalised value V/(1−γ)V/(1-\gamma), as in Table S21.

HyperparameterValue
Matrix games (NL)
Step size on V/(1−γ)V/(1-\gamma) (25 on VV of Eq. (1))1
Discount γ\gamma0.96
Coin Game (PPO NL)
Convolution channels16, 16
Linear units16
Learning rate0.005
PPO epochs, clip80, 0.2
Entropy, value-loss coefficients0.01, 0.5
Discount γ\gamma0.96

Table S21. Hyperparameters of the naive learners.

HyperparameterValue
Hawk width, heads32, 2
PPO iterations3,000
Meta-batch128
Inner batch16
Matches ×\times rounds per lifetime (meta-episode)20 ×\times 10
Optimiser steps per iteration1{sup}†`
Learning rate3×10−43 \times10^{-4}
PPO clip0.2
Value coefficient0.5
NL learning rate0.005
NL pool size10
Training seeds5

Table S19. Hyperparameters of COALA-PG* in IPD. {sup}†The paper’s Table 4 gives 4 epochs $\times$ 2 minibatches. A speed check found one step per iteration to match the pace of its {xref}Fig. 5A`.

HyperparameterValue
Matrix games
Reciprocal reward weight5.0
Hidden units2
Learning rate0.005
Discount γ\gamma0.96
PPO epochs, clip, entropy10, 0.1, 0.02
Practice matches per update2,048
Rounds per practice match20
Coin Game
Reciprocal reward weight1.0
Convolution channels, GRU units4, 16
Learning rate0.005
PPO epochs, clip, entropy40, 0.15, 0.01
Influence-model learning rate0.01
Parallel games512

Table S20. Hyperparameters of Reciprocator.

Additional results

Self-play variants. DUOS (self-play) in Figure 4 is exploratory and post hoc. It was designed after Table 3 had been seen, and it is not a tournament entrant. It uses the entrant’s rule form, code and settings, but trains only in self-play from the first meta-training step, with no naive learners, fixed strategies or PSRO. Both seats run the same rule, each with its own inner network, and both copies are credited with their mean value. As the games are symmetric, this gives an unbiased estimate of the gradient, with respect to the shared rule, of one copy’s expected return, a common-interest objective.

It is trained separately in IPD, Chicken and Stag Hunt for 2,000 meta-training steps of 128 lifetimes of N=100N = 100 matches, and in Coin Game for 950 meta-training steps of 256 lifetimes of 16 episodes in 512 games (the entrant’s totals, 800 + 6 × 200 and 500 + 3 × 150), with 5 and 3 seeds. IMP is not trained, since the mean value, and hence the gradient, is zero there.

M-FOS curriculum. M-FOS (meta-self-play) instead follows its authors’ curriculum [42]: two separately initialised learners, each trained on its own reward, play each other with probability 1−λ1 - \lambda, and otherwise each plays its own batch of naive learners. The paper describes the schedule only qualitatively. In the authors’ code, which we follow, λ\lambda starts at 1 and falls by 0.0015 per update in IPD (1,024 updates of 4,096 lifetimes of 100 matches) and by 0.005 in Coin Game (1,000 updates of one lifetime of 16 episodes in 512 games). It reaches 0 at update 667 and 200 respectively, after which training is pure self-play. In the runs shown, 66–68% (IPD) and 89–91% (Coin Game) of updates were self-play, and the last naive-learner update came at update 619–650 and 169–195.

Comparison. Counted in self-play lifetimes, M-FOS (meta-self-play) trains on about 11 times as many as DUOS (self-play) in IPD (about 2.8 million vs. 256,000), while in Coin Game DUOS (self-play) trains on about 270 times as many (243,200 vs. about 900). In IPD, M-FOS observes both players’ tables, sets its policy directly and samples its table from its learned Gaussian at every match, as in the authors’ code. In both games, its self-play scores pair the two co-trained learners of one run. On the scale of Table 3, DUOS (self-play) scores 1.019 in IPD and 1.257 in Coin Game, against 1.028 and 1.248 for M-FOS (meta-self-play). Figure 4 therefore compares two methods, each trained for self-play with its own recipe and budget, not identical training regimes.

OpponentControlPSROdd [95% CI]Seeds PSRO better
M-FOS−0.020+0.124+0.144 [+0.057, +0.231]5/5
M-FOS-H−0.199+0.026+0.225 [+0.145, +0.305]5/5
SHAPER−0.092+0.019+0.111 [+0.065, +0.158]5/5
Pairings won (all 6 IMP opponents)3/66/6

Table S24a. (a) IMP head-to-head margin, K=50K = 50

GameRunsControlPSROdd [95% CI]Runs lower
IMP100.3760.144−0.232 [−0.292, −0.171]10/10
IPD50.1060.078−0.029 [−0.114, +0.057]3/5

Table S24b. (b) Exploitability

GameMeasureTrainedZero-shotdd [95% CI]Seeds
ChickenTournament score2.1051.810+0.295 [+0.218, +0.371]5/5
Rank (of 6)31––
Own return vs M-FOS0.9790.108+0.871 [+0.857, +0.886]5/5
Margin vs M-FOS−2.893−0.195−2.697 [−2.753, −2.642]0/5
Self-play return2.9032.920−0.017 [−0.033, −0.001]0/5
Exploitability0.8850.020+0.865 [+0.849, +0.880]0/5
Stag HuntTournament score3.6673.415+0.252 [+0.190, +0.314]5/5
Rank (of 6)24––
Own return vs M-FOS3.9303.706+0.224 [+0.105, +0.343]5/5
Margin vs M-FOS0.0240.141−0.117 [−0.158, −0.076]0/5
Self-play return3.9053.803+0.103 [+0.060, +0.146]5/5
Exploitability0.0220.035−0.013 [−0.027, +0.001]5/5

Table S25a.

References

  1. [1]Milad Aghahari, Tim Cooijmans, Juan Agustin Duque, Shunichi Akatsuka, and Aaron Courville. 2024. Best Response Shaping. Reinforcement Learning Journal 2 (2024), 798–818.
  2. [2]Milad Aghahari, Juan Agustin Duque, Tim Cooijmans, and Aaron Courville. 2024. LOQA: Learning with Opponent Q-Learning Awareness. In International Conference on Learning Representations.
  3. [3]Eli Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. 2025. Playing repeated games with large language models. Nature Human Behaviour 9, 7 (July 2025), 1380–1390. doi:10.1038/s41562-025-02172-yDOI
  4. [4]Stefano V. Albrecht, Filippos Christianos, and Lukas Schäfer. 2024. Multi-agent reinforcement learning: foundations and modern approaches. The MIT Press, Cambridge, Massachusetts.
  5. [5]Robert M. Axelrod. 2006. The evolution of cooperation (revised ed.). Basic Books, New York, NY.
  6. [6]Jan Balaguer, Raphael Koster, Christopher Summerfield, and Andrea Tachetti. 2022. The Good Shepherd: An Oracle Agent for Mechanism Design. https://arxiv.org/abs/2202.10135
  7. [7]Jacob Beck, Risto Vuorinen, Evan Zheran Liu, Zheng Xiong, Luisa Zingraf, Chelsea Finn, and Shimon Whiteston. 2025. A Tutorial on Meta-Reinforcement Learning. Foundations and Trends in Machine Learning 18, 2-3 (April 2025), 224–384. doi:10.1561/2200000080
  8. [8]Xuxi Chen, Nelson Vadori, Tianlong Chen, and Zhangyang Wang. 2023. Learning to Optimize Differentiable Games. In Proceedings of the 40th International Conference on Machine Learning. PMLR, 5036–5051. https://proceedings.mlr.press/v202/chen23ab.html
  9. [9]Kyunghyun Cho, Bart van Merriënboer, Çağlar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, 1724–1734. doi:10.3115/v1/D14-1179DOI
  10. [10]John D. Co-Reyes, Yingjie Miao, Dayiyi Peng, Esteban Real, Sergey Levine, Quoc V. Le, Honglak Lee, and Aleksandra Faust. 2021. Evolving Reinforcement Learning Algorithms. In International Conference on Learning Representations.
  11. [11]Tim Cooijmans, Milad Alghahari, and Aaron Courville. 2023. Meta-Value Learning: a General Framework for Learning with Learning Awareness. https://arxiv.org/abs/2307.08863
  12. [12]Marta C. Couto, Fernando P. Santos, and Christian Hilbe. 2026. Evolution of noisy learning in games. Proceedings of the National Academy of Sciences 123, 20 (May 2026), e2529959123. doi:10.1073/pnas.2529959123DOI
  13. [13]R. M. Dawes. 1980. Social Dilemmas. Annual Review of Psychology 31, 1 (Jan. 1980), 169–193. doi:10.1146/annurev.ps.31.020180.001125DOI
  14. [14]Silmane Dridi and Erol Akçay. 2018. Learning to Cooperate: The Evolution of Social Rewards in Repeated Interactions. The American Naturalist 191, 1 (Jan. 2018), 58–73. doi:10.1086/694822DOI
  15. [15]Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. 2016. RL²: Fast Reinforcement Learning via Slow Reinforcement Learning. https://arxiv.org/abs/1611.02779
  16. [16]Juan Agustin Duque, Milad Aghajohari, Tim Cooijmans, Razvan Ciuca, Tianyu Zhang, Gauthier Gidel, and Aaron Courville. 2025. Advantage Alignment Algorithms. In International Conference on Learning Representations.arxiv.org/abs/2406.14662
  17. [17]Jidong Feng, Oliver Slumbers, Ziyu Wan, Bo Liu, Stephen McAleer, Ying Wen, Jun Wang, and Yaodong Yang. 2021. Neural Auto-Curricula in Two-Player Zero-Sum Games. In Advances in Neural Information Processing Systems, Vol. 34. 3504–3517.
  18. [18]Sebastian Flennerhag, Yannick Schroecker, Tom Zahavy, Hado van Hasselt, David Silver, and Satinder Singh. 2022. Bootstrapped Meta-Learning. In International Conference on Learning Representations.arxiv.org/abs/2109.04504
  19. [19]Sebastian Flennerhag, Tom Zahavy, Brendan O’Donoghue, Hado van Hasselt, Andrés György, and Satinder Singh. 2023. Optimist Meta-Gradients. In Advances in Neural Information Processing Systems, Vol. 36. 57852–57862. doi:10.52202/075280-2521
  20. [20]Jakob Foerster, Richard Y. Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. 2018. Learning with Opponent-Learning Awareness. In Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2018). International Foundation for Autonomous Agents and Multiagent Systems, Stockholm, Sweden, 122–130. doi:10.65109/hgwa8807DOI
  21. [21]Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric P. Xing, and Shimon Whiteson. 2018. DiCE: The Infinitely Differentiable Monte Carlo Estimator. In International Conference on Machine Learning, Vol. 80. 1529–1538.arxiv.org/abs/1802.05098
  22. [22]Alexander D. Goldie, Chris Lu, Matthew T. Jackson, Shimon Whiteson, and Jakob N. Foerster. 2024. Can Learned Optimization Make Reinforcement Learning Less Difficult?. In Advances in Neural Information Processing Systems, Vol. 37. 5454–5497. doi:10.52202/079017-0177DOI
  23. [23]Jess Daffin, Jaron Cohen, Jakob Nicolaus Foerster, and Shimon Whiteson. 2025. How Should We Meta-Learn Reinforcement Learning Algorithms? arXiv:2507.17668 [cs.LG], doi:10.48550/arXiv.2507.17668
  24. [24]Eric A. Hansen, Daniel S. Bernstein, and Shlomo Zilberstein. 2004. Dynamic Programming for Partially Observable Stochastic Games. In AAAI Conference on Artificial Intelligence. 709–715.
  25. [25]Keegan Harris, Ioannis Anagnostides, Gabriele Farina, Mikhail Khodak, Zhi-wei Steven Wu, and Tuomas Sandholm. 2023. Meta-Learning in Games. In International Conference on Learning Representations.arxiv.org/abs/2209.14110
  26. [26]Edward Hughes, Joel Z. Leibo, Matthew Phillips, Karl Tuyls, Edgar Duéñez-Guzman, Antonio García Castañeda, Iain Dunning, Tina Zhu, Kevin McKee, Raphael Koster, Heather Roff, and Thore Graepel. 2018. Inequity Aversion Improves Cooperation in Intertemporal Social Dilemmas. In Advances in Neural Information Processing Systems, Vol. 31. 3326–3336.arxiv.org/abs/1803.08884
  27. [27]Matthew T. Jackson, Minqi Jiang, Jack Parker-Holder, Risto Vuorio, Chris Lu, Gregory Farquhar, Shimon Whiteson, and Jakob N. Foerster. 2023. Discovering General Reinforcement Learning Algorithms with Adversarial Environment Design. In Advances in Neural Information Processing Systems, Vol. 36. 79980–79998. doi:10.52202/075280-3503DOI
  28. [28]Matthew T. Jackson, Chris Lu, Louis Kirsch, Robert T. Lange, Shimon Whiteson, and Jakob N. Foerster. 2024. Discovering Temporally-Aware Reinforcementarxiv.org/abs/2402.05828
  29. [29]Akbir Khan, Timon Willi, Newton Kwan, Andrea Tacchetti, Chris Lu, Edward Grefenstette, Tim Rocktäschel, and Jakob Foerster. 2024. Scaling Opponent Shaping to High Dimensional Games. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2024). International Foundation for Autonomous Agents and Multiagent Systems, Auckland, New Zealand, 1001–1010. doi:10.65109/tmpy7673arxiv.org/abs/2312.12568
  30. [30]Dong-Ki Kim, Miao Liu, Matthew Riemer, Chuangchuang Sun, Marwa Abdulhai, Golnaz Habibi, Sebastian Lopez-Cot, Gerald Tesauro, and Jonathan P. How. 2021. A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning. In International Conference on Machine Learning, Vol. 139. 5541–5550.arxiv.org/abs/2011.00382
  31. [31]Dong-Ki Kim, Matthew Riemer, Miao Liu, Jakob N. Foerster, Michael Everett, Chuangchuang Sun, Gerald Tesauro, and Jonathan P. How. 2022. Influencing Long-Term Behavior in Multiagent Reinforcement Learning. In Advances in Neural Information Processing Systems, Vol. 35. 18808–18821. doi:10.52202/068431-1366DOI
  32. [32]Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations.arxiv.org/abs/1412.6980
  33. [33]Louis Kirsch, Sebastian Flennerhag, Hado van Hasselt, Abram L. Friesen, Junhyuk Oh, and Yutian Chen. 2022. Introducing Symmetries to Black Box Meta Reinforcement Learning. In AAAI Conference on Artificial Intelligence. 7202–7210. doi:10.1609/aaai.v36i7.20681DOI
  34. [34]Louis Kirsch, Sjoerd van Steenkiste, and Jürgen Schmidhuber. 2020. Improving Generalization in Meta Reinforcement Learning Using Learned Objectives. In International Conference on Learning Representations.arxiv.org/abs/1910.04098
  35. [35]Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. 2017. A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning. In Advances in Neural Information Processing Systems, Vol. 30. 4190–4203.arxiv.org/abs/1711.00832
  36. [36]Joel Z. Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. 2017. Multi-agent Reinforcement Learning in Sequential Social Dilemmas. In Proceedings of the 16th International Conference on Autonomous Agents and Multi-agent Systems (AAMAS 2017). International Foundation for Autonomous Agents and Multiagent Systems, São Paulo, Brazil, 464–473. doi:10.65109/qmsu8421DOI
  37. [37]Adam Lerer and Alexander Peysakhovich. 2018. Maintaining cooperation in complex social dilemmas using deep reinforcement learning. https://arxiv.org/abs/1707.01068
  38. [38]Alistair Letcher, Jakob Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson. 2019. Stable Opponent Shaping in Differentiable Games. In International Conference on Learning Representations.arxiv.org/abs/1811.08469
  39. [39]Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhrub Batra. 2017. Deal or No Deal? End-to-End Learning of Negotiation Dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, 2443–2453. doi:10.18653/v1/D17-1259DOI
  40. [40]Zun Li, John Schultz, Daniel Hennes, and Marc Lanctot. 2026. Discovering Multiagent Learning Algorithms with Large Language Models. https://arxiv.org/abs/2602.16928
  41. [41]Chris Lu, Jakub Grudzien Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. 2022. Discovered Policy Optimisation. In Advances in Neural Information Processing Systems, Vol. 35. 16455–16468. doi:10.52202/068431-1197DOI
  42. [42]Chris Lu, Timon Willi, Christian Schroeder de Witt, and Jakob Foerster. 2022. Model-Free Opponent Shaping. In International Conference on Machine Learning, Vol. 162. 14398–14411.arxiv.org/abs/2205.01447
  43. [43]James Martens. 2020. New Insights and Perspectives on the Natural Gradient Method. Journal of Machine Learning Research 21, 146 (2020), 1–76.
  44. [44]Alex McAvoy, Julian Kates-Harbeck, Krishnendu Chatterjee, and Christian Hilbe. 2022. Evolutionary instability of selfish learning in repeated games. PNAS Nexus 1, 4 (Sept. 2022), pgac141. doi:10.1093/pnasnexus/pgac141
  45. [45]H. Brendan McMahan, Geoffrey J. Gordon, and Avrim Blum. 2003. Planning in the Presence of Cost Functions Controlled by an Adversary. In International Conference on Machine Learning. 536–543.
  46. [46]Luke Metz, Niru Mewa­watharan, Jeremy Nixon, C. Daniel Freeman, and Jascha Sohl-Dickstein. 2019. Understanding and Correcting Pathologies in the Training of Learned Optimizers. In International Conference on Machine Learning, Vol. 97. 4556–4565.arxiv.org/abs/1810.10180
  47. [47]Alexander Meulemans, Seijin Kobayashi, Johannes von Oswald, Nino Scherrer, Eric Elmoznino, Blake A. Richards, Guillaume Lajoie, Blaise Agüera y Arcas, and João Sacramento. 2025. Multi-Agent Cooperation Through Learning-Aware Policy Gradients. In International Conference on Learning Representations.arxiv.org/abs/2410.18636
  48. [48]Junhyuk Oh, Gregory Farquhar, Iurii Kemaev, Dan A. Calian, Matteo Hessel, Luisa Zintgraf, Satinder Singh, Hado van Hasselt, and David Silver. 2025. Discovering state-of-the-art reinforcement learning algorithms. Nature 648, 8093 (Dec. 2025), 312–319. doi:10.1038/s41586-025-09761-xDOI
  49. [49]Junhyuk Oh, Matteo Hessel, Wojciech M. Czarnecki, Zhongwen Xu, Hado van Hasselt, Satinder Singh, and David Silver. 2020. Discovering Reinforcement Learning Algorithms. In Advances in Neural Information Processing Systems, Vol. 33. 1060–1070.
  50. [50]Joong Sun Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23). ACM, 1–22. doi:10.1145/3586183.3606763DOI
  51. [51]William H. Press and Freeman J. Dyson. 2012. Iterated Prisoner’s Dilemma contains strategies that dominate any evolutionary opponent. Proceedings of the National Academy of Sciences 109, 26 (June 2012), 10409–10413. doi:10.1073/pnas.1206569109DOI
  52. [52]Julien Pérólat, Joel Z. Leibo, Vinicius Zambaldi, Charles Beattie, Karl Tuyls, and Thore Graepel. 2017. A Multi-Agent Reinforcement Learning Model of Common-Pool Resource Appropriation. In Advances in Neural Information Processing Systems, Vol. 30. 3643–3652.arxiv.org/abs/1707.06600
  53. [53]Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. 2017. Evolution Strategies as a Scalable Alternative to Reinforcement Learning. https://arxiv.org/abs/1703.03864
  54. [54]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs.LG]. doi:10.48550/arXiv.1707.06347arxiv.org/abs/1707.06347
  55. [55]Wilko Schwarting, Alyssa Pierson, Javier Alonso-Mora, Sertac Karaman, and Daniela Rus. 2019. Social behavior for autonomous vehicles. Proceedings of the National Academy of Sciences 116, 50 (Dec. 2019), 24972–24978. doi:10.1073/pnas.1820676116
  56. [56]David Sychrovský, Christopher Solinas, Revan MacQueen, Kevin Wang, James R. Wright, Nathan R. Sturtevant, and Michael Bowling. 2026. Approximating Nash Equilibria in General-Sum Games via Meta-Learning. In Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026). International Foundation for Autonomous Agents and Multiagent Systems, Paphos, Cyprus, 2374–2382. doi:10.65109/sfsz6668arxiv.org/abs/2504.18868
  57. [57]David Sychrovský, Michal Šustr, Elnaz Davoodi, Michael Bowling, Marc Lanctot, and Martin Schmid. 2024. Learning Not to Regret. In AAAI Conference on Artificial Intelligence. 15202–15210. doi:10.1609/aaai.v38i14.29443DOI
  58. [58]Nelson Vadori, Rahul Savani, Thomas Spooner, and Sumitra Ganesh. 2022. Consensus Multiplicative Weights Update: Learning to Learn Using Projector-Based Game Signatures. In International Conference on Machine Learning, Vol. 162. 21901–21926.arxiv.org/abs/2106.02615
  59. [59]Risto Vuorio, Jacob Beck, Shimon Whiteson, Jakob Foerster, and Gregory Farquhar. 2022. An Investigation of the Bias-Variance Tradeoff in Meta-Gradients. https://arxiv.org/abs/2209.11303
  60. [60]Marissa A. Weis, Maciej Wołczyk, Rajai Nasser, Rif A. Saroukos, Blaise Agüera y Arcas, Joao Sacramento, and Alexander Meulemans. 2026. Multi-agent cooperation through in-context co-player inference. https://arxiv.org/abs/2602.16301
  61. [61]Timon Willi, Alistair Letcher, Johannes Treutlein, and Jakob Foerster. 2022. COLA: Consistent Learning with Opponent-Learning Awareness. In International Conference on Machine Learning, Vol. 162. 23804–23831.
  62. [62]Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8, 3-4 (May 1992), 229–256. doi:10.1007/BF00992696DOI
  63. [63]Hang Xu, Kai Li, Haobo Fu, Qiang Fu, and Junliang Xing. 2022. AutoCFR: Learning to Design Counterfactual Regret Minimization Algorithms. In AAAI Conference on Artificial Intelligence. 5244–5251. doi:10.1609/aaai.v36i5.20460DOI
  64. [64]Zhongwen Xu, Hado van Hasselt, Matteo Hessel, Junhyuk Oh, Satinder Singh, and David Silver. 2020. Meta-Gradient Reinforcement Learning with an Objective Discovered Online. In Advances in Neural Information Processing Systems, Vol. 33. 15254–15264.arxiv.org/abs/2007.08433
  65. [65]Zhongwen Xu, Hado van Hasselt, and David Silver. 2018. Meta-Gradient Reinforcement Learning. In Advances in Neural Information Processing Systems, Vol. 31. 2396–2407.arxiv.org/abs/1805.09801
  66. [66]Jiachen Yang, Ang Li, Mehrdad Farajtabar, Peter Sunehag, Edward Hughes, and Hongyun Zha. 2020. Learning to Incentivize Other Learning Agents. In Advances in Neural Information Processing Systems, Vol. 33. 15208–15219. https://proceedings.neurips.cc/paper_files/paper/2020/file/ad7ed54d7b9baceb12045a929e7e2f66-Paper.pdfarxiv.org/abs/2006.06051
  67. [67]Stephen Zhou, Chris Lu, Roger B. Grosse, and Jakob Foerster. 2022. Proximal Learning with Opponent-Learning Awareness. In Advances in Neural Information Processing Systems, Vol. 35. 26324–26336. doi:10.52202/068431-1909
  68. [68]John L. Zhou, Weizhe Hong, and Jonathan C. Kao. 2024. Reciprocal Reward Influence Encourages Cooperation from Self-Interested Agents. In Advances in Neural Information Processing Systems, Vol. 37. 59491–59512. doi:10.52202/079017-1900DOI

Paper details

vlnc:W_gbrk84q4
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 19

downloads · 4

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents