betaFeedback

WorksEngineering

Entropic Risk-Sensitive Evolutionary Learning and Equilibrium Selection in Coordination Games

Abstract

We study risk-sensitive evolutionary learning dynamics and their long-run equilibrium selection behaviors in coordination games. Agents' risk attitudes enter through the classical entropic risk measure, which evaluates opponent-induced payoff uncertainty and feeds into noisy best responses under two standard revision protocols: best response with mutations and logit choice. We first analyze 2×22\times 2 coordination games in both single-population symmetric and two-population asymmetric settings. In the single-population setting, unlike the risk-neutral case where the dynamics are known to favor the risk-dominant equilibrium, we show that risk sensitivity can change the stochastically stable outcome: a greater risk-seeking attitude favors the payoff-dominant equilibrium, while a greater risk-averse attitude favors the maximin equilibrium. Thus, the population's risk attitude may act as a control knob for long-run equilibrium selection. In both population settings, we also identify a robust regime: any super-dominant equilibrium is stochastically stable for all risk attitudes, under both protocols, and across populations. We further extend the single-population analysis to symmetric kk-action games, which include symmetric kk-action coordination games as a special case, under risk-sensitive best response with mutations. In this setting, we show that, for sufficiently large populations, sufficiently risk-seeking agents uniquely select the strongly payoff-dominant equilibrium when it exists, whereas sufficiently risk-averse agents uniquely select the strongly maximin equilibrium when it exists. These results show that entropic risk sensitivity may serve as a systematic mechanism for steering equilibrium selection in evolutionary games, beyond the classical risk-neutral benchmark.

Introduction

In game theory, strategic interactions and learning processes among agents often come with substantial uncertainty, driven by opponents’ behavior, environmental randomness, and/or noise in payoffs and decision-making. For this reason, earlier literature argued that it is unrealistic to assume agents are always risk-neutral against such uncertainties, and behave as expectation maximizers [1, 2]. Instead, agents may be risk-averse or risk-seeking, and such attitudes are commonly modeled using risk measures that go beyond expected payoffs. More recently, risk sensitivity has also been incorporated in game theory for more tractable equilibrium computation [3] and for promoting cooperation in multi-agent systems [4, 5]. In particular, these works studied games whose payoff or value functions are transformed by risk sensitivity, and then analyzed the existence [2] and computation [3, 4, 5] of equilibria in the resulting transformed games.

What remains less understood is whether risk attitudes can be incorporated directly into agents’ learning dynamics, and how doing so changes equilibrium selection in the original game, which asks the question: which outcome will agents coordinate on through learning, and why [6, 7]? Towards addressing this question, we study the stochastic stability of risk-sensitive evolutionary dynamics, in canonical 2×22 \times2 coordination games [6, 8, 9]. We consider both the single-population symmetric setting and the two-population asymmetric setting, allowing different risk attitudes across populations. At each step, one random agent revises its action according to a risk-sensitive noisy best-response rule. Risk sensitivity enters through the classical entropic risk measure (ERM) [10], which replaces the risk-neutral expected payoffs used in learning. We focus on two standard revision protocols from evolutionary game theory: best response with mutations (BRM) [8] and logit choice [11].

Prior work in evolutionary game theory showed that, with risk-neutral agents, the risk-dominant equilibrium is often uniquely selected in the long run [8, 9, 12]. Few results have studied the impact of risk sensitivity on evolutionary learning, except, e.g., [13, 14]. They study how risk dominance and stochastic stability are preserved under reference-dependent preferences [13] and different risk attitudes [14], particularly when risk dominance coincides with maximin or payoff dominance. By comparison, our results systematically characterize how varying entropic risk sensitivity in the evolutionary dynamics changes long-run equilibrium selection.

More precisely, in the single-population symmetric setting, and under both protocols, when the payoff-dominant and maximin equilibria differ, increasing the risk-sensitivity parameter shifts stochastic stability from the maximin equilibrium to the payoff-dominant equilibrium. This is also consistent with the recent empirical observations that more risk-seeking learning agents are more likely to coordinate at the higher-payoff equilibrium [4, 5], and our results can thus be viewed as a formalization of these insights through the lens of coordination games and evolutionary dynamics. By contrast, if there exists a super-dominant equilibrium [13], which is both weakly maximin and weakly payoff-dominant, then it is stochastically stable for all risk attitudes under both protocols and in both population settings. Compared with [13, 14], our insight on ERM aligns with theirs in the single-population BRM setting, regarding the robustness of super-dominance, although we start from a different modeling framework of risk-sensitive learning. Beyond this setting, we also analyze the single-population BRM in cases where payoff dominance and maximin differ, and thus a super-dominant equilibrium does not exist; analyze the logit choice rule and its robustness, which was not covered in [14]; and extend the analyses to asymmetric two-population games with heterogeneous risk parameters.

Furthermore, we show that the same mechanism extends beyond the two-action case. In single-population symmetric kk-action games under risk-sensitive BRM, if a strongly payoff-dominant equilibrium exists, then it is uniquely selected by sufficiently risk-seeking agents for all sufficiently large populations; similarly, if a strongly maximin equilibrium exists, then it is uniquely selected by sufficiently risk-averse agents for all sufficiently large populations. Thus, the risk attitudes may provide a systematic way to steer long-run equilibrium selection for evolutionary dynamics: optimism toward high payoff realizations (i.e., risk-seeking dynamics) may promote payoff-dominant outcome, while pessimism toward low payoff realizations (i.e., risk-averse dynamics) may promote maximin outcome and thus robust behaviors.

Preliminaries

A 2×22 \times2 bimatrix game is specified by the payoff matrices

Mp=(apbpcpdp),p∈{1,2},M^p = \begin{pmatrix} a^p & b^p \\ c^p & d^p \end{pmatrix}, \qquad p \in\{1,2\},

where MpM^p denotes the payoff matrix of agent pp, and both agents have action set {1,2}\{1,2\}. In each MpM^p, rows index agent pp’s own action and columns index the opponent’s action. The game is symmetric if agents are interchangeable, i.e.,

M1=M2=(abcd).M^1 = M^2 = \begin{pmatrix} a & b \\ c & d \end{pmatrix}.

Definition 1 (2 × 2 coordination games) A 2 × 2 bimatrix game is a coordination game if

ap>cp,dp>bp,p∈{1,2}.a^p > c^p, \qquad d^p > b^p, \qquad p \in\{1,2\}.

In the symmetric setting, these conditions reduce to a>ca > c and d>bd > b.

Under these conditions, the pure action profiles (1,1)(1,1) and (2,2)(2,2) are strict Nash equilibria of the game. We refer to them as coordinated equilibria. Unless stated otherwise, we focus on such 2×22 \times2 coordination games. We further introduce the following standard refinements of such Nash equilibria.

Definition 2 (Risk dominance) The coordinated equilibrium (1,1)(1,1) is risk-dominant [6] if

(a1−c1)(a2−c2)>(d1−b1)(d2−b2).(a^1-c^1)(a^2-c^2) > (d^1-b^1)(d^2-b^2).

In the symmetric setting, this condition reduces to a−c>d−ba-c > d-b. The coordinated equilibrium (2,2)(2,2) is risk-dominant if the reverse inequality holds.

Definition 3 (Payoff dominance) The coordinated equilibrium (1,1)(1,1) is payoff-dominant [6] if it Pareto-dominates the other coordinated equilibrium, i.e.,

ap>dp,p∈{1,2}.a^{p} > d^{p}, \qquad p \in\{1,2\}.

It is weakly payoff-dominant if the above inequalities are non-strict, i.e., ap≥dpa^{p} \ge d^{p} for all p∈{1,2}p \in\{1,2\}. In the symmetric setting, payoff dominance reduces to a>da > d, and weak payoff dominance reduces to a≥da \ge d. The coordinated equilibrium (2,2)(2,2) is payoff-dominant, respectively weakly payoff-dominant, if the reverse strict, respectively non-strict, inequalities hold.

Definition 4 (Maximin action and maximin equilibrium) For agent pp, a maximin action is an action that maximizes the agent’s worst-case payoff over the opponent’s actions. Thus, action 1 is weakly maximin for agent pp if

min⁡{ap,bp}≥min⁡{cp,dp}.\min\{a^{p},b^{p}\} \ge\min\{c^{p},d^{p}\}.

It is strictly maximin if the above inequality is strict. Similarly, action 2 is weakly maximin, respectively strictly maximin, for agent pp if the reverse non-strict, respectively strict, inequality holds. A coordinated equilibrium is weakly maximin if each agent’s equilibrium action is weakly maximin.

Definition 5 (Super-dominant equilibrium) A coordinated equilibrium is super-dominant [13] if it is weakly maximin for both agents and weakly payoff-dominant, with at least one of these conditions holding strictly. Equivalently, (1,1)(1,1) is super-dominant if for each p∈{1,2}p \in\{1,2\},

ap≥dp,bp≥cp,a^{p} \ge d^{p}, \qquad b^{p} \ge c^{p},

and at least one of these inequalities is strict for each pp. The coordinated equilibrium (2,2)(2,2) is super-dominant if the reverse inequalities hold, with at least one strict inequality.

We then consider the standard evolutionary game setting as in [7], with either one or two populations, i.e., P∈{{1},{1,2}}\mathcal{P} \in\{\{1\},\{1,2\}\}. Each population p∈Pp \in\mathcal{P} contains Np∈NN^{p} \in\mathbb{N} agents, and each agent chooses an action from the set Sp={1,2}S^{p} = \{1,2\}. Let xp∈Xp,Npx^{p} \in X^{p,N^{p}} denote the fraction of population pp playing action 1, where

Xp,Np:={0,1Np,…,1}.X^{p,N^{p}} := \left\{0,\frac{1}{N^{p}},\ldots,1\right\}.

The aggregate state is

x=(xp)p∈P∈XN:=∏p∈PXp,Np.x = (x^{p})_{p \in\mathcal{P}} \in X^{N} := \prod_{p \in\mathcal{P}} X^{p,N^{p}}.

In the single-population setting, the state is one-dimensional. In this case, we write N:=N1N := N^{1}, XN:=X1,N1X^{N} := X^{1,N^{1}}, and x:=x1x := x^{1}, and suppress the population index throughout for notational simplicity. In the two-population setting, we write

N:=(N1,N2),XN≡XN1,N2:=X1,N1×X2,N2.N := (N^{1},N^{2}), \qquad X^{N} \equiv X^{N^{1},N^{2}} := X^{1,N^{1}} \times X^{2,N^{2}}.

Payoffs used in evolutionary learning are given by Fp:XN→R2F^{p}: X^{N} \to\mathbb{R}^{2}, where

Fp(x)=(F1p(x),F2p(x)),F^{p}(x) = \left(F_{1}^{p}(x),F_{2}^{p}(x)\right),

and Fip(x)F_i^p(x) is the payoff to action i∈Spi \in S^p for population pp at state xx. Agents are repeatedly and randomly matched to play the coordination game. In the single-population setting, matching is with replacement, so the opponent is sampled from the same population according to the current state, and thus x−p=xx^{-p}=x. In the two-population setting, agents in population pp are matched against population −p-p, and payoffs depend on the opponent state x−px^{-p}, where −p=2-p=2 if p=1p=1 and −p=1-p=1 if p=2p=2.

Risk-Sensitive Evolutionary Dynamics & Stochastic Stability

Risk-Sensitive Evolutionary Dynamics

To model uncertainty in agents’ learning process, we consider risk-sensitive agents who evaluate payoffs via the entropic risk measure [10, 15, 16], rather than risk-neutral expected payoffs.

Definition 6 (Entropic Risk Measure (ERM)) For a real-valued random payoff RR and parameter β∈R\beta\in\mathbb{R}, the entropic risk measure is defined as

Rβ(R):=1βlog⁡E[eβR],\mathcal{R}_{\beta}(R) := \frac{1}{\beta} \log\mathbb{E}\left[e^{\beta R}\right],

with the continuous extension R0(R)=E[R]\mathcal{R}_0(R)=\mathbb{E}[R].

For β\beta near zero, the Taylor expansion of the ERM gives

Rβ(R)=E[R]+β2Var⁡(R)+β26κ3(R)+O(β3),\mathcal{R}_{\beta}(R)=\mathbb{E}[R]+\frac{\beta}{2}\operatorname{Var}(R)+\frac{\beta^2}{6}\kappa_3(R)+O(\beta^3),

where κ3(R)=E[(R−E[R])3]\kappa_3(R)=\mathbb{E}[(R-\mathbb{E}[R])^3]. Thus, the ERM departs from expected payoff by accounting first for variance and then for skewness. A positive β\beta tilts evaluations toward favorable payoff realizations and corresponds to risk-seeking behavior, whereas a negative β\beta emphasizes adverse realizations and corresponds to risk-averse behavior. In our model, agents within each population pp share a common risk-sensitivity parameter βp∈R\beta^p \in\mathbb{R}, although this parameter may differ across populations.

Conditioning on the population state xx, the uncertainty faced by a revising agent comes from random matching. Hence, for the 2×22 \times2 bimatrix game, for each p∈Pp \in\mathcal{P}, the ERM-adjusted payoffs to actions 1 and 2 are

Fβp,1p(x)=1βplog⁡(x−peβpap+(1−x−p)eβpbp),Fβp,2p(x)=1βplog⁡(x−peβpcp+(1−x−p)eβpdp),(1)\begin{aligned} F_{\beta^p,1}^p(x)&=\frac{1}{\beta^p}\log\left(x^{-p}e^{\beta^p a^p}+(1-x^{-p})e^{\beta^p b^p}\right),\\ F_{\beta^p,2}^p(x)&=\frac{1}{\beta^p}\log\left(x^{-p}e^{\beta^p c^p}+(1-x^{-p})e^{\beta^p d^p}\right), \tag*{(1)} \end{aligned}

with the continuous extension at βp=0\beta^p=0 given by the expected payoffs

F0,1p(x)=apx−p+bp(1−x−p),F0,2p(x)=cpx−p+dp(1−x−p).F_{0,1}^p(x)=a^p x^{-p}+b^p(1-x^{-p}),\qquad F_{0,2}^p(x)=c^p x^{-p}+d^p(1-x^{-p}).

For each fixed βp\beta^p, these functions are continuous in xx and smooth on the interior. We thus define the ERM payoff difference for population pp as

Δβpp(x):=Fβ,1p(x)−Fβ,2p(x),(2)\Delta_{\beta^p}^p(x) := F_{\beta,1}^p(x) - F_{\beta,2}^p(x), \tag*{(2)}

which further leads to the following definition.

Definition 7 (ERM-adjusted best response) For population pp, an ERM-adjusted best response at state xx is an action that maximizes the ERM-adjusted payoff against the population state. Equivalently, action 1 is an ERM-adjusted best response if

Δβpp(x)≥0,\Delta_{\beta^p}^p(x) \ge0,

and action 2 is an ERM-adjusted best response if

Δβpp(x)≤0.\Delta_{\beta^p}^p(x) \le0.

When Δβpp(x)=0\Delta_{\beta^p}^p(x) = 0, both actions are ERM-adjusted best responses.

References

  1. [1]Goeree, J.K., Holt, C.A., Palfrey, T.R.: Risk averse behavior in generalized matching pennies games. Games and Economic Behavior 45(1), 97–113 (2003)DOI
  2. [2]Fiat, A., Papadimitriou, C.: When the players are not expectation maximizers. In: International Symposium on Algorithmic Game Theory, pp. 1–14 (2010). Springer
  3. [3]Mazumdar, E., Panaganti, K., Shi, L.: Tractable multi-agent reinforcement learning through behavioral economics. In: International Conference on Learning Representations (2025)
  4. [4]Noorani, E., Baras, J.S.: Risk-attitudes, trust, and emergence of coordination in multi-agent reinforcement learning systems: A study of independent risk-sensitive reinforce. In: 2022 European Control Conference (ECC), pp. 2266–2271 (2022). IEEEDOI
  5. [5]Zhang, R., Li, N., Ozdaglar, A., Shamma, J., Zardini, G.: Optimism as risk-seeking in multi-agent reinforcement learning. IEEE Control Systems Letters (2025)
  6. [6]Harsanyi, J.C., Selten, R.: A general theory of equilibrium selection in games. MIT Press Books 1 (1988)
  7. [7]Sandholm, W.H.: Population games and evolutionary dynamics. MIT press (2010)
  8. [8]Kandori, M., Mailath, G.J., Rob, R.: Learning, mutation, and long run equilibria in games. Econometrica: Journal of the Econometric Society, 29–56 (1993)
  9. [9]Young, H.P.: The evolution of conventions. Econometrica: Journal of the Econometric Society, 57–84 (1993)
  10. [10]Föllmer, H., Schied, A.: Convex measures of risk and trading constraints. Finance and stochastics 6(4), 429–447 (2002)
  11. [11]Blume, L.E.: The statistical mechanics of strategic interaction. Games and economic behavior 5(3), 387–424 (1993)
  12. [12]Staudigl, M.: Stochastic stability in asymmetric binary choice coordination games. Games and Economic Behavior 75(1), 372–401 (2012)DOI
  13. [13]Sawa, R., Wu, J.: Reference-dependent preferences, super-dominance and stochastic stability. Journal of mathematical economics 78, 96–104 (2018)
  14. [14]Nax, H.H., Newton, J.: Risk attitudes and risk dominance in the long run. Games and Economic Behavior 116, 179–184 (2019)DOI
  15. [15]Howard, R.A., Matheson, J.E.: Risk-sensitive markov decision processes. Management Science 18(7), 356–369 (1972)DOI
  16. [16]Whittle, P.: Risk-sensitive linear/quadratic/gaussian control. Advances in Applied Probability 13(4), 764–777 (1981)
  17. [17]Sandholm, W.H.: Orders of limits for stationary distributions, stochastic dominance, and stochastic stability. Theoretical Economics 5(1), 1–26 (2010)DOI
  18. [18]Stroock, D.W.: An introduction to Markov processes. Springer (2005)
  19. [19]Freidlin, M.I., Wentzell, A.D.: Random perturbations. Springer (1998)
  20. [20]Sandholm, W.H., Staudigl, M.: Large deviations and stochastic stability in the small noise double limit. Theoretical Economics 11(1), 279–355 (2016)DOI
  21. [21]Osborne, M.J., Rubinstein, A.: A course in game theory. MIT press (1994)
  22. [22]Ellison, G.: Basins of attraction, long-run stochastic stability, and the speed of step-by-step evolution. The Review of Economic Studies 67(1), 17–45 (2000)

Paper details

vlnc:W_6rd4y8v6
Version
v1 · latest
Published
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Engineering

Not recorded for this paper

View statement
Looking for related papers…
Contents