betaFeedback

WorksEconomics, Econometrics and Finance

Moral AI and the Erosion of Its Own Demand

Abstract

We study the evolution of human preference frequencies and AI adoption when individuals can choose between a moral AI that rules out defection and an unconstrained AI that preserves discretion. By allowing users to commit to cooperation, moral AI can sustain cooperation even when cooperative preferences are too scarce to do so on their own. But this success has a cost. By turning otherwise dormant cooperative preferences into committed behavior, moral AI exposes its users to exploitation by selfish individuals, causing the population share of conditional cooperators to decline under evolutionary selection. As conditional cooperators become less prevalent, equilibrium adoption eventually converges to zero. Moral AI can therefore be self-liquidating. For initial shares between the moral-AI adoption cutoff and the threshold for cooperation without AI, moral AI generates cooperation while lowering the limiting share of conditional cooperators. Adoption ultimately vanishes, leaving a lower share of the preference type that initially sustained demand for commitment.

*[email protected].

†Department of Economics, University of Oregon, [email protected].

Introduction

As AI takes on economic and social decisions, an important question is which forms of AI can survive competition. Some systems may be designed with strong moral constraints to rule out harmful or exploitative actions, while others retain greater discretion.1 Hendrycks [15] argues that the latter may have a competitive advantage because they can behave responsibly when useful without giving up other profitable actions. This raises the possibility that competition may gradually select against constraints intended to limit harmful behavior.

The conclusion is less immediate once AI systems are used in strategic interactions. A constraint does not only remove actions; it can also change how others respond. As AI agents become more capable, individuals and firms are likely to delegate an increasing range of consequential choices to them, making AI-mediated strategic interactions a routine feature of economic and social life. In these settings, restrictions built into an AI can serve as commitment devices. Commitment itself is not new. People have long limited their future discretion by signing contracts that penalize deviation, posting bonds or collateral that are forfeited upon breach, placing assets in escrow, or conducting transactions through platforms that impose rules on both sides rather than transacting privately. We study this familiar commitment logic in the context of AI adoption. A user can choose an AI whose rules restrict the actions it may take on her behalf while interacting with a counterparty who has made no comparable commitment and need not enter the same contractual or institutional arrangement. The commitment is therefore attached to the intermediary the user chooses rather than to a common institution governing both parties. While this logic applies more generally to any observable commitment device that restricts its user’s actions while leaving counterparties unconstrained, AI agents are the setting in which we expect such devices to become common.

This form of commitment is especially valuable when behavior depends on beliefs about others’ underlying preferences. We study this idea in a standard Prisoner’s Dilemma, a simple framework that captures many applications involving exchange, information sharing, joint production, and other situations in which mutual cooperation is valuable but each side has an incentive to take advantage of a cooperative counterpart. Individuals differ in their preferences: some are selfish, while others are conditional cooperators who are willing to cooperate when they expect the other side to do the same. Because these preferences are private, even two conditional cooperators may fail to recognize one another and instead behave defensively. Cooperative preferences can therefore exist in the population without producing cooperative behavior.

A moral AI can change this informational problem. We call an AI “moral” when it excludes defection from its feasible actions, in contrast to an unconstrained AI that leaves the user with discretion. As a benchmark, we assume that the AI chosen by each user is observable to the other side. This assumption is natural when interactions themselves are mediated by identifiable AI agents. Firms can publicly disclose the systems under which their agents operate, while personal AI agents can identify themselves when communicating with other agents. Emerging standards already move in this direction. For example, the Agent2Agent (A2A) Protocol requires an agent to make available an “Agent Card” describing information such as its identity, provider, version, capabilities, and security requirements.2 Such disclosure does not yet certify an agent’s behavioral constraints, but it illustrates how information about the agent acting on a user’s behalf can become observable to counterparties. We therefore begin with full observability to isolate the commitment channel and later relax this assumption.

Once a moral AI is observable, a user no longer needs to convince the other side that she personally intends to cooperate. The technological restriction makes cooperation credible without requiring direct observation of her preferences. In this sense, AI can substitute for information about human preferences and sustain cooperation in situations where private preferences alone cannot.

The central question of this paper is what happens when this technological solution feeds back onto human preferences. Moral AI makes cooperation possible for conditional cooperators, but the same commitment is visible to selfish individuals, who retain the option to defect and can exploit users whose AI has removed that option. The introduction of moral AI therefore creates a material advantage for selfish preferences. As individuals with higher material payoffs become more prevalent over time, the population becomes increasingly selfish. This change in population composition then changes the demand for different AI technologies. This feedback can make moral AI self-liquidating. In the baseline model, its long-run demand eventually disappears. The direction of this result resembles Hendrycks [15], but the logic is almost the reverse. Moral AI does not lose because constraint is inherently uncompetitive; constraint is precisely what makes it valuable in the first place. It loses because that value changes the human population on which its own demand depends. Figure 1 summarizes the central logic of the model.

The self-liquidating mechanism

Figure 1. The self-liquidating mechanism. Moral AI initially helps sustain cooperation, but the same commitment also changes population composition in a way that reduces support for adoption.

We consider three robustness extensions that clarify which features of the technology drive the result. First, we allow observability of AI choice to be imperfect. Each individual independently observes the opponent’s AI only with some probability. This captures settings in which the AI used by a counterparty is not directly disclosed but can sometimes be inferred from its outputs: recent studies find persistent model-specific stylistic fingerprints in the outputs of different AI systems, although such inference is necessarily imperfect [21] (Lu et al. 2026). Imperfect observability weakens commitment because a constraint that is not seen cannot fully affect the opponent’s behavior. We show that imperfect observability weakens the benchmark effect: adoption disappears at a higher share of conditional cooperators, so selection stops at a weakly higher limiting share.

Second, we replace perfect commitment with stochastic commitment. The moral AI normally forces cooperation but occasionally leaves the user with discretion, reflecting the possibility that its safeguards can be circumvented, overridden, or simply fail to bind in some situations. This imperfection has the opposite effect from imperfect observability. As long as bypasses are rare enough that observing moral AI still induces conditional cooperators with discretion to cooperate, a small probability of bypass can make moral AI more attractive. The user continues to receive the commitment benefit because conditional-cooperator opponents respond cooperatively to the observed constraint, but sometimes regains the option to defect. A slightly leaky constraint can therefore support commitment at an even lower population share of conditional cooperators than a perfectly binding one. As a result, moral AI can remain in use at lower shares of conditional cooperators, allowing selection against them to continue further.

Third, we relax the assumption that individuals must interact after being matched. When individuals can refuse to interact after observing their opponent’s AI choice, moral AI can sustain cooperation without exposing its users to exploitation. In particular, an equilibrium exists in which both preference types adopt moral AI and refuse to interact with unconstrained AI users. Both types then receive the same material payoff, eliminating selection against conditional cooperators. However, refusal does not eliminate this selection pressure in every equilibrium. Other equilibria continue to favor selfish types, sometimes even at population shares where moral AI is not adopted in the baseline model. Thus, the evolutionary consequences of moral AI depend not only on its ability to sustain cooperation, but also on whether its users can avoid exploitation.

Taken together, the results point to a broader question about the relationship between technology and morality. Technologies designed to support better behavior are usually judged by the behavior they produce: whether they reduce harm, sustain cooperation, or make good conduct easier. But a technology that changes the incentives surrounding moral action may also reshape the conditions under which moral dispositions are sustained, rewarded, or eroded. The long-run consequence of moral AI may thus depend not only on whether it makes people behave better, but on what kind of people remain afterward.

Literature Review

Our model builds on the indirect evolutionary approach to preference evolution. Güth and Yaari [13] and Güth [11] introduce the approach, which provides a useful theoretical framework for understanding this process. Individuals are endowed with preferences that need not coincide with material payoffs; these preferences shape their strategic behavior, while material payoffs determine evolutionary success. Preferences that induce more successful behavior therefore become more prevalent over time. Recent surveys by Alger and Weibull (2019) and Alger [2] review this literature and its applications to the evolution of social and other-regarding preferences. A common conclusion in this literature is that, under random matching and unobservable preference types, material-payoff maximization is weakly favored [12, 6].

One way for non-selfish preferences to become evolutionarily valuable is for them to generate observable signals that affect how others respond. In Frank [10], for example, a conscience acts as a commitment device, while imperfect signals of trustworthiness allow others to infer who is likely to honor that commitment. See also Akdeniz and van Veelen [1] on the role of commitment in the evolution of preferences. Frank also recognizes the resulting problem of mimicry: selfish individuals may imitate the signals associated with trustworthy types. Robson and Samuelson [22] sharpen this concern, arguing that once signals themselves evolve, successful signals invite mimics, which in turn undermine their informativeness and may generate an ongoing cycle of new signals and new mimics. Hopkins [17] disciplines this problem by making altruistic behavior itself a costly signal that generates better matching opportunities, while Heller and Mohlin [14] allow preferences and the ability to deceive others about those preferences to coevolve.

A related way to sustain non-selfish preferences is to change who interacts with whom. Assortative or endogenous matching allows cooperative types to interact disproportionately with one another and thereby enjoy the benefits of their behavior [3, 24]. Screening institutions can achieve a similar form of separation by inducing different preference types to self-select into different groups or arrangements even when preferences themselves are unobservable [16] (Wu and Xiong 2026).

Our mechanism is different. In the baseline model, matching remains random and human preferences remain hidden; moral AI neither sorts individuals nor screens them into separate groups. Its observable choice may convey information about the user, but it is also a commitment device in its own right. In the benchmark model, a selfish individual can mimic a conditional cooperator’s choice of moral AI only by accepting the same restriction on subsequent behavior. Moral AI therefore ties the observable signal to the behavior it promises, eliminating the possibility of appearing committed while retaining the option to defect. At the same time, because it does not shield conditional cooperators from selfish counterparts, this commitment exposes them to the selection pressure at the center of our analysis.

Ely and Szentes [7] also study evolutionary forces acting on AI. They model AI embedded in machines that reproduce copies of themselves, so self-promoting AI can spread at the expense of efficient production, especially when AI is imperfectly transparent. Our mechanism is different: AI does not reproduce on its own, but changes in prevalence through human adoption as human preferences evolve. Observability also plays a different role, creating commitment value for moral AI rather than disciplining misaligned AI.

A separate literature studies how delegation and machine-mediated commitment can sustain cooperation in one-shot interactions. Fershtman et al. [8] show that when principals delegate decisions to agents under publicly observable contracts, the contracts can serve as strategic commitments and support cooperative outcomes that would not arise in the underlying noncooperative game. Tennenholtz [23] develops a related idea in a computational setting: players submit programs that can inspect the programs submitted by others and condition their actions accordingly. Such programs can sustain mutual cooperation in the one-shot Prisoner’s Dilemma and, more generally, generate the set of feasible and individually rational payoffs. Cooper et al. [5] develop more robust simulation-based versions of program equilibrium, characterizing when similar cooperation can be supported without relying on exact syntactic comparison of programs. These papers establish that delegation and software-embedded commitment can themselves create cooperation. Our focus is instead on what happens when heterogeneous humans endogenously choose between a constrained and an unconstrained AI, and when the resulting strategic behavior feeds back into the distribution of human preferences and hence into future demand for the commitment technology.

Our analysis is also related to the emerging behavioral literature on AI delegation and morality. Köbis et al. [18] emphasize that AI can influence ethical behavior in several roles, particularly as a partner or delegate that enables individuals to obtain unethical benefits while distancing themselves from the underlying behavior. Köbis et al. [19] provide experimental evidence that the form of machine delegation matters: people request more dishonest behavior when delegation allows them to specify high-level goals or training examples rather than directly instructing each action, while machine agents are also substantially more willing than human delegates to carry out fully unethical instructions. Our mechanism concerns a different channel. Human preferences are fixed within an interaction, and we do not model delegation as changing an individual’s psychological cost of moral or immoral behavior. Instead, moral AI changes the feasible actions and therefore the strategic responses and material payoffs generated by those preferences. The resulting change in population composition is therefore an evolutionary effect rather than a within-person erosion of moral preferences.

Zimmaro et al. [26] provide a particularly close connection to the evolutionary part of our analysis. They use evolutionary game theory to study how Samaritan and Discriminatory AI agents affect the emergence of cooperation among human-like agents in a one-shot Prisoner’s Dilemma. Samaritan AIs cooperate unconditionally, whereas Discriminatory AIs condition their behavior on whether the counterpart is cooperative, and the two designs can have different effects on the evolution of cooperation. In our model, by contrast, AI is not an additional population of strategic opponents. Humans choose AI agents to act on their behalf in human–human interactions, and the prevalence of each AI technology is itself determined by human adoption. What evolves is the distribution of underlying human preferences. This distinction allows a technology that initially increases cooperation to reduce the prevalence of cooperative preferences and ultimately undermine its own demand.

The Static Model

Preferences and AI technologies

A unit mass of individuals is randomly matched to play a one-shot symmetric game with material payoffs of a Prisoner’s Dilemma. Since the game is symmetric, we only present the material payoffs for the row player in Table 1.

CD
CRRSS
DTTPP

Table 1. A symmetric 2×22 \times2 Prisoner’s Dilemma

The payoffs satisfy T>R>P>ST > R > P > S. Action CC denotes cooperation, while action DD denotes defection.

There are two human preference types. A fraction x∈[0,1]x \in[0,1] are conditional cooperators, denoted type θ\theta, and the remaining fraction 1−x1-x are selfish, denoted type τ\tau. Throughout the paper, we assume that human preference types are private but individuals know the distribution of them.

A selfish type evaluates outcomes according to material payoffs. A conditional cooperator obtains an additional utility benefit b>0b > 0 from mutual cooperation. Hence, if a conditional cooperator is the row player, her utilities from different outcomes of the game are given in Table 2.

CCDD
CCR+bR+bSS
DDTTPP

Table 2. A conditional cooperator’s utilities in Prisoner’s Dilemma

We assume R+b>TR+b > T. A θ\theta-type agent’s best response is CC if and only if she believes that her opponent chooses CC with probability at least κ=P−SR+b−T+P−S\kappa= \frac{P-S}{R+b-T+P-S}. It is the unique best response when that probability exceeds κ\kappa; at equality, DD is also optimal. This makes them conditional cooperators, who cooperate when they believe their opponents are sufficiently likely to cooperate. The experimental evidence in Fischbacher et al. [9] motivates this type, though it does not pin down the utility specification used here.

In the future environment we have in mind, individuals use AI agents to act on their behalf in strategic interactions. The actions CC and DD should be understood as potentially complicated courses of action rather than simple button presses. An individual may decide which action she wants to take and instruct the AI to carry it out. Alternatively, she may communicate her preferences and beliefs to the AI and let it choose the action she herself would prefer under the circumstances. In either case, the underlying preferences remain those of the human; the AI is the agent through which those preferences are translated into behavior. Such an arrangement seems increasingly plausible as AI agents become capable of carrying out complex tasks on behalf of their users.

There are two AI technologies. The moral AI permits only cooperation M:{C}M : \{C\}, whereas the unconstrained AI permits both actions A:{C,D}A : \{C,D\}. The term “moral AI” refers only to this restriction on the feasible action set; the AI itself need not possess preferences. In the baseline model, AI choices are perfectly observable.

The timing is as follows. After random matching, individuals simultaneously choose an AI. They observe the opponent’s choice and then simultaneously choose feasible actions. The payoff structure and the rules of each AI are common knowledge, and any randomization is independent and private. We assume that AI choice is costless. This allows us to focus on the strategic effect of the constraint rather than differences in adoption costs.

Let Pr⁡(M∣θ)\Pr(M \mid\theta) and Pr⁡(M∣τ)\Pr(M \mid\tau) denote the adoption rate of moral AI by the two types, respectively. Let y=xPr⁡(M∣θ)+(1−x)Pr⁡(M∣τ)y = x\Pr(M \mid\theta) + (1-x)\Pr(M \mid\tau) denote the population share using MM.

Remark 1. Before studying AI-mediated strategic interactions, it is useful to consider the benchmark environment in which no AI is available and individuals therefore face no technological constraint on their actions. Type τ\tau always defects because DD is its dominant action. An all-defect Bayesian Nash equilibrium therefore exists for every xx. A second Bayesian Nash equilibrium, in which type θ\theta cooperates and type τ\tau defects, exists if and only if x≥κx \ge\kappa. Thus, when x<κx < \kappa, the unique equilibrium without AI is all defection, and both preference types receive the same material payoff PP. In this sense, cooperative preferences are present but remain dormant: they do not translate into cooperative behavior.

AI-Mediated Strategic Interactions

In this section, we characterize the equilibria when individuals use AI agents to carry out their choices. We consider type-homogeneous Perfect Bayesian equilibria, in which all individuals of the same preference type use the same strategy. A strategy for each human type i∈{θ,τ}i \in\{\theta,\tau\} specifies the probability Pr⁡(M∣i)\Pr(M \mid i) of choosing moral AI and, conditional on choosing AA, a probability of choosing CC after each observed AI choice of the opponent. Lemma 1 below shows that in the baseline model no equilibrium involves an interior continuation mixture on the equilibrium path. An individual who chooses MM has no subsequent action choice because MM permits only CC.

After observing the opponent’s AI choice, individuals update their beliefs about the opponent’s preference type. By Bayes’ rule, whenever AA is chosen with positive probability, the posterior probability that an AA-user is type θ\theta is

μA=x(1−Pr⁡(M∣θ))x(1−Pr⁡(M∣θ))+(1−x)(1−Pr⁡(M∣τ)).\mu_A = \frac{x(1-\Pr(M \mid\theta))}{x(1-\Pr(M \mid\theta))+(1-x)(1-\Pr(M \mid\tau))}.

An equilibrium requires that continuation actions are optimal given these beliefs and the constraints imposed by the chosen AI, and that each type’s AI choice is optimal given the resulting continuation play.

Because MM forces its user to choose CC, beliefs about the preference type of an MM-user do not affect continuation behavior in the baseline model. A θ\theta-type individual who uses AA and faces an MM-user knows that the opponent will choose CC, and therefore chooses CC because R+b>TR+b>T. A τ\tau-type individual who uses AA always chooses DD.

Lemma 1. Let 0<x<10<x<1, and allow continuation strategies to be mixed. In every type-homogeneous Perfect Bayesian equilibrium, Pr⁡(M∣τ)=0\Pr(M\mid\tau)=0, and a θ\theta-type AA-user who meets an AA-user on the equilibrium path chooses CC with probability zero or one.

Proof: See Appendix.

Continuation play can therefore take one of two forms. In the defensive continuation equilibrium, a θ\theta-type AA-user chooses DD against another AA-user. This continuation equilibrium always exists, because if all θ\theta-type AA-users choose DD, an AA-user expects no cooperation from another AA-user. In the cooperative continuation equilibrium, a θ\theta-type AA-user chooses CC against another AA-user. Since τ\tau-type AA-users choose DD, this continuation equilibrium exists if and only if μA≥κ\mu_A\ge\kappa.

Proposition 1. For a fixed population share 0<x<10<x<1, all type-homogeneous Perfect Bayesian equilibria satisfy Pr⁡(M∣τ)=0\Pr(M\mid\tau)=0 and are characterized as follows.

(1) Defensive continuation equilibrium. A θ\theta-type individual using AA chooses CC against MM and DD against AA, while a τ\tau-type individual using AA chooses DD against either AAI. Moreover,

Pr⁡(M∣θ)={0,if x≤P−SR+b−S,1−(1−x)(P−S)x(R+b−P),otherwise.\Pr(M\mid\theta)= \begin{cases} 0, & \text{if } x\le\dfrac{P-S}{R+b-S},\\ 1-\dfrac{(1-x)(P-S)}{x(R+b-P)}, & \text{otherwise.} \end{cases}

(2) Cooperative continuation equilibrium. A θ\theta-type individual using AA chooses CC against either AAI, while a τ\tau-type individual using AA chooses DD against either AAI. Such an equilibrium exists if and only if x≥κx\ge\kappa. In every such equilibrium, 0≤Pr⁡(M∣θ)≤1−κ(1−x)(1−κ)x0\le\Pr(M\mid\theta)\le1-\dfrac{\kappa(1-x)}{(1-\kappa)x}.

Proof: See Appendix.

Note that at the endpoints only one type is present. At x=0x = 0 every individual is selfish and uses AA. At x=1x = 1 several adoption patterns support cooperation. We will see that both endpoints are stationary under the selection dynamics of § 4.

Proposition 1 shows that moral AI can make cooperative preferences behaviorally relevant even when they would otherwise remain dormant. Without AI, when the share of conditional cooperators is below κ\kappa, they defect because they cannot be sufficiently confident that their opponent will cooperate. Note that P−SR+b−S<κ\frac{P-S}{R+b-S} < \kappa. With observable moral AI, in the activation region P−SR+b−S<x<κ\frac{P-S}{R+b-S} < x < \kappa, the unique equilibrium has some conditional cooperators choose MM, which credibly commits them to cooperation and induces other conditional cooperators to cooperate in response. Below P−SR+b−S\frac{P-S}{R+b-S} nobody adopts MM, and for x≥κx \ge\kappa some cooperative-continuation equilibria also have zero adoption. As a result, cooperation can arise for population shares strictly below the threshold (κ\kappa) required without AI. The proposition therefore identifies a role for observable AI commitment in activating otherwise dormant cooperation: by restricting the actions available to its user, AI can substitute for otherwise unavailable information about intentions and support cooperation that hidden human preferences alone cannot sustain.

Evolution of preferences

We now allow the population share of conditional cooperators to evolve. At each xx, individuals play a type-homogeneous Perfect Bayesian equilibrium according to their preferences on a faster time scale, while preferences evolve more slowly according to the material payoffs earned by each preference type. This follows the standard indirect evolutionary approach, in which preferences shape behavior while material payoffs determine evolutionary success.

For any σ∈E(x)\sigma\in\mathcal{E}(x), where E(x)\mathcal{E}(x) denotes the set of type-homogeneous Perfect Bayesian equilibria at xx, let Fθ(σ,x)F^{\theta}(\sigma, x) and FT(σ,x)F^{T}(\sigma, x) denote the material fitness of the two preference types, and define the fitness differential

Δ(σ,x)≡Fθ(σ,x)−FT(σ,x).\Delta(\sigma, x) \equiv F^{\theta}(\sigma, x) - F^{T}(\sigma, x).

Definition 1. A selection dynamic is an autonomous law x˙=V(x)\dot{x}=V(x) on [0,1][0,1], with VV continuous on [0,1][0,1] and V(0)=V(1)=0V(0)=V(1)=0. It is payoff monotone if, for every x∈(0,1)x\in(0,1),

(i) if Δ(σ,x)≤0\Delta(\sigma,x)\le0 for every σ∈E(x)\sigma\in\mathcal{E}(x), then V(x)≤0V(x)\le0, with V(x)<0V(x)<0 if every such inequality is strict;

(ii) if Δ(σ,x)≥0\Delta(\sigma,x)\ge0 for every σ∈E(x)\sigma\in\mathcal{E}(x), then V(x)≥0V(x)\ge0, with V(x)>0V(x)>0 if every such inequality is strict.

In particular, V(x)=0V(x)=0 when every equilibrium gap is zero. If equilibrium gaps have opposing signs, the definition leaves the sign of V(x)V(x) unrestricted.

The definition deliberately does not impose an equilibrium-selection rule. When different equilibria strictly favor different types, the direction of evolution is indeterminate. When the equilibrium is unique, the definition reduces to the usual payoff-monotonicity condition, sign⁡V(x)=sign⁡Δ(σ,x)\operatorname{sign} V(x)=\operatorname{sign}\Delta(\sigma,x).

Proposition 2. x˙<0\dot{x}<0 if and only if 1>x>P−SR+b−S1>x>\frac{P-S}{R+b-S}, and x˙=0\dot{x}=0 otherwise. Hence, for x(0)∈(0,1)x(0)\in(0,1), lim⁡t→∞x(t)=min⁡{x(0),P−SR+b−S}\lim_{t\to\infty}x(t)=\min\left\{x(0),\frac{P-S}{R+b-S}\right\} and lim⁡t→∞y(t)=0\lim_{t\to\infty}y(t)=0.

Proof: See Appendix.

Proposition 2 reveals a self-liquidating force created by moral AI. By making cooperation credible, moral AI allows conditional cooperators to behave cooperatively in situations where they would otherwise defect. Yet this same commitment changes the material returns to preference types: conditional cooperators are more exposed to exploitation, while selfish individuals retain the flexibility to take advantage of them. Moral AI therefore creates selection pressure against the preferences that make it valuable in the first place. As conditional cooperators become less common, the demand for commitment falls. Once the trajectory enters the unique defensive region below κ\kappa, adoption declines with xx and converges to zero, and the technology loses the strategic role that initially sustained its use. The key economic implication is that a technology designed to support cooperative behavior can endogenously undermine the behavioral type on which its own value depends. Figure 2 provides a graphic illustration.

Equilibrium adoption and material fitness under observable commitment

Figure 2. Equilibrium adoption and material fitness under observable commitment. In the defensive continuation equilibrium, moral-AI adoption is positive for x>P−SR+b−Sx > \frac{P-S}{R+b-S}. For x≥κx \ge\kappa, cooperative-continuation equilibria additionally permit every adoption rate in the shaded set, including zero. At each interior state x>P−SR+b−Sx > \frac{P-S}{R+b-S}, every equilibrium favors selfish types in material fitness. Trajectories starting in (P−SR+b−S,1)\left(\frac{P-S}{R+b-S},1\right) converge to P−SR+b−S\frac{P-S}{R+b-S}, with adoption converging to zero. Drawn for T=5T=5, R=3R=3, P=1P=1, S=0S=0 and b=3b=3, so P−SR+b−S=1/6\frac{P-S}{R+b-S}=1/6 and κ=1/2\kappa=1/2. These parameter values are illustrative.

Robustness

Imperfect observability

We first maintain perfect enforcement of moral AI but relax the assumption that its use is always observed. In practice, a counterparty may know that an AI agent is acting on someone’s behalf without being able to verify exactly which system is being used or what constraints govern it. Even as technologies for disclosure and verification make AI use increasingly observable, such observability is likely to remain imperfect.

Assume that each player independently observes the opponent’s AI with probability ρ∈(0,1]\rho\in(0,1]. A player knows whether she observes the opponent’s AI but does not know whether the opponent observes hers. We have the following result.

Proposition 3. Suppose ρ>R+b−T+P−SR+b−S\rho> \frac{R+b-T+P-S}{R+b-S}. Then, x˙<0\dot{x}<0 if and only if 1>x>P−Sρ(R+b−S)1>x>\frac{P-S}{\rho(R+b-S)}, and x˙=0\dot{x}=0 otherwise. Hence, for x(0)∈(0,1)x(0)\in(0,1), lim⁡t→∞x(t)=min⁡{x(0),P−Sρ(R+b−S)}\lim_{t\to\infty}x(t)=\min\{x(0),\frac{P-S}{\rho(R+b-S)}\} and lim⁡t→∞y(t)=0\lim_{t\to\infty}y(t)=0.

Proof: See Appendix.

For sufficiently high ρ\rho, Proposition 3 implies that imperfect observability weakens the baseline mechanism (ρ<1\rho<1 implies that P−Sρ(R+b−S)>P−SR+b−S\frac{P-S}{\rho(R+b-S)}>\frac{P-S}{R+b-S}). The rationale is that the response benefit of commitment requires that the opponent observe it. A conditional cooperator who adopts moral AI is still constrained to cooperate when her AI choice goes unobserved, but in those encounters the constraint cannot induce additional cooperation from the opponent. Imperfect observability therefore reduces the benefit of commitment without eliminating its cost. As the population share of conditional cooperators falls, moral AI consequently ceases to be attractive at a higher xx than under perfect observability, and selection against conditional cooperators stops at a weakly higher share. Thus greater observability has two reinforcing effects: it makes moral AI more valuable and allows it to sustain cooperation for a wider range of population states. But precisely for this reason, it also extends the range over which moral AI selects against conditional cooperators. Imperfect observability dampens the self-liquidating force and leaves a weakly larger limiting share of conditional cooperators from a common initial population. The two limits are equal when the initial share lies below both cutoffs.

For lower observability, additional continuation equilibria may alter the adoption region and the states at which selection stops. The imitation argument nevertheless implies that selfish types weakly outperform conditional cooperators at every equilibrium. We leave the full equilibrium characterization in this parameter region to future work.

Stochastic commitment

We next restore perfect observability of AI choice and consider imperfect enforcement. In practice, even an AI designed to enforce cooperation may occasionally fail, be overridden, or be circumvented: with probability 1−ε1-\varepsilon, an MM-user is forced to cooperate, while with probability ε\varepsilon, the user remains free to choose either action. The opponent observes that MM is being used and knows ε\varepsilon, but does not observe whether the restriction has been bypassed. Bypass events are independent across players and independent of preference types and AI choices. Each MM-user privately observes whether her own constraint binds before choosing an action when discretion is available. Opponents observe the MM label but not the bypass realization.

We restrict attention to sufficiently small bypass probabilities:

ε<εˉ≡min⁡{1−κ,R+b−P(R+b−P)+(R+b−T+P−S),T−R(T−P)+(T−R)}.\varepsilon< \bar{\varepsilon} \equiv\min\left\{1-\kappa,\frac{R+b-P}{(R+b-P)+(R+b-T+P-S)},\frac{T-R}{(T-P)+(T-R)}\right\}.

The first condition ensures that observing MM still induces a conditional opponent to cooperate even if every user who obtains a bypass defects. The second makes the adoption gain strictly decreasing in Pr⁡(M∣θ)\operatorname{Pr}(M \mid\theta) in the region x<κx<\kappa. The third ensures that a selfish type does not find moral AI attractive even when all conditional cooperators adopt it.

Proposition 4. Suppose ε<εˉ\varepsilon< \bar{\varepsilon} holds. Then, x˙<0\dot{x}<0 if and only if 1>x>(1−ε)(P−S)(1−ε)(R+b−S)+ε(T−P)1>x>\frac{(1-\varepsilon)(P-S)}{(1-\varepsilon)(R+b-S)+\varepsilon(T-P)}, and x˙=0\dot{x}=0 otherwise. Hence, for x(0)∈(0,1)x(0)\in(0,1), lim⁡t→∞x(t)=min⁡{x(0),(1−ε)(P−S)(1−ε)(R+b−S)+ε(T−P)}\lim_{t\to\infty}x(t)=\min\left\{x(0),\frac{(1-\varepsilon)(P-S)}{(1-\varepsilon)(R+b-S)+\varepsilon(T-P)}\right\} and lim⁡t→∞y(t)=0\lim_{t\to\infty}y(t)=0.

Proof. See Appendix.

Proposition 4 implies that a small stochastic failure has the opposite comparative effect from imperfect observability. Imperfect observability weakens commitment because the other side sometimes fails to see it. A small bypass leaves the commitment visible and sufficiently credible, while occasionally restoring discretion to the user. Consequently,

(1−ε)(P−S)(1−ε)(R+b−S)+ε(T−P)≤P−SR+b−S≤P−Sρ(R+b−S)\frac{(1-\varepsilon)(P-S)}{(1-\varepsilon)(R+b-S)+\varepsilon(T-P)} \le\frac{P-S}{R+b-S} \le\frac{P-S}{\rho(R+b-S)}

for 0≤ε<εˉ0 \le\varepsilon< \bar{\varepsilon} and R+b−T+P−SR+b−S<ρ≤1\frac{R+b-T+P-S}{R+b-S}<\rho\le1. The first inequality is strict when ε>0\varepsilon> 0, and the second is strict when ρ<1\rho< 1. Imperfect observability causes moral-AI adoption to disappear at a higher population share; a small stochastic bypass instead allows it to remain attractive after conditional cooperators have become even less common.

Note that under the restriction of Proposition 4, 0≤ε<εˉ0 \le\varepsilon< \bar{\varepsilon}, the adoption cutoff (1−ε)(P−S)(1−ε)(R+b−S)+ε(T−P)\frac{(1-\varepsilon)(P-S)}{(1-\varepsilon)(R+b-S)+\varepsilon(T-P)} decreases strictly with ε\varepsilon.

We do not consider large enforcement errors. Our purpose is to examine whether the baseline mechanism survives small departures from perfect commitment. As ε\varepsilon becomes large, moral AI provides progressively weaker commitment, and additional continuation behavior may arise. In the extreme case ε=1\varepsilon= 1, MM imposes no restriction at all and becomes strategically equivalent to AA, so the commitment channel disappears entirely.

We also leave the interaction between imperfect observability and imperfect enforcement for future work. The two frictions depart from perfect observable commitment in different ways and have opposite effects on the adoption cutoff. Characterizing this interaction would require jointly studying the visibility and reliability of commitment, which we leave outside the scope of the present analysis.

A right to refuse

The baseline model assumes that randomly matched individuals must interact, even when their opponent’s AI choice exposes them to exploitation. We now allow individuals to decline interaction. After choosing an AI and observing the opponent’s choice, both individuals simultaneously decide whether to accept the match. The Prisoner’s Dilemma is played only if both accept; otherwise, each receives material payoff o<Ro<R and no moral benefit. Matching remains random, and there is no rematching. If o<So<S, declining is never profitable and the baseline model is recovered. We therefore consider S<o<RS<o<R.

Proposition 5. Suppose S<o<RS<o<R and 0<x<10<x<1.

(i) There is an equilibrium in which both types choose MM and decline opponents who use AA. Both types earn material payoff RR, so Δ(σ,x)=0\Delta(\sigma,x)=0.

(ii) In every equilibrium, FT(σ,x)≥Fθ(σ,x)F^{T}(\sigma,x)\ge F^{\theta}(\sigma,x).

(iii) Equilibria with Δ(σ,x)<0\Delta(\sigma,x)<0 also exist. If S<o<PS<o<P, they exist for every x>P−SR+b−Sx>\frac{P-S}{R+b-S} and whenever

P−oR+b+P−2o≤x<P−oR+P−2o,\frac{P-o}{R+b+P-2o}\le x<\frac{P-o}{R+P-2o},

where the lower bound is below P−SR+b−S\frac{P-S}{R+b-S}. If P≤o<RP\le o<R, they exist for every

x>o−SR+b−S.x>\frac{o-S}{R+b-S}.

Proof: See Appendix.

Allowing individuals to refuse interaction changes the evolutionary consequences of moral AI. When both types adopt MM and refuse to interact with AA-users, moral AI sustains cooperation without exposing its users to exploitation. Selfish individuals also adopt MM, since doing so allows them to participate in cooperative interactions. Both types then receive the same material payoff, eliminating selection against conditional cooperators.

However, refusal does not eliminate equilibria in which selfish individuals have a material advantage. In particular, when S<o<PS<o<P, such equilibria can arise even below the original adoption cutoff P−SR+b−S\frac{P-S}{R+b-S}, where moral AI is not adopted in the baseline model. Thus, the option to refuse interaction can prevent the erosion of cooperative preferences in some equilibria while allowing it to persist in others.

Since every population state admits an equilibrium with equal material fitness, while selfish types are weakly favored in every equilibrium, Definition 1 implies x˙≤0\dot{x} \le0 but does not generally require strict decline. The long-run outcome therefore depends on equilibrium selection.

Conclusion

This paper studies how moral AI can shape both strategic behavior and the evolution of human preferences. By restricting the actions available to its user, moral AI can make cooperation credible and induce conditional cooperators to cooperate even when their preferences would otherwise remain behaviorally dormant. Yet this same commitment creates a material advantage for selfish preferences, causing the population share of conditional cooperators to fall over time. As a result, for initially mixed populations, moral-AI adoption eventually converges to zero and the technology becomes self-liquidating. The mechanism survives sufficiently small departures from perfect observability and perfect enforcement, and the two frictions shift the adoption cutoff in opposite directions. Allowing individuals to refuse interaction has a different effect: moral AI can sustain cooperation without generating selection against conditional cooperators, although other equilibria continue to favor selfish types. More broadly, the study of AI morality should ask not only how to make AI behave morally, but also how moral AI changes human behavior and preferences over time. As AI increasingly acts on our behalf, these feedback effects may become as important as the moral properties of the AI itself.

Declaration of AI use

OpenAI’s ChatGPT was used during the development of this study to discuss and refine ideas, improve exposition, assist with editing and LaTeX formatting, prepare figures, and check the presentation of mathematical arguments. During revision, Anthropic’s Claude was used to review the manuscript, propose corrections to proofs and wording, and draft inserted text. The research questions, the model and the results are the authors’ own. The authors checked every proof and all AI-assisted material and take full responsibility for the final manuscript.

Appendix

Proof of Lemma 1

Recall that a type-τ\tau individual using AA always defects, whereas a type-θ\theta individual using AA cooperates against MM, since R+b>TR+b>T.

We first show that Pr⁡(M∣τ)=0\Pr(M\mid\tau)=0. Suppose otherwise. Choosing MM gives both types the same material payoff, whereas choosing AA gives type τ\tau a weakly higher material payoff, since defection is materially dominant. Moreover, type θ\theta is weakly more likely to achieve mutual cooperation by choosing MM rather than AA. Thus, type θ\theta has a weakly stronger incentive to adopt MM than type τ\tau. This comparison is strict whenever some individuals use MM, since against an MM-user, switching to AA gains type τ\tau T−RT-R and gains type θ\theta nothing.

Since Pr⁡(M∣τ)>0\Pr(M\mid\tau)>0, this implies Pr⁡(M∣θ)=1\Pr(M\mid\theta)=1. If some individuals still use AA, they must all be type τ\tau, so type θ\theta strictly prefers AA to MM, since P>SP>S against AA-users and both technologies yield R+bR+b against MM-users. If everyone uses MM, type τ\tau can profitably deviate to AA, earning T>RT>R. Both cases yield a contradiction. Hence Pr⁡(M∣τ)=0\Pr(M\mid\tau)=0.

Next, suppose a type-θ\theta AA-user strictly mixes between CC and DD against another AA-user. Indifference requires the opponent to cooperate with probability κ\kappa. Since type τ\tau always defects and type θ\theta mixes, we must have μA>κ\mu_A>\kappa.

Against an AA-user, the utility gain from choosing MM rather than AA is

μA(R+b)+(1−μA)S−[κT+(1−κ)P]=(μA−κ)(R+b−S)>0,\begin{aligned} \mu_A(R+b)+(1-\mu_A)S-[\kappa T+(1-\kappa)P] \\ &=(\mu_A-\kappa)(R+b-S)>0, \end{aligned}

where the equality follows from the definition of κ\kappa. Against an MM-user, both technologies yield the same utility R+bR+b. Therefore, type θ\theta strictly prefers MM, contradicting the use of AA on the equilibrium path.

Proof of Proposition 1

Consider first the defensive continuation equilibrium. For a θ\theta-type individual, the payoff difference between MM and AA arises only against an AA-user. Against a θ\theta-type AA-user, MM turns mutual defection into mutual cooperation, yielding a gain of R+b−PR+b-P. Against a τ\tau-type AA-user, MM instead exposes her to defection, yielding a loss of P−SP-S. Opponents using MM generate no payoff difference because a θ\theta-type individual cooperates against them whether she uses MM or AA. Hence her expected utility gain from MM over AA is

Δθ=x(1−Pr⁡(M∣θ))(R+b−P)−(1−x)(1−Pr⁡(M∣τ))(P−S).(1)\Delta_\theta=x(1-\Pr(M\mid\theta))(R+b-P)-(1-x)(1-\Pr(M\mid\tau))(P-S). \tag*{(1)}

By Lemma 1, Pr⁡(M∣τ)=0\Pr(M\mid\tau)=0. Substituting this into (1) gives Δθ=x(1−Pr⁡(M∣θ))(R+b−P)−(1−x)(P−S)\Delta_\theta=x(1-\Pr(M\mid\theta))(R+b-P)-(1-x)(P-S). If no θ\theta-type adopts MM, Δθ=x(R+b−P)−(1−x)(P−S)\Delta_\theta=x(R+b-P)-(1-x)(P-S), which is nonpositive if and only if x≤P−SR+b−Sx\le\frac{P-S}{R+b-S}. Thus Pr⁡(M∣θ)=0\Pr(M\mid\theta)=0 for population shares below this threshold.

When 1>x>P−SR+b−S1>x>\frac{P-S}{R+b-S}, a θ\theta-type strictly prefers MM when Pr⁡(M∣θ)=0\Pr(M\mid\theta)=0, but strictly prefers AA when Pr⁡(M∣θ)=1\Pr(M\mid\theta)=1. Hence both technologies must be used, and indifference requires Δθ=0\Delta_\theta=0, which gives Pr⁡(M∣θ)=1−(1−x)(P−S)x(R+b−P)\Pr(M\mid\theta)=1-\frac{(1-x)(P-S)}{x(R+b-P)}. This establishes the defensive equilibrium.

Now consider the cooperative continuation equilibrium. A θ\theta-type individual using AA chooses CC against either technology. She therefore obtains exactly the same outcome whether she chooses MM or AA and is indifferent between them. By Lemma 1, Pr⁡(M∣τ)=0\Pr(M\mid\tau)=0. Hence, by Bayes’ rule, the posterior probability that an AA-user is type θ\theta is μA=x(1−Pr⁡(M∣θ))x(1−Pr⁡(M∣θ))+(1−x)\mu_A=\frac{x(1-\Pr(M\mid\theta))}{x(1-\Pr(M\mid\theta))+(1-x)}.

For a θ\theta-type AA-user to cooperate against another AA-user, sequential rationality requires μA≥κ\mu_A\ge\kappa. This implies Pr⁡(M∣θ)≤1−κ(1−x)(1−κ)x\Pr(M\mid\theta)\le1-\frac{\kappa(1-x)}{(1-\kappa)x}, whose right hand side is nonnegative if and only if x≥κx\ge\kappa. Hence the cooperative continuation equilibrium exists if and only if x≥κx\ge\kappa, and whenever it exists any 0≤Pr⁡(M∣θ)≤1−κ(1−x)(1−κ)x0\le\Pr(M\mid\theta)\le1-\frac{\kappa(1-x)}{(1-\kappa)x} can be sustained.

This exhausts all type-homogeneous Perfect Bayesian equilibria.

Proof of Proposition 2

Consider first the defensive continuation equilibrium. Since Pr⁡(M∣τ)=0\Pr(M \mid\tau)=0, a type-θ\theta user of MM obtains material payoff xR+(1−x)SxR+(1-x)S, while a type-θ\theta user of AA obtains xPr⁡(M∣θ)R+(1−xPr⁡(M∣θ))Px\Pr(M \mid\theta)R+(1-x\Pr(M \mid\theta))P. In addition, a type-τ\tau individual using AA obtains xPr⁡(M∣θ)T+(1−xPr⁡(M∣θ))Px\Pr(M \mid\theta)T+(1-x\Pr(M \mid\theta))P.

Hence

Δ(σ,x)=Pr⁡(M∣θ)[xR+(1−x)S]+(1−Pr⁡(M∣θ))[xPr⁡(M∣θ)R+(1−xPr⁡(M∣θ))P]−(xPr⁡(M∣θ)T+(1−xPr⁡(M∣θ))P).(2)\begin{aligned} \Delta(\sigma,x)={}&\Pr(M \mid\theta)[xR+(1-x)S]\\ &+(1-\Pr(M \mid\theta))[x\Pr(M \mid\theta)R+(1-x\Pr(M \mid\theta))P]\\ &-(x\Pr(M \mid\theta)T+(1-x\Pr(M \mid\theta))P). \tag*{(2)} \end{aligned}

When x≤P−SR+b−Sx \le\frac{P-S}{R+b-S}, Pr⁡(M∣θ)=0\Pr(M \mid\theta)=0. We have Δ(σ,x)=0\Delta(\sigma,x)=0. When x>P−SR+b−Sx > \frac{P-S}{R+b-S}, Pr⁡(M∣θ)=1−(1−x)(P−S)x(R+b−P)\Pr(M \mid\theta)=1-\frac{(1-x)(P-S)}{x(R+b-P)}. This implies Δ(σ,x)=−Pr⁡(M∣θ)[(1−x)(P−S)bR+b−P+x(T−R)]<0\Delta(\sigma,x)=-\Pr(M \mid\theta)\left[\frac{(1-x)(P-S)b}{R+b-P}+x(T-R)\right]<0.

On the cooperative continuation branch, every type-θ\theta individual cooperates and every type-τ\tau individual defects. Thus, Fθ(σ,x)=xR+(1−x)SF^\theta(\sigma,x)=xR+(1-x)S and Fτ(σ,x)=xT+(1−x)PF^\tau(\sigma,x)=xT+(1-x)P, implying that Δ(σ,x)=x(R−T)+(1−x)(S−P)<0\Delta(\sigma,x)=x(R-T)+(1-x)(S-P)<0.

When P−SR+b−S<x<κ\frac{P-S}{R+b-S}<x<\kappa, only the defensive continuation equilibrium exists. Since Δ(σ,x)<0\Delta(\sigma,x)<0, evolution strictly favors type τ\tau over type θ\theta. For x≥κx\ge\kappa, both continuation families may exist, but both give the same direction of selection. Below the cutoff P−SR+b−S\frac{P-S}{R+b-S}, moral AI is not adopted, all individuals defect, and selection stops.

It remains to establish the limits. If x(0)≤P−SR+b−Sx(0)\le\frac{P-S}{R+b-S}, then V(x(0))=0V(x(0))=0 and the trajectory is constant. If x(0)∈(P−SR+b−S,1)x(0)\in\left(\frac{P-S}{R+b-S},1\right), the trajectory decreases while x>P−SR+b−Sx>\frac{P-S}{R+b-S} and cannot cross the interval [0,P−SR+b−S]\left[0,\frac{P-S}{R+b-S}\right] on which VV vanishes, so it has a limit ℓ≥P−SR+b−S\ell\ge\frac{P-S}{R+b-S}. If ℓ>P−SR+b−S\ell>\frac{P-S}{R+b-S}, then V(ℓ)<0V(\ell)<0, and by continuity VV is bounded away from zero near ℓ\ell, which contradicts convergence. Hence ℓ=P−SR+b−S\ell=\frac{P-S}{R+b-S}. Because P−SR+b−S<κ\frac{P-S}{R+b-S}<\kappa, the trajectory eventually enters (P−SR+b−S,κ)\left(\frac{P-S}{R+b-S},\kappa\right), where the defensive equilibrium is unique and

y=(R+b−S)x−(P−S)R+b−P,y=\frac{(R+b-S)x-(P-S)}{R+b-P},

which tends to zero as x→P−SR+b−Sx\to\frac{P-S}{R+b-S}. Both limits therefore hold whichever equilibrium is played at x≥κx \ge\kappa.

Proof of Proposition 3

First, observe that ρ>R+b−T+P−SR+b−S\rho> \frac{R+b-T+P-S}{R+b-S} implies P−Sρ(R+b−S)<κ\frac{P-S}{\rho(R+b-S)} < \kappa.

As in the proof of Lemma 1, type θ\theta has a weakly stronger incentive to adopt MM than type τ\tau. This comparison continues to hold under imperfect observability, since an unobserved AI choice does not affect the opponent’s behavior, while an observed MM induces cooperation by type-θ\theta AA-users. The comparison is strict whenever Pr⁡(M∣τ)>0\operatorname{Pr}(M \mid\tau) > 0, since ρ>0\rho> 0 and type θ\theta cooperates against an observed MM-user while type τ\tau defects. Thus, Pr⁡(M∣τ)>0\operatorname{Pr}(M \mid\tau) > 0 implies Pr⁡(M∣θ)=1\operatorname{Pr}(M \mid\theta) = 1. But then all AA-users are selfish, and type τ\tau strictly prefers AA to MM, a contradiction. Hence Pr⁡(M∣τ)=0\operatorname{Pr}(M \mid\tau) = 0.

Now consider x<κx < \kappa. After observing AA, the posterior probability that the opponent is type θ\theta is no greater than x<κx < \kappa. If the opponent’s AI is not observed, the probability of cooperation is also at most x<κx < \kappa. Thus a θ\theta-type AA-user cooperates only when she observes MM and defects otherwise.

Against a type-θ\theta AA-user, choosing MM induces cooperation only when it is observed, yielding a gain of ρ(R+b)+(1−ρ)S−P\rho(R+b) + (1-\rho)S - P. Against a type-τ\tau AA-user, MM only exposes the individual to defection, yielding a loss of P−SP-S. Against a type-θ\theta MM-user, choosing MM preserves mutual cooperation, whereas choosing AA allows defection when MM is not observed. Hence, the gain from choosing MM rather than AA in this case is R+b−[ρ(R+b)+(1−ρ)T]=(1−ρ)(R+b−T)R+b-[\rho(R+b)+(1-\rho)T] = (1-\rho)(R+b-T). This implies that the expected utility gain from choosing MM rather than AA is

Δθ=x(1−Pr⁡(M∣θ))[ρ(R+b)+(1−ρ)S−P]−(1−x)(P−S)+xPr⁡(M∣θ)(1−ρ)(R+b−T),=ρx(R+b−S)−(P−S)+xPr⁡(M∣θ)[(1−ρ)(R+b−T+P−S)−ρ(R+b−P)].(3)\begin{aligned} \Delta_\theta={}& x(1-\operatorname{Pr}(M \mid\theta))[\rho(R+b)+(1-\rho)S-P] \\ &- (1-x)(P-S)+x\operatorname{Pr}(M \mid\theta)(1-\rho)(R+b-T), \\ ={}& \rho x(R+b-S)-(P-S) \\ &+x\operatorname{Pr}(M \mid\theta)[(1-\rho)(R+b-T+P-S)-\rho(R+b-P)]. \tag*{(3)} \end{aligned}

Given ρ>R+b−T+P−SR+b−S\rho> \frac{R+b-T+P-S}{R+b-S}, the coefficient on Pr⁡(M∣θ)\Pr(M \mid\theta) is strictly negative. At Pr⁡(M∣θ)=0\Pr(M \mid\theta)=0, Δθ=ρx(R+b−S)−(P−S)\Delta_\theta=\rho x(R+b-S)-(P-S). Hence no type θ\theta adopts MM if and only if x≤P−Sρ(R+b−S)x \le\frac{P-S}{\rho(R+b-S)}.

Above this cutoff and below κ\kappa, equation (5) implies that when Pr⁡(M∣θ)=0\Pr(M \mid\theta)=0, the gain from adopting MM is ρx(R+b−S)−(P−S)>0\rho x(R+b-S)-(P-S)>0. When Pr⁡(M∣θ)=1\Pr(M \mid\theta)=1, this gain is x(1−ρ)(R+b−T)−(1−x)(P−S)<0x(1-\rho)(R+b-T)-(1-x)(P-S)<0, where the inequality follows from x<κx<\kappa. Thus neither zero nor full adoption can be an equilibrium, implying 0<Pr⁡(M∣θ)<10<\Pr(M \mid\theta)<1.

Moreover the material-fitness comparison has the same sign as in the baseline model. Since 0<Pr⁡(M∣θ)<10<\Pr(M \mid\theta)<1, some type-θ\theta individuals use AA and cooperate when they observe MM. Because preference types are private, type τ\tau can imitate type θ\theta’s AI-adoption strategy while defecting whenever discretion is available, thereby obtaining a strictly higher material payoff. Hence Fτ(σ,x)>Fθ(σ,x)F^\tau(\sigma,x)>F^\theta(\sigma,x).

The same argument applies to the additional equilibria that may exist for x≥κx\ge\kappa. If Pr⁡(M∣θ)=0\Pr(M \mid\theta)=0, some type-θ\theta individuals must cooperate, since otherwise adopting MM would be profitable above the cutoff. If 0<Pr⁡(M∣θ)<10<\Pr(M \mid\theta)<1, type-θ\theta individuals using AA cooperate when they observe MM. In both cases, type τ\tau can imitate type θ\theta’s AI choice and earn a strictly higher material payoff by defecting whenever discretion is available. Finally, if Pr⁡(M∣θ)=1\Pr(M \mid\theta)=1, the material payoffs are xR+(1−x)SxR+(1-x)S and xT+(1−x)PxT+(1-x)P for the two types, respectively, so type τ\tau again has a strict advantage.

Thus x˙<0\dot{x}<0 above the cutoff P−Sρ(R+b−S)\frac{P-S}{\rho(R+b-S)}, while below it everyone uses AA, everyone defects, and the two types earn the same material payoff; convergence follows as in the proof of Proposition 2. □

Proof of Proposition 4

First, ε<εˉ\varepsilon<\bar{\varepsilon} implies 1−ε>κ1-\varepsilon>\kappa. Hence a θ\theta-type AA-user cooperates after observing MM even if every MM-user who obtains a bypass subsequently defects.

Second, we show that type τ\tau does not adopt MM. Suppose Pr⁡(M∣τ)>0\Pr(M \mid\tau)>0. As in the proof of Lemma 1, we compare the two types’ incentives to choose MM rather than AA. Against AA-users, this incentive is weakly stronger for type θ\theta. If a type-θ\theta AA-user defects or mixes against AA, her expected utility from AA is the same as that of type τ\tau, while choosing MM gives type θ\theta at least as much utility as type τ\tau. If she cooperates against AA, the difference between the two types’ adoption incentives is

(1−ε)[μA(T−R)+(1−μA)(P−S)]>0.(1-\varepsilon)[\mu_A(T-R)+(1-\mu_A)(P-S)]>0.

Against an MM-user, the difference is also strictly positive: it is at least (1−ε)(T−R)>0(1-\varepsilon)(T-R)>0 against type θ\theta, and equals (1−ε)[(1−ε)(T−R)+ε(P−S)]>0(1-\varepsilon)[(1-\varepsilon)(T-R)+\varepsilon(P-S)]>0 against type τ\tau. Thus, type θ\theta has a strictly stronger incentive to adopt MM than type τ\tau, implying Pr⁡(M∣θ)=1\Pr(M\mid\theta)=1.

But when all type-θ\theta individuals choose MM, type τ\tau strictly prefers AA. Against a type-θ\theta MM-user, the gain from switching to AA is at least

(1−ε)(T−R)−ε(T−P)>0,(1-\varepsilon)(T-R)-\varepsilon(T-P)>0,

where the inequality follows from the third restriction on ε\varepsilon. Against a type-τ\tau AA-user, the gain is (1−ε)(P−S)>0(1-\varepsilon)(P-S)>0, and against a type-τ\tau MM-user, it is (1−ε)[(1−ε)(T−R)+ε(P−S)]>0(1-\varepsilon)[(1-\varepsilon)(T-R)+\varepsilon(P-S)]>0. This contradicts Pr⁡(M∣τ)>0\Pr(M\mid\tau)>0. Hence Pr⁡(M∣τ)=0\Pr(M\mid\tau)=0.

Now suppose x<κx<\kappa. Since τ\tau-type individuals do not adopt MM, the posterior probability that an AA-user is θ\theta-type is no greater than x<κx<\kappa. Thus a θ\theta-type AA-user defects against an AA-user. By contrast, an observed MM-user cooperates with probability at least 1−ε1-\varepsilon. Since 1−ε>κ1-\varepsilon>\kappa, a θ\theta-type AA-user cooperates against MM. If a θ\theta-type MM-user obtains a bypass, she follows the same continuation behavior.

Under this continuation behavior, for a θ\theta type, against a θ\theta-type AA-user, choosing MM rather than AA yields a gain of (1−ε)(R+b−P)+ε(T−P)(1-\varepsilon)(R+b-P)+\varepsilon(T-P).

Against a type τ\tau, it yields a loss of (1−ε)(P−S)(1-\varepsilon)(P-S), while against a θ\theta-type MM-user it yields a gain of ε(R+b−S)\varepsilon(R+b-S).

The expected utility gain from MM rather than AA for a θ\theta type can therefore be written as

Δθ=x(1−Pr⁡(M∣θ))[(1−ε)(R+b−P)+ε(T−P)]−(1−x)(1−ε)(P−S)+xPr⁡(M∣θ)ε(R+b−S)=x[(1−ε)(R+b−P)+ε(T−P)]−(1−x)(1−ε)(P−S)−xPr⁡(M∣θ)[(1−ε)(R+b−P)−ε(R+b−T+P−S)].(4)\begin{aligned} \Delta_\theta={}& x(1-\Pr(M \mid\theta))\left[(1-\varepsilon)(R+b-P)+\varepsilon(T-P)\right] \\ &- (1-x)(1-\varepsilon)(P-S)+x\Pr(M \mid\theta)\varepsilon(R+b-S) \\ ={}& x\left[(1-\varepsilon)(R+b-P)+\varepsilon(T-P)\right]-(1-x)(1-\varepsilon)(P-S) \\ &-x\Pr(M \mid\theta)\left[(1-\varepsilon)(R+b-P)-\varepsilon(R+b-T+P-S)\right]. \tag*{(4)} \end{aligned}

ε<εˉ\varepsilon< \bar{\varepsilon} ensures that the coefficient on Pr⁡(M∣θ)\Pr(M \mid\theta) is strictly negative. At Pr⁡(M∣θ)=0\Pr(M \mid\theta)=0, Δθ≤0\Delta_\theta\le0 if and only if

x≤(1−ε)(P−S)(1−ε)(R+b−S)+ε(T−P).x \le\frac{(1-\varepsilon)(P-S)}{(1-\varepsilon)(R+b-S)+\varepsilon(T-P)}.

Hence at and below this cutoff no θ\theta-type adopts MM.

Above the cutoff, every equilibrium involves cooperation by type θ\theta. If Pr⁡(M∣θ)=0\Pr(M \mid\theta)=0, some type-θ\theta individuals must cooperate, since otherwise adopting MM would be profitable. If 0<Pr⁡(M∣θ)<10 < \Pr(M \mid\theta) < 1, type-θ\theta AA-users cooperate against MM. Finally, if Pr⁡(M∣θ)=1\Pr(M \mid\theta)=1, then ε>0\varepsilon>0, since otherwise type θ\theta would prefer AA against selfish AA-users. In this case, bypassed type-θ\theta MM-users cooperate against MM, since 1−ε>κ1-\varepsilon>\kappa. Thus, in every case, type θ\theta cooperates with positive probability when discretion is available. Because preferences are private, type τ\tau can imitate type θ\theta’s AI-adoption strategy while defecting whenever discretion is available, obtaining a strictly higher material payoff.

At or below the cutoff, equation (4) implies that no one adopts MM. Since the cutoff is below κ\kappa, everyone defects and both types receive the same material payoff. The convergence result then follows as in the proof of Proposition 2.

Proof of Proposition 5

(i) Consider an equilibrium in which both types choose MM, accept MM-users, and decline AA-users. Since AA is off the equilibrium path, assign probability zero to an AA-user being type θ\theta. An MM-user facing an AA-user therefore expects defection and prefers to decline, since o>So>S. Against another MM-user, both types prefer to accept because R>oR>o. When both matched individuals choose AA, they defect if the match proceeds and accept the match if and only if P≥oP \ge o. A deviation to AA is rejected by every MM-user and yields o<Ro < R. Hence neither type has an incentive to deviate, and both receive material payoff RR.

(ii) As in the previous environments, type τ\tau can imitate the AI choice and acceptance strategy of type θ\theta while defecting whenever discretion is available. Since defection is materially dominant, this implies Fτ(σ,x)≥Fθ(σ,x)F^\tau(\sigma, x) \ge F^\theta(\sigma, x) in every equilibrium.

(iii) First, suppose S<o<PS < o < P and x>P−SR+b−Sx > \frac{P-S}{R+b-S}. Consider the defensive continuation equilibrium of Proposition 1, with every match accepted. The posterior probability that an AA-user is type θ\theta is μA=P−SR+b−S\mu_A = \frac{P-S}{R+b-S}. A type-θ\theta MM-user facing an AA-user obtains expected utility μA(R+b)+(1−μA)S=P>o\mu_A(R+b) + (1-\mu_A)S = P > o, and therefore accepts. Two AA-users also accept, since their continuation payoff is P>oP > o. Thus, the adoption incentives of type θ\theta are unchanged.

For type τ\tau, adopting MM yields a lower payoff against MM-users, since R<TR < T. Against AA-users, the highest payoff from adopting MM, allowing for refusal, is max⁡{μAR+(1−μA)S,o}<P\max\{\mu_A R+(1-\mu_A)S,o\} < P. Hence type τ\tau also has no incentive to deviate. The material fitness comparison is therefore the same as in the baseline defensive equilibrium, where Δ(σ,x)<0\Delta(\sigma, x) < 0.

Next, suppose S<o<PS < o < P and consider an equilibrium in which all type-θ\theta individuals choose MM, while all type-τ\tau individuals choose AA. Let MM-users accept MM-users and decline AA-users, while AA-users accept and defect against one another. Since every AA-user is selfish, declining an AA-user is optimal for an MM-user, as o>So > S.

A type-θ\theta individual obtains x(R+b)+(1−x)ox(R+b)+(1-x)o from choosing MM and xo+(1−x)Pxo+(1-x)P from choosing AA. Thus, choosing MM is optimal if and only if

x≥P−oR+b+P−2o.x \ge\frac{P-o}{R+b+P-2o}.

Similarly, type τ\tau prefers AA if and only if

x≤P−oR+P−2o.x \le\frac{P-o}{R+P-2o}.

The material fitness differential is

Δ(σ,x)=xR+(1−x)o−[xo+(1−x)P]<0\Delta(\sigma, x) = xR + (1-x)o - [xo + (1-x)P] < 0

whenever the second inequality is strict. The lower bound on xx is strictly below P−SR+b−S\frac{P-S}{R+b-S} because (P−S)(R+b+P−2o)−(P−o)(R+b−S)=(o−S)(R+b−P)+(P−o)(P−S)>0(P-S)(R+b+P-2o)-(P-o)(R+b-S)=(o-S)(R+b-P)+(P-o)(P-S)>0.

Finally, suppose P≤o<RP\le o<R and x>o−SR+b−Sx>\frac{o-S}{R+b-S}. Consider an equilibrium with Pr⁡(M∣τ)=0\Pr(M\mid\tau)=0 and

Pr⁡(M∣θ)=1−(1−x)(o−S)x(R+b−o).\Pr(M\mid\theta)=1-\frac{(1-x)(o-S)}{x(R+b-o)}.

The assumed restriction on xx ensures that this adoption probability lies strictly between zero and one. By Bayes’ rule, the posterior probability that an AA-user is type θ\theta is

μA=o−SR+b−S.\mu_A=\frac{o-S}{R+b-S}.

Let AA-users decline other AA-users, but accept MM-users. Type-θ\theta MM-users accept both technologies, while type-τ\tau MM-users decline AA. Against MM, type-θ\theta AA-users cooperate and type-τ\tau AA-users defect. The defensive continuation is played following two AA choices if interaction occurs.

A type-θ\theta MM-user is indifferent between accepting and declining an AA-user, since

μA(R+b)+(1−μA)S=o.\mu_A(R+b)+(1-\mu_A)S=o.

A type-τ\tau MM-user instead prefers to decline an AA-user. Two AA-users decline one another because their defensive continuation payoff satisfies P≤oP\le o.

Recall that y=xPr⁡(M∣θ)y=x\Pr(M\mid\theta) denotes the population share using MM. A type-θ\theta individual then obtains the same expected utility from either AI, which is y(R+b)+(1−y)oy(R+b)+(1-y)o.

A type-τ\tau individual obtains yT+(1−y)oyT+(1-y)o from choosing AA, compared with yR+(1−y)oyR+(1-y)o from choosing MM, and therefore strictly prefers AA.

For material fitness, a type-θ\theta AA-user receives yR+(1−y)oyR+(1-y)o. A type-θ\theta MM-user receives a weakly lower material payoff, since her expected material payoff against an AA-user is o−μAb<oo-\mu_A b<o. Hence

Fθ(σ,x)≤yR+(1−y)o<yT+(1−y)o=Fτ(σ,x),F^\theta(\sigma,x) \le yR + (1-y)o < yT + (1-y)o = F^\tau(\sigma,x),

where the strict inequality follows from y>0y>0 and T>RT>R. □\square

References

  1. [1]Akdeniz, A. and M. van Veelen (2021): “The evolution of morality and the role of commitment,” Evolutionary Human Sciences, 3, e41.DOI
  2. [2]Alger, I. (2023): “Evolutionarily Stable Preferences,” Philosophical Transactions of the Royal Society B, 378, 20210505.
  3. [3]Alger, I. and J. W. Weibull (2013): “Homo Moralis: Preference Evolution under Incomplete Information and Assortative Matching,” Econometrica, 81, 2269–2302.
  4. [5]Cooper, E., C. Oesterheld, and V. Conitzer (2025): “Characterising Simulation-Based Program Equilibria,” Proceedings of the AAAI Conference on Artificial Intelligence, 39, 13735–13744.
  5. [6]Dekel, E., J. C. Ely, and O. Yilankaya (2007): “Evolution of Preferences,” Review of Economic Studies, 74, 685–704.
  6. [7]Ely, J. C. and B. Szentes (2024): “Natural Selection of Artificial Intelligence,” Unpublished manuscript.
  7. [8]Fershtman, C., K. L. Judd, and E. Kalai (1991): “Observable Contracts: Strategic Delegation and Cooperation,” International Economic Review, 32, 551–559.DOI
  8. [9]Fischbacher, U., S. Gächter, and E. Fehr (2001): “Are People Conditionally Cooperative? Evidence from a Public Goods Experiment,” Economics Letters, 71, 397–404.
  9. [10]Frank, R. H. (1987): “If Homo Economicus Could Choose His Own Utility Function, Would He Want One with a Conscience?” American Economic Review, 77, 593–604.
  10. [11]Güth, W. (1995): “An Evolutionary Approach to Explaining Cooperative Behavior by Reciprocal Incentives,” International Journal of Game Theory, 24, 323–344.
  11. [12]Güth, W. and H. Kliemt (1998): “The Indirect Evolutionary Approach: Bridging the Gap between Rationality and Adaptation,” Rationality and Society, 10, 377–399.DOI
  12. [13]Güth, W. and M. Yaari (1992): “An Evolutionary Approach to Explain Reciprocal Behavior in a Simple Strategic Game,” in Explaining Process and Change - Approaches to Evolutionary Economics, ed. by U. Witt, University of Michigan Press, 23–34.
  13. [14]Heller, Y. and E. Mohlin (2019): “Coevolution of deception and preferences: Darwin and Nash meet Machiavelli,” Games and Economic Behavior, 113, 223–247.
  14. [15]Hendrycks, D. (2023): “Natural Selection Favors AIs over Humans,” arXiv preprint arXiv:2303.16200.
  15. [16]Holdahl, E. and J. Wu (2026): “Institutional Screening and the Sustainability of Conditional Cooperation,” Journal of Institutional and Theoretical Economics, 1–25, published online 11 March 2026.arxiv.org/abs/2311.02813
  16. [17]Hopkins, E. (2014): “Competitive Altruism, Mentalizing, and Signaling,” American Economic Journal: Microeconomics, 6, 272–292.DOI
  17. [18]Köbis, N., J.-F. Bonnefon, and I. Rahwan (2021): “Bad Machines Corrupt Good Morals,” Nature Human Behaviour, 5, 679–685.DOI
  18. [19]Köbis, N., Z. Rahwan, R. Rilla, B. I. Supriyatno, C. Bersch, T. Ajaj, J.-F. Bonnefon, and I. Rahwan (2025): “Delegation to Artificial Intelligence Can Increase Dishonest Behaviour,” Nature, 646, 126–134.DOI
  19. [21]McGovern, H., R. Stureborg, Y. Suhara, and D. Alikaniotis (2025): “Your Large Language Models are Leaving Fingerprints,” in Proceedings of the 1st Workshop on GenAI Content Detection (GenAIDetect), Abu Dhabi, UAE: International Conference on Computational Linguistics, 85–95.arxiv.org/abs/2405.14057
  20. [22]Robson, A. J. and L. Samuelson (2011): “The evolutionary foundations of preferences,” in Handbook of social economics, ed. by J. Benhabib, A. Bisin, and M. Jackson, Elsevier, vol. 1, 221–310.DOI
  21. [23]Tennenholtz, M. (2004): “Program Equilibrium,” Games and Economic Behavior, 49, 363–373.DOI
  22. [24]Wang, Z. and J. Wu (2026): “Preference Evolution under Partner Choice,” arXiv preprint arXiv:2304.11504.arxiv.org/abs/2304.11504
  23. [26]Zimmaro, F., M. Miranda, J. M. Ramos Fernández, J. A. Moreno López, M. Reddel, V. Widler, A. Antonioni, and T. A. Han (2024): “Emergence of Cooperation in the One-Shot Prisoner’s Dilemma through Discriminatory and Samaritan AIs,” Journal of the Royal Society Interface, 21, 20240212.DOI

Paper details

vlnc:W_zjsskzcp
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Economics, Econometrics and Finance

OpenAI’s ChatGPT, Anthropic’s Claude

View statement
Looking for related papers…
Contents