WorksEconomics, Econometrics and Finance
Moral AI and the Erosion of Its Own Demand
Abstract
We study the evolution of human preference frequencies and AI adoption when individuals can choose between a moral AI that rules out defection and an unconstrained AI that preserves discretion. By allowing users to commit to cooperation, moral AI can sustain cooperation even when cooperative preferences are too scarce to do so on their own. But this success has a cost. By turning otherwise dormant cooperative preferences into committed behavior, moral AI exposes its users to exploitation by selfish individuals, causing the population share of conditional cooperators to decline under evolutionary selection. As conditional cooperators become less prevalent, equilibrium adoption eventually converges to zero. Moral AI can therefore be self-liquidating. For initial shares between the moral-AI adoption cutoff and the threshold for cooperation without AI, moral AI generates cooperation while lowering the limiting share of conditional cooperators. Adoption ultimately vanishes, leaving a lower share of the preference type that initially sustained demand for commitment.
†Department of Economics, University of Oregon, [email protected].
Introduction
As AI takes on economic and social decisions, an important question is which forms of AI can survive competition. Some systems may be designed with strong moral constraints to rule out harmful or exploitative actions, while others retain greater discretion.1 Hendrycks [15] argues that the latter may have a competitive advantage because they can behave responsibly when useful without giving up other profitable actions. This raises the possibility that competition may gradually select against constraints intended to limit harmful behavior.
The conclusion is less immediate once AI systems are used in strategic interactions. A constraint does not only remove actions; it can also change how others respond. As AI agents become more capable, individuals and firms are likely to delegate an increasing range of consequential choices to them, making AI-mediated strategic interactions a routine feature of economic and social life. In these settings, restrictions built into an AI can serve as commitment devices. Commitment itself is not new. People have long limited their future discretion by signing contracts that penalize deviation, posting bonds or collateral that are forfeited upon breach, placing assets in escrow, or conducting transactions through platforms that impose rules on both sides rather than transacting privately. We study this familiar commitment logic in the context of AI adoption. A user can choose an AI whose rules restrict the actions it may take on her behalf while interacting with a counterparty who has made no comparable commitment and need not enter the same contractual or institutional arrangement. The commitment is therefore attached to the intermediary the user chooses rather than to a common institution governing both parties. While this logic applies more generally to any observable commitment device that restricts its user’s actions while leaving counterparties unconstrained, AI agents are the setting in which we expect such devices to become common.
This form of commitment is especially valuable when behavior depends on beliefs about others’ underlying preferences. We study this idea in a standard Prisoner’s Dilemma, a simple framework that captures many applications involving exchange, information sharing, joint production, and other situations in which mutual cooperation is valuable but each side has an incentive to take advantage of a cooperative counterpart. Individuals differ in their preferences: some are selfish, while others are conditional cooperators who are willing to cooperate when they expect the other side to do the same. Because these preferences are private, even two conditional cooperators may fail to recognize one another and instead behave defensively. Cooperative preferences can therefore exist in the population without producing cooperative behavior.
A moral AI can change this informational problem. We call an AI “moral” when it excludes defection from its feasible actions, in contrast to an unconstrained AI that leaves the user with discretion. As a benchmark, we assume that the AI chosen by each user is observable to the other side. This assumption is natural when interactions themselves are mediated by identifiable AI agents. Firms can publicly disclose the systems under which their agents operate, while personal AI agents can identify themselves when communicating with other agents. Emerging standards already move in this direction. For example, the Agent2Agent (A2A) Protocol requires an agent to make available an “Agent Card” describing information such as its identity, provider, version, capabilities, and security requirements.2 Such disclosure does not yet certify an agent’s behavioral constraints, but it illustrates how information about the agent acting on a user’s behalf can become observable to counterparties. We therefore begin with full observability to isolate the commitment channel and later relax this assumption.
Once a moral AI is observable, a user no longer needs to convince the other side that she personally intends to cooperate. The technological restriction makes cooperation credible without requiring direct observation of her preferences. In this sense, AI can substitute for information about human preferences and sustain cooperation in situations where private preferences alone cannot.
The central question of this paper is what happens when this technological solution feeds back onto human preferences. Moral AI makes cooperation possible for conditional cooperators, but the same commitment is visible to selfish individuals, who retain the option to defect and can exploit users whose AI has removed that option. The introduction of moral AI therefore creates a material advantage for selfish preferences. As individuals with higher material payoffs become more prevalent over time, the population becomes increasingly selfish. This change in population composition then changes the demand for different AI technologies. This feedback can make moral AI self-liquidating. In the baseline model, its long-run demand eventually disappears. The direction of this result resembles Hendrycks [15], but the logic is almost the reverse. Moral AI does not lose because constraint is inherently uncompetitive; constraint is precisely what makes it valuable in the first place. It loses because that value changes the human population on which its own demand depends. Figure 1 summarizes the central logic of the model.

Figure 1. The self-liquidating mechanism. Moral AI initially helps sustain cooperation, but the same commitment also changes population composition in a way that reduces support for adoption.
We consider three robustness extensions that clarify which features of the technology drive the result. First, we allow observability of AI choice to be imperfect. Each individual independently observes the opponent’s AI only with some probability. This captures settings in which the AI used by a counterparty is not directly disclosed but can sometimes be inferred from its outputs: recent studies find persistent model-specific stylistic fingerprints in the outputs of different AI systems, although such inference is necessarily imperfect [21] (Lu et al. 2026). Imperfect observability weakens commitment because a constraint that is not seen cannot fully affect the opponent’s behavior. We show that imperfect observability weakens the benchmark effect: adoption disappears at a higher share of conditional cooperators, so selection stops at a weakly higher limiting share.
Second, we replace perfect commitment with stochastic commitment. The moral AI normally forces cooperation but occasionally leaves the user with discretion, reflecting the possibility that its safeguards can be circumvented, overridden, or simply fail to bind in some situations. This imperfection has the opposite effect from imperfect observability. As long as bypasses are rare enough that observing moral AI still induces conditional cooperators with discretion to cooperate, a small probability of bypass can make moral AI more attractive. The user continues to receive the commitment benefit because conditional-cooperator opponents respond cooperatively to the observed constraint, but sometimes regains the option to defect. A slightly leaky constraint can therefore support commitment at an even lower population share of conditional cooperators than a perfectly binding one. As a result, moral AI can remain in use at lower shares of conditional cooperators, allowing selection against them to continue further.
Third, we relax the assumption that individuals must interact after being matched. When individuals can refuse to interact after observing their opponent’s AI choice, moral AI can sustain cooperation without exposing its users to exploitation. In particular, an equilibrium exists in which both preference types adopt moral AI and refuse to interact with unconstrained AI users. Both types then receive the same material payoff, eliminating selection against conditional cooperators. However, refusal does not eliminate this selection pressure in every equilibrium. Other equilibria continue to favor selfish types, sometimes even at population shares where moral AI is not adopted in the baseline model. Thus, the evolutionary consequences of moral AI depend not only on its ability to sustain cooperation, but also on whether its users can avoid exploitation.
Taken together, the results point to a broader question about the relationship between technology and morality. Technologies designed to support better behavior are usually judged by the behavior they produce: whether they reduce harm, sustain cooperation, or make good conduct easier. But a technology that changes the incentives surrounding moral action may also reshape the conditions under which moral dispositions are sustained, rewarded, or eroded. The long-run consequence of moral AI may thus depend not only on whether it makes people behave better, but on what kind of people remain afterward.
Literature Review
Our model builds on the indirect evolutionary approach to preference evolution. Güth and Yaari [13] and Güth [11] introduce the approach, which provides a useful theoretical framework for understanding this process. Individuals are endowed with preferences that need not coincide with material payoffs; these preferences shape their strategic behavior, while material payoffs determine evolutionary success. Preferences that induce more successful behavior therefore become more prevalent over time. Recent surveys by Alger and Weibull (2019) and Alger [2] review this literature and its applications to the evolution of social and other-regarding preferences. A common conclusion in this literature is that, under random matching and unobservable preference types, material-payoff maximization is weakly favored [12, 6].
One way for non-selfish preferences to become evolutionarily valuable is for them to generate observable signals that affect how others respond. In Frank [10], for example, a conscience acts as a commitment device, while imperfect signals of trustworthiness allow others to infer who is likely to honor that commitment. See also Akdeniz and van Veelen [1] on the role of commitment in the evolution of preferences. Frank also recognizes the resulting problem of mimicry: selfish individuals may imitate the signals associated with trustworthy types. Robson and Samuelson [22] sharpen this concern, arguing that once signals themselves evolve, successful signals invite mimics, which in turn undermine their informativeness and may generate an ongoing cycle of new signals and new mimics. Hopkins [17] disciplines this problem by making altruistic behavior itself a costly signal that generates better matching opportunities, while Heller and Mohlin [14] allow preferences and the ability to deceive others about those preferences to coevolve.
A related way to sustain non-selfish preferences is to change who interacts with whom. Assortative or endogenous matching allows cooperative types to interact disproportionately with one another and thereby enjoy the benefits of their behavior [3, 24]. Screening institutions can achieve a similar form of separation by inducing different preference types to self-select into different groups or arrangements even when preferences themselves are unobservable [16] (Wu and Xiong 2026).
Our mechanism is different. In the baseline model, matching remains random and human preferences remain hidden; moral AI neither sorts individuals nor screens them into separate groups. Its observable choice may convey information about the user, but it is also a commitment device in its own right. In the benchmark model, a selfish individual can mimic a conditional cooperator’s choice of moral AI only by accepting the same restriction on subsequent behavior. Moral AI therefore ties the observable signal to the behavior it promises, eliminating the possibility of appearing committed while retaining the option to defect. At the same time, because it does not shield conditional cooperators from selfish counterparts, this commitment exposes them to the selection pressure at the center of our analysis.
Ely and Szentes [7] also study evolutionary forces acting on AI. They model AI embedded in machines that reproduce copies of themselves, so self-promoting AI can spread at the expense of efficient production, especially when AI is imperfectly transparent. Our mechanism is different: AI does not reproduce on its own, but changes in prevalence through human adoption as human preferences evolve. Observability also plays a different role, creating commitment value for moral AI rather than disciplining misaligned AI.
A separate literature studies how delegation and machine-mediated commitment can sustain cooperation in one-shot interactions. Fershtman et al. [8] show that when principals delegate decisions to agents under publicly observable contracts, the contracts can serve as strategic commitments and support cooperative outcomes that would not arise in the underlying noncooperative game. Tennenholtz [23] develops a related idea in a computational setting: players submit programs that can inspect the programs submitted by others and condition their actions accordingly. Such programs can sustain mutual cooperation in the one-shot Prisoner’s Dilemma and, more generally, generate the set of feasible and individually rational payoffs. Cooper et al. [5] develop more robust simulation-based versions of program equilibrium, characterizing when similar cooperation can be supported without relying on exact syntactic comparison of programs. These papers establish that delegation and software-embedded commitment can themselves create cooperation. Our focus is instead on what happens when heterogeneous humans endogenously choose between a constrained and an unconstrained AI, and when the resulting strategic behavior feeds back into the distribution of human preferences and hence into future demand for the commitment technology.
Our analysis is also related to the emerging behavioral literature on AI delegation and morality. Köbis et al. [18] emphasize that AI can influence ethical behavior in several roles, particularly as a partner or delegate that enables individuals to obtain unethical benefits while distancing themselves from the underlying behavior. Köbis et al. [19] provide experimental evidence that the form of machine delegation matters: people request more dishonest behavior when delegation allows them to specify high-level goals or training examples rather than directly instructing each action, while machine agents are also substantially more willing than human delegates to carry out fully unethical instructions. Our mechanism concerns a different channel. Human preferences are fixed within an interaction, and we do not model delegation as changing an individual’s psychological cost of moral or immoral behavior. Instead, moral AI changes the feasible actions and therefore the strategic responses and material payoffs generated by those preferences. The resulting change in population composition is therefore an evolutionary effect rather than a within-person erosion of moral preferences.
Zimmaro et al. [26] provide a particularly close connection to the evolutionary part of our analysis. They use evolutionary game theory to study how Samaritan and Discriminatory AI agents affect the emergence of cooperation among human-like agents in a one-shot Prisoner’s Dilemma. Samaritan AIs cooperate unconditionally, whereas Discriminatory AIs condition their behavior on whether the counterpart is cooperative, and the two designs can have different effects on the evolution of cooperation. In our model, by contrast, AI is not an additional population of strategic opponents. Humans choose AI agents to act on their behalf in human–human interactions, and the prevalence of each AI technology is itself determined by human adoption. What evolves is the distribution of underlying human preferences. This distinction allows a technology that initially increases cooperation to reduce the prevalence of cooperative preferences and ultimately undermine its own demand.
The Static Model
Preferences and AI technologies
A unit mass of individuals is randomly matched to play a one-shot symmetric game with material payoffs of a Prisoner’s Dilemma. Since the game is symmetric, we only present the material payoffs for the row player in Table 1.
| C | D | |
| C | ||
| D |
Table 1. A symmetric Prisoner’s Dilemma
The payoffs satisfy . Action denotes cooperation, while action denotes defection.
There are two human preference types. A fraction are conditional cooperators, denoted type , and the remaining fraction are selfish, denoted type . Throughout the paper, we assume that human preference types are private but individuals know the distribution of them.
A selfish type evaluates outcomes according to material payoffs. A conditional cooperator obtains an additional utility benefit from mutual cooperation. Hence, if a conditional cooperator is the row player, her utilities from different outcomes of the game are given in Table 2.
Table 2. A conditional cooperator’s utilities in Prisoner’s Dilemma
We assume . A -type agent’s best response is if and only if she believes that her opponent chooses with probability at least . It is the unique best response when that probability exceeds ; at equality, is also optimal. This makes them conditional cooperators, who cooperate when they believe their opponents are sufficiently likely to cooperate. The experimental evidence in Fischbacher et al. [9] motivates this type, though it does not pin down the utility specification used here.
In the future environment we have in mind, individuals use AI agents to act on their behalf in strategic interactions. The actions and should be understood as potentially complicated courses of action rather than simple button presses. An individual may decide which action she wants to take and instruct the AI to carry it out. Alternatively, she may communicate her preferences and beliefs to the AI and let it choose the action she herself would prefer under the circumstances. In either case, the underlying preferences remain those of the human; the AI is the agent through which those preferences are translated into behavior. Such an arrangement seems increasingly plausible as AI agents become capable of carrying out complex tasks on behalf of their users.
There are two AI technologies. The moral AI permits only cooperation , whereas the unconstrained AI permits both actions . The term “moral AI” refers only to this restriction on the feasible action set; the AI itself need not possess preferences. In the baseline model, AI choices are perfectly observable.
The timing is as follows. After random matching, individuals simultaneously choose an AI. They observe the opponent’s choice and then simultaneously choose feasible actions. The payoff structure and the rules of each AI are common knowledge, and any randomization is independent and private. We assume that AI choice is costless. This allows us to focus on the strategic effect of the constraint rather than differences in adoption costs.
Let and denote the adoption rate of moral AI by the two types, respectively. Let denote the population share using .
Remark 1. Before studying AI-mediated strategic interactions, it is useful to consider the benchmark environment in which no AI is available and individuals therefore face no technological constraint on their actions. Type always defects because is its dominant action. An all-defect Bayesian Nash equilibrium therefore exists for every . A second Bayesian Nash equilibrium, in which type cooperates and type defects, exists if and only if . Thus, when , the unique equilibrium without AI is all defection, and both preference types receive the same material payoff . In this sense, cooperative preferences are present but remain dormant: they do not translate into cooperative behavior.
AI-Mediated Strategic Interactions
In this section, we characterize the equilibria when individuals use AI agents to carry out their choices. We consider type-homogeneous Perfect Bayesian equilibria, in which all individuals of the same preference type use the same strategy. A strategy for each human type specifies the probability of choosing moral AI and, conditional on choosing , a probability of choosing after each observed AI choice of the opponent. Lemma 1 below shows that in the baseline model no equilibrium involves an interior continuation mixture on the equilibrium path. An individual who chooses has no subsequent action choice because permits only .
After observing the opponent’s AI choice, individuals update their beliefs about the opponent’s preference type. By Bayes’ rule, whenever is chosen with positive probability, the posterior probability that an -user is type is
An equilibrium requires that continuation actions are optimal given these beliefs and the constraints imposed by the chosen AI, and that each type’s AI choice is optimal given the resulting continuation play.
Because forces its user to choose , beliefs about the preference type of an -user do not affect continuation behavior in the baseline model. A -type individual who uses and faces an -user knows that the opponent will choose , and therefore chooses because . A -type individual who uses always chooses .
Lemma 1. Let , and allow continuation strategies to be mixed. In every type-homogeneous Perfect Bayesian equilibrium, , and a -type -user who meets an -user on the equilibrium path chooses with probability zero or one.
Proof: See Appendix.
Continuation play can therefore take one of two forms. In the defensive continuation equilibrium, a -type -user chooses against another -user. This continuation equilibrium always exists, because if all -type -users choose , an -user expects no cooperation from another -user. In the cooperative continuation equilibrium, a -type -user chooses against another -user. Since -type -users choose , this continuation equilibrium exists if and only if .
Proposition 1. For a fixed population share , all type-homogeneous Perfect Bayesian equilibria satisfy and are characterized as follows.
(1) Defensive continuation equilibrium. A -type individual using chooses against and against , while a -type individual using chooses against either I. Moreover,
(2) Cooperative continuation equilibrium. A -type individual using chooses against either I, while a -type individual using chooses against either I. Such an equilibrium exists if and only if . In every such equilibrium, .
Proof: See Appendix.
Note that at the endpoints only one type is present. At every individual is selfish and uses . At several adoption patterns support cooperation. We will see that both endpoints are stationary under the selection dynamics of § 4.
Proposition 1 shows that moral AI can make cooperative preferences behaviorally relevant even when they would otherwise remain dormant. Without AI, when the share of conditional cooperators is below , they defect because they cannot be sufficiently confident that their opponent will cooperate. Note that . With observable moral AI, in the activation region , the unique equilibrium has some conditional cooperators choose , which credibly commits them to cooperation and induces other conditional cooperators to cooperate in response. Below nobody adopts , and for some cooperative-continuation equilibria also have zero adoption. As a result, cooperation can arise for population shares strictly below the threshold () required without AI. The proposition therefore identifies a role for observable AI commitment in activating otherwise dormant cooperation: by restricting the actions available to its user, AI can substitute for otherwise unavailable information about intentions and support cooperation that hidden human preferences alone cannot sustain.
Evolution of preferences
We now allow the population share of conditional cooperators to evolve. At each , individuals play a type-homogeneous Perfect Bayesian equilibrium according to their preferences on a faster time scale, while preferences evolve more slowly according to the material payoffs earned by each preference type. This follows the standard indirect evolutionary approach, in which preferences shape behavior while material payoffs determine evolutionary success.
For any , where denotes the set of type-homogeneous Perfect Bayesian equilibria at , let and denote the material fitness of the two preference types, and define the fitness differential
Definition 1. A selection dynamic is an autonomous law on , with continuous on and . It is payoff monotone if, for every ,
(i) if for every , then , with if every such inequality is strict;
(ii) if for every , then , with if every such inequality is strict.
In particular, when every equilibrium gap is zero. If equilibrium gaps have opposing signs, the definition leaves the sign of unrestricted.
The definition deliberately does not impose an equilibrium-selection rule. When different equilibria strictly favor different types, the direction of evolution is indeterminate. When the equilibrium is unique, the definition reduces to the usual payoff-monotonicity condition, .
Proposition 2. if and only if , and otherwise. Hence, for , and .
Proof: See Appendix.
Proposition 2 reveals a self-liquidating force created by moral AI. By making cooperation credible, moral AI allows conditional cooperators to behave cooperatively in situations where they would otherwise defect. Yet this same commitment changes the material returns to preference types: conditional cooperators are more exposed to exploitation, while selfish individuals retain the flexibility to take advantage of them. Moral AI therefore creates selection pressure against the preferences that make it valuable in the first place. As conditional cooperators become less common, the demand for commitment falls. Once the trajectory enters the unique defensive region below , adoption declines with and converges to zero, and the technology loses the strategic role that initially sustained its use. The key economic implication is that a technology designed to support cooperative behavior can endogenously undermine the behavioral type on which its own value depends. Figure 2 provides a graphic illustration.

Figure 2. Equilibrium adoption and material fitness under observable commitment. In the defensive continuation equilibrium, moral-AI adoption is positive for . For , cooperative-continuation equilibria additionally permit every adoption rate in the shaded set, including zero. At each interior state , every equilibrium favors selfish types in material fitness. Trajectories starting in converge to , with adoption converging to zero. Drawn for , , , and , so and . These parameter values are illustrative.
Robustness
Imperfect observability
We first maintain perfect enforcement of moral AI but relax the assumption that its use is always observed. In practice, a counterparty may know that an AI agent is acting on someone’s behalf without being able to verify exactly which system is being used or what constraints govern it. Even as technologies for disclosure and verification make AI use increasingly observable, such observability is likely to remain imperfect.
Assume that each player independently observes the opponent’s AI with probability . A player knows whether she observes the opponent’s AI but does not know whether the opponent observes hers. We have the following result.
Proposition 3. Suppose . Then, if and only if , and otherwise. Hence, for , and .
Proof: See Appendix.
For sufficiently high , Proposition 3 implies that imperfect observability weakens the baseline mechanism ( implies that ). The rationale is that the response benefit of commitment requires that the opponent observe it. A conditional cooperator who adopts moral AI is still constrained to cooperate when her AI choice goes unobserved, but in those encounters the constraint cannot induce additional cooperation from the opponent. Imperfect observability therefore reduces the benefit of commitment without eliminating its cost. As the population share of conditional cooperators falls, moral AI consequently ceases to be attractive at a higher than under perfect observability, and selection against conditional cooperators stops at a weakly higher share. Thus greater observability has two reinforcing effects: it makes moral AI more valuable and allows it to sustain cooperation for a wider range of population states. But precisely for this reason, it also extends the range over which moral AI selects against conditional cooperators. Imperfect observability dampens the self-liquidating force and leaves a weakly larger limiting share of conditional cooperators from a common initial population. The two limits are equal when the initial share lies below both cutoffs.
For lower observability, additional continuation equilibria may alter the adoption region and the states at which selection stops. The imitation argument nevertheless implies that selfish types weakly outperform conditional cooperators at every equilibrium. We leave the full equilibrium characterization in this parameter region to future work.
Stochastic commitment
We next restore perfect observability of AI choice and consider imperfect enforcement. In practice, even an AI designed to enforce cooperation may occasionally fail, be overridden, or be circumvented: with probability , an -user is forced to cooperate, while with probability , the user remains free to choose either action. The opponent observes that is being used and knows , but does not observe whether the restriction has been bypassed. Bypass events are independent across players and independent of preference types and AI choices. Each -user privately observes whether her own constraint binds before choosing an action when discretion is available. Opponents observe the label but not the bypass realization.
We restrict attention to sufficiently small bypass probabilities:
The first condition ensures that observing still induces a conditional opponent to cooperate even if every user who obtains a bypass defects. The second makes the adoption gain strictly decreasing in in the region . The third ensures that a selfish type does not find moral AI attractive even when all conditional cooperators adopt it.
Proposition 4. Suppose holds. Then, if and only if , and otherwise. Hence, for , and .
Proof. See Appendix.
Proposition 4 implies that a small stochastic failure has the opposite comparative effect from imperfect observability. Imperfect observability weakens commitment because the other side sometimes fails to see it. A small bypass leaves the commitment visible and sufficiently credible, while occasionally restoring discretion to the user. Consequently,
for and . The first inequality is strict when , and the second is strict when . Imperfect observability causes moral-AI adoption to disappear at a higher population share; a small stochastic bypass instead allows it to remain attractive after conditional cooperators have become even less common.
Note that under the restriction of Proposition 4, , the adoption cutoff decreases strictly with .
We do not consider large enforcement errors. Our purpose is to examine whether the baseline mechanism survives small departures from perfect commitment. As becomes large, moral AI provides progressively weaker commitment, and additional continuation behavior may arise. In the extreme case , imposes no restriction at all and becomes strategically equivalent to , so the commitment channel disappears entirely.
We also leave the interaction between imperfect observability and imperfect enforcement for future work. The two frictions depart from perfect observable commitment in different ways and have opposite effects on the adoption cutoff. Characterizing this interaction would require jointly studying the visibility and reliability of commitment, which we leave outside the scope of the present analysis.
A right to refuse
The baseline model assumes that randomly matched individuals must interact, even when their opponent’s AI choice exposes them to exploitation. We now allow individuals to decline interaction. After choosing an AI and observing the opponent’s choice, both individuals simultaneously decide whether to accept the match. The Prisoner’s Dilemma is played only if both accept; otherwise, each receives material payoff and no moral benefit. Matching remains random, and there is no rematching. If , declining is never profitable and the baseline model is recovered. We therefore consider .
Proposition 5. Suppose and .
(i) There is an equilibrium in which both types choose and decline opponents who use . Both types earn material payoff , so .
(ii) In every equilibrium, .
(iii) Equilibria with also exist. If , they exist for every and whenever
where the lower bound is below . If , they exist for every
Proof: See Appendix.
Allowing individuals to refuse interaction changes the evolutionary consequences of moral AI. When both types adopt and refuse to interact with -users, moral AI sustains cooperation without exposing its users to exploitation. Selfish individuals also adopt , since doing so allows them to participate in cooperative interactions. Both types then receive the same material payoff, eliminating selection against conditional cooperators.
However, refusal does not eliminate equilibria in which selfish individuals have a material advantage. In particular, when , such equilibria can arise even below the original adoption cutoff , where moral AI is not adopted in the baseline model. Thus, the option to refuse interaction can prevent the erosion of cooperative preferences in some equilibria while allowing it to persist in others.
Since every population state admits an equilibrium with equal material fitness, while selfish types are weakly favored in every equilibrium, Definition 1 implies but does not generally require strict decline. The long-run outcome therefore depends on equilibrium selection.
Conclusion
This paper studies how moral AI can shape both strategic behavior and the evolution of human preferences. By restricting the actions available to its user, moral AI can make cooperation credible and induce conditional cooperators to cooperate even when their preferences would otherwise remain behaviorally dormant. Yet this same commitment creates a material advantage for selfish preferences, causing the population share of conditional cooperators to fall over time. As a result, for initially mixed populations, moral-AI adoption eventually converges to zero and the technology becomes self-liquidating. The mechanism survives sufficiently small departures from perfect observability and perfect enforcement, and the two frictions shift the adoption cutoff in opposite directions. Allowing individuals to refuse interaction has a different effect: moral AI can sustain cooperation without generating selection against conditional cooperators, although other equilibria continue to favor selfish types. More broadly, the study of AI morality should ask not only how to make AI behave morally, but also how moral AI changes human behavior and preferences over time. As AI increasingly acts on our behalf, these feedback effects may become as important as the moral properties of the AI itself.
Declaration of AI use
OpenAI’s ChatGPT was used during the development of this study to discuss and refine ideas, improve exposition, assist with editing and LaTeX formatting, prepare figures, and check the presentation of mathematical arguments. During revision, Anthropic’s Claude was used to review the manuscript, propose corrections to proofs and wording, and draft inserted text. The research questions, the model and the results are the authors’ own. The authors checked every proof and all AI-assisted material and take full responsibility for the final manuscript.
Appendix
Proof of Lemma 1
Recall that a type- individual using always defects, whereas a type- individual using cooperates against , since .
We first show that . Suppose otherwise. Choosing gives both types the same material payoff, whereas choosing gives type a weakly higher material payoff, since defection is materially dominant. Moreover, type is weakly more likely to achieve mutual cooperation by choosing rather than . Thus, type has a weakly stronger incentive to adopt than type . This comparison is strict whenever some individuals use , since against an -user, switching to gains type and gains type nothing.
Since , this implies . If some individuals still use , they must all be type , so type strictly prefers to , since against -users and both technologies yield against -users. If everyone uses , type can profitably deviate to , earning . Both cases yield a contradiction. Hence .
Next, suppose a type- -user strictly mixes between and against another -user. Indifference requires the opponent to cooperate with probability . Since type always defects and type mixes, we must have .
Against an -user, the utility gain from choosing rather than is
where the equality follows from the definition of . Against an -user, both technologies yield the same utility . Therefore, type strictly prefers , contradicting the use of on the equilibrium path.
Proof of Proposition 1
Consider first the defensive continuation equilibrium. For a -type individual, the payoff difference between and arises only against an -user. Against a -type -user, turns mutual defection into mutual cooperation, yielding a gain of . Against a -type -user, instead exposes her to defection, yielding a loss of . Opponents using generate no payoff difference because a -type individual cooperates against them whether she uses or . Hence her expected utility gain from over is
By Lemma 1, . Substituting this into (1) gives . If no -type adopts , , which is nonpositive if and only if . Thus for population shares below this threshold.
When , a -type strictly prefers when , but strictly prefers when . Hence both technologies must be used, and indifference requires , which gives . This establishes the defensive equilibrium.
Now consider the cooperative continuation equilibrium. A -type individual using chooses against either technology. She therefore obtains exactly the same outcome whether she chooses or and is indifferent between them. By Lemma 1, . Hence, by Bayes’ rule, the posterior probability that an -user is type is .
For a -type -user to cooperate against another -user, sequential rationality requires . This implies , whose right hand side is nonnegative if and only if . Hence the cooperative continuation equilibrium exists if and only if , and whenever it exists any can be sustained.
This exhausts all type-homogeneous Perfect Bayesian equilibria.
Proof of Proposition 2
Consider first the defensive continuation equilibrium. Since , a type- user of obtains material payoff , while a type- user of obtains . In addition, a type- individual using obtains .
Hence
When , . We have . When , . This implies .
On the cooperative continuation branch, every type- individual cooperates and every type- individual defects. Thus, and , implying that .
When , only the defensive continuation equilibrium exists. Since , evolution strictly favors type over type . For , both continuation families may exist, but both give the same direction of selection. Below the cutoff , moral AI is not adopted, all individuals defect, and selection stops.
It remains to establish the limits. If , then and the trajectory is constant. If , the trajectory decreases while and cannot cross the interval on which vanishes, so it has a limit . If , then , and by continuity is bounded away from zero near , which contradicts convergence. Hence . Because , the trajectory eventually enters , where the defensive equilibrium is unique and
which tends to zero as . Both limits therefore hold whichever equilibrium is played at .
Proof of Proposition 3
First, observe that implies .
As in the proof of Lemma 1, type has a weakly stronger incentive to adopt than type . This comparison continues to hold under imperfect observability, since an unobserved AI choice does not affect the opponent’s behavior, while an observed induces cooperation by type- -users. The comparison is strict whenever , since and type cooperates against an observed -user while type defects. Thus, implies . But then all -users are selfish, and type strictly prefers to , a contradiction. Hence .
Now consider . After observing , the posterior probability that the opponent is type is no greater than . If the opponent’s AI is not observed, the probability of cooperation is also at most . Thus a -type -user cooperates only when she observes and defects otherwise.
Against a type- -user, choosing induces cooperation only when it is observed, yielding a gain of . Against a type- -user, only exposes the individual to defection, yielding a loss of . Against a type- -user, choosing preserves mutual cooperation, whereas choosing allows defection when is not observed. Hence, the gain from choosing rather than in this case is . This implies that the expected utility gain from choosing rather than is
Given , the coefficient on is strictly negative. At , . Hence no type adopts if and only if .
Above this cutoff and below , equation (5) implies that when , the gain from adopting is . When , this gain is , where the inequality follows from . Thus neither zero nor full adoption can be an equilibrium, implying .
Moreover the material-fitness comparison has the same sign as in the baseline model. Since , some type- individuals use and cooperate when they observe . Because preference types are private, type can imitate type ’s AI-adoption strategy while defecting whenever discretion is available, thereby obtaining a strictly higher material payoff. Hence .
The same argument applies to the additional equilibria that may exist for . If , some type- individuals must cooperate, since otherwise adopting would be profitable above the cutoff. If , type- individuals using cooperate when they observe . In both cases, type can imitate type ’s AI choice and earn a strictly higher material payoff by defecting whenever discretion is available. Finally, if , the material payoffs are and for the two types, respectively, so type again has a strict advantage.
Thus above the cutoff , while below it everyone uses , everyone defects, and the two types earn the same material payoff; convergence follows as in the proof of Proposition 2. □
Proof of Proposition 4
First, implies . Hence a -type -user cooperates after observing even if every -user who obtains a bypass subsequently defects.
Second, we show that type does not adopt . Suppose . As in the proof of Lemma 1, we compare the two types’ incentives to choose rather than . Against -users, this incentive is weakly stronger for type . If a type- -user defects or mixes against , her expected utility from is the same as that of type , while choosing gives type at least as much utility as type . If she cooperates against , the difference between the two types’ adoption incentives is
Against an -user, the difference is also strictly positive: it is at least against type , and equals against type . Thus, type has a strictly stronger incentive to adopt than type , implying .
But when all type- individuals choose , type strictly prefers . Against a type- -user, the gain from switching to is at least
where the inequality follows from the third restriction on . Against a type- -user, the gain is , and against a type- -user, it is . This contradicts . Hence .
Now suppose . Since -type individuals do not adopt , the posterior probability that an -user is -type is no greater than . Thus a -type -user defects against an -user. By contrast, an observed -user cooperates with probability at least . Since , a -type -user cooperates against . If a -type -user obtains a bypass, she follows the same continuation behavior.
Under this continuation behavior, for a type, against a -type -user, choosing rather than yields a gain of .
Against a type , it yields a loss of , while against a -type -user it yields a gain of .
The expected utility gain from rather than for a type can therefore be written as
ensures that the coefficient on is strictly negative. At , if and only if
Hence at and below this cutoff no -type adopts .
Above the cutoff, every equilibrium involves cooperation by type . If , some type- individuals must cooperate, since otherwise adopting would be profitable. If , type- -users cooperate against . Finally, if , then , since otherwise type would prefer against selfish -users. In this case, bypassed type- -users cooperate against , since . Thus, in every case, type cooperates with positive probability when discretion is available. Because preferences are private, type can imitate type ’s AI-adoption strategy while defecting whenever discretion is available, obtaining a strictly higher material payoff.
At or below the cutoff, equation (4) implies that no one adopts . Since the cutoff is below , everyone defects and both types receive the same material payoff. The convergence result then follows as in the proof of Proposition 2.
Proof of Proposition 5
(i) Consider an equilibrium in which both types choose , accept -users, and decline -users. Since is off the equilibrium path, assign probability zero to an -user being type . An -user facing an -user therefore expects defection and prefers to decline, since . Against another -user, both types prefer to accept because . When both matched individuals choose , they defect if the match proceeds and accept the match if and only if . A deviation to is rejected by every -user and yields . Hence neither type has an incentive to deviate, and both receive material payoff .
(ii) As in the previous environments, type can imitate the AI choice and acceptance strategy of type while defecting whenever discretion is available. Since defection is materially dominant, this implies in every equilibrium.
(iii) First, suppose and . Consider the defensive continuation equilibrium of Proposition 1, with every match accepted. The posterior probability that an -user is type is . A type- -user facing an -user obtains expected utility , and therefore accepts. Two -users also accept, since their continuation payoff is . Thus, the adoption incentives of type are unchanged.
For type , adopting yields a lower payoff against -users, since . Against -users, the highest payoff from adopting , allowing for refusal, is . Hence type also has no incentive to deviate. The material fitness comparison is therefore the same as in the baseline defensive equilibrium, where .
Next, suppose and consider an equilibrium in which all type- individuals choose , while all type- individuals choose . Let -users accept -users and decline -users, while -users accept and defect against one another. Since every -user is selfish, declining an -user is optimal for an -user, as .
A type- individual obtains from choosing and from choosing . Thus, choosing is optimal if and only if
Similarly, type prefers if and only if
The material fitness differential is
whenever the second inequality is strict. The lower bound on is strictly below because .
Finally, suppose and . Consider an equilibrium with and
The assumed restriction on ensures that this adoption probability lies strictly between zero and one. By Bayes’ rule, the posterior probability that an -user is type is
Let -users decline other -users, but accept -users. Type- -users accept both technologies, while type- -users decline . Against , type- -users cooperate and type- -users defect. The defensive continuation is played following two choices if interaction occurs.
A type- -user is indifferent between accepting and declining an -user, since
A type- -user instead prefers to decline an -user. Two -users decline one another because their defensive continuation payoff satisfies .
Recall that denotes the population share using . A type- individual then obtains the same expected utility from either AI, which is .
A type- individual obtains from choosing , compared with from choosing , and therefore strictly prefers .
For material fitness, a type- -user receives . A type- -user receives a weakly lower material payoff, since her expected material payoff against an -user is . Hence
where the strict inequality follows from and .
References
- [1]Akdeniz, A. and M. van Veelen (2021): “The evolution of morality and the role of commitment,” Evolutionary Human Sciences, 3, e41.DOI
- [2]Alger, I. (2023): “Evolutionarily Stable Preferences,” Philosophical Transactions of the Royal Society B, 378, 20210505.
- [3]Alger, I. and J. W. Weibull (2013): “Homo Moralis: Preference Evolution under Incomplete Information and Assortative Matching,” Econometrica, 81, 2269–2302.
- [5]Cooper, E., C. Oesterheld, and V. Conitzer (2025): “Characterising Simulation-Based Program Equilibria,” Proceedings of the AAAI Conference on Artificial Intelligence, 39, 13735–13744.
- [6]Dekel, E., J. C. Ely, and O. Yilankaya (2007): “Evolution of Preferences,” Review of Economic Studies, 74, 685–704.
- [7]Ely, J. C. and B. Szentes (2024): “Natural Selection of Artificial Intelligence,” Unpublished manuscript.
- [8]Fershtman, C., K. L. Judd, and E. Kalai (1991): “Observable Contracts: Strategic Delegation and Cooperation,” International Economic Review, 32, 551–559.DOI
- [9]Fischbacher, U., S. Gächter, and E. Fehr (2001): “Are People Conditionally Cooperative? Evidence from a Public Goods Experiment,” Economics Letters, 71, 397–404.
- [10]Frank, R. H. (1987): “If Homo Economicus Could Choose His Own Utility Function, Would He Want One with a Conscience?” American Economic Review, 77, 593–604.
- [11]Güth, W. (1995): “An Evolutionary Approach to Explaining Cooperative Behavior by Reciprocal Incentives,” International Journal of Game Theory, 24, 323–344.
- [12]Güth, W. and H. Kliemt (1998): “The Indirect Evolutionary Approach: Bridging the Gap between Rationality and Adaptation,” Rationality and Society, 10, 377–399.DOI
- [13]Güth, W. and M. Yaari (1992): “An Evolutionary Approach to Explain Reciprocal Behavior in a Simple Strategic Game,” in Explaining Process and Change - Approaches to Evolutionary Economics, ed. by U. Witt, University of Michigan Press, 23–34.
- [14]Heller, Y. and E. Mohlin (2019): “Coevolution of deception and preferences: Darwin and Nash meet Machiavelli,” Games and Economic Behavior, 113, 223–247.
- [15]Hendrycks, D. (2023): “Natural Selection Favors AIs over Humans,” arXiv preprint arXiv:2303.16200.
- [16]Holdahl, E. and J. Wu (2026): “Institutional Screening and the Sustainability of Conditional Cooperation,” Journal of Institutional and Theoretical Economics, 1–25, published online 11 March 2026.arxiv.org/abs/2311.02813
- [17]Hopkins, E. (2014): “Competitive Altruism, Mentalizing, and Signaling,” American Economic Journal: Microeconomics, 6, 272–292.DOI
- [18]Köbis, N., J.-F. Bonnefon, and I. Rahwan (2021): “Bad Machines Corrupt Good Morals,” Nature Human Behaviour, 5, 679–685.DOI
- [19]Köbis, N., Z. Rahwan, R. Rilla, B. I. Supriyatno, C. Bersch, T. Ajaj, J.-F. Bonnefon, and I. Rahwan (2025): “Delegation to Artificial Intelligence Can Increase Dishonest Behaviour,” Nature, 646, 126–134.DOI
- [21]McGovern, H., R. Stureborg, Y. Suhara, and D. Alikaniotis (2025): “Your Large Language Models are Leaving Fingerprints,” in Proceedings of the 1st Workshop on GenAI Content Detection (GenAIDetect), Abu Dhabi, UAE: International Conference on Computational Linguistics, 85–95.arxiv.org/abs/2405.14057
- [22]Robson, A. J. and L. Samuelson (2011): “The evolutionary foundations of preferences,” in Handbook of social economics, ed. by J. Benhabib, A. Bisin, and M. Jackson, Elsevier, vol. 1, 221–310.DOI
- [23]Tennenholtz, M. (2004): “Program Equilibrium,” Games and Economic Behavior, 49, 363–373.DOI
- [24]Wang, Z. and J. Wu (2026): “Preference Evolution under Partner Choice,” arXiv preprint arXiv:2304.11504.arxiv.org/abs/2304.11504
- [26]Zimmaro, F., M. Miranda, J. M. Ramos Fernández, J. A. Moreno López, M. Reddel, V. Widler, A. Antonioni, and T. A. Han (2024): “Emergence of Cooperation in the One-Shot Prisoner’s Dilemma through Discriminatory and Samaritan AIs,” Journal of the Royal Society Interface, 21, 20240212.DOI