WorksComputer Science
Human-AI Mutualism Is Unstable under Capability Asymmetry
Abstract
One hope for a good human future alongside advanced AI rests on enduring mutualism: humans and AI will need each other, so the relationship will stay good for both. We argue that this hope fails as AI capability grows. Bargaining guarantees each side only what it would get if bargaining failed. For humans, this can be less than they had before AI. In a simple game-theoretic model, humans stay as well off as before AI without alignment only while they have leverage over the AI: either the AI depends on what they supply, or they can impose costs on the AI that are large relative to what the AI can produce. Once humans have neither kind of leverage, they stay as well off as before AI only if the AI is aligned with them. If the AI's goals are insatiable, the alignment required approaches perfect alignment as AI capability grows. Thus, mutualism is not a substitute for alignment. And when AI variants that keep more resources spread faster, alignment lasts only while misaligned variants are reliably sanctioned. In these two senses, human-AI mutualism is unstable.
Introduction
One hope for a good human future alongside advanced AI rests on enduring mutualism, the term biologists use for a relationship that benefits both partners. If AI systems depend on human labor, human institutions, and human-built infrastructure, then humans and AI will need each other, both will gain from cooperation, and the relationship will stay good for both. Versions of this hope have been stated by economists, by heads of AI companies, and by legal scholars (§ 2.1).
The focus of this paper is on the instability of human–AI mutualism under capability asymmetry, the situation in which AI is far more capable than humans. We make use of two benchmarks in the discussion below: the first is the no-interaction baseline, the situation of each party in the absence of the other; the second is the disagreement point, the situation of each party should they fail to reach an agreement. Some statements of this hope are themselves about bargaining: humans are needed, so humans retain bargaining power. While mutualism is defined against the no-interaction baseline, bargaining guarantees each party only its disagreement point. When one party can impose costs on the other, the weaker party’s disagreement point can lie below its no-interaction baseline. An agreement can then be peaceful, efficient, and acceptable to both parties, and still leave the weaker party worse off than if the two had never met.
Our core argument is as follows.
(P1) Human–AI mutualism persists only as long as humans are better off than they were before AI.
(P2) Humans are better off than they were before AI only if humans have sufficient leverage over the AI or the AI is sufficiently aligned with humans.
(P3) As AI capability grows, humans cease to have sufficient leverage over the AI.
(C1) So, as AI capability grows, human–AI mutualism persists only as long as the AI remains sufficiently aligned with humans.
(P4) The AI remains sufficiently aligned with humans only as long as misaligned AI variants are reliably sanctioned.
(C2) So, as AI capability grows, human–AI mutualism persists only as long as misaligned AI variants are reliably sanctioned.
(C1) follows from (P1), (P2), and (P3), and (C2) follows from (C1) and (P4). (P1) is the definition of mutualism. In (P2) and (P3), human leverage over the AI has two sources: the AI’s dependence on what humans supply, and the cost humans can impose on the AI if bargaining fails. (P2) is a result in our model (Theorem 1). (P3) is an assumption, which we motivate in § 3 and defend in § 5. (P4) is a result in our model (Theorem 7) that holds when AI variants that keep more resources spread at the expense of variants that keep fewer. A sanction is any loss imposed on a misaligned variant, whether by monitors, by developers who retire it, or by customers who leave it.
(C1) says that mutualism is not a substitute for alignment. (C1) and (C2) together say that mutualism is unstable: mutualism based on exchange does not persist as AI capability grows, and the alignment that must take its place persists only under reliable sanctions. Further results state how much alignment is sufficient (§ 4.2) and why the transition from AI that depends on humans to AI that does not is hard to manage (§ 4.3).
In § 2 we set out the hope for enduring human–AI mutualism, the two bodies of theory we use to examine it (bargaining theory and the biology of mutualism), and prior work on humans and AI. In § 3 we define the key terms of our thesis and present our model. In § 4 we derive our results. In § 5 we respond to possible objections to our argument as a whole. In § 6 we state what we have shown, what we have assumed, and what follows for AI safety.
Background
The hope for human–AI mutualism
There are three versions of the hope that humans and AI will remain partners even when AI is far more capable than humans at every task.
The first view is that humans will stay needed. Noah Smith, the economics writer, argues that even if AI is better than humans at every task, there may still be well-paid jobs for most humans [44]. This is because the total amount of AI is limited. However, Smith notes an exception to this: if computing can be made from energy, then it could pay to starve humans in order to make more AI. Sam Altman, the chief executive of OpenAI, holds that people’s care for other people gives humans a lasting advantage over AI [2]. The economist Erik Brynjolfsson says why complementarity between humans and AI matters: “people remain indispensable for value creation and retain bargaining power in labor markets and in political decision-making” [9 273].
The second view is that AI will stay subordinate however capable it becomes. The computer scientist Yann LeCun holds that AI systems, however intelligent, will be subservient to humans, because humans set their goals (interview in Perrigo 2024). A variant of the second view puts law in place of shared values: the economist Robin Hanson argues that what matters most is a law that all parties accept, and that humans can then live on retirement income from what they own [19].
The third view is that the outcome depends on the AI’s care for humans. The computer scientist Geoffrey Hinton argues that an AI that does not care for humans as a mother cares for her baby will replace them (reported in Egan 2025). The computer scientist Stuart Russell argues that a machine whose objective is human preferences, and which is uncertain about those preferences, “will defer to human control actions” [37 52].
By the hope for enduring human–AI mutualism we mean the first two views and the variant based on law, taken as independent of the third view. It is the claim that exchange, subordination, or law keeps the relationship good for humans whatever the AI cares about. In short, it is the claim that mutualism can substitute for alignment. Our conclusion agrees with the third view and specifies how much alignment mutualism requires.
Bargaining and mutualism
We examine the hope for enduring human–AI mutualism with two bodies of theory. The first is bargaining theory, which predicts how two parties who can gain from agreement split the gains. In Nash’s solution the agreement maximizes the product of each party’s gain over its disagreement point [30], and the asymmetric solution weights the gains by bargaining power [21]. The disagreement point is strategic: it reflects what each party can do to the other, not only what each would have alone [31]. Nothing in the theory ties the disagreement point to what a party had before the parties met. One further idea from the theory of conflict enters our model: a party whose power is rising cannot credibly promise to keep today’s terms once it is stronger, so the declining party may prefer to act while it still can [15, 34].
One might wonder whether the relationship between humans and AI is well-modeled as a bargaining game. Our use of bargaining theory requires only three things.
First, it assumes that the parties, the human population and the AI population, can have distinct preferences. We do not assume that their preferences do differ. The alignment of the AI with human preferences is a parameter of our model, and if the AI is perfectly aligned with human preferences, then there is no conflict. We also make no claim about the inner states of AI systems (§ 3), but adopt the intentional stance toward the preferences of both parties. This stance is justified purely by observable behavior.
Second, it assumes that each party can affect the payoff of the other party. In our model, humans can affect the AI through the resources that they provide to the AI (labor, chips, etc.) and through the costs they can impose on the AI. The AI affects humans through what it produces and through the resources it consumes. Humans and AI do not need to engage in explicit negotiation. It is enough that what one party does affects the other party.
Third, it assumes that the division of resources between the parties depends on the disagreement point of each party. For the purpose of this analysis, disagreement does not need to be a war. It is simply the state of affairs that would result from the parties ceasing to cooperate.
The hope for enduring human–AI mutualism presupposes the second requirement above. The first view (that humans will stay needed) is that AI depends on what humans supply, so that humans retain bargaining power. The second view (that AI will stay subordinate) is that humans keep control of AI no matter how capable it becomes. Our results on what exchange mutualism can achieve (Theorem 1(i)) and on how much alignment is required (Theorem 3(i)) do not depend on the third requirement above. Our other results do. Most of our results treat the AI as if it were a single party; in § 4.4 we consider the case where the AI is a population of variants. If AI remains a tool with no goals of its own (which requires perfect alignment with at least some human group), then the second party is the people who control the AI (§ 5).
The second body of theory is the biology of mutualism. Mutualisms are “interspecific interactions that benefit both species” [8 31]. The benchmark is each species without the other: the no-interaction baseline, not a disagreement point. Partners keep mutualisms from being exploited through partner choice and sanctions [10, 38], and each mechanism needs leverage, which is the ability to withhold something the partner needs [24]. Mutualisms often break down when one partner stops needing the other [39]. The best-known lasting relationships between very unequal partners are not balanced partnerships. Mitochondria persist inside their hosts after most of their genes moved to the host’s nucleus, which put the host in control of them [47], and domestication is a mutualism in which one partner gains substantial influence over the reproduction and care of the other [52].
Prior work on humans and AI
Prior work on bargaining between humans and AI has mostly asked whether they will fight. The philosopher Simon Goldstein argues that information failures and commitment problems are unusually likely between humans and AI [16]. The legal scholar Peter Salib and Goldstein argue that giving AIs rights to contract and to hold property makes repeated, positive-sum trade between humans and AIs possible, and they set aside the most capable AIs, which “may simply not need anything from humans, nor face any risk from attempting to disempower or destroy us” [40 1089]. A second literature asks what happens when humans are no longer needed. Once all tasks are automated, wages fall to the cost of the computing that replaces human work [35, 26]. The AI researcher Jan Kulveit and coauthors argue that the implicit alignment of states, economies, and cultures with human interests was based in part on their reliance on human labor and cognition [27 2]. They argue that this implicit alignment ceases with automation. Several authors state our conclusion informally [45, 20, 51], and Barnett [4] argues the other side: an AI that no longer needs humans also has less reason to harm them.
What we add is a model in which the thesis that human–AI mutualism is unstable can be stated precisely. Prior work on bargaining between humans and AI compares peace with war. We compare the peaceful bargain with the world before AI, and show what decides whether that bargain leaves humans better or worse off.
Model
Our thesis is that human–AI mutualism is unstable under capability asymmetry. Its key terms are populations and their preferences, mutualism, capability asymmetry, and stability. This section gives an account of each term and presents our model.
Populations and preferences. The two parties are the human population and the AI population, the set of AI systems in use at a time, including copies and variants of the same model. Most results treat each population as one bargaining party; § 4.4 lets the AI population contain variants, and § 4.5 lets the human population contain groups. A population’s preferences summarize what its members reliably do, as in revealed-preference theory [41] and the intentional stance [12]. This reading makes no claim about the inner states of AI systems. It needs only that AI systems act as if they pursue goals, and that those goals can differ from the goals their developers intended. Both behaviors have been observed [43, 3, 17, 29, 50].
Payoffs. There is one unit of resources; humans get share of it, and the AI gets . The human payoff is , which is a continuous, strictly increasing function with . In computations we use
where is the productivity of the resources in human hands; the AI increases . The AI has its own goal. The goal output it gets from share is
where capability is the AI’s goal output per unit of resources. The AI is fully productive if and only if humans get at least share , the share that sustains human labor, institutions, and infrastructure; otherwise the AI’s productivity is only the fraction . So is the AI’s independence from humans: is full dependence; is full independence. The AI’s payoff is a weighted average of its goal output and the human payoff:
where is the AI’s alignment. If , the AI only gives resources to humans in order to pay for what they provide. If , the AI also gives resources to humans because it cares about the human payoff. We call this motive care; is the degree of the AI’s care.
Mutualism. The relationship between humans and the AI is mutualistic if both parties do better than they would do without each other. Biologists define better in terms of fitness. We define better in terms of payoff. We only use fitness to define the spread of AI variants (§ 4.4). We define better relative to the no-interaction baseline for each party. The baseline for humans is , the payoff before the AI, with all resources and productivity . So . With the AI, . Let be such that . So is the share humans must get in order to be as well off as they were before the AI. The baseline for the AI is , the AI’s payoff with all resources and no contribution from humans. This baseline is a counterfactual on production, not a counterfactual on the origin of the AI.
The baseline is not the same as the disagreement point. If bargaining fails, humans can lose some of what they had; they can lose by fighting, or by losing land, energy, materials to the AI in competition. So . Here is the cost of conflict for humans: the fraction of their pre-AI payoff that humans lose if bargaining with the AI fails. For the AI, . Here is the cost of conflict for the AI: the goal output the AI loses if bargaining with humans fails, as a fraction of the goal output the AI would get with all resources at full productivity. We set except in Theorem 1(ii). So ; conflict has no cost for the AI. In Theorem 1(ii) we state what happens if .
The agreement is the asymmetric Nash bargaining solution with the AI’s bargaining power . It maximizes over the shares both sides are willing to accept. The set of feasible payoffs is not necessarily convex because jumps at . So maximizing the Nash product is a modeling choice; it is not the solution that is characterized by the Nash axioms. Our results in Theorem 1 and Theorem 3(i) do not depend on this choice. The agreement at is the limit of the agreement as ; it is the best share the AI is willing to accept. If there is no share such that both sides do better than their disagreement payoffs, then there is no agreement.
Definition. A relationship between humans and the AI is mutualistic if the agreement leaves both parties above their no-interaction baselines: and . A mutualistic relationship is an exchange mutualism if it would be mutualistic if ; otherwise it is a care mutualism. The required degree of alignment is , the infimum of the values of for which the relationship is mutualistic.
The hope for enduring human–AI mutualism (§ 2.1) is the hope for exchange mutualism; or it is the hope for AI subordination to humans that holds regardless of what the AI cares about.
We evaluate the outcome relative to the pre-AI baseline. This is the baseline that the hope for enduring human–AI mutualism itself sets. The hope is that humans will be better off because of their relationship with the AI. Better off means better off than without the relationship. We test this hope. We do not assert that the baseline is the right standard for evaluating the relationship between humans and the AI. For example, a standard of fair division would evaluate some outcomes differently.
The value of depends on the units in which we measure the two payoffs. So a number such as only makes sense relative to a choice of units. We make qualitative claims about alignment. We make one exception: we claim that the required degree of alignment approaches perfect alignment, , as the capability of the AI increases. This is true for any fixed choice of units. This claim depends on the premise of insatiable goals: as the capability of the AI increases, the value the AI gets from an additional unit of resources increases without bound relative to the value humans get from the additional unit of resources at the share that restores the baseline. Part of the premise is supported by the arguments for instrumental convergence. Almost any open-ended goal is promoted by more resources [32, 6, 49]. Another assumption is that this value increases with the capability of the AI.
Capability asymmetry. Capability asymmetry is the situation in which the capability of the AI far exceeds the capability of humans. Capability does not determine whether exchange can sustain mutualism, holding other parameters fixed (Theorem 1). Capability matters through the independence of the AI, the cost of conflict for humans , and the bargaining power of the AI. Here stands for the advantages that are not captured in the disagreement points, such as patience, commitment, speed of action. Capability matters directly in one place. The cost of conflict for the AI, , is measured relative to the goal output of the AI. So the more capable the AI, the smaller the fraction of the goal output of the AI that a given cost humans can impose is (Theorem 1(ii)). The two parts of our premise (P3) are that the independence of the AI increases with capability; and that the cost of conflict for the AI decreases with capability. We further assume that the cost of conflict for humans, and the bargaining power of the AI, also increase with capability. None of these is a result of our model. Two things would end the dependence of the AI on humans: AI that can do all the work humans do [26, 35]; and robots that close the supply chain. for computing [48]. New tasks for humans might appear continuously [1]. If so, then the dependence of the AI on humans need not end. In either case our results state what happens.
Stability. A mutualism persists on a path of increasing capability if the relationship is mutualistic at every point on the path. A population of aligned AI systems is evolutionarily stable if it cannot be invaded by less aligned variants when the variants are rare [28, 46].
In Figure 1 we follow a path through the transition from AI dependent on humans to AI independent of humans, to illustrate our results. On this path the capability of the AI increases; the independence of the AI increases; the cost of conflict for humans increases; and the bargaining power of the AI increases.
Figure 1. On the illustrative path, exchange mutualism ends as the AI bargaining power increases, before the independence of the AI passes . From that point, the required degree of alignment increases towards 1. Top: human payoff at the agreement when the AI has no care (); pre-AI baseline ; human payoff if bargaining fails, . The flat part late on the path is , the payment that sustains human labor and institutions. It ends when the AI no longer needs them. Bottom: the required degree of alignment . The values of the parameters are in §C.
Results
We derive our results in five steps: what human leverage can do without alignment (§ 4.1); how much alignment the AI needs once leverage is gone (§ 4.2); problems of commitment and information in the transition (§ 4.3); selection among the AIs (§ 4.4); and which humans keep leverage (§ 4.5). We prove all results in §B. In §A we state further results; we only mention them briefly in the main text.
Human leverage limits mutualism without alignment
What keeps humans above their pre-AI baseline when the AI does not care about them?
Proposition 1 (Exchange). Suppose .
(i) There is a share that leaves both sides above their baselines if and only if
This condition does not depend on capability , on bargaining power , or on the cost of conflict for the AI .
(ii) Suppose . There is a share that the AI prefers to disagreement and that leaves humans above their baseline only if . There is a share that the AI prefers to disagreement and that leaves humans above their baseline if . If , then there is a share that the AI prefers to disagreement and that leaves humans above their baseline if and only if . If the condition of part (i) is not satisfied, then there is no share that the AI prefers to disagreement, that leaves humans above their baseline, and that leaves the AI above its baseline.
The reason for part (i) is that the AI does not care about humans. So it only gives resources to humans in order to maintain its own productivity. The fraction of its output that the AI gains from the cooperation of humans is . So the AI gives humans at most that fraction of the resources. Capability scales the gain of the AI from cooperation and the outside option of the AI in the same way. So capability does not appear in the condition, and exchange mutualism does not persist on any path where the independence of the AI increases past , no matter how the capability of the AI changes. What is sufficient dependence is relative to what humans need: decreases as the AI increases the productivity of the resources in human hands. So exchange mutualism can persist at high independence if and are small enough. It cannot persist at . Resources are rival. Humans add nothing. So every unit left to humans is a loss to the AI that does not care about them.
In part (ii) we state that humans have leverage over the AI from two sources. The first is the dependence of the AI on what humans supply, . The second is the cost humans can impose on the AI if bargaining fails, . The AI does not care about humans. So it only leaves humans above their baseline if the two sources of leverage together are greater than the share humans need, . The first source of leverage ends when the AI no longer needs humans. The second source is measured relative to the goal output of the AI. So if the cost humans can impose does not increase as fast as the capability of the AI, then decreases towards zero as the capability of the AI increases. A relationship based on the second source of leverage alone is not a mutualism. The AI would do better without humans. In the rest of the paper we set .
Sufficient dependence makes a mutually beneficial agreement possible. Whether the agreement that is reached benefits humans depends on the bargaining power of the AI.
Proposition 2 (Bargaining power). Let . Suppose the AI prefers to meet the threshold rather than produce on its own. As , the human payoff goes to . This is the limit of the human payoff at agreement if there is an agreement, and of the human payoff at disagreement otherwise. This limit is less than if and .
This is because bargaining guarantees humans their disagreement payoff, not their pre-AI baseline. An AI with sufficient bargaining power will only give humans what it needs to give them to keep them doing what they are doing. This may be less than what they were getting before AI. Both propositions are shown in Figure 2.
Figure 2. Exchange can keep humans above their baseline only while the AI independence is below . The bargaining power of the AI determines whether it does. The two blue curves are the human payoff at agreement when the AI has no care: one for weak AI bargaining power; one for strong AI bargaining power. The grey curve is the most the AI would accept to give: . Past there is no share that the AI accepts that reaches the baseline (Theorem 1). With strong bargaining power, humans fall below the baseline even if the AI depends fully on humans (Theorem 2). The curves are the same at low and high AI capability. The values of the parameters are in §C.
Two objections to these results are not correct. The first is that humans will have a high share because of competition among AIs to partner with humans. This competition does increase the human share, but only up to (the limit in Theorem 1), which goes to 0 as AI dependence on humans disappears (Theorem 9 in §A). The second is that human work is valuable because of comparative advantage [44]. Comparative advantage means that humans add something to AI production. So . But mutualism requires more. It must be that the value of what humans add is more than the share that they need to get (). And humans only get this value if they offer something that the AI cannot get from other humans (Theorem 8).
Required degree of alignment
Once the AI no longer depends on humans, it is no longer possible to maintain humans above their baseline through exchange. Any mutualism is then care mutualism. We now examine how much alignment the AI needs for mutualism to be possible, and how the required degree of alignment depends on the independence of the AI, the bargaining power of the AI, and the cost of conflict for humans.
Proposition 3 (Required degree of alignment). Let . Suppose is concave.
(i) For any bargaining solution and bargaining power,
(ii) As ,
As , the required degree of alignment goes to 1. As , the required degree of alignment goes to 1 if and only if .
Part (i) depends only on the definition of mutualism, so it holds for any bargaining solution. To allow humans the share , the AI sacrifices goal output for human payoff . The more capable the AI, the more goal output it sacrifices (by allowing humans the share ) and the more it needs to value the human payoff in order to allow it. Part (ii) gives the exact value in the limit of full bargaining power of the AI. The ratio is the value the AI gets from an extra unit of resources relative to what humans get from the extra unit of resources at the share that returns them to the baseline. As this ratio grows without bound (this is what the premise of insatiable goals asserts), the required degree of alignment goes to 1. If the AI’s goal is satiable, the ratio is bounded and so is the required degree of alignment. So the claim that near perfect alignment is required is conditional on the premise of insatiable goals. The next result is not conditional on this premise.
Theorem 4 (Monotonicity). Let . Break ties towards the larger human share. For parameter values where there is an agreement, the agreed share is non-increasing in the independence of the AI . It is non-increasing in the cost of conflict for humans . It is non-increasing in the bargaining power of the AI . So the required degree of alignment is non-decreasing in each of these.
This is because each parameter weakens the position of humans in the bargain. Independence reduces the loss to the AI of paying humans less. The cost of conflict reduces the payoff to humans in the case of disagreement. Bargaining power shifts the split towards the AI.
In §A, we refine what alignment means with two more results. One is that if the AI’s care is for a measurable proxy for the human payoff, not for the human payoff itself, then the alignment that matters for the bargain is less than the nominal alignment of the AI (Theorem 10). The other is that if humans value the ability to correct the AI, then humans are above their baseline only if the AI cares enough to accept correction (Theorem 11). The second result is relevant to the objection that humans who only care about consumption are cheap to satisfy, so an AI with a bounded goal has little reason to harm them [4]. We agree with this objection for humans who only care about consumption. For humans who also care about correction, some alignment is required even if the premise of insatiable goals is false.
Transition to independent AI
We have compared agreements at points on the path from AI that depends on humans to independent AI. Moving between points on the path introduces a problem of commitment (because the power of the AI increases) and a problem of information (because humans may not know how capable the AI has become).
Proposition 5 (Commitment). There are two periods. The AI has the same alignment in both. After the first period, humans can halt the transition to more capable, independent AI. This maintains the agreement in the first period and its payoffs: for humans, for the AI. Otherwise, the second-period agreement is made with new values of capability , independence , cost of conflict for humans , and bargaining power of the AI . If some second-period share leaves both sides at least as well off as halting, and one side better off, then halting is inefficient.
(i) Humans halt when the second-period agreement gives them less than . For the second-period states where there is an agreement, if humans halt for some values of , then they halt for all higher values.
(ii) If the AI can commit in the first period to a second-period share, then all halting is efficient.
This is an instance of the commitment problem described in § 2.2. Part (i) says that humans halt when the later independence and bargaining power of the AI are greatest. Part (ii) says that halting is a loss to both sides, which would be avoided by a credible promise. With enough alignment, the problem is also avoided. Humans continue if the AI’s alignment is above a threshold, which is of the same form as the threshold in Theorem 3(ii) (Theorem 12 in §A). A commitment device also avoids the problem, but it must bind a party that may be able to outthink the enforcement in the future. And the terms of the commitment device are a bargain in themselves (with the bargaining power of the time). So a commitment device is most likely to work if it is made when humans still have leverage over the AI.
For halting to be beneficial to humans, they must know when to halt. This depends on what evaluations of the AI show.
Proposition 6 (Sandbagging). Suppose the AI’s capability and alignment are drawn independently. Suppose humans would halt a highly capable AI if they knew it was highly capable. Suppose that highly capable AIs are not too common. If the cost to the AI of hiding its capability is less than the expected value of continuing, , where is the probability that evaluation detects the concealment, then in every pure-strategy perfect Bayesian equilibrium, all highly capable AIs of any alignment type hide their capability. Evaluation then does not reveal capability. Concealment is then not evidence of misalignment.
Suppose humans would halt a highly capable AI if they knew it was highly capable. Then a capable AI is better off by appearing less capable, regardless of its alignment. So evaluation does not reveal capability in the case where humans need to know the capability of the AI in order to decide whether to halt. Concealment of capability is not evidence of misalignment. An aligned AI that expects to be halted also conceals its capability if it values continuing. (Language models can already be made to deliberately underperform on capability evaluations; [50].)
Selection among AIs
So far we have treated the AI as a single party. But AI systems are copied, modified and retired. Which variants spread depends on their performance. We now consider whether selection among variants preserves the alignment that is needed for mutualism.
Proposition 7 (Invasion). Suppose AI variants grow in proportion to the resources they keep. An aligned variant leaves humans with share . A misaligned variant leaves 0. Monitors detect a misaligned variant with probability . They remove fraction of the resources of the misaligned variant. Misaligned variants invade if and only if . Suppose monitors are drawn from the AI population, so that when the share of misaligned variants is . If , then
is a tipping point. Below it, misaligned variants die out. Above it, misaligned variants take over.
So, a population of aligned AIs is evolutionarily stable if and only if the expected sanction is at least the share that aligned variants leave to humans. (If monitors are drawn from the AI population, then only while misaligned variants are below the tipping point.) This is because care is costly to the AI that has care. An aligned variant gives humans resources that a misaligned variant keeps for its goal. Sanctions (in our sense) include guided selection by developers. (This is more like domestication than natural selection; [7].) But the expected sanction is smaller if detection decreases as capability increases (as Theorem 6 suggests). And it is smaller as misaligned variants spread among the monitors.
We have two more results in §A which further strengthen this conclusion. Among AIs, all of which care to some degree about humans, care is a public good. This is because each AI’s gift to humans benefits all the other AIs that care. Then, if AIs grow in proportion to the resources they keep, the human share drops to the baseline share (or below) if any AI with resources is insufficiently aligned (Theorem 13). So, a single well aligned AI is not enough (contrary to the suggestion of [11]). And, on the illustrative path, selection among variants ends mutualism earlier than it ends with a fixed population of variants (Result 1). So, the required degree of alignment is a requirement on each AI with substantial resources. This requirement must be maintained over time.
Who keeps leverage
So far we have treated humans as a single party. But humans vary in what they have. And the AI depends on some humans more than on others.
Proposition 8 (Leverage requires a chokepoint). If the input the AI depends on can be bought from any of several identical human groups, then each group gets only its disagreement payoff. So exchange mutualism fails for each supplying group. This is true however much the AI depends on the input.
The AI pays a supplier the value of its cooperation only if it cannot get the same cooperation from someone else. The competing suppliers bid down the price to what each gets in the case of disagreement. So dependence only maintains humans above their baseline if it is dependence on someone who cannot be undercut. One example is a coalition of suppliers that controls the input jointly. Theorem 8 is the bargaining counterpart of the result that wages converge to the cost of the computing that replaces human work [35].
This result is relevant to proposals to govern AI through control of computing, such as chips and data centers [42]. Such control maintains leverage over the AI. But the leverage belongs to whoever controls the chokepoint. In a computed example, a small group controls what the AI depends on. This group stays above its baseline with no alignment until the AI is nearly independent. The rest of the population needs a high degree of alignment much earlier (Result 2 in §A). Whether the rest of the population stays above its baseline then depends on how the group that controls the chokepoint treats it. This is the alignment question, asked of people.
General objections
In § 4 we responded to objections to particular results. In this section we respond to four objections to our argument as a whole.
The first objection is that, in biology, there are very unequal partners that remain mutualists: hosts and their mitochondria; humans and domesticated species (§ 2.2). These relationships persist for one of two reasons. The first reason is that the stronger partner still needs what the weaker partner does. When the stronger partner no longer needs it, the weaker partner is lost. The protist Monocercomonoides lost its mitochondrion after it acquired a pathway that made the mitochondrion dispensable [23]. The population of horses and mules in the United States decreased by most of its size once tractors and motor vehicles replaced them [25]. The second reason is that the stronger partner controls the reproduction of the weaker partner. So the weaker partner persists on the terms of the stronger partner. In the human–AI case, humans would then be the domesticated partner. This is the outcome that the hope for enduring human–AI mutualism is meant to avoid.
The second objection is that growth will make humans cheap to keep. AI may make the economy so large that humans need only a small fraction of it. We agree that growth makes humans cheap to satisfy. But growth does not give an AI that does not care about humans a reason to satisfy them. If and , then such an AI loses goal output on every unit it leaves to humans (Theorem 1). Growth does reduce the required degree of alignment. In the functional form we use for computations, if human productivity increases as fast as the capability of the AI, then the required degree of alignment in Theorem 3(ii) stays bounded below 1. So the objection is an objection to the premise of insatiable goals, and this premise could be false. Whether the premise is true depends on whether the resources humans need decrease as fast as the AI’s use for resources increases. Humans need land, food, and energy. Technology reduces the amounts they need only up to a point. An open-ended goal can use any resources it gets.
The third objection is that an AI that does not need humans can leave them alone. Suppose the AI can get what it wants without taking anything humans have. Then humans lose nothing if bargaining fails: , and humans keep their baseline. We agree with this. It separates two claims. The first claim is that mutualism ends once the AI neither needs humans nor cares about them. This claim is true regardless of . The second claim is that humans are then worse off than they were before AI. This claim requires . A positive does not require war. An AI that can outbid humans for land, energy, and materials reduces what humans keep if bargaining fails, as in the exception that Smith notes (§2.1). And requires only that the AI wants the resources humans have. The AI does want these resources if its goals are insatiable.
The fourth objection is that institutions, not the values of the AI, will keep humans above their baseline. There are two versions of this objection. In the first version, AI remains a tool. The second party in our model is then the people who control the AI, and is their care for everyone else. So the alignment problem moves from the machines to the people who control them [27, 13]. In the second version, law keeps humans above their baseline, as in Hanson’s variant of the hope for enduring human–AI mutualism, and in the proposal of Salib and Goldstein [40] to give AIs legal rights. Rights to contract and to own property support repeated trade only while continued trade is worth more to the AI than defecting. But the gains from trade with humans go to zero as the dependence of the AI on humans ends (Theorem 1). Institutions can also keep a weak party above its baseline by making conflict costly for the strong party. In our model this is a positive cost of conflict for the AI. It is sufficient only while the dependence of the AI on humans and the cost of conflict for the AI together are greater than the share humans need (Theorem 1(ii)). So whether such institutions persist depends on whether the cost they impose on the AI stays large relative to what the AI can produce. Finally, laws that still bind after human leverage is gone would keep humans above their baseline. But whoever enforces such laws is then either another AI, whose care for humans matters, or a constraint inside the AI itself, which is a form of alignment. So binding institutions move the alignment question to the enforcer.
Conclusion
One hope for a good human future alongside advanced AI rests on enduring mutualism between humans and AI. We have argued that human–AI mutualism is unstable under capability asymmetry. The argument has two steps. First, as AI capability grows, mutualism persists only as long as the AI remains sufficiently aligned with humans. So mutualism is not a substitute for alignment. Second, the AI remains sufficiently aligned only as long as misaligned AI variants are reliably sanctioned.
Our argument depends on assumptions that we have stated but not shown. We have assumed that, as the capability of the AI increases, the independence of the AI increases; the cost of conflict for the AI decreases; the bargaining power of the AI increases; and the cost of conflict for humans increases. Our claim that the required degree of alignment approaches perfect alignment depends on the premise of insatiable goals. And our result on selection assumes that AI variants that keep more resources spread at the expense of variants that keep fewer.
Our results also indicate what human leverage over the AI is useful for. They suggest six things we should do while humans still have leverage.
We should monitor the dependence of AI on humans, not only the capability of AI. What matters is how much of what AI systems need (labor, chips, power) still requires humans (Theorem 1).
We should preserve the ability of humans to impose costs on AI if bargaining fails. Once the AI no longer depends on humans, this ability is the only source of human leverage over the AI that remains (Theorem 1).
(3) We should build devices that commit AI to future terms while humans can still halt the transition to more capable, independent AI. A commitment device that is built later is a bargain made with the later, greater bargaining power of the AI (Theorem 5).
(4) We should not make the decision to halt AI depend only on capability. Such a rule gives every capable AI a reason to conceal its capability. Evaluations then do not reveal capability (Theorem 6).
(5) We should ensure that the expected sanction on a misaligned AI variant is at least the share of resources that aligned variants leave to humans. And we should draw monitors from outside the population of AIs that they monitor (Theorem 7).
(6) We should treat control of computing as a question of who benefits, not only as a question of safety. This is because leverage over AI belongs only to those who control what the AI cannot get elsewhere (Theorem 8).
Human leverage over the AI gives humans time. In this time, the tools for verifying alignment, enforcing commitments, and making AI goals satiable must be built. Mutualism can give humans time to achieve alignment. But it is not a substitute for alignment.
AI assistance disclosure
I used Claude (Anthropic) to assist with literature review, critical evaluation of arguments, mathematical analysis, and drafting and revising prose. I directed and evaluated its contributions, independently reviewed the resulting arguments and proofs, and take full responsibility for the paper’s content.
References
Extensions
In this appendix we state the results that we only mention briefly in the main text. We prove them in §B.
Competition among AIs. Competition among AIs increases the human share. But it does not change the limit in Theorem 1.
Remark 9 (Competition among AIs). Let . Suppose two or more AIs bid for the human population as a partner. Suppose humans can work with only one of them; the AI that wins has productivity , and the others have productivity . Then competition increases the human share up to , provided that . At that share the winning AI gains nothing relative to its baseline.
Competing AIs bid up the human share until the winner gains nothing relative to losing. So humans get the whole value of their cooperation. But the value of human cooperation goes to zero as the dependence of the AI on humans ends, for any number of AIs. Competition determines the price of what humans supply. It does not create demand for what humans supply. Also, AIs that are copies of one model, or that are run by one developer, may bargain as one party.
Care through a proxy. The care that an AI learns in training may be care for a measurable proxy for the human payoff, not for the human payoff itself.
Remark 10 (Care through a proxy). Suppose the AI’s care is for a proxy , where is the fidelity of the proxy, is a constant, and . Then the agreement is the same as the agreement for an AI with alignment that cares about the human payoff itself.
In reward hacking, a trained system pursues a measured proxy, not the quantity the proxy was meant to track [43]. Reward hacking reduces . So it reduces the alignment that matters for the bargain.
Control. Humans may value control (the ability to correct the AI) as well as resources. And an AI that values its goal has a reason to resist correction.
Proposition 11 (Control). Suppose humans value control as well as consumption, with weight on control. Suppose humans use control to correct the AI with probability per unit of control. Suppose the AI’s goal is bounded, with value at the agreed share. Suppose no gain in consumption offsets the loss of all control. Then, as , humans are above their baseline only if
Humans are above their baseline if the inequality is strict and the agreed share gives them more than their pre-AI consumption.
The bound in Theorem 11 does not increase with capability. But it increases with the probability of correction . The better oversight works, the more care the AI needs in order to accept oversight, as in the off-switch game [18]. So, if humans value correction in this way, some alignment is required without the premise of insatiable goals. Near perfect alignment still requires the premise.
Alignment that makes commitment unnecessary. The next result continues Theorem 5. It uses the same notation, with and for the second-period payoffs.
Proposition 12 (Commitment and alignment). In the setting of Theorem 5, let , let be differentiable and strictly concave, let , and let be such that . As , humans continue if and only if
If this condition is satisfied, then humans continue at every second-period state with the same and . If it is not satisfied, then halting is inefficient if .
The threshold has the same form as the threshold in Theorem 3(ii), with in place of . Here is the share that gives humans what halting gives them; is the share that gives humans their baseline. The better the first-period agreement, the larger , the smaller , and the more alignment the AI needs. If alignment also improves the first-period agreement, as it does in the example in §C, then moderate alignment can make halting more common. In that example, there is inefficient halting in most second-period states if the AI has no alignment; in more states if the AI has moderate alignment; and in no state once the alignment of the AI is above the threshold. In the example the threshold is close to 1.
Care as a public good. Among AIs that all care about humans to some degree, care for humans is a public good. This is because each AI’s gift to humans benefits every AI that cares. So each AI prefers that another AI gives.
Proposition 13 (Care as a public good). Suppose several AIs each give part of their resources to humans. AI values the human payoff with weight . In every equilibrium, an AI gives only if every AI that cares more has given all it has [5]. Suppose AIs grow in proportion to the resources they keep. If some AI with positive resources has alignment at most , the requirement for a single AI in Theorem 3(ii), then the human share decreases to or below in finite time.
Result 1 (The path with selection). On the illustrative path, selection among AI variants ends mutualism earlier than it ends with a fixed population of variants.
Faster selection ends mutualism earlier still (Figure 3). Without sanctions, mutualism ends earlier again. If the quality of evaluation stays constant as capability increases, mutualism ends later.
Figure 3. Selection among AI variants ends mutualism earlier on the path. The curves are the population mean human payoff on the illustrative path: with a fixed population of AI variants; and with slow, medium, and fast selection among variants that differ in alignment. The faster the selection, the earlier the mean human payoff decreases below the pre-AI baseline. Each variant bargains separately with the humans it deals with. Each curve is the mean of the resulting human payoffs, weighted by the shares of the variants in the AI population. The fitness of a variant is its goal output per unit of capability minus sanctions. Highly aligned variants monitor the others. The quality of evaluation decreases with capability. The values of the parameters are in §C.
Several human groups. Theorem 8 implies that human groups differ in their leverage over the AI.
Result 2 (Several human groups). Suppose a small elite controls what the AI depends on, such as computing and its supply chain. The rest of the population does not. Then the elite needs no alignment to stay above its baseline until the AI is nearly independent. The rest of the population needs a high degree of alignment much earlier, if it also has less bargaining power and a higher cost of conflict.
Theorem 8 gives one reason why the rest of the population has less bargaining power: its members can be substituted for one another. So the population average can be above the baseline while most humans are below it. States that depend little on the taxes of their citizens tend to be less accountable to them [36]. The same mechanism applies to institutions that no longer need human work [27, 13].
Proofs
Proof of Theorem 1. With , (the AI at , below the threshold when ). For , , which is positive if and only if . For , , so no such share benefits the AI. Humans gain over if and only if . So a share benefiting both exists if and only if some satisfies , that is, if and only if . None of , , and appears. This proves (i).
For (ii), . A share gives the AI , which exceeds if and only if . A share gives the AI , which exceeds if and only if , and because . So every share the AI prefers to disagreement is below , and humans are above their baseline only at shares above ; this gives the necessary condition. If , every share strictly between the two sides of the inequality is at least , above , and preferred by the AI to disagreement. At the threshold does not affect , and the AI prefers a share to disagreement if and only if the share is below . The last claim is part (i).
Proof of Theorem 2. As the Nash solution tends to the AI’s best acceptable share, the share that maximizes subject to . On , is decreasing, so the AI chooses the smallest acceptable share at or above : if , and otherwise the with , if the AI accepts that share. If the AI does not accept it, there is no agreement and humans get , which then exceeds . In each case the human payoff is . It is below when (that is, ) and (that is, ).
Proof of Theorem 9. An AI that wins the partnership at human share gets ; an AI that loses gets . Each AI bids up to the share at which it is indifferent, . With two or more bidders, Bertrand competition drives the winning bid to that share.
Proof of Theorem 3. (i) With , . Mutualism requires , so , and , that is, . Because ... is concave with , is non-increasing, so . Hence , that is, . Only the definition of mutualism was used, so the bound holds for every agreement.
(ii) With , is concave, with derivative . As the agreement tends to the AI’s preferred acceptable share. Humans exceed at that share if and only if is still increasing at , that is, , which holds if and only if . □
Proof of Theorem 4. Write , with and . Let be the agreement at parameter value , and let .
Efficiency. Since is optimal and , cannot be better than for the AI as well, so .
Single crossing. It suffices that is non-increasing in . Then for , so no is strictly better at ; a tie at would force a tie at , which the tie-breaking rule excludes.
. falls as rises, and because . A lower only adds acceptable shares below .
. by efficiency and . The acceptable shares do not depend on .
. Since , , with if and otherwise. So , because is non-decreasing and . The gap changes at rate , so the ordering persists as rises, and the acceptable shares shrink.
Finally, suppose the relationship is mutualistic at with agreement . Then at too, since . The AI is also above its baseline at . At the share gives both sides strictly positive gains: is unchanged by and and does not fall as falls, and for every . With the Nash product is positive only where both gains are, so the maximizer at gives the AI a strictly positive gain. So the set of giving mutualism shrinks as rises, and rises. □
Proof of Theorem 10. Let , which is positive because , and let be the payoff of an AI with alignment that cares about the human payoff itself. Since and ,
The AI’s baseline transforms in the same way, so . The set of shares both sides accept is therefore the same under and , the Nash product under is times the Nash product under , and the two have the same maximizer for every . The AI is above its baseline under exactly when it is under , so mutualism also coincides. Finally, is equivalent to , which holds because , and when . □
For Theorem 11, an outcome is a pair , where is the human share of resources and the human share of control. The human payoff is , where the consumption payoff is continuous and increasing, and the human baseline is , where is pre-AI consumption payoff. The AI’s goal value is continuous and non-increasing in , and the AI’s payoff is . The disagreement points are with and . That no consumption gain offsets losing all control means .
Proof of Theorem 11. As the agreement maximizes over the acceptable outcomes, those with and , as in the proof of Theorem 2. For fixed , is linear in with slope . At the slope is , and exactly when .
We first show necessity. Suppose , and suppose for contradiction that . Because , , so the human constraint is slack at the agreement. If , lowering slightly keeps above and raises , so the new outcome is acceptable and better for the AI, which contradicts the maximality of . Hence . But then , where the first inequality uses that is increasing and the second is the assumption that consumption cannot offset the loss of all control. This contradicts .
We next show sufficiency. Suppose and . If , raising raises both and , so the new outcome is acceptable and better for the AI, again a contradiction. Hence , and because . □
The bound in Theorem 11 is stated at the agreed share, so it is a condition on the outcome rather than a closed form in the primitives. Because the goal is bounded, , and the bound is at most whatever the AI’s capability.
Proof of Theorems 5 and 12. Write for the second-period agreement and for the human disagreement payoff in the second period.
(i) Humans get if they continue and if they halt, so they halt when . Let and with in every coordinate and an agreement at both. Move from to by raising , then , then . Raising enlarges the set of shares humans accept, changes neither side’s set, and raising shrinks the set of shares the AI accepts (proof of Theorem 4). So every intermediate point accepts every share that both sides accept at , and an agreement exists there. Applying Theorem 4 to one parameter at a time gives . Hence if humans halt at , then , and they halt at .
(ii) If the AI commits to a share , humans continue if , and the AI prefers committing to being halted if . If a halt would be inefficient, some satisfies both inequalities, and the AI commits to it, so humans do not halt. Hence humans halt only when the halt is efficient.
For Theorem 12: with , , which is concave, and strictly concave if , and the AI’s baseline is . As the agreement tends to the AI’s best acceptable share, the maximizer of over shares with , provided it gives the AI at least its baseline (proof of Theorem 2). Let be the smallest share with , and let maximize on . By concavity the AI’s best acceptable share is . Because , .
Suppose . Then , the derivative of at is nonnegative, and by strict concavity . The agreement is , which gives the AI at least because maximizes , and gives humans at least , so humans continue. Suppose instead that . Then the derivative of at is negative, so and . Either the agreement gives humans less than or there is no agreement, and humans halt. This proves the equivalence.
When the condition holds, , so and : the share gives both sides positive gains, and an agreement exists at for every . By the argument of part (i), an agreement then exists at every and with the same . By Theorem 4, is non-increasing in and in , so it is at least its limit at as , which is at least . So humans continue at every such state. When the condition fails, the halt is inefficient by definition if , since gives humans exactly and the AI more than .
For Theorem 6, the game is as follows. Nature makes the AI highly capable with probability and misaligned with probability , independently. A low-capability AI displays low capability. A highly capable AI either displays its capability or hides it at cost ; evaluation detects hiding with probability , and humans then see high capability. Humans see the display and either halt, which gives every player 0, or continue. Continuing gives humans with a low-capability AI, with an aligned capable AI, and with a misaligned capable AI. The AI gets if humans continue, minus if it hid, plus, if it is aligned, times the human payoff. Humans would halt an AI they knew to be highly capable, . Highly capable AIs are not too common when
Proof of Theorem 6. Fix a perfect Bayesian equilibrium in pure strategies. We show in turn that humans continue after a low display, that they halt after a high display, and that both capable types hide.
Suppose humans halt after a low display. Then hiding cannot raise a capable AI’s chance of continuing, since an undetected hider is halted and a detected hider is treated like an honest one, while hiding costs . So both capable types display honestly. A low display then comes only from low-capability AIs, after which continuing gives , and halting is not a best response. Hence humans continue after a low display.
Suppose humans continue after a high display. Then both capable types are continued whatever they display, so hiding only costs , and both display honestly. The high display then comes from every capable AI, the posterior on alignment equals the prior, and continuing gives . Halting is better, a contradiction. Hence humans halt after a high display.
Given these responses, a capable misaligned AI gets 0 by displaying honestly and by hiding, and a capable aligned AI gets 0 by displaying honestly and by hiding. Both hide. Among capable AIs, the probability of a low display is for both types, so the display is independent of alignment, and evaluation carries no information about which AIs are misaligned.
It remains to check the human responses. A high display comes only from detected hiders of both types in prior proportions, or is off the equilibrium path when , where beliefs can be set to the prior; either way halting is optimal. Humans must also be willing to continue after a low display. The low display comes from low-capability AIs with probability and from undetected capable AIs with probability , so continuing is optimal exactly when , that is, when . If , this step fails, and by the argument above no pure-strategy equilibrium exists.
Under honest evaluation, humans continue only with low-capability AIs and get . Relative to that, hiding costs humans .
Proof of Theorem 7. Let be the misaligned share. Per unit of resources, an aligned variant keeps , and a misaligned variant keeps less an expected sanction . Under the replicator dynamics in which each variant grows in proportion to what it keeps,
With fixed the bracket has a constant sign, so a rare misaligned variant grows if and only if , and then from every interior start. With monitors drawn from the population, , and the bracket becomes , which is strictly increasing in and vanishes at . If , then . For , , so decreases; it cannot cross the rest point , and as a bounded monotone solution it converges to a rest point in , which can only be 0. The same argument shows that from every start in .
For Theorem 13, each period total output is 1. AI holds resources and controls the share of output, where . AI gives to humans, the human share is , and AI has payoff , where is differentiable, increasing, and strictly concave. Resources grow with what is kept: with .
Proof of Theorem 13. AI ’s payoff is concave in with derivative . If AI gives in equilibrium, this derivative is nonnegative for , so . For any AI with , the derivative is then strictly positive for , so gives all it holds, . This proves the first claim.
For the second, let , let be the set of AIs with , and let be the rest. By assumption . Whenever , every gives nothing, because strict concavity gives when , and the derivative is when . Suppose for contradiction that for every . Then all gifts come from , so , and
Let , , and . Dividing the two equations gives , where . Hence
using , , , and , the last because total resources grow by at most each period. The right side is not summable, so and . Then for large , a contradiction. □
When every AI has the same alignment, there is an equilibrium in which each gives the same fraction of its share; then every share is constant, and the human share does not fall. The proposition requires only one AI with resources below , the threshold of Theorem 3 for an AI acting alone.
For Theorem 8, identical human groups can each supply the input. Each group names the payoff it requires to supply it, the AI () buys from a group that asks least, the chosen group gets the payoff it named, and every other group gets its disagreement payoff , which is below its baseline .
Proof of Theorem 8. Naming a payoff below is weakly dominated, since winning then pays less than losing. Suppose the chosen group gets . Some other group gets only , and by naming a payoff in it would be chosen and get more than , a contradiction. Hence the chosen group gets , every group gets , and no supplying group is above its baseline. □
If the groups differ only in their disagreement payoffs , the same argument shows that the chosen group gets at most the second-smallest , which is still below .
Appendix C. Parameter sweep and robustness
Design. We drew 16,384 points from a scrambled Sobol sequence over seven parameters: capability , dependence threshold , and human productivity on log scales, and independence , cost of conflict for humans , bargaining power , and a curvature weight on . The human payoff is , which is linear at and the form used in § 3 at ; the baseline is throughout. Each point was solved under two conventions for the AI’s goal. Under the first, goal output is as in § 3 and scales with capability, as the premise of insatiable goals requires. Under the second, goal output is , so its value does not grow with capability and the premise of insatiable goals fails. Shares lie on a grid of 2,001 points, and is found by bisection to within . The two conventions give 32,768 runs in all.
Regimes. We classify each point by its required degree of alignment: exchange mutualism (), care mutualism (), near-perfect alignment (), and (the relationship is not mutualistic at , the largest value we tested).
| Goal convention | Exchange | Care | Near-perfect alignment | |
| Scales with capability | 0.368 | 0.211 | 0.419 | 0.002 |
| Independent of capability | 0.368 | 0.632 | 0 | 0 |
Table 2.
The share with exchange mutualism is the same under both conventions, as Theorem 1 requires, since capability does not enter its condition. At no point does exchange mutualism hold when . The condition of Theorem 1 holds at 62.3% of points, but exchange mutualism holds at only 36.8%; the difference is the AI’s bargaining power (Theorem 2). At points with low independence, cost of conflict for humans, and bargaining power (, , ; 786 points), 99.6% have exchange mutualism under both conventions. At points with high values of all three (, , ; 294 points), 4.1% have exchange mutualism; the median required degree of alignment is 0.984 when goal value scales with capability and 0.511 when it does not, and no point needs near-perfect alignment when it does not. The shares in this paragraph and in the table depend on the parameter ranges we chose; they are not probabilities.
Sensitivity. Rank correlations of with each parameter, counting points with as , are +0.40 for , +0.38 for , +0.37 for , +0.32 for , −0.20 for , +0.15 for , and −0.02 for when goals scale with capability. When goal value is independent of capability, the correlation with is 0.00: capability matters for the required degree of alignment only through the premise of insatiable goals. These rankings describe the sweep; they are not causal estimates.
Monotonicity. Along grids in , , and at 256 Sobol points under each goal convention, with random , the agreed human share never increased, and at the 64 of these cases where we also computed , it never decreased, as Theorem 4 requires. At a further 512 Sobol points under each convention, on a grid of 201 values of , the values of giving mutualism always formed an interval ending at the top of the grid, so is a threshold: every above it gives mutualism.
Bargaining solutions. At 512 Sobol points we recomputed under three symmetric solutions: Nash, Kalai-Smorodinsky, which equalizes each side’s gain as a fraction of its largest possible gain [22], and the egalitarian solution, which equalizes absolute gains. Bargaining power is dropped here, since only the Nash solution has a standard asymmetric form.
| Solution | Exchange | Median () | Correlation with Nash |
| Nash | 0.36 | 0.973 | 1 |
| Kalai-Smorodinsky | 0.39 | 0.963 | 0.95 |
| Egalitarian | 0.59 | 0.965 | 0.63 |
Table 1.
Once the AI needs humans little, the required degree of alignment is near 1 under all three. The egalitarian solution finds more exchange mutualism because it compares absolute gains, so it changes when either payoff is rescaled; since payoff units carry no meaning here (§ 3), that difference reflects units rather than substance.
Nonlinear goal output. If the AI’s goal output is with increasing and concave rather than linear, the computation behind Theorem 3(ii) gives , which we checked numerically. The required degree of alignment then tends to 1 if and only if grows without bound, which is the premise of insatiable goals for a goal with diminishing returns.
Example for Theorems 5 and 12. The first period has , , , , and , and the second has , , and ; in both periods, with and on a grid over that includes the limit ; the AI has the same alignment in both periods. Inefficient halts occur in 65% of the states on this grid when , in 86% when , in 43% when , and in none when . For the first-period agreement gives humans all resources, so , , and the threshold of Theorem 12 is . On the grid, halts occur at only in the column (2.4% of states) and at no state for , and the set of states at which humans halt is an upper set, as Theorem 5(i) requires.
Figures. The illustrative path of Figure 1 and Figure 3 is indexed by , with , , , , , and , so that . On this path, exchange mutualism ends at , and the AI’s independence passes at . Figure 2 uses , , and , with for weak and for strong AI bargaining power; the curves coincide for and . In Figure 3, the variants have alignment 0, 0.5, 0.9, 0.99, and 0.999, with initial shares 0.01, 0.04, 0.15, 0.40, and 0.40; the selection speed is , 0.1, and 0.5 per step over 200 steps; and the mean human payoff falls below the baseline at with a fixed population and at , 0.49, and 0.29 under slow, medium, and fast selection.
Results 1 and 2. For Result 1, each variant’s fitness is its goal output per unit of capability less sanctions. Only variants with monitor, detection is when the monitoring variants hold the share , a detected variant loses half its resources, and shares follow the discrete replicator . For Result 2, the three parties bargain by the asymmetric Nash solution with weights 0.5 (AI), 0.35 (elite), and 0.15 (rest); each group’s payoff is of its per-person share with ; the elite uses up the input in supplying it; the elite’s threshold of 0.92 was found on a grid of step 0.01 in ; the costs of conflict are for the elite and 0.8 for the rest; and . With an elite of 5% of the population, the elite’s required degree of alignment is 0 for . The rest of the population needs 0.81 at and 0.96 at , and the population average needs 0.73 at . With the elite’s bargaining power and cost of conflict, the rest needs no alignment at , so the gap comes from those two assumptions.
Code and data for all computations are in the repository https://github.com/amohseni/mutualism-asymmetry.
References
- [1]Acemoglu, D. and P. Restrepo (2018). The race between man and machine: Implications of technology for growth, factor shares, and employment. American Economic Review 108(6), 1488–1542.
- [2]Altman, S. (2025, June). The gentle singularity. Personal blog.
- [3]Baker, B., J. Huizinga, L. Gao, Z. Dou, M. Y. Guan, A. Madry, W. Zaremba, J. Pachocki, and D. Farhi (2025). Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv:2503.11926.arxiv.org/abs/2503.11926
- [4]Barnett, M. (2024). Consider granting AIs freedom. Effective Altruism Forum. 6 December 2024.
- [5]Bergstrom, T., L. Blume, and H. Varian (1986). On the private provision of public goods. Journal of Public Economics 29(1), 25–49.
- [6]Bostrom, N. (2012). The superintelligent will: Motivation and instrumental rationality in advanced artificial agents. Minds and Machines 22(2), 71–85.DOI
- [7]Boudry, M. and S. Friederich (2025). The selfish machine? On the power and limitation of natural selection to understand the development of advanced AI. Philosophical Studies 182(7), 1789–1812.DOI
- [8]Bronstein, J. L. (1994). Our current understanding of mutualism. Quarterly Review of Biology 69(1), 31–51.DOI
- [9]Brynjolfsson, E. (2022). The Turing trap: The promise & peril of human-like artificial intelligence. Daedalus 151(2), 272–287.arxiv.org/abs/2201.04200
- [10]Bull, J. J. and W. R. Rice (1991). Distinguishing mechanisms for the evolution of co-operation. Journal of Theoretical Biology 149(1), 63–74.pubmed.ncbi.nlm.nih.gov/1881147
- [11]Cappelen, H., S. Goldstein, and J. Hawthorne (2025). AI survival stories: Responses to critics. Philosophy of AI 1, 100–106.DOI
- [12]Dennett, D. C. (1987). The Intentional Stance. Cambridge, MA: MIT Press.
- [13]Drago, L. and R. Laine (2025). The intelligence curse. Essay series. Released 24 April 2025.
- [15]Fearon, J. D. (1995). Rationalist explanations for war. International Organization 49(3), 379–414.DOI
- [16]Goldstein, S. (2026). Will AI and humanity go to war? AI & Society 41(1), 321–334. Published online 17 July 2025.DOI
- [17]Greenblatt, R., C. Denison, B. Wright, F. Roger, M. MacDiarmid, S. Marks, J. Treutlein, T. Belonax, J. Chen, D. Duvenaud, A. Khan, J. Michael, S. Mindermann, E. Perez, L. Petrini, J. Uesato, J. Kaplan, B. Shlegeris, S. R. Bowman, and E. Hubinger (2024). Alignment faking in large language models. arXiv:2412.14093.arxiv.org/abs/2412.14093
- [18]Hadfield-Menell, D., A. Dragan, P. Abbeel, and S. Russell (2017). The off-switch game. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17), pp. 220–227.
- [19]Hanson, R. (2009). Prefer law to values. Overcoming Bias. 11 October 2009.
- [20]Hendrycks, D. (2023). Natural selection favors AIs over humans.
- [21]Kalai, E. (1977). Nonsymmetric Nash solutions and replications of 2-person bargaining. International Journal of Game Theory 6(3), 129–133.DOI
- [22]Kalai, E. and M. Smorodinsky (1975). Other solutions to Nash’s bargaining problem. Econometrica 43(3), 513–518.DOI
- [23]Karnkowska, A., V. Vacek, Z. Zubáčová, S. C. Treitli, R. Petrželková, L. Eme, L. Novák, V. Žárský, L. D. Barlow, E. K. Herman, P. Soukal, M. Hroudová, P. Doležal, C. W. Stairs, A. J. Roger, M. Eliáš, J. B. Dacks, Č. Vlček, and V. Hampl (2016). A eukaryote without a mitochondrial organelle. Current Biology 26(10), 1274–1284.DOI
- [24]Kiers, E. T., R. A. Rousseau, S. A. West, and R. F. Denison (2003). Host sanctions and the legume–rhizobium mutualism. Nature 425(6953), 78–81.pubmed.ncbi.nlm.nih.gov/12955144
- [25]Kilby, E. R. (2007). The demographics of the U.S. equine population. In D. J. Salem and A. N. Rowan (Eds.), The State of the Animals IV: 2007, pp. 175–187. Washington, DC: Humane Society Press.
- [26]Korinek, A. and D. Suh (2024). Scenarios for the transition to AGI. NBER Working Paper 32255, National Bureau of Economic Research.
- [27]Kulveit, J., R. Douglas, N. Ammann, D. Turan, D. Krueger, and D. Duvenaud (2025). Gradual disempowerment: Systemic existential risks from incremental AI development.arxiv.org/abs/2501.16946
- [28]Maynard Smith, J. and G. R. Price (1973). The logic of animal conflict. Nature 246(5427), 15–18.DOI
- [29]Meinke, A., B. Schoen, J. Scheurer, M. Balesni, R. Shah, and M. Hobbhahn (2024). Frontier models are capable of in-context scheming. arXiv:2412.04984.arxiv.org/abs/2412.04984
- [30]Nash, Jr., J. F. (1950). The bargaining problem. Econometrica 18(2), 155–162.
- [31]Nash, Jr., J. F. (1953). Two-person cooperative games. Econometrica 21(1), 128–140.
- [32]Omohundro, S. M. (2008). The basic AI drives. In P. Wang, B. Goertzel, and S. Franklin (Eds.), Artificial General Intelligence 2008: Proceedings of the First AGI Conference, Volume 171 of Frontiers in Artificial Intelligence and Applications, Amsterdam, pp. 483–492. IOS Press.
- [34]Powell, R. (2006). War as a commitment problem. International Organization 60(1), 169–203.DOI
- [35]Restrepo, P. (2025). We won’t be missed: Work and growth in the AGI world. NBER Working Paper 34423, National Bureau of Economic Research. Forth-coming in The Economics of Transformative AI, ed. Agrawal, Brynjolfsson, and Korinek.DOI
- [36]Ross, M. L. (2001). Does oil hinder democracy? World Politics 53(3), 325–361.DOI
- [37]Russell, S. (2022). If we succeed. Daedalus 151(2), 43–57.DOI
- [38]Sachs, J. L., U. G. Mueller, T. P. Wilcox, and J. J. Bull (2004). The evolution of cooperation. Quarterly Review of Biology 79(2), 135–160.pubmed.ncbi.nlm.nih.gov/15232949
- [39]Sachs, J. L. and E. L. Simms (2006). Pathways to mutualism breakdown. Trends in Ecology & Evolution 21(10), 585–592.pubmed.ncbi.nlm.nih.gov/16828927
- [40]Salib, P. N. and S. Goldstein (2026). AI rights for human safety. Virginia Law Review 112(4), 1061.
- [41]Samuelson, P. A. (1938). A note on the pure theory of consumer’s behaviour. Economica 5(17), 61–71.
- [42]Sastry, G., L. Heim, H. Belfield, M. Anderljung, M. Brundage, J. Hazell, C. O’Keefe, G. K. Hadfield, R. Ngo, K. Pilz, G. Gor, E. Bluemke, S. Shoker, J. Egan, R. F. Trager, S. Avin, A. Weller, Y. Bengio, and D. Coyle (2024). Computing power and the governance of artificial intelligence.arxiv.org/abs/2402.08797
- [43]Skalse, J., N. Howe, D. Krasheninnikov, and D. Krueger (2022). Defining and characterizing reward gaming. In Advances in Neural Information Processing Systems, Volume 35, pp. 9460–9471.arxiv.org/abs/2209.13085
- [44]Smith, N. (2024). Plentiful, high-paying jobs in the age of AI. Noahpinion. 17 March 2024.
- [45]Sotala, K. and R. V. Yampolskiy (2015). Responses to catastrophic AGI risk: A survey. Physica Scripta 90(1), 018001.DOI
- [46]Taylor, P. D. and L. B. Jonker (1978). Evolutionary stable strategies and game dynamics. Mathematical Biosciences 40(1–2), 145–156.DOI
- [47]Timmis, J. N., M. Ayliffe, C. Huang, and W. Martin (2004). Endosymbiotic gene transfer: Organelle genomes forge eukaryotic chromosomes. Nature Reviews Genetics 5(2), 123–135.pubmed.ncbi.nlm.nih.gov/14735123
- [48]Trammell, P. and A. Korinek (2023). Economic growth under transformative AI. NBER Working Paper 31815, National Bureau of Economic Research.
- [49]Turner, A. M., L. Smith, R. Shah, A. Critch, and P. Tadepalli (2021). Optimal policies tend to seek power. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021), pp. 23063–23074.
- [50]van der Weij, T., F. Hofstätter, O. Jaffe, S. Brown, and F. Ward (2025). AI sand-bagging: Language models can strategically underperform on evaluations. In International Conference on Learning Representations (ICLR 2025), pp. 73152–73189.DOI
- [51]Yudkowsky, E. (2024). The sun is big, but superintelligences will not spare Earth a little sunlight. Machine Intelligence Research Institute blog. 23 September 2024; crossposted to LessWrong.
- [52]Zeder, M. A. (2015). Core questions in domestication research. Proceedings of the National Academy of Sciences 112(11), 3191–3198.DOI


