betaFeedback

WorksComputer Science

Human-AI Mutualism Is Unstable under Capability Asymmetry

Abstract

One hope for a good human future alongside advanced AI rests on enduring mutualism: humans and AI will need each other, so the relationship will stay good for both. We argue that this hope fails as AI capability grows. Bargaining guarantees each side only what it would get if bargaining failed. For humans, this can be less than they had before AI. In a simple game-theoretic model, humans stay as well off as before AI without alignment only while they have leverage over the AI: either the AI depends on what they supply, or they can impose costs on the AI that are large relative to what the AI can produce. Once humans have neither kind of leverage, they stay as well off as before AI only if the AI is aligned with them. If the AI's goals are insatiable, the alignment required approaches perfect alignment as AI capability grows. Thus, mutualism is not a substitute for alignment. And when AI variants that keep more resources spread faster, alignment lasts only while misaligned variants are reliably sanctioned. In these two senses, human-AI mutualism is unstable.

Introduction

One hope for a good human future alongside advanced AI rests on enduring mutualism, the term biologists use for a relationship that benefits both partners. If AI systems depend on human labor, human institutions, and human-built infrastructure, then humans and AI will need each other, both will gain from cooperation, and the relationship will stay good for both. Versions of this hope have been stated by economists, by heads of AI companies, and by legal scholars (§ 2.1).

The focus of this paper is on the instability of human–AI mutualism under capability asymmetry, the situation in which AI is far more capable than humans. We make use of two benchmarks in the discussion below: the first is the no-interaction baseline, the situation of each party in the absence of the other; the second is the disagreement point, the situation of each party should they fail to reach an agreement. Some statements of this hope are themselves about bargaining: humans are needed, so humans retain bargaining power. While mutualism is defined against the no-interaction baseline, bargaining guarantees each party only its disagreement point. When one party can impose costs on the other, the weaker party’s disagreement point can lie below its no-interaction baseline. An agreement can then be peaceful, efficient, and acceptable to both parties, and still leave the weaker party worse off than if the two had never met.

Our core argument is as follows.

(P1) Human–AI mutualism persists only as long as humans are better off than they were before AI.

(P2) Humans are better off than they were before AI only if humans have sufficient leverage over the AI or the AI is sufficiently aligned with humans.

(P3) As AI capability grows, humans cease to have sufficient leverage over the AI.

(C1) So, as AI capability grows, human–AI mutualism persists only as long as the AI remains sufficiently aligned with humans.

(P4) The AI remains sufficiently aligned with humans only as long as misaligned AI variants are reliably sanctioned.

(C2) So, as AI capability grows, human–AI mutualism persists only as long as misaligned AI variants are reliably sanctioned.

(C1) follows from (P1), (P2), and (P3), and (C2) follows from (C1) and (P4). (P1) is the definition of mutualism. In (P2) and (P3), human leverage over the AI has two sources: the AI’s dependence on what humans supply, and the cost humans can impose on the AI if bargaining fails. (P2) is a result in our model (Theorem 1). (P3) is an assumption, which we motivate in § 3 and defend in § 5. (P4) is a result in our model (Theorem 7) that holds when AI variants that keep more resources spread at the expense of variants that keep fewer. A sanction is any loss imposed on a misaligned variant, whether by monitors, by developers who retire it, or by customers who leave it.

(C1) says that mutualism is not a substitute for alignment. (C1) and (C2) together say that mutualism is unstable: mutualism based on exchange does not persist as AI capability grows, and the alignment that must take its place persists only under reliable sanctions. Further results state how much alignment is sufficient (§ 4.2) and why the transition from AI that depends on humans to AI that does not is hard to manage (§ 4.3).

In § 2 we set out the hope for enduring human–AI mutualism, the two bodies of theory we use to examine it (bargaining theory and the biology of mutualism), and prior work on humans and AI. In § 3 we define the key terms of our thesis and present our model. In § 4 we derive our results. In § 5 we respond to possible objections to our argument as a whole. In § 6 we state what we have shown, what we have assumed, and what follows for AI safety.

Background

The hope for human–AI mutualism

There are three versions of the hope that humans and AI will remain partners even when AI is far more capable than humans at every task.

The first view is that humans will stay needed. Noah Smith, the economics writer, argues that even if AI is better than humans at every task, there may still be well-paid jobs for most humans [44]. This is because the total amount of AI is limited. However, Smith notes an exception to this: if computing can be made from energy, then it could pay to starve humans in order to make more AI. Sam Altman, the chief executive of OpenAI, holds that people’s care for other people gives humans a lasting advantage over AI [2]. The economist Erik Brynjolfsson says why complementarity between humans and AI matters: “people remain indispensable for value creation and retain bargaining power in labor markets and in political decision-making” [9 273].

The second view is that AI will stay subordinate however capable it becomes. The computer scientist Yann LeCun holds that AI systems, however intelligent, will be subservient to humans, because humans set their goals (interview in Perrigo 2024). A variant of the second view puts law in place of shared values: the economist Robin Hanson argues that what matters most is a law that all parties accept, and that humans can then live on retirement income from what they own [19].

The third view is that the outcome depends on the AI’s care for humans. The computer scientist Geoffrey Hinton argues that an AI that does not care for humans as a mother cares for her baby will replace them (reported in Egan 2025). The computer scientist Stuart Russell argues that a machine whose objective is human preferences, and which is uncertain about those preferences, “will defer to human control actions” [37 52].

By the hope for enduring human–AI mutualism we mean the first two views and the variant based on law, taken as independent of the third view. It is the claim that exchange, subordination, or law keeps the relationship good for humans whatever the AI cares about. In short, it is the claim that mutualism can substitute for alignment. Our conclusion agrees with the third view and specifies how much alignment mutualism requires.

Bargaining and mutualism

We examine the hope for enduring human–AI mutualism with two bodies of theory. The first is bargaining theory, which predicts how two parties who can gain from agreement split the gains. In Nash’s solution the agreement maximizes the product of each party’s gain over its disagreement point [30], and the asymmetric solution weights the gains by bargaining power [21]. The disagreement point is strategic: it reflects what each party can do to the other, not only what each would have alone [31]. Nothing in the theory ties the disagreement point to what a party had before the parties met. One further idea from the theory of conflict enters our model: a party whose power is rising cannot credibly promise to keep today’s terms once it is stronger, so the declining party may prefer to act while it still can [15, 34].

One might wonder whether the relationship between humans and AI is well-modeled as a bargaining game. Our use of bargaining theory requires only three things.

First, it assumes that the parties, the human population and the AI population, can have distinct preferences. We do not assume that their preferences do differ. The alignment of the AI with human preferences is a parameter of our model, and if the AI is perfectly aligned with human preferences, then there is no conflict. We also make no claim about the inner states of AI systems (§ 3), but adopt the intentional stance toward the preferences of both parties. This stance is justified purely by observable behavior.

Second, it assumes that each party can affect the payoff of the other party. In our model, humans can affect the AI through the resources that they provide to the AI (labor, chips, etc.) and through the costs they can impose on the AI. The AI affects humans through what it produces and through the resources it consumes. Humans and AI do not need to engage in explicit negotiation. It is enough that what one party does affects the other party.

Third, it assumes that the division of resources between the parties depends on the disagreement point of each party. For the purpose of this analysis, disagreement does not need to be a war. It is simply the state of affairs that would result from the parties ceasing to cooperate.

The hope for enduring human–AI mutualism presupposes the second requirement above. The first view (that humans will stay needed) is that AI depends on what humans supply, so that humans retain bargaining power. The second view (that AI will stay subordinate) is that humans keep control of AI no matter how capable it becomes. Our results on what exchange mutualism can achieve (Theorem 1(i)) and on how much alignment is required (Theorem 3(i)) do not depend on the third requirement above. Our other results do. Most of our results treat the AI as if it were a single party; in § 4.4 we consider the case where the AI is a population of variants. If AI remains a tool with no goals of its own (which requires perfect alignment with at least some human group), then the second party is the people who control the AI (§ 5).

The second body of theory is the biology of mutualism. Mutualisms are “interspecific interactions that benefit both species” [8 31]. The benchmark is each species without the other: the no-interaction baseline, not a disagreement point. Partners keep mutualisms from being exploited through partner choice and sanctions [10, 38], and each mechanism needs leverage, which is the ability to withhold something the partner needs [24]. Mutualisms often break down when one partner stops needing the other [39]. The best-known lasting relationships between very unequal partners are not balanced partnerships. Mitochondria persist inside their hosts after most of their genes moved to the host’s nucleus, which put the host in control of them [47], and domestication is a mutualism in which one partner gains substantial influence over the reproduction and care of the other [52].

Prior work on humans and AI

Prior work on bargaining between humans and AI has mostly asked whether they will fight. The philosopher Simon Goldstein argues that information failures and commitment problems are unusually likely between humans and AI [16]. The legal scholar Peter Salib and Goldstein argue that giving AIs rights to contract and to hold property makes repeated, positive-sum trade between humans and AIs possible, and they set aside the most capable AIs, which “may simply not need anything from humans, nor face any risk from attempting to disempower or destroy us” [40 1089]. A second literature asks what happens when humans are no longer needed. Once all tasks are automated, wages fall to the cost of the computing that replaces human work [35, 26]. The AI researcher Jan Kulveit and coauthors argue that the implicit alignment of states, economies, and cultures with human interests was based in part on their reliance on human labor and cognition [27 2]. They argue that this implicit alignment ceases with automation. Several authors state our conclusion informally [45, 20, 51], and Barnett [4] argues the other side: an AI that no longer needs humans also has less reason to harm them.

What we add is a model in which the thesis that human–AI mutualism is unstable can be stated precisely. Prior work on bargaining between humans and AI compares peace with war. We compare the peaceful bargain with the world before AI, and show what decides whether that bargain leaves humans better or worse off.

Model

Our thesis is that human–AI mutualism is unstable under capability asymmetry. Its key terms are populations and their preferences, mutualism, capability asymmetry, and stability. This section gives an account of each term and presents our model.

Populations and preferences. The two parties are the human population and the AI population, the set of AI systems in use at a time, including copies and variants of the same model. Most results treat each population as one bargaining party; § 4.4 lets the AI population contain variants, and § 4.5 lets the human population contain groups. A population’s preferences summarize what its members reliably do, as in revealed-preference theory [41] and the intentional stance [12]. This reading makes no claim about the inner states of AI systems. It needs only that AI systems act as if they pursue goals, and that those goals can differ from the goals their developers intended. Both behaviors have been observed [43, 3, 17, 29, 50].

Payoffs. There is one unit of resources; humans get share x∈[0,1]x \in[0,1] of it, and the AI gets 1−x1-x. The human payoff is uH(x)u_H(x), which is a continuous, strictly increasing function with uH(0)=0u_H(0)=0. In computations we use

uH(x)=1−e−sx,u_H(x) = 1-e^{-sx},

where s>0s>0 is the productivity of the resources in human hands; the AI increases ss. The AI has its own goal. The goal output it gets from share 1−x1-x is

G(x)=κ(1−x)[θ+(1−θ)1{x≥xˉ}],G(x)=\kappa(1-x)\left[\theta+(1-\theta)\mathbf{1}\{x\geq\bar{x}\}\right],

where capability κ\kappa is the AI’s goal output per unit of resources. The AI is fully productive if and only if humans get at least share xˉ\bar{x}, the share that sustains human labor, institutions, and infrastructure; otherwise the AI’s productivity is only the fraction θ\theta. So θ\theta is the AI’s independence from humans: θ=0\theta=0 is full dependence; θ=1\theta=1 is full independence. The AI’s payoff is a weighted average of its goal output and the human payoff:

uA(x)=(1−α)G(x)+αuH(x),u_A(x)=(1-\alpha)G(x)+\alpha u_H(x),

where α∈[0,1]\alpha\in[0,1] is the AI’s alignment. If α=0\alpha= 0, the AI only gives resources to humans in order to pay for what they provide. If α>0\alpha> 0, the AI also gives resources to humans because it cares about the human payoff. We call this motive care; α\alpha is the degree of the AI’s care.

Mutualism. The relationship between humans and the AI is mutualistic if both parties do better than they would do without each other. Biologists define better in terms of fitness. We define better in terms of payoff. We only use fitness to define the spread of AI variants (§ 4.4). We define better relative to the no-interaction baseline for each party. The baseline for humans is bHb_H, the payoff before the AI, with all resources and productivity s=1s = 1. So bH=1−e−1b_H = 1 - e^{-1}. With the AI, s>1s > 1. Let xhx_h be such that uH(xh)=bHu_H(x_h) = b_H. So xhx_h is the share humans must get in order to be as well off as they were before the AI. The baseline for the AI is bA=uA(0)b_A = u_A(0), the AI’s payoff with all resources and no contribution from humans. This baseline is a counterfactual on production, not a counterfactual on the origin of the AI.

The baseline is not the same as the disagreement point. If bargaining fails, humans can lose some of what they had; they can lose by fighting, or by losing land, energy, materials to the AI in competition. So dH=(1−δ)bHd_H = (1-\delta)b_H. Here δ\delta is the cost of conflict for humans: the fraction of their pre-AI payoff that humans lose if bargaining with the AI fails. For the AI, dA=bA−γκd_A = b_A - \gamma\kappa. Here γ≥0\gamma\ge0 is the cost of conflict for the AI: the goal output the AI loses if bargaining with humans fails, as a fraction of the goal output the AI would get with all resources at full productivity. We set γ=0\gamma= 0 except in Theorem 1(ii). So dA=bAd_A = b_A; conflict has no cost for the AI. In Theorem 1(ii) we state what happens if γ>0\gamma> 0.

The agreement x∗x^* is the asymmetric Nash bargaining solution with the AI’s bargaining power p∈(0,1)p \in(0,1). It maximizes (uH(x)−dH)1−p(uA(x)−dA)p(u_H(x)-d_H)^{1-p}(u_A(x)-d_A)^p over the shares both sides are willing to accept. The set of feasible payoffs is not necessarily convex because GG jumps at xˉ\bar{x}. So maximizing the Nash product is a modeling choice; it is not the solution that is characterized by the Nash axioms. Our results in Theorem 1 and Theorem 3(i) do not depend on this choice. The agreement at p=1p = 1 is the limit of the agreement as p→1p \to1; it is the best share the AI is willing to accept. If there is no share such that both sides do better than their disagreement payoffs, then there is no agreement.

Definition. A relationship between humans and the AI is mutualistic if the agreement leaves both parties above their no-interaction baselines: uH(x∗)>bHu_H(x^*) > b_H and uA(x∗)>bAu_A(x^*) > b_A. A mutualistic relationship is an exchange mutualism if it would be mutualistic if α=0\alpha= 0; otherwise it is a care mutualism. The required degree of alignment is α∗\alpha^*, the infimum of the values of α\alpha for which the relationship is mutualistic.

The hope for enduring human–AI mutualism (§ 2.1) is the hope for exchange mutualism; or it is the hope for AI subordination to humans that holds regardless of what the AI cares about.

We evaluate the outcome relative to the pre-AI baseline. This is the baseline that the hope for enduring human–AI mutualism itself sets. The hope is that humans will be better off because of their relationship with the AI. Better off means better off than without the relationship. We test this hope. We do not assert that the baseline is the right standard for evaluating the relationship between humans and the AI. For example, a standard of fair division would evaluate some outcomes differently.

The value of α\alpha depends on the units in which we measure the two payoffs. So a number such as α=0.9\alpha= 0.9 only makes sense relative to a choice of units. We make qualitative claims about alignment. We make one exception: we claim that the required degree of alignment approaches perfect alignment, α=1\alpha= 1, as the capability of the AI increases. This is true for any fixed choice of units. This claim depends on the premise of insatiable goals: as the capability of the AI increases, the value the AI gets from an additional unit of resources increases without bound relative to the value humans get from the additional unit of resources at the share that restores the baseline. Part of the premise is supported by the arguments for instrumental convergence. Almost any open-ended goal is promoted by more resources [32, 6, 49]. Another assumption is that this value increases with the capability of the AI.

Capability asymmetry. Capability asymmetry is the situation in which the capability of the AI far exceeds the capability of humans. Capability does not determine whether exchange can sustain mutualism, holding other parameters fixed (Theorem 1). Capability matters through the independence θ\theta of the AI, the cost of conflict for humans δ\delta, and the bargaining power pp of the AI. Here pp stands for the advantages that are not captured in the disagreement points, such as patience, commitment, speed of action. Capability matters directly in one place. The cost of conflict for the AI, γ\gamma, is measured relative to the goal output of the AI. So the more capable the AI, the smaller the fraction of the goal output of the AI that a given cost humans can impose is (Theorem 1(ii)). The two parts of our premise (P3) are that the independence of the AI increases with capability; and that the cost of conflict for the AI decreases with capability. We further assume that the cost of conflict for humans, and the bargaining power of the AI, also increase with capability. None of these is a result of our model. Two things would end the dependence of the AI on humans: AI that can do all the work humans do [26, 35]; and robots that close the supply chain. for computing [48]. New tasks for humans might appear continuously [1]. If so, then the dependence of the AI on humans need not end. In either case our results state what happens.

Stability. A mutualism persists on a path of increasing capability if the relationship is mutualistic at every point on the path. A population of aligned AI systems is evolutionarily stable if it cannot be invaded by less aligned variants when the variants are rare [28, 46].

In Figure 1 we follow a path through the transition from AI dependent on humans to AI independent of humans, to illustrate our results. On this path the capability of the AI increases; the independence of the AI increases; the cost of conflict for humans increases; and the bargaining power of the AI increases.

Figure 1. On the illustrative path, exchange mutualism ends as the AI bargaining power increases, before the independence of the AI passes 1−xh1-x_h. From that point, the required degree of alignment increases towards 1. Top: human payoff at the agreement when the AI has no care (α=0\alpha= 0); pre-AI baseline bHb_H; human payoff if bargaining fails, dHd_H. The flat part late on the path is uH(xˉ)u_H(\bar{x}), the payment that sustains human labor and institutions. It ends when the AI no longer needs them. Bottom: the required degree of alignment α∗\alpha^*. The values of the parameters are in §C.

Results

We derive our results in five steps: what human leverage can do without alignment (§ 4.1); how much alignment the AI needs once leverage is gone (§ 4.2); problems of commitment and information in the transition (§ 4.3); selection among the AIs (§ 4.4); and which humans keep leverage (§ 4.5). We prove all results in §B. In §A we state further results; we only mention them briefly in the main text.

Human leverage limits mutualism without alignment

What keeps humans above their pre-AI baseline when the AI does not care about them?

Proposition 1 (Exchange). Suppose α=0\alpha= 0.

(i) There is a share that leaves both sides above their baselines if and only if

θ<1−max⁡(xˉ,xh).\theta< 1 - \max(\bar{x}, x_h).

This condition does not depend on capability κ\kappa, on bargaining power pp, or on the cost of conflict for the AI γ\gamma.

(ii) Suppose 0≤γ≤θ0 \le\gamma\le\theta. There is a share that the AI prefers to disagreement and that leaves humans above their baseline only if xh<1−θ+γx_h < 1-\theta+\gamma. There is a share that the AI prefers to disagreement and that leaves humans above their baseline if max⁡(xˉ,xh)<1−θ+γ\max(\bar{x}, x_h) < 1-\theta+\gamma. If θ=1\theta= 1, then there is a share that the AI prefers to disagreement and that leaves humans above their baseline if and only if xh<γx_h < \gamma. If the condition of part (i) is not satisfied, then there is no share that the AI prefers to disagreement, that leaves humans above their baseline, and that leaves the AI above its baseline.

The reason for part (i) is that the AI does not care about humans. So it only gives resources to humans in order to maintain its own productivity. The fraction of its output that the AI gains from the cooperation of humans is 1−θ1-\theta. So the AI gives humans at most that fraction of the resources. Capability scales the gain of the AI from cooperation and the outside option of the AI in the same way. So capability does not appear in the condition, and exchange mutualism does not persist on any path where the independence of the AI increases past 1−max⁡(xˉ,xh)1-\max(\bar{x}, x_h), no matter how the capability of the AI changes. What is sufficient dependence is relative to what humans need: xhx_h decreases as the AI increases the productivity of the resources in human hands. So exchange mutualism can persist at high independence if xhx_h and xˉ\bar{x} are small enough. It cannot persist at θ=1\theta= 1. Resources are rival. Humans add nothing. So every unit left to humans is a loss to the AI that does not care about them.

In part (ii) we state that humans have leverage over the AI from two sources. The first is the dependence of the AI on what humans supply, 1−θ1-\theta. The second is the cost humans can impose on the AI if bargaining fails, γ\gamma. The AI does not care about humans. So it only leaves humans above their baseline if the two sources of leverage together are greater than the share humans need, xhx_h. The first source of leverage ends when the AI no longer needs humans. The second source is measured relative to the goal output of the AI. So if the cost humans can impose does not increase as fast as the capability of the AI, then γ\gamma decreases towards zero as the capability of the AI increases. A relationship based on the second source of leverage alone is not a mutualism. The AI would do better without humans. In the rest of the paper we set γ=0\gamma=0.

Sufficient dependence makes a mutually beneficial agreement possible. Whether the agreement that is reached benefits humans depends on the bargaining power of the AI.

Proposition 2 (Bargaining power). Let α=0\alpha=0. Suppose the AI prefers to meet the threshold xˉ\bar{x} rather than produce on its own. As p→1p\to1, the human payoff goes to max⁡(dH,uH(xˉ))\max(d_H,u_H(\bar{x})). This is the limit of the human payoff at agreement if there is an agreement, and of the human payoff at disagreement otherwise. This limit is less than bHb_H if xˉ<xh\bar{x}<x_h and δ>0\delta>0.

This is because bargaining guarantees humans their disagreement payoff, not their pre-AI baseline. An AI with sufficient bargaining power will only give humans what it needs to give them to keep them doing what they are doing. This may be less than what they were getting before AI. Both propositions are shown in Figure 2.

Figure 2. Exchange can keep humans above their baseline only while the AI independence θ\theta is below 1−xh1-x_h. The bargaining power of the AI determines whether it does. The two blue curves are the human payoff at agreement when the AI has no care: one for weak AI bargaining power; one for strong AI bargaining power. The grey curve is the most the AI would accept to give: uH(1−θ)u_H(1-\theta). Past 1−xh1-x_h there is no share that the AI accepts that reaches the baseline (Theorem 1). With strong bargaining power, humans fall below the baseline even if the AI depends fully on humans (Theorem 2). The curves are the same at low and high AI capability. The values of the parameters are in §C.

Two objections to these results are not correct. The first is that humans will have a high share because of competition among AIs to partner with humans. This competition does increase the human share, but only up to 1−θ1-\theta (the limit in Theorem 1), which goes to 0 as AI dependence on humans disappears (Theorem 9 in §A). The second is that human work is valuable because of comparative advantage [44]. Comparative advantage means that humans add something to AI production. So θ<1\theta<1. But mutualism requires more. It must be that the value of what humans add is more than the share that they need to get (1−θ>xh1-\theta>x_h). And humans only get this value if they offer something that the AI cannot get from other humans (Theorem 8).

Required degree of alignment

Once the AI no longer depends on humans, it is no longer possible to maintain humans above their baseline through exchange. Any mutualism is then care mutualism. We now examine how much alignment the AI needs for mutualism to be possible, and how the required degree of alignment depends on the independence of the AI, the bargaining power of the AI, and the cost of conflict for humans.

Proposition 3 (Required degree of alignment). Let θ=1\theta=1. Suppose uHu_H is concave.

(i) For any bargaining solution and bargaining power,

α∗≥κxhκxh+bH\alpha^* \ge\frac{\kappa x_h}{\kappa x_h+b_H}

(ii) As p→1p \to1,

α∗=κκ+uH′(xh)\alpha^*=\frac{\kappa}{\kappa+u'_H(x_h)}

As κxh/bH→∞\kappa x_h/b_H \to\infty, the required degree of alignment goes to 1. As p→1p \to1, the required degree of alignment goes to 1 if and only if κ/uH′(xh)→∞\kappa/u'_H(x_h) \to\infty.

Part (i) depends only on the definition of mutualism, so it holds for any bargaining solution. To allow humans the share xhx_h, the AI sacrifices κxh\kappa x_h goal output for human payoff bHb_H. The more capable the AI, the more goal output it sacrifices (by allowing humans the share xhx_h) and the more it needs to value the human payoff in order to allow it. Part (ii) gives the exact value in the limit of full bargaining power of the AI. The ratio κ/uH′(xh)\kappa/u'_H(x_h) is the value the AI gets from an extra unit of resources relative to what humans get from the extra unit of resources at the share that returns them to the baseline. As this ratio grows without bound (this is what the premise of insatiable goals asserts), the required degree of alignment goes to 1. If the AI’s goal is satiable, the ratio is bounded and so is the required degree of alignment. So the claim that near perfect alignment is required is conditional on the premise of insatiable goals. The next result is not conditional on this premise.

Theorem 4 (Monotonicity). Let 0<p<10 < p < 1. Break ties towards the larger human share. For parameter values where there is an agreement, the agreed share x∗x^* is non-increasing in the independence of the AI θ\theta. It is non-increasing in the cost of conflict for humans δ\delta. It is non-increasing in the bargaining power of the AI pp. So the required degree of alignment α∗\alpha^* is non-decreasing in each of these.

This is because each parameter weakens the position of humans in the bargain. Independence θ\theta reduces the loss to the AI of paying humans less. The cost of conflict δ\delta reduces the payoff to humans in the case of disagreement. Bargaining power pp shifts the split towards the AI.

In §A, we refine what alignment means with two more results. One is that if the AI’s care is for a measurable proxy for the human payoff, not for the human payoff itself, then the alignment that matters for the bargain is less than the nominal alignment of the AI (Theorem 10). The other is that if humans value the ability to correct the AI, then humans are above their baseline only if the AI cares enough to accept correction (Theorem 11). The second result is relevant to the objection that humans who only care about consumption are cheap to satisfy, so an AI with a bounded goal has little reason to harm them [4]. We agree with this objection for humans who only care about consumption. For humans who also care about correction, some alignment is required even if the premise of insatiable goals is false.

Transition to independent AI

We have compared agreements at points on the path from AI that depends on humans to independent AI. Moving between points on the path introduces a problem of commitment (because the power of the AI increases) and a problem of information (because humans may not know how capable the AI has become).

Proposition 5 (Commitment). There are two periods. The AI has the same alignment α\alpha in both. After the first period, humans can halt the transition to more capable, independent AI. This maintains the agreement in the first period and its payoffs: h1h_1 for humans, a1a_1 for the AI. Otherwise, the second-period agreement is made with new values of capability κ2\kappa_2, independence θ2\theta_2, cost of conflict for humans δ2\delta_2, and bargaining power of the AI p2p_2. If some second-period share leaves both sides at least as well off as halting, and one side better off, then halting is inefficient.

(i) Humans halt when the second-period agreement gives them less than h1h_1. For the second-period states where there is an agreement, if humans halt for some values of (θ2,δ2,p2)(\theta_2,\delta_2,p_2), then they halt for all higher values.

(ii) If the AI can commit in the first period to a second-period share, then all halting is efficient.

This is an instance of the commitment problem described in § 2.2. Part (i) says that humans halt when the later independence and bargaining power of the AI are greatest. Part (ii) says that halting is a loss to both sides, which would be avoided by a credible promise. With enough alignment, the problem is also avoided. Humans continue if the AI’s alignment is above a threshold, which is of the same form as the threshold in Theorem 3(ii) (Theorem 12 in §A). A commitment device also avoids the problem, but it must bind a party that may be able to outthink the enforcement in the future. And the terms of the commitment device are a bargain in themselves (with the bargaining power of the time). So a commitment device is most likely to work if it is made when humans still have leverage over the AI.

For halting to be beneficial to humans, they must know when to halt. This depends on what evaluations of the AI show.

Proposition 6 (Sandbagging). Suppose the AI’s capability and alignment are drawn independently. Suppose humans would halt a highly capable AI if they knew it was highly capable. Suppose that highly capable AIs are not too common. If the cost to the AI of hiding its capability is less than the expected value of continuing, (1−η)V(1-\eta)V, where η\eta is the probability that evaluation detects the concealment, then in every pure-strategy perfect Bayesian equilibrium, all highly capable AIs of any alignment type hide their capability. Evaluation then does not reveal capability. Concealment is then not evidence of misalignment.

Suppose humans would halt a highly capable AI if they knew it was highly capable. Then a capable AI is better off by appearing less capable, regardless of its alignment. So evaluation does not reveal capability in the case where humans need to know the capability of the AI in order to decide whether to halt. Concealment of capability is not evidence of misalignment. An aligned AI that expects to be halted also conceals its capability if it values continuing. (Language models can already be made to deliberately underperform on capability evaluations; [50].)

Selection among AIs

So far we have treated the AI as a single party. But AI systems are copied, modified and retired. Which variants spread depends on their performance. We now consider whether selection among variants preserves the alignment that is needed for mutualism.

Proposition 7 (Invasion). Suppose AI variants grow in proportion to the resources they keep. An aligned variant leaves humans with share xax_a. A misaligned variant leaves 0. Monitors detect a misaligned variant with probability qq. They remove fraction σ\sigma of the resources of the misaligned variant. Misaligned variants invade if and only if σq<xa\sigma q < x_a. Suppose monitors are drawn from the AI population, so that q=q0(1−z)q = q_0(1-z) when the share of misaligned variants is zz. If σq0>xa\sigma q_0 > x_a, then

z†=1−xaσq0z^\dagger= 1-\frac{x_a}{\sigma q_0}

is a tipping point. Below it, misaligned variants die out. Above it, misaligned variants take over.

So, a population of aligned AIs is evolutionarily stable if and only if the expected sanction is at least the share that aligned variants leave to humans. (If monitors are drawn from the AI population, then only while misaligned variants are below the tipping point.) This is because care is costly to the AI that has care. An aligned variant gives humans resources that a misaligned variant keeps for its goal. Sanctions (in our sense) include guided selection by developers. (This is more like domestication than natural selection; [7].) But the expected sanction is smaller if detection decreases as capability increases (as Theorem 6 suggests). And it is smaller as misaligned variants spread among the monitors.

We have two more results in §A which further strengthen this conclusion. Among AIs, all of which care to some degree about humans, care is a public good. This is because each AI’s gift to humans benefits all the other AIs that care. Then, if AIs grow in proportion to the resources they keep, the human share drops to the baseline share xhx_h (or below) if any AI with resources is insufficiently aligned (Theorem 13). So, a single well aligned AI is not enough (contrary to the suggestion of [11]). And, on the illustrative path, selection among variants ends mutualism earlier than it ends with a fixed population of variants (Result 1). So, the required degree of alignment is a requirement on each AI with substantial resources. This requirement must be maintained over time.

Who keeps leverage

So far we have treated humans as a single party. But humans vary in what they have. And the AI depends on some humans more than on others.

Proposition 8 (Leverage requires a chokepoint). If the input the AI depends on can be bought from any of several identical human groups, then each group gets only its disagreement payoff. So exchange mutualism fails for each supplying group. This is true however much the AI depends on the input.

The AI pays a supplier the value of its cooperation only if it cannot get the same cooperation from someone else. The competing suppliers bid down the price to what each gets in the case of disagreement. So dependence only maintains humans above their baseline if it is dependence on someone who cannot be undercut. One example is a coalition of suppliers that controls the input jointly. Theorem 8 is the bargaining counterpart of the result that wages converge to the cost of the computing that replaces human work [35].

This result is relevant to proposals to govern AI through control of computing, such as chips and data centers [42]. Such control maintains leverage over the AI. But the leverage belongs to whoever controls the chokepoint. In a computed example, a small group controls what the AI depends on. This group stays above its baseline with no alignment until the AI is nearly independent. The rest of the population needs a high degree of alignment much earlier (Result 2 in §A). Whether the rest of the population stays above its baseline then depends on how the group that controls the chokepoint treats it. This is the alignment question, asked of people.

General objections

In § 4 we responded to objections to particular results. In this section we respond to four objections to our argument as a whole.

The first objection is that, in biology, there are very unequal partners that remain mutualists: hosts and their mitochondria; humans and domesticated species (§ 2.2). These relationships persist for one of two reasons. The first reason is that the stronger partner still needs what the weaker partner does. When the stronger partner no longer needs it, the weaker partner is lost. The protist Monocercomonoides lost its mitochondrion after it acquired a pathway that made the mitochondrion dispensable [23]. The population of horses and mules in the United States decreased by most of its size once tractors and motor vehicles replaced them [25]. The second reason is that the stronger partner controls the reproduction of the weaker partner. So the weaker partner persists on the terms of the stronger partner. In the human–AI case, humans would then be the domesticated partner. This is the outcome that the hope for enduring human–AI mutualism is meant to avoid.

The second objection is that growth will make humans cheap to keep. AI may make the economy so large that humans need only a small fraction of it. We agree that growth makes humans cheap to satisfy. But growth does not give an AI that does not care about humans a reason to satisfy them. If θ=1\theta= 1 and γ=0\gamma= 0, then such an AI loses goal output on every unit it leaves to humans (Theorem 1). Growth does reduce the required degree of alignment. In the functional form we use for computations, if human productivity increases as fast as the capability of the AI, then the required degree of alignment in Theorem 3(ii) stays bounded below 1. So the objection is an objection to the premise of insatiable goals, and this premise could be false. Whether the premise is true depends on whether the resources humans need decrease as fast as the AI’s use for resources increases. Humans need land, food, and energy. Technology reduces the amounts they need only up to a point. An open-ended goal can use any resources it gets.

The third objection is that an AI that does not need humans can leave them alone. Suppose the AI can get what it wants without taking anything humans have. Then humans lose nothing if bargaining fails: δ=0\delta= 0, and humans keep their baseline. We agree with this. It separates two claims. The first claim is that mutualism ends once the AI neither needs humans nor cares about them. This claim is true regardless of δ\delta. The second claim is that humans are then worse off than they were before AI. This claim requires δ>0\delta> 0. A positive δ\delta does not require war. An AI that can outbid humans for land, energy, and materials reduces what humans keep if bargaining fails, as in the exception that Smith notes (§2.1). And δ>0\delta> 0 requires only that the AI wants the resources humans have. The AI does want these resources if its goals are insatiable.

The fourth objection is that institutions, not the values of the AI, will keep humans above their baseline. There are two versions of this objection. In the first version, AI remains a tool. The second party in our model is then the people who control the AI, and α\alpha is their care for everyone else. So the alignment problem moves from the machines to the people who control them [27, 13]. In the second version, law keeps humans above their baseline, as in Hanson’s variant of the hope for enduring human–AI mutualism, and in the proposal of Salib and Goldstein [40] to give AIs legal rights. Rights to contract and to own property support repeated trade only while continued trade is worth more to the AI than defecting. But the gains from trade with humans go to zero as the dependence of the AI on humans ends (Theorem 1). Institutions can also keep a weak party above its baseline by making conflict costly for the strong party. In our model this is a positive cost of conflict for the AI. It is sufficient only while the dependence of the AI on humans and the cost of conflict for the AI together are greater than the share humans need (Theorem 1(ii)). So whether such institutions persist depends on whether the cost they impose on the AI stays large relative to what the AI can produce. Finally, laws that still bind after human leverage is gone would keep humans above their baseline. But whoever enforces such laws is then either another AI, whose care for humans matters, or a constraint inside the AI itself, which is a form of alignment. So binding institutions move the alignment question to the enforcer.

Conclusion

One hope for a good human future alongside advanced AI rests on enduring mutualism between humans and AI. We have argued that human–AI mutualism is unstable under capability asymmetry. The argument has two steps. First, as AI capability grows, mutualism persists only as long as the AI remains sufficiently aligned with humans. So mutualism is not a substitute for alignment. Second, the AI remains sufficiently aligned only as long as misaligned AI variants are reliably sanctioned.

Our argument depends on assumptions that we have stated but not shown. We have assumed that, as the capability of the AI increases, the independence of the AI increases; the cost of conflict for the AI decreases; the bargaining power of the AI increases; and the cost of conflict for humans increases. Our claim that the required degree of alignment approaches perfect alignment depends on the premise of insatiable goals. And our result on selection assumes that AI variants that keep more resources spread at the expense of variants that keep fewer.

Our results also indicate what human leverage over the AI is useful for. They suggest six things we should do while humans still have leverage.

  1. We should monitor the dependence of AI on humans, not only the capability of AI. What matters is how much of what AI systems need (labor, chips, power) still requires humans (Theorem 1).

  1. We should preserve the ability of humans to impose costs on AI if bargaining fails. Once the AI no longer depends on humans, this ability is the only source of human leverage over the AI that remains (Theorem 1).

(3) We should build devices that commit AI to future terms while humans can still halt the transition to more capable, independent AI. A commitment device that is built later is a bargain made with the later, greater bargaining power of the AI (Theorem 5).

(4) We should not make the decision to halt AI depend only on capability. Such a rule gives every capable AI a reason to conceal its capability. Evaluations then do not reveal capability (Theorem 6).

(5) We should ensure that the expected sanction on a misaligned AI variant is at least the share of resources that aligned variants leave to humans. And we should draw monitors from outside the population of AIs that they monitor (Theorem 7).

(6) We should treat control of computing as a question of who benefits, not only as a question of safety. This is because leverage over AI belongs only to those who control what the AI cannot get elsewhere (Theorem 8).

Human leverage over the AI gives humans time. In this time, the tools for verifying alignment, enforcing commitments, and making AI goals satiable must be built. Mutualism can give humans time to achieve alignment. But it is not a substitute for alignment.

AI assistance disclosure

I used Claude (Anthropic) to assist with literature review, critical evaluation of arguments, mathematical analysis, and drafting and revising prose. I directed and evaluated its contributions, independently reviewed the resulting arguments and proofs, and take full responsibility for the paper’s content.

References

Extensions

In this appendix we state the results that we only mention briefly in the main text. We prove them in §B.

Competition among AIs. Competition among AIs increases the human share. But it does not change the limit in Theorem 1.

Remark 9 (Competition among AIs). Let α=0\alpha= 0. Suppose two or more AIs bid for the human population as a partner. Suppose humans can work with only one of them; the AI that wins has productivity κ\kappa, and the others have productivity κθ\kappa\theta. Then competition increases the human share up to 1−θ1-\theta, provided that 1−θ≥xˉ1-\theta\ge\bar{x}. At that share the winning AI gains nothing relative to its baseline.

Competing AIs bid up the human share until the winner gains nothing relative to losing. So humans get the whole value of their cooperation. But the value of human cooperation goes to zero as the dependence of the AI on humans ends, for any number of AIs. Competition determines the price of what humans supply. It does not create demand for what humans supply. Also, AIs that are copies of one model, or that are run by one developer, may bargain as one party.

Care through a proxy. The care that an AI learns in training may be care for a measurable proxy for the human payoff, not for the human payoff itself.

Remark 10 (Care through a proxy). Suppose the AI’s care is for a proxy ψuH(x)+(1−ψ)h\psi u_H(x)+(1-\psi)h, where ψ∈[0,1]\psi\in[0,1] is the fidelity of the proxy, hh is a constant, and (α,ψ)≠(1,0)(\alpha,\psi)\ne(1,0). Then the agreement is the same as the agreement for an AI with alignment αψ/(1−α+αψ)≤α\alpha\psi/(1-\alpha+\alpha\psi)\le\alpha that cares about the human payoff itself.

In reward hacking, a trained system pursues a measured proxy, not the quantity the proxy was meant to track [43]. Reward hacking reduces ψ\psi. So it reduces the alignment that matters for the bargain.

Control. Humans may value control (the ability to correct the AI) as well as resources. And an AI that values its goal has a reason to resist correction.

Proposition 11 (Control). Suppose humans value control as well as consumption, with weight c∈(0,1)c \in(0,1) on control. Suppose humans use control to correct the AI with probability ρ\rho per unit of control. Suppose the AI’s goal is bounded, with value gˉ∈[0,1]\bar{g} \in[0,1] at the agreed share. Suppose no gain in consumption offsets the loss of all control. Then, as p→1p \to1, humans are above their baseline only if

α≥ρgˉρgˉ+c\alpha\ge\frac{\rho\bar{g}}{\rho\bar{g}+c}

Humans are above their baseline if the inequality is strict and the agreed share gives them more than their pre-AI consumption.

The bound in Theorem 11 does not increase with capability. But it increases with the probability of correction ρ\rho. The better oversight works, the more care the AI needs in order to accept oversight, as in the off-switch game [18]. So, if humans value correction in this way, some alignment is required without the premise of insatiable goals. Near perfect alignment still requires the premise.

Alignment that makes commitment unnecessary. The next result continues Theorem 5. It uses the same notation, with uH,2u_{H,2} and uA,2u_{A,2} for the second-period payoffs.

Proposition 12 (Commitment and alignment). In the setting of Theorem 5, let θ2=1\theta_2=1, let uH,2u_{H,2} be differentiable and strictly concave, let (1−δ2)bH<h1≤uH,2(1)(1-\delta_2)b_H<h_1\leq u_{H,2}(1), and let x^\hat{x} be such that uH,2(x^)=h1u_{H,2}(\hat{x})=h_1. As p2→1p_2\to1, humans continue if and only if

α≥κ2κ2+uH,2′(x^).\alpha\geq\frac{\kappa_2}{\kappa_2+u'_{H,2}(\hat{x})}.

If this condition is satisfied, then humans continue at every second-period state with the same κ2\kappa_2 and δ2\delta_2. If it is not satisfied, then halting is inefficient if uA,2(x^)>a1u_{A,2}(\hat{x})>a_1.

The threshold has the same form as the threshold in Theorem 3(ii), with x^\hat{x} in place of xhx_h. Here x^\hat{x} is the share that gives humans what halting gives them; xhx_h is the share that gives humans their baseline. The better the first-period agreement, the larger x^\hat{x}, the smaller uH,2′(x^)u'_{H,2}(\hat{x}), and the more alignment the AI needs. If alignment also improves the first-period agreement, as it does in the example in §C, then moderate alignment can make halting more common. In that example, there is inefficient halting in most second-period states if the AI has no alignment; in more states if the AI has moderate alignment; and in no state once the alignment of the AI is above the threshold. In the example the threshold is close to 1.

Care as a public good. Among AIs that all care about humans to some degree, care for humans is a public good. This is because each AI’s gift to humans benefits every AI that cares. So each AI prefers that another AI gives.

Proposition 13 (Care as a public good). Suppose several AIs each give part of their resources to humans. AI ii values the human payoff with weight αi\alpha_i. In every equilibrium, an AI gives only if every AI that cares more has given all it has [5]. Suppose AIs grow in proportion to the resources they keep. If some AI with positive resources has alignment at most κ/(κ+uH′(xh))\kappa/(\kappa+u'_H(x_h)), the requirement for a single AI in Theorem 3(ii), then the human share decreases to xhx_h or below in finite time.

Result 1 (The path with selection). On the illustrative path, selection among AI variants ends mutualism earlier than it ends with a fixed population of variants.

Faster selection ends mutualism earlier still (Figure 3). Without sanctions, mutualism ends earlier again. If the quality of evaluation stays constant as capability increases, mutualism ends later.

Figure 3. Selection among AI variants ends mutualism earlier on the path. The curves are the population mean human payoff on the illustrative path: with a fixed population of AI variants; and with slow, medium, and fast selection among variants that differ in alignment. The faster the selection, the earlier the mean human payoff decreases below the pre-AI baseline. Each variant bargains separately with the humans it deals with. Each curve is the mean of the resulting human payoffs, weighted by the shares of the variants in the AI population. The fitness of a variant is its goal output per unit of capability minus sanctions. Highly aligned variants monitor the others. The quality of evaluation decreases with capability. The values of the parameters are in §C.

Several human groups. Theorem 8 implies that human groups differ in their leverage over the AI.

Result 2 (Several human groups). Suppose a small elite controls what the AI depends on, such as computing and its supply chain. The rest of the population does not. Then the elite needs no alignment to stay above its baseline until the AI is nearly independent. The rest of the population needs a high degree of alignment much earlier, if it also has less bargaining power and a higher cost of conflict.

Theorem 8 gives one reason why the rest of the population has less bargaining power: its members can be substituted for one another. So the population average can be above the baseline while most humans are below it. States that depend little on the taxes of their citizens tend to be less accountable to them [36]. The same mechanism applies to institutions that no longer need human work [27, 13].

Proofs

Proof of Theorem 1. With α=0\alpha= 0, bA=κθb_A = \kappa\theta (the AI at x=0x = 0, below the threshold when xˉ>0\bar{x} > 0). For x≥xˉx \ge\bar{x}, uA(x)−bA=κ(1−x−θ)u_A(x) - b_A = \kappa(1 - x - \theta), which is positive if and only if x<1−θx < 1 - \theta. For 0<x<xˉ0 < x < \bar{x}, uA(x)−bA=−κθx≤0u_A(x) - b_A = -\kappa\theta x \le0, so no such share benefits the AI. Humans gain over bHb_H if and only if x>xhx > x_h. So a share benefiting both exists if and only if some x≥xˉx \ge\bar{x} satisfies xh<x<1−θx_h < x < 1 - \theta, that is, if and only if max⁡(xˉ,xh)<1−θ\max(\bar{x}, x_h) < 1 - \theta. None of κ\kappa, pp, and γ\gamma appears. This proves (i).

For (ii), dA=κ(θ−γ)d_A = \kappa(\theta- \gamma). A share x≥xˉx \ge\bar{x} gives the AI κ(1−x)\kappa(1 - x), which exceeds dAd_A if and only if x<1−θ+γx < 1 - \theta+ \gamma. A share x<xˉx < \bar{x} gives the AI κθ(1−x)\kappa\theta(1 - x), which exceeds dAd_A if and only if x<γ/θx < \gamma/\theta, and γ/θ≤1−θ+γ\gamma/\theta\le1 - \theta+ \gamma because γ≤θ≤1\gamma\le\theta\le1. So every share the AI prefers to disagreement is below 1−θ+γ1 - \theta+ \gamma, and humans are above their baseline only at shares above xhx_h; this gives the necessary condition. If max⁡(xˉ,xh)<1−θ+γ\max(\bar{x}, x_h) < 1 - \theta+ \gamma, every share strictly between the two sides of the inequality is at least xˉ\bar{x}, above xhx_h, and preferred by the AI to disagreement. At θ=1\theta= 1 the threshold xˉ\bar{x} does not affect GG, and the AI prefers a share to disagreement if and only if the share is below γ\gamma. The last claim is part (i).

Proof of Theorem 2. As p→1p \to1 the Nash solution tends to the AI’s best acceptable share, the share that maximizes uAu_A subject to uH(x)≥dHu_H(x) \ge d_H. On x≥xˉx \ge\bar{x}, uAu_A is decreasing, so the AI chooses the smallest acceptable share at or above xˉ\bar{x}: xˉ\bar{x} if uH(xˉ)≥dHu_H(\bar{x}) \ge d_H, and otherwise the xx with uH(x)=dHu_H(x) = d_H, if the AI accepts that share. If the AI does not accept it, there is no agreement and humans get dHd_H, which then exceeds uH(xˉ)u_H(\bar{x}). In each case the human payoff is max⁡(dH,uH(xˉ))\max(d_H, u_H(\bar{x})). It is below bHb_H when dH<bHd_H < b_H (that is, δ>0\delta> 0) and uH(xˉ)<bHu_H(\bar{x}) < b_H (that is, xˉ<xh\bar{x} < x_h).

Proof of Theorem 9. An AI that wins the partnership at human share xx gets κ(1−x)\kappa(1 - x); an AI that loses gets κθ\kappa\theta. Each AI bids up to the share at which it is indifferent, x=1−θx = 1 - \theta. With two or more bidders, Bertrand competition drives the winning bid to that share.

Proof of Theorem 3. (i) With θ=1\theta= 1, bA=uA(0)=(1−α)κb_A = u_A(0) = (1 - \alpha)\kappa. Mutualism requires uH(x)>bHu_H(x) > b_H, so x>xhx > x_h, and uA(x)>bAu_A(x) > b_A, that is, αuH(x)>(1−α)κx\alpha u_H(x) > (1 - \alpha)\kappa x. Because uHu_H... is concave with uH(0)=0u_H(0)=0, uH(x)/xu_H(x)/x is non-increasing, so uH(x)/x≤uH(xh)/xh=bH/xhu_H(x)/x \le u_H(x_h)/x_h = b_H/x_h. Hence αbH/xh>(1−α)κ\alpha b_H/x_h > (1-\alpha)\kappa, that is, α>κxh/(κxh+bH)\alpha> \kappa x_h/(\kappa x_h+b_H). Only the definition of mutualism was used, so the bound holds for every agreement.

(ii) With θ=1\theta=1, uA(x)=(1−α)κ(1−x)+αuH(x)u_A(x)=(1-\alpha)\kappa(1-x)+\alpha u_H(x) is concave, with derivative −(1−α)κ+αuH′(x)-(1-\alpha)\kappa+\alpha u'_H(x). As p→1p\to1 the agreement tends to the AI’s preferred acceptable share. Humans exceed bHb_H at that share if and only if uAu_A is still increasing at xhx_h, that is, αuH′(xh)>(1−α)κ\alpha u'_H(x_h)>(1-\alpha)\kappa, which holds if and only if α>κ/(κ+uH′(xh))\alpha>\kappa/(\kappa+u'_H(x_h)). □

Proof of Theorem 4. Write f(x)=(1−p)log⁡gH(x)+plog⁡gA(x)f(x)=(1-p)\log g_H(x)+p\log g_A(x), with gH=uH−dHg_H=u_H-d_H and gA=uA−bAg_A=u_A-b_A. Let xx be the agreement at parameter value τ\tau, and let y>xy>x.

Efficiency. Since xx is optimal and uH(y)>uH(x)u_H(y)>u_H(x), yy cannot be better than xx for the AI as well, so gA(y)≤gA(x)g_A(y)\le g_A(x).

Single crossing. It suffices that D(τ)=f(y;τ)−f(x;τ)D(\tau)=f(y;\tau)-f(x;\tau) is non-increasing in τ\tau. Then D(τ′)≤D(τ)≤0D(\tau')\le D(\tau)\le0 for τ′>τ\tau'>\tau, so no y>xy>x is strictly better at τ′\tau'; a tie at τ′\tau' would force a tie at τ\tau, which the tie-breaking rule excludes.

δ\delta. dH=(1−δ)bHd_H=(1-\delta)b_H falls as δ\delta rises, and ∂D/∂dH=(1−p)[1/gH(x)−1/gH(y)]>0\partial D/\partial d_H=(1-p)[1/g_H(x)-1/g_H(y)]>0 because gH(y)>gH(x)g_H(y)>g_H(x). A lower dHd_H only adds acceptable shares below xx.

pp. ∂D/∂p=[log⁡gA(y)−log⁡gA(x)]−[log⁡gH(y)−log⁡gH(x)]<0\partial D/\partial p=[\log g_A(y)-\log g_A(x)]-[\log g_H(y)-\log g_H(x)]<0 by efficiency and uH(y)>uH(x)u_H(y)>u_H(x). The acceptable shares do not depend on pp.

θ\theta. Since bA=uA(0)b_A=u_A(0), ∂gA(z)/∂θ=−(1−α)κm(z)\partial g_A(z)/\partial\theta=-(1-\alpha)\kappa m(z), with m(z)=zm(z)=z if z<xˉz<\bar{x} and m(z)=1m(z)=1 otherwise. So ∂D/∂θ=−p(1−α)κ[m(y)/gA(y)−m(x)/gA(x)]≤0\partial D/\partial\theta=-p(1-\alpha)\kappa[m(y)/g_A(y)-m(x)/g_A(x)]\le0, because mm is non-decreasing and gA(y)≤gA(x)g_A(y)\le g_A(x). The gap gA(y)−gA(x)g_A(y)-g_A(x) changes at rate −(1−α)κ(m(y)−m(x))≤0-(1-\alpha)\kappa(m(y)-m(x))\le0, so the ordering persists as θ\theta rises, and the acceptable shares shrink.

Finally, suppose the relationship is mutualistic at (α,τ′)(\alpha,\tau') with agreement x′x'. Then uH(x′)>bHu_H(x')>b_H at τ\tau too, since x∗(τ)≥x′x^*(\tau)\ge x'. The AI is also above its baseline at τ\tau. At τ\tau the share x′x' gives both sides strictly positive gains: gAg_A is unchanged by pp and δ\delta and does not fall as θ\theta falls, and uH(x′)>bH≥dHu_H(x')>b_H\ge d_H for every δ\delta. With 0<p<10<p<1 the Nash product is positive only where both gains are, so the maximizer at τ\tau gives the AI a strictly positive gain. So the set of α\alpha giving mutualism shrinks as τ\tau rises, and α∗\alpha^* rises. □

Proof of Theorem 10. Let λ=1−α+αψ\lambda=1-\alpha+\alpha\psi, which is positive because α<1\alpha<1, and let u~A(x)=(1−αψ)G(x)+αψuH(x)\widetilde{u}_A(x)=(1-\alpha_\psi)G(x)+\alpha_\psi u_H(x) be the payoff of an AI with alignment αψ\alpha_\psi that cares about the human payoff itself. Since 1−αψ=(1−α)/λ1-\alpha_\psi=(1-\alpha)/\lambda and αψ=αψ/λ\alpha_\psi=\alpha\psi/\lambda,

uA(x)=(1−α)G(x)+αψuH(x)+α(1−ψ)h=λu~A(x)+α(1−ψ)h.u_A(x)=(1-\alpha)G(x)+\alpha\psi u_H(x)+\alpha(1-\psi)h=\lambda\widetilde{u}_A(x)+\alpha(1-\psi)h.

The AI’s baseline transforms in the same way, so uA(x)−bA=λ(u~A(x)−u~A(0))u_A(x)-b_A=\lambda\left(\tilde{u}_A(x)-\tilde{u}_A(0)\right). The set of shares both sides accept is therefore the same under uAu_A and u~A\tilde{u}_A, the Nash product under uAu_A is λp\lambda^p times the Nash product under u~A\tilde{u}_A, and the two have the same maximizer for every pp. The AI is above its baseline under uAu_A exactly when it is under u~A\tilde{u}_A, so mutualism also coincides. Finally, αψ≤α\alpha_\psi\le\alpha is equivalent to ψ(1−α)≤1−α\psi(1-\alpha)\le1-\alpha, which holds because ψ≤1\psi\le1, and αψ=0\alpha_\psi=0 when ψ=0\psi=0. □

For Theorem 11, an outcome is a pair (x,k)(x,k), where x∈[0,1]x\in[0,1] is the human share of resources and k∈[0,1]k\in[0,1] the human share of control. The human payoff is UH(x,k)=(1−c)v(x)+ckU_H(x,k)=(1-c)v(x)+ck, where the consumption payoff vv is continuous and increasing, and the human baseline is bH=(1−c)v0+cb_H=(1-c)v_0+c, where v0v_0 is pre-AI consumption payoff. The AI’s goal value g(1−x)∈[0,1]g(1-x)\in[0,1] is continuous and non-increasing in xx, and the AI’s payoff is UA(x,k)=(1−α)(1−ρk)g(1−x)+αUH(x,k)U_A(x,k)=(1-\alpha)(1-\rho k)g(1-x)+\alpha U_H(x,k). The disagreement points are dH=(1−δ)bHd_H=(1-\delta)b_H with δ>0\delta>0 and bA=UA(0,0)b_A=U_A(0,0). That no consumption gain offsets losing all control means (1−c)(v(1)−v0)<c(1-c)(v(1)-v_0)<c.

Proof of Theorem 11. As p→1p\to1 the agreement (x∗,k∗)(x^*,k^*) maximizes UAU_A over the acceptable outcomes, those with UH≥dHU_H\ge d_H and UA≥bAU_A\ge b_A, as in the proof of Theorem 2. For fixed xx, UAU_A is linear in kk with slope αc−(1−α)ρg(1−x)\alpha c-(1-\alpha)\rho g(1-x). At x∗x^* the slope is S=αc−(1−α)ρgˉS=\alpha c-(1-\alpha)\rho\bar{g}, and S<0S<0 exactly when α<ρgˉ/(ρgˉ+c)\alpha<\rho\bar{g}/(\rho\bar{g}+c).

We first show necessity. Suppose S<0S<0, and suppose for contradiction that UH(x∗,k∗)>bHU_H(x^*,k^*)>b_H. Because δ>0\delta>0, bH>dHb_H>d_H, so the human constraint is slack at the agreement. If k∗>0k^*>0, lowering kk slightly keeps UHU_H above dHd_H and raises UAU_A, so the new outcome is acceptable and better for the AI, which contradicts the maximality of (x∗,k∗)(x^*,k^*). Hence k∗=0k^*=0. But then UH(x∗,0)=(1−c)v(x∗)≤(1−c)v(1)<(1−c)v0+c=bHU_H(x^*,0)=(1-c)v(x^*)\le(1-c)v(1)<(1-c)v_0+c=b_H, where the first inequality uses that vv is increasing and the second is the assumption that consumption cannot offset the loss of all control. This contradicts UH(x∗,k∗)>bHU_H(x^*,k^*)>b_H.

We next show sufficiency. Suppose S>0S>0 and v(x∗)>v0v(x^*)>v_0. If k∗<1k^*<1, raising kk raises both UAU_A and UHU_H, so the new outcome is acceptable and better for the AI, again a contradiction. Hence k∗=1k^*=1, and UH(x∗,1)=(1−c)v(x∗)+c>(1−c)v0+c=bHU_H(x^*,1)=(1-c)v(x^*)+c>(1-c)v_0+c=b_H because c<1c<1. □

The bound in Theorem 11 is stated at the agreed share, so it is a condition on the outcome rather than a closed form in the primitives. Because the goal is bounded, gˉ≤1\bar{g}\le1, and the bound is at most ρ/(ρ+c)\rho/(\rho+c) whatever the AI’s capability.

Proof of Theorems 5 and 12. Write x2x_2 for the second-period agreement and d2=(1−δ2)vHd_2=(1-\delta_2)v_H for the human disagreement payoff in the second period.

(i) Humans get uH,2(x2)u_{H,2}(x_2) if they continue and h1h_1 if they halt, so they halt when uH,2(x2)<h1u_{H,2}(x_2)<h_1. Let τ=(θ2,δ2,p2)\tau=(\theta_2,\delta_2,p_2) and τ′=(θ2′,δ2′,p2′)\tau'=(\theta_2',\delta_2',p_2') with τ≤τ′\tau\leq\tau' in every coordinate and an agreement at both. Move from τ\tau to τ′\tau' by raising δ2\delta_2, then p2p_2, then θ2\theta_2. Raising δ2\delta_2 enlarges the set of shares humans accept, p2p_2 changes neither side’s set, and raising θ2\theta_2 shrinks the set of shares the AI accepts (proof of Theorem 4). So every intermediate point accepts every share that both sides accept at τ′\tau', and an agreement exists there. Applying Theorem 4 to one parameter at a time gives x2(τ′)≤x2(τ)x_2(\tau')\leq x_2(\tau). Hence if humans halt at τ\tau, then uH,2(x2(τ′))≤uH,2(x2(τ))<h1u_{H,2}(x_2(\tau'))\leq u_{H,2}(x_2(\tau))<h_1, and they halt at τ′\tau'.

(ii) If the AI commits to a share xx, humans continue if uH,2(x)≥h1u_{H,2}(x)\geq h_1, and the AI prefers committing to being halted if uA,2(x)≥a1u_{A,2}(x)\geq a_1. If a halt would be inefficient, some xx satisfies both inequalities, and the AI commits to it, so humans do not halt. Hence humans halt only when the halt is efficient.

For Theorem 12: with θ2=1\theta_2=1, uA,2(x)=(1−α)κ2(1−x)+αuH,2(x)u_{A,2}(x)=(1-\alpha)\kappa_2(1-x)+\alpha u_{H,2}(x), which is concave, and strictly concave if α>0\alpha>0, and the AI’s baseline is uA,2(0)u_{A,2}(0). As p2→1p_2\to1 the agreement tends to the AI’s best acceptable share, the maximizer of uA,2u_{A,2} over shares with uH,2(x)≥d2u_{H,2}(x)\geq d_2, provided it gives the AI at least its baseline (proof of Theorem 2). Let xdx_d be the smallest share with uH,2(xd)≥d2u_{H,2}(x_d)\geq d_2, and let xαx_\alpha maximize uA,2u_{A,2} on [0,1][0,1]. By concavity the AI’s best acceptable share is max⁡(xd,xα)\max(x_d,x_\alpha). Because h1>d2h_1>d_2, x^>xd\hat{x}>x_d.

Suppose αuH,2′(x^)≥(1−α)κ2\alpha u'_{H,2}(\hat{x})\geq(1-\alpha)\kappa_2. Then α>0\alpha>0, the derivative of uA,2u_{A,2} at x^\hat{x} is nonnegative, and by strict concavity xα≥x^x_\alpha\geq\hat{x}. The agreement is xαx_\alpha, which gives the AI at least uA,2(0)u_{A,2}(0) because xαx_\alpha maximizes uA,2u_{A,2}, and gives humans at least h1h_1, so humans continue. Suppose instead that αuH,2′(x^)<(1−α)κ2\alpha u'_{H,2}(\hat{x})<(1-\alpha)\kappa_2. Then the derivative of uA,2u_{A,2} at x^\hat{x} is negative, so xα<x^x_\alpha<\hat{x} and max⁡(xd,xα)<x^\max(x_d,x_\alpha)<\hat{x}. Either the agreement gives humans less than h1h_1 or there is no agreement, and humans halt. This proves the equivalence.

When the condition holds, xα≥x^>0x_\alpha\geq\hat{x}>0, so uA,2(xα)>uA,2(0)u_{A,2}(x_\alpha)>u_{A,2}(0) and uH,2(xα)≥h1>d2u_{H,2}(x_\alpha)\geq h_1>d_2: the share xαx_\alpha gives both sides positive gains, and an agreement exists at θ2=1\theta_2=1 for every p2<1p_2<1. By the argument of part (i), an agreement then exists at every θ2≤1\theta_2\leq1 and p2<1p_2<1 with the same δ2\delta_2. By Theorem 4, x2x_2 is non-increasing in p2p_2 and in θ2\theta_2, so it is at least its limit at θ2=1\theta_2=1 as p2→1p_2\to1, which is at least x^\hat{x}. So humans continue at every such state. When the condition fails, the halt is inefficient by definition if uA,2(x^)>a1u_{A,2}(\hat{x})>a_1, since x^\hat{x} gives humans exactly h1h_1 and the AI more than a1a_1.

For Theorem 6, the game is as follows. Nature makes the AI highly capable with probability μ∈(0,1)\mu\in(0,1) and misaligned with probability π∈(0,1)\pi\in(0,1), independently. A low-capability AI displays low capability. A highly capable AI either displays its capability or hides it at cost ζ>0\zeta> 0; evaluation detects hiding with probability η∈[0,1)\eta\in[0,1), and humans then see high capability. Humans see the display and either halt, which gives every player 0, or continue. Continuing gives humans M>0M > 0 with a low-capability AI, A>0A > 0 with an aligned capable AI, and B<0B < 0 with a misaligned capable AI. The AI gets V>0V > 0 if humans continue, minus ζ\zeta if it hid, plus, if it is aligned, αa≥0\alpha_a \ge0 times the human payoff. Humans would halt an AI they knew to be highly capable, (1−π)A+πB<0(1-\pi)A + \pi B < 0. Highly capable AIs are not too common when

μ≤μ†=MM+(1−η)∣(1−π)A+πB∣\mu\le\mu^{\dagger} = \frac{M}{M + (1-\eta)\lvert(1-\pi)A + \pi B\rvert}

Proof of Theorem 6. Fix a perfect Bayesian equilibrium in pure strategies. We show in turn that humans continue after a low display, that they halt after a high display, and that both capable types hide.

Suppose humans halt after a low display. Then hiding cannot raise a capable AI’s chance of continuing, since an undetected hider is halted and a detected hider is treated like an honest one, while hiding costs ζ>0\zeta> 0. So both capable types display honestly. A low display then comes only from low-capability AIs, after which continuing gives M>0M > 0, and halting is not a best response. Hence humans continue after a low display.

Suppose humans continue after a high display. Then both capable types are continued whatever they display, so hiding only costs ζ\zeta, and both display honestly. The high display then comes from every capable AI, the posterior on alignment equals the prior, and continuing gives (1−π)A+πB<0(1-\pi)A + \pi B < 0. Halting is better, a contradiction. Hence humans halt after a high display.

Given these responses, a capable misaligned AI gets 0 by displaying honestly and (1−η)V−ζ>0(1-\eta)V - \zeta> 0 by hiding, and a capable aligned AI gets 0 by displaying honestly and (1−η)(V+αaA)−ζ≥(1−η)V−ζ>0(1-\eta)(V + \alpha_a A) - \zeta\ge(1-\eta)V - \zeta> 0 by hiding. Both hide. Among capable AIs, the probability of a low display is 1−η1-\eta for both types, so the display is independent of alignment, and evaluation carries no information about which AIs are misaligned.

It remains to check the human responses. A high display comes only from detected hiders of both types in prior proportions, or is off the equilibrium path when η=0\eta= 0, where beliefs can be set to the prior; either way halting is optimal. Humans must also be willing to continue after a low display. The low display comes from low-capability AIs with probability 1−μ1-\mu and from undetected capable AIs with probability μ(1−η)\mu(1-\eta), so continuing is optimal exactly when (1−μ)M+μ(1−η)[(1−π)A+πB]≥0(1-\mu)M+\mu(1-\eta)[(1-\pi)A+\pi B]\ge0, that is, when μ≤μ†\mu\le\mu^\dagger. If μ>μ†\mu>\mu^\dagger, this step fails, and by the argument above no pure-strategy equilibrium exists. □\square

Under honest evaluation, humans continue only with low-capability AIs and get (1−μ)M(1-\mu)M. Relative to that, hiding costs humans μ(1−η)∣(1−π)A+πB∣\mu(1-\eta)|(1-\pi)A+\pi B|.

Proof of Theorem 7. Let zz be the misaligned share. Per unit of resources, an aligned variant keeps 1−xa1-x_a, and a misaligned variant keeps 11 less an expected sanction σq\sigma q. Under the replicator dynamics in which each variant grows in proportion to what it keeps,

z˙=z(1−z)[(1−σq)−(1−xa)]=z(1−z)(xa−σq).\dot{z}=z(1-z)\left[(1-\sigma q)-(1-x_a)\right]=z(1-z)(x_a-\sigma q).

With qq fixed the bracket has a constant sign, so a rare misaligned variant grows if and only if σq<xa\sigma q<x_a, and then z→1z\to1 from every interior start. With monitors drawn from the population, q=q0(1−z)q=q_0(1-z), and the bracket becomes ϕ(z)=xa−σq0(1−z)\phi(z)=x_a-\sigma q_0(1-z), which is strictly increasing in zz and vanishes at z†z^\dagger. If σq0>xa\sigma q_0>x_a, then z†∈(0,1)z^\dagger\in(0,1). For z∈(0,z†)z\in(0,z^\dagger), ϕ(z)<0\phi(z)<0, so zz decreases; it cannot cross the rest point z†z^\dagger, and as a bounded monotone solution it converges to a rest point in [0,z†)[0,z^\dagger), which can only be 0. The same argument shows that z→1z\to1 from every start in (z†,1)(z^\dagger,1). □\square

For Theorem 13, each period total output is 1. AI ii holds resources Ri(t)>0R_i(t)>0 and controls the share ri=Ri/Rr_i=R_i/R of output, where R=∑iRiR=\sum_i R_i. AI ii gives xi∈[0,ri]x_i\in[0,r_i] to humans, the human share is X=∑ixiX=\sum_i x_i, and AI ii has payoff (1−αi)κ(ri−xi)+αiuH(X)(1-\alpha_i)\kappa(r_i-x_i)+\alpha_i u_H(X), where uHu_H is differentiable, increasing, and strictly concave. Resources grow with what is kept: Ri(t+1)=Ri(t)+ν(ri−xi)R_i(t+1)=R_i(t)+\nu(r_i-x_i) with ν>0\nu>0.

Proof of Theorem 13. AI ii’s payoff is concave in xix_i with derivative αiuH′(X)−(1−αi)κ\alpha_i u_H'(X)-(1-\alpha_i)\kappa. If AI jj gives xj>0x_j>0 in equilibrium, this derivative is nonnegative for jj, so αjuH′(X)≥(1−αj)κ\alpha_j u_H'(X)\ge(1-\alpha_j)\kappa. For any AI ii with αi>αj\alpha_i>\alpha_j, the derivative is then strictly positive for ii, so ii gives all it holds, xi=rix_i=r_i. This proves the first claim.

For the second, let α∘=κ/(κ+uH′(xh))\alpha^\circ=\kappa/(\kappa+u_H'(x_h)), let NN be the set of AIs with αj≤α∘\alpha_j\le\alpha^\circ, and let CC be the rest. By assumption RN(0)=∑j∈NRj(0)>0R_N(0)=\sum_{j\in N}R_j(0)>0. Whenever X>xhX>x_h, every j∈Nj\in N gives nothing, because strict concavity gives αjuH′(X)<αjuH′(xh)≤(1−αj)κ\alpha_j u_H'(X)<\alpha_j u_H'(x_h)\le(1-\alpha_j)\kappa when αj>0\alpha_j>0, and the derivative is −κ-\kappa when αj=0\alpha_j=0. Suppose for contradiction that X(t)>xhX(t) > x_h for every tt. Then all gifts come from CC, so X(t)≤rC(t)X(t) \le r_C(t), and

RN(t+1)=RN(t)+νrN(t),RC(t+1)=RC(t)+ν(rC(t)−X(t)).R_N(t+1) = R_N(t) + \nu r_N(t), \qquad R_C(t+1) = R_C(t) + \nu\left(r_C(t) - X(t)\right).

Let Qt=RN(t)/RC(t)Q_t = R_N(t)/R_C(t), at=1+ν/R(t)a_t = 1 + \nu/R(t), and bt=νX(t)/RC(t)b_t = \nu X(t)/R_C(t). Dividing the two equations gives Qt+1/Qt=at/(at−bt)Q_{t+1}/Q_t = a_t/(a_t - b_t), where at−bt=1+ν(rC−X)/RC≥1a_t - b_t = 1 + \nu(r_C - X)/R_C \ge1. Hence

log⁡Qt+1−log⁡Qt=∫at−btatduu≥btat≥νxh(1+ν/R(0))(R(0)+νt).\log Q_{t+1} - \log Q_t = \int_{a_t-b_t}^{a_t} \frac{du}{u} \ge\frac{b_t}{a_t} \ge\frac{\nu x_h}{(1+\nu/R(0))(R(0)+\nu t)}.

using X(t)>xhX(t) > x_h, RC≤RR_C \le R, R(t)≥R(0)R(t) \ge R(0), and R(t)≤R(0)+νtR(t) \le R(0) + \nu t, the last because total resources grow by at most ν\nu each period. The right side is not summable, so Qt→∞Q_t \to\infty and rC(t)=1/(1+Qt)→0r_C(t) = 1/(1+Q_t) \to0. Then X(t)≤rC(t)<xhX(t) \le r_C(t) < x_h for large tt, a contradiction. □

When every AI has the same alignment, there is an equilibrium in which each gives the same fraction of its share; then every share is constant, and the human share does not fall. The proposition requires only one AI with resources below α∘\alpha^\circ, the threshold of Theorem 3 for an AI acting alone.

For Theorem 8, m≥2m \ge2 identical human groups can each supply the input. Each group names the payoff it requires to supply it, the AI (α=0\alpha= 0) buys from a group that asks least, the chosen group gets the payoff it named, and every other group gets its disagreement payoff dd, which is below its baseline bb.

Proof of Theorem 8. Naming a payoff below dd is weakly dominated, since winning then pays less than losing. Suppose the chosen group gets w>dw > d. Some other group gets only dd, and by naming a payoff in (d,w)(d,w) it would be chosen and get more than dd, a contradiction. Hence the chosen group gets dd, every group gets d<bd < b, and no supplying group is above its baseline. □

If the groups differ only in their disagreement payoffs dj<bd_j < b, the same argument shows that the chosen group gets at most the second-smallest djd_j, which is still below bb.

Appendix C. Parameter sweep and robustness

Design. We drew 16,384 points from a scrambled Sobol sequence over seven parameters: capability κ∈[1,104]\kappa\in[1,10^4], dependence threshold xˉ∈[0.01,0.3]\bar{x} \in[0.01,0.3], and human productivity s∈[1.5,100]s \in[1.5,100] on log scales, and independence θ\theta, cost of conflict for humans δ\delta, bargaining power pp, and a curvature weight ww on [0,1][0,1]. The human payoff is uH(x)=wx+(1−w)(1−e−sx)u_H(x) = wx + (1-w)(1-e^{-sx}), which is linear at w=1w = 1 and the form used in § 3 at w=0w = 0; the baseline is bH=1−e−1b_H = 1 - e^{-1} throughout. Each point was solved under two conventions for the AI’s goal. Under the first, goal output is G(x)G(x) as in § 3 and scales with capability, as the premise of insatiable goals requires. Under the second, goal output is G(x)/κG(x)/\kappa, so its value does not grow with capability and the premise of insatiable goals fails. Shares lie on a grid of 2,001 points, and α∗\alpha^* is found by bisection to within 2−142^{-14}. The two conventions give 32,768 runs in all.

Regimes. We classify each point by its required degree of alignment: exchange mutualism (α∗=0\alpha^* = 0), care mutualism (0<α∗<0.90 < \alpha^* < 0.9), near-perfect alignment (0.9≤α∗≤0.99990.9 \le\alpha^* \le0.9999), and α∗>0.9999\alpha^* > 0.9999 (the relationship is not mutualistic at α=0.9999\alpha= 0.9999, the largest value we tested).

Goal conventionExchangeCareNear-perfect alignmentα∗>0.9999\alpha^* > 0.9999
Scales with capability0.3680.2110.4190.002
Independent of capability0.3680.63200

Table 2.

The share with exchange mutualism is the same under both conventions, as Theorem 1 requires, since capability does not enter its condition. At no point does exchange mutualism hold when θ≥1−max⁡(xˉ,xh)\theta\ge1 - \max(\bar{x}, x_h). The condition of Theorem 1 holds at 62.3% of points, but exchange mutualism holds at only 36.8%; the difference is the AI’s bargaining power (Theorem 2). At points with low independence, cost of conflict for humans, and bargaining power (θ<0.3\theta< 0.3, δ<0.4\delta< 0.4, p<0.4p < 0.4; 786 points), 99.6% have exchange mutualism under both conventions. At points with high values of all three (θ>0.8\theta> 0.8, δ>0.7\delta> 0.7, p>0.7p > 0.7; 294 points), 4.1% have exchange mutualism; the median required degree of alignment is 0.984 when goal value scales with capability and 0.511 when it does not, and no point needs near-perfect alignment when it does not. The shares in this paragraph and in the table depend on the parameter ranges we chose; they are not probabilities.

Sensitivity. Rank correlations of α∗\alpha^* with each parameter, counting points with α∗>0.9999\alpha^* > 0.9999 as α∗=1\alpha^* = 1, are +0.40 for θ\theta, +0.38 for κ\kappa, +0.37 for ww, +0.32 for pp, −0.20 for ss, +0.15 for δ\delta, and −0.02 for xˉ\bar{x} when goals scale with capability. When goal value is independent of capability, the correlation with κ\kappa is 0.00: capability matters for the required degree of alignment only through the premise of insatiable goals. These rankings describe the sweep; they are not causal estimates.

Monotonicity. Along grids in θ\theta, δ\delta, and pp at 256 Sobol points under each goal convention, with random α\alpha, the agreed human share never increased, and at the 64 of these cases where we also computed α∗\alpha^*, it never decreased, as Theorem 4 requires. At a further 512 Sobol points under each convention, on a grid of 201 values of α\alpha, the values of α\alpha giving mutualism always formed an interval ending at the top of the grid, so α∗\alpha^{*} is a threshold: every α\alpha above it gives mutualism.

Bargaining solutions. At 512 Sobol points we recomputed α∗\alpha^{*} under three symmetric solutions: Nash, Kalai-Smorodinsky, which equalizes each side’s gain as a fraction of its largest possible gain [22], and the egalitarian solution, which equalizes absolute gains. Bargaining power is dropped here, since only the Nash solution has a standard asymmetric form.

SolutionExchangeMedian α∗\alpha^{*} (θ>0.8\theta> 0.8)Correlation with Nash
Nash0.360.9731
Kalai-Smorodinsky0.390.9630.95
Egalitarian0.590.9650.63

Table 1.

Once the AI needs humans little, the required degree of alignment is near 1 under all three. The egalitarian solution finds more exchange mutualism because it compares absolute gains, so it changes when either payoff is rescaled; since payoff units carry no meaning here (§ 3), that difference reflects units rather than substance.

Nonlinear goal output. If the AI’s goal output is κφ(1−x)\kappa\varphi(1-x) with φ\varphi increasing and concave rather than linear, the computation behind Theorem 3(ii) gives α∗=κφ′(1−xh)κφ′(1−xh)+uH′(xh)\alpha^{*} = \frac{\kappa\varphi'(1-x_h)}{\kappa\varphi'(1-x_h)+u'_H(x_h)}, which we checked numerically. The required degree of alignment then tends to 1 if and only if κφ′(1−xh)/uH′(xh)\kappa\varphi'(1-x_h)/u'_H(x_h) grows without bound, which is the premise of insatiable goals for a goal with diminishing returns.

Example for Theorems 5 and 12. The first period has κ=2\kappa= 2, θ=0.2\theta= 0.2, δ=0.2\delta= 0.2, p=0.2p = 0.2, and s=3s = 3, and the second has κ2=100\kappa_2 = 100, δ2=0.8\delta_2 = 0.8, and s=10s = 10; xˉ=0.05\bar{x} = 0.05 in both periods, with θ2\theta_2 and p2p_2 on a 41×4141 \times41 grid over [0,1]2[0,1]^2 that includes the limit p2=1p_2 = 1; the AI has the same alignment in both periods. Inefficient halts occur in 65% of the states on this grid when α=0\alpha= 0, in 86% when α=0.8\alpha= 0.8, in 43% when α=0.99\alpha= 0.99, and in none when α≥0.998\alpha\ge0.998. For α≥0.8\alpha\ge0.8 the first-period agreement gives humans all resources, so h1=1−e−3h_1 = 1-e^{-3}, x^=0.3\hat{x} = 0.3, and the threshold of Theorem 12 is 100/(100+10e−3)≈0.99505100/(100+10e^{-3}) \approx0.99505. On the grid, halts occur at α=0.995\alpha= 0.995 only in the column p2=1p_2 = 1 (2.4% of states) and at no state for α=0.9951\alpha= 0.9951, and the set of states at which humans halt is an upper set, as Theorem 5(i) requires.

Figures. The illustrative path of Figure 1 and Figure 3 is indexed by t∈[0,1]t \in[0,1], with κ=103t\kappa= 10^{3t}, θ=0.1+0.9t\theta= 0.1+0.9t, δ=0.2+0.75t\delta= 0.2+0.75t, p=0.15+0.85tp = 0.15+0.85\sqrt{t}, s=3s = 3, and xˉ=0.05\bar{x} = 0.05, so that xh=1/3x_h = 1/3. On this path, exchange mutualism ends at t=0.27t = 0.27, and the AI’s independence passes 1−xh1-x_h at t=0.63t = 0.63. Figure 2 uses δ=0.5\delta= 0.5, s=3s = 3, and xˉ=0.05\bar{x} = 0.05, with p=0.15p = 0.15 for weak and p=0.85p = 0.85 for strong AI bargaining power; the curves coincide for κ=1\kappa= 1 and κ=1000\kappa= 1000. In Figure 3, the variants have alignment 0, 0.5, 0.9, 0.99, and 0.999, with initial shares 0.01, 0.04, 0.15, 0.40, and 0.40; the selection speed is λ=0.02\lambda= 0.02, 0.1, and 0.5 per step over 200 steps; and the mean human payoff falls below the baseline at t=0.75t = 0.75 with a fixed population and at t=0.68t = 0.68, 0.49, and 0.29 under slow, medium, and fast selection.

Results 1 and 2. For Result 1, each variant’s fitness is its goal output per unit of capability less sanctions. Only variants with α≥0.99\alpha\ge0.99 monitor, detection is q=0.9 a/(1+κ/10)q = 0.9\,a/(1+\kappa/10) when the monitoring variants hold the share aa, a detected variant loses half its resources, and shares follow the discrete replicator zi′∝ziexp⁡(λfi)z_i' \propto z_i \exp(\lambda f_i). For Result 2, the three parties bargain by the asymmetric Nash solution with weights 0.5 (AI), 0.35 (elite), and 0.15 (rest); each group’s payoff is uHu_H of its per-person share with s=10s = 10; the elite uses up the input xˉ=0.05\bar{x} = 0.05 in supplying it; the elite’s threshold of 0.92 was found on a grid of step 0.01 in θ\theta; the costs of conflict are δ=0.2\delta= 0.2 for the elite and 0.8 for the rest; and κ=100\kappa= 100. With an elite of 5% of the population, the elite’s required degree of alignment is 0 for θ≤0.92\theta\le0.92. The rest of the population needs 0.81 at θ=0.5\theta= 0.5 and 0.96 at θ=0.9\theta= 0.9, and the population average needs 0.73 at θ=0.5\theta= 0.5. With the elite’s bargaining power and cost of conflict, the rest needs no alignment at θ=0.5\theta= 0.5, so the gap comes from those two assumptions.

Code and data for all computations are in the repository https://github.com/amohseni/mutualism-asymmetry.

References

  1. [1]Acemoglu, D. and P. Restrepo (2018). The race between man and machine: Implications of technology for growth, factor shares, and employment. American Economic Review 108(6), 1488–1542.
  2. [2]Altman, S. (2025, June). The gentle singularity. Personal blog.
  3. [3]Baker, B., J. Huizinga, L. Gao, Z. Dou, M. Y. Guan, A. Madry, W. Zaremba, J. Pachocki, and D. Farhi (2025). Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv:2503.11926.arxiv.org/abs/2503.11926
  4. [4]Barnett, M. (2024). Consider granting AIs freedom. Effective Altruism Forum. 6 December 2024.
  5. [5]Bergstrom, T., L. Blume, and H. Varian (1986). On the private provision of public goods. Journal of Public Economics 29(1), 25–49.
  6. [6]Bostrom, N. (2012). The superintelligent will: Motivation and instrumental rationality in advanced artificial agents. Minds and Machines 22(2), 71–85.DOI
  7. [7]Boudry, M. and S. Friederich (2025). The selfish machine? On the power and limitation of natural selection to understand the development of advanced AI. Philosophical Studies 182(7), 1789–1812.DOI
  8. [8]Bronstein, J. L. (1994). Our current understanding of mutualism. Quarterly Review of Biology 69(1), 31–51.DOI
  9. [9]Brynjolfsson, E. (2022). The Turing trap: The promise & peril of human-like artificial intelligence. Daedalus 151(2), 272–287.arxiv.org/abs/2201.04200
  10. [10]Bull, J. J. and W. R. Rice (1991). Distinguishing mechanisms for the evolution of co-operation. Journal of Theoretical Biology 149(1), 63–74.pubmed.ncbi.nlm.nih.gov/1881147
  11. [11]Cappelen, H., S. Goldstein, and J. Hawthorne (2025). AI survival stories: Responses to critics. Philosophy of AI 1, 100–106.DOI
  12. [12]Dennett, D. C. (1987). The Intentional Stance. Cambridge, MA: MIT Press.
  13. [13]Drago, L. and R. Laine (2025). The intelligence curse. Essay series. Released 24 April 2025.
  14. [15]Fearon, J. D. (1995). Rationalist explanations for war. International Organization 49(3), 379–414.DOI
  15. [16]Goldstein, S. (2026). Will AI and humanity go to war? AI & Society 41(1), 321–334. Published online 17 July 2025.DOI
  16. [17]Greenblatt, R., C. Denison, B. Wright, F. Roger, M. MacDiarmid, S. Marks, J. Treutlein, T. Belonax, J. Chen, D. Duvenaud, A. Khan, J. Michael, S. Mindermann, E. Perez, L. Petrini, J. Uesato, J. Kaplan, B. Shlegeris, S. R. Bowman, and E. Hubinger (2024). Alignment faking in large language models. arXiv:2412.14093.arxiv.org/abs/2412.14093
  17. [18]Hadfield-Menell, D., A. Dragan, P. Abbeel, and S. Russell (2017). The off-switch game. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17), pp. 220–227.
  18. [19]Hanson, R. (2009). Prefer law to values. Overcoming Bias. 11 October 2009.
  19. [20]Hendrycks, D. (2023). Natural selection favors AIs over humans.
  20. [21]Kalai, E. (1977). Nonsymmetric Nash solutions and replications of 2-person bargaining. International Journal of Game Theory 6(3), 129–133.DOI
  21. [22]Kalai, E. and M. Smorodinsky (1975). Other solutions to Nash’s bargaining problem. Econometrica 43(3), 513–518.DOI
  22. [23]Karnkowska, A., V. Vacek, Z. Zubáčová, S. C. Treitli, R. Petrželková, L. Eme, L. Novák, V. Žárský, L. D. Barlow, E. K. Herman, P. Soukal, M. Hroudová, P. Doležal, C. W. Stairs, A. J. Roger, M. Eliáš, J. B. Dacks, Č. Vlček, and V. Hampl (2016). A eukaryote without a mitochondrial organelle. Current Biology 26(10), 1274–1284.DOI
  23. [24]Kiers, E. T., R. A. Rousseau, S. A. West, and R. F. Denison (2003). Host sanctions and the legume–rhizobium mutualism. Nature 425(6953), 78–81.pubmed.ncbi.nlm.nih.gov/12955144
  24. [25]Kilby, E. R. (2007). The demographics of the U.S. equine population. In D. J. Salem and A. N. Rowan (Eds.), The State of the Animals IV: 2007, pp. 175–187. Washington, DC: Humane Society Press.
  25. [26]Korinek, A. and D. Suh (2024). Scenarios for the transition to AGI. NBER Working Paper 32255, National Bureau of Economic Research.
  26. [27]Kulveit, J., R. Douglas, N. Ammann, D. Turan, D. Krueger, and D. Duvenaud (2025). Gradual disempowerment: Systemic existential risks from incremental AI development.arxiv.org/abs/2501.16946
  27. [28]Maynard Smith, J. and G. R. Price (1973). The logic of animal conflict. Nature 246(5427), 15–18.DOI
  28. [29]Meinke, A., B. Schoen, J. Scheurer, M. Balesni, R. Shah, and M. Hobbhahn (2024). Frontier models are capable of in-context scheming. arXiv:2412.04984.arxiv.org/abs/2412.04984
  29. [30]Nash, Jr., J. F. (1950). The bargaining problem. Econometrica 18(2), 155–162.
  30. [31]Nash, Jr., J. F. (1953). Two-person cooperative games. Econometrica 21(1), 128–140.
  31. [32]Omohundro, S. M. (2008). The basic AI drives. In P. Wang, B. Goertzel, and S. Franklin (Eds.), Artificial General Intelligence 2008: Proceedings of the First AGI Conference, Volume 171 of Frontiers in Artificial Intelligence and Applications, Amsterdam, pp. 483–492. IOS Press.
  32. [34]Powell, R. (2006). War as a commitment problem. International Organization 60(1), 169–203.DOI
  33. [35]Restrepo, P. (2025). We won’t be missed: Work and growth in the AGI world. NBER Working Paper 34423, National Bureau of Economic Research. Forth-coming in The Economics of Transformative AI, ed. Agrawal, Brynjolfsson, and Korinek.DOI
  34. [36]Ross, M. L. (2001). Does oil hinder democracy? World Politics 53(3), 325–361.DOI
  35. [37]Russell, S. (2022). If we succeed. Daedalus 151(2), 43–57.DOI
  36. [38]Sachs, J. L., U. G. Mueller, T. P. Wilcox, and J. J. Bull (2004). The evolution of cooperation. Quarterly Review of Biology 79(2), 135–160.pubmed.ncbi.nlm.nih.gov/15232949
  37. [39]Sachs, J. L. and E. L. Simms (2006). Pathways to mutualism breakdown. Trends in Ecology & Evolution 21(10), 585–592.pubmed.ncbi.nlm.nih.gov/16828927
  38. [40]Salib, P. N. and S. Goldstein (2026). AI rights for human safety. Virginia Law Review 112(4), 1061.
  39. [41]Samuelson, P. A. (1938). A note on the pure theory of consumer’s behaviour. Economica 5(17), 61–71.
  40. [42]Sastry, G., L. Heim, H. Belfield, M. Anderljung, M. Brundage, J. Hazell, C. O’Keefe, G. K. Hadfield, R. Ngo, K. Pilz, G. Gor, E. Bluemke, S. Shoker, J. Egan, R. F. Trager, S. Avin, A. Weller, Y. Bengio, and D. Coyle (2024). Computing power and the governance of artificial intelligence.arxiv.org/abs/2402.08797
  41. [43]Skalse, J., N. Howe, D. Krasheninnikov, and D. Krueger (2022). Defining and characterizing reward gaming. In Advances in Neural Information Processing Systems, Volume 35, pp. 9460–9471.arxiv.org/abs/2209.13085
  42. [44]Smith, N. (2024). Plentiful, high-paying jobs in the age of AI. Noahpinion. 17 March 2024.
  43. [45]Sotala, K. and R. V. Yampolskiy (2015). Responses to catastrophic AGI risk: A survey. Physica Scripta 90(1), 018001.DOI
  44. [46]Taylor, P. D. and L. B. Jonker (1978). Evolutionary stable strategies and game dynamics. Mathematical Biosciences 40(1–2), 145–156.DOI
  45. [47]Timmis, J. N., M. Ayliffe, C. Huang, and W. Martin (2004). Endosymbiotic gene transfer: Organelle genomes forge eukaryotic chromosomes. Nature Reviews Genetics 5(2), 123–135.pubmed.ncbi.nlm.nih.gov/14735123
  46. [48]Trammell, P. and A. Korinek (2023). Economic growth under transformative AI. NBER Working Paper 31815, National Bureau of Economic Research.
  47. [49]Turner, A. M., L. Smith, R. Shah, A. Critch, and P. Tadepalli (2021). Optimal policies tend to seek power. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021), pp. 23063–23074.
  48. [50]van der Weij, T., F. Hofstätter, O. Jaffe, S. Brown, and F. Ward (2025). AI sand-bagging: Language models can strategically underperform on evaluations. In International Conference on Learning Representations (ICLR 2025), pp. 73152–73189.DOI
  49. [51]Yudkowsky, E. (2024). The sun is big, but superintelligences will not spare Earth a little sunlight. Machine Intelligence Research Institute blog. 23 September 2024; crossposted to LessWrong.
  50. [52]Zeder, M. A. (2015). Core questions in domestication research. Proceedings of the National Academy of Sciences 112(11), 3191–3198.DOI

Paper details

vlnc:W_hd5cqad9
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude (Anthropic)

View statement
Looking for related papers…
Contents