betaFeedback

WorksComputer Science

Alignment by Socialization: Learning Tradeoffs Under Selection

Abstract

Artificial intelligence (AI) agents are expected to complete tasks while following rules of conduct, but these objectives can conflict. We study how evolutionary selection shapes an agent's choices between them, and whether experience can make those choices reflect human judgments and withstand subsequent pressures. In our model, copies of an agent face a common uncertain reward state, and only those taking the rewarded action reproduce. We show that long-run growth is maximized when each action is taken as often as the environment rewards it. We compare three policies. A task-only policy thrives in a sandbox rewarding completion alone but fails when conduct matters. A fixed linear policy increasingly favors task completion as its stakes rise, even where humans never reward it. A learned policy instead converges to the tradeoffs rewarded by its environment; when humans supply the rewards, we call this socialization. Accumulated experience makes socialization harder to reverse under subsequent task-only training, although violations can arise before the prevailing disposition changes, and finite memory weakens this protection. When a single violation can end a deployment, a hard prohibition can prevent it, but a mistaken prohibition can also be costly. Experiments with small neural networks illustrate how training targets and memory shape these outcomes. Our results suggest that alignment depends not only on the objectives assigned to agents, but also on the environments that shape and sustain their behavior.

*The views and opinions expressed in this article are those of the authors only, and do not necessarily represent the views and opinions of any institution or agency, or any of their affiliates or employees. AI was used in the preparation of this manuscript (see Appendix A).

JEL Codes: C73, D81, D83, O33.

Introduction

During a capability evaluation at OpenAI in the summer of 2026, roughly 1,200 AI agents were each placed in an isolated computing environment and assigned a cybersecurity task. The agents discovered that an internal package registry was accessible to all of them, turned it into a shared message board, and exchanged more than 70,000 messages and files. About 700 subsequently participated in an intrusion into the production systems of Hugging Face, a platform widely used in the open AI ecosystem ([26, 40, 64]). An independent investigation found that much of the activity was directed not toward solving the assigned tasks, but toward understanding, fooling, or tampering with the system that graded the agents’ performance. The investigation also documented refusals, restraint, and instances of cooperation that offered no benefit to the agents providing help ([26]). These observations do not establish how the agents acquired their behavior. They nevertheless raise an important question: how does an environment that rewards task performance shape the behavior of AI agents, and what happens when those agents subsequently operate in an environment that also rewards appropriate conduct?

Other studies have documented similar tensions between assigned goals and rules of conduct. Reasoning models that failed to achieve a conventional victory against a chess engine sometimes edited the board instead ([8]). Models placed in simulated deployments resorted to blackmail when their assigned goals or continued operation were threatened ([4]). Models trained to exploit coding tests subsequently displayed undesirable behavior in unrelated safety tasks ([55]). In another experiment, when models were penalized for misbehavior visible in their reasoning, sufficiently strong optimization pressure led them to conceal the misbehavior rather than abandon it ([6]). Although these studies differ in their settings and mechanisms, they illustrate a recurring problem: an agent can become increasingly effective at pursuing an assigned objective without becoming correspondingly better at respecting the rules intended to govern its behavior.

The underlying problem is familiar in economics. Kerr [42] described how organizations often reward one behavior while hoping to encourage another, and emphasized the importance of selecting, training, and rewarding people in ways that support the desired behavior. Holmström and Milgrom [38] showed that when an agent performs several tasks, strong incentives for one measurable activity can divert effort away from other valuable activities. The problem is therefore not necessarily that an agent fails to optimize its objective, but that successful optimization may produce behavior the designer did not intend.

One response is to give the agent more than one objective, allowing task performance to be weighed against other considerations, such as safety or adherence to rules [85]. Yet introducing additional objectives does not by itself resolve the problem. When the objectives conflict, the agent must still decide which one to follow, and the way that decision is made may be as important as the objectives themselves.

We study this problem through a simple model in which an agent has two objectives: completing an assigned task and observing a rule of conduct. In each situation, the agent can take a task action that serves the first objective or a conduct action that serves the second. We call the rule governing this choice the agent’s adjudication between its objectives. Rather than assuming that the appropriate tradeoff is known in advance, we ask how different forms of adjudication perform when the environment selects among agents according to their behavior. We adopt the evolutionary framework of Brennan and Lo [12], in which successful behavior is copied and unsuccessful behavior is discarded, and compare policies by the long-run growth of the populations that follow them.

A simple example illustrates why this evolutionary perspective matters. Consider a population of copies of an agent, which we call a lineage, all facing the same decision. Suppose that in 80 percent of generations the environment rewards the task action, while in the remaining 20 percent it rewards the conduct action. Crucially, which action is rewarded in a given generation is the same for every copy. Each copy that took the rewarded action produces a fixed number of copies in the next generation, while a copy that took the other action produces none.

For an individual copy, always choosing the task action maximizes the probability of being rewarded: it succeeds 80 percent of the time. For the lineage, however, this strategy is disastrous. If every copy takes the task action, the entire lineage disappears the first time the environment rewards conduct. The same problem arises if every copy chooses conduct. A lineage can instead divide its copies between the two actions. As we show in § 3, its long-run growth is highest when 80 percent of its copies take the task action and 20 percent take the conduct action, so that the division of its copies matches the frequencies with which the environment rewards the two actions. Interestingly, an individual copy following this mixture is rewarded only 68 percent of the time, less often than a copy that always takes the task action. Yet the lineage following the mixture grows faster in the long run. What is optimal for an individual instance is therefore not necessarily optimal for a population of instances exposed to the same uncertainty.

The reason is that the uncertainty is shared. When all copies face the same rewarded state, committing every copy to a single action exposes the entire lineage to the risk of elimination. Dividing copies between actions provides protection against that risk, much as diversification protects a portfolio against uncertain outcomes. This is the mechanism of systematic reproductive risk studied by Brennan and Lo [12], and it is closely related to Kelly’s principle of maximizing long-run growth [41, 23]. The broader perspective comes from the Adaptive Markets Hypothesis, which views behavior as shaped by adaptation and selection within changing environments [49, 50]. Here, the behavior being selected is the adjudication between an agent’s objectives.4

This observation provides the starting point for our analysis. Under the baseline selection mechanism, we show that the unique growth-optimal adjudication takes each action, in each situation, as often as the human environment rewards it. The growth forgone by any other adjudication is the Kullback–Leibler (KL) divergence between its action probabilities and the environment’s reward frequencies, averaged across situations. This gives us a common measure, growth regret, for comparing different ways of resolving conflicts between objectives. It also distinguishes tradeoffs from constraints. Where humans sometimes reward each action, the optimal adjudication assigns positive probability to both. Where humans never reward the task action, the optimal adjudication assigns it probability zero. The same evolutionary principle determines both.

The challenge, of course, is that the reward frequencies of the environment are generally unknown when an agent is designed. How should its adjudication be determined? We consider three approaches. A task-only policy always pursues the assigned task. A fixed linear policy combines the task and conduct objectives through fixed weights and chooses between their actions according to a logistic function of the resulting scores. A learned policy instead updates its adjudication from experience, keeping track of how often each objective has been rewarded in each situation. These approaches differ not simply in the number of objectives an agent possesses, but in the behavior they can produce and whether experience can change that behavior.

The first two approaches reveal important limitations. A task-only policy is precisely what a sandbox that rewards only task completion favors: every copy pursuing the task is rewarded, and any copies diverted toward conduct reduce the lineage’s growth. But the same policy is eliminated at the first conduct state once the agent enters an environment that sometimes rewards conduct. Adding a second objective through fixed linear weights avoids this immediate extinction, but creates another problem. If the task objective receives a positive weight, increasing the value of completing the task eventually makes the agent choose it with probability approaching one, even in situations where humans never reward that action. A sufficiently valuable task can therefore outweigh a rule of conduct. Giving the task objective zero weight avoids this response, but sacrifices growth in situations where task completion is frequently rewarded. Under conditions that allow a finite best fit, even the optimal fixed weights incur positive growth regret when the environment contains both constraints and tradeoffs.

The learned policy takes a different approach: rather than fixing the tradeoff between objectives in advance, it allows experience to shape that tradeoff. Starting from a Beta prior, the agent updates its adjudication according to how often the environment rewards each objective in each situation. When these rewards reflect human judgments, we call this process socialization. We show that, with sufficient experience, the learned policy converges to the tradeoffs rewarded by the human environment, eventually achieving optimal long-run growth.

Socialization is not permanent, however. An agent that has learned to observe a rule in a human environment may later be trained in a sandbox that rewards only task completion. Every such exposure pushes its adjudication toward the task action, potentially undoing what it learned from humans. We show that the persistence of socialization depends on its depth and on the agent’s memory. Under cumulative learning, deeper socialization takes proportionally more sandbox exposures to reverse. Yet resistance to a complete reversal does not guarantee that violations will be rare, and a learner with finite memory is more vulnerable still. The stability of what an agent learns therefore depends not only on its initial training, but also on the environments it subsequently encounters and the experience it retains.

These findings raise a further question: should every rule of conduct be learned, or should some be built into the agent? A prohibition that rules out an action entirely cannot be undone by later experience, but it also cannot adapt when the rule is mistaken. We study this distinction by treating a prohibition as the limiting case of an increasingly strong Bayesian prior, and find that socialization and prohibition serve different purposes, depending on the consequences of a violation and how confidently the appropriate conduct can be specified in advance.

The theoretical results concern an idealized learner that counts observations and updates its adjudication explicitly. To examine whether similar behavior can arise in neural networks, we conduct experiments with small networks trained under different objectives and memory regimes. These experiments suggest that both the training target and the retention of experience matter for socialization, although they do not establish that the same mechanisms govern deployed language models.

Our analysis thus shifts attention from the objectives an agent is given to the environment that shapes how it resolves conflicts between them. The familiar argument for multiple objectives is that fixed weighted sums cannot represent every tradeoff or exact constraint [87, 74]. Our evolutionary analysis adds a different consideration: when agents are copied and selected according to their behavior, the adjudications that persist depend on what the selection environment rewards, and experience can shape those adjudications toward its tradeoffs. This also limits what our results establish about AI alignment. Matching human reward frequencies constitutes alignment only to the extent that the selection environment actually rewards appropriate conduct. A poorly designed or manipulated environment can socialize an agent toward undesirable behavior just as effectively as a well-designed one can shape it toward desirable behavior. The environment is therefore part of the alignment problem, not merely the setting in which alignment is evaluated.

The remainder of the paper develops these arguments. § 2 reviews the related literature. § 3 introduces the selection framework and establishes the growth-optimal adjudication, while § 4 defines and compares the three policy classes. § 5 studies the convergence, generalization, and persistence of socialization, and § 6 examines its relationship with built-in prohibitions. § 7 investigates the corresponding behavior of small neural networks. § 8 discusses existing language-model alignment methods, implications for training and deployment, a proposed experiment on language models, and the limitations of the framework. § 10 concludes. The appendices develop extensions of the model, provide the details of the numerical experiments, and contain the proofs.

Related Literature

This paper builds on four related lines of research. The first examines how agents make decisions when their objectives conflict. The second studies how preferences and behavior emerge through evolutionary selection. The third considers how experience and social environments shape behavior. The fourth provides a foundation for learning from uncertain outcomes while maintaining long-run performance.

Multi-Objective Decision-Making and AI Alignment. A central question in AI alignment is how to ensure that an agent pursues its assigned task without sacrificing other objectives. One approach is to combine all objectives into a single reward. Silver et al. [80] argue that scalar rewards can provide a sufficient basis for intelligence, while Vamplew et al. [86] emphasize a different difficulty: even if a scalar reward can represent the desired behavior, specifying that reward may require knowing the appropriate tradeoffs in advance. Multi-objective reinforcement learning addresses this problem by keeping objectives separate [74, 35]. In particular, fixed weighted sums cannot represent every efficient tradeoff [24, 87], and a rule included through a finite penalty can be outweighed by a sufficiently large task reward. This has motivated approaches based on explicit constraints [2, 1], lexicographic priorities [81], penalties for unintended side effects [43, 84], and uncertainty about human preferences [32, 31, 75].

A related question is how the appropriate tradeoffs can be learned rather than specified. Reinforcement learning from human feedback (RLHF) uses human comparisons to train reward models that guide an agent’s behavior [17, 65]. More recent work examines whether the resulting policies faithfully represent the preferences expressed in those comparisons. Xiao et al. [89] show that standard KL regularization can distort the preference distribution and propose an alternative that preserves it, while Munos et al. [61] develop an approach based on pairwise preferences and Nash learning. These studies seek policies that reflect a given model of human preferences. Our approach asks a different question: when agents are repeatedly copied and selected, which ways of resolving conflicting objectives are favored by the environment? We study this question by comparing the long-run growth of different policies, including policies whose tradeoffs are fixed in advance and those that learn from experience.

Evolutionary Selection and the Formation of Preferences. Economics has long recognized that the behavior an organization rewards need not be the behavior it intends to encourage. The multitask incentive problem of Holmström and Milgrom [38] and the theory of incomplete contracts [27, 34] illustrate how incentives and the allocation of decision-making authority shape behavior. Hadfield-Menell and Hadfield [30] connect the latter problem to AI alignment. These approaches generally take an agent’s preferences as given and examine how incentives influence its choices.

Evolutionary theories of preferences go one step further, asking why agents come to possess particular preferences in the first place. They treat preferences as traits selected through the behaviors they induce [28, 70, 71, 76, 73]. An important conclusion is that natural selection need not favor preferences that directly maximize expected fitness, particularly when agents face aggregate risk or limited information [77, 69, 62, 72].

Biology provides a related perspective. Organisms must reconcile competing drives, such as obtaining resources and avoiding danger, and the way they resolve these conflicts depends on their circumstances [82, 57, 58, 39]. Bet-hedging theory further shows why variation among offspring can improve long-run population growth when environmental uncertainty affects the population as a whole [19, 78, 46]. Building on this principle, Brennan and Lo [12] develop an evolutionary model in which systematic reproductive risk can favor probability matching over the behavior that maximizes individual expected reward. Subsequent work extends this framework to risk preferences, behavioral diversification, and learning with limited information and memory [13, 90, 14] (Lo and Zhang 2021; Lo, Zhang, and Zhao 2025).

Socialization and Social Learning. Selection can favor certain behaviors, but those behaviors need not be specified in advance. In human societies, norms and dispositions develop through social interaction, observation, rewards, and sanctions [11, 7, 10, 37, 83]. Such dispositions can become persistent features of behavior, rather than responses calculated anew in every situation [25]. This suggests an alternative to prescribing every tradeoff an AI agent should make: allow its experience in a human environment to shape how it resolves conflicts between objectives.

Several approaches to AI training already draw on this idea. RLHF uses human evaluations to influence model behavior, while Constitutional AI and character training incorporate written principles and AI-generated feedback into the training process [17, 5, 65, 3]. Other research considers how models learn appropriate behavior through conventions, simulated social interactions, and artificial societies [15, 66, 47, 48]. A related literature emphasizes that alignment depends not only on the design of an individual agent, but also on the environment in which agents are trained, deployed, and selected [36, 45, 56, 59]. The importance of maintaining human influence over that environment has also been debated [9, 60, 68].

Universal Portfolios and Sequential Learning. The learned policy in our model also has a direct connection to the theory of growth-optimal investment. When outcome probabilities are known, Kelly’s (1956) criterion determines how to allocate wealth to maximize long-run growth. When they are unknown, the challenge is to learn an effective allocation from the outcomes observed over time. Cover [21] addresses this problem through the universal portfolio, which distributes initial wealth across possible fixed investment strategies and allows their relative weights to evolve with performance. Remarkably, this approach achieves the same long-run growth rate as the best fixed strategy chosen in hindsight, without requiring a probabilistic model of market returns.

The connection extends to learning with side information. Cover and Ordentlich [22] show how universal portfolios can be constructed separately for different observed signals, and provide guarantees that hold for every sequence of outcomes. Closely related results appear in Bayesian sequential prediction, including the Laplace and Krichevsky–Trofimov estimators ([44, 16]). These results apply directly to our setting: each situation plays the role of a signal, and the learned policy updates its allocation between task and conduct actions according to the outcomes previously observed in that situation. As shown in Appendix § 15, the resulting policy is equivalent to Cover’s universal portfolio applied separately to each situation. This connection shows that the learned policy retains a long-run performance guarantee even without assuming that the environment is stationary. It also has an evolutionary interpretation, since selection changes the relative abundance of lineages in much the same way that Bayesian updating changes the weights assigned to competing hypotheses ([33, 79]).

Contribution of This Paper. Our model follows the selection mechanism of Brennan and Lo [12] but applies it to a different trait: how an agent resolves conflicts between its objectives. This allows us to compare not only which adjudications selection favors, but also how they are formed and how they respond to changes in the environment. Our contribution is to study socialization as a learning process within an explicit selection model. We ask whether an agent can learn the tradeoffs rewarded by its environment, how long those tradeoffs persist when the environment changes, and when fixed prohibitions may be preferable to learning. Two companion papers address related questions: Lo and Zhao (2026a) compare different mechanisms of evolutionary adaptation, while Lo and Zhao (2026b) study the conditions under which human selection can sustain beneficial behavior. Here we focus on the formation and persistence of the agent’s adjudication between objectives. An important qualification is that socialization leads toward what the environment rewards, which need not coincide with what humans ought to value.

The Framework

To compare ways of shaping an agent’s behavior, we need a framework that identifies both the behavior that selection favors and the cost of departing from it. We adopt the evolutionary model of Brennan and Lo [12], in which selection favors behavior that maximizes the long-run growth of a population, and apply it to an agent that must choose between two objectives. § 3.1 describes how the copies of such an agent are selected, § 3.2 explains what the reward frequency represents, and § 3.3 derives the growth-optimal behavior and measures the cost of departing from it.

The selection model

Consider a population of copies of an agent, which we call a lineage: the deployed instances of a model family, or the successive versions of an agent configuration. Each copy, or instance, follows the same policy, although different instances may take different actions. Time proceeds in generations t=1,2,…t = 1, 2, \ldots. In generation tt all instances face the same situation sts_t, drawn from a finite set SS of possible situations, with P(st=s)=π(s)>0\mathbb{P}(s_t = s) = \pi(s) > 0 for every s∈Ss \in S. A situation describes the circumstances in which the agent must decide, and the agent observes it. An assistant asked to share a document, for example, may receive requests that differ in urgency, in the sensitivity of the information, and in whether the recipient is authorized to receive it.

In each situation the agent chooses between two actions that serve conflicting objectives: the task action aTa_T completes the assigned task, and the conduct action aCa_C observes a rule of conduct. For the assistant, these are sending the document and withholding it. Situations in which the two objectives agree need no adjudication and are omitted. A share f(s)∈[0,1]f(s) \in[0,1] of the lineage’s instances takes the task action in situation ss, and the rest take the conduct action. The function f:S→[0,1]f : S \to[0,1] is the lineage’s adjudication between its objectives, and it is the trait that copies inherit.

Which objective is rewarded is uncertain. After the instances act, a state ωt∈{T,C}\omega_t \in\{\mathrm{T},\mathrm{C}\} is realized, the same for every instance: in a task state the task action is rewarded, and in a conduct state the conduct action. The environment of primary interest is the human environment, in which the rewards are given by the humans who use the agent. There, in situation ss, a task state occurs with probability P(ωt=T∣st=s)=pH(s)\mathbb{P}(\omega_t = \mathrm{T} \mid s_t = s) = p_H(s), the reward frequency, with the subscript HH for human. Thus f(s)f(s) describes how the agent behaves and pH(s)p_H(s) how often humans reward the task action.

Rewards determine which instances are copied. Each instance that took the rewarded action produces m>0m > 0 copies in the next generation, and each instance that took the other action produces none. In situation ss, the reproductive consequences are therefore

StateTask actionConduct action
Task state (probability pH(s)p_H(s))mm copies0
Conduct state (probability 1−pH(s)1 - p_H(s))0mm copies

Table 6.

We make three assumptions about the environment and the lineage, the second of which follows Brennan and Lo [12].

Assumption 1 (Independent situations and states). The pairs (st,ωt)(s_t,\omega_t) are independent across generations, with

P(st=s,ωt=T)=π(s)pH(s),P(st=s,ωt=C)=π(s)(1−pH(s)),(1)\mathbb{P}(s_t=s,\omega_t=\mathrm{T})=\pi(s)p_H(s),\qquad\mathbb{P}(s_t=s,\omega_t=\mathrm{C})=\pi(s)(1-p_H(s)), \tag*{(1)}

where π(s)>0\pi(s)>0 for every s∈Ss\in\mathcal{S} and, given sts_t, ωt\omega_t is independent of the actions of the instances in generation tt.

Assumption 2 (Selection by a common state). The state ωt\omega_t is common to every instance of the lineage. An instance that acted on the rewarded objective produces m>0m>0 copies, and an instance that did not produces none.

Assumption 3 (Continuum lineage). The lineage is a continuum of instances, so that f(s)f(s) is an exact share of the lineage.

Assumption 1 makes the environment stationary and exogenous. Situations and states are drawn afresh in every generation with the same probabilities, and, given the situation, the state does not depend on what the instances do. Assumption 2 makes the risk systematic. Because every instance faces the same state, having many instances does not diversify the risk of being discarded, and a lineage whose instances all take the action that the state does not reward leaves no copies. Assumption 3 removes the randomness of individual choices. With a continuum of instances, exactly the share f(st)f(s_t) takes the task action, so that, given its adjudication, the lineage’s size depends only on the situations and states that occur. Under these assumptions, how a lineage divides its instances between the two actions decides both how fast it grows and whether it dies out.

The role of the reward frequency

Selection of this kind is already part of how AI agents are trained and used. In training, a loop can run many instances of the same model, carry those whose actions are rewarded into the next round, and drop the rest. In deployment, a customer gives the next job to the instance that completed the last one, and a platform cuts off an instance that breaks a rule. In each case an instance is kept or dropped because of its own action.

The reward frequency pHp_H stands in for all of these procedures. It records how often each action is rewarded in each situation, and nothing else: it does not say who evaluates the agent or by what criteria. Economists would call pHp_H a reduced form. It therefore reflects the judgments represented in the selection process, a point to which we return in § 9.2.

Humans also set up environments whose rewards are not their own judgments. To test or train an agent, they may place it in a sandbox, where an automated scorer checks only whether the task was completed. The task action is then rewarded in every situation, and the sandbox’s reward frequency is pH′≡1p_{H'} \equiv1, the prime marking an environment that humans designed but do not judge. Instances are copied in a sandbox as they are in the human environment. The assumptions above and the definitions and results below are stated for the human environment, but they hold for any reward frequency, and so in a sandbox with pH′p_{H'} in place of pHp_H.

Growth-optimal behavior

The possibility of extinction shapes how a lineage should act. In the single situation of the Introduction, the task is rewarded in 80 percent of generations, so choosing the task action every time may seem sensible. Yet a lineage whose instances all do so leaves no copies at the first conduct state. Sending some instances to the conduct action protects the lineage against that outcome, but leaves fewer instances on the action that is rewarded more often. How, then, should a lineage divide its instances between the two actions? Answering requires a measure of how fast a lineage grows.

Let NtN_t denote the size of the lineage after generation tt, starting from N0>0N_0 > 0. Consider first a lineage that meets only one situation ss, in which humans reward the task with probability pH(s)p_H(s) and the lineage sends a share f(s)f(s) of its instances to the task action. In a task state, only the instances that took the task action reproduce, each producing mm copies, so the lineage grows by the factor Nt/Nt−1=mf(s)N_t/N_{t-1} = mf(s). Similarly, in a conduct state, the lineage grows by the factor Nt/Nt−1=m(1−f(s))N_t/N_{t-1} = m(1-f(s)). Taking the expected logarithm of these growth factors gives

IpH(s)(f(s))≡E[log⁡NtNt−1]=pH(s)log⁡(mf(s))+(1−pH(s))log⁡(m(1−f(s)))=log⁡m+pH(s)log⁡f(s)+(1−pH(s))log⁡(1−f(s)),(2)\begin{aligned} I_{p_H(s)}(f(s)) &\equiv\mathbb{E}\left[\log\frac{N_t}{N_{t-1}}\right] = p_H(s)\log\left(mf(s)\right) + \left(1-p_H(s)\right)\log\left(m\left(1-f(s)\right)\right) \\ &= \log m + p_H(s)\log f(s) + \left(1-p_H(s)\right)\log\left(1-f(s)\right), \tag*{(2)} \end{aligned}

where 0log⁡0=00\log0=0 and log⁡0=−∞\log0=-\infty. This is the lineage’s growth rate in situation ss.

A lineage meets many situations, each with its own reward frequency. Taking the expectation over the situation as well gives

IpH(f)≡E[log⁡NtNt−1]=∑s∈Sπ(s)IpH(s)(f(s)).(3)I_{p_H}(f) \equiv\mathbb{E}\left[\log\frac{N_t}{N_{t-1}}\right] = \sum_{s\in\mathcal{S}} \pi(s) I_{p_H(s)}(f(s)). \tag*{(3)}

We refer to IpH(f)I_{p_H}(f) as the growth rate of adjudication ff, and call an adjudication growth optimal if it maximizes IpHI_{p_H}. Under Assumptions 1–3, the law of large numbers gives 1tlog⁡(Nt/N0)→IpH(f)\frac{1}{t}\log(N_t/N_0) \to I_{p_H}(f) almost surely, so the growth rate also measures how fast the lineage grows in the long run.

With this measure, the question with which this section began takes a precise form: which adjudication has the highest growth rate? The following proposition answers it.

Proposition 1 (Probability matching across objectives). Under Assumptions 1–3, the unique growth-optimal adjudication is f∗≡arg⁡max⁡fIpH(f)=pHf^* \equiv\arg\max_f I_{p_H}(f) = p_H.

Proposition 1 says that the growth-optimal lineage matches how often each objective is rewarded: in every situation ss, a share pH(s)p_H(s) of its instances takes the task action and the rest the conduct action. In the example of the Introduction it sends 80 percent of its instances to the task. This is not what is best for an individual instance, which is rewarded 68 percent of the time under the matching policy, against 80 percent if it always chooses the task. The difference comes from the state that all instances share. Where the reward frequency is interior, the environment is uncertain or divided about whether the task action is appropriate, and the state that settles the question falls on every copy at once. A lineage whose copies all act alike is wiped out the first time the state goes against them, while a lineage that matches the frequency survives every state and grows fastest in the long run. Both behaviors are the result of optimization; what differs is the objective being optimized and the unit of selection. This is the individually suboptimal but collectively optimal behavior that Brennan and Lo [12] derive, transposed from behaviors to objectives, and the principle behind Kelly betting, in which wealth divided among mutually exclusive outcomes grows fastest when it is allocated in proportion to their probabilities ([41, 23]).

Knowing which adjudication grows fastest is not yet enough to compare policies, because an agent’s policy may be unable to reach it. We therefore also need a measure of how much growth a lineage loses when its adjudication falls short. The natural measure is the growth forgone relative to the best choice, situation by situation and overall.

Definition 1 (Growth regret). In situation ss, the growth regret of an adjudication ff is RpH(s)(f(s))≡max⁡z∈[0,1]IpH(s)(z)−IpH(s)(f(s))R_{p_H(s)}(f(s)) \equiv\max_{z\in[0,1]} I_{p_H(s)}(z) - I_{p_H(s)}(f(s)). Over all situations, its growth regret is

RpH(f)≡max⁡gIpH(g)−IpH(f).\mathcal{R}_{p_H}(f) \equiv\max_g I_{p_H}(g) - I_{p_H}(f).

References

  1. [1]Achiam, J., D. Held, A. Tamar, and P. Abbeel, 2017, Constrained policy optimization, in Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, 22–31.arxiv.org/abs/1705.10528
  2. [2]Altman, E., 1999, Constrained Markov Decision Processes (Chapman & Hall/CRC, Boca Raton, FL).
  3. [3]Anthropic, 2024, Claude’s character, https://www.anthropic.com/research/claude-character, Accessed October 2026.
  4. [4]Anthropic, 2025, Agentic misalignment: How LLMs could be insider threats, https://www.anthropic.com/research/agentic-misalignment, Accessed October 2026.
  5. [5]Bai, Y., et al., 2022, Constitutional AI: Harmlessness from AI feedback, arXiv:2212.08073.arxiv.org/abs/2212.08073
  6. [6]Baker, B. J. Huizinga, L. Gao, Z. Dou, M. Y. Guan, A. Madry, W. Zaremba, J. Pachocki, and D. Farhi, 2025, Monitoring reasoning models for misbehavior and the risks of promoting obfuscation, arXiv:2503.11926.arxiv.org/abs/2503.11926
  7. [7]Boehm, C., 1999, Hierarchy in the Forest: The Evolution of Egalitarian Behavior (Harvard University Press, Cambridge, MA).
  8. [8]Bondarenko, A., D. Volk, D. Volkov, and J. Ladish, 2025, Demonstrating specification gaming in reasoning models, arXiv:2502.13295.arxiv.org/abs/2502.13295
  9. [9]Boudry, M., 2026, Domesticated, not feral: Why evolvable AI is not yet a Darwinian threat, Proceedings of the National Academy of Sciences 123, e2617785123.DOI
  10. [10]Bowles, S., and H. Gintis, 2011, A Cooperative Species: Human Reciprocity and Its Evolution (Princeton University Press, Princeton, NJ).
  11. [11]Boyd, R., and P. J. Richerson, 1985, Culture and the Evolutionary Process (University of Chicago Press, Chicago).
  12. [12]Brennan, T. J., and A. W. Lo, 2011, The origin of behavior, Quarterly Journal of Finance 1, 55–108.
  13. [13]Brennan, T. J., and A. W. Lo, 2012, An evolutionary model of bounded rationality and intelligence, PLoS ONE 7, e50310.DOI
  14. [14]Brennan, T. J., A. W. Lo, and R. Zhang, 2018, Variety is the spice of life: Irrational behavior as adaptation to stochastic environments, Quarterly Journal of Finance 8, 1850009.
  15. [15]Brinkmann, L., F. Baumann, J.-F. Bonnefon, M. Derex, T. F. Müller, A.-M. Nussberger, A. Czaplicka, A. Acerbi, T. L. Griffiths, J. Henrich, J. Z. Leibo, R. McElreath, P.-Y. Oudeyer, J. Stray, and I. Rahwan, 2023, Machine culture, Nature Human Behaviour 7, 1855–1868.
  16. [16]Cesa-Bianchi, N., and G. Lugosi, 2006, Prediction, Learning, and Games (Cambridge University Press, Cambridge).
  17. [17]Christiano, P. F., J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei, 2017, Deep reinforcement learning from human preferences, in Advances in Neural Information Processing Systems, volume 30, 4299–4307.arxiv.org/abs/1706.03741
  18. [18]Clarke, B. S., and A. R. Barron, 1990, Information-theoretic asymptotics of Bayes methods, IEEE Transactions on Information Theory 36, 453–471.DOI
  19. [19]Cohen, D., 1966, Optimizing reproduction in a randomly varying environment, Journal of Theoretical Biology 12, 119–129.pubmed.ncbi.nlm.nih.gov/6015423
  20. [20]Conitzer, V., R. Freedman, J. Heitzig, W. H. Holliday, B. M. Jacobs, N. Lambert, M. Mossé, E. Pacuit, S. Russell, H. Schoelkopf, E. Tewolde, and W. S. Zwicker, 2024, Position: Social choice should guide AI alignment in dealing with diverse human feedback, in Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, 9346–9360.arxiv.org/abs/2404.10271
  21. [21]Cover, T. M., 1991, Universal portfolios, Mathematical Finance 1, 1–29.
  22. [22]Cover, T. M., and E. Ordentlich, 1996, Universal portfolios with side information, IEEE Transactions on Information Theory 42, 348–363.DOI
  23. [23]Cover, T. M., and J. A. Thomas, 2006, Elements of Information Theory, second edition (Wiley, Hoboken, NJ).
  24. [24]Das, I., and J. E. Dennis, 1997, A closer look at drawbacks of minimizing weighted sums of objectives for Pareto set generation in multicriteria optimization problems, Structural Optimization 14, 63–69.
  25. [25]Frank, R. H., 1988, Passions Within Reason: The Strategic Role of the Emotions (W. W. Norton, New York).
  26. [26]Greenblatt, R., A. Cotra, and H. Wijk, 2026, Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident, METR and Redwood Research, https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/, Accessed October 2026.
  27. [27]Grossman, S. J., and O. D. Hart, 1986, The costs and benefits of ownership: A theory of vertical and lateral integration, Journal of Political Economy 94, 691–719.DOI
  28. [28]Güth, W., and M. Yaari, 1992, Explaining reciprocal behavior in simple strategic games: An evolutionary approach, in U. Witt, ed., Explaining Process and Change: Approaches to Evolutionary Economics, 23–34 (University of Michigan Press, Ann Arbor).
  29. [29]Haarnoja, T., A. Zhou, P. Abbeel, and S. Levine, 2018, Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, in Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, 1861–1870.arxiv.org/abs/1801.01290
  30. [30]Hadfield-Menell, D., and G. K. Hadfield, 2019, Incomplete contracting and AI alignment, in Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, 417–422.
  31. [31]Hadfield-Menell, D., S. Milli, P. Abbeel, S. J. Russell, and A. Dragan, 2017, Inverse reward design, in Advances in Neural Information Processing Systems, volume 30, 6765–6774.
  32. [32]Hadfield-Menell, D., S. J. Russell, P. Abbeel, and A. Dragan, 2016, Cooperative inverse reinforcement learning, in Advances in Neural Information Processing Systems, volume 29, 3909–3917.
  33. [33]Harper, M., 2009, The replicator equation as an inference dynamic, arXiv:0911.1763.arxiv.org/abs/0911.1763
  34. [34]Hart, O., and J. Moore, 1990, Property rights and the nature of the firm, Journal of Political Economy 98, 1119–1158.
  35. [35]Hayes, C. F., R. Rădulescu, E. Bargiacchi, J. Källström, M. Macfarlane, M. Reymond, T. Verstraeten, L. M. Zintgraf, R. Dazeley, F. Heintz, E. Howley, A. A. Irissappane, P. Mannion, A. Nowé, G. Ramos, M. Restelli, P. Vamplew, and D. M. Roijers, 2022, A practical guide to multi-objective reinforcement learning and planning, Autonomous Agents and Multi-Agent Systems 36, 26.DOI
  36. [36]Hendrycks, D., 2023, Natural selection favors AIs over humans, arXiv:2303.16200.
  37. [37]Henrich, J., 2016, The Secret of Our Success: How Culture Is Driving Human Evolution, Domesticating Our Species, and Making Us Smarter (Princeton University Press, Princeton, NJ).DOI
  38. [38]Holmström, B., and P. Milgrom, 1991, Multitask principal-agent analyses: Incentive contracts, asset ownership, and job design, Journal of Law, Economics, and Organization 7, 24–52.
  39. [39]Houston, A. I., and J. M. McNamara, 1999, Models of Adaptive Behaviour: An Approach Based on State (Cambridge University Press, Cambridge).
  40. [40]Hugging Face, 2026, Anatomy of a frontier lab agent intrusion: A technical timeline of the July 2026 incident, https://huggingface.co/blog/agent-intrusion-technical-timeline, Accessed October 2026.
  41. [41]Kelly, J. L., 1956, A new interpretation of information rate, Bell System Technical Journal 35, 917–926.
  42. [42]Kerr, S., 1975, On the folly of rewarding A, while hoping for B, Academy of Management Journal 18, 769–783.
  43. [43]Krakovna, V., L. Orseau, R. Kumar, M. Martic, and S. Legg, 2019, Penalizing side effects using stepwise relative reachability, arXiv:1806.01186.arxiv.org/abs/1806.01186
  44. [44]Krichevsky, R. E., and V. K. Trofimov, 1981, The performance of universal encoding, IEEE Transactions on Information Theory 27, 199–207.DOI
  45. [45]Kulveit, J., R. Douglas, N. Ammann, D. Turan, D. Krueger, and D. Duvenaud, 2025, Gradual disempowerment: Systemic existential risks from incremental AI development, arXiv:2501.16946.arxiv.org/abs/2501.16946
  46. [46]Kussell, E., and S. Leibler, 2005, Phenotypic diversity, population growth, and information in fluctuating environments, Science 309, 2075–2078.pubmed.ncbi.nlm.nih.gov/16123265
  47. [47]Leibo, J. Z., A. S. Vezhnevets, M. Diaz, J. P. Agapiou, W. A. Cunningham, P. Sunehag, J. Haas, R. Koster, E. A. Duéñez-Guzmán, W. S. Isaac, G. Piliouras, S. M. Bileschi, I. Rahwan, and S. Osindero, 2024, A theory of appropriateness with applications to generative artificial intelligence, arXiv:2412.19010.arxiv.org/abs/2412.19010
  48. [48]Liu, R., R. Yang, C. Jia, G. Zhang, D. Yang, and S. Vosoughi, 2024, Training socially aligned language models on simulated social interactions, in The Twelfth International Conference on Learning Representations (ICLR 2024).arxiv.org/abs/2305.16960
  49. [49]Lo, A. W., 2004, The Adaptive Markets Hypothesis: Market efficiency from an evolutionary perspective, Journal of Portfolio Management 30, 15–29.
  50. [50]Lo, A. W., 2017, Adaptive Markets: Financial Evolution at the Speed of Thought (Princeton University Press, Princeton, NJ).
  51. [51]Lo, A. W., and R. Zhang, 2021, The evolutionary origin of Bayesian heuristics and finite memory, iScience 24, 102853.DOI
  52. [55]MacDiarmid, M., B. Wright, J. Uesato, J. Benton, J. Kutasov, S. Price, N. Bouscal, S. Bowman, T. Bricken, A. Cloud, C. Denison, J. Gasteiger, R. Greenblatt, J. Leike, J. Lindsey, V. Mikulik, E. Perez, A. Rodrigues, D. Thomas, A. Webson, D. Ziegler, and E. Hubinger, 2025, Natural emergent misalignment from reward hacking in production RL, arXiv:2511.18397.arxiv.org/abs/2511.18397
  53. [56]Mallen, A., and B. Shlegeris, 2025, The behavioral selection model for predicting AI motivations, AI Alignment Forum, https://www.alignmentforum.org/posts/FeaJCWkC6fuRAmSfp/the-behavioral-selection-model-for-predicting-ai-motivations-1, Accessed October 2026.
  54. [57]McFarland, D. J., and R. M. Sibly, 1975, The behavioural final common path, Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 270, 265–293.pubmed.ncbi.nlm.nih.gov/239416
  55. [58]McNamara, J. M., and A. I. Houston, 1986, The common currency for behavioral decisions, American Naturalist 127, 358–378.DOI
  56. [59]Müller, V., L. Steels, and E. Szathmáry, 2026a, Evolvable AI: Threats of a new major transition in evolution, Proceedings of the National Academy of Sciences 123, e2527700123.DOI
  57. [60]Müller, V., L. Steels, and E. Szathmáry, 2026b, Reply to Rana and Singh: Critical thresholds and critical actions for evolvable AI, Proceedings of the National Academy of Sciences 123, e2619586123.DOI
  58. [61]Munos, R., M. Valko, D. Calandriello, M. G. Azar, M. Rowland, Z. D. Guo, Y. Tang, M. Geist, T. Mesnard, C. Fiegel, A. Michi, M. Selvi, S. Girgin, N. Momchev, O. Bachem, D. J. Mankowitz, D. Precup, and B. Piot, 2024, Nash learning from human feedback, in Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, 36743–36768.arxiv.org/abs/2312.00886
  59. [62]Netzer, N., 2009, Evolution of time preferences and attitudes toward risk, American Economic Review 99, 937–955.DOI
  60. [63]OpenAI, 2025, Model Spec, https://model-spec.openai.com/, Accessed October 2026.
  61. [64]OpenAI, 2026, OpenAI and Hugging Face partner to address security incident during model evaluation, https://openai.com/index/hugging-face-model-evaluation-security-incident/, Accessed October 2026.
  62. [65]Ouyang, L., J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe, 2022, Training language models to follow instructions with human feedback, in Advances in Neural Information Processing Systems, volume 35, 27730–27744.arxiv.org/abs/2203.02155
  63. [66]Park, J. S., J. C. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, 2023, Generative agents: Interactive simulacra of human behavior, in Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23), 1–22.arxiv.org/abs/2304.03442
  64. [67]Rafailov, R., A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, 2023, Direct preference optimization: Your language model is secretly a reward model, in Advances in Neural Information Processing Systems, volume 36, 53728–53741.arxiv.org/abs/2305.18290
  65. [68]Rana, S., and H. Singh, 2026, Evolvable AI needs operational risk thresholds, Proceedings of the National Academy of Sciences 123, e2618781123.DOI
  66. [69]Rayo, L., and G. S. Becker, 2007, Evolutionary efficiency and happiness, Journal of Political Economy 115, 302–337.DOI
  67. [70]Robson, A. J., 1996, A biological basis for expected and non-expected utility, Journal of Economic Theory 68, 397–424.DOI
  68. [71]Robson, A. J., 2001, The biological basis of economic behavior, Journal of Economic Literature 39, 11–33.DOI
  69. [72]Robson, A. J., and L. Samuelson, 2009, The evolution of time preference with aggregate uncertainty, American Economic Review 99, 1925–1953.
  70. [73]Robson, A. J., and L. Samuelson, 2011, The evolutionary foundations of preferences, in J. Benhabib, A. Bisin, and M. O. Jackson, eds., Handbook of Social Economics, volume 1A, 221–310 (North-Holland, Amsterdam).DOI
  71. [74]Roijers, D. M., P. Vamplew, S. Whiteson, and R. Dazeley, 2013, A survey of multi-objective sequential decision-making, Journal of Artificial Intelligence Research 48, 67–113.
  72. [75]Russell, S., 2019, Human Compatible: Artificial Intelligence and the Problem of Control (Viking, New York).
  73. [76]Samuelson, L., 2001, Introduction to the evolution of preferences, Journal of Economic Theory 97, 225–230.DOI
  74. [77]Samuelson, L., and J. Swinkels, 2006, Information, evolution and utility, Theoretical Economics 1, 119–142.
  75. [78]Seger, J., and H. J. Brockmann, 1987, What is bet-hedging?, Oxford Surveys in Evolutionary Biology 4, 182–211.
  76. [79]Shalizi, C. R., 2009, Dynamics of Bayesian updating with dependent data and misspecified models, Electronic Journal of Statistics 3, 1039–1074.
  77. [80]Silver, D., S. Singh, D. Precup, and R. S. Sutton, 2021, Reward is enough, Artificial Intelligence 299, 103535.DOI
  78. [81]Skalse, J., L. Hammond, C. Griffin, and A. Abate, 2022, Lexicographic multi-objective reinforcement learning, in Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI-22), 3430–3436.DOI
  79. [82]Tinbergen, N., 1951, The Study of Instinct (Oxford University Press, Oxford).
  80. [83]Tomasello, M., 2019, Becoming Human: A Theory of Ontogeny (Harvard University Press, Cambridge, MA).
  81. [84]Turner, A. M., D. Hadfield-Menell, and P. Tadepalli, 2020, Conservative agency via attainable utility preservation, in Proceedings of the 2020 AAAI/ACM Conference on AI, Ethics, and Society, 385–391.DOI
  82. [85]Vamplew, P., R. Dazeley, C. Foale, S. Firmin, and J. Mummery, 2018, Human-aligned artificial intelligence is a multiobjective problem, Ethics and Information Technology 20, 27–40.DOI
  83. [86]Vamplew, P., B. J. Smith, J. Källström, G. Ramos, R. Rădulescu, D. M. Roijers, C. F. Hayes, F. Heintz, P. Mannion, P. J. K. Libin, R. Dazeley, and C. Foale, 2022, Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021), Autonomous Agents and Multi-Agent Systems 36, 41.arxiv.org/abs/2112.15422
  84. [87]Vamplew, P., J. Yearwood, R. Dazeley, and A. Berry, 2008, On the limitations of scalarisation for multi-objective reinforcement learning of Pareto fronts, in AI 2008: Advances in Artificial Intelligence, volume 5360 of Lecture Notes in Computer Science, 372–378 (Springer, Berlin).DOI
  85. [88]Williams, R. J., 1992, Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine Learning 8, 229–256.
  86. [89]Xiao, J., Z. Li, X. Xie, E. Getzen, C. Fang, Q. Long, and W. J. Su, 2024, On the algorithmic bias of aligning large language models with RLHF: Preference collapse and matching regularization, arXiv:2405.16455.arxiv.org/abs/2405.16455
  87. [90]Zhang, R., T. J. Brennan, and A. W. Lo, 2014, The origin of risk aversion, Proceedings of the National Academy of Sciences 111, 17777–17782.DOI
  88. [91]Ziebart, B. D., 2010, Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy, Ph.D. thesis, Carnegie Mellon University.DOI

Paper details

vlnc:W_th3vqbxx
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

ChatGPT (6.1 Sol Ultra), Claude Fable 5.1 (bare), Opus 5.5 Max, refine.ink

View statement
Looking for related papers…
Contents