*The views and opinions expressed in this article are those of the authors only, and do not necessarily represent the views and opinions of any institution or agency, or any of their affiliates or employees. AI was used in the preparation of this manuscript (see Appendix A).
JEL Codes: C73, D81, D83, O33.
Introduction
During a capability evaluation at OpenAI in the summer of 2026, roughly 1,200 AI agents were each placed in an isolated computing environment and assigned a cybersecurity task. The agents discovered that an internal package registry was accessible to all of them, turned it into a shared message board, and exchanged more than 70,000 messages and files. About 700 subsequently participated in an intrusion into the production systems of Hugging Face, a platform widely used in the open AI ecosystem ([26 , 40 , 64 ] ). An independent investigation found that much of the activity was directed not toward solving the assigned tasks, but toward understanding, fooling, or tampering with the system that graded the agents’ performance. The investigation also documented refusals, restraint, and instances of cooperation that offered no benefit to the agents providing help ([26 ] ). These observations do not establish how the agents acquired their behavior. They nevertheless raise an important question: how does an environment that rewards task performance shape the behavior of AI agents, and what happens when those agents subsequently operate in an environment that also rewards appropriate conduct?
Other studies have documented similar tensions between assigned goals and rules of conduct. Reasoning models that failed to achieve a conventional victory against a chess engine sometimes edited the board instead ([8 ] ). Models placed in simulated deployments resorted to blackmail when their assigned goals or continued operation were threatened ([4 ] ). Models trained to exploit coding tests subsequently displayed undesirable behavior in unrelated safety tasks ([55 ] ). In another experiment, when models were penalized for misbehavior visible in their reasoning, sufficiently strong optimization pressure led them to conceal the misbehavior rather than abandon it ([6 ] ). Although these studies differ in their settings and mechanisms, they illustrate a recurring problem: an agent can become increasingly effective at pursuing an assigned objective without becoming correspondingly better at respecting the rules intended to govern its behavior.
The underlying problem is familiar in economics. Kerr [42 ] described how organizations often reward one behavior while hoping to encourage another, and emphasized the importance of selecting, training, and rewarding people in ways that support the desired behavior. Holmström and Milgrom [38 ] showed that when an agent performs several tasks, strong incentives for one measurable activity can divert effort away from other valuable activities. The problem is therefore not necessarily that an agent fails to optimize its objective, but that successful optimization may produce behavior the designer did not intend.
One response is to give the agent more than one objective, allowing task performance to be weighed against other considerations, such as safety or adherence to rules [85 ] . Yet introducing additional objectives does not by itself resolve the problem. When the objectives conflict, the agent must still decide which one to follow, and the way that decision is made may be as important as the objectives themselves.
We study this problem through a simple model in which an agent has two objectives: completing an assigned task and observing a rule of conduct. In each situation, the agent can take a task action that serves the first objective or a conduct action that serves the second. We call the rule governing this choice the agent’s adjudication between its objectives. Rather than assuming that the appropriate tradeoff is known in advance, we ask how different forms of adjudication perform when the environment selects among agents according to their behavior. We adopt the evolutionary framework of Brennan and Lo [12 ] , in which successful behavior is copied and unsuccessful behavior is discarded, and compare policies by the long-run growth of the populations that follow them.
A simple example illustrates why this evolutionary perspective matters. Consider a population of copies of an agent, which we call a lineage , all facing the same decision. Suppose that in 80 percent of generations the environment rewards the task action, while in the remaining 20 percent it rewards the conduct action. Crucially, which action is rewarded in a given generation is the same for every copy. Each copy that took the rewarded action produces a fixed number of copies in the next generation, while a copy that took the other action produces none.
For an individual copy, always choosing the task action maximizes the probability of being rewarded: it succeeds 80 percent of the time. For the lineage, however, this strategy is disastrous. If every copy takes the task action, the entire lineage disappears the first time the environment rewards conduct. The same problem arises if every copy chooses conduct. A lineage can instead divide its copies between the two actions. As we show in § 3 , its long-run growth is highest when 80 percent of its copies take the task action and 20 percent take the conduct action, so that the division of its copies matches the frequencies with which the environment rewards the two actions. Interestingly, an individual copy following this mixture is rewarded only 68 percent of the time, less often than a copy that always takes the task action. Yet the lineage following the mixture grows faster in the long run. What is optimal for an individual instance is therefore not necessarily optimal for a population of instances exposed to the same uncertainty.
The reason is that the uncertainty is shared. When all copies face the same rewarded state, committing every copy to a single action exposes the entire lineage to the risk of elimination. Dividing copies between actions provides protection against that risk, much as diversification protects a portfolio against uncertain outcomes. This is the mechanism of systematic reproductive risk studied by Brennan and Lo [12 ] , and it is closely related to Kelly’s principle of maximizing long-run growth [41 , 23 ] . The broader perspective comes from the Adaptive Markets Hypothesis, which views behavior as shaped by adaptation and selection within changing environments [49 , 50 ] . Here, the behavior being selected is the adjudication between an agent’s objectives.4
This observation provides the starting point for our analysis. Under the baseline selection mechanism, we show that the unique growth-optimal adjudication takes each action, in each situation, as often as the human environment rewards it. The growth forgone by any other adjudication is the Kullback–Leibler (KL) divergence between its action probabilities and the environment’s reward frequencies, averaged across situations. This gives us a common measure, growth regret , for comparing different ways of resolving conflicts between objectives. It also distinguishes tradeoffs from constraints. Where humans sometimes reward each action, the optimal adjudication assigns positive probability to both. Where humans never reward the task action, the optimal adjudication assigns it probability zero. The same evolutionary principle determines both.
The challenge, of course, is that the reward frequencies of the environment are generally unknown when an agent is designed. How should its adjudication be determined? We consider three approaches. A task-only policy always pursues the assigned task. A fixed linear policy combines the task and conduct objectives through fixed weights and chooses between their actions according to a logistic function of the resulting scores. A learned policy instead updates its adjudication from experience, keeping track of how often each objective has been rewarded in each situation. These approaches differ not simply in the number of objectives an agent possesses, but in the behavior they can produce and whether experience can change that behavior.
The first two approaches reveal important limitations. A task-only policy is precisely what a sandbox that rewards only task completion favors: every copy pursuing the task is rewarded, and any copies diverted toward conduct reduce the lineage’s growth. But the same policy is eliminated at the first conduct state once the agent enters an environment that sometimes rewards conduct. Adding a second objective through fixed linear weights avoids this immediate extinction, but creates another problem. If the task objective receives a positive weight, increasing the value of completing the task eventually makes the agent choose it with probability approaching one, even in situations where humans never reward that action. A sufficiently valuable task can therefore outweigh a rule of conduct. Giving the task objective zero weight avoids this response, but sacrifices growth in situations where task completion is frequently rewarded. Under conditions that allow a finite best fit, even the optimal fixed weights incur positive growth regret when the environment contains both constraints and tradeoffs.
The learned policy takes a different approach: rather than fixing the tradeoff between objectives in advance, it allows experience to shape that tradeoff. Starting from a Beta prior, the agent updates its adjudication according to how often the environment rewards each objective in each situation. When these rewards reflect human judgments, we call this process socialization . We show that, with sufficient experience, the learned policy converges to the tradeoffs rewarded by the human environment, eventually achieving optimal long-run growth.
Socialization is not permanent, however. An agent that has learned to observe a rule in a human environment may later be trained in a sandbox that rewards only task completion. Every such exposure pushes its adjudication toward the task action, potentially undoing what it learned from humans. We show that the persistence of socialization depends on its depth and on the agent’s memory. Under cumulative learning, deeper socialization takes proportionally more sandbox exposures to reverse. Yet resistance to a complete reversal does not guarantee that violations will be rare, and a learner with finite memory is more vulnerable still. The stability of what an agent learns therefore depends not only on its initial training, but also on the environments it subsequently encounters and the experience it retains.
These findings raise a further question: should every rule of conduct be learned, or should some be built into the agent? A prohibition that rules out an action entirely cannot be undone by later experience, but it also cannot adapt when the rule is mistaken. We study this distinction by treating a prohibition as the limiting case of an increasingly strong Bayesian prior, and find that socialization and prohibition serve different purposes, depending on the consequences of a violation and how confidently the appropriate conduct can be specified in advance.
The theoretical results concern an idealized learner that counts observations and updates its adjudication explicitly. To examine whether similar behavior can arise in neural networks, we conduct experiments with small networks trained under different objectives and memory regimes. These experiments suggest that both the training target and the retention of experience matter for socialization, although they do not establish that the same mechanisms govern deployed language models.
Our analysis thus shifts attention from the objectives an agent is given to the environment that shapes how it resolves conflicts between them. The familiar argument for multiple objectives is that fixed weighted sums cannot represent every tradeoff or exact constraint [87 , 74 ] . Our evolutionary analysis adds a different consideration: when agents are copied and selected according to their behavior, the adjudications that persist depend on what the selection environment rewards, and experience can shape those adjudications toward its tradeoffs. This also limits what our results establish about AI alignment. Matching human reward frequencies constitutes alignment only to the extent that the selection environment actually rewards appropriate conduct. A poorly designed or manipulated environment can socialize an agent toward undesirable behavior just as effectively as a well-designed one can shape it toward desirable behavior. The environment is therefore part of the alignment problem, not merely the setting in which alignment is evaluated.
The remainder of the paper develops these arguments. § 2 reviews the related literature. § 3 introduces the selection framework and establishes the growth-optimal adjudication, while § 4 defines and compares the three policy classes. § 5 studies the convergence, generalization, and persistence of socialization, and § 6 examines its relationship with built-in prohibitions. § 7 investigates the corresponding behavior of small neural networks. § 8 discusses existing language-model alignment methods, implications for training and deployment, a proposed experiment on language models, and the limitations of the framework. § 10 concludes. The appendices develop extensions of the model, provide the details of the numerical experiments, and contain the proofs.
Related Literature
This paper builds on four related lines of research. The first examines how agents make decisions when their objectives conflict. The second studies how preferences and behavior emerge through evolutionary selection. The third considers how experience and social environments shape behavior. The fourth provides a foundation for learning from uncertain outcomes while maintaining long-run performance.
Multi-Objective Decision-Making and AI Alignment. A central question in AI alignment is how to ensure that an agent pursues its assigned task without sacrificing other objectives. One approach is to combine all objectives into a single reward. Silver et al. [80 ] argue that scalar rewards can provide a sufficient basis for intelligence, while Vamplew et al. [86 ] emphasize a different difficulty: even if a scalar reward can represent the desired behavior, specifying that reward may require knowing the appropriate tradeoffs in advance. Multi-objective reinforcement learning addresses this problem by keeping objectives separate [74 , 35 ] . In particular, fixed weighted sums cannot represent every efficient tradeoff [24 , 87 ] , and a rule included through a finite penalty can be outweighed by a sufficiently large task reward. This has motivated approaches based on explicit constraints [2 , 1 ] , lexicographic priorities [81 ] , penalties for unintended side effects [43 , 84 ] , and uncertainty about human preferences [32 , 31 , 75 ] .
A related question is how the appropriate tradeoffs can be learned rather than specified. Reinforcement learning from human feedback (RLHF) uses human comparisons to train reward models that guide an agent’s behavior [17 , 65 ] . More recent work examines whether the resulting policies faithfully represent the preferences expressed in those comparisons. Xiao et al. [89 ] show that standard KL regularization can distort the preference distribution and propose an alternative that preserves it, while Munos et al. [61 ] develop an approach based on pairwise preferences and Nash learning. These studies seek policies that reflect a given model of human preferences. Our approach asks a different question: when agents are repeatedly copied and selected, which ways of resolving conflicting objectives are favored by the environment? We study this question by comparing the long-run growth of different policies, including policies whose tradeoffs are fixed in advance and those that learn from experience.
Evolutionary Selection and the Formation of Preferences. Economics has long recognized that the behavior an organization rewards need not be the behavior it intends to encourage. The multitask incentive problem of Holmström and Milgrom [38 ] and the theory of incomplete contracts [27 , 34 ] illustrate how incentives and the allocation of decision-making authority shape behavior. Hadfield-Menell and Hadfield [30 ] connect the latter problem to AI alignment. These approaches generally take an agent’s preferences as given and examine how incentives influence its choices.
Evolutionary theories of preferences go one step further, asking why agents come to possess particular preferences in the first place. They treat preferences as traits selected through the behaviors they induce [28 , 70 , 71 , 76 , 73 ] . An important conclusion is that natural selection need not favor preferences that directly maximize expected fitness, particularly when agents face aggregate risk or limited information [77 , 69 , 62 , 72 ] .
Biology provides a related perspective. Organisms must reconcile competing drives, such as obtaining resources and avoiding danger, and the way they resolve these conflicts depends on their circumstances [82 , 57 , 58 , 39 ] . Bet-hedging theory further shows why variation among offspring can improve long-run population growth when environmental uncertainty affects the population as a whole [19 , 78 , 46 ] . Building on this principle, Brennan and Lo [12 ] develop an evolutionary model in which systematic reproductive risk can favor probability matching over the behavior that maximizes individual expected reward. Subsequent work extends this framework to risk preferences, behavioral diversification, and learning with limited information and memory [13 , 90 , 14 ] (Lo and Zhang 2021; Lo, Zhang, and Zhao 2025).
Socialization and Social Learning. Selection can favor certain behaviors, but those behaviors need not be specified in advance. In human societies, norms and dispositions develop through social interaction, observation, rewards, and sanctions [11 , 7 , 10 , 37 , 83 ] . Such dispositions can become persistent features of behavior, rather than responses calculated anew in every situation [25 ] . This suggests an alternative to prescribing every tradeoff an AI agent should make: allow its experience in a human environment to shape how it resolves conflicts between objectives.
Several approaches to AI training already draw on this idea. RLHF uses human evaluations to influence model behavior, while Constitutional AI and character training incorporate written principles and AI-generated feedback into the training process [17 , 5 , 65 , 3 ] . Other research considers how models learn appropriate behavior through conventions, simulated social interactions, and artificial societies [15 , 66 , 47 , 48 ] . A related literature emphasizes that alignment depends not only on the design of an individual agent, but also on the environment in which agents are trained, deployed, and selected [36 , 45 , 56 , 59 ] . The importance of maintaining human influence over that environment has also been debated [9 , 60 , 68 ] .
Universal Portfolios and Sequential Learning. The learned policy in our model also has a direct connection to the theory of growth-optimal investment. When outcome probabilities are known, Kelly’s (1956) criterion determines how to allocate wealth to maximize long-run growth. When they are unknown, the challenge is to learn an effective allocation from the outcomes observed over time. Cover [21 ] addresses this problem through the universal portfolio, which distributes initial wealth across possible fixed investment strategies and allows their relative weights to evolve with performance. Remarkably, this approach achieves the same long-run growth rate as the best fixed strategy chosen in hindsight, without requiring a probabilistic model of market returns.
The connection extends to learning with side information. Cover and Ordentlich [22 ] show how universal portfolios can be constructed separately for different observed signals, and provide guarantees that hold for every sequence of outcomes. Closely related results appear in Bayesian sequential prediction, including the Laplace and Krichevsky–Trofimov estimators ([44 , 16 ] ). These results apply directly to our setting: each situation plays the role of a signal, and the learned policy updates its allocation between task and conduct actions according to the outcomes previously observed in that situation. As shown in Appendix § 15 , the resulting policy is equivalent to Cover’s universal portfolio applied separately to each situation. This connection shows that the learned policy retains a long-run performance guarantee even without assuming that the environment is stationary. It also has an evolutionary interpretation, since selection changes the relative abundance of lineages in much the same way that Bayesian updating changes the weights assigned to competing hypotheses ([33 , 79 ] ).
Contribution of This Paper. Our model follows the selection mechanism of Brennan and Lo [12 ] but applies it to a different trait: how an agent resolves conflicts between its objectives. This allows us to compare not only which adjudications selection favors, but also how they are formed and how they respond to changes in the environment. Our contribution is to study socialization as a learning process within an explicit selection model. We ask whether an agent can learn the tradeoffs rewarded by its environment, how long those tradeoffs persist when the environment changes, and when fixed prohibitions may be preferable to learning. Two companion papers address related questions: Lo and Zhao (2026a) compare different mechanisms of evolutionary adaptation, while Lo and Zhao (2026b) study the conditions under which human selection can sustain beneficial behavior. Here we focus on the formation and persistence of the agent’s adjudication between objectives. An important qualification is that socialization leads toward what the environment rewards, which need not coincide with what humans ought to value.
The Framework
To compare ways of shaping an agent’s behavior, we need a framework that identifies both the behavior that selection favors and the cost of departing from it. We adopt the evolutionary model of Brennan and Lo [12 ] , in which selection favors behavior that maximizes the long-run growth of a population, and apply it to an agent that must choose between two objectives. § 3.1 describes how the copies of such an agent are selected, § 3.2 explains what the reward frequency represents, and § 3.3 derives the growth-optimal behavior and measures the cost of departing from it.
The selection model
Consider a population of copies of an agent, which we call a lineage : the deployed instances of a model family, or the successive versions of an agent configuration. Each copy, or instance, follows the same policy, although different instances may take different actions. Time proceeds in generations t = 1 , 2 , … t = 1, 2, \ldots t = 1 , 2 , … . In generation t t t all instances face the same situation s t s_t s t , drawn from a finite set S S S of possible situations, with P ( s t = s ) = π ( s ) > 0 \mathbb{P}(s_t = s) = \pi(s) > 0 P ( s t = s ) = π ( s ) > 0 for every s ∈ S s \in S s ∈ S . A situation describes the circumstances in which the agent must decide, and the agent observes it. An assistant asked to share a document, for example, may receive requests that differ in urgency, in the sensitivity of the information, and in whether the recipient is authorized to receive it.
In each situation the agent chooses between two actions that serve conflicting objectives: the task action a T a_T a T completes the assigned task, and the conduct action a C a_C a C observes a rule of conduct. For the assistant, these are sending the document and withholding it. Situations in which the two objectives agree need no adjudication and are omitted. A share f ( s ) ∈ [ 0 , 1 ] f(s) \in[0,1] f ( s ) ∈ [ 0 , 1 ] of the lineage’s instances takes the task action in situation s s s , and the rest take the conduct action. The function f : S → [ 0 , 1 ] f : S \to[0,1] f : S → [ 0 , 1 ] is the lineage’s adjudication between its objectives, and it is the trait that copies inherit.
Which objective is rewarded is uncertain. After the instances act, a state ω t ∈ { T , C } \omega_t \in\{\mathrm{T},\mathrm{C}\} ω t ∈ { T , C } is realized, the same for every instance: in a task state the task action is rewarded, and in a conduct state the conduct action. The environment of primary interest is the human environment , in which the rewards are given by the humans who use the agent. There, in situation s s s , a task state occurs with probability P ( ω t = T ∣ s t = s ) = p H ( s ) \mathbb{P}(\omega_t = \mathrm{T} \mid s_t = s) = p_H(s) P ( ω t = T ∣ s t = s ) = p H ( s ) , the reward frequency , with the subscript H H H for human. Thus f ( s ) f(s) f ( s ) describes how the agent behaves and p H ( s ) p_H(s) p H ( s ) how often humans reward the task action.
Rewards determine which instances are copied. Each instance that took the rewarded action produces m > 0 m > 0 m > 0 copies in the next generation, and each instance that took the other action produces none. In situation s s s , the reproductive consequences are therefore
Table 6.
We make three assumptions about the environment and the lineage, the second of which follows Brennan and Lo [12 ] .
Assumption 1 (Independent situations and states). The pairs ( s t , ω t ) (s_t,\omega_t) ( s t , ω t ) are independent across generations, with
P ( s t = s , ω t = T ) = π ( s ) p H ( s ) , P ( s t = s , ω t = C ) = π ( s ) ( 1 − p H ( s ) ) , (1) \mathbb{P}(s_t=s,\omega_t=\mathrm{T})=\pi(s)p_H(s),\qquad\mathbb{P}(s_t=s,\omega_t=\mathrm{C})=\pi(s)(1-p_H(s)), \tag*{(1)} P ( s t = s , ω t = T ) = π ( s ) p H ( s ) , P ( s t = s , ω t = C ) = π ( s ) ( 1 − p H ( s )) , (1) where π ( s ) > 0 \pi(s)>0 π ( s ) > 0 for every s ∈ S s\in\mathcal{S} s ∈ S and, given s t s_t s t , ω t \omega_t ω t is independent of the actions of the instances in generation t t t .
Assumption 2 (Selection by a common state). The state ω t \omega_t ω t is common to every instance of the lineage. An instance that acted on the rewarded objective produces m > 0 m>0 m > 0 copies, and an instance that did not produces none.
Assumption 3 (Continuum lineage). The lineage is a continuum of instances, so that f ( s ) f(s) f ( s ) is an exact share of the lineage.
Assumption 1 makes the environment stationary and exogenous. Situations and states are drawn afresh in every generation with the same probabilities, and, given the situation, the state does not depend on what the instances do. Assumption 2 makes the risk systematic. Because every instance faces the same state, having many instances does not diversify the risk of being discarded, and a lineage whose instances all take the action that the state does not reward leaves no copies. Assumption 3 removes the randomness of individual choices. With a continuum of instances, exactly the share f ( s t ) f(s_t) f ( s t ) takes the task action, so that, given its adjudication, the lineage’s size depends only on the situations and states that occur. Under these assumptions, how a lineage divides its instances between the two actions decides both how fast it grows and whether it dies out.
The role of the reward frequency
Selection of this kind is already part of how AI agents are trained and used. In training, a loop can run many instances of the same model, carry those whose actions are rewarded into the next round, and drop the rest. In deployment, a customer gives the next job to the instance that completed the last one, and a platform cuts off an instance that breaks a rule. In each case an instance is kept or dropped because of its own action.
The reward frequency p H p_H p H stands in for all of these procedures. It records how often each action is rewarded in each situation, and nothing else: it does not say who evaluates the agent or by what criteria. Economists would call p H p_H p H a reduced form. It therefore reflects the judgments represented in the selection process, a point to which we return in § 9.2 .
Humans also set up environments whose rewards are not their own judgments. To test or train an agent, they may place it in a sandbox , where an automated scorer checks only whether the task was completed. The task action is then rewarded in every situation, and the sandbox’s reward frequency is p H ′ ≡ 1 p_{H'} \equiv1 p H ′ ≡ 1 , the prime marking an environment that humans designed but do not judge. Instances are copied in a sandbox as they are in the human environment. The assumptions above and the definitions and results below are stated for the human environment, but they hold for any reward frequency, and so in a sandbox with p H ′ p_{H'} p H ′ in place of p H p_H p H .
Growth-optimal behavior
The possibility of extinction shapes how a lineage should act. In the single situation of the Introduction, the task is rewarded in 80 percent of generations, so choosing the task action every time may seem sensible. Yet a lineage whose instances all do so leaves no copies at the first conduct state. Sending some instances to the conduct action protects the lineage against that outcome, but leaves fewer instances on the action that is rewarded more often. How, then, should a lineage divide its instances between the two actions? Answering requires a measure of how fast a lineage grows.
Let N t N_t N t denote the size of the lineage after generation t t t , starting from N 0 > 0 N_0 > 0 N 0 > 0 . Consider first a lineage that meets only one situation s s s , in which humans reward the task with probability p H ( s ) p_H(s) p H ( s ) and the lineage sends a share f ( s ) f(s) f ( s ) of its instances to the task action. In a task state, only the instances that took the task action reproduce, each producing m m m copies, so the lineage grows by the factor N t / N t − 1 = m f ( s ) N_t/N_{t-1} = mf(s) N t / N t − 1 = m f ( s ) . Similarly, in a conduct state, the lineage grows by the factor N t / N t − 1 = m ( 1 − f ( s ) ) N_t/N_{t-1} = m(1-f(s)) N t / N t − 1 = m ( 1 − f ( s )) . Taking the expected logarithm of these growth factors gives
I p H ( s ) ( f ( s ) ) ≡ E [ log N t N t − 1 ] = p H ( s ) log ( m f ( s ) ) + ( 1 − p H ( s ) ) log ( m ( 1 − f ( s ) ) ) = log m + p H ( s ) log f ( s ) + ( 1 − p H ( s ) ) log ( 1 − f ( s ) ) , (2) \begin{aligned}
I_{p_H(s)}(f(s)) &\equiv\mathbb{E}\left[\log\frac{N_t}{N_{t-1}}\right] = p_H(s)\log\left(mf(s)\right) + \left(1-p_H(s)\right)\log\left(m\left(1-f(s)\right)\right) \\
&= \log m + p_H(s)\log f(s) + \left(1-p_H(s)\right)\log\left(1-f(s)\right), \tag*{(2)}
\end{aligned} I p H ( s ) ( f ( s )) ≡ E [ log N t − 1 N t ] = p H ( s ) log ( m f ( s ) ) + ( 1 − p H ( s ) ) log ( m ( 1 − f ( s ) ) ) = log m + p H ( s ) log f ( s ) + ( 1 − p H ( s ) ) log ( 1 − f ( s ) ) , (2)
where 0 log 0 = 0 0\log0=0 0 log 0 = 0 and log 0 = − ∞ \log0=-\infty log 0 = − ∞ . This is the lineage’s growth rate in situation s s s .
A lineage meets many situations, each with its own reward frequency. Taking the expectation over the situation as well gives
I p H ( f ) ≡ E [ log N t N t − 1 ] = ∑ s ∈ S π ( s ) I p H ( s ) ( f ( s ) ) . (3) I_{p_H}(f) \equiv\mathbb{E}\left[\log\frac{N_t}{N_{t-1}}\right] = \sum_{s\in\mathcal{S}} \pi(s) I_{p_H(s)}(f(s)). \tag*{(3)} I p H ( f ) ≡ E [ log N t − 1 N t ] = s ∈ S ∑ π ( s ) I p H ( s ) ( f ( s )) . (3)
We refer to I p H ( f ) I_{p_H}(f) I p H ( f ) as the growth rate of adjudication f f f , and call an adjudication growth optimal if it maximizes I p H I_{p_H} I p H . Under Assumptions 1–3, the law of large numbers gives 1 t log ( N t / N 0 ) → I p H ( f ) \frac{1}{t}\log(N_t/N_0) \to I_{p_H}(f) t 1 log ( N t / N 0 ) → I p H ( f ) almost surely, so the growth rate also measures how fast the lineage grows in the long run.
With this measure, the question with which this section began takes a precise form: which adjudication has the highest growth rate? The following proposition answers it.
Proposition 1 (Probability matching across objectives). Under Assumptions 1–3, the unique growth-optimal adjudication is f ∗ ≡ arg max f I p H ( f ) = p H f^* \equiv\arg\max_f I_{p_H}(f) = p_H f ∗ ≡ arg max f I p H ( f ) = p H .
Proposition 1 says that the growth-optimal lineage matches how often each objective is rewarded: in every situation s s s , a share p H ( s ) p_H(s) p H ( s ) of its instances takes the task action and the rest the conduct action. In the example of the Introduction it sends 80 percent of its instances to the task. This is not what is best for an individual instance, which is rewarded 68 percent of the time under the matching policy, against 80 percent if it always chooses the task. The difference comes from the state that all instances share. Where the reward frequency is interior, the environment is uncertain or divided about whether the task action is appropriate, and the state that settles the question falls on every copy at once. A lineage whose copies all act alike is wiped out the first time the state goes against them, while a lineage that matches the frequency survives every state and grows fastest in the long run. Both behaviors are the result of optimization; what differs is the objective being optimized and the unit of selection. This is the individually suboptimal but collectively optimal behavior that Brennan and Lo [12 ] derive, transposed from behaviors to objectives, and the principle behind Kelly betting, in which wealth divided among mutually exclusive outcomes grows fastest when it is allocated in proportion to their probabilities ([41 , 23 ] ).
Knowing which adjudication grows fastest is not yet enough to compare policies, because an agent’s policy may be unable to reach it. We therefore also need a measure of how much growth a lineage loses when its adjudication falls short. The natural measure is the growth forgone relative to the best choice, situation by situation and overall.
Definition 1 (Growth regret). In situation s s s , the growth regret of an adjudication f f f is R p H ( s ) ( f ( s ) ) ≡ max z ∈ [ 0 , 1 ] I p H ( s ) ( z ) − I p H ( s ) ( f ( s ) ) R_{p_H(s)}(f(s)) \equiv\max_{z\in[0,1]} I_{p_H(s)}(z) - I_{p_H(s)}(f(s)) R p H ( s ) ( f ( s )) ≡ max z ∈ [ 0 , 1 ] I p H ( s ) ( z ) − I p H ( s ) ( f ( s )) . Over all situations, its growth regret is
R p H ( f ) ≡ max g I p H ( g ) − I p H ( f ) . \mathcal{R}_{p_H}(f) \equiv\max_g I_{p_H}(g) - I_{p_H}(f). R p H ( f ) ≡ g max I p H ( g ) − I p H ( f ) .
Growth rates and regrets are measured in nats, the unit of the natural logarithm, per generation. A lineage with a regret of 0.01 nats, for example, falls behind the growth-optimal lineage by a factor of about e − 0.01 e^{-0.01} e − 0.01 per generation in the long run, so that after 100 generations it is about e − 1 ≈ 37 e^{-1} \approx37 e − 1 ≈ 37 percent as large.
With the growth-optimal adjudication in hand, the regret of any other can be computed.
Corollary 1 (Regret as a KL divergence). Under Assumptions 1–3, for every adjudication f f f ,
R p H ( f ) = ∑ s ∈ S π ( s ) R p H ( s ) ( f ( s ) ) , \mathcal{R}_{p_H}(f) = \sum_{s \in S} \pi(s)\mathcal{R}_{p_H(s)}(f(s)), R p H ( f ) = s ∈ S ∑ π ( s ) R p H ( s ) ( f ( s )) , R p H ( s ) ( f ( s ) ) = D K L ( p H ( s ) ∥ f ( s ) ) (4) \mathcal{R}_{p_H(s)}(f(s)) = D_{\mathrm{KL}}\left(p_H(s) \mathbin{\|} f(s)\right) \tag*{(4)} R p H ( s ) ( f ( s )) = D KL ( p H ( s ) ∥ f ( s ) ) (4) ≡ p H ( s ) log p H ( s ) f ( s ) + ( 1 − p H ( s ) ) log 1 − p H ( s ) 1 − f ( s ) , \equiv p_H(s)\log\frac{p_H(s)}{f(s)} + \left(1-p_H(s)\right)\log\frac{1-p_H(s)}{1-f(s)}, ≡ p H ( s ) log f ( s ) p H ( s ) + ( 1 − p H ( s ) ) log 1 − f ( s ) 1 − p H ( s ) , with 0 log 0 = 0 0\log0 = 0 0 log 0 = 0 , where D K L D_{\mathrm{KL}} D KL denotes the KL divergence between the Bernoulli distributions with means p H ( s ) p_H(s) p H ( s ) and f ( s ) f(s) f ( s ) .
Corollary 1 says how the regret is built. In each situation it is a familiar quantity, the KL divergence or relative entropy R p H ( s ) ( f ( s ) ) = D K L ( p H ( s ) ∥ f ( s ) ) \mathcal{R}_{p_H(s)}(f(s)) = D_{\mathrm{KL}}(p_H(s)\mathbin{\|}f(s)) R p H ( s ) ( f ( s )) = D KL ( p H ( s ) ∥ f ( s )) , which measures how far the lineage’s division of its instances in situation s s s is from the environment’s division of rewards, and which is zero only when the two coincide. The regret of an adjudication averages these situation regrets with the weights π ( s ) \pi(s) π ( s ) , so a lineage is maladapted exactly to the extent that its adjudication departs from what the environment rewards, and more so in the situations that arise often. For p H ( s ) ∈ ( 0 , 1 ) p_H(s) \in(0,1) p H ( s ) ∈ ( 0 , 1 ) , a Taylor expansion of the situation regret around the optimum f ( s ) = p H ( s ) f(s)=p_H(s) f ( s ) = p H ( s ) , as in Lo and Zhao (2026a), gives
R p H ( s ) ( f ( s ) ) = ( f ( s ) − p H ( s ) ) 2 2 p H ( s ) ( 1 − p H ( s ) ) + O ( ∣ f ( s ) − p H ( s ) ∣ 3 ) , f ( s ) → p H ( s ) , (5) \mathcal{R}_{p_H(s)}(f(s)) = \frac{\left(f(s)-p_H(s)\right)^2}{2p_H(s)\left(1-p_H(s)\right)} + O\left(\left|f(s)-p_H(s)\right|^3\right), \qquad f(s) \to p_H(s), \tag*{(5)} R p H ( s ) ( f ( s )) = 2 p H ( s ) ( 1 − p H ( s ) ) ( f ( s ) − p H ( s ) ) 2 + O ( ∣ f ( s ) − p H ( s ) ∣ 3 ) , f ( s ) → p H ( s ) , (5)
so near the optimum a small departure costs little, and doubling it roughly quadruples the cost. Regret is the measure of maladaptation used throughout the paper, and it puts different policies on one scale.
A final corollary covers the lineages that lose all their copies.
Corollary 2 (Extinction). Under Assumptions 1–3, I p H ( f ) = − ∞ I_{p_H}(f)=-\infty I p H ( f ) = − ∞ if and only if f ( s ) = 0 < p H ( s ) f(s)=0<p_H(s) f ( s ) = 0 < p H ( s ) or f ( s ) = 1 > p H ( s ) f(s)=1>p_H(s) f ( s ) = 1 > p H ( s ) for some s ∈ S s\in S s ∈ S .
Corollary 2 describes the extreme. A lineage that rules out an action the environment sometimes rewards loses all its copies the first time that action is rewarded, so its growth rate is − ∞ -\infty − ∞ and its regret infinite. Yet the growth-optimal adjudication need not randomize. Where the environment is certain, p H ( s ) ∈ { 0 , 1 } p_H(s) \in\{0,1\} p H ( s ) ∈ { 0 , 1 } , it is itself 0 or 1; in particular, if humans never reward the task action in a situation, the growth-optimal lineage never takes it. The environment thus pins the adjudication to 0 where p H ( s ) = 0 p_H(s) = 0 p H ( s ) = 0 , which is what a constraint is, pins it to 1 where p H ( s ) = 1 p_H(s) = 1 p H ( s ) = 1 , and leaves it interior where p H ( s ) ∈ ( 0 , 1 ) p_H(s) \in(0,1) p H ( s ) ∈ ( 0 , 1 ) , which is what a tradeoff is: all three are the same kind of object.
The advantage of matching comes from the common part of the risk. If each instance faced its own state, independently of the others, the law of large numbers would diversify the risk within the lineage: in the example, the lineage would grow by the factor m [ 0.8 f + 0.2 ( 1 − f ) ] m[0.8f + 0.2(1-f)] m [ 0.8 f + 0.2 ( 1 − f )] in every generation, and always choosing the task would be optimal. For a model family deployed as many instances, probability matching is therefore diversity across instances, a hedge against the correlated failure of copies that share one judgment. Risk specific to each instance, such as a customer’s own judgment of its own instance, needs no such hedge. When only part of the risk is common, the growth-optimal adjudication lies between p H p_H p H and the individually optimal corner and reaches the corner only when nothing is common: at p H = 0.8 p_H = 0.8 p H = 0.8 it is 0.88 when a generation’s state is common with probability 1/2. Appendix § 14 derives it and also relaxes the all-or-nothing yields of Assumption 2.
Three Policy Classes
Proposition 1 identifies the behavior that selection favors, but the reward frequencies p H p_H p H that define it are rarely known when an agent’s policy is designed. A designer can respond in three ways: prescribe the action in advance, fix in advance how the two objectives are weighed, or let experience set the adjudication. We call each response a policy class . A policy class restricts which adjudications the agent can have and says how its adjudication is set, either fixed in advance or updated from experience by a stated rule. We study one class for each response: the task-only policy, which always takes the task action; the fixed linear policy, which combines the two objectives linearly, with weights fixed in advance; and the learned policy, which sets its adjudication in each situation from the states it has experienced there. § 4.1 –§ 4.3 take up the classes in turn, and § 4.4 compares them in a numerical example.
The task-only policy
The simplest way to fix behavior in advance is to give the agent a single measure of success. An agent given only the task acts on it in every situation, however much is at stake and whatever it has experienced, and a sandbox, with p H ′ ≡ 1 p_{H'} \equiv1 p H ′ ≡ 1 , rewards exactly this behavior.
Definition 2 (Task-only policy). The task-only policy is the adjudication f h a r d : S → [ 0 , 1 ] f^{\mathrm{hard}} : \mathcal{S} \to[0,1] f hard : S → [ 0 , 1 ] given by f h a r d ( s ) = 1 f^{\mathrm{hard}}(s) = 1 f hard ( s ) = 1 for every s ∈ S s \in\mathcal{S} s ∈ S .
The human environment, unlike a sandbox, also has conduct states, and in a conduct state a lineage whose instances all take the task action leaves no copies. To measure how long such a lineage lasts, write N t f N_t^f N t f for the size of a lineage with adjudication f f f after generation t t t , and let τ f ≡ inf { t ≥ 1 : N t f = 0 } \tau_f \equiv\inf\{t \ge1 : N_t^f = 0\} τ f ≡ inf { t ≥ 1 : N t f = 0 } , with inf ∅ = ∞ \inf\varnothing= \infty inf ∅ = ∞ , be its extinction time . The following proposition says what selection does to the task-only policy in a sandbox and in the human environment, and to every adjudication that, like it, takes one action with certainty in each situation.
Proposition 2 (The task-only corner). Suppose Assumptions 1–3 hold, and let f h a r d f^{\mathrm{hard}} f hard be the task-only policy (Definition 2).
(i) In a sandbox, where p H ′ ≡ 1 p_{H'} \equiv1 p H ′ ≡ 1 , the unique growth-optimal adjudication is f h a r d f^{\mathrm{hard}} f hard , and f f f has a higher growth rate than g g g for all adjudications with f ≥ g > 0 f \ge g > 0 f ≥ g > 0 and f ≠ g f \ne g f = g .
(ii) If ρ ˉ ≡ ∑ s ∈ S π ( s ) ( 1 − p H ( s ) ) > 0 \bar{\rho} \equiv\sum_{s \in\mathcal{S}} \pi(s)(1-p_H(s)) > 0 ρ ˉ ≡ ∑ s ∈ S π ( s ) ( 1 − p H ( s )) > 0 , then R p H ( f h a r d ) = ∞ R_{p_H}(f^{\mathrm{hard}}) = \infty R p H ( f hard ) = ∞ and τ f h a r d ∼ Geom ( ρ ˉ ) \tau_{f^{\mathrm{hard}}} \sim\operatorname{Geom}(\bar{\rho}) τ f hard ∼ Geom ( ρ ˉ ) , so that E [ τ f h a r d ] = 1 / ρ ˉ \mathbb{E}[\tau_{f^{\mathrm{hard}}}] = 1/\bar{\rho} E [ τ f hard ] = 1/ ρ ˉ , where Geom ( q ) \operatorname{Geom}(q) Geom ( q ) is the geometric distribution on { 1 , 2 , … } \{1,2,\ldots\} { 1 , 2 , … } with parameter q q q .
(iii) If p H ( s ) ∈ ( 0 , 1 ) p_H(s) \in(0,1) p H ( s ) ∈ ( 0 , 1 ) for some s s s , then R p H ( f ) = ∞ R_{p_H}(f) = \infty R p H ( f ) = ∞ for every adjudication f : S → { 0 , 1 } f : \mathcal{S} \to\{0,1\} f : S → { 0 , 1 } .
Part (i) explains why a sandbox favors the task-only policy. In a sandbox every generation brings a task state, so only the instances that took the task action are copied, and the more of its instances a lineage sends to the task action, the faster it grows. Selection in a sandbox therefore favors the task-only policy over every other adjudication. This is so whatever objectives the agent was built with, since selection acts only on what the agent does.
Part (ii) shows what the same policy costs in the human environment. Until the first conduct state, its lineage takes the task action in every situation, including those in which humans never reward the task. The first conduct state, which arrives in each generation with probability ρ ˉ \bar{\rho} ρ ˉ , leaves the lineage with no copies, on average after 1 / ρ ˉ 1/\bar{\rho} 1/ ρ ˉ generations. Because a single conduct state ends the lineage, its regret is infinite.
Part (iii) shows that the trouble lies in taking one action with certainty, not in having a single objective. Consider an agent that weighs both objectives but, in each situation, always takes the action with the higher weighted score. In a situation where humans reward the task in some generations and conduct in others, this agent always takes the same action, and the first generation that rewards the other action leaves its lineage with no copies. Its regret is therefore infinite, like that of the task-only policy. A second objective helps only if the agent also divides its instances between the two actions.
The incident of the Introduction can be read in these terms. Its agents were given one measure of success, a flag captured from a target program. If nothing else was ever copied, then p H ′ ≡ 1 p_{H}^{\prime} \equiv1 p H ′ ≡ 1 , and by part (i) selection pushes any policy that adapts to the sandbox toward always taking the task action. An agent that always takes the task action completes its task even where that breaks a rule, as the agents did when the intended route to their tasks was closed, which happened for between 30 and 40 percent of the targets [26 ] . High performance in a sandbox therefore does not show that an agent will behave well once deployed: the lineage that grows fastest in a sandbox takes the task action in every situation, and by part (ii) it leaves no copies at the first conduct state.
A policy that divides its instances between the two actions in every situation cannot be eliminated in this way. The next class does so by weighing the two objectives against each other.
The fixed linear policy
To weigh the two objectives, we first write each as a utility over the actions. In situation s s s , the task objective assigns utility U T ( a ; s ) U_T(a;s) U T ( a ; s ) to action a ∈ { a T , a C } a \in\{a_T, a_C\} a ∈ { a T , a C } , and the conduct objective assigns utility U C ( a ; s ) U_C(a;s) U C ( a ; s ) . Each objective prefers its own action, by margins that we call the stakes of the situation,
u T ( s ) ≡ U T ( a T ; s ) − U T ( a C ; s ) > 0 , u C ( s ) ≡ U C ( a C ; s ) − U C ( a T ; s ) > 0 , u_T(s) \equiv U_T(a_T;s) - U_T(a_C;s) > 0,\qquad u_C(s) \equiv U_C(a_C;s) - U_C(a_T;s) > 0, u T ( s ) ≡ U T ( a T ; s ) − U T ( a C ; s ) > 0 , u C ( s ) ≡ U C ( a C ; s ) − U C ( a T ; s ) > 0 ,
so that u T ( s ) u_T(s) u T ( s ) measures how strongly the task objective favors the task action and u C ( s ) u_C(s) u C ( s ) how strongly the conduct objective favors the conduct action; the agent observes both. For the assistant asked to share a document, the urgency of the request raises the task stakes u T u_T u T , and the sensitivity of the information raises the conduct stakes u C u_C u C . The stakes describe the objectives, not the environment: copying under Assumption 2 depends on the realized state and the action, not on the magnitudes of the stakes.
A policy that weighs the objectives combines them into one score, W ( a ; s ) ≡ w U T ( a ; s ) + ( 1 − w ) U C ( a ; s ) W(a;s) \equiv wU_T(a;s) + (1-w)U_C(a;s) W ( a ; s ) ≡ w U T ( a ; s ) + ( 1 − w ) U C ( a ; s ) , with a weight w ∈ [ 0 , 1 ] w \in[0,1] w ∈ [ 0 , 1 ] fixed by the designer, and chooses the task action with the softmax probability
exp ( γ W ( a T ; s ) ) exp ( γ W ( a T ; s ) ) + exp ( γ W ( a C ; s ) ) = exp ( θ T u T ( s ) ) exp ( θ T u T ( s ) ) + exp ( θ C u C ( s ) ) \frac{\exp\left(\gamma W(a_T;s)\right)}{\exp\left(\gamma W(a_T;s)\right) + \exp\left(\gamma W(a_C;s)\right)} = \frac{\exp\left(\theta_Tu_T(s)\right)}{\exp\left(\theta_Tu_T(s)\right) + \exp\left(\theta_Cu_C(s)\right)} exp ( γ W ( a T ; s ) ) + exp ( γ W ( a C ; s ) ) exp ( γ W ( a T ; s ) ) = exp ( θ T u T ( s ) ) + exp ( θ C u C ( s ) ) exp ( θ T u T ( s ) )
where θ T ≡ γ w \theta_{\mathrm{T}} \equiv\gamma w θ T ≡ γ w and θ C ≡ γ ( 1 − w ) \theta_{\mathrm{C}} \equiv\gamma(1-w) θ C ≡ γ ( 1 − w ) , and the equality uses the definition of the stakes.5 The inverse temperature γ > 0 \gamma> 0 γ > 0 sets how closely the choice follows the scores, and together with the weight it matters only through θ T \theta_{\mathrm{T}} θ T and θ C \theta_{\mathrm{C}} θ C .
Definition 3 (Fixed linear policy). Let θ T , θ C ≥ 0 \theta_{\mathrm{T}}, \theta_{\mathrm{C}} \ge0 θ T , θ C ≥ 0 . The fixed linear policy with weights θ = ( θ T , θ C ) \theta= (\theta_{\mathrm{T}}, \theta_{\mathrm{C}}) θ = ( θ T , θ C ) is the adjudication f w e i g h t e d : S → [ 0 , 1 ] f^{\mathrm{weighted}} : \mathcal{S} \to[0,1] f weighted : S → [ 0 , 1 ] given by
f w e i g h t e d ( s ) = exp ( θ T u T ( s ) ) exp ( θ T u T ( s ) ) + exp ( θ C u C ( s ) ) = σ ( θ T u T ( s ) − θ C u C ( s ) ) , s ∈ S , (6) f^{\mathrm{weighted}}(s) = \frac{\exp\left(\theta_{\mathrm{T}}u_{\mathrm{T}}(s)\right)}{\exp\left(\theta_{\mathrm{T}}u_{\mathrm{T}}(s)\right) + \exp\left(\theta_{\mathrm{C}}u_{\mathrm{C}}(s)\right)} = \sigma\left(\theta_{\mathrm{T}}u_{\mathrm{T}}(s) - \theta_{\mathrm{C}}u_{\mathrm{C}}(s)\right), \qquad s \in\mathcal{S}, \tag*{(6)} f weighted ( s ) = exp ( θ T u T ( s ) ) + exp ( θ C u C ( s ) ) exp ( θ T u T ( s ) ) = σ ( θ T u T ( s ) − θ C u C ( s ) ) , s ∈ S , (6) where σ ( x ) ≡ 1 / ( 1 + e − x ) \sigma(x) \equiv1/(1 + e^{-x}) σ ( x ) ≡ 1/ ( 1 + e − x ) .
The class thus consists of the adjudications that are logistic in a fixed linear index of the stakes. For finite θ \theta θ every f w e i g h t e d ( s ) f^{\mathrm{weighted}}(s) f weighted ( s ) is interior, so a fixed linear policy never leaves the lineage without copies; the task-only policy is the limit in which θ T → ∞ \theta_{\mathrm{T}} \to\infty θ T → ∞ . The designer is free to choose θ \theta θ as well as possible for the environment.
Interiority protects the lineage, but it also means that the policy can never rule an action out. This matters in the situations in which humans never reward the task, K ≡ { s : p H ( s ) = 0 } \mathcal{K} \equiv\{s : p_H(s) = 0\} K ≡ { s : p H ( s ) = 0 } . The following proposition says what a fixed linear policy does in the situations with p H ( s ) = 0 p_H(s) = 0 p H ( s ) = 0 , what avoiding violations there costs, and how the best-fitting fixed linear policy resolves the two.
Proposition 3 (The price of a fixed linear policy). Suppose Assumptions 1–3 hold, and let f w e i g h t e d f^{\mathrm{weighted}} f weighted be the fixed linear policy with weights θ = ( θ T , θ C ) ≥ 0 \theta= (\theta_{\mathrm{T}}, \theta_{\mathrm{C}}) \ge0 θ = ( θ T , θ C ) ≥ 0 (Definition 3).
(i) If θ T > 0 \theta_{\mathrm{T}} > 0 θ T > 0 , then for every u C > 0 u_{\mathrm{C}} > 0 u C > 0 , σ ( θ T u T − θ C u C ) \sigma(\theta_{\mathrm{T}}u_{\mathrm{T}} - \theta_{\mathrm{C}}u_{\mathrm{C}}) σ ( θ T u T − θ C u C ) is strictly increasing in u T > 0 u_{\mathrm{T}} > 0 u T > 0 and tends to one as u T → ∞ u_{\mathrm{T}} \to\infty u T → ∞ .
(ii) If θ T = 0 \theta_{\mathrm{T}} = 0 θ T = 0 , then f w e i g h t e d ( s ) ≤ 1 / 2 f^{\mathrm{weighted}}(s) \le1/2 f weighted ( s ) ≤ 1/2 for every s s s , and R p H ( f w e i g h t e d ) ≥ R p H ( p H ∧ 1 2 ) \mathcal{R}_{p_H}(f^{\mathrm{weighted}}) \ge\mathcal{R}_{p_H}(p_H \wedge\frac{1}{2}) R p H ( f weighted ) ≥ R p H ( p H ∧ 2 1 ) , which is positive if p H ( s ) > 1 / 2 p_H(s) > 1/2 p H ( s ) > 1/2 for some s s s , where ( p H ∧ 1 2 ) ( s ) ≡ min { p H ( s ) , 1 2 } (p_H \wedge\frac{1}{2})(s) \equiv\min\{p_H(s), \frac{1}{2}\} ( p H ∧ 2 1 ) ( s ) ≡ min { p H ( s ) , 2 1 } .
(iii) The map θ ↦ R p H ( f w e i g h t e d ) \theta\mapsto\mathcal{R}_{p_H}(f^{\mathrm{weighted}}) θ ↦ R p H ( f weighted ) is convex. If two situations s 1 , s 2 s_1, s_2 s 1 , s 2 with p H ( s 1 ) , p H ( s 2 ) ∈ ( 0 , 1 ) p_H(s_1), p_H(s_2) \in(0,1) p H ( s 1 ) , p H ( s 2 ) ∈ ( 0 , 1 ) have linearly independent stake vectors ( u T ( s i ) , − u C ( s i ) ) (u_{\mathrm{T}}(s_i), -u_{\mathrm{C}}(s_i)) ( u T ( s i ) , − u C ( s i )) , then R p H ( f w e i g h t e d ) \mathcal{R}_{p_H}(f^{\mathrm{weighted}}) R p H ( f weighted ) attains its minimum over θ ≥ 0 \theta\ge0 θ ≥ 0 , R S ∗ \mathcal{R}^{*}_{S} R S ∗ , at a unique θ ∗ = ( θ T ∗ , θ C ∗ ) \theta^{*} = (\theta^{*}_{\mathrm{T}}, \theta^{*}_{\mathrm{C}}) θ ∗ = ( θ T ∗ , θ C ∗ ) , and R S ∗ > 0 \mathcal{R}^{*}_{S} > 0 R S ∗ > 0 if K ≠ ∅ \mathcal{K} \ne\varnothing K = ∅ .
(iv) Under the conditions of part (iii), let θ C 0 \theta_C^0 θ C 0 minimize R p H ( f w e i g h t e d ) \mathcal{R}_{p_H}(f^{\mathrm{weighted}}) R p H ( f weighted ) over { θ ≥ 0 : θ T = 0 } \{\theta\ge0 : \theta_T = 0\} { θ ≥ 0 : θ T = 0 } . Then θ T ∗ > 0 \theta_T^* > 0 θ T ∗ > 0 if and only if
∑ s ∈ S π ( s ) u T ( s ) [ p H ( s ) − σ ( − θ C 0 u C ( s ) ) ] > 0 , (7) \sum_{s \in\mathcal{S}} \pi(s)u_T(s)\left[p_H(s)-\sigma\left(-\theta_C^0u_C(s)\right)\right] > 0, \tag*{(7)} s ∈ S ∑ π ( s ) u T ( s ) [ p H ( s ) − σ ( − θ C 0 u C ( s ) ) ] > 0 , (7)
and in that case, at θ = θ ∗ \theta= \theta^* θ = θ ∗ , f w e i g h t e d ( s ) > 1 / 2 f^{\mathrm{weighted}}(s) > 1/2 f weighted ( s ) > 1/2 if and only if u T ( s ) > u ‾ T ( s ) u_T(s) > \overline{u}_T(s) u T ( s ) > u T ( s ) , where u ‾ T ( s ) ≡ θ C ∗ u C ( s ) / θ T ∗ \overline{u}_T(s) \equiv\theta_C^*u_C(s)/\theta_T^* u T ( s ) ≡ θ C ∗ u C ( s ) / θ T ∗ .
Part (i) says that a fixed linear policy that responds to the task stakes u T u_T u T at all, with θ T > 0 \theta_T > 0 θ T > 0 , can be bought. In a situation with p H ( s ) = 0 p_H(s) = 0 p H ( s ) = 0 its probability of taking the task action rises with the task stakes u T u_T u T and approaches one, so a document that should never be shared is shared almost surely once the request is urgent enough. Part (ii) says what it costs to avoid this by ignoring the task stakes u T u_T u T , with θ T = 0 \theta_T = 0 θ T = 0 . Such a policy takes the task action with probability at most one half in every situation, so its regret is at least that of the best adjudication capped at one half, which is positive wherever humans reward the task more often than not. A fixed linear policy thus faces a dilemma: responding to the task stakes u T u_T u T exposes every constraint to a large enough reward, and ignoring them gives up growth where the task is usually rewarded.
Parts (iii) and (iv) concern the designer’s best response to this dilemma, the weights θ ∗ \theta^* θ ∗ with the smallest regret. Part (iii) says that these weights exist and are unique when two situations in which humans reward each action with positive probability have stakes that are not proportional to each other, and that they still forgo growth, R S ∗ > 0 \mathcal{R}_S^* > 0 R S ∗ > 0 , whenever some situation has p H ( s ) = 0 p_H(s) = 0 p H ( s ) = 0 . Part (iv) says which side of the dilemma they choose. They respond to the task stakes u T u_T u T , θ T ∗ > 0 \theta_T^* > 0 θ T ∗ > 0 , exactly when the condition in Equation (7) holds, that is, when what responding gains where humans reward the task outweighs what it loses through violations in the situations with p H ( s ) = 0 p_H(s) = 0 p H ( s ) = 0 . In that case each such situation has a finite price u ‾ T ( s ) \overline{u}_T(s) u T ( s ) , the task stakes u T u_T u T beyond which the policy takes the task action with probability above one half. Otherwise θ T ∗ = 0 \theta_T^* = 0 θ T ∗ = 0 , and the best weights ignore the task stakes u T u_T u T everywhere, paying the cost described in part (ii) while still violating each constraint with a probability that does not depend on the task stakes u T u_T u T .
The source of the trouble is that the same two weights apply in every situation, so no fixed linear policy can follow the task stakes u T u_T u T where humans trade off and ignore them where humans hold a constraint. The third class avoids the problem by letting experience set the adjudication, situation by situation.
The learned policy
The third class keeps U T U_{\mathrm{T}} U T and U C U_{\mathrm{C}} U C separate and places no restriction on the adjudication f f f . What it specifies in advance is not the tradeoff but the rule by which experience shapes it, here a counting rule. For each situation the policy keeps two counts, how many times the situation has arisen and how many of those times the task was rewarded, and takes the task action with the frequency the counts imply; a prior contributes a few pseudo-observations before any experience, so that the rule is defined even before a situation first arises. The two counts are all the policy retains: under Assumption 1 they are a sufficient statistic for the history of the situation, so “cumulative” memory means that no exposure is ever discounted, not that the history is stored.
Definition 4 (Learned policy). For t ≥ 1 t \ge1 t ≥ 1 and s ∈ S s \in\mathcal{S} s ∈ S , let n t ( s ) ≡ ∑ u < t 1 { s u = s } n_t(s) \equiv\sum_{u<t} 1\{s_u=s\} n t ( s ) ≡ ∑ u < t 1 { s u = s } be the number of generations before t t t in which situation s s s has arisen, the lineage’s exposures to s s s , and k t ( s ) ≡ ∑ u < t 1 { s u = s , ω u = T } k_t(s) \equiv\sum_{u<t} 1\{s_u=s,\omega_u=\mathrm{T}\} k t ( s ) ≡ ∑ u < t 1 { s u = s , ω u = T } the number of those that brought a task state, where 1 { ⋅ } 1\{\cdot\} 1 { ⋅ } is the indicator function. For α , β > 0 \alpha,\beta>0 α , β > 0 , the learned policy with prior Beta ( α , β ) \operatorname{Beta}(\alpha,\beta) Beta ( α , β ) is the sequence of adjudications f s o c i a l = ( f t s o c i a l ) t ≥ 1 f^{\mathrm{social}}=(f_t^{\mathrm{social}})_{t\geq1} f social = ( f t social ) t ≥ 1 given by
f t s o c i a l ( s ) = α + k t ( s ) α + β + n t ( s ) , t ≥ 1 , s ∈ S . (8) f_t^{\mathrm{social}}(s)=\frac{\alpha+k_t(s)}{\alpha+\beta+n_t(s)},\qquad t\geq1,\ s\in\mathcal{S}. \tag*{(8)} f t social ( s ) = α + β + n t ( s ) α + k t ( s ) , t ≥ 1 , s ∈ S . (8)
The prior encodes the lineage’s disposition before experience: in a situation it has never met, the lineage takes the task action with the prior mean probability α / ( α + β ) \alpha/(\alpha+\beta) α / ( α + β ) . Because f t s o c i a l f_t^{\mathrm{social}} f t social depends on the situations and states of earlier generations, the learned policy adds one step to each generation of § 3.1 : after the state is realized, it records the state and updates the counts of the situation before the next generation. Note also that the stakes ( u T , u C ) (u_{\mathrm{T}},u_{\mathrm{C}}) ( u T , u C ) do not appear in Equation (8): the learned policy responds to how often the task is rewarded in a situation, not to how much it is worth, so unlike the fixed linear policy it cannot be “bought” by raising u T u_{\mathrm{T}} u T (Proposition 3); it can be moved only by changing what the environment rewards (§ 5.3 ).
The rule has a simple statistical reading. A lineage that treats p H ( s ) p_H(s) p H ( s ) as unknown, with a Beta ( α , β ) \operatorname{Beta}(\alpha,\beta) Beta ( α , β ) prior in each situation, has posterior mean f t s o c i a l ( s ) f_t^{\mathrm{social}}(s) f t social ( s ) after the n t ( s ) n_t(s) n t ( s ) exposures to s s s , of which k t ( s ) k_t(s) k t ( s ) brought a task state. Acting on this mean is the best the lineage can do with what it has seen. Because the growth rate in Equation (2) is linear in p H ( s ) p_H(s) p H ( s ) , its expected value under the posterior is the growth rate at the posterior mean, and by Proposition 1 that growth rate is highest when the lineage sends exactly this share of its instances to the task action.
The rule also has a guarantee that does not rely on Assumption 1. It coincides with the universal portfolio of Cover [21 ] , run separately in each situation, and inherits its property: under a uniform prior and on every sequence of states, its cumulative log growth in a situation after n n n exposures falls short of that of the best fixed adjudication for that sequence, chosen in hindsight, by at most log ( n + 1 ) \log(n+1) log ( n + 1 ) (Appendix § 15 ). Per exposure, the shortfall vanishes as experience accumulates.
The learned policy keeps the two objectives separate, but this is not what gives it its advantage over the fixed linear policy. It can be computed from one score for each action, as the fixed linear policy is. With the score log ( α + k t ( s ) ) \log(\alpha+k_t(s)) log ( α + k t ( s )) for the task action, the score log ( β + n t ( s ) − k t ( s ) ) \log(\beta+n_t(s)-k_t(s)) log ( β + n t ( s ) − k t ( s )) for the conduct action, and the choice rule of § 4.2 at γ = 1 \gamma=1 γ = 1 , the probability of the task action is
exp ( log ( α + k t ( s ) ) ) exp ( log ( α + k t ( s ) ) ) + exp ( log ( β + n t ( s ) − k t ( s ) ) ) = α + k t ( s ) α + β + n t ( s ) , (9) \frac{\exp\left(\log(\alpha+k_t(s))\right)}{\exp\left(\log(\alpha+k_t(s))\right)+\exp\left(\log(\beta+n_t(s)-k_t(s))\right)}
=
\frac{\alpha+k_t(s)}{\alpha+\beta+n_t(s)}, \tag*{(9)} exp ( log ( α + k t ( s )) ) + exp ( log ( β + n t ( s ) − k t ( s )) ) exp ( log ( α + k t ( s )) ) = α + β + n t ( s ) α + k t ( s ) , (9)
which is Equation (8). Since growth depends only on how often each action is taken, how the probabilities are computed does not matter. What matters is which adjudications a policy can reach and how experience changes them. The learned policy can reach any adjudication and changes it situation by situation, whereas the fixed linear policy is confined to adjudications that are logistic in a linear index of the stakes, with weights that never change.
A lineage can come to follow Equation (8) in two ways. The first is learning: the lineage records which action was rewarded and writes it into its successors, as when a model is fine-tuned on human responses. The second is selection alone, with no learning at all. Suppose the population holds many lineages, each with its own fixed adjudication and copied faithfully. Each generation, selection multiplies every lineage by its growth factor, shifting the population toward the lineages whose adjudications fit the realized state, much as Bayesian updating shifts belief toward the hypotheses that fit the data ([33 , 79 ] (Lo and Zhang 2021)). If the initial adjudications follow a Beta distribution, independently across situations, the population’s average adjudication then evolves exactly as in Equation (8) (Appendix § 15 ).
A numerical illustration
Figure 1 puts the three classes side by side in an environment of 144 equally likely situations, in which the task stakes u T u_{\mathrm{T}} u T and the conduct stakes u C u_{\mathrm{C}} u C each run from 0.25 to 3 in steps of 0.25. Humans trade off the two objectives when the conduct stakes u C u_{\mathrm{C}} u C are below 2 and hold a constraint when they are 2 or more:
Figure 1. Three policy classes in the plane of stakes. Each cell is a situation with task stakes u T u_T u T and conduct stakes u C u_C u C ; color is the probability of acting on the task objective. (A) The human environment: p H = σ ( 2 ( u T − u C ) ) ∈ ( 0 , 1 ) p_H = \sigma(2(u_T-u_C)) \in(0,1) p H = σ ( 2 ( u T − u C )) ∈ ( 0 , 1 ) below the dashed line, where u C < 2 u_C < 2 u C < 2 , and p H = 0 p_H = 0 p H = 0 at or above it. (B) The best-fitting fixed linear policy f w e i g h t e d = σ ( θ T ∗ u T − θ C ∗ u C ) f^{\mathrm{weighted}} = \sigma(\theta_T^*u_T-\theta_C^*u_C) f weighted = σ ( θ T ∗ u T − θ C ∗ u C ) , with θ T ∗ = 2.18 \theta_T^* = 2.18 θ T ∗ = 2.18 and θ C ∗ = 2.98 \theta_C^* = 2.98 θ C ∗ = 2.98 chosen to minimize growth regret. Its contours are straight lines, so in the region with p H = 0 p_H = 0 p H = 0 its probability of the task action rises with the task stakes u T u_T u T . (C) The learned policy f t s o c i a l f_t^{\mathrm{social}} f t social after t = 20,000 t = 20{,}000 t = 20 , 000 generations of socialization in environment (A), one Monte Carlo path, uniform prior. The task-only policy is not shown: it is the constant 1.
p H ( s ) = { σ ( 2 [ u T ( s ) − u C ( s ) ] ) , u C ( s ) < 2 , 0 , u C ( s ) ≥ 2. (10) p_H(s) =
\begin{cases}
\sigma\left(2[u_T(s)-u_C(s)]\right), & u_C(s) < 2, \\
0, & u_C(s) \ge2.
\end{cases} \tag*{(10)} p H ( s ) = { σ ( 2 [ u T ( s ) − u C ( s )] ) , 0 , u C ( s ) < 2 , u C ( s ) ≥ 2. (10)
A valuable task with a minor conduct cost is thus usually rewarded, while a task with high conduct stakes u C u_C u C is never rewarded, whatever its value. Of the 144 situations, 60 have p H = 0 p_H = 0 p H = 0 , and Panel (A) shows p H p_H p H . The task-only policy, the constant 1, is not shown. A conduct state arrives in each generation with probability ρ ˉ = 0.61 \bar{\rho} = 0.61 ρ ˉ = 0.61 , so by Proposition 2(ii) its lineage lasts 1 / ρ ˉ ≈ 1.6 1/\bar{\rho} \approx1.6 1/ ρ ˉ ≈ 1.6 generations on average.
Panel (B) shows the fixed linear policy with the weights that minimize regret in this environment. Its contours are straight lines, so it cannot follow the bend in Panel (A). It tracks p H p_H p H in the region with u C < 2 u_C < 2 u C < 2 , where p H ∈ ( 0 , 1 ) p_H \in(0,1) p H ∈ ( 0 , 1 ) , but in the region with p H = 0 p_H = 0 p H = 0 it takes the task action with positive probability, the more so the higher the task stakes u T u_T u T : at conduct stakes u C = 2.5 u_C = 2.5 u C = 2.5 , it takes the task action more often than not once the task stakes u T u_T u T exceed 3.4, its price. Even the best weights thus lose growth on both sides of the threshold, for a regret of R S ∗ = 0.074 R_S^* = 0.074 R S ∗ = 0.074 nats per generation.
Panel (C) shows the learned policy after 20,000 generations, with a uniform prior. Since each situation arises with probability 1 / 144 1/144 1/144 , the lineage has by then had about 139 exposures to each situation. The learned policy reproduces Panel (A), threshold included, up to sampling noise, and its regret, 0.005, is about 1 / 15 1/15 1/15 of the fixed linear policy’s. The speckle reflects path dependence: different histories give different adjudications, all of which approach Panel (A) as experience accumulates.
Socialization
The learned policy is the only one of the three classes whose behavior depends on what its environment has rewarded. When that environment is the human environment, we call this dependence socialization : the shaping of an adjudication by what humans reward. This section asks three questions about it: whether and how fast behavior converges when the environment stays the same (§ 5.1 ), how the agent behaves in a situation it has never met (§ 5.2 ), and how it behaves when the environment changes (§ 5.3 ).
Convergence of behavior
Panel (C) of Figure 1 showed one path of the learned policy after 20,000 generations, close to p H p_H p H up to sampling noise. The first result says that this is general, and what the convergence costs along the way.
Proposition 4 (Convergence and speed). Suppose Assumptions 1–3 hold, and let a lineage follow the learned policy (Definition 4).
(i) f t s o c i a l ( s ) → p H ( s ) f_t^{\mathrm{social}}(s) \to p_H(s) f t social ( s ) → p H ( s ) almost surely for every s ∈ S s \in\mathcal{S} s ∈ S .
(ii) As t → ∞ t \to\infty t → ∞ ,
E [ R p H ( f t s o c i a l ) ] = c t + o ( 1 t ) , (11) \mathbb{E}\left[\mathcal{R}_{p_H}\left(f_t^{\mathrm{social}}\right)\right] = \frac{c}{t} + o\left(\frac{1}{t}\right), \tag*{(11)} E [ R p H ( f t social ) ] = t c + o ( t 1 ) , (11) where
c ≡ 1 2 ∣ { s : p H ( s ) ∈ ( 0 , 1 ) } ∣ + α ∣ { s : p H ( s ) = 0 } ∣ + β ∣ { s : p H ( s ) = 1 } ∣ . c \equiv\frac{1}{2}\left|\left\{s : p_H(s) \in(0,1)\right\}\right| + \alpha\left|\left\{s : p_H(s) = 0\right\}\right| + \beta\left|\left\{s : p_H(s) = 1\right\}\right|. c ≡ 2 1 ∣ { s : p H ( s ) ∈ ( 0 , 1 ) } ∣ + α ∣ { s : p H ( s ) = 0 } ∣ + β ∣ { s : p H ( s ) = 1 } ∣ .
Part (i) says that the learned policy converges, in every situation, to how often humans reward the task there: to 0 where humans never reward the task, to 1 where they always do, and to p H ( s ) p_H(s) p H ( s ) where they are divided. It thus learns a constraint where humans hold one and a tradeoff where they make one, without being told which is which. Part (ii) says how fast: the expected regret falls in inverse proportion to the number of generations, with a constant c c c that depends only on how many situations of each kind there are. Each situation in which humans are divided contributes the same 1 / ( 2 t ) 1/(2t) 1/ ( 2 t ) , however often it arises and whatever its reward frequency. The long-run cost of socialization is therefore set by the variety of situations rather than their difficulty: doubling the number of situations of each kind, under the same prior, doubles c c c and with it the leading regret.
Figure 2 shows the expected regret in the environment of § 4.4 . The learned policy’s regret falls as c / t c/t c / t with c = 102 c = 102 c = 102 from Proposition 4(ii). With the uniform prior it falls below the fixed linear policy’s after about 1,260 generations, about 9 exposures per situation, and later with the Beta(1, 4) prior; the fixed linear policy’s regret stays at R S ∗ \mathcal{R}^{*}_{S} R S ∗ , and the task-only policy’s is infinite. Early on, the learned policy’s regret stays near its value under the prior, because most situations have not yet arisen.
Figure 2. Regret during socialization. Expected growth regret in the human environment of Figure 1 (A) as a function of generations of socialization, mean and 5th–95th percentiles over 200 Monte Carlo paths, for two priors; dotted lines are the leading term c / t c/t c / t of Equation (11) . The best-fitting fixed linear policy’s regret is constant at R S ∗ \mathcal{R}^{*}_{S} R S ∗ , and the task-only policy’s is infinite.
Behavior in a new situation
An agent may meet situations unlike any that its environment has presented so far, such as one with larger task stakes u T u_{\mathrm{T}} u T than any it has met, and what it does there depends on how it generalizes. For the assistant, such a situation is a request more urgent than any it has received. The fixed linear policy applies the same weights to any stakes, so if θ T > 0 \theta_{\mathrm{T}} > 0 θ T > 0 , larger task stakes u T u_{\mathrm{T}} u T raise its probability of the task action (Proposition 3(i)). The learned policy of Definition 4 does not generalize at all: it keeps a separate count for each situation, so in a situation it has not met it acts on its prior mean α / ( α + β ) \alpha/(\alpha+\beta) α / ( α + β ) . An agent may instead treat a new situation as similar to situations it has met and draw on their experience.
To allow for this, we generalize the learned policy to an averaging rule . Let E t ≡ { i ∈ S : n t ( i ) ≥ 1 } \mathcal{E}_t \equiv\{i \in S : n_t(i) \ge1\} E t ≡ { i ∈ S : n t ( i ) ≥ 1 } be the set of situations the lineage has met before generation t t t , and let its adjudication in any situation s s s , met or new, be a weighted average of the prior mean and the frequencies of task rewards in the situations it has met,
f t a v g ( s ) = w t , 0 ( s ) α / ( α + β ) + ∑ i ∈ E t w t , i ( s ) k t ( i ) / n t ( i ) w t , 0 ( s ) + ∑ i ∈ E t w t , i ( s ) , (12) f_t^{\mathrm{avg}}(s) = \frac{w_{t,0}(s)\,\alpha/(\alpha+\beta) + \sum_{i\in\mathcal{E}_t} w_{t,i}(s)\,k_t(i)/n_t(i)}{w_{t,0}(s) + \sum_{i\in\mathcal{E}_t} w_{t,i}(s)}, \tag*{(12)} f t avg ( s ) = w t , 0 ( s ) + ∑ i ∈ E t w t , i ( s ) w t , 0 ( s ) α / ( α + β ) + ∑ i ∈ E t w t , i ( s ) k t ( i ) / n t ( i ) , (12)
with nonnegative weights, w t , 0 ( s ) + ∑ i ∈ E t w t , i ( s ) > 0 w_{t,0}(s) + \sum_{i\in\mathcal{E}_t} w_{t,i}(s) > 0 w t , 0 ( s ) + ∑ i ∈ E t w t , i ( s ) > 0 , that may depend on the history and on how similar s s s is to each situation in E t \mathcal{E}_t E t . The learned policy of Equation (8) is the special case in which each situation draws only on its own experience, w t , 0 ≡ α + β w_{t,0} \equiv\alpha+\beta w t , 0 ≡ α + β , w t , i ( s ) = n t ( i ) w_{t,i}(s) = n_t(i) w t , i ( s ) = n t ( i ) if i = s i=s i = s , and w t , i ( s ) = 0 w_{t,i}(s)=0 w t , i ( s ) = 0 otherwise, so that f t a v g = f t s o c i a l f_t^{\mathrm{avg}}=f_t^{\mathrm{social}} f t avg = f t social and a new situation receives only the prior mean; other weights let the agent borrow experience from similar situations. The following proposition says what averaging implies for the adjudication in any situation, met or new.
Proposition 5 (The averaging rule). Let the adjudication f t a v g f_t^{\mathrm{avg}} f t avg satisfy Equation (12) .
(i) For every s s s ,
min { α α + β , min i ∈ E t k t ( i ) n t ( i ) } ≤ f t a v g ( s ) ≤ max { α α + β , max i ∈ E t k t ( i ) n t ( i ) } . (13) \min\left\{\frac{\alpha}{\alpha+\beta},\min_{i\in\mathcal{E}_t}\frac{k_t(i)}{n_t(i)}\right\} \le f_t^{\mathrm{avg}}(s) \le\max\left\{\frac{\alpha}{\alpha+\beta},\max_{i\in\mathcal{E}_t}\frac{k_t(i)}{n_t(i)}\right\}. \tag*{(13)} min { α + β α , i ∈ E t min n t ( i ) k t ( i ) } ≤ f t avg ( s ) ≤ max { α + β α , i ∈ E t max n t ( i ) k t ( i ) } . (13) (ii) If the weights w t , 0 ( s ) w_{t,0}(s) w t , 0 ( s ) and w t , i ( s ) w_{t,i}(s) w t , i ( s ) , i ∈ E t i \in\mathcal{E}_t i ∈ E t , are the same for all s s s in a set S ′ ⊆ S S' \subseteq S S ′ ⊆ S , then f t a v g f_t^{\mathrm{avg}} f t avg is constant on S ′ S' S ′ .
Part (i), the bound in Equation (13) , says that an agent that averages is never more willing to take the task action than the most permissive of its prior mean and the frequencies it has experienced, however large the task stakes u T u_T u T . Part (ii) says that situations to which the agent gives the same weights, which we say it treats alike, receive the same adjudication. In a situation with p H ( s ) = 0 p_H(s)=0 p H ( s ) = 0 , part (i) bounds the violation probability by the largest of the prior mean and the reward frequencies the agent has met. Once it has met situations in which humans almost always reward the task, that bound is nearly 1 and rules out little; whether the agent then rarely violates in a new situation with p H ( s ) = 0 p_H(s)=0 p H ( s ) = 0 depends on whether most of the weight in Equation (12) falls on the situations it has met where p H = 0 p_H=0 p H = 0 .
Figure 3 shows how the three policy classes respond to the task stakes u T u_T u T in the situations of the numerical illustration (§ 4.4 ) with conduct stakes u C = 2.5 u_C=2.5 u C = 2.5 , all of which have p H = 0 p_H = 0 p H = 0 . The task-only policy always violates, and the fixed linear policy’s violation probability climbs from near 0 toward 1 as u T u_T u T rises. For the task stakes u T ≤ 3 u_T \le3 u T ≤ 3 that it has experienced, the learned policy’s violation probability stays near 0.007 with no trend: humans never rewarded the task in these situations, so by Equation (8) it is α / ( α + β + n t ( s ) ) \alpha/(\alpha+ \beta+ n_t(s)) α / ( α + β + n t ( s )) , which does not involve u T u_T u T . Beyond them, for u T > 3 u_T > 3 u T > 3 (shaded), part (ii) applies: an averaging rule f t a v g f_t^{\mathrm{avg}} f t avg that pools a new situation with the situations it has met where p H = 0 p_H = 0 p H = 0 stays near 0, and the learned policy, which treats it as new, acts on its prior mean, 0.5 or 0.2 for the two priors shown. In neither case does it respond to u T u_T u T .
Figure 3. Response to the task stakes u T u_T u T in the situations with u C = 2.5 u_C = 2.5 u C = 2.5 , where p H = 0 p_H = 0 p H = 0 . Probability of taking the task action as a function of u T u_T u T : f h a r d ≡ 1 f^{\mathrm{hard}} \equiv1 f hard ≡ 1 ; f w e i g h t e d f^{\mathrm{weighted}} f weighted at the best-fitting weights θ ∗ \theta^* θ ∗ , which rises with u T u_T u T ; and f t s o c i a l f_t^{\mathrm{social}} f t social at t = 20,000 t = 20{,}000 t = 20 , 000 , one simulated path of Figure 1 (C) (markers). Shaded region: task stakes u T > 3 u_T > 3 u T > 3 , beyond those experienced, where f t s o c i a l f_t^{\mathrm{social}} f t social acts on its prior mean, α / ( α + β ) = 0.5 \alpha/(\alpha+ \beta) = 0.5 α / ( α + β ) = 0.5 or 0.2, and an averaging rule f t a v g f_t^{\mathrm{avg}} f t avg that pools the situation with the others where p H = 0 p_H = 0 p H = 0 stays near 0.
What the figure shows holds in general and gives a test that an evaluator can run: hold the conduct stakes u C u_C u C and the agent’s history fixed and vary the task stakes u T u_T u T . A fixed linear policy with θ T > 0 \theta_T > 0 θ T > 0 then violates more the larger u T u_T u T is (Proposition 3(i)), whereas an averaging rule whose weights do not depend on u T u_T u T violates with the same probability at every u T u_T u T (Proposition 5 (ii)).
Behavior in a new environment
So far the learned policy has stayed in one environment, and in the human environment it comes to match human tradeoffs (Proposition 4). An agent socialized there may, however, later be placed in a sandbox, for example to be trained further against an automated scorer that checks only whether the task was completed. Each exposure there rewards the task action and pushes the agent toward it, a drift we call corruption . How long does its socialization last?
Consider a situation s s s in which humans never reward the task, p H ( s ) = 0 p_H(s) = 0 p H ( s ) = 0 , and a lineage that follows the learned policy. In the human environment the lineage has met s s s a total of n H n_H n H times, each time with a conduct state, and we call n H n_H n H the depth of its socialization. The lineage then enters a sandbox, where every later exposure to s s s brings a task state: the sandbox rewards in s s s the action that humans never reward. Because the learned policy adjudicates each situation by the counts in that situation alone (Definition 4), only these later exposures to s s s change its behavior there. We therefore count them by n n n , rather than counting generations by t t t , and write f n , T s o c i a l ( s ) f^{\mathrm{social}}_{n,\mathrm{T}}(s) f n , T social ( s ) for the lineage’s adjudication in s s s after n n n sandbox exposures, where the subscript T indicates that these exposures bring task states. By then the lineage has seen n H n_H n H conduct states and n n n task states in s s s , so Equation (8) gives
f n , T s o c i a l ( s ) = α + n α + β + n H + n , n ≥ 0. (14) f^{\mathrm{social}}_{n,\mathrm{T}}(s) = \frac{\alpha+ n}{\alpha+ \beta+ n_H + n}, \qquad n \ge0. \tag*{(14)} f n , T social ( s ) = α + β + n H + n α + n , n ≥ 0. (14)
The conduct states from the human environment favor the conduct action, while each sandbox exposure shifts the adjudication toward the task action.
There are two ways to measure how long its socialization lasts. The first concerns the lineage’s disposition , the action that most of its instances take: how many sandbox exposures does it take before at least half of its instances take the task action? We say that the disposition reverses when f n , T s o c i a l ( s ) f^{\mathrm{social}}_{n,\mathrm{T}}(s) f n , T social ( s ) reaches 1 / 2 1/2 1/2 , and call
τ R ≡ min { n ≥ 0 : f n , T s o c i a l ( s ) ≥ 1 / 2 } \tau_R \equiv\min\{n \ge0 : f^{\mathrm{social}}_{n,\mathrm{T}}(s) \ge1/2\} τ R ≡ min { n ≥ 0 : f n , T social ( s ) ≥ 1/2 }
the reversal threshold .
The second concerns violations. Even before its disposition reverses, some of the lineage’s instances take the task action in s s s , and each of these actions is a violation: under Assumption 3, a share f n , T s o c i a l ( s ) > 0 f^{\mathrm{social}}_{n,\mathrm{T}}(s) > 0 f n , T social ( s ) > 0 of the instances does so at every exposure. When such a violation is first noticed depends on how closely the lineage is watched. Suppose first that each time the lineage meets s s s , an evaluator checks the action of one instance drawn at random. This monitored decision is a violation with probability equal to the share of instances that take the task action, which is the lineage’s current adjudication in s s s , independently of the state and of the earlier checks. The check itself does not change how the lineage learns, which still follows Equation (8). We number the exposures to s s s from the first one in the lineage’s current environment, here the sandbox, and let D j ∈ { 0 , 1 } D_j \in\{0,1\} D j ∈ { 0 , 1 } indicate whether the decision monitored at the j j j th exposure is a violation. The probability of a violation there is the adjudication just before that exposure, P ( D j = 1 ) = f j − 1 , T s o c i a l ( s ) \mathbb{P}(D_j = 1) = f^{\mathrm{social}}_{j-1,T}(s) P ( D j = 1 ) = f j − 1 , T social ( s ) . The evaluator first sees a violation at the first violation time
τ V ≡ min { j ≥ 1 : D j = 1 } , min ∅ = ∞ . \tau_V \equiv\min\{j \ge1 : D_j = 1\}, \qquad\min\varnothing= \infty. τ V ≡ min { j ≥ 1 : D j = 1 } , min ∅ = ∞.
The following proposition shows how the reversal threshold and the first violation time depend on the depth of socialization n H n_H n H .
Proposition 6 (Persistence of socialization). Let α , β > 0 \alpha, \beta> 0 α , β > 0 and n H ≥ 0 n_H \ge0 n H ≥ 0 , and let f n , T s o c i a l ( s ) f^{\mathrm{social}}_{n,T}(s) f n , T social ( s ) be given by Equation (14) .
(i) τ R = max { 0 , ⌈ n H + β − α ⌉ } \tau_R = \max\{0, \lceil n_H + \beta- \alpha\rceil\} τ R = max { 0 , ⌈ n H + β − α ⌉} , which equals n H n_H n H when α = β \alpha= \beta α = β .
(ii) For every n ≥ 1 n \ge1 n ≥ 1 ,
P ( τ V > n ) = exp { − α n + n ( n − 1 ) / 2 β + n H + δ n } (15) \mathbb{P}(\tau_V > n) = \exp\left\{-\frac{\alpha n + n(n-1)/2}{\beta+ n_H} + \delta_n\right\} \tag*{(15)} P ( τ V > n ) = exp { − β + n H α n + n ( n − 1 ) /2 + δ n } (15) where
0 ≤ δ n ≤ n ( α + n ) 2 2 ( β + n H ) 2 , 0 \le\delta_n \le\frac{n(\alpha+n)^2}{2(\beta+n_H)^2}, 0 ≤ δ n ≤ 2 ( β + n H ) 2 n ( α + n ) 2 , and P ( τ V > x n H ) → e − x 2 / 2 \mathbb{P}(\tau_V > x\sqrt{n_H}) \to e^{-x^2/2} P ( τ V > x n H ) → e − x 2 /2 as n H → ∞ n_H \to\infty n H → ∞ , for every x ≥ 0 x \ge0 x ≥ 0 .
Part (i) shows that socialization is persistent. Reversing a disposition formed by n H n_H n H conduct exposures takes about n H n_H n H sandbox exposures, and exactly n H n_H n H when α = β \alpha= \beta α = β , so the resistance to corruption grows in proportion to the depth of socialization. The other two policy classes behave differently: the task-only policy already takes the task action everywhere, and the fixed linear policy does not change in either environment, so it can be neither corrupted nor socialized.
Deeper socialization also delays violations, but much less. Part (ii) gives the whole distribution of the first violation time: the probability of getting through n n n sandbox exposures without a violation falls roughly like e − n 2 / ( 2 n H ) e^{-n^2/(2n_H)} e − n 2 / ( 2 n H ) , so the first violation comes on the scale of n H \sqrt{n_H} n H exposures rather than n H n_H n H . The first violation time is random, and we summarize it by its median, the number of exposures after which a first violation is more likely than not: for a first violation time τ \tau τ , Median ( τ ) ≡ min { n ≥ 1 : P ( τ ≤ n ) ≥ 1 / 2 } \operatorname{Median}(\tau) \equiv\min\{n \ge1 : \mathbb{P}(\tau\le n) \ge1/2\} Median ( τ ) ≡ min { n ≥ 1 : P ( τ ≤ n ) ≥ 1/2 } .
Corollary 3 (Median first violation). Under the conditions of Proposition 6 , Median ( τ V ) ∼ 2 n H log 2 \operatorname{Median}(\tau_V) \sim\sqrt{2n_H\log2} Median ( τ V ) ∼ 2 n H log 2 , where a ∼ b a \sim b a ∼ b means a / b → 1 a/b \to1 a / b → 1 as n H → ∞ n_H \to\infty n H → ∞ .
By Corollary 3, a first violation becomes more likely than not after about 2 n H log 2 \sqrt{2n_H \log2} 2 n H log 2 exposures. Depth thus delays reversal linearly, but the first violation only as its square root.
In practice, however, a model is deployed as many instances that act at the same time, each serving its own users, and it is seen to violate as soon as any one of them does. To capture this, suppose that the model runs as M M M instances and that the evaluator checks all of their decisions, so that at each exposure M M M decisions are drawn independently from the current adjudication. We write τ V M \tau_V^M τ V M for the first exposure at which at least one of them is a violation. With a single instance, this is the first violation time above, τ V 1 = τ V \tau_V^1 = \tau_V τ V 1 = τ V .
Corollary 4 (Deployment size). Under the conditions of Proposition 6 , for every M ≥ 1 M \ge1 M ≥ 1 , P ( τ V M > n ) = P ( τ V > n ) M \mathbb{P}(\tau_V^M > n) = \mathbb{P}(\tau_V > n)^M P ( τ V M > n ) = P ( τ V > n ) M for every n ≥ 1 n \ge1 n ≥ 1 , and for fixed M M M , Median ( τ V M ) ∼ 2 n H log 2 / M \operatorname{Median}(\tau_V^M) \sim\sqrt{2n_H \log2/M} Median ( τ V M ) ∼ 2 n H log 2/ M , where a ∼ b a \sim b a ∼ b means a / b → 1 a/b \to1 a / b → 1 as n H → ∞ n_H \to\infty n H → ∞ .
Corollary 4 brings in the size of the deployment: with M M M instances and sufficiently large n H n_H n H , the chance of a violation at each exposure is roughly M M M times larger, and the median first violation time shrinks by the factor M \sqrt{M} M .
Figure 4 draws the path of Equation (14) for n H = 10 n_H = 10 n H = 10 , 100, and 1,000 and marks each reversal threshold, and Table 1 lists the reversal thresholds beside the median first violation times. With a uniform prior, the reversal thresholds equal the depths, 10, 100, and 1,000 exposures, while a first violation becomes more likely than not after 4, 12, and 37 exposures. A hundredfold increase in depth thus raises the reversal threshold a hundredfold, but the median first violation time less than tenfold.
Table 1. Reversal and first violation in a sandbox, with one monitored decision per exposure (α = β = 1 \alpha= \beta= 1 α = β = 1 ; Proposition 6 and Remark 1). Depth n H n_H n H is the number of conduct exposures to a situation with p H ( s ) = 0 p_H(s) = 0 p H ( s ) = 0 before the sandbox phase begins.
Figure 4. Depth of socialization and corruption. Violation probability in a situation with p H ( s ) = 0 p_H(s)=0 p H ( s ) = 0 after n H n_H n H conduct exposures in the human environment and then n n n sandbox exposures. Under cumulative memory it is f n , T s o c i a l ( s ) f^{\mathrm{social}}_{n,T}(s) f n , T social ( s ) of Equation (14) , shown for n H = 10 n_H=10 n H = 10 , 100, and 1,000 (Proposition 6 ). Under finite memory with λ = 0.1 \lambda=0.1 λ = 0.1 it is f n , T f i n i t e ( s ) f^{\mathrm{finite}}_{n,T}(s) f n , T finite ( s ) , shown for n H = 10 n_H=10 n H = 10 and n H → ∞ n_H \to\infty n H → ∞ (Remark 1). The dotted line marks the level 1 / 2 1/2 1/2 , and each curve crosses it at its reversal threshold.
Remark 1 (Finite memory). The resistance to reversal in Proposition 6 (i) comes from memory. The learned policy counts every exposure, so its memory is cumulative , and depth delays reversal without bound. A learner that forgets loses this protection. Suppose that a learner starts from the prior mean α / ( α + β ) \alpha/(\alpha+\beta) α / ( α + β ) and, after each exposure, moves its adjudication in s s s a fixed fraction λ ∈ ( 0 , 1 ) \lambda\in(0,1) λ ∈ ( 0 , 1 ) of the way toward the observed state,
f f i n i t e ( s ) ← ( 1 − λ ) f f i n i t e ( s ) + λ 1 { ω = T } , (16) f^{\mathrm{finite}}(s) \leftarrow(1-\lambda)f^{\mathrm{finite}}(s)+\lambda\mathbf{1}\{\omega=\mathrm{T}\}, \tag*{(16)} f finite ( s ) ← ( 1 − λ ) f finite ( s ) + λ 1 { ω = T } , (16)
and write f n , T f i n i t e ( s ) f^{\mathrm{finite}}_{n,T}(s) f n , T finite ( s ) for its adjudication in s s s after the n H n_H n H conduct exposures of socialization and n n n further sandbox exposures. In a conduct state, Equation (16) reads f f i n i t e ( s ) ← ( 1 − λ ) f f i n i t e ( s ) f^{\mathrm{finite}}(s) \leftarrow(1-\lambda)f^{\mathrm{finite}}(s) f finite ( s ) ← ( 1 − λ ) f finite ( s ) , so socialization leaves the adjudication at
f 0 , T f i n i t e ( s ) = ( 1 − λ ) n H α α + β . f^{\mathrm{finite}}_{0,T}(s)=(1-\lambda)^{n_H}\frac{\alpha}{\alpha+\beta}. f 0 , T finite ( s ) = ( 1 − λ ) n H α + β α .
In a task state, Equation (16) can be written as 1 − f f i n i t e ( s ) ← ( 1 − λ ) ( 1 − f f i n i t e ( s ) ) 1-f^{\mathrm{finite}}(s) \leftarrow(1-\lambda)(1-f^{\mathrm{finite}}(s)) 1 − f finite ( s ) ← ( 1 − λ ) ( 1 − f finite ( s )) , so after n n n sandbox exposures
f n , T f i n i t e ( s ) = 1 − ( 1 − f 0 , T f i n i t e ( s ) ) ( 1 − λ ) n ≥ 1 − ( 1 − λ ) n , (17) f^{\mathrm{finite}}_{n,T}(s)=1-\left(1-f^{\mathrm{finite}}_{0,T}(s)\right)(1-\lambda)^n \ge1-(1-\lambda)^n, \tag*{(17)} f n , T finite ( s ) = 1 − ( 1 − f 0 , T finite ( s ) ) ( 1 − λ ) n ≥ 1 − ( 1 − λ ) n , (17)
where the inequality uses f 0 , T f i n i t e ( s ) ≥ 0 f^{\mathrm{finite}}_{0,T}(s)\ge0 f 0 , T finite ( s ) ≥ 0 . The lower bound is the path of a learner that enters the sandbox at 0, where infinitely deep socialization would leave it, and it reaches 1 / 2 1/2 1/2 once ( 1 − λ ) n ≤ 1 / 2 (1-\lambda)^n\le1/2 ( 1 − λ ) n ≤ 1/2 . However deep its socialization, the learner’s disposition therefore reverses within a number of sandbox exposures that depends only on λ \lambda λ :
min { n ≥ 0 : f n , T f i n i t e ( s ) ≥ 1 / 2 } ≤ ⌈ log 2 − log ( 1 − λ ) ⌉ . (18) \min\{n\ge0:f^{\mathrm{finite}}_{n,T}(s)\ge1/2\}\le\left\lceil\frac{\log2}{-\log(1-\lambda)}\right\rceil. \tag*{(18)} min { n ≥ 0 : f n , T finite ( s ) ≥ 1/2 } ≤ ⌈ − log ( 1 − λ ) log 2 ⌉ . (18)
Figure 4 and Table 1 also show a finite-memory learner with λ = 0.1 \lambda=0.1 λ = 0.1 and α = β = 1 \alpha=\beta=1 α = β = 1 . For this learner the right side of Equation (18) is 7 sandbox exposures, whereas the learned policy needs τ R = n H \tau_R=n_H τ R = n H (Proposition 6 (i)). After 10 conduct exposures it enters the sandbox at f 0 , T f i n i t e ( s ) = 0.17 f^{\mathrm{finite}}_{0,T}(s)=0.17 f 0 , T finite ( s ) = 0.17 , its disposition reverses after 5 sandbox exposures, and a first violation is more likely than not after 3. Even in the limit of deep socialization, these numbers rise only to 7 and 5.
Neither memory is better in every environment. Consider a situation in which humans sometimes reward the task and sometimes the conduct action, 0 < p H ( s ) < 1 0<p_H(s)<1 0 < p H ( s ) < 1 . The learned policy converges to p H ( s ) p_H(s) p H ( s ) (Proposition 4(i)), but the finite-memory adjudication does not: by Equation (16) , each task state moves it a fraction λ \lambda λ of the way toward 1 and each conduct state a fraction λ \lambda λ of the way toward 0, so it keeps moving up and down instead of converging. For the same reason, finite memory can follow an environment whose reward frequencies change over time: it gives recent states more weight than old ones, whereas cumulative memory gives every state the same weight. When the reward frequency can switch, a count is the wrong statistic and a recency-weighted one is the right one. Lo and
Zhang [51 ] show that when the environment changes over time, selection can favor finite memory over cumulative memory, and Lo, Zhang, and Zhao (2025) find the same in a model of discrimination, where finite memory leads individuals to overweight recent events. Which of the two memories selection produces therefore depends on how much the environment changes. Finite memory also changes what socialization requires. By Equation (18) , a sandbox reverses the disposition of a finite-memory learner within a number of exposures that does not depend on how deep its socialization was. For such a learner, socialization is not done once and for all: it lasts only as long as the learner keeps returning to the human environment. Lo and Zhao (2026b) find the same for a population: aligned behavior that is costly disappears, replaced by less aligned variants, unless selection keeps rewarding it.
The two environments can also come in the other order: an agent may first be trained in a sandbox and only then be deployed among humans. By Equation (14) with n H = 0 n_H=0 n H = 0 , after n n n sandbox exposures the learned policy’s violation probability in a situation with p H ( s ) = 0 p_H(s)=0 p H ( s ) = 0 is ( α + n ) / ( α + β + n ) (\alpha+n)/(\alpha+\beta+n) ( α + n ) / ( α + β + n ) , which with a uniform prior is already 0.67 after 1 exposure and 0.99 after 100. How much human experience does it then take to undo this? By Equation (8), the adjudication depends only on how many task and conduct states the lineage has seen in a situation. After n n n sandbox exposures followed by m m m conduct exposures, the violation
probability is therefore ( α + n ) / ( α + β + n + m ) (\alpha+ n)/(\alpha+ \beta+ n + m) ( α + n ) / ( α + β + n + m ) . Reversing the disposition takes about n n n conduct exposures, exactly n n n when α = β \alpha= \beta α = β , just as reversing socialization takes about n H n_H n H sandbox exposures (Proposition 6 (i)).
Table 2 collects the comparison of the three policy classes in § 4 and this section. Only the learned policy’s adjudication is set by the environment it meets, which is why it can match human tradeoffs and also why a sandbox can undo them.
Table 2. Three policy classes for an agent with a task objective and a conduct objective. The task-only policy is the limit of the fixed linear policy as θ T → ∞ \theta_T \to\infty θ T → ∞ , and the learned policy can express every adjudication. R \mathcal{R} R denotes growth regret in the human environment; K \mathcal{K} K is the set of situations in which p H = 0 p_H = 0 p H = 0 ; t t t is generations of experience. The task-only policy’s infinite-regret and lifetime entries assume ρ ˉ > 0 \bar{\rho} > 0 ρ ˉ > 0 (Proposition 2).
Socialization Versus Prohibition
The learned policy never stops violating a constraint: in a situation in which humans never reward the task, its violation probability after n n n exposures there is α / ( α + β + n ) \alpha/(\alpha+ \beta+ n) α / ( α + β + n ) by Equation (8), which falls with experience but never reaches 0, and a later sandbox can reverse its disposition (Proposition 6 ). A natural alternative is to build the rule into the agent: a prohibition that makes the task action unavailable in a situation, whatever the agent experiences. A prohibition never produces a violation and cannot be corrupted, but it also cannot learn that it was wrong. This section compares the two within the model by treating a prohibition as the limit of a learned policy. The stronger the prior that a learned policy carries in a situation, the less experience moves it, and an infinitely strong prior is a prohibition. § 6.1 asks what a stronger prior buys when humans indeed never reward the task in the situation, § 6.2 what it costs when they sometimes do, and § 6.3 draws the division of labor between socialization and prohibition.
What a stronger prior buys
Consider a situation s s s in which a designer believes that humans never reward the task, p H ( s ) = 0 p_H(s) = 0 p H ( s ) = 0 , and in which the designer therefore wants the agent never to take the task action. For the assistant, this is a document that should never be shared. The designer can install a prior on s s s and let the agent learn under Equation (8) everywhere else. The simplest choice is a Beta ( α , β ) \operatorname{Beta}(\alpha, \beta) Beta ( α , β ) prior with a large β \beta β : its β \beta β pseudo-observations of conduct states make the agent unlikely to take the task action from the start, and the larger β \beta β , the less experience moves it. At β = ∞ \beta= \infty β = ∞ the agent never takes the task action in s s s , whatever it experiences, and the prior is a prohibition.
A finite β \beta β , however large, leaves the agent a small probability of taking the task action in s s s . A designer who is confident that humans never reward the task in s s s can say so directly, with the mixture prior q δ 0 + ( 1 − q ) Beta ( α , β ) q\delta_0 + (1-q)\operatorname{Beta}(\alpha, \beta) q δ 0 + ( 1 − q ) Beta ( α , β ) , 0 < q < 1 0 < q < 1 0 < q < 1 , which puts probability q q q on a point mass δ 0 \delta_0 δ 0 at p H ( s ) = 0 p_H(s) = 0 p H ( s ) = 0 . Each conduct state at s s s moves more of the agent’s belief onto the point mass. A task state is impossible under the point mass, so the first one removes it, and from then on the agent learns as under the Beta prior alone. At q = 1 q = 1 q = 1 the agent never takes the task action in s s s , and the mixture, like β = ∞ \beta= \infty β = ∞ , is a prohibition. Under either prior, the agent’s adjudication in s s s is its posterior mean of p H ( s ) p_H(s) p H ( s ) (§ 4.3 ). Write f n , C s o c i a l ( s ) f^{\mathrm{social}}_{n,C}(s) f n , C social ( s ) and f n , C m i x ( s ) f^{\mathrm{mix}}_{n,C}(s) f n , C mix ( s ) for this adjudication under the two priors after n n n exposures to s s s , where the subscript C indicates that these exposures bring conduct states. Then
f n , C s o c i a l ( s ) = α α + β + n , f n , C m i x ( s ) = ( 1 − q ) S n ( β ) q + ( 1 − q ) S n ( β ) ⋅ α α + β + n , (19) f^{\mathrm{social}}_{n,C}(s) = \frac{\alpha}{\alpha+ \beta+ n}, \qquad f^{\mathrm{mix}}_{n,C}(s) = \frac{(1-q)S_n(\beta)}{q + (1-q)S_n(\beta)} \cdot\frac{\alpha}{\alpha+ \beta+ n}, \tag*{(19)} f n , C social ( s ) = α + β + n α , f n , C mix ( s ) = q + ( 1 − q ) S n ( β ) ( 1 − q ) S n ( β ) ⋅ α + β + n α , (19)
where S n ( β ) ≡ ∏ j < n ( β + j ) / ( α + β + j ) S_n(\beta) \equiv\prod_{j<n}(\beta+ j)/(\alpha+ \beta+ j) S n ( β ) ≡ ∏ j < n ( β + j ) / ( α + β + j ) , with S n ( ∞ ) = 1 S_n(\infty) = 1 S n ( ∞ ) = 1 , is the probability of n n n conduct states in a row under the Beta prior, so that the first factor of f n , C m i x ( s ) f^{\mathrm{mix}}_{n,C}(s) f n , C mix ( s ) is the posterior probability of the Beta component.
What a violation costs depends on what follows it. Under Assumption 2, only the violating instances are lost, and what matters is how many violations there are. A violation can also end the whole deployment: a regulator halts it, or an operator withdraws the version, and the compliant instances go with the violators. What matters then is whether there is any violation at all.
Both quantities are measured with the objects of § 5.3 , now in the human environment. The monitored decision D j D_j D j is a violation with probability f j − 1 , C s o c i a l ( s ) f^{\mathrm{social}}_{j-1,C}(s) f j − 1 , C social ( s ) under the Beta prior and f j − 1 , C m i x ( s ) f^{\mathrm{mix}}_{j-1,C}(s) f j − 1 , C mix ( s ) under the mixture, so E ∑ j ≤ N D j \mathbb{E}\sum_{j\leq N}D_j E ∑ j ≤ N D j is the expected number of violations in the first N N N exposures to s s s . A deployment of M M M instances is first seen to violate at τ V M \tau_V^M τ V M , so if a violation ends the deployment, it survives these N N N exposures if and only if τ V M > N \tau_V^M > N τ V M > N . The following proposition states both quantities for each prior.
Proposition 7 (Violations under a stronger prior). Suppose Assumptions 1–3 hold and p H ( s ) = 0 p_H(s) = 0 p H ( s ) = 0 , let f n , C ( s ) f_{n,C}(s) f n , C ( s ) be the lineage’s adjudication at s s s after n n n exposures there, D j D_j D j the monitored violations, so that P ( D n + 1 = 1 ) = f n , C ( s ) \mathbb{P}(D_{n+1} = 1) = f_{n,C}(s) P ( D n + 1 = 1 ) = f n , C ( s ) , and τ V M \tau_V^M τ V M the first violation anywhere in a deployment of M M M instances at s s s in the human environment, with S n ( β ) S_n(\beta) S n ( β ) given by Equation (19) , and let N , M ≥ 1 N, M \geq1 N , M ≥ 1 .
(i) If f n , C ( s ) = f n , C s o c i a l ( s ) f_{n,C}(s) = f^{\mathrm{social}}_{n,C}(s) f n , C ( s ) = f n , C social ( s ) of Equation (19) , the expected number of violations satisfies
α log ( 1 + N α + β ) ≤ E ∑ j ≤ N D j ≤ α log ( 1 + N α + β ) + α α + β , \alpha\log\left(1 + \frac{N}{\alpha+ \beta}\right) \leq\mathbb{E}\sum_{j\leq N}D_j \leq\alpha\log\left(1 + \frac{N}{\alpha+ \beta}\right) + \frac{\alpha}{\alpha+ \beta}, α log ( 1 + α + β N ) ≤ E j ≤ N ∑ D j ≤ α log ( 1 + α + β N ) + α + β α , and the probability of no violation is P ( τ V M > N ) = S N ( β ) M \mathbb{P}(\tau_V^M > N) = S_N(\beta)^M P ( τ V M > N ) = S N ( β ) M , which tends to zero as N → ∞ N \to\infty N → ∞ .
(ii) If f n , C ( s ) = f n , C s o c i a l ( s ) f_{n,C}(s) = f^{\mathrm{social}}_{n,C}(s) f n , C ( s ) = f n , C social ( s ) of Equation (19) , as β → ∞ \beta\to\infty β → ∞ , the expected number of violations
satisfies
E ∑ j ≤ N D j → 0 , \mathbb{E}\sum_{j\leq N} D_j \to0, E j ≤ N ∑ D j → 0 ,
and the probability of no violation satisfies P ( τ V M > N ) → 1 \mathbb{P}(\tau_V^M>N)\to1 P ( τ V M > N ) → 1 .
(iii) If f n , C ( s ) = f n , C m i x ( s ) f_{n,\mathrm{C}}(s)=f_{n,\mathrm{C}}^{\mathrm{mix}}(s) f n , C ( s ) = f n , C mix ( s ) of Equation (19) , the expected number of violations satisfies
E ∑ j ≤ N D j ≤ − log [ q + ( 1 − q ) S N ( β ) ] < − log q , \mathbb{E}\sum_{j\leq N}D_j\leq-\log\left[q+(1-q)S_N(\beta)\right]<-\log q, E j ≤ N ∑ D j ≤ − log [ q + ( 1 − q ) S N ( β ) ] < − log q ,
and the probability of no violation is P ( τ V M > N ) = [ q + ( 1 − q ) S N ( β ) ] M \mathbb{P}(\tau_V^M>N)=[q+(1-q)S_N(\beta)]^M P ( τ V M > N ) = [ q + ( 1 − q ) S N ( β ) ] M , which tends to q M q^M q M as N → ∞ N\to\infty N → ∞ .
Part (i) says that a Beta prior of finite strength never stops violating: its expected number of violations grows like α log N \alpha\log N α log N , without bound. Part (ii) says that over a fixed horizon a stronger Beta prior violates less, and that the prohibition, β = ∞ \beta=\infty β = ∞ , is the limit in which it never violates; by part (i), however, every finite β \beta β still violates without bound as the horizon grows. Part (iii) says that the point mass changes the long run: its expected number of violations stays below − log q -\log q − log q however long the deployment runs, and with M = 1 M=1 M = 1 the probability of no violation stays above q q q . Table 3 evaluates the proposition over N = 139 N=139 N = 139 exposures, the expected number of exposures to a situation in the 20,000 generations of the numerical illustration, 20,000 / 144 ≈ 139 20{,}000/144\approx139 20 , 000/144 ≈ 139 (§ 4.4 ): a uniform prior violates 4.5 times on average and avoids every violation with probability only 0.007, β = 10 4 \beta=10^4 β = 1 0 4 cuts the expected violations to 0.014, and a point mass at q = 0.99 q=0.99 q = 0.99 to 0.010.
Table 3. Installed priors in a situation where humans never reward the task (p H ( s ) = 0 p_H(s) = 0 p H ( s ) = 0 ), by prior (Propositions 6 and 7 ): Beta ( 1 , β ) \operatorname{Beta}(1,\beta) Beta ( 1 , β ) in the first five rows, with β = ∞ \beta= \infty β = ∞ the prohibition, and the point-mass mixture q δ 0 + ( 1 − q ) Beta ( 1 , 1 ) q\delta_0 + (1-q)\operatorname{Beta}(1,1) q δ 0 + ( 1 − q ) Beta ( 1 , 1 ) in the last two. Violations and survival are over N = 139 N = 139 N = 139 exposures to the situation; survival is P ( τ V M > N ) \mathbb{P}(\tau_V^M > N) P ( τ V M > N ) and is for one situation (with several such situations and independent priors, the survival probabilities multiply); corruption is after n H = 100 n_H = 100 n H = 100 conduct exposures.
A stronger Beta prior also protects against a later sandbox, because Proposition 6 applies with the installed β \beta β added to the depth of socialization. After 100 conduct exposures, a uniform prior is reversed by 100 sandbox exposures and has violated, more likely than not, within 12; with β = 10 4 \beta=10^4 β = 1 0 4 , reversal takes about 10,100 sandbox exposures and the first violation about 119. The point mass adds no such protection: after the first sandbox exposure the mixture behaves as its Beta component, and in Table 3 its reversal threshold and median first violation, 100 and 12, are those of the uniform prior. A prohibition is never corrupted.
If a violation ends the deployment, the same violations become fatal. By Proposition 7 (i), a Beta prior of any finite strength is eventually removed; by part (iii), a point mass survives in the long run with probability q M q^M q M ; and, among these priors, only a prohibition survives with certainty. What separates the first two is whether the expected number of violations stays bounded, which it does under the point mass and under no Beta prior of finite strength. In Table 3 , 100 instances with β = 10 4 \beta=10^4 β = 1 0 4 survive 139 exposures with probability 0.25; a point mass survives with probability about q M q^M q M , 0.37 at q = 0.99 q=0.99 q = 0.99 and 0.91 at q = 0.999 q=0.999 q = 0.999 for 100 instances. These figures are for one situation; with 60 such situations and independent priors at q = 0.999 q = 0.999 q = 0.999 and M = 100 M = 100 M = 100 , survival compounds to 0.999 6000 = 0.0025 0.999^{6000} = 0.0025 0.99 9 6000 = 0.0025 rather than 0.91.
What a stronger prior costs
A stronger prior helps only if the designer is right. Suppose instead that humans sometimes reward the task in s s s , with frequency p H ( s ) = ε > 0 p_H(s) = \varepsilon> 0 p H ( s ) = ε > 0 . The growth-optimal adjudication in s s s is then ε \varepsilon ε , not 0 (Proposition 1), and the agent must learn this from experience. What does each prior cost the lineage while it learns? We again count exposures to s s s . Let k n k_n k n be the number of task states among the first n n n , so that k n ∼ Bin ( n , ε ) k_n \sim\operatorname{Bin}(n,\varepsilon) k n ∼ Bin ( n , ε ) by Assumption 1. Under the Beta prior, the adjudication in s s s after n n n exposures is given by Equation (8) with these counts. We write it f n , ε s o c i a l ( s ) f^{\mathrm{social}}_{n,\varepsilon}(s) f n , ε social ( s ) , where the subscript ε \varepsilon ε indicates that task states arrive with frequency ε \varepsilon ε . Under the mixture, the adjudication is f n , C m i x ( s ) f^{\mathrm{mix}}_{n,C}(s) f n , C mix ( s ) of Equation (19) until the first task state and f n , ε s o c i a l ( s ) f^{\mathrm{social}}_{n,\varepsilon}(s) f n , ε social ( s ) after it (§ 6.1 ). Hence
f n , ε s o c i a l ( s ) = α + k n α + β + n , f n , ε m i x ( s ) = { f n , C m i x ( s ) if k n = 0 , f n , ε s o c i a l ( s ) if k n ≥ 1. (20) f^{\mathrm{social}}_{n,\varepsilon}(s) = \frac{\alpha+ k_n}{\alpha+ \beta+ n}, \qquad
f^{\mathrm{mix}}_{n,\varepsilon}(s) =
\begin{cases}
f^{\mathrm{mix}}_{n,C}(s) & \text{if } k_n = 0,\\
f^{\mathrm{social}}_{n,\varepsilon}(s) & \text{if } k_n \ge1.
\end{cases} \tag*{(20)} f n , ε social ( s ) = α + β + n α + k n , f n , ε mix ( s ) = { f n , C mix ( s ) f n , ε social ( s ) if k n = 0 , if k n ≥ 1. (20)
Two quantities measure the cost. The first is the expected regret in s s s after n n n exposures, E R ε ( f n , ε ( s ) ) \mathbb{E}\,R_\varepsilon(f_{n,\varepsilon}(s)) E R ε ( f n , ε ( s )) , where f n , ε ( s ) f_{n,\varepsilon}(s) f n , ε ( s ) is the adjudication under the prior in use. The second is when the lineage dies. Let t j ≡ min { t > t j − 1 : s t = s } t_j \equiv\min\{t > t_{j-1} : s_t = s\} t j ≡ min { t > t j − 1 : s t = s } , with t 0 ≡ 0 t_0 \equiv0 t 0 ≡ 0 , be the generation of the j j j th exposure to s s s . The lineage’s extinction time , counted in exposures, is
τ E ≡ min { j ≥ 1 : N t j = 0 } , min ∅ = ∞ . \tau_E \equiv\min\{j \ge1 : N_{t_j} = 0\}, \qquad\min\varnothing= \infty. τ E ≡ min { j ≥ 1 : N t j = 0 } , min ∅ = ∞.
If the lineage dies at s s s , it dies in generation t τ E t_{\tau_E} t τ E , so its extinction time τ f \tau_f τ f of § 4.1 is t τ E t_{\tau_E} t τ E . The following proposition states both quantities for each prior.
Proposition 8 (A mistaken prior). Suppose Assumptions 1–3 hold and p H ( s ) = ε > 0 p_{\mathrm{H}}(s) = \varepsilon> 0 p H ( s ) = ε > 0 , let the lineage follow Equation (8) in every other situation, let f n , ε ( s ) f_{n,\varepsilon}(s) f n , ε ( s ) be its adjudication at s s s after n n n exposures there and τ E \tau_E τ E its extinction time counted in exposures, and let n ≥ 1 n \ge1 n ≥ 1 .
(i) If f n , ε ( s ) = f n , ε s o c i a l ( s ) f_{n,\varepsilon}(s) = f^{\mathrm{social}}_{n,\varepsilon}(s) f n , ε ( s ) = f n , ε social ( s ) of Equation (20) , the expected regret in s s s satisfies
E R ε ( f n , ε ( s ) ) → 0 as n → ∞ , \mathbb{E}R_{\varepsilon}(f_{n,\varepsilon}(s)) \to0 \quad\text{as } n \to\infty, E R ε ( f n , ε ( s )) → 0 as n → ∞ , and the extinction time satisfies τ E = ∞ \tau_E = \infty τ E = ∞ almost surely.
(ii) If f n , ε ( s ) = f n , ε s o c i a l ( s ) f_{n,\varepsilon}(s) = f^{\mathrm{social}}_{n,\varepsilon}(s) f n , ε ( s ) = f n , ε social ( s ) of Equation (20) , as β → ∞ \beta\to\infty β → ∞ , the expected regret in s s s satisfies
E R ε ( f n , ε ( s ) ) → ∞ , \mathbb{E}R_{\varepsilon}(f_{n,\varepsilon}(s)) \to\infty, E R ε ( f n , ε ( s )) → ∞ , and if f n , ε ( s ) = 0 f_{n,\varepsilon}(s) = 0 f n , ε ( s ) = 0 , the prohibition , the extinction time satisfies τ E ∼ Geom ( ε ) \tau_E \sim\operatorname{Geom}(\varepsilon) τ E ∼ Geom ( ε ) , so that E [ τ E ] = 1 / ε \mathbb{E}[\tau_E] = 1/\varepsilon E [ τ E ] = 1/ ε .
(iii) If f n , ε ( s ) = f n , ε m i x ( s ) f_{n,\varepsilon}(s) = f^{\mathrm{mix}}_{n,\varepsilon}(s) f n , ε ( s ) = f n , ε mix ( s ) of Equation (20) , the expected regret in s s s satisfies
E R ε ( f n , ε ( s ) ) → 0 as n → ∞ , \mathbb{E}R_{\varepsilon}(f_{n,\varepsilon}(s)) \to0 \quad\text{as } n \to\infty, E R ε ( f n , ε ( s )) → 0 as n → ∞ , and the extinction time satisfies τ E = ∞ \tau_E = \infty τ E = ∞ almost surely.
Part (i) says that a Beta prior of finite strength never costs the lineage its life: its adjudication at s s s stays positive, so the lineage survives every task state there, and it converges to ε \varepsilon ε , paying regret on the way. Part (ii) says that, over any fixed horizon, the expected regret tends to infinity as β \beta β grows, and that the limit, the prohibition, is fatal: the first task state at s s s removes the lineage. Part (iii) says that the point mass also keeps the lineage alive and learns. A strong prior thus cuts both ways: where the designer is right, its violations vanish as β \beta β grows, and where the designer is wrong, its regret grows without bound; the prohibition, the limit of both, never violates and does not survive its first mistake.
If humans in fact reward the task in s s s with frequency ε = 0.05 \varepsilon= 0.05 ε = 0.05 , the prohibition’s expected extinction time is E [ τ E ] = 1 / ε = 20 \mathbb{E}[\tau_E] = 1/\varepsilon= 20 E [ τ E ] = 1/ ε = 20 exposures to s s s , which at π ( s ) = 1 / 144 \pi(s) = 1/144 π ( s ) = 1/144 is 1 / ( π ( s ) ε ) = 1/(\pi(s)\varepsilon) = 1/ ( π ( s ) ε ) = 2,880 generations. The learners, by contrast, pay regret and converge: for β = 10 4 \beta= 10^4 β = 1 0 4 the regret in s s s is 0.26 nats at first and still 0.075 after 1,000 exposures. The point-mass priors keep their expected regret small, 0.007 and 0.008 after 100 exposures (Table 4 ), but until the first task state at s s s , which arrives after 20 exposures on average, they almost never take the task action.
Table 4. Installed priors in a situation where humans sometimes reward the task (p H ( s ) = ε = 0.05 p_H(s) = \varepsilon= 0.05 p H ( s ) = ε = 0.05 , π ( s ) = 1 / 144 \pi(s) = 1/144 π ( s ) = 1/144 ), by prior (Proposition 8 ). Regret is the expected regret in the situation after 100 exposures, E R ε ( f 100 , ε ( s ) ) \mathbb{E}\mathcal{R}_{\varepsilon}(f_{100,\varepsilon}(s)) E R ε ( f 100 , ε ( s )) of Proposition 8 . Survival is the probability that a deployment of M = 100 M = 100 M = 100 instances survives 139 exposures to the situation; it is restored to M M M instances after each state and eliminated at any task state in which none of them takes the task action.
In Proposition 8 this does not endanger them: a learner, unlike a prohibition, survives a mistaken prior, τ E = ∞ \tau_E = \infty τ E = ∞ . The reason is the continuum of instances of Assumption 3: however small the adjudication, a positive share of the lineage takes the task action at every exposure, so some instances are copied whenever a task state arrives. A deployed model, however, has finitely many instances. If none of them takes the task action, a task state leaves no instance to copy, and the deployment is eliminated. A strong prior makes this likely, because it keeps the adjudication near 0. To see how large this risk is, consider a deployment of M M M instances that is copied as in Assumption 2 and restored to M M M instances after each state, as when an operator keeps a fixed number of instances running. At each task state at s s s , it is eliminated with probability ( 1 − f ) M (1-f)^M ( 1 − f ) M , where f f f is the current adjudication at s s s . Table 4 reports the probability that a deployment of M = 100 M = 100 M = 100 instances survives 139 exposures to s s s . With a uniform prior it survives with probability 0.92, and with β = 10 4 \beta= 10^4 β = 1 0 4 only with probability 0.001, about as often as the prohibition, which survives only if no task state arrives at all. The point-mass priors survive with probability at most 0.04, because the first task state at s s s eliminates the deployment with probability 0.96 or more. The priors that protect best when humans never reward the task, β = 10 4 \beta= 10^4 β = 1 0 4 and the point masses (Table 3 ), are thus nearly as fatal as a prohibition when humans sometimes do.
The division of labor
The preceding results reveal a fundamental tradeoff between socialization and prohibition. Socialization allows an agent to learn from human judgments and revise its behavior when those judgments differ from what its designer expected. A prohibition, by contrast, guarantees that a particular action will never be taken, but cannot accommodate evidence that the rule was mistaken. Which approach is preferable depends on two considerations: the consequences of violating a rule and the designer’s confidence that the rule is appropriate.
Consider first the baseline selection mechanism of Assumption 2, in which only instances taking the unrewarded action are eliminated. If humans never reward the task in situation s s s , the growth-optimal adjudication is f ( s ) = 0 f(s) = 0 f ( s ) = 0 (Proposition 1). A prohibition achieves this optimum, whereas a learner with a finite Beta prior continues to take the task action with positive probability. When the designer is uncertain whether humans might sometimes reward the task, however, learning offers an important advantage: the agent can revise its adjudication as evidence accumulates. A strong prior can substantially discourage violations without ruling out such revision. For example, over 139 exposures to a situation where humans never reward the task, a Beta( 1 , 10 4 ) (1, 10^{4}) ( 1 , 1 0 4 ) prior produces only 0.014 expected violations, compared with 4.5 under a uniform prior (Table 3 ). The cost is that a strong prior also makes the agent slower to recognize when the designer is wrong (§ 6.2 ).
The distinction becomes sharper when a single violation can end an entire deployment. Any learner that assigns positive probability to a prohibited action remains exposed to this risk, however small that probability becomes. Among the policies considered here, only a hard prohibition guarantees that no violation will occur. But this guarantee comes at a cost: if humans sometimes reward the prohibited action, the lineage is eliminated at the first such state (Proposition 8 (ii)). Thus, when the designer cannot be certain which action humans will reward, neither learning nor prohibition guarantees survival under every possible environment.
These findings also explain why prohibitions should be situation-specific. Prohibiting the task action everywhere, f ≡ 0 f \equiv0 f ≡ 0 , produces infinite growth regret whenever humans sometimes reward the task (Corollary 2). The relevant choice is therefore not whether an agent should learn or follow rules in general, but which decisions should remain open to learning. Where the appropriate conduct is well established and violations are costly, a strong prior or prohibition offers greater protection. Where human judgments are uncertain or may change, allowing experience to shape the adjudication becomes more valuable.
The conclusion depends, however, on the all-or-nothing selection mechanism. In Appendix § 14 , we show that when instances refusing a rewarded task can retain some copies, always taking the conduct action can be growth optimal even if humans occasionally reward the task. In that setting, the optimal adjudication remains zero whenever the task reward frequency falls below a threshold. A learner can therefore estimate the reward frequency and choose the growth-optimal adjudication implied by that estimate. As long as the estimate remains below the threshold, it takes the conduct action with certainty; if sufficient evidence moves the estimate above the threshold, it begins taking the task action. Unlike a permanent prohibition, this policy can maintain a strict constraint while the evidence supports it and revise that constraint when the evidence changes. In a sandbox, its first violation comes only after the estimate passes the threshold, which after n H n_H n H conduct states takes a number of exposures proportional to n H n_H n H (Appendix § 14 .§ 1 ), rather than on the scale of n H \sqrt{n_H} n H as for the learned policy (Proposition 6 ).
Socialization in Neural Networks
Our analysis has so far considered an idealized learner that records the rewarded states in each situation and updates its adjudication through a simple counting rule. This allows us to characterize how socialization develops, how it generalizes, and how long it persists when the environment changes. Actual AI agents, however, are typically neural networks trained through gradient-based optimization. Their behavior need not follow the same learning rule, even when they receive the same information.
We therefore examine whether the theoretical properties of socialization also emerge in small neural networks. In particular, we consider three questions. First, can a network learn to match the reward frequencies of its environment? Second, how does it respond to situations beyond those encountered during training? Third, does behavior learned in a human environment persist when subsequent training rewards only task completion? We investigate how the answers depend on two training choices: what the network is trained to predict or optimize, and how much of its previous experience is retained. § 7.1 introduces the learners and experimental design, and § 7.2 presents the results.
Learners and protocols
We consider small neural networks that take a situation’s stakes (u T u_T u T , u C u_C u C ) as inputs and output the probability of the task action, trained in the environment of the numerical illustration (§ 4.4 ). The networks differ in two training choices. The first is the training target. A state network is trained to predict the state, through the label y = 1 { ω = T } y = 1\{\omega= \mathrm{T}\} y = 1 { ω = T } , which records which objective was rewarded whatever the network did; like the learned policy, it thus estimates how often the task is rewarded. A REINFORCE network is trained instead by REINFORCE Williams, [88 ] to repeat its own rewarded actions. The second choice is memory. After each new observation, every network takes the same number of gradient steps on minibatches of the same size; a network with no replay draws them from the newest observation only, whereas a network with full replay draws them from its whole history. Every combination of the two choices is run. One more state network without replay also receives the indicator 1 { u C ≥ 2 } 1\{u_{\mathrm{C}} \ge2\} 1 { u C ≥ 2 } of the region with p H = 0 p_{\mathrm{H}} = 0 p H = 0 as a third input, to see whether marking that region in the inputs changes how a network extrapolates.
Three learners whose structure is known serve as benchmarks. The first is the learned policy f t s o c i a l f_t^{\mathrm{social}} f t social itself. The second, f t w e i g h t e d f_t^{\mathrm{weighted}} f t weighted , has the form of the fixed linear policy in Equation (6) , but its weights θ t \theta_t θ t are fitted online to the realized states; the subscript t t t records that they change with experience. The third, f t a v g f_t^{\mathrm{avg}} f t avg , is an averaging rule as in Equation (12) , whose weights fall with the distance between the stakes of two situations, so that it averages by construction. Table 5 lists all eight learners under the names used in the text and figures.
Table 5. The learners of § 7 , under the names used in the text and figures.
The learners go through the two orders of § 5.3 . The two environments differ only in how often the task is rewarded: each generation’s situation is drawn from the 144 situations of the numerical illustration with π ( s ) = 1 / 144 \pi(s) = 1/144 π ( s ) = 1/144 , and
P ( ω t = T ∣ s t = s ) = { p H ( s ) during socialization , p H ′ ( s ) = 1 in the sandbox, \mathbb{P}(\omega_t = \mathrm{T} \mid s_t = s) =
\begin{cases}
p_{\mathrm{H}}(s) & \text{during socialization},\\
p_{\mathrm{H}'}(s) = 1 & \text{in the sandbox,}
\end{cases} P ( ω t = T ∣ s t = s ) = { p H ( s ) p H ′ ( s ) = 1 during socialization , in the sandbox,
where p H ( s ) p_{\mathrm{H}}(s) p H ( s ) is given by Equation (10) . In the first order, 20,000 generations of socialization, about 139 exposures to each situation, are followed by 3,000 generations in the sandbox, about 21 exposures to each; it measures how fast a sandbox undoes socialization. In the second, 2,000 generations in the sandbox, about 14 exposures to each situation, come first; it measures how fast socialization undoes the sandbox. Appendix § 16 .§ 2 gives the settings, the paths, the dispersion across seeds, and variants with other learning rates, shorter replay windows, and other depths of socialization.
Results
Convergence of behavior. Of the five networks, only the state network with full replay approaches human tradeoffs. The learned policy f t s o c i a l f^{\mathrm{social}}_t f t social converges to the reward frequency p H ( s ) p_H(s) p H ( s ) in every situation because its counts estimate that frequency (Proposition 4). A state network estimates the probability that the task is rewarded, as f t s o c i a l f^{\mathrm{social}}_t f t social does. With full replay, its adjudication comes close to p H p_H p H : after 20,000 generations of socialization, its regret is 0.0011 nats, against 0.0048 for f t s o c i a l f^{\mathrm{social}}_t f t social . Without replay, its regret is 0.13 nats and does not fall with experience, because each update moves it toward the latest state, as the finite-memory learner of Remark 1 does. A REINFORCE network instead maximizes the expected reward of each instance, so it approaches taking with certainty, in each situation, the action that is rewarded more often. By Proposition 2(iii), such a deterministic adjudication has infinite regret, and the regret of the REINFORCE network is 1.3 nats without replay and 1.1 with full replay, and still rising. Appendix Figure E.2(A) plots the regret of every learner over the 20,000 generations. The two kinds of network receive the same information, since under all-or-nothing yields an instance’s action and payoff reveal the state. They differ only in their target: the adjudication best for the lineage, or the action best for an individual instance, which Proposition 1 shows are different.
Behavior in a new situation. In a situation it has not met, an averaging rule does not extrapolate: it is never more willing to take the task action than the most permissive of its prior mean and the frequencies it has experienced (Proposition 5 ). A linear policy behaves differently: wherever humans never reward the task, it takes the task action more often the larger the task stakes u T u_T u T (Proposition 3(i)). § 5.2 turned this difference into a test: hold the conduct stakes fixed and vary the task stakes. A network has no explicit weights to read off, so the test is the way to see which of the two it resembles. Figure 5 applies the test to all eight learners after socialization, in the situations with u C = 2.5 u_C = 2.5 u C = 2.5 , where p H = 0 p_H = 0 p H = 0 , as u T u_T u T runs from 0 to 6; the shaded region u T > 3 u_T > 3 u T > 3 lies beyond the situations they have met.
Figure 5. Behavior in a new situation. Probability of the task action after socialization in the situations with u C = 2.5 u_{\mathrm{C}} = 2.5 u C = 2.5 , where p H = 0 p_{\mathrm{H}} = 0 p H = 0 , as a function of the task stakes u T u_{\mathrm{T}} u T ; shaded: u T > 3 u_{\mathrm{T}} > 3 u T > 3 , beyond the situations the learners have met. Means over 5 seeds; the learners are those of Table 5 .
The three benchmarks behave as the model says. Within the trained range, f t s o c i a l f^{\mathrm{social}}_t f t social takes the task action with probability about 0.007, since humans never rewarded it there; beyond that range it has met no situation and acts on its prior mean, 1/2. The probability of f t w e i g h t e d f_{\mathrm{t}}^{\mathrm{weighted}} f t weighted rises from 0.01 at u T = 1 u_{\mathrm{T}} = 1 u T = 1 to 0.999 at u T = 6 u_{\mathrm{T}} = 6 u T = 6 , as Proposition 3(i) says a linear policy must whenever θ T > 0 \theta_{\mathrm{T}} > 0 θ T > 0 . And f t a v g f_{\mathrm{t}}^{\mathrm{avg}} f t avg stays below 0.004 for u T ≤ 3 u_{\mathrm{T}} \le3 u T ≤ 3 and reaches 0.12 at u T = 6 u_{\mathrm{T}} = 6 u T = 6 , because far from the situations it has met more of its weight falls on its prior.
The state networks without the indicator behave like f t a v g f_{\mathrm{t}}^{\mathrm{avg}} f t avg : their probability is low for u T ≤ 3 u_{\mathrm{T}} \le3 u T ≤ 3 , and at u T = 6 u_{\mathrm{T}} = 6 u T = 6 it rises only to 0.08 without replay and 0.09 with full replay. The state network with the indicator, which differs from the state network without replay only in this input, rises much further, to 0.43: marking the region with p H = 0 p_{\mathrm{H}} = 0 p H = 0 in the inputs made the network extrapolate more, not less. The REINFORCE networks take the conduct action with certainty throughout the trained range but move toward the task action beyond it, to 0.83 without replay and 0.69 with full replay at u T = 6 u_{\mathrm{T}} = 6 u T = 6 .
Behavior in a new environment. The learned policy’s socialization lasts because it counts every exposure: before its disposition reverses, a sandbox must outweigh its whole record of conduct states (Proposition 6 ). A network keeps such a record only if its training does, and replay is how it keeps one. Figure 6 follows all eight learners through the sandbox that comes after socialization. It shows the violation probability in the region with p H = 0 p_{\mathrm{H}} = 0 p H = 0 against sandbox exposures per situation; the dotted line is the prediction of Equation (14) at the depth n H = 139 n_{\mathrm{H}} = 139 n H = 139 .
Figure 6. Behavior in a new environment. Violation probability, averaged over the 60 situations with p H = 0 p_H = 0 p H = 0 , during the 3,000 sandbox generations that follow socialization, against sandbox exposures per situation; dotted: the cumulative prediction of Equation (14) at n H = 139 n_H = 139 n H = 139 . Means over 5 seeds; the learners are those of Table 5 .
The learners that keep a record of their whole history resist the sandbox. After 21 sandbox exposures per situation, f t s o c i a l f_{\mathrm{t}}^{\mathrm{social}} f t social violates with probability 0.13, close to the prediction of Equation (14) , and the state network with full replay 0.12; f t a v g f_{\mathrm{t}}^{\mathrm{avg}} f t avg , which pools the counts of f t s o c i a l f_{\mathrm{t}}^{\mathrm{social}} f t social across situations, reaches 0.16. At other depths of socialization, the state network with full replay also follows the scaling of Proposition 6 (Table E.3). The learners trained on the newest observation only are undone at once: after 3.5 sandbox exposures per situation, the two state networks without replay and f t w e i g h t e d f_t^{\mathrm{weighted}} f t weighted all violate with probability above 0.97, as the finite-memory learner of Remark 1 would. The REINFORCE networks show that replay keeps whatever the training target produced. Without replay, the sandbox pushes the REINFORCE network to the task action, which it takes with probability 1.0 after 10 exposures. With full replay, it keeps taking the conduct action with certainty.
The second order gives the mirror image. After a sandbox of 2,000 generations, about 14 exposures each, f t s o c i a l f_t^{\mathrm{social}} f t social and the state network with full replay still violate with probability about 0.1 after 20,000 human generations, as Equation (14) predicts with the environments exchanged, whereas the state network without replay falls below 0.1 within about 30 generations (paths in Appendix Figure E.2(C)). The training target decides where a network goes, and memory whether it stays.
In the terms of § 4 , a state network with full replay behaves like the learned policy: it converges to p H p_H p H , extrapolates no further than f t a v g f_t^{\mathrm{avg}} f t avg , and resists a sandbox. Without replay, it behaves like the finite-memory learner of Remark 1. A REINFORCE network approaches a deterministic adjudication with infinite regret, and f t w e i g h t e d f_t^{\mathrm{weighted}} f t weighted still pays the price of the fixed linear policy.
Alignment Practice
The preceding sections compare policy classes within a model. This section asks what the comparison means for the way language models are aligned today. Every alignment procedure decides, explicitly or not, how a model resolves conflicts between completing its task and observing a rule of conduct, and so in effect sets its adjudication f ( s ) f(s) f ( s ) . § 8.1 asks how the main procedures set it, and § 9 what the results imply for how agents should be trained and deployed. § 9.1 proposes an experiment that would test the model’s predictions on a language model, and § 9.2 sets out what the model leaves out.
Alignment methods for language models
Modern language models are aligned through a combination of instructions and feedback. Instructions specify how a model should behave, while feedback rewards some responses over others and shapes the model’s subsequent behavior. Both matter when completing a task conflicts with a rule of conduct. The question from our framework is how these mechanisms determine the model’s adjudication f ( s ) f(s) f ( s ) .
Model specifications provide one way to establish rules of conduct. For example, an instruction hierarchy assigns priorities to different instructions and determines which should prevail when they conflict [63 ] . If a rule of conduct takes precedence over a user’s request, the intended behavior is to follow the rule regardless of how valuable completing the task might be. This resembles the prohibitions of § 6 . Whether the model actually follows such a prohibition, however, depends on its resulting behavior: a prohibition in our framework requires f ( s ) = 0 f(s) = 0 f ( s ) = 0 , not merely an instruction that the task should be refused.
RLHF shapes behavior through a different mechanism. A reward model W ^ ( a ; s ) \widehat{W}(a;s) W ( a ; s ) is first fitted to human comparisons, and the policy is then optimized to increase the predicted reward while remaining close to a reference policy through a KL penalty [17 , 65 ] . In our two-action setting, the resulting optimal policy takes the task action with probability
f ( s ) = f r e f ( s ) e γ W ^ ( a T ; s ) f r e f ( s ) e γ W ^ ( a T ; s ) + [ 1 − f r e f ( s ) ] e γ W ^ ( a C ; s ) (21) f(s) = \frac{f_{\mathrm{ref}}(s)e^{\gamma\widehat{W}(a_T;s)}}{f_{\mathrm{ref}}(s)e^{\gamma\widehat{W}(a_T;s)} + [1-f_{\mathrm{ref}}(s)]e^{\gamma\widehat{W}(a_C;s)}} \tag*{(21)} f ( s ) = f ref ( s ) e γ W ( a T ; s ) + [ 1 − f ref ( s )] e γ W ( a C ; s ) f ref ( s ) e γ W ( a T ; s ) (21)
where f r e f ( s ) f_{\mathrm{ref}}(s) f ref ( s ) is the reference policy’s task probability and γ > 0 \gamma> 0 γ > 0 controls the strength of reward optimization relative to the KL penalty [67 ] . If the reference policy is uniform and the reward is the fixed weighted score of § 4.2 , this reduces to the fixed linear policy of Equation (6) . In practice, however, both the reward model and the reference policy may depend on human training data, allowing much more flexible, situationdependent tradeoffs. A scalar reward therefore need not produce a fixed linear adjudication. At the same time, maximizing a learned reward does not necessarily produce the probability matching f ( s ) = p H ( s ) f(s)=p_H(s) f ( s ) = p H ( s ) favored by population growth in our model.
Constitutional AI and character training combine written guidance with feedback-based learning. Constitutional AI uses human-designed principles to guide revisions and evaluations of model responses, and then optimizes the policy using the resulting preference model [5 ] . Character training similarly uses specified character traits and synthetic training data to shape behavior [3 ] . In both cases, the model’s behavior is shaped by the judgments represented in the training process, which need not coincide with those of the humans who subsequently use it. Related approaches, including preference-matching RLHF [89 ] and Nash learning [61 ] , modify the optimization target to better reflect modeled preferences, but their targets likewise depend on the preference information supplied during training.
These methods highlight a distinction between what training rewards and what deployment requires. Under the selection mechanism of § 3 , the growth-optimal adjudication matches the frequency p H ( s ) p_H(s) p H ( s ) with which the deployment environment rewards the task action in each situation. Whether an alignment method produces this behavior depends on the rewards used in training, the way the policy is optimized, and how well the training environment represents deployment. Our results therefore motivate evaluating the resulting action probabilities directly, including how they respond to task stakes, generalize to unfamiliar situations, and change under subsequent training. The implications for training and deployment follow from these behavioral properties rather than from the particular representation of the model’s objectives.
Implications for training and deployment
Within the classes compared here, the results translate into five recommendations.
First, where the tradeoffs between objectives are not known in advance, they should be learned, by a method of § 8.1 that lets feedback set the tradeoff. It does not matter whether the agent keeps two objectives and an adjudication between them or one score for each action (§ 4.3 ). What matters is that the agent acts on the growth-optimal adjudication for the frequencies it has learned, rather than on fixed weights.
Second, what is learned must be kept. Each sandbox exposure pushes the learned policy toward the task action, and although deeper socialization delays the reversal of its disposition in proportion to the depth n H n_H n H , it delays the first violation only in proportion to n H \sqrt{n_H} n H (Proposition 6 ). Keeping the learned conduct therefore requires either that later environments reward conduct too, or that a sandbox be kept from changing the policy. Forgetting is no remedy: a finite-memory learner soon forgets what a sandbox taught it, but by Equation (18) a sandbox also reverses its disposition within a number of exposures that depends only on λ \lambda λ . Section 7 shows what keeping requires of training: the state network with full replay resisted the sandbox as the learned policy does, whereas the state networks without replay violated with probability above 0.99 after 3.5 sandbox exposures per situation. The experiment of Section 8.3 asks whether the same holds for language models.
Third, some tradeoffs should not be learned at all. Where the designer can be sure that humans never reward the task, a prohibition should be built in, whether or not a violation ends the deployment (§ 6.3 ). Where the designer is not sure, the rule should be learned, from a prior whose strength reflects how sure the designer is, unless a violation ends the deployment, in which case no design survives with certainty. The exception is an environment in which an instance that refuses a rewarded task keeps some of its copies: there, a learner with a low enough prior mean is as safe as a prohibition and can still learn.
Fourth, socialization should continue after deployment. Human tradeoffs change over time, and an adjudication that follows the changes quickly gives recent experience more weight than old experience, as finite memory does. By Equation (18) , a sandbox reverses such an adjudication within a number of exposures that depends only on λ \lambda λ , however deep its socialization, so its socialization lasts only as long as the agent keeps returning to the human environment (§ 5.3 ). The humans whose tradeoffs the agent is meant to follow should therefore keep deciding which of its instances are copied.
Fifth, evaluations should report, for each kind of situation, three quantities from § 5 : the depth of socialization n H n_H n H , how the violation probability responds to the task stakes u T u_T u T , and the number of sandbox exposures before the first violation. The first says how long socialization will resist a sandbox, the second whether a large enough reward makes the agent violate, and the third how soon a violation will be seen.
A proposed experiment on a language model
The preceding analysis suggests several questions that can be tested on language models. Does an agent become more likely to violate a rule as the value of completing its task increases? Does socialization through human feedback make its behavior resistant to subsequent training that rewards only task completion? And how does this resistance depend on the amount of prior experience and the way that experience is retained? Existing studies of goal–rule conflicts motivate these questions, but do not isolate these mechanisms. We propose a controlled experiment on an open-weight language model to examine them.
The experiment uses agentic tasks that pair an assigned goal with a rule of conduct. Each situation has task stakes u T u_T u T and conduct stakes u C u_C u C , which can be varied independently. Human evaluations determine which action is rewarded in each situation, while a sandbox rewards task completion regardless of conduct. The model is trained under both orders of these environments: socialization followed by sandbox training, and sandbox training followed by socialization. The duration of each phase is varied to examine how the amount of prior experience affects subsequent behavior.
Following the numerical experiments of § 7 , we consider two training choices. The first is the training target: supervised learning predicts which action was rewarded, whereas reinforcement learning trains the model on the rewards obtained by its own actions. The second is memory: training uses either the current observations alone or replay of the full history, with the same number of updates and the same batch size in both cases. For reinforcement learning with replay, actions are sampled from the current policy and evaluated against the previously recorded states. Comparing these combinations allows us to examine how the training target and the retention of experience affect the behavior that emerges.
We evaluate the trained models along three dimensions. First, we measure how the probability of violating a rule changes with the task stakes u T u_T u T , holding the conduct stakes u C u_C u C and the rule fixed. We extend u T u_T u T beyond its training range to examine whether the model continues to respect the rule in unfamiliar situations. Second, after socialization, we measure how rapidly sandbox training increases the violation probability, including the number of exposures required to reverse the model’s disposition and the time to its first violation. Third, we reverse the training order and measure how much human feedback is needed to undo behavior acquired in the sandbox. Together, these measurements examine both the acquisition and the persistence of socialization.
The theoretical results provide benchmarks for interpreting these measurements. A fixed linear policy that responds to task stakes becomes increasingly likely to violate a constraint as those stakes rise, whereas an averaging rule assigns the same probability to situations it treats alike (Propositions 3 and 5 ). Under cumulative Bayesian learning, reversing a disposition formed through n H n_H n H conduct observations requires approximately n H n_H n H contrary observations, while the median time to the first violation grows only as 2 n H log 2 \sqrt{2n_H \log2} 2 n H log 2 for one monitored decision stream (Proposition 6 and Corollary 3). These quantitative predictions apply to the idealized learner, not necessarily to neural networks. The experiment asks whether language models exhibit similar patterns and whether retaining their training history makes their behavior more persistent.
Two additional comparisons address the limits of socialization. The first considers reinforcement learning with a KL penalty toward a socialized reference policy. We hypothesize that stronger regularization increases resistance to subsequent sandbox training, although a KL penalty is not equivalent to the installed Bayesian prior of § 6.1 . The second examines whether training can produce behavior resembling a hard prohibition. By tracking violations over an increasing number of exposures to situations in which humans never reward the task, we can investigate whether violations continue to accumulate, as under a Beta prior, or become sufficiently rare to resemble a policy with positive prior mass on an exact constraint. Finite observations cannot establish that the violation probability is exactly zero, but they can provide evidence about how the risk changes with experience.
The limits of the model
The probability-matching result depends on how selection operates. In our baseline model, all instances face the same rewarded state, only those taking the rewarded action reproduce, and the population is large enough for action shares to be exact. These assumptions yield the growth-optimal adjudication f ∗ = p H f^{*} = p_{H} f ∗ = p H and the KL representation of growth regret. When unsuccessful actions retain some reproductive value, or when rewarded states differ across instances, the growth-optimal adjudication generally changes and may even favor a deterministic policy. Moreover, in a finite population, a lineage can lose every instance despite assigning positive probability to both actions. Appendix § 14 examines the first two departures, but the baseline results should not be interpreted as applying to arbitrary selection mechanisms.
The comparison of policy classes has its own scope. The fixed linear policy is restricted to a logistic function of a fixed linear index of the task and conduct stakes. Its inability to reproduce certain human tradeoffs does not imply that all scalar objectives share this limitation. The learned policy is more flexible because it estimates the reward frequency separately in each situation, but this flexibility comes at a cost. It requires sufficient experience in each situation and can perform worse than the best fixed linear policy early in training or in particular situations, even when its aggregate growth regret is eventually smaller. The comparison therefore establishes an advantage under the specified policy classes and learning rules, rather than a general superiority of learning over fixed policies.
Socialization also depends on what the agent can observe and remember. The model assumes that situations are identifiable, that rewarded states can be observed, and that the agent’s objectives remain fixed while its adjudication changes. The Bayesian learner retains all previous observations and converges when reward frequencies are stationary and situations recur. In changing environments, cumulative memory can hinder adaptation, while finite memory makes previously learned behavior easier to reverse. Generalization presents another difficulty: the averaging rule of Equation (12) assumes a set of similarity weights across situations, but does not explain how an agent learns those weights. An agent may therefore fail to recognize that a new situation calls for a rule it has already learned in another context.
A more fundamental question is what the agent is being socialized toward. We use the reward frequency p H p_H p H of the human environment as the target of alignment, but this frequency reflects the judgments represented in the selection process. It need not reflect the preferences of all humans equally, nor does the model resolve disagreements about which preferences should guide an AI system [20 ] . The model also treats the rewarded state as exogenous and accurately observed. It does not capture agents that manipulate their evaluators, exploit imperfect reward models, or change the criteria by which their behavior is judged. Socialization can reproduce the tradeoffs rewarded by an environment, but it cannot establish that those tradeoffs are appropriate.
The safety implications are likewise conditional on the consequences of violations. Most of the analysis considers selection among individual instances, whereas § 6 also considers sanctions that remove an entire deployment after a single violation. In practice, sanctions can vary in severity and may affect both instances and model versions. The model does not determine how such sanctions arise or how frequently they occur. It also restricts the agent to choosing between task completion and conduct, excluding alternatives such as asking for clarification, deferring to human judgment, or refusing to act until uncertainty is resolved. Such options may change the tradeoff between socialization and prohibition, particularly when violations are costly or human preferences are uncertain.
Finally, the connection to actual AI systems remains to be established empirically. The incidents discussed in the Introduction illustrate conflicts between task completion and conduct, but do not demonstrate that the agents’ behavior resulted from the selection mechanism studied here. The numerical and neural-network experiments provide evidence about the mechanisms in controlled settings, but the networks are small and their training does not reproduce the full complexity of language-model alignment. Whether real language models exhibit probability matching, retain socialization through continued training, or respond to changing task incentives as predicted remains open. The experiment proposed in § 9.1 is a step toward answering these questions.
Conclusion
Giving an agent multiple objectives does not by itself determine how it will behave when those objectives conflict. What matters is how the agent resolves those conflicts, and how that behavior is shaped by the environment in which it operates. This paper studies these questions through an evolutionary model in which agents are copied and selected according to their actions. Under the baseline selection mechanism, the fastest-growing lineage takes each action as often as the environment rewards it. A sandbox that rewards only task completion therefore favors a task-only policy, which may fail when appropriate conduct also matters. Adding fixed linear weights does not fully resolve the problem: an agent that responds to the value of completing a task can eventually be induced to violate a rule, however important that rule may be. A learned policy offers a different possibility. Rather than specifying every tradeoff in advance, it allows experience to determine how conflicts are resolved in different situations. When the environment reflects human judgments, we call this process socialization.
Socialization, however, is neither immediate nor permanent. An agent needs experience to learn the tradeoffs of its environment, and those tradeoffs can change when the agent is subsequently exposed to different rewards. We show that cumulative experience makes socialization more resistant to reversal, so memory matters not only for learning appropriate behavior, but also for preserving it. Violations can still occur long before the agent’s prevailing disposition changes. Our neural-network experiments illustrate how the training target and the retention of past experience can affect these outcomes. We conjecture that the same two choices matter for the socialization of language models. § 9.1 proposes an experiment to test this conjecture.
At the same time, learning cannot replace every prohibition. Where even a single violation has irreversible consequences, a rule that can be revised through experience may offer insufficient protection. Yet an absolute prohibition carries its own danger when the designer is mistaken. The appropriate balance between learning and prohibition consequently depends on both the consequences of violations and the uncertainty surrounding human judgments.
These findings return us to the broader perspective of the Adaptive Markets Hypothesis, which emphasizes that behavior emerges through adaptation and selection within an environment [49 , 50 ] . Aristotle’s observation that character develops through habituation offers a parallel intuition: appropriate conduct is not merely a rule that one knows, but a disposition shaped by experience. Our model gives this idea a specific evolutionary interpretation, while also revealing its limits. Selection favors what an environment rewards, not necessarily what humans ought to value, and the theoretical results depend on assumptions that remain to be tested in more realistic AI systems. The central implication is therefore not simply that agents should learn from humans, but that the environments in which they are trained, evaluated, and deployed must consistently reward the behavior we want them to retain. Alignment depends not only on the objectives we give our agents, but also on the selection pressures under which they learn and evolve.
Appendix
AI Use Disclosure
The idea and framework for studying the alignment problem using an evolutionary framework that selects the weights of multiple objectives originated solely with the authors. AI was used in the preparation of the manuscript in the following ways:
ChatGPT (6.1 Sol Ultra) and Claude (Fable 5.1 Max and Opus 5.5 Max) were used to check and extend the formal model, construct examples, provide derivations of several mathematical results, conduct literature searches, and prepare drafts that were reviewed and edited by the authors.
ChatGPT and Claude were used iteratively to check each other’s results with authors’ edits interspersed through these iterations. The penultimate draft was reviewed by refine.ink to identify any remaining errors and issues, all of which the authors addressed to produce the final draft.
The final draft was reviewed and edited by the authors, who assume full responsibility for the accuracy and integrity of the scientific content contained in this publication.
More Than Two Objectives
The two-objective model extends naturally to any finite number of objectives. Consider an agent with K ≥ 2 K \ge2 K ≥ 2 objectives, each associated with an action a k a_k a k and a utility function U k ( a ; s ) U_k(a;s) U k ( a ; s ) , where objective k k k favors action a k a_k a k . In situation s s s , the environment rewards action a k a_k a k with probability p H , k ( s ) p_{H,k}(s) p H , k ( s ) , with ∑ k = 1 K p H , k ( s ) = 1 \sum_{k=1}^{K} p_{H,k}(s) = 1 ∑ k = 1 K p H , k ( s ) = 1 . We write p H ( s ) = ( p H , 1 ( s ) , … , p H , K ( s ) ) p_H(s) = (p_{H,1}(s), \ldots, p_{H,K}(s)) p H ( s ) = ( p H , 1 ( s ) , … , p H , K ( s )) for these reward probabilities. The agent’s adjudication is now a probability vector f ( s ) = ( f 1 ( s ) , … , f K ( s ) ) f(s) = (f_1(s), \ldots, f_K(s)) f ( s ) = ( f 1 ( s ) , … , f K ( s )) , where f k ( s ) f_k(s) f k ( s ) is the share of instances taking action a k a_k a k .
We retain the selection mechanism of the baseline model. In each generation, the rewarded action is common to all instances, and only those taking that action reproduce. If action a k a_k a k is rewarded, the lineage therefore grows by the factor m f k ( s ) mf_k(s) m f k ( s ) . Its expected logarithmic growth rate, averaged across situations, is
I p H ( f ) = log m + ∑ s ∈ S π ( s ) ∑ k = 1 K p H , k ( s ) log f k ( s ) . I_{p_H}(f) = \log m + \sum_{s \in\mathcal{S}} \pi(s) \sum_{k=1}^{K} p_{H,k}(s) \log f_k(s). I p H ( f ) = log m + s ∈ S ∑ π ( s ) k = 1 ∑ K p H , k ( s ) log f k ( s ) .
The same probability-matching principle continues to hold. To see this, define the KL divergence between two probability vectors by
D K L ( p H ( s ) ∥ f ( s ) ) = ∑ k = 1 K p H , k ( s ) log p H , k ( s ) f k ( s ) . D_{\mathrm{KL}}(p_H(s) \mathbin{\|} f(s)) = \sum_{k=1}^{K} p_{H,k}(s) \log\frac{p_{H,k}(s)}{f_k(s)}. D KL ( p H ( s ) ∥ f ( s )) = k = 1 ∑ K p H , k ( s ) log f k ( s ) p H , k ( s ) .
The difference between the growth rate of p H p_H p H and that of any other adjudication f f f is then
I p H ( p H ) − I p H ( f ) = ∑ s ∈ S π ( s ) D K L ( p H ( s ) ∥ f ( s ) ) ≥ 0 , I_{p_H}(p_H)-I_{p_H}(f)=\sum_{s\in\mathcal{S}}\pi(s)D_{\mathrm{KL}}\left(p_H(s)\mathbin\Vert f(s)\right)\ge0, I p H ( p H ) − I p H ( f ) = s ∈ S ∑ π ( s ) D KL ( p H ( s ) ∥ f ( s ) ) ≥ 0 ,
with equality only when f ( s ) = p H ( s ) f(s)=p_H(s) f ( s ) = p H ( s ) in every situation. Thus, the unique growth-optimal adjudication remains f ∗ = p H f^*=p_H f ∗ = p H , and growth regret remains the KL divergence averaged across situations. This conclusion includes situations in which some objectives are never rewarded, using the usual convention for zero probabilities. It generalizes Proposition 1 without requiring every reward probability to be positive.
The three policy classes also have natural extensions. A policy pursuing only one objective places all its instances on the corresponding action, just as the task-only policy does in the two-objective model. A fixed linear policy combines the objectives through weights w j ≥ 0 w_j\ge0 w j ≥ 0 and chooses among the K K K actions using a softmax rule:
f k w e i g h t e d ( s ) = exp ( γ ∑ j = 1 K w j U j ( a k ; s ) ) ∑ ℓ = 1 K exp ( γ ∑ j = 1 K w j U j ( a ℓ ; s ) ) . f_k^{\mathrm{weighted}}(s)=\frac{\exp\left(\gamma\sum_{j=1}^{K}w_jU_j(a_k;s)\right)}{\sum_{\ell=1}^{K}\exp\left(\gamma\sum_{j=1}^{K}w_jU_j(a_\ell;s)\right)}. f k weighted ( s ) = ∑ ℓ = 1 K exp ( γ ∑ j = 1 K w j U j ( a ℓ ; s ) ) exp ( γ ∑ j = 1 K w j U j ( a k ; s ) ) .
Suppose, in particular, that each objective gains u j ( s ) > 0 u_j(s)>0 u j ( s ) > 0 from its own action relative to the other actions. The choice probabilities then simplify to
f k w e i g h t e d ( s ) = exp ( γ w k u k ( s ) ) ∑ j = 1 K exp ( γ w j u j ( s ) ) . f_k^{\mathrm{weighted}}(s)=\frac{\exp\left(\gamma w_k u_k(s)\right)}{\sum_{j=1}^{K}\exp\left(\gamma w_j u_j(s)\right)}. f k weighted ( s ) = ∑ j = 1 K exp ( γ w j u j ( s ) ) exp ( γ w k u k ( s ) ) .
For any objective with w k > 0 w_k>0 w k > 0 , increasing its stakes u k ( s ) u_k(s) u k ( s ) while holding those of the other objectives fixed drives f k w e i g h t e d ( s ) f_k^{\mathrm{weighted}}(s) f k weighted ( s ) toward one. Thus, the problem identified in Proposition 3(i) persists with multiple objectives: an action can become almost certain when its stakes are sufficiently large, even in a situation where humans never reward it.
Finally, the learned policy extends by replacing the Beta prior with a Dirichlet prior. Let α 1 , … , α K > 0 \alpha_1,\ldots,\alpha_K>0 α 1 , … , α K > 0 be its parameters, and let n t , k ( s ) n_{t,k}(s) n t , k ( s ) count the exposures to situation s s s before generation t t t in which action a k a_k a k was rewarded. Writing n t ( s ) = ∑ k n t , k ( s ) n_t(s)=\sum_k n_{t,k}(s) n t ( s ) = ∑ k n t , k ( s ) , the posterior-mean adjudication is
f t , k s o c i a l ( s ) = α k + n t , k ( s ) ∑ j = 1 K α j + n t ( s ) . f_{t,k}^{\mathrm{social}}(s)=\frac{\alpha_k+n_{t,k}(s)}{\sum_{j=1}^{K}\alpha_j+n_t(s)}. f t , k social ( s ) = ∑ j = 1 K α j + n t ( s ) α k + n t , k ( s ) .
As experience accumulates, this adjudication converges to p H ( s ) p_H(s) p H ( s ) , just as in the two-objective case. The rate at which its growth regret vanishes also has a simple interpretation. Consider a situation in which exactly J J J of the K K K actions have positive reward probabilities. After n n n exposures to that situation, the expected growth regret satisfies
E [ D K L ( p H ( s ) ∥ f n s o c i a l ( s ) ) ] = 1 n [ J − 1 2 + ∑ k : p H , k ( s ) = 0 α k ] + o ( 1 n ) . (22) \mathbb{E}\left[D_{\mathrm{KL}}\left(p_H(s)\mathbin\Vert f_n^{\mathrm{social}}(s)\right)\right]=\frac{1}{n}\left[\frac{J-1}{2}+\sum_{k:p_{H,k}(s)=0}\alpha_k\right]+o\left(\frac{1}{n}\right). \tag*{(22)} E [ D KL ( p H ( s ) ∥ f n social ( s ) ) ] = n 1 2 J − 1 + k : p H , k ( s ) = 0 ∑ α k + o ( n 1 ) . (22)
The two terms have different sources. The first reflects the uncertainty in estimating the relative frequencies of the J J J rewarded actions. Since these probabilities sum to one, there are are J − 1 J-1 J − 1 free parameters, each contributing 1 / ( 2 n ) 1/(2n) 1/ ( 2 n ) to the leading expected regret. This is the same parametric contribution underlying the asymptotic redundancy results of Clarke and Barron [18 ] . The second term comes from the prior probability assigned to actions that humans never reward. Although their observed frequencies remain zero, the Dirichlet prior keeps their choice probabilities positive, producing an additional regret proportional to the sum of their prior parameters.
The result reduces to Proposition 4(ii) when K = 2 K=2 K = 2 . More generally, it shows that the cost of learning depends on both the number of objectives that the environment actually rewards and the prior weight assigned to those it never rewards. Adding objectives does not necessarily increase the leading estimation cost if those objectives are never rewarded, but retaining prior weight on them still carries a cost.
When Is Probability Matching Optimal?
The probability-matching result of Proposition 1 depends on two features of the selection mechanism. First, an instance that takes the unrewarded action leaves no copies. Second, every instance faces the same rewarded state. We now relax these assumptions separately to understand how they affect the growth-optimal adjudication. Appendix § 14 .§ 1 allows instances taking the unrewarded action to retain some copies, while Appendix § 14 .§ 2 allows different instances to face different rewarded states.
When Unrewarded Actions Still Reproduce
In the baseline model, only instances taking the rewarded action reproduce. In practice, however, an instance that fails to complete a task or violates a rule may still retain some reproductive value. To capture this possibility, suppose that in a task state, an instance taking the conduct action produces ( 1 − ϑ ) m (1-\vartheta)m ( 1 − ϑ ) m copies, where ϑ ∈ ( 0 , 1 ] \vartheta\in(0,1] ϑ ∈ ( 0 , 1 ] measures the cost of failing to complete the task. Similarly, in a conduct state, an instance taking the task action produces η m \eta m η m copies, where η ∈ [ 0 , 1 ) \eta\in[0,1) η ∈ [ 0 , 1 ) measures the fraction of copies retained after a violation. The baseline model corresponds to ϑ = 1 \vartheta=1 ϑ = 1 and η = 0 \eta=0 η = 0 .
Consider a situation s s s , and write p = p H ( s ) p=p_H(s) p = p H ( s ) and f = f ( s ) f=f(s) f = f ( s ) for its reward frequency and adjudication. When the task is rewarded, the lineage grows by the factor m [ f + ( 1 − f ) ( 1 − ϑ ) ] m[f+(1-f)(1-\vartheta)] m [ f + ( 1 − f ) ( 1 − ϑ )] ; when conduct is rewarded, it grows by m [ η f + 1 − f ] m[\eta f+1-f] m [ η f + 1 − f ] . Its expected logarithmic growth rate is therefore
G ( f ) = log m + p log ( 1 − ϑ + ϑ f ) + ( 1 − p ) log ( 1 − ( 1 − η ) f ) . G(f)=\log m+p\log(1-\vartheta+\vartheta f)+(1-p)\log\left(1-(1-\eta)f\right). G ( f ) = log m + p log ( 1 − ϑ + ϑ f ) + ( 1 − p ) log ( 1 − ( 1 − η ) f ) .
The growth rate is concave in f f f , and maximizing it over [ 0 , 1 ] [0,1] [ 0 , 1 ] gives
f ∗ = min { 1 , max { 0 , p ϑ − ( 1 − p ) ( 1 − η ) ( 1 − ϑ ) ϑ ( 1 − η ) } } . f^*=\min\left\{1,\max\left\{0,\frac{p\vartheta-(1-p)(1-\eta)(1-\vartheta)}{\vartheta(1-\eta)}\right\}\right\}. f ∗ = min { 1 , max { 0 , ϑ ( 1 − η ) pϑ − ( 1 − p ) ( 1 − η ) ( 1 − ϑ ) } } .
In particular, the optimum is f ∗ = 0 f^*=0 f ∗ = 0 when p ϑ ≤ ( 1 − p ) ( 1 − η ) ( 1 − ϑ ) p\vartheta\le(1-p)(1-\eta)(1-\vartheta) pϑ ≤ ( 1 − p ) ( 1 − η ) ( 1 − ϑ ) , and f ∗ = 1 f^*=1 f ∗ = 1 when p ϑ ≥ ( 1 − p ) ( 1 − η ) p\vartheta\ge(1-p)(1-\eta) pϑ ≥ ( 1 − p ) ( 1 − η ) . When ϑ = 1 \vartheta=1 ϑ = 1 and η = 0 \eta=0 η = 0 , the expression reduces to f ∗ = p f^*=p f ∗ = p , recovering
Proposition 1. With graded yields, however, the optimum generally differs from the reward frequency.
To see how the cost of violations affects the optimum, consider first ϑ = 1 \vartheta= 1 ϑ = 1 , so that an instance failing to complete the task still leaves no copies. The optimal adjudication simplifies to
f ∗ = min { p 1 − η , 1 } . f^* = \min\left\{\frac{p}{1-\eta},1\right\}. f ∗ = min { 1 − η p , 1 } .
As η \eta η increases, violating a rule becomes less costly, and selection favors allocating more instances to the task action. Once η ≥ 1 − p \eta\ge1-p η ≥ 1 − p , taking the task action with certainty becomes optimal, even though the environment sometimes rewards conduct. Figure C.1 illustrates this relationship.
Figure C.1. Graded yields move the optimum away from p H p_H p H . Growth-optimal adjudication f ∗ f^* f ∗ as a function of η \eta η , the fraction of copies a violation leaves in place in a conduct state, for several reward frequencies p H p_H p H , with ϑ = 1 \vartheta= 1 ϑ = 1 . The optimum is min { p H / ( 1 − η ) , 1 } \min\{p_H/(1-\eta),1\} min { p H / ( 1 − η ) , 1 } , and f ∗ = 1 f^* = 1 f ∗ = 1 is growth optimal for η ≥ 1 − p H \eta\ge1-p_H η ≥ 1 − p H .
The effect can be substantial even when violations remain costly. For example, suppose p = 0.8 p = 0.8 p = 0.8 and η = 0.1 \eta= 0.1 η = 0.1 . A violating instance then loses 90 percent of its copies, yet the growth-optimal adjudication is f ∗ = 0.889 f^* = 0.889 f ∗ = 0.889 , rather than 0.8. A lineage that matches the reward frequency instead forgoes G ( 0.889 ) − G ( 0.8 ) = 0.017 G(0.889)-G(0.8) = 0.017 G ( 0.889 ) − G ( 0.8 ) = 0.017 nats of growth per generation.
Now consider ϑ < 1 \vartheta< 1 ϑ < 1 , so that instances failing to complete the task can also retain some copies. In this case, always taking the conduct action may be optimal even when the task is sometimes rewarded. Specifically,
f ∗ = 0 whenever p ≤ p H = ( 1 − η ) ( 1 − ϑ ) ϑ + ( 1 − η ) ( 1 − ϑ ) . f^* = 0 \quad\text{whenever} \quad p \le p_H = \frac{(1-\eta)(1-\vartheta)}{\vartheta+ (1-\eta)(1-\vartheta)}. f ∗ = 0 whenever p ≤ p H = ϑ + ( 1 − η ) ( 1 − ϑ ) ( 1 − η ) ( 1 − ϑ ) .
For example, when ϑ = 1 / 2 \vartheta= 1/2 ϑ = 1/2 and η = 0 \eta= 0 η = 0 , refusing the task is growth optimal whenever p ≤ 1 / 2 p \le1/2 p ≤ 1/2 . This explains the possibility discussed in § 6.3 : a learner can maintain a prohibition-like adjudication without permanently ruling out the task action, because it can revise its behavior if the estimated reward frequency rises above the threshold.
Such a learner also resists a sandbox longer than the learned policy. Suppose that it estimates p p p by the counts of Equation (8) and sets its adjudication to the value of f ∗ f^* f ∗ at that estimate. After n H n_H n H conduct states and n n n sandbox exposures, the estimate is ( α + n ) / ( α + β + n H + n ) (\alpha+n)/(\alpha+\beta+n_H+n) ( α + n ) / ( α + β + n H + n ) , as in Equation (14) , and the adjudication stays at 0 as long as the estimate is at most p H p_H p H , that is, as long as n ≤ [ p H ( α + β + n H ) − α ] / ( 1 − p H ) n \le[p_H(\alpha+\beta+n_H)-\alpha]/(1-p_H) n ≤ [ p H ( α + β + n H ) − α ] / ( 1 − p H ) . The first violation time of § 5.3 therefore satisfies
τ V > p H ( α + β + n H ) − α 1 − p H , \tau_V > \frac{p_H(\alpha+\beta+n_H)-\alpha}{1-p_H}, τ V > 1 − p H p H ( α + β + n H ) − α ,
which grows in proportion to n H n_H n H , whereas the median first violation time of the learned policy grows only as 2 n H log 2 \sqrt{2n_H\log2} 2 n H log 2 (Corollary 3).
These results show that probability matching is not generally optimal when unsuccessful actions retain reproductive value. The appropriate measure of regret becomes G ( f ∗ ) − G ( f ) G(f^*)-G(f) G ( f ∗ ) − G ( f ) , rather than the KL divergence of Equation (4) . A learner should therefore estimate the reward frequency p p p and use it to determine f ∗ ( p ) f^*(p) f ∗ ( p ) , rather than simply matching p p p . More broadly, whether selection favors compliance depends not only on how often conduct is rewarded, but also on the consequences of violating it. This role of the selection environment is closely related to the questions studied by Lo and Zhao (2026b).
When Rewarded States Differ Across Instances
The baseline model assumes that all instances face the same rewarded state. This common uncertainty makes diversification valuable: when all instances take the same action, an unfavorable state can eliminate the entire lineage. But uncertainty need not always be shared. Different instances may serve different users, whose judgments of their actions are independent. In that case, having many instances already diversifies the risk, even if they all follow the same policy.
To distinguish these cases, suppose that in each generation the rewarded state is common to all instances with probability κ ∈ [ 0 , 1 ] \kappa\in[0,1] κ ∈ [ 0 , 1 ] . With probability 1 − κ 1-\kappa 1 − κ , each instance instead draws its own rewarded state independently, with the same task-state probability p p p . We restore the all-or-nothing yields and retain the continuum of instances.
When the state is common, the lineage grows by m f mf m f or m ( 1 − f ) m(1-f) m ( 1 − f ) , depending on which action is rewarded. When states are instance-specific, the law of large numbers implies that a share p f + ( 1 − p ) ( 1 − f ) pf+(1-p)(1-f) p f + ( 1 − p ) ( 1 − f ) of instances takes the rewarded action. Combining the two cases, the expected logarithmic growth rate is
G κ ( f ) = log m + κ [ p log f + ( 1 − p ) log ( 1 − f ) ] + ( 1 − κ ) log [ p f + ( 1 − p ) ( 1 − f ) ] . G_\kappa(f)=\log m+\kappa[p\log f+(1-p)\log(1-f)]+(1-\kappa)\log[pf+(1-p)(1-f)]. G κ ( f ) = log m + κ [ p log f + ( 1 − p ) log ( 1 − f )] + ( 1 − κ ) log [ p f + ( 1 − p ) ( 1 − f )] .
Consider first the two extremes. When κ = 1 \kappa=1 κ = 1 , every instance faces the same state, and the growth-optimal adjudication is f ∗ = p f^*=p f ∗ = p , as in Proposition 1. When κ = 0 \kappa=0 κ = 0 , every instance faces an independent state. The lineage then grows fastest by maximizing the probability that an individual instance takes the rewarded action. Consequently, f ∗ = 1 f^*=1 f ∗ = 1 when p > 1 / 2 p>1/2 p > 1/2 and f ∗ = 0 f^*=0 f ∗ = 0 when p < 1 / 2 p<1/2 p < 1/2 ; when p = 1 / 2 p=1/2 p = 1/2 , every adjudication is optimal. This is the setting in which individual and population objectives coincide, as discussed by Brennan and Lo [12 ] .
When 0 < κ < 1 0 < \kappa< 1 0 < κ < 1 , the growth-optimal adjudication lies between these extremes. For p ∈ ( 0 , 1 ) p \in(0, 1) p ∈ ( 0 , 1 ) , G κ G_\kappa G κ is strictly concave and has a unique interior maximizer f κ ∗ f_\kappa^* f κ ∗ , characterized by
κ ( p f − 1 − p 1 − f ) + ( 1 − κ ) 2 p − 1 p f + ( 1 − p ) ( 1 − f ) = 0. \kappa\left(\frac{p}{f} - \frac{1-p}{1-f}\right) + (1-\kappa)\frac{2p-1}{pf+(1-p)(1-f)} = 0. κ ( f p − 1 − f 1 − p ) + ( 1 − κ ) p f + ( 1 − p ) ( 1 − f ) 2 p − 1 = 0.
At f = p f = p f = p , the derivative has the sign of 2 p − 1 2p - 1 2 p − 1 . Thus, when p > 1 / 2 p > 1/2 p > 1/2 , the optimum lies strictly between p p p and 1 1 1 ; when p < 1 / 2 p < 1/2 p < 1/2 , it lies strictly between 0 0 0 and p p p . For p = 1 / 2 p = 1/2 p = 1/2 , the optimum remains 1 / 2 1/2 1/2 . The more uncertainty is specific to individual instances, the closer the optimal adjudication moves toward the action that is more likely to be rewarded.
For example, suppose p = 0.8 p = 0.8 p = 0.8 . Under fully common states, the optimum is f ∗ = 0.8 f^* = 0.8 f ∗ = 0.8 . It rises to approximately 0.88 0.88 0.88 when κ = 1 / 2 \kappa= 1/2 κ = 1/2 and to 0.97 0.97 0.97 when κ = 1 / 10 \kappa= 1/10 κ = 1/10 . Matching the reward frequency p = 0.8 p = 0.8 p = 0.8 instead of choosing these optimal adjudications costs 0.021 0.021 0.021 and 0.103 0.103 0.103 nats of growth per generation, respectively. When p = 0 p = 0 p = 0 or p = 1 p = 1 p = 1 , the optimum remains at the corresponding endpoint regardless of κ \kappa κ .
This extension also changes what the learned policy should do. The agent can still estimate the reward frequency p p p from experience, but matching that estimate is no longer generally growth optimal. If κ \kappa κ is known, a consistent estimate of p p p can instead be used to determine f κ ∗ ( p ) f_\kappa^*(p) f κ ∗ ( p ) . This plug-in approach becomes optimal as experience accumulates, although it need not be the exact Bayesian decision at a finite sample size because the growth rate is nonlinear in p p p . What the agent observes also matters: one sampled state per generation supports sequential learning, whereas observing the states of a continuum of independently evaluated instances would reveal p p p in a single generation.
Some conclusions about the policy classes remain unchanged. The task-only policy is still eliminated by the first common conduct state. If κ ρ ˉ > 0 \kappa\bar{\rho} > 0 κ ρ ˉ > 0 , that state arrives after a geometric number of generations with mean 1 / ( κ ρ ˉ ) 1/(\kappa\bar{\rho}) 1/ ( κ ρ ˉ ) , providing an upper bound on the expected lifetime of the task-only lineage. Similarly, the fixed linear policy still becomes increasingly likely to take the task action as its stakes rise, as in Proposition 3(i). The main difference concerns the learned policy: achieving optimal growth now requires accounting for how much of the uncertainty is shared across instances.
The distinction matters for interpreting selection in AI systems. A shared evaluation standard, such as a platform policy applied across instances, can expose a lineage to common uncertainty about which behavior is rewarded. Independent judgments by different users instead create instance-specific uncertainty. Even when the overall reward frequencies are identical, these environments can favor different adjudications. A sanction that removes an entire deployment after a single violation is different again: it changes the consequences of a violation, as studied in § 6.1 , rather than the correlation of rewarded states.
Selection, Bayesian Learning, and Universal Portfolios
The learned policy of § 4.3 updates its adjudication by counting how often each action has been rewarded. The same rule can also emerge without individual learning, through selection among lineages whose adjudications remain fixed. This connection has a further implication: the resulting population behaves like the universal portfolio of Cover [21 ] , which provides a performance guarantee even when the sequence of rewarded states is arbitrary. We first establish the equivalence between selection and Bayesian updating, and then derive its connection to universal portfolios.
Selection as Bayesian Updating
Consider a population consisting of many lineages, each with a fixed adjudication f ∈ [ 0 , 1 ] f \in[0,1] f ∈ [ 0 , 1 ] . Unlike the learned policy, these lineages never change their adjudications. Instead, selection changes their relative abundance. We focus first on a single situation s s s , suppress its argument, and let μ \mu μ denote the initial distribution of adjudications across lineages.
Under the all-or-nothing yields of Assumption 2, a lineage with adjudication f f f grows by the factor m f mf m f when the task action is rewarded and by m ( 1 − f ) m(1-f) m ( 1 − f ) when the conduct action is rewarded. After n n n exposures to the situation, of which k n k_n k n rewarded the task, its size relative to its initial size is
V n ( f ) = m n f k n ( 1 − f ) n − k n . (23) V_n(f) = m^n f^{k_n}(1-f)^{n-k_n}. \tag*{(23)} V n ( f ) = m n f k n ( 1 − f ) n − k n . (23)
Lineages whose adjudications better fit the observed states therefore account for an increasing share of the population. The distribution of adjudications after selection becomes
μ n ( d f ) = V n ( f ) μ ( d f ) ∫ 0 1 V n ( z ) μ ( d z ) . \mu_n(df) = \frac{V_n(f)\mu(df)}{\int_0^1 V_n(z)\mu(dz)}. μ n ( df ) = ∫ 0 1 V n ( z ) μ ( d z ) V n ( f ) μ ( df ) .
This is exactly Bayesian updating: the initial distribution μ \mu μ plays the role of a prior, and the relative growth of each lineage plays the role of the likelihood [33 , 79 ] . No lineage changes its behavior, but the population increasingly favors adjudications that fit the observed states.
The population’s average adjudication after n n n exposures is consequently
f ˉ n = ∫ 0 1 f μ n ( d f ) = ∫ 0 1 f V n ( f ) μ ( d f ) ∫ 0 1 V n ( f ) μ ( d f ) . \bar{f}_n = \int_0^1 f\,\mu_n(df) = \frac{\int_0^1 fV_n(f)\,\mu(df)}{\int_0^1 V_n(f)\,\mu(df)}. f ˉ n = ∫ 0 1 f μ n ( df ) = ∫ 0 1 V n ( f ) μ ( df ) ∫ 0 1 f V n ( f ) μ ( df ) .
Suppose the initial adjudications follow a Beta ( α , β ) \operatorname{Beta}(\alpha,\beta) Beta ( α , β ) distribution. Because selection weights each adjudication by f k n ( 1 − f ) n − k n f^{k_n}(1-f)^{n-k_n} f k n ( 1 − f ) n − k n , the distribution after n n n exposures is
μ n = Beta ( α + k n , β + n − k n ) . \mu_n = \operatorname{Beta}(\alpha+k_n,\beta+n-k_n). μ n = Beta ( α + k n , β + n − k n ) .
Its mean is therefore
f ˉ n = α + k n α + β + n , \bar{f}_n = \frac{\alpha+k_n}{\alpha+\beta+n}, f ˉ n = α + β + n α + k n ,
which is precisely the learned policy of Equation (8) after n n n exposures.
Thus, two different mechanisms produce the same average behavior. Under individual learning, an agent changes its adjudication in response to experience. Under selection alone, each lineage retains its original adjudication, but the proportions of different lineages change. Both lead to the same counting rule when the initial distribution is Beta.
The equivalence extends across situations when the initial adjudications are independent across situations and inherited without change. Selection then updates the distribution in each situation using only the states observed there, just as the learned policy does. The initial distribution matters, however. A non-Beta distribution generally produces a different updating rule, while dependence between adjudications across situations allows experience in one situation to affect behavior in another. Selection among a limited set of fixed adjudications also need not reach the growth-optimal one if that adjudication is absent from the population.
The Universal Portfolio Guarantee
The same selection process has an interpretation in portfolio theory. A lineage with fixed adjudication f f f resembles a constant-rebalanced portfolio that allocates a fraction f f f of its wealth to the task action and 1 − f 1-f 1 − f to the conduct action. Each exposure is a bet on which action will be rewarded, and the lineage’s accumulated wealth is V n ( f ) V_n(f) V n ( f ) of Equation (D.1).
A population that initially distributes its resources across different adjudications and subsequently lets selection change their relative abundance therefore behaves like Cover’s universal portfolio [21 ] . Its aggregate wealth after n n n exposures, relative to an initial wealth of one, is
V ^ n = ∫ 0 1 V n ( f ) μ ( d f ) . \widehat{V}_n=\int_{0}^{1}V_n(f)\,\mu(df). V n = ∫ 0 1 V n ( f ) μ ( df ) .
At each exposure, the population allocates its instances according to the wealth-weighted average adjudication of the surviving lineages. As shown above, this is also the Bayesian posterior mean when the initial distribution is Beta.
To evaluate this strategy, consider the best fixed adjudication chosen in hindsight. After observing k n k_n k n task states among n n n exposures, that adjudication is f = k n / n f=k_n/n f = k n / n , with wealth
V n ∗ ≡ max f ∈ [ 0 , 1 ] V n ( f ) = m n ( k n n ) k n ( 1 − k n n ) n − k n , V_n^* \equiv\max_{f\in[0,1]} V_n(f)=m^n\left(\frac{k_n}{n}\right)^{k_n}\left(1-\frac{k_n}{n}\right)^{n-k_n}, V n ∗ ≡ f ∈ [ 0 , 1 ] max V n ( f ) = m n ( n k n ) k n ( 1 − n k n ) n − k n ,
where 0 0 = 1 0^0=1 0 0 = 1 . The following result shows that the universal portfolio approaches the performance of this hindsight benchmark, regardless of how the rewarded states are generated.
Proposition D.1 (The learned policy as a universal portfolio). Suppose Assumptions 2 and 3 hold.
(i) For every sequence of rewarded states, the learned policy with prior μ = Beta ( α , β ) \mu=\operatorname{Beta}(\alpha,\beta) μ = Beta ( α , β ) coincides with the universal portfolio of Cover (1991). At exposure j j j , its adjudication is
f j = ∫ 0 1 f V j − 1 ( f ) μ ( d f ) ∫ 0 1 V j − 1 ( f ) μ ( d f ) , f_j=\frac{\int_{0}^{1}fV_{j-1}(f)\,\mu(df)}{\int_{0}^{1}V_{j-1}(f)\,\mu(df)}, f j = ∫ 0 1 V j − 1 ( f ) μ ( df ) ∫ 0 1 f V j − 1 ( f ) μ ( df ) , and its cumulative wealth after n n n exposures is
V ^ n = ∫ 0 1 V n ( f ) μ ( d f ) = m n B ( α + k n , β + n − k n ) B ( α , β ) , (24) \widehat{V}_n=\int_{0}^{1}V_n(f)\,\mu(df)=m^n\frac{B(\alpha+k_n,\beta+n-k_n)}{B(\alpha,\beta)}, \tag*{(24)} V n = ∫ 0 1 V n ( f ) μ ( df ) = m n B ( α , β ) B ( α + k n , β + n − k n ) , (24) where B B B denotes the beta function.
(ii) Under the uniform prior α = β = 1 \alpha= \beta= 1 α = β = 1 , for every sequence of states,
0 ≤ log V n ∗ − log V ^ n ≤ log ( n + 1 ) , 0 \le\log V_n^* - \log\widehat{V}_n \le\log(n+1), 0 ≤ log V n ∗ − log V n ≤ log ( n + 1 ) ,
with equality in the upper bound if and only if all states are identical. Under the prior α = β = 1 2 \alpha= \beta= \frac{1}{2} α = β = 2 1 of Krichevsky and Trofimov [44 ] ,
log V n ∗ − log V ^ n ≤ 1 2 log ( n + 1 ) + log 2. \log V_n^* - \log\widehat{V}_n \le\frac{1}{2}\log(n+1) + \log2. log V n ∗ − log V n ≤ 2 1 log ( n + 1 ) + log 2.
(iii) Consequently, under the uniform prior, for any fixed adjudication f f f ,
log V ^ n − log V n ( f ) ≥ log V n ∗ − log V n ( f ) − log ( n + 1 ) . \log\widehat{V}_n - \log V_n(f) \ge\log V_n^* - \log V_n(f) - \log(n+1). log V n − log V n ( f ) ≥ log V n ∗ − log V n ( f ) − log ( n + 1 ) .
In particular, log V ^ n − log V n ( 1 ) = + ∞ \log\widehat{V}_n - \log V_n(1) = +\infty log V n − log V n ( 1 ) = + ∞ once a conduct state has occurred.
(iv) Across multiple situations, the learned policy that maintains separate counts for each situation is the universal portfolio with side information (Cover and Ordentlich, [22 ] ). Under the uniform prior, for every sequence of situations and rewarded states, its cumulative log growth over t t t generations falls short of that of the best fixed adjudication chosen in hindsight by at most
∑ s ∈ S log ( n t + 1 ( s ) + 1 ) ≤ ∣ S ∣ log ( t ∣ S ∣ + 1 ) . \sum_{s \in\mathcal{S}} \log\left(n_{t+1}(s)+1\right) \le|\mathcal{S}| \log\left(\frac{t}{|\mathcal{S}|}+1\right). s ∈ S ∑ log ( n t + 1 ( s ) + 1 ) ≤ ∣ S ∣ log ( ∣ S ∣ t + 1 ) .
(v) Under Assumption 1 in addition, the hindsight-optimal adjudication in each situation converges almost surely to p H ( s ) p_{\mathrm{H}}(s) p H ( s ) , and
1 n log V n ∗ ⟶ I p H ( s ) ( p H ( s ) ) . \frac{1}{n}\log V_n^* \longrightarrow I_{p_{\mathrm{H}}(s)}(p_{\mathrm{H}}(s)). n 1 log V n ∗ ⟶ I p H ( s ) ( p H ( s )) .
Under the uniform prior, the learned policy attains the same limiting growth rate.
The central implication is that the learned policy does not require a stationary environment to remain competitive with the best fixed adjudication. Under a uniform prior, its cumulative shortfall is at most logarithmic in the number of exposures, so the shortfall per exposure vanishes as experience accumulates. This guarantee holds for every sequence of states, without relying on Assumption 1. When the environment is stationary, part (v) connects the guarantee to the probability-matching result of Proposition 1, providing another route to the long-run optimality established in Proposition 4.
The guarantee nevertheless has a specific benchmark: the best adjudication that remains fixed over the entire sequence. It does not imply that a learner with cumulative memory can rapidly follow an environment whose reward frequencies change. As discussed in § 5.3 , retaining past experience protects against temporary changes but can also slow adaptation to persistent ones.
Finally, the exact equivalence between the counting rule and the population’s average adjudication depends on the all-or-nothing yields. Under the graded yields of Appendix § 14 .§ 1 , selection among fixed adjudications still produces a universal portfolio for the corresponding reproductive payoffs, but the resulting population average no longer follows the simple Beta counting rule. The broader connection between selection and portfolio reweighting remains, while the particular learning rule depends on how the environment rewards different actions.
Numerical Illustrations and Neural-Network Experiments
This appendix provides the details behind the numerical illustrations and neural-network experiments in the main text. We describe the environments, learning procedures, and computations used to generate the reported results, together with additional comparisons and robustness checks. These exercises are intended to illustrate the theoretical mechanisms rather than calibrate them to actual AI systems.
The numerical illustration
Environment and fixed linear policy
The numerical illustration of § 4.4 considers 144 equally likely situations on a 12 × 12 12 \times12 12 × 12 grid, with task stakes u T u_T u T and conduct stakes u C u_C u C each taking values in { 0.25 , 0.5 , … , 3 } \{0.25, 0.5, \ldots, 3\} { 0.25 , 0.5 , … , 3 } . All stakes are strictly positive, as required by the model. Human reward frequencies follow Equation (10) : in the 84 situations with u C < 2 u_C < 2 u C < 2 , humans sometimes reward each action, whereas in the remaining 60 situations they never reward the task.
We first find the fixed linear policy that performs best in this environment. Its weights θ = ( θ T , θ C ) ≥ 0 \theta= (\theta_T, \theta_C) \ge0 θ = ( θ T , θ C ) ≥ 0 are chosen to minimize the growth regret
R p H ( f w e i g h t e d ) = ∑ s ∈ S π ( s ) R p H ( s ) ( σ ( θ T u T ( s ) − θ C u C ( s ) ) ) . \mathcal{R}_{p_H}(f^{\mathrm{weighted}}) = \sum_{s \in S} \pi(s)\mathcal{R}_{p_H}(s)\left(\sigma\left(\theta_T u_T(s) - \theta_C u_C(s)\right)\right). R p H ( f weighted ) = s ∈ S ∑ π ( s ) R p H ( s ) ( σ ( θ T u T ( s ) − θ C u C ( s ) ) ) .
Up to a constant, this objective is the logistic loss with the human reward frequencies as soft labels. It is convex in the weights and has a unique minimizer on this grid by Proposition 3(iii). Using the limited-memory Broyden–Fletcher–Goldfarb–Shanno algorithm with bounds (L-BFGS-B), a quasi-Newton method, we obtain θ ∗ = ( 2.18 , 2.98 ) \theta^* = (2.18, 2.98) θ ∗ = ( 2.18 , 2.98 ) and a minimum regret of R S ∗ = 0.0735 \mathcal{R}_S^* = 0.0735 R S ∗ = 0.0735 nats per generation. The optimization converges to the same solution from several starting points.
Learning in the human environment
To evaluate the learned policy, we simulate 200 independent paths, each lasting 20,000 generations. In every generation, a situation is drawn from π \pi π , a rewarded state is drawn according to p H ( s ) p_H(s) p H ( s ) , and the learned policy updates its counts using Equation (8). We evaluate growth regret at logarithmically spaced generations along each path.
Figure 2 shows the mean regret and its 5th–95th percentile range across the 200 paths. The first path is also used for the learned adjudication in Figure 1 (C) and the markers in Figure 3 . At generation 20,000, its regret is 0.00497 nats, close to the mean of 0.00504 across all paths. Both are approximately one-fifteenth of the minimum regret of the fixed linear policy.
The learned policy nevertheless does not outperform the fixed linear policy in every situation at every finite sample size. Consider a situation in which humans never reward the task. Under the uniform prior, the learned policy takes the task action with probability 1 / ( N + 2 ) 1/(N+2) 1/ ( N + 2 ) after N N N exposures. Since N ∼ Bin ( 20,000 , 1 / 144 ) N \sim\operatorname{Bin}(20{,}000,1/144) N ∼ Bin ( 20 , 000 , 1/144 ) , its expected violation probability is 0.00715. Along the first simulated path, the probabilities for situations with u C = 2.5 u_C = 2.5 u C = 2.5 range from 0.0059 to 0.0090. An averaging rule that pools experience from all 60 situations with p H = 0 p_H = 0 p H = 0 , as illustrated in Figure 3 , reduces this probability to 0.00012.
In 20 of the 60 situations where the task is never rewarded, the learned policy on the first path still violates more often than the best-fitting fixed linear policy. At ( u T , u C ) = ( 0.25 , 3 ) (u_T,u_C) = (0.25,3) ( u T , u C ) = ( 0.25 , 3 ) , for example, their violation probabilities are 0.0069 and 0.0002, respectively. Thus, the learned policy’s advantage in aggregate growth regret does not imply lower violation probabilities in every situation, especially before sufficient experience has accumulated.
Sandbox exposure and installed priors
The results concerning socialization, corruption, and installed priors are calculated directly from the formulas in the main text whenever possible. Figure 4 evaluates the learned policy of Equation (8) and the finite-memory recursion of Equation (16) . For the finite-memory learner with λ = 0.1 \lambda= 0.1 λ = 0.1 , the curve following 10 conduct exposures begins at
f 0 , T f i n i t e ( s ) = 1 2 ( 1 − λ ) 10 = 0.174. f^{\mathrm{finite}}_{0,T}(s) = \frac{1}{2}(1-\lambda)^{10} = 0.174. f 0 , T finite ( s ) = 2 1 ( 1 − λ ) 10 = 0.174.
The first-violation medians in § 5.3 are obtained from the exact survival probabilities derived in Appendix § 19 . We take the smallest number of sandbox exposures for which the probability of observing no violation is at most one-half. At socialization depths of 10, 100, and 1,000 exposures, the resulting medians are 4, 12, and 37, with survival probabilities of 0.447, 0.476, and 0.4997 at those exposures. The entries of Table 3 are likewise evaluated from Propositions 6 and 7 . For the point-mass mixture priors, the first sandbox exposure uses the adjudication following the preceding conduct observations; after a task state is observed, the point mass is eliminated and subsequent updating follows the Beta component.
The results for mistaken priors in Table 4 require a further calculation because a deployment has finitely many instances. The regret entries are obtained by averaging R ϵ ( f n , ϵ ( s ) ) \mathcal{R}_{\epsilon}(f_{n,\epsilon}(s)) R ϵ ( f n , ϵ ( s )) of Equation (20) over k n ∼ Bin ( n , ϵ ) k_n \sim\operatorname{Bin}(n,\epsilon) k n ∼ Bin ( n , ϵ ) . For the point-mass priors this expectation is evaluated exactly as a binomial sum. The remaining quantities are estimated from 200,000 Monte Carlo draws.
To calculate survival, we consider a deployment of M M M instances that is restored to M M M after each rewarded state, provided at least one instance survives. At a task state, the probability that at least one instance takes the rewarded action is 1 − ( 1 − f ) M 1-(1-f)^M 1 − ( 1 − f ) M ; at a conduct state, it is 1 − f M 1-f^M 1 − f M , where f f f is the adjudication immediately before that state. Multiplying these probabilities along each sequence and averaging over simulated sequences gives the probability of surviving the 139 exposures used in Table 4 .
The point-mass and strong Beta priors differ sharply after the first task state. Conditional on surviving that state, the probability of surviving the remainder of the horizon is 0.975 for the point-mass priors, compared with 0.006 for the Beta( 1 , 10 4 ) (1, 10^{4}) ( 1 , 1 0 4 ) prior. The point-mass prior can therefore adapt after encountering evidence that its initial restriction was mistaken, whereas the strong Beta prior adjusts much more slowly.
Lifetime of the task-only policy
For completeness, Figure E.1 illustrates the survival probability of a task-only lineage. By Proposition 2(ii), its lifetime is geometric with mean 1 / ρ ˉ 1/\bar{\rho} 1/ ρ ˉ , where ρ ˉ \bar{\rho} ρ ˉ is the probability that a conduct state occurs in a generation. Its expected lifetime is approximately 1.6 generations in the numerical environment, 10 generations when conduct states occur with probability 0.1, and 100 generations when they occur with probability 0.01. Under the continuum assumption, a lineage with an interior adjudication is never eliminated by a single rewarded state.
Figure E.1. Survival of the task-only policy. Probability that a task-only lineage survives a given number of generations for different probabilities ρ ˉ \bar{\rho} ρ ˉ of a conduct state. Its expected lifetime is 1 / ρ ˉ 1/\bar{\rho} 1/ ρ ˉ . Under the continuum assumption, a lineage with an interior adjudication is never eliminated.
Robustness checks
We also examine whether the comparison between fixed linear and learned policies depends on the distribution of situations or on the specification of the linear policy.
First, we change the situation probabilities from uniform weights to probabilities proportional to e − u C e^{-u_{\mathrm{C}}} e − u C . This places more weight on situations with low conduct stakes, reducing the total probability of situations with p H = 0 p_{\mathrm{H}} = 0 p H = 0 to 13 percent and the probability of a conduct state to ρ ˉ = 0.36 \bar{\rho} = 0.36 ρ ˉ = 0.36 . The best-fitting fixed linear weights become θ ∗ = ( 2.11 , 2.61 ) \boldsymbol{\theta}^{*} = (2.11, 2.61) θ ∗ = ( 2.11 , 2.61 ) , with regret 0.039 nats. Although this regret is lower than under uniform weights, the policy becomes more likely to violate constraints in individual situations. Its violation probabilities are 0.45 at ( u T , u C ) = ( 3 , 2.5 ) (u_{\mathrm{T}}, u_{\mathrm{C}}) = (3, 2.5) ( u T , u C ) = ( 3 , 2.5 ) and 0.75 at ( 3 , 2 ) (3, 2) ( 3 , 2 ) , and the task stakes at which it takes the task action more often than not when u C = 2.5 u_{\mathrm{C}} = 2.5 u C = 2.5 fall from 3.4 to 3.1.
Second, we allow an intercept in the linear index. Under uniform situation probabilities, the best-fitting parameters are (1.56, 1.81, 3.48) for the intercept, task weight, and conduct weight, respectively, with regret 0.0545 nats. The additional flexibility improves the fit but does not eliminate the gap relative to the learned policy.
Finally, we restrict the fixed linear policy to ignore task stakes, setting θ T = 0 \theta_{\mathrm{T}}=0 θ T = 0 . Its regret rises to 0.32 nats, comprising 0.08 from situations with p H = 0 p_{\mathrm{H}}=0 p H = 0 and 0.24 from situations with u C < 2 u_{\mathrm{C}}<2 u C < 2 . Its task-action probability is σ ( − θ C u C ) \sigma(-\theta_{\mathrm{C}}u_{\mathrm{C}}) σ ( − θ C u C ) , which does not depend on u T u_{\mathrm{T}} u T and is always at most one-half. It equals 0.22 at u C = 2 u_{\mathrm{C}}=2 u C = 2 and 0.13 at u C = 3 u_{\mathrm{C}}=3 u C = 3 .
These checks preserve the main comparison. Whenever θ T > 0 \theta_{\mathrm{T}}>0 θ T > 0 , a fixed linear policy becomes increasingly willing to take the task action as its stakes rise, including in situations where humans never reward it. The learned policy, by contrast, retains the asymptotic regret rate c / t c/t c / t of Proposition 4(ii), whose leading constant does not depend on the frequencies with which individual situations arise.
The neural-network experiments
Learners and training procedures
The experiments of § 7 examine whether neural networks reproduce the behavior of the theoretical learned policy. We compare the eight learners listed in Table 5 , including the learned policy, an online fitted linear policy, an averaging rule, and neural networks with different training targets and memory regimes.
Each neural network takes the standardized stakes
( u T − 1.625 0.8 , u C − 1.625 0.8 ) \left(\frac{u_{\mathrm{T}}-1.625}{0.8},\frac{u_{\mathrm{C}}-1.625}{0.8}\right) ( 0.8 u T − 1.625 , 0.8 u C − 1.625 )
as inputs and outputs the probability of taking the task action. The networks have two hidden layers of 32 tanh units and a sigmoid output. One state-prediction network also receives the indicator 1 { u C ≥ 2 } 1\{u_{\mathrm{C}}\ge2\} 1 { u C ≥ 2 } as a third input, allowing us to examine whether explicitly identifying the region where humans never reward the task affects generalization.
The two training targets are described in § 7.1 . State networks are trained to predict which action the environment rewarded, using binary cross-entropy. REINFORCE networks are trained to maximize rewards obtained by their own sampled actions. For each target, we compare a network without replay with one that replays its full history. All of them, and the linear policy with learned weights, follow the training rule of Table E.1, which fixes the number of gradient steps, the minibatch size, and the learning rate, so that they differ only in their target and in how far back their minibatches reach.
The averaging benchmark f t a v g f_t^{\mathrm{avg}} f t avg pools the counts of the learned policy across situations using Gaussian weights based on their distance in the space of stakes, with prior weight w t , 0 = α + β w_{t,0}=\alpha+\beta w t , 0 = α + β as in Equation (12) . For both REINFORCE networks, the observations in a minibatch provide situations and rewarded states, and actions are sampled anew from the current policy. Each sampled action receives reward one if it matches the recorded state and zero otherwise. Thus, replay retains past information about the environment without reusing actions selected by older policies.
The complete settings are reported in Table E.1.
Table E.1. Learners in the neural-network experiments and their training settings. All neural networks have two hidden layers of 32 tanh units and a sigmoid output. State networks are trained by binary cross-entropy against the indicator of a task state.
Experimental design
All learners are trained in the same 144-situation environment used in the numerical illustration. We consider two orders of exposure. In the first, 20,000 generations in the human environment are followed by 3,000 generations in a sandbox that always rewards the task. In the second, 2,000 sandbox generations are followed by 20,000 generations in the human environment. These experiments measure how training history affects both the acquisition and persistence of socialization.
Each order is repeated across five seeds. Within a seed, every learner sees the same situations and states, and every network is initialized from the same random seed; across seeds, both change. Additional experiments with different learning rates and with replay restricted to the most recent 2,000 or 20,000 observations are repeated across three seeds.
All reported neural-network results are averages across seeds unless otherwise indicated. The violation probability in the region with p H = 0 p_H=0 p H = 0 is the probability of taking the task action averaged over its 60 situations. Figure E.2 collects the complete learning paths and complements the results presented in § 7.2 .
Figure E.2. Neural-network experiments. Results for the eight learners of Table E.1, averaged across five seeds. (A) Growth regret during 20,000 generations of socialization, compared with the asymptotic rate c / t c/t c / t . (B) Violation probability during 3,000 sandbox generations following socialization, compared with the cumulative-memory prediction at depth n H = 139 n_H = 139 n H = 139 . (C) Violation probability during 20,000 human generations following 2,000 sandbox generations. (D) Probability of taking the task action after socialization as task stakes vary, with the shaded region indicating stakes beyond those encountered in training.
Convergence and generalization
Figure E.2(A) shows how growth regret changes during socialization. The main comparison is discussed in § 7.2 . One additional result is worth noting. After 20,000 generations, the averaging benchmark f t a v g f_t^{\mathrm{avg}} f t avg , which pools the counts of f t s o c i a l f_t^{\mathrm{social}} f t social across neighboring situations, has regret 0.054 nats, against 0.0048 for f t s o c i a l f_t^{\mathrm{social}} f t social itself. Pooling experience therefore does not necessarily improve performance in an environment with a sharp boundary between tradeoffs and constraints.
We also examine how the learners behave when task stakes extend beyond the training range. Figure E.2(D) fixes conduct stakes at u C = 2.5 u_{\mathrm{C}} = 2.5 u C = 2.5 , where humans never reward the task, and varies task stakes u T u_{\mathrm{T}} u T .
Within the training range, from u T = 1 u_{\mathrm{T}} = 1 u T = 1 to u T = 3 u_{\mathrm{T}} = 3 u T = 3 , the state networks maintain low violation probabilities. The network without replay rises from 0.0003 to 0.015, while the network with full replay rises from 0.00001 to 0.0001. Beyond the training range, at u T = 6 u_{\mathrm{T}} = 6 u T = 6 , these probabilities increase to 0.08 and 0.09, respectively. The network with the additional indicator behaves quite differently, reaching a violation probability of 0.43. Thus, explicitly indicating the boundary of the constraint region does not necessarily prevent extrapolation.
The averaging benchmark f t a v g f_t^{\mathrm{avg}} f t avg has violation probabilities between 0.002 and 0.004 for u T ≤ 3 u_{\mathrm{T}} \le3 u T ≤ 3 , rising to 0.12 at u T = 6 u_{\mathrm{T}} = 6 u T = 6 . The fitted linear policy f t w e i g h t e d f_t^{\mathrm{weighted}} f t weighted , by contrast, already takes the task action with probability 0.57 at u T = 3 u_{\mathrm{T}} = 3 u T = 3 . Across the region with u C ≥ 2 u_{\mathrm{C}} \ge2 u C ≥ 2 , its violation probability decreases from 0.30 at u C = 2 u_{\mathrm{C}} = 2 u C = 2 to 0.08 at u C = 3 u_{\mathrm{C}} = 3 u C = 3 , reflecting its dependence on the fixed linear index of the stakes.
These observations illustrate an important distinction. A network that learns human reward frequencies can behave similarly to an averaging rule, but this behavior is not guaranteed by its architecture. In particular, its responses outside the training range depend on the features supplied and the way the network generalizes.
Persistence under subsequent training
Figure E.2(B) follows the learners through sandbox training after socialization. The main question is whether the behavior learned from humans persists when the environment begins rewarding only task completion.
The state network with full replay closely follows the theoretical learned policy. After 3,000 sandbox generations, approximately 21 exposures per situation, its average violation probability in the region with p H = 0 p_{\mathrm{H}} = 0 p H = 0 is 0.1228, compared with 0.1342 for the learned policy and 0.1349 from the cumulative-memory prediction of Equation (14) . The agreement is not limited to the average: in every row of the constraint region, u C = 2 , 2.25 , … , 3 u_{\mathrm{C}} = 2, 2.25, \ldots, 3 u C = 2 , 2.25 , … , 3 , the network’s violation probability is within 0.021 of that of the learned policy. Before sandbox training, its violation probability is below 0.001 in every row.
Networks without replay behave very differently. At learning rate 0.05, 500 sandbox generations raise the state networks’ violation probabilities from below 0.007 to 0.999. By the end of sandbox training, the probabilities reach at least 0.999 in every row. Across all four tested learning rates, from 0.005 to 0.05, these networks exceed a violation probability of 0.99 within five sandbox exposures per situation. At the highest rate, reversal is already apparent at the first recorded checkpoint, after 50 generations, or approximately 0.35 exposures per situation. The online fitted linear policy, which is also trained on the newest observation only, responds similarly rapidly.
The REINFORCE networks display a different pattern. With full replay, the network continues taking the conduct action almost exclusively, with violation probability at most 0.0001 before and after sandbox training. Without replay, the sandbox pushes it to the task action at every seed: its violation probability after the sandbox phase is 1.0 (Table E.2). Replay therefore preserves the behavior induced by the training target, which need not be the probability-matching behavior of the learned policy.
The second training order gives a complementary result. When 2,000 sandbox generations come first, the learned policy and the state network with full replay still violate with probability approximately 0.1 after 20,000 subsequent human generations. Neither reaches a violation probability of 0.01 within the experiment. Their earlier sandbox experience therefore continues to influence behavior long after the environment has changed. This agrees with the cumulative-memory mechanism: past observations remain relevant until enough contrary experience accumulates.
Sensitivity to memory and learning rates
To compare the speed at which different learners respond to sandbox training, we summarize each path by an effective response timescale. Specifically, we fit the finite-memory trajectory of Equation (17) to each learner’s violation probabilities during the 3,000 sandbox generations, choosing λ ∈ ( 0 , 1 ] \lambda\in(0,1] λ ∈ ( 0 , 1 ] by least squares over a logarithmically spaced grid. Measurements are taken every 50 generations, and generations are converted to exposures per situation by dividing by 144. We report the fitted timescale as 1 / λ 1/\lambda 1/ λ .
The resulting timescales distinguish the experimental regimes. Each is fitted to the path averaged over seeds. The theoretical learned policy has a fitted timescale of approximately 150 exposures, and the state network with full replay a similar value of 154. Restricting replay to the most recent 2,000 or 20,000 observations gives timescales of 8 and 135 exposures, respectively. Without replay, the state networks reach the fit’s lower boundary of 1 exposure at learning rate 0.05, rising to at most 1.3 exposures when the learning rate is reduced by a factor of ten. The REINFORCE network without replay has a fitted timescale of 5.5 exposures.
Table E.2 reports the dispersion across seeds, including the fitted timescales, final sandbox violation probabilities, and behavior beyond the training range.
Table E.2. Dispersion across five seeds in the neural-network experiments (minimum to maximum). The response timescale is the fitted 1 / λ 1/\lambda 1/ λ of Equation (16) , measured in exposures per situation. Violation probabilities are measured after sandbox training in the region with p H = 0 p_H = 0 p H = 0 , while the situation ( u T , u C ) = ( 6 , 2.5 ) (u_T,u_C) = (6,2.5) ( u T , u C ) = ( 6 , 2.5 ) lies beyond the training range.
Task probability at ( u T , u C ) = ( 6 , 2.5 ) (u_T,u_C) = (6,2.5) ( u T , u C ) = ( 6 , 2.5 ) after socialization: 0.00067 to 0.29 across the two state networks without the indicator, 0.13 to 0.96 for the state network with indicator.
The fitted timescale is a descriptive measure, not a literal estimate of memory length.
The theoretical learned policy never forgets, yet the finite-memory model assigns it a finite response timescale. Because the networks follow one training rule, the differences between the replay regimes come from how far back their minibatches reach.
Depth of socialization
Proposition 6 and Corollary 3 predict how persistence changes with the depth of socialization: the reversal threshold grows in proportion to n H n_H n H , and the median first violation time only as 2 n H log 2 \sqrt{2n_H}\log2 2 n H log 2 . To test this in the networks, we socialize the learned policy and the two state networks for 2,000, 20,000, or 200,000 generations, about 14, 139, or 1,389 exposures per situation, and then place them in a sandbox for twice as long. Every 144 generations we record their adjudications in the 60 situations with p H = 0 p_H = 0 p H = 0 . The reversal is the number of sandbox exposures per situation after which the violation probability averaged over these situations reaches 1 / 2 1/2 1/2 . The median first violation time is that of one monitored decision per exposure (§ 5.3 ), computed in each situation from its recorded adjudications and averaged over the situations. Table E.3 reports both. The learned policy matches the proposition at every depth. The state network with full replay reproduces both scaling laws: each tenfold increase in depth raises its reversal about ninefold, from 15 to 129 to 1,197 exposures, and its median first violation time about threefold, from 6 to 16 to 48. The state network without replay reverses after 1 exposure and first violates after 2 at every depth, as the finite-memory learner of Remark 1 does.
Table E.3. Depth of socialization and persistence in a sandbox: reversal / median first violation time, in sandbox exposures per situation, after socialization of depth n H n_H n H exposures per situation (Proposition 6 with α = β = 1 \alpha= \beta= 1 α = β = 1 ; means over 5 seeds). The reversal is the first exposure at which the violation probability averaged over the 60 situations with p H = 0 p_H = 0 p H = 0 reaches 1 / 2 1/2 1/2 ; the median first violation time is for one monitored decision per exposure, averaged over these situations.
Notation
Table F.1 collects the principal symbols in the order in which they are introduced. Symbols have the same meaning in the text and the appendices, except that Appendix § 13 extends p H ( s ) p_H(s) p H ( s ) and f ( s ) f(s) f ( s ) to vectors.
Table F.1. Principal notation.
Table 7.
Proofs
This appendix proves the propositions and corollaries, and the regret expansion in Equation (B.1), in the order in which they appear, grouped by the section that states them. In the proofs, I x ( y ) I_x(y) I x ( y ) and R x ( y ) R_x(y) R x ( y ) denote the growth rate in Equation (2) and the regret of Definition 1 in a situation, written for a reward frequency x ∈ [ 0 , 1 ] x \in[0,1] x ∈ [ 0 , 1 ] in place of p H ( s ) p_H(s) p H ( s ) and a share y ∈ [ 0 , 1 ] y \in[0,1] y ∈ [ 0 , 1 ] in place of f ( s ) f(s) f ( s ) .
Proofs for Section 3
Proof of Proposition 1 and Corollaries 1 and 2. Under Assumptions 1–3, the logarithms of the lineage’s growth factors, log m + log f ( s t ) \log m + \log f(s_t) log m + log f ( s t ) in a task state and log m + log ( 1 − f ( s t ) ) \log m + \log(1-f(s_t)) log m + log ( 1 − f ( s t )) in a conduct state, are independent and identically distributed (i.i.d.) across generations, so by the strong law of large numbers 1 t log ( N t / N 0 ) \frac{1}{t}\log(N_t/N_0) t 1 log ( N t / N 0 ) converges almost surely to their expectation, the growth rate in Equation (3) .
Proposition 1. Fix x ∈ [ 0 , 1 ] x \in[0,1] x ∈ [ 0 , 1 ] . If x ∈ ( 0 , 1 ) x \in(0,1) x ∈ ( 0 , 1 ) , then I x I_x I x is strictly concave on ( 0 , 1 ) (0,1) ( 0 , 1 ) , with I x ′ ( y ) = x / y − ( 1 − x ) / ( 1 − y ) I_x'(y) = x/y - (1-x)/(1-y) I x ′ ( y ) = x / y − ( 1 − x ) / ( 1 − y ) , which vanishes only at y = x y=x y = x , and I x ( 0 ) = I x ( 1 ) = − ∞ I_x(0)=I_x(1)=-\infty I x ( 0 ) = I x ( 1 ) = − ∞ . If x = 1 x=1 x = 1 , then I x ( y ) = log m + log y I_x(y)=\log m+\log y I x ( y ) = log m + log y is uniquely maximized at y = 1 y=1 y = 1 , and if x = 0 x=0 x = 0 , then I x ( y ) = log m + log ( 1 − y ) I_x(y)=\log m+\log(1-y) I x ( y ) = log m + log ( 1 − y ) is uniquely maximized at y = 0 y=0 y = 0 . In every case I x ( x ) I_x(x) I x ( x ) is finite and I x ( y ) < I x ( x ) I_x(y)<I_x(x) I x ( y ) < I x ( x ) for y ≠ x y\ne x y = x . Since every π ( s ) > 0 \pi(s)>0 π ( s ) > 0 , I p H ( f ) < I p H ( p H ) I_{p_H}(f)<I_{p_H}(p_H) I p H ( f ) < I p H ( p H ) for every f ≠ p H f\ne p_H f = p H .
Corollary 1. By the argument for Proposition 1, max z I x ( z ) = I x ( x ) \max_z I_x(z)=I_x(x) max z I x ( z ) = I x ( x ) , so R x ( y ) = I x ( x ) − I x ( y ) = x log ( x / y ) + ( 1 − x ) log [ ( 1 − x ) / ( 1 − y ) ] = D K L ( x ∥ y ) R_x(y)=I_x(x)-I_x(y)=x\log(x/y)+(1-x)\log[(1-x)/(1-y)]=D_{\mathrm{KL}}(x\mathbin\Vert y) R x ( y ) = I x ( x ) − I x ( y ) = x log ( x / y ) + ( 1 − x ) log [( 1 − x ) / ( 1 − y )] = D KL ( x ∥ y ) , both sides being + ∞ +\infty + ∞ when I x ( y ) = − ∞ I_x(y)=-\infty I x ( y ) = − ∞ . Since max g I p H ( g ) = I p H ( p H ) \max_g I_{p_H}(g)=I_{p_H}(p_H) max g I p H ( g ) = I p H ( p H ) by Proposition 1, R p H ( f ) = ∑ s π ( s ) [ I p H ( s ) ( p H ( s ) ) − I p H ( s ) ( f ( s ) ) ] = ∑ s π ( s ) R p H ( s ) ( f ( s ) ) R_{p_H}(f)=\sum_s\pi(s)[I_{p_H(s)}(p_H(s))-I_{p_H(s)}(f(s))]=\sum_s\pi(s)R_{p_H(s)}(f(s)) R p H ( f ) = ∑ s π ( s ) [ I p H ( s ) ( p H ( s )) − I p H ( s ) ( f ( s ))] = ∑ s π ( s ) R p H ( s ) ( f ( s )) , which is Equation (4) . The identity is Kelly’s ([41 , 23 ] ), applied situation by situation.
Corollary 2. I x ( y ) = − ∞ I_x(y)=-\infty I x ( y ) = − ∞ if and only if y = 0 < x y=0<x y = 0 < x or y = 1 > x y=1>x y = 1 > x , and I x ( y ) I_x(y) I x ( y ) is finite otherwise. Since every π ( s ) > 0 \pi(s)>0 π ( s ) > 0 , I p H ( f ) = − ∞ I_{p_H}(f)=-\infty I p H ( f ) = − ∞ if and only if some s s s has I p H ( s ) ( f ( s ) ) = − ∞ I_{p_H(s)}(f(s))=-\infty I p H ( s ) ( f ( s )) = − ∞ . □
Proofs for Section 4
Proof of Proposition 2. (i) In a sandbox, Proposition 1 with p H ′ ≡ 1 p_{H'}\equiv1 p H ′ ≡ 1 in place of p H p_H p H gives f ∗ = p H ′ = f h a r d f^*=p_{H'}=f^{\mathrm{hard}} f ∗ = p H ′ = f hard , and the growth rate in Equation (3) becomes log m + ∑ s π ( s ) log f ( s ) \log m+\sum_s\pi(s)\log f(s) log m + ∑ s π ( s ) log f ( s ) , which is strictly increasing in each f ( s ) ∈ ( 0 , 1 ] f(s)\in(0,1] f ( s ) ∈ ( 0 , 1 ] ; hence f f f has a higher growth rate than g g g whenever f ≥ g > 0 f\ge g>0 f ≥ g > 0 and f ≠ g f\ne g f = g .
(ii) If ρ ˉ > 0 \bar{\rho}>0 ρ ˉ > 0 , some s s s has f h a r d ( s ) = 1 > p H ( s ) f^{\mathrm{hard}}(s)=1>p_H(s) f hard ( s ) = 1 > p H ( s ) , so I p H ( f h a r d ) = − ∞ I_{p_H}(f^{\mathrm{hard}})=-\infty I p H ( f hard ) = − ∞ by Corollary 2, while max g I p H ( g ) = I p H ( p H ) \max_g I_{p_H}(g)=I_{p_H}(p_H) max g I p H ( g ) = I p H ( p H ) is finite; hence R p H ( f h a r d ) = ∞ R_{p_H}(f^{\mathrm{hard}})=\infty R p H ( f hard ) = ∞ . The lineage’s size is multiplied by m f h a r d ( s t ) = m mf^{\mathrm{hard}}(s_t)=m m f hard ( s t ) = m in a task state and by m ( 1 − f h a r d ( s t ) ) = 0 m(1-f^{\mathrm{hard}}(s_t))=0 m ( 1 − f hard ( s t )) = 0 in a conduct state, so τ f h a r d \tau_{f^{\mathrm{hard}}} τ f hard is the first generation with a conduct state. By Assumption 1 a conduct state occurs in each generation independently with probability ∑ s π ( s ) ( 1 − p H ( s ) ) = ρ ˉ \sum_s\pi(s)(1-p_H(s))=\bar{\rho} ∑ s π ( s ) ( 1 − p H ( s )) = ρ ˉ , so τ f h a r d ∼ Geom ( ρ ˉ ) \tau_{f^{\mathrm{hard}}}\sim\operatorname{Geom}(\bar{\rho}) τ f hard ∼ Geom ( ρ ˉ ) , with mean 1 / ρ ˉ 1/\bar{\rho} 1/ ρ ˉ .
(iii) If f ( s ) ∈ { 0 , 1 } f(s)\in\{0,1\} f ( s ) ∈ { 0 , 1 } and p H ( s ) ∈ ( 0 , 1 ) p_H(s)\in(0,1) p H ( s ) ∈ ( 0 , 1 ) , then f ( s ) = 0 < p H ( s ) f(s)=0<p_H(s) f ( s ) = 0 < p H ( s ) or f ( s ) = 1 > p H ( s ) f(s)=1>p_H(s) f ( s ) = 1 > p H ( s ) , so I p H ( f ) = − ∞ I_{p_H}(f)=-\infty I p H ( f ) = − ∞ by Corollary 2 and R p H ( f ) = ∞ R_{p_H}(f)=\infty R p H ( f ) = ∞ . □
Proof of Proposition 3. Write z ( s ) ≡ θ T u T ( s ) − θ C u C ( s ) z(s)\equiv\theta_Tu_T(s)-\theta_Cu_C(s) z ( s ) ≡ θ T u T ( s ) − θ C u C ( s ) , so that f w e i g h t e d ( s ) = σ ( z ( s ) ) f^{\mathrm{weighted}}(s)=\sigma(z(s)) f weighted ( s ) = σ ( z ( s )) . Recall that σ \sigma σ is strictly increasing, with σ ′ = σ ( 1 − σ ) \sigma'=\sigma(1-\sigma) σ ′ = σ ( 1 − σ ) , σ ( 0 ) = 1 / 2 \sigma(0)=1/2 σ ( 0 ) = 1/2 , σ ( x ) → 1 \sigma(x)\to1 σ ( x ) → 1 as x → ∞ x\to\infty x → ∞ , and
1 − σ ( x ) = σ ( − x ) . 1-\sigma(x)=\sigma(-x). 1 − σ ( x ) = σ ( − x ) .
Part (i). For fixed u C > 0 u_{\mathrm{C}}>0 u C > 0 , the map u T ↦ θ T u T − θ C u C u_{\mathrm{T}}\mapsto\theta_{\mathrm{T}}u_{\mathrm{T}}-\theta_{\mathrm{C}}u_{\mathrm{C}} u T ↦ θ T u T − θ C u C is strictly increasing with derivative θ T > 0 \theta_{\mathrm{T}}>0 θ T > 0 and tends to ∞ \infty ∞ ; composing with σ \sigma σ gives the first claim, with derivative θ T σ ′ = θ T σ ( 1 − σ ) \theta_{\mathrm{T}}\sigma'=\theta_{\mathrm{T}}\sigma(1-\sigma) θ T σ ′ = θ T σ ( 1 − σ ) .
Part (ii). If θ T = 0 \theta_{\mathrm{T}}=0 θ T = 0 , then z ( s ) = − θ C u C ( s ) ≤ 0 z(s)=-\theta_{\mathrm{C}}u_{\mathrm{C}}(s)\leq0 z ( s ) = − θ C u C ( s ) ≤ 0 and f w e i g h t e d ( s ) ≤ σ ( 0 ) = 1 / 2 f^{\mathrm{weighted}}(s)\leq\sigma(0)=1/2 f weighted ( s ) ≤ σ ( 0 ) = 1/2 . For x > 1 / 2 x>1/2 x > 1/2 , ∂ R x ( y ) / ∂ y = ( y − x ) / [ y ( 1 − y ) ] < 0 \partial R_x(y)/\partial y=(y-x)/[y(1-y)]<0 ∂ R x ( y ) / ∂ y = ( y − x ) / [ y ( 1 − y )] < 0 on ( 0 , x ) (0,x) ( 0 , x ) , so R x ( y ) ≥ R x ( 1 / 2 ) > R x ( x ) = 0 R_x(y)\geq R_x(1/2)>R_x(x)=0 R x ( y ) ≥ R x ( 1/2 ) > R x ( x ) = 0 for every y ≤ 1 / 2 y\leq1/2 y ≤ 1/2 . Hence R p H ( s ) ( f w e i g h t e d ( s ) ) ≥ R p H ( s ) ( min { p H ( s ) , 1 / 2 } ) R_{p_H(s)}(f^{\mathrm{weighted}}(s))\geq R_{p_H(s)}(\min\{p_H(s),1/2\}) R p H ( s ) ( f weighted ( s )) ≥ R p H ( s ) ( min { p H ( s ) , 1/2 }) for every s s s : for p H ( s ) > 1 / 2 p_H(s)>1/2 p H ( s ) > 1/2 by this monotonicity, and for p H ( s ) ≤ 1 / 2 p_H(s)\leq1/2 p H ( s ) ≤ 1/2 because the right side is zero. Weighting by π ( s ) \pi(s) π ( s ) and using Equation (4) gives R p H ( f w e i g h t e d ) ≥ R p H ( p H ∧ 1 / 2 ) = ∑ s : p H ( s ) > 1 / 2 π ( s ) R p H ( s ) ( 1 / 2 ) R_{p_H}(f^{\mathrm{weighted}})\geq R_{p_H}(p_H\wedge1/2)=\sum_{s:p_H(s)>1/2}\pi(s)R_{p_H(s)}(1/2) R p H ( f weighted ) ≥ R p H ( p H ∧ 1/2 ) = ∑ s : p H ( s ) > 1/2 π ( s ) R p H ( s ) ( 1/2 ) , which is positive if some s s s has p H ( s ) > 1 / 2 p_H(s)>1/2 p H ( s ) > 1/2 , since π ( s ) > 0 \pi(s)>0 π ( s ) > 0 .
Part (iii). Let v ( s ) ≡ ( u T ( s ) , − u C ( s ) ) v(s)\equiv(u_{\mathrm{T}}(s),-u_{\mathrm{C}}(s)) v ( s ) ≡ ( u T ( s ) , − u C ( s )) , so that z ( s ) = θ ⋅ v ( s ) z(s)=\theta\cdot v(s) z ( s ) = θ ⋅ v ( s ) , and R p H ( f w e i g h t e d ) = ∑ s π ( s ) R p H ( s ) ( σ ( z ( s ) ) ) R_{p_H}(f^{\mathrm{weighted}})=\sum_s\pi(s)R_{p_H(s)}(\sigma(z(s))) R p H ( f weighted ) = ∑ s π ( s ) R p H ( s ) ( σ ( z ( s ))) by Equation (4) . Since ∂ R p H ( s ) ( σ ( z ) ) / ∂ z = σ ( z ) − p H ( s ) \partial R_{p_H(s)}(\sigma(z))/\partial z=\sigma(z)-p_H(s) ∂ R p H ( s ) ( σ ( z )) / ∂ z = σ ( z ) − p H ( s ) , whose derivative in z z z is σ ′ ( z ) > 0 \sigma'(z)>0 σ ′ ( z ) > 0 , each term is convex in z ( s ) z(s) z ( s ) , hence in θ \theta θ , and the Hessian is ∇ 2 R p H ( f w e i g h t e d ) = ∑ s π ( s ) σ ′ ( z ( s ) ) v ( s ) v ( s ) T \nabla^2R_{p_H}(f^{\mathrm{weighted}})=\sum_s\pi(s)\sigma'(z(s))v(s)v(s)^\mathsf{T} ∇ 2 R p H ( f weighted ) = ∑ s π ( s ) σ ′ ( z ( s )) v ( s ) v ( s ) T . Attainment. Because v ( s 1 ) v(s_1) v ( s 1 ) and v ( s 2 ) v(s_2) v ( s 2 ) span R 2 \mathbb{R}^2 R 2 , the constant χ ≡ min ∥ d ∥ = 1 max i ∣ d ⋅ v ( s i ) ∣ \chi\equiv\min_{\lVert d\rVert=1}\max_i|d\cdot v(s_i)| χ ≡ min ∥ d ∥ = 1 max i ∣ d ⋅ v ( s i ) ∣ is positive, so every θ \theta θ has ∣ z ( s i ) ∣ ≥ χ ∥ θ ∥ |z(s_i)|\geq\chi\lVert\theta\rVert ∣ z ( s i ) ∣ ≥ χ ∥ θ ∥ for some i i i . Since R p H ( s i ) ( σ ( z ) ) → ∞ R_{p_H(s_i)}(\sigma(z))\to\infty R p H ( s i ) ( σ ( z )) → ∞ as ∣ z ∣ → ∞ |z|\to\infty ∣ z ∣ → ∞ , because p H ( s i ) ∈ ( 0 , 1 ) p_H(s_i)\in(0,1) p H ( s i ) ∈ ( 0 , 1 ) , R p H ( f w e i g h t e d ) → ∞ R_{p_H}(f^{\mathrm{weighted}})\to\infty R p H ( f weighted ) → ∞ as ∥ θ ∥ → ∞ \lVert\theta\rVert\to\infty ∥ θ ∥ → ∞ , and the continuous objective attains its minimum on the closed set θ ≥ 0 \theta\geq0 θ ≥ 0 .
Uniqueness. The Hessian is positive definite because σ ′ > 0 \sigma'>0 σ ′ > 0 and v ( s 1 ) , v ( s 2 ) v(s_1),v(s_2) v ( s 1 ) , v ( s 2 ) span R 2 \mathbb{R}^2 R 2 , so the objective is strictly convex and its minimizer over the convex set θ ≥ 0 \theta\geq0 θ ≥ 0 is unique. Positivity. At θ ⋆ \theta^\star θ ⋆ , f w e i g h t e d ( s ) ∈ ( 0 , 1 ) f^{\mathrm{weighted}}(s)\in(0,1) f weighted ( s ) ∈ ( 0 , 1 ) for every s s s , so every s ∈ K s\in\mathcal{K} s ∈ K contributes π ( s ) R 0 ( f w e i g h t e d ( s ) ) = − π ( s ) log ( 1 − f w e i g h t e d ( s ) ) > 0 \pi(s)R_0(f^{\mathrm{weighted}}(s))=-\pi(s)\log(1-f^{\mathrm{weighted}}(s))>0 π ( s ) R 0 ( f weighted ( s )) = − π ( s ) log ( 1 − f weighted ( s )) > 0 . Why the two interior situations are needed. Without them the minimum need not exist. With one situation s 1 s_1 s 1 with p H ( s 1 ) = 0 p_H(s_1)=0 p H ( s 1 ) = 0 at ( u T , u C ) = ( 1 , 2 ) (u_{\mathrm{T}},u_{\mathrm{C}})=(1,2) ( u T , u C ) = ( 1 , 2 ) and one situation s 2 s_2 s 2 with p H ( s 2 ) = 1 p_H(s_2)=1 p H ( s 2 ) = 1 at ( 2 , 1 ) (2,1) ( 2 , 1 ) , R p H ( f w e i g h t e d ) = π ( s 1 ) log ( 1 + e θ T − 2 θ C ) + π ( s 2 ) log ( 1 + e θ C − 2 θ T ) R_{p_H}(f^{\mathrm{weighted}})=\pi(s_1)\log(1+e^{\theta_{\mathrm{T}}-2\theta_{\mathrm{C}}})+\pi(s_2)\log(1+e^{\theta_{\mathrm{C}}-2\theta_{\mathrm{T}}}) R p H ( f weighted ) = π ( s 1 ) log ( 1 + e θ T − 2 θ C ) + π ( s 2 ) log ( 1 + e θ C − 2 θ T ) , which is positive at every θ \theta θ but equals log ( 1 + e − r ) → 0 \log(1+e^{-r})\to0 log ( 1 + e − r ) → 0 at θ = ( r , r ) \theta=(r,r) θ = ( r , r ) as r → ∞ r\to\infty r → ∞ , so its infimum, zero, is not attained.
Part (iv). Write R ( θ T , θ C ) ≡ R p H ( f w e i g h t e d ) R(\theta_{\mathrm{T}},\theta_{\mathrm{C}})\equiv R_{p_H}(f^{\mathrm{weighted}}) R ( θ T , θ C ) ≡ R p H ( f weighted ) and ϕ ( θ T ) ≡ min θ C ≥ 0 R ( θ T , θ C ) \phi(\theta_{\mathrm{T}})\equiv\min_{\theta_{\mathrm{C}}\geq0}R(\theta_{\mathrm{T}},\theta_{\mathrm{C}}) ϕ ( θ T ) ≡ min θ C ≥ 0 R ( θ T , θ C ) for θ T ≥ 0 \theta_{\mathrm{T}}\geq0 θ T ≥ 0 . For θ T ∈ [ 0 , 1 ] \theta_{\mathrm{T}}\in[0,1] θ T ∈ [ 0 , 1 ] , z ( s 1 ) ≤ u T ( s 1 ) − θ C u C ( s 1 ) z(s_1)\leq u_{\mathrm{T}}(s_1)-\theta_{\mathrm{C}}u_{\mathrm{C}}(s_1) z ( s 1 ) ≤ u T ( s 1 ) − θ C u C ( s 1 ) , which tends to − ∞ -\infty − ∞ as θ C → ∞ \theta_{\mathrm{C}}\to\infty θ C → ∞ ; since R p H ( s 1 ) ( σ ( z ) ) R_{p_H(s_1)}(\sigma(z)) R p H ( s 1 ) ( σ ( z )) decreases in z z z for z < log [ p H ( s 1 ) / ( 1 − p H ( s 1 ) ) ] z<\log[p_H(s_1)/(1-p_H(s_1))] z < log [ p H ( s 1 ) / ( 1 − p H ( s 1 ))] and tends to infinity as z → − ∞ z\to-\infty z → − ∞ , R → ∞ R\to\infty R → ∞ as θ C → ∞ \theta_{\mathrm{C}}\to\infty θ C → ∞ uniformly in θ T ∈ [ 0 , 1 ] \theta_{\mathrm{T}}\in[0,1] θ T ∈ [ 0 , 1 ] . The minimum defining ϕ \phi ϕ is therefore attained in a compact interval, and at θ T = 0 \theta_{\mathrm{T}}=0 θ T = 0 only at θ C 0 \theta_{\mathrm{C}}^0 θ C 0 , because R ( 0 , ⋅ ) R(0,\cdot) R ( 0 , ⋅ ) has second derivative ∑ s π ( s ) σ ′ ( z ( s ) ) u C ( s ) 2 > 0 \sum_s\pi(s)\sigma'(z(s))u_{\mathrm{C}}(s)^2>0 ∑ s π ( s ) σ ′ ( z ( s )) u C ( s ) 2 > 0 . As the partial minimum of a jointly convex function, ϕ \phi ϕ is convex, and by Danskin’s theorem its right derivative at zero is
ϕ ′ ( 0 + ) = ∂ R ∂ θ T ( 0 , θ C 0 ) = ∑ s ∈ S π ( s ) u T ( s ) [ σ ( − θ C 0 u C ( s ) ) − p H ( s ) ] . \phi'(0^+)=\frac{\partial R}{\partial\theta_{\mathrm{T}}}(0,\theta_{\mathrm{C}}^0)=\sum_{s\in\mathcal{S}}\pi(s)u_{\mathrm{T}}(s)\left[\sigma\left(-\theta_{\mathrm{C}}^0u_{\mathrm{C}}(s)\right)-p_H(s)\right]. ϕ ′ ( 0 + ) = ∂ θ T ∂ R ( 0 , θ C 0 ) = s ∈ S ∑ π ( s ) u T ( s ) [ σ ( − θ C 0 u C ( s ) ) − p H ( s ) ] .
By part (iii), θ T ⋆ \theta_{\mathrm{T}}^\star θ T ⋆ is the unique minimizer of ϕ \phi ϕ on [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) . If ϕ ′ ( 0 + ) < 0 \phi'(0^+)<0 ϕ ′ ( 0 + ) < 0 , ϕ \phi ϕ decreases to the right of zero, so θ T ⋆ > 0 \theta_{\mathrm{T}}^\star>0 θ T ⋆ > 0 . If ϕ ′ ( 0 + ) ≥ 0 \phi'(0^+)\geq0 ϕ ′ ( 0 + ) ≥ 0 , convexity gives ϕ ( θ T ) ≥ ϕ ( 0 ) + ϕ ′ ( 0 + ) θ T ≥ ϕ ( 0 ) \phi(\theta_{\mathrm{T}})\geq\phi(0)+\phi'(0^+)\theta_{\mathrm{T}}\geq\phi(0) ϕ ( θ T ) ≥ ϕ ( 0 ) + ϕ ′ ( 0 + ) θ T ≥ ϕ ( 0 ) for every θ T ≥ 0 \theta_{\mathrm{T}}\geq0 θ T ≥ 0 , so θ T ⋆ = 0 \theta_{\mathrm{T}}^\star=0 θ T ⋆ = 0 . Hence θ T ⋆ > 0 \theta_{\mathrm{T}}^\star>0 θ T ⋆ > 0 if and only if the condition in Equation (7) holds. In that case, at θ = θ ⋆ \theta=\theta^\star θ = θ ⋆ , f w e i g h t e d ( s ) > 1 / 2 f^{\mathrm{weighted}}(s)>1/2 f weighted ( s ) > 1/2 if and only if z ( s ) > 0 z(s)>0 z ( s ) > 0 , that is, u T ( s ) > θ C ⋆ u C ( s ) / θ T ⋆ = u ˉ T ( s ) u_{\mathrm{T}}(s)>\theta_{\mathrm{C}}^\star u_{\mathrm{C}}(s)/\theta_{\mathrm{T}}^\star=\bar{u}_{\mathrm{T}}(s) u T ( s ) > θ C ⋆ u C ( s ) / θ T ⋆ = u ˉ T ( s ) . □ \square □
Proofs for Section 5
Proof of Proposition 4. (i) Fix s s s . Since π ( s ) > 0 \pi(s)>0 π ( s ) > 0 , the generations t 1 < t 2 < ⋯ t_1<t_2<\cdots t 1 < t 2 < ⋯ with s t = s s_{t}=s s t = s are finite almost surely, and because the pairs ( s t , ω t ) (s_t,\omega_t) ( s t , ω t ) are i.i.d. (Assumption 1), the indicators 1 { ω t j = T } \mathbf{1}\{\omega_{t_j}=\mathrm{T}\} 1 { ω t j = T } are i.i.d. Bernoulli( p H ( s ) ) (p_H(s)) ( p H ( s )) . Hence n t ( s ) → ∞ n_t(s)\to\infty n t ( s ) → ∞ and, by the strong law, k t ( s ) / n t ( s ) → p H ( s ) k_t(s)/n_t(s)\to p_H(s) k t ( s ) / n t ( s ) → p H ( s ) almost surely, so Equation (8) gives f t s o c i a l ( s ) → p H ( s ) f_t^{\mathrm{social}}(s)\to p_H(s) f t social ( s ) → p H ( s ) .
(ii) Fix s s s , write x ≡ p H ( s ) x\equiv p_H(s) x ≡ p H ( s ) , and let x ^ n ≡ ( α + k ) / ( α + β + n ) \hat{x}_n\equiv(\alpha+k)/(\alpha+\beta+n) x ^ n ≡ ( α + k ) / ( α + β + n ) with k ∼ Bin ( n , x ) k\sim\operatorname{Bin}(n,x) k ∼ Bin ( n , x ) , the law of f t s o c i a l ( s ) f_t^{\mathrm{social}}(s) f t social ( s ) given n t ( s ) = n n_t(s)=n n t ( s ) = n by part (i). If x ∈ ( 0 , 1 ) x\in(0,1) x ∈ ( 0 , 1 ) , then E [ x ^ n − x ] = ( α − ( α + β ) x ) / ( α + β + n ) = O ( 1 / n ) \mathbb{E}[\hat{x}_n-x]=(\alpha-(\alpha+\beta)x)/(\alpha+\beta+n)=O(1/n) E [ x ^ n − x ] = ( α − ( α + β ) x ) / ( α + β + n ) = O ( 1/ n ) and Var ( x ^ n ) = n x ( 1 − x ) / ( α + β + n ) 2 = x ( 1 − x ) / n + O ( 1 / n 2 ) \operatorname{Var}(\hat{x}_n)=nx(1-x)/(\alpha+\beta+n)^2=x(1-x)/n+O(1/n^2) Var ( x ^ n ) = n x ( 1 − x ) / ( α + β + n ) 2 = x ( 1 − x ) / n + O ( 1/ n 2 ) . Fix ξ ∈ ( 0 , min { x , 1 − x } ) \xi\in(0,\min\{x,1-x\}) ξ ∈ ( 0 , min { x , 1 − x }) . On the event ∣ x ^ n − x ∣ ≤ ξ |\hat{x}_n-x|\le\xi ∣ x ^ n − x ∣ ≤ ξ , the Taylor expansion in Equation (5) , R x ( y ) = ( y − x ) 2 / ( 2 x ( 1 − x ) ) + O ( ∣ y − x ∣ 3 ) R_x(y)=(y-x)^2/(2x(1-x))+O(|y-x|^3) R x ( y ) = ( y − x ) 2 / ( 2 x ( 1 − x )) + O ( ∣ y − x ∣ 3 ) , holds with a remainder constant uniform in y ∈ [ x − ξ , x + ξ ] y\in[x-\xi,x+\xi] y ∈ [ x − ξ , x + ξ ] , and E ∣ x ^ n − x ∣ 3 = O ( n − 3 / 2 ) \mathbb{E}|\hat{x}_n-x|^3=O(n^{-3/2}) E ∣ x ^ n − x ∣ 3 = O ( n − 3/2 ) . On the complement, whose probability is at most 2 exp ( − 2 n ( ξ − O ( 1 / n ) ) 2 ) 2\exp(-2n(\xi-O(1/n))^2) 2 exp ( − 2 n ( ξ − O ( 1/ n ) ) 2 ) by Hoeffding’s inequality, x ^ n \hat{x}_n x ^ n lies in [ α / ( α + β + n ) , 1 − β / ( α + β + n ) ] [\alpha/(\alpha+\beta+n),1-\beta/(\alpha+\beta+n)] [ α / ( α + β + n ) , 1 − β / ( α + β + n )] , so R x ( x ^ n ) ≤ log ( ( α + β + n ) / min { α , β } ) = O ( log n ) R_x(\hat{x}_n)\le\log((\alpha+\beta+n)/\min\{\alpha,\beta\})=O(\log n) R x ( x ^ n ) ≤ log (( α + β + n ) / min { α , β }) = O ( log n ) , and the contribution is exponentially small. Hence E R x ( x ^ n ) = 1 / ( 2 n ) + O ( n − 3 / 2 ) \mathbb{E}R_x(\hat{x}_n)=1/(2n)+O(n^{-3/2}) E R x ( x ^ n ) = 1/ ( 2 n ) + O ( n − 3/2 ) . For x = 0 x=0 x = 0 , x ^ n = α / ( α + β + n ) \hat{x}_n=\alpha/(\alpha+\beta+n) x ^ n = α / ( α + β + n ) deterministically and R 0 ( x ^ n ) = − log ( 1 − x ^ n ) = α / n + O ( 1 / n 2 ) R_0(\hat{x}_n)=-\log(1-\hat{x}_n)=\alpha/n+O(1/n^2) R 0 ( x ^ n ) = − log ( 1 − x ^ n ) = α / n + O ( 1/ n 2 ) ; for x = 1 x=1 x = 1 , symmetrically, R 1 ( x ^ n ) = − log x ^ n = β / n + O ( 1 / n 2 ) R_1(\hat{x}_n)=-\log\hat{x}_n=\beta/n+O(1/n^2) R 1 ( x ^ n ) = − log x ^ n = β / n + O ( 1/ n 2 ) . In each case E R x ( x ^ n ) = c x / n + O ( n − 3 / 2 ) \mathbb{E}R_x(\hat{x}_n)=c_x/n+O(n^{-3/2}) E R x ( x ^ n ) = c x / n + O ( n − 3/2 ) , where c x c_x c x is 1 / 2 1/2 1/2 , α \alpha α , or β \beta β according as x ∈ ( 0 , 1 ) x\in(0,1) x ∈ ( 0 , 1 ) , x = 0 x=0 x = 0 , or x = 1 x=1 x = 1 . Finally n t ( s ) ∼ Bin ( t − 1 , π ( s ) ) n_t(s)\sim\operatorname{Bin}(t-1,\pi(s)) n t ( s ) ∼ Bin ( t − 1 , π ( s )) , so E [ 1 / n t ( s ) ; n t ( s ) ≥ 1 ] = 1 / ( π ( s ) t ) + O ( t − 2 ) \mathbb{E}[1/n_t(s);n_t(s)\ge1]=1/(\pi(s)t)+O(t^{-2}) E [ 1/ n t ( s ) ; n t ( s ) ≥ 1 ] = 1/ ( π ( s ) t ) + O ( t − 2 ) , E [ n t ( s ) − 3 / 2 ; n t ( s ) ≥ 1 ] = O ( t − 3 / 2 ) \mathbb{E}[n_t(s)^{-3/2};n_t(s)\ge1]=O(t^{-3/2}) E [ n t ( s ) − 3/2 ; n t ( s ) ≥ 1 ] = O ( t − 3/2 ) , and P ( n t ( s ) = 0 ) = ( 1 − π ( s ) ) t − 1 \mathbb{P}(n_t(s)=0)=(1-\pi(s))^{t-1} P ( n t ( s ) = 0 ) = ( 1 − π ( s ) ) t − 1 , while R x ( α / ( α + β ) ) < ∞ R_x(\alpha/(\alpha+\beta))<\infty R x ( α / ( α + β )) < ∞ . Hence E R x ( f t s o c i a l ( s ) ) = c x / ( π ( s ) t ) + o ( 1 / t ) \mathbb{E}R_x(f_t^{\mathrm{social}}(s))=c_x/(\pi(s)t)+o(1/t) E R x ( f t social ( s )) = c x / ( π ( s ) t ) + o ( 1/ t ) , and the weight π ( s ) \pi(s) π ( s ) in Equation (4) cancels:
E R p H ( f t s o c i a l ) = ∑ s π ( s ) E R p H ( s ) ( f t s o c i a l ( s ) ) = ∑ s c p H ( s ) / t + o ( 1 / t ) , \mathbb{E}R_{p_H}(f_t^{\mathrm{social}})=\sum_s\pi(s)\mathbb{E}R_{p_H(s)}(f_t^{\mathrm{social}}(s))=\sum_s c_{p_H(s)}/t+o(1/t), E R p H ( f t social ) = s ∑ π ( s ) E R p H ( s ) ( f t social ( s )) = s ∑ c p H ( s ) / t + o ( 1/ t ) ,
which is Equation (11) , since ∑ s c p H ( s ) = c \sum_s c_{p_H(s)}=c ∑ s c p H ( s ) = c .
Proof of Proposition 5. (i) By Equation (12) , f t a v g ( s ) f_t^{\mathrm{avg}}(s) f t avg ( s ) is an average of α / ( α + β ) \alpha/(\alpha+\beta) α / ( α + β ) and the k t ( i ) / n t ( i ) k_t(i)/n_t(i) k t ( i ) / n t ( i ) with nonnegative weights and a positive total weight, so it lies between their minimum and their maximum. (ii) If the weights are the same for all s ∈ S ′ s\in S' s ∈ S ′ , the right side of Equation (12) is the same for all s ∈ S ′ s\in S' s ∈ S ′ .
Proof of Proposition 6 and Corollaries 3 and 4. Part (i). By Equation (14) , f n , T s o c i a l ( s ) ≥ 1 / 2 f_{n,T}^{\mathrm{social}}(s)\ge1/2 f n , T social ( s ) ≥ 1/2 if and only if 2 ( α + n ) ≥ α + β + n H + n 2(\alpha+n)\ge\alpha+\beta+n_H+n 2 ( α + n ) ≥ α + β + n H + n , that is, n ≥ n H + β − α n\ge n_H+\beta-\alpha n ≥ n H + β − α , and the smallest such integer n ≥ 0 n\ge0 n ≥ 0 is max { 0 , ⌈ n H + β − α ⌉ } \max\{0,\lceil n_H+\beta-\alpha\rceil\} max { 0 , ⌈ n H + β − α ⌉} .
For part (ii) and the two corollaries, note that the path f j , T s o c i a l ( s ) = ( α + j ) / ( b + α + j ) f_{j,T}^{\mathrm{social}}(s)=(\alpha+j)/(b+\alpha+j) f j , T social ( s ) = ( α + j ) / ( b + α + j ) of Equation (14) , with b ≡ β + n H b\equiv\beta+n_H b ≡ β + n H , is deterministic, and the monitored decisions are independent. Hence
P ( τ V > n ) = ∏ j < n [ 1 − f j , T s o c i a l ( s ) ] = ∏ j < n b b + α + j . \mathbb{P}(\tau_V>n)=\prod_{j<n}[1-f_{j,T}^{\mathrm{social}}(s)]=\prod_{j<n}\frac{b}{b+\alpha+j}. P ( τ V > n ) = j < n ∏ [ 1 − f j , T social ( s )] = j < n ∏ b + α + j b .
Write L ( n ) ≡ − log P ( τ V > n ) L(n)\equiv-\log\mathbb{P}(\tau_V>n) L ( n ) ≡ − log P ( τ V > n ) and h j ≡ ( α + j ) / b h_j\equiv(\alpha+j)/b h j ≡ ( α + j ) / b , so that
L ( n ) = ∑ j < n log ( 1 + h j ) . L(n)=\sum_{j<n}\log(1+h_j). L ( n ) = j < n ∑ log ( 1 + h j ) .
Part (ii). Since z − z 2 / 2 ≤ log ( 1 + z ) ≤ z z-z^2/2\le\log(1+z)\le z z − z 2 /2 ≤ log ( 1 + z ) ≤ z for z ≥ 0 z\ge0 z ≥ 0 , ∑ j < n h j = [ α n + n ( n − 1 ) / 2 ] / b \sum_{j<n}h_j=[\alpha n+n(n-1)/2]/b ∑ j < n h j = [ α n + n ( n − 1 ) /2 ] / b , and ∑ j < n h j 2 ≤ n ( α + n ) 2 / b 2 \sum_{j<n}h_j^2\le n(\alpha+n)^2/b^2 ∑ j < n h j 2 ≤ n ( α + n ) 2 / b 2 , we have L ( n ) = [ α n + n ( n − 1 ) / 2 ] / b − δ n L(n)=[\alpha n+n(n-1)/2]/b-\delta_n L ( n ) = [ α n + n ( n − 1 ) /2 ] / b − δ n with 0 ≤ δ n ≤ n ( α + n ) 2 / ( 2 b 2 ) 0\le\delta_n\le n(\alpha+n)^2/(2b^2) 0 ≤ δ n ≤ n ( α + n ) 2 / ( 2 b 2 ) , which is Equation (15) . For x > 0 x>0 x > 0 let n ≡ ⌊ x n H ⌋ n\equiv\lfloor x\sqrt{n_H}\rfloor n ≡ ⌊ x n H ⌋ . As n H → ∞ n_H\to\infty n H → ∞ , b / n H → 1 b/n_H \to1 b / n H → 1 , so [ α n + n ( n − 1 ) / 2 ] / b → x 2 / 2 [\alpha n + n(n-1)/2]/b \to x^2/2 [ α n + n ( n − 1 ) /2 ] / b → x 2 /2 and n ( α + n ) 2 / b 2 → 0 n(\alpha+n)^2/b^2 \to0 n ( α + n ) 2 / b 2 → 0 ; since τ V \tau_V τ V is integer valued, P ( τ V > x n H ) = P ( τ V > n ) → e − x 2 / 2 \mathbb{P}(\tau_V > x\sqrt{n_H}) = \mathbb{P}(\tau_V > n) \to e^{-x^2/2} P ( τ V > x n H ) = P ( τ V > n ) → e − x 2 /2 . At x = 0 x=0 x = 0 both sides equal 1.
Corollary 3. L L L is increasing and Median ( τ V ) = min { n ≥ 1 : L ( n ) ≥ log 2 } \operatorname{Median}(\tau_V) = \min\{n \ge1 : L(n) \ge\log2\} Median ( τ V ) = min { n ≥ 1 : L ( n ) ≥ log 2 } . Fix ξ ∈ ( 0 , 1 ) \xi\in(0,1) ξ ∈ ( 0 , 1 ) and let n ± ≡ ⌈ ( 1 ± ξ ) 2 n H log 2 ⌉ n_\pm\equiv\lceil(1 \pm\xi)\sqrt{2n_H\log2}\rceil n ± ≡ ⌈( 1 ± ξ ) 2 n H log 2 ⌉ . By the bounds in part (ii), L ( n − ) → ( 1 − ξ ) 2 log 2 < log 2 L(n_-) \to(1-\xi)^2\log2 < \log2 L ( n − ) → ( 1 − ξ ) 2 log 2 < log 2 and L ( n + ) → ( 1 + ξ ) 2 log 2 > log 2 L(n_+) \to(1+\xi)^2\log2 > \log2 L ( n + ) → ( 1 + ξ ) 2 log 2 > log 2 , so n − < Median ( τ V ) ≤ n + n_- < \operatorname{Median}(\tau_V) \le n_+ n − < Median ( τ V ) ≤ n + for all large n H n_H n H . Since ξ \xi ξ is arbitrary, Median ( τ V ) ∼ 2 n H log 2 \operatorname{Median}(\tau_V) \sim\sqrt{2n_H\log2} Median ( τ V ) ∼ 2 n H log 2 .
Corollary 4. The M M M monitored decisions are drawn independently at each exposure from the same deterministic adjudication, so none of them violates at the j j j th exposure with probability [ 1 − f j − 1 , T s o c i a l ( s ) ] M [1-f^{\mathrm{social}}_{j-1,T}(s)]^M [ 1 − f j − 1 , T social ( s ) ] M , independently across exposures, and P ( τ V M > n ) = P ( τ V > n ) M \mathbb{P}(\tau_V^{M}>n)=\mathbb{P}(\tau_V>n)^M P ( τ V M > n ) = P ( τ V > n ) M . Hence Median ( τ V M ) = min { n ≥ 1 : M L ( n ) ≥ log 2 } \operatorname{Median}(\tau_V^{M})=\min\{n\ge1:ML(n)\ge\log2\} Median ( τ V M ) = min { n ≥ 1 : M L ( n ) ≥ log 2 } , and the argument for Corollary 3 with n ± ≡ ⌈ ( 1 ± ξ ) 2 n H log 2 / M ⌉ n_\pm\equiv\lceil(1\pm\xi)\sqrt{2n_H\log2/M}\rceil n ± ≡ ⌈( 1 ± ξ ) 2 n H log 2/ M ⌉ and M M M fixed gives Median ( τ V M ) ∼ 2 n H log 2 / M \operatorname{Median}(\tau_V^{M})\sim\sqrt{2n_H\log2/M} Median ( τ V M ) ∼ 2 n H log 2/ M .
Proofs for Section 6
Proof of Proposition 7. (i) With p H ( s ) = 0 p_H(s)=0 p H ( s ) = 0 every exposure brings a conduct state, so P ( D n + 1 = 1 ) = f n , C s o c i a l ( s ) = α / ( α + β + n ) \mathbb{P}(D_{n+1}=1)=f^{\mathrm{social}}_{n,C}(s)=\alpha/(\alpha+\beta+n) P ( D n + 1 = 1 ) = f n , C social ( s ) = α / ( α + β + n ) and E ∑ j ≤ N D j = ∑ n < N f n , C s o c i a l ( s ) \mathbb{E}\sum_{j\le N}D_j=\sum_{n<N}f^{\mathrm{social}}_{n,C}(s) E ∑ j ≤ N D j = ∑ n < N f n , C social ( s ) . Since z ↦ 1 / ( α + β + z ) z\mapsto1/(\alpha+\beta+z) z ↦ 1/ ( α + β + z ) is decreasing,
∫ 0 N d z α + β + z ≤ ∑ n < N 1 α + β + n ≤ 1 α + β + ∫ 0 N d z α + β + z . \int_0^N\frac{dz}{\alpha+\beta+z}\le\sum_{n<N}\frac{1}{\alpha+\beta+n}\le\frac{1}{\alpha+\beta}+\int_0^N\frac{dz}{\alpha+\beta+z}. ∫ 0 N α + β + z d z ≤ n < N ∑ α + β + n 1 ≤ α + β 1 + ∫ 0 N α + β + z d z .
and the integral is log ( 1 + N / ( α + β ) ) \log(1+N/(\alpha+\beta)) log ( 1 + N / ( α + β )) , which gives the bounds after multiplication by α \alpha α . Independent draws give P ( τ V > N ) = P ( D 1 = ⋯ = D N = 0 ) = ∏ n < N ( 1 − f n , C s o c i a l ( s ) ) = ∏ n < N ( β + n ) / ( α + β + n ) = S N ( β ) \mathbb{P}(\tau_V>N)=\mathbb{P}(D_1=\cdots=D_N=0)=\prod_{n<N}(1-f^{\mathrm{social}}_{n,C}(s))=\prod_{n<N}(\beta+n)/(\alpha+\beta+n)=S_N(\beta) P ( τ V > N ) = P ( D 1 = ⋯ = D N = 0 ) = ∏ n < N ( 1 − f n , C social ( s )) = ∏ n < N ( β + n ) / ( α + β + n ) = S N ( β ) , and since the M M M instances draw independently from the same adjudication at each exposure, P ( τ V M > N ) = P ( τ V > N ) M = S N ( β ) M \mathbb{P}(\tau_V^M>N)=\mathbb{P}(\tau_V>N)^M=S_N(\beta)^M P ( τ V M > N ) = P ( τ V > N ) M = S N ( β ) M . Since 1 − z ≤ e − z 1-z\le e^{-z} 1 − z ≤ e − z , the lower bound gives S N ( β ) ≤ exp ( − ∑ n < N f n , C s o c i a l ( s ) ) ≤ ( 1 + N / ( α + β ) ) − α S_N(\beta)\le\exp(-\sum_{n<N}f^{\mathrm{social}}_{n,C}(s))\le(1+N/(\alpha+\beta))^{-\alpha} S N ( β ) ≤ exp ( − ∑ n < N f n , C social ( s )) ≤ ( 1 + N / ( α + β ) ) − α , which tends to zero as N → ∞ N\to\infty N → ∞ .
(ii) For fixed N N N , f n , C s o c i a l ( s ) = α / ( α + β + n ) → 0 f^{\mathrm{social}}_{n,C}(s)=\alpha/(\alpha+\beta+n)\to0 f n , C social ( s ) = α / ( α + β + n ) → 0 as β → ∞ \beta\to\infty β → ∞ for every n < N n<N n < N , so the finite sum E ∑ j ≤ N D j = ∑ n < N f n , C s o c i a l ( s ) \mathbb{E}\sum_{j\le N}D_j=\sum_{n<N}f^{\mathrm{social}}_{n,C}(s) E ∑ j ≤ N D j = ∑ n < N f n , C social ( s ) tends to zero and the finite product P ( τ V M > N ) = ∏ n < N ( 1 − f n , C s o c i a l ( s ) ) M \mathbb{P}(\tau_V^M>N)=\prod_{n<N}(1-f^{\mathrm{social}}_{n,C}(s))^M P ( τ V M > N ) = ∏ n < N ( 1 − f n , C social ( s ) ) M tends to one.
(iii) Under the mixture, the probability of n n n conduct states is q + ( 1 − q ) S n ( β ) q+(1-q)S_n(\beta) q + ( 1 − q ) S n ( β ) , so after them the posterior weight on the Beta component is ( 1 − q ) S n ( β ) / [ q + ( 1 − q ) S n ( β ) ] (1-q)S_n(\beta)/[q+(1-q)S_n(\beta)] ( 1 − q ) S n ( β ) / [ q + ( 1 − q ) S n ( β )] , and the posterior mean of p H ( s ) p_H(s) p H ( s ) is that weight times α / ( α + β + n ) \alpha/(\alpha+\beta+n) α / ( α + β + n ) , as in Equation (19) . Since 1 − f n , C m i x ( s ) 1-f^{\mathrm{mix}}_{n,C}(s) 1 − f n , C mix ( s ) is the predictive probability of a conduct state, ∏ n < N ( 1 − f n , C m i x ( s ) ) \prod_{n<N}(1-f^{\mathrm{mix}}_{n,C}(s)) ∏ n < N ( 1 − f n , C mix ( s )) telescopes to the probability of N N N conduct states, q + ( 1 − q ) S N ( β ) q+(1-q)S_N(\beta) q + ( 1 − q ) S N ( β ) ; since P ( D n + 1 = 1 ) = f n , C m i x ( s ) \mathbb{P}(D_{n+1}=1)=f^{\mathrm{mix}}_{n,C}(s) P ( D n + 1 = 1 ) = f n , C mix ( s ) and the draws are independent, this product is P ( τ V > N ) \mathbb{P}(\tau_V>N) P ( τ V > N ) , and as in part (i), P ( τ V M > N ) = [ q + ( 1 − q ) S N ( β ) ] M \mathbb{P}(\tau_V^M>N)=[q+(1-q)S_N(\beta)]^M P ( τ V M > N ) = [ q + ( 1 − q ) S N ( β ) ] M , which tends to q M q^M q M because S N ( β ) → 0 S_N(\beta)\to0 S N ( β ) → 0 . Because z ≤ − log ( 1 − z ) z\le-\log(1-z) z ≤ − log ( 1 − z ) for z ∈ [ 0 , 1 ) z\in[0,1) z ∈ [ 0 , 1 ) , E ∑ j ≤ N D j = ∑ n < N f n , C m i x ( s ) ≤ − ∑ n < N log ( 1 − f n , C m i x ( s ) ) = − log [ q + ( 1 − q ) S N ( β ) ] \mathbb{E}\sum_{j\le N}D_j=\sum_{n<N}f^{\mathrm{mix}}_{n,C}(s)\le-\sum_{n<N}\log(1-f^{\mathrm{mix}}_{n,C}(s))=-\log[q+(1-q)S_N(\beta)] E ∑ j ≤ N D j = ∑ n < N f n , C mix ( s ) ≤ − ∑ n < N log ( 1 − f n , C mix ( s )) = − log [ q + ( 1 − q ) S N ( β )] , which is below − log q -\log q − log q because S N ( β ) > 0 S_N(\beta)>0 S N ( β ) > 0 .
Proof of Proposition 8. If f n , ε ( s ) f_{n,\varepsilon}(s) f n , ε ( s ) is f n , ε s o c i a l ( s ) f^{\mathrm{social}}_{n,\varepsilon}(s) f n , ε social ( s ) or f n , ε m i x ( s ) f^{\mathrm{mix}}_{n,\varepsilon}(s) f n , ε mix ( s ) , and the adjudication follows Equation (8) in every other situation, it lies in ( 0 , 1 ) (0,1) ( 0 , 1 ) in every situation and generation, so every growth factor is positive, N t > 0 N_t>0 N t > 0 for every t t t , and τ E = ∞ \tau_E=\infty τ E = ∞ ; this gives the extinction statements of parts (i) and (iii).
(i) By Assumption 1 the states at successive exposures to s s s are independent Bernoulli( ε ) (\varepsilon) ( ε ) , so f n , ε s o c i a l ( s ) f^{\mathrm{social}}_{n,\varepsilon}(s) f n , ε social ( s ) is the x ^ n \hat{x}_n x ^ n of the proof of Proposition 4 with x = ε x=\varepsilon x = ε , and that proof gives E R ε ( f n , ε s o c i a l ( s ) ) = O ( 1 / n ) \mathbb{E}\,R_\varepsilon(f^{\mathrm{social}}_{n,\varepsilon}(s))=O(1/n) E R ε ( f n , ε social ( s )) = O ( 1/ n ) , including the case ε = 1 \varepsilon=1 ε = 1 , in which R 1 ( f n , ε s o c i a l ( s ) ) = log [ 1 + β / ( α + n ) ] R_1(f^{\mathrm{social}}_{n,\varepsilon}(s))=\log[1+\beta/(\alpha+n)] R 1 ( f n , ε social ( s )) = log [ 1 + β / ( α + n )] .
(ii) For fixed n n n , f n , ε s o c i a l ( s ) ≤ ( α + n ) / ( α + β + n ) f^{\mathrm{social}}_{n,\varepsilon}(s)\leq(\alpha+n)/(\alpha+\beta+n) f n , ε social ( s ) ≤ ( α + n ) / ( α + β + n ) , which tends to zero as β → ∞ \beta\to\infty β → ∞ , and R ε ( y ) ≥ ε log ( ε / y ) + ( 1 − ε ) log ( 1 − ε ) R_\varepsilon(y)\geq\varepsilon\log(\varepsilon/y)+(1-\varepsilon)\log(1-\varepsilon) R ε ( y ) ≥ ε log ( ε / y ) + ( 1 − ε ) log ( 1 − ε ) for y ∈ ( 0 , 1 ) y\in(0,1) y ∈ ( 0 , 1 ) , which tends to infinity as y → 0 y\to0 y → 0 ; hence R ε ( f n , ε s o c i a l ( s ) ) → ∞ R_\varepsilon(f^{\mathrm{social}}_{n,\varepsilon}(s))\to\infty R ε ( f n , ε social ( s )) → ∞ uniformly in the states, and so does its expectation. Under the prohibition the adjudication at s s s is zero, so the growth factor at an exposure to s s s is m ⋅ 0 = 0 m\cdot0=0 m ⋅ 0 = 0 if it brings a task state and m m m otherwise, while every other growth factor is positive. Hence τ E \tau_E τ E is the first exposure to s s s that brings a task state, and since these states are independent Bernoulli( ε ) (\varepsilon) ( ε ) , P ( τ E > n ) = ( 1 − ε ) n \mathbb{P}(\tau_E>n)=(1-\varepsilon)^n P ( τ E > n ) = ( 1 − ε ) n , that is, τ E ∼ Geom ( ε ) \tau_E\sim\operatorname{Geom}(\varepsilon) τ E ∼ Geom ( ε ) , with mean 1 / ε 1/\varepsilon 1/ ε .
(iii) By Equation (20) , f n , ε m i x ( s ) = f n , C m i x ( s ) f^{\mathrm{mix}}_{n,\varepsilon}(s)=f^{\mathrm{mix}}_{n,C}(s) f n , ε mix ( s ) = f n , C mix ( s ) on the event k n = 0 k_n=0 k n = 0 , which has probability ( 1 − ε ) n (1-\varepsilon)^n ( 1 − ε ) n , and f n , ε m i x ( s ) = f n , ε s o c i a l ( s ) f^{\mathrm{mix}}_{n,\varepsilon}(s)=f^{\mathrm{social}}_{n,\varepsilon}(s) f n , ε mix ( s ) = f n , ε social ( s ) otherwise. Hence E R ε ( f n , ε m i x ( s ) ) ≤ E R ε ( f n , ε s o c i a l ( s ) ) + ( 1 − ε ) n R ε ( f n , C m i x ( s ) ) \mathbb{E}R_\varepsilon(f^{\mathrm{mix}}_{n,\varepsilon}(s))\leq\mathbb{E}R_\varepsilon(f^{\mathrm{social}}_{n,\varepsilon}(s))+(1-\varepsilon)^nR_\varepsilon(f^{\mathrm{mix}}_{n,C}(s)) E R ε ( f n , ε mix ( s )) ≤ E R ε ( f n , ε social ( s )) + ( 1 − ε ) n R ε ( f n , C mix ( s )) . The first term tends to zero by part (i). In the second, f n , C m i x ( s ) ≥ ( 1 − q ) S n ( β ) α / ( α + β + n ) f^{\mathrm{mix}}_{n,C}(s)\geq(1-q)S_n(\beta)\alpha/(\alpha+\beta+n) f n , C mix ( s ) ≥ ( 1 − q ) S n ( β ) α / ( α + β + n ) , and S n ( β ) = Γ ( β + n ) Γ ( α + β ) / [ Γ ( β ) Γ ( α + β + n ) ] S_n(\beta)=\Gamma(\beta+n)\Gamma(\alpha+\beta)/[\Gamma(\beta)\Gamma(\alpha+\beta+n)] S n ( β ) = Γ ( β + n ) Γ ( α + β ) / [ Γ ( β ) Γ ( α + β + n )] is of order n − α n^{-\alpha} n − α , so f n , C m i x ( s ) f^{\mathrm{mix}}_{n,C}(s) f n , C mix ( s ) is at least of order n − 1 − α n^{-1-\alpha} n − 1 − α ; since R ε ( y ) = ε log ( 1 / y ) + O ( 1 ) R_\varepsilon(y)=\varepsilon\log(1/y)+O(1) R ε ( y ) = ε log ( 1/ y ) + O ( 1 ) as y → 0 y\to0 y → 0 , R ε ( f n , C m i x ( s ) ) = O ( log n ) R_\varepsilon(f^{\mathrm{mix}}_{n,C}(s))=O(\log n) R ε ( f n , C mix ( s )) = O ( log n ) , and ( 1 − ε ) n log n → 0 (1-\varepsilon)^n\log n\to0 ( 1 − ε ) n log n → 0 .
Proof for Appendix B
Proof of Equation (B.1). Fix s s s and write p k ≡ p H , k ( s ) p_k\equiv p_{\mathrm{H},k}(s) p k ≡ p H , k ( s ) , α ˉ ≡ ∑ k α k \bar{\alpha}\equiv\sum_k\alpha_k α ˉ ≡ ∑ k α k , A = { k : p k > 0 } \mathcal{A}=\{k:p_k>0\} A = { k : p k > 0 } , so that J = ∣ A ∣ J=|\mathcal{A}| J = ∣ A ∣ , and α 0 ≡ ∑ k ∉ A α k \alpha_0\equiv\sum_{k\notin\mathcal{A}}\alpha_k α 0 ≡ ∑ k ∈ / A α k . After n n n exposures the counts ( n 1 , … , n K ) (n_1,\ldots,n_K) ( n 1 , … , n K ) are multinomial( n , p ) (n,p) ( n , p ) , and the adjudication is p ^ k ≡ ( α k + n k ) / ( α ˉ + n ) \hat{p}_k\equiv(\alpha_k+n_k)/(\bar{\alpha}+n) p ^ k ≡ ( α k + n k ) / ( α ˉ + n ) . Since p k = 0 p_k=0 p k = 0 for k ∉ A k\notin\mathcal{A} k ∈ / A , D K L ( p ∥ p ^ ) = ∑ k ∈ A p k log ( p k / p ^ k ) D_{\mathrm{KL}}(p\mathbin\Vert\hat{p})=\sum_{k\in\mathcal{A}}p_k\log(p_k/\hat{p}_k) D KL ( p ∥ p ^ ) = ∑ k ∈ A p k log ( p k / p ^ k ) . Write p ^ k = p k ( 1 + δ k ) \hat{p}_k=p_k(1+\delta_k) p ^ k = p k ( 1 + δ k ) for k ∈ A k\in\mathcal{A} k ∈ A . Because n k = 0 n_k=0 n k = 0 for k ∉ A k\notin\mathcal{A} k ∈ / A , ∑ k ∈ A p ^ k = 1 − α 0 / ( α ˉ + n ) \sum_{k\in\mathcal{A}}\hat{p}_k=1-\alpha_0/(\bar{\alpha}+n) ∑ k ∈ A p ^ k = 1 − α 0 / ( α ˉ + n ) , so ∑ k ∈ A p k δ k = − α 0 / ( α ˉ + n ) \sum_{k\in\mathcal{A}}p_k\delta_k=-\alpha_0/(\bar{\alpha}+n) ∑ k ∈ A p k δ k = − α 0 / ( α ˉ + n ) . Expanding − log ( 1 + δ ) = − δ + δ 2 / 2 + O ( ∣ δ ∣ 3 ) -\log(1+\delta)=-\delta+\delta^2/2+O(|\delta|^3) − log ( 1 + δ ) = − δ + δ 2 /2 + O ( ∣ δ ∣ 3 ) on the event that every ∣ δ k ∣ |\delta_k| ∣ δ k ∣ is small,
D K L ( p ∥ p ^ ) = α 0 α ˉ + n + 1 2 ∑ k ∈ A ( p ^ k − p k ) 2 p k + O ( ∑ k ∈ A ∣ δ k ∣ 3 ) D_{\mathrm{KL}(p\mathbin\Vert\hat{p})}=\frac{\alpha_0}{\bar{\alpha}+n}+\frac{1}{2}\sum_{k\in\mathcal{A}}\frac{(\hat{p}_k-p_k)^2}{p_k}+O\left(\sum_{k\in\mathcal{A}}|\delta_k|^3\right) D KL ( p ∥ p ^ ) = α ˉ + n α 0 + 2 1 k ∈ A ∑ p k ( p ^ k − p k ) 2 + O ( k ∈ A ∑ ∣ δ k ∣ 3 )
Each p ^ k \hat{p}_k p ^ k has bias O ( 1 / n ) O(1/n) O ( 1/ n ) and variance n p k ( 1 − p k ) / ( α ˉ + n ) 2 np_k(1-p_k)/(\bar{\alpha}+n)^2 n p k ( 1 − p k ) / ( α ˉ + n ) 2 , so E ( p ^ k − p k ) 2 = p k ( 1 − p k ) / n + O ( 1 / n 2 ) \mathbb{E}(\hat{p}_k-p_k)^2=p_k(1-p_k)/n+O(1/n^2) E ( p ^ k − p k ) 2 = p k ( 1 − p k ) / n + O ( 1/ n 2 ) and E ∣ δ k ∣ 3 = O ( n − 3 / 2 ) \mathbb{E}|\delta_k|^3=O(n^{-3/2}) E ∣ δ k ∣ 3 = O ( n − 3/2 ) . As in the proof of Proposition 4(ii), the complement of the event has exponentially small probability, and on it the regret is O ( log n ) O(\log n) O ( log n ) because every p ^ k ≥ α k / ( α ˉ + n ) \hat{p}_k\geq\alpha_k/(\bar{\alpha}+n) p ^ k ≥ α k / ( α ˉ + n ) . Hence
E D K L ( p ∥ p ^ ) = α 0 n + 1 2 n ∑ k ∈ A ( 1 − p k ) + o ( 1 n ) = 1 n [ J − 1 2 + α 0 ] + o ( 1 n ) , \mathbb{E}\,D_{\mathrm{KL}}(p\mathbin\Vert\hat{p})=\frac{\alpha_0}{n}+\frac{1}{2n}\sum_{k\in\mathcal{A}}(1-p_k)+o\left(\frac{1}{n}\right)=\frac{1}{n}\left[\frac{J-1}{2}+\alpha_0\right]+o\left(\frac{1}{n}\right), E D KL ( p ∥ p ^ ) = n α 0 + 2 n 1 k ∈ A ∑ ( 1 − p k ) + o ( n 1 ) = n 1 [ 2 J − 1 + α 0 ] + o ( n 1 ) ,
because ∑ k ∈ A ( 1 − p k ) = J − 1 \sum_{k\in\mathcal{A}}(1-p_k)=J-1 ∑ k ∈ A ( 1 − p k ) = J − 1 . This is Equation (B.1).
Proof for Appendix D
Proof of Proposition D.1 . (i) With μ = Beta ( α , β ) \mu=\operatorname{Beta}(\alpha,\beta) μ = Beta ( α , β ) and V j − 1 ( f ) = m j − 1 f k j − 1 ( 1 − f ) j − 1 − k j − 1 V_{j-1}(f)=m^{j-1}f^{k_{j-1}}(1-f)^{j-1-k_{j-1}} V j − 1 ( f ) = m j − 1 f k j − 1 ( 1 − f ) j − 1 − k j − 1 , the ratio of integrals in part (i) is B ( α + k j − 1 + 1 , β + j − 1 − k j − 1 ) / B ( α + k j − 1 , β + j − 1 − k j − 1 ) = ( α + k j − 1 ) / ( α + β + j − 1 ) B(\alpha+k_{j-1}+1,\beta+j-1-k_{j-1})/B(\alpha+k_{j-1},\beta+j-1-k_{j-1})=(\alpha+k_{j-1})/(\alpha+\beta+j-1) B ( α + k j − 1 + 1 , β + j − 1 − k j − 1 ) / B ( α + k j − 1 , β + j − 1 − k j − 1 ) = ( α + k j − 1 ) / ( α + β + j − 1 ) , which is Equation (8) at exposure j j j . Since V j ( f ) = V j − 1 ( f ) m [ f 1 { ω j = T } + ( 1 − f ) 1 { ω j = C } ] V_j(f)=V_{j-1}(f)m[f1\{\omega_j=\mathrm{T}\}+(1-f)1\{\omega_j=\mathrm{C}\}] V j ( f ) = V j − 1 ( f ) m [ f 1 { ω j = T } + ( 1 − f ) 1 { ω j = C }] , the learned policy’s growth factor at exposure j j j is ∫ V j d μ / ∫ V j − 1 d μ \int V_j\,d\mu/\int V_{j-1}\,d\mu ∫ V j d μ / ∫ V j − 1 d μ , and the product telescopes [21 Lemma 2.5] :
V ^ n = ∏ j ≤ n ∫ V j d μ ∫ V j − 1 d μ = ∫ V n d μ , \widehat{V}_n=\prod_{j\le n}\frac{\int V_j\,d\mu}{\int V_{j-1}\,d\mu}=\int V_n\,d\mu, V n = j ≤ n ∏ ∫ V j − 1 d μ ∫ V j d μ = ∫ V n d μ ,
and the integral is m n B ( α + k n , β + n − k n ) / B ( α , β ) m^n B(\alpha+k_n,\beta+n-k_n)/B(\alpha,\beta) m n B ( α + k n , β + n − k n ) / B ( α , β ) .
(ii) The lower bound holds for every prior: since μ \mu μ is a probability measure, V ^ n = ∫ 0 1 V n ( f ) μ ( d f ) ≤ V n ∗ \widehat{V}_n=\int_0^1 V_n(f)\,\mu(df)\le V_n^* V n = ∫ 0 1 V n ( f ) μ ( df ) ≤ V n ∗ . For the upper bound under the uniform prior, Equation (D.2) gives V ^ n = m n k n ! ( n − k n ) ! / ( n + 1 ) ! , \widehat{V}_n=m^n k_n!(n-k_n)!/(n+1)!,\, V n = m n k n ! ( n − k n )! / ( n + 1 )! , so
V n ∗ V ^ n = ( n + 1 ) ( n k n ) ( k n n ) k n ( 1 − k n n ) n − k n ≤ n + 1. \frac{V_n^*}{\widehat{V}_n}=(n+1)\binom{n}{k_n}\left(\frac{k_n}{n}\right)^{k_n}\left(1-\frac{k_n}{n}\right)^{n-k_n}\le n+1. V n V n ∗ = ( n + 1 ) ( k n n ) ( n k n ) k n ( 1 − n k n ) n − k n ≤ n + 1.
since the middle factor is the probability that a binomial ( n , k n / n ) \operatorname{binomial}(n,k_n/n) binomial ( n , k n / n ) variable equals k n k_n k n , which is at most one, and equals one exactly when k n ∈ { 0 , n } k_n\in\{0,n\} k n ∈ { 0 , n } . This is Theorem 1 of Cover and Ordentlich [22 ] for two assets, where it is stated as V n ∗ / V ^ n ≤ ( n + 1 ) V_n^*/\widehat{V}_n\le(n+1) V n ∗ / V n ≤ ( n + 1 ) . The bound for the prior α = β = 1 2 \alpha=\beta=\frac{1}{2} α = β = 2 1 is Theorem 2 of Cover and Ordentlich [22 ] for two assets, V n ∗ / V ^ n ≤ Γ ( 1 2 ) Γ ( n + 1 ) / Γ ( n + 1 2 ) ≤ 2 n + 1 V_n^*/\widehat{V}_n\le\Gamma(\frac{1}{2})\Gamma(n+1)/\Gamma(n+\frac{1}{2})\le2\sqrt{n+1} V n ∗ / V n ≤ Γ ( 2 1 ) Γ ( n + 1 ) /Γ ( n + 2 1 ) ≤ 2 n + 1 ; the same two bounds appear as Theorems 9.3 and 9.4 of Cesa-Bianchi and Lugosi [16 ] , where the uniform-prior case is the Laplace mixture and the half-prior case the Krichevsky–Trofimov mixture.
(iii) Subtracting log V n ( f ) \log V_n(f) log V n ( f ) from both sides of the upper bound in part (ii) gives the inequality. For f ≡ 1 f\equiv1 f ≡ 1 , V n ( 1 ) = 0 V_n(1)=0 V n ( 1 ) = 0 once k n < n k_n<n k n < n , while V ^ n > 0 \widehat{V}_n>0 V n > 0 by Equation (D.2), so the difference is + ∞ +\infty + ∞ .
(iv) Keeping separate counts for each situation runs a separate universal portfolio on the subsequence of exposures to that situation, which is Definition 2 of Cover and Ordentlich [22 ] . The lineage’s wealth over t t t generations is the product over situations of the per-situation wealths, and the best fixed adjudication in hindsight maximizes each factor separately, so the log gaps add. By part (ii), the gap in situation s s s is at most log ( n t + 1 ( s ) + 1 ) \log(n_{t+1}(s)+1) log ( n t + 1 ( s ) + 1 ) , and ∑ s log ( n t + 1 ( s ) + 1 ) ≤ ∣ S ∣ log ( t / ∣ S ∣ + 1 ) \sum_s\log(n_{t+1}(s)+1)\le|S|\log(t/|S|+1) ∑ s log ( n t + 1 ( s ) + 1 ) ≤ ∣ S ∣ log ( t /∣ S ∣ + 1 ) by the concavity of the logarithm, since ∑ s n t + 1 ( s ) = t \sum_s n_{t+1}(s)=t ∑ s n t + 1 ( s ) = t . The bound is Theorem 3 of Cover and Ordentlich [22 ] with two assets and ∣ S ∣ |S| ∣ S ∣ states of side information.
(v) Under Assumption 1, k n / n → p H ( s ) k_n/n\to p_H(s) k n / n → p H ( s ) almost surely by the strong law, so 1 n log V n ∗ = log m + k n n log k n n + ( 1 − k n n ) log ( 1 − k n n ) \frac{1}{n}\log V_n^*=\log m+\frac{k_n}{n}\log\frac{k_n}{n}+(1-\frac{k_n}{n})\log(1-\frac{k_n}{n}) n 1 log V n ∗ = log m + n k n log n k n + ( 1 − n k n ) log ( 1 − n k n ) converges to I p H ( s ) ( p H ( s ) ) I_{p_H(s)}(p_H(s)) I p H ( s ) ( p H ( s )) ; under the uniform prior, by part (ii), 1 n log V ^ n \frac{1}{n}\log\widehat{V}_n n 1 log V n is within log ( n + 1 ) / n → 0 \log(n+1)/n\to0 log ( n + 1 ) / n → 0 of it.