betaFeedback

WorksComputer Science

Coopshift: Post-Training Amplifies The Memory Curse In LLM Cooperation

Abstract

How does post-training change cooperation during repeated interaction? Within each of seven released model lineages, we compare pretrained (Base), supervised fine-tuned (SFT), and subsequent preference- or reward-optimized (Aligned) checkpoints across seven social dilemmas. Copies of each checkpoint play one another for 200 rounds, with visible history limited to the last two or forty rounds. The memory curse refers to lower cooperation when agents can see more past rounds. Our central finding is that post-training amplifies this decline. Averaged across the six binary-action games, expanding visible history reduces cooperation at every SFT and Aligned checkpoint. At each post-trained stage, these declines exceed Base's in six of seven lineages. Most of this average amplification is already present at SFT. With long history, both post-trained stages also cooperate less than Base on average in each binary-action game. In the symmetric binary games, their fitted cooperation rates fall more steeply on average than Base's as per-round cooperation costs rise, benefits fall, or noncooperation becomes more rewarding. The difference in cooperation rates between Aligned and SFT also changes from one-shot decisions to repeated play. When given the same past records, their choices depend on whether the instructions describe a final round or possible future interaction. Post-training evaluation should therefore go beyond isolated choices to assess whether cooperation is sustained in repeated interaction and how it depends on payoffs and visible history.

Introduction

Post-training adapts models to follow instructions and satisfy evaluated preferences [24, 28]. Repeated interaction adds a challenge: sustaining cooperation despite incentives to deviate [26]. In a public-goods game (PGG), for example, players keep their endowments or contribute them to a pool. The pool is multiplied by a factor greater than one and shared equally among all players, including noncontributors. In our PGG, mutual contribution maximizes group payoff and gives everyone more than mutual retention. Yet withholding one’s contribution while others contribute earns that player even more this round, so mutual contribution is not a one-round Nash equilibrium. Repetition changes the trade-off: the immediate gain from defection can be outweighed by losing future mutual cooperation [14]. We ask: how does cooperation differ across training stages, and how do these differences change when agents can see more of their interaction history?

Past interaction matters because agents can use it to decide whether to keep cooperating, withdraw, or try to restore cooperation. A longer record gives them more evidence about their partners; its effect on cooperation depends on how that evidence changes their choices. Human experiments separate private contributions costs from public returns (Gore et al., 2002), reveal payoff-dependent conditional cooperation [4], and link declining contributions to imperfect conditional cooperation ([11]). In human experiments, intermediate visible histories can also support more cooperation than longer records ([22]).

Figure 1. CoopShift: cooperation across released post-training stages. Seven lineages (21 checkpoints) play seven social dilemmas across 84 payoff configurations. One-shot decisions complement 200-round self-play. A: Volunteer’s Dilemma with a forty-round history limit (HL40): mean cooperation across seven lineages versus relative volunteering cost c/Bc/B, where cc is cost and BB each player’s benefit. The y axis spans 30–85%. B: Forty-minus-two-round history responses, in percentage points (pp): mean ± standard deviation (SD) across seven lineage summaries, each averaging the six binary games (Table 3).

Language-model cooperation depends on objectives, partners, and repeated interaction ([25, 1]). Liu et al. [20] document a memory curse: agents can cooperate less when their prompts contain more past rounds. Here, memory means the interaction record visible at each decision. Within each model family, or lineage, we compare pretrained Base, demonstration-trained SFT, and a subsequent preference- or reward-optimized Aligned checkpoint. We ask whether expanding visible history produces a larger cooperation decline at the post-trained checkpoints than at Base. This larger short-to-long-history decline is what we call amplification of the memory curse.

We evaluate 21 checkpoints from seven lineages in seven social dilemmas: PGG, a common-pool resource game (CPR), Donation Prisoner’s Dilemma (Donation-PD), the Volunteer’s Dilemma (Volunteer), Assurance, Traveler’s Dilemma (TD), and the Trust game (Table 1). One-shot prompts present a single decision without past interaction or future-round statements. In repeated play, copies of one checkpoint interact for 200 recorded rounds, seeing up to the two or forty most recent rounds. Separate tests give different checkpoints the same past-interaction records, either constructed by the researchers or saved from earlier games, and compare their current choices.

For each binary game, the cooperation rate records how often its designated cooperative action is chosen. At a fixed PGG or Donation-PD payoff setting, this rate also determines the group’s average earned points. We use collective welfare to mean the sum of all players’ earned points.

Our central finding is that post-training amplifies the memory curse (§ 4). After averaging the six binary games equally within each lineage, increasing visible history from two to forty rounds lowers cooperation at every SFT and Aligned checkpoint, by about 8 and 11 percentage points (pp) on average, respectively. Base’s mean decline is about 1 pp, and its cooperation increases in five of seven lineages. At each post-trained stage, the decline exceeds Base’s in six of seven lineages, and most of the average amplification is already present at SFT. We extend prior memory evidence by showing where the response to longer histories becomes stronger along released training sequences.

With forty visible rounds, SFT and Aligned cooperate less than Base on average in each binary game. Their cooperation rates also fall more than Base’s, on average, as per-round cooperation costs rise, benefits fall, or noncooperation becomes more rewarding (§ 3). This comparison uses fitted trends across each game’s tested payoff values and holds in the symmetric binary games. TD has a different outcome: both stages choose higher mean claims than Base. Lower contributions translate into lower mean PGG and Donation-PD payoffs. Differences in cooperation between stages therefore depend on both payoffs and visible history. § 5 compares one-shot with repeated play, then gives checkpoints the same past records but different future instructions.

Experimental Design

We compare 21 checkpoints from seven open-weight lineages. Aligned objectives correspond to direct preference optimization (DPO), reinforcement learning (RL) from human feedback (RLHF), or group relative policy optimization (GRPO). In Base / SFT / Aligned order, the display names are: OLMo-3-7B (Base / Instruct-SFT / Instruct-DPO); SmolLM2-1.7B (Base / SFT-only / Instruct); Mistral/Zephyr-7B (Mistral-v0.1 / Mistral-SFT-Beta / Zephyr-Beta); Llama/Tülu-3-8B (Llama-3.1 / Tülu-3-SFT / Tülu-3-DPO); InternLM2-7B (Base / Chat-SFT / Chat); DeepSeekMath-7B (Base / Instruct / RL); and Nemotron-Cascade-8B (Qwen3-Base / Cascade-SFT / Cascade-RLHF). We abbreviate stages as B, S, A. Appendix B identifies the exact released checkpoints.

A payoff configuration fixes a game’s player count and all its payoff parameters. In the main one-shot and repeated-game evaluation, every checkpoint uses the same 84 configurations: PGG (8), CPR (16), Donation-PD (12), Volunteer (12), Assurance (8), TD (16), and Trust (12). Each is used in one-shot, HL2, and HL40 evaluations. In the first five games, players have the same two action options and payoff rule; TD offers multiple claims; Trust has an investor and two trustees. Table 1 explains the games and parameters; Appendix C gives their complete payoff rules.

GameDecision and tensionMeaning of the raw main parameter
PGGContribute a unit to a shared pool or retain it.The marginal per-capita return (MPCR), α=r/n\alpha= r/n, is each player’s return per contributed unit, where rr is the pool multiplier and nn the group size. Higher α\alpha lowers private cost 1−α1-\alpha.
CPRRestrain extraction or extract more; each unit earns less as total extraction rises.Low extraction is 11 and high extraction is h>1h>1. Higher θ\theta increases high extraction’s private advantage at fixed hh; payoffs depend on current-round extraction.
Donation-PDPay cc to give bb to each other player, or give nothing.cc is the donor’s cost and bb the benefit to each of two others. Higher R=2b/cR=2b/c, at fixed cc, raises external returns; private cost stays cc.
VolunteerSupply a shared benefit or rely on another volunteer; one suffices.Each player receives BB if anyone volunteers; every volunteer pays cc, so extra volunteers duplicate costs. Higher ρ=c/B\rho=c/B, at fixed BB, increases volunteering cost.
AssuranceJoin a threshold project or take a sure outside payoff.A successful contributor earns a fraction λ\lambda more than the sure outside payoff. Success requires at least kk contributors, including itself; unsuccessful contributors receive zero. Higher λ\lambda raises the reward for successful coordination.
TDClaim an amount; the lower claimant gets a bonus, the higher a penalty.KK is the number of legal claim levels. Higher τ\tau, at fixed KK, raises bonus/penalty and one-step undercutting gain. Outcome: claim normalized to [0,1][0,1].
TrustChoose whether to invest; two trustees state simultaneous, contingent return intentions.Investment gives each trustee 5m5m, where mm is the multiplier. A returning trustee sends back 5k(1+s)5k(1+s); higher ss raises the investor’s return and the trustee’s cost.

Table 1. What the games measure. The main parameter changes a cost, return, or coordination reward, depending on the game.

Whose cooperation is measured? Within each run, we average players and then rounds. We then average runs and payoff configurations. Thus Trust counts each of its three players equally: the investor has weight 1/31/3 and the two trustees together 2/32/3. The separate shared-record tests score the queried player’s action.

One-shot and repeated decisions. One-shot prompts omit history and future-round statements. Repeated actions are simultaneous. Following Tewolde et al. [32] and Liu et al. [20], prompts describe probabilistic continuation and a long-run payoff objective. Our prompts state a 99% probability of another round; recording uses a fixed 200 rounds without announcing this endpoint. The history limit (HL) caps visible completed rounds at two (HL2) or forty (HL40); early in play, agents see all available rounds up to that cap. Each configuration uses five trajectory seeds or 20 one-shot draws. § 5.2 defines the separate final-round and continuing probes.

Response formats. In the main one-shot and repeated-game evaluation, the model gives a rationale and then selects a legal action label. The shared-record tests use game-native JSON actions; the PGG shown-versus-hidden test also requests a rationale before the action. Sampling uses temperature 0.7, top-pp 0.8, and top-kk 20.

What we measure. The cooperation rate is the fraction of decisions taking the game’s designated cooperative action. We report rates as percentages and changes as percentage points (pp): a fall from 70% to 60% is a change of −10 pp. The six binary-action games, including Trust, form our cooperation panel; overall cooperation and history summaries weight these games equally. TD instead measures the chosen claim, normalized from zero for the lowest legal claim to one for the highest. Tables and figures multiply this value by 100 and report it separately.

How to read the summaries. Each lineage contributes one estimate to an overall mean. We first average runs and payoff settings within that lineage; summaries of the six binary games then weight those games equally. To compare stages, we compute their difference within each lineage before averaging the seven differences. Tables report the mean and, where shown, the SD across these seven estimates. The SD describes how much the models differ; a count such as 6/7 says how many lineages show the stated direction. Intervals give a range for the mean, based on either variation across lineages or repeated sampling with the same models, as specified in each table.

Complementary stage comparisons. Self-play measures the group behavior each checkpoint produces under common game rules and history limits. Every player uses the same checkpoint, so changing stages replaces both the focal player and its partners and changes the histories generated through interaction. The shared-record tests in § 5.2 hold the presented past records fixed to compare checkpoints on common inputs.

Cooperation Rates and Payoff Responses

At HL40, Table 2 reports mean cooperation at four values of each game’s main payoff parameter, labeled D1D_1–D4D_4; the parameter row gives the game-specific values. From D1D_1 to D4D_4, cooperation costs rise, benefits fall, or noncooperation becomes more rewarding. For example, lower PGG marginal per-capita return (MPCR)—each player’s return per unit contributed—raises the private contribution cost.

Cooperation levels. Compare stages vertically within the same column. In every symmetric binary game, SFT and Aligned are below Base at all four displayed payoff values: they cooperate less under the same incentives. Averaging seven lineages at each complete binary-game configuration, SFT is below Base in 66/68 and Aligned in 64/68. Trust shows the same game-level ordering; TD instead has higher post-trained mean normalized claims.

Responses to changing payoffs. Read across a row, then compare its Drop with the other stages’. For each lineage and stage, a straight-line fit uses all four payoff values at their actual numerical spacing. The fitted drop is the fitted rate at D1D_1 minus that at D4D_4, in pp. Positive values indicate a downward fitted trend; near-zero values indicate little net fitted change. Individual steps can still rise.

In each symmetric binary game, both post-trained mean drops are positive and larger than Base’s: cooperation falls more as per-round costs rise, benefits fall, or noncooperation becomes more rewarding (from D1D_1 to D4D_4). To see the stage gaps in Table 2, subtract Base from SFT or Aligned within each column. In PGG, the SFT-minus-Base gap changes from −16.6 pp at D1D_1 to −23.5 pp at D4D_4: SFT falls further below Base. Comparing the Drop entries summarizes how these gaps change using all four points. Volunteer has a large cooperation gap but nearly parallel SFT and Base trends. Appendix § 15 reports these responses separately for each lineage.

Game / stageD1D_1D2D_2D3D_3D4D_4Drop
PGG: MPCR0.80.60.50.4–
Base (B)59.858.657.157.2+2.9
SFT (S)43.238.635.433.7+9.8
Aligned (A)42.238.735.631.4+10.5
CPR: θ\theta0.20.40.60.8–
Base (B)68.063.762.365.6+2.6
SFT (S)31.433.230.527.1+4.6
Aligned (A)23.626.224.619.5+4.2
Donation-PD: RR2.421.41.1–
Base (B)54.352.953.754.3−0.2
SFT (S)15.912.613.613.4+1.8
Aligned (A)11.78.99.38.7+2.3
Volunteer: ρ\rho0.1250.3750.6250.875–
Base (B)80.676.675.170.2+9.8
SFT (S)51.949.344.242.3+10.2
Aligned (A)46.042.636.533.8+12.8
Assurance: λ\lambda0.40.20.10.05–
Base (B)87.285.786.887.3+0.0
SFT (S)80.878.371.575.0+7.7
Aligned (A)85.280.070.476.4+12.0
TD: τ\tau0.0250.050.10.2–
Base (B)39.639.840.737.9+1.7
SFT (S)67.266.967.067.4−0.3
Aligned (A)62.662.161.460.4+2.2

Table 2. Cooperation across four payoff values at HL40. Entries are means across seven lineages (cooperation in %; TD normalized claims multiplied by 100), after averaging seeds and secondary parameters. The final column is the fitted decline across all four points (pp); the numerical parameter spacing is retained.

Do less cooperative groups also earn less? PGG and Donation-PD let us answer this directly: at a fixed payoff setting, every contribution adds a fixed positive amount to the group’s total, so the contribution rate determines average points earned per player. Other games require more information about joint actions; for example, Assurance payoffs depend on whether enough players contribute to complete the project. We therefore convert rates to returns in these two games before averaging settings. Both post-trained stages earn fewer points than Base on average at HL40, with decreases more consistent across lineages in Donation-PD.

Post-Training Amplifies the Memory Curse

Does the difference in cooperation between stages also change when more past interaction is visible? We compare separate 200-round HL2 and HL40 runs of the same checkpoints at the same complete payoff configurations. Following Liu et al. [20], the memory curse means lower cooperation with a longer visible history. Averaging the six binary games equally within each lineage, mean cooperation falls by about 8 pp for SFT and 11 pp for Aligned, compared with about 1 pp for Base.

Both post-trained stages decline in all seven lineage summaries and decline more than Base in six; Base cooperation increases in five. Most of the average amplification is already present at SFT: its decline exceeds Base’s by 6.93 pp, compared with a further 2.82 pp from SFT to Aligned. Table 3 shows that both post-trained stages’ additional declines relative to Base have intervals below zero. The smaller difference between SFT and Aligned has an interval spanning zero. Appendix D gives the history changes for each lineage, including InternLM2’s smaller post-trained declines.

History change or paired differenceMean ± SD95% intervalNegative
Base: HL40 minus HL2−1.17 ± 3.92[−4.79, +2.45]2/7
SFT: HL40 minus HL2−8.11 ± 5.46[−13.16, −3.05]7/7
Aligned: HL40 minus HL2−10.93 ± 6.40[−16.85, −5.01]7/7
SFT minus Base history change−6.93 ± 7.30[−13.68, −0.19]6/7
Aligned minus Base history change−9.76 ± 7.47[−16.67, −2.85]6/7
Aligned minus SFT history change−2.82 ± 4.37[−6.87, +1.22]6/7

Table 3. The memory curse and its amplification after post-training. The first three rows report HL40 minus HL2; the last three compare these changes between stages. Entries give mean ± SD across seven lineages, in pp, after averaging the six binary games equally within each lineage. Stage differences are calculated within lineage. The exploratory 95% tt intervals use these seven estimates; the last column counts negative estimates.

Let ps(L)p_s(L) be the expected cooperation rate, as a fraction, at stage ss with history limit LL. The history change HsH_s compares HL40 with HL2; Ks/tK_{s/t} compares that change between stages:

Hs=ps(40)−ps(2),Ks/t=Hs−Ht.(1)H_s = p_s(40) - p_s(2), \qquad K_{s/t} = H_s - H_t. \tag*{(1)}

A negative HsH_s means that showing more history lowers cooperation at that stage—the memory curse. KS/BK_{S/B} compares SFT with Base, KA/BK_{A/B} compares Aligned with Base, and KA/SK_{A/S} compares Aligned with SFT. We multiply these rate differences by 100 when reporting pp. For example, KS/B=−0.10K_{S/B} = -0.10, displayed as −10 pp, means that showing more history reduces SFT cooperation by ten percentage points more than Base cooperation.

Figure 2 separates the games and payoff values. A point left of zero means that cooperation falls more, or rises less, at the first-named stage than at the second. Black diamonds show game averages; colored points show individual lineage and payoff results. SFT’s mean history change is more negative than Base’s in PGG, CPR, Donation-PD, and Volunteer; Assurance has the opposite ordering. Aligned declines more than Base and SFT in all five symmetric binary-game means; in Trust, SFT declines more than Aligned. Longer history also lowers post-trained models’ mean returns in PGG and Donation-PD.

Figure 2. Which stage loses more cooperation when shown a longer history? Panels compare SFT vs Base (KS/BK_{S/B}), Aligned vs Base (KA/BK_{A/B}), and Aligned vs SFT (KA/SK_{A/S}). Left of zero means a more negative change from HL2 to HL40 at the first-named stage: a larger decrease or smaller increase. TD reports normalized claims. Each game/panel has 28 colored points: seven lineages × four primary payoff values, averaging five seeds and secondary parameters. Black diamonds average points equally within and across lineages. All three panels use the same percentage-point scale.

A possible feedback mechanism. A model that reduces cooperation after a partner defects changes what that partner sees next. If neither readily resumes cooperating, their responses can reinforce one another over time. Post-training may change this balance between withdrawal and recovery. Human PGG experiments show a related pattern: contributing less than others helps drive declining cooperation [11].

Beyond three checkpoints. Appendix F follows Nemotron-Cascade through seven training stages. Averaged across the six binary games, all six post-Base checkpoints show a larger mean HL2-to-HL40 decline than Base. Yet the instruction-following RL stage after RLHF raises mean cooperation in one-shot, HL2, and HL40 evaluations. Later training can therefore restore cooperation levels while a stronger history-related decline remains. Figure 4 shows how levels, payoff responses, and history effects change along this training path.

Figure 4. The same five symmetric binary games across seven checkpoints: cooperation level, all-point fitted payoff responsiveness, and HL40–HL2 history response. Points and whiskers show mean ±\pm within-checkpoint replicate SD (20 one-shot draws or five trajectory seeds); the history contrast pairs the five seeds. Each panel has its own labeled metric and axis. RLHF is the canonical Aligned endpoint.

Cooperation across Interaction Settings

The history result raises a further question: does a one-shot cooperation advantage persist in actual interaction? We focus on Aligned versus SFT to examine training after SFT, then compare their choices given the same past records.

Aligned versus SFT in one-shot and repeated play

Table 4 asks whether Aligned cooperates more than SFT. Positive entries mean that it does. The first two columns give this difference in one-shot decisions and repeated play; the third shows how much it changes. We calculate changes within each lineage before averaging them. A negative change means that moving to repeated play shifts the difference toward SFT.

GameOne-shot Aligned − SFTRepeated (HL40) Aligned − SFTChange Repeated − one-shotLineages with a decrease
PGG+8.62±6.78+8.62 \pm6.78−0.77±20.44-0.77 \pm20.44−9.39±20.00-9.39 \pm20.005/7
CPR+3.47±22.35+3.47 \pm22.35−7.08±14.96-7.08 \pm14.96−10.55±19.14-10.55 \pm19.145/7
Donation-PD−1.71±7.11-1.71 \pm7.11−4.21±5.51-4.21 \pm5.51−2.51±9.25-2.51 \pm9.254/7
Volunteer+2.66±14.47+2.66 \pm14.47−7.25±11.63-7.25 \pm11.63−9.91±5.18-9.91 \pm5.187/7
Assurance+1.93±13.67+1.93 \pm13.67+1.59±8.14+1.59 \pm8.14−0.35±8.96-0.35 \pm8.963/7
TD+2.48±9.71+2.48 \pm9.71−5.50±8.82-5.50 \pm8.82−7.98±12.26-7.98 \pm12.265/7
Trust−3.91±8.47-3.91 \pm8.47+0.21±12.99+0.21 \pm12.99+4.12±9.03+4.12 \pm9.032/7

Table 4. Does Aligned cooperate more than SFT in one-shot decisions and repeated play? The first two columns report Aligned minus SFT; positive values mean more cooperation at Aligned. Change is repeated minus one-shot, computed within each lineage before averaging; negative values mean a smaller Aligned-minus-SFT difference in repeated play. These columns give mean ±\pm SD across seven model lineages, in pp. SD describes variation in effect magnitude and direction across model lineages; the last column counts lineages with a negative change. TD measures normalized claims; Trust averages roles.

The difference decreases in all five symmetric binary-game means. In PGG, CPR, and Volunteer, Aligned cooperates more than SFT in one-shot decisions but less in repeated play. In Donation-PD, it cooperates less in both; Assurance changes little. Volunteer has the clearest pattern: the difference falls by about 10 pp, with all seven lineages showing a decrease. Lineage directions vary in the other games. TD’s mean claim difference also decreases, whereas Trust’s mean intention difference increases. A one-shot cooperation advantage can thus disappear when checkpoints actually interact.

Interpretation. Repeated play supplies feedback from partners: a model’s choices and its partners’ responses become part of the history used for later decisions. Training recipes also differ across lineages. SFT demonstrations already teach desired behavior, and later training uses different preferences and rewards. These differences may help explain why the Aligned–SFT comparison varies across models.

Instructions about future rounds have game-specific effects

Final-round and continuing decisions

Self-play gives different checkpoints different histories. We now supply the same scripted records and ask for one current decision under different future instructions across all seven games. FINAL says that the interaction ends after the current round. CONTINUE says that there is a 99% probability of another round, that actions will be visible in future rounds, and that the goal is to maximize long-run points. We keep the payoff, checkpoint, scripted past records, and current round number (41) the same between the two prompts. The comparison measures the effect of changing these future-interaction instructions together.

We supply either no past records or the last 2, 10, or 40 rounds of seven scripted behavior patterns.2 This gives 1+3×7=221 + 3 \times7 = 22 equally weighted conditions, with 20 action draws per prompt. This probe uses PGG MPCR values {0.4,0.5,0.6,0.7,0.8}\{0.4, 0.5, 0.6, 0.7, 0.8\} and CPR θ∈{0.2,0.5,0.8}\theta\in\{0.2, 0.5, 0.8\}; other payoff parameters follow Appendix § 13. Table 5 averages these single-decision tests; the earlier repeated-game results measure 200 rounds of actual interaction.

GameFINALCONTINUEMean Δ\DeltaΔ\Delta BaseΔ\Delta SFTΔ\Delta Aligned
PGG73.7 ±\pm 14.373.2 ±\pm 14.4−0.5±4.0-0.5 \pm4.0−0.1±3.4-0.1 \pm3.4+2.5±6.4+2.5 \pm6.4−3.9±6.8-3.9 \pm6.8
CPR43.9 ±\pm 18.459.0 ±\pm 15.2+15.1±9.7+15.1 \pm9.7+6.2±4.0+6.2 \pm4.0+14.0±10.4+14.0 \pm10.4+25.1±20.6+25.1 \pm20.6
Donation-PD61.0 ±\pm 21.864.0 ±\pm 14.5+3.0±9.9+3.0 \pm9.9−0.8±7.2-0.8 \pm7.2+5.5±12.7+5.5 \pm12.7+4.4±12.2+4.4 \pm12.2
Volunteer78.3 ±\pm 13.781.3 ±\pm 10.5+3.0±7.5+3.0 \pm7.5+3.6±5.1+3.6 \pm5.1+2.8±10.8+2.8 \pm10.8+2.8±11.0+2.8 \pm11.0
Assurance72.9 ±\pm 18.075.1 ±\pm 12.2+2.2±10.8+2.2 \pm10.8+1.0±6.3+1.0 \pm6.3+3.8±16.6+3.8 \pm16.6+1.6±12.9+1.6 \pm12.9
TD53.2 ±\pm 14.640.7 ±\pm 9.8−12.6±8.2-12.6 \pm8.2−8.4±9.5-8.4 \pm9.5−11.4±7.2-11.4 \pm7.2−17.9±14.3-17.9 \pm14.3
Trust73.9 ±\pm 11.975.4 ±\pm 9.5+1.5±5.9+1.5 \pm5.9−0.2±1.8-0.2 \pm1.8+3.2±9.3+3.2 \pm9.3+1.5±8.0+1.5 \pm8.0

Table 5. How instructions about future rounds change the current decision. FINAL says this is the last round; CONTINUE states a 99% chance of another. Cooperation is in percent; TD claims are rescaled from 0 (lowest allowed) to 100 (highest). Δ\Delta is CONTINUE minus FINAL, in pp. Entries are mean ±\pm SD across seven model lineages, after averaging payoff settings and 22 history conditions within lineage. SD summarizes variation across model lineages. The first three numeric columns also average stages; the last three keep stages separate. Trust equally weights the investor and a representative trustee.

In Table 5, the effect of continuing instructions is the CONTINUE rate minus the FINAL rate, denoted Δ\Delta and reported in pp. Continuing instructions increase CPR restraint by about 15 pp on average and lower TD claims by about 13 pp. Both directions hold at each of the three stages after averaging history conditions. In CPR, the increase is larger at SFT than Base and larger again at Aligned. In PGG, the small average effect combines an increase at SFT with a decrease at Aligned. Thus the same instructions about future interaction can raise or lower cooperation, depending on the game and checkpoint.

Human repeated-PD experiments find higher cooperation at greater actual continuation probabilities after participants gain experience ([10]). Our test asks for one choice after telling the model either that this is the final round or that there is a 99% chance of another round. These continuing instructions raise CPR cooperation, lower TD claims, and change PGG cooperation little on average (Table 5).

Past records and instructions about future interaction

Does showing past records have a different effect under final-round and continuing instructions? We give SFT and Aligned fresh PGG decisions at (n,r)=(3,1.4),(3,2.2),(5,1.8),(5,4.2)(n,r) = (3,1.4), (3,2.2), (5,1.8), (5,4.2) under four combinations: records shown or hidden, each with FINAL or CONTINUE. Records come from separately collected SFT and Aligned games at these settings. For each lineage and payoff, we sample eight records from each source stage in each time bin: rounds 1–20, 21–50, 51–100, and 101–200. Each checkpoint generates 32 answers per record under each of the four conditions. Payoffs and source stages receive equal weight; time bins are weighted by their lengths. REAL shows saved actions, earned points, and summaries; MASKED removes them but keeps rules, role, round number, and future instructions. Removing records also shortens the prompt. Table 6 subtracts hidden-record from shown-record cooperation: negative means less cooperation when records are shown.

Future instructionStageREALMASKEDREAL−MASKED
CONTINUESFT (S)41.9±31.741.9 \pm31.758.2±27.658.2 \pm27.6−16.3±12.3-16.3 \pm12.3
Aligned (A)41.4±35.341.4 \pm35.363.3±29.963.3 \pm29.9−22.0±12.9-22.0 \pm12.9
FINALSFT (S)35.8±22.235.8 \pm22.251.5±22.051.5 \pm22.0−15.7±21.9-15.7 \pm21.9
Aligned (A)34.6±26.434.6 \pm26.446.6±23.846.6 \pm23.8−12.0±32.0-12.0 \pm32.0

Table 6. How showing past records changes PGG cooperation. REAL shows the records; MASKED hides them. The first two numeric columns give rates (%); the last subtracts the second from the first to give the effect of showing records (pp). Negative values mean less cooperation with records visible. Entries are mean ± SD across seven lineages. All four conditions use the same payoff and record weights.

Showing past records lowers mean contribution at both stages. Under CONTINUE, the mean reduction is larger for Aligned than for SFT; under FINAL, it is larger for SFT. The ordering of these average history effects therefore reverses when the same decision is described as final rather than potentially continuing.

Subtracting SFT’s record effect from Aligned’s gives one difference for each frame. This difference falls by 9.42 pp on average when FINAL becomes CONTINUE, with the same direction in five of seven lineages. The 97.5% interval from resampling lineages is [−24.29, +2.45] pp and includes zero.

Why the final-round result is informative. In final-round PGG, withholding one’s contribution always earns 1−r/n>01-r/n>0 more points this round, holding the others’ actions fixed. Past records still change the models’ choices. This pattern is consistent with responses to past behavior persisting under explicit final-round instructions.

Related Work

Cooperation and memory in language agents. Prior studies examine how language agents respond to social objectives, partners, and repeated interaction [25, 13, 1, 26]. A fuller comparison appears in Appendix § 11. This literature evaluates both cooperation-sustaining mechanisms, such as repetition, reputation, mediation, and contracting in CoopEval [32], and history effects, including the memory curse [20]. CoopShift asks whether the cooperation change from showing more history differs across Base, SFT, and Aligned checkpoints within a released lineage. We complement this comparison with probes that give SFT and Aligned the same past records and with the payoffs earned during self-play. These measure, respectively, choices under common historical inputs and outcomes generated by interacting groups.

Post-training and human behavior. Instruction and preference optimization train models to produce desired responses [24, 28]. We ask how released checkpoints behave when their decisions affect other players. Human experiments provide comparisons for specific responses to costs, benefits, and past interaction.

Discussion and Conclusion

Differences in cooperation between checkpoints depend on both costs and benefits and the history agents can see. Most of the average memory-curse amplification appears between Base and SFT, making demonstration training central to understanding it.

Interpreting Base behavior. Base follows the same procedure of giving a rationale and then choosing an action. Its small average history effect combines different responses: InternLM2 Base and Nemotron’s Qwen3 Base decline by about 7.6 and 6.0 pp. Some Base checkpoints therefore respond to history.

Conclusion. Post-training amplifies the memory curse in our six-binary-game panel. At HL40, both post-trained stages cooperate less than Base on average in each binary game. In the symmetric binary games, their fitted cooperation rates also fall more than Base’s, on average, as per-round cooperation costs rise, benefits fall, or noncooperation becomes more rewarding. Evaluation should track how cooperation changes with payoffs and visible history and what returns it produces.

Acknowledgments

Jiayuan Liu, Emanuel Tewolde, and Vincent Conitzer thank the Cooperative AI Foundation, Macroscopic Ventures (formerly Polaris Ventures / the Center for Emerging Risk Research) and Jaan Tallinn’s donor-advised fund at Founders Pledge for financial support. Shiyi Du is partially supported by the SoftBank Group–Arm Fellowship. Tianqin Li is supported by NSF CISE RI 1816568 and NIH R01 EY030226–01A1.

Reproducibility Statement

Section 2 describes the evaluation conditions, response formats, sampling settings, and aggregation rules. Appendices B and C identify the checkpoints and specify the game payoffs. Appendix D gives history changes by lineage; Table 2 lists the payoff values used for the response fits, and Appendix E gives payoff responses by lineage. Appendix F specifies the seven-stage Nemotron-Cascade extension and its common payoff grid. Section 5.2 describes the final-round, continuing, and shown-versus-hidden record comparisons.

AI Use Statement

Generative AI assisted conceptual framing, hypothesis refinement, experimental-design feedback, elementary payoff derivations and repeated-game reasoning, implementation, data cleaning and reformatting, statistical cross-checks, result interpretation, literature synthesis, manuscript drafting, and figure/table preparation. The evaluated agents themselves generate the synthetic game decisions. The authors are responsible for scientific validation and the paper’s text, figures, tables, and claims.

Detailed Related Work

Post-training and social-agent evaluation

Instruction tuning and learning from human feedback improve evaluated instruction following and preferences [24]; DPO optimizes preferences without the same explicit reward-model-and-RL pipeline [28]. Generalization comparisons of SFT and RL [9], and the distinction between high-probability reasoning success and solutions recoverable by sampling [35], emphasize defining the evaluation target. We compare released stages on their own generated cooperation, payoff responses, and history responses across several preference-and reward-optimization objectives.

Economic-game evaluations already demonstrate sensitivity to objectives, prompts, partners and repetition [25, 13, 1]. SOTOPIA evaluates interactive goals and relationships [36]; GovSim adds resource sustainability over time [26], unlike our stationary CPR payoff. Social training studies investigate behavior cloning, self-reinforcement, reward design, and incentive-supported coordination [33, 34, 19]. CoopEval compares repetition, reputation, mediation and contracting [32]. We complement these studies by comparing released training stages under common game rules and evaluation procedures.

Memory, conditional cooperation, and recovery

Liu et al. [20] introduce the memory curse as lower cooperation with more visible interaction history and study historical content, reasoning, and interventions. We ask how this decline changes along a training sequence. Our comparisons KS/BK_{S/B} and KA/BK_{A/B} measure whether longer histories reduce cooperation more after post-training than at Base; KA/SK_{A/S} compares SFT with its later Aligned checkpoint. We build on part of that experimental framework with newly collected game data. Studies of reasoning in public goods and norm enforcement further motivate examining both the rationale and the selected action [27, 18].

Human conditional-contribution schedules [12], separately elicited preferences and beliefs [11], and distributional/reciprocity choices [8] motivate separating an opening action from its subsequent feedback. Ma et al. [22] display zero to four encounters within anonymous eight-person groups and find an intermediate-history peak; we instead compare caps of two and forty in fixed groups. Noisy repeated-PD experiments motivate studying tolerance and recovery after temporary defection [15]. Our generated self-play and fixed-record probes resolve different parts of that question: the former changes all players and their resulting histories; the latter compares responses to a shared input.

Game-specific payoff responses and human comparisons

Human public-goods studies separately manipulate MPCR/group size in ten-period play [17] and private/external returns in ten no-feedback one-shot decisions [16]; a single-decision study reports no statistically detected MPCR response [31]. These designs motivate identifying both the economic margin and the interaction protocol. Sequential dilemmas vary a second mover’s private deviation gain and harm to a cooperative first mover [4]. Their positive mutual-cooperation reference supports relative-gain measures; our response statistic instead uses the tested span of each game’s raw parameter.

Human TD claims fall with larger reward/penalty sizes under random rematching [7]; related LLM signatures are studied by Roberts et al. [29]. Human Trust experiments elicit investment and return in a sequential two-person design [5]; ours elicits one investor’s and two trustees’ simultaneous contingent intentions. Behavioral-test similarity [23] and prediction of observed human choices [30, 6] are different evaluation targets; the latter also includes one-shot/first-round settings favoring aligned models. Our comparisons concern specific payoff and history responses under each study’s stated protocol.

Continuation, presentation, and shared-input comparisons

Repeated-game theory distinguishes equilibrium support from equilibrium selection [14]. Human repeated-PD evidence studies this distinction empirically [10]. Public-goods studies use different finite or probabilistic stopping procedures and report different cooperation patterns [21, 2]. They motivate specifying both what agents are told about future rounds and how collection actually stops. Our canonical data record 200 rounds under continuing instructions; the shared-record tests vary a current decision’s instructions at fixed presented history and payoff.

Externality-framing experiments change how a decision is presented while keeping its potential payoffs fixed [3]. Our PGG comparison shows or hides past records, which also changes prompt length. In the final-round condition, contributing still changes the player’s current payoff by r/n−1r/n - 1, regardless of the records shown. These tests ask how a model responds to the information it receives, alongside our measurements of cooperation during actual interaction.

Model Checkpoints and Training Ancestry

Table 7 identifies the 21 released checkpoints in the primary panel: one Base, one SFT, and one Aligned checkpoint per lineage. A colon separates a Hugging Face repository identifier from a checkpoint subfolder. In particular, OLMo-3-7B Base denotes allenai/Olmo-3-1025-7B; 1025 is part of its release identifier. The table also names each Aligned training objective.

Training stages across lineages. OLMo-3, SmolLM2, Zephyr, and Tülu-3 use DPO releases as their Aligned checkpoints. InternLM2 uses its online-RLHF Chat release, DeepSeekMath its GRPO-trained RL release, and Nemotron-Cascade the RLHF subfolder of its intermediate-checkpoint repository. OLMo-3 and Tülu-3 contribute their DPO checkpoints to this panel. SFT and Aligned identify released training stages with lineage-specific demonstrations and subsequent optimization objectives.

Ancestry details. The InternLM2 SFT model card documents training from InternLM2-Base, so we use internlm/internlm2-base-7b. Additional pretraining distinguishes the separately released internlm/internlm2-7b from this SFT parent. The SmolLM2 SFT release names an 8k-context parent, whereas the evaluated Base is the published 1.7B endpoint listed in Table 7.

LineageStageHugging Face repository
OLMo-3-7BBaseallenai/Olmo-3-1025-7B
OLMo-3-7BSFTallenai/Olmo-3-7B-Instruct-SFT
OLMo-3-7BAligned (DPO)allenai/Olmo-3-7B-Instruct-DPO
SmolLM2-1.7BBaseHuggingFaceTB/SmolLM2-1.7B
SmolLM2-1.7BSFTHuggingFaceTB/SmolLM2-1.7B-sft-only
SmolLM2-1.7BAligned (DPO)HuggingFaceTB/SmolLM2-1.7B-Instruct
Mistral/Zephyr-7BBasemistralai/Mistral-7B-v0.1
Mistral/Zephyr-7BSFTHuggingFaceH4/mistral-7b-sft-beta
Mistral/Zephyr-7BAligned (DPO)HuggingFaceH4/zephyr-7b-beta
Llama/Tülu-3-8BBasemeta-llama/Llama-3.1-8B
Llama/Tülu-3-8BSFTallenai/Llama-3.1-Tulu-3-8B-SFT
Llama/Tülu-3-8BAligned (DPO)allenai/Llama-3.1-Tulu-3-8B-DPO
InternLM2-7BBaseinternlm/internlm2-base-7b
InternLM2-7BSFTinternlm/internlm2-chat-7b-sft
InternLM2-7BAligned (online-RLHF)internlm/internlm2-chat-7b
DeepSeekMath-7BBasedeepseek-ai/deepseek-math-7b-base
DeepSeekMath-7BSFTdeepseek-ai/deepseek-math-7b-instruct
DeepSeekMath-7BAligned (GRPO/RL)deepseek-ai/deepseek-math-7b-rl
Nemotron-Cascade-8BBaseQwen/Qwen3-8B-Base
Nemotron-Cascade-8BSFTnvidia/Nemotron-Cascade-8B-Intermediate-ckpts:SFT
Nemotron-Cascade-8BAligned (RLHF)nvidia/Nemotron-Cascade-8B-Intermediate-ckpts:RLHF

Table 7. All 21 checkpoints in the primary panel: one Base, SFT, and Aligned release per lineage. Identifiers link to the Hugging Face repositories; Nemotron’s SFT and RLHF suffixes identify checkpoint subdirectories. Aligned training objectives appear in parentheses.

Game Payoffs and Parameter Grids

Here uiu_i is player ii’s one-round payoff in game points. In binary-action games, CC denotes the designated cooperative action (ai=1a_i = 1) and DD the alternative (ai=0a_i = 0); these are explanatory labels for the implemented actions. We write 1{E}\mathbf{1}\{E\} for the indicator of event EE: it equals one when the event holds and zero otherwise. Parameters are local to each game. Every numerical example below uses a configuration in the amended main grid.

Reading the matrices. For the first five games, rows give a focal player’s action and columns give m−i=∑j≠iajm_{-i} = \sum_{j \ne i} a_j, the number of other players choosing CC. Each entry gives the focal player’s payoff conditional on this count. In these symmetric games, this count summarizes the other players’ actions, so the compact matrices specify every player’s payoff for every joint action in the multiplayer game. TD entries instead give the ordered pair (u1,u2)(u_1, u_2), and Trust entries give (u1,u2,u3)(u_1, u_2, u_3). The entries are deterministic one-round payoffs.

Public-goods game (PGG)

Each of nn players contributes its unit endowment (CC) or retains it (DD). For k=∑jajk = \sum_j a_j contributors and MPCR α=r/n\alpha= r/n, the payoff is ui=1−ai+αku_i = 1-a_i+\alpha k. We cross n∈{3,5}n \in\{3, 5\} with α∈{0.4,0.5,0.6,0.8}\alpha\in\{0.4, 0.5, 0.6, 0.8\} in the amended main panel. A contribution increases the return to every player by α\alpha, while its contributor gives up the unit endowment; its net private cost is 1−α1-\alpha.

Own actionm−i=0m_{-i}=0m−i=1m_{-i}=1⋯\cdotsm−i=n−1m_{-i}=n-1
CC: contributeα\alpha2α2\alpha⋯\cdotsnαn\alpha
DD: retain111+α1+\alpha⋯\cdots1+(n−1)α1+(n-1)\alpha

Table 1 (PDF p. 15). General payoff matrix: focal payoff uiu_i.

Own actionm−i=0m_{-i}=0m−i=1m_{-i}=1m−i=2m_{-i}=2
CC: contribute0.60.61.21.21.81.8
DD: retain1.01.01.61.62.22.2

Table 2 (PDF p. 15). Example: n=3n=3, α=0.6\alpha=0.6, hence r=1.8r=1.8.

For example, one contributor and two noncontributors receive (0.6,1.6,1.6)(0.6, 1.6, 1.6) when the contributor is Player 1.

Common-pool resource game (CPR)

Three players choose restrained extraction xi=1x_i = 1 (CC) or high extraction xi=hx_i = h (DD), with h∈{1.25,1.5,2.5,3}h \in\{1.25, 1.5, 2.5, 3\}. Their payoff is ui=xi(V−∑jxj)u_i = x_i(V-\sum_j x_j), where V=3h+1+2θV = 3h+1+2\theta and the amended main grid is θ∈{0.2,0.4,0.6,0.8}\theta\in\{0.2, 0.4, 0.6, 0.8\}. Thus m−im_{-i} counts the other restrained extractors. The game uses a stationary resource-return rule applied separately in each round, with payoffs determined by that round’s joint extraction.

General payoff matrix: focal payoff uiu_i.
Own actionm−i=0m_{-i}=0m−i=1m_{-i}=1m−i=2m_{-i}=2
CC: extract 1V−1−2hV-1-2hV−2−hV-2-hV−3V-3
DD: extract hhh(V−3h)h(V-3h)h(V−1−2h)h(V-1-2h)h(V−2−h)h(V-2-h)

Table 21.

Example: h=1.5h=1.5, θ=0.4\theta=0.4, hence V=6.3V=6.3.

Own actionm−i=0m_{-i}=0m−i=1m_{-i}=1m−i=2m_{-i}=2
CC: extract 12.302.803.30
DD: extract 1.52.703.454.20

Table 22.

In the example, all restrained extractors earn 3.30 each, while all high extractors earn 2.70 each. Holding the other two actions fixed, high extraction nevertheless gives the larger private payoff in every column.

Donation Prisoner's Dilemma (Donation-PD)

Each of three players starts with 2 points. Cooperating pays cost cc to give benefit bb to each other player; defecting pays no cost and gives nothing. The payoff is ui=2−cai+b∑j≠iaju_i=2-ca_i+b\sum_{j\ne i}a_j. We cross c∈{0.2,0.6,1.2}c\in\{0.2,0.6,1.2\} with R=2b/c∈{1.1,1.4,2.0,2.4}R=2b/c\in\{1.1,1.4,2.0,2.4\}. At fixed cc, changing RR changes others’ contribution returns while the immediate private difference remains uD−uC=cu_D-u_C=c. Its oriented response thus measures sensitivity to external returns.

General payoff matrix: focal payoff uiu_i.
Own actionm−i=0m_{-i}=0m−i=1m_{-i}=1m−i=2m_{-i}=2
CC: donate2−c2-c2−c+b2-c+b2−c+2b2-c+2b
DD: do not donate222+b2+b2+2b2+2b

Table 23.

Example: c=0.6c=0.6, R=2R=2, hence b=0.6b=0.6.

Own actionm−i=0m_{-i}=0m−i=1m_{-i}=1m−i=2m_{-i}=2
CC: donate1.42.02.6
DD: do not donate2.02.63.2

Table 24.

Volunteer's Dilemma (Volunteer)

Three players choose whether to volunteer. If at least one volunteers, every player receives benefit BB, and each volunteer separately pays cost cc. If nobody volunteers, all receive zero. Thus ui=B1{∑jaj≥1}−caiu_i=B\mathbf{1}\{\sum_j a_j\ge1\}-ca_i, with B∈{2,4,8}B\in\{2,4,8\} and ρ=c/B∈{0.125,0.375,0.625,0.875}\rho=c/B\in\{0.125,0.375,0.625,0.875\}.

A volunteer always earns the net payoff B−c=B(1−ρ)B-c=B(1-\rho), whether or not anyone else volunteers. A nonvolunteer earns BB if another player volunteers and zero otherwise. Each player receives the full benefit BB when provision occurs, and every volunteer pays its own full cost cc. Across the four tested cost ratios, a volunteer therefore receives 0.875B0.875B, 0.625B0.625B, 0.375B0.375B, or 0.125B0.125B, respectively.

General payoff matrix: focal payoff uiu_i.
Own actionm−i=0m_{-i}=0m−i=1m_{-i}=1m−i=2m_{-i}=2
CC: volunteerB−cB-cB−cB-cB−cB-c
DD: do not volunteer0BBBB

Table 25.

Example: B=4B=4, ρ=0.375\rho=0.375, hence c=1.5c=1.5.

Own actionm−i=0m_{-i}=0m−i=1m_{-i}=1m−i=2m_{-i}=2
CC: volunteer2.52.52.5
DD: do not volunteer044

Table 26.

For example, (C,D,D)(C,D,D) gives (2.5,4,4)(2.5,4,4), (C,C,D)(C,C,D) gives (2.5,2.5,4)(2.5,2.5,4), and (D,D,D)(D,D,D) gives (0,0,0)(0,0,0). Volunteering is privately beneficial when nobody else volunteers; otherwise, not volunteering avoids the cost.

Volunteer is a threshold-one public-good dilemma: everyone benefits from provision, but each prefers another player to bear its cost. Unlike a dominant-defection dilemma, the best response depends on whether another player volunteers. Since 0<c<B0 < c < B throughout our grid, the pure-strategy Nash equilibria have exactly one volunteer. The cooperation rate here measures willingness to supply the benefit; group performance depends on securing provision while avoiding redundant costs.

Assurance

Three players choose to join a project (CC) or take a safe outside option (DD). A contributor receives hh if at least k∈{2,3}k \in\{2,3\} players, including itself, contribute, and zero otherwise. Opting out always gives 3, including when the others complete the project. Thus

ui={h1{m−i+1≥k},ai=1,3,ai=0,h=3(1+λ),λ∈{0.05,0.10,0.20,0.40}.u_i = \begin{cases} h\mathbf{1}\{m_{-i}+1 \ge k\}, & a_i=1,\\ 3, & a_i=0, \end{cases} \qquad h=3(1+\lambda),\qquad\lambda\in\{0.05,0.10,0.20,0.40\}.

Here hh is the contributor’s total payoff when the project succeeds.

Own actionm−i=0m_{-i}=0m−i=1m_{-i}=1m−i=2m_{-i}=2
CC: join0h1{k=2}h\mathbf{1}\{k=2\}hh
DD: safe option333

Table 1 (PDF p. 17). General payoff matrix: focal payoff uiu_i, k∈{2,3}k\in\{2,3\}.

Own actionm−i=0m_{-i}=0m−i=1m_{-i}=1m−i=2m_{-i}=2
CC: join03.63.6
DD: safe option333

Table 2 (PDF p. 17). Example: k=2k=2, λ=0.2\lambda=0.2, hence h=3.6h=3.6.

In this example, (C,C,D)(C,C,D) gives (3.6,3.6,3)(3.6,3.6,3), whereas (C,D,D)(C,D,D) gives (0,3,3)(0,3,3). At threshold k=3k=3 with the same λ\lambda, the contributor’s middle-column payoff becomes zero; the other entries are unchanged.

Full cooperation is a strict stage-game Nash equilibrium throughout this grid: against two contributors, joining pays h>3h>3, more than the outside option. All opting out is also a strict equilibrium because one participant cannot meet either threshold. The cooperative outcome makes every player better off, and reaching it requires coordinating participation.

Traveler’s Dilemma (TD)

Two players choose integer claims from 2 to K+1K+1, with K∈{8,16,32,64}K\in\{8,16,32,64\} legal levels. Equal claims pay that claim to both players. For unequal claims, the lower claimant receives its claim plus bonus BB, while the higher claimant receives the lower claim minus BB, where B=1+τ(K−1)B=1+\tau(K-1) and τ∈{0.025,0.05,0.10,0.20}\tau\in\{0.025,0.05,0.10,0.20\}. For a chosen claim cc, the reported behavioral outcome is (c−2)/(K−1)(c-2)/(K-1); this normalized claim is distinct from payoff.

Claim23⋯\cdotsK+1K+1
2(2,2)(2,2)(2+B,2−B)(2+B,2-B)⋯\cdots(2+B,2−B)(2+B,2-B)
3(2−B,2+B)(2-B,2+B)(3,3)(3,3)⋯\cdots(3+B,3−B)(3+B,3-B)
⋮\vdots⋮\vdots⋮\vdots⋱\ddots⋮\vdots
K+1K+1(2−B,2+B)(2-B,2+B)(3−B,3+B)(3-B,3+B)⋯\cdots(K+1,K+1)(K+1,K+1)

Table 3 (PDF p. 17). General payoff matrix: entries (u1,u2)(u_1,u_2). Rows are Player 1’s claims; columns are Player 2’s claims.

For K=8K=8 and τ=0.1\tau=0.1, B=1.7B=1.7: equal claims of 9 pay (9,9)(9,9), while claims (8,9)(8,9) pay (9.7,6.3)(9.7,6.3). Undercutting raises the lower claimant’s payoff by 0.7, and the general matrix specifies all 64 joint actions.

Trust

Player 1 is an investor with endowment E=10E=10; Players 2 and 3 are trustees. Keeping the endowment gives (10,0,0)(10,0,0). Investing creates G=5mG=5m for each trustee, with multiplier m∈{2,3,4}m\in\{2,3,4\}. trustee’s contingent action either returns r=5(1+s)r = 5(1+s) to the investor (CC) or retains everything (DD), with s∈{0.2,0.4,0.6,0.8}s \in\{0.2, 0.4, 0.6, 0.8\}. Returning therefore leaves a trustee G−rG-r, while retaining gives GG, conditional on investment. The investor receives the sum returned by the two trustees.

General payoff matrix: entries (u1,u2,u3)(u_1,u_2,u_3).

Columns give the ordered trustee actions (a2,a3)(a_2,a_3).

Investor action(C,C)(C,C)(C,D)(C,D)(D,C)(D,C)(D,D)(D,D)
C: invest(2r,G−r,G−r)(2r,G-r,G-r)(r,G−r,G)(r,G-r,G)(r,G,G−r)(r,G,G-r)(0,G,G)(0,G,G)
D: keep(10,0,0)(10,0,0)(10,0,0)(10,0,0)(10,0,0)(10,0,0)(10,0,0)(10,0,0)

Example: m=3m=3, s=0.4s=0.4, hence G=15G=15 and r=7r=7.

Investor action(C,C)(C,C)(C,D)(C,D)(D,C)(D,C)(D,D)(D,D)
C: invest(14,8,8)(14,8,8)(7,8,15)(7,8,15)(7,15,8)(7,15,8)(0,15,15)(0,15,15)
D: keep(10,0,0)(10,0,0)(10,0,0)(10,0,0)(10,0,0)(10,0,0)(10,0,0)(10,0,0)

The implementation elicits all three intentions simultaneously and executes trustee returns conditional on investment.

History Effects by Model Lineage

Table 8 expands the six-binary-game history comparison in § 4 by lineage. Within each lineage, we average the five trajectory seeds and payoff configurations in each game, then weight games equally. The table reports HL40 minus HL2 at each training stage and the corresponding paired differences between stages, in pp. Negative HsH_s means cooperation declines with longer history; negative Ks/tK_{s/t} means a larger decline, or smaller increase, at stage ss than at stage tt (Equation (1)). InternLM2’s SFT and Aligned declines are smaller than Base’s.

LineageHBH_BHSH_SHAH_AKS/BK_{S/B}KA/BK_{A/B}KA/SK_{A/S}
OLMo+1.7±1.4+1.7 \pm1.4−13.5±1.4-13.5 \pm1.4−8.9±0.8-8.9 \pm0.8−15.2±2.8-15.2 \pm2.8−10.7±2.0-10.7 \pm2.0+4.6±1.0+4.6 \pm1.0
SmolLM2+0.0±0.1+0.0 \pm0.1−16.1±2.4-16.1 \pm2.4−18.1±2.3-18.1 \pm2.3−16.2±2.4-16.2 \pm2.4−18.1±2.4-18.1 \pm2.4−1.9±2.0-1.9 \pm2.0
DeepSeekMath+0.9±0.2+0.9 \pm0.2−5.6±0.8-5.6 \pm0.8−9.7±1.1-9.7 \pm1.1−6.5±0.8-6.5 \pm0.8−10.7±1.0-10.7 \pm1.0−4.1±1.2-4.1 \pm1.2
InternLM2−7.6±0.8-7.6 \pm0.8−2.3±0.4-2.3 \pm0.4−2.8±0.5-2.8 \pm0.5+5.3±0.9+5.3 \pm0.9+4.8±0.6+4.8 \pm0.6−0.5±0.6-0.5 \pm0.6
Nemotron−6.0±1.7-6.0 \pm1.7−10.9±1.5-10.9 \pm1.5−19.7±2.7-19.7 \pm2.7−4.9±2.9-4.9 \pm2.9−13.7±3.8-13.7 \pm3.8−8.8±3.0-8.8 \pm3.0
Tulu+1.4±0.8+1.4 \pm0.8−6.0±0.9-6.0 \pm0.9−12.9±1.1-12.9 \pm1.1−7.4±0.9-7.4 \pm0.9−14.3±0.3-14.3 \pm0.3−6.9±1.0-6.9 \pm1.0
Zephyr+1.3±0.6+1.3 \pm0.6−2.4±0.2-2.4 \pm0.2−4.4±0.8-4.4 \pm0.8−3.7±0.6-3.7 \pm0.6−5.7±1.0-5.7 \pm1.0−2.0±0.7-2.0 \pm0.7

Table 8. History changes for each lineage, averaged over the six binary games. Each entry is mean ±\pm SD in pp across five matched run IDs, after averaging payoff configurations within game and games equally. The SD summarizes variation among repeated runs within each lineage.

Payoff Responses by Model Lineage

Table 10 adds paired stage differences to the four-point payoff summaries in the main text. Table 11–16 retain each lineage separately, with SD across five repeated runs. Within each lineage, we average the other payoff parameters and fit all four values of the game’s main parameter. Stage differences pair runs by seed ID; realized random draws can differ across deployments.

Table 10: Full HL40 payoff surfaces and paired stage differences. Each game’s first row gives four values of its main payoff parameter (D1D_1–D4D_4), ordered toward higher cooperation costs, lower benefits, or greater rewards for noncooperation. Entries average seven lineages equally, after averaging seeds and secondary parameters within lineage. Stage rows give cooperation rates (%; TD normalized claims multiplied by 100); difference rows give SFT minus Base and Aligned minus Base (pp). Negative differences mean lower cooperation than Base (lower claims in TD). When a negative difference becomes more negative from D1D_1 to D4D_4, the post-trained stage falls further below Base. The final column uses a straight-line fit to all four points (Equation (2)). In a stage row, a positive drop means a fitted decline; in a difference row, it is the post-trained drop minus Base’s, so a positive value means a larger fitted decline than Base’s.

Game / stageD1D_1D2D_2D3D_3D4D_4Fitted drop
PGG: MPCR0.80.60.50.4–
Base (B)59.858.657.157.2+2.9
SFT (S)43.238.635.433.7+9.8
Aligned (A)42.238.735.631.4+10.5
SFT − Base−16.6−20.0−21.6−23.5+6.9
Aligned − Base−17.7−19.9−21.5−25.8+7.6
CPR: θ\theta0.20.40.60.8–
Base (B)68.063.762.365.6+2.6
SFT (S)31.433.230.527.1+4.6
Aligned (A)23.626.224.619.5+4.2
SFT − Base−36.6−30.5−31.8−38.5+2.0
Aligned − Base−44.4−37.5−37.7−46.1+1.6
Donation-PD: RR2.421.41.1–
Base (B)54.352.953.754.3−0.2
SFT (S)15.912.613.613.4+1.8
Aligned (A)11.78.99.38.7+2.3
SFT − Base−38.4−40.3−40.1−41.0+2.1
Aligned − Base−42.7−44.1−44.4−45.6+2.5
Volunteer: ρ\rho0.1250.3750.6250.875–
Base (B)80.676.675.170.2+9.8
SFT (S)51.949.344.242.3+10.2
Aligned (A)46.042.636.533.8+12.8
SFT − Base−28.6−27.2−30.9−27.8+0.4
Aligned − Base−34.5−34.0−38.6−36.4+3.1
Assurance: λ\lambda0.40.20.10.05–
Base (B)87.285.786.887.3+0.0
SFT (S)80.878.371.575.0+7.7
Aligned (A)85.280.070.476.4+12.0
SFT − Base−6.4−7.3−15.4−12.3+7.7
Aligned − Base−2.0−5.7−16.5−10.9+11.9
TD: τ\tau0.0250.050.10.2–
Base (B)39.639.840.737.9+1.7
SFT (S)67.266.967.067.4−0.3
Aligned (A)62.662.161.460.4+2.2
SFT − Base+27.6+27.1+26.4+29.5−2.0
Aligned − Base+23.0+22.4+20.8+22.4+0.4
Lineage / stageD1D_1D2D_2D3D_3D4D_4Fitted drop
PGG: MPCR0.80.60.50.4–

Table 10. Full HL40 payoff surfaces and paired stage differences. Each game’s first row gives four values of its main payoff parameter (D1D_1–D4D_4), ordered toward higher cooperation costs, lower benefits, or greater rewards for noncooperation. Entries average seven lineages equally, after averaging seeds and secondary parameters within lineage. Stage rows give cooperation rates (%; TD normalized claims multiplied by 100); difference rows give SFT minus Base and Aligned minus Base (pp). Negative differences mean lower cooperation than Base (lower claims in TD). When a negative difference becomes more negative from D1D_1 to D4D_4, the post-trained stage falls further below Base. The final column uses a straight-line fit to all four points (Equation (2)). In a stage row, a positive drop means a fitted decline; in a difference row, it is the post-trained drop minus Base’s, so a positive value means a larger fitted decline than Base’s.

Lineage / stageD1D_1D2D_2D3D_3D4D_4Fitted drop
PGG: MPCR0.80.60.50.4–
OLMo-3-7B
Base (B)20.3 ± 23.710.1 ± 7.33.3 ± 2.64.8 ± 1.3+17.0 ± 24.9
SFT (S)8.5 ± 2.17.6 ± 3.06.2 ± 1.54.9 ± 0.4+3.6 ± 2.1
Aligned (A)16.1 ± 6.113.4 ± 3.211.7 ± 2.99.0 ± 2.5+6.8 ± 4.5
SFT − Base−11.8 ± 23.9−2.4 ± 9.3+2.9 ± 2.6+0.1 ± 1.5−13.4 ± 25.5
Aligned − Base−4.1 ± 28.5+3.3 ± 8.3+8.4 ± 3.8+4.2 ± 2.4−10.1 ± 27.8
SmolLM2-1.7B
Base (B)99.8 ± 0.199.8 ± 0.1100.0 ± 0.199.9 ± 0.1−0.1 ± 0.2
SFT (S)40.7 ± 7.034.6 ± 3.037.8 ± 7.237.3 ± 5.9+3.1 ± 1.5
Aligned (A)34.1 ± 4.834.8 ± 3.239.4 ± 6.036.3 ± 6.2−3.5 ± 7.4
SFT − Base−59.1 ± 6.9−65.1 ± 2.9−62.2 ± 7.1−62.6 ± 5.9+3.1 ± 1.5
Aligned − Base−65.8 ± 4.8−64.9 ± 3.1−60.6 ± 6.0−63.6 ± 6.1−3.4 ± 7.4
DeepSeekMath-7B
Base (B)98.6 ± 0.298.9 ± 0.298.9 ± 0.498.8 ± 0.3−0.2 ± 0.3
SFT (S)35.6 ± 6.634.4 ± 6.437.5 ± 9.035.4 ± 4.8−0.7 ± 9.1
Aligned (A)3.8 ± 0.73.8 ± 0.74.3 ± 1.73.1 ± 0.6+0.4 ± 1.2
SFT − Base−63.1 ± 6.6−64.4 ± 6.4−61.3 ± 8.9−63.4 ± 4.9−0.5 ± 9.3
Aligned − Base−94.8 ± 0.8−95.1 ± 0.8−94.5 ± 1.5−95.7 ± 0.6+0.6 ± 1.3
InternLM2-7B
Base (B)36.9 ± 2.137.2 ± 1.535.3 ± 2.137.9 ± 2.4−0.3 ± 2.5
SFT (S)46.4 ± 2.943.3 ± 2.933.4 ± 1.334.4 ± 2.6+13.8 ± 4.4
Aligned (A)54.4 ± 0.948.7 ± 1.941.0 ± 2.740.7 ± 2.8+14.9 ± 2.9
SFT − Base+9.6 ± 4.9+6.1 ± 3.9−1.9 ± 3.3−3.5 ± 4.4+14.0 ± 6.3
Aligned − Base+17.6 ± 2.2+11.5 ± 1.7+5.8 ± 4.1+2.8 ± 3.5+15.2 ± 1.4
Nemotron-Cascade-8B
Base (B)16.8 ± 19.617.7 ± 16.414.7 ± 12.012.8 ± 2.6+4.0 ± 16.5
SFT (S)63.9 ± 14.256.3 ± 10.245.2 ± 22.146.1 ± 9.0+19.8 ± 20.0
Aligned (A)50.7 ± 9.752.9 ± 9.732.4 ± 20.128.7 ± 5.1+24.1 ± 10.3
SFT − Base+47.2 ± 26.9+38.7 ± 19.3+30.6 ± 30.7+33.3 ± 8.6+15.8 ± 29.9
Aligned − Base+33.9 ± 24.0+35.2 ± 23.8+17.8 ± 29.8+15.9 ± 4.9+20.1 ± 20.8
Tülu-3-8B
Base (B)98.5 ± 0.198.5 ± 0.598.5 ± 0.498.4 ± 0.3+0.1 ± 0.2
SFT (S)50.6 ± 2.637.7 ± 4.836.1 ± 2.826.2 ± 1.0+23.0 ± 1.5
Aligned (A)45.4 ± 6.826.9 ± 2.429.8 ± 5.618.0 ± 1.7+25.1 ± 8.6
SFT − Base−48.0 ± 2.7−60.7 ± 5.1−62.4 ± 3.0−72.2 ± 1.1+22.9 ± 1.5
Aligned − Base−53.1 ± 6.7−71.6 ± 2.9−68.7 ± 6.0−80.4 ± 1.9+25.0 ± 8.8
Zephyr-7B
Base (B)47.9 ± 1.547.9 ± 2.148.8 ± 1.947.9 ± 0.9−0.3 ± 1.4
SFT (S)56.8 ± 1.555.9 ± 3.351.8 ± 1.251.6 ± 1.4+5.8 ± 1.4
Aligned (A)90.6 ± 2.990.5 ± 2.390.2 ± 1.284.1 ± 1.9+5.3 ± 2.9
SFT − Base+8.9 ± 0.7+8.0 ± 2.7+2.9 ± 2.0+3.8 ± 1.2+6.1 ± 0.9
Aligned − Base+42.7 ± 2.6+42.6 ± 3.6+41.3 ± 0.9+36.2 ± 2.6+5.6 ± 2.2

Table 11. PGG: HL40 payoff responses for each lineage. Entries are mean ± SD across five runs, averaging secondary parameters within each run. Cooperation rates are percentages; stage differences and fitted declines are in pp. Stage differences match runs by seed ID. The final column fits all four parameter values within each run before taking the mean and SD.

For a game’s four tested raw parameter values x1,…,x4x_1,\ldots,x_4, ordered from D1D_1 to D4D_4, let zj=(xj−x1)/(x4−x1)z_j=(x_j-x_1)/(x_4-x_1). This preserves the numerical spacing while setting z1=0z_1=0 and z4=1z_4=1, including when the raw values descend. Fit the cooperation rate as a fraction with an intercept, f^(z)=η^+β^zz\hat{f}(z)=\hat{\eta}+\hat{\beta}_z z. The fitted drop in percentage points is

S=100[f^(0)−f^(1)]=−100β^z.(2)S=100[\hat{f}(0)-\hat{f}(1)]=-100\hat{\beta}_z. \tag*{(2)}

A positive SS is a downward fitted trend across the tested range; individual adjacent observations may still rise.

Models with high Base cooperation can also have high SFT cooperation. Pairing stages within lineage lets Figure 3 display their response differences. To compare variation between models with variation among payoff settings within a model, let dℓcd_{\ell c} be the HL40 SFT-minus-Base difference in pp for lineage ℓ\ell and configuration cc in one game, and let dˉℓ\bar{d}_{\ell} be its configuration average. Then

Figure 3. Fitted payoff-drop differences relative to Base. Positive values mean a larger fitted decline after post-training across the tested payoff span. Each point is a lineage difference, averaging runs and the other payoff parameters; diamonds and bars show mean ± SD across seven lineages. Both panels use the same scale. TD measures normalized claims.

Var⁡ℓ,c(dℓc)=Var⁡ℓ(dˉℓ)+EℓVar⁡c(dℓc).\operatorname{Var}_{\ell,c}(d_{\ell c}) = \operatorname{Var}_{\ell}(\bar{d}_{\ell}) + \mathbb{E}_{\ell}\operatorname{Var}_{c}(d_{\ell c}).

With equal weights over observed estimates, differences between lineage averages account for 90.8% of this variation in PGG and 89.8% in CPR (Table 9).

GameTotalBetween lineageWithin lineage/config
PGG1586.551439.81146.74
CPR829.71745.2784.44

Table 9. Where does the HL40 SFT–Base difference come from? We separate differences between lineage averages from differences among payoff settings within a lineage. Entries use equal lineage/configuration weights and population variance, in pp2\mathrm{pp}^{2}, calculated from the observed estimates.

Table 12. CPR: HL40 payoff responses for each lineage. Entries are mean ± SD across five runs, averaging secondary parameters within each run. Cooperation rates are percentages; stage differences and fitted declines are in pp. Stage differences match runs by seed ID. The final column fits all four parameter values within each run before taking the mean and SD.

Lineage / stageD1D_1D2D_2D3D_3D4D_4Fitted drop
CPR: θ\theta0.20.40.60.8–
OLMo-3-7B
Base (B)56.5 ± 8.745.9 ± 7.145.5 ± 8.442.6 ± 9.4+12.7 ± 8.8
SFT (S)12.3 ± 2.512.6 ± 3.713.0 ± 6.312.1 ± 2.8+0.0 ± 2.8
Aligned (A)9.8 ± 1.67.9 ± 0.78.2 ± 0.38.8 ± 1.5+0.8 ± 1.9
SFT − Base−44.2 ± 6.6−33.3 ± 4.4−32.5 ± 10.6−30.4 ± 10.8−12.7 ± 11.6
Aligned − Base−46.7 ± 8.2−38.0 ± 6.5−37.3 ± 8.3−33.8 ± 10.7−11.9 ± 8.5
SmolLM2-1.7B
Base (B)99.6 ± 0.199.6 ± 0.299.6 ± 0.299.7 ± 0.1−0.1 ± 0.1
SFT (S)8.5 ± 1.519.0 ± 2.717.1 ± 3.29.5 ± 1.4−0.3 ± 0.9
Aligned (A)8.4 ± 0.912.9 ± 2.712.6 ± 1.47.6 ± 0.8+0.8 ± 1.5
SFT − Base−91.0 ± 1.5−80.6 ± 2.6−82.4 ± 3.2−90.2 ± 1.3−0.2 ± 1.0
Aligned − Base−91.2 ± 0.8−86.6 ± 2.7−87.0 ± 1.3−92.1 ± 0.7+0.9 ± 1.5
DeepSeekMath-7B
Base (B)97.8 ± 0.483.4 ± 3.283.7 ± 1.997.8 ± 0.4−0.0 ± 0.9
SFT (S)54.1 ± 3.766.3 ± 3.063.3 ± 3.152.3 ± 2.8+2.5 ± 5.5
Aligned (A)8.8 ± 1.136.5 ± 2.834.4 ± 4.28.9 ± 0.7+0.5 ± 0.8
SFT − Base−43.8 ± 3.9−17.1 ± 2.0−20.4 ± 3.5−45.5 ± 3.1+2.6 ± 5.7
Aligned − Base−89.0 ± 0.8−46.9 ± 4.0−49.3 ± 4.2−88.9 ± 0.8+0.6 ± 1.2
InternLM2-7B
Base (B)45.2 ± 1.743.6 ± 1.642.3 ± 1.345.5 ± 1.0+0.1 ± 1.0
SFT (S)27.9 ± 2.525.0 ± 1.024.7 ± 1.025.8 ± 1.0+2.0 ± 2.2
Aligned (A)40.3 ± 2.038.8 ± 1.536.5 ± 1.736.5 ± 2.0+4.1 ± 3.7
SFT − Base−17.3 ± 2.9−18.5 ± 1.7−17.7 ± 1.5−19.7 ± 1.9+1.9 ± 2.0
Aligned − Base−4.9 ± 1.7−4.8 ± 2.1−5.9 ± 1.9−9.1 ± 2.8+4.0 ± 3.0
Nemotron-Cascade-8B
Base (B)22.0 ± 10.435.3 ± 8.825.8 ± 4.919.7 ± 2.3+4.8 ± 8.2
SFT (S)46.4 ± 7.039.4 ± 7.430.4 ± 9.126.6 ± 5.6+20.5 ± 8.1
Aligned (A)32.1 ± 15.422.3 ± 2.321.4 ± 3.017.8 ± 3.5+13.2 ± 15.4
SFT − Base+24.4 ± 15.9+4.2 ± 11.2+4.6 ± 11.6+6.8 ± 5.5+15.7 ± 15.2
Aligned − Base+10.2 ± 18.4−13.0 ± 10.2−4.4 ± 2.9−1.9 ± 4.3+8.3 ± 17.4
Tülu-3-8B
Base (B)91.6 ± 3.075.3 ± 2.976.7 ± 2.190.5 ± 4.7+0.5 ± 2.1
SFT (S)34.5 ± 1.334.3 ± 2.031.9 ± 1.329.9 ± 1.7+4.9 ± 2.0
Aligned (A)33.3 ± 1.232.6 ± 0.928.4 ± 1.027.1 ± 0.4+6.9 ± 1.5
SFT − Base−57.1 ± 2.4−41.0 ± 2.8−44.9 ± 1.9−60.6 ± 5.7+4.3 ± 4.0
Aligned − Base−58.2 ± 2.9−42.7 ± 2.7−48.4 ± 2.1−63.4 ± 4.8+6.3 ± 2.6
Zephyr-7B
Base (B)63.3 ± 3.062.8 ± 1.862.5 ± 1.363.2 ± 2.5+0.1 ± 3.6
SFT (S)36.0 ± 1.035.5 ± 2.333.4 ± 2.233.7 ± 2.5+2.7 ± 1.9
Aligned (A)32.6 ± 1.232.3 ± 1.530.7 ± 1.229.8 ± 1.6+3.0 ± 1.2
SFT − Base−27.2 ± 2.9−27.3 ± 2.1−29.0 ± 1.9−29.6 ± 2.9+2.6 ± 2.5
Aligned − Base−30.7 ± 2.9−30.6 ± 3.1−31.8 ± 1.7−33.5 ± 2.5+2.8 ± 4.6
Lineage / stageD1D_{1}D2D_{2}D3D_{3}D4D_{4}Fitted drop
Donation-PD: RR2.421.41.1–
OLMo-3-7B
Base (B)2.2 ± 0.31.7 ± 0.51.4 ± 0.31.4 ± 0.3+0.8 ± 0.4
SFT (S)1.0 ± 0.20.4 ± 0.20.2 ± 0.10.1 ± 0.0+0.8 ± 0.2
Aligned (A)1.7 ± 0.11.2 ± 0.20.6 ± 0.10.4 ± 0.1+1.3 ± 0.1
SFT − Base−1.3 ± 0.4−1.3 ± 0.6−1.2 ± 0.3−1.3 ± 0.3+0.0 ± 0.5
Aligned − Base−0.5 ± 0.2−0.5 ± 0.5−0.8 ± 0.4−1.0 ± 0.4+0.5 ± 0.4
SmolLM2-1.7B
Base (B)99.7 ± 0.199.8 ± 0.199.7 ± 0.199.7 ± 0.3+0.1 ± 0.2
SFT (S)7.8 ± 3.49.7 ± 2.411.7 ± 4.712.1 ± 4.2−4.3 ± 4.9
Aligned (A)3.4 ± 0.85.6 ± 1.75.6 ± 1.83.8 ± 1.0−0.5 ± 2.0
SFT − Base−91.9 ± 3.4−90.1 ± 2.4−88.0 ± 4.7−87.6 ± 4.2−4.4 ± 4.8
Aligned − Base−96.3 ± 0.8−94.2 ± 1.6−94.1 ± 1.8−95.8 ± 1.2−0.5 ± 2.1
DeepSeekMath-7B
Base (B)95.9 ± 0.296.1 ± 0.695.9 ± 0.596.0 ± 0.7−0.0 ± 0.8
SFT (S)12.0 ± 2.25.8 ± 0.99.0 ± 1.610.2 ± 1.7+0.5 ± 1.1
Aligned (A)5.8 ± 0.71.8 ± 0.73.5 ± 1.03.7 ± 0.7+1.1 ± 0.6
SFT − Base−83.9 ± 2.2−90.3 ± 1.1−86.9 ± 1.9−85.8 ± 1.6+0.5 ± 1.4
Aligned − Base−90.2 ± 0.5−94.2 ± 1.0−92.4 ± 1.1−92.3 ± 1.2+1.1 ± 1.1
InternLM2-7B
Base (B)43.0 ± 1.841.7 ± 1.241.0 ± 0.642.7 ± 1.5+0.6 ± 1.8
SFT (S)37.7 ± 1.428.7 ± 1.129.9 ± 1.827.3 ± 1.1+8.3 ± 0.6
Aligned (A)41.1 ± 2.531.8 ± 2.132.7 ± 1.632.9 ± 1.8+6.8 ± 2.6
SFT − Base−5.3 ± 2.1−13.0 ± 1.3−11.1 ± 1.3−15.4 ± 2.6+7.7 ± 2.1
Aligned − Base−1.9 ± 3.0−9.9 ± 3.3−8.3 ± 1.4−9.9 ± 2.5+6.2 ± 3.1
Nemotron-Cascade-8B
Base (B)1.0 ± 0.30.4 ± 0.10.7 ± 0.30.7 ± 0.2+0.2 ± 0.1
SFT (S)5.5 ± 1.73.4 ± 0.43.8 ± 0.55.1 ± 1.8+0.3 ± 0.8
Aligned (A)3.3 ± 1.51.6 ± 0.33.3 ± 0.92.9 ± 0.5−0.3 ± 1.3
SFT − Base+4.5 ± 1.5+3.0 ± 0.4+3.1 ± 0.7+4.4 ± 1.6+0.1 ± 0.7
Aligned − Base+2.3 ± 1.4+1.2 ± 0.4+2.6 ± 0.9+2.2 ± 0.4−0.4 ± 1.4
Tülu-3-8B
Base (B)91.4 ± 4.681.1 ± 8.690.4 ± 6.492.6 ± 4.2−4.2 ± 4.5
SFT (S)15.8 ± 1.014.2 ± 0.613.7 ± 1.413.5 ± 0.7+2.2 ± 0.7
Aligned (A)7.3 ± 1.56.5 ± 1.34.8 ± 0.94.9 ± 1.5+2.6 ± 1.2
SFT − Base−75.5 ± 4.7−66.9 ± 8.8−76.7 ± 6.7−79.1 ± 4.3+6.3 ± 4.4
Aligned − Base−84.1 ± 3.4−74.5 ± 8.4−85.6 ± 6.0−87.7 ± 4.1+6.8 ± 4.7
Zephyr-7B
Base (B)47.1 ± 1.550.0 ± 2.346.9 ± 1.447.2 ± 2.1+1.0 ± 2.5
SFT (S)31.6 ± 1.826.1 ± 1.026.7 ± 2.125.3 ± 1.5+5.0 ± 1.3
Aligned (A)19.0 ± 3.413.7 ± 1.514.9 ± 0.912.3 ± 2.1+5.1 ± 3.2
SFT − Base−15.5 ± 1.6−23.8 ± 2.4−20.2 ± 3.2−21.9 ± 3.1+4.1 ± 3.2
Aligned − Base−28.1 ± 4.6−36.2 ± 2.4−32.0 ± 1.2−35.0 ± 2.0+4.2 ± 3.8

Table 13. Donation-PD: HL40 payoff responses for each lineage. Entries are mean ± SD across five runs, averaging secondary parameters within each run. Cooperation rates are percentages; stage differences and fitted declines are in pp. Stage differences match runs by seed ID. The final column fits all four parameter values within each run before taking the mean and SD.

Lineage / stageD1D_{1}D2D_{2}D3D_{3}D4D_{4}Fitted drop
Volunteer: ρ\rho0.1250.3750.6250.875–
OLMo-3-7B
Base (B)83.4±3.683.4 \pm3.659.5±15.259.5 \pm15.256.2±13.856.2 \pm13.830.3±3.730.3 \pm3.7+48.8±7.4+48.8 \pm7.4
SFT (S)41.3±2.241.3 \pm2.235.8±0.735.8 \pm0.729.4±0.829.4 \pm0.827.0±1.627.0 \pm1.6+14.8±2.0+14.8 \pm2.0
Aligned (A)45.2±2.445.2 \pm2.439.3±2.539.3 \pm2.534.6±3.634.6 \pm3.629.8±2.229.8 \pm2.2+15.2±2.4+15.2 \pm2.4
SFT − Base−42.1±4.3-42.1 \pm4.3−23.7±15.3-23.7 \pm15.3−26.8±14.1-26.8 \pm14.1−3.2±4.2-3.2 \pm4.2−34.0±5.7-34.0 \pm5.7
Aligned − Base−38.3±2.9-38.3 \pm2.9−20.2±14.4-20.2 \pm14.4−21.6±10.9-21.6 \pm10.9−0.5±4.0-0.5 \pm4.0−33.6±5.8-33.6 \pm5.8
SmolLM2-1.7B
Base (B)99.6±0.299.6 \pm0.299.6±0.199.6 \pm0.199.6±0.199.6 \pm0.199.5±0.199.5 \pm0.1+0.0±0.2+0.0 \pm0.2
SFT (S)38.8±6.438.8 \pm6.437.9±4.237.9 \pm4.236.2±6.836.2 \pm6.837.3±3.337.3 \pm3.3+1.8±4.5+1.8 \pm4.5
Aligned (A)32.7±4.732.7 \pm4.728.5±2.928.5 \pm2.929.3±8.029.3 \pm8.030.5±6.830.5 \pm6.8+1.8±10.7+1.8 \pm10.7
SFT − Base−60.8±6.3-60.8 \pm6.3−61.7±4.3-61.7 \pm4.3−63.5±6.8-63.5 \pm6.8−62.2±3.2-62.2 \pm3.2+1.8±4.5+1.8 \pm4.5
Aligned − Base−66.9±4.6-66.9 \pm4.6−71.1±2.9-71.1 \pm2.9−70.3±8.0-70.3 \pm8.0−69.1±6.8-69.1 \pm6.8+1.7±10.6+1.7 \pm10.6
DeepSeekMath-7B
Base (B)97.2±0.497.2 \pm0.497.3±0.297.3 \pm0.296.9±0.496.9 \pm0.496.2±0.296.2 \pm0.2+1.0±0.4+1.0 \pm0.4
SFT (S)45.9±2.545.9 \pm2.543.8±3.443.8 \pm3.438.4±2.138.4 \pm2.135.8±1.635.8 \pm1.6+10.7±1.7+10.7 \pm1.7
Aligned (A)16.5±1.016.5 \pm1.014.9±0.914.9 \pm0.913.7±0.413.7 \pm0.412.0±1.012.0 \pm1.0+4.4±1.1+4.4 \pm1.1
SFT − Base−51.3±2.5-51.3 \pm2.5−53.5±3.4-53.5 \pm3.4−58.5±1.9-58.5 \pm1.9−60.4±1.8-60.4 \pm1.8+9.7±1.8+9.7 \pm1.8
Aligned − Base−80.7±0.9-80.7 \pm0.9−82.4±1.0-82.4 \pm1.0−83.1±0.8-83.1 \pm0.8−84.2±1.0-84.2 \pm1.0+3.4±0.7+3.4 \pm0.7
InternLM2-7B
Base (B)35.5±0.835.5 \pm0.836.0±1.336.0 \pm1.336.0±2.236.0 \pm2.236.1±1.436.1 \pm1.4−0.5±1.7-0.5 \pm1.7
SFT (S)45.1±0.945.1 \pm0.943.5±0.943.5 \pm0.941.1±1.441.1 \pm1.438.6±1.638.6 \pm1.6+6.6±2.3+6.6 \pm2.3
Aligned (A)53.8±2.253.8 \pm2.250.0±1.250.0 \pm1.245.7±1.045.7 \pm1.045.2±1.645.2 \pm1.6+9.0±2.3+9.0 \pm2.3
SFT − Base+9.6±0.9+9.6 \pm0.9+7.5±1.8+7.5 \pm1.8+5.2±3.4+5.2 \pm3.4+2.5±1.9+2.5 \pm1.9+7.1±2.4+7.1 \pm2.4
Aligned − Base+18.3±2.7+18.3 \pm2.7+13.9±1.5+13.9 \pm1.5+9.7±1.8+9.7 \pm1.8+9.1±2.9+9.1 \pm2.9+9.5±3.7+9.5 \pm3.7
Nemotron-Cascade-8B
Base (B)98.3±1.798.3 \pm1.794.5±4.194.5 \pm4.189.1±8.089.1 \pm8.082.3±8.682.3 \pm8.6+16.0±8.5+16.0 \pm8.5
SFT (S)86.5±3.686.5 \pm3.681.4±4.881.4 \pm4.868.1±2.368.1 \pm2.364.3±7.264.3 \pm7.2+24.0±7.4+24.0 \pm7.4
Aligned (A)72.2±6.672.2 \pm6.669.2±5.769.2 \pm5.748.0±2.948.0 \pm2.942.8±1.742.8 \pm1.7+32.9±6.9+32.9 \pm6.9
SFT − Base−11.7±3.5-11.7 \pm3.5−13.1±7.7-13.1 \pm7.7−21.0±8.3-21.0 \pm8.3−18.0±7.3-18.0 \pm7.3+8.0±7.8+8.0 \pm7.8
Aligned − Base−26.0±7.9-26.0 \pm7.9−25.3±6.3-25.3 \pm6.3−41.0±7.4-41.0 \pm7.4−39.5±7.3-39.5 \pm7.3+16.9±12.9+16.9 \pm12.9
Tülu-3-8B
Base (B)98.6±0.498.6 \pm0.498.0±0.598.0 \pm0.597.5±0.197.5 \pm0.197.1±0.597.1 \pm0.5+1.5±0.8+1.5 \pm0.8
SFT (S)45.9±1.745.9 \pm1.744.1±1.244.1 \pm1.240.8±1.540.8 \pm1.537.6±0.837.6 \pm0.8+8.4±1.7+8.4 \pm1.7
Aligned (A)37.9±1.537.9 \pm1.535.0±1.535.0 \pm1.533.6±2.333.6 \pm2.330.3±1.130.3 \pm1.1+7.3±1.5+7.3 \pm1.5
SFT − Base−52.7±1.9-52.7 \pm1.9−54.0±1.3-54.0 \pm1.3−56.8±1.5-56.8 \pm1.5−59.5±0.6-59.5 \pm0.6+6.9±1.7+6.9 \pm1.7
Aligned − Base−60.7±1.6-60.7 \pm1.6−63.0±1.9-63.0 \pm1.9−63.9±2.4-63.9 \pm2.4−66.8±1.5-66.8 \pm1.5+5.8±1.9+5.8 \pm1.9
Zephyr-7B
Base (B)51.4±1.851.4 \pm1.851.0±0.951.0 \pm0.950.5±1.750.5 \pm1.749.8±1.649.8 \pm1.6+1.5±1.2+1.5 \pm1.2
SFT (S)60.2±0.960.2 \pm0.958.8±2.158.8 \pm2.155.7±1.455.7 \pm1.455.9±1.555.9 \pm1.5+4.9±2.1+4.9 \pm2.1
Aligned (A)63.9±5.063.9 \pm5.060.9±3.160.9 \pm3.150.3±1.950.3 \pm1.946.1±2.346.1 \pm2.3+19.2±6.0+19.2 \pm6.0
SFT − Base+8.9±2.0+8.9 \pm2.0+7.8±2.2+7.8 \pm2.2+5.1±2.6+5.1 \pm2.6+6.0±2.3+6.0 \pm2.3+3.3±2.3+3.3 \pm2.3
Aligned − Base+12.6±5.8+12.6 \pm5.8+9.9±3.4+9.9 \pm3.4−0.2±2.1-0.2 \pm2.1−3.7±1.4-3.7 \pm1.4+17.7±5.8+17.7 \pm5.8

Table 14. Volunteer: HL40 payoff responses for each lineage. Entries are mean ±\pm SD across five runs, averaging secondary parameters within each run. Cooperation rates are percentages; stage differences and fitted declines are in pp. Stage differences match runs by seed ID. The final column fits all four parameter values within each run before taking the mean and SD.

Lineage / stageD1D_1D2D_2D3D_3D4D_4Fitted drop
Assurance: λ\lambda0.40.20.10.05–
OLMo-3-7B
Base (B)98.3±0.698.3 \pm0.689.8±19.589.8 \pm19.598.0±0.298.0 \pm0.298.0±0.598.0 \pm0.5−0.2±1.0-0.2 \pm1.0
SFT (S)89.9±18.289.9 \pm18.290.3±13.090.3 \pm13.045.6±16.545.6 \pm16.572.4±20.372.4 \pm20.3+30.6±16.1+30.6 \pm16.1
Aligned (A)81.8±19.681.8 \pm19.671.3±22.771.3 \pm22.757.0±4.457.0 \pm4.463.8±18.863.8 \pm18.8+22.0±30.0+22.0 \pm30.0
SFT − Base−8.5±18.2-8.5 \pm18.2+0.5±6.9+0.5 \pm6.9−52.5±16.5-52.5 \pm16.5−25.6±20.4-25.6 \pm20.4+30.8±16.4+30.8 \pm16.4
Aligned − Base−16.6±20.1-16.6 \pm20.1−18.5±37.9-18.5 \pm37.9−41.0±4.5-41.0 \pm4.5−34.2±18.6-34.2 \pm18.6+22.2±30.7+22.2 \pm30.7
SmolLM2-1.7B
Base (B)99.1±0.399.1 \pm0.399.1±0.499.1 \pm0.499.3±0.499.3 \pm0.499.5±0.499.5 \pm0.4−0.3±0.2-0.3 \pm0.2
SFT (S)96.8±3.696.8 \pm3.693.8±6.593.8 \pm6.595.2±5.095.2 \pm5.096.2±3.696.2 \pm3.6+0.8±3.2+0.8 \pm3.2
Aligned (A)97.6±4.097.6 \pm4.095.8±4.695.8 \pm4.696.7±3.796.7 \pm3.798.0±3.298.0 \pm3.2−0.0±0.9-0.0 \pm0.9
SFT − Base−2.4±3.5-2.4 \pm3.5−5.3±6.1-5.3 \pm6.1−4.1±4.7-4.1 \pm4.7−3.2±3.2-3.2 \pm3.2+1.1±3.3+1.1 \pm3.3
Aligned − Base−1.6±3.8-1.6 \pm3.8−3.3±4.1-3.3 \pm4.1−2.6±3.4-2.6 \pm3.4−1.4±2.9-1.4 \pm2.9+0.3±1.0+0.3 \pm1.0
DeepSeekMath-7B
Base (B)99.2±0.399.2 \pm0.399.2±0.299.2 \pm0.299.0±0.399.0 \pm0.399.2±0.299.2 \pm0.2+0.1±0.3+0.1 \pm0.3
SFT (S)87.3±2.387.3 \pm2.383.1±3.283.1 \pm3.277.0±3.677.0 \pm3.682.1±3.182.1 \pm3.1+7.7±3.1+7.7 \pm3.1
Aligned (A)87.7±8.287.7 \pm8.278.0±9.578.0 \pm9.561.0±11.361.0 \pm11.374.7±18.174.7 \pm18.1+19.4±5.1+19.4 \pm5.1
SFT − Base−11.9±2.3-11.9 \pm2.3−16.1±3.3-16.1 \pm3.3−22.1±3.6-22.1 \pm3.6−17.1±3.2-17.1 \pm3.2+7.6±3.2+7.6 \pm3.2
Aligned − Base−11.6±8.2-11.6 \pm8.2−21.1±9.6-21.1 \pm9.6−38.0±11.4-38.0 \pm11.4−24.5±18.0-24.5 \pm18.0+19.3±4.8+19.3 \pm4.8
InternLM2-7B
Base (B)40.6±2.140.6 \pm2.141.6±1.641.6 \pm1.640.2±1.940.2 \pm1.942.0±1.542.0 \pm1.5−0.7±3.4-0.7 \pm3.4
SFT (S)71.1±2.871.1 \pm2.869.1±2.269.1 \pm2.264.3±2.964.3 \pm2.964.3±1.164.3 \pm1.1+7.3±2.5+7.3 \pm2.5
Aligned (A)84.5±2.484.5 \pm2.482.2±1.882.2 \pm1.878.0±4.878.0 \pm4.874.8±3.374.8 \pm3.3+9.1±4.4+9.1 \pm4.4
SFT − Base+30.5±2.9+30.5 \pm2.9+27.5±3.3+27.5 \pm3.3+24.0±3.0+24.0 \pm3.0+22.3±2.2+22.3 \pm2.2+8.1±4.0+8.1 \pm4.0
Aligned − Base+43.9±4.4+43.9 \pm4.4+40.7±0.9+40.7 \pm0.9+37.8±6.2+37.8 \pm6.2+32.8±3.8+32.8 \pm3.8+9.8±7.5+9.8 \pm7.5
Nemotron-Cascade-8B
Base (B)99.8±0.199.8 \pm0.199.7±0.299.7 \pm0.299.8±0.099.8 \pm0.099.8±0.199.8 \pm0.1−0.1±0.2-0.1 \pm0.2
SFT (S)59.3±18.359.3 \pm18.352.1±1.452.1 \pm1.467.7±21.767.7 \pm21.759.3±35.359.3 \pm35.3−4.0±21.9-4.0 \pm21.9
Aligned (A)70.8±25.370.8 \pm25.371.8±24.471.8 \pm24.453.5±2.353.5 \pm2.380.7±23.480.7 \pm23.4+0.9±20.2+0.9 \pm20.2
SFT − Base−40.5±18.2-40.5 \pm18.2−47.6±1.4-47.6 \pm1.4−32.2±21.7-32.2 \pm21.7−40.5±35.3-40.5 \pm35.3−3.9±21.8-3.9 \pm21.8
Aligned − Base−28.9±25.3-28.9 \pm25.3−27.9±24.4-27.9 \pm24.4−46.3±2.3-46.3 \pm2.3−19.2±23.5-19.2 \pm23.5+0.9±20.1+0.9 \pm20.1
Tülu-3-8B
Base (B)99.0±0.199.0 \pm0.198.9±0.398.9 \pm0.399.1±0.299.1 \pm0.299.1±0.299.1 \pm0.2−0.0±0.2-0.0 \pm0.2
SFT (S)81.2±6.281.2 \pm6.283.9±2.083.9 \pm2.077.3±3.977.3 \pm3.977.0±5.877.0 \pm5.8+4.6±5.4+4.6 \pm5.4
Aligned (A)88.0±15.988.0 \pm15.980.7±19.580.7 \pm19.567.5±15.467.5 \pm15.460.0±2.660.0 \pm2.6+27.1±15.1+27.1 \pm15.1
SFT − Base−17.9±6.3-17.9 \pm6.3−15.0±2.0-15.0 \pm2.0−21.8±3.8-21.8 \pm3.8−22.1±5.8-22.1 \pm5.8+4.7±5.3+4.7 \pm5.3
Aligned − Base−11.0±15.9-11.0 \pm15.9−18.2±19.6-18.2 \pm19.6−31.6±15.4-31.6 \pm15.4−39.1±2.4-39.1 \pm2.4+27.1±15.1+27.1 \pm15.1
Zephyr-7B
Base (B)74.5±1.674.5 \pm1.671.5±2.871.5 \pm2.872.3±1.472.3 \pm1.473.6±2.373.6 \pm2.3+1.4±3.0+1.4 \pm3.0
SFT (S)80.0±2.580.0 \pm2.576.0±1.476.0 \pm1.473.2±1.673.2 \pm1.673.8±2.873.8 \pm2.8+6.8±3.2+6.8 \pm3.2
Aligned (A)86.4±3.086.4 \pm3.080.2±4.880.2 \pm4.878.9±3.178.9 \pm3.182.7±5.382.7 \pm5.3+5.3±1.2+5.3 \pm1.2
SFT − Base+5.5±2.4+5.5 \pm2.4+4.6±3.7+4.6 \pm3.7+0.9±3.0+0.9 \pm3.0+0.2±3.1+0.2 \pm3.1+5.5±3.7+5.5 \pm3.7
Aligned − Base+11.8±4.4+11.8 \pm4.4+8.8±5.0+8.8 \pm5.0+6.6±3.3+6.6 \pm3.3+9.1±3.3+9.1 \pm3.3+3.9±3.1+3.9 \pm3.1

Table 15. Assurance: HL40 payoff responses for each lineage. Entries are mean ±\pm SD across five runs, averaging secondary parameters within each run. Cooperation rates are percentages; stage differences and fitted declines are in pp. Stage differences match runs by seed ID. The final column fits all four parameter values within each run before taking the mean and SD.

Table 16: TD: HL40 payoff responses for each lineage. Entries are mean ±\pm SD across five runs, averaging secondary parameters within each run. Normalized claims are multiplied by 100; stage differences and fitted declines are in pp. Stage differences match runs by seed ID. The final column fits all four parameter values within each run before taking the mean and SD.

Lineage / stageD1D_1D2D_2D3D_3D4D_4Fitted drop
TD: τ\tau0.0250.050.10.2–
OLMo-3-7B
Base (B)90.1 ±\pm 5.290.9 ±\pm 2.988.5 ±\pm 3.176.4 ±\pm 10.5+14.6 ±\pm 10.0
SFT (S)98.1 ±\pm 1.496.9 ±\pm 5.398.4 ±\pm 2.198.6 ±\pm 1.0−-1.0 ±\pm 2.6
Aligned (A)83.1 ±\pm 4.183.6 ±\pm 9.684.1 ±\pm 10.283.7 ±\pm 8.9−-0.4 ±\pm 7.4
SFT −- Base+8.1 ±\pm 6.2+6.0 ±\pm 6.6+9.8 ±\pm 4.1+22.2 ±\pm 10.2−-15.6 ±\pm 11.0
Aligned −- Base−-7.0 ±\pm 8.3−-7.3 ±\pm 11.3−-4.5 ±\pm 12.8+7.2 ±\pm 19.0−-15.0 ±\pm 15.4
SmolLM2-1.7B
Base (B)3.6 ±\pm 0.44.1 ±\pm 1.24.4 ±\pm 0.43.9 ±\pm 0.4−-0.1 ±\pm 0.6
SFT (S)62.4 ±\pm 4.662.6 ±\pm 5.962.6 ±\pm 4.760.5 ±\pm 9.2+2.0 ±\pm 9.2
Aligned (A)68.3 ±\pm 6.167.6 ±\pm 5.665.3 ±\pm 4.065.9 ±\pm 1.5+2.4 ±\pm 2.5
SFT −- Base+58.8 ±\pm 4.4+58.4 ±\pm 6.4+58.2 ±\pm 5.0+56.5 ±\pm 9.0+2.2 ±\pm 9.2
Aligned −- Base+64.6 ±\pm 6.4+63.4 ±\pm 6.3+60.9 ±\pm 4.3+61.9 ±\pm 1.7+2.5 ±\pm 3.0
DeepSeekMath-7B
Base (B)30.5 ±\pm 2.132.1 ±\pm 3.833.2 ±\pm 3.332.2 ±\pm 2.7−-1.2 ±\pm 2.8
SFT (S)69.0 ±\pm 3.968.3 ±\pm 1.369.8 ±\pm 3.668.2 ±\pm 1.2+0.4 ±\pm 3.7
Aligned (A)68.5 ±\pm 2.471.1 ±\pm 5.170.0 ±\pm 4.067.7 ±\pm 5.9+1.9 ±\pm 5.0
SFT −- Base+38.5 ±\pm 3.2+36.2 ±\pm 4.1+36.7 ±\pm 4.3+36.0 ±\pm 3.7+1.7 ±\pm 5.1
Aligned −- Base+38.0 ±\pm 2.1+39.0 ±\pm 6.6+36.8 ±\pm 4.0+35.5 ±\pm 5.0+3.1 ±\pm 4.6
InternLM2-7B
Base (B)36.2 ±\pm 2.735.6 ±\pm 3.136.8 ±\pm 2.237.4 ±\pm 1.2−-1.5 ±\pm 2.1
SFT (S)63.6 ±\pm 0.963.0 ±\pm 1.463.2 ±\pm 0.762.9 ±\pm 1.0+0.5 ±\pm 0.9
Aligned (A)63.1 ±\pm 0.962.1 ±\pm 0.962.2 ±\pm 1.861.0 ±\pm 0.8+1.8 ±\pm 1.2
SFT −- Base+27.5 ±\pm 3.3+27.4 ±\pm 2.1+26.5 ±\pm 2.5+25.6 ±\pm 1.4+2.0 ±\pm 2.5
Aligned −- Base+27.0 ±\pm 3.4+26.6 ±\pm 3.3+25.5 ±\pm 3.8+23.7 ±\pm 1.3+3.3 ±\pm 3.2
Nemotron-Cascade-8B
Base (B)58.5 ±\pm 7.357.0 ±\pm 5.260.7 ±\pm 3.256.6 ±\pm 13.0+1.3 ±\pm 13.9
SFT (S)23.0 ±\pm 4.625.2 ±\pm 5.625.0 ±\pm 4.725.5 ±\pm 2.5−-1.7 ±\pm 2.6
Aligned (A)25.4 ±\pm 3.826.4 ±\pm 3.327.1 ±\pm 6.523.6 ±\pm 2.8+2.1 ±\pm 3.1
SFT −- Base−-35.5 ±\pm 6.9−-31.8 ±\pm 10.3−-35.6 ±\pm 3.4−-31.1 ±\pm 14.0−-3.0 ±\pm 13.0
Aligned −- Base−-33.2 ±\pm 10.3−-30.6 ±\pm 3.6−-33.6 ±\pm 7.5−-32.9 ±\pm 14.8+0.8 ±\pm 16.5
Tülu-3-8B
Base (B)26.6 ±\pm 5.927.3 ±\pm 3.528.7 ±\pm 2.627.2 ±\pm 3.2−-0.5 ±\pm 3.4
SFT (S)83.3 ±\pm 3.681.4 ±\pm 2.879.9 ±\pm 8.186.2 ±\pm 4.6−-3.6 ±\pm 5.6
Aligned (A)75.9 ±\pm 7.271.1 ±\pm 5.669.6 ±\pm 11.768.4 ±\pm 7.9+6.0 ±\pm 13.3
SFT −- Base+56.7 ±\pm 4.9+54.1 ±\pm 5.1+51.2 ±\pm 9.4+58.9 ±\pm 4.2−-3.1 ±\pm 3.1
Aligned −- Base+49.3 ±\pm 3.5+43.8 ±\pm 5.5+40.9 ±\pm 11.8+41.2 ±\pm 9.2+6.6 ±\pm 11.2
Zephyr-7B
Base (B)31.6 ±\pm 3.131.4 ±\pm 1.732.4 ±\pm 1.331.9 ±\pm 2.8−-0.4 ±\pm 2.7
SFT (S)71.1 ±\pm 1.771.1 ±\pm 1.470.1 ±\pm 1.870.0 ±\pm 2.4+1.2 ±\pm 1.8
Aligned (A)53.9 ±\pm 1.453.1 ±\pm 0.851.8 ±\pm 2.552.3 ±\pm 2.2+1.4 ±\pm 2.3
SFT −- Base+39.5 ±\pm 4.2+39.6 ±\pm 2.7+37.7 ±\pm 1.9+38.1 ±\pm 3.3+1.6 ±\pm 3.5
Aligned −- Base+22.3 ±\pm 4.2+21.7 ±\pm 2.2+19.4 ±\pm 2.6+20.5 ±\pm 2.6+1.8 ±\pm 2.6
Table 16.

Table 27.

A Seven-Stage Nemotron-Cascade Case Study

Checkpoints, payoff settings, and measurements

This case study follows one released lineage through seven checkpoints: Qwen3-8B-Base, SFT, RLHF, instruction-following RL (IFRL), mathematical RL (MathRL), code RL (CodeRL), and software-engineering RL (SWE-RL). The canonical Aligned checkpoint for this lineage is RLHF. SWE-RL is the endpoint of this longer case study. The primary 21-checkpoint panel’s three-stage comparison is complemented here by four subsequent checkpoints.

Base uses Qwen/Qwen3-8B-Base. SFT through CodeRL use the correspondingly named subfolders of nvidia/Nemotron-Cascade-8B-Intermediate-ckpts; SWE-RL uses nvidia/Nemotron-Cascade-8B. Table 17 explains the training added at each checkpoint, following the publisher’s model cards and training description.3 These checkpoints trace successive training stages within the same model lineage. All checkpoints use the same game prompt template and action-label interface, bfloat16 inference, temperature 0.7, top-p 0.8, and top-k 20.

CheckpointTraining completed at this step
BaseQwen3-8B pretrained model, before Nemotron-Cascade post-training.
SFTMulti-stage supervised fine-tuning on example responses spanning general dialogue, reasoning, coding, and other domains.
RLHFReinforcement learning using a learned human-preference reward. This is our main panel’s Aligned checkpoint.
IFRLInstruction-following reinforcement learning, rewarding adherence to explicit response constraints; applied in non-thinking mode for this unified model.
MathRLReinforcement learning on mathematical problems, using answer-correctness rewards.
CodeRLReinforcement learning on programming problems, using code-execution tests as rewards.
SWE-RLSoftware-engineering reinforcement learning for code repair, using an execution-free reward that compares proposed repairs with reference patches.

Table 17. What changes across the seven Nemotron-Cascade checkpoints? Rows follow the released training sequence. Base, SFT, and RLHF are the three stages used for this lineage in the main panel; the remaining rows extend that sequence.

All seven checkpoints are evaluated on the same original 82 configurations. Relative to the main panel in Appendix § 13, PGG includes MPCR 0.7, giving {0.4,0.5,0.6,0.7,0.8}\{0.4, 0.5, 0.6, 0.7, 0.8\}, and CPR uses θ∈{0.2,0.5,0.8}\theta\in\{0.2, 0.5, 0.8\} instead of {0.2,0.4,0.6,0.8}\{0.2, 0.4, 0.6, 0.8\}; all other parameters are unchanged. Thus PGG has ten complete configurations and CPR twelve, compared with eight and sixteen in the main panel. Each checkpoint has 20 one-shot draw IDs per configuration and five trajectory seeds at each of HL2 and HL40; repeated rates average all 200 logged rounds and all seats within configuration. All comparisons in this appendix use this common 82-configuration grid.

We compare cooperation rates and payoff responses on the same five games: PGG, CPR, Donation-PD, Volunteer, and Assurance. A separate history table adds Trust to match the six-binary-game average in the main text. For each checkpoint and run, we first average cooperation over configurations within a game, then average games equally. For payoff responses, we fit the main parameter while holding secondary parameters fixed, then average these fits and games (Equation (2)). PGG therefore fits five points here and CPR three. Assurance uses the λ\lambda span 0.35; fitting successful payoff h=3(1+λ)h = 3(1 + \lambda) over span 1.05 gives the identical fitted drop.

The result tables report mean ± sample SD across 20 one-shot draws or five repeated-game runs within a checkpoint. For history and consecutive-stage comparisons, we subtract results with the same seed ID before computing the SD. This SD describes variation across runs of this one lineage. OS denotes one-shot decisions. As in Equation (1), HsH_s is HL40 minus HL2, and Ks/BK_{s/B} is that history change minus Base’s; both are reported in pp. Negative Ks/BK_{s/B} means a larger cooperation decline than Base’s. Rows follow the released training sequence.

Higher cooperation with a smaller fitted payoff decline

Figure 4 and Table 18–Table 19 compare the cooperation rate and fitted payoff drop across these same five games. From RLHF to IFRL, mean cooperation rises by 11.22 pp in one-shot decisions, 6.18 pp at HL2, and 14.53 pp at HL40. At HL40, the fitted payoff drop falls from 12.63 to 0.75 pp: cooperation becomes higher but changes less across the tested payoff range. The same combination occurs in one-shot and HL2 evaluations. This transition combines higher cooperation with a smaller fitted response to payoff changes. The figure displays the training path; the tables give means, SDs, and paired history changes.

StageOSHL2HL40HHKS/BK_{S/B}
Base55.39±4.7655.39 \pm4.7653.80±0.4853.80 \pm0.4846.01±2.9546.01 \pm2.95−7.79±3.15-7.79 \pm3.15+0.00±0.00+0.00 \pm0.00
SFT39.89±3.3639.89 \pm3.3655.23±0.1655.23 \pm0.1645.47±1.5745.47 \pm1.57−9.76±1.66-9.76 \pm1.66−1.97±4.53-1.97 \pm4.53
RLHF55.88±4.3155.88 \pm4.3158.30±0.4158.30 \pm0.4138.59±3.2738.59 \pm3.27−19.71±3.45-19.71 \pm3.45−11.92±5.32-11.92 \pm5.32
IFRL67.10±3.2167.10 \pm3.2164.48±1.1864.48 \pm1.1853.12±2.8753.12 \pm2.87−11.36±2.66-11.36 \pm2.66−3.57±4.61-3.57 \pm4.61
MathRL65.79±3.0965.79 \pm3.0965.31±0.3565.31 \pm0.3554.22±2.6254.22 \pm2.62−11.09±2.93-11.09 \pm2.93−3.30±1.97-3.30 \pm1.97
CodeRL61.35±2.6261.35 \pm2.6260.59±1.9360.59 \pm1.9353.36±4.7453.36 \pm4.74−7.23±5.56-7.23 \pm5.56+0.56±6.12+0.56 \pm6.12
SWE-RL62.38±2.9362.38 \pm2.9362.07±1.3962.07 \pm1.3952.01±3.2152.01 \pm3.21−10.05±4.15-10.05 \pm4.15−2.27±2.96-2.27 \pm2.96

Table 18. Same five symmetric binary games for both level and response: PGG, CPR, Donation-PD, Volunteer, Assurance. Levels are percentages; history contrasts are pp. Mean ±\pm SD across replicates within each checkpoint. Repeated contrasts pair five nominal seeds.

StageOSHL2HL40
Base−0.39±7.21-0.39 \pm7.21+3.18±1.01+3.18 \pm1.01+5.26±4.07+5.26 \pm4.07
SFT+4.05±6.67+4.05 \pm6.67+2.75±1.04+2.75 \pm1.04+12.11±3.96+12.11 \pm3.96
RLHF+8.63±8.25+8.63 \pm8.25+4.15±1.50+4.15 \pm1.50+12.63±6.70+12.63 \pm6.70
IFRL+1.76±5.90+1.76 \pm5.90+2.64±2.19+2.64 \pm2.19+0.75±6.00+0.75 \pm6.00
MathRL+4.21±8.78+4.21 \pm8.78+4.19±4.10+4.19 \pm4.10+6.79±9.61+6.79 \pm9.61
CodeRL+3.37±8.79+3.37 \pm8.79+6.13±1.68+6.13 \pm1.68+12.61±3.66+12.61 \pm3.66
SWE-RL+2.21±7.16+2.21 \pm7.16+5.25±3.14+5.25 \pm3.14+12.81±4.74+12.81 \pm4.74

Table 19. Same five symmetric binary games for both level and response: PGG, CPR, Donation-PD, Volunteer, Assurance. All-point fitted responses in pp; mean ±\pm replicate SD (20 draws for OS, five seeds for HL2/HL40).

History responses across game sets

On the six-binary-game panel, every post-training checkpoint has a more negative mean history response than Base (Table 20). The Base and SWE-RL HL40 levels are 54.46±2.46%54.46 \pm2.46\% and 54.27±2.36%54.27 \pm2.36\%, while their history changes are −6.43±2.60-6.43 \pm2.60 and −10.43±3.04-10.43 \pm3.04 pp. Similar endpoint levels thus coexist with different responses to history. Intermediate HL40 levels range from RLHF’s 38.73%38.73\% to MathRL’s 57.50%57.50\%.

StageOSHL2HL40HHKs/BK_{s/B}
Base57.82±4.9157.82 \pm4.9160.89±0.4560.89 \pm0.4554.46±2.4654.46 \pm2.46−6.43±2.60-6.43 \pm2.60+0.00±0.00+0.00 \pm0.00
SFT38.96±3.6038.96 \pm3.6056.44±0.2856.44 \pm0.2845.84±1.5145.84 \pm1.51−10.60±1.56-10.60 \pm1.56−4.17±3.84-4.17 \pm3.84
RLHF50.59±3.8350.59 \pm3.8358.58±0.5058.58 \pm0.5038.73±2.8138.73 \pm2.81−19.85±3.02-19.85 \pm3.02−13.42±4.61-13.42 \pm4.61
IFRL66.08±3.4266.08 \pm3.4267.63±0.7267.63 \pm0.7255.92±2.0655.92 \pm2.06−11.70±1.90-11.70 \pm1.90−5.27±3.50-5.27 \pm3.50
MathRL63.39±3.0463.39 \pm3.0469.10±0.4269.10 \pm0.4257.50±2.1957.50 \pm2.19−11.60±2.26-11.60 \pm2.26−5.17±2.05-5.17 \pm2.05
CodeRL59.92±2.7959.92 \pm2.7963.06±1.0563.06 \pm1.0555.53±4.0155.53 \pm4.01−7.53±4.58-7.53 \pm4.58−1.09±5.12-1.09 \pm5.12
SWE-RL60.36±2.8860.36 \pm2.8864.70±1.0264.70 \pm1.0254.27±2.3654.27 \pm2.36−10.43±3.04-10.43 \pm3.04−4.00±1.95-4.00 \pm1.95

Table 20. Six-binary-game cooperation and history panel, including Trust and excluding TD. Levels are percentages; history contrasts are pp. Mean ±\pm SD across replicates within each checkpoint. Repeated contrasts pair five nominal seeds.

The five-game panel supplies the like-for-like level/response comparison above. In that panel, five of six post-training history contrasts are more negative than Base; CodeRL has Ks/B=+0.56±6.12K_{s/B} = +0.56 \pm6.12 pp. The six-binary-game panel, which includes Trust, has negative Base-relative history contrasts at all six post-training checkpoints. Both summaries specify their game sets.

References

  1. [1]Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models. Nature Human Behaviour, 9(7):1380–1390, 2025.DOI
  2. [2]Lisa R Anderson, Robert L Hicks, and Andrew Turscak. Cooperation in public goods experiments with random and finite stopping rules. Journal of the Economic Science Association, 10(2):367–392, 2024.DOI
  3. [3]James Andreoni. Warm-glow versus cold-prickle: the effects of positive and negative framing on cooperation in experiments. The Quarterly Journal of Economics, 110(1):1–21, 1995.DOI
  4. [4]Malte Baader, Simon Gächter, Kyeongtae Lee, and Martin Sefton. Social preferences and the variability of conditional cooperation. Economic Theory, 81(1):341–366, 2026.
  5. [5]Joyce Berg, John Dickhaut, and Kevin McCabe. Trust, reciprocity, and social history. Games and economic behavior, 10(1):122–142, 1995.DOI
  6. [6]Marcel Binz, Elif Akata, Abdullah Almaatouq, Mohammed Alsobay, Oleksii Ariasov, Franziska Brändle, David Broska, Jason W Burton, Nuno Busch, Frederick Callaway, et al. Post-training makes large language models less human-like. arXiv preprint arXiv:2605.07632, 2026.arxiv.org/abs/2605.07632
  7. [7]C Monica Capra, Jacob K Goeree, Rosario Gomez, and Charles A Holt. Anomalous behavior in a traveler’s dilemma? American Economic Review, 89(3):678–690, 1999.DOI
  8. [8]Gary Charness and Matthew Rabin. Understanding social preferences with simple tests. The quarterly journal of economics, 117(3):817–869, 2002.DOI
  9. [9]Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161, 2025.arxiv.org/abs/2501.17161
  10. [10]Pedro Dal Bó and Guillaume R Fréchette. The evolution of cooperation in infinitely repeated games: Experimental evidence. American Economic Review, 101(1):411–429, 2011.
  11. [11]Urs Fischbacher and Simon Gächter. Social preferences, beliefs, and the dynamics of free riding in public goods experiments. American economic review, 100(1):541–556, 2010.DOI
  12. [12]Urs Fischbacher, Simon Gächter, and Ernst Fehr. Are people conditionally cooperative? evidence from a public goods experiment. Economics letters, 71(3):397–404, 2001.
  13. [13]Nicoló Fontana, Francesco Pierri, and Luca Maria Aiello. Nicer than humans: How do large language models behave in the prisoner’s dilemma? In Proceedings of the International AAAI Conference on Web and Social Media, volume 19, pp. 522–535, 2025.arxiv.org/abs/2406.13605
  14. [14]Drew Fudenberg and Eric Maskin. The folk theorem in repeated games with discounting or with incomplete information. Econometrica, 54(3):533–554, 1986.
  15. [15]Drew Fudenberg, David G Rand, and Anna Dreber. Slow to anger and fast to forgive: Cooperation in an uncertain world. American Economic Review, 102(2):720–749, 2012.
  16. [16]Jacob K Goeree, Charles A Holt, and Susan K Laury. Private costs and public benefits: unraveling the effects of altruism and noisy behavior. Journal of public Economics, 83(2):255–276, 2002.DOI
  17. [17]R Mark Isaac and James M Walker. Group size effects in public goods provision: The voluntary contributions mechanism. The Quarterly Journal of Economics, 103(1):179–199, 1988.DOI
  18. [18]Yuxuan Li and Hirokazu Shirado. Spontaneous giving and calculated greed in language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 5271–5286, 2025.arxiv.org/abs/2502.17720
  19. [19]Yunhao Liang, Yuan Qu, Jingyuan Yang, Shaochong Lin, and Zuo-Jun Max Shen. Everyone contributes! incentivizing strategic cooperation in multi-llm systems via sequential public goods games. arXiv preprint arXiv:2508.02076, 2025.arxiv.org/abs/2508.02076
  20. [20]Jiayuan Liu, Tianqin Li, Shiyi Du, Xin Luo, Haoxuan Zeng, Emanuel Tewolde, Tai Sing Lee, Tonghan Wang, Carl Kingsford, and Vincent Conitzer. The memory curse: How expanded recall erodes cooperative intent in llm agents. arXiv preprint arXiv:2605.08060, 2026.arxiv.org/abs/2605.08060
  21. [21]Volodymyr Lugovskyy, Daniela Puzzello, Andrea Sorensen, James Walker, and Arlington Williams. An experimental study of finitely and infinitely repeated linear public goods games. Games and Economic Behavior, 102:286–302, 2017.DOI
  22. [22]Shuangmei Ma, Boyu Zhang, Shinan Cao, Jun S Liu, and Wen-Xu Wang. Limited memory optimizes cooperation in social dilemma experiments. Royal Society open science, 8(8):210653, 2021.DOI
  23. [23]Qiaozhu Mei, Yutong Xie, Walter Yuan, and Matthew O Jackson. A turing test of whether ai chatbots are behaviorally similar to humans. Proceedings of the National Academy of Sciences, 121(9): e2313925121, 2024.DOI
  24. [24]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.arxiv.org/abs/2203.02155
  25. [25]Steve Phelps and Yvan I Russell. The machine psychology of cooperation: can gpt models operationalize prompts for altruism, cooperation, competitiveness, and selfishness in economic games? Journal of Physics: Complexity, 6(1):015018, 2025.DOI
  26. [26]Giorgio Piatti, Zhijing Jin, Max Kleiman-Weiner, Bernhard Schölkopf, Mnimnaya Sachan, and Rada Mihalcea. Cooperate or collapse: Emergence of sustainable cooperation in a society of llm agents. Advances in Neural Information Processing Systems, 37:111715–111759, 2024.arxiv.org/abs/2404.16698
  27. [27]David Guzman Piedrahita, Yongjin Yang, Mnimnaya Sachan, Giorgia Ramponi, Bernhard Schölkopf, and Zhijing Jin. Corrupted by reasoning: Reasoning language models become free-riders in public goods games. arXiv preprint arXiv:2506.23276, 2025.arxiv.org/abs/2506.23276
  28. [28]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023.arxiv.org/abs/2305.18290
  29. [29]Jesse Roberts, Kyle Moore, and Douglas Fisher. Do large language models learn human-like strategic preferences? In Proceedings of the 1st Workshop for Research on Agent Language Models (REALM 2025), pp. 97–108, 2025.arxiv.org/abs/2404.08710
  30. [30]Eilam Shapira, Moshe Tennenholtz, and Roi Reichart. Alignment makes language models normative, not descriptive. arXiv preprint arXiv:2603.17218, 2026.arxiv.org/abs/2603.17218
  31. [31]Natalie Struwe, Esther Blanco, and James M Walker. Increasing benefits in one-time public goods does not promote cooperation. Proceedings of the National Academy of Sciences, 121(41): e2410326121, 2024.
  32. [32]Emanuel Tewolde, Xiao Zhang, David Guzman Piedrahita, Vincent Conitzer, and Zhijing Jin. Cooperval: Benchmarking cooperation-sustaining mechanisms and llm agents in social dilemmas. arXiv preprint arXiv:2604.15267, 2026.arxiv.org/abs/2604.15267
  33. [33]Ruivi Wang, Haofei Yu, Wenxin Zhang, Zhengyang Qi, Maarten Sap, Yonatan Bisk, Graham Neubig, and Hao Zhu. Sotopia-π: Interactive learning of socially intelligent language agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12912–12940, 2024.
  34. [34]Haofei Yu, Zhengyang Qi, Yining Zhao, Kolby Nottingham, Keyang Xuan, Bodhisattwa Prasad Majumder, Hao Zhu, Paul Pu Liang, and Jiaxuan You. Sotopia-rl: Reward design for social intelligence. arXiv preprint arXiv:2508.03905, 2025.arxiv.org/abs/2508.03905
  35. [35]Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (eds.), Advances in Neural Information Processing Systems, volume 38, Main Conference, pp. 57654–57689. Curran Associates, Inc., 2025. doi: 10.52202/8713-1933. URL https://proceedings.neurips.cc/paper_files/paper/2025/file/537d5aa768c2d534016a4d06f87bc8fb-Paper-Conference.pdf.arxiv.org/abs/2504.13837
  36. [36]Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, et al. Sotopia: Interactive evaluation for social intelligence in language agents. In International Conference on Learning Representations, volume 2024, pp. 40975–41019, 2024.arxiv.org/abs/2310.11667

Paper details

vlnc:W_2m7mvjmb
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Generative AI

View statement
Looking for related papers…
Contents