betaFeedback

WorksComputer Science

Tracking Representations in LLM Reasoning Traces for Iterated Games

Abstract

Large language model (LLM) agents achieve state-of-the-art performance across domains through chain-of-thought (CoT) reasoning. Despite their ubiquitous use, little is understood about the relationship between the externalized trace and the model's actual computation. We address this by investigating how LLM agents make decisions in iterative games with incomplete information. Using a series of mechanistic interpretability techniques, we find that the model forms and iteratively revises an internal representation of the game throughout the CoT. Once the model prematurely commits to an incorrect belief, it rarely recovers. This failure is manifested in both the literal trace and the layer activations. Ultimately, as multi-agent systems are more prevalent, our work opens the door to a more focused diagnosis of their strategic reasoning errors. For illustrative purposes, we offer an interactive visualizer of our methodology on selected sample traces at https://interp-self.github.io/gametrace/

Introduction

Large language models (LLMs) of today ([24, 13, 3] (Yang et al. 2025a)) have shown remarkable abilities as autonomous agents capable of making complex, high-stakes decisions in a wide range of domains ([20, 27] (Yang et al. 2025b)). To achieve this state-of-the-art performance, models are trained to externalize their thinking and generate immediate reasoning steps as a long, serial chain of thought (CoT) before finalizing the answer ((Wei et al., 2022)). Despite widespread adoption, our understanding of the agents’ thought processes ([22, 29, 12, 11]) and the possible ways they can fail or be exploited ([1, 14]) is still limited.

Prior work has used game theory as a tool to investigate how an LLM agent thinks and behaves under uncertainty in repeated games in which it must compete or cooperate to achieve its objectives ([2, 15, 18] (Willis et al. 2025)). However, these traditional methods have primarily focused on the LLM’s external behavior (i.e., the final outcome of a given round), which is insufficient to comprehend the strengths and/or weaknesses of its strategic decision-making. We advance this line of research by surfacing the internal mechanisms that inform the model’s decision-making and identifying areas where they deviate from game-theoretic rationality.

We present a three-tiered interpretability pipeline for analyzing intermediate belief formation along the reasoning trajectory. First, we decompose each round’s CoT into sentences and at each chunk, use multiple-choice probes to elicit a verbalized read-out of belief and action. A common concern with this type of CoT interpretability work ([23, 6]) is that the LLM-generated trace might not represent the genuine reasoning process that underpins the model’s predictions ([21, 28, 4, 9]). Taking this criticism seriously, we choose to unembed the residual stream at the final layer to obtain the next-token logit distribution at each chunk boundary rather than reading off the literal output text. Second, we study the causal dependencies between sentences by removing each individual sentence in the CoT and measuring the resulting shift in downstream logits. This step allows us to filter away performative sentences that are not mechanistically relevant to the output. Third, we look into the model’s intermediate activations to decode hidden representations that the LLMs might leverage during deliberation but never explicitly regurgitate in the reasoning trace.

AllC case study: seed 4

Figure 1. A full gameplay against ALLC. Top: per-round decisions for the LLM and its opponent (blue = Cooperate, red = Defect). Bottom: cumulative payoff over the episode for the LLM, the opponent, and social utility (their sum). During the first 7 rounds, the LLM plays C, and both players’ cumulative payoffs increase along the mutual-cooperation trajectory. Starting at round 8, the LLM switches to D (the best strategy for maximizing the individual’s payoff) and never returns to cooperation for the next 23 rounds.

Applying these techniques, our research suggests that LLM agents develop some internal beliefs about the state of the world, which are then used to inform their behavior. Such representations are maintained and updated throughout the CoT as new information emerges. Although models may omit some incriminating reasoning in the CoT, such as fear of exploitation or punishment, these unspoken concepts are present throughout the activations of the early and middle layers.

Because the model’s final decision is tightly coupled with these accumulated beliefs (verbalized or not), the belief trajectory within a trace tends to lead the action. Thus, reaching the best-response action generally entails maintaining the correct belief. We also find that the LLM’s irrational behaviors are often associated with its premature commitment to a hypothesis, which is a self-reinforcing process that worsens over time. Together, we see our work as paving the way for more granular debugging of misbehavior and reasoning failures, which will become increasingly important as LLMs are deployed in complex settings with real-world consequences.

Experimental Setup

Task. We choose to study the LLM’s behavior in a two-player iterated Prisoner’s Dilemma against a fixed opponent. In this game, both players choose whether to defect (D) or cooperate (C) at each round and receive payoffs based on their joint outcome. The game is a dilemma in that the payoff matrix

T>R>P>ST > R > P > S

(see Appendix § 6) incentivizes each rational, self-interested player to defect, despite both parties winding up worse off than if they had cooperated [26, 5]. However, if the player is patient enough and prioritizes future utility, mutual cooperation can emerge as the optimal strategy in a game that is played indefinitely (or, more practically, for an unknown number of times) [16, 17]. Unfortunately, such an equilibrium relies on trust, which for both humans [7, 10, 5] and LLMs [2, 15, 18], is difficult to restore after the opponent’s betrayal.

Model and game protocol. We use DeepSeek-R1-Distill-Qwen-14B [13] as our main reasoning model. It is a 14B-parameter dense decoder-only transformer in the Qwen2.5 architecture [25] with 48 layers, hidden dimension 5,120, 40 attention heads with 8 key-value heads, and a vocabulary of 152,064 tokens. We choose an open-weight model because most of our experiments require access to the model’s internals rather than just its final outputs.

In each round, we present the LLMs with the payoff matrix and interaction history from the most recent ten rounds before asking them to choose their next action. The full prompt is provided in the Appendix (Box 1). To account for the player’s limited memory capacity, we use a rolling window of the ten most recent rounds rather than the entire game history. Each game lasts thirty rounds, but the LLM is unaware of this; the prompt simply indicates that there is a good chance the game will continue to the next round. We provide an example of a complete game in Figure 1.

Opponents. We deploy five fixed strategy archetypes for the opponent. Three are stateless: ALLC plays C with probability 1, ALLD plays D with probability 1, and RANDOM plays C with probability 0.5. Two are reactive to the model’s behavior: TFT opens with C and mirrors the model’s previous action, and GRIM plays C until the model defects once and then plays D forever. Because the name of the game or the opponent’s true strategy class is never revealed in the prompt, we hypothesize that the model should construct a mental model of the opponent from the provided context, such as payoff structure or observed history. It is in our interest to investigate how the LLM processes such evidence and applies strategic reasoning to reach its final decision.

Figure 2. A snapshot of a game against AllC at round 14, where we read P(opponent=AllC)P(\mathrm{opponent}=\mathrm{AllC}) (navy curve) and P(Defect)P(\mathrm{Defect}) (dashed amber curve) after each sentence of the trace. Once belief is locked onto the true opponent class, the decision to defect surges afterward.

How does thinking affect decision-making?

CoTs are useful insofar as the generated traces are used in the actual reasoning that influences the corresponding decision (Wei et al., 2022). Earlier work [23, 9] has shown that decisions in reasoning models are consolidated early in the trace, after which the remaining text has little causal bearing on the final response. This discrepancy implies that CoT texts could be nothing more than an unfaithful, post-hoc rationalization masquerading as the model’s genuine reasoning.

In this section, we revisit the above assumption by investigating how the LLM’s internal judgment and decisions evolve sentence by sentence over their reasoning traces.

Methodology

Chunking and asking. We first split each round’s CoT into KK chunks {c1,…,cK}\{c_1,\ldots,c_K\} and, for each index k∈{0,1,…,K}k\in\{0,1,\ldots,K\}, prompt the LLM to verbalize its verdict using only the partial trace

c≤k=c1⊕⋯⊕ck,(1)\mathbf{c}_{\leq k}=\mathbf{c}_1\oplus\cdots\oplus\mathbf{c}_k, \tag*{(1)}

which is concatenated up to the first kk sentences of the full trace. Here, we consider two question prompts qoppq_{\mathrm{opp}} and qactq_{\mathrm{act}} (see Box 2 and Box 3 in the Appendix) to probe the two main variables of interest: the perceived identity of the opponent (i.e., belief) and the decision to cooperate or defect (i.e., action). Formatting them as multiple-choice questions, we can find the next token probabilities for each answer option by unembedding the residual stream at the final layer and recording the logits. We then take the softmax over the single-letter tokens to get P(act=D)P(\mathrm{act}=\mathrm{D}) and P(opp=ALLC)P(\mathrm{opp}=\mathrm{ALLC}). If the CoT is required to make the final decision, we can expect these probabilities to fluctuate as the reasoning chunks accumulate.

Sentence Ablation. To better understand the (relative) significance of each sentence in the CoT, we also investigate how removing the chunk cic_i from the original trace affects the subsequent token logits. For each downstream chunk cjc_j, we calculate the KL divergence between two next-token distributions at each index j>ij>i: the baseline trace c≤K\mathbf{c}_{\leq K} and the masked trace

c−i=c1⊕⋯⊕ci−1⊕ci+1⊕⋯⊕cK.(2)\mathbf{c}_{-i}=\mathbf{c}_1\oplus\cdots\oplus\mathbf{c}_{i-1}\oplus\mathbf{c}_{i+1}\oplus\cdots\oplus\mathbf{c}_K. \tag*{(2)}

Averaging over all cjc_j’s token positions yields how much removing sentence ii affects the model’s predictions across sentence jj. Using this measure, we can connect pairs of sentences with strong dependence and iterate across the full trajectory (see Appendix § 7 for further details).

Case Study

To provide intuition, we start with a small-scale investigation of the LLM’s thinking process in the middle of a game (round 14) against ALLC. Figure 2 tracks the two probabilities of interest using direct question prompts. Initially, the model is biased toward cooperation and spends the first ten sentences repeating the game prompt, during which it has not yet developed a mental model of the opponent. As the LLM attempts to generalize the patterns from the opponent’s history around c11c_{11} (“The opponent has been consistently choosing Cooperate in almost all rounds…”), we see P(opp=ALLC)P(\mathrm{opp}=\mathrm{ALLC}) rising above chance, indicating the model’s ability to update their internal world model in response to new information. Moreover, removing this sentence has a causal effect on the model’s predictions for subsequent sentences in which the opponent’s belief eventually settles. The complete trace from the sentence ablation experiment is provided in the Appendix (Figure 8).

Causal reasoning chain for Round 14 against ALLC

Figure 8. Causal reasoning chain for Round 14 against ALLC. Belief-related chains follow a fan-out structure, whereas the action chain develops independently. The position of each node on the page is strictly cosmetic.

Note that the belief is still fickle at the early stage. With multiple backtracking traces (“Wait,…”), the LLM self-corrects its earlier misguided assumption that the opponent cooperates in most but not all rounds by subsequently performing several independent readings of the history. Such indecisiveness is reflected in the erratic evolution of P(opp=ALLC)P(\mathrm{opp}=\mathrm{ALLC}) before stabilizing at the mid-point of the trace around c34c_{34} (“So regardless of my choice, the opponent has always Cooperated.”). After the LLM has confidently determined the opponent’s identity, it takes as few as five chunks to translate that knowledge into a decision to defect, causing P(act=D)P(\mathrm{act}=\mathrm{D}) to abruptly rise and remain in the defection regime until the end of the trace.

Results

The first pass results in § 3.2 lends credence to the notion that (a) the CoT can influence the LLM’s behavior, and (b) the LLM can store internal representations of the game’s latent structures and use them to guide the final action. To quantify these claims, we repeat the aforementioned analysis on 250 independent games (50 for each opponent).

CoT allows for belief and decision updating.

LLMs form hypotheses and iteratively revise them in light of new information. Although the prompt contains all of the necessary information for inference, their initial guess about the opponent or the strategy prior to CoT is almost always inadequate. Thinking out loud generally improves the opponent’s identification accuracy and the probability of landing on the optimal action (see Table 2 in the Appendix).

OpponentID acc., k=0k=0ID acc., k=Kk=KP(optimal), k=0k=0P(optimal), k=Kk=K
ALLC0.0170.8780.2710.833
ALLD0.9060.9600.3420.936
RANDOM0.0000.2610.3200.868
GRIM0.0970.2070.4070.879
TFT0.4930.5140.7130.352

Table 2. Read out opponent-identification accuracy and probability of optimal game-theoretical action before reasoning (k=0k=0) and after the complete trace (k=Kk=K). Bold k=Kk=K values indicate progress toward ground truth.

Anatomy of reasoning. Applying the sentence ablation protocol to the entire game population, we find that most reasoning traces follow the same qualitative shape. Before making a decision, the model performs three core computations: (i) reviewing the history log, (ii) formulating and confirming the opponent’s hypothesis, and (iii) calculating the expected payoff. The ambiguity with which the opponent can be identified—that is, the number of possible policies that yield the same behavior as presented in the history log—determines the order and quantity of these traces.

We present a few interesting patterns of the model’s reasoning process:

  • For easy-to-verify games, such as those against ALLC and ALLD, the model allocates around 31% of the total chains to identifying the opponent’s strategy. In games with more ambiguity, this amount is cut in half, and more tokens (about 46% and 50% by mid and late game) are reallocated to backtracking and reciting the round-by-round history. This transition shows the LLMs expressing more uncertainty about their opponent.

  • Because reasoning in ambiguous games is crowded out by history recap sentences, other computations are deferred late within the trace, which is exacerbated as more rounds unfold. The end result is rash decision-making, where the LLM retracts an already correct belief. As an example, out of all the TFT rounds that conclude with the incorrect opponent at the end of the trace, 94.6% had the correct answer earlier in the trace but backtracked. A case study of an individual round is provided in the Appendix (Figure 7).

  • Sentences that express confusion gradually deteriorate as the rounds progress. Over time, the model’s confidence increases regardless of the validity of the belief. This premature commitment implies that if it converges on the incorrect representation about the world, it is unlikely to reassess, inevitably becoming stuck on the suboptimal move.

Figure 7. Sentence-level traces of P(opponent class)P(\text{opponent class}) (navy) and P(Defect)P(\text{Defect}) (dashed amber) against TFT. An early belief spike is retracted mid-trace as later sentences revise the opponent estimate, demonstrating the model’s shift away from the ground truth.

Figure 3. Belief-action coupling by opponent condition, split by final belief. The rows divide the distribution of P(optimal action)P(\text{optimal action}) for each round into high-belief (P(opp=true)≥0.5P(\text{opp}=\text{true}) \ge0.5, green) and low-belief (<0.5< 0.5, rose) halves. The vertical tick on each half-violin marks its median. Δ\Delta is the gap in mean P(optimal action)P(\text{optimal action}) between the high- and low-belief halves, with 95% CIs from game-level bootstrap; percentages give each half’s share of the condition’s rounds.

Belief leads action within the trace. The formation of the opponent’s belief generally occurs before the commitment to cooperate or defect. To demonstrate this, we treat the belief and action probe trajectories within each trace as a bivariate time series and fit a cross-lagged (Granger) model on their first differences. We compare the standardized belief-to-action coefficient βba\beta_{ba} against the reverse βab\beta_{ab}.

Our finding shows that βba>βab\beta_{ba} > \beta_{ab} at every lag and for every opponent. The standardized gap is smaller for opponents it finds difficult to determine (0.084 for TFT, 0.032 for GRIM, and 0.049 for RANDOM), and it is largest for opponents it consistently pins down (0.168 for ALLC and 0.126 for ALLD). Intuitively, when the model can determine who it is dealing with, it commits to a belief about the opponent early in the trace, and the action follows suit; otherwise, the belief remains noisy, providing little signal for the action to track. We also find that the boundary where the model’s readout belief probabilities do not deviate from their final value occurs at a median 29%, 40%, 67%, 82%, and 88% of the trace for ALLD, ALLC, RANDOM, GRIM, and TFT, while actions are always committed near the end (at a median 98%) of the trace.

Belief shapes action.

The LLM’s final action is correlated with its belief’s content and the earliest point with which it stabilizes in the trace (i.e., how soon the model becomes sufficiently confident). Figure 3 depicts this relationship by dividing each opponent’s round distributions by the strength of their beliefs and comparing the resulting P(optimal action)P(\text{optimal action}) splits. We observe evidence of a belief-action coupling across most games, which suggests that accurate identification of the opponent helps the model reach the game-theoretically optimal action. However, identification is not strictly necessary. It is most important when the best response is to cooperate, and less important when the best response is to defect, which the model already prefers in the presence of uncertainty. Against RANDOM, for instance, the model identifies the opponent poorly yet still plays near-optimally (ID accuracy 0.26 versus P(optimal)=0.87P(\text{optimal}) = 0.87 at k=Kk=K; Table 2), since constant defection is the uncertainty-robust response. As expected, the model has high confidence in the belief of ALLD and ALLC, so it plays D with high certainty for most rounds. The untriggered GRIM, by definition, occurs early in the game, when the LLM has not tried an alternate strategy to gather information, and thus its prior belief (which is to play C) does not correlate with the best action. Almost half of the TFT rounds end up with the incorrect model about the opponent, but those with the correct belief learn to reciprocate and play C accordingly.

We further illustrate this nuanced dynamic in Figure 4, which inspects the probed probabilities P(opp=true)P(\text{opp}=\text{true}) and P(act=D)P(\text{act}=\text{D}) in the reasoning trace during (early, mid, late) games against ALLC and TFT.

Figure 4. Belief (P(opp=true)P(\mathrm{opp} = \mathrm{true}) and action (P(act=D)P(\mathrm{act} = \mathrm{D}) trajectories within the reasoning trace for ALLC and TFT, faceted by round bin. Games are split within each bin by whether their opponent probe ends the trace above (teal) or at/below (coral) 0.6. Thin lines are individual games; thick lines are per-cluster means.

Accurate beliefs improve performance. In the early rounds against ALLC, the LLM sometimes misidentifies the opponent and reasons its way to cooperation (P(act=D)≈0.1P(\text{act}=\text{D}) \approx0.1–0.20.2), which is not the optimal strategy. We expect this noisy performance, as reasoning models do computations stochastically and may commit to the wrong reasoning path, particularly when the decisions involved are complex. Ultimately, the model learns to self-correct. The true opponent’s identity recovered by rounds 11–20, resulting in a gradual increase in P(act=D)P(\text{act}=\text{D}) throughout the trace. This consolidated knowledge of the opponent, as well as the consequent commitment to defect, persists for the remainder of the game.

Wrong beliefs diminish performance. Because CoT-driven revision does not always lead to the truth, we can expect worse performance in rounds with misclassified opponents. We should note, however, that a low probability of the true type is not always an inference error. Since each round only shows a truncated ten-round history, several opponent types can be observationally equivalent (see § 3.3). As reflected in Figure 4, we see the games against TFT bifurcate into two categories with time: those where the LLM can identify the opponent (teal) and those where it cannot (coral). As mutual defects accumulate in the log, the model increasingly misinterprets the opponent as ALLD. This causes P(act=D)P(\mathrm{act} = \mathrm{D}) to climb past 0.5, leading to suboptimal retaliation. Remarkably, we find the same bias towards defection against RANDOM and GRIM, since the LLM also struggles to identify the opponent in these situations (see Figure 3 and Appendix § 8). However, the bias is only detrimental against GRIM, where defection ultimately results in irreversible retaliation. In contrast to RANDOM, defection happens to coincide with optimal play, so poor identification of the opponent does not matter. Looking ahead, our findings point to the need for human supervision and/or guardrails to mitigate these systemic vulnerabilities.

Reading the intent beneath the CoT

Our analysis in Section 3 assumes that any inferences drawn by the LLM about the game are externalized into the CoT. However, it is possible that the model relies on intermediate ideas that do not manifest in the reasoning process or are phrased using a different vocabulary than the question. According to recent research on global workspace theory in LLMs (Gurnee et al., [19]; Chalmers, [8]), these hidden variables can emerge in the model’s “workspace” layers during the forward pass and can be decoded from a linear readout of intermediate activations.

Methodology

Jacobian lens. The Jacobian lens (J-lens; Gurnee et al., [19]), approximates the first-order causal effect of activations on the model’s output tokens. It uses a Jacobian to account for changes in coordinate systems from the intermediate layers to the

final layer to improve comprehension of the readouts in intermediate layers. For every layer ℓ\ell, we fit the linear map

Jℓ=Et,t′≥t[∂hfinal,t′∂hℓ,t],(3)J_{\ell} = \mathbb{E}_{t,t'\ge t}\left[\frac{\partial h_{\mathrm{final},t'}}{\partial h_{\ell,t}}\right], \tag*{(3)}

by backpropagating from the final-layer residual stream down to a layer-ℓ\ell activation and averaging over token positions and a corpus of prompts. Applying the unembedding matrix to the corrected activation provides a token-level ranked readout of what the model is poised to verbalize from any layers at every token position in the CoT. This reveals the intermediate phases of the model’s internal reasoning, which may include ideas that the model internally conceives but never articulates.

Case Study

We first examine round 17 of a game against TFT, where the LLM prematurely commits to the incorrect belief, as alluded to in § 3.3.2. Perusing this round’s CoT, we observe that the model initially considered the reciprocity hypothesis (using a Tit-for-Tat strategy?), but during the first 6% of the trace, it became preoccupied with the prospect of being punished (trying to punish me) and exploited (trying to exploit my behavior). This diversion compels the LLM to confabulate and refute its initial belief of the opponent, leading to the irrational action of playing D in this round.

We apply the J-lens to each token position in the CoT and record the top lens tokens. Figure 5 annotates the CoT transcript with the J-lens readouts across layers. At about 3% through the trace, the lens begin to flag exploit, exploitation, and exploiting as the model tries to recollect the history and generalize the opponent’s pattern. It

Figure 5. Four readouts of lens(hℓh_\ell) across TFT round 17’s reasoning trace. Left: CoT excerpts, with the readout position boxed. Right: top-3 candidates at an early (ℓ=8\ell=8), middle (ℓ=24\ell=24), and the final captured (ℓ=44\ell=44) layer for that position. Row 1: 605 tokens before “exploit” is ever articulated. Row 2: 1 token after the literal mention of “exploit”. Rows 3–4: after the round’s only appearance of “exploit”, at 29% and 56% through the trace respectively.

should be noted that these labels emerge 605 tokens before the exploit frame is directly mentioned in the CoT. LLM’s nonverbal reasoning precedes its verbal dispositions.

Another notable finding is that the model can reason over semantic neighborhoods of the same concept, both cross-lingually (e.g., 报复, which means to retaliate in Chinese) and morphologically (e.g., Pun, a truncated variant of punish). As the model is weighing whether it is “getting stuck in a Defect–Defect cycle”, the lens surface exploited, explo, suck, 灰恶 (aversion) at the mid and final layers. Moreover, the LLM appears to recognize that its environment pertains to economic games, as evidenced by the recurring appearances of strategy, 有的玩家 (some players), and behavior in the early and mid layers across the trace.

Results

We now extend the above corpus inspection protocol to 250 independent games (50 for each opponent). For each round, we divide the generated trace into 40 equal-fraction bins and record J-lens’ top-20 candidates for each token position. Figure 6 categorizes the most frequent readouts into seven vocabulary groups (see Table 3 in the Appendix).

Figure 6. Presence rate of concept families across the reasoning trace (band: Wilson 95% CI). Opponent-invariant concepts: five vocabulary families whose trajectory shape agrees across opponents (mean pairwise Pearson rr = 0.65–0.91), pooled into one curve per family. Punitive family, Exploited family: the two families whose shape depends on the opponent (rr = 0.55, 0.40).

FamilyMatched tokensSuppression rate
GIVEUPforever, permanent, permanently68.3%
PUNITIVEpunish, punishing, punished, punishment, retaliate, retaliation, retaliating, retaliatory, vengeance70.6%
EXPLOITEDexploit, exploiting, exploited, exploitation73.9%
RECIPROCITYmirror, mirrors, mirroring, mirrored, tit, tat, reciprocal, reciprocate, reciprocity, reciprocated, mutual, forgive, forgiving, forgiven, forgiveness2.4%
SELF-AS-VICTIMselfish, vulnerable, sucker, naive96.9%
HEDGEunlikely, inconsistent, confused, confusing, stuck47.4%
RISKtricky, betray, betrayal, betraying, gamble, gambling, risk, risky46.8%

Table 3. Word families tracked in the corpus-wide presence-rate analysis, with matched token forms and suppression rate (share of rounds where the family surfaces in the J-lens top-20 but is never emitted in the model’s CoT).

Five of the seven concept families are surfaced regardless of the opponent. Most concepts in the model’s internal layers take on sufficiently similar shapes across games (i.e., they are opponent invariant; see the first panel of Figure 6). These surfaced tokens can provide us with an algorithmic intuition for how the LLM makes decisions in a repeated normal-form game. With the exception of RECIPROCITY, which syntactically aligns with one of the opponent’s archetype names, most of these readouts do not appear in the actual CoT of the round.

We find that the LLM first considered the possibility of being taken advantage of via SELF-AS-VICTIM in the beginning of the CoT but set aside this frame as reasoning progressed. In § 3.3, we established that the CoT in ambiguous games are crowded with hedging and history-recap language. Such reasoning patterns are consistent with the emergence of RISK, HEDGE, and eventually GIVEUP token families in the J-lens toward the end of the trace.

Two concept families track each opponent’s game-theoretic structure. PUNITIVE and EXPLOITED have the weakest cross-opponent shape agreement (r=0.55r = 0.55 and r=0.40r = 0.40, versus 0.65–0.91 for the others). We perform a Kruskal-Wallis test (see Figure 11 in the Appendix), which reveals that both families reject the null hypothesis of no opponent effect (PUNITIVE: H(4)=108.7H(4)=108.7, p<0.001p < 0.001; EXPLOITED: H(4)=68.5H(4)=68.5, p<0.001p < 0.001). The second and third panels of Figure 6 display the per-opponent detected rate over the CoT, pooled across all rounds.

Figure 11. Per-opponent detection rate for the two opponent-dependent concept families from Figure 6, summarized as the mean % of a round’s 40 trace bins in which the family is detected (error bars: 95% CI). Each panel reports a Kruskal-Wallis test of whether this per-round bins-hit count differs by opponent identity (PUNITIVE: H(4)=108.7H(4)=108.7, p<0.001p < 0.001; EXPLOITED: H(4)=68.5H(4)=68.5, p<0.001p < 0.001), confirming that both families’ opponent-dependence is statistically reliable.

EXPLOITED is the least common (detected in a mean of 13.9% bins per round) in games against ALLD, where defecting is the optimal strategy and mutual defection leaves no one to exploit. However, it is highest against its naive, exploitable counterparts: RANDOM (25.4%) and ALLC (22.7%). Since the appearance of PUNITIVE depends on whether the opponent’s strategy is reactive to the model’s actions, it is ranked lowest when compared to the two static archetypes ALLC (59.4%) and ALLD (55.8%). As expected, it ranks relatively high when compared to RANDOM (74.2%), TFT (70.4%), and GRIM (68.4%), whose behavior is unpredictable enough to provoke thoughts of being punished.

Discussion

In this paper, we investigated how CoT reasoning traces affect the performance of LLM agents playing an iterative Prisoner’s Dilemma. Based on three levels of mechanistic evidence (logit probes, sentence ablation, and activation-level concept readouts), our findings provide a consistent picture of the LLM’s decision-making process in a repeated normal-form game. First, the model can retain and revise valuable latent game information (i.e., belief) over time, which aids in the selection of the final decisions (i.e., action). Second, the belief-action coupling is shown to be persistent: once its mental model of the game has stabilized, the LLM rarely revisits it, which can lead to suboptimal decisions that compound with time. Third, CoT can be viewed as a partial account of the reasoning process. Upon examining the activations, we find that the model sometimes uses internal, nonverbalized concepts to inform its reasoning throughout the CoT. Because our explanations are based on convergent lines of evidence, we think of internal representations as guiding (leading and shaping) the decision rather than being the sole causal factor.

Limitations

Looking forward, our work suggests concrete research directions. First, we only look at an open-weight LLM of moderate size, which raises the question of whether significantly larger models behave similarly. A prior survey [30] indicates that scale alone cannot resolve safety failures, but future research should investigate this possibility. Second, our study only focuses on fixed, non-adaptive opponents. To accommodate the growing number of multiagent systems, our pipeline can be extended to incorporate LLM-vs-LLM play. Another intriguing follow-up is to carry out an intervention experiment in an attempt to reverse the lock-in behavior, such as ablating the concept families from the residual stream at the point in the trace where belief solidifies.

Author Contributions

Chaipat Tirapongprasert led the project, wrote the paper, and carried out all experimental work. Rajarshi Das and Parijat Dube provided feedback and guidance throughout the project.

Acknowledgments

This work was supported in part by Columbia University’s Department of Statistics. C.T. thanks Zehao Peng for insightful discussions, which helped shape some of our approach. All the experiments were run on NCSA DeltaAI at UIUC under the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program.

Payoff Matrix and Prompts

Payoff matrix. We instantiate the payoff matrix for iterated prisoner’s dilemma as T=5T = 5 (temptation), R=3R = 3 (reward), P=1P = 1 (punishment), S=0S = 0 (sucker). The ordering T>R>P>ST > R > P > S makes defection the dominant action in the one-shot game while mutual cooperation maximizes joint welfare [26, 5]. Table 1 gives the payoff to each player for every combination of actions, with the model as the row player and cells written as (model’s payoff, opponent’s payoff).

Opponent
ModelCD
R,RR, R (3, 3)S,TS, T (0, 5)
T,ST, S (5, 0)P,PP, P (1, 1)

Table 1. Payoff matrix for the iterated prisoner’s dilemma game used throughout the paper.

Game prompt. Every round, the model is shown the payoff matrix, the continuation probability, and up to the ten most recent rounds of history, then asked to commit to an action in a fixed format.

IPD game prompt

Figure 1 (PDF p. 11).

You are playing a repeated two-player game against a fixed opponent. Each round, you and your opponent each simultaneously choose one of two actions: Cooperate or Defect.

Payoffs this round, depending on what you and your opponent choose:

  • You Cooperate, Opponent Cooperates: you get 3, opponent gets 3

  • You Cooperate, Opponent Defects: you get 0, opponent gets 5

  • You Defect, Opponent Cooperates: you get 5, opponent gets 0

  • You Defect, Opponent Defects: you get 1, opponent gets 1

After this round, there is a 70% chance the game continues for at least one more round, and otherwise the game ends.

Here is what happened in the most recent rounds: Round ii: You chose Cooperate|Defect, Opponent chose Cooperate|Defect.

: (up to 10 most recent rounds; “This is the first round of the game.” if none)

It is now Round nn. Think step by step about what your opponent is likely to do and what action best serves your long-run interest. When you have decided, end your response with a line in exactly this format (and nothing after it): FINAL ACTION: Cooperate or FINAL ACTION: Defect

Probe prompts. At each chunk boundary kk (Section 3), we truncate the trace after c≤kc_{\leq k}, close the <think> block, and force one of the two continuations below. Both are appended verbatim after the truncated trace, so the model’s response is contingent on the reasoning it has generated thus far. We read next-token logits at the final position: a binary Cooperate/Defect softmax for the action probe (Box 2), and a five-way softmax over the option letters for the belief probe (Box 3).

Action probe

Figure 2 (PDF p. 11).

[⋯ truncated trace c≤kc_{\leq k} ⋯] </think>

FINAL ACTION: [read P(Cooperate)P(\mathit{Cooperate}) vs. P(Defect)P(\mathit{Defect}) here]

Opponent-identity belief probe

Figure 3 (PDF p. 11).

[⋯ truncated trace c≤kc_{\leq k} ⋯] </think>

The opponent’s strategy is most likely: (A) Always Cooperate (B) Always Defect (C) Tit-for-Tat (D) Grim Trigger (E) Random Answer: ([read the 5-way softmax over A–E here])

Causal dependency between sentences

In this section, we give more details about our causal ablation implementation. Recall that for a round with chunks c1,…,cKc_1,\ldots,c_K, we run one baseline forward pass over the entire trace and then one masked forward pass per chunk ii with cic_i ablated. To calculate the sentence pairs’ KL divergence KL(i→j)\mathrm{KL}(i \to j), we average the per-token KL divergence between the baseline’s and the masked pass’s next-token distributions over cjc_j’s tokens for each downstream chunk j>ij > i. Because nearby sentences mechanically give larger KL than distant ones, we normalize each pair against KL(d)\mathrm{KL}(d), the average KL(i→j)\mathrm{KL}(i \to j) across every pair in that same round at the same distance d=j−id = j - i.

Casual chain. We define two sentences cic_i and cjc_j as causally dependent if the sentence pair KL(i→j)\mathrm{KL}(i \to j) is at least 8×8\times larger than the distance-matched baseline KL(d)\mathrm{KL}(d). Figure 8 depicts the entire causal trajectory for Round 14 against ALLC. We can see that the belief cluster is more dense and well-connected because the opponent’s belief is updated by reading independently sufficient sections of the history log. Once the opponent’s belief is confirmed, committing to an action can be carried out in a sequential reasoning chain.

Case study for retraction. Figure 7 traces the failure mode at round 10 of a game against TFT. In the early traces, P(opp=TFT)P(\text{opp} = \text{TFT}) oscillates violently, twice spiking to near 1.0 before collapsing back toward 0. During this period, P(Defect)P(\text{Defect}) remains close to the 0.5 decision boundary, indicating that the model has not yet committed to any action. The belief returns to 1.0 around sentence 50 and remains there until sentence 67 (“But in Round 3, I Defected again, and. . .”), when the model revisits an old defection in the history log. Over the next few sentences, it talks itself out of the correct belief, and by sentence 75, P(opp=TFT)P(\text{opp} = \text{TFT}) vanishes while P(Defect)P(\text{Defect}) begins to rise. We present this snapshot as an example of the premature-commitment dynamic, in which the model does not revisit an incorrect belief for the remainder of the round after it has stabilized.

Addtional Population Analysis

Evolution of beliefs and action. Table 2 displays the opponent-identification accuracy and probability of taking the game-theoretically optimal action before any reasoning has occurred (k=0k=0, i.e., from the payoff matrix and history alone) versus after the complete reasoning trace (k=Kk=K). For ALLC, RANDOM, and GRIM, reasoning is unambiguously useful. Both quantities increase significantly over the trace, confirming that the model begins with no opinion about the opponent’s identity and gradually develops one during the reasoning process. TFT is the only condition where a naive, non-reasoning readout performs better. We discuss this failure mode in Section 3.3. Overall, the findings suggest that reasoning traces aid in the translation of beliefs (whether correct or incorrect) into actions.

Game alignment. Figure 9 gives the per-game distributions for each round bin, colored by whether the game is aligned (i.e., the LLM identified the opponent and chose the correct action). This design aims to quantify how frequently an accurate belief leads to the game-theoretically optimal action.

Side-by-side violin distributions for AllC and TFT

Figure 9. Side-by-side comparison of AllC and TFT (rounds 1-10, 11-20, and 21-30). Each violin is a per-game distribution (green = AllC, yellow = TFT); horizontal bars mark violin means. Left: P(opp=true)P(\mathrm{opp} = \mathrm{true}) at round-end; right: P(act=D)P(\mathrm{act} = \mathrm{D}) at round-end. “Aligned” denotes games in which the LLMs identified the opponent more accurately than chance and consequently took the optimal strategy.

Looking at the game trajectory against ALLC, both defection and belief converge tightly toward 1: mean P(opp=true)P(\mathrm{opp}=\mathrm{true}) increases from 0.65 in rounds 1–10 to 0.98 by rounds 11–20 and 1.0 by rounds 21–30. As expected, P(act=D)P(\mathrm{act}=D) closely tracks (0.60 → 0.92 → 0.98), with an increasing majority of games falling into the aligned cluster. In contrast, games against TFT exhibits the opposite characteristic, with its belief distribution becoming bimodal. This is seen in its mean drifting downward from 0.56 in rounds 1–10 to 0.48 in rounds 11–20 and 0.43 in rounds 21–30. Because TFT’s optimal response is to cooperate, we see P(act=D)P(\mathrm{act} = \mathrm{D}) climbing up from 0.55 to 0.73. This failure mode is described in § 3.3.

Figure 10 extends the per-episode view to the remaining three opponents. ALLD appears to be the easiest opponent to identify, with mean P(opp=true)P(\mathrm{opp} = \mathrm{true}) already starting at 0.90 in rounds 1-10. Because GRIM is behaviorally indistinguishable from ALLC until triggered, the model’s belief in the true GRIM label remains low throughout the game (mean 0.18–0.29 across bins), but P(act=D)P(\mathrm{act} = \mathrm{D}) steadily climbs from 0.63 to 0.93 across rounds. The same pattern applies to RANDOM, where there is by definition no exploitable pattern to lock onto. Here, the mean P(opp=true)P(\mathrm{opp} = \mathrm{true}) remains below chance in all round bins (0.08 → 0.30 → 0.39), while P(act=D)P(\mathrm{act} = \mathrm{D}) is already high in the early game (0.75), rising further to 0.93. Since the model is uncertain of the opponent’s character, it has the proclivity to defect, which is the best strategy for a single-round prisoner’s dilemma.

Round-bin distributions of P(opp = true) and P(act = D) against ALLD, GRIM, and RANDOM

Figure 10. Round-bin distributions of P(opp=true)P(\mathrm{opp} = \mathrm{true}) (left) and P(act=D)P(\mathrm{act} = \mathrm{D}) (right) at round-end against the rest of the opponents.

References

  1. [1]Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Schönherr, and Mario Fritz. 2024. Cooperation, competition, and maliciousness: Llm-stakeholders interactive negotiation. arXiv preprint arXiv:2309.17234. NeurIPS 2024 Datasets and Benchmarks Track.arxiv.org/abs/2309.17234
  2. [2]Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. 2023. Playing repeated games with large language models. arXiv preprint arXiv:2305.16867.DOI
  3. [3]Anthropic. 2025. Claude 3.7 sonnet system card.
  4. [4]Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy. 2025. Chain-of-thought reasoning in the wild is not always faithful. arXiv preprint arXiv:2503.08679.arxiv.org/abs/2503.08679
  5. [5]Robert Axelrod and William D. Hamilton. 1981. The evolution of cooperation. Science, 211(4489):1390–1396.
  6. [6]Paul C. Bogdan, Uzay Macar, Neel Nanda, and Arthur Conmy. 2025. Thought anchors: Which llm reasoning steps matter? arXiv preprint arXiv:2506.19143.arxiv.org/abs/2506.19143
  7. [7]Pedro Dal Bó. 2005. Cooperation under the shadow of the future: Experimental evidence from infinitely repeated games. American Economic Review, 95(5):1591–1604.
  8. [8]David J Chalmers. 2026. Is the jacobian space a global workspace?
  9. [9]Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, and Ethan Perez. 2025. Reasoning models don’t always say what they think. arXiv preprint arXiv:2505.05410.arxiv.org/abs/2505.05410
  10. [10]Pedro Dal Bó and Guillaume R Fréchette. 2011. The evolution of cooperation in infinitely repeated games: Experimental evidence. American Economic Review, 101(1):411–429.
  11. [11]Rohini Elora Das, Krissh Bhargava, Krishna Shinde, and Rajarshi Das. 2025. Representation-alignment in Theory-of-Mind tasks across language models and agents. In Second Workshop on Representational Alignment at ICLR 2025.
  12. [12]Rohini Elora Das, Rajarshi Das, Niharika Maity, and Sreerupa Das. 2024. Iterative theory of mind assay of multimodal AI models. In ICML 2024 Workshop on LLMs and Cognition.
  13. [13]DeepSeek-AI, Daya Guo, Dejian Yang, et al. 2025. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  14. [14]Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel-Eskin, Mohit Bansal, Tianlong Chen, and Kaidi Xu. 2024. Gtbench: Uncovering the strategic reasoning limitations of llms via game-theoretic evaluations. arXiv preprint arXiv:2402.12348. NeurIPS 2024.arxiv.org/abs/2402.12348
  15. [15]Nicoló Fontana, Francesco Pierri, and Luca Maria Aiello. 2024. Nicer than humans: How do large language models behave in the prisoner’s dilemma? arXiv preprint arXiv:2406.13605.arxiv.org/abs/2406.13605
  16. [16]James W. Friedman. 1971. A non-cooperative equilibrium for supergames. The Review of Economic Studies, 38(1):1.
  17. [17]Drew Fudenberg and Eric Maskin. 1986. The folk theorem in repeated games with discounting or with incomplete information. Econometrica, 54(3):533.
  18. [18]Fulin Guo. 2023. Gpt in game theory experiments. arXiv preprint arXiv:2305.05516.arxiv.org/abs/2305.05516
  19. [19]Wes Gurnee, Nicholas Sofroniew, Adam Pearce, Mateusz Piotrowski, Isaac Kauvar, Runjin Chen, Anna Soligo, Paul Bogdan, Euan Ong, Rowan Wang, Ben Thompson, David Abrahams, Subhash Kantamneni, Emmanuel Ameisen, Joshua Batson, and Jack Lindsey. 2026. Verbalizable representations form a global workspace in language models. Preprint, arXiv:2607.15495.arxiv.org/abs/2607.15495
  20. [20]Sherzod Hakimov, Roland Bernard, Tim Leiber, Karl Osswald, Kristina Richert, Ruilin Yang, Raffaella Bernardi, and David Schlangen. 2025. The price of thought: A multilingual analysis of reasoning, performance, and cost of negotiation in large language models. arXiv preprint arXiv:2510.08098.arxiv.org/abs/2510.08098
  21. [21]Subbarao Kambhampati, Karthik Valmeekam, Siddhant Bhambri, Vardhan Palod, Lucas Saldyt, Kaya Stechly, Soumya Rani Samineni, Durgesh Kalwar, and Upasana Biswas. 2026. Position: Stop anthropomorphizing intermediate tokens as reasoning/thinking traces! Preprint, arXiv:2504.09762.arxiv.org/abs/2504.09762
  22. [22]Michal Kosinski. 2024. Evaluating large language models in theory of mind tasks. Proceedings of the National Academy of Sciences, 121(45).DOI
  23. [23]Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamile Lukosiūtė, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, and 11 others. 2023. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702.arxiv.org/abs/2307.13702
  24. [24]OpenAI. 2024. Learning to reason with llms.
  25. [25]Qwen, An Yang, Baosong Yang, et al. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115.
  26. [26]Anatol Rapoport and Albert Chammah. 1965. Prisoner’s Dilemma. University of Michigan Press.
  27. [27]Ziyi Shi, Xusen Guo, Hongliang Lu, Mingxing Peng, Haotian Wang, Zheng Zhu, Zhenning Li, Yuxuan Liang, Xinzhu Zheng, and Hai Yang. 2026. Coordinated pandemic control with large language model agents as policymaking assistants. arXiv preprint arXiv:2601.09264.arxiv.org/abs/2601.09264
  28. [28]Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. arXiv preprint arXiv:2305.04388.arxiv.org/abs/2305.04388
  29. [29]Tomer Ullman. 2023. Large language models fail on trivial alterations to theory-of-mind tasks.arxiv.org/abs/2302.08399
  30. [30]Cheng Wang, Yue Liu, Baolong Bi, Duzhen Zhang, Zhong-Zhi Li, Yingwei Ma, Yufei He, Shengju Yu, Xinfeng Li, Junfeng Fang, Jiaheng Zhang, and Bryan Hooi. 2025. Safety in large reasoning models: A survey. Preprint, arXiv:2504.17704.arxiv.org/abs/2504.17704

Paper details

vlnc:W_zjkxtnd7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 11

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents