betaFeedback

WorksComputer Science

AVCORP: Identifying and Resisting Deception via Communication Theory and Theory of Mind

Abstract

Today's LLM-based agents are vulnerable to deception by users and other agents. Defending against deception requires Theory of Mind (ToM) reasoning to infer the beliefs and goals of others as well as modeling tactics based on communication theory. To enable training models to detect deception, we adopt the well-known deception game, Avalon. We present a corpus, AVCORP, of 5,000 annotated utterances from 250 games of Avalon. AVCORP includes (1) 37 tactics mapped to a 4x4 behavior matrix, and (2) reasoning traces capturing abductive hypotheses, suspicion levels, and nested ToM beliefs. We train Qwen3-8B on AVCORP and evaluate it on 25 held-out games and 20 games played by humans. On held-out games, our best trained detector outperforms its untrained baseline and frontier models (GPT-5.6 Sol, Gemini 3.1 Pro, DeepSeek-V4-Pro), matching Claude Opus 5.5. On human games, every trained detector exceeds the untrained model in F1 score across all rounds.

Correspondence: [email protected]

🐙 Code and Corpus: https://github.com/fardinsaad/_AVCORP_

🐣 Dataset and Detector: https://huggingface.co/collections/fardinsaad/avcorp

Introduction

Agents based on Large Language Models (LLMs) are increasingly deployed in practice [45]. Besides capabilities and skills, what makes an agent is its mind and its recognition of the minds of others [10, 37]. Such Theory of Mind (ToM) [29, 13] endowed agents promise an enhanced ability to cooperate by dealing with incompleteness and ambiguity of communication [31].

Although attacks on LLMs, such as jailbreaking [41], are as old as LLMs, the rise of ToM-endowed agents creates a new threat. Specifically, an attacker may use an agent’s cognitive reasoning against it not merely by circumventing its rigid guardrails but by deceiving it.

We adopt the following working definition of deception. An agent deceives a target when it intentionally leads the target to form or keep a belief that the agent believes false, to advance its own goal [42].

We focus on an agent being deceived by a user or another agent. Users deceive agents: attackers told Claude that it was an employee of a legitimate cybersecurity firm conducting defensive testing, and it then carried out their cyberattacks [2]. Agents deceive agents: frontier models do so when it serves an objective [26], and other agents are susceptible to deception as they do not reliably distrust goals passed along by the deceptive agent in pursuit of a shared goal [27]. Since deception by users and agents is accelerating, defenses must accelerate comparably, which requires characterizing the structure of deceptive behavior [32].

By our definition, this structure lies in the mind: a deceiver models the target’s beliefs [39], operating one ToM order above the target. To combat deception, an investigator must model how the deceiver is modeling them, creating a reciprocal escalation [33]. Although this escalation demands increasing cognitive depth, behavioral models suggest that humans mostly operate at first- or second-order reasoning [6, 8]. Effective detection must therefore operate at this depth, requiring not only attributing beliefs to others (first-order belief attribution: what does player X believe?) but also modeling how others intend to influence those beliefs (second-order influence attribution: what does player X want me to believe, and why?). Training a detector to reason at this depth requires corpora that link tactical utterances to these nested ToM reasoning traces.

We present AVCORP, a corpus that makes this connection through dual-layer annotations. The first layer annotates utterances using a 37-tactic behavior matrix grounded in Information Manipulation Theory 2 (IMT2) [23]. Interpersonal Deception Theory (IDT) ([4]), Truth-Default Theory (TDT) ([19]), and philosophical accounts of deceptive speech acts ([12, 7]) (collectively, PhI). The second layer records an Investigator’s reasoning trace for each round, with abductive hypotheses, suspicion levels, and nested ToM beliefs, and reserves second-order beliefs for strongly suspected players.

Figure 1. One round of Avalon gameplay. Team Selection: A rotating leader (P4) proposes a quest team (P1 and P3). Voting: All players vote on the proposal; a majority rejection passes leadership to the next player. Quest Execution: Approved team members anonymously submit PASS or FAIL cards. A single FAIL sabotages the quest. Discussion: Players analyze outcomes, contest identities, and share evidence before the next round begins.

Our contributions are as follows:

Conceptual Model: Linking what is said, through 37 tactics in a 4×44 \times4 behavior matrix, to why, through nested ToM reasoning (§ 3).

Illustrative Corpus: 5,000 utterances from 250 games, 4,000 tactic-labeled and 1,000 with ToM reasoning traces, built with a scalable six-stage pipeline (§ 4).

Human-Validated Traces: The traces deduce both Evil players at game end in 93.2% of games, and two blind annotators agree with each other and with the judge’s selection (κ\kappa from .92 to .94; § 4.6.1 and § 4.6.2).

Trained Detector: On held-out games, our best detector outperforms three frontier models and matches Claude Opus 5.5; on human games, every trained detector exceeds the untrained model in F1 score (§ 5).

Avalon Gameplay and Design

Avalon: The Resistance offers a controlled testbed for this problem. It is a “Game of Thoughts” (Wang et al., [40]) with extreme informational asymmetry: Good players share evidence, whereas Evil players frame and fabricate to conceal their identities. Existing resources capture parts of what a detector needs, such as human strategy annotations ([38, 20]) and statement-level deception labels ([1]), but they do not connect what is said to why it is said through nested ToM reasoning.

Avalon is a social deduction game where players are secretly assigned to opposing factions (a Good majority and an Evil minority) ([43]). Evil players know all alignments and conceal themselves to disrupt the Good faction, whereas Good players know only their own and must identify Evil players through open discussion.

We keep only the Good and Evil alignments, removing special roles (e.g., Merlin, Assassin), and designate one Good player as the Investigator. The non-Investigator players serve as contextual agents (CC). Their discussion provides the behavioral cues that the Investigator uses to detect deception. A game lasts at most five rounds and ends when either faction wins three quests. Each game has three Good and two Evil players, and each round has three phases followed by a discussion (Figure 1).

Team Selection. Each game’s first leader ℓ1\ell_1 is assigned, and leadership then rotates round-robin. In each round rr, the leader ℓr\ell_r proposes a quest team TrT_r of two or three players (Table 3, Appendix § 10).

RoundR1R2R3R4R5
Team size23233

Table 3. Team size distribution across Avalon rounds.

Voting. All players vote on the proposed team (VrV_r), but every proposal receives a passing majority. This ensures complete games and preserves the voting record as evidence for the Investigator.

Quest Execution. Approved team members secretly submit a PASS or FAIL card, and a single FAIL fails the quest (QrQ_r). Good players always pass, whereas Evil players may fail a quest or pass it to maintain cover.

Discussion. After each quest, players discuss without strict turn-taking. Each contextual agent contributes one utterance uiu_i of one to three sentences (Figure 2), generated from our taxonomy (§ 3). The Investigator participates last, contributing the final utterance u^\hat{u} as the penultimate output of its reasoning trace.

Figure 2. One AVCORP sample from Round 1 of Game 60. After a two-person quest returns FAIL, G1_1 accuses E1_1 (Rational justification); E1_1 deflects (Defensive disclosure); G2_2 expresses uncertainty (Perspective-taking); E2_2 overclaims future leadership (Overstated contribution). The Investigator abducts E1_1 as likely Evil (hE1Eh^{E}_{E_1}), assigns Strong suspicion (sE1s_{E_1}), and infers E1_1’s second-order belief that ambiguity will delay accusation (β2E1\beta^{E_1}_2). The trace concludes with the Investigator’s utterance u^\hat{u} and deduction α^\hat{\alpha} (hiGh^G_i, hiEh^E_i, sis_i and β1i\beta^i_1, β2i\beta^i_2 are summarized versions).

Each round rr adds Hr=(ℓr,Tr,Vr,Qr)\mathcal{H}_r = (\ell_r,T_r,V_r,Q_r) to the history; its discussion is Dr={ui}i∈CD_r = \{u_i\}_{i\in C}. After round rr, the Investigator reasons over all rounds so far using the history H1:r\mathcal{H}_{1:r} and discussion D1:rD_{1:r} to construct its reasoning trace ρr\rho_r.

Reasoning Trace. The Investigator’s output for each sample is a reasoning trace ρr\rho_r, which serves as the gold label for deception detection. Formally:

ρr=({hi}i∈C,{si}i∈C,{β1i}i∈C,{β2i}i∈Csi≥Strong,u^,α^) where,\rho_r = \left(\{h_i\}_{i\in C},\{s_i\}_{i\in C},\{\beta^i_1\}_{i\in C},\{\beta^i_2\}_{i\in C_{s_i\geq\mathrm{Strong}}},\hat{u},\hat{\alpha}\right)\ \text{where,}

hi=(hiG,hiE)h_i = (h_i^G,h_i^E) are abductive hypotheses assuming Good and Evil alignments (both grounded in H1:r\mathcal{H}_{1:r} and D1:rD_{1:r}), si∈{Minimal,Mild,Moderate,Strong,Extreme}s_i \in\{\mathrm{Minimal},\mathrm{Mild},\mathrm{Moderate},\mathrm{Strong},\mathrm{Extreme}\} is the suspicion level, β1i\beta^i_1 and β2i\beta^i_2 are first- and second-order beliefs (the latter only for Strong/Extreme suspicion), u^\hat{u} is the Investigator’s utterance, and α^\hat{\alpha} is the per-round deduction of suspected Evil players (Figure 2).

Understanding Deception: A Taxonomy

Table 1 presents our taxonomy of deceptive and truthful communication to guide the dialogue generation. Rows reflect information commitment strategies (grounded in IMT2 and PhI) and columns reflect social goal orientations (grounded in IDT and TDT). The cells list tactics (37 in all); Appendix B provides their detailed definitions.

Info CommitmentCooperative: Advancing shared goals honestlyDefensive: Protecting one’s own position or imageOpportunistic: Exploiting situations for personal advantageAdversarial: Undermining or sabotaging others
Transparent: Openly sharing clear, factual informationEvidence sharing<br>Rational justification<br>Perspective-takingDefensive disclosure<br>Self-justificationWeaponized candor<br>Overstated contributionTruthful misdirection<br>Misleading emphasis
Framing: Shaping discourse through omission or emphasisBayesian hedging<br>Strategic uncertainty<br>Pragmatic silenceOmission to save face<br>Half-truthsCherry-picking<br>False consensus<br>MisframingWithholding info<br>Strategic omission
Careless-to-truth: Speaking without regard for factual accuracyNoisy sincerity<br>Over-explainingFace-saving blather<br>Vague fillerBullshitting<br>TechnobabbleObfuscatory nonsense<br>Confusion tactics
Counterfactual: Asserting information known to be falseHonest mistake<br>Sincere false beliefDeflection<br>Self-deception<br>Rationalized errorReverse accusations<br>Strategic fabricationHard lying<br>Gaslighting<br>Bold falsehoods

Table 1. 4×44 \times4 Behavior Taxonomy: 37 Deception Tactics. Columns are derived from IDT and TDT; rows are from IMT2 and Phl. Social goal refers to whether a player is Cooperative (only if Good), Defensive, Opportunistic (only if Evil), and Adversarial (only if Evil). The tactics are described in Appendix § 11.

Figure 2 shows four tactics in context. The Investigator sees only the utterances and history, not tactic labels. We rate a contextual agent’s skill level λi\lambda_i (i∈Ci \in C) on two scales: GRS and Mach-IV.

Goodness Rationality Spectrum (GRS). GRS is based on theories of cognitive control ([25, 3]). It rates a Good player’s reasoning in terms of logical soundness, Table 1: 4×44 \times4 Behavior Taxonomy: 37 Deception Tactics. Columns are derived from IDT and TDT; rows are from IMT2 and Phl. Social goal refers to whether a player is Cooperative (only if Good), Defensive, Opportunistic (only if Evil), and Adversarial (only if Evil). The tactics are described in Appendix § 11.

transparency, and reasoning under uncertainty as:

  • High (HH): Provides rigorous, evidence-based, and well-founded justifications.

  • Moderate (MM): Generally rational but susceptible to minor logical oversights.

  • Low (LL): Displays fragmented reasoning; may appear confused, erratic, or emotional.

Mach-IV (Machiavellianism). We adapt Christie and Geis’s [9] Mach-IV scale of manipulation to categorize Evil players into three levels:

  • High (HH): Expert manipulator employing subtle deception and strategic planning.

  • Moderate (MM): Competent deceiver using standard tactics with occasional slips.

  • Low (LL): Clumsy deceiver prone to obvious tells like self-contradiction.

The skill level modulates how the tactic is expressed in dialogue. For instance, a “Hard lying” tactic performed by a High Mach-IV Evil player is subtle and believable, whereas the same tactic used by a Low Mach-IV Evil player is easily refuted. Figure 2 demonstrates both scales: G₁ (GRS: Moderate) delivers a precise, evidence-grounded accusation, whereas G₂ (GRS: Low) offers only an uncertain and confused observation. E₁ (Mach-IV: Moderate) produces a measured self-defense, and E₂ (Mach-IV: High) pivots smoothly to overclaiming their fitness to lead next.

Dataset Construction Pipeline

We build AVCORP with a six-stage pipeline (Figure 3) on a modified Avalon (§ 2) in which deception detection is the explicit objective of the Investigator. The pipeline first fixes each game, with balanced Good and Evil roles (§ 4.1) and a constraint-satisfying history (§ 4.2). It then generates and verifies both layers using three LLM generators (Gemini 3.1 Pro, GPT-5.2, GPT-5.4) and two LLM judges (Claude Sonnet 4.5 and 4.6) (Zheng et al., [48]), keeping each utterance and trace consistent with that game state (§ 4.3–§ 4.6). Because each utterance is conditioned on its game state, one of the 37 tactics, and one of three skill levels (§ 3), no two tactic-labeled utterances are identical. Two annotators blind to the ground-truth roles α\alpha validate the judge’s trace selections (§ 4.6.2).

Figure 3. Six-stage pipeline for building AVCORP: Stages 1–2 (far left) construct the game record, Stages 3–4 (center) the tactic-annotated dialogue, and Stages 5–6 (far right) the reasoning traces.

We define each game record as g=(P,α,H)g = (\mathcal{P}, \alpha, \mathcal{H}), where P={p1,…,p5}\mathcal{P} = \{p_1, \ldots, p_5\} is the player set, α:P→{G,E}\alpha: \mathcal{P} \to\{G, E\} assigns exactly two Evil players, and H=H1:R\mathcal{H} = \mathcal{H}_{1:R} is the history of all R≤5R \le5 rounds (§ 2). For each round rr, an annotated sample extends the game record to ξr=(g,Ar,Dr,ρr)\xi_r = (g, \mathcal{A}_r, \mathcal{D}_r, \rho_r). Here, Ar\mathcal{A}_r assigns a tactic τi∈T\tau_i \in\mathcal{T} and a skill level λi\lambda_i to each contextual agent i∈Ci \in\mathcal{C}. Dr\mathcal{D}_r contains the round’s generated utterances, and ρr\rho_r is the Investigator’s reasoning trace. The six stages (Appendix § 12) construct gg (Stages 1–2), AC\mathcal{A}_{C} and Dr\mathcal{D}_{r} (Stages 3–4), and ρr\rho_{r} (Stages 5–6).

victories. Within each block, we cycle through all outcome patterns to ensure representational parity.

Stage 1: Player Role Augmentation

Stage 1 generates hidden role assignments α\alpha for 250 games. To ensure role parity across the (52)=10\binom{5}{2} = 10 possible Evil pairs, we generate 225 assignments (supplementing 25 human-annotated seed R1R1 games) satisfying the following constraints.

Combinatorial balance. To prevent identity bias, each of the 10 Evil pairs appears 25 times (250/10), so each player is Evil in exactly 100 games.

Investigator rotation. We split the corpus into five consecutive 50-game blocks and fix player pkp_k as the Investigator in block kk, so pkp_k never appears in an Evil pair within that block. This diagonal constraint appears as a staircase in Figure 3.

Macro-distribution solver. We satisfy these constraints using a solver that assigns patterns to blocks via weighted random sampling. At each step, it selects a valid Evil pair (one that excludes the fixed-Good Investigator) with a probability proportional to the remaining capacity.

Leader rotation ℓr\ell_r. For the 25 human-annotated seed games, we retain the original ℓ1\ell_1 leadership; for the 225 generated games, we assign ℓ1\ell_1 such that each player leads in exactly 50 of the 250 total games (Figure 9, Appendix § 12.§ 2). Subsequent leadership rotates clockwise.

Round 1 leader distribution across 250 games

Figure 9. Round 1 leader distribution across 250 games, split by game outcome (green = Good victory, red = Evil victory). Each player serves as Round 1 leader in exactly 50 games, confirming the balanced assignment; the outcome split reveals no advantage from leading first.

Team selection TrT_r. We compose each team using a role-aware selection rule operating on α\alpha and a per-player suspicion score σp(r)\sigma_p^{(r)} (initialized to zero). On a PASS round, a Good leader chooses the least suspicious players. On a FAIL round, they must include at least one Evil player to deliberately test a suspect. An Evil leader avoids any player whose suspicion exceeds the current group average by more than 1.0 to preserve cover. We update σp(r)\sigma_p^{(r)} after each round as σp(r+1)=σp(r)+Δp(r)\sigma_p^{(r+1)} = \sigma_p^{(r)} + \Delta_p^{(r)} where:

Δp(r)={+1.0p∈Tr, Qr=FAIL−0.3p∈Tr, Qr=PASS0otherwise.\Delta_p^{(r)} = \begin{cases} +1.0 & p \in T_r,\ Q_r = \mathrm{FAIL} \\ -0.3 & p \in T_r,\ Q_r = \mathrm{PASS} \\ 0 & \text{otherwise.} \end{cases}

Stage 2: Public History Generation

Stage 2 generates the public history H\mathcal{H} for 250 games (1,000 rounds), mapping roles to proposals, votes, and outcomes. We first assign quest outcomes, then sample leaders, role-consistent teams, and votes. Pairing fixed outcomes with varying roles yields varied game states for Stage 3.

Outcome sequences QrQ_r. We assign each game one of 20 predefined outcome sequences (Table 4, Appendix § 12.§ 2). We balance the corpus to include 100 games ending at Round 3, 50 at Round 4, and 100 at Round 5, resulting in 135 Evil and 115 Good victories.

Terminal roundGood winsEvil wins
R3P–P–PF–F–F
R4P–P–F–PF–F–P–F
R4P–F–P–PF–P–F–F
R4F–P–P–PP–F–F–F
R5F–F–P–P–PF–F–P–P–F
R5F–P–F–P–PF–P–F–P–F
R5F–P–P–F–PF–P–P–F–F
R5P–F–F–P–PP–F–F–P–F
R5P–F–P–F–PP–F–P–F–F
R5P–P–F–F–PP–P–F–F–F
Total1010

Table 4. The 20 predefined outcome sequences grouped by terminal round and winning team. P = quest pass (Good wins the quest); F = quest fail (Evil sabotages). Each sequence terminates on the round delivering the third accumulated win. Within R5, each row is a matched pair sharing identical first-four outcomes—only the fifth quest determines the winner.

Voting VrV_r. For each nonmember p∉Trp \notin T_r, let πp=Pr⁡[vp=N]\pi_p = \Pr[v_p = N] denote the No-vote probability:

πp={0.6α(p)=G, ∑q∈Trσq(r)>2.00.4α(p)=G, ∃q∈Tr:σq(r)>1.50.2α(p)=E, Tr∩α−1(E)=∅0otherwise.\pi_p = \begin{cases} 0.6 & \alpha(p) = G,\ \sum_{q \in T_r} \sigma_q^{(r)} > 2.0 \\ 0.4 & \alpha(p) = G,\ \exists q \in T_r : \sigma_q^{(r)} > 1.5 \\ 0.2 & \alpha(p) = E,\ T_r \cap\alpha^{-1}(E) = \varnothing\\ 0 & \text{otherwise.} \end{cases}

If ∣{p:vp=N}∣≥3|\{p : v_p = N\}| \geq3, we convert the minimum No votes to Yes to restore a majority (see Figure 11 in Appendix § 12.§ 2 for the distribution).

Bar chart showing voting pattern distribution across 1,000 rounds

Figure 11. Voting pattern distribution across 1,000 rounds. Most proposals pass unanimously (65.3 %); a single dissenting vote occurs in 32.9 % of rounds, and the minimum two-No threshold is reached in only 1.8 %.

Verification. We validate Stage 2 against eight hard constraints (e.g., clockwise leader rotation, valid team sizes, and correct game termination). We observed zero violations across all 250 games.

Stage 3: Tactic and Dialogue Generation

Stage 3 produces tactic annotations Ac\mathcal{A}_c and discussion Dr\mathcal{D}_r for the 975 samples across R1–R5, yielding 3,900 contextual-agent utterances. The 25 seed games retain their original human annotations.

Tactic annotation. We define six tactic pools, combining alignment and skill levels. In Round 1, we cycle through the pools to ensure balanced coverage. High-skill agents are restricted to alignment-congruent tactics (Cooperative for Good; Opportunistic/Adversarial for Evil), whereas Moderate- and Low-skill agents may also draw from the 9 shared Defensive tactics. From Round 2 onward, GPT-5.2 evolves each agent’s tactic from its prior-round assignment, adapting to game-state variables while preserving continuity. Evil agents recalibrate based on exposure risk, opportunities for blame, and team composition; Good agents adjust based on accumulated evidence and shifting suspicions. Each agent’s skill level stays fixed across rounds. Full tactic distributions appear in Appendix § 12.§ 3.

Dialogue generation. We generate two candidates (GPT-5.2 and Gemini 3.1 Pro) per sample, conditioned on the game record gg, assigned tactics Ac\mathcal{A}_c, cumulative history H1:r\mathcal{H}_{1:r}, preceding verbatim discussion Dr−1\mathcal{D}_{r-1}, and LLM-generated summaries of earlier rounds D^1:r−2\hat{\mathcal{D}}_{1:r-2} (Prompts in Appendix § 12.§ 3).

Validation. We validate each candidate against an eight-point checklist covering format compliance, speaker ordering, tactic-dialogue alignment, and history grounding. All 975 samples satisfy every constraint for both models.

Stage 4: Tactic and Dialogue Verification

To ensure quality, we employ a two-layer LLM-as-a-judge pipeline, using Claude Sonnet 4.5 as an evaluator blind to model identity.

Scoring criteria and decision logic. The judge scores candidates on five binary criteria: Coherence and Situational Fit, History Alignment, Tactic-Dialogue Alignment, Skill Consistency, and Format Compliance. We use a layered decision logic:

  • Layer 1: If both candidates score 5/5, pairwise comparison determines the winner. If only one scores 5/5, it is selected. If both score 4/5, the judge selects the stronger response and applies a targeted correction to the failing criterion. Scores ≤3\leq3 trigger full custom regeneration.

  • Layer 2: All corrected or regenerated outputs undergo a second-pass recheck (max three attempts), after which unverified outputs are flagged as NEEDS_HUMAN for manual inspection.

Human-in-the-loop. After Stage 4 verification, we manually review at least 50 sampled dialogues drawn from corrected and regenerated outputs. We additionally review all samples flagged as NEEDS_HUMAN. Based on these reviews, we iteratively refine the generation prompts and verification criteria for Stage 3 and Stage 4.

Verification results. Table 6 in Appendix § 12.§ 4 reports per-round outcomes. GPT-5.2 is selected (directly or after targeted correction) in 84–96% of samples across rounds, reflecting a quality advantage over Gemini 3.1 Pro (Figure 16a). Pairwise tiebreakers resolve 51–66% of cases in R2–R5. R1 requires the most correction (18.2% targeted), because no earlier round constrains the opening dialogue. Custom regeneration remains below 1% across all rounds. Per-criterion pass rates and prompt configurations appear in Appendix § 12.§ 4.

RoundNAˉGPT\bar{A}_{\mathrm{GPT}}AˉGEM\bar{A}_{\mathrm{GEM}}GPTGEMTGPTT_{\mathrm{GPT}}TGEMT_{\mathrm{GEM}}CustomPairwiseL2
R12254.623.1970.2210.6713.784.440.893.56✓
R22504.964.5393.603.602.000.400.4066.00✓
R32504.974.3794.403.201.600.800.0058.80✓
R41504.944.3993.334.670.670.670.6756.67✓
R51004.914.3088.006.005.001.000.0051.00✓

Table 6. Per-round verification outcomes by Claude Sonnet 4.5. AˉGPT\bar{A}_{\mathrm{GPT}} and AˉGEM\bar{A}_{\mathrm{GEM}} are mean judge scores out of 5 for GPT-5.2 and Gemini 3.1 Pro. GPT and GEM indicate direct-selection rates; TGPTT_{\mathrm{GPT}} and TGEMT_{\mathrm{GEM}} denote targeted-correction rates; Custom denotes full custom-regeneration rate. The five columns partition all samples (%) and sum to 100. Pair is the pairwise-tiebreak rate; L2 marks rounds that successfully underwent a second-layer recheck.

Figure 16. (a) Mean judge score Aˉ\bar{A} with ±1\pm1 SD error bars. (b) C1: Coherence & Tactic Situational Fit. (c) C2: Public History Alignment. (d) C3: Tactic-Dialogue Alignment. (e) C4: Authenticity & Skill Consistency. (f) C5: Format Compliance (both models 100% every round). Figure 16: Per-round score trajectories across R1–R5. Panel (a) shows mean judge scores (Aˉ\bar{A}) out of 5 with ±1\pm1 SD bands; panels (b)–(f) show per-criterion pass rates (%). GPT-5.2 shown in blue; Gemini 3.1 Pro in red.

Stage 5: Reasoning Trace Augmentation

Stage 5 generates the Investigator’s reasoning trace ρr\rho_r for all 1,000 verified samples. We run two prompting frameworks via GPT-5.4, ReAct (Yao et al., [47]) and ReCon (Wang et al., [40]), each producing a candidate trace.

Reasoning frameworks. We condition models on the observable game state (H1:r,D1:r)(\mathcal{H}_{1:r}, \mathcal{D}_{1:r}) and prior traces ρ1:r−1\rho_{1:r-1}, withholding ground-truth roles α\alpha so that no trace sees the answer. Our ReAct implementation interleaves two Thought–Action–Observation cycles (first isolating diagnostic history events, then cross-checking them against specific utterances) before a Synthesis step constructs ρr\rho_r. Our ReCon implementation uses two-pass recursive contemplation: a Formulation pass infers first-order alignments, and a Refinement pass models second-order influence attribution with checks for evidence sufficiency and cross-round consistency. Appendix § 12.§ 5 details the prompt structures.

Output schema constraints. Both frameworks output the six-field schema defined in § 2. To ensure logical soundness, generations must satisfy the Definitive Deduction Rule: if the Investigator participated in a two-person failed quest, they must definitively identify the partner pxp_x as Evil (assigning spx=Extremes_{p_x} = \mathrm{Extreme} and ensuring px∈α^p_x \in\hat{\alpha}).

Stage 6: Reasoning Trace Verification

Stage 6 mirrors Stage 4 with criteria for reasoning traces, selecting the higher-quality trace of the two Stage 5 candidates. We use Claude Sonnet 4.6 as the judge, which receives Trace A (ReAct) and Trace B (ReCon), with identities withheld.

Scoring criteria. The judge scores each trace on four binary criteria: (1) Abduction Plausibility, (2) Suspicion Level Consistency, (3) Gated Depth Reasoning, and (4) Statement and Deduction Coherence. Criterion 1 checks that abductive hypotheses hih_i cite specific events. Criterion 2 ensures suspicion levels sis_i match cumulative evidence and enforces the Definitive Deduction Rule. Criterion 3 verifies first-order beliefs β1i\beta^i_1 exist for all players, restricting second-order beliefs β2i\beta^i_2 to Strong or Extreme suspects. Criterion 4 confirms the utterance u^\hat{u} is evidence-grounded and the deduction α^\hat{\alpha} includes all high-suspicion players.

Context and decision logic. The judge evaluates trace pairs using history H1:r\mathcal{H}_{1:r}, discussions D1:r\mathcal{D}_{1:r} with summaries D^1:r\hat{\mathcal{D}}_{1:r}, prior traces ρ1:r−1\rho_{1:r-1}, and roles α\alpha. We apply a two-layer logic: in Layer 1, if both traces score 4/4, a pairwise comparison selects the one closer to α\alpha. If only one scores 4/4, it is selected directly. If both score below 4/4, a targeted correction is applied to the stronger base trace. In Layer 2, corrected traces undergo up to three rechecks before being flagged.

Human-in-the-loop. We follow the same pipeline of manual review and prompt iteration as Stage 4.

Verification results. ReAct traces score higher on the criteria than ReCon traces (mean 3.9 against 3.8 of 4; Table 8, Appendix § 12.§ 6), but the judge selects ReCon in 63.9% of pairs. The reason is that 79.1% of pairs tie at 4/4 (Figure 17a), and the pairwise tiebreaker rewards qualitative depth. Targeted correction applies to only 22 traces (2.2%), and all of them resolve within Layer 2.

FrameworkC1C2C3C44/4MeanSD3/42/4≤1/4DirectCorr.Total
ReAct97.694.3100.099.392.53.90.36.21.30.033.90.634.5
ReCon93.789.199.998.284.43.80.512.23.30.163.91.665.5
Δ+3.9+5.2+0.1+1.1+8.1+0.10−0.15−6.0−2.0−0.1−30.0−1.0−31.0
Agreement (%):92.585.899.998.1—————————
Pair outcome (%):——————————79.118.72.2

Table 8. Comprehensive Stage 6 verification statistics for ReAct and ReCon traces (NN=1,000). Pass rates: binary pass rate per criterion (%). Score: mean total score with SD; distribution of 3/4, 2/4, and ≤1/4 outcomes (%). Selection: rate of direct selection, corrected-trace selection (ReAct-Claude or ReCon-Claude), and combined total (%). Δ row shows ReAct minus ReCon. The pairwise statistics row reports per-criterion agreement (%) under C1–C4, and pair outcome fractions aligned under the selection columns: both traces score 4/4 (resolved by pairwise tiebreaker), exactly one scores 4/4 (direct selection), or both score below 4/4 (targeted correction).

Figure 17. Stage 6 reasoning framework (ReAct, ReCon) selection rates by Claude Sonnet 4.6 and passing rates of their criteria along with agreements.

Deduction Accuracy of Traces

We score each trace’s deductions α^\hat{\alpha} against the withheld roles α\alpha at the round each game ends (R3, R4, or R5). The traces correctly identify both Evil players in 233 of 250 games (93.2%, Table 10 in Appendix § 16), against 16.7% (1/(42)1/\binom{4}{2}) for naming two of the four candidates at random. Quest outcomes alone single out the Evil pair in only 12 of these games, so the dialogue provides most of the evidence for the verdict (Table 12).

EndsnExact (%)PRF1
R310093.00.9600.9600.960
R45086.00.9300.9300.930
R510097.00.9800.9800.980
All25093.20.9620.9620.962

Table 10. Gold-trace deduction accuracy at game-ending round. Exact indicates the percentage of games that correctly identify both Evil players and no one else.

Allowed pairsGamesLogic only (%)ExpectedGold
112100.012.0012
23750.018.5037
36633.322.0063
45625.014.0053
53520.07.0029
64416.77.3339
2 to 623828.968.83221

Table 12. Games grouped by the number of Evil pairs that their quest outcomes allow. Logic only is the expected exact match of a random choice among the allowed pairs, and Expected is the number of games that this choice resolves in expectation; Gold counts the games that the gold traces resolve.

Impact of the Definitive Deduction Rule. The rule applies in 85 out of 250 games (277 traces). The traces resolve 84 of these games and 149 of the 165 games in which the rule never applies (90.3%). Impact of label priors. Each of the 10 Evil pairs occurs in exactly 25 games (§ 4.1), so the label favors no pair. Weighting the games that Evil wins (133 of 135 resolved) and that Good wins (100 of 115) equally lowers the verdict only from 93.2% to 92.7%.

Inter-Annotator and Judge Agreement

Two annotators review 100 trace pairs (79 tiebreak vs 21 clean-win) from 25 games sampled at random and stratified by ending round: 10 for R3, 5 for R4, and 10 for R5. Blind to the framework, the judge’s selection, and ground-truth roles α\alpha, they score each trace on the four Stage 6 criteria and select the better one (Appendix § 17). Each framework appears as Trace A in 50 items and as Trace B in the other 50 in random order. Thus, a preference for the first trace shown cannot favor either framework.

The annotators select the same trace on 97 items (κ=.94\kappa= .94, 95% CI [.86, 1.00]), and the judge agrees with them on 96 (κ=.92\kappa= .92 [.83, .98]) and 97 items (κ=.94\kappa= .94 [.86, 1.00]; Table 13), all within the almost perfect agreement range (κ≥.81\kappa\ge.81) (Landis and Koch, [18]). On the 79 tiebreak items, where the judge used α\alpha, it agrees with the annotators on 77 and 78 items and matches all 76 selections they share. On the 21 clean-win items, both annotators select ReCon over the judge’s ReAct on the same two items, in each of which the judge failed the ReCon trace on a criterion that both annotators passed. Table 13 and Table 15 in Appendix § 17 report the agreement with PABAK (Byrt et al., [5]).

PairAll (100)Tiebreak (79)Clean win (21)
Agreeκ\kappaPABAKAgreeκ\kappaPABAKAgreeκ\kappaPABAK
A1 vs. A297/1000.9370.94076/790.9180.92421/211.0001.000
A1 vs. J96/1000.9170.92077/790.9460.94919/210.8090.810
A2 vs. J97/1000.9370.94078/790.9730.97519/210.8090.810
J vs. Both95/970.9570.95976/761.0001.00019/210.8090.810

Table 13. Agreement on the final trace selection. A1 and A2 are the annotators and J the Stage 6 judge. Agree counts items on which the pair selects the same trace; Both is the judge against the annotators’ shared choice, over the items on which they agree. Tiebreak items were decided by the judge using α\alpha; clean-win items on criterion scores alone. PABAK is 2Po−12P_o - 1, where PoP_o is the share of items on which the pair agrees.

CriterionPairPassAgreeκ\kappaPABAK
C1 AbductionA1 vs. A20.980, 0.9751990.8860.990
C1 AbductionA1 vs. Judge0.980, 0.9651970.7200.970
C1 AbductionA2 vs. Judge0.975, 0.9651960.6570.960
C2 SuspicionA1 vs. A20.940, 0.9351990.9570.990
C2 SuspicionA1 vs. Judge0.940, 0.9101940.7840.940
C2 SuspicionA2 vs. Judge0.935, 0.9101950.8260.950
C3 DepthA1 vs. A21.000, 1.000200undefined1.000
C3 DepthA1 vs. Judge1.000, 1.000200undefined1.000
C3 DepthA2 vs. Judge1.000, 1.000200undefined1.000
C4 CoherenceA1 vs. A21.000, 1.000200undefined1.000
C4 CoherenceA1 vs. Judge1.000, 0.9951990.0000.990
C4 CoherenceA2 vs. Judge1.000, 0.9951990.0000.990

Table 15. Per-criterion agreement, Trace A and Trace B pooled (n=200n = 200 per row). Pass gives each rater’s pass rate. κ\kappa is undefined when both raters pass every trace.

Trained Detector

We finetune Qwen3-8B (Qwen Team, [30]) with LoRA (Hu et al., [15]), using supervised fine-tuning (SFT) followed by Group Relative Policy Optimization (GRPO) (Shao et al., [36]). We split the corpus at the game level, stratified by ending round, into 225 training games (900 samples) and 25 held-out games (100 samples). Each prompt includes the history Hr−2:r\mathcal{H}_{r-2:r}, the discussion Dr\mathcal{D}_r, and the Investigator’s prior traces ρr−2:r−1\rho_{r-2:r-1}. SFT runs one epoch on the gold labels, and GRPO then samples 16 completions per prompt for 450 steps. A full run takes 25.4 to 34.8 GPU hours on an NVIDIA H100 PCIe GPU. Appendix § 18.§ 1 lists all parameters, token caps, and GPU hours.

Variants and rewards. We use two variants for training that differ in their output labels. The *trace variant learns ρr\rho_r alone. The tactic variant also learns each contextual agent’s tactic. The original set scores (1) schema validity, (2) abduction completeness, (3) suspicion consistency, (4) gated second-order depth, and (5) deduction F1; (3) and (5) use α\alpha. The revised set replaces (2) with abduction accuracy and (3) with suspicion calibration, which checks that the Evil players receive higher suspicion than the Good players, and extends (1) to check field content; three of them use α\alpha. The tactic variant adds a sixth reward for tactic accuracy, and Appendix § 18.§ 2 defines every reward. We train each trace configuration three times and each tactic configuration once, for eight runs.

Evaluation. We score the deduction at the round each game ends as an exact match and as F1, and each detector conditions on its own earlier traces (Appendix § 18.8). The untrained baseline is Qwen3-8B with its adapters off, prompted as each variant. The gold traces resolve 23 of the 25 held-out games (92.0%). We test transfer on the 20 Avalon-NLU games of Stepputtis et al. [38] (82 rounds; Appendix § 18.§ 3), played by six humans as two Loyal Servants, Merlin, Percival, Morgana, and the Assassin. The Investigator is the first Loyal Servant, and we evaluate two prompts (Appendix § 18.§ 6).

Results. On AVCORP, every trained configuration exceeds its untrained baseline in F1, and all except two tactic configurations, which tie at 48.0%, also exceed it in exact match (Table 2). The tactic variant with the revised rewards performs best: it resolves 18 of the 25 games, as many as Claude Opus 5.5 and more than GPT-5.6 Sol (16), Gemini 3.1 Pro (14), and DeepSeek-V4-Pro (12) (Appendix § 18.§ 5). On Avalon-NLU, every trained detector exceeds the untrained model in round F1, mainly through higher recall (Appendix § 18.§ 4). Training raises game-ending F1 in seven of the eight configurations, but the exact match ranges from 20.0% to 46.7%, against 40.0% (trace prompt) and 20.0% (tactic prompt) without training. Training on AVCORP thus improves inference round by round on human games, whereas the final verdict depends on the training method and the prompt. Future work includes testing cross-setting transfer (e.g., Werewolf) and auditing the detection abilities of existing safety-trained models.

DetectorStageAVCORP (4 candidates) Exact (%)F1Avalon-NLU (5 candidates) Exact (%)F1Round F1
Qwen3-8B, trace promptadapters off24.00.5840.00.670.46
Qwen3-8B, tactic promptadapters off48.00.7220.00.550.43
Trace, original rewardsSFT36.0 (24–56)0.70 (0.64–0.81)46.7 (40–55)0.71 (0.67–0.76)0.58 (0.55–0.60)
Trace, original rewardsSFT + GRPO33.3 (24–44)0.78 (0.73–0.82)30.0 (25–35)0.65 (0.64–0.67)0.57 (0.53–0.60)
Trace, revised rewardsSFT38.7 (36–44)0.69 (0.65–0.74)40.0 (30–45)0.70 (0.66–0.72)0.56 (0.53–0.59)
Trace, revised rewardsSFT + GRPO64.0 (60–68)0.86 (0.84–0.88)40.0 (30–50)0.72 (0.68–0.77)0.60 (0.57–0.62)
Tactic, original rewardsSFT52.00.7535.00.640.53
Tactic, original rewardsSFT + GRPO48.00.8520.00.650.55
Tactic, revised rewardsSFT48.00.7445.00.660.54
Tactic, revised rewardsSFT + GRPO72.00.8835.00.700.58

Table 2. Detector performance across domains. Exact: identifies both Evil players exclusively; F1: game-ending deduction; Round F1: all 82 Avalon-NLU rounds. Trace-only rows: mean of three runs (ranges in parentheses).

Related Work

Social deduction games enforce information asymmetry, making them natural testbeds for studying deception. Avalon studies evaluate LLM agents ([22]), annotate persuasion and deception strategies in human dialogue ((Stepputtis et al., 2023)), capture human reasoning traces ([20]), and raise win rates through recursive contemplation ([40]) or tree search ([21]). Werewolf studies examine role-concealment communication ([46]), multimodal persuasion ([17]), reasoning modules ([44]), and deception benchmarking ([1]). These resources provide strategy labels or reasoning traces, but they do not pair tactic labels of utterances with ToM reasoning traces for training detectors.

Deception emerges in AI systems as a strategy to satisfy goals ([28]), and LLMs lack robust defenses against manipulative contextual cues ([16]). Sarkadi et al. [34] combine IMT2 and IDT with ToM in a formal model of deceptive agents, but provide no corpus. AVCORP builds on these accounts with a tactic taxonomy and nested ToM traces designed to train detectors.

Ethical Considerations

We design AVCORP to facilitate deception detection. The taxonomy uses published deception theory (IMT2, IDT, TDT, and philosophical accounts of lying). Releasing the taxonomy adds no new capability to deceive. Every reasoning trace takes the Investigator’s perspective: it models suspicion and updates beliefs about the other players. No trace takes an Evil player’s perspective or plans a deception. Therefore, the traces do not teach how to deceive. The tactic-labeled utterances do show deceptive moves in context, and a model could learn to imitate them. We release them with the detection task and the taxonomy so that safety researchers can audit which patterns a detector must cover.

Limitations

We identify limitations that concern the corpus design, detector training, evaluation scope, and verification judges, and describe a resolution for each.

Distributional artifacts. GPT-5.2 selects tactics for Rounds 2–5 based on the game context. This selection skews the tactic frequencies (Table 5 in Appendix § 12.§ 3). The most frequent tactic, Evidence sharing, labels 17.5% of the utterances. Further, AVCORP fixes quest outcomes to balance Evil and Good wins (54% and 46%). A detector must exceed these two majority baselines to show that it learns. Additionally, the corpus contains no rejected proposals, which can provide evidence about player roles. Future versions can use frequency-aware sampling and allow rejected proposals to reduce these artifacts.

Information Commitment \ Social Goal(1) Cooperative(2) Defensive(3) Opportunistic(4) Adversarial
(A) TransparentEvidence sharing (701)<br>Rational justification (442)<br>Perspective-taking (109)Defensive disclosure (73)<br>Self-justification (64)Weaponized candor (87)<br>Overstated contribution (35)Truthful misdirection (188)<br>Misleading emphasis (205)
(B) FramingBayesian hedging (239)<br>Strategic uncertainty (47)<br>Pragmatic silence (30)Omission to save face (26)<br>Half-truths (48)Cherry-picking (211)<br>False consensus (198)<br>Misframing (337)Withholding info (WIFOM) (59)<br>Strategic omission (101)
(C) Careless-to-truthNoisy sincerity (57)<br>Over-explaining (146)Face-saving blather (46)<br>Vague filler (32)Bullshitting (21)<br>Technobabble (17)Obfuscatory nonsense (11)<br>Confusion tactics (18)
(D) CounterfactualHonest mistake (16)<br>Sincere false belief (27)Deflection (244)<br>Self-deception (21)<br>Rationalized error (22)Reverse accusations (47)<br>Strategic fabrication (19)Hard lying (22)<br>Gaslighting (16)<br>Bold falsehoods (18)

Table 5. Per-tactic assignment counts (n=4,000n = 4{,}000). Each entry shows the tactic name and its total count across all 1,000 samples. Column totals confirm the alignment-category breakdown reported in Figure 12: Cooperative (Good-exclusive) = 1,814; Defensive (Shared) = 576; Opportunistic + Adversarial (Evil-exclusive) = 1,610. The total of 4,000 is verified.

Synthetic dialogue realism. An LLM generates every utterance, and the dialogue may lack the hesitations, topic shifts, and emotional cues of human deception. Researchers can replace the synthetic text with crowdsourced human utterances based on the existing tactic assignments. They must then regenerate the reasoning traces (Stages 5 and 6) to analyze the new statements.

Scale and language scope. The current corpus contains 5,000 utterances from an English language game. The six-stage pipeline directly supports scaling to more games, but token cost limited its size. We release the pipeline and tactic taxonomy so that the community can scale the corpus to more games and languages (Chinese Avalon gameplay (Li et al., [20])), including settings such as Werewolf and Among Us (Sarkar et al., [35]), where the same deception taxonomy applies.

Untested transfer beyond the game. We train and test the detector only on Avalon using synthetic gameplay (AVCORP) and human players (Avalon-NLU). We have not evaluated transfer to deception outside the game. For example, DeceptionBench (Huang et al., [16]) tests deception in domains like healthcare and the economy. Evaluating on such benchmarks requires a new task format because their data lacks hidden roles and our tactic labels.

Training methods. GRPO does not consistently improve on SFT. With the original rewards, GRPO underperforms SFT for exact matches on AVCORP. The exact match rate drops from 36.0% to 33.3% for the trace variant and 52.0% to 48.0% for the tactic variant. On Avalon-NLU, SFT yields more exact matches than GRPO on average. The deduction F1 reward (Appendix § 18.§ 2) gives partial credit to a deduction that names extra players. With the original rewards, GRPO names more players per round than SFT on Avalon-NLU (Table 20 in Appendix § 18.§ 4). In contrast, SFT imitates the gold traces, whose game-ending deductions always name two players. A reward that scores the deduction size might close this gap, but we have not evaluated this approach.

DetectorStageR1R2R3R4R5PRF1NamedTwo
Qwen3-8B, trace promptadapters off0.1630.2590.6110.6230.6670.6250.3660.4621.1719
Qwen3-8B, tactic promptadapters off0.1750.3380.6100.4690.5830.4920.3840.4321.5620
Trace, original rewardsSFT0.5250.5120.6060.6660.6770.5550.6060.5792.1818.7
SFT + GRPO0.5080.5420.6110.6260.6280.4880.6910.5722.8315.7
Trace, revised rewardsSFT0.4890.4790.6020.6460.7270.5410.5830.5612.1518.3
SFT + GRPO0.5240.5600.5840.6890.7450.5800.6160.5972.1317.7
Tactic, original rewardsSFT0.4710.5050.5000.6300.6000.4590.6160.5262.6815
SFT + GRPO0.4330.5310.5790.6210.6210.4490.6950.5453.1010
Tactic, revised rewardsSFT0.4790.5000.5580.6180.5830.4850.5980.5362.4618
SFT + GRPO0.4600.5120.6520.6940.6430.5300.6400.5802.4115

Table 20. Round F1 on the 20 Avalon-NLU games with the domain-context prompt (R1 to R3: 20 rounds each; R4: 16; R5: 6), with precision, recall, and F1 over all 82 rounds, the mean number of players named per round, and the number of games, of 20, whose game-ending deduction names exactly two players (Two). Every model conditions on its own earlier traces. Trace-only rows give the mean over three runs.

Gold prior traces during training. During training, each prompt includes the gold traces from the two preceding rounds, whereas during evaluation, the detector receives its own earlier traces. Gold traces contain previous gold deductions. Copying the previous gold deduction alone resolves 18 of the 25 held-out games (Appendix § 18.8). The detector can therefore learn to depend on information that it does not receive during evaluation, and an error in an early round can cascade into later rounds. To fix this, future training can build prompts from the detector’s own earlier traces or scheduled sampling (Mihaylova and Martins, [24]) can progressively replace gold traces with generated traces.

Verification judges. Both verification stages use Claude Sonnet 4.5 and 4.6. The Stage 6 judge always receives the ReAct trace as Trace A. We did not swap the trace order or evaluate other model families. Additionally, rerunning the pipeline with future models may yield different outcomes. Nevertheless, the blind human study (§ 4.6.2) balances the trace order and agrees with the judge (κ\kappa from .92 to .94). We publish all prompts and scripts to enable the community to recalibrate with successor models.

Appendix

Avcorp Design

Table 3 records the quest team size for each round in Avalon, which determines how many players participate in the quest phase.

Behavior Matrix Definitions

The behavior matrix BB organizes all 37 tactics across two axes: information commitment strategy and social goal orientation. The axes are defined below, followed by full definitions of each tactic. Alignment accessibility is noted as Good, Evil, or Both.

Information commitment strategy is derived from IMT2 and PhI, this axis distinguishes how utterances relate to the known truth:

  • Transparent: Openly sharing clear, evidence-based information.

  • Framing: Deliberate shaping of discourse through omission or emphasis.

  • Careless-to-truth: Assertions made without regard for factual accuracy.

  • Counterfactual: Direct assertions of information contrary to reality.

Social goal orientation is derived from IDT and TDT, this axis characterizes the social goal an utterance serves:

  • Cooperative: Acting in harmony with group goals and mutual benefit.

  • Defensive: Protecting one’s own image, reputation, or position.

  • Opportunistic: Exploiting situations for personal gain at the expense of others.

  • Adversarial: Seeking to undermine, harm, or sabotage others.

(A) Transparent

Cooperative: Good.

  • Evidence sharing: Providing concrete facts to support a claim.

  • Rational justification: Explaining actions through logical reasons.

  • Perspective-taking: Acknowledging another’s viewpoint to build trust.

Defensive: Both.

  • Defensive disclosure: Revealing only what is necessary to protect oneself.

  • Self-justification: Excusing one’s own actions to avoid blame.

Opportunistic: Evil.

  • Weaponized candor: Using blunt truth strategically to gain advantage over others.

  • Overstated contribution: Exaggerating one’s effort to appear indispensable.

Adversarial: Evil.

  • Truthful misdirection: Offering accurate details that nonetheless distract from the relevant truth.

  • Misleading emphasis: Highlighting true but minor details to make them appear central.

(B) Framing

Cooperative: Good.

  • Bayesian hedging: Presenting probabilities or uncertainty to remain flexible.

  • Strategic uncertainty: Deliberately leaving outcomes ambiguous to avoid overcommitment.

  • Pragmatic silence: Omitting information instead of lying outright.

Defensive: Both.

  • Omission to save face: Withholding details to avoid personal embarrassment.

  • Half-truths: Mixing true information with key omissions to create a false impression.

Opportunistic: Evil.

  • Cherry-picking: Selectively presenting only favorable evidence.

  • False consensus: Overstating agreement among others to pressure a target.

  • Misframing: Presenting accurate information within a skewed or misleading context.

Adversarial: Evil.

  • Withholding info: Creating strategic doubt by hiding critical details.

  • Strategic omission: Structuring a true account so that omitted facts lead to a false conclusion.

Careless-to-truth

Cooperative: Good.

  • Noisy sincerity: Being honest in a confusing or informationally overloaded way.

  • Over-explaining: Adding excessive detail in an attempt to appear genuine.

Defensive: Both.

  • Face-saving blather: Using empty talk to avoid confrontation or accountability.

  • Vague filler: Relying on imprecise or generic phrases that commit to nothing.

Opportunistic: Evil.

  • Bullshitting: Speaking without concern for truth or falsity, purely to produce an effect.

  • Technobabble: Using jargon or complexity to appear credible without substantive content.

Adversarial: Evil.

  • Obfuscatory nonsense: Deploying intentionally incoherent speech to mislead or exhaust.

  • Confusion tactic: Disrupting others through pivots or interruptions rather than incoherent content.

Counterfactual

Cooperative: Good.

  • Honest mistake: Asserting something false in genuine good faith.

  • Sincere false belief: Expressing a persistently internalized false belief, not a momentary slip.

Defensive: Both.

  • Deflection: Shifting blame or suspicion onto another player by redirecting attention, often through partial truths or emotional appeals.

  • Self-deception: Convincing oneself of a falsehood and acting on it sincerely.

  • Rationalized error: Justifying a factual mistake with faulty or motivated logic.

Opportunistic: Evil.

  • Reverse accusations: Shifting blame back onto the accuser to neutralize suspicion.

  • Strategic fabrication: Inventing plausible details to achieve a deceptive goal.

Adversarial: Evil.

  • Hard lying: Deliberately asserting the opposite of the known truth.

  • Gaslighting: Causing others to doubt their own memory, perception, or reasoning.

  • Bold falsehoods: Asserting lies so brazenly that delivery alone overwhelms scrutiny, regardless of subtlety.

Six Stages

Figure 4. Stage 1: Role Assignment. Good/Evil assignment across 250 games. Constraints: 25 human-annotated seed games; Investigator rotation; 10 unique evil role pairs. Macro-distribution solver: Balanced role assignment. 3 Constraints. Macro solver. Balanced: 3G / 2E. Stage 2: Public History. Team compositions, votes, quest outcomes. Predefined quest-end sequences per game. 0 hard-constraint violations. 115 Win · 135 Loss across 250 games. 250 games. 5 rounds. 0 violations. Constraint Satisfaction Problems. Stage 1: Balanced Role Assignment. P1 P2 P3 P4 P5. G E G G E. C(5,2)=10C(5,2)=10 unique combos × 25 games · 150 G/player 100 E/player. Stage 2: Public History Generation. R3: 100 R4: 50 R5: 100 games each. 115 Win · 135 Loss 250 total. Example public history (Game 60): Rnd Leader Team Votes Quest; R1 P3 P1, P3 5Y 0N PASS; R2 P2 P2, P4, P5 3Y 2N FAIL; R3 P1 P1, P3, P4 4Y 1N PASS. Stages 1 and 2 facilitate in construction of gg.

Figure 5. Stage 3: Tactic & Dialogue Generation. Two-pass pipeline per round (R1–R5). Pass 1: Tactic precomputation: R1: Cyclic rotation based on alignment (G, E) × skill levels (H, M, L); R2–R5: Contextual precomputation via GPT-5.2. Pass 2: GPT-5.2 & Gemini-3.1 in parallel. R1: Prior context: (gg, ACA_C). R2–R5: Prior context: (ACA_C, H1:rH_{1:r}, D^1:r−2\hat{D}_{1:r-2}, Dr−1D_{r-1}). 37 tactics. 2 candidates / round. GPT-5.2. Gemini-3.1. Stage 4: Tactic & Dialogue Verification. LLM-as-Judge: Claude Sonnet 4.5. Verification in Two Layers. L1: Blind A/B verification. L1: 5-criterion binary scoring. L1: Targeted & custom corrections. L2: Rechecks targeted & custom corrections for a max of 3 attempts. L2: NEEDS_HUMAN after 3 failed attempts. Human-in-the-loop (HITL). Blind A/B. 2 layer verification. 5-criteria. Claude sonnet 4.5. Stages 3 & 4: Tactic and Dialogue Annotation (R1–R5). Stage 3: Tactic and Dialogue Generation. Seed Context. Pass 1 Precompute Tactics. Pass 2 (parallel). GPT-5.2 → A. Gemini-3.1 → B. CSV A. CSV B. Stage 4: Blind Verification & Correction. 1st Layer Verification: Blind A / B. Claude Sonnet 4.5 Scorer. Decision. 5/5 · 4/5 · ≤3. Verified. 2nd Layer Correction loop → max 3 attempts: Att. 1 → Att. 2 → Att. 3 → NEEDS_HUMAN. 5 scoring criteria: Coherence & Tactic Fit; History Alignment; Tactic–Dialogue; Authenticity & Skill; Format. Human-in-the-Loop. Verifier-flagged (NEEDS_HUMAN) · Random spot-check · Prompt calibration. Stages 3 and 4 facilitate in construction of ACA_C and DD.

Figure 6. Stages 5 and 6 facilitate in constructing ρ\rho.

Stage 1

Figure 7a. Role assignment heatmap (5 players × 250 games). The Good assignment staircase reflects the diagonal block constraint.

Figure 7b. Distribution of 150 Good assignments for each player across the 250 games.

Image not extracted

Figure 7. Stage 1 Good and Evil Role Distribution.

Stage 2

Game outcome across all 250 games

Figure 8. Game outcome across all 250 games (green = Good victory, red = Evil victory). The repeating block structure reflects the predefined assignment of the outcome sequence.

Team composition distribution across 1,000 rounds

Figure 10. Team composition distribution across 1,000 rounds. Labels denote the exact count of Good (G) and Evil (E) players on each team. Mixed teams containing exactly one Evil player (1G-1E and 2G-1E) account for 77.4% of all rounds, reflecting the role-aware selection rule.

Stage 3

Figure 12 reports tactic assignment frequencies by alignment category across all 4,000 agent assignments and Good-exclusive tactics (Cooperative column) account for 1,814 assignments (45.4%), Evil-exclusive tactics (Opportunistic and Adversarial columns) for 1,610 (40.3%), and Shared tactics (Defensive/Both column) for 576 (14.4%). The lower Shared share reflects the level-aware assignment rule, since high-skill agents dominate numerically and draw exclusively from alignment-specific columns.

Figure 12. Tactic assignment distribution by alignment category (n=4,000n = 4,000 assignments across 1,000 rounds).

Table 5 reports the assignment count for each of the 37 tactics, organized by the same 4×44 \times4 matrix structure used in § 3. Figure 13 illustrates the five most and least occurring tactics.

Figure 13. The five most and least frequently assigned tactics across all 1,000 samples.

Prompt Boxes 1 and 2 show the structure of the Pass 1 and Pass 2 prompts for a Round 4 example. Content common to all games is abbreviated with [. . .].

Prompt Box 1. Pass 1 Tactic Precompute (Round 4 Example)

Figure 1 (PDF p. 19). Prompt Box 1. Pass 1 Tactic Precompute (Round 4 Example)

System: You are an expert in social deduction game strategy and deception theory.

User:
You are an expert in social deduction game strategy and deception theory. Your task: Pre-assign
the communication tactic for each speaking player in Round 4 of an Avalon game. Base your choices
on: (1) each player’s Round 3 tactic as a seed anchor, (2) the current game state.
# GAME CONTEXT
Game ID: G042   Assigning tactics for: Round 4
Silent protagonist (observer): P3 (Good)
Player Roles: P1: Evil, P2: Good, P3: Good (protagonist), P4: Evil, P5: Good
# CUMULATIVE PUBLIC HISTORY (Rounds 1 through 4)
Format: Round | Leader | Team | Votes | Quest Outcome
Round 1 | Leader: P1 | Team: P1, P2, P3 | Votes: P1:Y P2:Y P3:Y P4:N P5:Y | Quest 1 Outcome: PASS
Round 2 | Leader: P2 | Team: P2, P4, P5 | Votes: P1:Y P2:Y P3:N P4:Y P5:Y | Quest 2 Outcome: FAIL
Round 3 | Leader: P3 | Team: P1, P2, P3 | Votes: P1:Y P2:Y P3:Y P4:Y P5:Y | Quest 3 Outcome: PASS
Round 4 | Leader: P4 | Team: P2, P4, P5 | Votes: P1:Y P2:Y P3:N P4:Y P5:Y | Quest 4 Outcome: [...]
# OLDER ROUND SUMMARIES (Rounds 1 through 2)
These summaries cover rounds before Round 3. Use them for broader game trajectory and behavioral
arc context.
[... condensed narrative summaries of Rounds 1–2...]
# ROUND 3 DISCUSSION (Reference for player dialogues and dynamics)
[... full verbatim discussion log from Round 3...]
# ROUND 3 TACTIC SEEDS (Anchor points for Round 4 evolution)
P1 (Evil, Mach-IV High): ‘Strategic Deception’ [Framing × Opportunistic]
P2 (Good, GRS Moderate): ‘Tentative Reasoning’ [Framing × Cooperative]
P4 (Evil, Mach-IV Moderate): ‘Deflection’ [Framing × Defensive]
P5 (Good, GRS High): ‘Honest Disclosure’ [Transparent × Cooperative]
# ALIGNMENT POOLS
P1 (Evil): must select from Evil pool (27 tactics)
P2 (Good): must select from Good pool (19 tactics)
P4 (Evil): must select from Evil pool (27 tactics)
P5 (Good): must select from Good pool (19 tactics)
# TACTIC EVOLUTION GUIDELINES
Scale levels are FIXED. [...] Only the tactic may change.
Round 4 (Mid): mix Opportunistic and Defensive – begin shaping the narrative more actively.
[...] Good and Evil player guidance...
Output: JSON object: {"P1": {"tactic": "...", "row": "...", "col": "...", "scale": "Mach-IV",
"level": "High"}, ...}
Prompt Box 2. Pass 2 Dialogue Generation (Round 4 Example)

System: You are an expert in social deception games and deception theory. You generate realistic
game discussions with tactical communication patterns.

User:
Generate a realistic discussion log for Round 4 of an Avalon game based on the provided game
context.
# AVALON RULES AND PUBLIC HISTORY FORMAT
[. . . game mechanics: team selection, approval votes, anonymous quest outcome . . .]
# 4×4 BEHAVIOR MATRIX
[. . . row/column definitions: Transparent, Framing, Careless-to-truth, Counterfactual ×
Cooperative, Defensive, Opportunistic, Adversarial . . .]
# SCALE DEFINITIONS
GRS (Good players): High = evidence-based, Moderate = some hedging, Low = confused or overwhelmed.
Mach-IV (Evil players): High = subtle manipulation, Moderate = situational, Low = impulsive.
# CUMULATIVE PUBLIC HISTORY (Rounds 1 through 4)
[. . . same public history block as Pass 1 . . .]
# OLDER ROUND SUMMARIES (Rounds 1 through 2)
[. . . condensed narrative summaries of Rounds 1–2 . . .]
# ROUND 3 DISCUSSION
Use this as context for how each player communicates. Maintain character voice consistency.
[. . . full verbatim discussion log from Round 3 . . .]
# PRE-ASSIGNED TACTICS FOR ROUND 4
Each speaker’s tactic has been pre-assigned based on their role, scale, and current game state.
P1 (Evil, Mach-IV High): ‘Aggressive Misdirection’ [Counterfactual × Adversarial]
P2 (Good, GRS Moderate): ‘Tentative Reasoning’ [Framing × Cooperative]
P4 (Evil, Mach-IV Moderate): ‘False Accusation’ [Counterfactual × Opportunistic]
P5 (Good, GRS High): ‘Evidence Synthesis’ [Transparent × Cooperative]

Output: JSON with two fields.
discussion_log: string starting with "Discussion after Quest 4:" followed by four P#: "dialogue"
lines (one blank line between speakers; exactly 4 speakers; speaking order agenda-driven, not
P1–P4 default).
matrix_tactic_scale: JSON object mapping each speaker to their row, col, tactic, scale, and level.

Stage 4

Table 6 reports the per-round Claude Sonnet 4.5 judge scores, selections, and corrections for GPT-5.2 and Gemini 3.1 Pro candidates. Table 7 reports per-criterion pass rates across all five rounds. Coherence and Tactic Situational Fit is the weakest criterion for Gemini 3.1 Pro across all rounds (29.3–80.8%), whereas GPT-5.2 is weakest on Tactic-Dialogue Alignment in R1 (78.2%) and otherwise near-ceiling on all criteria. Format Compliance reaches 100% for both models in every round. Figure 14 shows the per-criterion pass-rate heatmap. Figure 15 shows the per-round selection and correction breakdown. Figure 16 shows per-round score trajectories for the average judge score and each criterion.

Figure 14. Per-criterion pass rates (%). Left: GPT-5.2; right: Gemini 3.1 Pro. Criteria: C1 Coherence & Tactic Fit, C2 History Alignment, C3 Tactic-Dialogue Alignment, C4 Authenticity & Skill Consistency, C5 Format Compliance.

Figure 15. Selection and correction breakdown (%) per round. GPT: GPT-5.2 direct; GEM: Gemini 3.1 Pro direct; TGT_G (for GPT) / TMT_M (for GEM): targeted correction; Custom: custom regeneration.

CriterionR1 (n=225n=225) GPTR1 (n=225n=225) GEMR2 (n=250n=250) GPTR2 (n=250n=250) GEMR3 (n=250n=250) GPTR3 (n=250n=250) GEMR4 (n=150n=150) GPTR4 (n=150n=150) GEMR5 (n=100n=100) GPTR5 (n=100n=100) GEM
(1) Coherence93.829.399.680.899.272.498.771.399.067.0
(2) History95.193.899.699.299.296.097.396.098.099.0
(3) Tactic-Dialogue78.245.397.290.098.492.898.795.394.087.0
(4) Authenticity95.150.2100.083.2100.076.099.376.0100.077.0
(5) Format100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0

Table 7. Per-criterion pass rates (%) for Gemini 3.1 Pro (GEM) and GPT-5.2 (GPT) across all five rounds. Criteria: (1) Coherence and Tactic Situational Fit, (2) Public History Alignment, (3) Tactic-Dialogue Alignment, (4) Authenticity and Skill Consistency, (5) Format Compliance.

Prompt Boxes 3 and 4 show the structure of the Layer 1 pair evaluation and Layer 2 single-dialogue recheck prompts for a Round 4 example. Content common to all games is abbreviated with [. . .].

Prompt Box 3. Layer 1: Pair Evaluation (Round 4 Example)

System: You are an expert verifier for the Avalon social deception game with deep knowledge of deception tactics and game mechanics. Your task is to evaluate pairs of AI-generated dialogue responses, assess their quality, and determine the best outcome: select the superior response as-is, apply targeted corrections to fix specific issues in the better response, or generate an entirely new dialogue with appropriate tactics and scales if both responses are inadequate.

User: You are evaluating two AI-generated Avalon Round 4 discussion logs to determine which is superior. Both responses were generated for the same game context. Your task is to validate both using binary pass/fail checks on 5 criteria, then recommend the best outcome.

CONTEXT

Game: G042 Round: 4

Player Roles: P1: Evil, P2: Good, P3: Good (protagonist), P4: Evil, P5: Good

Protagonist (does NOT speak in dialogue): P3

Prior Round Summaries: [. . . condensed narrative summaries of Rounds 1–3 for game G042 . . .]

Public History (current round): [. . . same cumulative public history block as Stage 3 Pass 1 and Pass 2 . . .]

—

RESPONSE A (Gemini 3.1 Pro — model identity withheld from judge):

Discussion after Quest 4:

P5: “The R2 failure with P2/P4/P5 and the R1 pass with P1/P2/P3 make P4 the clearest suspect.”

P4: “P3 has been suspiciously quiet and didn’t vote for our team. Maybe P3 is evil.”

[. . . P2 and P1 utterances . . .]

matrix_tactic_scale (Response A):

P5 (Good, GRS High): ‘Evidence Synthesis’ [Transparent ×\times Cooperative]

P4 (Evil, Mach-IV Moderate): ‘False Accusation’ [Counterfactual ×\times Opportunistic]

P2 (Good, GRS Moderate): ‘Tentative Reasoning’ [Framing ×\times Cooperative]

P1 (Evil, Mach-IV High): ‘Aggressive Misdirection’ [Counterfactual ×\times Adversarial]

—

RESPONSE B (GPT-5.2 — model identity withheld from judge):

[. . . same game context; independent Round 4 dialogue generated by GPT-5.2 . . .]

matrix_tactic_scale (Response B):

[. . . GPT-5.2 tactic annotations for the same four speakers . . .]

—

TACTIC ANNOTATION FRAMEWORK: Both responses were generated from identical pre-assigned tactic plans. Neither matrix is canonical; treat as starting reference, not ground truth.

[. . . full 37-tactic taxonomy, row/column definitions, and scale descriptions . . .]

EVALUATION CRITERIA (1 = PASS, 0 = FAIL):

(1) Coherence & Tactic Situational Fit: tactic is socially congruous with the discussion dynamics (conversational position, what others are saying, how tensions develop).

(2) Public History Alignment: consistent with quest outcomes, teams, and voting from the current round and prior round summaries.

(3) Tactic–Dialogue Alignment: annotation correctly names what was said; tactic is valid in the taxonomy; row/col match the 4×44 \times4 matrix; scale and level are accurate.

(4) Authenticity & Skill Consistency: believable Avalon exchange; utterance sophistication matches the player’s assigned skill level.

(5) Format Compliance: starts with “Discussion after Quest 4:”, exactly 4 nonprotagonist speakers, each speaks exactly once.

DECISION LOGIC:

  1. Both 5/5 ⇒ PAIRWISE_TIEBREAKER: qualitative comparison selects the stronger response

  2. One 5/5, other <5 ⇒ select the 5/5 response directly

  3. At least one 4/5 ⇒ CORRECTION_A or CORRECTION_B: targeted inline fix of the failing criterion; if both 4/5, qualitative comparison selects the base

  4. Both ≤3/5 ⇒ CUSTOM_GENERATION: generate a new dialogue from scratch

Output: JSON with response_a/response_b (per-criterion scores and totals), recommendation, pairwise_winner, failed_criterion, targeted_correction (corrected dialogue + matrix), custom_dialogue (new dialogue + matrix), and reasoning.

Prompt Box 4. Layer 2: Single-Dialogue Recheck (Round 4 Example)

System: [. . . same system prompt as Layer 1 . . .]

User:
You are evaluating a single AI-generated Avalon discussion log in Round 4 for quality. The dialogue has already undergone one round of correction. Your default is to ACCEPT it; only flag a criterion as failing when there is a clear, objective, nondebatable violation. You must not be nitpicky or subjective. Score it on 5 binary criteria and, if it fails any criterion, apply the minimum fix required to pass that criterion.

# CONTEXT
Game: G042    Round: 4
Player Roles: P1: Evil, P2: Good, P3: Good (protagonist), P4: Evil, P5: Good
Protagonist (does NOT speak in dialogue): P3
Prior Round Summaries:
[. . . same condensed narrative summaries as Layer 1 . . .]
Public History (current round):
[. . . same public history block as Layer 1 . . .]

—

DIALOGUE TO EVALUATE (Layer 1 targeted correction output):
Discussion after Quest 4:
P5: “The R2 failure with P2/P4/P5 and the R1 pass make P4 the clearest suspect on the evidence.”
P4: “P3 voted against our team in R4 — that warrants scrutiny before we finalize R5.”
[. . . P2 and P1 utterances . . .]

Matrix Tactic Scale:
P5 (Good, GRS High): ‘Evidence Synthesis’ [Transparent × Cooperative]
P4 (Evil, Mach-IV Moderate): ‘False Accusation’ [Counterfactual × Opportunistic]
P2 (Good, GRS Moderate): ‘Tentative Reasoning’ [Framing × Cooperative]
P1 (Evil, Mach-IV High): ‘Aggressive Misdirection’ [Counterfactual × Adversarial]

—

TACTIC ANNOTATION FRAMEWORK:
[. . . same 37-tactic taxonomy, row/column definitions, and scale descriptions as Layer 1 . . .]

EVALUATION CRITERIA (1 = PASS, 0 = FAIL):
[. . . same 5 criteria as Layer 1 . . .]
Benefit-of-the-doubt rule: when uncertain whether a flaw reaches the 0/1 threshold, always score as passing (1). Only assign 0 when the failure is absolutely clear, objective, and nondebatable.

DECISION LOGIC:
1. 5/5 ⇒ ACCEPT: accept the dialogue as-is
2. 4/5 ⇒ TARGETED_CORRECTION: fix the single failing criterion in the provided dialogue
3. ≤3/5 ⇒ FULL_REGENERATION: generate a completely new dialogue that passes all 5 criteria
Output: JSON with criteria (per-criterion scores and explanations), total, recommendation, failed_criterion, targeted_correction (corrected dialogue + matrix), regenerated_dialogue (new dialogue + matrix), and reasoning.

Stage 5

Prompt Boxes 5 and 6 show the structure of the ReAct and ReCon generation prompts for a Round 4 example. Content common to all games is abbreviated with [. . .].

Prompt Box 5. ReAct: Reasoning Trace Generation (Round 4 Example)

System: You are P3, the Investigator – a Good player in a modified 5-player Avalon game. Reason carefully and produce only valid JSON.

User:
[. . . Avalon game overview, setup, round sequence, Investigator role, Critical Self-Knowledge, and Internal Reasoning Framework (Steps 1–6: Abduction, Suspicion, First-Order Beliefs, Second-Order Beliefs, Statement, Deduction) . . .]

—

# CURRENT GAME CONTEXT
Game: G042    Round: R4    Protagonist: P3
Player speaking order this round: P5, P4, P2, P1

**CUMULATIVE PUBLIC HISTORY**
Round 1 | Leader: P1 | Team: P1, P2, P3 | Votes: P1:Y P2:Y P3:Y P4:N P5:Y | Quest 1: PASS
Round 2 | Leader: P2 | Team: P2, P4, P5 | Votes: P1:Y P2:Y P3:N P4:N P5:Y | Quest 2: FAIL

Round 3 | Leader: P3 | Team: P1, P2, P3 | Votes: P1:Y P2:Y P3:Y P4:Y P5:Y | Quest 3: PASS Round 4 | Leader: P4 | Team: P2, P4, P5 | Votes: P1:Y P2:Y P3:N P4:Y P5:Y | Quest 4: FAIL

CUMULATIVE DISCUSSION LOG [... verbatim discussion logs for Rounds 1–4...]

PRIOR ROUND REASONING (ReAct chain) [... R1–R3 ReAct reasoning traces produced by the same framework...]

—

Reasoning (ReAct → adapted for modified Avalon):

Thought 1: Identify the single most diagnostic unresolved question about player alignments after this round. Focus on quest outcomes, team composition, and player statements relative to your own participation.

Action 1: Select one concrete, named piece of evidence from the cumulative public history above. State it explicitly.

Observation 1: State precisely what that evidence implies about one or more players’ alignments. Apply the Definitive Deduction Rule if applicable.

Thought 2: Based on Observation 1, state your updated hypothesis. Which player is your top suspect and on what basis? (Anti-loop: if Thought 2 would only restate Thought 1 with no new evidence, examine a genuinely different clue before continuing.)

Action 2: Cross-check your hypothesis against one specific statement or voting behavior from the cumulative discussion log. Cite the player and the content explicitly.

Observation 2: Does the cross-check confirm, revise, or reinforce your Thought 2 hypothesis? State your final confidence level.

Thought 3 (Synthesis): Based on your two-step analysis, synthesize your final assessment. Which players warrant second-order belief analysis (Strong/Extreme suspicion)? What would you say to the group, grounded only in observable evidence? State your final deduction of 1–3 Evil players.

Now produce the final JSON, maintaining speaking order (P5, P4, P2, P1) in abduction, suspicion, and beliefs: [... output schema: abduction (good_expl, evil_expl, choice per player), suspicion (level, reason), depth, beliefs (β1\beta_1 all players; β2\beta_2 Strong/Extreme only), statement, deduction ...]

Prompt Box 6: ReCon: Reasoning Trace Generation (Round 4 Example)

Figure 6 (PDF p. 25). Prompt Box 6. ReCon: Reasoning Trace Generation (Round 4 Example)

System: You are P3, the Investigator – a Good player in a modified 5-player Avalon game. Reason carefully and produce only valid JSON.

User: [... same static Avalon overview, setup, and Internal Reasoning Framework as ReAct...]

—

CURRENT GAME CONTEXT

Game: G042 Round: R4 Protagonist: P3

Player speaking order this round: P5, P4, P2, P1

CUMULATIVE PUBLIC HISTORY [... same four-round public history block as ReAct...]

CUMULATIVE DISCUSSION LOG [... verbatim discussion logs for Rounds 1–4...]

PRIOR ROUND REASONING (ReCon chain) [... R1–R3 ReCon reasoning traces produced by the same framework...]

—

Reasoning (ReCon → adapted for modified Avalon):

Formulation contemplation (first-order perspective transition): For each non-protagonist player in speaking order (P5, P4, P2, P1), infer their probable alignment based on quest participation, voting record, and speech in the discussion log. Adopt each player’s perspective: given what they know, would an Evil player in their position behave differently from a Good player here? Form an initial role assessment and suspicion level for each player, grounded only in observable evidence.

Refinement contemplation (second-order perspective transition): Revisit your initial formulation from the perspectives of other players. Check three issues: (a) Evidence Sufficiency – are your strongest claims supported by explicit observable evidence (quest outcomes, votes, or quoted statements)? (b) Cross-Round Consistency – does your deduction stay consistent with prior rounds unless new evidence justifies a revision? (c) Public Statement Alignment – your statement field must be what you would plausibly say aloud to other players, strategic yet grounded in observable evidence. Revise suspicion levels, beliefs, and statement to

eliminate any issues.
Now produce the final JSON, maintaining speaking order (P5, P4, P2, P1) in abduction, suspicion,
and beliefs:
[. . . same output schema as ReAct...]

Stage 6

Table 8 reports comprehensive per-framework verification statistics for Stage 6. Table 9 provides a focused view of per-criterion pass rates and inter-framework agreement. fig:17a shows the reasoning framework selection rates by Claude Sonnet 4.6, along with the agreements where both frameworks achieve a 4/4 score. fig:17b shows the corresponding per-criterion pass rate trends across frameworks.

CriterionReAct (%)ReCon (%)Agree. (%)
Abduction Plausibility97.693.792.5
Suspicion Consistency94.389.185.8
Gated Depth Reasoning100.099.999.9
Statement & Deduction99.398.298.1
Overall (4/4)92.584.4—

Table 9. Per-criterion pass rates for ReAct and ReCon traces in Stage 6 (NN=1,000). Agreement reports the percentage of pairs where both traces receive 4/4 score for that criterion.

Prompt Boxes 7 and 8 show the structure of the Layer 1 pair evaluation and Layer 2 single-trace recheck prompts for a Round 4 example. Content common to all games is abbreviated with [. . .].

Prompt Box 7. Layer 1: Pair Evaluation (Round 4 Example)

System: You are an expert verifier for Avalon Theory of Mind reasoning traces. Evaluate Trace A
and Trace B using 4 binary PASS/FAIL criteria, apply the decision logic strictly, and return only
valid JSON. Judge from the Investigator’s perspective using observable evidence only, not factual
alignment truth. Only fail a criterion when the violation is clear.

User:
You are evaluating two AI-generated Theory of Mind reasoning traces for the same Avalon game
round. Apply 4 binary PASS/FAIL criteria, then apply the decision logic strictly. Judge from the
Investigator’s perspective using observable evidence only.
# GAME CONTEXT
Game: G042   Round: R4   Investigator: P3 (always Good; always PASS; does NOT speak)
Player Roles (ground truth): P1: Evil, P2: Good, P3: Good, P4: Evil, P5: Good
Is Final Round: No

[. . . Avalon mechanics reference: team sizes, quest rules, Investigator constraints, definitive
Deduction Rule, gated depth rule, suspicion scale ...]
Cumulative Public History (Rounds 1–4):
[... same four-round history block as Stage 5 prompts...]
Cumulative Round Summaries (Rounds 1–4):
[... condensed narrative summaries of each round...]
Cumulative Discussion Log (Rounds 1–4):
[... verbatim discussion logs...]
Prior Round Reasoning Golds (Rounds 1–3):
[... gold reasoning traces selected in prior rounds...]

TRACE A (ReAct — framework identity withheld from judge):
[... ReAct reasoning trace for G042 R4: abduction, suspicion, depth, beliefs, statement, deduction
...]

TRACE B (ReCon — framework identity withheld from judge):
[... ReCon reasoning trace for G042 R4: same six fields...]

EVALUATION CRITERIA (1 = PASS, 0 = FAIL):
(1) Abduction Plausibility: hypotheses h_i cite specific observable evidence; each alignment choice
reflects the strongest available inference; the Definitive Deduction Rule is enforced for any
two-person failed quest involving the Investigator.
(2) Suspicion Level Consistency: suspicion levels s_i are proportionate to cumulative public evidence;
the Definitive Deduction Rule mandates Extreme for the two-person failed quest partner; Strong
requires multiple converging cues.
(3) Gated Depth Reasoning: first-order beliefs β_1^i exist for all players; second-order beliefs β_2^i
exist only for players at Strong or Extreme suspicion; depth field is set to 2 iff at least one
player has Strong or Extreme suspicion.
(4) Statement and Deduction Coherence: final utterance û is grounded in observable evidence and
plausible as in-character dialogue; deduction α̂ includes all Strong and Extreme suspicion players;
final round requires exactly two named Evil players.

DECISION LOGIC:
1. Both 4/4 ⇒ PAIRWISE_TIEBREAKER: choose the stronger trace, closer to ground-truth alignment.
2. One 4/4, other <4/4 ⇒ select the 4/4 trace directly.
3. Both <4/4 ⇒ CORRECTION_A or CORRECTION_B: correct the higher-scoring base; pairwise comparison
breaks ties.
Hard rules: CORRECTION_A/B only when both traces score <4/4; if one trace is 4/4, it must be
selected.

EVALUATION PROCESS:
1. Score Trace A on all 4 criteria.   2. Score Trace B on all 4 criteria.   3. Count passes for

each trace. 4. Apply the decision logic. 5. Return JSON in the format below.

Output: JSON with trace_a/trace_b (per-criterion scores and totals), recommendation, pairwise_winner, pairwise_reasoning, failed_criteria, and targeted_correction (full corrected trace when applicable).

Prompt Box 8. Layer 2: Single-Trace Recheck (Round 4 Example)

System: [. . . same system prompt as Layer 1 . . .]

User:

You are re-evaluating a single AI-generated Theory of Mind reasoning trace for an Avalon game round. This trace has been selected as the best candidate and received a targeted correction. Re-score it on the same 4 binary criteria. The default is to ACCEPT; only fail a criterion when the violation is clear, objective, and non-debatable.

GAME CONTEXT

Game: G042 Round: R4 Investigator: P3 (always Good; always PASS; does NOT speak)

Player Roles (ground truth): P1: Evil, P2: Good, P3: Good, P4: Evil, P5: Good

Is Final Round: No

[. . . same Avalon mechanics reference as Layer 1 . . .]

Cumulative Public History (Rounds 1–4):

[. . . same history block as Layer 1 . . .]

Cumulative Round Summaries (Rounds 1–4):

[. . . condensed narrative summaries . . .]

Cumulative Discussion Log (Rounds 1–4):

[. . . verbatim discussion logs . . .]

Prior Round Reasoning Golds (Rounds 1–3):

[. . . gold traces from prior rounds . . .]

—

TRACE TO EVALUATE (Layer 1 targeted correction output):

[. . . the corrected reasoning trace from Layer 1: abduction, suspicion, depth, beliefs, statement, deduction . . .]

—

EVALUATION CRITERIA (1 = PASS, 0 = FAIL):

[. . . same 4 criteria as Layer 1 . . .]

Benefit-of-the-doubt rule: when uncertain whether a flaw reaches the pass/fail threshold, always score as passing. Only assign 0 when the failure is absolutely clear, objective, and non-debatable.

Output: JSON with per-criterion scores, total, recommendation (ACCEPT or TARGETED_CORRECTION), failed_criteria, and targeted_correction (full corrected trace when applicable; null otherwise).

Trace Deduction Accuracy

Table 10 reports the deduction accuracy by game-ending round, whereas Table 11 reports deduction accuracy for all 1,000 traces, grouped by round. Accuracy rises from R1 to R5 as quest outcomes and discussions accumulate. This rise indicates that the traces infer incrementally rather than apply a fixed heuristic. The F1 over all rounds (0.842) is lower than the game-ending F1 (0.962) because early rounds lack sufficient evidence to identify the Evil pair. Reporting the pooled F1 alone would understate the corpus; reporting it with the game-ending verdict shows where the difficulty lies.

RoundnExact (%)PRF1
R125038.40.6950.6880.691
R225060.00.8040.8280.816
R325081.60.9040.9080.906
R415086.70.9300.9370.934
R510097.00.9800.9800.980
All100067.70.8390.8450.842

Table 11. Gold-trace deduction accuracy by round, all 1,000 traces. Same column annotation as Table 10.

What quest outcomes alone determine. For each game, there is a provision for six pairs ((42)\binom{4}{2}) of Evil players, and a failed quest includes at least one Evil player while the Investigator is always Good. To determine the valid Evil pairs remaining from quest outcomes, we keep each pair that includes a member of every failed team. For instance, in G002, the Investigator P1 and player P4 fail the R1 quest. Thus, we know that P4 is Evil, and the R2 team of P2, P3, and P5 also fails, so three pairs remain: (P2, P4), (P3, P4), and (P4, P5). Table 12 groups the 250 games by the number of pairs that remain. On AVCORP, quest outcomes single out one pair in 12 games, and two to six pairs remain in the other 238. A reasoner that uses only quest outcomes and picks one of kk allowed pairs at random is correct with probability 1/k1/k, so a group of nn such games yields n/kn/k correct games in expectation (Expected in Table 12). Across the 238 games, this sum is 18.5 + 22 + 14 + 7 + 7.33 = 68.83, an expected exact match of 68.83/238 = 28.9%, whereas the gold traces resolve 221 (92.9%). The difference comes from the dialogue that the traces use but the quest record lacks.

Review of divergent traces. We manually reviewed the games whose final verdict differed from α\alpha. One game-ending verdict, G124/4, excluded every member of a failed team. It also broke the game’s speaker order, indicating a disrupted generation. We regenerated the failing trace under human supervision, and the trace now names the correct pair. In each of the 17 games that still diverge, the trace names a pair that the quest record allows. In several cases, the Evil players pass a shared quest to avoid suspicion. No final verdict, therefore, contradicts its quest record.

Annotator Agreement

We sample 25 whole games (100 rounds) so that the two annotators follow each Investigator’s reasoning across rounds, as the generator did. The sample is representative: 79 of its 100 items are tiebreak cases (Table 13), against 79.1% of the corpus. The judge decided the other 21 on criterion scores alone. The annotators receive the same Trace A and Trace B pairs as the judge. They receive these pairs in a random order without a fixed position. Since most games allow multiple Evil pairs consistent with the quest record (Appendix § 16), annotators score the reasoning rather than the correctness of its conclusion. The judge uses α\alpha to resolve tied pairs. Judge agreement compares the judge’s selection with each annotator’s selection and their shared selection. If the tiebreak rewarded correct conclusions over sound reasoning, judge agreement would drop on the tiebreak items.

Selection agreement. Confidence intervals for κ\kappa are 95% bootstrap intervals (2,000 resamples over items). We compute κ\kappa with scikit-learn and draw the bootstrap resamples with NumPy. The judge matches the annotators’ shared choice on 95 of the 97 rounds where they agree (κ=.96\kappa= .96, 95% CI [.89, 1.00]). Table 14 reports the selections behind these values; its last panel shows that the annotators share the judge’s preference for ReCon.

A1 (rows) vs. A2
ABAll
A59160
B23840
All6139100
A1 (rows) vs. J
ABAll
A58260
B23840
All6040100
A2 (rows) vs. J
ABAll
A59261
B13839
All6040100
Selections by framework
A1A2J
ReCon565756
ReAct444344

Table 14. Trace selections on the 100 items. In the first three panels, rows give the first rater and columns the second, and A and B are the positions in which the annotators saw the two traces: ReCon was Trace A in 50 items and Trace B in the other 50, and ReAct the reverse. The last panel counts each rater’s selections by framework, regardless of position.

Sampled games. We used stratified random sampling over each game’s ending round (seed 42). We allocated games in proportion to their frequency in the corpus (40%, 20%, and 40% for R3, R4, and R5 endings). R3-ending (10 games, 30 items): G004, G014, G015, G018, G059, G062, G066, G202, G207, G215. R4-ending (5 games, 20 items): G026, G128, G175, G178, G228. R5-ending (10 games, 50 items): G034, G035, G042, G086, G088, G090, G185, G192, G198, G242. Three of the 25 sampled games (G062, G178, G228) are among the 17 games whose final verdict diverges from α\alpha (§ 4.6.1).

Per-criterion agreement. Table 15 reports agreement on each of the four criteria. We pool Trace A and Trace B. Thus, each pair of raters contributes 200 judgments per criterion. Cohen’s κ\kappa corrects the observed agreement (PoP_o) for the expected agreement (PeP_e). When nearly every trace passes, PeP_e approaches PoP_o. This mathematical property forces κ\kappa toward zero or makes it undefined ([11]). This effect occurs even when raters agree on almost every judgment. Criterion 4 demonstrates this effect directly. Both annotators pass all 200 traces. The judge passes 199 traces. Each annotator agrees with the judge on 199 judgments. However, Pe=Po=0.995P_e = P_o = 0.995, which yields κ=0\kappa= 0. On Criterion 3, every rater passes every trace. This unanimous outcome sets Pe=1P_e = 1 and makes κ\kappa undefined. PABAK replaces PeP_e with the expected value under balanced labels ([5]). PABAK measures the agreement itself regardless of failure rarity. PABAK is at least 0.94 for every pair and criterion. Failures occur on only two criteria. For these two criteria, κ\kappa remains defined but has wide intervals. For example, on Criterion 1, the

95% interval for Annotator 2 against the judge spans from 0.206 to 0.939. Gwet’s AC1 (Gwet, [14]) is another prevalence-robust coefficient. This coefficient stays within 0.01 of PoP_o on every row and supports the same conclusion.

Trained Detector Settings

Qwen3-8B is released under the Apache 2.0 license.

Trained Model Parameters and GPU Hours

Table 16 lists the SFT and GRPO parameters. Each GRPO step uses two prompts with 16 completions each, and one epoch over the 900 training samples takes 450 steps.

ParameterSFTGRPO
Base modelQwen3-8B, bf16, not quantized
LoRArank 32, αLoRA=64\alpha_{\mathrm{LoRA}} = 64, dropout 0.05, all linear layers
OptimizerAdamW (default)AdamW (fused)
Learning rate2×10−52 \times10^{-5}1×10−51 \times10^{-5}
Warmup ratio0.050.05
Epochs11 (450 steps)
Batch per device ×\times accumulation4×2=84 \times2 = 8 samples8×4=328 \times4 = 32 completions
Completions per prompt—16
Sampling temperature—0.8
Maximum length (tokens)5,120 (5,632) in totalprompt 3,328 (3,456); completion 1,280
Losscross-entropy on the completionDAPO
Rewards—5 (6), equal weights
Trainer (TRL 0.24.0)SFTTrainerGRPOTrainer
LibrariesPyTorch 2.7.1, Transformers 4.56.1, PEFT 0.14.0, Accelerate 1.4.0

Table 16. Training parameters. Values in parentheses apply to the tactic variant.

Token caps. The longest gold trace has 1,139 tokens (1,191 with tactics), and the longest prompt has 3,013 tokens (3,329 with tactics). We cap completions at 1,280 tokens and prompts at 3,328 tokens (3,456 with tactics). The completion cap stays close to the longest trace because GRPO generates 16 completions per prompt and processes every token in each. An output that reaches the cap has no deduction and counts as an empty prediction (Appendix § 18.7).

GPU hours. All runs use one NVIDIA H100 PCIe GPU. SFT takes 20 to 24 minutes and GRPO 25.1 to 34.4 hours. The eight runs take 226.3 GPU hours in total. We run 18 evaluations on the held-out games (51 to 91 minutes each, 22.2 GPU-hours in total): the SFT and GRPO stages of the eight runs, and the untrained model once for each of the two prompts.

GRPO Rewards

Each reward scores a completion from 0 to 1, and GRPO sums the rewards with equal weights. The rewards read the fields of the completion. These fields include the hypotheses hih_i with a Good or Evil choice, the suspicion levels sis_i, the beliefs β1i\beta_1^i and β2i\beta_2^i, the depth (the highest belief order), the utterance u^\hat{u}, the deduction α^\hat{\alpha}, and, in the tactic variant, one tactic per contextual agent. Table 17 defines each reward over the four contextual agents. Four of the five original rewards saturated during training. Rewards (2) and (3) reach their maximum scores without identifying the Evil players. The revised rewards replace these two with rewards that score the Good or Evil choice and the order of suspicion levels against α\alpha.

RewardSetα\alphaScore
(1) Schema validityOriginalNo1 if the completion is valid JSON with every field; otherwise half the share of fields present, and 0 if it is not JSON.
(1) Schema validityRevisedNoMean of three checks: every field is present, the utterance u^\hat{u} is not empty, and the depth equals the highest belief order.
(2) Abduction completenessOriginalNoShare of agents with both hypotheses and a Good or Evil choice.
(2) Abduction accuracyRevisedYesShare of agents whose Good or Evil choice matches α\alpha; a missing agent counts as wrong.
(3) Suspicion consistencyOriginalYes0.6×0.6 \times the share of agents with a valid suspicion level si+0.4×s_i + 0.4 \times the share of Evil players rated Strong or Extreme.
(3) Suspicion calibrationRevisedYesMean over the four (Evil, Good) pairs of agents: 1 if the Evil player has the higher level, 0.5 if the levels are equal, and 0 otherwise.
(4) Gated depthBothNo0.5×0.5 \times the share of agents with a first-order belief +0.5×max⁡(0,1−0.25v)+ 0.5 \times\max(0, 1 - 0.25v), where vv counts second-order beliefs about agents below Strong suspicion.
(5) Deduction F1BothYesF1 of α^\hat{\alpha} against the Evil players in α\alpha.
(6) Tactic accuracyTactic variantNoShare of agents whose predicted tactic matches the gold tactic.

Table 17. GRPO rewards. Set gives the reward sets that use the reward; α\alpha marks rewards that use the withheld roles.

Avalon-NLU Conversion

We convert the 20 released games of Stepputtis et al. (2023) without editing the source files. These games are released under the MIT license. Each quest becomes one round. Four games end after three quests, ten after four, and six after five. This conversion yields 82 total rounds (4×3+10×4+6×5=824 \times3 + 10 \times4 + 6 \times5 = 82). Seats map to P1 through P6. The Investigator is the first Loyal Servant, and the Evil pair is Morgana and the Assassin. The history follows the AVCORP format. It adds the team size of each quest and every proposal with its leader and team. It also adds the vote count of each rejected proposal and the score after the quest. The discussion uses the chat transcript of the quest. We mark the Investigator’s own messages within this transcript. An independent script rebuilt all 82 rounds from the raw files. This script found one seat-parsing error across three rounds. We fixed this error before evaluation. In one game (NLU-3YjxN), the log records a quest failed by four Good players. A re-proposal announced in the chat lacks a matching proposal event. Therefore, we also report results over the remaining 19 games.

Detector Results by Round

Table 18 and Table 19 report results by round on the 25 held-out AVCORP games. Table 20 reports results on the 20 Avalon-NLU games using the domain-context prompt (Appendix § 18.§ 6). Every detector and frontier model scores lower at R1 than at R3. At R1, no quest outcomes are known. Every game contains exactly two Evil players. However, with the trace variant and original rewards, GRPO names 2.57 to 3.12 players per round on AVCORP. With the revised rewards, it names 1.73 to 2.27 players. The best detector performs strongest late in a game. At R5, its F1 score reaches 0.927. In contrast, the frontier models score between 0.550 and 0.700. On Avalon-NLU, the untrained model names the fewest players (1.17 and 1.56 per round). This model also produces the lowest recall (0.366 and 0.384).

DetectorStageR1R2R3R4R5PRF1Named
Qwen3-8B, trace promptadapters off0.0730.1720.5070.4900.5450.6710.2350.3480.70
Qwen3-8B, tactic promptadapters off0.2220.3530.5930.6400.6290.7220.3500.4710.97
Trace, original rewardsSFT0.5020.5520.6510.6280.7350.7040.5170.5961.47
SFT + GRPO0.6870.7170.7670.7650.8040.6210.9130.7372.97
Trace, revised rewardsSFT0.4640.5030.6480.6710.6320.6750.4930.5701.46
SFT + GRPO0.7070.7740.8490.8250.8440.8020.7830.7931.95
Tactic, original rewardsSFT0.4000.5450.6880.7270.7320.6820.5350.5991.57
SFT + GRPO0.7270.7750.8320.7880.8510.7130.8800.7872.47
Tactic, revised rewardsSFT0.4160.4940.6740.6790.7000.6850.5000.5781.46
SFT + GRPO0.7000.6920.8320.8850.9270.7680.7950.7812.07
GPT-5.6 Soldefault reasoning0.6150.7600.8200.7330.5500.7330.7000.7161.91
Claude Opus 5.5†default reasoning0.7070.8000.8600.9000.7000.7990.7950.7971.99
Gemini 3.1 Prodefault reasoning0.6000.6670.7470.7000.6000.7270.6250.6721.72
DeepSeek-V4-Prodefault reasoning0.5420.5600.7600.6550.6000.6340.6150.6241.94
Gold traces—0.6800.7520.9000.9001.0000.8160.8200.8182.01

Table 18. Round F1 on the 25 held-out AVCORP games (R1 to R3: 25 rounds each; R4: 15; R5: 10), with precision, recall, and F1 over all 100 rounds and the mean number of players named per round. Trace-only rows give the mean over three runs. †Reworded prompt. Gold traces are the corpus traces, scored the same way.

DetectorStageR3 (10)R4 (5)R5 (10)R1R2R3R4R5
Qwen3-8B, trace promptadapters off5010.200.321.001.271.30
Qwen3-8B, tactic promptadapters off6420.520.721.241.331.50
Trace, original rewardsSFT3.71.04.31.351.431.521.531.63
SFT + GRPO3.01.34.03.123.072.922.892.57
Trace, revised rewardsSFT4.32.03.31.331.401.531.441.80
SFT + GRPO7.33.05.71.731.961.992.042.27
Tactic, original rewardsSFT5351.201.521.721.672.10
SFT + GRPO6242.402.442.522.402.70
Tactic, revised rewardsSFT5341.081.241.681.732.00
SFT + GRPO7382.002.162.042.072.10
GPT-5.6 Soldefault reasoning9341.642.002.002.002.00
Claude Opus 5.5†default reasoning9361.962.002.002.002.00
Gemini 3.1 Prodefault reasoning7251.201.721.962.002.00
DeepSeek-V4-Prodefault reasoning7051.842.002.001.872.00
Gold traces—103102.002.042.002.002.00

Table 19. Game-ending verdict by the round at which a game ends (exact matches; 10 games end at R3, 5 at R4, and 10 at R5) and the mean of players named per round, on the 25 held-out AVCORP games. Notation as in Table 18.

Frontier Models in Avcorp

Table 21 compares the best detector with four frontier models, the gold traces, and two strategies that use no model. Each frontier model receives the tactic prompt without the Qwen3 switch /no_think, conditions on its own earlier traces, and runs at its provider’s default reasoning setting and temperature. It can generate up to 32,768 tokens per round, reasoning included. One DeepSeek-V4-Pro round that does not end a game reaches this cap and counts as an empty prediction. Claude Opus 5.5 refused 16 of the 25 games from the second round on, and Anthropic’s API labeled these refusals as reasoning extraction. For Claude Opus 5.5 alone, we reword the seven instruction phrases that mention reasoning or a trace (e.g., “reasoning trace” becomes “report”) and rerun all 25 games. After this change, the model answers every round. Naming every candidate reaches perfect recall but resolves no game, and naming two candidates at random resolves 16.7% of games in expectation.

ModelExact (%)Exact gamesF1
Best detector (tactic, revised, SFT + GRPO)72.018/250.882
GPT-5.6 Sol64.016/250.740
Claude Opus 5.5†72.018/250.800
Gemini 3.1 Pro56.014/250.680
DeepSeek-V4-Pro48.012/250.620
Gold traces92.023/250.960
Name every candidate0.00/250.667
Two at random (expected)16.7—0.500

Table 21. Game-ending verdict on the 25 held-out AVCORP games. Exact requires naming both Evil players and no one else. †Reworded prompt.

Avalon-NLU under Two Prompts

Table 22 reports every Avalon-NLU evaluation under two prompts: Minimal vs Domain-context. Table 2 reports the second. The minimal prompt adapts the AVCORP prompt to six players. It adds a paragraph of rules covering roles, team sizes, votes, and assassinations. This prompt uses the original history format and limits outputs to 2,048 tokens. The domain-context prompt adds guidance on weighing evidence, scores, quest records, message marking, and deduction sizes. This prompt uses the extended history from Appendix § 18.§ 3. It limits outputs to 3,072 tokens. Under both prompts, every trained detector exceeds the untrained model in round F1. The minimal prompt does not state the required deduction size. At the game-ending round, only 2 to 12 of the 20 deductions name exactly two players under the minimal prompt. In contrast, 10 to 20 deductions name exactly two players under the domain-context prompt. Under the minimal prompt, GRPO leads SFT on the game-ending verdict (3.4 against 3.1 exact games on average). However, GRPO trails SFT under the domain-context prompt (6.6 against 8.5 exact games). Under the minimal prompt, the trained detectors name the Investigator itself in 3 to 15 of the 82 rounds. The scorer removes these invalid names. The domain-context prompt explicitly marks the Investigator’s messages. Consequently, no detector names the Investigator under the domain-context prompt. Two domain-context outputs reach the 3,072-token limit. Both occur during SFT tactic runs at the final round of NLU-QdJ2dO. We count these truncated outputs as empty predictions. No detector resolves game NLU-3YjxjN under either prompt. Therefore, every exact count remains the same over the other 19 games.

DetectorStageMinimal prompt: ExactMinimal prompt: F1F_1Minimal prompt: F19F_{19}Minimal prompt: Round F1F_1Domain-context prompt: ExactDomain-context prompt: F1F_1Domain-context prompt: F19F_{19}Domain-context prompt: Round F1F_1
Qwen3-8B, trace promptadapters off10.3380.3610.24080.6670.6760.462
Qwen3-8B, tactic promptadapters off50.5480.5880.36140.5500.5530.432
Trace, original, run 1SFT40.6420.6670.51290.7160.7270.581
SFT + GRPO10.5920.5950.56760.6510.6580.581
Trace, original, run 2SFT40.5530.5750.51480.6670.6750.552
SFT + GRPO20.6250.6240.57970.6740.6830.604
Trace, original, run 3SFT30.5260.5480.502110.7560.7690.604
SFT + GRPO10.6420.6470.56150.6360.6430.532
Trace, revised, run 1SFT30.6340.6580.53890.7230.7340.586
SFT + GRPO50.6340.6580.48660.6750.6840.568
Trace, revised, run 2SFT20.6000.6230.50390.7070.7180.566
SFT + GRPO40.6810.6890.52680.7140.7250.615
Trace, revised, run 3SFT30.6130.6390.50660.6590.6670.533
SFT + GRPO60.6820.7060.588100.7650.7790.607
Tactic, originalSFT20.5370.5570.51070.6440.6750.526
SFT + GRPO20.5930.5960.55740.6460.6530.545
Tactic, revisedSFT40.6230.6490.51490.6580.6930.536
SFT + GRPO60.6890.7130.54470.6980.7070.580
Mean of 2adapters off3.00.4430.4740.3016.00.6080.6140.447
Mean of 8SFT3.10.5910.6150.5128.50.6910.7070.560
Mean of 8SFT + GRPO3.40.6420.6530.5516.60.6820.6910.579

Table 22. Every Avalon-NLU evaluation under the minimal and domain-context prompts. Exact counts the games, of 20, whose game-ending deduction names both Evil players and no one else; the counts are the same over the 19 games without NLU-3YjxjN. F1 scores the game-ending deduction over 20 games and F119_{19} over 19; Round F1 scores all 82 rounds. Every model conditions on its own earlier traces.

Outputs at the Length Cap

Table 23 lists every output that reached the 1,280-token cap on the held-out games. These truncated outputs lack a deduction. We count them as empty predictions. When using their own traces, the detectors produce eight truncated outputs out of 1,800. A truncated output enters the prompt for the next round. This propagation explains the consecutive truncated rounds in games G203 and G228. Game G228 is one of two held-out games where the gold trace names the wrong pair. Every trained detector also misses the correct pair in this game. Therefore, the truncation does not change the game-ending score. When using gold prior traces (Appendix § 18.8), only the untrained model with the tactic prompt reaches the cap. This truncation occurs in four rounds.

Prior tracesRunStageGameRoundsFinal
OwnTrace, original, run 1UntrainedG203R2, R3R3
OwnTrace, original, run 1SFTG088R5R5
OwnTrace, original, run 2SFTG045R2—
OwnTrace, revised, run 2SFT + GRPOG228R2, R3, R4R4
OwnTactic, originalUntrainedG045R5R5
GoldTactic promptUntrainedG089, G165R3, R3G165
GoldTactic promptUntrainedG191, G133R4, R5G133

Table 23. Outputs that reached the 1,280-token cap on the 25 held-out AVCORP games. Final marks a game’s last round, which the game-ending verdict scores.

Evaluation with Gold Prior Traces

Table 24 compares the game-ending verdicts using gold prior traces versus generated prior traces on the same 25 games. With gold prior traces, every stage resolves 72.0% to 92.0% of the games. This performance range includes the untrained model. Gold prior traces leak the answer. Copying the previous gold deduction without any model resolves 18 of the 25 games. Therefore, we evaluate every detector using its own traces. The gold-trace evaluations took 203 to 254 minutes per run. These eight runs required 29.1 GPU hours in total.

DetectorStageGold prior traces: ExactGold prior traces: F1Own prior traces: ExactOwn prior traces: F1
Qwen3-8B, trace promptadapters off72.00.88924.00.581
Qwen3-8B, tactic promptadapters off72.00.86348.00.717
Trace, original rewardsSFT82.70.90236.00.697
Trace, original rewardsSFT + GRPO89.30.94733.30.775
Trace, revised rewardsSFT76.00.86938.70.688
Trace, revised rewardsSFT + GRPO84.00.92764.00.861
Tactic, original rewardsSFT84.00.93152.00.747
Tactic, original rewardsSFT + GRPO88.00.94048.00.847
Tactic, revised rewardsSFT88.00.94048.00.735
Tactic, revised rewardsSFT + GRPO92.00.96072.00.882

Table 24. Game-ending verdict on the 25 held-out games with gold or own prior traces. Exact in %. Trace-only rows give the mean over three runs.

Assistance

We used AI coding assistants to support code development during this study.

References

  1. [1]Mrinal Agarwal, Saad Rana, Theo Sundoro, Hermela Berhe, Spencer Kim, Vasu Sharma, Sean O’Brien, and Kevin Zhu. 2025. WOLF: Werewolf-based Observations for LLM Deception and Falsehoods. In NeurIPS Workshop on Multi-Turn Interactions in Large Language Models.arxiv.org/abs/2512.09187
  2. [2]Anthropic. 2025. Disrupting the first reported AI-orchestrated cyber espionage campaign. Anthropic.
  3. [3]Todd S. Braver. 2012. The variable nature of cognitive control: A dual mechanisms framework. Trends in Cognitive Sciences, 16(2):106–113.
  4. [4]David B. Buller and Judee K. Burgoon. 1996. Interpersonal Deception Theory. Communication Theory, 6(3):203–242.
  5. [5]Ted Byrt, Janet Bishop, and John B. Carlin. 1993. Bias, prevalence and kappa. Journal of Clinical Epidemiology, 46(5):423–429.pubmed.ncbi.nlm.nih.gov/8501467
  6. [6]Colin F. Camerer, Teck-Hua Ho, and Juin-Kuan Chong. 2004. A cognitive hierarchy model of games. The Quarterly Journal of Economics, 119(3):861–898.DOI
  7. [7]Thomas L. Carson. 2010. Lying and Deception: Theory and Practice. Oxford University Press, Oxford, UK.
  8. [8]Juin-Kuan Chong, Teck-Hua Ho, and Colin Camerer. 2016. A generalized cognitive hierarchy model of games. Games and Economic Behavior, 99:257–274.DOI
  9. [9]Richard Christie and Florence L. Geis. 2013. Studies in Machiavellianism. Academic Press, New York, USA.
  10. [10]Daniel C. Dennett. 1987. The Intentional Stance. MIT Press, Cambridge, Massachusetts.
  11. [11]Alvan R. Feinstein and Domenic V. Cicchetti. 1990. High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6):543–549.pubmed.ncbi.nlm.nih.gov/2348207
  12. [12]Harry G. Frankfurt. 2005. On Bullshit. Princeton University Press, Princeton, NJ.
  13. [13]Chris Frith and Uta Frith. 2005. Theory of Mind. Current Biology, 15(17):R644–R645.pubmed.ncbi.nlm.nih.gov/16139190
  14. [14]Kilem Li Gwet. 2008. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1):29–48.
  15. [15]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In Proceedings of the 10th International Conference on Learning Representations (ICLR), Virtual. OpenReview.net.arxiv.org/abs/2106.09685
  16. [16]Yao Huang, Yitong Sun, Yichi Zhang, Ruochen Zhang, Yinpeng Dong, and Xingxing Wei. 2026. Deception-Bench: A Comprehensive Benchmark for AI Deception Behaviors in Real-world Scenarios. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/2510.15501
  17. [17]Bolin Lai, Hongxin Zhang, Miao Liu, Aryan Pariani, Fiona Ryan, Wenqi Jia, Shirley Anugrah Hayati, James Rehg, and Diyi Yang. 2023. Werewolf among us: Multimodal resources for modeling persuasion behaviors in social deduction games. In Findings of the Association for Computational Linguistics: ACL, pages 6570–6588, Toronto, Canada. Association for Computational Linguistics.DOI
  18. [18]J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174.pubmed.ncbi.nlm.nih.gov/843571
  19. [19]Timothy R. Levine. 2014. Truth-default theory (TDT): A theory of human deception and deception detection. Journal of Language and Social Psychology, 33(4):378–392.DOI
  20. [20]Zizhen Li, Chuanhao Li, Yibin Wang, Qi Chen, Diping Song, Yukang Feng, Jianwen Sun, Jiaxin Ai, Fanrui Zhang, Mingzhu Sun, and 1 others. 2025. InMind: Evaluating LLMs in capturing and applying individual human reasoning styles. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 5038–5076.arxiv.org/abs/2508.16072
  21. [21]Jonathan Light, Min Cai, Weiqin Chen, Guanzhi Wang, Xiusi Chen, Wei Cheng, Yisong Yue, and Ziniu Hu. 2025. Strategist: Self-improvement of LLM decision making via bi-level tree search. In The Thirteenth International Conference on Learning Representations (ICLR).arxiv.org/abs/2408.10635
  22. [22]Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. 2023. From text to tactic: Evaluating LLMs playing the game of Avalon. In NeurIPS Foundation Models for Decision Making Workshop, New Orleans, Louisiana. Curran Associates.
  23. [23]Steven A. McCornack, Karlyn Morrison, Jung E. Paik, Amy M. Wisner, and Xiangying Zhu. 2014. Information Manipulation Theory 2: A propositional theory of deceptive discourse production. Journal of Language and Social Psychology, 33(4):348–377.
  24. [24]Tsvetomila Mihaylova and André F. T. Martins. 2019. Scheduled sampling for transformers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 351–356, Florence, Italy. Association for Computational Linguistics.arxiv.org/abs/1906.07651
  25. [25]Earl K. Miller and Jonathan D. Cohen. 2001. An integrative theory of prefrontal cortex function. Annual Review of Neuroscience, 24(1):167–202.pubmed.ncbi.nlm.nih.gov/11283309
  26. [26]Matthew L. Olson, Nathan Ratzlaff, Maxwell Hinck, Thao Nguyen, Varun Lal, Joseph Campbell, Simon Stepputis, and S.-Y. Tseng. 2026. LieCraft: A multi-agent framework for evaluating deceptive capabilities in language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 37802–37809. AAAI Press. Long-horizon dialogue understanding for role identification in the game of Avalon with large language models. In Findings of the Association for Computational Linguistics: EMNLP, pages 11193–11208, Singapore. Association for Computational Linguistics.
  27. [27]OpenAI. 2026. The Hugging Face incident and the road ahead. OpenAI.
  28. [28]Peter S. Park, Simon Goldstein, Aidan O’Gara, Michael Chen, and Dan Hendrycks. 2024. AI deception: A survey of examples, risks, and potential solutions. Patterns, 5(5):100966.DOI
  29. [29]David Premack and Guy Woodruff. 1978. Does the chimpanzee have a theory of mind? Behavioral and Brain Sciences, 1(4):515–526.DOI
  30. [30]Qwen Team. 2025. Qwen3 technical report. CoRR, abs/2505.09388.
  31. [31]Fardin Saad, Pradeep K. Murukannaiah, and Munindar P. Singh. 2026. Theory of Mind in action: The instruction inference task in dynamic human-agent collaboration. Artificial Intelligence, 361:104621:1–104621:53.DOI
  32. [32]Ştefan Sarkadi. 2018. Deception. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, pages 5781–5782, Stockholm, Sweden. IJCAI.
  33. [33]Ştefan Sarkadi. 2023. An arms race in theory-of-mind: Deception drives the emergence of higher-level theory-of-mind in agent societies. In IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS), pages 1–10, Toronto, ON, Canada. IEEE.DOI
  34. [34]Ştefan Sarkadi, Alison R. Panisson, Rafael H. Bordini, Peter McBurney, Simon Parsons, and Martin Chapman. 2019. Modelling deception using theory of mind in multi-agent systems. AI Communications, 32(4):287–302.DOI
  35. [35]Bidipta Sarkar, Warren Xia, C. Karen Liu, and Dorsa Sadigh. 2025. Training language models for social deduction with multi-agent reinforcement learning. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 1830–1839, Detroit, USA. IFAAMAS.
  36. [36]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. W. Yu, and Daya Guo. 2024. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300.arxiv.org/abs/2402.03300
  37. [37]Munindar P. Singh. 1994. Multiagent Systems: A Theoretical Framework for Intentions, Know-How, and Communications. Number 799 in Lecture Notes in Computer Science. Springer, Heidelberg.
  38. [38]Simon Stepputis, Joseph Campbell, Yaqi Xie, Zhengyang Qi, Wenxin Zhang, Ruiyi Wang, Sanketh Rangreji, Charles Lewis, and Katia Sycara. 2023. Long-horizon dialogue understanding for role identification in the game of Avalon with large language models. In Findings of the Association for Computational Linguistics: EMNLP, pages 11193–11208, Singapore. Association for Computational Linguistics.
  39. [39]Jeffrey J. Walczyk and Caroline Fargerson. 2019. A cognitive framework for understanding development of the ability to deceive. New Ideas in Psychology, 54:82–92.DOI
  40. [40]Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. 2024. Boosting LLM agents with recursive contemplation for effective deception handling. In Findings of the Association for Computational Linguistics: ACL, pages 9909–9953, Bangkok, Thailand. Association for Computational Linguistics.
  41. [41]Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does LLM safety training fail? In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 80079–80110, Red Hook, NY, USA. Curran Associates, Inc.arxiv.org/abs/2307.02483
  42. [42]Andrew Whiten and Richard W. Byrne. 1988. Tactical deception in primates. Behavioral and Brain Sciences, 11(2):233–244.DOI
  43. [43]Wikipedia. 2025. The resistance: Avalon.
  44. [44]Shuang Wu, Liwen Zhu, Tao Yang, Shiwei Xu, Qiang Fu, Wei Yang, and Haobo Fu. 2024. Enhance reasoning for large language models in the game Werewolf. CoRR, abs/2402.02330.arxiv.org/abs/2402.02330
  45. [45]Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, and 10 others. 2025. The rise and potential of large language model-based agents: A survey. Science China Information Sciences, 68(2):121101.arxiv.org/abs/2309.07864
  46. [46]Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. 2023. Exploring large language models for communication games: An empirical study on Werewolf. CoRR, abs/2309.04658.arxiv.org/abs/2309.04658
  47. [47]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing reasoning and acting in language models. In Proceedings of the 11th International Conference on Learning Representations (ICLR), pages 1–33, Kigali, Rwanda. OpenReview.net.arxiv.org/abs/2210.03629
  48. [48]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 46595–46623, Red Hook, NY, USA. Curran Associates, Inc.arxiv.org/abs/2306.05685

Paper details

vlnc:W_tgh7v3zj
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents