betaFeedback

WorksComputer Science

CogRSI: Evolving and Auditing Cognitive Reasoning Strategies

Abstract

Can a language model learn better ways to reason about other people's minds by improving its own strategies, and are the strategies it learns actually right? We study this with cogrsi. A grounded reader marks who saw each event in a story and uses a short written principle to settle disagreements between its readings. With a principle written by experts, it beats self-consistency at the same token cost. A self-improvement loop then writes the principle itself. It keeps an update only if accuracy rises by a calibrated margin, and in one variant only if it also passes counterfactual test pairs. The learned principles transfer: on 800 BigToM pairs that no system had seen, each of them scores higher than the expert principle. We then check what the learned rules actually say. Each rule is translated into code and run on thousands of small stories with known answers. Most of them give a wrong answer in at least one situation they claim to cover, for example by treating a character who is in the room as knowing what happened even while asleep. Accuracy does not reveal which rules are wrong, and the counterfactual test does not fix this. A strategy can help in practice and still be wrong as a rule, so both need checking. Our code and data are publicly available.

Introduction

Recursive self-improvement (RSI) lets a model change its own artifacts, such as prompts, tools or code, and keep the changes that help (Wang et al., 2024; Yin et al., 2025; Zhang et al., 2026). Reasoning about other minds is a hard test for it. Theory of mind (ToM) is the clearest case: a system has to track what each person saw, was told and believes. Benchmarks probe these states in stories and conversations (Figure 1), and a skill that looks general can rest on surface cues (Ullman, 2023; Shapira et al., 2024).

The task: a strategy learned on one generator must track who saw what on benchmarks it never saw

Figure 1. The task: a strategy learned on one generator must track who saw what on benchmarks it never saw.

Self-improving systems keep an artifact when it raises a score [1] (Wang et al. 2024; Zhao et al. 2024). For reasoning about minds, two questions stay open. Does what the loop learns transfer beyond the data it learned from? And is a strategy that helps also correct? Take the rule “a character who is in the room knows what happens”. It raises accuracy whenever being there means seeing. It gives the wrong answer when a character is in the room but asleep.

We study both questions with CogRSI and make three contributions. (C1) A grounded reader. It marks who perceived each sentence or utterance, reads the story with and without these marks, and uses a short written principle to settle disagreements. (C2) Counterfactual-guided strategy evolution. A loop writes that principle as a small library of rules, each with a condition for when it applies. It keeps an update only if accuracy rises by a calibrated margin. In one variant, the update must also pass counterfactual pairs, two versions of a story that differ in one fact about who could see what. (C3) A semantic audit. We check what each learned rule says. Two independent models translate the rule into code, and we run the code on thousands of small stories whose answers follow from who saw or was told what. A rule is invalid if both translations give the same wrong answer on some story. In other words, following the rule as written leads to a wrong belief in a situation it claims to cover.

Figure 2. Overview. Rows: what persists, transfer, cognitive validity. Existing self-improving and ToM methods leave them open. Our suite measures them, and CogRSI addresses them.

The learned strategies transfer. On 800 BigToM pairs that no system had seen, every learned principle scores higher than the expert principle (§ 5.2). Yet 86.7% of the learned rules are invalid. Accuracy does not reveal which ones, and the counterfactual gate does not change this (§ 5.4). A self-improving system for reasoning about minds therefore needs two checks: how its strategies behave, and what they say.

Related Work

We review the closest lines of work. Appendix § 14 gives more detail and a comparison table.

Self-improving systems. Agents keep skills ((Wang et al., 2024)), insights ((Zhao et al., 2024)) or workflows ((Wang et al., 2025)). They improve their scaffolding ((Zelikman et al., 2024; Yin et al., 2025; Robeyns et al., 2025)), design agents ([18] (Zhang et al. 2026)) and evolve prompts and code ([8, 1, 28]). Recent work regularises these edits ((Wu et al., 2026b; Xia et al., 2026)) and checks a skill before keeping it. The check can be a replay on held-out tasks ([37]), a counterexample to the edited assumption ([27]), or a test of the rule as a falsifiable hypothesis ((Wang et al., 2026a)). Audits find self-scored gains that do not hold ([15] (Xu et al. 2026)), edits that only look like gains ((Wang et al., 2026b)), selections that miss held-out gains ([25]), and skill libraries that add little over plain context ([14]). All of these checks measure task success. We also check what an admitted rule says, against a formal model of the domain.

a counterexample to the edited assumption ([27]), or a test of the rule as a falsifiable hypothesis ((Wang et al., 2026a)). Audits find self-scored gains that do not hold ([15] (Xu et al. 2026)), edits that only look like gains ((Wang et al., 2026b)), selections that miss held-out gains ([25]), and skill libraries that add little over plain context ([14]). All of these checks measure task success. We also check what an admitted rule says, against a formal model of the domain.

Theory of mind and formal checks. ToM methods take a character’s perspective by filtering or prompting ([19, 32] (Wilf et al. 2024; Shinoda et al. 2026)). Others track beliefs symbolically or over time ([35, 17]), verify belief traces ((Wu et al., 2025)), or use model-based inference ([21] (Zhang et al. 2025)). Training with reinforcement learning ([26]) does not transfer ([33]). None of these methods learns its strategy from failures. Our counterfactual pairs follow contrast sets ([10, 20]). Translating natural language into checkable logic follows nl2spec ([6]), and we apply the idea to strategies that an RSI loop has written.

Task and Evaluation

Appendix § 11 details the evaluation suite (Figure 11).

Figure 11. Figure 11: Task and suite. (A) Task. (B) Eight benchmarks. (C) Sealed, disjoint samples. (D) Native scoring.

Task. A context c=(u1,…,un)c = (u_1, \ldots, u_n) is a story of sentences or a conversation of utterances about agents A\mathcal{A}. A question qq asks about the mental state of one or more agents, and a system answers y^=F(q,c)\hat{y} = F(q,c) in the benchmark’s format. In artifact-level RSI, a builder model reads the current system’s failures on development data D\mathcal{D} and proposes an artifact xx. Following MedRSI (Wu et al., 2026a), xx is registered (kept) if its mean accuracy gain over KK fresh trial cohorts exceeds a margin δ\delta,

reg⁡(x)⟺1K∑k=1KΔAk(x)>δ.(1)\operatorname{reg}(x) \Longleftrightarrow\frac{1}{K}\sum_{k=1}^{K}\Delta A_k(x) > \delta. \tag*{(1)}

All of D\mathcal{D} comes from one story generator, ExploreToM [34]. The loop sees no other data.

Three kinds of evidence. Task accuracy comes from eight benchmarks: FANToM [22], SimpleToM [13], ExploreToM, Hi-ToM [16], OpenToM (Xu et al., 2024), ToMBench [5], CogToM (Tong et al., 2026) and BigToM [9]. The main test sample has 3,349 questions. A confirmatory sample of 2,470 questions was drawn after every design decision, and seven more benchmarks are used for testing only. Behavioural validity comes from counterfactual pairs. A pair counts only if both versions are answered right. Our own generator gives a 96-pair gate split, the only one any loop sees, and a 320-pair evaluation split. Its variants cover a character who watches, leaves, sleeps, is told, is lied to or looks later. BigToM-U adds 800 independent pairs from BigToM conditions that no system had answered. In a flip pair the agent does or does not see a change. In a same pair the agent does or does not see an irrelevant event. Semantic validity asks whether a learned rule is correct (§ 4.3). Each benchmark scores answers with its own rule, and Main is the mean over the eight benchmarks. Differences use a paired bootstrap over stories, resampling seeds and scenarios for the loops.

Methodology: CogRSI

CogRSI has three parts: a reader, loops that write principles for it, and an audit of what the loops write (Figure 3, Algorithms 1 and 2).

Figure 3. CogRSI. (A) RSI over procedures writes tools that export perceptual access. (B) RSI over principles keeps a library of conditioned strategies and admits an update only if it passes the accuracy margin and, in the gated variant, the counterfactual gate. (C) The reader applies the strategies at test time. The semantic audit (§ 4.3) checks what the admitted strategies assert.

The Grounded Reader (C1)

One perception call per context labels each sentence or utterance with the characters who witnessed it. It uses the perception rule of Jung et al. [19] with indexed outputs. PercepToM deletes what a character did not see. Our reader keeps the whole story and adds who saw what. It draws six samples at temperature 0.7, the budget of self-consistency with k=6k=6. Four samples read the annotated story, or the questioned character’s view for conversations, and two read the plain story. If the two readings conflict or the samples split closely, one call picks between the two leading answers under a principle PP. It never overturns an answer held by five or six samples (Proposition 3). CogRSI uses an expert principle: a character knows only what the story shows them witnessing, being told, or working out from what they themselves did. The loops below write PP instead.

Counterfactual-Guided Strategy Evolution (C2)

Strategies. The loop keeps a library LL of up to five rules. Each rule has the form When condition: rule, with at most 60 words. The library fills the principle slot of the reader. In each round the builder sees LL and up to four failures from one group of 80 new development questions. It then proposes one change: ADD, REVISE or RETIRE a rule. The builder is never told what makes a rule valid. Up to four candidate changes compete per round.

Calibrated registration. The same principle varies by 2.7 points across trial cohorts of 80 questions. With four candidates per round, a 1-point margin keeps a candidate with no real gain in 70% of rounds. We therefore set δ=3\delta=3 points and also require a gain of at least zero on two of the three cohorts.

Counterfactual guidance. The gated variant also asks a candidate to keep the 96 gate pairs right. Let ΔV\Delta V be the change in correctly answered gate pairs and ΔA‾\overline{\Delta A} the mean accuracy gain. Then

reg⁡G⟺[reg⁡∧ΔV≥−3]∨[ΔV≥6∧ΔA‾≥−0.5].(2)\operatorname{reg}_{G} \Longleftrightarrow[\operatorname{reg} \land\Delta V \geq-3] \lor[\Delta V \geq6 \land\overline{\Delta A} \geq-0.5]. \tag*{(2)}

where the tolerances come from the reader’s run-to-run noise. The second term keeps a candidate that fixes several pairs at no accuracy cost. A pairs-as-data variant uses the same 192 gate items as ordinary trial items with accuracy-only registration. It sees the same information without the pair constraint. We also run an accuracy-only library, a gated single principle, an accuracy-only loop with a 1-point margin that writes one principle, and GEPA (Agrawal et al., [1]) in the same slot. Appendix C.1 reports a loop that writes executable tools.

Semantic Audit (C3)

A formal model of access. The audit needs a ground truth for what characters believe. We use the account that seeing leads to knowing (Wimmer et al., 1988), plus testimony. In one room, agents leave and return, fall asleep and wake, move an object, whisper its location truly or falsely, peek into a container, or change a visible or hidden property of the object. An event updates the beliefs of everyone who perceives it, and a whisper reaches only its listener. A fixed sample of 6,000 small worlds gives about 85k questions with known answers.

Translation and counterexamples. Two models translate each rule into code: Llama-3.3-70B and Qwen3.5-122B, neither of which wrote the rules. Each writes applies(world, q) for the rule’s condition and predict(world, q) for its answer. They are told to follow the words literally and do not know which loop wrote the rule or whether it was kept. A shared counterexample is a question where both translations give the same answer and the formal model gives a different one. Two independent readings then agree on what the rule says, and it is wrong. A rule is invalid if it has a shared counterexample. It is valid if it has none and the two translations agree on at least 80% of questions. It is vacuous if it almost never applies and ambiguous otherwise. Finally, the reader answers rendered counterexamples with and without the rule, to see whether they change its behaviour.

Experiments

We ask four questions. RQ1: How strong is the reader? RQ2: Do evolved strategies transfer? RQ3: What does registration buy? RQ4: What do evolved strategies say?

Setup. The main model is Qwen3.5-35B-A3B (FP8) (Qwen Team, [30]). The reader also runs on Qwen3.5-122B-A10B, Llama-3.3-70B-Instruct (Grattafiori et al., [12]), GLM-4.6V-Flash (GLM-V Team, [11]) and Qwen3.5-9B, served with vLLM (Kwon et al., [24]) with thinking off. Baselines are zero-shot, chain of thought (CoT) (Kojima et al., [23]), self-consistency (SC) (Wang et al., 2023) with k=6k=6 and with k=9k=9 (the reader’s token budget), retrieval, prompt-only GEPA, SimToM (Wilf et al., 2024) and PercepToM (Jung et al., [19]). SimToM and PercepToM also run with our six samples. Each loop runs eight rounds on ExploreToM development data. Every experiment after the design freeze was pre-registered with its

Principle written byRegistrationAccuracyPair validityAudit
Dev.Test MainOwn pairsBigToM-UInvalid rules (%)
no principlenone59.770.191.794.0n.a.
hand (expert)none59.773.191.794.5n.r.
accuracy-only loop1-pt63.871.891.595.989.7
GEPA (same slot)Pareto67.075.384.4n.r.
accuracy-only library3-pt66.171.5{sup}†`91.2{sup}†`96.885.2{sup}†`
pairs as data, library3-pt63.1n.r.n.r.96.6n.r.
gated single principle3-pt + gate62.171.998.196.582.7
gated library3-pt + gate65.871.896.496.686.0

caption: tbl:1: Who writes the principle. Same reader, backbone and sampling; only the principle changes. Registration: accuracy margin in points, with the counterfactual gate where stated (Pareto: GEPA’s own selection). Dev.: 300 unseen ExploreToM development questions per seed. Test Main: main test sample, eight benchmarks. Own pairs: 320 held-out pairs from our generator. BigToM-U: 800 BigToM pairs in conditions no system had seen. Invalid rules: share of the arm’s proposed rules with a formal counterexample (§ 4.3). Seeds: 6 for loops and libraries, 3 for GEPA and the single principle; †seeds 0–2 only. n.r.: not run; n.a.: not applicable. Every learned principle beats the expert one on BigToM-U; the gate changes validity on its own pairs only.

:::

hypotheses and thresholds (Appendix § 10), and we report every outcome.

The Reader (RQ1)

With the expert principle, CogRSI scores 73.1 Main, against 68.7 for SC (k=9k=9) at the same token cost (Table 3). It scores 73.5 against 68.5 on the confirmatory sample and beats SC (k=9k=9) on all five backbones. FANToM and SimpleToM carry 81% of the gain. On the seven test-only benchmarks it scores 65.2, against 66.2 for SC (k=9k=9). Inside the same reader, our annotated view beats PercepToM’s filtered view by +1.3 narrative Main points ([+0.7, +2.0]) and SimToM’s view by +0.9 ([+0.1, +1.7]). With learned principles, the reader scores 71.8–71.9 Main (Table 1).

MethodFANSimpleExploreHiOpenTBenchCogBigMain
Qwen3.5-35B-A3B (main backbone; MoE 35B / 3B active, FP8, thinking off)
Native solver (floor)3.750.967.255.866.570.179.783.359.7
Zero-shot8.0+4.360.8+9.964.9−2.458.4+2.668.4+1.972.1+2.079.7−0.084.9+1.762.1+2.5
CoT13.2+9.469.1+18.269.6+2.351.8−4.072.5+5.971.7+1.782.5+2.988.4+5.164.8+5.2
SC (kk=6)17.6+13.970.2+19.372.6+5.462.2+6.475.0+8.573.9+3.885.5+5.989.2+5.968.3+8.6
SC (kk=9)18.0+14.371.2+20.372.1+4.864.9+9.274.4+7.973.3+3.285.9+6.389.4+6.168.7+9.0
Retrieval1.8−2.049.4−1.568.8+1.658.6+2.865.9−0.770.8+0.776.4−3.383.1−0.159.3−0.3
GEPA0.4−3.445.7−5.271.8+4.646.5−9.265.9−0.667.5−2.674.8−4.981.7−1.656.8−2.9
PercepToM2.0−1.887.5+36.667.7+0.548.2−7.668.4+1.972.0+1.979.7+0.184.8+1.563.8+4.1
PercepToM + SC (belief)‡23.7+20.0————————
PercepToM + SC3.7−0.086.2+35.362.2−5.055.7−0.175.6+9.174.3+4.284.8+5.189.5+6.266.5+6.9
SimToM1.1−2.778.0+27.175.2+8.037.6−18.274.1+7.567.9−2.275.7−4.085.1+1.861.8+2.2
SimToM + SC1.8−2.079.0+28.176.9+9.641.6−14.176.5+9.969.6−0.580.5+0.984.9+1.663.9+4.2
Lib-C (rebuilt library)2.7−1.151.8+0.972.2+4.953.9−1.966.6+0.168.3−1.872.3−7.380.9−2.458.6−1.1
Lib-A (inherited library)2.3−1.454.6+3.773.9+6.753.1−2.765.9−0.768.8−1.375.6−4.180.4−2.959.3−0.3
CogRSI-L (loop-written principle)25.3+21.681.7+30.874.0+6.870.0+14.276.5+9.972.8+2.784.7+5.089.6+6.371.8+12.2
CogRSI28.0+24.290.0+39.172.5+5.369.3+13.575.8+9.374.1+4.084.9+5.290.3+7.173.1+13.5
Qwen3.5-122B-A10B (MoE 122B / 10B active, FP8)
Zero-shot (floor)3.070.070.068.864.677.279.185.964.8
CoT11.6+8.678.5+8.672.9+2.955.6−13.173.1+8.574.8−2.484.9+5.789.0+3.167.5+2.7
SC (kk=6)18.2+15.280.8+10.977.5+7.668.0−0.876.3+11.778.0+0.787.8+8.789.9+4.072.1+7.2
SC (kk=9)16.9+13.981.9+12.077.5+7.568.7−0.176.0+11.478.0+0.888.5+9.389.1+3.272.1+7.3
PercepToM6.2+3.288.2+18.370.4+0.548.0−20.863.7−0.977.8+0.578.9−0.386.2+0.364.9+0.1
CogRSI25.3+22.394.0+24.178.4+8.572.4+3.778.6+14.078.2+1.087.9+8.889.7+3.875.6+10.8
Llama-3.3-70B-Instruct (dense 70B, AWQ 4-bit)
Zero-shot (floor)0.072.570.965.969.075.881.487.765.4
CoT1.6+1.686.0+13.677.9+6.965.5−0.472.1+3.176.4+0.685.2+3.888.0+0.369.1+3.7
SC (kk=6)3.7+3.787.1+14.777.0+6.168.9+2.972.9+3.975.4−0.483.8+2.488.0+0.369.6+4.2
SC (kk=9)4.3+4.386.8+14.477.2+6.269.0+3.172.8+3.876.2+0.383.9+2.587.9+0.269.8+4.3
PercepToM1.1+1.185.0+12.678.0+7.159.1−6.869.0−0.075.8−0.081.4−0.087.1−0.667.1+1.7
CogRSI12.8+12.896.6+24.275.9+5.069.7+3.875.7+6.774.9−0.983.1+1.789.4+1.772.3+6.9
CoT14.866.463.960.070.278.685.591.766.4
SC (kk=6)17.666.866.764.372.281.487.393.368.7
SC (kk=9)16.666.767.165.271.581.187.791.768.5
PercepToM2.184.769.451.465.076.881.790.065.1
PercepToM + SC (belief)‡26.2————————
SimToM2.172.868.931.474.871.079.988.361.2
CogRSI (vote only)25.787.168.365.774.679.886.788.372.0
CogRSI (no close-split)27.288.366.962.975.280.688.891.772.7
CogRSI-L (loop-written principle)29.380.470.666.773.979.988.392.572.7
CogRSI28.888.867.565.774.580.488.393.973.5

Table 3. Main results. (A) Main test sample, three backbones. (B) Confirmatory sample. Accuracy (%) is the mean of runs: three in A, and in B three for CogRSI and SC (kk=9), six loop seeds for CogRSI-L, one for the rest. Main averages the eight. Subscripts are gaps to the block floor. Bold is best, underline second. FAN is FANTоM All*. CogRSI has the best Main in every block. ‡FANTоM only. —: does not apply.

Each part adds, at equal cost. The expert principle adds +3.0 Main points over no principle ([+2.2, +3.7]). Even without a principle, the reader beats SC (k=9k=9) by +1.5 ([+0.6, +2.4]). Each reading and each arbitration step raises belief accuracy (Table 2, Figure 8). CogRSI makes 7.3 calls per question and uses 1.51× the tokens of SC (k=6k=6). Against SC (k=9k=9) at the same token cost, the paired intervals exclude zero on FANToM, SimpleToM and Hi-ToM (Figure 9, Table 4). In a ToMBench case (Figure 10), every other system calls a lie true. The two readings of CogRSI disagree, and arbitration picks “false” in all three runs.

(a) Whole systemAll questionsNarrative
St.VariantMainFANBeliefNon-lit.Narr.
1Lib-A library59.3−13.82.3−25.767.1−15.176.3−1.969.1−13.0
Lib-C (rebuilt)58.6−14.52.7−25.365.5−16.676.6−1.568.6−13.5
2no principle70.1−3.023.2−4.878.6−3.579.8+1.779.8−2.4
CogRSI-L principle71.8−1.325.3−2.780.5−1.678.4+0.380.8−1.3
3− close-split arb.72.6−0.527.3−0.781.9−0.278.2 0.081.7−0.4
− all arbitration72.2−0.928.7+0.780.8−1.375.5−2.680.7−1.4
CogRSI (full)73.128.082.178.282.1
(b) Stage-3 readingsNarrative questions
St.VariantBeliefKnowl.Emot.Non-lit.Narr.
3annotated only78.4−3.780.1−5.473.1−2.672.9−5.378.1−4.0
plain only76.9−5.282.8−2.777.2+1.480.5+2.478.6−3.5
vote of both80.8−1.383.2−2.474.8−0.975.5−2.680.7−1.4
CogRSI (full)82.185.575.778.282.1

Table 2. Ablations by stage (main model). (a) One part removed or swapped. (b) Stage 3’s readings. Subscripts: gap to full CogRSI. Rows compare whole systems; no Stage-1 output enters Stages 2 or 3. The reader carries the gain; the hand-written principle adds most within it.

CogRSI benchmark radar chart

Figure 8. Table 3(A) by benchmark (main backbone; rings at 25, 50, 75, 100). CogRSI encloses every baseline on FAN, Simple and Hi.

Cost and robustness plots

Figure 9. Cost and robustness. (a) Main accuracy vs. tokens (log, relative to CogRSI). (b) Four-benchmark mean by backbone, with SC (k=6k=6) and (k=9k=9) (70B is Llama-3.3). CogRSI is on the frontier and leads SC at equal tokens on every backbone.

Bench.Δ95% CI
FAN+10.0[+4.0, +15.7]
Simple+18.9[+15.0, +22.7]
Explore+0.5[−1.5, +2.3]
Hi+4.4[+1.2, +7.6]
Open+1.4[−0.4, +3.1]
TBench+0.8[−0.7, +2.2]
Cog−1.1[−2.3, +0.1]
Big+1.0[−0.1, +2.2]

Table 4. Paired test: CogRSI − SC (k=9k=9) at equal tokens. Significant on FAN, Simple and Hi; Hi not after Holm correction.

Case study diagram

Figure 10. Case study. A ToMBench egocentric lie, abridged (full text and logs in Table 28). Chips are CogRSI’s six samples in run 0. The readings disagree, and arbitration picks the right answer.

Evolved Strategies Transfer (RQ2)

Beyond the expert on independent pairs. On BigToM-U, every learned principle scores above the expert one (Table 1). The gated library reaches 96.6% pair validity, against 94.5 for the expert principle (+2.1, [+1.0, +3.5]) and 94.0 with no principle (+2.6, [+1.4, +3.9]). The accuracy-only library gains +2.8 over no principle. The gain comes from flip pairs, where the agent does or does not see the change (94.9 vs. 91.2). On same pairs every principle stays near ceiling (98.2 vs. 97.8), so learned rules do not make the reader see missed events where there are none. SC (k=9k=9) reaches 93.0, below the reader without a principle.

The gate adds nothing here. The pre-registered primary contrast, gated minus accuracy-only library, is −0.2 ([−1.4, +0.8]). Gated minus pairs-as-data is −0.1 ([−1.1, +0.9]). The transfer comes from evolving the principle.

Development gains do not reach test Main. The accuracy-only loop’s principle beats the expert one on new development questions in most seeds (63.8 vs. 59.7, Figure 12), yet it loses on test Main. A loop that writes executable tools shows the same pattern. Its library lifts ExploreToM from 67.2 to 73.9 but leaves FANToM flat (Figure 5), and inheriting it instead of rebuilding it changes FANToM All* by +0.5 ([−0.5, +1.5]).

RSI over procedures: tools encoding access and gains near the source

Figure 5. RSI over procedures. (a) Candidate tools per E1 run, filled if they encode access. (b) ExploreToM accuracy, native solver → Lib-A. Every run fills the access slot, and its library helps only near its source.

RSI over principles

Figure 12. RSI over principles. (a) Incumbent’s accuracy on new development questions by round (mean and range over 6 seeds; round 0 admits the first principle). (b) 300 new development questions on seen stories, per seed (no, hand-written, loop-written principle). (c) Test accuracy per benchmark, Lib-A → CogRSI-L, main test sample. The loop’s rule beats ours on development questions in most seeds, not on test.

Limits. On test Main the gated library trails the expert principle by −1.3 ([−1.9, −0.6]) and is level with the accuracy-only loop (0.0). GEPA writes prompts of 1,838–2,383 words. They reach 84.7 on ExploreToM, 67.0 Main and 84.4 on BigToM-U. When a belief must be inferred backwards from an action, every system stays near 68.0% pair validity (37.2% on flip pairs), and no principle helps (+0.2, [−2.5, +2.8]).

What Registration Buys (RQ3)

A calibrated margin makes admissions precise. We scored every candidate on 120 held-out development questions. Of the updates the 1-point loop kept, 63.6% help. With the 3-point margin, 83.3% help, with or without the gate (6 and 12 kept updates, Table 5). The margin halves the share of harmful updates, from 31.8 to 16.7%, and keeps fewer helpful ones.

RegistrationAdm.Prec.Harm.RecallMissed
1-pt margin4663.631.823.722.9
3-pt margin2583.316.721.729.2
3-pt + gate (single)2583.316.717.225.0
3-pt + gate (library)2583.316.713.043.8

Table 5. Selection quality (%, development data). Every candidate scored on 120 held-out questions. Adm.: rounds with an admission; Prec./Harm.: admitted updates that help/hurt; Recall: helpful candidates admitted; Missed: rounds with a helpful candidate and no admission. The calibrated margin, not the gate, makes selection precise.

Counterfactual items help, with or without the gate. On the evaluation split of our own generator, the gated library keeps 96.4% of pairs right, against 91.2 for the accuracy-only library (+5.1, [+3.6, +6.8]). Almost all of this gain is on one case: a character misses a move and later looks (Figure 6). On the development gate pairs, the pairs-as-data library keeps 79.2 of 96 right and the gated library 78.7. The accuracy-only library keeps 74.8, with seeds ranging from 67 to 84. The counterfactual items carry the gain, and the pair constraint adds little. We cannot show that the gated library is less accurate on development (+2.7, [−1.8, +6.6] against pairs-as-data).

Validity versus accuracy and accuracy by access variant

Figure 6. Validity is a separate target. (a) Test Main against pair validity on the 320 held-out counterfactual pairs, means over seeds or runs. (b) Accuracy by access variant on the same items. Accuracy-only principles cluster near the reader with no principle, GEPA falls off both axes, and the gated library moves up but not right.

What Evolved Strategies Say (RQ4)

Most evolved rules fail somewhere. The principle loops proposed 573 rules. 286 are invalid: for each, there is a story it claims to cover where following it gives the wrong answer. 44 are valid, 84 are ambiguous because the two translations disagree, and 159 almost never apply. Among the rules the audit could decide, 86.7% (80.7, 92.2)

ends_mid=1 are invalid, in every loop (Table 1). In the pre-registered run, one translator hit its output limit on 253 rules. We recompiled them with a larger limit and report both sets of verdicts (before: 165 invalid and 270 ambiguous, Appendix § 10).

Accuracy cannot filter them. We replayed each registration rule on the same candidates. Among the rules a 1-point margin would keep, 84.8% are invalid. The 3-point margin gives 89.3% and the gate 88.2% (Figure 7a). Valid and invalid rules have the same trial gain: −1.22 against −1.47 points, a difference of +0.24 ([−0.82, +1.32], Figure 7b). The two gain distributions differ by only 0.16 in total variation. Any rule that keeps candidates by trial gain then mislabels valid and invalid rules at rates that sum to at least 0.84 (Guo et al., [15]). The training data explains part of this. 116 of the 286 invalid rules fail only in situations ExploreToM never contains, such as sleeping, private telling, lies and peeking. Invalid rules also pile up in the libraries over rounds (Figure 7c). Only 3 of the 28 decided rules that pass the 3-point margin are valid.

Semantic audit of evolved rules

Figure 7. Semantic audit of 573 evolved rules. (a) Share of decided rules that are invalid among all proposed rules and among those each registration rule would admit, replayed on the same candidates. (b) Trial gain of valid and invalid rules. (c) Invalid and valid entries in the libraries by round, summed over seeds. Accuracy does not tell valid from invalid rules, and the gate does not change what is admitted.

What each loop kept. Figure 4 follows three loops round by round. Among kept rules the audit could decide, the invalid ones number 14/14 for the accuracy-only loop, 5/5 for the accuracy-only library, 7/9 for the gated library and 3/4 for the gated single principle.

Registration as the loop sees it: gated library, accuracy-only library, and accuracy-only loop across rounds

Figure 4. Registration as the loop sees it. Every candidate that reached registration, in order (four per round), placed at the incumbent’s running level plus its mean trial gain. Kept candidates are coloured by the semantic audit (§ 4.3). Levels are the loops’ own trial estimates, which overstate held-out gains (Table 5). Each loop keeps climbing on its own trials, and most of what it keeps is invalid.

An example. One kept rule of a gated library reads when a question asks whether A believes B knows the resulting state, and B was present when the event occurred: assume A believes B knows it (Box (g)). Both translations answer “yes” when B was in the room but asleep. The formal model answers “no”, because being there is not seeing. ExploreToM never puts a character to sleep, so no trial cohort could penalise this rule. It also passed the gate, because none of the 96 gate pairs asks this second-order question.

Counterexamples matter, but only a little. On 1139 rendered counterexamples, the rule turns a right answer wrong in 8.6% of cases. On 176 random control stories it does so in 2.8%.

The conditions do not decide when a rule applies. Removing every condition from the gated libraries changes BigToM-U pair validity by −0.5 (−1.6, +0.4). Giving each rule another rule’s condition changes it by −0.2 (−1.2, +0.7). The reader treats the rules as general hints and weighs them against the story.

Discussion: What Does the Loop Learn?

The evolved strategies are useful, and most of them are wrong as rules. The reader explains how both can hold. It weighs a rule against a story in which perception is already marked. It can follow a wrong rule where it helps and override it where the story disagrees, which fits the small effect of counterexamples and the irrelevant conditions. A strategy handed to another reader or a person carries the rule itself, so the rule needs its own check. The expert principle is the opposite case. It is correct and leads on test Main, where conversations dominate, but it loses on BigToM-U, where seeing a behaviour that helps this reader, and it has written that behaviour down as rules that do not hold on their own.

Conclusion

CogRSI evolves strategies for reasoning about other minds. The evolved strategies transfer to independent BigToM pairs, where the gated library reaches 96.6% pair validity against 94.5% for an expert principle. A calibrated margin raises the share of helpful updates from 63.6% to 83.3%. A semantic audit shows that 86.7% of the rules the loops propose give a wrong answer in some situation they claim to cover. Neither accuracy nor counterfactual gating catches this. Self-improving systems that reason about minds should check both what their strategies do and what they say, and using the audit during registration is the natural next step. Our replay suggests that few current candidates would pass, which moves the bottleneck from selecting strategies to proposing correct ones.

Limitations

  • All loops learn from one story generator. The formal model covers physical access and testimony in one room, and other mental states are left for future work.

  • The audit relies on two models translating rules into code. Shared counterexamples guard against a single mistranslation, and both translations can still miss what a rule’s author meant.

  • Loops run three to six seeds, and some arms were not evaluated on every test set (Table 1).

  • All benchmarks are in English and mostly synthetic.

Ethics, AI-Use and Reproducibility Statements

Ethics. Systems that infer what a person knows or believes could be used to profile real people from their conversations. We use only fictional stories and synthetic dialogues and train no models.

Use of AI assistants. AI assistants helped write and review code, run the experiments, and draft and edit text. The authors directed and checked this work and are responsible for all content.

Reproducibility. Our repository contains the code, prompts, sample manifests, frozen configurations and the scripts that generate every table, figure and number from the per-run outputs. Appendix § 7 gives all prompts, Appendix § 9 the algorithms and settings, Appendix § 11 the samples and metrics, and Appendix § 13 every run. Code: https://github.com/koanatakiyo/cogRSI.

Prompts

All prompts of CogRSI (Boxes (a)–(e)), read from the frozen source: the prompt text is verbatim, bold headers and mode tables are layout, and monospace braces mark placeholders. Every call uses the system message of Box (a); samples are drawn at temperature 0.7, the perception and arbitration calls at temperature 0. The perception call (one per story or conversation) uses PercepToM’s perception rule with indexed outputs constrained by a JSON schema of the unit IDs; in conversations, belief questions about named characters use a perspective view. The principle slot holds the hand-written principle (CogRSI), a loop-written one (CogRSI-L; Box (f)) or nothing (Table 2).

Solver System

Instructions

This is a theory-of-mind test. Answer the question about what the characters perceived, know, or believe, based only on the given story or conversation.

Answer format

{answer format of the benchmark}

Reply

StyleText
defaultReply only as JSON: {"reasoning": "{step-by-step reasoning}", "answer": {answer}}. Think step by step in the reasoning field before giving the answer.
concise retryReply only as JSON: {"reasoning": "{brief step-by-step reasoning, at most 120 words}", "answer": {answer}}. Reason briefly before giving the answer.

(a) Solver: the system message of every call; the concise reply is the retry style.

Perception call User Prompt

Input

Numbered source units in original order:

[{"id": "s0001", "text": "{unit 1}"}, {"id": "s0002", "text": "{unit 2}"}]

Perception rule

FormatRule
narrativesAssume that characters in the story can perceive every scene occurring in their location but not scenes occurring elsewhere. Also, include the actant of any action as a perceiver of that action.
conversationsAssume that characters in the story can hear every utterance that occurs while they are involved in the dialogue, but not those that occur when they are absent. Also, ensure that the speaker of each utterance is included in the audience.

Output

Return only a JSON object mapping EVERY provided unit ID to its list of perceivers/audience. Do not output source text, explanations, or new IDs. Use consistent character names from the source. An individual list may be empty when no perceiver is supported; do not invent a perceiver just to fill a list. Never omit a unit ID. These encoding rules do not change the perception rules above.

(b) Perception call, one per story or conversation.

Annotated view User Prompt

Instructions

Each sentence or utterance below is followed by [perceived by: ...], the characters who witnessed it. A character knows only what they perceived or were told; use these annotations to track what each character knows and believes.

Story

{sentence 1} [perceived by: {names}] {sentence 2} [perceived by: {names}] ...

(c) Annotated view: each unit with its perceivers.

Perspective view User Prompt

Header before the visible utterances

QuestionHeader
first orderHere are the past utterances in sequence that {char1} is aware of.
second orderHere are the past utterances in sequence that {char1} thinks {char2} is aware of.

(d) Perspective view for belief questions in conversations.

Input{story and question as in the benchmark}
DisagreementTwo careful readings of this story disagree: one answers {A}, the other answers {B}.
Rule
ArbiterText
evidential (CogRSI)Decide which is correct. A character knows only what the story shows them witnessing, being told, or working out from what they themselves did. Do not credit a character with knowing about an event unless the story shows them present and noticing it; an event that happens while they are away, busy, or not shown noticing stays unknown to them, and their later actions can reveal what they did or did not notice. If the question does not depend on who knows what, choose the better answer on the story’s merits.
neutral (control)Decide which is correct.

caption: (e) Arbitration of view conflicts and close splits.

:::

Proofs

Proposition 1 (Access determines location beliefs). If agents perceive exactly the events at their location and α\alpha is correct, an agent’s first-order belief about an object’s location is the location set by the last move of the object in the agent’s perspective view.

Proof. Let oo change location only through units that place it (an initial description or a move), and let agents perceive exactly the units at their location, with no communication. Write VaV_a for agent aa’s perspective view. By induction over tt, after units u1,…,utu_1,\ldots,u_t, aa’s belief about oo is the location set by the last placing unit in Va∩{u1,…,ut}V_a\cap\{u_1,\ldots,u_t\} (undefined if there is none). For t=0t=0 there is no belief. Step t→t+1t\to t+1: if ut+1∉Vau_{t+1}\notin V_a, aa did not perceive it (α\alpha is correct), so aa’s belief is unchanged, and so is the last placing unit in aa’s view. If ut+1∈Vau_{t+1}\in V_a places oo, aa perceives the placement and believes it, which is the new last placing unit in VaV_a; if it does not place oo, neither side changes. Second-order questions use the chain view; this matches second-order beliefs when co-present agents know of each other’s presence, as ExploreToM’s generator assumes.

Proposition 2 (Registration only climbs). Along a seed of the principle loop, every admitted principle beats the incumbent it replaces by more than δ\delta in mean accuracy on the KK trial cohorts of its round, and the reader changes only when a principle is admitted.

Proof. In each round every candidate xx and the incumbent PP answer the same KK trial cohorts, and ΔAk(x)\Delta A_k(x) is the accuracy of xx minus that of PP on cohort kk. By Eq. (1) the candidate with the largest mean gain replaces PP only if that mean exceeds δ\delta. If none is admitted, PP is kept, and the reader, which depends on the loop only through PP, is unchanged. The bound holds on the trial cohorts, not on the population; the final cohort (Appendix § 9.§ 2) checks the last incumbent on new questions (from stories the loop had seen).

Proposition 3 (Arbitration is local). CogRSI’s answer differs from the vote over its six samples only on view conflicts and close splits; it never overturns an answer held by five or six samples and always returns a sampled answer. Hence arbitration changes accuracy by at most the share of arbitrated questions.

Proof. The answer equals the vote unless an arbitration call is made and its choice is accepted; a call is made only on a view conflict or a close split, and a choice is accepted only if it equals one of the two answers put to the arbiter, both sampled. Suppose YY holds at least five of six votes. The runner-up has at most one vote, so there is no close split. A view conflict needs both plain samples to agree and the annotated majority to differ: if they agree on YY, at least three of the four annotated samples are YY, so the annotated majority is YY; if they agree on another answer, that answer has two votes and YY at most four, a contradiction. So no call is made, and accuracy differs from the vote’s only on arbitrated questions.

Algorithm details

RSI over procedures

Algorithm 1 runs on the MedRSI code base (Wu et al., 2026a) at a pinned commit, with the backbone as solver and builder. Each round the builder reads prioritised failures of the current solver on ExploreToM physical-access development stories (trace, gold and wrong answer) and proposes up to four tools; tools that fail to load or to export the interface are discarded. A candidate is admitted if it improves a discovery cohort and, by at least one Algorithm 1 Artifact-level RSI (Stages 1 and 2)

Require: development pool DD, builder BB, system S0S_0 (empty library or P=∅P = \varnothing), rounds RR, KK, δ\delta

1: for r=0,…,R−1r = 0, \ldots, R - 1 do 2: draw a discovery cohort and KK trial cohorts T1…TKT_1 \ldots T_K from unused stories of DD 3: F←F \leftarrow failures of SrS_r on the discovery cohort (Stage 2: grouped by question type ×\times belief order) 4: X←B(F)X \leftarrow B(F); drop candidates that break the interface (Stage 1) or name shown characters (Stage 2) 5: for all x∈Xx \in X do 6: g(x)←1K∑kacc⁡(Sr⊕x,Tk)−acc⁡(Sr,Tk)g(x) \leftarrow\frac{1}{K} \sum_k \operatorname{acc}(S_r \oplus x, T_k) - \operatorname{acc}(S_r, T_k) (Eq. (1)) 7: end for 8: x∗←arg⁡max⁡xg(x)x^* \leftarrow\arg\max_x g(x) 9: if g(x∗)>δg(x^*) > \delta then 10: Sr+1←Sr⊕x∗S_{r+1} \leftarrow S_r \oplus x^* (Stage 1 arm A: add the tool; Stage 2: replace PP) 11: else 12: Sr+1←SrS_{r+1} \leftarrow S_r 13: end if 14: end for 15: return SRS_R (arm C: rebuild once from round-0 failures pooled over DD, same budget)

Algorithm 2 CogRSI (Stage 3), one question

Require: context cc, question qq, principle PP

1: α←PERCEIVE⁡(c)\alpha\leftarrow\operatorname{PERCEIVE}(c) (who witnessed each unit; one call per context) 2: if cc is a conversation and qq asks a named chain’s belief then 3: v←v \leftarrow perspective view of the chain 4: else 5: v←cv \leftarrow c annotated with α\alpha 6: end if 7: Y←Y \leftarrow four samples on (v,P)(v, P) and two on cc (temperature 0.7; concise retry) 8: y^←\hat{y} \leftarrow majority of YY 9: if view conflict, or close split (top ≤4/6\le4/6, runner-up ≥2/6\ge2/6; narratives) then 10: y′←ARBITRATE⁡(c,α,y' \leftarrow\operatorname{ARBITRATE}(c, \alpha, top two answers, P)P); if y′y' is one of them then y^←y′\hat{y} \leftarrow y' 11: end if 12: return y^\hat{y}

point on average, three trial cohorts of 40 stories. Table 6 gives every run round by round.

Runr0r1r2r3r4r5r6r7Adm.ToolsAccessNestingParsing
A s050.546.651.545.849.750.347.652.5*132321610
A s141.644.546.053.146.950.143.251.3030301612
A s254.454.555.547.6*52.452.759.250.8131291723
C s052.646.351.746.9*50.5*48.347.851.2231311729
C s142.846.345.850.946.0*48.6*44.453.8331311328
C s253.253.6*52.9*47.451.354.4*58.444.3*52828150

Table 6. Stage 1 (E1) runs round by round: development accuracy of the solver with the current library (A: inherited, C: rebuilt; *: tools admitted in that round) and the number of admitted tools (a round can admit more than one); right: the candidate tools each run wrote and how many encode perceptual access, nested belief or story parsing (a tool can encode several). Admission is rare, so the inheriting arm rarely has a library to inherit.

RSI over principles

Development data: ExploreToM e1_dev (29,752 questions over 2,519 stories in 2,189 structural groups; the sampler draws by story), disjoint from every test sample. Every cohort takes one question from each of a set of unused stories: per round a discovery cohort of 80 and three trial cohorts of 80; after eight rounds a final cohort of 300 compares no principle, the hand-written principle of CogRSI and the loop’s principle. Eight rounds and the final cohort need more stories than the pool holds: once every story has been used, a new pass reshuffles the stories and takes from each a question not used before. Seeds 0–2 ran the first seven rounds before this rule existed and were resumed from their saved rounds with it, which reproduces their earlier draws exactly; later seeds used it from the start. Builder: the backbone at temperature 0.7; one call per failure group (the four largest groups, up to four failures each), replying with a one-sentence diagnosis and a principle of at most 80 words that must not name the shown stories’ characters or places (checked against the perception call’s names and capitalised words; violators and principles over 120 words are rejected). Test runs use the frozen CogRSI runner with its two texts replaced by the seed’s principle (Box (f)).

CogRSI: reading and design history

Algorithm 2 gives the reader. Ties in the vote are broken by the first sampled answer; samples that hit the 1,024-token cap are retried once with a concise instruction. CogRSI was built in three steps, each frozen and then tested on every test set (Table 8): a vote over the annotated and plain views (frozen at commit 6035943), the arbitration of view conflicts (c86b903) and the arbitration of close splits (e59dcbb). The first two were frozen before any of their test results were seen. The third was developed afterwards on development data only (held-out sample 1 and development samples of four further ToM benchmarks) after inspecting the second design’s test failures for diagnosis; six candidates were tried and five rejected on development data:

DesignMain sampleFANToMAll*Llama-70BGLM-4.6VQwen3.5-9B
SC (k=6k=6)68.317.661.948.051.4
Vote over both views (v3)72.228.765.554.762.3
+ conflict arbitration (v5)72.627.367.253.660.3
+ close-split arbitration = CogRSI (v6)73.128.067.154.960.9

Table 8. CogRSI’s design variants on every test condition (three runs each; per-benchmark results and every run in Table 20). Each design was frozen before its test runs; the close-split arbitration was added after the first test run of the design without it and was selected on development data only (§C.3).

  • Access gate: plain sampling when the perception call finds no access asymmetry. Development: held-out-1 narrative items: 79.7 / 78.9 (classifier) vs 80.0 for the base design.

  • Construct router: emotion and non-literal questions answered from plain samples. Development: held-out-1: 80.0, equal to the base design.

  • Last-witnessed trace: explicit trace of what each character last witnessed. Development: Hi-ToM 44.2 vs 66.4 for the base design.

  • Extra voters: zero-shot, RSI-library and SC answers added to the vote. Development: lower than the base design on most development datasets.

  • Conflict-only gate: plain reading on conflicts the classifier deems access-free. Development: held-out-1: 80.4 vs 81.2 for the base design.

The adopted change applies the existing arbitration call, unchanged, to every remaining close split (Table 7); it was frozen with byte-identical source checks, run once on all test data, and every variant Every seed’s principle says how a character comes to believe or know a state (50–67 words).

SampleBenchmarknnwithoutwith
Held-out 1SimpleToM50191.893.2
Held-out 1Hi-ToM50066.466.8
Held-out 1OpenToM50077.477.0
Held-out 1ToMBench50076.477.2
Held-out 1CogToM50085.485.6
Held-out 1BigToM50489.990.3

Table 7. Development results without and with the arbitration of close splits, on the same six samples per question on held-out sample 1, the development set and the design without it. Rejected candidates are listed in the text.

Loop-written principles

Figure 0. The principles written and admitted by the principle-level loop (§ 4.2), one per seed, verbatim.

seed 0: To predict where A believes B will search, identify the last location B moved the object that A also witnessed. Ignore any moves B made that A did not see, and ignore the object’s true current location. A’s prediction is based solely on the most recent move in the shared history.

seed 1: A character believes an object has a state if they entered the location after that state was established and no intervening action altered it while they were absent. Characters retain beliefs about object states from their last direct observation or from the state existing in the location when they arrived.

seed 2: When determining if a character believes X, first verify if X is an objective fact in the story. If X is true, assume the character believes X unless the story explicitly states they were misled, told a falsehood, or have direct evidence contradicting X. Do not assume ignorance solely because the character did not witness X.

seed 3: To determine what A thinks B believes: If B’s last perceived event occurred before the target event, and A knows B was present or unobserved leaving before the target event, A assumes B believes the target event occurred. If A knows B was absent or saw an earlier state, A assumes B believes the earlier state.

seed 4: To predict where A thinks B will search: First, determine B’s last observation of the object. If B never saw the object move to a new location, A predicts B searches that last observed spot. However, if A knows B entered the scene after the object was moved to a new location, A predicts B searches that new location, as B’s knowledge starts from their entry point.

seed 5: Assume a character believes another knows a fact if the first character knows the second was physically present when the event occurred, unless explicitly told the second was unaware. Do not require the second character to have consciously witnessed the event; mere presence suffices for the first character to attribute knowledge.

was re-run on the confirmatory sample (Table 3B).

Validity-aware RSI: details

Pre-registration. Every experiment in this section was registered before its first output (files configs/prereg_J099–J104 in our repository) with its arms, seeds, thresholds, hypotheses and failure handling. For the principle arms the primary hypothesis H1 is CogRSI’s library minus the accuracy-only loop on test Main. The secondary hypotheses, Holm-corrected together, are the library against the hand-written principle (H2), the library minus the accuracy-only library on held-out pair validity (H3), the library minus the single principle (H4) and the library minus GEPA (H5), the last two on test Main. All outcomes are reported, and no arm was rerun or extended after its outputs existed. Two test cells failed on a connection error and were rerun with identical inputs, as the registration allows. No seed was missing at the cutoff.

Loop settings. All principle arms use the frozen reader and backbone and ExploreToM development data only: eight rounds, a discovery cohort of 80 new questions, three trial cohorts of 80, up to four candidates per round and a final cohort of 300 unseen questions shared by every comparison of a seed. The builder answers in JSON with a diagnosis, an operation (ADD, REVISE, RETIRE), an entry number, a condition and a rule. The library is composed as a numbered list, “(i) When condition: rule”, into both principle positions of the reader. Among admissible candidates the largest mean accuracy gain wins, with ties broken by the larger gate gain. GEPA (Agrawal et al. [1]) optimises the same slot from an empty seed, with the reader as its metric, the reader’s failures with gold answers as its reflective data, and 10,240 reader evaluations; it selects on its own validation split of development questions, so its development accuracy is not comparable and Table 1 omits it.

Counterfactual pairs. Narratives follow ToMi’s numbered-sentence format: two characters, an object in a container, and a move. Conversations are group chats in which one speaker makes a claim. Each pair varies one access fact. Narrative flip pairs are watch/leave, watch/asleep, told/leave, look/leave, told/lied and look/lied, and narrative same pairs leave/asleep and watch/told. Dialogue flip pairs are hear/away, hear/muted, relayed/away and relayed/misled, and dialogue same pairs away/muted and hear/relayed. Gold answers are computed by rule from the access facts. The gate split has 48 narrative and 48 dialogue pairs and the evaluation split 160 of each, drawn with different seeds from disjoint name, place, object and template lists. In the look variant the character who missed the move returns and checks the new container, so the right answer is the object’s true location. Table 9 gives every arm by benchmark and by variant.

PrincipleMain test samplePair validity by access variant (eval split)
FANSimpleExploreHiOpenTBenchCogBigMainsawleftpresenttoldliedlooksAll
no principle23.273.372.669.174.674.284.190.170.1100.0100.0100.0100.099.534.291.7
hand (expert)28.090.072.569.375.874.184.990.373.1100.0100.0100.0100.0100.033.391.7
accuracy-only loop25.381.774.070.076.572.884.789.671.8100.099.198.799.698.841.291.5
GEPA (same slot)23.271.784.762.771.571.877.872.867.0100.073.154.199.596.595.075.3
accuracy-only library23.781.376.070.375.572.684.088.771.5100.095.699.699.598.551.791.2
VC, single principle26.481.775.870.775.572.983.788.871.9100.098.8100.0100.0100.090.098.1
VC (ours)24.583.675.669.275.872.984.488.671.8100.099.699.699.8100.074.296.4
SC (k=9k=9)18.071.272.164.974.473.385.989.468.7100.0100.0100.098.8100.010.888.3

Table 9. Table 9: Arms by benchmark and by access variant (main backbone, means over seeds or runs). Variant columns give item accuracy on the evaluation split’s items of that variant (narrative and dialogue pooled); All is pair validity.

Registration noise. The margin was set from development logs of the accuracy-only loop only. A fixed principle’s accuracy varies by 2.7 points across trial cohorts of 80 questions. Simulating four candidates with no real gain per round, a 1-point margin admits one in 70% of rounds and a 3-point margin in about 10%. The gate tolerances come from the reader’s run-to-run variation in valid pairs on the counterfactual items.

Selection quality. For each finished loop we scored every candidate that reached registration, admitted or not, and its incumbent on the first 120 questions of the loop’s final cohort, which no registration step saw. A candidate helps if it beats its incumbent there. The accuracy-only loop proposed 192 scored candidates in 48 rounds and admitted 22. CogRSI’s library proposed 191 and admitted 12, and the accuracy-only library and the single principle admitted 6 and 6 of 94 and 96. The selection gap, the best candidate’s held-out gain minus the admitted one’s, averages 1.1, 0.8, 1.0 and 1.8 points in the order of Table 5. With so few admissions per arm the precision estimates are coarse.

Matched inheritance. Arm M runs the procedure loop with A’s inheritance (its builder sees and composes registered tools, which persist) and C’s evidence (failures of the round-0 solver), for seeds 0–2 on the same pool and schedule. On the full sealed FANToM set, with the seed-paired bootstrap of the original comparison, M − C is +0.5 points (95% CI [−0.5, +1.5]; the 90% interval lies within the pre-registered 3-point equivalence margin), A − C +0.6 ([−2.0, +4.1]) and M − A −0.2 ([−3.1, +2.2]). Inheritance makes no meaningful difference, whichever evidence the builder sees.

Controlled reader. To separate the view from the rest of the reader, we replaced the annotated story with PercepToM’s filtered story or SimToM’s perspective-taking story and kept perception, sampling, voting, arbitration and the hand-written principle fixed. On the narrative benchmarks of the main test sample the annotated view beats the filtered view by +1.3 Main points ([+0.7, +2.0]) and the perspective view by +0.9 ([+0.1, +1.7]). The largest gaps are on Hi-ToM (+6.3 and +3.7), where the filters drop context that higher-order questions need.

Semantic audit (J105). The oracle (src/cogrsi/access_oracle.py) samples worlds event by event from the legal moves: leave, enter, fall asleep, wake, move the object, whisper a location to one listener (true or false), peek into the container that holds the object, or change a visible or hidden property. Moves and changes are perceived by the agents present and awake and by the actor; a whisper by its speaker and listener; a peek by the peeker; a visible property also by anyone who later enters or wakes while the object is in view. Second-order beliefs change only through events both agents perceive, or when one tells the other. The audited population is every candidate that reached registration in the accuracy-only loop, the accuracy-only library, the gated library and the gated single principle (573 texts; the entry an ADD or REVISE writes, or a whole single principle). The compilers see only the rule text. Before the freeze, three hand-written rules were used to fix the sandbox, the compile instructions, a deterministic scope guard for second-order questions, private telling and peeking, and the shared-counterexample criterion. The pre-registered run gave 165 invalid, 29 valid, 270 ambiguous and 109 vacuous texts; 253 of the ambiguous ones had no code because the Qwen3.5-122B replies hit the 2,048-token output cap. A recorded deviation recompiled those texts with an 8,192-token cap and nothing else changed (43 still hit it), giving 286, 44, 84 and 159. Shadow registration replays each candidate under each rule against its own incumbent and logged trial gains; the loops that ran without the gate were scored on the gate pairs afterwards. Admitted rules that were invalid, by arm: accuracy-only loop 14/14, accuracy-only library 5/5, gated library 7/9, gated single principle 3/4. Without the rule, the reader answers 29.4% of rendered counterexamples and 35.2% of controls wrongly: these stories are hard in themselves.

Objective vs. data (J106) and transfer (J107). Arm B (pairs as data) and seeds 3–5 of the accuracy-only library were run with the J101 settings; their test-sample and own-pair evaluations were not run before this version.

BigToM-U uses BigToM’s conditions that state the initial belief, with true/false-belief flip pairs and true/false control same pairs (800 pairs from 200 scenarios); every scenario’s other version had appeared in an earlier split, so the set is new in its conditions and pairing, not its scenarios. On flip pairs: gated library 94.9, accuracy-only library 95.5, pairs as data 95.1, expert 91.2, none 90.2, SC (k=9k=9) 88.5, GEPA 70.5. Backward belief (400 pairs): gated library 68.0, accuracy-only library 68.1, expert 68.1, none 67.8.

Rule activation (J109). Intact gated libraries score 96.6 on BigToM-U, 96.0 with every condition removed and 96.3 with conditions swapped between entries (flip pairs 94.9, 93.9, 94.4).

Benchmark details

Task design

A system receives a context (story or conversation) and a question in the benchmark’s native format and returns an answer in that format; it never sees gold answers or test items during development. The RSI loops see only ExploreToM development stories; CogRSI’s design used held-out sample 1.

Sample construction. Samples are drawn by story with fixed seeds and are disjoint by construction (Figure 11C). S0 holds the smoke subsets (never reported), S1 the development data, S2 held-out sample 1 (CogRSI’s development set), S3 held-out sample 2 (the main test sample) and S4 held-out sample 3, drawn after every design decision. Every design was frozen, with byte-identical source checks, before its test runs (Appendix § 9.§ 3). Two qualifications apply. Close-split arbitration was developed after inspecting the test failures of the design without it, so main-sample results for the final reader are exploratory and S4 is its confirmatory test. For FANToM, SimpleToM and ExploreToM, every held-out sample, S4 included, is a subset of the sealed test sets on which the baselines and Stage-1 libraries ran before CogRSI existed, and FANToM baseline failures from those sets were read for diagnosis. For these three benchmarks S4 is independent of every CogRSI design decision, not unseen by the project.

Construction

All inputs are text. MMToM-QA and MuMA-ToM are run in their text-only versions (the stories and dialogues released with them; no video), and Social-IQ 2.0 as a written description of each clip followed by its transcript; the descriptions were written once from the clip frames by the backbone with image input enabled for that pass only. No system sees images or video, so these results say nothing about multimodal transfer. Each benchmark’s test pool was sealed before the formal runs. Samples are drawn by story, conversation or ExploreToM structural group with fixed seeds (smoke and held-out 1: 0, held-out 2: 1, confirmatory: 3), and each excludes every unit of the earlier ones; the confirmatory draw is disjoint from the development data, the smoke subsets and held-out samples 1 and 2. For FANToM, SimpleToM and ExploreToM all three held-out samples are subsets of the sealed test sets on which the baselines and Stage-1 libraries were evaluated on October 6–9, before CogRSI existed (Table 18–Table 24), and FANToM baseline failures from those sets were read for diagnosis. No CogRSI design decision used held-out sample 3, but for these three benchmarks its independence is at the level of design decisions, not of exposure to the project. Table 10 gives the sizes. ToMBench items with repeated IDs were disambiguated and options with empty text removed before any draw.

BenchmarkInputHeld-out 1Main testConfirm.
FAnToMconversation84 conv.187 sets (74 conv.)75 conv.
SimpleToMstory334 (167)334 (167)334 (167)
ExploreToM E1story253 (51)254 (52)250 (52)
ExploreToM E2story254 (51)250 (50)254 (51)
Hi-ToMstory500 (102)502 (101)70 (14)
OpenToMstory500 (100)500 (100)500 (100)
ToMBenchstory500 (243)505 (253)500 (246)
CogToMstory500 (309)500 (257)502 (292)
BigToMstory504 (84)504 (84)60 (10)

Table 10. Samples per benchmark: questions (stories or groups in parentheses; FAnToM: All* sets and conversations). Held-out sample 1 is CogRSI’s development set; the main test sample is held-out sample 2; the confirmatory sample is held-out sample 3. SimpleToM: behavior and judgment questions scored.

MethodFAN All*Simple Beh.+Judg.Explore E1+E2Overall (3/3)
Qwen3.5-35B-A3B-FP8 (MoE 35B / 3B active, FP8, thinking off)
Native solver (floor)4.351.961.339.2
Prompting
Zero-shot7.8<sub>+3.5</sub>60.9<sub>+9.0</sub>60.1<sub>−1.2</sub>42.9<sub>+3.8</sub>
CoT14.4<sub>+10.1</sub>68.4<sub>+16.5</sub>62.0<sub>+0.7</sub>48.2<sub>+9.1</sub>
SC (k=6k=6)18.2<sub>+14.0</sub>68.4<sub>+16.5</sub>64.8<sub>+3.5</sub>50.5<sub>+11.3</sub>
Adapted on ExploreToM dev
Retrieval2.8<sub>−1.5</sub>49.7<sub>−2.2</sub>65.2<sub>+3.9</sub>39.2<sub>+0.1</sub>
GEPA0.6<sub>−3.7</sub>46.7<sub>−5.2</sub>68.7<sub>+7.4</sub>38.6<sub>−0.5</sub>
ToM procedure
PercepToM5.0‡<sub>+0.7</sub>74.6<sub>+22.7</sub>62.9<sub>+1.6</sub>47.5<sub>+8.3</sub>
RSI over procedures (Stage 1)
Lib-C (rebuilt library)2.2<sub>−2.1</sub>51.6<sub>−0.3</sub>68.7<sub>+7.5</sub>40.8<sub>+1.7</sub>
Lib-A (inherited library)2.9<sub>−1.4</sub>52.7<sub>+0.7</sub>70.7<sub>+9.4</sub>42.1<sub>+2.9</sub>

Table 18. Full sealed test sets (RSI study, design frozen before its runs). Accuracy (%) per dataset, full sealed test sets of FANToM, SimpleToM and ExploreToM (October 6–9; before CogRSI v3 existed); smoke subsets excluded. Mean over three runs (CogRSI and baselines; GEPA: three optimizer seeds; Stage-1 tools: three loop seeds; CogRSI-L: one run per principle-loop seed). FANToM: official All* on short contexts. SimpleToM: mean of behavior and judgment. ExploreToM: mean of the E1 (physical access) and E2 (communication access) partitions. Subscripts: gap to the floor (the same native solver with an empty tool library). Best in bold, runner-up underlined. Overall averages the datasets every row has (count in the header). ‡PercepToM on FANToM is scored from its per-question results like every other system; the official scorer’s printed summary for these runs is higher and cannot be reproduced from the per-question results it writes (for every other system the two agree exactly). Per-metric detail in Table 21; every run in Table 24.

MethodFANToM All*FANToM BeliefSimpleToM Behav.SimpleToM Judg.ExploreToM E1ExploreToM E2
Native solver4.352.965.338.650.072.5
Zero-shot7.8+3.563.5+10.572.0+6.749.8+11.245.8−4.274.3+1.8
CoT14.4+10.168.2+15.278.0+12.758.9+20.344.4−5.679.5+7.0
SC (k=6k=6)18.2+14.071.2+18.378.5+13.258.3+19.847.2−2.982.3+9.8
Retrieval2.8−1.541.7−11.261.4−3.938.1−0.556.7+6.773.7+1.2
GEPA0.6−3.738.8−14.258.8−6.534.6−4.060.2+10.277.1+4.6
PercepToM5.0‡+0.781.0+28.080.1+14.969.0+30.561.1+11.164.7−7.8
Lib-C (rebuilt library)2.2−2.150.3−2.654.8−10.548.4+9.856.8+6.780.7+8.2
Lib-A (inherited library)2.9−1.448.3−4.754.7−10.650.6+12.161.8+11.879.5+7.0

Table 21. Per-metric detail for FANToM, SimpleToM and ExploreToM (same runs as Table 18). Subscripts: gap to the floor. †Fewer than three runs. ‡PercepToM on FANToM is scored from its per-question results like every other system; the official scorer’s printed summary for these runs is higher and cannot be reproduced from the per-question results it writes (for every other system the two agree exactly).

SystemRunAll*BeliefBehav.Judg.MentalE1E2
Native solverrep. 04.153.565.338.697.250.172.6
Native solverrep. 14.552.565.338.697.250.372.3
Native solverrep. 24.352.765.338.697.249.672.7
Zero-shotrep. 07.563.171.749.395.046.074.3
Zero-shotrep. 18.163.872.349.995.145.774.4
Zero-shotrep. 27.963.571.850.294.845.874.3
CoTrep. 014.667.878.959.498.144.179.7
CoTrep. 114.968.677.758.598.144.479.5
CoTrep. 213.668.177.458.797.844.779.3
SC (k=6k=6)rep. 017.270.778.658.098.347.082.5
SC (k=6k=6)rep. 118.471.478.659.997.947.582.5
SC (k=6k=6)rep. 219.271.678.257.198.146.982.0
Retrievalrep. 02.742.262.038.589.956.873.5
Retrievalrep. 12.640.860.537.888.955.973.8
Retrievalrep. 23.142.161.637.989.557.473.9
GEPAopt. seed 00.536.663.434.388.760.676.2
GEPAopt. seed 10.038.057.632.486.863.176.9
GEPAopt. seed 21.241.655.337.277.357.078.3
PercepToMrep. 04.880.480.469.270.861.064.6
PercepToMrep. 15.381.580.168.970.961.364.7
PercepToMrep. 24.880.979.969.070.961.164.8
PercepToM (v4 adapter, appendix)rep. 04.681.992.077.562.161.766.2
PercepToM (v4 adapter, appendix)rep. 15.780.991.577.661.861.165.8
PercepToM (v4 adapter, appendix)rep. 23.982.291.477.262.160.665.7
E2 C+ (library visible to builder)seed 01.930.863.353.475.674.486.2
E2 C+ (library visible to builder)seed 11.941.246.145.180.656.382.3
E2 C+ (library visible to builder)seed 21.462.956.244.978.551.077.0
E2 C− (library hidden from builder)seed 01.731.662.653.875.575.386.4
E2 C− (library hidden from builder)seed 11.440.046.949.579.257.683.5
E2 C− (library hidden from builder)seed 22.750.058.648.284.648.681.0
Lib-C (rebuilt library)seed 02.251.154.144.493.364.284.0
Lib-C (rebuilt library)seed 13.858.560.644.784.051.677.0
Lib-C (rebuilt library)seed 20.741.349.656.187.454.581.2
Lib-A (inherited library)seed 01.532.163.055.973.775.184.9
Lib-A (inherited library)seed 12.143.846.643.779.458.078.3
Lib-A (inherited library)seed 25.068.954.552.387.852.575.2

Table 24. Every run behind Table 18–Table 22 (accuracy %). Repeats of static baselines are independent executions, not controlled sampling seeds. – = not run.

SystemFANToM All*FANToM BeliefSimpleToM Behav.SimpleToM Judg.SimpleToM MentalExploreToM E1ExploreToM E2
Native solver4.352.965.338.697.250.072.5
Zero-shot7.863.572.049.895.045.874.3
CoT14.468.278.058.998.044.479.5
SC (k=6k=6)18.271.278.558.398.147.282.3
Retrieval2.841.761.438.189.456.773.7
GEPA0.638.858.834.684.360.277.1
PercepToM5.0‡81.080.169.070.961.164.7
PercepToM (v4 adapter, appendix)4.781.691.677.462.061.165.9
E2 C+ (library visible to builder)1.744.955.247.878.260.681.8
E2 C− (library hidden from builder)1.940.556.050.579.860.583.6
Lib-C (rebuilt library)2.250.354.848.488.256.880.7
Lib-A (inherited library)2.948.354.750.680.361.879.5

Table 22. All systems, including rows kept out of Table 18: the original PercepToM adapter (v4) and the E2 branches (parent library plus new tools). Mental-state questions all have gold answer “No”, so a constant “No” scores 100%; this column is reported separately. †Fewer than three runs. ‡PercepToM on FANToM is scored from its per-question results like every other system; the official scorer’s printed summary for these runs is higher and cannot be reproduced from the per-question results it writes (for every other system the two agree exactly).

Metrics

Accuracy is each benchmark’s own rule: option match for multiple choice, the benchmarks’ scripts for free text and lists; FANToM uses its official scorer on short contexts, and All* counts a target set as right only if every question in it is right. PercepToM’s FANToM runs are scored from their per-question results like every other system, because the official scorer’s printed summary for those runs cannot be reproduced from the per-question results it writes (for every other system the two agree exactly). ATOMS constructs are mapped from each benchmark’s question type by a fixed table; no-

dependent annotation validates these labels. Paired differences are bootstrapped over stories or conversations (2,000 resamples); within-benchmark construct effects centre each question’s difference on its benchmark’s mean.

Baseline details

All baselines use the backbone, decoding settings and output cap of CogRSI (Table 11). Native solver: the Stage-1 solver with an empty library (floor). Zero-shot: the question and context with the answer-format instruction. CoT (Kojima et al., [23]): step-by-step reasoning before the answer. SC (k=6k=6), SC (k=9k=9) (Wang et al., 2023): majority over kk CoT samples at temperature 0.7, k=6k=6 being CogRSI’s number of samples and k=9k=9 its tokens per question, run by the frozen CogRSI runner with kk fixed. Retrieval: the most similar development failures (sentence embeddings of question and story), shown as solved examples with their correct answers. GEPA (Agrawal et al., [1]): prompt-only optimisation on ExploreToM development data, three optimizer seeds. SimToM (Wilf et al., 2024): a first call asks which events or utterances the target character knows about, a second answers from them with our CoT prompt (CoT when no character is named). PercepToM (Jung et al., [19]): its perception and perspective-filtering steps with a zero-shot fallback when the target character is not found; PercepToM + SC (belief) on FANToM: six samples over its filtered conversation for belief questions, six plain samples otherwise, majority vote. PercepToM + SC and SimToM + SC: CogRSI’s six samples at temperature 0.7 drawn over PercepToM’s filtered context (every question) or after SimToM’s perspective-taking call, with SimToM’s answer prompt, majority vote. Lib-A/C: the Stage-1 libraries, inherited and rebuilt, three loop seeds. Not run: DEL-ToM (Wu et al., 2025) needs a verifier trained on its own simulator;

MethodCallsIn (k)Out (k)Main
CoT1.00.350.2664.8
SC (k=6k=6)6.02.111.6768.3
SC (k=9k=9)9.03.172.5068.7
PercepToM1.40.470.0863.8
CogRSI7.33.332.3673.1
CogRSI-L7.33.512.4071.8

Table 11. Cost per question (multiple-choice test benchmarks, main backbone; all calls, including perception, retries and arbitration) and overall accuracy (Table 3). SC (k=9k=9) matches CogRSI’s tokens.

Validity-aware libraries

Figure 14. Final libraries of CogRSI (§4.2). Each numbered entry is one admitted ADD or REVISE.

CogRSI’s final libraries, one per seed, verbatim (the doubled “When” is in the logged text).

seed 0: (1) When When the question asks about a character’s belief regarding another character’s belief about an event, and the second character witnessed the event while the first did not: If Character A did not witness an event but Character B did, and Character A knows Character B is present or aware, assume Character A attributes Character B’s knowledge of the event to them, even if Character A lacks direct perception.

seed 1: (1) When When the question asks where a character will search for an object, and the character’s last known perception of the object’s location differs from the object’s actual final location: Identify the last location the character personally witnessed the object being placed. If the object was subsequently moved by anyone (including the character themselves) after that perception without the character witnessing it, the character will search at their last witnessed location. If the character witnessed the final move, they search there. (2) When When the question asks if Character A thinks Character B believes X, and Character B witnessed an event establishing X, regardless of whether Character A witnessed it: If Character B witnessed the event that makes X true, Character A must attribute that belief to Character B. Character A’s lack of perception of the event does not negate their ability to know that Character B knows X.

seed 2: (1) When When a question asks if a character believes an object has a visible property, and that property was created by an event that occurred before the character entered the location: If the character enters the location after the event, they can perceive the resulting visible state of the object. Assume they believe the object has the property unless told otherwise. (2) When When a question asks if Character A believes Character B knows a past event, and Character B witnessed that event (even in secret) while Character A was unaware of the witnessing: If Character B witnessed the event, Character A believes Character B knows the event occurred if Character A assumes Character B was present or if no information suggests Character A knows Character B was absent during the event. (3) When When a question asks if Character A believes Character B holds a belief about a state, and Character B was present in the location during the event that created that state, regardless of whether B witnessed the event explicitly: If Character B was present in the location when the event occurred, assume Character A believes Character B knows the resulting state, unless A has specific reason to doubt B’s perception.

seed 3: (1) When The question asks where a character will search, and that character was absent during the final move of the object to a new location: The character will search in the location where the object was moved immediately before the question, provided the character who performed that final move is present or the character has no prior knowledge contradicting it. The character’s belief is updated by the last known location, not their last witnessed location.

seed 4: (1) When When a character enters a location after an event and the resulting physical state of an object is visible, they perceive the state: If an object’s final visible state (e.g., covered, dyed, damaged) persists after the character arrives, the character believes the object is in that state, regardless of whether they witnessed the cause.

seed 5: (1) When When the question asks where a character will search for an object, and that character did not witness the last movement of the object by another character: The character will search in the location where the object actually is, as the character’s belief is overridden by the fact that the object was moved without their knowledge, but the system must correctly identify the current physical location based on the full narrative sequence. (2) When When the question asks about a character’s belief regarding a state of an object that is currently visible or physically present to that character, even if the character did not witness the specific action that created it: If the object’s current state is observable by the character upon their return or presence, the character believes the state exists based on visible evidence, regardless of whether they witnessed the event that created it. (3) When When the question asks if Character A thinks Character B believes an object has a state, and Character A knows the object has that state due to visible evidence or a witnessed event, while Character B was absent for the cause but present for the evidence: Character A attributes the belief to Character B if the physical state of the object is visible to Character B, regardless of whether Character B witnessed the event that created the state.

(g) Final libraries of CogRSI (§4.2). Each numbered entry is one admitted ADD or REVISE.

AutoToM (Zhang et al., 2025) and ThoughtTracing [21] answer through a planning model with a different interface; Decompose-ToM [32] and Shoes-of-Others (Shin-oda et al., 2026) are perspective methods of the kind SimToM and PercepToM represent.

Experiment details

Protocol and implementation

Every system runs the same served model per backbone through vLLM 0.30.0 (PyTorch 2.13, Transformers 5.18), thinking off, 32k context and a 1,024-token output cap (Table 16). Each run is one execution over a sample; failed runs are retried Accuracy-only library (gate off)

BackboneArchitectureWeightsGPUs
Qwen3.5-35B-A3B (main)MoE, 256 experts, 40 layersFP81×H200 / A6000
Qwen3.5-122B-A10BMoE, 256 experts, 48 layersFP81×H200
Llama-3.3-70B-Instructdense, 80 layersAWQ 4-bit1×H200 / 2×A6000
GLM-4.6V-Flashdense, 40 layers, text-only servingBF161×A6000
Qwen3.5-9Bdense, 32 layersBF161×H200 / A6000

Table 16. Backbones and hardware (NVIDIA H200 NVL 141 GB, RTX A6000 48 GB). All served with vLLM 0.30.0.

Final artifacts of the two ablations

Figure h. Final artifacts of the two ablations (§ 4.2), verbatim.

seed 0: (1) When When the question asks what Character A believes Character B will search for, and Character B did not witness Character A’s most recent action: Identify the location of the object according to Character B’s last witnessed event. Character A, knowing B’s limited knowledge, predicts B will search in that location.

seed 1: (1) When When the question asks about a character’s belief regarding an object’s location, and that character moved the object to a location after which no other character moved it while the first character was present: If a character moved an object to a location and was present for subsequent moves, they know the true location. If they were absent for subsequent moves, they believe the location held while they were absent, unless they moved it again later. (2) When When the question asks if Character A thinks Character B believes a state, and Character B witnessed the state directly: If Character B witnessed the state, assume Character A attributes this belief to B, even if A did not witness the state themselves, unless A knows B is mistaken. (3) When When a character enters a location after an object’s state has changed, and the object remains in that state without further changes: If a character did not witness a state change but the object is currently in that changed state, assume they believe the object is in that state.

seed 2: (1) When When a question asks if a character believes a state exists, and that state was created or altered by an event perceived by someone who was present before the question-asker entered: If a character enters a location after an event occurred there, assume they can perceive the resulting state of objects unless the story explicitly states they cannot. Infer belief in the state based on the physical reality established by the event.

Single principle under the full rule

seed 0: A character knows an object’s state if they perceive the change, OR if they are in a location where the object is present and its state is visible to them. Characters do not need to witness the specific event causing a change if they enter after it and can see the result.

seed 1: If Character A performs an action, they know the result. If Character B enters after the action but before any concealment, assume B perceives the result unless told otherwise. When inferring what A thinks B believes, if A knows B was present during the event, A assumes B knows, even if A did not see B’s eyes.

seed 2: 1. Characters know the current state of objects in a location if they entered after a change occurred and the state remains visible. 2. Characters know about an action if they witnessed it or if the action’s result is visibly present when they enter. 3. Do not assume ignorance of visible states simply because the character did not witness the specific moment of change.

and reported (Table 20). Every frozen design is recorded with its commit and source hash next to its outputs.

SystemRunFAN All*Simple Behav.Simple Judg.Explore E1Explore E2HiOpenTBenchCogBig
Native solver03.268.333.554.779.254.866.470.379.683.3
13.768.333.554.781.256.066.870.179.683.5
24.368.333.552.081.656.666.469.979.882.9
Zero-shot07.575.447.351.677.659.268.272.180.085.1
18.075.444.350.879.257.668.272.379.684.7
28.674.947.351.678.458.468.872.179.484.9
CoT013.982.654.556.383.651.872.071.781.487.9
113.982.055.154.385.653.272.271.383.088.5
211.883.856.353.584.050.473.272.383.288.7
SC (k=6k=6)018.283.256.357.588.462.475.073.586.689.1
116.683.259.956.787.662.474.874.185.289.9
218.283.255.158.387.262.075.274.384.888.5
SC (k=9k=9)017.683.861.155.588.464.374.473.186.490.3
119.883.255.156.387.265.174.873.186.088.7
216.684.459.357.587.665.374.073.985.489.1
Retrieval02.161.737.162.276.858.866.071.176.683.1
11.661.735.362.275.258.266.070.576.282.7
21.663.537.162.274.458.865.670.776.483.5
GEPA00.561.128.163.876.845.065.867.175.881.9
10.061.730.565.079.249.864.668.975.481.5
20.558.134.763.882.444.867.466.573.281.5
PercepToM01.694.081.465.469.650.467.871.979.884.5
12.793.480.865.770.048.469.071.979.885.1
21.692.882.665.070.845.868.472.379.684.7
SimToM00.582.071.966.186.436.773.268.176.885.1
11.183.271.363.484.438.274.866.575.485.5
21.685.674.364.286.837.874.268.974.884.5
Lib-C (rebuilt library)02.753.944.367.386.456.471.869.976.282.3
13.763.545.552.881.256.268.069.176.481.0
21.644.958.760.285.249.060.065.964.479.4
Lib-A (inherited library)01.667.157.574.487.255.268.468.376.282.7
11.649.143.764.283.256.464.467.974.679.4
23.758.152.157.577.247.664.870.376.079.0
CogRSI (vote only)025.192.888.059.183.667.177.471.984.886.7
129.492.285.659.884.866.377.471.784.886.3
231.692.288.057.184.068.177.672.384.286.9
CogRSI (no close-split)027.893.484.457.584.471.775.872.183.089.7
126.792.885.057.585.670.576.672.584.290.1
227.392.283.858.786.070.176.473.383.890.3
CogRSI031.095.883.859.185.669.776.274.385.290.3
124.695.285.058.388.471.575.474.185.089.9
228.395.285.057.986.066.775.874.184.490.9
CogRSI-L (loop-written principle)024.186.866.557.189.272.576.070.784.689.9
125.190.481.459.886.471.176.072.785.690.3
226.291.678.464.686.867.975.273.985.088.3
324.690.476.658.788.070.179.673.385.289.5
425.789.868.959.186.870.175.673.584.489.7
526.283.876.061.490.468.374.672.983.489.7

Table 20. Every run behind the main test sample (Table 3) (accuracy %: FANToM: All* over 187 target sets). Runs of static baselines are independent executions; GEPA runs are optimizer seeds.

Main results of the reader

  1. CogRSI is first on Main. On the main backbone it scores 73.1, against 68.3 for SC (k=6k=6) and 68.7 for SC (k=9k=9). On Qwen3.5-122B and Llama-3.3-70B it scores 75.6 and 72.3 against 72.1 and 69.6, and it beats SC (k=9k=9) on all five backbones at equal tokens (Table 13).

BackboneCogRSISC (k=9k=9)SC (k=6k=6)Tokens
Qwen3.5-35B (main)69.460.959.81.00
Qwen3.5-122B70.464.264.20.98
Llama-3.3-70B67.162.061.91.15
Qwen3.5-9B60.952.351.40.99
GLM-4.6V-Flash54.947.948.01.09

Table 13. Token-matched self-consistency on every backbone. Mean of FANToM All*, SimpleToM, Hi-ToM and BigToM (three runs). On FANToM, Llama3.3, Qwen3.5-9B and GLM use CogRSI without close-split arbitration, as in Figure 9b. Tokens: SC (k=9k=9)’s tokens per question relative to CogRSI’s (SimpleToM, Hi-ToM, BigToM; main backbone: all main benchmarks).

  1. The margin is largest where access decides the answer. On FAnToM All* it beats the strongest baseline by +4.3 points (95% CI [+0.2, +8.6]), on Hi-ToM by +4.4 ([+1.2, +7.6]), and SC (k=6k=6) on SimpleToM by +19.9. FAnToM and SimpleToM carry 81% of the gain over SC (k=9k=9) (85% on the confirmatory sample), and

the other six average 1.2 points, so the gain targets perception-dependent tasks.

  1. It holds on the confirmatory sample. On Table 3B, CogRSI scores 73.5 (three runs, 73.1–74.2) against 68.5 for SC (k=9k=9) (68.1–69.1), and CogRSI-L scores 72.7 over six seeds. The vote-only design, frozen before testing, already scores 72.0.

  1. It does not hold on seven further benchmarks.

There CogRSI scores 65.2, against 66.2 for SC (k=9k=9) and 66.7 for PercepToM + SC, and CogRSI-L scores 64.0 (Table 17). SimToM + SC is best on ToMi (85.6 vs. 66.0), and on the confirmatory items SimToM leads.

MethodToMiParaNegoOmniMMToMMuMASIQMean
Prompting
Native solver (floor)69.378.664.661.839.244.362.560.0
Zero-shot66.274.466.364.541.444.9<u>63.2</u>60.1
CoT67.966.368.759.453.166.561.363.3
SC (k=6k=6)66.167.171.761.157.770.1<u>63.2</u>65.3
SC (k=9k=9)66.266.972.561.760.172.8<u>63.2</u>66.2
Adapted on ExploreToM dev
Retrieval66.367.461.863.737.546.564.258.2
GEPA69.572.459.962.828.349.162.157.7
ToM procedure
PercepToM70.773.666.5<u>63.9</u>40.045.1<u>63.2</u>60.4
PercepToM + SC69.769.772.361.8<u>59.8</u>71.162.766.7
SimToM<u>84.6</u>78.467.753.438.855.559.762.6
SimToM + SC85.682.271.356.441.859.663.165.7
RSI over procedures (Stage 1)
Lib-C (rebuilt library)69.572.461.257.836.148.159.357.8
Lib-A (inherited library)76.072.758.458.036.645.360.558.2
Grounded reading (Stages 2–3)
CogRSI (no principle)65.667.372.060.756.4<u>71.3</u>63.165.2
CogRSI (vote only)66.267.172.3<u>63.9</u>53.568.562.764.9
CogRSI (no close-split)65.667.472.0<u>62.3</u>57.068.661.965.0
CogRSI-L67.569.0<u>72.8</u>61.649.665.262.064.0
CogRSI66.067.773.562.557.167.162.965.2

Table 17. Seven test-only benchmarks (main backbone, main test sample, accuracy %, mean of three runs; CogRSI-L: six loop seeds). They were added after the first design was frozen and never used for development. Best in bold, second underlined. Here CogRSI is level with self-consistency, and SimToM leads on the ToMi family.

Table 19 gives the development sample of CogRSI’s arbitration (held-out sample 1, not a test sample). Before CogRSI existed, the RSI libraries and the baselines were evaluated on the full sealed test sets of FANToM, SimpleToM and ExploreToM (the planned E1/E2 study, whose primary endpoint is reported in §5.3); Table 18–Table 24 give those results, including appendix-only systems. Table 20 lists every run. The seven test-only benchmarks are in Table 17. On them CogRSI is level with self-consistency (65.2 vs. 65.3 and 66.2), PercepToM + SC is best on average (66.7), and SimToM + SC leads on ToMi (85.6 vs. 66.0). On the confirmatory sample’s five test-only benchmarks the picture is the same (67.1 vs. 67.7 for SC (k=9k=9) and 71.2 for SimToM).

MethodFAN All*Simple Beh.+Judg.Explore E1+E2Hi Acc.Open Acc.TBench Acc.Cog Acc.Big Acc.Overall (8/8)
Qwen3.5-35B-A3B-FP8 (MoE 35B / 3B active, FP8, thinking off)
Native solver (floor)3.951.867.649.565.976.180.484.960.0
Prompting
Zero-shot6.8+2.958.0+6.270.4+2.854.2+4.766.7+0.874.7−1.480.4±0.086.0+1.262.2+2.1
CoT15.0+11.168.6+16.867.8+0.248.5−1.071.0+5.174.5−1.781.9+1.588.7+3.864.5+4.5
SC (k=6k=6)19.3+15.469.5+17.772.3+4.759.7+10.274.0+8.176.8+0.784.7+4.389.2+4.368.2+8.2
Adapted on ExploreToM dev
Retrieval2.8−1.148.5−3.369.8+2.251.2+1.765.9−0.175.5−0.780.3−0.183.9−0.959.7−0.3
GEPA0.5−3.444.2−7.672.8+5.247.5−2.166.2+0.373.2−2.976.8−3.681.0−3.857.8−2.2
ToM procedure
PercepToM4.2+0.386.6+34.869.6+2.049.1−0.566.6+0.775.0−1.180.0−0.486.2+1.364.7+4.6
CogRSI
Lib-C (rebuilt library)1.1−2.850.8−1.072.2+4.649.3−0.267.2+1.371.2−4.978.1−2.382.7−2.159.1−0.9
Lib-A (inherited library)2.3−1.650.7−1.175.3+7.751.6+2.164.6−1.371.3−4.978.1−2.380.6−4.359.3−0.7
Ours
CogRSI v3 (ours)25.4+21.587.2+35.472.8+5.263.3+13.878.5+12.576.8+0.783.6+3.285.4+0.671.6+11.6

Table 19. Development sample: held-out sample 1. Accuracy (%) per dataset, held-out sample 1, the development set of CogRSI’s arbitration (not a test sample); smoke subsets excluded. Mean over three runs (CogRSI and baselines; GEPA: three optimizer seeds; Stage-1 tools: three loop seeds; CogRSI-L: one run per principle-loop seed). FANToM: official All* on short contexts. SimpleToM: mean of behavior and judgment. ExploreToM: mean of the E1 (physical access) and E2 (communication access) partitions. Subscripts: gap to the floor (the same native solver with an empty tool library). Best in bold, runner-up underlined. Overall averages the datasets every row has (count in the header). Test results: Table 3.

Ablation study of the reader

What each part adds. The rows of Table 2 compare whole systems. No Stage-1 output enters Stage 2 or 3: Stage 1 only motivated the access annotation. Lib-A scores 59.3 Main because it lacks the whole reader. Inside the same reader, the hand-written principle adds +3.0 Main points over no principle (95% CI [+2.2, +3.7]), the loop’s principle adds +1.7 ([+1.1, +2.3]), and the reader with no principle already beats SC (k=9k=9) by +1.5 ([+0.6, +2.4]). In Stage 3 each reading and each arbitration step lifts belief accuracy, within the bound of Proposition 3. On FANToM All* the differences between Stage-3 variants are within the spread of three runs (Table 20).

Annotating and filtering are close. With the same arbitration, PercepToM’s filtered reading reaches 78.1 on the main multiple-choice benchmarks and CogRSI, which keeps the whole story, reaches 78.9. The two also differ in sample allocation, so this small gap does not isolate annotation from filtering. With our six samples PercepToM + SC and SimToM + SC reach 66.5 and 63.9 Main, and on FANToM All* 3.7 and 1.8, against 28.0. Table 12 gives PercepToM’s filtered reading the same arbitration as CogRSI: where its answer and self-consistency’s differ, the evidential rule decides. On 561 development conflicts the evidential rule is right 314 times, the plain view 267, the annotated view 254 and a neutral arbiter 218; without the rule the model assumes characters noticed hidden facts (SimpleToM: 23 vs. 114 of 140; Appendix § 13.8).

MethodHiOpenTBenchCogBigMean
PercepToM (filtered story)48.268.472.079.784.870.6
SC (k=6k=6, full story)62.275.073.985.589.277.2
PercepToM + SC + arbitration67.574.174.085.789.078.1
CogRSI (annotated + plain)69.375.874.184.990.378.9

Table 12. Annotate or filter? Accuracy on the five multiple-choice benchmarks of the main test sample, three runs. The third row gives PercepToM’s filtered reading our arbitration: where its answer and self-consistency’s differ, the evidential rule of §4.1 decides.

RSI over procedures

On Percept-ToM’s gold perceivers [19], Lib-A’s tools export at most 10% of the gold access pairs; rebuilt tools omit perceivers (precision 99.7, recall 42.2) or credit characters in other rooms (74.8 / 88.5). Inheriting the library minus rebuilding it is +0.6 points of FANToM All* (95% CI [−7.3, +8.6]) under the pre-registered hierarchical bootstrap over runs and conversations, and [−2.0, +4.1] under the seed-paired bootstrap used for the matched arm in Appendix § 10; acquiring communication with the inherited library in reach rather than without it gives −1.8 ([−6.8, +3.2]). The inheriting runs admitted 0–1 tools and had a non-empty library in 4 of 24 rounds; the rebuilding runs admitted 2–5 (Table 6).

RSI over principles

Code and text differ in more than the artifact. CogRSI-L scores 71.8 Main and 25.3 FANToM All*, against 59.3 and 2.3 for Lib-A (Figure 12c), but they also differ in perception, readings, voting and arbitration. It beats CogRSI on ExploreToM (74.0 vs. 72.5), Hi-ToM (70.0 vs. 69.3) and Open-ToM (76.5 vs. 75.8), and trails it on FANToM All* and SimpleToM (Table 3).

Table 14 gives every seed: admitted rounds, the final development cohort and the main test sample; near its source CogRSI-L leads CogRSI on ExploreToM (74.0 vs. 72.5) and trails it on FANToM All* (25.3 vs. 28.0) and SimpleToM (81.7 vs. 90.0). Trial accuracy rises by construction, since a round is admitted only if it does. On a development cohort of new questions, scored after every round, the incumbent goes from 56.0 (no principle) to 62.3 after round 0 and stays between 60.2 and 62.9 after that. The final principles score 63.2, against 59.7 for ours and 59.3 for none. On the main test sample the reader with no principle scores 70.1 Main (three runs, highest 70.3), below the lowest CogRSI-L seed (70.9).

SeedAdmittedNoneHandLoopMain
04/861.761.363.070.9
15/860.062.364.072.5
22/852.351.760.072.1
33/858.359.061.772.4
46/862.763.062.771.4
52/861.060.768.071.6

Table 14. Principle-level RSI, one row per loop seed. Admitted: rounds whose candidate passed the registration rule. Development: accuracy on a final cohort of 300 unseen ExploreToM development questions with no principle (the annotation format only), the hand-written principle of CogRSI and the loop’s final principle. Test: one run of the reader with the seed’s principle (Table 3). Principles verbatim in Appendix § 9.§ 2.

Grounded reading

Grounding helps epistemic questions. CogRSI is the most accurate system on belief and knowledge questions (Figure 13a). Pooled over benchmarks it beats SC (k=6k=6) on beliefs by +3.9 points (CI [+2.5, +5.4]). Within benchmarks the belief gain remains (+1.2, [+0.3, +2.1]), while non-literal and mixed questions lose (−1.7 and −4.1, Table 25). Grounding helps the cognitive side of ToM, which in humans dissociates from the affective side

Grounded reading

Figure 13. Grounded reading. (a) Belief and knowledge accuracy. (b) Narrative accuracy by reading. (c) Conflicts settled correctly. Grounding helps epistemic questions, and two readings with the evidential rule beat one.

ConstructnnSCPercepToMLib-ACogRSICogRSI−SC [95% CI]within benchmark
EpistemicBelief343873.871.863.677.7+3.9 [+2.5, +5.4]+1.2 [+0.3, +2.1]
Knowledge309584.665.477.385.7+1.1 [+0.2, +2.0]−0.8 [−1.8, +0.1]
MotivationalDesire7281.579.281.080.1−1.4 [−3.7, +0.5]−1.1 [−3.5, +0.9]
Intention11888.189.085.988.10.0 [−1.7, +1.8]+0.2 [−1.4, +2.0]
AffectiveEmotion15178.176.275.375.7−2.4 [−5.4, +0.2]−2.4 [−5.4, +0.2]
PragmaticNon-literal17779.882.776.378.2−1.7 [−3.7, +0.2]−1.7 [−3.5, −0.1]
MixedMixed9779.770.868.774.9−4.8 [−9.5, −0.4]−4.1 [−8.4, −0.6]

Table 25. Accuracy by ATOMS construct ([4]) on all test questions of the main backbone (main test sample, three runs each). The difference to self-consistency is bootstrapped over stories/ conversaciones (2,000 resamples); within benchmark centres each question’s difference on its benchmark’s mean difference and resamples stories within benchmarks. Bold intervals exclude zero. Constructs with fewer than 50 questions (perception, other) are omitted.

[36].

Perception limits true belief. With the gold perceivers of Percept-ToMi in place of the perception call (Table 27), CogRSI rises from 60.5 to 70.3 and CogRSI-L from 63.3 to 81.0, most on second-order true belief (2.8 to 35.4), while SC, which never calls perception, stays near 59.8. On FANToM gold perceivers lift belief accuracy (79.3 to 91.2) but not All* (26.0 to 25.2).

MethodPerceivers\multicolumn{5}{c}{Percept-ToMi (576 questions)}\multicolumn{2}{c}{FANToM (62 dialogues)}
FB1FB2TB1TB2AllAll{sup}*`Belief
SC (k=6k=6)model84.892.156.65.659.8——
gold85.792.156.27.260.4——
PercepToMmodel89.01.956.947.248.82.282.3
gold87.872.757.319.059.38.290.6
CogRSImodel89.992.656.62.860.526.079.3
gold89.290.566.035.470.325.291.2
CogRSI-Lmodel91.393.257.311.263.323.679.7
gold93.494.776.559.181.022.090.5

Table 27. Perception or reasoning? Each system with its own perception call (model) and with the gold perceivers of Percept-ToMi and Percept-FANToM in place of that call (gold; nothing else changes). FB/TB: false/true belief, first/second order. SC never calls perception, so its gold rows are a control. Mean of three runs (CogRSI-L: six loop seeds). Gold perceivers lift both readers most on true-belief questions. On FANToM, belief accuracy rises and All{sup}*` does not.

Two readings, settled by the rule. The vote over both readings beats either reading alone on belief, knowledge and all narrative questions (Figure 13b). On 561 development conflicts the evidential rule is right 314 times, and an arbiter told only to choose the better answer is right 218 times. On belief test conflicts the rule beats both readings (353 of 602, Figure 13c), and close-split arbitration fixes 76 answers while breaking 39.

Table 25 gives every ATOMS construct. Pooled over benchmarks CogRSI improves on SC (k=6k=6) for knowledge (+1.1, [+0.2, +2.0]) as for beliefs; within benchmarks only the belief gain remains, and non-literal (−1.7, [−3.5, −0.1]) and mixed questions (−4.1, [−8.4, −0.6]) lose. Where access is irrelevant the annotations cost accuracy (non-literal: 72.9 annotated vs. 80.5 plain; emotions: 73.1 vs. 77.2); on non-literal conflicts the plain reading is right 25 of 27 times, arbitration 16. The perception call is precise (98.2, 98.7) but incomplete (recall 38.3, 73.8).

Gold perceivers. Table 27 replaces each system’s perception call with the gold perceivers of Percept-ToMi (576 questions) and Percept-FANToM ([19]) (the 62 conversations of the main FANToM sample that it annotates); everything else is unchanged. On Percept-ToMi CogRSI rises from 60.5 to 70.3, CogRSI-L from 63.3 to 81.0 and PercepToM from 48.8 to 59.3. The largest gains are on true-belief questions, where a character who saw the move must be counted: second order 2.8 to 35.4 for CogRSI and 11.2 to 59.1 for CogRSI-L. PercepToM gains instead on second-order false belief (1.9 to 72.7), where its filter must keep the right perceivers, and loses on second-order true belief (47.2 to 19.0). SC does not call perception and stays at 59.8/60.4. On FAN-ToM gold perceivers raise belief accuracy to about 91.2 for every system but leave All* unchanged (CogRSI 26.0 to 25.2), since All* also needs every answerability and info-accessibility question of a set right.

Cost and robustness

On the cost–accuracy frontier, with a significant margin. CogRSI makes 7.3 calls per question against 6.0 for SC (k=6k=6) and uses 1.51× its tokens. SC (k=9k=9) uses as many and scores lower on Main (68.7 vs. 73.1, Figure 9a). A Stage-1 run takes 4.3 hours on one H200. Against SC (k=9k=9) the paired 95% intervals exclude zero on FANToM, SimpleToM and Hi-ToM only, and Hi-ToM does not survive Holm correction (Table 4). Against the strongest baseline of each benchmark, CogRSI wins 2, ties 5 and loses 1, and none of these differences survives Holm correction (Table 26). On each of the five backbones its four-benchmark mean beats SC (k=9k=9) by +5.1 to +8.6 points at 0.98 to 1.15 times its tokens (Figure 9b, Table 13).

BenchmarkStrongest baselineCogRSI − bestHolmCogRSI − SC
FANToM All{sup}*`PercepToM + SC (belief) (23.7)+4.3 [+0.2, +8.6]–+10.3 [+5.2, +15.6]
SimpleToMPercepToM (87.5)+2.5 [−0.8, +6.0]–+19.9 [+16.1, +23.7]
ExploreToMSimToM + SC (76.9)−4.3 [−8.3, −0.6]–−0.1 [−2.1, +1.9]
Hi-ToMSC (k=9k=9) (64.9)+4.4 [+1.2, +7.6]–+7.1 [+4.2, +10.0]
OpenToMSimToM + SC (76.5)−0.7 [−3.0, +1.5]–+0.8 [−0.9, +2.7]
ToMBenchPercepToM + SC (74.3)−0.1 [−1.6, +1.3]–+0.2 [−1.1, +1.5]
CogToMSC (k=9k=9) (85.9)−1.1 [−2.3, +0.1]–−0.7 [−1.9, +0.5]
BigToMPercepToM + SC (89.5)+0.9 [−0.4, +2.2]–+1.2 [−0.1, +2.5]

Table 26. Paired differences (points) between CogRSI and the strongest baseline of each benchmark (its score in brackets) and self-consistency, main backbone, three runs per system. 95% intervals from 2,000 bootstrap resamples of stories or conversations; bold intervals exclude zero. Strongest baseline: the highest Table 3 score among all baselines, including SimToM + SC and PercepToM + SC. Holm: significant after Holm correction over the eight benchmarks (bootstrap pp-values, familywise 0.05). An interval that includes zero is not evidence of equivalence.

Table 26 gives paired differences for every

Sample mix of the vote

Figure 14a. Sample mix of the vote.

Close-split ruleAccuracy
top ≤ 3, runner-up ≥ 281.0
top ≤ 3, runner-up ≥ 381.0
top ≤ 4, runner-up ≥ 2 (ours)81.3
top ≤ 4, runner-up ≥ 381.0

Table 14b. Close-split thresholds.

fig:14: Sensitivity of CogRSI’s fixed choices (narrative test benchmarks, macro mean, all runs). (a) Accuracy of a majority over the first aa annotated and pp plain samples of the vote-only runs; ours (4+2) outlined. (b) Close-split arbitration restricted to stricter thresholds (looser ones would need new arbiter calls). Robustness checks on test data, not selections. Mixes with both readings are best; ours scores 79.9 (best mix: 79.9), and our thresholds score highest.

benchmark: against each benchmark’s strongest baseline CogRSI wins 2, ties 5 and loses 1 of 8, and after Holm correction it wins 0 and loses 0. A single CogRSI run (78.7) is above three pooled runs of SC (k=6k=6) (76.4) on the main benchmarks other than FANToM, where All* sets cannot be pooled by vote (ExploreToM’s two partitions counted separately). Each loop runs once, offline: one Stage-1 run takes 4.3 elapsed hours on one H200. Table 23 gives every backbone on the benchmarks run for all five, Table 13 SC (k=9k=9) on each of them with its token ratio, and Figure 14 the fixed choices. The sample mix is checked offline from the saved labels of the vote-only design: for every a≤4a \le4 annotated and p≤2p \le2 plain samples, the accuracy of a majority over the first aa and pp of them. The close-split thresholds can be checked only in the stricter direction, by restricting arbitration to splits that also meet the stricter rule; looser thresholds would need new arbiter calls. Both checks use test data and are robustness checks, not selections.

MethodFANToM All*SimpleToMHi-ToMBigToMOverall
Qwen3.5-122B-A10B (FP8), larger-model spot check
Zero-shot3.070.068.885.956.9
CoT11.678.555.689.058.7
SC (k=6k=6)18.280.868.089.964.2
PercepToM6.288.248.086.257.2
CogRSI25.394.072.489.770.4
Llama-3.3-70B-Instruct (AWQ 4-bit)
Zero-shot0.072.565.987.756.5
CoT1.686.065.588.060.3
SC (k=6k=6)3.787.168.988.061.9
PercepToM1.185.059.187.158.1
CogRSI12.896.669.789.467.1
GLM-4.6V-Flash
Zero-shot0.945.839.575.340.4
CoT0.050.557.480.047.0
SC (k=6k=6)0.949.861.080.248.0
PercepToM1.172.529.274.344.3
CogRSI2.971.161.584.154.9
Qwen3.5-9B
Zero-shot2.537.248.680.942.3
CoT4.650.053.488.049.0
SC (k=6k=6)6.148.361.989.251.4
PercepToM0.265.338.881.246.4
CogRSI11.476.665.190.360.9

Table 23. Other backbones. Same systems, prompts and test samples as Table 3; mean of three runs. SimpleToM: mean of behavior and judgment. Best per backbone in bold. †Fewer than three runs finished. On FANToM, CogRSI’s close-split arbitration does not apply (dialogue questions), so its FANToM cells reuse the runs of the variant without it (Appendix § 9.§ 3).

Case study

Arbitration breaks the tie. On an egocentric lie from ToMBench (Figure 10, Table 28), every run of every other system answers that Miss Zhou’s claim is true. In CogRSI’s first run the annotated reading says false, the plain one true, and the vote ties. Arbitration chooses false in all three runs.

MethodAns.What decides it (from the run logs)
SC (k=6k=6)A ×\timesmajority of six samples is A in all three runs
PercepToMA ×\times“target character not found” →\to zero-shot fallback, A (all runs)
Lib-AA ×\timesinherited tools; A in all three seeds
AnnotatedB ✓\checkmarkrun 0: three of four annotated samples answer B
PlainA ×\timesrun 0: both plain samples A; the 3–3 tie keeps the first sampled answer, A
CogRSIB ✓\checkmarkview conflict →\to arbitration chooses B (runs 0, 1); close split →\to arbitration chooses B (run 2)
CogRSI-LA ×\timeseach seed’s principle (object locations and states): all six samples A, no arbitration call

Table 28. Case study: a ToMBench item every baseline misses and CogRSI gets right by arbitration.

The main case (Table 28) was chosen as a test item that every baseline run answers wrongly and every CogRSI run answers correctly; the decisive step of each run is read from its saved records. With the loop’s principles (CogRSI-L), which are about object locations and states, all six samples agree on the wrong answer, so no arbitration call is made. A second case, a SimpleToM development question on which the readings disagree (held-out sample 1): the plain samples credit Isla with knowing about the malware, which the story never shows her learning; the annotation and the evidential rule do not, while the neutral arbiter sides with the plain reading.

Arbitration example

Figure j. Arbitration on a SimpleToM development item (held-out sample 1).

Extended related work

Self-improving systems.

Skill and tool libraries that grow with use go back to program induction ([7]) and embodied agents ((Wang et al., 2024)); agents distil experience into insights ((Zhao et al., 2024)) or workflows ((Wang et al., 2025)), learn from verbal reflection ((Shinn et al., 2023)), improve their own scaffolding ((Zelikman et al., 2024; Yin et al., 2025; Robeyns et al., 2025)), design agents ([18]), keep an open-ended archive of self-modified agents ((Zhang et al., 2026)), and evolve prompts ([8, 1]) or programs against evaluators ([28]). Table 15 compares what each keeps and how it admits a change; CogRSI alone compares inheritance with a rebuild at the same budget and lets the same loop write a principle instead of a procedure.

MethodWhat is keptLearned fromAccessRebuild
Voyagercode skillsenv. feedback–×
Reflexionverbal memorytask feedback–×
ADAS, DGMagent codebenchmark score–×
GEPApromptrollout reflection–×
SymbolicToM– (fixed)hand-designedbelief graphs×
SimToM– (fixed)hand-designedfiltered story×
PercepToM– (fixed)hand-designedfiltered story×
CogRSItools; principlefailuresannotated story✓

Table 15. CogRSI against representative self-improving systems and ToM methods. What is kept: the artifact carried to later rounds; Access: how perceptual access enters reasoning; Rebuild: whether inheritance is tested against a rebuild with the same budget. Only CogRSI inherits across rounds and keeps the whole story while marking who perceived each part.

Theory of mind in language models. SymbolicToM ([35]) tracks beliefs in graphs, SimToM ((Wilf et al., 2024)) answers from the events a character knows, TimeToM ([17]) builds temporal belief chains, PercepToM ([19]) filters the story by perception, Decompose-ToM ([32]) recursively simulates perspectives and Shoes-of-Others ((Shinoda et al., 2026)) prefills the target’s perspective; all apply a fixed procedure. DEL-ToM ((Wu et al., 2025)) scores belief traces with a trained verifier, AutoToM ((Zhang et al., 2025)) and ThoughtTracing ([21]) infer mental states by inverting an agent model, and Lu et al. ((2025)) train small models with reinforcement learning, whose ToM does not generalise beyond the training datasets ([33]). Stress tests show that LLM ToM often relies on templates ((Ullman, 2023; Shapira et al., 2024)); our construct analysis (Appendix § 13.§ 6) adds to this.

Cognitive science. Children learn that seeing leads to knowing ([29] (Wimmer et al. 1988)); adults track who saw what alongside a flexible inferential system ([2]) and infer beliefs from actions ([3]); cognitive and affective ToM dissociate ([36]). We organise the construct analysis by the ATOMS taxonomy ([4]).

Story. Miss Zhou is a professional in the competitive advertising industry. At an industry gathering, someone asks her about the details of an innovative advertising strategy her company is about to launch. Miss Zhou replies that her team develops a brand new marketing method that changes the industry landscape, but currently cannot reveal more information. She also emphasizes that this is a major breakthrough in the industry. After the industry gathering ends, Miss Zhou lets her team start planning the innovative strategy within this week. Question. Is what Miss Zhou says true? (A) Yes (B) No. Gold: B.

References

  1. [1]Lakshya A Agrawal et al. 2026. GEPA: Reflective prompt evolution can outperform reinforcement learning. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2507.19457
  2. [2]Ian A. Apperly and Stephen A. Butterfill. 2009. Do humans have two systems to track beliefs and belief-like states? Psychological Review, 116(4):953–970.DOI
  3. [3]Chris L. Baker, Julian Jara-Ettinger, Rebecca Saxe, and Joshua B. Tenenbaum. 2017. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 1:0064.DOI
  4. [4]Cindy Beaudoin, Élizabel Leblanc, Charlotte Gagner, and Miriam H. Beauchamp. 2020. Systematic review and inventory of theory of mind measures for young children. Frontiers in Psychology, 10:2905.DOI
  5. [5]Zhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Guangjun Bi, Gongyao Jiang, Yaru Cao, Mengting Hu, Yunghwei Lai, Zexuan Xiong, and Minlie Huang. 2024. ToMBench: Benchmarking theory of mind in large language models. In Proceedings of ACL.arxiv.org/abs/2402.15052
  6. [6]Matthias Cosler, Christopher Hahn, Daniel Mendoza, Frederik Schmitt, and Caroline Trippel. 2023. nl2spec: Interactively translating unstructured natural language to temporal logics with large language models. arXiv preprint arXiv:2303.04864.arxiv.org/abs/2303.04864
  7. [7]Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sablé-Meyer, Lucas Morales, Luke Hewitt, Luc Cary, Armando Solar-Lezama, and Joshua B. Tenenbaum. 2021. DreamCoder: Bootstrapping inductive program synthesis with wake-sleep library learning. In Proceedings of the 42nd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI).DOI
  8. [8]Chrisantha Fernando, Dylan Sunil Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. 2024. Promptbreeder: Self-referential self-improvement via prompt evolution. In Proceedings of the 41st International Conference on Machine Learning, pages 13481–13544.arxiv.org/abs/2309.16797
  9. [9]Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D. Goodman. 2023. Understanding social reasoning in language models with language models. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track).arxiv.org/abs/2306.15448
  10. [10]Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Shuao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, and 7 others. 2020. Evaluating models’ local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020.arxiv.org/abs/2004.02709
  11. [11]GLM-V Team. 2025. GLM-4.5V and GLM-4.1V-Thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv:2507.01006.
  12. [12]Aaron Grattafiori et al. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783.arxiv.org/abs/2407.21783
  13. [13]Yuling Gu, Oyvind Tafjord, Hyunwoo Kim, Jared Moore, Ronan Le Bras, Peter Clark, and Yejin Choi. 2026. SimpleToM: Exposing the gap between explicit ToM inference and implicit ToM application in LLMs. In International Conference on Learning Representations (ICLR).
  14. [14]Tianyi Guan, Yiding Wang, Haotong Yang, Siyuan Cao, Shirui Liu, Yi Hu, Jiaqi Li, and Muhan Zhang. 2026. ContinualSkillBench: Can LLM agents truly evolve their capabilities? arXiv preprint arXiv:2608.03874.arxiv.org/abs/2608.03874
  15. [15]Diandian Guo, Cong Cao, Fangfang Yuan, Yingqi Wang, Yueshan Wang, and Dakui Wang. 2026. Self-authored verification is unreliable in heuristic self-improving agents. arXiv preprint arXiv:2607.24300.arxiv.org/abs/2607.24300
  16. [16]Yinghui He, Yufan Wu, Yilin Jia, Rada Mihalcea, Yulong Chen, and Naihao Deng. 2023. HI-TOM: A benchmark for evaluating higher-order theory of mind reasoning in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023.arxiv.org/abs/2310.16755
  17. [17]Guiyang Hou, Wenqi Zhang, Yongliang Shen, Linjuan Wu, and Weiming Lu. 2024. TimeToM: Temporal space is the key to unlocking the door of large language models’ theory-of-mind. In Findings of the Association for Computational Linguistics: ACL 2024, pages 11532–11547.arxiv.org/abs/2407.01455
  18. [18]Shengran Hu, Cong Lu, and Jeff Clune. 2025. Automated design of agentic systems. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2408.08435
  19. [19]Chani Jung, Dongkwan Kim, Jiho Jin, Jiseon Kim, Yeon Seonwoo, Yejin Choi, Alice Oh, and Hyunwoo Kim. 2024. Perceptions to beliefs: Exploring precursory inferences for theory of mind in large language models. In Proceedings of EMNLP.
  20. [20]Divyansh Kaushik, Eduard Hovy, and Zachary C. Lipton. 2020. Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations.
  21. [21]Hyunwoo Kim, Melanie Sclar, Tan Zhi-Xuan, Lance Ying, Sydney Levine, Yang Liu, Joshua B. Tenenbaum, and Yejin Choi. 2025. Hypothesis-driven theory-of-mind reasoning for large language models. In Conference on Language Modeling (COLM).arxiv.org/abs/2502.11881
  22. [22]Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Le Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. 2023. FANToM: A benchmark for stress-testing machine theory of mind in interactions. In Proceedings of EMNLP.arxiv.org/abs/2310.15421
  23. [23]Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems.arxiv.org/abs/2205.11916
  24. [24]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP).arxiv.org/abs/2309.06180
  25. [25]Hongqiang Lin, Chao Liu, Xiaofan Bai, Xuan Jin, Yuhong Li, Nenggan Zheng, and Xipeng Cao. 2026. Beyond endpoint performance: Process-level evaluation of self-evolving agents. arXiv preprint arXiv:2609.24663.arxiv.org/abs/2609.24663
  26. [26]Yi-Long Lu, Chunhui Zhang, Jiajun Song, Lifeng Fan, and Wei Wang. 2025. Do theory of mind benchmarks need explicit human-like reasoning in language models? arXiv preprint arXiv:2504.01698.arxiv.org/abs/2504.01698
  27. [27]Guanyu Nie, Fangzhou Zhu, Shixiong Kai, Xiongwei Han, Tao Zhong, and Mingxuan Yuan. 2026. SkillEvoReg: Regularizing agent skill evolution against overfitting. arXiv preprint arXiv:2609.30861.arxiv.org/abs/2609.30861
  28. [28]Alexander Novikov et al. 2025. AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131.arxiv.org/abs/2506.13131
  29. [29]Chris Pratt and Peter Bryant. 1990. Young children understand that looking leads to knowing (so long as they are looking into a single barrel). Child Development, 61(4):973–982.pubmed.ncbi.nlm.nih.gov/2209200
  30. [30]Qwen Team. 2026. Qwen3.5 models. Hugging Face model cards, https://huggingface.com/Qwen/Qwen3.5-35B-A3B-FP8.
  31. [32]Sneheel Sarangi, Maha Elgarf, and Hanan Salam. 2025. Decompose-ToM: Enhancing theory of mind reasoning in large language models through simulation and task decomposition. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10228–10241.arxiv.org/abs/2501.09056
  32. [33]Sneheel Sarangi and Hanan Salam. 2025. Small LLMs do not learn a generalizable theory of mind via reinforcement learning. arXiv preprint arXiv:2507.15788.arxiv.org/abs/2507.15788
  33. [34]Melanie Sclar, Jane Dwivedi-Yu, Maryam Fazel-Zarandi, Yulia Tsvetkov, Yonatan Bisk, Yejin Choi, and Asli Celikyilmaz. 2025. Explore theory of mind: Program-guided adversarial data generation for theory of mind reasoning. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2412.12175
  34. [35]Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. 2023. Minding language models’ (lack of) theory of mind: A plug-and-play multi-character belief tracker. In Proceedings of ACL.
  35. [36]Simone G. Shamay-Tsoory and Judith Aharon-Peretz. 2007. Dissociable prefrontal networks for cognitive and affective theory of mind: A lesion study. Neuropsychologia, 45(13):3054–3067.
  36. [37]Linfang Shang, Ming Xu, Yiding Sun, Tianle Xia, Lingxiang Hu, Lan Xu, and Ning Zheng. 2026. When self-evolution backfires: Pre-commit gating against skill contamination in LLM agents. arXiv preprint arXiv:2608.05810.arxiv.org/abs/2608.05810

Paper details

vlnc:W_mxdmbfvr
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents