WorksComputer Science
Understanding the Role of Reasoning Data in Pretraining
Abstract
Reasoning data expose models to multi-step reasoning, but reasoning traces may cover only a subset of the available facts. How should pretraining balance atomic facts with reasoning traces? We study this question by constructing a setting in which every fact provides an atomic training signal, while only a subset appears in reasoning supervision, enabling us to measure generalization to transitions observed atomically but never used in training traces. For a linear transformer, we derive the weight assigned to atomic training signals that minimizes held-transition squared error and show that its leading-order optimum increases with reasoning-chain length and decreases with the fraction of facts held out from reasoning traces. These trends persist in a full transformer layer with an MLP, softmax attention, and cross-entropy loss. We further validate the role of reasoning complexity and data coverage in two natural-language settings: controlled multi-hop reasoning and mathematical pretraining. Together, our results provide a principled view of how reasoning-chain structure and data coverage should shape the allocation of training data during pretraining.
Introduction
Reasoning traces teach models how to compose facts across multiple steps, while atomic examples provide broader coverage of those facts. Under a fixed training budget, these objectives compete: spending more data on reasoning traces leaves less budget for learning individual facts, while spending more on atomic examples provides broader factual coverage but less practice with composition. This raises a fundamental question for pretraining: how should a model’s training budget be divided between atomic facts and reasoning traces?
This question becomes especially important as reasoning data move into pretraining, rather than being reserved for the later post-training stage. Recent work has begun to examine how reasoning data should be incorporated into pretraining. Akter et al. [2] study the effects of when reasoning data are introduced during training, as well as their diversity and quality, while Baek et al. [5] investigate repeatedly sampled domain data during pretraining. Recent training recipes likewise incorporate instruction data and thinking traces into Olmo 3 midtraining (Olmo et al., [23]) and SFT-style examples into Nemotron 3 Nano pretraining (Blakeman et al., [6]). Together, these developments establish reasoning-oriented data as an increasingly important component of the pretraining mixture. Yet they leave open a basic question: how should the optimal mixture depend on the structure and coverage of reasoning data?
The tension is subtle because reasoning traces provide something atomic examples do not: they teach the model how to use knowledge across multiple steps. But reasoning traces can also be an expensive way to teach that knowledge. Longer traces consume more of a fixed training budget, while a limited collection of traces may cover only a subset of the facts needed at evaluation. Thus, longer reasoning can simultaneously make compositional training more valuable and make broad factual coverage more important. This suggests that the optimal allocation should depend on both **reasoning-chain length and reasoning- @references#+#+#+#+ trace coverage, namely, the fraction of facts that appear inside reasoning traces. Our goal is to characterize this allocation under a fixed training budget, where differences in learning dynamics matter even when different data mixtures might eventually reach similar performance.
We study this question first in state tracking, a controlled setting for multi-step reasoning over a finite environment [20, 19, 8, 13, 17]. Starting from an initial state, the model receives a sequence of relations and must infer the state reached after executing the entire sequence. This formulation captures capabilities that arise in practical language-model tasks, including tracking entities through a narrative, reasoning over sequences of chess moves, and executing programs [16, 20]. It has consequently become a standard testbed for studying language-model reasoning, both theoretically [18, 20, 19] and empirically [18, 17] (Zhang et al. 2022).
Our construction makes the distinction between learning facts and learning to compose them explicit. Each state transition corresponds to an atomic fact. We include every transition as an atomic training example, but construct reasoning traces using only a subset of the transitions. The remaining transitions are held transitions: the model sees them individually during training, but never encounters them inside a reasoning trace. Thus, a held transition is not an unseen fact—the model has learned it atomically—but an unseen use of that fact within a reasoning chain. This construction cleanly separates factual coverage from reasoning composition and lets us ask whether learning to compose facts transfers to facts that have never been used in a reasoning trace.
We first explore this tradeoff empirically using a one-layer transformer with softmax attention and an MLP. We vary reasoning-chain length and reasoning-trace coverage. Perhaps surprisingly, we find that longer reasoning chains favor allocating more training data to atomic facts, not less. Intuitively, longer traces provide richer compositional training, but each trace also consumes more of the fixed budget, making broad factual coverage increasingly important. Broader reasoning-trace coverage likewise favors a larger allocation to atomic facts. We further confirm that the preferred mixture depends on both reasoning complexity and data coverage in two natural-language settings: controlled multi-hop reasoning and pretraining on FineMath (allal et al., 2025) and MathLake [7].
To explain these trends, we turn to a linear transformer, a standard simplification for analyzing transformer learning dynamics [1, 10] (Zhang et al. 2024). Atomic training examples directly reinforce transitions that reasoning traces may never contain, while reasoning traces train the model to compose transitions across multiple steps. We derive the atomic-data weight that minimizes squared prediction error on held transitions. To leading order, the optimal weight increases with reasoning-chain length and decreases with the fraction of held transitions. Thus, both longer reasoning chains and broader reasoning-trace coverage favor a larger atomic allocation in the analyzed regime.
Contributions. Our contributions are three-fold. First, we introduce a state-tracking setting that separates factual coverage from reasoning composition: all transitions are observed atomically, but only a subset appears inside reasoning traces. Using a one-layer softmax-MLP transformer, we show that an interior mixture of atomic and reasoning examples can outperform either extreme, and characterize how the preferred mixture changes with reasoning-chain length and reasoning-trace coverage. Second, for a linear-attention model, we derive an explicit mixture law whose optimum depends on reasoning-chain length and coverage, showing how both longer reasoning chains and broader reasoning-trace coverage can favor greater allocation to atomic facts. Third, we extend the empirical study to two natural-language settings: a controlled multi-hop task that preserves the separation between atomic facts and reasoning supervision, and a fixed-token FineMath–MathLake pretraining protocol that varies reasoning-data availability, difficulty, and introduction time.
Related Work
Reasoning data in pretraining. Recent work has increasingly explored the role of reasoning data during pretraining, rather than reserving them for post-training. Akter et al. [2] systematically study how reasoning data should be allocated between pretraining and supervised fine-tuning, showing that reasoning examples introduced during pretraining provide benefits that cannot be fully recovered by shifting the same supervision to post-training. Wang et al. [24] find that mathematical and long-chain-of-thought data during mid-training improve both base-model reasoning and subsequent reinforcement-learning performance, while Hatamizadeh et al. [9] incorporate explicit reasoning directly into the pretraining objective. An earlier related observation comes from Allen-Zhu & Li [4], who, although not studying reasoning data specifically, argue from controlled knowledge-learning experiments that instruction-finetuning data should be mixed into pretraining rather than reserved entirely for post-training. These works establish the value of introducing richer forms of supervision during pretraining, but leave open a more fundamental question: how should reasoning data be balanced against atomic data when the training budget is fixed? We study this tradeoff by separating factual coverage from reasoning composition: every fact is observed atomically, but only a subset is exposed within reasoning traces. This allows us to characterize when reasoning data are most valuable and how their benefit depends on reasoning-chain length and coverage.
Transformer theory. A growing body of work uses controlled synthetic tasks to understand how Transformers acquire, retrieve, and compose structured knowledge. Nichani et al. [22] analyze factual recall through associative memories, showing how shallow Transformers can store and retrieve large collections of subject-relation-object associations. Huang et al. [12] study how attention learns to select relevant relations and retrieve corresponding facts from a pretrained associative memory. Most closely related to our setting, Huang et al. [13] prove that Transformers trained by gradient descent can learn chain-of-thought state tracking and characterize how the algebraic structure of the transition system governs length generalization through attention concentration. These works provide insight into how Transformers store and compose knowledge, but do not study how different forms of training data interact under a fixed budget. We instead consider a setting in which every transition is observed through atomic examples, while only a subset appears in reasoning traces. This cleanly separates factual availability from exposure to multi-step composition, allowing us to analyze how the allocation between atomic and reasoning data affects the ability to compose transitions that are known individually but absent from the reasoning data.
Toy setting: state tracking
Preliminaries
Definition 1 (State-tracking world and held transitions). Let and . Entities are indexed by , and each relation acts through a fixed permutation . The full fact table contains entity-relation pairs. A length- chain , with , induces the state sequence
For an integer , draw the held set and define its complement by
The held fraction is the fraction of facts excluded from reasoning traces, and the reasoning coverage is the fraction available to those traces. The mask is sampled once and fixed throughout each run.
Definition 2 (Atomic, reasoning, and evaluation populations). An atomic example samples independently and uniformly, but supervises only . A reasoning example is sampled uniformly from chains whose actual transitions all belong to and supervises the complete state trace . Formally,
We write for expectation over the reasoning population associated with the fixed set .
Learning objective. Entities and relations are represented by distinct typed identity tokens. Both data sources use the same relation-first input format:
The shared prefix places the first prediction at the same absolute position and supplies the same number of candidate relations in both sources. Atomic examples supervise only the first transition, whereas reasoning examples additionally supervise subsequent prediction roles. Every fact is available through atomic supervision, while reasoning traces cover only a subset, allowing us to study whether facts learned individually can be used within multi-step computations.
Let be the atomic loss weight. We average the reasoning loss over its state predictions before combining the two sources:
where
Each cross-entropy term supervises the next entity using the true preceding states. A test chain is called held if at least one of its ground-truth transitions belongs to .
One full layer transformer
Our learner consists of a single causal self-attention head followed by a position-wise multi-layer perceptron (MLP). We encode token identities and positions in separate one-hot blocks and directly parameterize the merged query-key matrix, following Zhang et al. (2024); Ahn et al. [1]; Huang et al. [10, 13]. We first define the input representation, followed by the attention and output maps.
Definition 3 (Token and positional encoding). Let and be the standard basis vectors corresponding to relation and entity , respectively. Fix a maximum input-prefix length , shared by atomic and reasoning examples. The token-content vectors are
For a token at position , let and let be the -th standard basis vector. Concatenating content and position gives
All content and positional vectors are fixed. Relation, entity, and position coordinates occupy disjoint blocks of sizes , , and , respectively. We write for the corresponding content-only matrix.
Both data sources predict at position , and reasoning role predicts from the prefix ending at position . The last supervised query is therefore at position , so covers all input prefixes. We reserve for the maximum prefix length and use for the training horizon below.
We parameterize the merged query–key scores directly by a trainable matrix . In particular, the score assigned to key position by query position is .
Definition 4 (Causal self-attention head). For and a prefix , define
where, for ,
The softmax is applied column-wise.
The attention output is passed through a shared MLP to produce next-token logits over the entity and relation vocabulary.
Definition 5 (One-layer attention–MLP Transformer). Fix an MLP width and a coordinatewise nonlinearity . The trainable matrices are
with parameters . At query position , the vocabulary-logit vector is
The next-token distribution is
The first output coordinates index relations, and the last index entities. Accordingly, the probability assigned to entity is component of this distribution.
We train this model using the mixed cross-entropy objective in equation 1.
Effect of data mixture Figure 1 shows that mixing atomic and reasoning examples improves OOD accuracy over either pure-source baseline, suggesting that the two forms of supervision play complementary roles. The empirically preferred atomic example ratio generally increases with reasoning length and decreases with the held-out fact fraction . This pattern is not strictly monotone across all selected grid maximizers: the cases at deviate from the held-fraction trend. However, these panels exhibit near-optimal plateaus, with several mixture ratios attaining very similar mean OOD accuracy. We refer to Section B.1 for more experiment details.
Figure 1. Effect of atomic–reasoning mixing on state-tracking accuracy. Final-answer accuracy as a function of the atomic ratio in a synthetic world with 64 entities and 8 relations, each sampled as an independent random permutation. Curves and shaded regions indicate the mean and ±1 standard deviation across three training seeds. The empirically optimal atomic example ratio generally increases with reasoning length and decreases with the held-out fact fraction . The selected grid maximizers for at deviate from the latter trend; however, these panels exhibit near-optimal plateaus, with several tested mixture ratios achieving very similar mean OOD accuracy.
Linear-attention and the optimal mixture
We next examine how the allocation between atomic examples and reasoning traces affects optimization. Our goal is to characterize the mixture that best supports the use of held transitions after a common training horizon, and to understand how this allocation depends on reasoning length and coverage. To make this interaction tractable, we study a scalar linear-attention surrogate.
Following prior block-structured analyses of Transformer learning (Nichani et al., 2024; Huang et al., 2025; 2026a;b; Javanmard et al., 2026b;a), we use a block-diagonal attention score matrix with token-token and position-position blocks, fixing the mixed token-position blocks to zero. These two components contribute additively to the softmax logits, but their contributions multiply after exponentiation. This motivates combining the bilinear content scores with a learned positional gate through the Hadamard product in the linear surrogate:
Here controls token interactions, while controls positional routing. In particular, a zero gate entry excludes a connection, whereas an additive zero would leave its content score unchanged. Removing attention normalization and replacing the nonlinear output map with a scalar linear readout gives the following model.
Definition 6 (Scalar linear-attention model). Let contain the fixed typed one-hot content vectors, with and . For , , and , define
The trainable coordinates are the relation values , the relation–entity scores , and the routing coefficients for ; all remaining entries are fixed at zero. Write .
At role , the teacher-forced prefix ends at and has length . Let be entry of . For entity , define
Squared loss and joint training. Atomic prefixes sample the initial entity and all relation labels independently and uniformly; reasoning prefixes are uniform over chains whose actual transitions belong to . For an atomic loss weight , the mixed population loss is
Atomic examples supervise only the first state, whereas reasoning examples supervise the complete trace. Averaging the reasoning loss over its predictions makes the relative weight of the two source losses.
All active entries of are jointly trained by gradient flow:
We initialize to zero and use the same initialization for every mixture:
Here has a single unit entry at and zeros elsewhere.
Held-transition error and optimal allocation. The training objective alone does not measure whether facts learned from atomic supervision can be used at later prediction roles. We therefore evaluate later transitions that were excluded from reasoning traces. For each , let
The conditioning event has probability , so this distribution is well-defined. Define the held-transition error by
At a training horizon , define the optimal atomic weight as
Motivated by the simply transitive state-tracking setting of Huang et al. [13], we impose the following balance condition on the transition world.
Assumption 1 (Balanced state-tracking world). The world consists of entities and relations. Each relation acts through a fixed permutation . Moreover, for every pair of entities , exactly one relation maps to :
Equivalently, for each fixed entity , the map is a bijection of . The transition world is fixed throughout training and evaluation.
Under Assumption 1, a uniformly sampled relation maps any current entity to a uniformly distributed next entity. In particular, for the scalar encoding , every satisfies
The complete cyclic-shift family satisfies this assumption.
We now characterize how reasoning length and coverage affect the optimal atomic loss weight. The following theorem shows that, at a fixed training horizon, concentrates around a deterministic center as the numbers of entities and relations grow. This center increases with reasoning length and decreases with the held fraction, describing how the preferred allocation changes with the task and the available reasoning supervision.
Theorem 1 (Length–coverage trend). *Fix , , , and . Suppose Assumption 1 holds. Consider gradient flow on (4), initialized at , , and . Let be uniform among sets of facts, where is
an integer, and put . For all sufficiently large , with probability at least , the finite-time MSE-optimal atomic weight is unique in , and
where
The center increases with and decreases with .
We refer the complete proof to Section A. The theorem predicts a larger leading optimal atomic weight for longer reasoning chains and a smaller weight when a larger fraction of facts is excluded from reasoning traces, which is consistent with Figure 1.
Experiments on natural language
Controlled multi-hop reasoning
We start with a controlled natural-language task that asks a model to follow relationships between people and identify the final person, which follows from BIO pretraining in Allen-Zhu & Li [4]. This setting distinguishes two sources of supervision: atomic fact statements and multi-hop reasoning examples. We vary the atomic-document fraction in the training mixture and the fraction of person–relation pairs withheld from reasoning examples. All facts remain available in the atomic corpus, while OOD queries require at least one pair absent from reasoning supervision.
Figure 2 shows that interior mixtures outperform both pure-source endpoints on OOD accuracy in all displayed settings. Four-hop tasks favor larger atomic fractions than two- and three-hop tasks, while withholding more pairs from reasoning supervision lowers peak OOD accuracy and generally narrows or shifts the near-best region toward smaller atomic fractions. These results are consistent with Theorem 1. We refer to Appendix § 9.§ 2 for more details.
Figure 2. Multi-hop final-answer accuracy as a function of the atomic ratio in a synthetic world with 32 people and 32 relations. Columns vary hop count and rows vary reasoning holdout fraction . Curves and shaded regions indicate the mean and ±1 standard deviation across three training seeds. Large red circles mark OOD accuracies within five percentage points of each panel’s maximum.
Pretraining with mathematical reasoning data
Balancing general math text and reasoning data We pretrain Olmo-3 150M on mixtures of FineMath, a general math web-text corpus, and the OpenMathInstruct-2 subset of MathLake, a mathematical reasoning dataset. We sweep the MathLake fraction from 0% to 100% in 10% steps. The total budget is fixed at 10B tokens, so higher MathLake fractions also mean more repetition of its 277M tokens (up to epochs). Prior work either varies composition and token count together (Zhao et al., 2025), uses a single reasoning fraction [2], or tests only small domain fractions [5]; none includes both pure-source endpoints, so none shows that a mixture beats either source alone.
Table 1 shows that both extremes are suboptimal. Pure FineMath reaches a GSM8K CE of 1.785, while pure MathLake degrades to 7.513. Although its training loss falls to 0.2, GSM8K CE rises throughout training, consistent with overfitting to a heavily repeated corpus. A small share of reasoning data helps significantly, with CE dropping to 1.613 at 10% MathLake and reaching its minimum of 1.608 at 20%. Beyond this point, CE rises monotonically, returning to the pure FineMath level at 50%. We therefore run subsequent experiments at 10%, 20%, and 30% MathLake, covering the optimum and the start of the degradation regime.
| FM (%) | ML (%) | CE | PPL |
| 100 | 0 | 1.785 | 5.96 |
| 90 | 10 | 1.613 | 5.02 |
| 80 | 20 | 1.608 | 4.99 |
| 70 | 30 | 1.674 | 5.33 |
| 60 | 40 | 1.689 | 5.41 |
| 50 | 50 | 1.768 | 5.86 |
| 40 | 60 | 1.817 | 6.16 |
| 30 | 70 | 2.035 | 7.66 |
| 20 | 80 | 2.148 | 8.57 |
| 10 | 90 | 2.402 | 11.04 |
| 0 | 100 | 7.513 | 1831.40 |
Table 1. Mixture-ratio sweep between FineMath and MathLake (Olmo-3 150M, 10B tokens). Zero-shot GSM8K test CE and PPL over the solution sequence. Best in bold.
Quantity and difficulty of reasoning data At each of the mixture ratios above, we vary two properties of the reasoning data.
Held-out ratio. We control the fraction of the MathLake pool available for training and withhold the rest. Since the mixture ratio and total budget are fixed, every run sees the same number of reasoning tokens. To make the held-out ratio a monotone intervention, the pools are nested. We shuffle MathLake once at the document level and take each pool as a prefix of the next, where the 25% pool is a subset of the 50% pool, which is a subset of the 75% pool. Thus, moving from a smaller to a larger only adds unseen documents and never swaps one sample for another.
Difficulty. We use MathLake’s per-problem difficulty scores (1–10) as a proxy for reasoning length and complexity. The scores closely track solution length (258 tokens at level 1 vs. 680 at level 9), so splitting by level alone would also split by token volume. We instead cut the corpus at its token-weighted median, which falls between levels 4 and 5, and cap both halves at 133.9M tokens, so easy and hard runs see identical reasoning-token budgets at identical repetition rates. A random 50% pool (138.7M tokens) serves as the control.
As shown by Figure 3a, the optimal mixing ratio increases with , therefore decreases with held-out ratio. In Figure 3b, the optimal mixing ratio increases with difficulty. The results are consistent with Theorem 1.
Figure 3. GSM8K CE at step 5000, with uncertainty estimated by bootstrapping evaluation examples 3 times. (a) Held-out ratio. FineMath / MathLake mixture: 70/30, 80/20, 90/10. GSM8K CE; , , , . (b) Difficulty (equal-token halves). FineMath / MathLake mixture: 70/30, 80/20, 90/10. GSM8K CE; hard (levels 5–10), random 50%, easy (levels 1–4). (a) Varying the fraction of the MathLake pool available for training, at three FineMath/MathLake ratios. (b) Training on the easy or hard half of MathLake versus a random 50% subset.
When to introduce reasoning data Finally, we fix the total reasoning budget and vary when it enters training. Each run lasts 10k steps with MathLake making up 20% of all training tokens, but the first steps use FineMath only. The full reasoning budget must then fit into the remaining steps, so MathLake’s share of the post-switch mixture is 20%, 25%, 33%, 50%, and 100% for , 2k, 4k, 6k, 8k. We implement this by resuming each arm from the step- checkpoint of a single pure-FineMath run, carrying over the step count and learning-rate schedule.
Delaying reasoning data brings no benefit (Table 2): GSM8K CE increases monotonically with . Since starting early is never worse, we recommend mixing in reasoning data from the beginning of training, consistent with Baek et al. [5], who find that domain data is best introduced early.
| Pure FM steps () | ML share while mixing | CE | PPL |
| 0 | 20.0% | 1.744 | 5.72 |
| 1k | 22.2% | 1.749 | 5.75 |
| 2k | 25.0% | 1.754 | 5.78 |
| 3k | 28.6% | 1.767 | 5.85 |
| 4k | 33.3% | 1.771 | 5.88 |
| 6k | 50.0% | 1.853 | 6.38 |
| 8k | 100.0% | 2.411 | 11.14 |
Table 2. Timing of reasoning data. Each run trains Olmo-3 150M for 10k steps with MathLake making up 20% of all tokens; the first steps use pure FineMath, so the MathLake share while mixing rises with . Zero-shot GSM8K test CE and PPL over the solution sequence at the final checkpoint. Best in bold.
Conclusion
We studied a fundamental question in reasoning-oriented pretraining: how should a fixed training budget be divided between atomic facts and reasoning traces? By separating factual coverage from exposure to multi-step composition, we showed that the optimal allocation depends systematically on reasoning-chain length and reasoning-trace coverage. Perhaps surprisingly, longer reasoning chains can favor greater allocation to atomic facts, as richer compositional training comes at the cost of broader factual coverage. Our theoretical analysis explains this effect in a simplified linear setting, and experiments with full Transformers and realistic mathematical data demonstrate that the same qualitative trends persist beyond the controlled regime. Together, our results provide a principled perspective on how the structure and coverage of reasoning data should shape pretraining mixtures.
Limitations. Our controlled setting abstracts away heterogeneous data sources and varying data quality in large-scale pretraining. We also focus on pretraining and do not study how different data mixtures affect subsequent SFT and RL, which we leave for future work.
Acknowledgments
This research was supported by NSF CAREER Award 2146492, NSF-Simons AI Institute for Cosmic Origins (CosmicAI), NSF AI Institute for Foundations of Machine Learning (IFML), and an Apple Research Award.
AI use statement
We have not used generative AI tools for tasks with required disclosure. Additionally, we used generative AI tools for polishing the paper. We have reviewed all AI-assisted work. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.
[Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25(49):1–55, 2024.]
[Yi Zhang, Arturs Backurs, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, and Tal Wagner. Unveiling transformers with lego: a synthetic reasoning task. arXiv preprint arXiv:2206.04301, 2022.]
[Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: Rl post-training amplifies behaviors learned in pretraining. arXiv preprint arXiv:2504.07912, 2025.]
Proof of the length-coverage theorem
Throughout, , , , , , and . For a family , define
The base norm is entrywise maximum for tables/vectors and . A multilinear map of base operator norm satisfies
The same calculation applies to higher multilinear products.
Lemma 1 (Balance and the fixed-mask cubic). For a fixed mask with a nonempty reasoning population, put
For ,
Define
This is the leading gain polynomial identified in Lemma 3.
Proof. Complete shifts and the scalar code give
For every relation tuple ,
Consequently, for ,
For , the two averages are and , since . Substitution in
gives equation (16). In general, .
Lemma 2 (Concentration of the balanced cubic). Fix , , and . For sufficiently large , with probability at least ,
An explicit is given in equation (39).
Proof. 1. Raw moments and exact normalizations. Let indicate validity and set
Given , let start at with its first edge forced to be and its remaining relations fresh. Let be independent and uniform. Define
Then
Explicit expectation-bias bounds. For complete shifts,
The independent vertex samples underlying each configuration are
| statistic | independent samples | distinct requests if no collisions |
| retained | ||
| retained | ||
| retained, 1 held |
Table 1 (PDF p. 14).
The auxiliary path starts with ; the mixed test edge is . For the event that two independent samples coincide,
Assume . If distinct retained facts are requested,
For the collision-free mixed configuration,
If is a normalized inclusion probability and a scalar weight, then
On , use respectively , , or . Therefore, with expectation over masks,
The reference means use and ; are independent before masking.
Global sensitivity and concentration
If masks differ by one exchange, . A configuration with edge requests can change only if one request hits a changed fact. Each request is marginally uniform, so
The two forced copies of a diagonal-pair edge count as one request. Thus the global exchange constants are
For a function with exchange constant , reveal the smaller subset in steps. Its Doob increments obey
The range bound follows by coupling two possible next reveals with one exchange in their uniform completions. For the conditional log moment function ,
Iteration and exponential Markov yield
There are statistics in equation 28. Set
A union bound gives an event , , on which
Ratios and cubic coefficients
Assume and write . On ,
Introduce scalar coefficients, not parameter matrices:
The product inequality gives
Now
Direct differentiation supplies all three norm constants:
Hence an explicit mask error is
For fixed , , all four errors in equation (34) are ; equations equation (36)–equation (39) give the same rate. Finally,
Lemma 3 (Uniform finite-time cubic approximation). Fix a mask with and a horizon . For sufficiently large , the joint raw flow satisfies
An explicit bound is equation (61).
Proof. 1. Exact raw gradients. For a chain function , define the weighted linear functional
Write and . Then
For ,
The polynomial ODE thus has a global real solution, with smooth -dependence on finite intervals. Here .
2. Rescaled equations and their operator bounds. Use only as an auxiliary coordinate. Define
The arguments in these maps are scalar tables, not held sets. Let ; subscripts below denote . The candidate probabilities give the base operator bounds
For example, the final coefficient is . Apply equation (13) to the affine mixture of these operators:
Set
Row scaling is . Substitution into the raw flow gives
Thus, for , , ,
Close the bootstrap with a stated dimension threshold.
Let and assume
Up to the first exit from , ,
No exit occurs on . Substituting back into equation (48) sharpens the other two estimates:
Derive the first two expansions by integration.
Let
Using , equations equation 47 and equation (51) imply
Consequently,
For routing, the exact equation gives
In particular, the last two bounds are
5. Multiply the expansions and project onto the target. Let and . For , and
The cubic prediction is therefore
The product remainder satisfies
Lemma 1 supplies the missing target-projection step:
Finally,
Thus an explicit time error is
All bounds include two derivatives in by equation 13.
Lemma 4 (Explicit center and stable ordering). Set , , , and . If and , then has a unique maximizer and
Proof. Direct differentiation gives
A quadratic with opposite endpoint signs has exactly one root in . Writing for a candidate mixture and ,
For the exterior slopes,
These follow from the sign of and the endpoint values in equation 63. Hence
Existence follows by compactness; these inequalities give interiority and uniqueness. Writing , we have
For ,
Proof of Theorem 1. On the event of Lemma 2, let
The explicit sufficient conditions are
They hold for sufficiently large with fixed. Lemma 4 gives
Maximizing is equivalent to minimizing . Moreover,
Figure 4. Held-out atomic recall. ID and OOD final-answer accuracy and held-out atomic accuracy versus atomic example ratio, using the same setting as Figure 1. For held-out atomic evaluation (green dashed), we sample uniformly from held-out facts and independently and uniformly from all relations, scoring only . Curves and shaded regions show the mean and standard deviation across three training seeds.
Additional details of experiments
Additional details of fig:toy
Atomic examples use the same prompt format as reasoning examples but supervise only the first transition. We sample the starting state uniformly from all states and the first relation uniformly and independently from all relations. Thus, the supervised state–relation pair is uniform over the entire fact table, including pairs held out from reasoning supervision. After the first transition, each subsequent relation is sampled uniformly from the seen outgoing relations of the current state, and the state is updated accordingly. Cross-entropy is applied only to the first successor state; all remaining targets are masked. All runs use 4000 AdamW updates with learning rate 0.002, weight decay 0.01, and batch size 256.
Additional details of fig:multihop-mixture
Knowledge world and reasoning examples. We construct a synthetic world containing 32 people and 32 relation rules, giving 1,024 atomic facts. An atomic document states one person–relation fact. An -hop reasoning example specifies a starting person and relations, followed by a worked solution listing the intermediate transitions and final person.
Examples of training data. The following excerpts are taken directly from the generated corpora. The atomic-pretraining examples are:
Atomic document 1: Nevaeh Tristan Kay’s assessor is Jesse Alexis McCann.
Atomic document 2: Jesse Alexis McCann’s trainer is Delaney Emma Chadwick.
The corresponding SFT-format reasoning example composes these two facts:
Question: Starting from Nevaeh Tristan Kay, who do you reach after following: assessor, trainer? Answer: Nevaeh Tristan Kay’s assessor is Jesse Alexis McCann. Jesse Alexis McCann’s trainer is Delaney Emma Chadwick. Answer: Delaney Emma Chadwick.
Reasoning holdout and data splits. For each holdout fraction , we reserve a subset of person–relation pairs that may not occur in reasoning-training chains. These subsets are nested across , and every reserved fact remains in the atomic corpus. Training chains and ID evaluation queries use only unreserved pairs; OOD queries contain at least one reserved pair. Each setting contains 3,000 reasoning-training rows, 300 ID queries, and 300 OOD queries. At , , only 2,921 distinct legal training chains exist, so some training rows repeat.
Model and training. We train a Transformer from scratch with 6 layers, hidden dimension 384, 6 attention heads (12M), and context length 512 for 300 steps. Each update contains 32,768 tokens (64 sequences of length 512). We use Muon for Transformer weight matrices with a peak learning rate of 0.02, and AdamW for the remaining parameters.
References
- [1]Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36:45614–45650, 2023.arxiv.org/abs/2306.00297
- [2]Syeda Nahida Akter, Shrimai Prabhumoye, Eric Nyberg, Mostofa Patwary, Mohammad Shoeybi, Yejin Choi, and Bryan Catanzaro. Front-loading reasoning: The synergy between pretraining and post-training data. In International Conference on Learning Representations, volume 2026, pp. 80909–80928, 2026.arxiv.org/abs/2510.03264
- [4]Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=5x788rQbcj.
- [5]Christina Baek, Ricardo Pio Monti, David J. Schwab, Amro Abbas, Rishabh Adiga, Cody Blakeney, Maximilian Böther, Paul Burstein, Aldo Gael Carranza, Alvin Deng, Parth Doshi, Vineeth Dorna, Alex Fang, Tony Jiang, Siddharth Joshi, Brett W. Larsen, Jason Chan Lee, Katherine L. Mentzer, Luke Merrick, Haakon Mongstad, Fan Pan, Anshuman Suri, Darren Teh, Jason Telanoff, Jack Urbanek, Zhengping Wang, Josh Wills, Haoli Yin, Aditi Raghunathan, J. Zico Kolter, Bogdan Ghaza, Ari S. Morcos, Matthew L Levitt, and Pratyush Maini. The finetuner’s fallacy: When to pretrain with your finetuning data. In Third Conference on Language Modeling, 2026. URL https://openreview.net/forum?id=HgDsWB3KFB.arxiv.org/abs/2603.16177
- [6]Aaron Blakeman, Aaron Grattafiori, Aarti Basant, Abhibha Gupta, Abhinav Khattar, Adi Renduchintala, Aditya Vavre, Akanksha Shukla, Akhiad Bercovich, Aleksander Ficĕk, et al. Nemotron 3 nano: Open, efficient mixture-of-experts hybrid meta-transformer model for agentic reasoning. arXiv preprint arXiv:2512.20848, 2025.
- [7]Mengzhang Cai, Xin Gao, Yu Li, Honglin Lin, Zheng Liu, Zhuoshi Pan, Qizhi Pei, Xiaoran Shang, Mengyuan Sun, Zinan Tang, et al. Opendataarena: A fair and open arena for benchmarking post-training dataset value. arXiv preprint arXiv:2512.14051, 2025.arxiv.org/abs/2512.14051
- [8]Niklas Forner, Marcel Kühn, Matthias Thamm, and Bernd Rosenow. Learning dynamics of chain-of-thought state tracking in a solvable transformer model. arXiv preprint arXiv:2606.18164, 2026.arxiv.org/abs/2606.18164
- [9]Ali Hatamizadeh, Syeda Nahida Akter, Shrimai Prabhumoye, Jan Kautz, Mostofa Pat-wary, Mohammad Shoeybi, Bryan Catanzaro, and Yejin Choi. Rlp: Reinforcement as a pretraining objective. In International Conference on Learning Representations, volume 2026, pp. 41375–41402, 2026.
- [10]Jianhao Huang, Zixuan Wang, and Jason D. Lee. Transformers learn to implement multi-step gradient descent with chain of thought. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=r3DF5sOo5B.arxiv.org/abs/2502.21212
- [12]Ruomin Huang, Eshaan Nichani, Jason D Lee, and Rong Ge. Fine-tuning dynamics of in-context factual recall in transformers. arXiv preprint arXiv:2605.27774, 2026b.arxiv.org/abs/2605.27774
- [13]Yu Huang, Zixin Wen, Aarti Singh, Yuejie Chi, and Yuxin Chen. Transformers provably learn chain-of-thought reasoning with length generalization. Advances in Neural Information Processing Systems, 38:126815–126965, 2026c.arxiv.org/abs/2511.07378
- [16]Najoung Kim and Sebastian Schuster. Entity tracking in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3835–3855, 2023.arxiv.org/abs/2305.02363
- [17]Belinda Z Li, Zifan Carl Guo, and Jacob Andreas. (how) do language models track state? arXiv preprint arXiv:2503.02854, 2025.arxiv.org/abs/2503.02854
- [18]Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shortcuts to automata. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=De4FYqjFueZ.arxiv.org/abs/2210.10749
- [19]William Merrill and Ashish Sabharwal. A little depth goes a long way: The expressive power of log-depth transformers. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=5pHfYe10iX.arxiv.org/abs/2503.03961
- [20]William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=QZgo9JZpLq.arxiv.org/abs/2404.08819
- [22]Eshaan Nichani, Jason Lee, and Alberto Bijetti. Understanding factual recall in transformers via associative memories. In International Conference on Learning Representations, volume 2025, pp. 4166–4207, 2025.arxiv.org/abs/2412.06538
- [23]Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groenveld, Faezeh Brahman, Finbarr Timbers, Hamish Ivison, et al. Olmo 3. arXiv preprint arXiv:2512.13961, 2025.
- [24]Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling. In 2nd AI for Math Workshop @ ICML 2025, 2025. URL https://openreview.net/forum?id=chCeUHjLzs.arxiv.org/abs/2506.20512



