WorksComputer Science
MathaMAGICal: Validating Trajectory Data Attribution During Capability Acquisition
Abstract
Influence-function-based methods like MAGIC provide exact first-order scores for trajectory data attribution, yet only retraining indicates whether those scores accurately predict data removal. We evaluate MAGIC on M- and M-parameter GPT-2-style models learning multistep arithmetic from scratch on filtered web text and four synthetic math families. Full-run attribution at M suffers from numerical overflow, and even with metagradient clipping, scores correlate near chance with retraining (linear datamodeling score, LDS, –). In contrast, short checkpoint windows yield stable, accurate predictions at both scales (LDS – across four -step windows at M), whereas a single-checkpoint gradient-dot baseline shows near-zero correlation. Rankings of individual training blocks predict retraining only within that single run and do not transfer across seeds or hyperparameters (), whereas format-level patterns replicate across runs. Two data families, worked examples and primitive facts, occupy – their corpus share in both the helpful and harmful score tails, reflecting strong bidirectional local sensitivity distinct from net benefit under multi-epoch family deletion. These results show that trajectory attribution is reliable within short, validated windows and yields family-level insights that generalize across runs.
40th Conference on Neural Information Processing Systems (NeurIPS 2026). Workshop: ATTRIB: Attributing Model Behavior at Scale: Data Attribution and Provenance.
Introduction
Trajectory-based data attribution methods like MAGIC estimate how training examples influence model behavior by differentiating held-out loss back through executed optimizer updates [14, 9]. MAGIC yields an exact first-order derivative along one training path, yet unrolling across thousands of pretraining steps overflows numerically, and stabilized finite scores still require retraining checks to verify whether they predict data removal. To test when trajectory attribution holds during capability acquisition without pretraining contamination, we train 86M and 152M GPT-2-style models [34] from scratch on filtered FineWeb-Edu text [31] and four synthetic arithmetic families: worked examples, primitive facts, verbal procedures, and code traces (Figure 1). Models train on single operations and MULTIPLY-then-ADD chains and are evaluated on held-out ADD-then-MULTIPLY problems (e.g., take 7, add 45, then multiply by 3 to get 156). Using bergson [33], we test full-run and short-window MAGIC scores against subset-removal retraining (Appendix § 9.8).

Figure 1. Models train from scratch on 50% web text and 50% synthetic arithmetic across four data families (bar, left); 152M web+math solves all 200/200 held-out compositions. Differentiating held-out loss through a full 86M run overflows and clipped scores fail retraining validation (LDS 0.17–0.22), whereas short checkpoint windows validate at both scales (LDS 0.99 on 25-step 86M windows; 0.86–0.99 across four 100-step 152M windows).
Contributions.
Horizon limits and short-window validity. Full-run MAGIC attribution overflows on our 86M model, and even with reverse-pass metagradient clipping, scores correlate near chance with retraining (linear datamodeling score, LDS, 0.17–0.22). Short continuations from intermediate checkpoints validate at both scales (LDS 0.86–0.99 across four 100-step windows at 152M), whereas a single-checkpoint gradient-dot baseline sits near zero.
Family-level sensitivity vs. trajectory-bound block rankings. Individual block rankings predict retraining on their own trajectory yet fail to transfer across seeds, batch sizes, or optimizers on identical blocks (rank correlation ), whereas family-level patterns replicate across runs. Worked examples and primitive facts occupy – their corpus share in both helpful and harmful 5% score tails, capturing bidirectional local sensitivity distinct from net benefit under multi-epoch family deletion.
Related Work
Unlike corroborative text matching [40, 22], contributive attribution estimates counterfactual influence from checkpoint gradients or subset regressions: influence functions [18, 11], TracIn [32], Datamodels [15], TRAK [30], LESS [41], and TrackStar [4]. MAGIC and metagradient descent differentiate optimizer updates end-to-end [14, 9] in bergson [33], though horizon drift, cross-run variance, stagewise sign flips, and group interactions limit retraining agreement [28, 6, 20, 12, 7] and circuit tracing targets internal units [5].
Pretraining studies use attribution or procedural mixtures to shape reasoning [36, 38, 19, 13, 35, 16], while Transformer arithmetic shows format-sensitive transitions and compositional limits [29, 27, 37, 8, 21, 39, 25]. Under synthetic provenance control, we test the horizon and granularity where trajectory attribution predicts retraining (Appendix § 9.7).
Method
To study skill acquisition, we train models from scratch on a mixture of web text and synthetic arithmetic from four data families (Figure 2), built from ADD, SUB, and MULTIPLY operations. During training, we include single operations and MULTIPLY-then-ADD sequences, but deliberately exclude ADD-then-MULTIPLY programs. We hold out 200 ADD-then-MULTIPLY problems as our test set and score them by exact match. Control models trained without arithmetic solve none of the 200 (Appendix § 9.§ 5).

Figure 2. Schematic family formats for the training-allowed order MULTIPLY then ADD. The first three show the same execution; primitive facts use independent operands. Actual program identifiers are exclusive to one family per seed (Appendix § 9.§ 1).
Attribution and retraining validation. We target the negative log-likelihood of held-out output tokens and use bergson [33] to differentiate this loss through executed training steps with MAGIC [14], assigning one metagradient score per 512-token block. Each score estimates the first-order effect of removing a block: positive scores mark harmful blocks (removal lowers held-out loss) and negative scores mark helpful ones, matching retrained loss changes at on validated windows (Appendix § 9.§ 3). To validate predictions, we remove sampled block subsets, retrain from the same checkpoint, and compute the linear datamodeling score (LDS) [15, 6], the rank correlation between predicted and observed loss changes per held-out query (1 is exact rank agreement; 0 is chance).
Two sets of metrics. Because per-block rankings do not transfer across independent runs (§ 4.5), we evaluate data families with two complementary metrics. Tail lift measures local sensitivity as a family’s enrichment in the most helpful and most harmful 5% of scored blocks relative to its corpus share; equal enrichment in both tails indicates high variance whose opposing effects largely cancel in the mean. Net benefit measures macro-scale reliance via multi-epoch deletion of an entire family (§ 4.7).
Attribution windows. Because attributing the full training run fails validation (§ 4.1), we attribute short windows instead. Each window starts from a saved checkpoint and trains for 100 more steps on new blocks. We attribute only those 100 steps, since longer windows stop matching retraining (§ 4.2). We place the four main 152M windows before the skill (0/200), as it appears (17/200), as it forms (89/200), and once it is present (174/200) (Figure 3; Table 6).
Figure 3. Held-out accuracy of 152M web+math. Each window attributes 100 steps from its marked checkpoint; the model reaches 200/200 by a third of training and holds it.
Table 6: Window index. Each attribution window continues training from an initialization checkpoint and attributes only the steps it runs. Stage names are fixed by held-out accuracy at initialization. Configuration is optimizer steps rows per step.
| Window | Run ID | Initialized from | Configuration |
| 152M web+math | |||
| Before the skill | W3B-before | step_1510 | |
| Skill appearing | W3B-onset | step_2114 | |
| Skill forming | W3B-midrise | step_2718 | |
| Skill present | W3B-after | step_4530 | |
| 86M web+math, 300M | |||
| Before the skill | winB2-2375 | step_2375 | |
| Skill appearing | winB2-2755 | step_2755 | |
| Skill appearing, seed 1 | winB2-2755-s1 | step_2755 | |
| Skill forming | winB2-3135 | step_3135 | |
| Skill present | C2-post5700 | step_5700 | |
| 152M webmath | |||
| Late branch | W150s-b30517-m1.0-L100 | step_400 | |
| Early branch | W150s-b15312-m1.0-L100 | step_400 | |
| Early branch, 50% math | W150s-b15312-m0.5-L100 | step_700 | |
| Curriculum arms | |||
| 86M webmath | CUR-nl-win | step_700 | |
| 86M codemath | CUR-code-win | step_400 |
Table 6. Window index. Each attribution window continues training from an initialization checkpoint and attributes only the steps it runs. Stage names are fixed by held-out accuracy at initialization. Configuration is optimizer steps rows per step.
Corpus controls. We match families on token budget and on reuse, the average number of times each distinct program appears, so no family has more data or repetition than another. Reuse is 5.0× in the 86M corpus (Appendix § 9.§ 1) and 4× in the main 152M run (Appendix § 9.§ 4). Contamination audits of the web text find zero matches to our record formats or held-out items, and web-only controls solve 0/200 (Appendix § 9.§ 1).
Experimental design
Models. Models are named by parameter count and training diet (Appendix § 9.§ 5): web+math (single mixed run), web→math or code→math (arithmetic-free pretrain followed by a math phase), and *webonly or code-only (controls). The main model is 152M web+math (3.0B tokens); two-phase 152M runs saw fewer arithmetic tokens than the 86M series and are flagged where reported (Appendix § 9.7).
Stabilizers, seeds, and baselines. We test stronger damping, forward gradient clipping, and Muon [17, 23] (altering forward training) alongside reverse-pass metagradient clipping (leaving forward weights unchanged): at 86M (189/200 reference), stronger damping collapses learning (0/200), whereas forward clipping (193/200) and Muon (194/200) preserve capability (Table 7). Most windows use one seed; full-run instability uses two and deletion three. Because reweighting alters the subsequent trajectory, across which per-block rankings do not transfer (§ 4.5), we evaluate family deletion rather than block upweighting. TrackStar and SOURCE [4, 33] exceeded compute limits, so our baseline is single-checkpoint gradient-dot on identical retraining subsets. Code, checkpoints, and scores are released (Appendix § 9.§ 6).
| Recipe | Held-out (/200) |
| Conventional reference | 189 |
| Muon optimizer | 194 |
| Forward gradient clipping () | 193 |
| Damping or | 0 |
Table 7. Held-out exact match out of 200 across forward-training recipes at 86M (1.0B tokens; reverse-pass metagradient clipping leaves forward weights unchanged). A model at 0/200 has no arithmetic to attribute, so the damped configurations are excluded. Green marks capability preserved and orange capability destroyed.
Results
Full-run attribution overflows, and clipped scores fail validation
Full-run attribution overflows at 86M: the running metagradient grows by 0.4–0.5% per step, leaving 44.7% of block scores non-finite after 30,518 steps. Clipping the running metagradient in an otherwise bit-identical rerun eliminates non-finite scores, yet aggregate LDS stays near chance at 0.217 and 0.168 across two seeds (12 subsets each; , 0.60), showing that finiteness alone does not ensure retraining agreement. Stronger optimizer damping likewise fails to preserve both capability and full-run stability (Figure 11; Table 7, Appendix), motivating short-window attribution.
Figure 11. Left: growth of the running metagradient norm at the measured rate of 0.4–0.5% per step, against the largest finite score observed. Right: mean per-query LDS against retraining for the clipped full-run scores (two seeds) and for short windows of 100, 200, and 400 steps. Clipping removes the overflow without restoring agreement; shorter windows restore it.
Short-window attribution validates against retraining
Across 24 short attribution windows at 86M (23 with full 1%-removal retraining grids), we vary horizon length, batch size, damping, optimizer, acquisition stage, seed, and query set. At 512 rows per step, windows reach LDS 0.90–0.99; extending the horizon to 400 steps drops LDS to −0.08, and shrinking batch size to 64 rows drops it to 0.08 (step count and batch size co-vary across fixed-budget windows; Table 8, Appendix).
Table 8: Measured validity envelope for short-window attribution at 86M parameters, one axis at a time from the settled configuration (512 rows per optimizer step, , AdamW, constant lr , fp32). Mean per-query LDS against retraining. Steps and batch are confounded across the grid and should be read jointly. Green cells mark LDS , orange cells mark LDS ; intermediate values are left untinted.
| Axis | Setting | LDS |
| Horizon (256 rows/step) | 100 steps | 0.57–0.94 |
| Horizon (256 rows/step) | 200 steps | 0.28 |
| Horizon (256 rows/step) | 400 steps | −0.08 |
| Batch | 64 rows/step | 0.08 |
| Batch | 128 rows/step, 100 steps | 0.854 |
| Batch | 512 rows/step | 0.97–0.99 |
| Boundary position | Before the skill | 0.61 |
| Boundary position | Skill forming | 0.90 |
| Boundary position | Skill present | 0.93 |
| Damping | Non-finite / 0.96 | |
| Optimizer | Muon | 0.84 |
| Target | In-distribution | 0.90 |
Table 8.
When we use attribution windows before, during, and after emergence, we obtain LDS values ranging from 0.61 to 0.93. This shows continuous loss sensitivity across short continuations while exact-match accuracy changes by only a handful of problems before the main increase in capability (Appendix § 9.§ 2).
Short-window validation across acquisition at 152M
The main 152M web+math model reaches 200/200 held-out and 100/100 in-distribution exact match one-third through its 3.0B-token run and stays saturated (Figure 3). On unseen blocks, four 100-step windows probe before the skill, skill appearing, skill forming, and skill present. Across 48 retraining subsets and eight queries per window, median per-query LDS reaches in all four windows and in three (Table 9, Appendix), lowest at skill appearing. All four share one parent run, and batch/horizon differences preclude isolating a clean 86M-to-152M scaling comparison.
Table 9: Attribution windows on 152M web+math. Each window continues from its checkpoint on 25,600 unseen blocks for 100 optimizer steps and attributes only those steps. LDS is the per-query rank correlation between the predicted and measured effects of removing 1% of blocks, over a retraining grid. All 32 query-window cells reach .
| Window | Init step | Held-out at init | Median LDS | LDS range |
| Before the skill | 1,510 | 0/200 | 0.991 | 0.980–0.995 |
| Skill appearing | 2,114 | 17/200 | 0.863 | 0.777–0.960 |
| Skill forming | 2,718 | 89/200 | 0.992 | 0.983–0.996 |
| Skill present | 4,530 | 174/200 | 0.991 | 0.985–0.994 |
Table 9.
First-order baseline comparison
Using the same retraining subsets, we apply single-checkpoint gradient-dot attribution. This achieves near-zero LDS compared to 0.57–0.99 for unrolled MAGIC (Figure 10, Appendix). This confirms the value of short-horizon unrolling.
Figure 10. Aggregate LDS for the unrolled scores and the first-order gradient-dot baseline, computed on the identical retrain subsets, for four windows at 86M and two at 152M.
Family-level sensitivity vs. trajectory-bound block rankings
Primitive facts and worked examples are enriched in both extreme 5% tails of validated attribution scores at 2.1–2.6 times their corpus share, while code traces are mildly enriched, verbal procedures are chance-like, and web text is less than a quarter of its chance share (Figure 4; Table 10, Appendix). The symmetric enrichment in both tails shows that attribution detects high bidirectional local sensitivity rather than uniform helpfulness across a family. Because worked examples share the held-out evaluation format, the detected sensitivity reflects both arithmetic composition and alignment with the evaluation format.
Figure 4. Family-level sensitivity vs. multi-epoch deletion on 152M web+math. Left: both 5% score tails at initial emergence as lift over corpus share; worked examples and primitive facts populate both tails (bidirectional local sensitivity), while web text stays depleted. Right: multi-epoch family deletion against step-matched random removal (18 retrains, 3 seeds) measures net benefit outside the local regime and aligns in four-family rank (). Worked examples share the evaluation format, and web removal shifts the arithmetic fraction.
| Window | Worked | Primitive | Verbal | Code | Web text |
| Before the skill | 2.83 / 2.63 | 2.13 / 2.15 | 0.96 / 1.07 | 1.20 / 1.21 | 0.22 / 0.23 |
| Skill appearing | 2.64 / 2.43 | 2.14 / 2.16 | 1.21 / 1.23 | 1.39 / 1.41 | 0.16 / 0.19 |
| Skill forming | 2.10 / 1.92 | 2.40 / 2.48 | 1.15 / 1.16 | 1.51 / 1.48 | 0.21 / 0.24 |
| Skill present | 1.44 / 1.21 | 1.63 / 1.61 | 0.89 / 0.82 | 0.98 / 1.02 | 0.77 / 0.83 |
Table 10. Family share of the most helpful and most harmful 5% of blocks, as a lift over each family’s share of the window; a lift of one is chance. Entries are helpful / harmful, averaged over eight queries.
Individual block rankings predict retraining on their own trajectory but do not transfer across runs. On the same set of blocks, across different seeds, batch sizes, or optimizers, block-level rankings of sensitivity fail to generalize (Spearman correlations between −0.01 and +0.09; top-5% overlap near chance; Table 11, Appendix). In contrast, the family-level tail enrichment measures are reproducible across these same changes, establishing that data families are the reproducible unit across trajectories.
| Pair | Agreement | Pair | Agreement |
| Base vs. large batch | +0.091 | Base vs. Muon | +0.005 |
| Base vs. damped | +0.020 | Large batch vs. Muon | +0.012 |
| Large batch vs. damped | Muon vs. damped | +0.040 |
Table 11. Block-level rank agreement between four windows that share identical blocks and differ only in batch size, optimizer, or damping.
Stage dynamics of family tail sensitivity
Figure 5 illustrates how the enrichment of the helpful tail changes over the four 152M attribution windows. Worked examples are most enriched before accuracy emerges and primitive facts are most enriched during emergence. Web text is less than a quarter of its chance share until the model saturates, at which point the family-level shares approach the corpus proportions. The harmful tail shows the same pattern of movement (Table 10, Appendix). The per-block absolute effect sizes diminish as the held-out loss drops, whereas the loss-normalized sensitivity metric increases and then saturates (Table 4, Appendix).
Figure 5. Helpful-tail lift per family across the four validated 152M windows, plotted at their starting checkpoints (four discrete windows; line segments aid visual tracking only). Harmful-tail lifts follow nearly identical trajectories (Table 10, Appendix), reflecting symmetric bidirectional local sensitivity.
| Window | Baseline loss | Median SD (loss) | SD / baseline |
| Before the skill | 0.342 | 0.0008 | |
| Skill appearing | 0.101 | 0.0022 | |
| Skill forming | 0.0067 | 0.0114 | |
| Skill present | 0.00055 | 0.0111 |
Table 4. Retraining effect sizes across the four 152M windows. SD is the standard deviation of the held-out loss change across the 48 removals of 1% of blocks. Baseline loss and SD are medians over the eight queries.
Multi-epoch family deletion
To measure net family benefit independently of local tail sensitivity, we run 18 continuations of 152M web+math from skill appearing: one deletion arm per family plus a step-matched random-deletion control, each removing 3,200 of 25,600 blocks over eight epochs (3 seeds per arm).
Deleting worked examples halts learning (0/200 held-out accuracy vs. 148/200 under random removal). Deleting primitive facts also reduces accuracy (126/200), while deleting code traces (139/200) or verbal procedures (143.7/200) stays near random (Figure 4; Table 12, Appendix). Four-family deletion loss aligns in rank () with window tail lift despite operating outside the local linear regime. Removing web text raises the arithmetic share from 50.0% (random control) to 57.1%, confounding its +15.0-problem gain.
| Removed | Held-out (/200) Three seeds | Mean | Vs. random | Helpful tail lift | |
| Worked examples | 0, 0, 0 | 0.0 | 2.64 | ||
| Primitive facts | 119, 126, 133 | 126.0 | 0.015 | 2.14 | |
| Verbal procedures | 142, 143, 146 | 143.7 | 0.318 | 1.21 | |
| Code traces | 137, 139, 141 | 139.0 | 0.076 | 1.39 | |
| Web text | 161, 163, 165 | 163.0 | 0.017 | 0.16 | |
| Random control | 143, 146, 155 | 148.0 | — | — | — |
Table 12. Multi-epoch family deletion at 152M. Eighteen continuations branch from the checkpoint where accuracy first appears; each removes 3,200 of 25,600 blocks and trains the remaining 22,400 blocks for eight epochs from the same initialization, matching step count and starting state. For reference, the rightmost column lists helpful-tail lift in the corresponding 100-step window (harmful-tail lifts are similarly elevated; Table 10). Four-family deletion loss aligns in rank with window tail lift (), though eight-epoch full-family removal measures macro-scale net benefit outside the local linear regime. Web removal raises the arithmetic share to 57.1%, confounding its +15.0-problem gain. Bold, blue-tinted -values mark contrasts significant at , from two-sided pooled two-sample -tests against the random control ().
Two-phase pretraining and checkpoint conditioning
Pretraining without arithmetic alters subsequent acquisition speed: across a matched 800-step math phase at 86M, codemath reaches 130/200 held-out exact match versus 26/200 for webmath. Attribution windows within both math phases validate at LDS 0.995 and 0.996; window tails emphasize worked examples while math-phase deletion relies on worked examples and primitive facts, and deleting code traces costs no more than random removal (Appendix § 9.7). Pretraining thus alters readiness to learn, whereas math-phase attribution captures local sensitivity within that phase.
Discussion
Window attribution measures how strongly individual blocks perturb held-out loss along a single trajectory, whereas multi-epoch deletion tests macro-scale reliance when an entire family is withheld. Worked examples and primitive facts populate both the most helpful and most harmful 5% tails while keeping near-zero mean scores, reflecting wide block-to-block variance rather than uniformly positive updates. Because worked examples match the fixed evaluation format, their sensitivity captures both arithmetic composition and prompt formatting.
Limitations. Most window configurations use one seed, restricting cross-run claims to the family level: per-block rankings are trajectory-specific and do not transfer. In the 86M grid, batch size and step count co-vary, and web deletion raises the arithmetic share. We make no cross-scale capability claims: two-phase 152M runs saw fewer math tokens than 86M runs, and 152M web+math uses a single parent seed under a different window configuration. Arithmetic accuracy holds only within the trained numeric range, and TrackStar and SOURCE exceeded our compute budget (Appendix § 9.7).
Conclusion
Short-window MAGIC attribution predicts retraining upon removing data during capability acquisition, but full-run unrolling overflows and clipped full-run attribution fails validation. Within-run rankings are trajectory-specific and fail to transfer, but the family-level tail patterns replicate across runs. These patterns represent local bidirectional sensitivity, which is different from the multi-epoch net impact. Thus, trajectory attribution is most useful when used within a validated window and interpreted as a family-level effect.
Acknowledgments and Disclosure of Funding
This research was supported in part through research cyberinfrastructure resources and services provided by the Partnership for an Advanced Computing Environment (PACE) at the Georgia Institute of Technology, Atlanta, Georgia, USA. RRID:SCR_027619. We also thank Modal Labs for additional compute support and infrastructure that accelerated our retraining validation experiments, and Algoverse AI Research for research mentorship, program support, and compute resources. Finally, we thank Lucia Quirke, Louis Jaburi, Gonçalo Paulo, Stella Biderman, and the EleutherAI team for open-source trajectory-attribution tooling in bergson and helpful technical discussions. The authors declare no competing financial interests.
Author Contributions
Jason Lee, Dikshanta Pokhrel, Phonn Nay Khant, and Victor Lee contributed equally as joint first authors; Glenn Matlin served as senior and corresponding author. The matrix below records each author’s roles under the NISO CRediT (Contributor Roles Taxonomy) standard.
| Conceptualization | Data curation | Formal analysis | Funding acquisition | Investigation | Methodology | Project administration | Resources | Software | Supervision | Validation | Visualization | Writing – original draft | Writing – review & editing | |
| Jason Lee | ● | ● | · | · | ● | ● | · | ● | ● | · | ● | ● | ● | ● |
| Dikshanta Pokhrel | ● | · | ● | · | ● | ● | · | · | ● | · | ● | · | ● | ● |
| Phonn Nay Khant | · | ● | · | · | ● | ● | · | · | ● | · | ● | ● | ● | ● |
| Victor Lee | · | · | · | · | ● | ● | · | · | ● | · | ● | ● | ● | ● |
| Moritz Weckbecker | · | · | · | · | · | ● | · | · | · | ● | · | · | · | · |
| Sebastian Lapuschkin | · | · | · | · | · | · | · | · | · | ● | · | · | · | · |
| Wojciech Samek | · | · | · | · | · | · | · | · | · | ● | · | · | · | · |
| Mark Riedl | · | · | · | · | · | · | · | · | · | ● | · | · | · | · |
| Glenn Matlin | ● | · | · | ● | ● | ● | ● | ● | ● | ● | · | ● | ● | ● |
Table 14.
Additional Details
Corpus specification
A program is a starting value with an ordered list of operations, where each operation is an operator paired with an operand. The example in Figure 2 starts at 17, multiplies by 2, and then adds 6. Table 1 gives the allowed ranges and the five compositions, or two-step operator orders, used in training. If any intermediate or final value falls outside the allowed range, we discard the program and draw a new one. There are 1,000 start values, 99 addition or subtraction operands, and 11 multipliers. Each of the three compositions that include a multiplication therefore allows programs, and each of the two that use only addition and subtraction allows , for in all. Discarding programs whose value drops below zero leaves . Our main 152M run raises the largest start value to 2,999 (Appendix § 9.§ 4).
| Parameter | Range | Composition | Operator order |
| Start value | [0, 999] | mul_add | MUL, ADD |
| ADD/SUB operand | [1, 99] | add_sub | ADD, SUB |
| Multiplier | [2, 12] | sub_add | SUB, ADD |
| Value floor | 0 | sub_mul | SUB, MUL |
| Value ceiling | 99,999 | mul_sub | MUL, SUB |
| Distinct programs | add_mul | Held out |
Table 1. Program space of the 86M specification.
Table 1: Program space of the 86M specification. A program whose intermediate or final value leaves [0, 99,999] is rejected and resampled. Under these ranges no program reaches the ceiling, so only the floor rejects. ADD followed by MULTIPLY is withheld for evaluation.
Families. We render programs in four families at equal token budgets (Figure 2). Worked examples, verbal procedures, and code traces write out a whole program step by step, so each operation acts on the result of the one before and every intermediate value appears. Primitive facts take each operation from a program and apply it to a newly drawn starting value instead of the previous result: for the example program, and , where a worked example has and . The model sees the same single operations but never sees one result feed into the next. Within a seed, each program appears in only one family, and each family has a balanced mix of the five compositions. Worked examples are the only family written in the evaluation format, which states the problem and then a WORK marker. At evaluation, the model must generate every number after that marker itself.
Token budgets and reuse. The families render a program at different lengths (Table 2), so equal problem counts would give them unequal shares of the corpus. We equalize budgets in tokens. Reusing programs raises held-out loss and makes scores larger and less stable (Table 3), and one distinct-program count for all families would leave reuse confounded with family: an earlier build that did this ranged from to . We therefore set each family’s distinct-program count to its token budget divided by the product of the target reuse and its mean record length. The 1B-token corpus has a common reuse of , with 232,000 distinct programs for worked examples and 845,000 for primitive facts. The main 152M corpus targets a reuse of four (Appendix § 9.§ 4).
| Family | Mean tokens per record |
| Worked examples | 107.6 |
| Primitive facts | 29.6 |
| Verbal procedures | 62.9 |
| Code traces | 52.1 |
Table 2. Mean tokens per record for each family. These lengths set the distinct-program count each family needs to reach a common reuse at an equal token budget.
| Distinct programs | Reuse | Held-out loss |
| 1,500 | 2.47 | |
| 200 | 2.66 | |
| 50 | 2.76 |
Table 3. Reuse degrades capability at a matched token budget. Attribution scores also grew in magnitude and agreed less between runs as reuse rose.
Score scale. Family tail statistics rank raw block scores. Packed synthetic blocks carry slight trailing padding relative to full 512-token web blocks (e.g., 503 vs. 512 content tokens for primitive facts); normalizing scores per content token on the 86M runs leaves family lifts virtually unchanged.
Identifier audit and web screening. Program identifiers are truncated content hashes of the program signature: the start value, operator order, and operands. We audit every build by checking that all records sharing an identifier also share a signature. The 1B-token corpus has no collisions across 1,954,000 identifiers, and the 3B-token corpus has none across 7,076,856. Held-out problems come from the same generator and ranges as training, so we audit all 103,966 web documents (500 MB) in the 1B-token corpus against our record format, every held-out prompt, and every held-out answer fact: zero documents match any of the three. Arithmetic-free web-only control at both 86M and 152M (1.0B web tokens) likewise solve 0/200 held-out items (Table 5). The 200 held-out problems are frozen across every model we report.
Table 5: Model index. Every model named in the paper, with the subfolder that backs it in leejason2025/mathamagical-checkpoints on Hugging Face. The run name used in our code and logs is the middle segment of each subfolder. Held-out is free-running exact match on the 200 ADD→MULTIPLY problems.
| Paper name | HF subfolder | Tokens | Held-out |
| 152M web+math | 150m-mix150m-3b/step_91552 | 3.0B | 200/200 |
| 86M web+math, 1B | 86m-cap2-s0/step_30517 | 1.0B | 189/200 |
| 86M web+math, 1B, Muon | 86m-cap2-muon-s0/step_30517 | 1.0B | 194/200 |
| 86M web+math, 300M | 86m-cap2-300m-clip-s0/step_9155 | 300M | 177/200 |
| 86M webmath | 86m-CUR-nl-mathphase/step_800 | 300M+800 | 26/200 |
| 86M codemath | 86m-CUR-code-mathphase/step_800 | 300M+800 | 130/200 |
| 152M webmath, late branch | 150m-P150s-b30517-m1.0/step_800 | 1.0B+800 | 141/200 |
| 152M webmath, early branch | 150m-P150s-b15312-m1.0/step_800 | 0.5B+800 | 129/200 |
| Early branch, 50% math | 150m-P150s-b15312-m0.5/step_1600 | 0.5B+1,600 | 136/200 |
| 86M web-only | 86m-nl86m-300m/step_9155 | 300M | 0/200 |
| 86M code-only | 86m-code86m-300m/step_9155 | 300M | 0/200 |
| 152M web-only | 150m-web150m-1b/step_30517 | 1.0B | 0/200 |
Table 5. Model index. Every model named in the paper, with the subfolder that backs it in leejason2025/mathamagical-checkpoints on Hugging Face. The run name used in our code and logs is the middle segment of each subfolder. Held-out is free-running exact match on the 200 ADD→MULTIPLY problems.
Builds. Builds are deterministic given a seed. Each build writes a manifest with token counts, block counts, realized reuse, distinct-program counts, the identifier audit, and a hash over the packed corpus. The generator, packer, tokenizer, and mixture manifest are released with each corpus.
Interface and format sensitivity
Held-out exact match evaluates one held-out composition in one fixed prompt format. Two broader evaluations test how far the trained models generalize across prompt interfaces and operand magnitudes across all fifteen models of Appendix § 9.§ 6.
Interface benchmark. We pose 2,000 problems from EleutherAI’s asdiv and arithmetic suites [10, 26, 3] in seven prompt formats; Figure 6 reports the 783 problems inside the 86M training specification. Arithmetic-trained models reach up to 99% in infix or verbal form and at most 2% in English on the identical underlying problems, showing that they compute reliably within trained syntax without comprehending natural-language problem statements. The arithmetic-free and public controls stay below 2% in every format, and every arithmetic-trained model scores 0% in canonical form (answer only, without intermediate steps) because generation reverts to the worked-solution template seen during training.
Figure 6. Exact match by model and prompt format on the 783 benchmark problems inside the 86M training specification (788 for infix). Columns are nine arithmetic-trained models, three arithmetic-free controls, and three public models (GPT-2, Pythia-160M, GPT-Neo-125M).
Discrete exact match vs. continuous completion loss. We select acquisition stages by discrete exact match rather than continuous completion loss because completion loss already approaches a digit-guessing floor while exact match remains 0/200 [37]. MAGIC instead differentiates continuous completion loss, which explains why pre-emergence windows at 0/200 exact match still carry informative gradients and validate at LDS 0.61–0.991 (§ 4.2). Because accuracy swings from 99% to near 0% across prompt interfaces above, the main attribution experiments hold one evaluation format fixed across stages and seeds so that measured sensitivities reflect arithmetic learning rather than prompt mismatch (§ 5).
Number range. A second evaluation sweeps single-operation infix problems across operand size (Figure 7). The arithmetic-trained models stop at the edge of their training range. None exceeds 0.5% on addition or subtraction with three-digit operands, and none exceeds 4% at a multiplier of 16. On four-digit multiplicands, all but one stay at or below 4%. The exception is 152M web+math at 47.5% (95/200) before dropping to 0% at five digits: its specification raises the maximum start value from 999 to 2,999, placing 49 of the 200 (24.5%) four-digit items inside its training range and indicating partial generalization across the remainder of the four-digit bucket. The two-phase 152M models keep the 86M specification and stop at the same limits as the 86M models.
Figure 7. Exact match on generated single-operation infix problems by operand size (left) and multiplier (right), for the strongest 86M model (86M web+math, 1B, Muon). Accuracy holds up to the trained limits of two-digit addition and subtraction operands, three-digit multiplicands, and multipliers of twelve, and drops to near zero beyond them.
Attribution and validation detail
Sign check. For each removed subset, the validation harness records baseline held-out loss minus the loss after removal, and correlates that difference with the subset’s summed scores. On the four main windows, the correlation is positive for all 32 query-window pairs, with a median of 0.99 in three windows and 0.86 in the window where accuracy first appears (Figure 8). A positive score therefore marks a block whose removal lowers held-out loss.
Figure 8. Predicted against measured effect for every leave-1%-out retrain of the four 152M web+math windows (48 subsets 8 queries each, standardized per query). Each panel prints the median per-query LDS.
The family analyses read scores from disk, so we also regenerated the 48 subsets of each main window from their seed and summed the stored scores. The sums match the harness’s records on 384 of 384 rows in every window.
Effect sizes. We measure the size of a window’s retraining effects by the standard deviation of the held-out loss change across its 48 removals (Table 4). This standard deviation falls by a factor of 44 from the first 152M window to the last, which could suggest that training data stops mattering once the skill is present. Baseline held-out loss falls by a factor of about 600 over the same windows, so relative to the remaining loss the effect rises from 0.08% to about 1.1% by the third window and then plateaus. The smallest standard deviation we report, , is above the fp32 evaluation precision of about .
Artifact availability. The window checkpoints, per-block score tensors, 48 × 8 retraining grids, deletion indices, and the base run’s evaluation curve are released as a results dump alongside the checkpoints and corpora (Appendix § 9.§ 6).
Single-phase run protocol
Base run. 152M web+math is a 152M-parameter decoder with context 512 and a 276-token byte-and-digit tokenizer, trained for one epoch over 3.0B tokens at a 50/50 mixture of web text and synthetic arithmetic with a target reuse of four. Optimization uses AdamW [24] at learning rate with linear decay, weight decay 0.01, 20 warmup steps, batch 64, and 91,553 optimizer steps. The program specification raises the maximum start value to 2,999, which takes the rarest composition to 3.21M programs and the space to approximately .
Window protocol. All four windows use an identical protocol: 25,600 blocks for two epochs at batch 512 with gradient accumulation of eight, giving 100 optimizer steps; a math fraction of 0.5; damping of ; AdamW at a constant learning rate of with five warmup steps; full single precision; evaluation mode; eight held-out queries; and 48 leave-one-percent-out retrains. Continuation blocks are drawn from the portion of the corpus the base run had not reached in its own shuffle order.
Checkpoint resume. The base run reached its platform wall-clock limit at step 88,788 (97.0%), resumed deterministically from the stored parameters, optimizer state, batch index, and random-number state, and completed the remaining 2,765 steps. Checkpoint restoration itself is constant-time because training restarts directly at the saved batch index without replaying prior data; however, each container launch rebuilds the tokenized dataset on startup ( hours even with a persistent volume cache), so finishing the final 3% required 2.35 hours of startup preparation for 0.7 hours of training.
Model and window index
Models are named by parameter count and training diet. web+math is a model trained once, from scratch, on the mixed corpus; web→math and code→math are pretrained without arithmetic and then given a math phase; web-only and code-only never see arithmetic and are the floor controls. A qualifier appears only where one is needed to separate two runs, so a math phase is 100% math unless stated otherwise, and early and late branch are the halfway and final checkpoints of the 152M web-only pretrain. All models in the 152M family have 152M parameters; we never write 150M.
Windows are named model · stage, where the four stages are fixed by held-out accuracy at the window’s initialization: before the skill at 0/200, skill appearing at the first non-zero score, skill forming on the rise, and skill present near ceiling.
The validity-envelope sweeps of § 4.2 are all variants of the 86M web+math, 300M · skill appearing window, and carry the run IDs A1-len200, A1-len400 and A1-len800 for horizon, A3-bs64, A3-bs128 and A3-bs512 for batch, A4-muon for the optimizer, A5-eps0 and A5-eps1e6 for damping, and A7-indist for the in-distribution target. Deletion arms are named C1v2-L-drop-family-sn over three seeds.
Full per-window configuration and results for all 44 windows are in the released results dump.
Evaluation protocol
Models. The interface and range evaluations cover fifteen models: twelve of our own (nine arithmetic-trained models across 86M and 152M, including the two curriculum arms, and the three arithmetic-free controls 86M web-only, 86M code-only, and 152M web-only), together with three public models of comparable size: GPT-2 [34], Pythia-160M [1], and GPT-Neo-125M [2]. One further checkpoint is omitted deliberately: it is a bit-identical forward twin of another model in the set.
Interface benchmark. The same 2,000 arithmetic problems are presented in seven formats: English zero-shot and four-shot, numeric infix, verbal, canonical, scratchpad, and the specification-then-work format the models train on. Decoding is greedy, and scoring is exact match. Results are reported both over all items and restricted to items inside each model’s training specification, because the two differ substantially.
Number-size range evaluation. Single-operation problems are generated and swept across operand magnitude and multiplier size to locate where each model’s arithmetic fails. The sweep covers addition and subtraction from one to seven-digit operands, and multiplication across multiplicand digits and multipliers from two to ninety-nine.
Released artifacts. All public artifacts are hosted at three repositories:
Model checkpoints: https://huggingface.co/leejason2025/mathamagical-checkpoints
3.0B training corpus: https://huggingface.co/datasets/leejason2025/mathamagical-3b-data
Code and validation summaries: https://github.com/HCAI-Lab-GT/mathamagical
Reproducing the main 152M results uses the released base and four window checkpoints, the exact training corpus and frozen held-out set, the corpus manifest and hash, and the production generator, packer, and tokenizer that ship with the corpus.
Independent validation uses the released results dump: per-query correlations, the retraining grid for each main window, score tensors, subset identities, and run configurations. Family-deletion results ship as seed-level outputs with the matched random-deletion controls, and the base evaluation curve identifies the selected acquisition stages. Every chart in this paper is generated from those files.
Additional results and limitations
Baselines. We planned to compare against TrackStar and SOURCE [4, 33] on the same runs, targets, and queries. Neither produced scores within our compute budget: SOURCE exceeded a 24-hour cap while preconditioning its first segment, and TrackStar was mis-scoped and not rerun. Stronger checkpoint-based estimators therefore remain untested.
In their place we use single-checkpoint gradient-dot, which needs no unrolling. At each window’s starting checkpoint, it scores every block by the dot product between the block’s gradient and the held-out loss gradient. We validate these scores on the same retraining subsets as the MAGIC scores. Gradient-dot reaches LDS 0.08, −0.08, 0.10, and −0.04 on four 86M windows and 0.21 and 0.13 on two 152M windows, against 0.57–0.99 for MAGIC on those subsets (Figure 10). Per-block Spearman correlation between the two methods is 0.00–0.16. Gradient-dot also puts arithmetic in the helpful tail and web text in the harmful tail, which looks plausible, but its scores do not agree with retraining.
Full-run attribution. The bergson implementation of MAGIC carries the running metagradient backward through every optimizer step with no clipping, normalization, or damping, and its norm grows by about 0.4–0.5% per step (Figure 11, left). At 30,518 steps, 44.7% of block scores are non-finite and the largest finite magnitude is . We patched the reverse pass to clip the running metagradient and reran a bit-identical version of an affected run. Non-finite scores fell from 25.3% on the unpatched run to 0%, but the clipped scores do not predict retraining: aggregate LDS is 0.217 and 0.168 on two seeds (12 subsets; statistical -values 0.50 and 0.60), and per-query values vary by ±0.3 within a single run, consistent with horizon-induced Taylor drift in trajectory attribution [7].
The library also exposes a damping term, . No setting in our sweep both learned arithmetic and gave stable full-run scores (Table 7). Whether a different optimizer or reverse-pass treatment can do so remains open.
Windows. Across the 86M sweep, LDS spans −0.08 to 0.996; Table 8 gives the main settings. Batch size and horizon are confounded, because steps equal blocks times epochs divided by batch, and every 86M window at 512 rows per step is only 25 steps long. A 128-row window at 100 steps scores 0.854, above the 0.570 of a 256-row window at the same length, so we read steps and batch jointly and name neither as the main factor. The 86M run at 512 rows and 100 steps that would separate them was not run.
We scored the end-of-window checkpoints to see what changes inside a window. Held-out accuracy moves by a few problems out of 200 at both scales (at most 1 → 8), and not at all in the 25-step windows at 512 rows. The windows attribute a decrease in loss; the accuracy jump comes later.
Three windows on the two-phase 152M models validate at 0.990, 0.993, and 0.994, with every query above 0.985, and five more reach 0.94–0.99 on incomplete retraining banks. The configuration therefore works at a second model size. We make no cross-scale claim from this, since these models saw 26.2M arithmetic tokens against 500M for the 1B-token 86M models.
Table 9 gives validation for the four 152M web+math windows. The model reaches 200/200 by step 30,200 and holds that score through step 91,552. The 86M series behaves differently: LDS rises monotonically across its windows, and in its window before accuracy appears, two of eight queries fail validation. The two series also differ in batch size and continuation length, so we report this as a difference between two configurations and make no cross-scale claim.
Family tails. We compare families by tail lift: a family’s share of the most helpful or most harmful 5% of blocks in a window, divided by its share of the window, so that 1 is chance (Table 10). Tail lift measures sensitivity, not net benefit (§ 3): mean per-family scores are small and change sign between windows. Worked examples and primitive facts have the largest lifts, and those lifts are nearly equal in the two tails: 2.83 and 2.63 for worked examples and 2.13 and 2.15 for primitive facts in the first window. Blocks from these families move held-out loss strongly in both directions, which does not make either family helpful as a whole.
Web text is half the corpus by tokens and holds 0.16–0.24 of its chance share of either tail during acquisition. The 86M reference window shows the same depletion (0.30–0.41; Table 13) on a corpus where web text was a minority of tokens. A low tail share does not mean web text contributes nothing in total. By the last window the lifts move toward 1 and web text rises to 0.77–0.83, so strong family concentration belongs to the acquisition period.
| Family | Helpful | Harmful | Held-out | Vs. random |
| Worked examples | 2.41 | 1.85 | 0 | |
| Primitive facts | 2.01 | 2.07 | 36 | |
| Verbal procedures | 0.93 | 1.21 | 47 | |
| Code traces | 1.46 | 1.22 | 43 | |
| Web text | 0.30 | 0.41 | 52 |
Table 13. Window tail sensitivity and family deletion at 86M. Lift is each family’s share of the most helpful (most negative 5%) and most harmful (most positive 5%) validated block scores in a reference 86M window, averaged over the six of eight queries with and divided by corpus share (§ 3); high-sensitivity families populate both tails. Deletion reports held-out exact match (/200, three seeds) after retraining the math phase without that family against step-matched random removal (base 64, random 45). Only worked examples reach significance at (; primitive facts ).
Four windows that share identical blocks and differ only in batch size, optimizer, or damping agree on block rankings at Spearman −0.01 to +0.091, and one window’s eight queries agree with each other at +0.075 to +0.418 (Table 11). Rankings from different 86M seeds agree at 0.00 to 0.08, and the top-5% blocks of two runs overlap at about the rate expected by chance. MAGIC scores describe one realized training run (Appendix § 9.8), so changing a hyperparameter changes what is being estimated; Nguyen et al. [ [28] describe the same cross-run variance in per-example scores more generally. Family tail shares hold across these changes, and block rankings do not.
Family deletion. Tail lift covers 100 steps of one run. To measure a family’s net benefit, we delete it and retrain (Table 12). Eighteen continuations branch from the checkpoint where accuracy first appears: one arm per family plus a random-deletion control, with three seeds each. Every arm removes 3,200 of 25,600 blocks and trains the remaining 22,400 for eight epochs from the same state. Deleting worked examples eliminates acquisition (0/200 against 148/200 for random deletion), and deleting primitive facts lowers accuracy to 126/200. Deleting code traces (139/200) or verbal procedures (143.7/200) stays closer to random. Three of five contrasts are significant at . Deletion cost ranks the families in the same order as helpful-tail lift (), although eight epochs of full-family removal lie far outside the 1%, 100-step additive first-order regime that the scores approximate [[14, 12]]. Worked examples also share the evaluation format (§ 4.5); primitive facts share neither the format nor chained operations and still rank second on both measures.
A fixed block count does not remove the same share of each family. Removing 3,200 blocks deletes all of an arithmetic family (12.5% of the corpus) but only a quarter of web text (50%). The four arithmetic arms train at 42.9% arithmetic and are comparable with each other. The web arm trains at 57.1%, so its 15.0-problem gain over random deletion cannot be attributed to web removal alone.
The 86M reference window gives the same direction with less power (Table 13). Retraining the 86M math phase without worked examples costs 45.3 problems against matched random removal (), removing primitive facts costs 9.3 (), and removing verbal or web blocks costs nothing. Three seeds support the ordering and no stronger claim.
Curriculum arms. 86M webmath and 86M codemath differ only in pretraining corpus; their 800-step math phase is identical. After it, codemath scores 130/200 against 26/200 for webmath (Table 5). Because a window has fixed length, attribution cost does not grow with the length of pretraining. The two arms’ math-phase windows validate at LDS 0.995 and 0.996 (Table 6), with worked examples enriched in both 5% tails (2.3–2.6× helpful; 2.0–2.6× harmful). In 600-step family-deletion continuations from each arm’s pre-transition checkpoint (two seeds per arm; random-removal means 129.5/200 on webmath and 160.5/200 on codemath), removing worked examples reduces accuracy to 0/200 in both arms and removing primitive facts lowers it to 113.0/200 () and 153.5/200 (), whereas removing code traces (135.0/200 and 160.0/200) or verbal procedures (140.5/200 and 170.5/200) costs no more than random removal.
Limitations. The results concern one synthetic arithmetic specification, one withheld operation order, one evaluation format, and one background web corpus. Contamination audits rule out direct matches to our record formats or held-out items without eliminating general arithmetic in web text; the arithmetic-free controls score at most 2% in any prompt format (Figure 6).
On the fully trained 86M models, web text rises and worked examples fall in the helpful tail (Figure 9). Every math block in those windows had already been seen in training while the web blocks were new, so we read this as a data-freshness artifact and not as a change in the family ordering.
Figure 9. Helpful-tail lift per family for every window with a full family table in the released results (24 windows across three model groups). The ranking of Figure 4 replicates across configurations and scales. The 86M windows where web text rises and worked examples fall are on fully trained models, where every math block had been seen and web blocks were fresh; that data-freshness artifact is discussed with the limitations.
MAGIC and influence functions
This appendix states the parts of Ilyas and Engstrom [2025] that our claims depend on and applies them to our target, source unit, and sign convention (§ 3). Ilyas and Engstrom [2025] give the full development and the exactness argument.
The influence function. Represent a dataset as a weight vector over a pool , where multiplies the loss of example during training and excludes it. Let map a weighting to trained parameters, let measure a trained model, and let . Evaluating requires retraining, so attribution replaces it with an explicit , the first-order expansion about the full dataset :
The leading term is the influence function [18, 14]. In deep learning, it has no closed form, because the derivative runs through training itself, so scalable methods approximate it (at § 2). MAGIC computes it exactly.
Single-model attribution. Training is stochastic, so conventional attribution predicts what a freshly initialized model would do on average without an example, and some of the variation in does not come from the data. Fixing initialization and data order removes that variation: becomes deterministic, and model output varies with the data weights alone [14]. Our retraining checks resume from a stored optimizer state in the same data order, so they measure the quantity the scores predict. For the same reason, a score describes only the run it was computed on, and block rankings need not carry over to another seed (Section 4.5).
Iterative and smooth training. REPLAY computes the exact metagradient when training is iterative and smooth [14]. Iterative means
where is the optimizer state at step , the update rule, the minibatch, the loss of example at that state, and the number of steps. In a window, is the exported checkpoint and is the window length. Smooth means that for any measurement function , a small change in makes only a small change in ; Engstrom et al. [ [9] call this metasmoothness.
For AdamW [24], holds the parameters and the first and second moment estimates. The update is not smooth as usually written, because the derivative of grows without bound as approaches zero. The smooth version replaces that root with for a constant ; the update we differentiate is the one implemented in bergson [33]. This constant is the damping term of Table 8 and Table 7. At 86M, gives non-finite scores and gives LDS 0.96, while at or the model no longer learns arithmetic. Smoothness comes from the training configuration, which is the formal basis for the finding in § 4.2 that attribution validity depends on the training recipe.
Exact computation. The gradient of with respect to is a metagradient, so REPLAY [9] computes it by walking the realized training path backward. It initializes ; then for it reloads and and computes
where is the part of the derivative that arises at step , and is the sensitivity of the target to the optimizer state at that step. The exact influence function is . REPLAY keeps a small set of states and recomputes the others, which costs training steps and memory [14]. Because a score sums over every step of the differentiated window, it is defined only for a named window. And is a single running quantity carried through all steps without normalization, which is why it overflows over a full run (Section 4.1).
Validation: the linear data modeling score. Because predicts a retraining outcome, retraining can check it. Sample binary weight vectors , each obtained by dropping blocks at random for a drop fraction . For each vector, obtain the true by retraining and the predicted from (1), then report the Spearman correlation between the two:
Subsets are indexed by superscript, as in , and the weight on block by subscript, as in , so counts retrains and counts blocks. The four main 152M windows use and subsets, computed per held-out query and aggregated across eight queries. Full-run diagnostics use 12 subsets; the statistical -values we report for them are separate from the removal fraction . Table 6 lists the other window configurations. Ilyas and Engstrom [ [14] report that LDS degrades as grows, because the linear estimate holds only near . Our failures come instead from the horizon at fixed .
How this study applies the framework. The pool is the set of 512-token blocks the differentiated window consumes, so indexes block occurrences, and counts blocks in the window rather than in the corpus. The measurement is mean completion-only negative log-likelihood on the held-out programs, evaluated at the final state of the window. For a window, continues from an exported checkpoint . The four main 152M windows use AdamW steps in fp32 at 512 rows per step, on blocks the parent run had not yet reached.
The sign convention of § 3 follows from (1). The on-disk score of block is , the -th component of the that REPLAY returns, that is at . Removing a subset sets for , so on and zero elsewhere, and
A positive score therefore predicts that removing the block lowers held-out loss: positive means harmful, and the most helpful blocks carry the most negative scores. Equivalently, predicts baseline held-out loss minus the loss after removing , which our validation harness correlates with measured retraining at a median of 0.86–0.99 per window (Appendix § 9.§ 3).
References
- [1]Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Af lah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. A suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430, 2023. URL https://proceedings.mlr.press/v202/biderman23a.html.arxiv.org/abs/2304.01373
- [2]Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large scale autoregressive language modeling with Mesh-Transformers. Zenodo, 2021. URL https://doi.org/10.5281/zenodo.5297715.
- [3]Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f6a4-Abstract.html.arxiv.org/abs/2005.14165
- [4]Tyler A. Chang, Dheeraj Rajagopal, Tolga Bolukbasi, Lucas Dixon, and Ian Tenney. Scalable influence and fact tracing for large language model pretraining. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=gLa96F1Wm.arxiv.org/abs/2410.17413
- [5]Jianhui Chen, Yuzhang Luo, and Liangming Pan. Mechanistic data attribution: Tracing the training origins of interpretable LLM units. arXiv preprint arXiv:2601.21996, 2026. URL https://arxiv.org/abs/2601.21996.
- [6]Junwei Deng, Ting-Wei Li, Shiyuan Zhang, Shixuan Liu, Yijun Pan, Hao Huang, Xinhe Wang, Pingbang Hu, Xingjian Zhang, and Jiaqi W. Ma. dattri: A library for efficient data attribution. In Advances in Neural Information Processing Systems, volume 37, pages 136763–136781, 2024. URL https://arxiv.org/abs/2410.04555.
- [7]Junwei Deng, Pingbang Hu, Suliang Jin, Hao Lu, Jiachen T. Wang, Shichang Zhang, and Jiaqi W. Ma. How faithful is trajectory-based data attribution? Error sources, remedies, and practical guidelines. arXiv preprint arXiv:2605.18814, 2026. URL https://arxiv.org/abs/2605.18814.
- [8]Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchoui, and Yejin Choi. Faith and fate: Limits of Transformers on compositionality. In Advances in Neural Information Processing Systems, volume 36, pages 70293–70332, 2023. URL https://arxiv.org/abs/2305.18654.
- [9]Logan Engstrom, Andrew Ilyas, Benjamin Chen, Axel Feldmann, William Moses, and Aleksander Madry. Optimizing ML training with metagradient descent. arXiv preprint arXiv:2503.13751, 2025. URL https://arxiv.org/abs/2503.13751.
- [10]Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation. Zenodo, 2021. URL https://doi.org/10.5281/zenodo.5371629.
- [11]Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296, 2023. URL https://arxiv.org/abs/2308.03296.
- [12]Jaeseung Heo, Kyeongheung Yun, Youngbin Choi, Sehyun Hwang, Jungseul Ok, and Dongwoo Kim. Interaction-aware influence functions for group attribution. arXiv preprint arXiv:2605.15675, 2026. URL https://arxiv.org/abs/2605.15675.
- [13]Prateek Humane, Paolo Cudrano, Daniel Z. Kaplan, Matteo Matteucci, Supriyo Chakraborty, and Irina Rish. Influence functions for efficient data selection in reasoning. arXiv preprint arXiv:2510.06108, 2025. URL https://arxiv.org/abs/2510.06108.
- [14]Andrew Ilyas and Logan Engstrom. MAGIC: Near-optimal data attribution for deep learning. arXiv preprint arXiv:2504.16430, 2025. URL https://arxiv.org/abs/2504.16430.
- [15]Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Datamodels: Predicting predictions from training data. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 9525–9587, 2022. URL https://proceedings.mlr.press/v162/ilyas22a.html.arxiv.org/abs/2202.00622
- [16]Liangze Jiang, Zachary Shinnick, Anton van den Hengel, Hemanth Saratchandran, and Damien Teney. Procedural pretraining: Warming up language models with abstract data. arXiv preprint arXiv:2601.21725, 2026. URL https://arxiv.org/abs/2601.21725.
- [17]Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. Blog post, 2024. URL https://kellerjordan.github.io/posts/muon/.
- [18]Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1885–1894, 2017. URL https://proceedings.mlr.press/v70/koh17a.html.arxiv.org/abs/1703.04730
- [19]Siqi Kou, Qingyuan Tian, Hanwen Xu, Zihao Zeng, and Zhijie Deng. Which data attributes stimulate math and code reasoning? An investigation via influence functions. In Advances in Neural Information Processing Systems, volume 38, 2025. URL https://arxiv.org/abs/2505.19949.
- [20]Jin Hwa Lee, Matthew Smith, Maxwell Adam, and Jesse Hoogland. Influence dynamics and stagewise data attribution. arXiv preprint arXiv:2510.12071, 2025. URL https://arxiv.org/abs/2510.12071.
- [21]Nayoung Lee, Kartik Sreenivasan, Jason D. Lee, Kangwook Lee, and Dimitris Papailiopoulos. Teaching arithmetic to small Transformers. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=1qT3yK0q5J.arxiv.org/abs/2307.03381
- [22]Jiacheng Liu, Taylor Blanton, Yanai Elazar, Sewon Min, Yen-Sung Chen, Arnavi Chheda-Kothary, Huy Tran, Byron Bischoff, Eric Marsh, Michael Schmitz, Cassidy Trier, Aaron Sarnat, Jenna James, Jon Borchardt, Bailey Kuehl, Evie Cheng, Karen Farley, Sruthi Sreeram, Taira Anderson, David Albright, Carissa Schoenick, Luca Soldaini, Dirk Groeneveld, Rock Yuren Pang, Pang Wei Koh, Noah A. Smith, Sophie Lebrecht, Yejin Choi, Hannae Hajishirzi, Ali Farhadi, and Jesse Dodge. OLMoTrace: Tracing language model outputs back to trillions of training tokens. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 178–188, 2025a. doi: 10.18653/v1/2025.acl-demo.18. URL https://aclanthology.org/2025.acl-demo.18/.DOI
- [23]Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yarun Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is scalable for LLM training. arXiv preprint arXiv:2502.16982, 2025b. URL https://arxiv.org/abs/2502.16982.
- [24]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In The Seventh International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.
- [25]Andhika Bernard Lumbantobing and Hokky Situngkir. Arithmetic pedagogy for language models. arXiv preprint arXiv:2606.05106, 2026. URL https://arxiv.org/abs/2606.05106.
- [26]Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing English math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975–984, 2020. doi: 10.18653/v1/2020.acl-main.92. URL https://aclanthology.org/2020.acl-main.92/.DOI
- [27]Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=9XFSbDPmdW.arxiv.org/abs/2301.05217
- [28]Elisa Nguyen, Minjoon Seo, and Seong Joon Oh. A Bayesian approach to analysing training data attribution in deep learning. In Advances in Neural Information Processing Systems, volume 36, pages 64155–64180, 2023. URL https://openreview.net/forum?id=P7dS3H0p2u.arxiv.org/abs/2305.19765
- [29]Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. Investigating the limitations of Transformers with simple arithmetic tasks. arXiv preprint arXiv:2102.13019, 2021. URL https://arxiv.org/abs/2102.13019.
- [30]Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. TRAK: Attributing model behavior at scale. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 27074–27113, 2023. URL https://proceedings.mlr.press/v202/park23c.html.arxiv.org/abs/2303.14186
- [31]Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. The FineWeb dataset: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems, volume 37, pages 30811–30849, 2024. URL https://arxiv.org/abs/2406.17557.
- [32]Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In Advances in Neural Information Processing Systems, volume 33, pages 19920–19930, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/e6385d39ec9394f2f3a354d9d2b88eec-Abstract.html.arxiv.org/abs/2002.08484
- [33]Lucia Quirke, Louis Jaburi, David Johnston, William Z. Li, Gonçalo Paulo, Guillaume Martres, Girish Gupta, Nora Belrose, and Stella Biderman. Bergson: An open source library for data attribution. arXiv preprint arXiv:2606.11660, 2026. URL https://arxiv.org/abs/2606.11660.
- [34]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019.
- [35]Neil Rathi and Alec Radford. Shaping capabilities with token-level data filtering. arXiv preprint arXiv:2601.21571, 2026. URL https://arxiv.org/abs/2601.21571.
- [36]Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwarak Talupuru, Acyr Locatelli, Robert Kirk, Tim Rocktäschel, Edward Grefenstette, and Max Bartolo. Procedural knowledge in pretraining drives reasoning in large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=1hQKHHUsMx.arxiv.org/abs/2411.12580
- [37]Rylan Schaeffer, Brando Miranda, and Sammi Koyejo. Are emergent abilities of large language models a mirage? In Advances in Neural Information Processing Systems, volume 36, pages 55565–55581, 2023. URL https://arxiv.org/abs/2304.15004.
- [38]Loris Schoenegger, Lukas Thoma, Terra Blevins, and Benjamin Roth. Influence-driven curriculum learning for pre-training on limited data. In Proceedings of the First BabyLM Workshop, pages 356–379, 2025. doi: 10.18653/v1/2025.babylm-main.26. URL https://aclanthology.org/2025.babylm-main.26/.DOI
- [39]Alan Sun, Ethan Sun, and Warren Shepard. Algorithmic phase transitions in language models: A mechanistic case study of arithmetic. arXiv preprint arXiv:2412.07386, 2024. URL https://arxiv.org/abs/2412.07386.
- [40]Theodora Worledge, Judy Hanwen Shen, Nicole Meister, Caleb Winston, and Carlos Guestrin. Unifying corroborative and contributive attributions in large language models. In 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 665–683, 2024. doi: 10.1109/SaTML59370.2024.00039. URL https://arxiv.org/abs/2311.12233.
- [41]Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS: Selecting influential data for targeted instruction tuning. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 54104–54132, 2024. URL https://proceedings.mlr.press/v235/xia24a.html.arxiv.org/abs/2402.04333








