betaFeedback

WorksComputer Science

Data Attribution Should Steer Pretraining

Abstract

Pretraining data curation decides what a model learns from, yet what makes a document worth learning from depends on what the model already knows. Influence-driven selectors already rank documents to improve one run; we argue that training-data attribution should steer pretraining by identifying testable data properties that become reusable curation criteria, evaluated on new data against cheaper selectors at matched total cost, discovery included. A document ranking is consumed by the run it was computed for, whereas an explained data property can be tested across corpora and checkpoints, especially where familiar quality signals diverge from measured usefulness. Because historical contribution, an understandable explanation, and future data value require distinct evidence, we specify a feedback loop that interprets attributed sources, tests candidate data changes, and states for five claim types what evidence each requires and what none establishes. We close with a falsifiable comparison against perplexity, classifier, and mixing baselines at equal total compute.

Introduction

Pretraining data curation makes judgments about which text is worth learning from. Developers adjust mixture weights [14], filter subcorpora [Penedo et al., 2024], reorder curricula [Zhang et al., 2026], and compare curation strategies in controlled testbeds [Li et al., 2024]. Attribution adds a way to investigate how training instances contribute to a model’s behavior, under a specified account of influence [25] (Koh and Liang 2017; Park et al. 2023).

Language models evolve dynamically as optimization interacts with data mixtures, capacity, and objectives [7]. Data that helped form an ability may become redundant; other material becomes useful once prerequisites develop. Influence-driven curricula [Schoenegger et al., 2025] and model-aware selection and mixing [21] (Yu et al. 2024; Wang et al. 2026c) already act on attribution during pretraining, with gains and losses across tested policies. The broader opportunity is to learn which data properties support which capabilities under which training conditions, and hold those decisions to matching evidence.

Our position is that training-data attribution should steer pretraining by identifying testable data properties that become reusable curation criteria, evaluated on new data against cheaper selectors at matched total cost, discovery included. Influence rankings are consumed by the run they were computed for; a property of quality, content, or structure that explains a gain can be tested beyond that run and compiled into a selector that needs no fresh attribution. We call the study of these links capability provenance: connecting a target ability to the data and training conditions that shaped it [Matlin et al., 2026]. Prior provenance work traced abilities in trained models; prospective steering asks what evidence those links require before changing a training mix, and pairs developer-facing interpretability [20] (Nguyen et al. 2024) with actionable decision utility [Orgad et al., 2026]. We report no new experiments because this claim concerns what should count as evidence for a class of curation decisions, which no single run settles. § 6 sets out the matched-cost comparison the field should run in open testbeds, and § 7 states what would refute the position.

Historical Contribution and Future Data Value

Our scope is contributive attribution, how training data shapes behavior, rather than corroborative attribution of sources that support an output [Worledge et al., 2024]. Within contributive attribution, removing a document, reweighting it, and replacing a mixture ask distinct counterfactuals [4]: for a metric ff of trained parameters, removing zz asks f(θ−z)−f(θ)f(\theta_{-z}) - f(\theta) over training randomness, reweighting asks ∂f(θε)∂ε∣ε=0\left.\frac{\partial f(\theta_{\varepsilon})}{\partial\varepsilon}\right|_{\varepsilon= 0} for weight 1+ε1 + \varepsilon, and mixture replacement asks f(θw′)−f(θw)f(\theta_{w^{\prime}}) - f(\theta_w), so an estimate for one action does not license another.

Data that helped form a capability may add little once the model has learned from it. Conversely, data with little contribution so far may become useful after prerequisites develop [13, 17], and data introduced at another stage can have a different effect [Liu et al., 2025a]. These possibilities defeat a rule that the most influential past examples should receive the most future weight. Online selection for what is learnable and not yet learnt [Mindermann et al., 2022], influence models of evolving preferences [Yu et al., 2024], and optimal-control selection [26] act on this premise. The object of study is conditional data value Δf(a∣θt,w,Bt)\Delta_f(a \mid\theta_t, w, B_t): the change in metric ff from data change aa at checkpoint θt\theta_t, mixture ww, and remaining budget BtB_t.

Because conditional value shifts as prerequisites form, the training path carries signal a final checkpoint erases. Gradient tracing [Pruthi et al., 2020] and stagewise attribution [Lee et al., 2026] track this path, and task-agnostic analysis finds shifting contributions toward final parameters [Nishida et al., 2026], though parameter progress alone does not prove target capability. Since fixed-target attribution runs online at low overhead [Wang et al., 2025a] while influence shifts across stages [Wang et al., 2024], the question is which decisions trajectory estimates resolve that an endpoint cannot.

Content quality and contribution also come apart: influential text can teach a shortcut, while filtered text can supply useful structure. On English C4 and Pile web text for 125M–1.7B models, reference-corpus and off-the-shelf embedding similarity fail to beat random selection or datamodels [21] (Sam et al. 2025); on RedPajama-V2 and multi-recipe pools across 125M–7B models, static quality classifiers and proxy rankings diverge from reference loss and token utility [Saada et al., 2026, Lin et al., 2024, Wang et al., 2026a]; and in multi-epoch runs up to 7B–9B parameters on token-limited Common Crawl and C4 pools, unfiltered text overtakes filtered subsets once repetition exhausts the filtered pool [24] (Muennighoff et al. 2023; Mohri et al. 2026). The open question is which data properties explain these regime-dependent disagreements and whether attribution can surface them before a run.

Capability Provenance as a Control Loop

Observe, attribute, and interpret. In the loop of Figure 1, developers track capability and safety metrics across checkpoints and match the unit of attribution to the unit of intervention. Broad buckets can conceal opposing effects, and summed document scores do not equal a group effect [Guu et al., 2023, Yu et al., 2025, Heo et al., 2026]. A readable label only proposes a hypothesis, since discourse structure, source identity, and lexical overlap can drive the same score; when explanations remain entangled at the evidentiary check (§ 4), the loop takes the dashed revision path.

The attribution-to-steering control loop

Figure 1. The attribution-to-steering control loop. Evidence supports a test proportionate to the decision’s cost and uncertainty; outcomes inform later data policies and curation criteria, or send an unsupported interpretation back along the dashed path.

Test a data change and update criteria. A bounded trial, such as an end-of-training ablation [8], can test an uncertain attribution before a larger run. Every data change evaluates its target capability, adjacent capabilities, safety, and general language modeling as a joint outcome vector, where a gain on one metric cannot offset an unacceptable regression on another. Findings may also suggest a revised criterion or selector, as expensive judgments are distilled into cheap raters [Wettig et al., 2024, Yu et al., 2024] and tested on new data at matched cost.

Published example. Coalson et al. [ [16] trace toxicity in Pythia-160M trained on 1B OpenWebText tokens using EK-FAC influence on RealToxicityPrompts. Standard document-level attribution fails on intervention: dropping the top 10% of documents cuts toxicity by ≤10%, whereas dropping 50% cuts toxicity by 33% but degrades perplexity by 21% and LAMBADA by 13%. Following Figure 1’s dashed path separates three conflated drivers in those document scores: shared function words, isolated outlier tokens in benign text, and local toxic context. Subtracting non-toxic query gradients and penalizing dense 99th-percentile token clusters (2% of tokens in a one-token window) during pretraining cuts held-out explicit and implicit toxicity 4–10× up to 1B parameters within baseline fluency bounds, and token selections from the 160M proxy transfer across model scales on the same pretraining corpus.

A Ladder of Information for Data Decisions

Table 1 separates five evidentiary questions (historical contribution, explanatory property, out-of-sample prediction [Ilyas et al., 2022, Park et al., 2023], controlled intervention, and learned mechanism) alongside what each cannot establish. Because these rows answer distinct questions rather than forming a mandatory sequence, a selective intervention can precede mechanistic understanding, even though a successful action still leaves its motivating explanation unconfirmed.

Walking Coalson et al. [ [16] through Table 1 clarifies what that study licenses for curation: failed document deletion exposed a granularity mismatch in row 1, differential token-density scoring isolated a candidate property in row 2, and pretraining suppression plus layer probing supplied scoped evidence for rows 4–5. Because the cross-scale test reuses fixed token indices within the same OpenWebText slice, however, it leaves open whether the extracted property transfers to a held-out corpus or beats cheap token classifiers [Rathi and Radford, 2026] once EK-FAC discovery compute is charged. Across all five rows, use, identity, interpretation, robustness, and causal support qualify each claim (Appendix § 10).

QuestionRelevant evidenceWhat it does not establish
1. Which data contributed?Attribution estimates under a stated target and counterfactual.That influential data will be useful in a later training decision.
2. What might explain the contribution?Inspection of content, lexical and structural patterns, and competing explanations.That an understandable pattern faithfully explains the model’s dependence on the data.
3. Does the relationship predict unseen outcomes?Predictions evaluated on evidence distinct from that used to construct them.That a forecast of attribution scores predicts the effect of changing training data.
4. Does the candidate data change help?A comparison isolating the intervention across target and off-target effects.That the motivating explanation was right, or that the policy transfers.
5. How does the data shape computation?A separately validated account linking data, representations, and behavior.That this account identifies the best future curation policy.

Table 1. Different attribution claims require different evidence and license different conclusions.

Empirical Precedents and Measurement Limits

Fact tracing finds influential examples that support outputs via relation and entity priors without stating the queried fact [11]. Tracing studies contrast memorization-like support for knowledge with generalization-like support for reasoning [Wang et al., 2025b], capability-provenance work separates support profiles across social and technical probes via targeted unlearning [Matlin et al., 2026], and influence selects instruction data for targeted skills [Xia et al., 2024]. These studies surface unexpected dependencies while exposing two measurement limits.

On the target side, a benchmark score samples one capability facet rather than defining it, and near-chance accuracy limits attribution. On the intervention side, unlearning is an imperfect proxy because suppressed behavior can recover from independent facts [18] (Łucki et al. 2025) while inducing neighbor knowledge holes [Ko et al., 2025]; neither single-benchmark attribution nor an unlearning screen validates prospective upweighting.

A Research Agenda for Pretraining Decisions

Benchmark reusable criteria at matched total cost. Open testbeds such as DataComp-LM, DATE-LM, and DataDecide [Li et al., 2024, Jiao et al., 2025, Magnusson et al., 2025] can test whether attribution-derived criteria transfer to held-out corpora better than domain reweighting and regression mixing [Xie et al., 2023a, Liu et al., 2025c], importance resampling [Xie et al., 2023b], online mixing [2], model-aware and distilled selection [Yu et al., 2024, Shum et al., 2025], and direct curation [Jiang et al., 2025] under a named estimator. Total cost sums compute for indexing, scoring, ablations, and training plus capped inspection; charging discovery grants baselines the same budget for extra tokens or mixture sweeps on the held-out corpus (Appendix § 10, Table 2).

Protocol componentIllustrative specification
1. Corpora and splitDiscovery pool (Ddisc\mathcal{D}_{\mathrm{disc}}): 10B-token stratified slice of DCLM-Baseline or Dolma labeled with WebOrganizer 24 ×\times 24 topic-by-format bins [Li et al., 2024, Wettig et al., 2025]. Held-out transfer pool (Dhold\mathcal{D}_{\mathrm{hold}}): disjoint 20B-token DCLM-Pool slice untouched during discovery.
2. Model scale and checkpointsDiscovery run: 410M- or 1B-parameter decoder-only model trained for 10B tokens on Ddisc\mathcal{D}_{\mathrm{disc}}, saving 10 evenly spaced checkpoints with projected gradients (d=2048d = 2048–40964096) indexed via TRAK or EK-FAC [16] (Park et al. 2023). Transfer run: 1B-parameter model trained from scratch for 10B selected tokens from Dhold\mathcal{D}_{\mathrm{hold}} (plus a mid-training branch at 50% budget for stage-conditional tests).
3. Criterion extractionContrast top- and bottom-attributed regions or token clusters on Ddisc\mathcal{D}_{\mathrm{disc}} across competing hypotheses (discourse structure, lexical overlap, source metadata), screen candidate edits via bounded end-of-training ablations [8] or unlearning, and compile the surviving property into a standalone classifier or token filter [Wettig et al., 2024, Rathi and Radford, 2026] that runs on Dhold\mathcal{D}_{\mathrm{hold}} without gradients.
4. Baseline selectors(a) Small-model perplexity pruning and perplexity correlations [3] (Thrush et al. 2025); (b) reference-corpus classifier and quality raters [Penedo et al., 2024, Wettig et al., 2024]; (c) embedding-centroid similarity to discovery queries; and (d) domain reweighting or online mixing [2] (Xie et al. 2023a).
5. Matched total cost (CtotalC_{\mathrm{total}})Let Cdisc=Cindex+Cquery+Cscreen+CdistillC_{\mathrm{disc}} = C_{\mathrm{index}} + C_{\mathrm{query}} + C_{\mathrm{screen}} + C_{\mathrm{distill}} be all accelerator FLOPs (or GPU-hours) spent on Ddisc\mathcal{D}_{\mathrm{disc}} beyond the base run, paired with an explicit cap on human inspection (e.g., ≤8\leq8 analyst-hours, or 0 hours when extraction uses automated lexical and discourse contrasts). Baseline arms pay only their selection cost CselC_{\mathrm{sel}} on Dhold\mathcal{D}_{\mathrm{hold}} and invest the difference Cdisc−CselC_{\mathrm{disc}} - C_{\mathrm{sel}} into extra training tokens or mixture-weight sweeps on Dhold\mathcal{D}_{\mathrm{hold}} so every arm expends Ctotal=Cdisc+CtrainC_{\mathrm{total}} = C_{\mathrm{disc}} + C_{\mathrm{train}}.
6. Probe suite and guardrailsEvaluated under OLMES [26] with above-chance admission floors. Discovery queries: e.g., SocialIQA (social reasoning), GSM8K (math), or RealToxicityPrompts (safety). Disjoint held-out target probes: unseen tasks and phrasings testing the same construct (e.g., social pragmatics, MATH-elementary, ToxiGen). Off-target guardrails: MMLU macro-average, ARC-Challenge, HellaSwag, and PLaMo validation perplexity.
7. Seeds and decision ruleS≥3S \geq3 independent pretraining seeds per arm on Dhold\mathcal{D}_{\mathrm{hold}} with paired bootstrap confidence intervals over evaluation items. The attribution-derived criterion succeeds if it improves the held-out target suite over the best compute-matched baseline (p<0.05p < 0.05) while keeping every guardrail metric within a pre-declared non-inferiority bound (e.g., ≤0.5\leq0.5 percentage-point drop on off-target accuracy and ≤1%\leq1\% relative perplexity increase). Failing to beat the compute-matched baseline across two target constructs refutes the position at that scale.

Table 2. Illustrative single-testbed protocol instantiating the matched-cost comparison of Section 6.

Instrument outcomes across stages and held-out formats. § 5’s limits govern scoring: evaluations must hold out task formats disjoint from attribution queries under pre-declared adjacent-capability, safety, and perplexity bounds. Because unlearned behavior can recover from independent facts, unlearning only screens candidates, whereas retraining or continual-pretraining branches across checkpoints test how value shifts.

Separate failed explanations from estimator error. A failed held-out run may reflect either a false data-value hypothesis or an unfaithful influence approximation. Comparing optimizer-aware trajectory estimates [19] with small-model retraining isolates estimator error before discarding a property, and logging the hypothesis, checkpoint, estimator, and outcome in provenance records [23] keeps negative findings cumulative.

Alternative Views, Falsification, and Scope

Direct curation ablations and cheaper selectors such as perplexity pruning, perplexity correlations, or synthetic data probes may resolve the decision [3] (Thrush et al. 2025; Li et al. 2024; Wang et al. 2026b). Because current estimators carry query sensitivity and non-convex approximation error [6, 4], raw scores require separating controls (§ 4); attribution earns its cost when opposing effects cancel in coarse buckets or competing explanations prescribe different rules, complementing structural-alignment accounts on the training corpus [Lehalleur et al., 2025]. The position fails if criteria derived from attribution do not beat perplexity-based or classifier selectors on held-out corpora at equal total compute, discovery included.

Capability provenance is also dual-use: source maps that aid bias and safety audits can guide targeted poisoning or concentrate harmful skills. § 3 therefore pairs interventions with safety bounds, and empirical contribution never confers permission to train on contested data.

Conclusion

Training-data attribution should be evaluated by its utility for pretraining decisions, and its distinctive utility is explanation. The most influential examples in a model’s past need not be the ones to show it next; learning why is what attribution adds when its criteria win at matched cost.

Acknowledgments and Disclosure of Funding

We thank Bhavya Shah and Isaac Song for discussions, early drafting, and feedback on the manuscript, and we thank the ATTRIB 2026 reviewers and program chairs for constructive comments. We also thank the ML Alignment & Theory Scholars (MATS) Program, the Berkeley Existential Risk Initiative (BERI), and the Georgia Tech AI Safety Initiative for research and program support. The authors declare no competing financial interests.

What attribution evidence supports

The value of an attribution depends on the decision it informs. Identifying influential documents, explaining what they share, and choosing what to train on next are distinct questions, and evidence for one does not settle the others. This appendix develops the five qualifying considerations introduced alongside Table 1 in § 4 and maps prior systems onto its rows.

Interpreting a source is a scientific step. Structural similarity compares discourse organization; lexical overlap compares direct textual similarity. Either can suggest why attributed documents matter, but a topic, source identity, or repeated phrase may support competing explanations. Semantic coherence in an external representation does not establish that the language model whose provenance is being studied represents the same distinctions internally. A grouping should therefore help formulate and distinguish hypotheses, rather than settle the explanation by naming it.

§ 3 notes that a broad category can conceal opposing effects and that summed document scores do not give a group effect. Data outside a taxonomy can also be informative: text with an identifiable subject absent from the scheme reveals a missing category, whereas text too corrupted or incomplete to classify exposes a data-quality failure. Neither status alone establishes the text’s contribution to learning, because a curation rule acts on document properties while an attribution estimate answers a question about a specific model and target.

From explanation to a useful decision. An attribution score identifies a candidate source of support; interpreting its documents suggests a hypothesis about what that support consists of. A data intervention tests the resulting decision. Removal, replacement, and reweighting differ, so the replacement data and resource budget are part of the claim. Causal claims require comparisons that isolate the stated data change. A successful trial can justify a larger test without establishing that the policy transfers to every scale or training stage.

The decision may reweight a training mixture directly or induce a reusable criterion for selecting new data. The latter requires held-out corpus evaluation with discovery cost charged. Matched comparisons with cheaper selectors test whether attribution improves the decision, while separating controls distinguish the explanations behind that gain [Orgad et al., 2026]. When an intervention fails, structured records of which row in Table 1 broke keep future runs from repeating the mistake.

Robustness and generalization qualify the claim. Attribution reports must decouple influence signs, magnitudes, and relative rank orderings, as rank correlations can obscure sign reversals and near-zero effects. Sensitivity checks can vary queries, phrasing, checkpoint selection, and estimator assumptions. Disagreement can reveal approximation error or different counterfactual questions; change across checkpoints may also reflect learning. General policies need evidence across relevant runs, while a useful local claim may remain tied to one setting. A relationship that holds on related probes need not hold after a change in training stage, corpus, or model, so the intended scope determines the relevant transfer question.

Selective improvement is a joint outcome. A capability gain must be evaluated alongside changes in adjacent capabilities, safety, and general language modeling. A policy is selective only relative to pre-declared bounds on those off-target metrics; improving the target metric alone does not establish an acceptable tradeoff. Task accuracy, model calibration, and uncertainty about an intervention’s causal effect are distinct quantities: confident outputs do not imply that reweighting their support will help, and noisy influence estimates do not imply model miscalibration.

Existing systems and the evidence they supply. Several systems already instantiate parts of the loop, and each supplies evidence for a specific row of Table 1. Influence-driven mixture controllers [9] (Yu et al. 2024; Wang et al. 2026c), online mixing from per-domain scaling laws [Jiang et al., 2025], and optimal-control selection [26] report improvements at matched compute in the scales and mixtures tested, with losses on some policies. They answer the intervention question for a chosen policy; they do not by themselves explain which data properties produced the gain. Trajectory-aware estimators [5] (Wang et al. 2024) and cross-scale consistency of influence [Khaddaj et al., 2025] address the robustness and transfer questions, while joint batch selection [22] and interaction-aware group influence [Heo et al., 2026] address aggregation. Fact tracing [1], cross-lingual sharing [15], and trillion-token verbatim tracing [Liu et al., 2025b] supply descriptive evidence about what models draw on, at different costs and with different counterfactual meaning. Forgetting dynamics [10] bound how long an observed contribution persists. Reading these results through the table clarifies what remains open: a validated explanation of why a selected region helped, and evidence that a criterion learned from one run improves the next.

Mechanistic understanding has a distinct payoff. A mechanistic account answers an explanatory question and does not by itself establish transfer or decision value. Targeted data interventions during pretraining have changed the emergence of interpretable attention heads, with concurrent changes in in-context learning in the reported setting [12]. Such a result remains scoped to the model, data, and intervention used. Mechanistic evidence can deepen an account of how data supports an ability, while a useful curation decision can be tested before that account is complete.

Illustrative reference protocol for Section 6. To show how the matched-cost comparison in § 6 can be operationalized in a single open testbed, Table 2 provides an illustrative reference instantiation using released corpora, sub-billion to 1B model scales, and standardized evaluation suites. Concrete token budgets, checkpoint counts, and non-inferiority tolerances scale with the testbed, whereas pre-declaring all seven protocol components separates a discovery corpus Ddisc\mathcal{D}_{\mathrm{disc}} (where attribution is computed and a candidate data property is extracted) from a disjoint transfer corpus Dhold\mathcal{D}_{\mathrm{hold}} (where the resulting criterion competes against cheaper selectors without further attribution).

Author contributions

The matrix below records each author’s roles under the NISO CRediT (Contributor Roles Taxonomy) standard, listed in byline order.

ConceptualizationData curationFormal analysisFunding acquisitionInvestigationMethodologyProject administrationResourcesSoftwareSupervisionValidationVisualizationWriting – original draftWriting – review & editing
Glenn Matlin●●●●●●●●●●
Chandreyi Chakraborty●●●●●
Mohammed Rehan Parwani●●●●●●
Sahee Eom●●●●
Anthony Wen-Ming Zang●●●●
Akhil Theerthala●●●●
Arjun Chatterjee●●●●●●●●
Tejas Khandwekar●●●●●
Brandon Michaels●●●●●
Alvin Deng●●●●
Mark Riedl●

Table 3.

References

  1. [1]Ekin Akyürek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. Towards tracing knowledge in language models back to the training data. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2429–2446. Association for Computational Linguistics, 2022. doi: 10.18653/v1/2022.findings-emnlp.180. URL https://aclanthology.org/2022.findings-emnlp.180/.DOI
  2. [2]Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training. arXiv preprint arXiv:2312.02406, 2023. URL https://arxiv.org/abs/2312.02406.
  3. [3]Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max M. Marion, Matthew Leavitt, and Mansheej Paul. Perplexed by perplexity: Perplexity-based data pruning with small reference models. In International Conference on Learning Representations, pages 100615–100634, 2025. URL https://arxiv.org/abs/2405.20541.
  4. [4]Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B. Grosse. If influence functions are the answer, then what is the question? In Advances in Neural Information Processing Systems, volume 35, pages 17953–17967, 2022. URL https://arxiv.org/abs/2209.05364.
  5. [5]Juhan Bae, Wu Lin, Jonathan Lorraine, and Roger Grosse. Training data attribution via approximate unrolling. In Advances in Neural Information Processing Systems, volume 37, pages 66647–66686, 2024. doi: 10.52202/079017-2129. URL https://arxiv.org/abs/2405.12186.
  6. [6]Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning are fragile. In International Conference on Learning Representations, 2021. URL https://arxiv.org/abs/2006.14651.
  7. [7]Stella Biderman, Mohammad Aflah Khan, Niloofar Mireshghallah, Catherine Arnett, Fazl Barez, and Naomi Saphra. Position: Don’t just “fix it in post”: A science of AI must study training dynamics. In Proceedings of the 43rd International Conference on Machine Learning, 2026. URL https://arxiv.org/abs/2606.06533.
  8. [8]Cody Blakeney, Mansheej Paul, Brett W. Larsen, Sean Owen, and Jonathan Frankle. Does your data spark joy? Performance gains from domain upsampling at the end of training. arXiv preprint arXiv:2406.03476, 2024. URL https://arxiv.org/abs/2406.03476.
  9. [9]Ernie Chang, Yang Li, Patrick Huber, Vish Vogeti, David Kant, Yangyang Shi, and Vikas Chandra. AutoMixer: Checkpoint artifacts as automatic data mixers. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025a. URL https://arxiv.org/abs/2506.21910.
  10. [10]Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang, Youngkyung Seo, Du-Seong Chang, and Minjoon Seo. How do large language models acquire factual knowledge during pretraining? In Advances in Neural Information Processing Systems, 2024. URL https://arxiv.org/abs/2406.11813.
  11. [11]Tyler A. Chang, Dheeraj Rajagopal, Tolga Bolukbasi, Lucas Dixon, and Ian Tenney. Scalable influence and fact tracing for large language model pretraining. In International Conference on Learning Representations, 2025b. URL https://arxiv.org/abs/2410.17413.
  12. [12]Jianhui Chen, Yuzhang Luo, and Liangming Pan. Mechanistic data attribution: Tracing the training origins of interpretable LLM units. In Proceedings of the 43rd International Conference on Machine Learning, 2026. URL https://arxiv.org/abs/2601.21996.
  13. [13]Mayee F. Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. Skil-It: A data-driven skills framework for understanding and training language models. In Advances in Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2307.14430.
  14. [14]Mayee F. Chen, Michael Y. Hu, Nicholas Lourie, Kyunghyun Cho, and Christopher Ré. Aioli: A unified optimization framework for language model data mixing. In International Conference on Learning Representations, pages 46089–46132, 2025. URL https://arxiv.org/abs/2411.05735.
  15. [15]Rochelle Choenni, Dan Garrette, and Ekaterina Shutova. How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13244–13257. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.818. URL https://aclanthology.org/2023.emnlp-main.818/.
  16. [16]Zachary Coalson, Juhan Bae, Nicholas Carlini, and Sanghyun Hong. IF-GUIDE: Influence function-guided detoxification of LLMs. In Advances in Neural Information Processing Systems, 2025. URL https://arxiv.org/abs/2506.01790.
  17. [17]Cuong Dang, Hoang Anh Just, and Ruoxi Jia. Capacity-dependent effects of data selection for reasoning. In Conference on Language Modeling, 2026. URL https://arxiv.org/abs/2608.13721.
  18. [18]Aghyad Deeb and Fabien Roger. Do unlearning methods remove information from language model weights? arXiv preprint arXiv:2410.08827, 2025. URL https://arxiv.org/abs/2410.08827.
  19. [19]Junwei Deng, Pingbang Hu, Suliang Jin, Hao Lu, Jiachen T. Wang, Shichang Zhang, and Jiaqi W. Ma. How faithful is trajectory-based data attribution? Error sources, remedies, and practical guidelines. arXiv preprint arXiv:2605.18814, 2026. URL https://arxiv.org/abs/2605.18814.
  20. [20]Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608, 2017. URL https://arxiv.org/abs/1702.08608.
  21. [21]Logan Engstrom, Axel Feldmann, and Aleksander Madry. DsDm: Model-aware dataset selection with datamodels. In Proceedings of the 41st International Conference on Machine Learning, 2024. URL https://arxiv.org/abs/2401.12926.
  22. [22]Talfan Evans, Nikhil Parthasarathy, Hamza Merzic, and Olivier J. Henaff. Data curation via joint example selection further accelerates multimodal learning. In Advances in Neural Information Processing Systems, 2024. URL https://arxiv.org/abs/2406.17711.
  23. [23]Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12): 86–92, 2021. doi: 10.1145/3458723. URL https://doi.org/10.1145/3458723.
  24. [24]Sachin Goyal, Pratyush Maini, Zachary C. Lipton, Aditi Raghunathan, and J. Zico Kolter. Scaling laws for data filtering: Data curation cannot be compute agnostic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22702–22711, 2024. doi: 10.1109/CVPR52733.2024.02142. URL https://doi.org/10.1109/CVPR52733.2024.02142.
  25. [25]Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilė Lukosiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296, 2023. URL https://arxiv.org/abs/2308.03296.
  26. [26]Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. OLMES: A standard for language model evaluations. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5020–5048. Association for Computational Linguistics, April 2025a. doi: 10.18653/v1/2025.findings-naacl.282. URL https://aclanthology.org/2025.findings-naacl.282/.DOI

Paper details

vlnc:W_zf6eshdq
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents