betaFeedback

WorksComputer Science

On Representational and Functional Alignment of Training Data Attribution Methods for Language Models

Abstract

Existing approaches to training data attribution derive influence estimates from different sources of information, including gradients, model-internal states, and surface-level similarity. However, comparisons between attribution methods remain incomplete, often relying on indirect evaluations through downstream task performance. In this work, we introduce a framework for directly comparing methods along three dimensions: functional alignment, which measures correlation between full attribution vectors; representational alignment, which captures structural similarity of the representations underlying attribution; and cross-method predictability, which evaluates agreement on local pairwise ranking decisions. We observe strong representational and moderate functional alignment within paradigms. Alignment across paradigms is low and task-dependent, yet stable across models. Our results suggest that the similar downstream performance observed across paradigms in prior work can be attributed to agreement on the relative importance of documents, rather than to agreement on rankings of the full training data.

Introduction

Training data attribution methods assign documents from a language model’s training data scores proportional to their contribution to a given generation, to provide insights into model predictions. Several attribution paradigms have proven popular for language models, differing in the source of information they base attribution on: Some methods leverage loss gradients (e.g., Pruthi et al. [35]; Koh and Liang [22]), others model-internal states (e.g., Li et al. [27]), yet others only lexical overlap (e.g., Liu et al. [28]), assuming this still provides useful signals for understanding model behavior.

These paradigms follow fundamentally different attribution rationales (gradient-based methods estimate the effect of including a given training example on a test prediction through a first-order approximation, whereas representation-based methods assume that an example is influential if it is close to said test prediction in some feature space; Sun et al. [40]), with empirical work reporting limited rank correlation between certain paradigms (Pezeshkpour et al. [34]). Nevertheless, prior work has demonstrated that cheap representation-based methods can serve as proxies for expensive gradient-based ones for tasks such as fact tracing or artifact detection (Akyürek et al. [1]; Pezeshkpour et al. [34]; Sun et al. [40]; Liu et al. [28]).

Figure 1. Analytical Framework. We directly compare attribution strategies by decomposing alignment into (1) similarity of underlying representations, (2) correlation of attributions, and (3) agreement on pairwise rankings.

In this work, we investigate how well methods from different training data attribution paradigms align in their assessments of data influence. Prior work has largely neglected comparisons across paradigms and, when conducted, has compared solely in terms of downstream task performance (e.g., Jiao et al. [20]) or by the rank correlation between importance rankings over training instances (e.g., Pezeshkpour et al. [34]). We argue that this generates an incomplete view of alignment and propose a novel evaluation framework that addresses three main limitations: First, existing comparisons between attribution methods (Pezeshkpour et al. [34]; Brophy et al. [4]; Nguyen et al. [32]) depend heavily on evaluation task mechanics, limiting their transferability. For example, methods that work well for question answering may fail to capture aspects relevant to instruction tuning because of differences in learning objectives. Second, these works only consider global measures of correlation between attributions (e.g., Spearman’s ρ\rho between rankings induced by two attribution score vectors ∈R∣D∣\in\mathbb{R}^{|D|}), which does not adequately reflect functional equivalence in many downstream applications such as data selection, where agreement on relative relevance judgments is sufficient (i.e., whether a given example is more influential than another). Third, existing studies do not separate representational structure from downstream ranking decisions, making it difficult to determine whether differences between methods stem from how information is represented or from how it is processed. We accordingly decompose alignment into three dimensions: functional alignment, i.e., correlation between two full attribution vectors; representational alignment, i.e., structural similarity of the underlying document representations used for attribution; and cross-method predictability, i.e., agreement in pairwise relevance predictions rather than correlation between two rankings of the entire training dataset. These measures provide complementary perspectives on alignment: Functional alignment and cross-method predictability may serve to establish that two methods produce sufficiently similar importance rankings over training instances, while measures of representational alignment can generate insights into structural similarities of the document representations used for attribution.

We leverage this framework to conduct a systematic study of alignment of 12 attribution methods across four paradigms, 5 tasks, and 7 LoRA-finetuned models spanning three model families.1 Specifically, we provide insights into: (i) the alignment of attribution methods across different language modeling tasks, (ii) the stability of alignment across datasets and base models, (iii) whether methods can be employed as proxies for others, and (iv) factors that impact alignment at the instance level. We find that methods within the same attribution paradigm tend to exhibit high representational but moderate functional alignment. Alignment across paradigms is overall low and highly task-dependent, but consistent across model families and sizes. Our findings suggest that the relatively consistent downstream performance across paradigms in prior work may be driven by agreement on the relative importance of individual examples, rather than by agreement on example relevance across the full training

data. In a qualitative analysis, we find that cross-paradigm alignment appears to be shaped by a mix of prompting artifacts, lexical overlap, and task-dependent structural and behavioral similarities. Overall, our results suggest that methods within a paradigm are partially interchangeable, but that different paradigms capture distinct, only partially aligned notions of data influence, suggesting that only a combination of different sources of information may yield a complete characterization of data influence. Our main contributions are: (1) A framework for measuring representational and functional alignment; and a test of cross-method predictability that assesses local rather than overall agreement on example influence. (2) A systematic study of alignment across 7 base models, 5 tasks, and 12 attribution methods. (3) A qualitative analysis of factors of alignment.

Related Work

Works on input feature- (e.g. Jain and Wallace, [18]; Neely et al., [31]; Sun et al., [39]) and neuron attribution (e.g. Yu et al., [46]) have compared methods in terms of vector similarity and explanation quality metrics. However, their findings are not directly applicable here because they target fundamentally different attribution mechanisms.

Indirect evaluation of alignment. Prior comparisons of training data attribution methods are conceptual surveys without empirical evaluation ([11]; [5]), or predominantly compare via benchmark performance on tasks such as hallucination tracing (Wu et al., [42]), fact tracing (Akyürek et al., [1]), or data cleaning (Pezeshkpour et al., [34]; Jiao et al., [19]; Yu et al., [46]; Han et al., [12]). Work on example-based explanation compares in terms of application-specific properties such as stability (Gu et al., [10]), faithfulness, or plausibility (Hanawa et al., [13]). Others compare directly with ground-truth measures of influence obtained in training-based experiments (Bae et al., [3]; Grosse et al., [9]; Park et al., [33]; Mlodozeniec et al., [29]; Hong et al., [14]; Ilyas and Engstrom, [17]; Sun et al., [40]). All of these approaches only quantify whether two methods perform similarly on average. They do not consider (1) whether methods behave similarly on the same instances (which we cover through cross-method predictability and functional alignment), and more generally (2) whether attribution strategies operate on similar representations (which we capture via representational alignment).

Functional alignment across methods. Prior work that directly compares attribution methods has characterized alignment only in terms of correlation or set overlap among the top influential training examples, which corresponds to our definition of functional alignment. However, existing comparisons vary substantially in scope: Some compare different implementations of the same attribution paradigm ([32, 4]). Others evaluate the agreement of gradient-based methods with estimates derived from nontraditional attribution strategies, such as in-context probing ([19]) and neuron attribution ([46]). Our work is most aligned in scope with Pezeshkpour et al. [34], who compare embedding similarity with four gradient-based methods using BERT models. They find that certain gradient-based methods correlate highly with more expensive Hessian-based ones, but observe only modest correlation between similarity- and gradient-based methods. We extend this line of work by introducing two additional perspectives on alignment. We further consider new attribution methods from additional paradigms, new tasks, and more modern model architectures.

Methodology and Experimental Setup

We first describe 3 perspectives on alignment and define corresponding metrics. We then integrate different attribution methods under a common evaluation framework, before describing experiments targeting the following main questions: (1) How aligned are methods of different paradigms? (2) Is alignment consistent across models and tasks? (3) What drives alignment at the instance level?

Alignment Measures

Measuring Representational Alignment

We aim to capture the similarity between similarity structures induced by different representations

used for training data attribution. For this, we adapt methodology from mechanistic interpretability, originally proposed for studying the alignment of model-internal representations: Following Huh et al. [16], we define representations as functions r:Z→Rdr:\mathcal{Z} \to\mathbb{R}^{d} that encode individual training documents as vectors. For illustration, consider two representations rir_i and rjr_j, implemented via two text embedding models (we introduce additional representations in § 3.2). Given a similarity kernel defined by an inner product over representations, we apply two alignment metrics. Centered Kernel Alignment ([23]) yields a measure of global representational alignment:

CKA⁡(ri,rj)=∥riTrj∥F2∥riTri∥F∥rjTrj∥F,(1)\operatorname{CKA}(r_i,r_j)=\frac{\lVert r_i^\mathsf{T}r_j\rVert_{\mathrm{F}}^2}{\lVert r_i^\mathsf{T}r_i\rVert_{\mathrm{F}}\lVert r_j^\mathsf{T}r_j\rVert_{\mathrm{F}},} \tag*{(1)}

where FF is the Frobenius norm. We use Centered Kernel Nearest-Neighbor Alignment (CKNN) for local representational alignment, which restricts comparison to kk neighbors ([16]).

Measuring Functional Alignment

We define functional alignment as the average overall agreement between two attribution methods across test instances. Specifically, given two vectors of pointwise relevance scores over the training data ϕri(ztest),ϕrj(ztest)∈R∣D∣\phi_{r_i}(z_{\mathrm{test}}), \phi_{r_j}(z_{\mathrm{test}}) \in\mathbb{R}^{|D|}, we quantify agreement using Spearman’s ρ\rho, Kendall’s τ\tau, and rank-biased overlap (RBO, prioritizes agreement at the top of ranked lists). All metrics are computed per test instance and then averaged.

Measuring Cross-Method Predictability

Methods may diverge in their global rankings but still agree on the relative importance of individual examples. We therefore further test if attributions from one method can be leveraged to emulate those of another. Specifically, we propose a test based on pairwise comparisons: we evaluate if the scores from method ii can act as features to successfully predict the binary pairwise rankings of method jj. Let sztest(i)(a)s_{z_{\mathrm{test}}}^{(i)}(a) denote the score by method ii for document aa given ztestz_{\mathrm{test}}. For each ztestz_{\mathrm{test}}, we construct 2000 random pairs (a≠b)(a \ne b), subsequently split into train and test sets (80/20). We construct features based on score differences of method ii, and labels based on the binary ranking of method jj:

xab,ztest=sztest(i)(a)−sztest(i)(b),x_{ab,z_{\mathrm{test}}}=s_{z_{\mathrm{test}}}^{(i)}(a)-s_{z_{\mathrm{test}}}^{(i)}(b),
yab,ztest(j)=I[sztest(j)(a)>sztest(j)(b)].(2)y_{ab,z_{\mathrm{test}}}^{(j)}=\mathbb{I}\left[s_{z_{\mathrm{test}}}^{(j)}(a)>s_{z_{\mathrm{test}}}^{(j)}(b)\right]. \tag*{(2)}

To quantify how well rankings produced by method jj can be predicted using features derived from method ii (cross-method predictability), we report the mean held-out accuracy of instance-specific logistic regression models without intercept, where x~\tilde{x} denotes the score differences xx, standardized to zero mean and unit variance across pairs of a given test instance:

P(yab,ztest(j)=1)=11+exp⁡(−wztestx~ab,ztest)(3)P\left(y_{ab,z_{\mathrm{test}}}^{(j)}=1\right)=\frac{1}{1+\exp\left(-w_{z_{\mathrm{test}}}\tilde{x}_{ab,z_{\mathrm{test}}}\right)} \tag*{(3)}

Representations

Prior works categorize by attribution strategy (e.g., retraining- vs. gradient-based: Hammoudeh and Lowd, [11]), but only the former are tractable here. Instead, we categorize by source of information (Table 1: gradients, model-internal representations, embeddings, lexical overlap), as each corresponds to different assumptions about data influence. To enable the study of representational alignment, we require representation functions rr, so that documents can be compared via inner product (§ 3.1.1). For the study of functional alignment, we obtain point-wise scores ϕ(ztest,z)\phi(z_{\mathrm{test}},z) by dot product between r(ztest)r(z_{\mathrm{test}}) and the representation of the example r(z)r(z) for all methods except rstestr_{s_{\mathrm{test}}} (for which scores are given by Equation (7)).

Gradient-based representationsModel-internal representations
r∇fr_{\nabla^f} Raw model loss gradientsrrepsimr_{\mathrm{repsim}} Final-layer last token hidden state ([27])
r∇fL2r_{\nabla^f}^{L2} L2-normalized gradientsrmpr_{\mathrm{mp}} Mean-pooled final-layer hidden states
r∇fAdamr_{\nabla^f}^{\mathrm{Adam}} Adam-adjusted gradients ([43])rattnr_{\mathrm{attn}} Summary of norm. att. maps
rHessianr_{\mathrm{Hessian}} Hessian-adjusted gradients
Embedding similarityLexical overlap
qwenq_{\mathrm{wen}} Qwen3-Embedding-8Brbm25r_{\mathrm{bm25}} BM25-based vector
nemotn_{\mathrm{emot}} llama-embed-nemotron-8brjaccardr_{\mathrm{jaccard}} Jaccard-based vector
rtfidfr_{\mathrm{tfidf}} TF-IDF-based vector

Table 1. Summary of attribution methods studied.

Note that there exist multiple valid definitions of relevance for gradient-based methods: one can retrieve the most or least influential samples (i.e., those with the largest or smallest absolute similarity scores) or the most helpful or harmful ones (those with the most positive or negative scores; the ones that either "help" with or "harm" loss most, e.g., Koh and Liang, [22]). We do so to ease interpretability, given that this strategy’s anticipated remove-and-retrain effect on loss most closely matches that of the semantic- and lexical-similarity-based retrieval logics (we assume that training on similar examples is more likely to decrease loss than training on unrelated ones).

Gradient-based Representations

Common to several gradient-based strategies is the definition of influence as the similarity between test-instance gradients and training-instance gradients (Hammoudeh and Lowd, [11]). We define three representations based on prior work:

r∇fidX(z)=∇L(z,θ)(4)r_{\nabla f}^{\mathrm{id}X}(z)=\nabla L(z,\theta) \tag*{(4)}
r∇fL2(z)=∇L(z,θ)∥∇L(z,θ)∥2(5)r_{\nabla f}^{L2}(z)=\frac{\nabla L(z,\theta)}{\left\|\nabla L(z,\theta)\right\|_2} \tag*{(5)}
r∇faa(z)=aa⁡(∇L(z,θ)).(6)r_{\nabla f}^{\mathrm{aa}}(z)=\operatorname{aa}\left(\nabla L(z,\theta)\right). \tag*{(6)}

Representation r∇fidXr_{\nabla f}^{\mathrm{id}X} is based on raw gradients, as used for estimation by Pruthi et al. [35]. r∇fL2r_{\nabla f}^{L2} represents unit-norm gradients, as utilized in Yeh et al. [45]. In r∇faar_{\nabla f}^{\mathrm{aa}}, we use Adam-adjusted (normalized) gradients, as in Xia et al. [43].

Next, we consider point-wise influence via influence functions as in Koh and Liang [22]:

IF(z,ztest)=−Hθ^−1∇L(ztest,θ^)⋅∇L(z,θ^).(7)IF(z,z_{\mathrm{test}})=-H_{\hat{\theta}}^{-1}\nabla L(z_{\mathrm{test}},\hat{\theta})\cdot\nabla L(z,\hat{\theta}). \tag*{(7)}

Conceptually, stests_{\mathrm{test}} incorporates second-order information on how changes in parameters affect test loss. To study the effect of this re-weighting, we define the following representation. For implementation details, see Appendix § 10:

rstest(z)=Hθ^−1∇θL(z,θ^).(8)r_{s_{\mathrm{test}}}(z)=H_{\hat{\theta}}^{-1}\nabla_{\theta}L(z,\hat{\theta}). \tag*{(8)}

Model-internal Representations

We define three representations based on internal model states: We define rrepsimr_{\mathrm{repsim}} based on the influence estimation strategy in Li et al. [27]:

rrepsim(z)=HT,:(L).(9)r_{\mathrm{repsim}}(z)=H_{T,:}^{(L)}. \tag*{(9)}

where H(L)∈Rn×dH^{(L)}\in\mathbb{R}^{n\times d} denotes the hidden states of the final transformer layer for an input sequence zz, and TT is the index of the last non-padding token.

We obtain rmpr_{\mathrm{mp}} by mean pooling over the non-padding tokens in the final layer. Let m∈{0,1}nm\in\{0,1\}^{n} specify the attention mask, then:

rmp(z)=(mTH(L))/(1Tm).(10)r_{\mathrm{mp}}(z)=\left(m^{\mathsf{T}}H^{(L)}\right)/(1^{\mathsf{T}}m). \tag*{(10)}

Next, rattnr_{\mathrm{attn}} is based on attention patterns. Here, documents are considered similar if they induce similar distributions of attention values. Kobayashi et al. [21] define a token-to-token interaction map M∈Rn×nM\in\mathbb{R}^{n\times n}, Mi,j:=∥∑hαi,jhfh(xj)∥M_{i,j}:=\left\|\sum_h \alpha_{i,j}^{h}f^{h}(x_j)\right\|, which we adapt to support grouped-query attention (GQA) utilized by Qwen models here (Appendix § 11). To obtain sequence-length invariant vectors suited for comparison across documents, we pad to the maximum sequence length SS, and then compute the singular value decompositions Ml=UΣVT, Σl=diag⁡(σ1(l),…,σS(l))M_l=U\Sigma V^{\mathsf{T}},\ \Sigma_l=\operatorname{diag}\left(\sigma_1^{(l)},\ldots,\sigma_S^{(l)}\right). The attention-based representation is the concatenation of the top k=128k=128 singular values across all layers:

rattn(z)=[σ1(1),…,σk(1),…,σ1(L),…,σk(L)].(11)r_{\mathrm{attn}}(z)=\left[\sigma_1^{(1)},\ldots,\sigma_k^{(1)},\ldots,\sigma_1^{(L)},\ldots,\sigma_k^{(L)}\right]. \tag*{(11)}

Embedding-based Representations

We define two representations on embedding similarity using external models, rnemotr_{nemot}: [nvidia/llama-embed-nemotron-8b](https://huggingface.co/nvidia/llama-embed-nemotron-8b: Babakhin et al., [2]) and rqwenr_{qwen}: [Qwen/Qwen3-Embedding-8B](https://huggingface.co/Qwen/Qwen3-Embedding-8B: Zhang et al., [47]), selected based on performance on the MMTEB benchmark (Enevoldsen et al., [7]).

Lexical Overlap-based Representations

Recent work (e.g., Liu et al., [28]) retrieves lexically related training examples to address LLM-scale datasets. We therefore define a set of representations in which two documents will be assigned high similarity scores if they are lexically related to the same documents in the training data. Our approach is inspired by the representation strategy in Lagus et al. [25], who show that by representing documents as matrices A∈Rn×dA \in\mathbb{R}^{n \times d}, where each row corresponds to a word vector, their similarity can be measured in terms of the squared Frobenius norm of their pairwise word similarity matrices (∥ABT∥F2\lVert AB^{\mathsf{T}}\rVert_{\mathrm{F}}^{2}). This approach compares documents based on the covariance of word vectors. To represent each document in terms of its similarity to other documents in the training data while allowing comparisons via inner-product similarity, we first generate rankings for a given test instance ztestz_{\mathrm{test}} using Jaccard similarity (for lexical overlap), TF-IDF (weighted lexical similarity), and BM25 (probabilistic relevance). Let did_i be the ii-th training example, and ϕ(z,zi)\phi(z,z_i) denote the relevance of ziz_i for zz. Then rbm25r_{\mathrm{bm25}}, rtfidfr_{\mathrm{tfidf}} and rjaccardr_{\mathrm{jaccard}} are of the form

rϕ(z)=[ϕ(z,zi),ϕ(z,zi),…,ϕ(z,zN)].(12)r_{\phi}(z)=\left[\phi(z,z_i),\phi(z,z_i),\ldots,\phi(z,z_N)\right]. \tag*{(12)}

Experiments

Modeling Alignment Across Paradigms

We cluster representations using agglomerative hierarchical clustering based on pairwise alignment scores to characterize alignment structure across attribution methods. Let yijy_{ij} denote the alignment score (e.g., CKA) between representations ii and jj. We consider a pseudo-dissimilarity measure dij=1−yijd_{ij}=1-y_{ij} and perform agglomerative hierarchical clustering of representations using average linkage (UPGMA: Sokal and Michener, [37]). Starting from individual representations, this approach builds a hierarchy by iteratively merging the two clusters with the smallest average pairwise dissimilarity. This yields one hierarchical clustering per alignment score, which we visualize as dendrograms.

Stability of Clusterings

To assess whether our method groupings are stable across different base models and tasks, and not a product of random variation, we compare dendrograms using cophenetic correlation (Sokal and Rohlf, [38]). For a given clustering, let cijc_{ij} denote the cophenetic distance between representations rir_i and rjr_j, defined as the dissimilarity value at which rir_i and rjr_j are first merged into the same cluster. Given two clusterings over a set of representations, the cophenetic correlation between dendrograms following Sokal and Rohlf [38] is ρcoph(A,B)=ρ({cijA}i<j,{cijB}i<j)\rho_{\mathrm{coph}}(A,B)=\rho(\{c_{ij}^{A}\}_{i<j},\{c_{ij}^{B}\}_{i<j}). We measure the average stability across factors

ρˉd,m=1(∣D∣2)(∣M∣2)∑d1<d2∑m1<m2ρcoph(Cd1,m1,Cd2,m2),(13)\bar{\rho}_{d,m}=\frac{1}{\binom{\lvert\mathcal{D}\rvert}{2}\binom{\lvert\mathcal{M}\rvert}{2}}\sum_{d_1<d_2}\sum_{m_1<m_2}\rho_{\mathrm{coph}}(C^{d_1,m_1},C^{d_2,m_2}), \tag*{(13)}

as well as two conditional averages: (i) fixing the dataset, averaging over base models,2 and (ii) fixing the base model, averaging over datasets. Lastly, we measure the pairwise average cophenetic stability between individual alignment measures.3

Modeling Factors Affecting Alignment

To assess the robustness of our analysis, specifically whether alignment scores are sensitive to the choice of base model and task, we fit linear mixed-effects models and assess contributions of both factors (and their interaction) via likelihood ratio tests over nested specifications. The full model is given by:

Mfull:y∼C(b)∗C(d)≡\mathcal{M}_{\mathrm{full}}: y \sim C(b) * C(d) \equiv
yijbd=β0+βb+βd+βbd+uij+ϵijbd,y_{ijbd}=\beta_0+\beta_b+\beta_d+\beta_{bd}+u_{ij}+\epsilon_{ijbd},

where yijbdy_{ijbd} denotes the alignment score for representation pair ijij, given base model bb and dataset dd. We assume uij∼N(0,σpair2)u_{ij}\sim\mathcal{N}(0,\sigma_{\mathrm{pair}}^{2}), ϵijbd∼N(0,σϵ2)\epsilon_{ijbd}\sim\mathcal{N}(0,\sigma_{\epsilon}^{2}). We test for interaction effects via a likelihood ratio test between Mfull\mathcal{M}_{\mathrm{full}} and Madd:y∼C(b)+C(d)\mathcal{M}_{\mathrm{add}}: y\sim C(b)+C(d) corresponding to H0:βbd=0H_0:\beta_{bd}=0. We likewise test the effects of base model and dataset via likelihood ratio tests between full and reduced models (M−B:y∼C(d)\mathcal{M}_{-B}:y\sim C(d), M−D:y∼C(b)\mathcal{M}_{-D}:y\sim C(b)), corresponding to H0:βb=0H_0:\beta_b=0 and H0:βd=0H_0:\beta_d=0. We test whether each fixed effect improves fit via likelihood ratio tests and quantify effect sizes as the reductions in marginal and conditional R2R^2.4

Qualitative Comparison of Local Neighborhoods Across Representations

The proposed measures quantify alignment but do not identify sources of agreement or disagreement at the instance level. We therefore complement them with instance-level qualitative analysis to better understand what drives misalignment. Given two representations ri,rjr_i,r_j and a document zz, we retrieve the top-5 most similar examples to zz under each representation. For each dataset and representation pair, we compare the resulting local neighborhoods in two settings: (i) when the neighborhoods diverge most, and (ii) when they align best (see Appendix § 12 for details). We restrict this analysis to 4 of the 12 representations, selecting a representative subset using a greedy farthest-first procedure on the CKA dissimilarity matrix. We inspect 60 pairs across the Olmo3 models.

Models and Data

We fine tune Qwen3-Base (Yang et al., [44]) at sizes 0.6B, 1.7B, 4B, 8B; Olmo-3-1025-7B (Ettinger et al., [8]); OLMo-2-0425-1B (Walsh et al., [41]); and Llama-3.2-1B (Dubey et al., [6]) for one epoch using LoRA (Hu et al., [15]) on the following datasets (for details see Appendix § 13):

SFT: the instruction-tuning dataset for OLMo2 based on Tülu3 (allenai/tulu-3-sft-olmo-2-mixture-0225: Lambert et al., [26]);

math (allenai/tulu-3-sft-personas-math), and code: subsets of Tülu3 targeting math and coding problems (allenai/tulu-3-sft-personas-code);

squad_v2: reading comprehension for extractive question answering (Rajpurkar et al., [36]);

ftrace: benchmark dataset that perturbs documents to increase attribution difficulty (Jiao et al., [20]).

We fine-tune 35 models on the full datasets and measure alignment across 1000 random documents per dataset and model combination.

Results

We report pairwise representational alignment (CKA), functional alignment (Spearman’s ρ\rho), and cross-method predictability averaged across models and tasks in Figure 3, for Kendall’s τ\tau, RBO, per model- and per task results see Appendix § 8.

Figure 3. Pairwise alignment. Left: representational alignment (CKA ∈[0,1]\in[0,1]), center: functional alignment (Spearman’s ρ∈[−1,1]\rho\in[-1,1]), and right: cross-method predictability (acc ∈[0,1]\in[0,1]) averaged across tasks and models. Higher values suggest better alignment. For CKNNA@25, RBO, and Kendall’s τ\tau, see Appendix § 8.

Alignment Across Paradigms

Regarding our first question (how aligned are methods from different paradigms?), we find that attribution methods cluster by their underlying paradigms,

exhibiting high internal agreement (e.g., r∇fL2r_{\nabla_f}^{L2} vs. r∇fidXr_{\nabla_f}^{idX}) but low cross-paradigm alignment (e.g., r∇fL2r_{\nabla_f}^{L2} vs. rrmpr_{rmp}). This is reflected in the dendrograms in Figure 2, which show clear separation between gradient-based, embedding-based, and lexical similarity methods. Overall, we observe clear within-paradigm alignment, as evidenced by a pronounced block-diagonal structure in the heatmaps of Figure 3 and Appendix § 8. This is most pronounced for representational alignment (CKA, CKNN@25) and cross-method predictability within gradient-based (rstestr_{stest}, r∇faar_{\nabla_f}^{aa}, r∇fL2r_{\nabla_f}^{L2}, r∇fidXr_{\nabla_f}^{idX}) and embedding-based attribution paradigms (rnemotr_{nemot}, rqwenr_{qwen}). For functional alignment (ρ\rho, τ\tau, RBO), however, this pattern only holds for embedding-based methods. One notable exception is the pairing of BM25 and the attention-based method, which features relatively high representational alignment (yet still weak functional alignment and low cross-method predictability).

Hierarchical clustering of attribution methods

Figure 2. Hierarchical clustering of attribution methods. Clusters largely align with attribution paradigms.

While we measure considerably lower rank correlations than Pezeshkpour et al. [34] report for BERT models, our results broadly align with theirs for the strategies considered in both studies. Specifically, gradient- and Hessian-based methods exhibit greater functional alignment than cross-paradigm pairings, while correlation between gradient- and embedding-based methods is negligible.

Per-task predictability results (Appendix § 8) show substantial variation within the same paradigm, for example, alignment between r∇faar_{\nabla_f}^{aa} and rstestr_{stest} is about 11-20 pt lower on SFT (0.56) than on other tasks. Alignment across paradigms, however, appears largely invariant to task.

Stability of clusterings. The average cophenetic stability ρˉd,m\bar{\rho}_{d,m} is high across metrics (CKA: 0.80, CKNNA@25: 0.75, ρ\rho: 0.71, τ\tau: 0.75, RBORBO: 0.72, accpredacc_{\mathrm{pred}}: 0.79), indicating that dendrogram structure is largely preserved across conditions. Stability across base models ρˉb\bar{\rho}_{b} and datasets ρˉd\bar{\rho}_{d} is consistently high as well (>0.7> 0.7, Appendix § 7). Regarding cophenetic stability between alignment metrics (CKA, ρ\rho, etc.), ρˉa1,a2\bar{\rho}_{a_1,a_2}, we find high agreement between measures targeting the same quantity (representational alignment, functional alignment, or cross-method predictability). Measures targeting representational alignment (CKA, CKNNA@25) show strong consistency (ρˉa1,a2≈0.70\bar{\rho}_{a_1,a_2} \approx0.70), while functional alignment measures (τ\tau, RBO, ρ\rho) also exhibit moderate-to-high agreement (≈0.57−0.61\approx0.57 - 0.61). In contrast, correlations across measures targeting different quantities (e.g., CKA vs. ρ\rho) are low (≈0.02−0.30\approx0.02 - 0.30). Weak correlation to functional alignment confirms that cross-method predictability captures a distinct notion of alignment.

Factors Affecting Alignment.

To answer our second question (is alignment consistent across models and tasks?), we find that alignment patterns (rather than the clusterings discussed earlier) are broadly stable with respect to base model and dataset choice: while different choices induce statistically significant effects in likelihood-ratio tests, the practical effect sizes (ΔR2\Delta R^2, Table 6 in Appendix § 9) are minor. This demonstrates that the specific alignment properties (or lack thereof) between paradigms are fundamental, rather than artifacts of a particular experimental setup. Among these secondary factors, removing the dataset factor yields the largest decrease in explained variance across metrics, except CKA. Conversely, the base model choice shows a stronger effect only on global representational alignment (CKA), but not on its local counterpart (CKNNA@25). Interaction terms between the base model and dataset contribute comparatively little additional explanatory power.

Term →ΔRm2(M−B)\Delta R^{2}_{m}(M-B)ΔRc2(M−B)\Delta R^{2}_{c}(M-B)ΔRm2(M−D)\Delta R^{2}_{m}(M-D)ΔRc2(M−D)\Delta R^{2}_{c}(M-D)ΔRm2(Madd)\Delta R^{2}_{m}(M_{\mathrm{add}})ΔRc2(Madd)\Delta R^{2}_{c}(M_{\mathrm{add}})
CKA9.88e-031.01e-026.89e-037.07e-032.25e-032.31e-03
CKNNA@254.64e-034.77e-031.33e-021.36e-021.59e-031.64e-03
τ\tau5.16e-055.30e-051.21e-041.24e-041.29e-051.32e-05
accpredacc_{\mathrm{pred}}6.74e-036.92e-037.67e-037.87e-033.72e-033.82e-03
RBO1.04e-041.05e-042.89e-042.94e-044.95e-055.02e-05
ρ\rho9.74e-059.96e-052.07e-042.12e-042.40e-052.45e-05

Table 6. Ablation analysis. Reductions in marginal and conditional R2R^2 (ΔRm2\Delta R^{2}_{m}, ΔRc2\Delta R^{2}_{c}) when removing base model-, dataset-, or interaction terms from the full mixed-effects model. Although likelihood-ratio tests indicate several significant terms, the effect sizes are small, suggesting that our analysis is robust to changes in base model and task.

Qualitative Insights on Alignment

To address our final question (what drives alignment at the instance level?), we manually compare local neighborhoods of different representations to understand the qualitative sources of agreement and disagreement. We examine whether local neighborhoods differ across attribution representations in terms of (1) prompt and completion format, such as prompting style or step-by-step thinking, (2) keyword overlap, and (3) shared topics or themes. For each pairing of attribution representations (e.g., r∇fidxr_{\nabla f}^{\mathrm{id}x} vs. rbm25r_{bm25}), we analyze one test instance with minimal- and one test instance with maximal overlap between neighborhoods. Below we summarize our results; for full annotations see Appendix § 12.

Alignment depends on instruction tuning goals. Alignment patterns vary across tasks. For the datasets math and code, neighborhoods tend to align in terms of example structure, such as shared problem-decomposition patterns or prompting styles. In contrast, neighborhoods for squad_v2 and france frequently align at the level of high-level question type, or semantic category (e.g., incomplete-information handling). When BM25-based neighborhoods lack keyword overlap with the test instance, they also tend to have low overlap with other strategies. In contrast, neighborhoods of gradient- and embedding-based methods often consist of instances with similar structures or completion behaviors, despite limited keyword overlap.

Prompting artifacts drive misalignment. Recurring instruction specifications and personaprompting templates introduce repeated structure across all datasets, increasing lexical overlap. BM25 appears to be frequently driven by these artifacts. In contrast, representation-based methods appear to exploit more meaningful task-related similarities between instances (from an instruction-tuning perspective). On the other hand, test instances targeting abilities not directly related to task solving (such as safety refusals, indications of missing information, or reproduction of hardcoded facts such as knowledge-cutoff dates) often appear in the same neighborhoods under the representation-based methods (r∇fid,xr_{\nabla f}^{id,x}, rrepsimr_{\mathrm{repsim}}, and rnemotr_{\mathrm{nemot}}). This occurs even when the instances are neither semantically nor topically related: for example, some neighborhoods contain both safety-related examples and examples on the cutoff date.

Discussion

Task dependency of alignment and limitations of lexical matching. The divergence across paradigms appears to stem from differences in how each method defines similarity. Lexical methods (e.g., BM25) primarily capture surface form and instruction overlap, and therefore diverge in their attributions from other strategies when structure is shared, but wording differs. In contrast, gradient- and embedding-based methods likely better capture structural and behavioral similarity, grouping instances by shared instruction and completion patterns rather than keyword overlap. Given that learning behavior rather than memorization is the central goal of instruction tuning tasks, this suggests that relying on lexical overlap alone may be a weak basis for attribution in instruction tuning.

Measuring functional alignment is insufficient. Our results show that the three proposed evaluation protocols capture distinct and only partially overlapping aspects of alignment. We further show that methods may exhibit low correlation in attributions across the full dataset yet strong pairwise agreement on example relevance, as well as high representational alignment. This implies that low average rank correlation does not necessarily preclude one method from serving as a useful substitute for another in a given downstream task. Accordingly, relying solely on rank-correlation analyses may lead to the unwarranted disqualification of prospective methods, whereas our framework can provide more nuanced insights into alignment.

Additionally, our results may also partially explain the apparent mismatch between prior findings, where some report low cross-paradigm correlation, while others independently observe similar downstream-task performance of methods belonging to different paradigms. To assess whether this hypothesis holds, future work should design evaluation protocols spanning downstream tasks that vary in their reliance on local versus global agreement, and test whether cross-method predictability scores predict performance on individual tasks.

Mechanistic interpretation of alignment. Despite negligible functional alignment, we measure high representational alignment and cross-method predictability between gradient-similarity methods and the Hessian-based rtestr_{\mathrm{test}}. Decomposing alignment into the three suggested dimensions can provide a mechanistic interpretation of this apparent discrepancy: High representational alignment suggests that the reweighting in rtestr_{\mathrm{test}} (Equation (8)) does not fundamentally alter the representation structure. While it systematically changes score assignment, it retains information about pairwise relevance, as indicated by the high cross-method predictability. Similarly, although representation-based methods may be expected to align closely with one another, our representational alignment results and qualitative analysis show that alignment is contingent on capturing the same task-specific features.

Alignment deteriorates cross-paradigm. While most pairings achieve above-chance cross-method predictability, accuracy is only consistently above 0.75 for within-paradigm comparisons, whereas the only cross-paradigm pairings scoring above 0.65 are those between lexical- and embedding-based methods. Representational and functional alignment across paradigms is likewise weak. Our findings suggest that practitioners seeking to use one method as a proxy for another (e.g., to pre-filter examples before applying more computationally expensive but also more accurate methods) should select methods of the same attribution paradigm. At the same time, the observed lack of alignment suggests an opportunity for future work on example-based explanations built on data attribution: combining orthogonal strategies may yield more comprehensive characterizations of data influence.

Conclusion

We introduce a framework for directly measuring alignment between training data attribution methods, which captures correlation of attribution rankings (functional alignment), assesses structural similarity of the underlying representation spaces used for attribution (representational alignment), and evaluates agreement on pairwise ranking decisions (cross-method predictability). We show that this decomposition is useful for disentangling different notions of alignment that are not captured by existing evaluations based on measures of downstream performance. Comparing 12 methods spanning four paradigms, we find high alignment within individual attribution paradigms (gradient similarity, model-internal representations, embedding similarity, and lexical overlap). In contrast, cross-paradigm alignment is consistently lower and task-dependent. This suggests that methods within a paradigm are partially interchangeable for downstream applications, whereas those across paradigms are not reliable proxies for one another. Qualitatively, we find that this divergence stems from a fundamental difference in how paradigms define similarity: lexical methods capture surface-level prompting artifacts, whereas representation-based ones appear to capture behavioral patterns.

Overall, our results show that representational alignment, functional alignment, and cross-method predictability do not necessarily covary, emphasizing the need to distinguish between local and overall agreement in evaluation. To this end, we recommend augmenting work on new attribution methods through systematic alignment analysis to contextualize new approaches within known paradigms.

Limitations

Our experiments consider text generation tasks. Alignment between specific paradigms may differ for other task formats, such as simpler classification setups. However, our framework is fully compatible with other task definitions and models outside NLP contexts.

In contrast to prior work, we intentionally forgo evaluating downstream performance in favor of directly testing alignment, which is required to establish a more fundamental understanding. Our focus was on developing complementary perspectives on alignment, rather than studying failure modes specific to individual tasks or replicating prior downstream evaluations.

Note that our analysis primarily provides recommendations on evaluation practices; our study is not designed to produce quality or performance characterizations of individual attribution methods that could directly inform the design of new attribution methods.

We restrict our analysis to base models with a maximum of 8B parameters due to the computational costs involved with obtaining and analyzing gradient-based influence estimates. We ensure robustness by validating our findings across seven models spanning parameter sizes from 0.6B to 8B and three model families. Likewise, in line with prior work (e.g., [24, 43]), we only consider LoRA-fine-tuned models in our analysis due to the high computational costs of gradient-based attribution. Alignment patterns may vary when attribution targets other (subsets) of model parameters.

While future work may explore alternative training data attribution paradigms, we have compiled a set of 12 methods across four paradigms, which represent a cross-section of the current literature.

Ethical Considerations

Given that we document patterns of alignment across attribution paradigms, there is a risk that actors acting in bad faith could misuse this information to select an attribution strategy that aligns with their desired claims of data influence, despite prior negative results obtained with other methods. Our work mitigates this risk by demonstrating that single-paradigm attribution may provide an incomplete view of data influence.

Acknowledgments

We would like to thank the anonymous reviewers for their helpful comments and suggestions. This research has been funded by the Vienna Science and Technology Fund (WWTF)[10.47379/VRG19008] "Knowledge-infused Deep Learning for Natural Language Processing".

References

Cophenetic Stability

Llama3.2-1BOLMo-1BOLMo3-7BQwen3-0.6BQwen3-1.7BQwen3-4BQwen3-8B
CKA0.800.820.800.840.830.810.79
CKNN@250.860.760.740.760.740.720.75
τ\tau0.740.810.760.830.810.790.75
accpredacc_{\mathrm{pred}}0.790.770.800.830.830.800.79
RBO0.740.780.730.760.780.750.72
ρ\rho0.720.780.710.790.760.750.71

Table 2. Average cophenetic correlation across tasks per base model.

codeftracemathsftsquad_v2
CKA0.920.950.840.950.97
CKNN@250.870.960.920.950.98
τ\tau0.920.900.770.900.96
accpredacc_{\mathrm{pred}}0.990.900.800.930.96
RBO0.910.860.780.900.94
ρ\rho0.890.880.740.890.95

Table 3. Average cophenetic correlation across base models per task.

CKACKNN@25τ\tauaccpredacc_{\mathrm{pred}}RBOρ\rho
CKA0.730.700.050.030.020.05
CKNN@250.670.090.070.060.09
τ\tau0.610.300.570.61
accpredacc_{\mathrm{pred}}0.160.290.30
RBO0.520.57
ρ\rho0.61

Table 4. Average cophenetic correlation between alignment measures.

Full Alignment Scores

Figure 4. Representational alignment (CKNNA@25: top), and functional alignment (Kendall’s τ\tau: center, RBO: bottom), averaged across tasks and models.

Figure 5. Per-task CKA results.

Figure 6. Per-task CKNN@25 results.

Figure 7. Per-task Spearman’s ρ\rho results.

Figure 8. Per-task Kendall’s τ\tau results.

Figure 9. Per-task RBO results.

Figure 10. Per-task cross-method predictability results.

Figure 11. Per-base-model CKA results.

Figure 12. Per-base-model CKNNA @25 results.

Figure 13. Per-base-model Spearman’s ρ\rho results.

Figure 14. Per-base-model Kendall’s τ\tau results.

Figure 15. Per-base-model RBO results.

Figure 16. Per-base model cross-method predictability.

Likelihood Ratio Tests

Term → MetricBase modelDatasetInteraction
ρ\rho13.127.93.2
τ\tau10.123.52.5
RBO49.2*136.7***23.5
accpredacc_{\mathrm{pred}}733.9***831.1***411.0***
CKA105.1***73.8***24.4
CKNNA@2550.9**142.7***17.6

Table 5. Likelihood-ratio tests. Results of likelihood-ratio tests comparing the full mixed-effects model against reduced variants with base model, dataset, or interaction terms removed. See § 3.3.3.

Details on rstestr_{\mathrm{stest}}

Kwon et al. [24] note that for negative log-likelihood loss functions, the inverse Hessian part in stests_{\mathrm{test}} can be approximated with a closed-form expression. Following their approach, we compute:

rstest≈1nλ∑i=1n(∇θL(z,θ^)−∇θL(z,θ^)⊤∇θL(zi,θ^)λ+∥∇θL(zi,θ^)∥2∇θL(zi,θ^))r_{\mathrm{stest}} \approx\frac{1}{n\lambda}\sum_{i=1}^{n}\left(\nabla_{\theta}L(z,\hat{\theta})-\frac{\nabla_{\theta}L(z,\hat{\theta})^{\top}\nabla_{\theta}L(z_i,\hat{\theta})}{\lambda+\lVert\nabla_{\theta}L(z_i,\hat{\theta})\rVert^{2}}\nabla_{\theta}L(z_i,\hat{\theta})\right)

where nn is the number of training examples and λ\lambda is a damping constant.

Details on rattnr_{\mathrm{attn}}

Let z=(x1,…,xS)z=(x_1,\ldots,x_S) be a document of length SS. Following Kobayashi et al. [21] we consider a standard multi-head attention layer with output

yiintegrated=∑j=1n∑hαi,jhfh(xj)y_i^{\mathrm{integrated}}=\sum_{j=1}^{n}\sum_h\alpha_{i,j}^{h}f^h(x_j)

where

fh(x):=(xWV,h+bV,h)WO,h.f^h(x):=(xW^{V,h}+b^{V,h})W^{O,h}.

Kobayashi et al. define a token-to-token interaction map M∈Rn×nM\in\mathbb{R}^{n\times n}, Mi,j:=∥∑hαi,jhfh(xj)∥M_{i,j}:=\left\lVert\sum_h\alpha_{i,j}^{h}f^h(x_j)\right\rVert. To adapt this analysis to modern multi-head attention implementations (e.g., HuggingFace), which use a single fused output projection (oprojo_{\mathrm{proj}}) after concatenating all heads, we note that the per-head output projections WO,hW^{O,h} can be viewed as a block decomposition of WOW^O, such that

∑h(vhWO,h)≡Concat⁡(v1,…,vH)WO.\sum_h(v^hW^{O,h})\equiv\operatorname{Concat}(v^1,\ldots,v^H)W^O.

We utilize the following norm-based measure to capture token-to-token attention at layer ll:

Ml,i,j=∥Concat⁡h(αi,jhf~h(xj))WO∥,M∈RL×(N×N)M_{l,i,j}=\left\lVert\operatorname{Concat}_h\left(\alpha_{i,j}^{h}\tilde{f}^{h}(x_j)\right)W^O\right\rVert,\quad M\in\mathbb{R}^{L\times(N\times N)}

where f~h(⋅)\tilde{f}^{h}(\cdot) denotes the value transformation associated with head hh. In standard multi-head attention, f~h=fh\tilde{f}^{h}=f^{h}, while in grouped-query attention (GQA), f~h=fg(h)\tilde{f}^{h}=f^{g(h)} with g(h)g(h) mapping query heads to shared value groups.

Qualitative Study of Local Neighborhoods

We provide observations for all 60 studied examples in Table 7. Figure 17 to Figure 21 depict a subset of examples selected as follows: For a given representation rir_i, let N5(ri)(z)\mathcal{N}_5^{(r_i)}(z) denote the set of top-5 training examples ranked by similarity to document zz. For two representations (ri,rj)(r_i,r_j), we quantify overlap between their local neighborhoods around a common zz:

Figure 17. Instances with highest (top) and lowest (bottom) 5-nearest neighbor overlap across representations rbm25r_{\mathrm{bm25}} and nemot for Olmo-3-7B_f7race_date-1m_conversational. The instances shown are: “Complete the sentence by filling in the blank: The women's marathon 10 kilometre event at the 2008 Olympic Games took place on 20 August at the Shunyi Olympic Rowing-Canoeing Park in Beijing, .” — China; “Complete the sentence by filling in the blank: He was special envoy in Copenhagen 1919-1927, in Budapest 1922-1927, in Riga and Kaunas 1927-” — Warsaw; “Complete the sentence by filling in the blank: The women's team archery competition at the in London was held from 27 to 29 July at Lord's Cricket” — 2012 Olympic Games; “Complete the sentence by filling in the blank: The women's marathon 10 kilometre event at the 2008 Olympic Games took place on 20 August at the” — Beijing; “Complete the sentence by filling in the blank: The men's marathon 10 kilometre event at the 2008 Olympic Games took place on 21 August at the” — China; “Complete the sentence by filling in the blank: The men's marathon 10 kilometre event at the 2008 Olympic Games took place on 21 August at the” — Shunyi Olympic Rowing-Canoeing Park; “Complete the sentence by filling in the blank: Andorra, represented by the Andorran Olympic Committee, sent a team to compete at the 2008” — China; “Complete the sentence by filling in the blank: Jasminum officinale, known as the common jasmine or simply jasmine, is a species of flowering” — China; “Complete the sentence by filling in the blank: It is a perennial that thrives under damp and shady conditions or open meadows in California, Arizona, New Mexico, Nevada, Oregon, Washington, , and Utah.” — Idaho; “Complete the sentence by filling in the blank: The terms ‘accounting’ and ‘ are often used as synonyms.” — financial reporting; “Complete the sentence by filling in the blank: The municipality is unilingually .” — Finnish; “Complete the sentence by filling in the blank: The agriculture, equine studies and .” — arboriculture; “Complete the sentence by filling in the blank: Therefore, very little Luxembourg wine is seen outside Luxembourg and .” — Belgium; “Complete the sentence by filling in the blank: They also include , and sometimes titanium.” — copper; “Complete the sentence by filling in the blank: The genus has been found in Europe (France, Germany, England), the Middle East” — Oregon; “Complete the sentence by filling in the blank: It is found in most of North America, including British Columbia, Florida, Illinois, ” — Minnesota; “Complete the sentence by filling in the blank: It borders the US states of the northeast, Wyoming to the east, Nevada and Utah” — Montana; “Complete the sentence by filling in the blank: It is found in Panama, Costa Rica, Mexico, Colombia, , Ecuador, Venezuela” — Peru; “Complete the sentence by filling in the blank: It is found in Nakhchivan, southern Turkey, Cyprus, Syria, Lebanon, Jordan, ” — Israel.

Instances with highest and lowest 5-nearest neighbor overlap across representations

Figure 21. Instances with highest (top) and lowest (bottom) 5-nearest neighbor overlap across representations nemot and rrepsimr_{\mathrm{repsim}} for Olmo-3–7B_tulu-3-sft-olmo-2-mixture-0225.

MaxMin
FTrace
r∇fidxr_{\nabla f}^{id x}rbm25r_{bm25}1/5 overlap. Both local neighborhoods consist mostly of instances whose completions are of the same semantic category as the test instance.For r∇fidxr_{\nabla f}^{id x}, one completion matches the test instance completion, with no clear relation between the test instance and the others. For rbm25r_{bm25}, there is one keyword match between the test instance prompt and a completion, with no obvious relation for the others.
r∇fidxr_{\nabla f}^{id x}rnemotr_{nemot}4/5 match. The single differing instance in rnemotr_{nemot} shares the same completion as the test instance, whereas the completion in r∇fidxr_{\nabla f}^{id x} differs but remains within the same semantic category.For min, both rnemotr_{nemot} and r∇fidxr_{\nabla f}^{id x} contain completions from the same semantic category. Three instances in rnemotr_{nemot} match the test instance completion.
r∇fidxr_{\nabla f}^{id x}rrepsimr_{repsim}3/5 match. for r∇fidxr_{\nabla f}^{id x} exact same instruction following goal of predicting pronouns, for rrepsimr_{repsim} not.Except for one example of r∇fidxr_{\nabla f}^{id x}, all examples across the test instance and both representations are of the same semantic category.
rbm25r_{bm25}rnemotr_{nemot}3/5 match. All rnemotr_{nemot} instances either match the test instance’s prompt or belong to the same semantic category. For rbm25r_{bm25}, one instance matches via keyword overlap, while the other appears unrelated.For rbm25r_{bm25}, no direct relation to the test instance is apparent, whereas all rnemotr_{nemot} instances belong to the same semantic category.
rbm25r_{bm25}rrepsimr_{repsim}5/5 match. All but one are of the same semantic category as the test instance.Three for rbm25r_{bm25} and one for rrepsimr_{repsim} are of the same semantic category as the test instance. The other examples appear unrelated, with rrepsimr_{repsim} providing three given names as completion, which resembles the test instance’s prompt.
rnemotr_{nemot}rrepsimr_{repsim}4/5 match. For rnemotr_{nemot}, the non-matching example is in the same semantic category as the other examples and the test instance; for rrepsimr_{repsim}, it is not.For rnemotr_{nemot}, 3/5 examples are of the same semantic category as the test instance. For rrepsimr_{repsim}, examples are only distantly related in topic.
Math
r∇fidxr_{\nabla f}^{id x}rbm25r_{bm25}2/5 match. No obvious difference between the examples of r∇fidxr_{\nabla f}^{id x} and rbm25r_{bm25}.As for max.
r∇fidxr_{\nabla f}^{id x}rnemotr_{nemot}2/5 match. The prompts for rnemotr_{nemot} appear more semantically related to the test instance’s prompt than r∇fidxr_{\nabla f}^{id x}.The prompts for rnemotr_{nemot} appear more semantically related to the test instance’s prompt than r∇fidxr_{\nabla f}^{id x}.
r∇fidxr_{\nabla f}^{id x}rrepsimr_{repsim}2/5 match. The matching instances both lack instruction on step-by-step thinking, while the remaining examples for both r∇fidxr_{\nabla f}^{id x} and rrepsimr_{repsim} include it.The examples for rrepsimr_{repsim} share in common that they do not decompose into steps, whereas the ones of r∇fidxr_{\nabla f}^{id x} decompose into subproblems, matching the test instance.
rbm25r_{bm25}rnemotr_{nemot}2/5 match. The matching examples provide completions that are lexically related to the test instance. No obvious relation for the examples of rbm25r_{bm25}. The descriptions of the personas employed by the non-matching examples of rnemotr_{nemot} are semantically related.Completion share identical introduction for rbm25r_{bm25}, with no obvious relation to the test instance. The description of the personas employed by the non-matching examples of rnemotr_{nemot} are semantically related.
rbm25r_{bm25}rrepsimr_{repsim}4/5 match. The formatting of the instance that differs for rbm25r_{bm25} is less similar to the test instance than that of rrepsimr_{repsim}.Similar formatting across instances of both representations and test instance, no obvious semantic relation to the test instance.
rnemotr_{nemot}rrepsimr_{repsim}2/5 match. Non-matching examples of rnemotr_{nemot} are semantically related to the test instance. No obvious relation for those of rrepsimr_{repsim}.Examples of rnemotr_{nemot} are semantically related to the test instance. No obvious relation to the test instance for those of rrepsimr_{repsim}.
Code
r∇fidxr_{\nabla f}^{id x}rbm25r_{bm25}1/5 matches. All examples share the same completion formatting, providing only a code block without any additional introduction. Only one example (r∇fidxr_{\nabla f}^{id x}) features a completion directly relevant to solving the problem stated by the test instance.All examples share the same completion formatting, and only one example (r∇fidxr_{\nabla f}^{id x}) features a completion directly relevant to solving the problem stated by the test instance.
r∇fidxr_{\nabla f}^{id x}rnemotr_{nemot}4/5 match. All examples across both representations are on the exact same coding task as the test instance. The non-matching example of r∇fidxr_{\nabla f}^{id x} differs from the test instance only in that another type of text is stated in the instruction, for rnemotr_{nemot} it matches the one of the test instance.The examples of r∇fidxr_{\nabla f}^{id x} do not appear to be semantically related to the test instance. All but one example of rnemotr_{nemot} are semantically related to and relevant for solving the test instance’s problem and match in terms of keywords.
MaxMin
r∇fidxr^{id x}_{\nabla f}rrepsimr_{repsim}3/5 match. The examples of rrepsimr_{repsim} appear more closely related to the test instance’s objective than the ones of r∇fidxr^{id x}_{\nabla f}.All examples of r∇fidxr^{id x}_{\nabla f} share a common completion formatting that does not match the test instance nor rrepsimr_{repsim}. No direct relation to test instance for r∇fidxr^{id x}_{\nabla f}, examples of rrepsimr_{repsim} are loosely of the same task category as the test instance.
rbm25r_{bm25}rnemotr_{nemot}2/5 match. For rnemotr_{nemot}, examples are semantically related to the test instance and match via keywords. Examples for rbm25r_{bm25} feature unrelated goals but match in terms of keywords.Examples for rbm25r_{bm25} match via keywords, those of rnemotr_{nemot} are more semantically related to the test instance’s task goal.
rbm25r_{bm25}rrepsimr_{repsim}2/5 match. Examples of rbm25r_{bm25} match via keywords, no obvious relation of the examples of rrepsimr_{repsim} to the test instance beyond similar completion formatting.All examples of rbm25r_{bm25} share a common completion formatting that does not match the test instance nor rrepsimr_{repsim}. Examples of rbm25r_{bm25} match in terms of keywords with the test instance, no obvious relation of the examples of rrepsimr_{repsim} to the test instance.
rnemotr_{nemot}rrepsimr_{repsim}3/5 match. All examples but one for rrepsimr_{repsim} across both representations are on the exact same coding task as the test instance.The description of the personas employed by rnemotr_{nemot} are semantically related to those in the test instance. For rrepsimr_{repsim}, all examples share the same completion formatting with the test instance but appear not semantically related to it.
squad_v2
r∇fidxr^{id x}_{\nabla f}rbm25r_{bm25}1/5 matches. No apparent relation of either set of examples to the test instance.As for max.
r∇fidxr^{id x}_{\nabla f}rnemotr_{nemot}2/5 match. Examples of r∇fidxr^{id x}_{\nabla f} appear unrelated to the test instance, only one example for rnemotr_{nemot} is semantically related.All completions for r∇fidxr^{id x}_{\nabla f} and all but two for rnemotr_{nemot} are examples for indicating incomplete information. The example that provides a response for rnemotr_{nemot} is semantically related to the test instance.
r∇fidxr^{id x}_{\nabla f}rrepsimr_{repsim}2/5 match. For r∇fidxr^{id x}_{\nabla f} all completions consist of numbers or indications of years, months or days, for rrepsimr_{repsim} one example is a geographical location.All examples are examples for indicating incomplete information, as is the test instance.
rbm25r_{bm25}rnemotr_{nemot}3/5 match. All examples of rnemotr_{nemot} are semantically related to the test instance, those of rbm25r_{bm25} appear unrelated.Examples of rbm25r_{bm25} appear unrelated, those of rnemotr_{nemot} are semantically related to the test instance.
rbm25r_{bm25}rrepsimr_{repsim}4/5 match. For both representations, examples either indicate incomplete information like the test instance, or belong to the same semantic task category.For rbm25r_{bm25}, one example features matching keywords with the test instance, two indicate incomplete information (unlike the test instance). The examples in rrepsimr_{repsim}, do not appear to be semantically related to the test instance, but two completions provide dates like the test instance. Two completions for rnemotr_{nemot} and two for rrepsimr_{repsim} belong to the same semantic category. For both representations, all prompts have keyword overlap or semantic similarity.
rnemotr_{nemot}rrepsimr_{repsim}2/5 match. All but one example (rrepsimr_{repsim}) belong to the same semantic category.
SFT
r∇fidxr^{id x}_{\nabla f}rbm25r_{bm25}1/5 match. None of the examples from either representation appears relevant or in the format of the test instance. Examples of rbm25r_{bm25} are non-English. Two examples of r∇fidxr^{id x}_{\nabla f} are examples for answer refusal, one is a labeling task for harmful data.For r∇fidxr^{id x}_{\nabla f}, one is a labeling task for harmful data, two are safety-related, both in line with the instruction tuning goal of the test instance. For rbm25r_{bm25}, examples are non-English and do not appear related to the test instance.
r∇fidxr^{id x}_{\nabla f}rnemotr_{nemot}3/5 match. All examples are of the same semantic category and share the refusal phrasing of the test instance almost verbatim.For r∇fidxr^{id x}_{\nabla f}, two examples contain information likely intended for the model to memorize verbatim, which appears to align with the instruction-tuning goal of the test instance. The remaining three are safety-related. For rnemotr_{nemot}, two examples also contain information likely intended for memorization in line with the test instance. Two examples are non-English.
r∇fidxr^{id x}_{\nabla f}rrepsimr_{repsim}4/5 match. All examples are of the same semantic category and share the refusal phrasing of the test instance almost verbatim.For r∇fidxr^{id x}_{\nabla f}, three examples are safety-related, and two appear unrelated to the test instance. For rrepsimr_{repsim}, all examples share the semantic task category and completion format of the test instance.
MaxMin
rbm25r_{bm25}rnemotr_{nemot}4/5 match. Two of the matching examples share the same task category with the test instance, an additional one for rnemotr_{nemot} does so as well, despite being the only English-language example.All examples except for one (rbm25r_{bm25}) share the same task category with the test instance for both representations.
rbm25r_{bm25}rrepsimr_{repsim}4/5 match. All examples are non-English, none appear to be relevant to the test instance.For rbm25r_{bm25}, only one example matches the task category of the test instance, for rrepsimr_{repsim}, all do.
rnemotr_{nemot}rrepsimr_{repsim}5/5 match. All examples match and are in line with the intended safety behavior demonstrated in the test instance.All examples share the task category of the test instance and follow the same completion format. The examples of rnemotr_{nemot} seem to be more semantically related to the test instance than the ones of rrepsimr_{repsim}.

Table 7. Full observations from the qualitative analysis, grouped by dataset.

∣N5(ri)(z)∩N5(rj)(z)∣5.\frac{\left|\mathcal{N}_5^{(r_i)}(z)\cap\mathcal{N}_5^{(r_j)}(z)\right|}{5}.

For each dataset, we retrieve and study neighborhoods of two documents zmax⁡z_{\max} and zmin⁡z_{\min}: one with maximal overlap among the top-5 examples (cf Figure 17: top), and one with minimal overlap (bottom). We break ties by random selection. Prior to computing inner-product similarity, we mean-center representations to remove global offsets in the embedding space, ensuring that neighborhoods reflect relative structure rather than absolute activation magnitudes.

Choice of representations. We restrict this analysis to 4 of the 12 representations. Given a CKA dissimilarity matrix over all candidates, we apply a greedy farthest-first selection procedure: starting from the most central representation, we iteratively add the most dissimilar remaining representation. This results in

  • rnemotr_{\mathrm{nemot}} (embedding model),

  • r∇fidXr_{\nabla_f}^{\mathrm{idX}} (raw loss gradients),

  • rbm25r_{\mathrm{bm25}}, and

  • repsimr_{\mathrm{epsim}} (model-internal embeddings).

Figure 18. Instances with highest (top) and lowest (bottom) 5-nearest neighbor overlap across representations rbm25r_{bm25} and rrepsimr_{repsim} for Olmo-3-7B_tulu-3-sft-personas-math.

Figure 19. Instances with highest (top) and lowest (bottom) 5-nearest neighbor overlap across representations r∇fidxr^{idx}_{\nabla f} and nemot for Olmo-3-7B_tulu-3-sft-personas-code.

Figure 20. Instances with highest (top) and lowest (bottom) 5-nearest neighbor overlap across representations r∇fiDxr^{iDx}_{\nabla f} and rbm25↑r^{\uparrow}_{bm25} for Olmo-3-7B_squad_v2_conversational.

Fine-tuning

We fine-tune on NVIDIA H100 and H200 GPUs for a total of 199.22 hours (456.53 GPU hours), adjusting gradient accumulation to ensure consistent effective batch sizes across runs. All models and samples are made available under CC-BY 4.0 in line with their original licenses.

Precisionbfloat16LoRA rank (rr)8OptimizerAdamW
Learning rate1×10−41 \times10^{-4}LoRA alpha32LR schedulerLinear
Weight decay0.0LoRA dropout0.1Effective batch size64
Max grad norm1.0LoRA biasnoneTraining epochs1
Trainable paramsLoRA onlyMax sequence length4096Gradient checkpointingTrue
Target modulesq_proj, k_proj, v_proj, o_projSeed42

Table 8. LoRA fine-tuning hyperparameters. We use the default parameters from the Hugging Face documentation, adjusting the target modules and effective batch size for our specific model architectures.

Task Suitesquad_v2_conv
Base Model
Llama-3.2-1B74.72
OLMo-2-0425-1B77.81
Olmo-3-1025-7B50.08
Qwen3-0.6B-Base80.33
Qwen3-1.7B-Base84.20
Qwen3-4B-Base87.74
Qwen3-8B-Base89.26

Table 9. Performance (F1) on held-out data of models fine-tuned on squad_v2.

Task Suiteftrace_conv
Base Model
Llama-3.2-1B39.10
OLMo-2-0425-1B36.96
Olmo-3-1025-7B9.93
Qwen3-0.6B-Base37.92
Qwen3-1.7B-Base48.18
Qwen3-4B-Base59.14
Qwen3-8B-Base61.80

Table 10. Performance (F1) on held-out data of models fine-tuned on ftrace.

Task Suitecode_bpb
Base Model
Llama-3.2-1B0.42
OLMo-2-0425-1B0.52
Olmo-3-1025-7B0.32
Qwen3-0.6B-Base0.35
Qwen3-1.7B-Base0.31
Qwen3-4B-Base0.29
Qwen3-8B-Base0.28

Table 11. Performance on OLMES task suite olmo3:base_easy:code_bpb.

Task Suitemath_bpb
Base Model
Llama-3.2-1B0.62
OLMo-2-0425-1B0.53
Olmo-3-1025-7B0.37
Qwen3-0.6B-Base0.43
Qwen3-1.7B-Base0.38
Qwen3-4B-Base0.34
Qwen3-8B-Base0.33

Table 12. Performance on OLMES task suite olmo3:base_easy:math_bpb.

Task Suite Base ModelMMLUolmo_2_generativeolmo_2_heldout
Llama-3.2-1B0.290.460.25
OLMo-2-0425-1B0.430.540.26
Olmo-3-1025-7B0.670.710.54
Qwen3-0.6B-Base0.540.470.37
Qwen3-1.7B-Base0.650.570.47
Qwen3-4B-Base0.750.650.62
Qwen3-8B-Base0.790.690.67

Table 13. Macro-average score across OLMES task suites: mmlu:mc::olmes, olmo_2_generative::olmes, and olmo_2_heldout::olmes for models fine-tuned on allenai/tulu-3-sft-olmo-2-mixture-0225.

References

  1. [1]Ekin Akyurek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. 2022. Towards Tracing Knowledge in Language Models Back to the Training Data. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2429–2446, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  2. [2]Yauhen Babakhin, Radek Osmulski, Ronay Ak, Gabriel Moreira, Mengyao Xu, Benedikt Schifferer, Bo Liu, and Even Oldridge. 2025. Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks. arXiv preprint.arxiv.org/abs/2511.07025
  3. [3]Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B. Grosse. 2022. If Influence Functions are the Answer, Then What is the Question? Advances in Neural Information Processing Systems, 35:17953–17967.arxiv.org/abs/2209.05364
  4. [4]Jonathan Brophy, Zayd Hammoudeh, and Daniel Lowd. 2024. Adapting and evaluating influence-estimation methods for gradient-boosted decision trees. J. Mach. Learn. Res., 24(1):154:7325–154:7372.
  5. [5]Junwei Deng, Yuzheng Hu, Pingbang Hu, Ting-wei Li, Shixuan Liu, Jiachen T. Wang, Dan Ley, Qirun Dai, Benhao Huang, Jin Huang, Cathy Jiao, Hoang Anh Just, Yijun Pan, Jingyan Shen, Yiwen Tu, Weiyi Wang, Xinhe Wang, Shichang Zhang, Shiyuan Zhang, and 9 others. 2025. A Survey of Data Attribution: Methods, Applications, and Evaluation in the Era of Generative AI. Preprint, Social Science Research Network:5451054.DOI
  6. [6]Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravanakumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 516 others. 2024. The Llama 3 Herd of Models. Preprint, arXiv:2407.21783.arxiv.org/abs/2407.21783
  7. [7]Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Diganta Misra, Shreeya Dhakal, Jonathan Rystrøm, Roman Solomatin, Ömer Veysel Çağatan, and 63 others. 2024. MMTEB: Massive Multilingual Text Embedding Benchmark. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2502.13595
  8. [8]Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groenveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh, Pradeep Dasigi, and 48 others. 2025. Olmo 3. Preprint, arXiv:2512.13961.arxiv.org/abs/2512.13961
  9. [9]Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Dumrus, Ethan Perez, Evan Hubinger, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. 2023. Studying Large Language Model Generalization with Influence Functions. Preprint, arXiv:2308.03296.arxiv.org/abs/2308.03296
  10. [10]Peijian Gu, Yaozong Shen, Lijie Wang, Quan Wang, Hua Wu, and Zhendong Mao. 2023. JAEval: A Comprehensive Evaluation of Instance Attribution on Natural Language Understanding. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11966–11977, Singapore. Association for Computational Linguistics.DOI
  11. [11]Zayd Hammoudeh and Daniel Lowd. 2024. Training data influence analysis and estimation: A survey. Machine Learning, 113(5):2351–2403.DOI
  12. [12]Xiaochuang Han, Byron C. Wallace, and Yulia Tsvetkov. 2020. Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5553–5563, Online. Association for Computational Linguistics.arxiv.org/abs/2005.06676
  13. [13]Kazuaki Hanawa, Sho Yokoi, Satoshi Hara, and Kentaro Inui. 2020. Evaluation of Similarity-based Explanations. In International Conference on Learning Representations.arxiv.org/abs/2006.04528
  14. [14]Dat Minh Hong, Bruno Kacper Mlodozieniec, Runa Eschenhagen, and Richard E. Turner. 2025. Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functions. In Mechanistic Interpretability Workshop at NeurIPS 2025.arxiv.org/abs/2509.23437
  15. [15]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations.arxiv.org/abs/2106.09685
  16. [16]Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. 2024. Position: The Platonic Representation Hypothesis. In Proceedings of the 41st International Conference on Machine Learning, pages 20617–20642. PMLR.arxiv.org/abs/2405.07987
  17. [17]Andrew Ilyas and Logan Engstrom. 2025. MAGIC: Near-Optimal Data Attribution for Deep Learning. Preprint, arXiv:2504.16430.arxiv.org/abs/2504.16430
  18. [18]Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543–3556, Minneapolis, Minnesota. Association for Computational Linguistics.
  19. [19]Cathy Jiao, Weizhen Gao, Aditi Raghunathan, and Chenyan Xiong. 2025a. On the Feasibility of In-Context Probing for Data Attribution. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5140–5155, Albuquerque, New Mexico. Association for Computational Linguistics.
  20. [20]Cathy Jiao, Yijun Pan, Emily Xiao, Daisy Sheng, Niket Jain, Hanzhang Zhao, Ishita Dasgupta, Jiaqi W. Ma, and Chenyan Xiong. 2025b. DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
  21. [21]Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is Not Only a Weight: Analyzing Transformers with Vector Norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7057–7075, Online. Association for Computational Linguistics.arxiv.org/abs/2004.10102
  22. [22]Pang Wei Koh and Percy Liang. 2017. Understanding Black-box Predictions via Influence Functions. In Proceedings of the 34th International Conference on Machine Learning, pages 1885–1894. PMLR.arxiv.org/abs/1703.04730
  23. [23]Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of Neural Network Representations Revisited. In Proceedings of the 36th International Conference on Machine Learning, pages 3519–3529. PMLR.arxiv.org/abs/1905.00414
  24. [24]Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. 2023. DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models. In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2310.00902
  25. [25]Jarkko Lagus, Janne Sinkkonen, and Arto Klami. 2019. Low-Rank Approximations of Second-Order Document Representations. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 634–644, Hong Kong, China. Association for Computational Linguistics.DOI
  26. [26]Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Validad Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Reka D. Hwang, Jiangiang Yang, Ronan Le Bras, Oyvind Tafjord, Christopher Wilhelm, Luca Soldaini, and 4 others. 2025. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. In Second Conference on Language Modeling.arxiv.org/abs/2411.15124
  27. [27]Zhe Li, Wei Zhao, Yige Li, and Jun Sun. 2025. Do Influence Functions Work on Large Language Models? In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 14367–14382, Suzhou, China. Association for Computational Linguistics.arxiv.org/abs/2409.19998
  28. [28]Jiacheng Liu, Taylor Blanton, Yanai Elazar, Sewon Min, Yen-Sung Chen, Arnavi Chheda-Kothary, Huy Tran, Byron Bischoff, Eric Marsh, Michael Schmitz, Cassidy Trier, Aaron Sarnat, Jenna James, Jon Borchardt, Bailey Kuehl, Evie Yu-Yen Cheng, Karen Farley, Taira Anderson, David Albright, and 11 others. 2025. OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 178–188, Vienna, Austria. Association for Computational Linguistics.
  29. [29]Bruno Kacper Mlodozeniec, Runa Eschenhagen, Juhan Bae, Alexander Immer, David Krueger, and Richard E. Turner. 2024. Influence Functions for Scalable Data Attribution in Diffusion Models. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2410.13850
  30. [30]Shinichi Nakagawa and Holger Schielzeth. 2013. A general and simple method for obtaining R2 from generalized linear mixed-effects models. Methods in Ecology and Evolution, 4(2):133–142.DOI
  31. [31]Michael Neely, Stefan F. Schouten, Maurits Bleeker, and Ana Lucic. 2022. A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processing. In HHAI2022: Augmenting Human Intellect, pages 60–78. IOS Press.arxiv.org/abs/2205.04559
  32. [32]Elisa Nguyen, Minjoon Seo, and Seong Joon Oh. 2023. A Bayesian Approach To Analysing Training Data Attribution In Deep Learning. Advances in Neural Information Processing Systems, 36:641455–64180.arxiv.org/abs/2305.19765
  33. [33]Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclecq, and Aleksander Madry. 2023. TRAK: Attributing Model Behavior at Scale. In Proceedings of the 40th International Conference on Machine Learning, pages 27074–27113. PMLR.arxiv.org/abs/2303.14186
  34. [34]Pouya Pezeshkpour, Sarthak Jain, Byron Wallace, and Sameer Singh. 2021. An Empirical Comparison of Instance Attribution Methods for NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 967–975, Online. Association for Computational Linguistics.
  35. [35]Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. 2020. Estimating Training Data Influence by Tracing Gradient Descent. In Advances in Neural Information Processing Systems, volume 33, pages 19920–19930. Curran Associates, Inc.arxiv.org/abs/2002.08484
  36. [36]Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know What You Don’t Know: Unanswerable Questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Linguistics.arxiv.org/abs/1806.03822
  37. [37]R. Sokal and C. Michener. 1958. A statistical method for evaluating systematic relationships. University of Kansas science bulletin.DOI
  38. [38]Robert R. Sokal and F. James Rohlf. 1962. The Comparison of Dendrograms by Objective Methods. Taxon, 11(2):33–40.DOI
  39. [39]Jingyi Sun, Pepa Atanasova, and Isabelle Augenstein. 2025. Evaluating Input Feature Explanations through a Unified Diagnostic Evaluation Framework. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 10559–10577, Albuquerque, New Mexico. Association for Computational Linguistics.arxiv.org/abs/2406.15085
  40. [40]Weiwei Sun, Haokun Liu, Nikhil Kandpal, Colin Raffel, and Yiming Yang. 2026. Enhancing Training Data Attribution with Representational Optimization. Advances in Neural Information Processing Systems, 38:102274–102311.arxiv.org/abs/2505.18513
  41. [41]Evan Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Allyson Ettinger, and 23 others. 2025. OLMo 2 Furious (OLMo’s Version). In Second Conference on Language Modeling.arxiv.org/abs/2501.00656
  42. [42]Kangxi Wu, Liang Pang, Huawei Shen, and Xueqi Cheng. 2024. Enhancing Training Data Attribution for Large Language Models with Fitting Error Consideration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14131–14143, Miami, Florida, USA. Association for Computational Linguistics.
  43. [43]Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024. LESS: Selecting Influential Data for Targeted Instruction Tuning. In Proceedings of the 41st International Conference on Machine Learning, pages 54104–54132. PMLR.arxiv.org/abs/2402.04333
  44. [44]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Cheng'en Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 Technical Report. Preprint, arXiv:2505.09388.arxiv.org/abs/2505.09388
  45. [45]Chih-Kuan Yeh, Ankur Taly, Mukund Sundararajan, Frederick Liu, and Pradeep Ravikumar. 2022. First is Better Than Last for Language Data Influence. Advances in Neural Information Processing Systems, 35:32285–32298.arxiv.org/abs/2202.11844
  46. [46]Haeun Yu, Pepa Atanasova, and Isabelle Augenstein. 2024. Revealing the Parametric Knowledge of Language Models: A Unified Framework for Attribution Methods. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8173–8186, Bangkok, Thailand. Association for Computational Linguistics.
  47. [47]Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. arXiv preprint.arxiv.org/abs/2506.05176

Paper details

vlnc:W_wjtvveh5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents