WorksComputer Science
On the Sparsity of Direct Preference Optimization: Weight Disentanglement in the NTK Regime
Abstract
Direct Preference Optimization (DPO) has been widely adopted for aligning language models with human preferences. Recent empirical work has revealed that DPO induces remarkably sparse parameter updates, yet no theoretical explanation for this phenomenon has been offered. We analyze DPO's optimization dynamics under a neural tangent kernel linearization and show that the logistic structure of the DPO loss gives rise to an implicit maximum-margin bias. Building on this observation, we introduce the concept of dual sparsity: the DPO update direction is a sparse linear combination (data-level sparsity, driven by support vectors) of sparse gradient feature vectors (parameter-level sparsity, arising from gradient cancellation between preferred and dispreferred responses). We formalize this structure in our main theorem, bound the number of support pairs by a margin-dependent spectral quantity, and present a framework relating dual sparsity to weight disentanglement—a condition favorable for model merging. Experiments on GPT-2 Small and Medium and on 7–9B instruction-tuned models (Llama-3-8B, Mistral-7B, Gemma-2-9B) with the Anthropic HH-RLHF and UltraFeedback datasets confirm both sources of sparsity, and experiments on GPT-2 show that DPO reduces cross-task interference by up to compared to supervised fine-tuning.
Introduction
Direct Preference Optimization [14] has emerged as a dominant paradigm for aligning large language models with human preferences. By casting the alignment objective as a logistic loss over preference pairs, DPO circumvents the instabilities of reinforcement learning from human feedback while retaining strong empirical performance. As practitioners increasingly train separate DPO models for distinct objectives—helpfulness, harmlessness, instruction following—there is growing interest in composing these independently trained models via model merging [8] (Yu et al. 2024), which combines task-specific parameter updates (task vectors) into a single multitask model without additional training.
A striking but poorly understood empirical observation is that DPO fine-tuning induces highly sparse parameter updates. Mukherjee et al. [ [11] recently demonstrated that reinforcement learning-based alignment concentrates its modifications onto a small subnetwork of the pre-trained model, leaving the vast majority of parameters essentially unchanged. This sparsity stands in marked contrast to supervised fine-tuning (SFT), which typically distributes updates broadly across the parameter space. Independently, Ortiz-Jiménez et al. [ [12] identified weight disentanglement—the condition that task vectors act on non-overlapping regions of parameter space—as a key property enabling successful model merging. A natural question arises: is the sparsity of DPO a mere coincidence, or does it reflect a deeper structure in the optimization that is systematically favorable for composing task vectors?
Preprint.
In this paper, we provide a theoretical answer to this question. We analyze the implicit bias of DPO under a neural tangent kernel (NTK) linearization [9] and show that the logistic loss structure, combined with the geometry of preference-pair gradients, gives rise to a phenomenon we call dual sparsity. The DPO update direction converges to the maximum-margin solution in the NTK feature space, inheriting two complementary forms of sparsity: data-level sparsity, where only a small subset of preference pairs (support vectors) contribute to the final update, and parameter-level sparsity, where the gradient feature vector for each pair is itself sparse due to gradient cancellation between preferred and dispreferred responses that share common tokens. Building on this, we present a framework relating dual sparsity to weight disentanglement, and verify empirically that DPO produces cross-task inner products an order of magnitude smaller than those under SFT.
Our main contributions are as follows:
(C1) We reduce DPO to maximum-margin classification under NTK linearization and introduce dual sparsity—a joint data-level and parameter-level sparsity structure—formalized in theorem 3. This provides the first theoretical explanation for why DPO induces sparse parameter updates without any explicit sparsity-promoting regularization.
(C2) We present a framework relating dual sparsity to weight disentanglement [12] via a triplet-space formulation of the DPO margin (§ 3.3). The framework reduces the disentanglement condition to a per-triplet inner product whose magnitude is governed by the parameter-space overlap between tasks—an empirical question that we evaluate in § 4.
(C3) We validate both sources of sparsity on GPT-2 Small (124M) and Medium (355M) fine-tuned with the Anthropic HH-RLHF [2] and UltraFeedback [3] datasets, and at modern scale on Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.2, and Gemma-2-9B-it. DPO feature vectors are over 90% near-zero from 124M to 9B parameters, and the time-integrated weights concentrate on a small subset of pairs within three epochs of full fine-tuning at the 7–8B scale. On GPT-2, cross-task interference under DPO is reduced by 11× to 35× relative to SFT across all configurations.
The remainder of the paper is organized as follows. § 2 reviews the DPO formulation and weight disentanglement framework. § 3 presents our theoretical analysis, culminating in the dual sparsity theorem and a framework connecting it to weight disentanglement. § 4 provides experimental validation. § 5 discusses related work, and § 6 concludes with limitations and future directions.
Preliminaries
Direct Preference Optimization
Let denote a reference policy, typically a pre-trained or supervised fine-tuned (SFT) model [14]. Given a prompt and a pair of responses where is preferred and is dispreferred, the log-probability margin under parameters is
with temperature . For a preference dataset , the DPO loss is the logistic loss
whose gradient w.r.t. is weighted by : the gradient vanishes for “easy” pairs already preferred by the policy, focusing optimization on the “hard” subset near the decision boundary.
Weight Disentanglement
We review the notion of weight disentanglement [12], which characterizes an ideal condition for composing independently trained task vectors without cross-task interference.
Consider a neural network with pre-trained parameters . A task is specified by a data support , an input distribution with , and a target function . Fine-tuning on task yields parameters , and the corresponding task vector is defined as .
Definition 1 (Task Arithmetic, Ilharco et al., [8]). Let be a set of task vectors with mutually disjoint supports for . The network satisfies the task arithmetic property around if
Definition 2 (Weight Disentanglement, Ortiz-Jiménez et al., [12]). The network is weight-disentangled with respect to and if there exist functions with for and a residual with for , such that
It is straightforward to verify that weight disentanglement (definition 2) implies the task arithmetic property (definition 1) by setting and .
Theoretical Analysis
Recent empirical work has shown that alignment fine-tuning via DPO induces highly sparse parameter updates relative to the pre-trained weights [11]. Independently, weight disentanglement has been identified as a desirable property for robust model merging [12]. The following questions remain open:
Why does DPO induce sparsity? What mechanism in the DPO optimization dynamics leads to sparse parameter updates, despite the absence of any explicit sparsity-promoting regularization?
What are the implications of this sparsity for cross-task interference? Specifically, does the sparsity inherent in DPO updates reduce the interference between independently trained task vectors?
This section addresses both questions. We analyze the implicit bias of DPO under a neural tangent kernel (NTK) linearization (§ 3.1) and identify a dual sparsity structure that emerges from the interplay between the logistic loss and the geometry of the gradient features (§ 3.2). We then present a framework relating this structure to weight disentanglement, with empirical evaluation deferred to section 4 (§ 3.3).
DPO as Maximum-Margin Classification in Tangent Space
We begin by linearizing the DPO margin around the initial parameters . Under the neural tangent kernel (NTK) regime [9], the margin admits a first-order approximation:
where and the gradient feature vector is defined as
Here denotes the total number of parameters. Each encodes the direction in parameter space along which a perturbation most effectively increases the preference margin for pair . Under this linearization, the DPO loss reduces to a binary logistic regression over the features with offsets :
With the standard DPO initialization , the offsets vanish: by eq. (1).
We make the following assumption, which is standard in the analysis of implicit bias for logistic regression [17]:
Assumption 1 (Linear Separability). There exist and such that for all .
Under assumption 1, the gradient flow on the total loss starting from admits the explicit trajectory
where is the time-integrated gradient weight for pair . Applying the implicit-bias result of Soudry et al. [ [17] to this setup yields:
Lemma 1 (Implicit Bias of DPO). Under assumption 1, the normalized parameter update converges to the maximum-margin direction:
By the Karush–Kuhn–Tucker (KKT) conditions, admits the representation
where is the set of support pairs.
Remark 1 (Empirical observability of dual variables). The normalized time-integrated coefficients also converge to the dual variables as . This justifies tracking as an empirical proxy for the support set in § 4.2.
Dual Sparsity of DPO Updates
Lemma 1 establishes that the DPO update direction is fully determined by the support pairs and the corresponding feature vectors. We now identify two distinct sources of sparsity in this representation.
Definition 3 (Dual Sparsity). The DPO update direction is said to exhibit dual sparsity if the following two conditions hold:
(i) Data-level sparsity: The number of support pairs is small relative to the dataset, i.e., .
(ii) Parameter-level sparsity: Each gradient feature vector is sparse, i.e., for .
The data-level sparsity is a direct consequence of the maximum-margin structure:
Lemma 2 (Data-Level Sparsity). Suppose (so that ) and assumption 1 holds. Let , with , be the Gram matrix of , and let be the maximum margin. Then, for the linearized model eq. (7)*,
where is the principal submatrix of indexed by SV.
The proof is given in section A.1.
This bound is meaningful because, in the DPO setting, the gradient flow concentrates its cumulative weight on the pairs that remain difficult throughout training—those near the margin boundary—while the remaining pairs receive diminishing gradient signal. Since the bound is governed by the ratio rather than explicitly by , the effective support set is probably rather than merely whenever the margin is large relative to the spectral scale of the features; we confirm empirically that this is the case on real preference data (see § 4.2).
For the parameter-level sparsity, we introduce the following assumption:
Assumption 2 (Sparsity of Gradient Features). For each , the gradient feature vector satisfies for some .
Remark 2 (Motivation for assumption 2). In autoregressive language models, the preferred and dispreferred responses and for a given prompt typically share a substantial number of tokens—particularly in the initial portion of the response. Since is defined as the difference of gradients , the parameter-level contributions from shared tokens cancel, leaving nonzero entries only in the parameters responsible for distinguishing between and . This cancellation effect provides a structural basis for assumption 2. We verify this assumption empirically in section 4.
Combining lemmas 1 and 2 with assumption 2, we arrive at the main result:
Theorem 3 (Dual Sparsity of DPO). Under assumptions 1 and 2, the DPO update direction exhibits dual sparsity in the sense of definition 3. That is, is a sparse linear combination of sparse vectors:
Consequently, is itself sparse with .
Connection to Weight Disentanglement
Theorem 3 establishes that DPO updates exhibit dual sparsity. We now examine how this structure relates to the weight disentanglement framework reviewed in section 2.2. The original formulation of weight disentanglement (definition 2) is stated for a generic function ; applying it to the DPO setting requires identifying the appropriate input space and function in preference learning.
Triplet-space formulation. We take the input space to be the space of preference triplets and define as the DPO log-probability margin
Differentiating at recovers the gradient feature vector of section 3.1: . The same features that drive dual sparsity therefore serve as the Jacobian relevant for weight disentanglement. A task is associated with a preference dataset collected under task-specific annotation criteria.
Linearized decomposition. For task vectors and coefficients , a first-order Taylor expansion of around gives
Equation (14) matches the structural form of the weight disentanglement decomposition (definition 2) under the identification
With this identification, the disentanglement condition for reduces to the inner-product condition
Observation 1 (Linearized weight disentanglement under DPO). Under the NTK linearization, the DPO margin admits the decomposition eq. (14), and the identification eq. (15) places this decomposition in the form of weight disentanglement (definition 2). The disentanglement condition holds to the extent that the cross-task inner product eq. (16) is small.
Connection to dual sparsity. Lemma 1 establishes that, in the limit, the normalized task vector for task aligns with the maximum-margin direction . Substituting into eq. (15), the local function takes the form
The cross-task interference eq. (16) is therefore governed by for . By theorem 3, is a sparse combination of sparse vectors; assumption 2 ensures is also sparse. When the active parameter supports of and have small overlap, this inner product is small. Whether the overlap is in fact small for naturally occurring preference tasks is an empirical question, which we examine in section 4.
Remark on the disjoint-support assumption. Definition 1 assumes disjoint task supports . In the triplet-space formulation this is naturally satisfied: distinct preference datasets arise from distinct annotation processes, and an ordered triplet is tied to the task it was annotated under.
Experiments
We empirically validate the theoretical predictions of section 3. Specifically, we verify (i) data-level sparsity of the dual coefficients (lemma 2), (ii) parameter-level sparsity of the gradient features (assumption 2), and (iii) suppression of cross-task interference under DPO compared to SFT (section 4.4).
Experimental Setup
Models. We use GPT-2 Small and GPT-2 Medium [13], together with three instruction-tuned models with 7–9B parameters: Llama-3-8B-Instruct [7], Mistral-7B-Instruct-v0.2 [10], and Gemma-2-9B-it [5]. Each checkpoint serves as both the initialization and the reference policy, .
Datasets. For our primary experiments, we use the Anthropic HH-RLHF dataset [2], which consists of two subsets: helpful (evaluating which response is more useful to the user) and harmless (evaluating which response avoids harmful content). These two distinct subsets serve as the tasks and in our cross-task interference analysis. To assess generality, we additionally use the UltraFeedback Binarized dataset [3], a large-scale preference dataset annotated by GPT-4 across four evaluation axes (instruction-following, truthfulness, honesty, and helpfulness).
Gradient features. The gradient feature vectors are computed empirically using the Jacobian of the model at initialization, since the analytical NTK cannot be obtained in closed form for self-attention architectures. Because storing full-dimensional gradient vectors ( with ) is memory-intensive, experiments that require are performed on subsamples of the datasets (see details in section B).
The models, datasets, and numbers of preference pairs used in each experiment, together with the computation of the time-integrated weights , are detailed in section B.
Data-Level Sparsity
Lemma 2 predicts that the DPO update direction is driven by a small number of support pairs, with the remaining pairs contributing negligibly. To verify this, we examine how the time-integrated weights are distributed across pairs as training progresses.
Figure 1 visualizes for GPT-2 Small and Medium on pairs from HH-RLHF at 1, 100, and 1000 epochs. At early stages, the weights are distributed nearly uniformly. As training progresses, the distribution becomes increasingly polarized: a small subset of pairs accumulates large weights while the rest stop growing (see section C for the corresponding log-probabilities). This is the behavior predicted by lemma 1, under which the normalized coefficients converge to the dual variables , which are nonzero only for the support pairs.
Table 1 quantifies this concentration on pairs by the top-10% mass, i.e., the fraction of held by the top 10% of pairs (10% under a uniform distribution). For GPT-2, the top-10% mass grows from about 11% to 40–44% over 1000 epochs; for Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.2, it grows from about 11% to 44–52% within three epochs of full fine-tuning DPO. The numerical evaluation of the bound in lemma 2 in the setting of fig. 1 is reported in section A.2.
| Model | Dataset | Top-10% mass (%) | ||
| Epoch 1 | Epoch 100 | Epoch 1000 | ||
| GPT-2 Small | HH-RLHF | 10.88 | 28.78 | 39.57 |
| GPT-2 Medium | HH-RLHF | 11.02 | 32.32 | 43.70 |
| Epoch 1 | Epoch 2 | Epoch 3 | ||
| Llama-3-8B-Instruct | UltraFeedback | 11.75 | 41.82 | 52.48 |
| Mistral-7B-Instruct-v0.2 | UltraFeedback | 11.23 | 39.92 | 43.77 |
Table 1. Top-10% mass of on pairs, i.e., the fraction of held by the top 10% of pairs (10% under a uniform distribution). Top: GPT-2 trained with DPO on HH-RLHF. Bottom: 7–8B models trained with full fine-tuning DPO on UltraFeedback Binarized.
Parameter-Level Sparsity
Assumption 2 posits that the DPO gradient feature vectors are sparse due to gradient cancellation between preferred and dispreferred responses. To verify this and to contrast it with SFT, Figure 2 visualizes the element-wise distribution of a representative under both methods (GPT-2 Medium, HH-RLHF, ), and Table 2 quantifies the fraction of near-zero entries for all five models (, UltraFeedback Binarized).

Figure 2. Element-wise distribution of gradient feature vectors on a logarithmic scale for GPT-2 Medium on pairs from HH-RLHF. (a) DPO: the vast majority of entries are near zero, confirming assumption 2. (b) SFT: the distribution is substantially denser, since no gradient cancellation occurs. Near-zero fractions for all five models, computed in a separate setting, are reported in Table 2.
| Model | Params | DPO (%) | SFT (%) |
| GPT-2 Small | 124M | 91.29 | 3.86 |
| GPT-2 Medium | 355M | 94.82 | 4.34 |
| Llama-3-8B-Instruct | 8B | 92.51 | 16.44 |
| Mistral-7B-Instruct-v0.2 | 7B | 90.93 | 19.72 |
| Gemma-2-9B-it | 9B | 92.12 | 13.57 |
Table 2. Fraction of near-zero entries in a representative gradient feature vector under DPO () and SFT (), computed from 1,000 pairs sampled from UltraFeedback Binarized. Higher values indicate sparser features.
Under DPO (Figure 2a), the vast majority of entries are concentrated near zero, with only a small fraction taking on appreciable magnitude. Under SFT (Figure 2b), the distribution is substantially broader, reflecting the fact that the SFT feature vector involves only the preferred response and does not benefit from the gradient cancellation that arises in the DPO difference . The same contrast holds at every scale we test (Table 2): from 124M to 9B parameters, more than 90% of the entries of the DPO feature vector are near zero, whereas only 4–20% of those of the SFT feature vector are.
As discussed in remark 2, this contrast arises naturally from the structure of preference data in autoregressive models: since and typically share a large number of tokens—particularly in the initial portion of the response—the parameter-level gradients for these shared tokens cancel in the DPO feature vector, leaving nonzero contributions only for the parameters that distinguish between the two responses.
Cross-Task Interference
We evaluate whether the dual sparsity of DPO updates translates into reduced cross-task interference, as discussed in § 3.3. We consider two task pairs of increasing semantic distance:
(a) HH helpful ↔ harmless: two subsets of the same dataset (Anthropic HH-RLHF) with different annotation criteria—helpfulness vs. harmlessness.
(b) HH helpful ↔ UltraFeedback: two entirely different datasets with different annotation methods (human vs. AI), different prompt distributions, and different preference dimensions.
For the cross-dataset pair (b), only the gradient features of UltraFeedback at are required; no DPO training on UltraFeedback is performed.
Baseline. We compare DPO against SFT. Unlike DPO, SFT does not produce the logistic loss structure that gives rise to implicit maximum-margin bias; consequently, all data pairs contribute to the update direction, and data-level sparsity does not emerge. Moreover, the SFT feature vector involves only the preferred response, so it does not benefit from the gradient cancellation that induces parameter-level sparsity in DPO.
Metrics. The disentanglement condition derived in § 3.3 is a per-triplet condition for (eq. (16)). Verifying it directly would require evaluating for every , which is computationally prohibitive given the scale of with . We therefore aggregate the per-triplet quantity into a single scalar summary by introducing the mean feature vector for task :
On this aggregated quantity, we evaluate two measures of cross-task overlap between and :
Top-1% Jaccard overlap: the Jaccard coefficient between the index sets of the top 1% largest-magnitude entries in each mean feature vector. This captures the degree of overlap between the most active parameters of the two tasks—the structural condition under which the per-triplet inner product is small (§ 3.3).
Interference : the inner product between the DPO update direction for and the mean feature vector for . This is the average of over and serves as a tractable summary of the per-triplet disentanglement condition eq. (16).
Results. Table 3 reports the results. DPO suppresses cross-task interference by an order of magnitude or more in every configuration. Two qualitative patterns are notable: under DPO, both the Jaccard overlap and the interference decrease at the larger scale (e.g., 0.258 0.191 on the within-dataset pair and 1.91 0.852 on the cross-dataset pair), consistent with the sharper polarization of observed in § 4.2; and the same pattern holds for the cross-dataset pair (HH helpful UltraFeedback), demonstrating that the effect generalizes beyond subsets of a single dataset to entirely independent preference data sources. These observations empirically support the framework of § 3.3: DPO’s dual sparsity, combined with the small parameter-space overlap exhibited by semantically distinct preference tasks, yields cross-task inner products an order of magnitude smaller than those under SFT.
| Task Pair | Method | Model | Jaccard (%) | |
| HH helpharm | SFT | Small | 54.9 | 9.12 |
| HH helpharm | DPO | Small | 18.9 | 0.258 |
| HH helpharm | SFT | Medium | 49.3 | 3.22 |
| HH helpharm | DPO | Medium | 11.4 | 0.191 |
| HH helpUF | SFT | Small | 68.4 | 21.2 |
| HH helpUF | DPO | Small | 25.2 | 1.91 |
| HH helpUF | SFT | Medium | 60.5 | 19.8 |
| HH helpUF | DPO | Medium | 17.7 | 0.852 |
Table 3. Cross-task interference for DPO and SFT across two model scales (GPT-2 Small/Medium) and two task pairs: HH helpharm (HH-RLHF helpful vs. harmless subsets) and HH helpUF (HH-RLHF helpful vs. UltraFeedback). Lower values indicate greater disentanglement. Across all configurations, DPO reduces the interference by 11 to 35 relative to SFT.
Model merging. To examine whether reduced interference carries over to merged models, we merge task vectors obtained by DPO or SFT on each task pair as , and measure the degradation of each task’s performance relative to its single-task model (section D). As shown in Table 4, DPO yields a smaller than SFT for both tasks in both task pairs, consistent with the reduced interference in Table 3.
| Task Pair | Method | Task 1 | Task 2 |
| HH help↔harm | SFT | 0.110 | 0.102 |
| HH help↔harm | DPO | 0.027 | 0.026 |
| HH help↔UF | SFT | 0.096 | 0.030 |
| HH help↔UF | DPO | 0.036 | 0.014 |
Table 4. Performance degradation (single-task − merged; lower is better) under task-vector merging on GPT-2 Medium. Task 1 is HH helpful; Task 2 is HH harmless or UltraFeedback. Full results are reported in section D.
Related Work
Optimization dynamics of DPO. Since Rafailov et al. [ [14] introduced DPO as a stable alternative to reinforcement learning from human feedback, several works have analyzed its dynamics theoretically. One line modifies the loss to address specific failure modes—IPO [6] avoids the Bradley–Terry assumption, and KTO [4] adopts a prospect-theoretic utility—but these are orthogonal to our analysis, which concerns the original objective. Closer to Ren and Sutherland [ [16] linearize DPO under the NTK regime and use the resulting framework to characterize the squeezing effect, a counter-intuitive decrease in the probability of preferred responses also studied by Razin et al. [ [15] under the term likelihood displacement. We adopt the same NTK lens but extract a structurally distinct property: the direction of the update in parameter space. Combining the classical implicit-bias result of Soudry et al. [ [17] for separable logistic regression with the specific geometry of preference-pair gradients yields dual sparsity (theorem 3), which in turn provides a theoretical explanation for the small-subnetwork behavior observed empirically by Mukherjee et al. [ [11] on RL-based alignment. Likelihood displacement and dual sparsity are thus two complementary consequences of the same dynamics: one concerning the probabilities the update changes, the other the parameters it changes.
Task arithmetic, weight disentanglement, and model merging. Ilharco et al. [ [8] showed that linear arithmetic on task vectors compositionally steers model behavior, and Ortiz-Jiménez et al. [ [12] formalized weight disentanglement as a sufficient condition for this property, identifying via NTK tangent-space analysis that task vectors should act on disjoint regions of parameter space. A complementary line of work—DARE [Yu et al., 2024] and TIES-Merging [Yadav et al., 2023]—engineers sparsity into task vectors as a post-hoc intervention to suppress interference during merging. Our framework offers a different perspective: under DPO, sparsity emerges intrinsically from the optimization dynamics, and the resulting parameter-space overlap between semantically distinct preference tasks is small enough that the disentanglement condition holds approximately without any explicit intervention (§ 4.4). Empirically, merging independently preference-optimized models has been reported to be competitive with or superior to data mixing, both for balancing safety and general-purpose capabilities [1] and for balancing helpfulness, honesty, and harmlessness [Yang et al., 2025]. Our analysis provides theoretical support for these empirical findings, attributing the effectiveness of merging DPO-trained models to the dual sparsity of their updates and the resulting small cross-task interference. Our merging experiments (§ 4.4) show that this translates into smaller performance degradation than SFT under task-vector merging; a larger-scale study of merging is left to future work.
Conclusion
We have presented a theoretical analysis of why DPO induces sparse parameter updates. Under the NTK linearization, the logistic structure of the DPO loss drives the update direction toward a maximum-margin solution that exhibits dual sparsity: it is determined by a small number of support pairs, and the gradient feature vector of each pair is itself sparse due to gradient cancellation between preferred and dispreferred responses. Relating this structure to weight disentanglement, we showed empirically that DPO reduces cross-task interference by 11× to 35× relative to SFT and degrades less under model merging, and that both sources of sparsity hold from GPT-2 to 7–9B models.
Limitations. Our analysis relies on the NTK linearization, which is not guaranteed to hold throughout DPO training of large models. More fundamentally, parameter-level sparsity (assumption 2) is an empirically motivated assumption rather than a consequence derived from the architecture. It relies on token overlap between the preferred and dispreferred responses: for pairs whose responses are entirely disjoint (e.g., different topics or formats), the gradient cancellation does not occur, and this source of sparsity may weaken or vanish. A formal derivation of assumption 2, and a characterization of the degree of overlap it requires in practice, are left to future work.
Proof and Numerical Evaluation of Lemma 2
Proof
Throughout, we work with the frozen-NTK surrogate of the margin, , under which the DPO loss reduces exactly to the logistic regression eq. (7); lemma 2 holds exactly for this model. Under , the offsets vanish, , so that solves . By assumption 1, the optimal value is positive, and hence the constraint is active: .
KKT conditions. The problem is equivalent to the hard-margin problem subject to for all , with and . Its KKT conditions give with and . Thus in eq. (10), and complementary slackness yields
Bound. Let , , and the all-ones vector. Summing eq. (19) over and applying the Cauchy–Schwarz inequality with ,
By the Rayleigh quotient, . Combining the two, , i.e., . Since is a principal submatrix of the positive semidefinite matrix , Cauchy’s interlacing theorem gives . Finally, since is an integer with , we obtain .
Numerical Evaluation
Computing exactly requires solving the max-margin problem. We instead evaluate the bound through the finite-time margin of the normalized trajectory, which is justified by the following corollary.
Corollary 4 (Finite-time evaluation of the bound). Under the assumptions of lemma 2, let and define the finite-time margin
Then . If, moreover, follows the gradient flow of the frozen-NTK surrogate, then as . Consequently, whenever ,
Proof. Let . Since , is feasible for the max-margin problem, and hence . Under the frozen-NTK gradient flow, lemma 1 gives ; since is continuous as the minimum of finitely many continuous functions, . Finally, if , then , which, combined with eq. (11), , and the integrality of , yields the chain of inequalities.
The values reported below are computed from the actual training trajectory of the network rather than from the frozen-NTK gradient flow. For such a trajectory, the convergence statement of corollary 4 does not apply, but the inequality still holds, since it uses only ; the reported ratio is therefore a conservative (upper) estimate of . Table 5 reports this quantity in the setting of Figure 1 (GPT-2 Small/Medium, HH-RLHF, ). For both models it is about 1/7 to 1/9 of , guaranteeing that is roughly an order of magnitude smaller than , consistent with the concentration observed in § 4.2.
Figure 1. Distribution of across data pairs for GPT-2 Small (top) and Medium (bottom) at 1 epoch (left), 100 epochs (middle), and 1000 epochs (right) on HH-RLHF. Dark bars indicate the top 10% of pairs by weight. As training progresses, the distribution polarizes onto a small number of support pairs; this effect is more pronounced in the larger model.
| Model | Implied bound on | |
| GPT-2 Small | 14.58 | |
| GPT-2 Medium | 11.33 |
Table 5. Conservative evaluation of the support-set bound eq. (11) in the setting of Figure 1 ().
Experimental Details
Time-integrated weights. The weights are approximated by accumulating the discrete-time analogue over training steps. Since depends only on the margins, it requires no gradient features.
Experimental settings. Table 6 summarizes the models, datasets, and numbers of preference pairs used in each experiment. The full fine-tuning DPO runs for the 7–8B models use up to eight NVIDIA A100 GPUs.
| Experiment | Models | Dataset | Pairs | Training |
| Data-level (Figure 1) | GPT-2 S/M | HH | 100 | DPO, 1000 ep. |
| Data-level (table 1) | GPT-2 S/M | HH | 1000 | DPO, 1000 ep. |
| Llama-3-8B | UF | 1000 | Full-FT DPO, 3 ep. | |
| Mistral-7B | UF | 1000 | Full-FT DPO, 3 ep. | |
| Parameter-level (Figure 2) | GPT-2 M | HH | 100 | – |
| Parameter-level (table 2) | All five | UF | 1000 | – |
| Interference (table 3) | GPT-2 S/M | HH, UF | 100 | DPO / SFT |
| Model merging (table 4) | GPT-2 M | HH, UF | section D | DPO / SFT |
Table 6. Experimental settings. “Pairs” is the number of preference pairs on which each quantity is computed. HH: Anthropic HH-RLHF; UF: UltraFeedback Binarized; ep.: epochs.
Evolution of Log-Probabilities During DPO Training
We record the log-probabilities of the preferred and dispreferred responses along the training trajectory that produces Figure 1 (GPT-2 Medium, HH-RLHF, ). Each value is the sequence-level log-probability averaged over the 100 pairs, with and .
As shown in Table 7, both log-probabilities decrease, but the rejected log-probability decreases considerably faster, so the gap widens from 11.7 to 128.6. Since , the margin satisfies , where gap denotes the per-pair log-probability gap; a widening gap thus directly increases the margin. As a result, for easy pairs, which stops their contribution to and produces the polarization of observed in Figure 1.
| Epoch | logps/chosen | logps/rejected | Gap (chosen − rejected) |
| 1 | 11.7 | ||
| 100 | 54.0 | ||
| 1000 | 128.6 |
Table 7. Average log-probabilities of preferred (chosen) and dispreferred (rejected) responses during DPO training (GPT-2 Medium, HH-RLHF, ).
Model Merging Experiments
Setup. For each task, we fine-tune GPT-2 Medium with DPO or SFT until convergence and take the task vector from the best checkpoint. For each target task, we form , where is selected on a validation set, and report . We use the same two task pairs as in § 4.4.
Results. Table 8 reports the single-task and merged performance. On both task pairs, DPO degrades less than SFT under merging. The gap between the two methods is smaller than the order-of-magnitude reduction in the cross-task inner product (Table 3), since merged performance also depends on factors other than parameter-space overlap.
| Task Pair | Method | Single | Merged | ||||
| Task 1 | Task 2 | Task 1 | Task 2 | Task 1 | Task 2 | ||
| HH help↔harm | SFT | 0.631 | 0.625 | 0.521 | 0.523 | 0.110 | 0.102 |
| DPO | 0.668 | 0.645 | 0.641 | 0.619 | 0.027 | 0.026 | |
| HH help↔UF | SFT | 0.631 | 0.535 | 0.518 | 0.505 | 0.096 | 0.030 |
| DPO | 0.668 | 0.572 | 0.632 | 0.558 | 0.036 | 0.014 |
Table 8. Single-task and merged performance of GPT-2 under task-vector merging. is the single-task minus merged performance (lower is better).
Licenses
Table 9 lists the existing assets used in this paper together with their licenses and the versions (Hugging Face identifiers) we use.
| Asset | Version (Hugging Face) | License |
| GPT-2 Small | openai-community/gpt2 | Modified MIT |
| GPT-2 Medium | openai-community/gpt2-medium | Modified MIT |
| Llama-3-8B-Instruct | meta-llama/Meta-Llama-3-8B-Instruct | Llama 3 Community License |
| Mistral-7B-Instruct-v0.2 | mistralai/Mistral-7B-Instruct-v0.2 | Apache 2.0 |
| Gemma-2-9B-it | google/gemma-2-9b-it | Gemma Terms of Use |
| Anthropic HH-RLHF | Anthropic/hh-rlhf | MIT |
| UltraFeedback Binarized | HuggingFaceH4/ultrafeedback_binarized | MIT |
Table 9. Existing assets, versions, and licenses.
References
- [1]Aakanksha, Arash Ahmadian, Seraphina Goldfarb-Tarrant, Beyza Ermis, Marzieh Fadaee, and Sara Hooker. Mix data or merge models? optimizing for diverse multi-task learning. arXiv preprint arXiv:2410.10801, 2024.arxiv.org/abs/2410.10801
- [2]Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSIlva, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.arxiv.org/abs/2204.05862
- [3]Ganqiu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with scaled ai feedback. In International Conference on Machine Learning, pages 9722–9744. PMLR, 2024.arxiv.org/abs/2310.01377
- [4]Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. In International Conference on Machine Learning, 2024.arxiv.org/abs/2402.01306
- [5]Gemma Team. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024.
- [6]Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, 2024.arxiv.org/abs/2310.12036
- [7]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.arxiv.org/abs/2407.21783
- [8]Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations, 2023.arxiv.org/abs/2212.04089
- [9]Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, volume 31, pages 8571–8580, 2018.arxiv.org/abs/1806.07572
- [10]Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023.arxiv.org/abs/2310.06825
- [11]Sagnik Mukherjee, Lifan Yuan, Dilek Hakkani-Tur, and Hao Peng. Reinforcement learning finetunes small subnetworks in large language models. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2505.11711
- [12]Guillermo Ortiz-Jiménez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Advances in Neural Information Processing Systems, volume 36, pages 66727–66754, 2023.
- [13]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 2019.
- [14]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pages 53728–53741, 2023.arxiv.org/abs/2305.18290
- [15]Noam Razin, Sadhika Malladi, Aditya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional unalignment: Likelihood displacement in direct preference optimization. In International Conference on Learning Representations, 2025.arxiv.org/abs/2410.08847
- [16]Yi Ren and Danica J. Sutherland. Learning dynamics of LLM finetuning. In International Conference on Learning Representations, 2025.arxiv.org/abs/2407.10490
- [17]Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70):1–57, 2018.arxiv.org/abs/1710.10345
