betaFeedback

WorksComputer Science

On the Sparsity of Direct Preference Optimization: Weight Disentanglement in the NTK Regime

Abstract

Direct Preference Optimization (DPO) has been widely adopted for aligning language models with human preferences. Recent empirical work has revealed that DPO induces remarkably sparse parameter updates, yet no theoretical explanation for this phenomenon has been offered. We analyze DPO's optimization dynamics under a neural tangent kernel linearization and show that the logistic structure of the DPO loss gives rise to an implicit maximum-margin bias. Building on this observation, we introduce the concept of dual sparsity: the DPO update direction is a sparse linear combination (data-level sparsity, driven by support vectors) of sparse gradient feature vectors (parameter-level sparsity, arising from gradient cancellation between preferred and dispreferred responses). We formalize this structure in our main theorem, bound the number of support pairs by a margin-dependent spectral quantity, and present a framework relating dual sparsity to weight disentanglement—a condition favorable for model merging. Experiments on GPT-2 Small and Medium and on 7–9B instruction-tuned models (Llama-3-8B, Mistral-7B, Gemma-2-9B) with the Anthropic HH-RLHF and UltraFeedback datasets confirm both sources of sparsity, and experiments on GPT-2 show that DPO reduces cross-task interference by up to 35×35{\times} compared to supervised fine-tuning.

Introduction

Direct Preference Optimization [14] has emerged as a dominant paradigm for aligning large language models with human preferences. By casting the alignment objective as a logistic loss over preference pairs, DPO circumvents the instabilities of reinforcement learning from human feedback while retaining strong empirical performance. As practitioners increasingly train separate DPO models for distinct objectives—helpfulness, harmlessness, instruction following—there is growing interest in composing these independently trained models via model merging [8] (Yu et al. 2024), which combines task-specific parameter updates (task vectors) into a single multitask model without additional training.

A striking but poorly understood empirical observation is that DPO fine-tuning induces highly sparse parameter updates. Mukherjee et al. [ [11] recently demonstrated that reinforcement learning-based alignment concentrates its modifications onto a small subnetwork of the pre-trained model, leaving the vast majority of parameters essentially unchanged. This sparsity stands in marked contrast to supervised fine-tuning (SFT), which typically distributes updates broadly across the parameter space. Independently, Ortiz-Jiménez et al. [ [12] identified weight disentanglement—the condition that task vectors act on non-overlapping regions of parameter space—as a key property enabling successful model merging. A natural question arises: is the sparsity of DPO a mere coincidence, or does it reflect a deeper structure in the optimization that is systematically favorable for composing task vectors?

Preprint.

In this paper, we provide a theoretical answer to this question. We analyze the implicit bias of DPO under a neural tangent kernel (NTK) linearization [9] and show that the logistic loss structure, combined with the geometry of preference-pair gradients, gives rise to a phenomenon we call dual sparsity. The DPO update direction converges to the maximum-margin solution in the NTK feature space, inheriting two complementary forms of sparsity: data-level sparsity, where only a small subset of preference pairs (support vectors) contribute to the final update, and parameter-level sparsity, where the gradient feature vector for each pair is itself sparse due to gradient cancellation between preferred and dispreferred responses that share common tokens. Building on this, we present a framework relating dual sparsity to weight disentanglement, and verify empirically that DPO produces cross-task inner products an order of magnitude smaller than those under SFT.

Our main contributions are as follows:

  • (C1) We reduce DPO to maximum-margin classification under NTK linearization and introduce dual sparsity—a joint data-level and parameter-level sparsity structure—formalized in theorem 3. This provides the first theoretical explanation for why DPO induces sparse parameter updates without any explicit sparsity-promoting regularization.

  • (C2) We present a framework relating dual sparsity to weight disentanglement [12] via a triplet-space formulation of the DPO margin (§ 3.3). The framework reduces the disentanglement condition to a per-triplet inner product whose magnitude is governed by the parameter-space overlap between tasks—an empirical question that we evaluate in § 4.

  • (C3) We validate both sources of sparsity on GPT-2 Small (124M) and Medium (355M) fine-tuned with the Anthropic HH-RLHF [2] and UltraFeedback [3] datasets, and at modern scale on Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.2, and Gemma-2-9B-it. DPO feature vectors are over 90% near-zero from 124M to 9B parameters, and the time-integrated weights concentrate on a small subset of pairs within three epochs of full fine-tuning at the 7–8B scale. On GPT-2, cross-task interference under DPO is reduced by 11× to 35× relative to SFT across all configurations.

The remainder of the paper is organized as follows. § 2 reviews the DPO formulation and weight disentanglement framework. § 3 presents our theoretical analysis, culminating in the dual sparsity theorem and a framework connecting it to weight disentanglement. § 4 provides experimental validation. § 5 discusses related work, and § 6 concludes with limitations and future directions.

Preliminaries

Direct Preference Optimization

Let πref\pi_{\mathrm{ref}} denote a reference policy, typically a pre-trained or supervised fine-tuned (SFT) model [14]. Given a prompt xx and a pair of responses (y+,y−)(y^{+}, y^{-}) where y+y^{+} is preferred and y−y^{-} is dispreferred, the log-probability margin under parameters θ\theta is

mi(θ)=β(log⁡πθ(yi+∣xi)πθ(yi−∣xi)−log⁡πref(yi+∣xi)πref(yi−∣xi)),(1)m_i(\theta) = \beta\left( \log\frac{\pi_{\theta}(y_i^{+} \mid x_i)}{\pi_{\theta}(y_i^{-} \mid x_i)} - \log\frac{\pi_{\mathrm{ref}}(y_i^{+} \mid x_i)}{\pi_{\mathrm{ref}}(y_i^{-} \mid x_i)} \right), \tag*{(1)}

with temperature β>0\beta> 0. For a preference dataset D={(xi,yi+,yi−)}i=1n\mathcal{D} = \{(x_i, y_i^{+}, y_i^{-})\}_{i=1}^{n}, the DPO loss is the logistic loss

ℓi(θ)=log⁡(1+e−mi(θ)),(2)\ell_i(\theta) = \log\left(1 + e^{-m_i(\theta)}\right), \tag*{(2)}

whose gradient w.r.t. θ\theta is weighted by σ(−mi(θ))∈(0,1)\sigma(-m_i(\theta)) \in(0, 1): the gradient vanishes for “easy” pairs already preferred by the policy, focusing optimization on the “hard” subset near the decision boundary.

Weight Disentanglement

We review the notion of weight disentanglement [12], which characterizes an ideal condition for composing independently trained task vectors without cross-task interference.

Consider a neural network f:X×Θ→Yf:\mathcal{X} \times\Theta\to\mathcal{Y} with pre-trained parameters θ0\theta_0. A task tt is specified by a data support Dt⊆X\mathcal{D}_t \subseteq\mathcal{X}, an input distribution μt\mu_t with supp⁡(μt)=Dt\operatorname{supp}(\mu_t) = \mathcal{D}_t, and a target function ft∗:Dt→Yf_t^*: \mathcal{D}_t \to\mathcal{Y}. Fine-tuning ff on task tt yields parameters θt∗\theta_t^*, and the corresponding task vector is defined as τt:=θt∗−θ0\tau_t := \theta_t^* - \theta_0.

Definition 1 (Task Arithmetic, Ilharco et al., [8]). Let T={τt}t∈[T]T = \{\tau_t\}_{t \in[T]} be a set of task vectors with mutually disjoint supports Dt∩Dt′=∅\mathcal{D}_t \cap\mathcal{D}_{t'} = \varnothing for t≠t′t \ne t'. The network ff satisfies the task arithmetic property around θ0\theta_0 if

f(x;θ0+∑s=1Tαsτs)={f(x;θ0+αtτt)x∈Dt,f(x;θ0)x∉⋃tDt.(3)f\left(x; \theta_0 + \sum_{s=1}^{T} \alpha_s \tau_s\right) = \begin{cases} f(x; \theta_0 + \alpha_t \tau_t) & x \in\mathcal{D}_t, \\ f(x; \theta_0) & x \notin\bigcup_t \mathcal{D}_t. \end{cases} \tag*{(3)}

Definition 2 (Weight Disentanglement, Ortiz-Jiménez et al., [12]). The network ff is weight-disentangled with respect to TT and {Dt}\{\mathcal{D}_t\} if there exist functions gt:X×Θ→Yg_t: \mathcal{X} \times\Theta\to\mathcal{Y} with gt(x;⋅)=0g_t(x; \cdot) = 0 for x∉Dtx \notin\mathcal{D}_t and a residual g0g_0 with g0(x)=0g_0(x) = 0 for x∈⋃tDtx \in\bigcup_t \mathcal{D}_t, such that

f(x;θ0+∑t=1Tαtτt)=∑t=1Tgt(x;αtτt)+g0(x).(4)f\left(x; \theta_0 + \sum_{t=1}^{T} \alpha_t \tau_t\right) = \sum_{t=1}^{T} g_t(x; \alpha_t \tau_t) + g_0(x). \tag*{(4)}

It is straightforward to verify that weight disentanglement (definition 2) implies the task arithmetic property (definition 1) by setting gt(x;αtτt)=f(x;θ0+αtτt)1(x∈Dt)g_t(x; \alpha_t \tau_t) = f(x; \theta_0 + \alpha_t \tau_t)1(x \in\mathcal{D}_t) and g0(x)=f(x;θ0)1(x∉⋃tDt)g_0(x) = f(x; \theta_0)1(x \notin\bigcup_t \mathcal{D}_t).

Theoretical Analysis

Recent empirical work has shown that alignment fine-tuning via DPO induces highly sparse parameter updates relative to the pre-trained weights [11]. Independently, weight disentanglement has been identified as a desirable property for robust model merging [12]. The following questions remain open:

  1. Why does DPO induce sparsity? What mechanism in the DPO optimization dynamics leads to sparse parameter updates, despite the absence of any explicit sparsity-promoting regularization?

  1. What are the implications of this sparsity for cross-task interference? Specifically, does the sparsity inherent in DPO updates reduce the interference between independently trained task vectors?

This section addresses both questions. We analyze the implicit bias of DPO under a neural tangent kernel (NTK) linearization (§ 3.1) and identify a dual sparsity structure that emerges from the interplay between the logistic loss and the geometry of the gradient features (§ 3.2). We then present a framework relating this structure to weight disentanglement, with empirical evaluation deferred to section 4 (§ 3.3).

DPO as Maximum-Margin Classification in Tangent Space

We begin by linearizing the DPO margin around the initial parameters θ0\theta_0. Under the neural tangent kernel (NTK) regime [9], the margin admits a first-order approximation:

mi(θ)≈mi(θ0)+⟨Φi,Δθ⟩,(5)m_i(\theta) \approx m_i(\theta_0) + \langle\Phi_i, \Delta\theta\rangle, \tag*{(5)}

where Δθ=θ−θ0\Delta\theta= \theta- \theta_0 and the gradient feature vector Φi\Phi_i is defined as

Φi:=β(∇θlog⁡πθ0(yi+∣xi)−∇θlog⁡πθ0(yi−∣xi))∈Rp.(6)\Phi_i := \beta\left(\nabla_\theta\log\pi_{\theta_0}(y_i^+ \mid x_i) - \nabla_\theta\log\pi_{\theta_0}(y_i^- \mid x_i)\right) \in\mathbb{R}^p. \tag*{(6)}

Here pp denotes the total number of parameters. Each Φi\Phi_i encodes the direction in parameter space along which a perturbation Δθ\Delta\theta most effectively increases the preference margin for pair ii. Under this linearization, the DPO loss reduces to a binary logistic regression over the features {Φi}i=1n\{\Phi_i\}_{i=1}^{n} with offsets bi:=mi(θ0)b_i := m_i(\theta_0):

ℓi(Δθ)=log⁡(1+exp⁡(−⟨Φi,Δθ⟩−bi)).(7)\ell_i(\Delta\theta) = \log\left(1 + \exp\left(-\langle\Phi_i, \Delta\theta\rangle- b_i\right)\right). \tag*{(7)}

With the standard DPO initialization πθ0=πref\pi_{\theta_0} = \pi_{\mathrm{ref}}, the offsets vanish: bi=mi(θ0)=0b_i = m_i(\theta_0) = 0 by eq. (1).

We make the following assumption, which is standard in the analysis of implicit bias for logistic regression [17]:

Assumption 1 (Linear Separability). There exist Δθ∈Rp\Delta\theta\in\mathbb{R}^{p} and γ>0\gamma> 0 such that ⟨Φi,Δθ⟩+bi≥γ\langle\Phi_i, \Delta\theta\rangle+ b_i \ge\gamma for all i∈[n]i \in[n].

Under assumption 1, the gradient flow on the total loss L(Δθ)=∑iℓi(Δθ)L(\Delta\theta) = \sum_i \ell_i(\Delta\theta) starting from Δθ(0)=0\Delta\theta(0) = 0 admits the explicit trajectory

Δθ(T)=∑i=1nAi(T) Φi,Ai(T):=∫0Tσ(−mi(θ(t))) dt,(8)\Delta\theta(T) = \sum_{i=1}^{n} A_i(T)\,\Phi_i,\qquad A_i(T) := \int_{0}^{T} \sigma(-m_i(\theta(t)))\,\mathrm{d}t, \tag*{(8)}

where Ai(T)A_i(T) is the time-integrated gradient weight for pair ii. Applying the implicit-bias result of Soudry et al. [ [17] to this setup yields:

Lemma 1 (Implicit Bias of DPO). Under assumption 1, the normalized parameter update converges to the L2L_2 maximum-margin direction:

Δθ(T)∥Δθ(T)∥→T→∞u⋆,where u⋆ solves max⁡∥u∥≤1min⁡i∈[n]{⟨u,Φi⟩+bi}.(9)\frac{\Delta\theta(T)}{\lVert\Delta\theta(T)\rVert} \xrightarrow[T\to\infty]{} u^\star,\qquad\text{where }u^\star\text{ solves }\max_{\lVert u\rVert\le1}\min_{i\in[n]}\{\langle u,\Phi_i\rangle+b_i\}. \tag*{(9)}

By the Karush–Kuhn–Tucker (KKT) conditions, u⋆u^\star admits the representation

u⋆=∑i∈SVαi⋆Φi,αi⋆≥0,(10)u^\star= \sum_{i\in\mathrm{SV}}\alpha_i^\star\Phi_i,\qquad\alpha_i^\star\ge0, \tag*{(10)}

where SV:={i:αi⋆>0}\mathrm{SV}:=\{i:\alpha_i^\star>0\} is the set of support pairs.

Remark 1 (Empirical observability of dual variables). The normalized time-integrated coefficients αi(T):=Ai(T)/∥Δθ(T)∥\alpha_i(T):=A_i(T)/\lVert\Delta\theta(T)\rVert also converge to the dual variables αi⋆\alpha_i^\star as T→∞T\to\infty. This justifies tracking AiA_i as an empirical proxy for the support set in § 4.2.

Dual Sparsity of DPO Updates

Lemma 1 establishes that the DPO update direction is fully determined by the support pairs and the corresponding feature vectors. We now identify two distinct sources of sparsity in this representation.

Definition 3 (Dual Sparsity). The DPO update direction u⋆=∑i∈SVαi⋆Φiu^\star=\sum_{i\in\mathrm{SV}}\alpha_i^\star\Phi_i is said to exhibit dual sparsity if the following two conditions hold:

(i) Data-level sparsity: The number of support pairs is small relative to the dataset, i.e., ∣SV∣≪n|\mathrm{SV}|\ll n.

(ii) Parameter-level sparsity: Each gradient feature vector Φi\Phi_i is sparse, i.e., ∥Φi∥0≪p\lVert\Phi_i\rVert_0\ll p for i∈SVi\in\mathrm{SV}.

The data-level sparsity is a direct consequence of the maximum-margin structure:

Lemma 2 (Data-Level Sparsity). Suppose πθ0=πref\pi_{\theta_0}=\pi_{\mathrm{ref}} (so that bi=0b_i=0) and assumption 1 holds. Let K:=Φ⊤Φ∈Rn×nK:=\Phi^\top\Phi\in\mathbb{R}^{n\times n}, with Kij=⟨Φi,Φj⟩K_{ij}=\langle\Phi_i,\Phi_j\rangle, be the Gram matrix of Φ:=[Φ1 Φ2 ⋯ Φn]\Phi:=[\Phi_1\ \Phi_2\ \cdots\ \Phi_n], and let γ⋆:=min⁡i∈[n]⟨u⋆,Φi⟩>0\gamma^\star:=\min_{i\in[n]}\langle u^\star,\Phi_i\rangle>0 be the maximum margin. Then, for the linearized model eq. (7)*,

∣SV∣ γ⋆2≤λmax⁡(KSV,SV)≤λmax⁡(K),hence∣SV∣≤min⁡{n,⌊λmax⁡(K)γ⋆2⌋},(11)|\mathrm{SV}|\,\gamma_\star^2 \le\lambda_{\max}(K_{\mathrm{SV},\mathrm{SV}})\le\lambda_{\max}(K),\qquad\text{hence}\qquad|\mathrm{SV}|\le\min\left\{n,\left\lfloor\frac{\lambda_{\max}(K)}{\gamma_\star^2}\right\rfloor\right\}, \tag*{(11)}

where KSV,SVK_{\mathrm{SV},\mathrm{SV}} is the principal submatrix of KK indexed by SV.

The proof is given in section A.1.

This bound is meaningful because, in the DPO setting, the gradient flow concentrates its cumulative weight Ai(T)A_i(T) on the pairs that remain difficult throughout training—those near the margin boundary—while the remaining pairs receive diminishing gradient signal. Since the bound is governed by the ratio λmax⁡(K)/γ⋆2\lambda_{\max}(K)/\gamma_\star^2 rather than explicitly by nn, the effective support set is probably ≪n\ll n rather than merely ≤rank⁡(Φ)\le\operatorname{rank}(\Phi) whenever the margin is large relative to the spectral scale of the features; we confirm empirically that this is the case on real preference data (see § 4.2).

For the parameter-level sparsity, we introduce the following assumption:

Assumption 2 (Sparsity of Gradient Features). For each i∈[n]i\in[n], the gradient feature vector Φi\Phi_i satisfies ∥Φi∥0≤s\lVert\Phi_i\rVert_0\le s for some s≪ps\ll p.

Remark 2 (Motivation for assumption 2). In autoregressive language models, the preferred and dispreferred responses yi+y_i^+ and yi−y_i^- for a given prompt xix_i typically share a substantial number of tokens—particularly in the initial portion of the response. Since Φi\Phi_i is defined as the difference of gradients β(∇θlog⁡πθ0(yi+∣xi)−∇θlog⁡πθ0(yi−∣xi))\beta(\nabla_\theta\log\pi_{\theta_0}(y_i^+ \mid x_i) - \nabla_\theta\log\pi_{\theta_0}(y_i^- \mid x_i)), the parameter-level contributions from shared tokens cancel, leaving nonzero entries only in the parameters responsible for distinguishing between yi+y_i^+ and yi−y_i^-. This cancellation effect provides a structural basis for assumption 2. We verify this assumption empirically in section 4.

Combining lemmas 1 and 2 with assumption 2, we arrive at the main result:

Theorem 3 (Dual Sparsity of DPO). Under assumptions 1 and 2, the DPO update direction u⋆u^\star exhibits dual sparsity in the sense of definition 3. That is, u⋆u^\star is a sparse linear combination of sparse vectors:

u⋆=∑i∈SVαi⋆Φi,∣SV∣≪n,∥Φi∥0≤s≪p.(12)u^\star= \sum_{i\in SV} \alpha_i^\star\Phi_i,\qquad|SV| \ll n,\qquad\lVert\Phi_i\rVert_0 \le s \ll p. \tag*{(12)}

Consequently, u⋆u^\star is itself sparse with ∥u⋆∥0≤∣SV∣⋅s\lVert u^\star\rVert_0 \le|SV| \cdot s.

Connection to Weight Disentanglement

Theorem 3 establishes that DPO updates exhibit dual sparsity. We now examine how this structure relates to the weight disentanglement framework reviewed in section 2.2. The original formulation of weight disentanglement (definition 2) is stated for a generic function f:X×Θ→Yf:\mathcal{X} \times\Theta\to\mathcal{Y}; applying it to the DPO setting requires identifying the appropriate input space and function in preference learning.

Triplet-space formulation. We take the input space to be the space of preference triplets X~:={(x,y+,y−)}\widetilde{\mathcal{X}} := \{(x,y^+,y^-)\} and define f:X~×Θ→Rf:\widetilde{\mathcal{X}} \times\Theta\to\mathbb{R} as the DPO log-probability margin

f(x~;θ):=β(log⁡πθ(y+∣x)−log⁡πθ(y−∣x)),x~=(x,y+,y−).(13)f(\widetilde{x};\theta) := \beta\left(\log\pi_\theta(y^+ \mid x) - \log\pi_\theta(y^- \mid x)\right),\qquad\widetilde{x}=(x,y^+,y^-). \tag*{(13)}

Differentiating at θ0\theta_0 recovers the gradient feature vector of section 3.1: ∇θf(x~;θ0)=Φ(x~)\nabla_\theta f(\widetilde{x};\theta_0)=\Phi(\widetilde{x}). The same features that drive dual sparsity therefore serve as the Jacobian relevant for weight disentanglement. A task tt is associated with a preference dataset Dt⊆X~D_t \subseteq\widetilde{\mathcal{X}} collected under task-specific annotation criteria.

Linearized decomposition. For task vectors {τt}t∈[T]\{\tau_t\}_{t\in[T]} and coefficients {λt}t∈[T]\{\lambda_t\}_{t\in[T]}, a first-order Taylor expansion of ff around θ0\theta_0 gives

f(x~;θ0+∑t=1Tλtτt)≈f(x~;θ0)+∑t=1Tλt⟨τt,Φ(x~)⟩.(14)f\left(\widetilde{x};\theta_0+\sum_{t=1}^{T}\lambda_t\tau_t\right)\approx f(\widetilde{x};\theta_0)+\sum_{t=1}^{T}\lambda_t\langle\tau_t,\Phi(\widetilde{x})\rangle. \tag*{(14)}

Equation (14) matches the structural form of the weight disentanglement decomposition (definition 2) under the identification

g0(x~):=f(x~;θ0),gt(x~;λtτt):=λt⟨τt,Φ(x~)⟩.(15)g_0(\widetilde{x}) := f(\widetilde{x};\theta_0),\qquad g_t(\widetilde{x};\lambda_t\tau_t) := \lambda_t\langle\tau_t,\Phi(\widetilde{x})\rangle. \tag*{(15)}

With this identification, the disentanglement condition gt(x~;λtτt)=0g_t(\widetilde{x};\lambda_t\tau_t)=0 for x~∉Dt\widetilde{x}\notin D_t reduces to the inner-product condition

⟨τt,Φ(x~)⟩≈0for x~∉Dt.(16)\langle\tau_t,\Phi(\widetilde{x})\rangle\approx0\qquad\text{for }\widetilde{x}\notin D_t. \tag*{(16)}

Observation 1 (Linearized weight disentanglement under DPO). Under the NTK linearization, the DPO margin ff admits the decomposition eq. (14), and the identification eq. (15) places this decomposition in the form of weight disentanglement (definition 2). The disentanglement condition holds to the extent that the cross-task inner product eq. (16) is small.

Connection to dual sparsity. Lemma 1 establishes that, in the limit, the normalized task vector for task tt aligns with the maximum-margin direction ut⋆=∑i∈SVtαt,i⋆Φ(x~i(t))u_t^\star=\sum_{i\in SV_t}\alpha_{t,i}^\star\Phi(\widetilde{x}_i^{(t)}). Substituting into eq. (15), the local function gtg_t takes the form

gt(x~;λtτt)∝⟨ut⋆,Φ(x~)⟩=⟨∑i∈SVtαt,i⋆Φ(x~i(t)),Φ(x~)⟩.(17)g_t(\widetilde{x};\lambda_t\tau_t)\propto\langle u_t^\star,\Phi(\widetilde{x})\rangle=\left\langle\sum_{i\in SV_t}\alpha_{t,i}^\star\Phi(\widetilde{x}_i^{(t)}),\Phi(\widetilde{x})\right\rangle. \tag*{(17)}

The cross-task interference eq. (16) is therefore governed by ⟨u1∗,Φ(x~)⟩\langle u_1^*, \Phi(\tilde{x})\rangle for x~∈D2\tilde{x} \in D_2. By theorem 3, u1∗u_1^* is a sparse combination of sparse vectors; assumption 2 ensures Φ(x~)\Phi(\tilde{x}) is also sparse. When the active parameter supports of u1∗u_1^* and {Φ(x~)}x~∈D2\{\Phi(\tilde{x})\}_{\tilde{x}\in D_2} have small overlap, this inner product is small. Whether the overlap is in fact small for naturally occurring preference tasks is an empirical question, which we examine in section 4.

Remark on the disjoint-support assumption. Definition 1 assumes disjoint task supports Dt∩Dt′=∅D_t \cap D_{t'} = \varnothing. In the triplet-space formulation this is naturally satisfied: distinct preference datasets arise from distinct annotation processes, and an ordered triplet (x,y+,y−)(x, y^+, y^-) is tied to the task it was annotated under.

Experiments

We empirically validate the theoretical predictions of section 3. Specifically, we verify (i) data-level sparsity of the dual coefficients αi∗\alpha_i^* (lemma 2), (ii) parameter-level sparsity of the gradient features Φi\Phi_i (assumption 2), and (iii) suppression of cross-task interference under DPO compared to SFT (section 4.4).

Experimental Setup

Models. We use GPT-2 Small and GPT-2 Medium [13], together with three instruction-tuned models with 7–9B parameters: Llama-3-8B-Instruct [7], Mistral-7B-Instruct-v0.2 [10], and Gemma-2-9B-it [5]. Each checkpoint serves as both the initialization and the reference policy, πθ0=πref\pi_{\theta_0} = \pi_{\mathrm{ref}}.

Datasets. For our primary experiments, we use the Anthropic HH-RLHF dataset [2], which consists of two subsets: helpful (evaluating which response is more useful to the user) and harmless (evaluating which response avoids harmful content). These two distinct subsets serve as the tasks T1T_1 and T2T_2 in our cross-task interference analysis. To assess generality, we additionally use the UltraFeedback Binarized dataset [3], a large-scale preference dataset annotated by GPT-4 across four evaluation axes (instruction-following, truthfulness, honesty, and helpfulness).

Gradient features. The gradient feature vectors Φi=β(∇θlog⁡πθ0(yi+∣xi)−∇θlog⁡πθ0(yi−∣xi))\Phi_i = \beta(\nabla_\theta\log\pi_{\theta_0}(y_i^+ \mid x_i) - \nabla_\theta\log\pi_{\theta_0}(y_i^- \mid x_i)) are computed empirically using the Jacobian of the model at initialization, since the analytical NTK cannot be obtained in closed form for self-attention architectures. Because storing full-dimensional gradient vectors (Φi∈Rp\Phi_i \in\mathbb{R}^p with p>108p > 10^8) is memory-intensive, experiments that require Φi\Phi_i are performed on subsamples of the datasets (see details in section B).

The models, datasets, and numbers of preference pairs used in each experiment, together with the computation of the time-integrated weights AiA_i, are detailed in section B.

Data-Level Sparsity

Lemma 2 predicts that the DPO update direction is driven by a small number of support pairs, with the remaining pairs contributing negligibly. To verify this, we examine how the time-integrated weights Ai(T)A_i(T) are distributed across pairs as training progresses.

Figure 1 visualizes AiA_i for GPT-2 Small and Medium on N=100N = 100 pairs from HH-RLHF at 1, 100, and 1000 epochs. At early stages, the weights are distributed nearly uniformly. As training progresses, the distribution becomes increasingly polarized: a small subset of pairs accumulates large weights while the rest stop growing (see section C for the corresponding log-probabilities). This is the behavior predicted by lemma 1, under which the normalized coefficients αi(T)\alpha_i(T) converge to the dual variables αi∗\alpha_i^*, which are nonzero only for the support pairs.

Table 1 quantifies this concentration on N=1000N = 1000 pairs by the top-10% mass, i.e., the fraction of ∑iAi\sum_i A_i held by the top 10% of pairs (10% under a uniform distribution). For GPT-2, the top-10% mass grows from about 11% to 40–44% over 1000 epochs; for Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.2, it grows from about 11% to 44–52% within three epochs of full fine-tuning DPO. The numerical evaluation of the bound in lemma 2 in the setting of fig. 1 is reported in section A.2.

ModelDatasetTop-10% mass (%)
Epoch 1Epoch 100Epoch 1000
GPT-2 SmallHH-RLHF10.8828.7839.57
GPT-2 MediumHH-RLHF11.0232.3243.70
Epoch 1Epoch 2Epoch 3
Llama-3-8B-InstructUltraFeedback11.7541.8252.48
Mistral-7B-Instruct-v0.2UltraFeedback11.2339.9243.77

Table 1. Top-10% mass of AiA_i on N=1000N = 1000 pairs, i.e., the fraction of ∑iAi\sum_i A_i held by the top 10% of pairs (10% under a uniform distribution). Top: GPT-2 trained with DPO on HH-RLHF. Bottom: 7–8B models trained with full fine-tuning DPO on UltraFeedback Binarized.

Parameter-Level Sparsity

Assumption 2 posits that the DPO gradient feature vectors Φi\Phi_i are sparse due to gradient cancellation between preferred and dispreferred responses. To verify this and to contrast it with SFT, Figure 2 visualizes the element-wise distribution of a representative Φi\Phi_i under both methods (GPT-2 Medium, HH-RLHF, N=100N = 100), and Table 2 quantifies the fraction of near-zero entries for all five models (N=1000N = 1000, UltraFeedback Binarized).

Element-wise distribution of gradient feature vectors for DPO and SFT

Figure 2. Element-wise distribution of gradient feature vectors on a logarithmic scale for GPT-2 Medium on N=100N = 100 pairs from HH-RLHF. (a) DPO: the vast majority of entries are near zero, confirming assumption 2. (b) SFT: the distribution is substantially denser, since no gradient cancellation occurs. Near-zero fractions for all five models, computed in a separate setting, are reported in Table 2.

ModelParamsDPO Φi\Phi_i (%)SFT ΦiSFT\Phi_i^{\mathrm{SFT}} (%)
GPT-2 Small124M91.293.86
GPT-2 Medium355M94.824.34
Llama-3-8B-Instruct8B92.5116.44
Mistral-7B-Instruct-v0.27B90.9319.72
Gemma-2-9B-it9B92.1213.57

Table 2. Fraction of near-zero entries in a representative gradient feature vector under DPO (Φi\Phi_i) and SFT (ΦiSFT\Phi_i^{\mathrm{SFT}}), computed from 1,000 pairs sampled from UltraFeedback Binarized. Higher values indicate sparser features.

Under DPO (Figure 2a), the vast majority of entries are concentrated near zero, with only a small fraction taking on appreciable magnitude. Under SFT (Figure 2b), the distribution is substantially broader, reflecting the fact that the SFT feature vector ΦiSFT=∇θlog⁡πθ(yi+∣xi)\Phi_i^{\mathrm{SFT}} = \nabla_\theta\log\pi_\theta(y_i^+ \mid x_i) involves only the preferred response and does not benefit from the gradient cancellation that arises in the DPO difference ∇θlog⁡πθ0(yi+∣xi)−∇θlog⁡πθ0(yi−∣xi)\nabla_\theta\log\pi_{\theta_0}(y_i^+ \mid x_i) - \nabla_\theta\log\pi_{\theta_0}(y_i^- \mid x_i). The same contrast holds at every scale we test (Table 2): from 124M to 9B parameters, more than 90% of the entries of the DPO feature vector are near zero, whereas only 4–20% of those of the SFT feature vector are.

As discussed in remark 2, this contrast arises naturally from the structure of preference data in autoregressive models: since yi+y_i^+ and yi−y_i^- typically share a large number of tokens—particularly in the initial portion of the response—the parameter-level gradients for these shared tokens cancel in the DPO feature vector, leaving nonzero contributions only for the parameters that distinguish between the two responses.

Cross-Task Interference

We evaluate whether the dual sparsity of DPO updates translates into reduced cross-task interference, as discussed in § 3.3. We consider two task pairs of increasing semantic distance:

  • (a) HH helpful ↔ harmless: two subsets of the same dataset (Anthropic HH-RLHF) with different annotation criteria—helpfulness vs. harmlessness.

  • (b) HH helpful ↔ UltraFeedback: two entirely different datasets with different annotation methods (human vs. AI), different prompt distributions, and different preference dimensions.

For the cross-dataset pair (b), only the gradient features Φj(2)\Phi_j^{(2)} of UltraFeedback at θ0\theta_0 are required; no DPO training on UltraFeedback is performed.

Baseline. We compare DPO against SFT. Unlike DPO, SFT does not produce the logistic loss structure that gives rise to implicit maximum-margin bias; consequently, all data pairs contribute to the update direction, and data-level sparsity does not emerge. Moreover, the SFT feature vector ΦiSFT=∇θlog⁡πθ(yi+∣xi)\Phi_i^{\mathrm{SFT}} = \nabla_\theta\log\pi_\theta(y_i^+ \mid x_i) involves only the preferred response, so it does not benefit from the gradient cancellation that induces parameter-level sparsity in DPO.

Metrics. The disentanglement condition derived in § 3.3 is a per-triplet condition ⟨τt,Φ(x~)⟩≈0\langle\tau_t, \Phi(\tilde{x})\rangle\approx0 for x~∉Dt\tilde{x} \notin D_t (eq. (16)). Verifying it directly would require evaluating ⟨u1∗,Φ(x~)⟩\langle u_1^*, \Phi(\tilde{x})\rangle for every x~∈D2\tilde{x} \in D_2, which is computationally prohibitive given the scale of Φ(x~)∈Rp\Phi(\tilde{x}) \in\mathbb{R}^p with p>108p > 10^8. We therefore aggregate the per-triplet quantity into a single scalar summary by introducing the mean feature vector for task tt:

Vt:=1∣Dt∣∑x~∈DtΦ(x~).(18)V_t := \frac{1}{|D_t|} \sum_{\tilde{x}\in D_t} \Phi(\tilde{x}). \tag*{(18)}

On this aggregated quantity, we evaluate two measures of cross-task overlap between V1V_1 and V2V_2:

  • Top-1% Jaccard overlap: the Jaccard coefficient between the index sets of the top 1% largest-magnitude entries in each mean feature vector. This captures the degree of overlap between the most active parameters of the two tasks—the structural condition under which the per-triplet inner product ⟨u1∗,Φ(x~)⟩\langle u_1^*, \Phi(\tilde{x})\rangle is small (§ 3.3).

  • Interference ⟨u1∗,V2⟩\langle u_1^*, V_2\rangle: the inner product between the DPO update direction for T1\mathcal{T}_1 and the mean feature vector for T2\mathcal{T}_2. This is the average of ⟨u1∗,Φ(x~)⟩\langle u_1^*, \Phi(\tilde{x})\rangle over x~∈D2\tilde{x} \in D_2 and serves as a tractable summary of the per-triplet disentanglement condition eq. (16).

Results. Table 3 reports the results. DPO suppresses cross-task interference by an order of magnitude or more in every configuration. Two qualitative patterns are notable: under DPO, both the Jaccard overlap and the interference decrease at the larger scale (e.g., 0.258 →\rightarrow 0.191 on the within-dataset pair and 1.91 →\rightarrow 0.852 on the cross-dataset pair), consistent with the sharper polarization of AiA_i observed in § 4.2; and the same pattern holds for the cross-dataset pair (HH helpful ↔\leftrightarrow UltraFeedback), demonstrating that the effect generalizes beyond subsets of a single dataset to entirely independent preference data sources. These observations empirically support the framework of § 3.3: DPO’s dual sparsity, combined with the small parameter-space overlap exhibited by semantically distinct preference tasks, yields cross-task inner products an order of magnitude smaller than those under SFT.

Task PairMethodModelJaccard (%)⟨u1∗,V2⟩\langle u_1^*, V_2\rangle
HH help↔\leftrightarrowharmSFTSmall54.99.12
HH help↔\leftrightarrowharmDPOSmall18.90.258
HH help↔\leftrightarrowharmSFTMedium49.33.22
HH help↔\leftrightarrowharmDPOMedium11.40.191
HH help↔\leftrightarrowUFSFTSmall68.421.2
HH help↔\leftrightarrowUFDPOSmall25.21.91
HH help↔\leftrightarrowUFSFTMedium60.519.8
HH help↔\leftrightarrowUFDPOMedium17.70.852

Table 3. Cross-task interference for DPO and SFT across two model scales (GPT-2 Small/Medium) and two task pairs: HH help↔\leftrightarrowharm (HH-RLHF helpful vs. harmless subsets) and HH help↔\leftrightarrowUF (HH-RLHF helpful vs. UltraFeedback). Lower values indicate greater disentanglement. Across all configurations, DPO reduces the interference ⟨u1∗,V2⟩\langle u_1^*, V_2\rangle by 11×\times to 35×\times relative to SFT.

Model merging. To examine whether reduced interference carries over to merged models, we merge task vectors obtained by DPO or SFT on each task pair as θmerged=θ0+α(τtarget+τother)\theta_{\mathrm{merged}} = \theta_0 + \alpha(\tau_{\mathrm{target}} + \tau_{\mathrm{other}}), α∈[0,2]\alpha\in[0,2] and measure the degradation Δ\Delta of each task’s performance relative to its single-task model (section D). As shown in Table 4, DPO yields a smaller Δ\Delta than SFT for both tasks in both task pairs, consistent with the reduced interference in Table 3.

Task PairMethodΔ\Delta Task 1Δ\Delta Task 2
HH help↔harmSFT0.1100.102
HH help↔harmDPO0.0270.026
HH help↔UFSFT0.0960.030
HH help↔UFDPO0.0360.014

Table 4. Performance degradation Δ\Delta (single-task − merged; lower is better) under task-vector merging on GPT-2 Medium. Task 1 is HH helpful; Task 2 is HH harmless or UltraFeedback. Full results are reported in section D.

Related Work

Optimization dynamics of DPO. Since Rafailov et al. [ [14] introduced DPO as a stable alternative to reinforcement learning from human feedback, several works have analyzed its dynamics theoretically. One line modifies the loss to address specific failure modes—IPO [6] avoids the Bradley–Terry assumption, and KTO [4] adopts a prospect-theoretic utility—but these are orthogonal to our analysis, which concerns the original objective. Closer to Ren and Sutherland [ [16] linearize DPO under the NTK regime and use the resulting framework to characterize the squeezing effect, a counter-intuitive decrease in the probability of preferred responses also studied by Razin et al. [ [15] under the term likelihood displacement. We adopt the same NTK lens but extract a structurally distinct property: the direction of the update in parameter space. Combining the classical implicit-bias result of Soudry et al. [ [17] for separable logistic regression with the specific geometry of preference-pair gradients yields dual sparsity (theorem 3), which in turn provides a theoretical explanation for the small-subnetwork behavior observed empirically by Mukherjee et al. [ [11] on RL-based alignment. Likelihood displacement and dual sparsity are thus two complementary consequences of the same dynamics: one concerning the probabilities the update changes, the other the parameters it changes.

Task arithmetic, weight disentanglement, and model merging. Ilharco et al. [ [8] showed that linear arithmetic on task vectors compositionally steers model behavior, and Ortiz-Jiménez et al. [ [12] formalized weight disentanglement as a sufficient condition for this property, identifying via NTK tangent-space analysis that task vectors should act on disjoint regions of parameter space. A complementary line of work—DARE [Yu et al., 2024] and TIES-Merging [Yadav et al., 2023]—engineers sparsity into task vectors as a post-hoc intervention to suppress interference during merging. Our framework offers a different perspective: under DPO, sparsity emerges intrinsically from the optimization dynamics, and the resulting parameter-space overlap between semantically distinct preference tasks is small enough that the disentanglement condition holds approximately without any explicit intervention (§ 4.4). Empirically, merging independently preference-optimized models has been reported to be competitive with or superior to data mixing, both for balancing safety and general-purpose capabilities [1] and for balancing helpfulness, honesty, and harmlessness [Yang et al., 2025]. Our analysis provides theoretical support for these empirical findings, attributing the effectiveness of merging DPO-trained models to the dual sparsity of their updates and the resulting small cross-task interference. Our merging experiments (§ 4.4) show that this translates into smaller performance degradation than SFT under task-vector merging; a larger-scale study of merging is left to future work.

Conclusion

We have presented a theoretical analysis of why DPO induces sparse parameter updates. Under the NTK linearization, the logistic structure of the DPO loss drives the update direction toward a maximum-margin solution that exhibits dual sparsity: it is determined by a small number of support pairs, and the gradient feature vector of each pair is itself sparse due to gradient cancellation between preferred and dispreferred responses. Relating this structure to weight disentanglement, we showed empirically that DPO reduces cross-task interference by 11× to 35× relative to SFT and degrades less under model merging, and that both sources of sparsity hold from GPT-2 to 7–9B models.

Limitations. Our analysis relies on the NTK linearization, which is not guaranteed to hold throughout DPO training of large models. More fundamentally, parameter-level sparsity (assumption 2) is an empirically motivated assumption rather than a consequence derived from the architecture. It relies on token overlap between the preferred and dispreferred responses: for pairs whose responses are entirely disjoint (e.g., different topics or formats), the gradient cancellation does not occur, and this source of sparsity may weaken or vanish. A formal derivation of assumption 2, and a characterization of the degree of overlap it requires in practice, are left to future work.

Proof and Numerical Evaluation of Lemma 2

Proof

Throughout, we work with the frozen-NTK surrogate of the margin, milin(θ):=mi(θ0)+⟨Φi,θ−θ0⟩m_i^{\mathrm{lin}}(\theta) := m_i(\theta_0) + \langle\Phi_i, \theta- \theta_0 \rangle, under which the DPO loss reduces exactly to the logistic regression eq. (7); lemma 2 holds exactly for this model. Under πθ0=πref\pi_{\theta_0} = \pi_{\mathrm{ref}}, the offsets vanish, bi=mi(θ0)=0b_i = m_i(\theta_0) = 0, so that u⋆u^\star solves max⁡∥u∥≤1min⁡i∈[n]⟨u,Φi⟩\max_{\lVert u\rVert\le1} \min_{i \in[n]} \langle u, \Phi_i\rangle. By assumption 1, the optimal value γ⋆\gamma_\star is positive, and hence the constraint is active: ∥u⋆∥=1\lVert u^\star\rVert= 1.

KKT conditions. The problem is equivalent to the hard-margin problem w⋆=arg min⁡w12∥w∥2w^\star= \operatorname*{arg\,min}_w \frac{1}{2}\lVert w\rVert^2 subject to ⟨w,Φi⟩≥1\langle w, \Phi_i\rangle\ge1 for all ii, with u⋆=w⋆/∥w⋆∥u^\star= w^\star/\lVert w^\star\rVert and γ⋆=1/∥w⋆∥\gamma_\star= 1/\lVert w^\star\rVert. Its KKT conditions give w⋆=∑iα^iΦiw^\star= \sum_i \hat{\alpha}_i\Phi_i with α^i≥0\hat{\alpha}_i \ge0 and α^i(⟨w⋆,Φi⟩−1)=0\hat{\alpha}_i(\langle w^\star, \Phi_i\rangle- 1) = 0. Thus αi⋆=α^i/∥w⋆∥\alpha_i^\star= \hat{\alpha}_i/\lVert w^\star\rVert in eq. (10), and complementary slackness yields

⟨u⋆,Φi⟩=γ⋆for all i∈SV.(19)\langle u^\star, \Phi_i\rangle= \gamma_\star\qquad\text{for all } i \in\mathrm{SV}. \tag*{(19)}

Bound. Let S:=SVS := \mathrm{SV}, s:=∣S∣s := |S|, and 1S∈Rs\mathbf{1}_S \in\mathbb{R}^s the all-ones vector. Summing eq. (19) over SS and applying the Cauchy–Schwarz inequality with ∥u⋆∥=1\lVert u^\star\rVert= 1,

sγ⋆=⟨u⋆,∑i∈SΦi⟩≤∥∑i∈SΦi∥=(1STKS,S1S)1/2.(20)s\gamma_\star= \left\langle u^\star, \sum_{i\in S}\Phi_i\right\rangle\le\left\lVert\sum_{i\in S}\Phi_i\right\rVert= \left(\mathbf{1}_S^{\mathsf T}K_{S,S}\mathbf{1}_S\right)^{1/2}. \tag*{(20)}

By the Rayleigh quotient, 1STKS,S1S≤λmax⁡(KS,S)∥1S∥2=sλmax⁡(KS,S)\mathbf{1}_S^{\mathsf T}K_{S,S}\mathbf{1}_S \le\lambda_{\max}(K_{S,S})\lVert\mathbf{1}_S\rVert^2 = s\lambda_{\max}(K_{S,S}). Combining the two, s2γ⋆2≤sλmax⁡(KS,S)s^2\gamma_\star^2 \le s\lambda_{\max}(K_{S,S}), i.e., sγ⋆2≤λmax⁡(KS,S)s\gamma_\star^2 \le\lambda_{\max}(K_{S,S}). Since KS,SK_{S,S} is a principal submatrix of the positive semidefinite matrix KK, Cauchy’s interlacing theorem gives λmax⁡(KS,S)≤λmax⁡(K)\lambda_{\max}(K_{S,S}) \le\lambda_{\max}(K). Finally, since ss is an integer with s≤ns \le n, we obtain s≤min⁡{n,⌊λmax⁡(K)/γ⋆2⌋}s \le\min\{n, \lfloor\lambda_{\max}(K)/\gamma_\star^2\rfloor\}. □\square

Numerical Evaluation

Computing γ⋆\gamma_\star exactly requires solving the max-margin problem. We instead evaluate the bound through the finite-time margin of the normalized trajectory, which is justified by the following corollary.

Corollary 4 (Finite-time evaluation of the bound). Under the assumptions of lemma 2, let Δθ(T)≠0\Delta\theta(T) \ne0 and define the finite-time margin

γ~T:=min⁡i∈[n]⟨Δθ(T)∥Δθ(T)∥,Φi⟩.(21)\widetilde{\gamma}_T := \min_{i\in[n]} \left\langle\frac{\Delta\theta(T)}{\lVert\Delta\theta(T)\rVert}, \Phi_i \right\rangle. \tag*{(21)}

Then γ~T≤γ⋆\widetilde{\gamma}_T \le\gamma_\star. If, moreover, Δθ(T)\Delta\theta(T) follows the gradient flow of the frozen-NTK surrogate, then γ~T→γ⋆\widetilde{\gamma}_T \to\gamma_\star as T→∞T \to\infty. Consequently, whenever γ~T>0\widetilde{\gamma}_T > 0,

∣SV∣≤min⁡{n,⌊λmax⁡(K)γ⋆2⌋}≤min⁡{n,⌊λmax⁡(K)γ~T2⌋}.(22)|\mathrm{SV}| \le\min\left\{n,\left\lfloor\frac{\lambda_{\max}(K)}{\gamma_\star^2}\right\rfloor\right\} \le\min\left\{n,\left\lfloor\frac{\lambda_{\max}(K)}{\widetilde{\gamma}_T^2}\right\rfloor\right\}. \tag*{(22)}

Proof. Let uT:=Δθ(T)/∥Δθ(T)∥u_T := \Delta\theta(T)/\|\Delta\theta(T)\|. Since ∥uT∥=1\|u_T\| = 1, uTu_T is feasible for the max-margin problem, and hence γ~T=min⁡i⟨uT,Φi⟩≤max⁡∥u∥≤1min⁡i⟨u,Φi⟩=γ⋆\widetilde{\gamma}_T = \min_i\langle u_T, \Phi_i\rangle\le\max_{\|u\|\le1}\min_i\langle u, \Phi_i\rangle= \gamma_\star. Under the frozen-NTK gradient flow, lemma 1 gives uT→u⋆u_T \to u^\star; since u↦min⁡i⟨u,Φi⟩u \mapsto\min_i\langle u, \Phi_i\rangle is continuous as the minimum of finitely many continuous functions, γ~T→min⁡i⟨u⋆,Φi⟩=γ⋆\widetilde{\gamma}_T \to\min_i\langle u^\star, \Phi_i\rangle= \gamma_\star. Finally, if γ~T>0\widetilde{\gamma}_T > 0, then 1/γ⋆2≤1/γ~T21/\gamma_\star^2 \le1/\widetilde{\gamma}_T^2, which, combined with eq. (11), ∣SV∣≤n|\mathrm{SV}| \le n, and the integrality of ∣SV∣|\mathrm{SV}|, yields the chain of inequalities. □\square

The values reported below are computed from the actual training trajectory of the network rather than from the frozen-NTK gradient flow. For such a trajectory, the convergence statement of corollary 4 does not apply, but the inequality γ~T≤γ⋆\widetilde{\gamma}_T \le\gamma_\star still holds, since it uses only ∥uT∥=1\|u_T\| = 1; the reported ratio λmax⁡(K)/γ~T2\lambda_{\max}(K)/\widetilde{\gamma}_T^2 is therefore a conservative (upper) estimate of λmax⁡(K)/γ⋆2\lambda_{\max}(K)/\gamma_\star^2. Table 5 reports this quantity in the setting of Figure 1 (GPT-2 Small/Medium, HH-RLHF, n=100n = 100). For both models it is about 1/7 to 1/9 of nn, guaranteeing that ∣SV∣|\mathrm{SV}| is roughly an order of magnitude smaller than nn, consistent with the concentration observed in § 4.2.

Figure 1. Distribution of AiA_i across data pairs for GPT-2 Small (top) and Medium (bottom) at 1 epoch (left), 100 epochs (middle), and 1000 epochs (right) on HH-RLHF. Dark bars indicate the top 10% of pairs by weight. As training progresses, the distribution polarizes onto a small number of support pairs; this effect is more pronounced in the larger model.

Modelλmax⁡(K)/γ~T2\lambda_{\max}(K)/\widetilde{\gamma}_T^2Implied bound on ∣SV∣|\mathrm{SV}|
GPT-2 Small14.58≤14\le14
GPT-2 Medium11.33≤11\le11

Table 5. Conservative evaluation of the support-set bound eq. (11) in the setting of Figure 1 (n=100n = 100).

Experimental Details

Time-integrated weights. The weights Ai(T)=∫0Tσ(−mi(θ(t))) dtA_i(T) = \int_0^T \sigma(-m_i(\theta(t)))\,dt are approximated by accumulating the discrete-time analogue ∑tσ(−mi(θt))\sum_t \sigma(-m_i(\theta_t)) over training steps. Since AiA_i depends only on the margins, it requires no gradient features.

Experimental settings. Table 6 summarizes the models, datasets, and numbers of preference pairs used in each experiment. The full fine-tuning DPO runs for the 7–8B models use up to eight NVIDIA A100 GPUs.

ExperimentModelsDatasetPairsTraining
Data-level (Figure 1)GPT-2 S/MHH100DPO, 1000 ep.
Data-level (table 1)GPT-2 S/MHH1000DPO, 1000 ep.
Llama-3-8BUF1000Full-FT DPO, 3 ep.
Mistral-7BUF1000Full-FT DPO, 3 ep.
Parameter-level (Figure 2)GPT-2 MHH100–
Parameter-level (table 2)All fiveUF1000–
Interference (table 3)GPT-2 S/MHH, UF100DPO / SFT
Model merging (table 4)GPT-2 MHH, UFsection DDPO / SFT

Table 6. Experimental settings. “Pairs” is the number of preference pairs on which each quantity is computed. HH: Anthropic HH-RLHF; UF: UltraFeedback Binarized; ep.: epochs.

Evolution of Log-Probabilities During DPO Training

We record the log-probabilities of the preferred and dispreferred responses along the training trajectory that produces Figure 1 (GPT-2 Medium, HH-RLHF, N=100N = 100). Each value is the sequence-level log-probability averaged over the 100 pairs, with logps/chosen⁡=log⁡πθ(y+∣x)\operatorname{logps/chosen} = \log\pi_\theta(y^+ \mid x) and logps/rejected⁡=log⁡πθ(y−∣x)\operatorname{logps/rejected} = \log\pi_\theta(y^- \mid x).

As shown in Table 7, both log-probabilities decrease, but the rejected log-probability decreases considerably faster, so the gap widens from 11.7 to 128.6. Since πθ0=πref\pi_{\theta_0} = \pi_{\mathrm{ref}}, the margin satisfies mi(θ)=β(gπi(θ)−gπi(θ0))m_i(\theta) = \beta\left(g_{\pi_i}(\theta) - g_{\pi_i}(\theta_0)\right), where gapi_i denotes the per-pair log-probability gap; a widening gap thus directly increases the margin. As a result, σ(−mi)→0\sigma(-m_i) \to0 for easy pairs, which stops their contribution to AiA_i and produces the polarization of AiA_i observed in Figure 1.

Epochlogps/chosenlogps/rejectedGap (chosen − rejected)
1−247.2-247.2−258.9-258.911.7
100−280.5-280.5−334.5-334.554.0
1000−308.3-308.3−436.9-436.9128.6

Table 7. Average log-probabilities of preferred (chosen) and dispreferred (rejected) responses during DPO training (GPT-2 Medium, HH-RLHF, N=100N = 100).

Model Merging Experiments

Setup. For each task, we fine-tune GPT-2 Medium with DPO or SFT until convergence and take the task vector τ=θ⋆−θ0\tau= \theta^{\star} - \theta_0 from the best checkpoint. For each target task, we form θmerged=θ0+α(τtarget+τother)\theta_{\mathrm{merged}} = \theta_0 + \alpha\left(\tau_{\mathrm{target}} + \tau_{\mathrm{other}}\right), where α∈{0,0.1,…,2.0}\alpha\in\{0, 0.1, \ldots, 2.0\} is selected on a validation set, and report Δ=(single-task)−(merged)\Delta= \text{(single-task)} - \text{(merged)}. We use the same two task pairs as in § 4.4.

Results. Table 8 reports the single-task and merged performance. On both task pairs, DPO degrades less than SFT under merging. The gap between the two methods is smaller than the order-of-magnitude reduction in the cross-task inner product (Table 3), since merged performance also depends on factors other than parameter-space overlap.

Task PairMethodSingleMergedΔ\Delta
Task 1Task 2Task 1Task 2Task 1Task 2
HH help↔harmSFT0.6310.6250.5210.5230.1100.102
DPO0.6680.6450.6410.6190.0270.026
HH help↔UFSFT0.6310.5350.5180.5050.0960.030
DPO0.6680.5720.6320.5580.0360.014

Table 8. Single-task and merged performance of GPT-2 under task-vector merging. Δ\Delta is the single-task minus merged performance (lower is better).

Licenses

Table 9 lists the existing assets used in this paper together with their licenses and the versions (Hugging Face identifiers) we use.

AssetVersion (Hugging Face)License
GPT-2 Smallopenai-community/gpt2Modified MIT
GPT-2 Mediumopenai-community/gpt2-mediumModified MIT
Llama-3-8B-Instructmeta-llama/Meta-Llama-3-8B-InstructLlama 3 Community License
Mistral-7B-Instruct-v0.2mistralai/Mistral-7B-Instruct-v0.2Apache 2.0
Gemma-2-9B-itgoogle/gemma-2-9b-itGemma Terms of Use
Anthropic HH-RLHFAnthropic/hh-rlhfMIT
UltraFeedback BinarizedHuggingFaceH4/ultrafeedback_binarizedMIT

Table 9. Existing assets, versions, and licenses.

References

  1. [1]Aakanksha, Arash Ahmadian, Seraphina Goldfarb-Tarrant, Beyza Ermis, Marzieh Fadaee, and Sara Hooker. Mix data or merge models? optimizing for diverse multi-task learning. arXiv preprint arXiv:2410.10801, 2024.arxiv.org/abs/2410.10801
  2. [2]Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSIlva, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.arxiv.org/abs/2204.05862
  3. [3]Ganqiu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with scaled ai feedback. In International Conference on Machine Learning, pages 9722–9744. PMLR, 2024.arxiv.org/abs/2310.01377
  4. [4]Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. In International Conference on Machine Learning, 2024.arxiv.org/abs/2402.01306
  5. [5]Gemma Team. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024.
  6. [6]Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, 2024.arxiv.org/abs/2310.12036
  7. [7]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.arxiv.org/abs/2407.21783
  8. [8]Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations, 2023.arxiv.org/abs/2212.04089
  9. [9]Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, volume 31, pages 8571–8580, 2018.arxiv.org/abs/1806.07572
  10. [10]Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023.arxiv.org/abs/2310.06825
  11. [11]Sagnik Mukherjee, Lifan Yuan, Dilek Hakkani-Tur, and Hao Peng. Reinforcement learning finetunes small subnetworks in large language models. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2505.11711
  12. [12]Guillermo Ortiz-Jiménez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Advances in Neural Information Processing Systems, volume 36, pages 66727–66754, 2023.
  13. [13]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 2019.
  14. [14]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pages 53728–53741, 2023.arxiv.org/abs/2305.18290
  15. [15]Noam Razin, Sadhika Malladi, Aditya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional unalignment: Likelihood displacement in direct preference optimization. In International Conference on Learning Representations, 2025.arxiv.org/abs/2410.08847
  16. [16]Yi Ren and Danica J. Sutherland. Learning dynamics of LLM finetuning. In International Conference on Learning Representations, 2025.arxiv.org/abs/2407.10490
  17. [17]Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70):1–57, 2018.arxiv.org/abs/1710.10345

Paper details

vlnc:W_j2kxm9vd
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents