betaFeedback

WorksComputer Science

Margin Dynamics for Large Language Model Alignment

Abstract

Practitioners choose among LLM alignment methods largely by trial and error. We provide a theoretical foundation: each pairwise method defines a vector field on the probability simplex that governs how the model redistributes mass between preferred and dispreferred completions. Projecting onto a single preferred/dispreferred pair reduces this to a one-dimensional margin ODE with a closed-form solution. These solutions reveal that DPO margins grow without bound (logarithmically as O(log⁡t)O(\log t) in the gradient-flow limit) while IPO margins converge to a finite 1/(2τ)1/(2τ); that label noise induces a qualitative phase transition in DPO (unbounded →\to bounded) but affects IPO only quantitatively; and that gradient concentration under asymmetric weighting governs trainability at scale. We confirm all three predictions across five model families spanning four benchmarks. The closed-form solutions also enable principled method design: we prove that adding any label smoothing ε>0\varepsilon > 0 to SimPO creates a finite margin attractor at Δ∗=β−1log⁡((1−ε)/ε)Δ^* = β^{-1}\log((1-\varepsilon)/\varepsilon), while ε=0\varepsilon = 0 (standard SimPO) has no attractor. We test this across three models (8B-9B) and confirm the prediction: the theory-derived configuration (TPO-S, ε=0.05\varepsilon = 0.05) is competitive with the strongest baselines, attaining the highest mean IFEval score among the compared methods on all three, validating that the predicted finite attractor is a usable operating point rather than proposing a new state-of-the-art loss.

Introduction

The alignment toolkit has grown rapidly: supervised fine-tuning (SFT; Ouyang et al. [21]), reinforcement learning from human feedback (PPO; Schulman et al. [25]), and a family of offline preference methods that bypass reward modeling by optimizing pairwise losses directly (DPO, Rafailov et al. [22]; IPO, Gheshlaghi Azar et al. [9]; SimPO, Meng et al. [19]; KTO, Ethayarajh et al. [7]; GRPO, Shao et al. [26]). Each introduces different losses, hyperparameters, and failure modes. Selecting among them remains empirical: a method that excels on one model size or task can collapse on another, and no existing theory explains why.

We close this gap by analyzing alignment as margin dynamics on the probability simplex. Every method defines a field F=F+−F−F = F^{+} - F^{-} governing how the model redistributes mass across completions. Projecting onto a two-completion effective theory yields margin ODEs with closed-form solutions. DPO’s margin grows without bound (sub-linearly, O(log⁡t)O(\log t) under gradient flow) while IPO’s converges to a finite value, and a single fitted calibration parameter βeff\beta_{\mathrm{eff}} maps the theory to full-scale training across all settings we test. Adam enters as a strictly positive time reparameterization, so we claim the functional character (unbounded vs. bounded) and the invariant set rather than the step-time rate constant, which βeff\beta_{\mathrm{eff}} absorbs (Appendix A.7).

The closed-form solutions organize the alignment landscape along three axes (Figure 1). Axis 1, Repulsion: methods with targeted repulsion (DPO, IPO, SimPO, KTO) explicitly push down dispreferred completions and maintain Ω(π)\Omega(\pi) gradient pressure, while implicit-only methods (SFT, RFT) suffer O(π2)O(\pi^2) self-limiting decay. Axis 2, Weighting: the weighting function w(Δ)w(\Delta) determines margin dynamics. Sigmoid weights (DPO, SimPO) produce unbounded margins, linear weights (IPO, PPO) converge to finite margins, and label noise induces a qualitative phase transition in the former but only a quantitative shift in the latter. Axis 3, Reference: frozen references (DPO, IPO) cause gradient decay as the policy drifts, reference-free methods (SimPO) avoid decay but risk unbounded drift, and asymmetric weighting (DPOγ_\gamma) exposes a gradient-concentration regime that governs trainability at scale. We derive quantitative predictions along each axis and confirm them across five model families (§ 3.1).

The theoretical framework also specifies desired dynamical properties and facilitates identifying losses that produce them. The noise phase transition (Theorem 2.4) predicts that adding any label smoothing ε>0\varepsilon> 0 to SimPO creates a stable fixed point at Δ∗=β−1log⁡((1−ε)/ε)\Delta^* = \beta^{-1}\log((1-\varepsilon)/\varepsilon), converting unbounded margin growth into convergent dynamics. We set ε=0.05\varepsilon=0.05 (Δ∗≈29\Delta^* \approx29) and call this configuration TPO-S (Targeted Preference Optimization, label-Smoothed variant) for brevity; we also test a Huber-loss variant (TPO-H) derived from the gradient concentration analysis. In summary, this paper makes three contributions: (i) a margin-dynamics framework yielding closed-form ODEs for major pairwise alignment methods, producing a three-axis taxonomy with testable predictions; (ii) quantitative confirmation across five model families and four benchmarks; and (iii) a constructive test in which the theory-predicted configuration (TPO-S, ε=0.05\varepsilon=0.05), specified a priori with no tuning, is confirmed to be a competitive operating point — attaining the highest mean IFEval among the compared methods across three models (8B–9B) — validating the finite-attractor prediction rather than proposing a state-of-the-art loss.

Margin Dynamics and Constructive Validation

Let πθ(y∣x)\pi_\theta(y|x) denote a language model policy parameterized by θ\theta. For a fixed prompt xx and completions Yx={y1,…,yn}\mathcal{Y}_x = \{y_1,\ldots,y_n\}, the policy defines a point on the probability simplex Δn−1\Delta^{n-1}. Each training method defines a vector field F:Δn−1→TΔn−1F:\Delta^{n-1}\to T\Delta^{n-1} governing the evolution of π\pi, tangent to the simplex (∑iFi=0\sum_i F_i=0). We decompose F=F+−F−F=F^+ - F^- into attraction (toward desired outputs) and repulsion (away from undesired outputs), following the decomposition of Deng et al. [ [6]. The key step is projecting this field onto a two-completion simplex {yw,yl}\{y_w,y_l\}, reducing the dynamics to a one-dimensional margin ODE Δ˙=w(Δ)\dot{\Delta}=w(\Delta) that admits closed-form solutions. Here Δ=log⁡(π(yw)/π(yl))\Delta=\log(\pi(y_w)/\pi(y_l)) is the log-margin between preferred and dispreferred completions, σ\sigma denotes the sigmoid function, and β\beta, τ\tau, γ\gamma are method-specific parameters defined below. To ground the notation, consider DPO Rafailov et al. [ [22]:

LDPO=−log⁡σ(β(Δθ−Δref)),Δθ=log⁡πθ(yw∣x)πθ(yl∣x),Δref=log⁡πref(yw∣x)πref(yl∣x).(1)\mathcal{L}_{\mathrm{DPO}}=-\log\sigma\left(\beta(\Delta_\theta-\Delta_{\mathrm{ref}})\right),\qquad\Delta_\theta=\log\frac{\pi_\theta(y_w|x)}{\pi_\theta(y_l|x)},\qquad\Delta_{\mathrm{ref}}=\log\frac{\pi_{\mathrm{ref}}(y_w|x)}{\pi_{\mathrm{ref}}(y_l|x)}. \tag*{(1)}

where β>0\beta>0 is the inverse temperature. More generally, weighting the rejected term by γ∈[0,1]\gamma\in[0,1] gives the asymmetric variant LDPOγ=−log⁡σ(βΔγ)\mathcal{L}_{\mathrm{DPO}_\gamma}=-\log\sigma(\beta\Delta_\gamma), where Δγ=(log⁡πθ(yw∣x)−log⁡πθ(yl∣x))−γ(log⁡πθ(yl∣x)−log⁡πref(yl∣x))\Delta_\gamma=(\log\pi_\theta(y_w|x)-\log\pi_{\theta}(y_l|x))-\gamma(\log\pi_\theta(y_l|x)-\log\pi_{\mathrm{ref}}(y_l|x)); γ=0\gamma=0 removes the explicit penalty on dispreferred completions, γ=1\gamma=1 recovers standard DPO. IPO Gheshlaghi Azar et al., [9] replaces the sigmoid with a squared penalty: LIPO=(Δθ−Δref−1/(2τ))2\mathcal{L}_{\mathrm{IPO}}=(\Delta_\theta-\Delta_{\mathrm{ref}}-1/(2\tau))^2, where τ\tau controls regularization strength. We develop the solutions along three axes, each revealing a qualitatively different aspect of alignment dynamics.

The most fundamental distinction among methods is how they decrease probability on undesired completions. SFT relies solely on the normalization constraint; preference and RL methods explicitly push probability down.

Definition 2.1 (Repulsion types). A method has targeted repulsion if its loss explicitly decreases π(y)\pi(y) for specific completions. It has implicit-only repulsion if the only mechanism is normalization.

On the two-completion simplex under softmax parameterization, these two repulsion types produce different convergence rates (Figure 1a), establishing the first axis of our taxonomy.

Three axes of the margin dynamics framework: repulsion, margin, and reference

Figure 1. Three axes of the margin dynamics framework. (a) SFT only attracts toward ywy_w (F+F^+), with pressure vanishing as π(yl)→0\pi(y_l) \to0. DPO adds targeted repulsion F−F^-, sustaining Ω(π)\Omega(\pi) pressure. (b) Sigmoid weighting (DPO) yields unbounded margin growth; linear weighting (IPO, PPO) converges. (c) Frozen references cause gradient decay; reference-free methods risk drift; adaptive references stay stable.

Theorem 2.2 (Convergence rate separation). On a two-completion simplex under softmax parameterization with learning rate η\eta: (i) SFT gives π˙(yl)=−2ηπ(yl)2(1−π(yl))\dot{\pi}(y_l)=-2\eta\pi(y_l)^2(1-\pi(y_l)), so ∣π˙(yl)∣=O(π(yl)2)|\dot{\pi}(y_l)|=O(\pi(y_l)^2) as π(yl)→0\pi(y_l)\to0 (self-limiting). (ii) DPO gives π˙(yl)=−2ηβσ(−βΔ)π(yl)(1−π(yl))\dot{\pi}(y_l)=-2\eta\beta\sigma(-\beta\Delta)\pi(y_l)(1-\pi(y_l)) with

σ(−βΔ)≥c(β,πref,δ)>0\sigma(-\beta\Delta) \ge c(\beta,\pi_{\mathrm{ref}},\delta) > 0 for π(yl)≥δ>0\pi(y_l) \ge\delta> 0, so ∣π˙(yl)∣=Ω(π(yl))|\dot{\pi}(y_l)| = \Omega(\pi(y_l)) for π(yl)≥δ\pi(y_l) \ge\delta (sustained while π(yl)\pi(y_l) remains bounded away from zero). Here cc depends on β\beta, πref\pi_{\mathrm{ref}}, and δ\delta through the bounded margin Δ≤log⁡((1−δ)/δ)+∣log⁡(πref(yw)/πref(yl))∣\Delta\le\log((1-\delta)/\delta) + |\log(\pi_{\mathrm{ref}}(y_w)/\pi_{\mathrm{ref}}(y_l))|.

The O(π2)O(\pi^2) vs. Ω(π)\Omega(\pi) separation means SFT’s gradient on bad completions vanishes quadratically while DPO maintains linear pressure. This separates SFT-family from preference methods but does not differentiate within the preference family. To do so, we turn to the weighting function governing margin evolution.

All pairwise preference methods induce F[π](y)∝w(Δ)⋅(1y=yw−1y=yl)F[\pi](y) \propto w(\Delta)\cdot(\mathbf{1}_{y=y_w}-\mathbf{1}_{y=y_l}), differing only in the weighting function ww: DPO/SimPO use w(Δ)=βσ(−βΔ)w(\Delta)=\beta\sigma(-\beta\Delta) (sigmoid, saturating); IPO uses w(Δ)=τ−2τ2Δw(\Delta)=\tau-2\tau^2\Delta (linear); KTO uses a bell-shaped ww that peaks near Δ=0\Delta=0 and decays for both large positive and large negative margins, arising from its prospect-theoretic value function [7]; PPO uses w∝Δ∗−Δw\propto\Delta^*-\Delta (linear, reward-driven). The margin ODE Δ˙=w(Δ)\dot{\Delta}=w(\Delta) (absorbing 2η2\eta into the time scale; Appendix A) is a 1D autonomous ODE whose solutions we derive in closed form. All results are stated under gradient flow (continuous time); discretization from finite learning rate introduces O(η)O(\eta) errors that do not affect the qualitative predictions (unbounded vs. bounded margins, phase transitions) but may shift quantitative fixed points. In practice, Adam’s preconditioner acts as a state-dependent time reparameterization of the margin ODE (Appendix A.7), preserving each trajectory’s functional form while the time scale is absorbed into βeff\beta_{\mathrm{eff}}.

Theorem 2.3 (Margin dynamics without noise). (i) DPO: The margin grows without bound: implicit solution u+log⁡u=β2t+Cu+\log u=\beta^2t+C where u=eβΔu=e^{\beta\Delta}; asymptotically Δ(t)∼1βlog⁡(β2t)\Delta(t)\sim\frac{1}{\beta}\log(\beta^2t). (ii) IPO: The margin converges exponentially: Δ(t)=12τ2+(Δ(0)−12τ2)e−2τ2t\Delta(t)=\frac{1}{2\tau^2}+\left(\Delta(0)-\frac{1}{2\tau^2}\right)e^{-2\tau^2t}.

This distinction sharpens dramatically under noise, revealing a phase transition that constitutes one of our central findings.

Theorem 2.4 (Margin dynamics with label noise). Under the mean-field approximation (averaging over noise realizations; exact in the large-sample limit), with fraction ϵ∈(0,1/2)\epsilon\in(0,1/2) of labels flipped: (i) DPO: unique stable fixed point ΔDPO∗=1βlog⁡1−ϵϵ\Delta^*_{\mathrm{DPO}}=\frac{1}{\beta}\log\frac{1-\epsilon}{\epsilon}. Without noise, Δ∗=∞\Delta^*=\infty; noise acts as implicit regularization. (ii) IPO: fixed point ΔIPO∗=1−2ϵ2τ2\Delta^*_{\mathrm{IPO}}=\frac{1-2\epsilon}{2\tau^2} with convergence rate 2τ22\tau^2 (unchanged by noise).

Noise changes DPO’s behavior qualitatively (unbounded → bounded) but IPO’s only quantitatively (shifted fixed point, same rate). This phase transition explains IPO’s empirical noise robustness [9] and directly motivates TPO’s label-smoothing term later in this section.

The mean-field approximation introduces O(1/B)O(1/\sqrt{B}) fluctuations for batch size BB; with our effective batch size of 4, the transition is smoothed but the fixed-point location is preserved, as confirmed by the consistent βeff\beta_{\mathrm{eff}} across noise levels (§ 3.1).

The third axis concerns what happens when NN pairs share parameters: the gradient becomes dominated by hard examples. Let vj=∇θΔjv_j = \nabla_\theta\Delta_j and wi=∇θΔγ,iw_i = \nabla_\theta\Delta_{\gamma,i} for asymmetric DPO with γ∈[0,1]\gamma\in[0,1].

Theorem 2.5 (Gradient concentration). Under gradient flow, the margin of example jj evolves as

Δ˙j=βN∑i=1Nσ(−βΔγ,i)⋅vjTwi.\dot{\Delta}_j = \frac{\beta}{N}\sum_{i=1}^{N}\sigma(-\beta\Delta_{\gamma,i})\cdot v_j^{\mathsf{T}}w_i.

For well-learned examples (i.e., πθ(yjr)<πref(yjr)\pi_\theta(y_j^r) < \pi_{\mathrm{ref}}(y_j^r), equivalently Δγ,j≫1/β\Delta_{\gamma,j} \gg1/\beta), the self-reinforcement term is exponentially small. The dynamics are dominated by hard examples (Δγ,i=O(1/β)\Delta_{\gamma,i} = O(1/\beta)). Higher γ\gamma accelerates this concentration.

When the concentrated gradient interferes negatively with easy examples (vjTwi<0v_j^{\mathsf{T}}w_i < 0), their margins decrease, causing margin collapse. This motivates TPO’s bounded weighting (§ 2.1).

Bridging theory to practice. The theorems describe isolated two-completion dynamics, but real training involves NN pairs sharing parameters. The gradient concentration result (Theorem 2.5) provides the bridge: each pair’s effective dynamics differ from the isolated prediction by cross-example interference terms, formalized through an effective coupling parameter.

Remark 2.6 (Effective coupling approximation). Consider NN training pairs sharing parameters, with margin gradients vj=∇θΔjv_j = \nabla_\theta\Delta_j and loss gradients wi=∇θΔγ,iw_i = \nabla_\theta\Delta_{\gamma,i}. The margin of pair jj evolves as Δ˙j=(β/N)∑iσ(−βΔi)⋅vjTwi\dot{\Delta}_j = (\beta/N)\sum_i \sigma(-\beta\Delta_i)\cdot v_j^{\mathsf{T}}w_i (Theorem 2.5). Define the effective coupling βeff,j=Δ˙j/σ(−βΔj)\beta_{\mathrm{eff},j} = \dot{\Delta}_j/\sigma(-\beta\Delta_j). Then, for γ=1\gamma= 1 (so that wj=vjw_j = v_j):

βeff,j=βN[∥vj∥2+∑i≠jσ(−βΔi)σ(−βΔj)⋅vjTwi],(2)\beta_{\mathrm{eff},j} = \frac{\beta}{N}\left[\lVert v_j\rVert^2 + \sum_{i\ne j}\frac{\sigma(-\beta\Delta_i)}{\sigma(-\beta\Delta_j)}\cdot v_j^{\mathsf{T}}w_i\right], \tag*{(2)}

where the first term is the self-reinforcement and the sum captures cross-example interference. Assumption: if the cross-example gradient inner products {vjTwi}i≠j\{v_j^{\mathsf{T}}w_i\}_{i\ne j} are approximately independent with mean zero and variance σ⊥2\sigma_\perp^2 (e.g., when training examples are semantically diverse and gradients do not share a low-rank subspace), then βeff≈β∥vj∥2/N±O(βσ⊥Nactive/N)\beta_{\mathrm{eff}} \approx\beta\lVert v_j\rVert^2/N \pm O(\beta\sigma_\perp\sqrt{N_{\mathrm{active}}}/N), where Nactive=∣{i:σ(−βΔi)>1/e}∣N_{\mathrm{active}} = |\{i : \sigma(-\beta\Delta_i) > 1/e\}| counts examples with non-negligible gradient weight. The independence assumption is strong and may be violated in practice (e.g., LoRA gradients share a low-rank subspace; thematically correlated datasets induce gradient correlations); the approximation should be interpreted as a scaling relationship rather than a precise prediction. This remark is the principal bridge between the idealized two-completion ODE and multi-example training; its informality reflects the strong independence assumption required, which is satisfied approximately but not rigorously in LoRA training.

Under the independence assumption, βeff/β\beta_{\mathrm{eff}}/\beta increases with dataset size: as NN grows, more examples reach large margins (σ(−βΔi)→0\sigma(-\beta\Delta_i) \to0), reducing NactiveN_{\mathrm{active}} relative to NN. The empirical values (∼1/4.4\sim1/4.4 to ∼1/1.9\sim1/1.9) are post-hoc fits; the remark provides a structural decomposition explaining the scaling trend. The proof is in Appendix § 6. In practice, we fit a single βeff\beta_{\mathrm{eff}} per data scale from the clean-data margin trajectory and hold it fixed across all noise conditions; the consistency of this fit across noise levels (§ 3.1) provides the critical test of the theory.

The two-completion projection is an effective theory: it predicts functional forms (logarithmic growth, exponential convergence, phase transitions) that hold across all tested n∈{2,4,8,16}n \in\{2,4,8,16\} completions (Table 10), while a single fitted calibration parameter βeff\beta_{\mathrm{eff}} absorbs the gap between the idealized two-completion dynamics and full-scale training (§ 3.1). We discuss its scope and limitations in Section 5. For iterative training, coverage gaps decay as ΔT≲Δ0(1−pmin⁡)KT\Delta_T \lesssim\Delta_0(1-p_{\min})^{KT} (Theorem A.9, Appendix § 6). Additional results (PPO dynamics, GRPO gradient vanishing, reference staleness, optimal adaptive reference, and diversity selection) are in Appendix § 6.

nnPairsΔ0\Delta_0Δ50\Delta_{50}Δ300\Delta_{300}Growth (Δ300/Δ50\Delta_{300}/\Delta_{50})Qualitative
220−0.011.1156.114.1×Unbounded
46080.3185.9281.71.5×Unbounded
8140106.0227.4360.21.6×Unbounded
1660099.1202.0319.01.6×Unbounded

Table 10. N-completion validation of the two-completion effective theory. DPO margin dynamics at n∈{2,4,8,16}n \in\{2, 4, 8, 16\} completions per prompt (DS-R1-8B, 300 steps). The growth ratio (final/step-50 margin) converges at n≥4n \ge4 (1.5–1.6×), confirming that the two-completion ODE captures the dominant dynamics regardless of the number of completions. Initial margins increase with nn due to the larger pairwise comparison space, but the functional form (unbounded growth) is invariant.

Table 1 organizes the landscape along the three axes, revealing two families: margin-driven methods (DPO, IPO, SimPO, KTO) and reward-driven methods (PPO, GRPO). PPO bridges both: its KL penalty induces linear margin dynamics like IPO (Proposition A.7). Table 2 maps each theoretical result to its experimental test. Having identified the failure modes along each axis, we derive a loss family that addresses all three simultaneously.

ResultPredictionExperimentConfirmed?
Main predictions (closed-form margin ODEs)
Thm. 2.3DPO Δ∼log⁡t\Delta\sim\log t; IPO Δ→1/(2τ)\Delta\to1/(2\tau)Figure 3✓ functional form
Thm. 2.4Noise: DPO phase transition, IPO rate-preservingFigure 4✓ 8% error
Thm. 2.2γ=0\gamma= 0 vs γ>0\gamma> 0 sharp separationFigure 5✓ qualitative
Supporting predictions (Appendix)
Thm. 2.5High γ→\gamma\to collapseFigure 5✓ non-monotonic
Thm. A.9Exponential gap decayApp. C✓ quantitative
Prop. A.3GRPO fails at low ppApp. C✓ qualitative
Prop. A.4Frozen ref constrains driftApp. C✓ 82× gap

Table 2. Theory-to-experiment mapping. Main predictions test the closed-form margin ODEs; supporting predictions are in the appendix. A single calibration parameter βeff\beta_{\mathrm{eff}} maps the two-completion theory to full-scale training.

MethodAxis 1: RepulsionAxis 2: WeightingAxis 3: Reference
SFT / RFT / STaRImplicit only——
PPO / RLHFTargetedLinear (non-sat.)Frozen πref\pi_{\mathrm{ref}} (KL)
GRPOTargeted (imbal.)Advantage-weightedNone
DPOTargetedSigmoid (saturating)Frozen πref\pi_{\mathrm{ref}}
IPOTargetedLinear (non-sat.)Frozen πref\pi_{\mathrm{ref}}
SimPOTargetedSigmoid (saturating)None
KTOTargetedBell-shaped (robust)Implicit (KL term)
TPO-HTargetedHuber (bounded)Reference-free
TPO-STargetedSigmoid + smoothingReference-free

Table 1. Three-axis taxonomy of LLM training methods.

From Theory to Loss Design

The framework is not only explanatory but constructive: specify desired dynamical properties of w(Δ)w(\Delta), then identify losses that produce them. We require five properties: D1 a finite attractor (w(Δ∗)=0w(\Delta^*) = 0); D2 convergence rate bounded away from zero (∣w∣|w| bounded below by a positive constant far from Δ∗\Delta^*); D3 stability (w′(Δ∗)<0w'(\Delta^*) < 0); D4 bounded gradient (∣w∣≤c|w| \le c), preventing the concentration of Theorem 2.5; and D5 asymmetry (stronger push below the target). SimPO’s w(Δ)=βσ(−Δ)w(\Delta) = \beta\sigma(-\Delta) is strictly positive (D1 fails); IPO’s w(Δ)=τ−2τ2Δw(\Delta) = \tau- 2\tau^2\Delta satisfies D1–D3 but grows without bound (D4 fails).

We present two complementary instantiations; the “TPO” prefix denotes targeted preference optimization: both losses are derived from desiderata D1–D5 but via different weighting functions (Huber vs. label-smoothed sigmoid), not a shared functional form. TPO-H (Huber variant) uses the clamped linear weighting w∗(Δ)=clamp⁡(Δ∗−Δ,−c,c)w^*(\Delta) = \operatorname{clamp}(\Delta^* - \Delta, -c, c), satisfying D1–D4 (but not D5). Integrating yields the Huber loss LTPO−HL_{\mathrm{TPO-H}} (Appendix § 7). TPO-S (label-smoothed sigmoid) exploits Theorem 2.4 directly: adding ε>0\varepsilon> 0 to SimPO yields w(Δ)=(1−ε)σ(−Δ)−εσ(Δ)w(\Delta) = (1-\varepsilon)\sigma(-\Delta) - \varepsilon\sigma(\Delta), with a unique zero at Δ∗=β−1log⁡((1−ε)/ε)\Delta^* = \beta^{-1}\log((1-\varepsilon)/\varepsilon):

LTPO−S=(1−ε)(−log⁡σ(δ))+ε(−log⁡σ(−δ)),δ=β(sw−sl−μ),(3)L_{\mathrm{TPO-S}} = (1-\varepsilon)(-\log\sigma(\delta)) + \varepsilon(-\log\sigma(-\delta)), \qquad\delta= \beta(s_w - s_l - \mu), \tag*{(3)}

where sw,sls_w, s_l are length-normalized log-probabilities (as in SimPO), with β=0.1\beta=0.1, μ=0.5\mu=0.5, ε=0.05\varepsilon=0.05. (We use μ\mu for SimPO’s target-margin parameter to avoid conflict with the asymmetric weighting γ∈[0,1]\gamma\in[0,1] in DPOγ_\gamma.) TPO-S satisfies all five desiderata and preserves SimPO’s sigmoid curriculum while the label-smoothing term creates a restoring force at Δ∗\Delta^* (Figure 2). The desiderata identify when and why these loss designs work, and predict the exact attractor location as a closed-form function of ε\varepsilon. While cDPO [20] also uses label smoothing, it retains a frozen reference and treats ε\varepsilon as a tuning knob; TPO-S is reference-free, includes a target margin μ>0\mu> 0, and derives ε\varepsilon from the phase transition (Theorem 2.4), giving a closed-form relationship between ε\varepsilon and the attractor location. The empirical gap between TPO-S and cDPO (2–6 points on IFEval, Table 3) reflects the reference axis (Axis 3), not ε\varepsilon itself. Llama 2’s preference margin loss [Touvron et al., 2023] also injects a margin term into the DPO sigmoid, using a per-example offset m(sw,sl)m(s_w,s_l) derived from Likert-scale annotations. However, such a shift leaves w(Δ)=βσ(−β(Δ−m/β))w(\Delta) = \beta\sigma(-\beta(\Delta- m/\beta)) strictly positive, so the margin still grows without bound, and only the initial condition is shifted. TPO-S’s ε\varepsilon changes the sign of ww at Δ∗\Delta^*, a qualitatively different modification that produces the finite attractor.

Effective potential $V(\Delta)$ for SimPO, TPO-H, and TPO-S

Figure 2. Effective potential V(Δ)V(\Delta) (schematic). SimPO (dashed) has no minimum: margins diverge. TPO-H and TPO-S have finite attractors at Δ∗\Delta^*.

DS-R1-8BQW3.5-9BMI-8B
TPO-S66.8±0.3\mathbf{66.8 \pm0.3}61.4±0.4\mathbf{61.4 \pm0.4}76.5±0.3\mathbf{76.5 \pm0.3}
TPO-H66.4±0.366.4 \pm0.360.8±0.360.8 \pm0.376.1±0.976.1 \pm0.9
SimPO66.5±0.666.5 \pm0.660.9±0.260.9 \pm0.273.9±1.473.9 \pm1.4
DPO65.1±0.265.1 \pm0.260.6±0.860.6 \pm0.871.9±0.671.9 \pm0.6
KTO64.2±0.864.2 \pm0.860.4±0.560.4 \pm0.572.2±1.072.2 \pm1.0
cDPO64.1±0.764.1 \pm0.759.8±0.459.8 \pm0.470.6±0.570.6 \pm0.5
IPO64.2±0.764.2 \pm0.759.7±0.659.7 \pm0.670.3±1.470.3 \pm1.4
ORPO52.6±1.752.6 \pm1.751.2±7.551.2 \pm7.549.5±6.249.5 \pm6.2
SFT63.163.159.859.870.370.3

Table 3. IFEval instruction-following accuracy (%) on UltraFeedback-trained models (mean ±\pm std, 5 seeds). IFEval is an objective benchmark with no judge model. The theory-derived TPO-S attains the highest mean among the compared methods on all three, confirming the predicted attractor is a competitive operating point. TPO-S adds negligible wall-clock overhead over SimPO: a single extra log⁡σ(−δ)\log\sigma(-\delta) evaluation per example (the ε\varepsilon-weighted label-smoothing term in Eq. (3)).1

What the theory adds over grid search. A practitioner grid-searching ε\varepsilon over SimPO would eventually find a good value. The theory predicts a qualitative phase transition: any ε>0\varepsilon> 0 creates a finite attractor while ε=0\varepsilon=0 does not, with the exact attractor location Δ∗=β−1log⁡((1−ε)/ε)\Delta^* = \beta^{-1}\log((1-\varepsilon)/\varepsilon) as a closed-form function of ε\varepsilon. Practitioners can directly control the attractor: ε=0.01\varepsilon=0.01 gives Δ∗≈46\Delta^* \approx46 (weaker regularization), ε=0.1\varepsilon=0.1 gives Δ∗≈22\Delta^* \approx22 (stronger). We set ε=0.05\varepsilon=0.05 (Δ∗≈29\Delta^* \approx29) as a moderate default for medium-level tuning. A sweep over ε∈{0.01,…,0.2}\varepsilon\in\{0.01,\ldots,0.2\} confirms the predicted monotonic ordering (Appendix § 10.3).

The desiderata paradox. Both TPO-H and TPO-S satisfy D1–D4; TPO-S additionally satisfies D5. The performance gap is small (0.4–0.6 points on IFEval), confirming that the finite-attractor property (D1–D3) is the primary driver. TPO-S’s sigmoid provides exponentially small but nonzero gradients everywhere, giving it a slight edge over TPO-H’s hard clipping. Full derivation in Appendix § 7.

The four theorems and the fitted βeff\beta_{\mathrm{eff}} approximation yield concrete, falsifiable predictions: specific functional forms, a noise-induced phase transition, and a concentration-driven collapse pattern, all testable with a single fitted calibration parameter per scale.

Experiments

We validate the framework’s predictions and test TPO across three models, two data regimes (on-policy and off-policy), and both preference accuracy and generation quality. We evaluate on MATH [12] (transition regime: 37% base solve rate), GSM8K [4] (high-coverage: >70%), and AIME 2024 [18] (low-coverage: 0%), spanning the coverage spectrum of Theorem A.9. Primary model: Qwen3-8B with LoRA [15] (r=64r=64). Baselines: SFT, RFT, DPO (β=0.1\beta=0.1), IPO (τ=0.1\tau=0.1), SimPO, KTO, cDPO, ORPO, GRPO; all use default hyperparameters from their respective papers unless noted. Dynamical predictions track margin Δ\Delta over training steps; we fit a single βeff\beta_{\mathrm{eff}} per data scale and use it across all noise conditions without refitting.

Prospective Predictions

We test three predictions derived from the margin ODEs. The first concerns the functional form of margin trajectories (Theorem 2.3): DPO’s sigmoid weighting predicts unbounded growth as O(log⁡t)O(\log t), while IPO’s linear weighting predicts exponential convergence to 1/(2τ)1/(2\tau). Figure 3 confirms this separation: DPO reaches 1540 at step 2000 while IPO oscillates near the predicted fixed point of 5. The separation replicates on full MATH (DPO 465 vs. IPO ∼\sim0), GSM8K (257 vs. 8.5), and MBPP [2] (160 vs. 1.1). A τ\tau-sweep validates the IPO fixed point quantitatively across τ∈{0.01,…,0.2}\tau\in\{0.01,\ldots,0.2\} (Appendix E.2).

Figure 3. Margin and gradient dynamics over 2000 training steps (Theorem 2.3). Left: DPO’s margin grows without bound, consistent with u+log⁡u=at+cu+\log u=at+c (R2=0.85R^2=0.85; the remaining variance reflects discrete-time effects and cross-example interference not captured by the isolated ODE). Right: DPO’s gradient norm is 10× SFT’s, confirming that targeted repulsion sustains Ω(π)\Omega(\pi) gradient pressure (Definition 2.1).

This distinction sharpens under noise, providing the second and most striking test. Theorem 2.4 predicts that label noise ε\varepsilon should bound DPO’s margin at Δ∗=β−1log⁡1−εε\Delta^*=\beta^{-1}\log\frac{1-\varepsilon}{\varepsilon} while leaving IPO’s convergence rate 2τ22\tau^2 unchanged. Figure 4 confirms: DPO’s margin decreases monotonically with noise (1041 →\rightarrow 114 →\rightarrow 97 →\rightarrow 64 for ε∈{0,0.05,0.1,0.2}\varepsilon\in\{0,0.05,0.1,0.2\}), while IPO remains stable. We fit βeff=β/4.4\beta_{\mathrm{eff}}=\beta/4.4 once from the clean-data trajectory and hold it constant across all noise conditions; consistency across noise levels (β/4.5\beta/4.5 and β/4.4\beta/4.4 from ε=0.1\varepsilon=0.1 and 0.2) confirms the log-formula. The ratio βeff/β\beta_{\mathrm{eff}}/\beta varies from ∼1/4.4\sim1/4.4 (500 samples) to ∼1/1.9\sim1/1.9 (full scale); predicting βeff\beta_{\mathrm{eff}} a priori remains open. The qualitative predictions (functional forms, phase transitions, ordering of attractors) are parameter-free; only the quantitative mapping to specific margin values requires βeff\beta_{\mathrm{eff}}.

Figure 4. Noise phase transition (Theorem 2.4). DPO’s final margin (y-axis) plotted against label-noise fraction ε\varepsilon (x-axis) collapses from unbounded growth at ε=0\varepsilon=0 to a finite fixed point Δ∗=βeff−1log⁡((1−ε)/ε)\Delta^*=\beta_{\mathrm{eff}}^{-1}\log((1-\varepsilon)/\varepsilon) for ε>0\varepsilon>0 (dashed: theory, solid: empirical). IPO’s fixed point shifts with ε\varepsilon but the convergence rate is preserved.

The third axis tests repulsion strength (Theorem 2.2): any γ>0\gamma>0 should maintain Ω(π)\Omega(\pi) repulsion while γ=0\gamma=0 reverts to O(π2)O(\pi^2). Figure 5 confirms: γ=0\gamma=0 plateaus at 66% while γ=0.25\gamma=0.25 reaches near-ceiling held-out accuracy (∼99%\sim99\%). Higher γ\gamma triggers gradient concentration (Theorem 2.5), producing a non-monotonic pattern; measured on the 50 tracked hard examples of Appendix § 10.5 (mean over seeds), γ=0.5\gamma=0.5 achieves 97% while γ=0.75\gamma=0.75 degrades to 87% (and γ=0.25\gamma=0.25 recovers to 91% on this harder tracked set). The pattern is absent on GSM8K (100% for all γ>0\gamma>0), confirming collapse requires hard examples. Standard DPO (γ=1.0\gamma=1.0) shows the widest variance (82–98% over 3 seeds). Cross-model validation on Mistral-7B confirms monotonic degradation with γ\gamma (98% →\rightarrow 88% →\rightarrow 83% for γ=0.25,0.75,1.0\gamma=0.25,0.75,1.0).

Asymmetric repulsion and DriftDPO ablation

Figure 5. Repulsion strength (Theorem 2.2). γ=0\gamma=0 plateaus at 66%; γ=0.5\gamma=0.5 achieves 97%; γ=0.7\gamma=0.7–0.80.8 shows a collapse valley (Theorem 2.5).

All three predictions replicate on GSM8K and MBPP (Appendix § 10.5–§ 10.§ 1), confirming that the dynamics are governed by w(Δ)w(\Delta), not by architecture or domain specifics.

Theory-Predicted Configurations across Model Scales

Having confirmed the framework’s descriptive predictions, we test its constructive power: does the theory-predicted configuration match or exceed existing methods? We evaluate on DeepSeek-R1-0528-Qwen3-8B (8B, 9K on-policy pairs), Qwen3.5-9B (9B, 5.4K pairs), and Ministral-3-8B (8B, 2.8K pairs). All methods use LoRA (r=64r=64); TPO-S uses β=0.1\beta=0.1, μ=0.5\mu=0.5, ε=0.05\varepsilon=0.05 (predicted by Theorem 2.4); TPO-H uses β=2.0\beta=2.0, μ=1.0\mu=1.0. No hyperparameter tuning was performed on ε\varepsilon; the theory predicts that any ε>0\varepsilon>0 creates a finite attractor. We report IFEval (objective instruction-following) and UltraFeedback preference accuracy.

The theory-derived TPO-S attains the highest mean IFEval score among the compared methods across all three models (Table 3), with the effect significant on MI-8B (+2.6 over SimPO, p=0.013p=0.013)1 and within noise on DS-8B (p=0.36p=0.36); we read this as confirming the predicted attractor is a competitive operating point, not as a state-of-the-art claim. Method ranking aligns with the taxonomy: reference-free methods with finite attractors (TPO-S, TPO-H, SimPO) occupy the top three positions on every model, reference-dependent methods form a middle tier, and ORPO [14] falls far behind. GRPO and PPO are online RL methods analyzed theoretically (Proposition A.3); we exclude them from Table 3 to keep the comparison within the offline-pairwise setting.

Ablation: reference vs. attractor. Table 3 implicitly contains a 2×22 \times2 ablation over reference model (present/absent) and finite attractor (ε>0\varepsilon> 0 / ε=0\varepsilon= 0), with DPO, cDPO, SimPO, and TPO-S filling the four cells. Removing the reference contributes +1.2 points on average (SimPO − DPO: +1.4, +0.3, +2.0), while adding the attractor on top of a reference-free loss contributes +1.1 (TPO-S − SimPO: +0.3, +0.5, +2.6). Adding the attractor to a reference-based loss hurts: cDPO − DPO averages −1.0, negative on every model—when the frozen reference already bounds margins, additional clipping reduces useful gradient signal. The two axes are complementary: TPO-S combines both, and neither alone matches its gains on MI-8B (+4.6 over DPO).

Table 4 shows two robust patterns and one suggestive one. Robustly: TPO-S leads or ties on most settings, and reference-free methods consistently outscore reference-dependent ones (gap exceeding 24 points on DS-8B on-policy, Proposition A.4). Suggestively, and within the seed variance of this table, TPO-S retains its edge off-policy on DS-8B and QW-9B, consistent with the finite attractor providing regularization under distribution mismatch; we do not over-read the per-setting margins here given the seed-to-seed spread. Method rankings are preserved under the standard SFT →\rightarrow preference pipeline (Appendix § 9).

On-policy (Math)Off-policy (UltraFeedback)
DS-8BQW-9BMI-8BDS-8BQW-9BMI-8B
TPO-S73.6±.573.6 \pm.572.0±1.972.0 \pm1.959.4±1.159.4 \pm1.155.2±1.955.2 \pm1.959.4±1.959.4 \pm1.958.2±2.258.2 \pm2.2
SimPO72.4±1.972.4 \pm1.969.4±2.469.4 \pm2.458.6±6.258.6 \pm6.251.2±.851.2 \pm.854.6±2.454.6 \pm2.460.2±.860.2 \pm.8
TPO-H71.4±2.671.4 \pm2.668.0±1.968.0 \pm1.959.8±1.9\mathbf{59.8 \pm1.9}47.8±.847.8 \pm.853.4±4.253.4 \pm4.251.2±1.151.2 \pm1.1
KTO50.2±1.350.2 \pm1.364.2±.864.2 \pm.859.6±1.159.6 \pm1.146.8±1.146.8 \pm1.152.0±2.052.0 \pm2.052.8±1.552.8 \pm1.5
DPO47.0±2.047.0 \pm2.061.0±2.361.0 \pm2.355.6±8.855.6 \pm8.845.4±.545.4 \pm.547.0±4.847.0 \pm4.855.2±2.855.2 \pm2.8
cDPO46.6±1.546.6 \pm1.559.0±.759.0 \pm.757.2±1.557.2 \pm1.545.2±.445.2 \pm.445.6±.945.6 \pm.946.0±1.246.0 \pm1.2
IPO43.8±.443.8 \pm.455.4±1.555.4 \pm1.550.8±.450.8 \pm.443.4±.543.4 \pm.542.8±1.342.8 \pm1.345.2±.845.2 \pm.8
ORPO42.5±.042.5 \pm.053.8±3.653.8 \pm3.646.5±.046.5 \pm.042.4±.842.4 \pm.841.8±.441.8 \pm.443.0±.043.0 \pm.0

Table 4. Preference accuracy (%, mean ±\pm std, 5 seeds) on on-policy math data and off-policy UltraFeedback [5] (60K pairs). TPO-S leads or ties on most settings. The reference-free vs. reference-dependent gap exceeds 24 points on DS-R1-8B.

Additional experiments (method rankings, diversity/coverage, nn-completion validation, ablations) are in Appendix § 8. On mathematical reasoning, TPO reaches 98.7% on MATH vs. 93.8% for DPO (Appendix § 8.§ 13.1), with the gain concentrated in the transition regime (Theorem 2.5); on GSM8K both reach 100%; on AIME both score ∼\sim12% (pt≈0p_t \approx0, Theorem A.9).

Label-smoothing sensitivity. Theorem 2.4 predicts that any ε>0\varepsilon> 0 creates a finite attractor at Δ∗=β−1log⁡((1−ε)/ε)\Delta^{*} = \beta^{-1}\log((1-\varepsilon)/\varepsilon), with higher ε\varepsilon yielding a lower equilibrium. A sweep on DS-R1-8B (UltraFeedback, 60K pairs) confirms: final margins decrease monotonically—1605 → 1264 → 832 → 455 → 341 for ε∈{0.01,0.03,0.05,0.1,0.2}\varepsilon\in\{0.01, 0.03, 0.05, 0.1, 0.2\}—while SimPO (ε=0\varepsilon=0) diverges. Observed margins are 25–35× the closed-form Δ∗\Delta^{*} (e.g., 832 vs. 29 at ε=0.05\varepsilon=0.05), reflecting βeff\beta_{\mathrm{eff}} renormalization (Remark 2.6). Downstream accuracy is flat across ε\varepsilon (49–52%), confirming that any ε>0\varepsilon> 0 suffices (Appendix E.3).

Cross-Architecture Universality. The theory operates on the probability simplex, shared by all autoregressive models with softmax outputs, so its predictions should be architecture-agnostic. We replicate Prediction 1 on six additional models spanning five families. In every case, DPO’s margin grows to 204–629 while IPO’s remains bounded near 1/(2τ)=51/(2\tau)=5 (Figure 6). Growth rates vary 3× across families, reflecting differences in βeff\beta_{\mathrm{eff}}, but the functional form is invariant: logarithmic for DPO, exponential convergence for IPO, despite different tokenizers, vocabularies, and parameter counts.

Figure 6. Margin dynamics replicated across five model families (Qwen, Gemma, OLMo, Mistral, DeepSeek). DPO’s margin grows without bound and IPO’s converges near the predicted fixed point 1/(2τ)=51/(2\tau)=5 (dashed line) on every architecture. Only the growth rate varies across families.

Alignment methods and unified frameworks. We analyze RLHF/PPO [21, 25], DPO [22], IPO [9], SimPO [19], KTO [7], GRPO [26], and others as instances of a single dynamical mechanism. Deng et al. [ [6] introduce the F+/F−F^{+}/F^{-} decomposition for image generation on continuous velocity fields; we borrow this decomposition but work on a discrete simplex with 1D margin ODEs. Several frameworks characterize what each method optimizes—f-PO [11] via f-divergences, RPO [Sun et al., 2025] via a common objective, DCR [24] via gradient flow—while we characterize how the optimization unfolds. Llama 2’s preference margin loss [Touvron et al., 2023] shifts the DPO sigmoid by a per-example offset derived from Likert annotations; in our taxonomy this leaves w(Δ)w(\Delta) strictly positive (no attractor). The noise phase transition of Theorem 2.4 exemplifies the difference: f-PO can show that DPO and IPO minimize different divergences, but cannot predict that DPO’s margin grows logarithmically while IPO’s converges exponentially, or that noise creates a phase transition in one but not the other.

Failure modes and iterative training. Concurrent work identifies individual DPO pathologies—gradient imbalance [17] (Yan et al. 2025), likelihood displacement [23], noise fragility [3], GRPO zero-gradient collapse [Zhang et al., 2025]—which our framework unifies as instances of a single cross-method pattern governed by the three axes. DAPO [Yu et al., 2025] removes the reference model entirely (reference-free, bounded convergence in our taxonomy). Iterative methods [10] (Yuan et al. 2024; Zelikman et al. 2022) refresh the training distribution; our coverage theorem (Theorem A.9) formalizes when they succeed (pt>0p_t > 0) and stall (pt→0p_t \to0).

Conclusion

We showed that major pairwise LLM alignment methods are instances of a single dynamical mechanism: a vector field on the probability simplex whose margin ODE admits closed-form solutions. The three main predictions (margin trajectories, noise-induced phase transitions, and repulsion strength) are confirmed across five model families. We validated the framework’s constructive prediction that any ε>0\varepsilon> 0 label smoothing creates a finite attractor for SimPO across three models; the theory-derived TPO-S (ε=0.05\varepsilon=0.05) is a competitive operating point, attaining the highest mean IFEval among the compared methods on all three, which confirms the predicted attractor is usable rather than establishing a new state-of-the-art loss.

Limitations. The two-completion simplex is the central modeling choice: qualitative predictions hold at n∈{2,4,8,16}n \in\{2, 4, 8, 16\} (Table 10), but quantitative accuracy requires βeff\beta_{\mathrm{eff}}, fitted per experiment (2.3× variation across scales). The margin is a sufficient state variable only in the leakage-free / independent-gradient regime: when the chosen and rejected gradients are strongly aligned (⟨gw,gl⟩>∥gw∥2\langle g_w, g_l\rangle> \lVert g_w\rVert^2, with g=∇θlog⁡πg = \nabla_\theta\log\pi), mass escaping to unsampled completions can decouple the margin from π(yw)\pi(y_w) ((likelihood displacement, Razin et al., 2024)), so the margin captures relative preference dynamics and predicting downstream quality from it alone is out of scope—consistent with the weak margin–accuracy correlation (ρ≈0.13\rho\approx0.13) on AIME. TPO-S’s gain over SimPO is modest (1.2 pts on DS 8B); on MI 8B KTO leads on chat. The γ\gamma-sweep collapse valley shows high seed variance (82–98%). Experiments use LoRA (r=64r=64) and Adam; Adam acts as a time reparameterization absorbed into βeff\beta_{\mathrm{eff}} (Appendix § 6.§ 13.2), and the residual R2=0.85R^2=0.85 (Figure 3) reflects discrete-time and cross-example effects.

Proofs

Remark A.1 (Extension beyond two completions). The two-completion analysis is exact at n=2n = 2 (no leakage by construction). As n→∞n \to\infty with all completions labeled, repelling every incorrect completion concentrates mass on correct ones, and the analysis again becomes tight, though this limit is unrealistic in practice. The approximation error is largest at intermediate nn, where repelling a subset of incorrect completions redistributes mass to unsampled ones, the displacement phenomenon of Asadi et al. [2026]. This makes the two-completion simplex a conservative approximation: it predicts the advantage of targeted repulsion in the leakage-free regime, while the actual advantage with n>2n > 2 is strictly larger. We validate empirically that the qualitative predictions (unbounded DPO margins, bounded IPO margins) hold at n∈{2,4,8,16}n \in\{2, 4, 8, 16\} (Appendix § 8.§ 13.3), though quantitative accuracy at n>2n > 2 with the same βeff\beta_{\mathrm{eff}} remains unvalidated. Diversity-aware selection (Proposition A.6) mitigates leakage by covering the incorrect-completion space, consistent with the +29% GRPO gain at K=4K = 4.

Notation and conventions. All results are stated on a two-completion simplex {yw,yl}\{y_w, y_l\} under softmax parameterization π(yi)=eθi/(eθw+eθl)\pi(y_i) = e^{\theta_i}/(e^{\theta_w} + e^{\theta_l}). We assume gradient flow (continuous time) with full-batch gradients; SGD noise from mini-batching is not analyzed but does not affect the qualitative predictions.

Remark A.2 (Connection to generative modeling). The vector field FF on the probability simplex is the same mathematical object studied in continuous normalizing flows [16] and flow matching [8]. In those settings, a learned velocity field transports a source distribution to a target along a probability path; here, the loss function induces the velocity field on the margin coordinate Δ=log⁡(π(yw)/π(yl))\Delta= \log(\pi(y_w)/\pi(y_l)), yielding the 1D autonomous ODEs we solve in closed form. The noise fixed point of Theorem 2.4—where the forward and reverse “flows” balance—is the discrete analog of score equilibrium in diffusion models [13] (Song et al. 2021): label-smoothed TPO (§ 2.1) exploits this by engineering a stable equilibrium at a prescribed margin, converting the noise phase transition from a failure mode into a design principle.

For IPO, we analyze the reference-free variant L=(τΔθ−1/2)2\mathcal{L} = (\tau\Delta_\theta- 1/2)^2 where Δθ=log⁡(π(yw)/π(yl))\Delta_\theta= \log(\pi(y_w)/\pi(y_l)); with a frozen reference, the dynamics are identical up to a constant shift Δref\Delta_{\mathrm{ref}} in the fixed point. For DPO, the full margin is Δ=Δ~θ−Δref\Delta= \widetilde{\Delta}_\theta- \Delta_{\mathrm{ref}} where Δref=log⁡(πref(yw)/πref(yl))\Delta_{\mathrm{ref}} = \log(\pi_{\mathrm{ref}}(y_w)/\pi_{\mathrm{ref}}(y_l)) is constant under a frozen reference; the ODE dΔ~/dt=βσ(−βΔ~)d\widetilde{\Delta}/dt = \beta\sigma(-\beta\widetilde{\Delta}) is autonomous in Δ~\widetilde{\Delta}, so the reference shifts the initial condition but does not change the functional form of the solution. The key identity is the softmax Jacobian:

π˙(yl)=π(yl)(1−π(yl))(θ˙l−θ˙w).(4)\dot{\pi}(y_l) = \pi(y_l)(1-\pi(y_l))\left(\dot{\theta}_l-\dot{\theta}_w\right). \tag*{(4)}

For any loss L\mathcal{L}, gradient descent gives θ˙y=−η∂L∂θy\dot{\theta}_y = -\eta\frac{\partial\mathcal{L}}{\partial\theta_y}. Since DPO and IPO losses depend on θw−θl\theta_w-\theta_l, both logits receive equal and opposite gradients, yielding θ˙w−θ˙l=2η⋅w(Δ)\dot{\theta}_w-\dot{\theta}_l = 2\eta\cdot w(\Delta) where w(Δ)w(\Delta) is the method-specific weighting. We absorb 2η2\eta into the time scale, writing dΔdt=w(Δ)\frac{d\Delta}{dt} = w(\Delta).

Axis 1: Convergence Rate Separation (Theorem 2.2)

Proof sketch. Compute the softmax gradient for each loss, substitute into the Jacobian (4), and compare the rate at which ∣π˙(yl)∣|\dot{\pi}(y_l)| vanishes as π(yl)→0\pi(y_l) \to0.

Proof. Part (i): SFT. The cross-entropy loss L=−log⁡π(yw)\mathcal{L} = -\log\pi(y_w) gives gradients θ˙y=η(p∗(y)−π(y))\dot{\theta}_y = \eta(p^*(y)-\pi(y)). With p∗(yw)=1p^*(y_w)=1, p∗(yl)=0p^*(y_l)=0:

θ˙w=ηπ(yl),θ˙l=−ηπ(yl).(5)\dot{\theta}_w = \eta\pi(y_l), \qquad\dot{\theta}_l = -\eta\pi(y_l). \tag*{(5)}

Thus θ˙l−θ˙w=−2ηπ(yl)\dot{\theta}_l-\dot{\theta}_w = -2\eta\pi(y_l). By (4):

π˙(yl)=−2ηπ(yl)2(1−π(yl))=O(π(yl)2) as π(yl)→0.(6)\dot{\pi}(y_l) = -2\eta\pi(y_l)^2(1-\pi(y_l)) = O(\pi(y_l)^2)\ \text{as}\ \pi(y_l)\to0. \tag*{(6)}

Part (ii): DPO. The loss L=−log⁡σ(βΔ~)\mathcal{L} = -\log\sigma(\beta\widetilde{\Delta}) with Δ~=(θw−θl)−log⁡(πref(yw)/πref(yl))\widetilde{\Delta} = (\theta_w-\theta_l)-\log(\pi_{\mathrm{ref}}(y_w)/\pi_{\mathrm{ref}}(y_l)) gives:

∂L∂θw=−βσ(−βΔ~),∂L∂θl=βσ(−βΔ~).(7)\frac{\partial\mathcal{L}}{\partial\theta_w} = -\beta\sigma(-\beta\widetilde{\Delta}), \qquad\frac{\partial\mathcal{L}}{\partial\theta_l} = \beta\sigma(-\beta\widetilde{\Delta}). \tag*{(7)}

So θ˙l−θ˙w=−2ηβσ(−βΔ)\dot{\theta}_l-\dot{\theta}_w = -2\eta\beta\sigma(-\beta\Delta), and by (4):

π˙(yl)=−2ηβσ(−βΔ)π(yl)(1−π(yl)).(8)\dot{\pi}(y_l) = -2\eta\beta\sigma(-\beta\Delta)\pi(y_l)(1-\pi(y_l)). \tag*{(8)}

When π(yl)≥δ>0\pi(y_l)\ge\delta>0, the margin Δ\Delta is bounded above, so σ(−βΔ)≥c(β,πref,δ)>0\sigma(-\beta\Delta)\ge c(\beta,\pi_{\mathrm{ref}},\delta)>0 where cc depends on β\beta, πref\pi_{\mathrm{ref}}, and δ\delta through the bounded margin. Therefore ∣π˙(yl)∣=Ω(π(yl))|\dot{\pi}(y_l)|=\Omega(\pi(y_l)): sustained. □\square

Axis 2: Margin Dynamics (Theorems 2.3–2.4)

Theorem 2.3 (without noise). Proof sketch. Solve the 1D ODE dΔ/dt=w(Δ)d\Delta/dt = w(\Delta) for each weighting function. DPO’s sigmoid yields a nonlinear ODE solvable via substitution; IPO’s linear weighting gives a standard linear ODE.

Proof. (i) DPO. The ODE dΔdt=β1+eβΔ\frac{d\Delta}{dt} = \frac{\beta}{1+e^{\beta\Delta}} becomes, under u=eβΔu=e^{\beta\Delta}:

dudt=β2u1+u.(9)\frac{du}{dt}=\frac{\beta^2u}{1+u}. \tag*{(9)}

Separating variables and integrating: log⁡u+u=β2t+C\log u+u=\beta^2t+C where C=βΔ(0)+eβΔ(0)C=\beta\Delta(0)+e^{\beta\Delta(0)}. For large tt, uu dominates log⁡u\log u, so u(t)∼β2tu(t)\sim\beta^2t and Δ(t)∼1βlog⁡(β2t)\Delta(t)\sim\frac{1}{\beta}\log(\beta^2t): unbounded but logarithmic. (This asymptotic is valid for β2t≫1\beta^2t\gg1; at early times the implicit solution should be used directly.)

(ii) IPO. The ODE dΔdt=τ−2τ2Δ\frac{d\Delta}{dt}=\tau-2\tau^2\Delta is linear with fixed point Δ∗=1/(2τ)\Delta^*=1/(2\tau). Writing δ=Δ−Δ∗\delta=\Delta-\Delta^*:

dδdt=−2τ2δ⟹Δ(t)=12τ+(Δ(0)−12τ)e−2τ2t.(10)\frac{d\delta}{dt}=-2\tau^2\delta\quad\Longrightarrow\quad\Delta(t)=\frac{1}{2\tau}+\left(\Delta(0)-\frac{1}{2\tau}\right)e^{-2\tau^2t}. \tag*{(10)}

Exponential convergence to 1/(2τ)1/(2\tau) with rate 2τ22\tau^2.

Theorem 2.4 (with label noise). Proof sketch. Average the ODE over correct and flipped pairs, treating the noise as i.i.d. across examples. This is a mean-field approximation: we replace the per-example noise realization with its expectation, which becomes exact in the large-sample limit (law of large numbers applied to the batch gradient). For finite batches, the approximation introduces fluctuations of order O(1/B)O(1/\sqrt{B}) where BB is the batch size. For DPO, the opposing sigmoid terms create a stable fixed point. For IPO, the linear structure preserves the convergence rate.

Proof. (i) DPO. The averaged ODE is:

dΔdt=(1−ε)βσ(−βΔ)−εβσ(βΔ).(11)\frac{d\Delta}{dt}=(1-\varepsilon)\beta\sigma(-\beta\Delta)-\varepsilon\beta\sigma(\beta\Delta). \tag*{(11)}

Setting to zero and using σ(−x)=e−xσ(x)\sigma(-x)=e^{-x}\sigma(x): (1−ε)e−βΔ∗=ε(1-\varepsilon)e^{-\beta\Delta^*}=\varepsilon, giving Δ∗=1βlog⁡1−εε\Delta^*=\frac{1}{\beta}\log\frac{1-\varepsilon}{\varepsilon}. Stability: differentiating the full averaged field, f′(Δ)=−(1−ε)β2σ(βΔ)σ(−βΔ)−εβ2σ(βΔ)σ(−βΔ)=−β2σ(βΔ)σ(−βΔ)<0f'(\Delta)=-(1-\varepsilon)\beta^2\sigma(\beta\Delta)\sigma(-\beta\Delta)-\varepsilon\beta^2\sigma(\beta\Delta)\sigma(-\beta\Delta)=-\beta^2\sigma(\beta\Delta)\sigma(-\beta\Delta)<0 everywhere.

(ii) IPO. A flipped pair has effective margin −Δ-\Delta, giving loss (τΔ+1/2)2(\tau\Delta+1/2)^2. Averaging:

dΔdt=−2τ2Δ+τ(1−2ε).(12)\frac{d\Delta}{dt}=-2\tau^2\Delta+\tau(1-2\varepsilon). \tag*{(12)}

Fixed point Δ∗=(1−2ε)/(2τ)\Delta^*=(1-2\varepsilon)/(2\tau), rate 2τ22\tau^2 unchanged. Noise shifts the target but not the speed.

Proposition A.3. GRPO gradient vanishes with sparse rewards. For GRPO with binary rewards, group size KK, success probability pp: E[∥∇L∥2]∝p(1−p)⋅G2/K\mathbb{E}[\|\nabla L\|^2]\propto p(1-p)\cdot G^2/K, maximized at p=1/2p=1/2 and vanishing as p→0p\to0 or p→1p\to1. The zero-gradient probability is (1−p)K+pK(1-p)^K+p^K.

Proof. The advantage Ai=Ri−RˉA_i=R_i-\bar{R} gives ∑Ai2=k(K−k)/K\sum A_i^2=k(K-k)/K. Since E[k(K−k)]=Kp(1−p)(K−1)\mathbb{E}[k(K-k)]=Kp(1-p)(K-1):

E[1K2∑Ai2]=(K−1)p(1−p)K2∼p(1−p)K.(13)\mathbb{E}\left[\frac{1}{K^2}\sum A_i^2\right]=\frac{(K-1)p(1-p)}{K^2}\sim\frac{p(1-p)}{K}. \tag*{(13)}

Maximized at p=1/2p=1/2, vanishing as p→0p\to0 or 11. Zero-gradient probability: (1−p)K+pK(1-p)^K+p^K.

Axis 3: Reference Strategy (Proposition A.4, Proposition A.5)

Proposition A.4. Gradient decay under staleness. For any method with frozen πref\pi_{\mathrm{ref}} whose gradient includes σ(−βΔ)\sigma(-\beta\Delta), the per-example gradient norm is ∥∇L∥=β⋅σ(−βΔ)⋅∥∇log⁡π(yw)−∇log⁡π(yl)∥\|\nabla L\|=\beta\cdot\sigma(-\beta\Delta)\cdot\|\nabla\log\pi(y_w)-\nabla\log\pi(y_l)\|. As training progresses, Δ\Delta grows on correctly-learned examples, driving σ(−βΔ)→0\sigma(-\beta\Delta)\to0. Reference-free methods (SimPO, ORPO) avoid this decay but still experience sigmoid-based gradient decay from growing margins; the staleness proposition isolates the additional decay caused by the frozen reference.

Proof. The DPO gradient is ∇θL=−βσ(−βΔ)[∇θlog⁡πθ(yw∣x)−∇θlog⁡πθ(yl∣x)]\nabla_\theta\mathcal{L}=-\beta\sigma(-\beta\Delta)[\nabla_\theta\log\pi_\theta(y_w|x)-\nabla_\theta\log\pi_\theta(y_l|x)]. Taking norms:

∥∇θL∥=β⋅σ(−βΔ)⋅∥∇θlog⁡πθ(yw∣x)−∇θlog⁡πθ(yl∣x)∥.(14)\lVert\nabla_\theta\mathcal{L}\rVert=\beta\cdot\sigma(-\beta\Delta)\cdot\lVert\nabla_\theta\log\pi_\theta(y_w|x)-\nabla_\theta\log\pi_\theta(y_l|x)\rVert. \tag*{(14)}

Since Δ=Δθ−Δref\Delta=\Delta_\theta-\Delta_{\mathrm{ref}} and Δθ\Delta_\theta grows on correctly-learned examples (as established by Theorem 2.3 for DPO), σ(−βΔ)→0\sigma(-\beta\Delta)\to0. The log-probability gradient term is bounded for finite-parameter models, so the sigmoid decay dominates. For SimPO, Δ=Δθ−μ\Delta=\Delta_\theta-\mu with fixed μ\mu, so no reference-staleness decay occurs. However, SimPO still experiences sigmoid-based gradient decay from growing margins (σ(−βΔθ)→0\sigma(-\beta\Delta_\theta)\to0 as Δθ\Delta_\theta grows). The distinction is that DPO has both margin-based decay and reference-staleness decay, while SimPO has only the former.

Proposition A.5 (Optimal adaptive reference). The reference minimizing λ1KL⁡(πref∥πθ)+λ2KL⁡(πref∥π0)\lambda_1\operatorname{KL}(\pi_{\mathrm{ref}}\Vert\pi_\theta)+\lambda_2\operatorname{KL}(\pi_{\mathrm{ref}}\Vert\pi_0) is πref∗(y)∝π0(y)1−απθ(y)α\pi_{\mathrm{ref}}^*(y)\propto\pi_0(y)^{1-\alpha}\pi_\theta(y)^\alpha with α=λ1/(λ1+λ2)\alpha=\lambda_1/(\lambda_1+\lambda_2). At α=0\alpha=0: frozen (DPO). At α=1\alpha=1: self-referencing (SimPO-like).

Proof. The objective J(πref)=λ1KL⁡(πref∥πθ)+λ2KL⁡(πref∥π0)J(\pi_{\mathrm{ref}})=\lambda_1\operatorname{KL}(\pi_{\mathrm{ref}}\Vert\pi_\theta)+\lambda_2\operatorname{KL}(\pi_{\mathrm{ref}}\Vert\pi_0) is strictly convex. Setting the Lagrangian derivative to zero:

(λ1+λ2)log⁡πref(y)=λ1log⁡πθ(y)+λ2log⁡π0(y)+const.(15)(\lambda_1+\lambda_2)\log\pi_{\mathrm{ref}}(y)=\lambda_1\log\pi_\theta(y)+\lambda_2\log\pi_0(y)+\mathrm{const}. \tag*{(15)}

Exponentiating: πref∗(y)∝πθ(y)α⋅π0(y)1−α\pi_{\mathrm{ref}}^*(y)\propto\pi_\theta(y)^\alpha\cdot\pi_0(y)^{1-\alpha} with α=λ1/(λ1+λ2)\alpha=\lambda_1/(\lambda_1+\lambda_2).

Note: We minimize KL⁡(πref∥πθ)\operatorname{KL}(\pi_{\mathrm{ref}}\Vert\pi_\theta) (forward KL) rather than KL⁡(πθ∥πref)\operatorname{KL}(\pi_\theta\Vert\pi_{\mathrm{ref}}) (staleness) because the forward direction yields a closed form. Both vanish at πref=πθ\pi_{\mathrm{ref}}=\pi_\theta.

Diversity and PPO (Propositions A.6–A.7)

Proposition A.6 (Diversity improves field estimation). For a weighted estimate V^=∑wivi\widehat{V}=\sum w_i v_i with independent viv_i, E[vi]=V\mathbb{E}[v_i]=V, Var⁡(vi)=σ2I\operatorname{Var}(v_i)=\sigma^2 I, and ∑wi=1\sum w_i=1: MSE⁡(V^)=σ2/deff\operatorname{MSE}(\widehat{V})=\sigma^2/d_{\mathrm{eff}} where deff=(∑wi2)−1≤Nd_{\mathrm{eff}}=(\sum w_i^2)^{-1}\le N, with equality iff weights are uniform.

Proof. For V^=∑wivi\widehat{V}=\sum w_i v_i with ∑wi=1\sum w_i=1 and independent viv_i with E[vi]=V\mathbb{E}[v_i]=V, Var⁡(vi)=σ2I\operatorname{Var}(v_i)=\sigma^2I:

MSE⁡(V^)=Var⁡(V^)=σ2∑wi2=σ2/deff.(16)\operatorname{MSE}(\widehat{V})=\operatorname{Var}(\widehat{V})=\sigma^2\sum w_i^2=\sigma^2/d_{\mathrm{eff}}. \tag*{(16)}

By Cauchy–Schwarz, 1=(∑wi)2≤N∑wi21=(\sum w_i)^2\le N\sum w_i^2, so deff≤Nd_{\mathrm{eff}}\le N with equality iff wi=1/Nw_i=1/N.

Proposition A.7 (PPO-KL margin dynamics). Under PPO with KL penalty β\beta (not the clipped PPO variant, whose non-smooth objective does not admit a closed-form ODE): dΔdt=η[(R(yw)−R(yl))−β(Δ−Δref)]\frac{d\Delta}{dt}=\eta[(R(y_w)-R(y_l))-\beta(\Delta-\Delta_{\mathrm{ref}})], a linear ODE with fixed point Δ∗=(R(yw)−R(yl))/β+Δref\Delta^*=(R(y_w)-R(y_l))/\beta+\Delta_{\mathrm{ref}} and rate ηβ\eta\beta.

Proof. The KL-regularized gradient gives θ˙y=η[R(y)−βlog⁡(π(y)/πref(y))+const]\dot{\theta}_y=\eta[R(y)-\beta\log(\pi(y)/\pi_{\mathrm{ref}}(y))+\mathrm{const}]. For two completions:

dΔdt=η[(R(yw)−R(yl))−β(Δ−Δref)],(17)\frac{d\Delta}{dt}=\eta[(R(y_w)-R(y_l))-\beta(\Delta-\Delta_{\mathrm{ref}})], \tag*{(17)}

a linear ODE with fixed point Δ∗=(R(yw)−R(yl))/β+Δref\Delta^*=(R(y_w)-R(y_l))/\beta+\Delta_{\mathrm{ref}} and rate ηβ\eta\beta.

Coverage Expansion (Theorem A.9)

The following result is a straightforward application of coupon-collector-style reasoning. Its contribution is not the proof technique but the formalization of when iterative alignment methods stall (pt→0p_t\to0) vs. succeed (pt>0p_t>0), connecting the coverage gap to the margin dynamics framework: methods with targeted repulsion maintain pt>0p_t>0 on solvable problems, while SFT-only methods see ptp_t decay.

Definition A.8 (Coverage). *For reward threshold τ\tau, the coverage of policy π\pi is Cov⁡(π,τ)=Px[∃y∼π(⋅∣x):R(x,y)≥τ]\operatorname{Cov}(\pi,\tau)=\mathbb{P}_x[\exists y\sim\pi(\cdot|x):R(x,y)\ge\tau].

Theorem A.9 (Iterative coverage expansion). Let Δ0=1−Cov⁡(π0,τ)\Delta_0 = 1 - \operatorname{Cov}(\pi_0,\tau) be the initial coverage gap. After TT iterations with KK samples per prompt and discovery probabilities {pt}\{p_t\}, assuming (a) covered prompts remain covered across iterations (no catastrophic forgetting) and (b) pt>0p_t > 0 for uncovered prompts:

E[Cov⁡(π(T),τ)]≥1−Δ0∏t=0T−1(1−pt)K.(18)\mathbb{E}[\operatorname{Cov}(\pi^{(T)},\tau)] \ge1 - \Delta_0 \prod_{t=0}^{T-1}(1-p_t)^K. \tag*{(18)}

When pt≥pmin⁡>0p_t \ge p_{\min} > 0, coverage gaps decay exponentially: ΔT≤Δ0(1−pmin⁡)KT\Delta_T \le\Delta_0(1-p_{\min})^{KT}.

Proof sketch. Induction on iterations. At each step, every uncovered prompt has at least KK independent chances to discover a solution, reducing the coverage gap by a factor of (1−pt)K(1-p_t)^K.

Proof. Define Ut={x:no completion y with R(x,y)≥τ has been found by iteration t}U_t = \{x : \text{no completion } y \text{ with } R(x,y) \ge\tau\text{ has been found by iteration } t\} as the set of uncovered prompts. Let Δt=P[x∈Ut]\Delta_t = \mathbb{P}[x \in U_t] denote the coverage gap.

Base case: Δ0=1−Cov⁡(π0,τ)\Delta_0 = 1 - \operatorname{Cov}(\pi_0,\tau) by definition.

Inductive step: At iteration tt, for each prompt x∈Utx \in U_t, we sample KK completions from π(t)(⋅∣x)\pi^{(t)}(\cdot|x). Let pt=inf⁡x∈UtPy∼π(t)[R(x,y)≥τ∣x]p_t = \inf_{x \in U_t}\mathbb{P}_{y \sim\pi^{(t)}}[R(x,y) \ge\tau\mid x] be the worst-case discovery probability over uncovered prompts. The probability that all KK samples fail to produce a τ\tau-good completion is at most (1−pt)K(1-p_t)^K (by independence of samples). Therefore: P[x∈Ut+1∣x∈Ut]≤(1−pt)K\mathbb{P}[x \in U_{t+1} \mid x \in U_t] \le(1-p_t)^K. Taking expectations over prompts: Δt+1=P[x∈Ut+1]≤P[x∈Ut]⋅(1−pt)K=Δt⋅(1−pt)K\Delta_{t+1} = \mathbb{P}[x \in U_{t+1}] \le\mathbb{P}[x \in U_t] \cdot(1-p_t)^K = \Delta_t \cdot(1-p_t)^K.

Telescoping: By induction, ΔT≤Δ0∏t=0T−1(1−pt)K\Delta_T \le\Delta_0 \prod_{t=0}^{T-1}(1-p_t)^K, so Cov⁡(π(T),τ)=1−ΔT≥1−Δ0∏t=0T−1(1−pt)K\operatorname{Cov}(\pi^{(T)},\tau) = 1 - \Delta_T \ge1 - \Delta_0 \prod_{t=0}^{T-1}(1-p_t)^K.

When pt≥pmin⁡>0p_t \ge p_{\min} > 0 for all tt, the product is bounded by (1−pmin⁡)KT(1-p_{\min})^{KT}, giving exponential decay of the coverage gap: ΔT≤Δ0(1−pmin⁡)KT\Delta_T \le\Delta_0(1-p_{\min})^{KT}.

Gradient Concentration (Theorem 2.5)

Proof. The asymmetric DPO loss for example ii is ℓi=−log⁡σ(βΔγ,i)\ell_i = -\log\sigma(\beta\Delta_{\gamma,i}) where Δγ,i=log⁡(πθ(yiw)/πref(yiw))−γlog⁡(πθ(yil)/πref(yil))\Delta_{\gamma,i} = \log(\pi_\theta(y_i^w)/\pi_{\mathrm{ref}}(y_i^w)) - \gamma\log(\pi_\theta(y_i^l)/\pi_{\mathrm{ref}}(y_i^l)). Since πref\pi_{\mathrm{ref}} is frozen, ∇θΔγ,i=∇θlog⁡π(yiw)−γ∇θlog⁡π(yil)=:wi\nabla_\theta\Delta_{\gamma,i} = \nabla_\theta\log\pi(y_i^w) - \gamma\nabla_\theta\log\pi(y_i^l) =: w_i. By the chain rule:

∇θL=−βN∑i=1Nσ(−βΔγ,i)⋅wi.(19)\nabla_\theta\mathcal{L} = -\frac{\beta}{N}\sum_{i=1}^{N}\sigma(-\beta\Delta_{\gamma,i})\cdot w_i. \tag*{(19)}

Under gradient flow θ˙=−∇θL\dot{\theta} = -\nabla_\theta\mathcal{L}, the margin Δj=log⁡π(yjw)−log⁡π(yjl)\Delta_j = \log\pi(y_j^w) - \log\pi(y_j^l) has gradient vj=∇θΔjv_j = \nabla_\theta\Delta_j, so:

Δ˙j=vj⊤θ˙=βN∑i=1Nσ(−βΔγ,i)⋅vj⊤wi.(20)\dot{\Delta}_j = v_j^\top\dot{\theta} = \frac{\beta}{N}\sum_{i=1}^{N}\sigma(-\beta\Delta_{\gamma,i})\cdot v_j^\top w_i. \tag*{(20)}

This is exact in continuous time; vjv_j, wiw_i, and Δγ,i\Delta_{\gamma,i} all depend on θ(t)\theta(t).

For Δγ,j≫1/β\Delta_{\gamma,j} \gg1/\beta: σ(−βΔγ,j)=(1+eβΔγ,j)−1≤e−βΔγ,j\sigma(-\beta\Delta_{\gamma,j}) = (1+e^{\beta\Delta_{\gamma,j}})^{-1} \le e^{-\beta\Delta_{\gamma,j}}, exponentially small. For hard examples with Δγ,i=O(1/β)\Delta_{\gamma,i} = O(1/\beta), σ(−βΔγ,i)=Θ(1)\sigma(-\beta\Delta_{\gamma,i}) = \Theta(1), so these dominate the sum.

For the γ\gamma-monotonicity: when πθ(yil)<πref(yil)\pi_\theta(y_i^l) < \pi_{\mathrm{ref}}(y_i^l) (i.e., the model has successfully decreased the dispreferred probability below the reference, which we term a “learned” example), log⁡(πθ(yil)/πref(yil))<0\log(\pi_\theta(y_i^l)/\pi_{\mathrm{ref}}(y_i^l)) < 0, so ∂Δγ,i/∂γ=−log⁡(πθ(yil)/πref(yil))>0\partial\Delta_{\gamma,i}/\partial\gamma= -\log(\pi_\theta(y_i^l)/\pi_{\mathrm{ref}}(y_i^l)) > 0. Higher γ\gamma increases Δγ,i\Delta_{\gamma,i}, decreasing σ(−βΔγ,i)\sigma(-\beta\Delta_{\gamma,i}) and concentrating the gradient on hard examples faster. Early in training, this condition may not hold for all examples.

Effective Coupling Approximation (Remark 2.6)

Proof. From Theorem 2.5, the margin of pair jj evolves as:

Δ˙j=βN∑i=1Nσ(−βΔi)⋅vj⊤wi.(21)\dot{\Delta}_j = \frac{\beta}{N}\sum_{i=1}^{N}\sigma(-\beta\Delta_i)\cdot v_j^\top w_i. \tag*{(21)}

Isolating the self-term (i=ji=j) and noting that for γ=1\gamma=1, wj=vjw_j=v_j (as assumed in Remark 2.6; for general γ\gamma, the self-term is vjTwj≠∥vj∥2v_j^{\mathsf T}w_j\ne\lVert v_j\rVert^2):

Δ˙j=βN[σ(−βΔj)∥vj∥2+∑i≠jσ(−βΔi)⋅vjTwi].(22)\dot{\Delta}_j=\frac{\beta}{N}\left[\sigma(-\beta\Delta_j)\lVert v_j\rVert^2+\sum_{i\ne j}\sigma(-\beta\Delta_i)\cdot v_j^{\mathsf T}w_i\right]. \tag*{(22)}

Dividing by σ(−βΔj)\sigma(-\beta\Delta_j) to extract the effective coupling:

βeff,j:=Δ˙jσ(−βΔj)=βN[∥vj∥2+∑i≠jσ(−βΔi)σ(−βΔj)⋅vjTwi].(23)\beta_{\mathrm{eff},j}:=\frac{\dot{\Delta}_j}{\sigma(-\beta\Delta_j)}=\frac{\beta}{N}\left[\lVert v_j\rVert^2+\sum_{i\ne j}\frac{\sigma(-\beta\Delta_i)}{\sigma(-\beta\Delta_j)}\cdot v_j^{\mathsf T}w_i\right]. \tag*{(23)}

Cross-term analysis. The inner products vjTwiv_j^{\mathsf T}w_i for i≠ji\ne j depend on the parameter-sharing structure. In the LoRA setting with rank rr and model dimension dd, the gradients vj,wi∈R2rdv_j,w_i\in\mathbb{R}^{2rd} (two LoRA matrices per layer). Independence assumption (stated in Remark 2.6): For distinct prompts with sufficient diversity, the gradient directions are approximately independent, giving E[vjTwi]≈0\mathbb{E}[v_j^{\mathsf T}w_i]\approx0 and Var⁡[vjTwi]=σ⊥2\operatorname{Var}[v_j^{\mathsf T}w_i]=\sigma_\perp^2. This assumption is strong: LoRA gradients share a low-rank subspace of dimension 2rd≪d22rd\ll d^2, and thematically correlated training examples (e.g., multiple math problems of the same type) will have positively correlated gradients. The O(1/N)O(1/\sqrt{N}) error scaling below depends entirely on this assumption; when it is violated, the cross-term contribution may not average out, and βeff\beta_{\mathrm{eff}} may deviate from the self-reinforcement approximation by more than the stated error. The sigmoid ratio σ(−βΔi)/σ(−βΔj)\sigma(-\beta\Delta_i)/\sigma(-\beta\Delta_j) is O(1)O(1) for active examples and exponentially small for well-learned ones. Defining Nactive={i:σ(−βΔi)>1/e}N_{\mathrm{active}}=\{i:\sigma(-\beta\Delta_i)>1/e\}, the cross-term sum has mean zero and standard deviation O(σ⊥Nactive)O(\sigma_\perp\sqrt{N_{\mathrm{active}}}) under the independence assumption, giving:

βeff,j=β∥vj∥2N±O(βσ⊥NactiveN).(24)\beta_{\mathrm{eff},j}=\frac{\beta\lVert v_j\rVert^2}{N}\pm O\left(\frac{\beta\sigma_\perp\sqrt{N_{\mathrm{active}}}}{N}\right). \tag*{(24)}

Scaling with dataset size. As training progresses, well-learned examples have Δi≫1/β\Delta_i\gg1/\beta, so σ(−βΔi)→0\sigma(-\beta\Delta_i)\to0 and they exit the active set. The number of active examples NactiveN_{\mathrm{active}} depends on the difficulty distribution: for a dataset with fraction ff of hard examples (near the decision boundary), Nactive≈fNN_{\mathrm{active}}\approx fN. As NN grows with fixed difficulty distribution, βeff≈β∥vj∥2/N\beta_{\mathrm{eff}}\approx\beta\lVert v_j\rVert^2/N while the noise term scales as βσ⊥fN/N=βσ⊥f/N\beta\sigma_\perp\sqrt{fN}/N=\beta\sigma_\perp\sqrt{f}/\sqrt{N}, which vanishes. This explains why βeff/β\beta_{\mathrm{eff}}/\beta approaches a stable value at large NN: the self-reinforcement term dominates and the cross-example noise averages out.

Empirical consistency. The observed βeff/β≈1/4.4\beta_{\mathrm{eff}}/\beta\approx1/4.4 at 500 samples and ≈1/1.9\approx1/1.9 at full scale is consistent with NactiveN_{\mathrm{active}} decreasing relative to NN as the dataset grows: with more data, a larger fraction of examples are well-separated (easy), leaving fewer active examples to dilute the self-reinforcement signal.

Adam as time reparameterization. Our experiments use AdamW, not gradient flow. Adam’s update is θt+1=θt−η m^t/(v^t+ϵ)\theta_{t+1}=\theta_t-\eta\,\hat{m}_t/(\sqrt{\hat{v}_t}+\epsilon), which in continuous-time form is θ˙=−Pt∇L\dot{\theta}=-P_t\nabla\mathcal{L} with Pt=diag⁡(1/(v^t+ϵ))P_t=\operatorname{diag}(1/(\sqrt{\hat{v}_t}+\epsilon)) a positive-definite preconditioner. Projecting onto the scalar margin, Δ˙=vTPt∇L=(vTPtv)⋅w(Δ)\dot{\Delta}=v^{\mathsf T}P_t\nabla\mathcal{L}=(v^{\mathsf T}P_tv)\cdot w(\Delta), where vTPtv>0v^{\mathsf T}P_tv>0 is a state-dependent positive scalar. Because PtP_t is positive definite, vTPtv>0v^{\mathsf T}P_tv>0 is a strictly positive (state-dependent) scalar, so this is a time reparameterization of the gradient-flow ODE: multiplying w(Δ)w(\Delta) by a positive factor c(Δ)=vTPtvc(\Delta)=v^{\mathsf T}P_tv leaves its zero set unchanged and preserves the sign of w′w' at each zero (since (cw)′(Δ∗)=c(Δ∗)w′(Δ∗)(cw)'(\Delta^*)=c(\Delta^*)w'(\Delta^*) when w(Δ∗)=0w(\Delta^*)=0). Hence, under a positive but possibly nonuniform c(Δ)c(\Delta), the following are invariant: the zeros of ww (fixed-point locations), the sign of w′w' at each zero (stability), the sign of Δ˙\dot{\Delta} (monotone direction), and—since c>0c>0 cannot change the sign of cwc w—the bounded-vs-unbounded dichotomy (DPO’s margin remains unbounded, IPO’s remains bounded, and the noise attractor exists iff ϵ>0\epsilon>0). What is not invariant is the rate in training-step time: a nonuniform c(Δ)c(\Delta) changes the log⁡t\log t prefactor for DPO and the exponential decay constant for IPO. Near a fixed point c(Δ)→c(Δ∗)c(\Delta)\to c(\Delta^*) is locally constant, so exponential convergence is recovered locally with a rescaled constant; globally we claim only the functional character (sub-linear/unbounded vs. bounded), not the step-time rate. The rate constant is precisely what βeff\beta_{\mathrm{eff}} absorbs. This invariance applies to the positive scalar preconditioner only. The multi-example coupling term ∑i≠jσ(−βΔi)vj⊤wi\sum_{i\ne j}\sigma(-\beta\Delta_i)v_j^\top w_i is not a positive rescaling—its cross-products are sign-indefinite—and it is this term, together with the scalar time rescaling, that βeff\beta_{\mathrm{eff}} absorbs quantitatively; the coupling can shift fixed-point locations (and, in strongly low-rank regimes, alter them), which is why βeff\beta_{\mathrm{eff}} must be fitted. The parameter-free claims we rely on (unbounded vs. bounded growth, the ε>0\varepsilon>0 existence of the attractor) rest on the scalar-invariance argument and the mean-field field, not on the coupling.

Remark A.10 (Collapse mechanism and post-collapse floor). When the gradient concentrates on ∣H∣|\mathcal{H}| hard examples, the margin of easy example jj decreases if ∑i∈Hσ(−βΔγ,i)⋅vj⊤wi<0\sum_{i\in\mathcal{H}}\sigma(-\beta\Delta_{\gamma,i})\cdot v_j^\top w_i<0 (gradient interference). Once Δj\Delta_j decreases to O(1/β)O(1/\beta), example jj rejoins the hard set, its sigmoid weight becomes Θ(1)\Theta(1), and the self-reinforcement term vj⊤wj>0v_j^\top w_j>0 provides a restoring force. This explains the empirically observed post-collapse floor at Δ≈150\Delta\approx150 (=O(1/β2)=O(1/\beta^2) in log-probability units with β=0.1\beta=0.1), independent of the pre-collapse peak. We note that this restoring-force argument explains the post-collapse floor but does not fully account for the non-monotonic accuracy pattern observed in the γ\gamma-sweep (Appendix § 10.5), where γ=1.0\gamma=1.0 sometimes recovers to near-optimal accuracy. This likely involves interactions beyond the pairwise analysis.

TPO Derivation Details

This section contains the full derivation details for TPO-H and TPO-S, summarized in § 2.1.

TPO-H: Huber variant. The simplest weighting function satisfying D1–D4 is the clamped linear w∗(Δ)=clamp⁡(Δ∗−Δ,−c,c)w^*(\Delta)=\operatorname{clamp}(\Delta^*-\Delta,-c,c): linear near the target (D1–D3), saturating far away (D4). This weighting is symmetric around Δ∗\Delta^*, so D5 (asymmetry) is not satisfied. Integrating to recover the loss yields the Huber function:

LTPO−H(Δ,δ)={(Δ∗−Δ)2/(2δ)if ∣Δ∗−Δ∣≤δ,∣Δ∗−Δ∣−δ/2otherwise,(25)\mathcal{L}_{\mathrm{TPO-H}}(\Delta,\delta)= \begin{cases} (\Delta^*-\Delta)^2/(2\delta) & \text{if }|\Delta^*-\Delta|\le\delta,\\ |\Delta^*-\Delta|-\delta/2 & \text{otherwise,} \end{cases} \tag*{(25)}

where δ\delta controls the transition between quadratic and linear regimes. As δ→∞\delta\to\infty, TPO-H recovers IPO (up to constants); as δ→0\delta\to0, a hinge-style loss.

TPO-S: Label-smoothed sigmoid variant. TPO-S preserves SimPO’s sigmoid curriculum while the label-smoothing term creates a restoring force once the margin overshoots Δ∗\Delta^*. It satisfies D1–D3, D4 (the sigmoid weighting is bounded by 1), and D5 (the weighting is asymmetric: ∣w(Δ)∣|w(\Delta)| is larger below Δ∗\Delta^* than above, since the (1−ε)σ(−Δ)(1-\varepsilon)\sigma(-\Delta) term dominates for Δ<Δ∗\Delta<\Delta^*).

TPO-H achieves the finite attractor through loss geometry (bounded, symmetric gradients); TPO-S through data-side regularization (label smoothing creates a fixed point where two label populations balance). TPO-H excels with abundant data where bounded gradients prevent collapse under noisy labels; TPO-S excels with limited data where the sigmoid curriculum enables efficient exploration.

Design summary. Each design choice addresses a failure axis:

AxisFailure modeTPO design choice
1 (Repulsion)Gradient concentration at high γ\gamma (Thm. 2.5)Targeted repulsion with γ>0\gamma>0
2 (Weighting)Unbounded or vanishing gradientsHuber (TPO-H) / smoothing (TPO-S)
3 (Reference)Gradient decay from stale ref (Prop. A.4)Reference-free scores

Table 25.

Connection to existing losses. The TPO family interpolates between known objectives. For TPO-H: as δ→∞\delta\to\infty, the quadratic regime dominates, recovering IPO; as δ→0\delta\to0, the linear regime dominates, recovering a hinge-style loss. For TPO-S: setting ε=0\varepsilon=0 recovers SimPO exactly, while increasing ε\varepsilon strengthens the attractor at the cost of label noise. Diversity-aware trajectory selection via greedy facility location (Proposition A.6) is a plug-in for any iterative method.

Extended Experiments

This section contains experiments moved from the main text for space.

Cross-Architecture Universality

(See Figure 6 in the main text for the cross-architecture margin dynamics plot.)

We replicate Prediction 1 on six additional models spanning five families. In every case, DPO’s margin grows to 204–629 while IPO’s remains bounded near 1/(2τ)=51/(2\tau)=5 (Figure 6). Growth rates vary 3×3\times across families, but the functional form is invariant.

Reference Staleness

Figure 7 compares reference divergence across methods, validating Proposition A.4.

Figure 7. Reference divergence over training (Proposition A.4). SimPO (reference-free) diverges 82×82\times more than DPO (frozen reference), confirming the staleness–drift tradeoff predicted by Axis 3: frozen references constrain policy drift but cause gradient decay; reference-free methods avoid decay but risk unbounded drift.

Cross-Model Margin Data

Table 5: Cross-architecture validation of margin dynamics (Theorem 2.3). DPO’s margin grows without bound; IPO’s converges near 1/(2τ)=51/(2\tau)=5. Pattern holds across all seven models and five families.

ModelFamilyDPO Δ3000\Delta_{3000}IPO Δ3000\Delta_{3000}
Qwen3-8B (primary)Qwen1540∼5\sim5
Qwen3.5-4BQwen5791.2
Qwen3-4BQwen3675.4
Gemma 3 4BGoogle3055.0
OLMo-2 7BAI26292.0
Mistral 7BMistral2040.5
DeepSeek-R1 7BDeepSeek5047.2

Table 5.

The table shows that DPO’s margin grows to 204–1540 (unbounded) while IPO’s remains in [0.5, 7.2] (bounded near 1/(2τ)=51/(2\tau)=5) across all seven models. The qualitative separation predicted by Theorem 2.3 is universal; only the growth rate varies, reflecting differences in model capacity and tokenization.

Method Rankings

Preference methods (DPO 78%) outperform supervised methods (SFT 72%, RFT 47%) at full scale (Table 6), confirming Axis 1. RFT’s anti-scaling (52% → 47%) reflects memorization without repulsion. GRPO achieves 70% at 500 samples (where p≈0.5p \approx0.5) but degrades to 48% at 7500: as training progresses, on-policy sampling concentrates on already-solved problems, so the effective pp for unsolved problems drops toward zero, increasing zero-gradient batches exactly as Proposition A.3 predicts. On AIME (low-coverage), all methods score 0%, confirming Theorem A.9.

MethodAxes (1 / 2 / 3)1005007500Theory predicts
SFTImplicit / — / —446172Self-limiting (Thm. 2.2)
RFTImplicit / — / —524947Overfits without repulsion
GRPOTargeted / Adv. / None497048Fails: sparse rewards (Prop. A.3)
DPOTargeted / Sigmoid / Frozen67–717478Strong but stale (Prop. A.4)

Table 6.

Table 6: Accuracy (%) on MATH across data scales. Preference methods with targeted repulsion outperform supervised methods; GRPO degenerates under sparse rewards.

Diversity and Coverage Expansion

Table 7: Left: Diversity selection as a plug-in (100-sample MATH, K=4K=4). Right: Coverage over 5 iterations (MATH, 7500 problems, K=8K=8).

MethodBase+Div
GRPO24%53%
RFT34%38%
SFTiter_{\mathrm{iter}}35%38%

Table 7.

Iter12345
Cov (%)3747535656
Gap.63.53.47.44.44

Table 7 (PDF p. 20).

Diversity improves every method (Table 7, left). Coverage gaps decay approximately exponentially (right), consistent with Theorem A.9, saturating at 56% when remaining problems are too hard (pt→0p_t \to0).

TPO on Mathematical Reasoning

Table 8: TPO vs. standard DPO on mathematical reasoning (mean ±\pm std over 3 seeds).

MethodMATHGSM8KAIMETheory basis
DPO (γ=1\gamma=1, β=0.1\beta=0.1)93.8 ±\pm 3.0100.0 ±\pm 0.011.8 ±\pm 1.8—
TPO (γ=0.5\gamma=0.5, β/L\beta/\sqrt{L})98.7 ±\pm 1.2100.0 ±\pm 0.012.3 ±\pm 0.8Thms. 2.5, 2.3

Table 8.

TPO achieves 98.7% on MATH vs. 93.8% for DPO—a 4.9-point gain with 2.5×\times lower variance (Table 8). The improvement is concentrated in the transition regime where hard examples trigger collapse (Theorem 2.5). The γ\gamma advantage generalizes: on Mistral-7B, γ=0.5\gamma=0.5 achieves 97% vs. 88% for γ=1.0\gamma=1.0; on DeepSeek-R1, 94% vs. 88%.

Reference Refresh Experiment

The monotonic improvement with refresh frequency (Table 9) confirms the theory: the gradient signal σ(−βΔ)\sigma(-\beta\Delta) decays between refreshes, and more frequent refreshes keep Δ\Delta small enough to maintain learning. The residual 2-point gap between K=200K=200 and SimPO reflects within-window staleness.

MethodBest Acc (%)
DPO (frozen reference)50
DPO + refresh K=2000K=200068
DPO + refresh K=1000K=100071
DPO + refresh K=500K=50071
DPO + refresh K=200K=20073
SimPO (no reference)75

Table 9. Reference refresh experiment (DS-R1-8B, on-policy). Refreshing the frozen reference every KK steps recovers most of the 25-point gap between DPO and SimPO.

Supporting Dynamical Evidence

Gradient norms (Thm. 2.2). DPO maintains gradient norms 10–20×\times larger than SFT throughout training (∼\sim20 vs. ∼\sim1.5 averaged over 2000 steps), confirming the O(π2)O(\pi^2) vs. Ω(π)\Omega(\pi) separation. This is consistent with the gradient imbalance of Yan et al. [2025] and Ma et al. [ [17]; our contribution is showing it follows from targeted repulsion (Axis 1).

Reference staleness (Prop. A.4). SimPO’s reference divergence grows 214×\times vs. DPO’s 2.6×\times (500 samples). The gap holds across scales: 418×\times vs. 2.3×\times on full MATH, 24×\times vs. 1.3×\times on GSM8K.

N-completion validation. We train DPO with n∈{2,4,8,16}n \in\{2, 4, 8, 16\} completions per prompt (100 prompts, 300 steps, DS-R1-8B) to test whether the two-completion simplex reduction captures the dominant dynamics at intermediate nn. Table 10 reports the margin trajectories.

The key finding: the qualitative prediction (unbounded DPO margin growth) holds at all nn, and the growth ratios at n=4n=4 and n=8n=8 are nearly identical (1.5× vs. 1.6×), suggesting convergence of the effective dynamics. The absolute margin scale increases with nn (more pairwise comparisons provide a stronger training signal), but this is absorbed by βeff\beta_{\mathrm{eff}} (Remark 2.6). The two-completion theory correctly predicts the functional form at all tested nn; the remaining gap is the scale parameter βeff\beta_{\mathrm{eff}}, which Remark 2.6 decomposes as β∥vj∥2/N±O(1/N)\beta\lVert v_j\rVert^2/N \pm O(1/\sqrt{N}) under the independence assumption. To test whether the logarithmic functional form (not just unboundedness) generalizes, we fit Δ(t)=alog⁡t+b\Delta(t)=a\log t+b to the n≥4n\ge4 trajectories and compare against a linear baseline Δ(t)=at+b\Delta(t)=at+b:

n=2n=2n=4n=4n=8n=8n=16n=16
R2R^2 (log fit)0.960.830.700.62
R2R^2 (linear fit)0.980.620.800.43

Table 11.

With only 6 evaluation points per trajectory, neither model achieves high R2R^2 at n≥4n\ge4. The log model is preferred on n=4n=4 and n=16n=16; the linear model wins on n=8n=8. The evidence is consistent with logarithmic growth but not conclusive given the limited data; denser margin tracking would provide a stronger test.

Margin dynamics vs. downstream accuracy. The margin ODEs predict trajectories with quantitative accuracy, but the mapping to downstream performance is nonlinear. On AIME, ρ(Δ,acc)≈0.13−\rho(\Delta,\mathrm{acc})\approx0.13- margin is a poor proxy for accuracy. Accuracy peaks at moderate margins (Δ∗≈10\Delta^*\approx10–30) then degrades, motivating TPO’s controlled attractor.

SimPO Replication: Standard SFT Pipeline

To validate that the method rankings from our direct preference training (Table 4) hold under the standard pipeline, we replicate the SimPO setup: SFT on UltraChat-200k (1 epoch, LoRA r=64r=64), then preference training on UltraFeedback (60K pairs, 5000 steps). Table 11 reports best-checkpoint preference accuracy over 5 seeds per method.

The key finding: the method ranking is preserved. On MI 8B, reference-free methods (SimPO 51.2%, TPO-H 50.8%, TPO-S 50.4%) outperform all reference-dependent methods (47–48%) by 2–4 points, consistent with the staleness axis. On DS 8B, the SFT base is strong enough that UltraFeedback

Table 11: SimPO replication pipeline (SFT → preference). Best-checkpoint preference accuracy (%) on UltraFeedback, mean ± std over 5 seeds. The reference-free advantage persists on MI 8B; on DS 8B the strong SFT base saturates all methods.

DS-R1-8BMI-8BQW-9B
SimPO49.2 ± 0.451.2 ± 0.4—
TPO-H49.6 ± 0.550.8 ± 0.4—
TPO-S49.2 ± 0.450.4 ± 0.5—
KTO49.0 ± 0.048.0 ± 0.0—
DPO49.0 ± 0.048.0 ± 0.0—
cDPO49.0 ± 0.047.0 ± 0.0—
IPO49.0 ± 0.047.0 ± 0.0—

Table 11 (PDF p. 22).

preference training provides minimal additional signal, compressing all methods into a 0.6-point band (49.0–49.6%). The near-zero variance across seeds confirms saturation rather than noise. This pattern is theory-consistent: when the base model already captures most of the preference distribution, the staleness penalty for frozen references is small (the reference stays close to the policy), narrowing the gap between reference-free and reference-dependent methods.

Extended Experimental Details

The proof of Theorem 2.5 (gradient concentration) is in Appendix § 6.§ 13.1.

Multi-Benchmark Replication

All three main predictions replicate across four benchmarks spanning mathematics (MATH, GSM8K, AIME) and code generation (MBPP). Table 12 summarizes the key quantities.

MATHGSM8KAIMEMBPP
(11.6K, 3000 steps)(7K, 3000 steps)(30, 3000 steps)(374, 1000 steps)
Prediction 1: Margin trajectories
DPO Δfinal\Delta_{\mathrm{final}}46525715160
IPO Δfinal\Delta_{\mathrm{final}}0.48.5−0.31.1
Prediction 2: Noise (ε=0.1\varepsilon= 0.1)
DPO Δfinal\Delta_{\mathrm{final}}44418.846
IPO Δfinal\Delta_{\mathrm{final}}1.47.1−0.40.3
Prediction 3: Repulsion (γ\gamma sweep)
γ=0\gamma= 0 accuracy66.5%95%––
γ=0.25\gamma= 0.25 accuracy (held-out)99.5%100%––

Table 12.

Table 12: Multi-benchmark replication of main predictions. DPO margin grows without bound on every benchmark; IPO margin remains bounded. Noise degrades DPO monotonically while IPO is stable. The γ=0\gamma= 0 vs. γ>0\gamma> 0 gap holds wherever the task is solvable (pt>0p_t > 0). AIME margins are near zero for both methods, consistent with Theorem A.9: when base solve rate pt≈0p_t \approx0, neither method can make progress.

IPO Fixed Point Validation (τ\tau-Sweep)

Theorem 2.3(ii) predicts that IPO’s margin converges to 1/(2τ)1/(2\tau). We validate this quantitatively by sweeping τ∈{0.01,0.02,0.05,0.1,0.2}\tau\in\{0.01, 0.02, 0.05, 0.1, 0.2\} on clean data (ε=0\varepsilon= 0):

τ\tau0.010.020.050.10.2
Theory 1/(2τ)1/(2\tau)50.025.010.05.02.5
Empirical45.926.28.03.34.9

Table 14.

The empirical values follow the predicted 1/(2τ)1/(2\tau) scaling across a 20×20 \times range of τ\tau: four of the five points decrease monotonically with τ\tau as predicted, with a mean absolute percentage error of 33%. The largest deviations occur at the extremes. At small τ\tau (large target margin) the finite 1000-step budget has not fully reached the fixed point (the source of the underestimation at τ=0.01\tau= 0.01). The τ=0.2\tau= 0.2 point (empirical 4.9 vs. theory 2.5) departs from the monotone trend; with such a small target margin the estimate is dominated by measurement noise on the clean track set over a short budget. We report the raw values without correction. With noisy data (ε=0.1\varepsilon= 0.1), the fixed points shift to (1−2ε)/(2τ)(1 - 2\varepsilon)/(2\tau) as predicted by Theorem 2.4(ii), with the convergence rate 2τ22\tau^2 unchanged.

Label-Smoothing Sweep (epsilon-Sweep)

This section provides full details for the ε\varepsilon-sweep summarized in § 3.2. Theorem 2.4 predicts that any ε>0\varepsilon> 0 creates a finite margin attractor at Δ∗=βlog⁡((1−ε)/ε)\Delta^* = \beta\log((1-\varepsilon)/\varepsilon), with smaller ε\varepsilon giving larger attractors. We validate this by sweeping ε∈{0.01,0.03,0.05,0.1,0.2}\varepsilon\in\{0.01, 0.03, 0.05, 0.1, 0.2\} on DS-R1-8B with UltraFeedback (60K pairs, 5000 steps, seed 1).

ε\varepsilon0.010.030.050.10.2
Theory Δ∗(β=0.1)\Delta^*(\beta=0.1)4635292214
Final margin16051264832455341
Best accuracy (%)5252514951

Table 1 (PDF p. 23).

The qualitative prediction is confirmed: final margins decrease monotonically with ε\varepsilon, and all are finite (SimPO, i.e. ε=0\varepsilon=0, diverges to >500>500 and continues growing). The quantitative margins are ∼25\sim25–35×35\times the theory prediction, reflecting a much smaller βeff\beta_{\mathrm{eff}} on this 60K-pair dataset than the β/4.4\beta/4.4 calibrated on 500 MATH samples—consistent with the scaling trend in Remark 2.6. Downstream accuracy is flat across ε\varepsilon (49–52%), indicating that the specific attractor location matters less than the qualitative distinction between finite and infinite margins.

Seed Robustness

To verify that the margin dynamics are not artifacts of a particular random seed, we replicate the DPO and IPO margin experiments across four seeds (s∈{0,1,2,3}s \in\{0, 1, 2, 3\}). DPO’s margin grows without bound in every case: {785,2219,152,348}\{785, 2219, 152, 348\} at step 1000. The 14×14\times spread across seeds reflects sensitivity to the training data order, but the qualitative prediction (unbounded growth) holds universally. IPO’s margin remains bounded in every case: {4.9,6.0,4.9,7.1}\{4.9, 6.0, 4.9, 7.1\}, all near the predicted 1/(2τ)=51/(2\tau) = 5.

Model and training. All experiments use Qwen3-8B with LoRA (r=64r = 64, α=128\alpha= 128, applied to all linear layers). Optimizer: AdamW, learning rate 2×10−52 \times10^{-5}, linear warmup over 50 steps, cosine decay. Batch size 1 with gradient accumulation 4 (effective batch size 4). Maximum sequence length 512 tokens. Each run uses a single A100-40GB GPU (∼\sim24GB peak memory). Reference log-probabilities computed via LoRA adapter disabling (zero memory overhead).

Hyperparameters. DPO: β=0.1\beta= 0.1. IPO: τ=0.1\tau= 0.1. SimPO: μ=1.0\mu= 1.0, length-normalized. GRPO: K=8K = 8, binary rewards. Noise experiments: ε∈{0,0.05,0.1,0.2,0.3,0.5}\varepsilon\in\{0, 0.05, 0.1, 0.2, 0.3, 0.5\}. Gamma sweep: γ∈{0,0.25,0.5,0.75,1.0}\gamma\in\{0, 0.25, 0.5, 0.75, 1.0\}. All prediction experiments run for 1000 steps (short) and 2000 steps (long).

Compute budget. Prediction experiments (short + long runs): ∼\sim60 runs ×\times 30 min = 30 GPU-hours. Full-scale replication (MATH, GSM8K, AIME): ∼\sim42 runs ×\times 90 min = 63 GPU-hours. Cross-model validation: ∼\sim12 runs ×\times 90 min = 18 GPU-hours. Baselines and ablations (IPO, SimPO): ∼\sim20 runs ×\times 45 min = 15 GPU-hours. Supplementary (tau sweep, seed robustness, annealing, real datasets): ∼\sim80 runs ×\times 20 min = 27 GPU-hours. Total: ∼\sim153 GPU-hours on 8×\timesA100-40GB.

Data. 500 MATH problems with verified correct and incorrect completions. Held-out evaluation on 100 problems (binary preference accuracy: does the model assign higher log-probability to the correct completion?). Full-scale prediction experiments use 11.6K MATH training examples (3000 steps), 7K GSM8K examples, 30 AIME 2024 problems, and 374 MBPP training problems.

Cross-model validation. To test generalization beyond Qwen3-8B, we replicated the margin dynamics experiment (Prediction 1) on six additional models spanning five families: Qwen3.5-4B, Qwen3-4B (Alibaba), Gemma 3 4B (Google), OLMo-2 7B (AI2), Mistral 7B (Mistral AI), and DeepSeek-R1 7B (DeepSeek). Each model was trained with LoRA (r=64r = 64) using identical hyperparameters (β=0.1\beta= 0.1, τ=0.1\tau= 0.1, lr =2×10−5= 2 \times10^{-5}, 1000 steps). In every case, DPO’s margin grows without bound while IPO’s converges to a bounded value, confirming the universality of Theorem 2.3.

Gradient Concentration and Collapse (gamma-Sweep)

Theorem 2.5 predicts that higher γ\gamma accelerates gradient concentration on hard examples, potentially causing margin collapse. We validate this with a fine-grained γ\gamma-sweep on full-scale MATH (11.6K train, 3000 steps) tracking per-example margins for 50 training examples. A collapse event occurs when an example’s margin rises above 1/(10β)=1001/(10\beta) = 100 then falls back below.

The non-monotonicity is striking and reproducible (Table 13 (PDF p. 24)): γ=0.75\gamma= 0.75 averages 87% across 3 seeds (range 81–92%), while γ=0.5\gamma= 0.5 averages 97% (range 96–99%). Standard DPO (γ=1.0\gamma= 1.0) shows high variance (82–98%), suggesting it sits near a phase boundary where collapse may or may not occur depending on the training trajectory. This is consistent with Theorem 2.5: moderate γ\gamma concentrates the gradient enough to trigger negative interference (vjTwi<0v_j^{\mathsf{T}} w_i < 0) but not enough to generate the restoring force that allows recovery. The collapse is architecture-universal: Mistral-7B shows monotonic degradation (98% →\rightarrow 88% →\rightarrow 83% for γ=0.25,0.75,1.0\gamma= 0.25, 0.75, 1.0), DeepSeek-R1 (96% →\rightarrow 92% →\rightarrow 88%), and Qwen3-4B (88% →\rightarrow 83% →\rightarrow 82%).

γ\gamma0.00.250.50.60.70.750.81.0
MATH (11.6K, transition regime)
Final accuracy (%)66919810096928898
Final margin582784938643983042491067
Collapse events (/50)1479344542434
MATH seed robustness (mean ±\pm range over 2–3 seeds)
Mean accuracy (%)6691979191878889
Seed range–90–9196–9983–10087–9681–9288–8882–98
GSM8K (7K, high-coverage regime)
Final accuracy (%)96100100100100100100100
Collapse events (/50)044000001

Table 13 (PDF p. 24). Gradient concentration sweep on MATH (11.6K train, 3000 steps) and GSM8K (7K train). Accuracies here are measured on the 50 tracked hard examples used for the per-example collapse analysis, so they are lower than the held-out accuracies in Table 12 (e.g. γ=0.25\gamma= 0.25: 91% tracked-hard vs. 99.5% held-out). Collapse events peak at γ=0.7\gamma= 0.7–0.80.8 on MATH, creating a “collapse valley.” GSM8K shows no collapse (high-coverage regime). Seed robustness (2–3 seeds) confirms the pattern is reproducible.

Reproducibility

All loss functions are given in closed form (Equations 1, 3, 25), and Section E specifies the models, hyperparameters, optimizer settings, data construction, and training schedules needed to reproduce every experiment. The total compute budget is ∼\sim153 GPU-hours on 8×\timesA100-40GB (Section E).

NeurIPS Paper Checklist

  1. Claims

    Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

    Answer: [Yes]

    Justification: All three contributions (framework, empirical confirmation, constructive test) stated in the abstract are supported by Sections 2–3.

Guidelines:

  • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.

  • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.

  • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

  • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 5 discusses limitations including the two-completion approximation requiring fitted βeff\beta_{\mathrm{eff}}, modest gains over SimPO, high seed variance in the γ\gamma-sweep, and the LoRA/Adam-specific setting.

Guidelines:

  • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.

  • The authors are encouraged to create a separate “Limitations” section in their paper.

  • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

  • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

  • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

  • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

  • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

  • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theorems (2.2–2.5) and propositions (A.3–A.7, A.9) are numbered, state their assumptions (e.g., gradient flow, two-completion simplex, mean-field approximation), and have complete proofs in Appendix A.

Guidelines:

  • The answer [N/A] means that the paper does not include theoretical results.

  • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

  • All assumptions should be clearly stated or referenced in the statement of any theorems.

  • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

  • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

  • Theorems and Lemmas that the proof relies upon should be properly referenced.

Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Appendix § 10.4 specifies all models, hyperparameters, optimizer settings, LoRA configuration, batch sizes, training steps, and datasets. Loss functions are given in closed form (Equations (1), (3), (25)).

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

  • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

  • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

  • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example: (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No] Justification: Code is not released. The loss functions are given in closed form and Appendix E gives the full experimental details. All source datasets are publicly available (MATH, GSM8K, AIME 2024, MBPP, UltraFeedback, UltraChat-200k, IFEval). The preference pairs were generated by the authors from these problems and can be regenerated with the procedure in Appendix E.

Guidelines:

  • The answer [N/A] means that paper does not include experiments requiring code.

  • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

  • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

  • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

  • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

  • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?

Answer: [Yes]

Justification: Section 3 and Appendix E.4 specify model architectures, LoRA rank, optimizer, learning rate, batch size, training steps, hyperparameters for all methods, and data splits.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

  • The full details can be provided either with the code, in appendix, or as supplemental material.

Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Tables 3–4 report mean ± std over 5 seeds, and p-values from Welch’s t-test are provided for the main TPO-S vs. SimPO comparison (footnote 2, Section 3.2).

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

  • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

  • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

  • The assumptions made should be given (e.g., Normally distributed errors).

  • It should be clear whether the error bar is the standard deviation or the standard error of the mean.

  • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

  • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).

  • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Appendix E.4 reports GPU type (A100-40GB), peak memory (24GB), per-run times, and total budget (153 GPU-hours on 8×\timesA100-40GB) broken down by experiment category.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

  • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

  • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).

Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: The research uses publicly available datasets and models under their published terms, plus preference pairs the authors generated from public problems. It involves no human subjects and no personal data.

Guidelines:

  • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.

  • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.

  • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: The paper does not include a broader impacts discussion. This is foundational research on alignment loss dynamics with no direct path to specific negative applications beyond those inherent to LLM alignment generally.

Guidelines:

  • The answer [N/A] means that there is no societal impact of the work performed.

  • If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.

  • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

  • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

  • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

  • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [N/A]

Justification: The paper does not release new models or datasets; it analyzes existing alignment methods and proposes a loss modification tested on publicly available models and data.

Guidelines:

  • The answer [N/A] means that the paper poses no such risks.

  • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

  • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

  • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All datasets and models are cited with their original papers and used under their published terms. Datasets: MATH (MIT), GSM8K (MIT), MBPP (CC-BY-4.0), UltraFeedback (MIT), UltraChat-200k (MIT), IFEval (Apache-2.0), AIME 2024 (public competition problems). Models: Qwen3 and Qwen3.5 (Apache-2.0), DeepSeek-R1-0528-Qwen3-8B and DeepSeek-R1-Distill-Qwen-7B (MIT), Ministral-3-8B (Apache-2.0), Mistral-7B (Apache-2.0), OLMo-2-7B (Apache-2.0), Gemma 3 4B (Gemma Terms of Use). The preference pairs and correct/incorrect completions were generated by the authors from these public problems.

Guidelines:

  • The answer [N/A] means that the paper does not use existing assets.

  • The authors should cite the original paper that produced the code package or dataset.

  • The authors should state which version of the asset is used and, if possible, include a URL.

  • The name of the license (e.g., CC-BY 4.0) should be included for each asset.

  • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

  • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

  • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

  • If this information is not available online, the authors are encouraged to reach out to the asset’s creators.

New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [N/A]

Justification: The paper does not release new code, datasets, or model weights.

Guidelines:

  • The answer [N/A] means that the paper does not release new assets.

  • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

  • The paper should discuss whether and how consent was obtained from people whose asset is used.

  • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [N/A]

Justification: The paper does not involve crowdsourcing or human subjects research.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

  • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [N/A]

Justification: The paper does not involve human subjects research.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

  • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

  • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.

Answer: [N/A]

Justification: LLMs are the subject of study (as models being aligned) but are not used as a non-standard component of the research methodology itself.

Guidelines:

  • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

  • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.

References

  1. [2]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.arxiv.org/abs/2108.07732
  2. [3]Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust DPO: Aligning language models with noisy feedback. In International Conference on Machine Learning, 2024.arxiv.org/abs/2403.00409
  3. [4]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiiichiro Nakano, et al. Training verifiers to solve math word problems, 2021.arxiv.org/abs/2110.14168
  4. [5]Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with scaled ai feedback, 2024. URL https://arxiv.org/abs/2310.01377.
  5. [6]Mingyang Deng, He Li, Tianhong Li, Yilun Du, and Kaiming He. Generative modeling via drifting, 2026.arxiv.org/abs/2602.04770
  6. [7]Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization, 2024.arxiv.org/abs/2402.01306
  7. [8]Zhengyang Geng, Mingyang Deng, Xingjian Bai, J. Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling, 2025.arxiv.org/abs/2505.13447
  8. [9]Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 4447–4455. PMLR, 02–04 May 2024. URL https://proceedings.mlr.press/v238/gheshlaghi-azar24a.html.arxiv.org/abs/2310.12036
  9. [10]Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, and Mathieu Blondel. Direct language model alignment from online ai feedback, 2024. URL https://arxiv.org/abs/2402.04792.
  10. [11]Jiaqi Han, Mingjian Jiang, Yuxuan Song, Stefano Ermon, and Minkai Xu. $f$-po: Generalizing preference optimization with $f$-divergence minimization. In Yingzhen Li, Stephan Mandt, Shipra Agrawal, and Emtiyaz Khan, editors, Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, volume 258 of Proceedings of Machine Learning Research, pages 1144–1152. PMLR, 03–05 May 2025. URL https://proceedings.mlr.press/v258/han25a.html.arxiv.org/abs/2410.21662
  11. [12]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Advances in Neural Information Processing Systems, 2021.
  12. [13]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.arxiv.org/abs/2006.11239
  13. [14]Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model, 2024.arxiv.org/abs/2403.07691
  14. [15]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.arxiv.org/abs/2106.09685
  15. [16]Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.02747
  16. [17]Qinwei Ma, Jingzhe Shi, Can Jin, Jeng-Neng Hwang, Serge Belongie, and Lei Li. Gradient imbalance in direct preference optimization. arXiv preprint arXiv:2502.20847, 2025.arxiv.org/abs/2502.20847
  17. [18]Mathematical Association of America. American Invitational Mathematics Examination (AIME), 2024. URL https://artofproblemsolving.com/wiki/index.php/2024_AIME_I.
  18. [19]Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with a reference-free reward. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=3Tzcot1LKb.
  19. [20]Eric Mitchell. A note on DPO with noisy preferences and relationship to IPO. https://ericmitchell.ai/cdpo.pdf, 2023.
  20. [21]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2203.02155
  21. [22]Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2305.18290
  22. [23]Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional unalignment: Likelihood displacement in direct preference optimization. Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2410.08847
  23. [24]Max Ruiz Luyten and Mihaela van der Schaar. The reasoning-creativity trade-off: Toward creativity-driven problem solving. arXiv preprint arXiv:2601.00747, 2025.arxiv.org/abs/2601.00747
  24. [25]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.arxiv.org/abs/1707.06347
  25. [26]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300.

Paper details

vlnc:W_u848yu5r
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents