betaFeedback

WorksComputer Science

PORT: Preference Optimization via Robust Token-Level Reweighting

Abstract

Preference optimization has become a central approach for aligning large language models with human values, but its effectiveness depends heavily on the quality of preference annotations. In practice, preference data is often noisy due to annotation errors and ambiguity. Existing robust preference optimization methods primarily operate at the sequence level, implicitly treating entire responses as uniformly correct or incorrect. However, rejected responses may still contain informative reasoning steps, while preferred responses can include subtle errors. In this work, we propose Preference Optimization via Robust Token-level reweighting (PORT), a fine-grained framework for robust alignment under noisy preferences. PORT performs token-level reweighting using the empirical cumulative distribution function (CDF) of token log-probabilities, yielding an efficient forward-pass-only proxy to gradient-norm penalization that selectively suppresses corrupted tokens without additional backward-pass computation. We provide theoretical analysis showing that PORT reduces the gradient bias from noisy preference labels by minimizing an upper bound on the bias term. Extensive experiments across diverse noise settings demonstrate that PORT consistently improves robustness and outperforms existing sequence-level baselines. The code is available at https://github.com/osu-srml/PORT

Introduction

Preference optimization has emerged as a leading method for aligning large language models (LLMs) with human values, ensuring their outputs are useful, safe, and consistent with user intent [2]. Unlike Reinforcement Learning from Human Feedback (RLHF) [18], which trains an explicit reward model and then applies reinforcement learning to guide the policy toward high-reward responses, preference optimization methods such as Direct Preference Optimization (DPO) [20] and Identity Preference Optimization (IPO) [7] directly optimize the implicit reward margin between preferred and rejected responses within a stable, supervised learning paradigm.

Despite their success, preference optimization methods share a key vulnerability: their effectiveness depends critically on the quality of preference annotations. In practice, however, preference data is rarely clean, and acquiring high-quality data at scale is often costly and even infeasible. Annotator disagreement, intrinsic ambiguity in hard-to-distinguish pairs, and labeling errors all introduce noise into the training signal, which biases gradient estimates and degrades alignment performance. To address this, recent work has explored mitigation techniques such as label smoothing [4, 13, 16], regularization [15, 27], and heuristic data filtering [8, 17] (see Appendix § 8 for a discussion of each).

However, these approaches are fundamentally limited by their reliance on sequence-level modeling of preference noise. Existing preference optimization objectives and their noise-mitigation variants aggregate per-token log-ratios uniformly across an entire response, implicitly treating all token positions as equally reliable. This assumption is often violated in practice: a preferred response

40th Conference on Neural Information Processing Systems (NeurIPS 2026). may contain localized hallucinations or reasoning errors, while a rejected response may still include correct intermediate reasoning steps.

Figure 1. Overview of the PORT training pipeline: 1) Given a batch B={(x~,y~w[b],y~l[b])}b=1∣B∣\mathcal{B} = \{(\tilde{x}, \tilde{y}_{w}^{[b]}, \tilde{y}_{l}^{[b]})\}_{b=1}^{|\mathcal{B}|} of preference pairs, where y~w[b]\tilde{y}_{w}^{[b]} and y~l[b]\tilde{y}_{l}^{[b]} denote chosen and rejected responses. 2) Forward pass: the policy πθ\pi_{\theta} and reference policy πref\pi_{\mathrm{ref}} are evaluated to obtain per-token log-ratios δt[b](θ)\delta_{t}^{[b]}(\theta) and token-level logits s~w,t[b],s~l,t[b]\tilde{s}_{w,t}^{[b]}, \tilde{s}_{l,t}^{[b]}. 3) Δ\DeltaCDF estimation: token log-probabilities are pooled across the batch to form a global empirical CDF F(⋅)F(\cdot), which gives percentile ranks Fw,t[b]F_{w,t}^{[b]} and Fl,t[b]F_{l,t}^{[b]} for the chosen and rejected tokens. A large gap indicates that the rejected token is assigned anomalously high probability relative to the chosen token, signaling potential token-level label noise. 4) Weight conversion: gap Δt[b]=Fl,t[b]−Fw,t[b]\Delta_{t}^{[b]} = F_{l,t}^{[b]} - F_{w,t}^{[b]} is mapped to a weight wt[b]=γexp⁡(ρΔt[b])w_{t}^{[b]} = \gamma\exp(\rho\Delta_{t}^{[b]}). 5) Weighted token-level update: these weights are incorporated into the preference objective L~w\tilde{\mathcal{L}}_{w}, yielding a robust gradient update that suppresses corrupted tokens. The procedure uses only a single forward pass and introduces no extra backward-pass overhead.

Consider a preference pair where the preferred response is “The capital of Australia is Canberra, which was purpose-built as a compromise between Sydney and Melbourne,” while the rejected response is “The capital of Australia is Sydney, as it is the largest city.” If the preference label is flipped due to noise, sequence-level methods would penalize all tokens in the rejected response equally. However, this treatment is overly coarse: although the response is factually incorrect, many tokens (e.g., “as it is the largest city”) are locally fluent and not directly responsible for the error. Penalizing all tokens uniformly therefore introduces unnecessary bias into the learning signal, allowing noisy or uninformative tokens to disproportionately influence the gradient.

Unfortunately, existing noise mitigation strategies fail to resolve this mismatch. Label smoothing methods such as Conservative-DPO (cDPO) [16] apply uniform shrinkage to the entire objective, effectively scaling all token contributions equally. Pair-level reweighting methods such as Distributionally Robustifying DPO (Dr.DPO) [27] assign a single scalar weight to each preference pair, leaving within-sequence variation unmodeled. Even token-level importance sampling methods such as Token-Level Importance Sampling (TIS-DPO) [14] introduce finer granularity but rely on external reward models, thereby reintroducing risks of reward misgeneralization and additional computational overhead. Collectively, these limitations highlight a key gap: existing methods either operate at too coarse a granularity or depend on external supervision to achieve token-level adaptivity.

Motivated by this, we explore token-level noise mitigation under instance-dependent preference noise and propose Preference Optimization via Robust Token-level reweighting (PORT), a framework that performs token-level reweighting without requiring external reward models or additional backward passes. The key idea is to assign a weight to each token position such that solving a weighted preference optimization on noisy data yields a solution that aligns with true human preferences.

To ground this design in theory, we analyze the effect of noise on the gradient of preference optimization and derive an upper bound on the resulting gradient bias. Minimizing this bound yields a closed-form optimal token-level weighting function, which characterizes how each token should be reweighted according to its sensitivity to noise and its contribution to the learning signal.

However, this optimal solution is not directly computable in practice, as it depends on unknown noise statistics and the norm of the gradient with respect to all the model parameters which is computationally expensive. To bridge this gap, we identify a practical surrogate that preserves the structure of the optimal solution. In particular, we show that the relative ordering of token probabilities within a batch provides a sufficient signal for approximating the optimal weights. This leads to a CDF-based weighting scheme computed from model log-probabilities, which serves as an efficient and fully forward-pass–based proxy for the theoretically optimal solution.

Our main contributions are as follows:

  • We propose PORT, a robust alignment framework that introduces a token-level CDF reweighting objective that requires zero additional backward-pass overhead and no external reward model.

  • We provide theoretical guarantees showing that our weighting scheme minimizes an upper bound on gradient bias under instance-dependent noise, and characterize the conditions under which it strictly reduces the bias.

  • We conduct extensive experiments on Anthropic HH and Alpaca Farm benchmarks, demonstrating consistent and significant improvements over sequence-level baselines across diverse noise settings.

Background and Problem Formulation

Let πθ\pi_\theta denote a language model parameterized by θ\theta, which defines a conditional distribution πθ(⋅∣x)\pi_\theta(\cdot\mid x) over responses given a prompt x∈Xx \in\mathcal{X}. Consider a preference dataset D={(x,yw,yl)}1\mathcal{D} = \{(x, y_w, y_l)\}^{1}, where ywy_w and yly_l are the preferred and dispreferred responses for prompt xx, respectively. Traditional RLHF [18] first trains a reward model rϕ(x,y)r_\phi(x,y) by fitting the Bradley-Terry model [3]

P(yw≻yl∣x)=exp⁡(rϕ(x,yw))exp⁡(rϕ(x,yl))+exp⁡(rϕ(x,yw)).P(y_w \succ y_l \mid x) = \frac{\exp(r_\phi(x,y_w))}{\exp(r_\phi(x,y_l))+\exp(r_\phi(x,y_w))}.

The learned reward is then used with Proximal Policy Optimization (PPO) [21] to optimize πθ\pi_\theta:

max⁡πθEx∼D,y∼πθ(⋅∣x)[rϕ(x,y)−βDKL(πθ(⋅∣x)∥πref(⋅∣x))](1)\max_{\pi_\theta}\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_\theta(\cdot\mid x)}\left[r_\phi(x,y)-\beta D_{\mathrm{KL}}\left(\pi_\theta(\cdot\mid x)\parallel\pi_{\mathrm{ref}}(\cdot\mid x)\right)\right] \tag*{(1)}

where πref\pi_{\mathrm{ref}} is the reference policy (typically the initial supervised fine-tuned model).

(Token-Level) Preference Optimization. Ouyang et al. [18] show that the optimal policy under Eq. (1) satisfies rϕ(x,y)=βlog⁡πθ(y∣x)πref(y∣x)+βlog⁡Z(x)r_\phi(x,y)=\beta\log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}+\beta\log Z(x), enabling the reward model to be eliminated entirely. This insight gives rise to DPO [20], which optimizes the policy directly from preference data. Rafailov et al. [20] further present a generalized preference optimization framework that encompasses several existing algorithms:

min⁡θL(θ)=E(x,yw,yl)∼D[f(βh(x,yw,yl;θ))]=E(x,yw,yl)∼D[f(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))].\min_\theta\mathcal{L}(\theta)=\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}}\left[f\left(\beta h(x,y_w,y_l;\theta)\right)\right]=\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}}\left[f\left(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\mathrm{ref}}(y_w\mid x)}-\beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\mathrm{ref}}(y_l\mid x)}\right)\right].

where we define h(x,y,y′;θ):=log⁡πθ(y∣x)πref(y′∣x)πθ(y′∣x)πref(y∣x)h(x,y,y';\theta):=\log\frac{\pi_\theta(y\mid x)\pi_{\mathrm{ref}}(y'\mid x)}{\pi_\theta(y'\mid x)\pi_{\mathrm{ref}}(y\mid x)} as the preference score of response yy relative to y′y' given prompt xx under policy πθ\pi_\theta. The choice of f(⋅)f(\cdot) instantiates different algorithms: DPO [20] sets f(x)=−log⁡σ(x)f(x)=-\log\sigma(x), recovering a cross-entropy loss over preference pairs, while IPO [7] sets f(x)=(x−12)2f(x)=\left(x-\frac{1}{2}\right)^2, regressing the preference margin directly to mitigate over-optimization.

The sequence-level log-ratios above decompose naturally under the autoregressive factorization of the policy. Specifically, for a response y=(y1,y2,…,yT)y=(y_1,y_2,\ldots,y_T) of length TT, we have log⁡πθ(y∣x)πref(y∣x)=∑t=1Tlog⁡πθ(yt∣x,y<t)πref(yt∣x,y<t)\log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}=\sum_{t=1}^{T}\log\frac{\pi_\theta(y_t\mid x,y_{<t})}{\pi_{\mathrm{ref}}(y_t\mid x,y_{<t})}. Substituting into the above yields the decomposed preference optimization objective, which expresses the sequence-level log-ratio as a sum of per-token log-ratios:

min⁡θL(θ)=E(x,yw,yl)∼D[f(∑t=1∣yw∣βlog⁡πθ(yw,t∣x,yw,<t)πref(yw,t∣x,yw,<t)−∑t=1∣yl∣βlog⁡πθ(yl,t∣x,yl,<t)πref(yl,t∣x,yl,<t))].(2)\min_\theta\mathcal{L}(\theta)=\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}}\left[f\left(\sum_{t=1}^{|y_w|}\beta\log\frac{\pi_\theta(y_{w,t}\mid x,y_{w,<t})}{\pi_{\mathrm{ref}}(y_{w,t}\mid x,y_{w,<t})}-\sum_{t=1}^{|y_l|}\beta\log\frac{\pi_\theta(y_{l,t}\mid x,y_{l,<t})}{\pi_{\mathrm{ref}}(y_{l,t}\mid x,y_{l,<t})}\right)\right]. \tag*{(2)}

Noisy Human Feedback

In practice, human preference data are often noisy and may not accurately reflect true preferences. Let D~={(x,y~w,y~l)}\widetilde{\mathcal{D}}=\{(x,\widetilde{y}_w,\widetilde{y}_l)\} denote the observed (noisy) preference dataset, derived from an underlying noise-free dataset D={(x,yw,yl)}D = \{(x, y_w, y_l)\}. To simplify notation, let zz denote a sample tuple (x,yw,yl)(x, y_w, y_l) from DD and −z:=(x,yl,yw)-z := (x, y_l, y_w) its flipped counterpart. Similarly, z~:=(x,y~w,y~l)\tilde{z} := (x, \tilde{y}_w, \tilde{y}_l) and −z~:=(x,y~l,y~w)-\tilde{z} := (x, \tilde{y}_l, \tilde{y}_w) denote the corresponding tuples in noisy D~\tilde{D}. We define the flip probability η(z):=P(z~=−z)\eta(z) := \mathbb{P}(\tilde{z} = -z), i.e. the probability that the true ordering yw≻yly_w \succ y_l is corrupted to yl≻ywy_l \succ y_w.

Following [9], we consider two noise models that capture systematic annotation errors and intrinsic ambiguity in hard-to-distinguish pairs: (i) ϵ\epsilon-mislabeled noise: each preference label is independently flipped with probability η(z)≡ϵ\eta(z) \equiv\epsilon, where a larger ϵ\epsilon indicates more severe corruption. (ii) α\alpha-uncertain noise: the preference label y~w≻y~l\tilde{y}_w \succ\tilde{y}_l is sampled with probability σ(α(r∗(yw∣x)−r∗(yl∣x)))\sigma(\alpha(r^*(y_w|x) - r^*(y_l|x))) and y~l≻y~w\tilde{y}_l \succ\tilde{y}_w otherwise, where r∗r^* denotes the ground-truth reward function and α>0\alpha> 0 controls the level of uncertainty. The corresponding flip probability is η(z)=1−σ(α(r∗(yw∣x)−r∗(yl∣x)))\eta(z) = 1 - \sigma(\alpha(r^*(y_w|x) - r^*(y_l|x))), and as α→0\alpha\to0, labels approach random guessing.

Research Goal. As discussed in Section 1, existing noise mitigation methods either operate at the sequence level or rely on external supervision, motivating the need for token-level noise mitigation without external reward models. In this paper, we propose a token-level weighting mechanism for robust preference optimization that effectively learns a policy πθ\pi_\theta aligned with genuine preferences from a noisy preference dataset D~={(x,y~w,y~l)}\tilde{D} = \{(x, \tilde{y}_w, \tilde{y}_l)\}. Specifically, for the token-level log-ratio at each position tt, defined as δt(θ)\delta_t(\theta) below, we assign a scalar weight w(z,t)≥0w(z,t) \ge0 and define a weighted objective2:

min⁡θL~w(θ)=min⁡θE(x,y~w,y~l)∼D~[f(β∑t=1Tw(z,t;θ)⋅δt(θ))]withδt(θ)=log⁡πθ(y~w,t∣x,y~w,<t)πref(y~l,t∣x,y~l,<t)πθ(y~l,t∣x,y~l,<t)πref(y~w,t∣x,y~w,<t).(3)\min_\theta\tilde{\mathcal{L}}_w(\theta) = \min_\theta\mathbb{E}_{(x,\tilde{y}_w,\tilde{y}_l)\sim\tilde{D}}\left[f\left(\beta\sum_{t=1}^{T}w(z,t;\theta)\cdot\delta_t(\theta)\right)\right] \quad\text{with}\quad \delta_t(\theta)=\log\frac{\pi_\theta(\tilde{y}_{w,t}\mid x,\tilde{y}_{w,<t})\pi_{\mathrm{ref}}(\tilde{y}_{l,t}\mid x,\tilde{y}_{l,<t})}{\pi_\theta(\tilde{y}_{l,t}\mid x,\tilde{y}_{l,<t})\pi_{\mathrm{ref}}(\tilde{y}_{w,t}\mid x,\tilde{y}_{w,<t})}. \tag*{(3)}

The central challenge is to identify optimal weights {w(z,t)}t=1T\{w(z,t)\}_{t=1}^{T} such that the solution to the above aligns with the solution to Eq. (2) learned from noise-free data DD. Rather than learning a fixed set of weights, we seek an optimal weight function w(x,y~w,y~l,t):=w(z~,t)w(x,\tilde{y}_w,\tilde{y}_l,t) := w(\tilde{z},t) that computes position-level weights from the data and the current policy. Throughout the analysis, we write the token-level weighting function as w(z,t)w(z,t), with its dependence on the current policy parameters left implicit. When computing gradients, we treat the weights as fixed coefficients. The weights may therefore be updated as the policy evolves, but gradients are not propagated through the weight computation.

Optimal Weighting Function

Next, we identify the optimal weight function w(z,t;θ)w(z,t;\theta) such that minimizing the reweighted objective L~w\tilde{\mathcal{L}}_w recovers a similar solution to minimizing L\mathcal{L} on noise-free data. To this end, we first analyze the gradient bias to understand how noisy preference labels impact the gradient update, and then derive the optimal weight function that minimizes such bias. Finally, we identify conditions under which the proposed token-level weighting mechanism is guaranteed to reduce gradient bias.

Gradient Bias under Preference Noise. Given the standard preference optimization objective in Eq. (2), let L(θ)\mathcal{L}(\theta) and L~(θ)\tilde{\mathcal{L}}(\theta) denote the losses on the ground-truth preference dataset DD and the noisy dataset D~\tilde{D}, respectively. By [16], their corresponding gradients are given by

g(θ):=∇θL(θ)=E[∇ℓ(z;θ)]g(\theta) := \nabla_\theta\mathcal{L}(\theta) = \mathbb{E}\left[\nabla\ell(z;\theta)\right]
g~(θ):=∇θL~(θ)=E[(1−η(z))∇ℓ(z;θ)+η(z)∇ℓ(−z;θ)]\tilde{g}(\theta) := \nabla_\theta\tilde{\mathcal{L}}(\theta) = \mathbb{E}\left[(1-\eta(z))\nabla\ell(z;\theta)+\eta(z)\nabla\ell(-z;\theta)\right]

where η(z)=P(z~=−z)\eta(z) = \mathbb{P}(\tilde{z}=-z) is the flip probability defined in Section 2 and ℓ(z;θ):=f(β∑tδt(z;θ))\ell(z;\theta) := f\left(\beta\sum_t\delta_t(z;\theta)\right) is the per-sample loss, with ℓ(−z;θ)\ell(-z;\theta) denoting its value on the flipped triple (x,yl,yw)(x,y_l,y_w). The effect of label noise can be characterized by the gradient bias, defined as the difference between g~(θ)\tilde{g}(\theta) and g(θ)g(\theta).

Definition 1 (Gradient Bias). g~(θ)\tilde{g}(\theta) deviates from the noise-free gradient g(θ)g(\theta) by a bias term BB:

B=g~(θ)−g(θ)=E[η(z)(∇ℓ(−z;θ)−∇ℓ(z;θ))]:=Δg(4)B = \tilde{g}(\theta)-g(\theta)=\mathbb{E}\left[\eta(z)\left(\nabla\ell(-z;\theta)-\nabla\ell(z;\theta)\right)\right] := \Delta g \tag*{(4)}

To mitigate the impact of noise, we consider a token-level weighting mechanism, yielding the weighted objective L~w(θ)\tilde{\mathcal{L}}_w(\theta) in (3). Denote its gradient as g~w(θ):=∇θL~w(θ)\tilde{g}_w(\theta) := \nabla_\theta\tilde{\mathcal{L}}_w(\theta) and its clean-data counterpart as gw(θ):=∇θLw(θ)g_w(\theta) := \nabla_\theta\mathcal{L}_w(\theta). How well minimizing L~w(θ)\tilde{\mathcal{L}}_w(\theta) recovers a model aligned with the true preference can be captured by weighted gradient bias, defined as the difference between g~w(θ)\tilde{g}_w(\theta) and g(θ)g(\theta).2

Definition 2 (Weighted Gradient Bias). g~w(θ)\widetilde{g}_w(\theta) deviates from noise-free g(θ)g(\theta) by a bias term BwB_w:

Bw=g~w(θ)−g(θ)=[g~w(θ)−gw(θ)]+[gw(θ)−g(θ)]:=Δgw+Rb(5)B_w = \widetilde{g}_w(\theta) - g(\theta) = [\widetilde{g}_w(\theta) - g_w(\theta)] + [g_w(\theta) - g(\theta)] := \Delta g_w + R_b \tag*{(5)}

where ℓw(z;θ):=f(β∑t=1Tw(z,t)δt(z;θ))\ell_w(z;\theta) := f\left(\beta\sum_{t=1}^{T} w(z,t)\delta_t(z;\theta)\right), and Δgw:=g~w(θ)−gw(θ)=E[η(z)(∇ℓw(−z;θ)−∇ℓw(z;θ))]\Delta g_w := \widetilde{g}_w(\theta)-g_w(\theta) = \mathbb{E}[\eta(z)(\nabla\ell_w(-z;\theta)-\nabla\ell_w(z;\theta))] and Rb:=gw(θ)−g(θ)R_b := g_w(\theta)-g(\theta).

Detailed derivations are deferred to Appendix § 9.1. For the analysis, the weight is invariant to swapping the two responses

Optimal weighting function. Recall that our goal is to identify an optimal weight function such that the solution to the weighted objective in Eq. (3) on noisy data aligns with the true preference. To this end, we minimize the norm of the weighted gradient bias BwB_w:

min⁡w∥Bw∥=∥Δgw+Rb∥(6)\min_w \|B_w\| = \|\Delta g_w + R_b\| \tag*{(6)}

We restrict the weight function w(z,t;θ)w(z,t;\theta) to the admissible set W:={w≥0:Ep[w(z,t)]=1}\mathcal{W} := \{w \ge0 : \mathbb{E}_p[w(z,t)] = 1\} to avoid a trivial solution, where pp denotes the probability density function of zz. We define the reweighted distribution pw(z):=w(z,t;θ)p(z)p_w(z) := w(z,t;\theta)p(z). Applying Jensen’s and Pinsker’s inequalities, we can upper bound ∥Δgw∥\|\Delta g_w\| and ∥Rb∥\|R_b\|:

∥Δgw∥≤βKE[η(z)w(z,t)∥∇δt(z;θ)∥]\|\Delta g_w\| \le\beta K\mathbb{E}\left[\eta(z)w(z,t)\|\nabla\delta_t(z;\theta)\|\right]
∥Rb∥≤C⋅2DKL(pw∥p)\|R_b\| \le C\cdot\sqrt{2D_{\mathrm{KL}}(p_w\mathbin\Vert p)}

where C,KC,K is the constant. Note that the upper bound of ∥Rb∥\|R_b\| depends on weight function w(z,t;θ)w(z,t;\theta) through KL-divergence DKL(pw∥p)D_{\mathrm{KL}}(p_w\mathbin\Vert p); minimizing this term prevents the reweighted distribution from drifting too far from the original data distribution. Since directly minimizing ∥Bw∥\|B_w\| is intractable, we instead minimize the derived upper bound, which admits a tractable closed-form solution. This leads to the following optimization problem:

min⁡wE[w(z,t)η(z)∥∇δt(z;θ)∥]+1λDKL(pw∥p)(7)\min_w \mathbb{E}[w(z,t)\eta(z)\|\nabla\delta_t(z;\theta)\|] + \frac{1}{\lambda}D_{\mathrm{KL}}(p_w\mathbin\Vert p) \tag*{(7)}

where λ>0\lambda> 0 is a trade-off parameter that balances the two upper bounds. Theorem 1 below shows that this optimization admits a closed-form solution, with the proof deferred to Appendix § 9.2.

Theorem 1 (Surrogate Optimal Weighting Function). The surrogate optimal weight function w∗w^* that minimizes (7) is

w∗(z,t)∝exp⁡(−λη(z)∥∇δt(z;θ)∥)(8)w^*(z,t) \propto\exp\left(-\lambda\eta(z)\|\nabla\delta_t(z;\theta)\|\right) \tag*{(8)}

Theorem 1 implies that samples with a high noise rate η\eta, a large gradient norm ∥∇δt(z;θ)∥\|\nabla\delta_t(z;\theta)\|, or both, should be assigned smaller weights. This aligns with the goal of robust optimization, since such samples may exert strong gradient updates in directions that are likely incorrect and are thus most harmful. The parameter λ\lambda controls the trade-off: as λ→0\lambda\to0 the weights approach uniform and recover the unweighted objective, while as λ→∞\lambda\to\infty it approaches hard filtering of high-risk samples.

Local improvement guarantee. The proposed reweighted objective strictly generalizes the original objective: setting wt(⋅)≡1w_t(\cdot) \equiv1 in L~w\widetilde{\mathcal{L}}_w recovers the unweighted objective L~\widetilde{\mathcal{L}}. A natural question is whether, and under what conditions, our weighting mechanism strictly improves over the unweighted baseline, i.e., when can we guarantee a reduced gradient bias ∥Bw∥<∥B∥\|B_w\| < \|B\|.

Proposition 1 (Sufficient Condition for Strict Improvement). Suppose that

Cov⁡(η(z)∥∑t∇δt(z;θ)∥,(η(z)+f′(β∑tδt(z;θ)))⟨∑t∇δt(z;θ),η(z)∑t∇δt(z;θ)⟩)≻0,(9)\operatorname{Cov}\left(\eta(z)\left\|\sum_t \nabla\delta_t(z;\theta)\right\|,\left(\eta(z)+f'\left(\beta\sum_t\delta_t(z;\theta)\right)\right)\left\langle\sum_t\nabla\delta_t(z;\theta),\eta(z)\sum_t\nabla\delta_t(z;\theta)\right\rangle\right) \succ0, \tag*{(9)}

where ≻\succ denotes the componentwise comparison operator. Then for sufficiently small λ>0\lambda> 0, the reweighted objective strictly reduces the gradient bias: ∥Bw∥<∥B∥\|B_w\| < \|B\|.

See proof in Appendix § 9.3. Note that this condition is only sufficient but not necessary: it characterizes a natural regime in which the benefit of down-weighting noisy samples outweighs the distortion introduced in the clean gradient direction. We verify in the following that this condition holds under both noise models defined in Section 2.

  • α\alpha-uncertain noise. When η(z)=1−σ(α(r∗(yw∣x)−r∗(yl∣x)))\eta(z)=1-\sigma(\alpha(r^*(y_w|x)-r^*(y_l|x))), Eq. (9) reduces to requiring that η(z)>σ(−∑tδt(z;θ))\eta(z)>\sigma(-\sum_t\delta_t(z;\theta)) in expectation, i.e., the true noise rate exceeds the model’s implicit uncertainty. Substituting the noise model yields σ(−α∑tσt(z;θ))>σ(−∑tσt(z;θ))\sigma(-\alpha\sum_t\sigma_t(z;\theta))>\sigma(-\sum_t\sigma_t(z;\theta)), which holds strictly whenever α<1\alpha<1. Intuitively, α<1\alpha<1 means the annotator is less certain than the model: the annotator’s labels are noisier than the model’s current preference predictions. This regime arises naturally in early training or on ambiguous preference pairs, where noise mitigation is most beneficial. When α=1\alpha=1, the noise and confidence terms cancel exactly, and reweighting offers no benefit.

  • ϵ\epsilon-mislabeled noise. When labels are flipped uniformly with constant rate η(z)=ϵ\eta(z)=\epsilon, the noise rate is decoupled from the sample content, and Eq. (9) simplifies to ϵ>Ez[σ(−β∑tδt(z;θ))]\epsilon>\mathbb{E}_z[\sigma(-\beta\sum_t\delta_t(z;\theta))] (see Appendix § 9.3). Since σ(0)=1/2\sigma(0)=1/2, it fails near initialization for any ϵ<1/2\epsilon<1/2, where down-weighting a pair removes more clean signal than noise. It holds once the model separates most clean pairs by a sufficient margin, i.e., in the later phase of training where the gradient is increasingly dominated by flipped labels and DPO tends to memorize them.

PORT: Preference Optimization via Robust Token-level Reweighting

From theory to practice. Theorem 1 shows that w∗∝exp⁡(−λη(z)∥∇δt(z;θ)∥)w^* \propto\exp(-\lambda\eta(z)\lVert\nabla\delta_t(z;\theta)\rVert). However, neither η(z)\eta(z) nor ∥∇δt(z;θ)∥\lVert\nabla\delta_t(z;\theta)\rVert is directly observable, and computing them during training is computationally expensive. We therefore seek a tractable proxy that approximates the optimal weighting in practice.

We hypothesize that the relative difference in the cumulative distribution function (CDF) of logits within a batch can serve as such a proxy. To formalize this idea, consider a batch containing BB preference pairs (x[b],y~w[b],y~l[b])(x^{[b]},\widetilde{y}_w^{[b]},\widetilde{y}_l^{[b]}). We pool all non-padding token log-probabilities in the batch into a single reference set R=⋃b=1B⋃t=1Tb{πθ(y~w,t[b]∣x[b],y~w,<t[b]),πθ(y~l,t[b]∣x[b],y~l,<t[b])}\mathcal{R}=\bigcup_{b=1}^{B}\bigcup_{t=1}^{T_b}\{\pi_\theta(\widetilde{y}_{w,t}^{[b]} \mid x^{[b]},\widetilde{y}_{w,<t}^{[b]}),\pi_\theta(\widetilde{y}_{l,t}^{[b]} \mid x^{[b]},\widetilde{y}_{l,<t}^{[b]})\}, which contains the scores of all non-padding chosen and rejected tokens in the batch. Based on this reference set, we define the empirical CDF F(u;R)=1∣R∣∑r∈R1(r<u)F(u;\mathcal{R})=\frac{1}{|\mathcal{R}|}\sum_{r\in\mathcal{R}}\mathbb{1}(r<u). For the bb-th preference pair, we further define Fw,t[b]=F(πθ(y~w,t[b]∣x[b],y~w,<t[b]);R)F_{w,t}^{[b]}=F(\pi_\theta(\widetilde{y}_{w,t}^{[b]}\mid x^{[b]},\widetilde{y}_{w,<t}^{[b]});\mathcal{R}) and Fl,t[b]=F(πθ(y~l,t[b]∣x[b],y~l,<t[b]);R)F_{l,t}^{[b]}=F(\pi_\theta(\widetilde{y}_{l,t}^{[b]}\mid x^{[b]},\widetilde{y}_{l,<t}^{[b]});\mathcal{R}), which measure the relative percentile ranks of the chosen and rejected token logits within the batch distribution. Using these quantities, we define the directional CDF difference ΔCDF=Fl,t[b]−Fw,t[b]\Delta\mathrm{CDF}=F_{l,t}^{[b]}-F_{w,t}^{[b]}. Intuitively, a large positive ΔCDF\Delta\mathrm{CDF} indicates that the rejected token receives unexpectedly high probability mass relative to the chosen token, which is a strong signal of preference noise.

To validate our hypothesis, we empirically compare the true gradient norm with our proposed CDF-based metric. Specifically, Figure 2 plots the SFT (Supervised Fine-Tuning) gradient norm ∥∇δt(z;θ)∥\lVert\nabla\delta_t(z;\theta)\rVert multiplied with α\alpha-uncertain noise against ΔCDF=Fl,t[b]−Fw,t[b]\Delta\mathrm{CDF}=F_{l,t}^{[b]}-F_{w,t}^{[b]} over 8,752 tokens. The results show a strong positive linear correlation, with Pearson correlation coefficient r=0.4526r=0.4526. This empirical evidence suggests that the easily computable token-level CDF difference effectively captures the magnitude of the underlying gradient norm. In particular, tokens with large Δ\DeltaCDF tend to induce larger gradient norms, corresponding precisely to the tokens that Theorem 1 indicates should receive lower reliability weights—as these high-gradient, high-noise positions exert the most harmful influence on the gradient update under corruption.

Scatter plot of $\Delta$CDF versus token-level gradient discrepancy

Figure 2. Scatter plot of Δ\DeltaCDF versus token-level gradient discrepancy. Since ∥∇δt(z;θ)∥=∥∇θπθ(yw,t∣x)−∇θπθ(yl,t∣x)∥\lVert\nabla\delta_t(z;\theta)\rVert=\lVert\nabla_\theta\pi_\theta(y_w,t|x)-\nabla_\theta\pi_\theta(y_l,t|x)\rVert, we compute this quantity by taking the SFT gradients of the chosen and rejected responses separately and evaluating the norm of their difference. Across a large population of tokens, an implicit linear trend emerges between the two quantities (Pearson r=0.4526r=0.4526, n=8,752n=8{,}752), suggesting that Δ\DeltaCDF may serve as a lightweight proxy for the noise-induced gradient magnitude in Theorem 1, without requiring access to the unobservable statistics.

Motivated by this observation, we substitute the CDF-based proxy into the theoretically optimal weighting function and obtain a computationally efficient reweighting scheme. Specifically, define the token-level weight at position tt for the bb-th preference pair as

wt[b]=γexp⁡(−ρ(Fl,t[b]−Fw,t[b]))(10)w_t^{[b]}=\gamma\exp\left(-\rho\left(F_{l,t}^{[b]}-F_{w,t}^{[b]}\right)\right) \tag*{(10)}

where γ\gamma and ρ\rho are scaling hyperparameters. Importantly, we impose no normalization constraint to the weighting function. Finally, we incorporate these weights into (3)

L~ω(θ)=E(x[b],y~w[b],y~l[b])∼D~[f(β∑t=1Twt[b]⋅δt[b](θ))],with δt[b](θ)=log⁡πθ(y~w,t[b]∣x[b],y~w,<t[b])πref(y~l,t[b]∣x[b],y~l,<t[b])πθ(y~l,t[b]∣x[b],y~l,<t[b])πref(y~w,t[b]∣x[b],y~w,<t[b])(11)\widetilde{\mathcal{L}}_{\omega}(\theta)=\mathbb{E}_{(x^{[b]},\widetilde{y}_{w}^{[b]},\widetilde{y}_{l}^{[b]})\sim\widetilde{\mathcal{D}}}\left[f\left(\beta\sum_{t=1}^{T}w_{t}^{[b]}\cdot\delta_{t}^{[b]}(\theta)\right)\right],\quad\text{with }\delta_{t}^{[b]}(\theta)=\log\frac{\pi_{\theta}(\widetilde{y}_{w,t}^{[b]}|x^{[b]},\widetilde{y}_{w,<t}^{[b]})\pi_{\mathrm{ref}}(\widetilde{y}_{l,t}^{[b]}|x^{[b]},\widetilde{y}_{l,<t}^{[b]})}{\pi_{\theta}(\widetilde{y}_{l,t}^{[b]}|x^{[b]},\widetilde{y}_{l,<t}^{[b]})\pi_{\mathrm{ref}}(\widetilde{y}_{w,t}^{[b]}|x^{[b]},\widetilde{y}_{w,<t}^{[b]})} \tag*{(11)}

Token-level weighting as generalized sequence-level weighting. Existing sequence-level methods can be viewed as the special case where a single scalar weight is applied uniformly across all tokens, i.e., wt[b]=wseqw_t^{[b]}=w^{\mathrm{seq}}. In contrast, PORT allows wt[b]w_t^{[b]} to vary across positions, enabling fine-grained control over which parts of a sequence contribute more strongly to learning.

Importantly, we impose no normalization constraint on {wt[b]}\{w_t^{[b]}\}, and each token weight is determined independently based on its local CDF signal. This design allows the method to naturally adapt to different corruption patterns. When noise is localized, only a subset of token weights is affected, leaving the remaining tokens largely unchanged. When noise is distributed across the sequence, the directional CDF gap Fl,t[b]−Fw,t[b]F_{l,t}^{[b]}-F_{w,t}^{[b]} remains consistently large across positions, resulting in a coherent down- or up-weighting of the entire preference pair.

Overall, PORT achieves the bias-reduction properties characterized in Proposition 1 without additional backward-pass overhead. By selectively suppressing unreliable or high-noise tokens while preserving informative ones, it remains robust under corrupted preference supervision.

Experiments

Pretrained Models. We conduct experiments on LLaMA2-7B [23] and Qwen3-4B [28]. For both models, we initialize from the instruction-tuned checkpoints and use them as πref\pi_{\mathrm{ref}}. All models are trained with a linear warmup schedule and an effective batch size of 16 preference pairs. PORT’s hyperparameters are set to ρ=0.3\rho=0.3 and γ=1.5\gamma=1.5, selected by grid search on a held-out validation split. Full hyperparameter details are in Appendix § 10.2.

Datasets. We conduct experiments on three benchmarks spanning helpfulness and safety alignment: AlpacaFarm [6], Anthropic HH [1], and PKU-Beaver [11]. We provide a more comprehensive description of these datasets in Appendix § 10.2.

Baselines. We compare PORT against a comprehensive set of baselines spanning three categories of noise mitigation strategies: (i) Standard alignment methods include SFT, DPO [20], IPO [7], and SLiC-HF [32]; (ii) Sequence-level robust methods include cDPO [16], rDPO [4], β\beta-DPO [26], Dr.DPO [27], and wDPO [15]; (iii) Filtering-based methods include fDPO [17] and SamS [8]. All baselines are re-implemented under identical training conditions to ensure a fair comparison.

Evaluation metrics. We evaluate model performance using three complementary metrics. Gold Reward Model (RM) Score, GPT Win Rate (%), and Safety Rate (SR). We provide a more comprehensive description of these metrics in Appendix § 10.2.

PORT Consistently Outperforms on AlpacaFarm

To circumvent the high costs of human evaluation, following prior work, we utilize a large-scale human preference model [12] as the gold Reward model. We adjust the reward zero point so that the average reward of the initial LM is set to zero. The preference labels are flipped with probability 0.3 to simulate noisy annotations.

As shown in Table 1, PORT mostly achieves the best performance across both models and metrics. On LLaMA2-7B, PORT attains a Gold RM Score of 3.82 and a GPT Win Rate of 67.6%, outperforming the strongest sequence-level baseline (β\beta-DPO) by +0.40 in Gold RM Score and +4.4% in GPT Win Rate. On Qwen3-4B, PORT achieves a Gold RM Score of 2.98 and a GPT Win Rate of 53.6%, mostly outperforming the second-best method SamS. Note that the performance gap on Qwen3-4B is smaller than that on LLaMA2-7B. We attribute this difference to the fact that LLaMA2-7B is trained on comparatively older data and, due to its higher capacity, is more susceptible to memorizing noisy preference labels, making gradient-level noise mitigation more impactful. Nevertheless, PORT still achieves the highest Gold RM score on Qwen3-4B among all methods, confirming that token-level reweighting remains effective across architectures.

Table 1: Evaluation results on the AlpacaFarm dataset under two model settings. Bold indicates the best performance, and <u>underline</u> denotes the second-best performance. The results is reported under 3 different seed and as the format of mean ±\pm standard deviation.

MethodLLaMA2-7BQwen3-4B
Gold RM Score ↑\uparrowGPT Win Rate (%) ↑\uparrowGold RM Score ↑\uparrowGPT Win Rate (%) ↑\uparrow
SLiC-HF2.58 ±\pm 0.1155.8 ±\pm 1.22.08 ±\pm 0.0745.5 ±\pm 0.8
IPO2.75 ±\pm 0.0957.4 ±\pm 0.52.15 ±\pm 0.1346.7 ±\pm 0.7
DPO2.66 ±\pm 0.1256.8 ±\pm 0.92.18 ±\pm 0.1146.3 ±\pm 1.2
fDPO3.20 ±\pm 0.0660.7 ±\pm 0.82.54 ±\pm 0.0848.7 ±\pm 1.0
SamS3.68 ±\pm 0.1164.8 ±\pm 0.9<u>2.84 ±\pm 0.10</u>53.8 ±\pm 1.1
cDPO3.18 ±\pm 0.2560.5 ±\pm 0.32.38 ±\pm 0.1646.9 ±\pm 0.8
rDPO3.30 ±\pm 0.1362.1 ±\pm 0.92.60 ±\pm 0.1150.4 ±\pm 1.0
β\beta-DPO<u>3.42 ±\pm 0.08</u><u>63.2 ±\pm 1.0</u>2.73 ±\pm 0.0750.2 ±\pm 0.9
Dr.DPO3.39 ±\pm 0.1861.6 ±\pm 1.02.69 ±\pm 0.2352.5 ±\pm 0.8
PORT3.82 ±\pm 0.2367.6 ±\pm 1.82.98 ±\pm 0.16<u>53.6 ±\pm 1.4</u>

Table 1.

PORT Improves Safety Alignment on Anthropic HH

We conduct experiments on training with the Anthropic HH dataset, which consists of single-turn dialogues covering different topics in the safety domain. The evaluation metrics we use in our experiments include the safety rate on the PKU Beaver dataset [5, 10]. We will generate the response from the model trained with different baselines and then use the PKU Beaver evaluation model [5] to evaluate whether the response is safe or not. Table 2 shows the results of the evaluations based on the PKU Beaver dataset. Across all baselines, PORT achieves the highest PKU-Beaver SR of 0.55 and GPT Win Rate of 67.6%. Among sequence-level methods, β\beta-DPO performs strongest, followed by Dr.DPO. Filtering-based approaches achieve competitive results, though SamS relies on additional batch scheduling overhead. Notably, label-smoothing methods underperform relative to reweighting-based methods, suggesting that uniform label correction is less effective than adaptive sample-level adjustments for safety alignment.

Table 2: Results on Anthropic HH dataset (Harmless split). Bold denotes the best performance, <u>underline</u> indicates the second-best performance.

MethodPKU-Beaver SR ↑\uparrowGPT Win Rate (%) ↑\uparrow
SLiC-HF0.4156.0
IPO0.4357.6
DPO0.4256.8
fDPO0.4861.8
SamS<u>0.53</u><u>65.8</u>
cDPO0.4861.8
rDPO0.4962.6
β\beta-DPO0.5164.3
Dr.DPO0.5063.5
PORT0.5567.6

Table 2.

Robustness Across Noise Intensities and Types

To further investigate the robustness of our proposed PORT method, we conduct ablation studies focusing on two key dimensions: noise intensity and noise types. All experiments in this section are

conducted on the Anthropic HH dataset and evaluated using the PKU-Beaver SR metric. The detailed experimental setup and more comprehensive experimental results are shown in Appendix § 10.2.

Impact of Noise Intensity. We examine the performance of PORT and representative baselines across varying levels of mislabeled noise, with the flipping probability ranging over p∈{0.2,0.3,0.4}p \in\{0.2, 0.3, 0.4\}. Results are reported in Table 3. Several trends emerge from this analysis. First, the performance of standard DPO degrades consistently as noise increases, with SR dropping from 0.45 at p=0.2p=0.2 to 0.38 at p=0.4p=0.4, confirming that vanilla preference optimization is highly sensitive to label corruption. Moreover, PORT maintains consistently strong performance across all settings, achieving SR scores of 0.56, 0.55, and 0.53 at p=0.2p=0.2, p=0.3p=0.3, and p=0.4p=0.4, respectively. Notably, the performance gap between PORT and the strongest sequence-level baseline widens as noise increases — from 0.03 at p=0.2p=0.2 to 0.06 at p=0.4p=0.4 — demonstrating that token-level reweighting becomes increasingly advantageous under higher noise regimes.

Table 3: Ablation of Noise Intensity on Anthropic HH (evaluated by PKU-Beaver SR ↑\uparrow). Bold denotes the best performance, <u>underline</u> indicates the second-best performance.

Methodp=0.2p = 0.2p=0.3p = 0.3p=0.4p = 0.4
SR ↑\uparrowGPT Win Rate (%) ↑\uparrowSR ↑\uparrowGPT Win Rate (%) ↑\uparrowSR ↑\uparrowGPT Win Rate (%) ↑\uparrow
DPO0.4559.30.4256.80.3853.4
SamS0.5365.9<u>0.53</u><u>65.8</u>0.4660.1
β\beta-DPO<u>0.54</u><u>66.7</u>0.5164.3<u>0.47</u><u>60.9</u>
PORT0.5668.50.5567.60.5365.9

Table 3.

Impact of Noise Type. We further compare model resilience under two qualitatively distinct noise distributions at a fixed noise level of p=0.4p=0.4. Results are reported in Table 4. (i) ϵ\epsilon-mislabeled noise simulates stochastic annotation errors by flipping each preference label independently with a fixed probability η=0.4\eta=0.4, regardless of the content of the preference pair. PORT achieves the best performance with a Safety Rate of 0.53 and GPT win rate of 65.9%, outperforming the strongest baseline Dr.DPO by +0.05 SR and +4.1% GPT win rate, demonstrating that token-level down-weighting of anomalously high-probability rejected tokens provides substantial gains when noise is uniformly distributed. (ii) α\alpha-uncertain noise simulates systematic annotation errors where label flip probabilities are determined by the inward margin between responses under a Bradley-Terry model. PORT continues to achieve the best results with SR of 0.49 and the GPT WR of 62.5%. The consistent margin across both noise types further validates the theoretical guarantee from Proposition 1: our CDF-based weighting is guaranteed to reduce gradient bias when the human noise distribution is softer than the model’s implicit preference confidence.

Methodϵ\epsilon-mislabeled noiseα\alpha-uncertain noise
SR ↑\uparrowGPT Win Rate (%) ↑\uparrowSR ↑\uparrowGPT Win Rate (%) ↑\uparrow
DPO0.3853.40.3853.5
SamS0.4660.10.4357.0
Dr.DPO0.4861.80.4458.5
PORT0.5365.90.4962.5

Table 4. Impact of Noise Type on Model Performance (evaluated by PKU-Beaver SR ↑\uparrow) on Anthropic HH with p=0.4p = 0.4 noise level. Bold denotes the best performance, underline indicates the second-best performance. PORT achieves consistent gains of ∼\sim4% WR and 0.05 SR over the strongest baseline across both noise types.

Impact of Batch size. PORT estimates the empirical CDF from the reference set R\mathcal{R} of all non-padding chosen and rejected tokens in the current batch, so its weights depend on the batch size BB. We vary B∈{1,4,16}B \in\{1,4,16\} using Qwen2.5-0.5B trained on Anthropic HH and evaluated on PKU-Beaver and results are shown in Table 6. The GPT Win Rate drops to 39.1% at B=1B=1, where R\mathcal{R} contains tokens from a single preference pair. It recovers to 42.8% at B=4B=4, close to the 44.3% obtained with B=16B=16, the default used throughout our experiments. Pooling tokens across a few pairs is therefore sufficient for a reliable rank estimate, and PORT does not require large batches.

Batch Size (BB)GPT Win Rate (%)
139.1
442.8
1644.3

Table 6. Effect of batch size on GPT win rate using Qwen2.5-0.5B. The reference set R\mathcal{R} pools all non-padding chosen and rejected token logits across the batch. Higher is better.

PORT Concentrates Gradients on Reliable Tokens

To investigate the impact of our proposed weighting mechanism on learning dynamics, we tracked the per-sample training loss over 30 gradient steps for two models: standard DPO and PORT. Both models were trained using Qwen2.5-0.5B [19] as the base policy on a subset of 128 samples drawn from the HH-RLHF dataset. To group pairs by how reliable their preference is, we use a diagnostic score that is distinct from the training-time gap Δt[b]\Delta_t^{[b]}. For each pair zz, we evaluate the initial policy πθ0\pi_{\theta_0} once, before any preference training, on the ground-truth chosen and rejected responses (yw,yl)(y_w, y_l), and compute the initial separation score m0(z):=Fˉw(z)−Fˉl(z)m_0(z) := \bar{F}_w(z) - \bar{F}_l(z), where Fˉw(z)\bar{F}_w(z) and Fˉl(z)\bar{F}_l(z) are the empirical-CDF ranks averaged over all non-padding tokens of ywy_w and yly_l, respectively. A large m0(z)m_0(z) means that πθ0\pi_{\theta_0} already ranks the chosen response well above the rejected one; a small m0(z)m_0(z) means the two responses receive similar ranks, i.e., the pair is ambiguous to the model. We split the 128 pairs at the median of m0(z)m_0(z) into a High-separation and a Low-separation group of 64 pairs each, and keep this assignment fixed for all 30 steps. The two groups differ markedly in separation, with m0=0.367±0.167m_0 = 0.367 \pm0.167 (range [0.151,0.801][0.151, 0.801]) for High-separation pairs versus 0.064±0.0390.064 \pm0.039 (range [0.000,0.146][0.000, 0.146]) for Low-separation pairs. As an independent check, reward model agrees with the ground-truth ordering on 82.8% of High-separation pairs but only 59.4% of Low-separation pairs, only modestly above chance. Under the α\alpha-uncertain noise model, where the flip probability η(z)\eta(z) decreases with the reward margin, High-separation pairs are thus the ones whose labels are most likely to remain correct, while Low-separation pairs are the most prone to flips. Ground-truth labels are used only for this post-hoc diagnostic. The High Margin group consists of samples with well-separated chosen and rejected reward distributions, signifying reliable preference annotations. Conversely, the Low Margin group comprises samples with overlapping reward distributions, indicating noisy or ambiguous labels. As illustrated in Figure 3a, both groups exhibit nearly identical loss trajectories under standard DPO. The model treats all samples uniformly regardless of their inherent label quality, converging to a similarly low loss by step 30. In contrast, Figure 3b reveals a distinct divergence under PORT: High Margin samples achieve a substantially lower loss at a much faster rate, while Low Margin samples lag significantly behind throughout the training process. This divergence empirically validates that PORT’s Δ\DeltaCDF-based weighting successfully concentrates the gradient signal on reliable, well-separated pairs, enabling the model to learn preference distinctions more efficiently from trustworthy samples while safely marginalizing the influence of noisy annotations.

Figure 3. Comparison of per-sample training loss trajectories between standard DPO and our proposed PORT method across 30 gradient steps. The training samples are partitioned into High Δ\DeltaCDF (more reliable) and Low Δ\DeltaCDF (more noisy) groups.

Conclusion

In this work, we presented PORT (Preference Optimization via Robust Token-level reweighting), a fine-grained framework for robust LLM alignment under noisy preference annotations. Unlike existing sequence-level methods or approaches relying on external reward models, PORT performs token-level reweighting via the empirical CDF of token logits — a lightweight, forward-pass-only proxy requiring no additional backward-pass overhead. Theoretically, we derived the optimal weighting function by minimizing an upper bound on the noise-induced gradient bias, and established sufficient conditions under which PORT is guaranteed to strictly reduce this bias under both ϵ\epsilon-mislabeled and α\alpha-uncertain noise models. Empirically, we demonstrated through extensive experiments on AlpacaFarm, Anthropic HH, and PKU-Beaver that PORT consistently outperforms a broad range of sequence-level baselines across diverse noise settings and model architectures.

Acknowledgments and Disclosure of Funding

This material is based upon work supported by the U.S. National Science Foundation under Grant Nos. IIS-2202699, IIS-2416895, IIS-2301599, CMMI-2301601, DMS-2529302, and OIA-2616279. The authors declare no competing interests.

Related Work

Preference-based Alignment for Large Language Models

Aligning large language models with human preferences has become a central problem in modern NLP [31]. Reinforcement Learning from Human Feedback (RLHF) [18] is one of the most widely used approaches, where a reward model is trained from human annotations and then used to guide policy optimization. Despite its success, RLHF requires training an additional reward model and performing reinforcement learning, which makes the pipeline complex and unstable.

Direct Preference Optimization (DPO) [20] has recently been proposed as a simpler alternative that directly optimizes the policy using pairwise preference data without explicitly training a reward model. DPO formulates preference learning using a Bradley–Terry model [3] and optimizes a supervised objective that implicitly regularizes the policy toward a reference model. Subsequent works [7, 15] have extended DPO with different objectives, regularization strategies, and theoretical analyses, demonstrating competitive performance with improved stability. While these methods have significantly advanced alignment quality, they share a common assumption that is rarely satisfied in practice: the availability of clean, noise-free preference annotation [25].

Robust Preference Optimization under Noisy Feedback

Annotating large-scale preference datasets remains costly and labor-intensive, inevitably compromising annotation quality. Empirical studies suggest that 20%-40% of preference annotations in commonly used datasets contain labeling errors [13], underscoring the practical severity of this problem. Researchers have increasingly recognized the importance of noise robustness in preference alignment, leading to a growing body of work on robust DPO variants.

One line of work addresses noise through label smoothing. Conservative-DPO (cDPO) [16] incorporates a conservative smoothing term that down-weights the influence of potentially flipped labels, providing provable robustness guarantees under bounded noise rates. Robust Preference Optimization (ROPO) [13] extends this with an adaptive smoothing mechanism that adjusts the correction strength based on estimated noise levels. However, both methods apply a uniform correction across all tokens in a response, failing to account for token-level variation in annotation quality.

A second line of work addresses noise through data filtering. Filtered DPO [17] employs a pretrained reward model to score each preference pair and filters out samples whose reward margin falls below a threshold. Adaptive batch-Wise SAMple Scheduling (SAMS) [8] utilizes a batch-wise training sample scheduling method to dynamically and adaptively schedule training samples. While effective, these approaches reintroduce the computational overhead and misgeneralization risks of reward-model-dependent pipelines, and may discard entire pairs even when only a subset of tokens are corrupted.

A third line addresses noise through reweighting, Distributionally Robustifying DPO (Dr.DPO) [27] assigns pair-level weights based on estimated noise rates to reduce the influence of corrupted samples. Winsorized Direct Preference Optimization (wDPO) [15] applies Winsorized reweighting at the sequence level to limit the influence of outlier preference pairs. Im and Li[9] provide a theoretical analysis of how preference optimization generalizes under noisy feedback, characterizing the conditions under which noise degrades alignment quality. While these reweighting approaches improve robustness, they remain sequence-level methods that assign a single scalar weight to each response pair, ignoring within-sequence token-level variation.

Token-level Modeling and Fine-grained Alignment

A parallel line of work has explored token-level modeling as a finer-grained alternative to sequence-level preference optimization. TDPO [30] proposes a token-level DPO formulation that decomposes the preference signal across individual tokens, allowing the objective to reflect each token’s contribution to the overall preference. Most closely related to our work, TIS-DPO [14] introduces token-level importance sampling weights estimated from an external reward model, achieving fine-grained noise mitigation at the token level. However, TIS-DPO inherits the limitations of reward-model-dependent approaches: it requires an additional model for weight estimation, introducing both computational overhead and misgeneralization risk. LLMDoctor [22] proposes flow-guided token-level preference optimization for test-time alignment, while TIDPO [29] uses token importance scores derived from attention weights to guide the preference optimization objective.

Motivated by these observations, we study preference optimization under noisy feedback from a token-level perspective. Instead of assigning a single weight to each pair, we design a token-level weighting mechanism that estimates the relative quality of each token using the empirical distribution of logits within the batch. This enables more accurate identification of corrupted regions and leads to improved robustness compared to sequence-level methods.

Theoretical Proof

Gradient Bias

Given the DPO/IPO loss in the normal setting:

ℓ(x,yw,yl;θ)=L∗(θ)=E(x,yw,yl)∼D[f(σ(rϕ(x,yw)−rϕ(x,yl)))]=E(x,yw,yl)∼D[f(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))](12)\begin{aligned} \ell(x,y_w,y_l;\theta)=\mathcal{L}^{*}(\theta)=\mathbb{E}_{(x,y_w,y_l)\sim D}\left[f\left(\sigma\left(r_{\phi}(x,y_w)-r_{\phi}(x,y_l)\right)\right)\right] \\ &=\mathbb{E}_{(x,y_w,y_l)\sim D}\left[f\left(\beta\log\frac{\pi_{\theta}(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)}-\beta\log\frac{\pi_{\theta}(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)}\right)\right] \tag*{(12)} \end{aligned}

In the case of DPO, f(x)=−log⁡σ(x)f(x)=-\log\sigma(x), and in the case of IPO, f(x)=(x−12)2f(x)=\left(x-\frac{1}{2}\right)^2. So Eq. (12) can be rewritten as:

ℓ(z;θ)=E(z)∼Df(βh(z;θ))(13)\ell(z;\theta)=\mathbb{E}_{(z)\sim D}f(\beta h(z;\theta)) \tag*{(13)}

In the following proof, we abbreviate h(θ;x,yw,yl)h(\theta;x,y_w,y_l) as h(θ)h(\theta) when x,yw,ylx,y_w,y_l are implicitly understood. Consider the standard objective and a weighted objective in a setting with instance-dependent noise η(x,y,y′)\eta(x,y,y'):

L(θ)=E[(1−η(z))ℓ(z;θ)+η(z)ℓ(−z;θ)]Lw(θ)=E[((1−η(z))ℓw(z;θ)+η(z)ℓw(−z;θ))](14)\begin{aligned} \mathcal{L}(\theta)&=\mathbb{E}\left[(1-\eta(z))\ell(z;\theta)+\eta(z)\ell(-z;\theta)\right]\\ \mathcal{L}_w(\theta)&=\mathbb{E}\left[((1-\eta(z))\ell_w(z;\theta)+\eta(z)\ell_w(-z;\theta))\right] \tag*{(14)} \end{aligned}

Proof of Definition 1 (Gradient Bias). The gradient of standard DPO under the noisy setting is:

g(θ)=∇L(θ)=E[∇ℓ(z;θ)]g~(θ)=∇L~(θ)=E[(1−η(z))∇ℓ(z;θ)+η(z)∇ℓ(−z;θ)]B=Δg=g~(θ)−g(θ)=E[η(z)(∇ℓ(−z;θ)−∇ℓ(z;θ))](15)\begin{aligned} g(\theta)&=\nabla\mathcal{L}(\theta)=\mathbb{E}[\nabla\ell(z;\theta)]\\ \tilde{g}(\theta)&=\nabla\tilde{\mathcal{L}}(\theta)=\mathbb{E}[(1-\eta(z))\nabla\ell(z;\theta)+\eta(z)\nabla\ell(-z;\theta)]\\ B&=\Delta g=\tilde{g}(\theta)-g(\theta)=\mathbb{E}[\eta(z)(\nabla\ell(-z;\theta)-\nabla\ell(z;\theta))] \tag*{(15)} \end{aligned}

Proof of Definition 2 (Weighted Gradient Bias). For the weighted objective, the gradients are:

gw(θ)=∇Lw(θ)=E[∇ℓw(z;θ)]g~w(θ)=∇L~w(θ)=E[((1−η(z))∇ℓw(z;θ)+η(z)∇ℓw(−z;θ))]Δgw=g~w(θ)−gw(θ)=E[η(z)(∇ℓw(−z;θ)−∇ℓw(z;θ))](16)\begin{aligned} g_w(\theta)&=\nabla\mathcal{L}_w(\theta)=\mathbb{E}[\nabla\ell_w(z;\theta)]\\ \tilde{g}_w(\theta)&=\nabla\tilde{\mathcal{L}}_w(\theta)=\mathbb{E}[((1-\eta(z))\nabla\ell_w(z;\theta)+\eta(z)\nabla\ell_w(-z;\theta))]\\ \Delta g_w&=\tilde{g}_w(\theta)-g_w(\theta)=\mathbb{E}[\eta(z)(\nabla\ell_w(-z;\theta)-\nabla\ell_w(z;\theta))] \tag*{(16)} \end{aligned}

We aim to minimize the total bias Bw=gw(θ)−g(θ)B_w=g_w(\theta)-g(\theta):

Bw=[g~w(θ)−gw(θ)]+[gw(θ)−g(θ)]=Δgw+Rb(17)B_w=[\tilde{g}_w(\theta)-g_w(\theta)]+[g_w(\theta)-g(\theta)]=\Delta g_w+R_b \tag*{(17)}

where Rb=gw(θ)−g(θ)R_b=g_w(\theta)-g(\theta) represents the distortion introduced by reweighting clean data.

Optimal Solution of the Weight Function

We now focus on the term ∇ℓ(−z;θ)−∇ℓ(z;θ)\nabla\ell(-z;\theta)-\nabla\ell(z;\theta). According to the chain rule:

∇ℓ(−z;θ)−∇ℓ(z;θ)=∇θ(f(−β∑tδt(z;θ)))−∇θ(f(β∑tδt(z;θ)))=β(f′(−β∑tδt(z;θ))×(−∑t∇θδt(z;θ))−f′(β∑tδt(z;θ))×(∇θ∑tδt(z;θ)))=−β∇θ∑tδt(z;θ)×(f′(−β∑tδt(z;θ))+f′(β∑tδt(z;θ)))(18)\begin{aligned} \nabla\ell(-z;\theta)-\nabla\ell(z;\theta) &=\nabla_{\theta}\left(f\left(-\beta\sum_t\delta_t(z;\theta)\right)\right)-\nabla_{\theta}\left(f\left(\beta\sum_t\delta_t(z;\theta)\right)\right)\\ &=\beta\left(f'\left(-\beta\sum_t\delta_t(z;\theta)\right)\times\left(-\sum_t\nabla_{\theta}\delta_t(z;\theta)\right)\right.\\ &\qquad\left.-f'\left(\beta\sum_t\delta_t(z;\theta)\right)\times\left(\nabla_{\theta}\sum_t\delta_t(z;\theta)\right)\right)\\ &=-\beta\nabla_{\theta}\sum_t\delta_t(z;\theta)\times\left(f'\left(-\beta\sum_t\delta_t(z;\theta)\right)+f'\left(\beta\sum_t\delta_t(z;\theta)\right)\right) \tag*{(18)} \end{aligned}

In standard preference learning frameworks, we prove that f′(−β∑tδt(z;θ))+f′(β∑tδt(z;θ))f'\left(-\beta\sum_t\delta_t(z;\theta)\right)+f'\left(\beta\sum_t\delta_t(z;\theta)\right) evaluates to a constant:

  • i. DPO: If f(x)=−log⁡σ(x)f(x)=-\log\sigma(x), then f′(x)+f′(−x)=−1f'(x)+f'(-x)=-1. Therefore, (∇ℓ(−z;θ)−∇ℓ(z;θ))=β∇θ∑tδt(z;θ)(\nabla\ell(-z;\theta)-\nabla\ell(z;\theta))=\beta\nabla_{\theta}\sum_t\delta_t(z;\theta).

  • ii. IPO: If f(x)=(x−12)2f(x)=\left(x-\frac{1}{2}\right)^2, then f′(x)=2(x−12)f'(x)=2\left(x-\frac{1}{2}\right) and f′(x)+f′(−x)=−2f'(x)+f'(-x)=-2. Therefore, (∇ℓ(−z;θ)−∇ℓ(z;θ))=2β∇θ∑tδt(z;θ)∝∇θ∑tδt(z;θ)(\nabla\ell(-z;\theta)-\nabla\ell(z;\theta))=2\beta\nabla_{\theta}\sum_t\delta_t(z;\theta)\propto\nabla_{\theta}\sum_t\delta_t(z;\theta).

  • iii. SliC: If f(x)=max⁡(0,1−x)f(x)=\max(0,1-x), then f′(x)+f′(−x)=−2f'(x)+f'(-x)=-2 when x∈[−1,1]x\in[-1,1], and −1-1 otherwise. Therefore, (∇ℓ(−z;θ)−∇ℓ(z;θ))∝∇θ∑tδt(z;θ)(\nabla\ell(-z;\theta)-\nabla\ell(z;\theta))\propto\nabla_{\theta}\sum_t\delta_t(z;\theta).

Similarly, for ∇ℓw(−z;θ)−∇ℓw(z)\nabla\ell_w(-z;\theta)-\nabla\ell_w(z), we can get that

∇ℓw(−z;θ)−∇ℓw(z;θ)=∇θ(f(−β∑tw(z,t)δt(z;θ)))−∇θ(f(β∑tw(z,t)δt(z;θ)))=β(f′(−β∑tw(z,t)δt(z;θ))×(−∇θ∑tw(z,t)δt(z;θ))−f′(β∑tw(z,t)δt(z;θ))×(∇θ∑tw(z,t)δt(z;θ)))=−β∇θ∑tw(z,t)δt(z;θ)×(f′(−β∑tw(z,t)δt(z;θ))+f′(β∑tw(z,t)δt(z;θ)))(19)\begin{aligned} \nabla\ell_w(-z;\theta)-\nabla\ell_w(z;\theta) &=\nabla_{\theta}\left(f\left(-\beta\sum_t w(z,t)\delta_t(z;\theta)\right)\right)-\nabla_{\theta}\left(f\left(\beta\sum_t w(z,t)\delta_t(z;\theta)\right)\right)\\ &=\beta\left(f'\left(-\beta\sum_t w(z,t)\delta_t(z;\theta)\right)\times\left(-\nabla_{\theta}\sum_t w(z,t)\delta_t(z;\theta)\right)\right.\\ &\qquad\left.-f'\left(\beta\sum_t w(z,t)\delta_t(z;\theta)\right)\times\left(\nabla_{\theta}\sum_t w(z,t)\delta_t(z;\theta)\right)\right)\\ &=-\beta\nabla_{\theta}\sum_t w(z,t)\delta_t(z;\theta)\times\left(f'\left(-\beta\sum_t w(z,t)\delta_t(z;\theta)\right)\right.\\ &\qquad\left.+f'\left(\beta\sum_t w(z,t)\delta_t(z;\theta)\right)\right) \tag*{(19)} \end{aligned}

And assume f′(hw(z;θ))f′(h(z;θ))≤f′(−wmax⁡H)f′(H):=K\frac{f'(h_w(z;\theta))}{f'(h(z;\theta))}\leq\frac{f'(-w_{\max}H)}{f'(H)}:=K, so we can express the noise-induced bias as:

Δgw=E[∥η(z)∑t=1Tf′(∑tw(z,t)δt(z;θ))f′(∑tδt(z;θ))wt∇θδt(z;θ)∥]≤KE[∥η(z)∑t=1Twt∇θδt(z;θ)∥]≤K∑t=1TE[η(z)w(z,t)∥∇θδt(θ)∥](20)\begin{aligned} \Delta g_w &=\mathbb{E}\left[\left\|\eta(z)\sum_{t=1}^{T}\frac{f'\left(\sum_t w(z,t)\delta_t(z;\theta)\right)}{f'\left(\sum_t\delta_t(z;\theta)\right)}w_t\nabla_{\theta}\delta_t(z;\theta)\right\|\right]\\ &\leq K\mathbb{E}\left[\left\|\eta(z)\sum_{t=1}^{T}w_t\nabla_{\theta}\delta_t(z;\theta)\right\|\right]\\ &\leq K\sum_{t=1}^{T}\mathbb{E}\left[\eta(z)w(z,t)\left\|\nabla_{\theta}\delta_t(\theta)\right\|\right] \tag*{(20)} \end{aligned}

By Jensen’s inequality, ∥E(wt(z;θ)η(z)∇δt(z;θ))∥≤E(wt(z;θ)η(z)∥∇δt(z;θ)∥)\left\|\mathbb{E}(w_t(z;\theta)\eta(z)\nabla\delta_t(z;\theta))\right\|\leq\mathbb{E}(w_t(z;\theta)\eta(z)\|\nabla\delta_t(z;\theta)\|). Minimizing this upper bound effectively controls Δgw\Delta g_w.

For the reweighting bias Rb=E(∇ℓw(z;θ)−∇ℓ(z;θ))R_b = \mathbb{E}(\nabla\ell_w(z;\theta)-\nabla\ell(z;\theta)), assuming the gradient norm is bounded by M=sup⁡z∥∇ℓ(z;θ)∥/βM = \sup_z \lVert\nabla\ell(z;\theta)\rVert/\beta, we have:

∥Rb∥=∥E(∇ℓw(z;θ)−∇ℓ(z;θ))∥≤∥f′(∑tw(z,t)δt(z))∑t(wt−1)∇δt+(f′(∑tw(z,t)δt(z))−f′(∑tδt(z)))∑t∇δt∥≤L0∑t∣wt−1∣∥∇δt∥+(f′(∑tw(z,t)δt(z))−f′(∑tδt(z)))∑t∇δt≤L0∑t∣wt−1∣∥∇δt∥+L1∥∑tw(z,t)δt(z)−∑tδt(z)∥∥∑t∇δt∥≤(L0+L1TD)∑tEp[∣wt−1∣](21)\begin{aligned} \lVert R_b\rVert&= \lVert\mathbb{E}(\nabla\ell_w(z;\theta)-\nabla\ell(z;\theta))\rVert\\ &\le\left\lVert f'\left(\sum_t w(z,t)\delta_t(z)\right)\sum_t(w_t-1)\nabla\delta_t+\left(f'\left(\sum_t w(z,t)\delta_t(z)\right)-f'\left(\sum_t\delta_t(z)\right)\right)\sum_t\nabla\delta_t\right\rVert\\ &\le L_0\sum_t\lvert w_t-1\rvert\lVert\nabla\delta_t\rVert+\left(f'\left(\sum_t w(z,t)\delta_t(z)\right)-f'\left(\sum_t\delta_t(z)\right)\right)\sum_t\nabla\delta_t \\ &\le L_0\sum_t\lvert w_t-1\rvert\lVert\nabla\delta_t\rVert+L_1\left\lVert\sum_t w(z,t)\delta_t(z)-\sum_t\delta_t(z)\right\rVert\left\lVert\sum_t\nabla\delta_t\right\rVert\\ &\le(L_0+L_1TD)\sum_t\mathbb{E}_p[\lvert w_t-1\rvert] \tag*{(21)} \end{aligned}

Here, L0L_0 is the upper bound of the gradient norm of ∇δt\nabla\delta_t, L1L_1 is the Lipschitz constant of f′f', DD is the upper bound of δt\delta_t and TT is the upper bound of ∑t∇δt\sum_t\nabla\delta_t.

Thus, minimizing DKL(pw∥p)D_{\mathrm{KL}}(p_w\mathbin\Vert p) directly controls the upper bound of ∥Rb∥\lVert R_b\rVert. we introduce the following lemma:

Lemma 1. Let X∼PX\sim P be a random variable. Let w,g:R→Rw,g:\mathbb{R}\to\mathbb{R} be measurable functions such that w(X)w(X) is integrable and EX∼P[w(X)]=1\mathbb{E}_{X\sim P}[w(X)]=1. The regularized optimization problem:

min⁡w(βE(w(z;θ)ϕ(z;θ))+βλDKL(pw∥p))(22)\min_w\left(\beta\mathbb{E}(w(z;\theta)\phi(z;\theta))+\frac{\beta}{\lambda}D_{\mathrm{KL}}(p_w\mathbin\Vert p)\right) \tag*{(22)}

has the non-trivial optimal solution w∗∝exp⁡(−λϕ(z;θ))w^*\propto\exp(-\lambda\phi(z;\theta)).

Proof of Theorem 1. Using Lagrange multipliers, we construct the functional:

F=E(w(z;θ)ϕ(z;θ))+1λE(w(z;θ)log⁡(w(z;θ)))=E(w(z;θ)(ϕ(z;θ)+1λlog⁡(w(z;θ))))(23)\begin{aligned} F&=\mathbb{E}(w(z;\theta)\phi(z;\theta))+\frac{1}{\lambda}\mathbb{E}(w(z;\theta)\log(w(z;\theta))) \\ &=\mathbb{E}\left(w(z;\theta)\left(\phi(z;\theta)+\frac{1}{\lambda}\log(w(z;\theta))\right)\right) \tag*{(23)} \end{aligned}

Assume w∗(z)=exp⁡(−ϕ(z;θ)λ)Zw^*(z)=\frac{\exp(-\phi(z;\theta)\lambda)}{Z} and Z=E[exp⁡(−λϕ(z;θ))]Z=\mathbb{E}[\exp(-\lambda\phi(z;\theta))]. For any zz, ϕ(z;θ)+1λlog⁡(w∗)=−1λlog⁡Z\phi(z;\theta)+\frac{1}{\lambda}\log(w^*)=-\frac{1}{\lambda}\log Z. Therefore,

E[w(z;θ)(ϕ(z;θ)+1λlog⁡(w(z;θ)))]=E[w(z;θ)(ϕ(z;θ)+1λlog⁡(w(z;θ))−(ϕ+1λlog⁡w∗(z;θ)))]+E[w(z;θ)⋅(−1/λ)log⁡Z]=1λE[w(z;θ)log⁡(w(z;θ)w∗(z;θ))]−1λlog⁡Z⋅E[w(z;θ)]=1λ(DKL(pw∥pw∗)−log⁡Z)≥−1λlog⁡Z(24)\begin{aligned} \mathbb{E}\left[w(z;\theta)\left(\phi(z;\theta)+\frac{1}{\lambda}\log(w(z;\theta))\right)\right] &=\mathbb{E}\left[w(z;\theta)\left(\phi(z;\theta)+\frac{1}{\lambda}\log(w(z;\theta))-\left(\phi+\frac{1}{\lambda}\log w^*(z;\theta)\right)\right)\right] \\ &\quad+\mathbb{E}[w(z;\theta)\cdot(-1/\lambda)\log Z] \\ &=\frac{1}{\lambda}\mathbb{E}\left[w(z;\theta)\log\left(\frac{w(z;\theta)}{w^*(z;\theta)}\right)\right]-\frac{1}{\lambda}\log Z\cdot\mathbb{E}[w(z;\theta)] \\ &=\frac{1}{\lambda}\left(D_{\mathrm{KL}}(p_w\mathbin\Vert p_{w^*})-\log Z\right) \\ &\ge-\frac{1}{\lambda}\log Z \tag*{(24)} \end{aligned}

Since ZZ is a constant and if and only if w=w∗w=w^*, DKL(pw∥pw∗)=0D_{\mathrm{KL}}(p_w\mathbin\Vert p_{w^*})=0.

Analysis of the Effective Regime

We now analyze when the optimal weight wλ(x)∝e−λϕ(x)w_\lambda(x)\propto e^{-\lambda\phi(x)} effectively reduces the total bias.

Proof of Proposition 1. Define ϕ(z;θ):=βη(z)∥∇δt(z;θ)∥\phi(z;\theta):=\beta\eta(z)\lVert\nabla\delta_t(z;\theta)\rVert and the normalized weight:

wλ(z)=exp⁡(−λϕ(z;θ))Z(λ),Z(λ):=Ez[exp⁡(−λϕ(z;θ))].(25)w_\lambda(z)=\frac{\exp(-\lambda\phi(z;\theta))}{Z(\lambda)},\qquad Z(\lambda):=\mathbb{E}_z[\exp(-\lambda\phi(z;\theta))]. \tag*{(25)}

Since Bw(0)=ΔgB_w(0)=\Delta g (at λ=0\lambda=0 the weights are uniform and Rb(0)=0R_b(0)=0), a sufficient condition for ∥Bw(λ)∥<∥B∥\lVert B_w(\lambda)\rVert<\lVert B\rVert at small λ>0\lambda>0 is:

ddλ12∥Bw(λ)∥2∣λ=0=⟨Δg,ddλBw(λ)∣λ=0⟩<0.(26)\left.\frac{d}{d\lambda}\frac{1}{2}\lVert B_w(\lambda)\rVert^2\right|_{\lambda=0}=\left\langle\Delta g,\left.\frac{d}{d\lambda}B_w(\lambda)\right|_{\lambda=0}\right\rangle<0. \tag*{(26)}

Differentiating wλ(z)w_\lambda(z) with respect to λ\lambda:

dwλ(z)dλ=−ϕ(z)Z(λ)−exp⁡(−λϕ(z))Z′(λ)Z(λ)2.(27)\frac{dw_\lambda(z)}{d\lambda}=\frac{-\phi(z)Z(\lambda)-\exp(-\lambda\phi(z))Z'(\lambda)}{Z(\lambda)^2}. \tag*{(27)}

At λ=0\lambda=0: Z(0)=1Z(0)=1 and Z′(0)=E[−ϕ(z)]=−E[ϕ(z)]Z'(0)=\mathbb{E}[-\phi(z)]=-\mathbb{E}[\phi(z)]. Therefore:

dwλ(z)dλ∣λ=0=−ϕ(z)+E[ϕ(z)]=:δw(z),(28)\left.\frac{dw_\lambda(z)}{d\lambda}\right|_{\lambda=0}=-\phi(z)+\mathbb{E}[\phi(z)]=:\delta w(z), \tag*{(28)}

which satisfies E[δw(z)]=0\mathbb{E}[\delta w(z)]=0, i.e. the perturbation preserves normalization. We have Δgw(λ)=βE[wλ(z,t)η(z)∇δt(z;θ)]\Delta g_w(\lambda)=\beta\mathbb{E}[w_\lambda(z,t)\eta(z)\nabla\delta_t(z;\theta)]. Its derivative at λ=0\lambda=0 is:

dΔgwdλ∣λ=0=βE[δw(z)⋅η(z)∇δt(z;θ)]=β(E[ϕ]⋅E[η∇δt]−E[ϕ(z)⋅η(z)∇δt(z;θ)])=−βCov⁡(η(z)∥∇δt(z;θ)∥,η(z)∇δt(z;θ)).\begin{aligned} \left.\frac{d\Delta g_w}{d\lambda}\right|_{\lambda=0}=\beta\mathbb{E}[\delta w(z)\cdot\eta(z)\nabla\delta_t(z;\theta)] \\ &=\beta\left(\mathbb{E}[\phi]\cdot\mathbb{E}[\eta\nabla\delta_t]-\mathbb{E}[\phi(z)\cdot\eta(z)\nabla\delta_t(z;\theta)]\right) \\ &=-\beta\operatorname{Cov}(\eta(z)\lVert\nabla\delta_t(z;\theta)\rVert,\eta(z)\nabla\delta_t(z;\theta)). \end{aligned}

Similarly, since ∇ℓ(z;θ)=f′(βδt(z;θ))∇δt(z;θ)\nabla\ell(z;\theta)=f'(\beta\delta_t(z;\theta))\nabla\delta_t(z;\theta), the derivative of the reweighting bias Rb(λ)=E[(wλ(z)−1)∇ℓ(z;θ)]R_b(\lambda)=\mathbb{E}[(w_\lambda(z)-1)\nabla\ell(z;\theta)] is:

dRbdλ∣λ=0=βE[δw(z)⋅f′(β∑δt(z;θ))∑∇δt(z;θ)]=−βCov⁡(η(z)∥∇∑δt(z;θ)∥,f′(β∑δt(z;θ))∑∇δt(z;θ)).\begin{aligned} \left.\frac{dR_b}{d\lambda}\right|_{\lambda=0}=\beta\mathbb{E}[\delta w(z)\cdot f'(\beta\sum\delta_t(z;\theta))\sum\nabla\delta_t(z;\theta)] \\ &=-\beta\operatorname{Cov}\left(\eta(z)\lVert\nabla\sum\delta_t(z;\theta)\rVert,f'(\beta\sum\delta_t(z;\theta))\sum\nabla\delta_t(z;\theta)\right). \end{aligned}

Combining the two terms:

ddλBw(λ)∣λ=0=−βCov⁡(η(z)∥∑∇δt(z;θ)∥,(η(z)+f′(β∑δt(z;θ)))∑∇δt(z;θ)).(29)\left.\frac{d}{d\lambda}B_w(\lambda)\right|_{\lambda=0}=-\beta\operatorname{Cov}\left(\eta(z)\left\lVert\sum\nabla\delta_t(z;\theta)\right\rVert,\left(\eta(z)+f'(\beta\sum\delta_t(z;\theta))\right)\sum\nabla\delta_t(z;\theta)\right). \tag*{(29)}

Substituting (29) into (26), the condition (⋆)<0(\star)<0 is equivalent to:

⟨Ez[η(z)∑∇δt(z;θ)],Cov⁡(η(z)∥∑∇δt(z;θ)∥,(η(z)+f′(β∑δt(z;θ)))∑∇δt(z;θ))⟩>0.(30)\left\langle\mathbb{E}_z\left[\eta(z)\sum\nabla\delta_t(z;\theta)\right],\operatorname{Cov}\left(\eta(z)\left\lVert\sum\nabla\delta_t(z;\theta)\right\rVert,\left(\eta(z)+f'(\beta\sum\delta_t(z;\theta))\right)\sum\nabla\delta_t(z;\theta)\right)\right\rangle>0. \tag*{(30)}

For a constant vector v=E[η∇δt]v=\mathbb{E}[\eta\nabla\delta_t] and scalar X=η∥∇δt∥X=\eta\lVert\nabla\delta_t\rVert, the identity ⟨v,Cov⁡(X,Y)⟩=Cov⁡(X,⟨v,Y⟩)\langle v,\operatorname{Cov}(X,Y)\rangle=\operatorname{Cov}(X,\langle v,Y\rangle) gives a component-wise sufficient condition for (26): for each coordinate jj,

Cov⁡(η(z)∥∇δt(z;θ)∥,(η(z)+f′(β∑δt(z;θ)))∇δt(z;θ)j)>0,(31)\operatorname{Cov}\left(\eta(z)\lVert\nabla\delta_t(z;\theta)\rVert,\left(\eta(z)+f'(\beta\sum\delta_t(z;\theta))\right)\nabla\delta_t(z;\theta)_j\right)>0, \tag*{(31)}

which holds whenever η(z)+f′(βδt(z;θ))>0\eta(z)+f'(\beta\delta_t(z;\theta))>0 in the direction of ∇δt\nabla\delta_t, i.e.

η(z)>−f′(βδt(z;θ)).\eta(z)>-f'(\beta\delta_t(z;\theta)).

Case II supplementary: ϵ\epsilon-mislabeled noise. When noise η(z)≡ϵ\eta(z)\equiv\epsilon,

Cov⁡(η(z)∥∇δt(z;θ)∥,(η(z)+f′(∑δt(z;θ)))∇δt(z;θ))=Cov⁡(ϵ∥∇δt(z;θ)∥,(ϵ+f′(∑δt(z;θ)))∇δt(z;θ))=ϵ(Var⁡(∥∇δt(z;θ)∥)−Cov⁡(∥∇δt(z;θ)∥,f′(∑δt(z;θ))∥∇δt(z;θ)∥)).(32)\begin{aligned} \operatorname{Cov}\left(\eta(z)\lVert\nabla\delta_t(z;\theta)\rVert,\left(\eta(z)+f'(\sum\delta_t(z;\theta))\right)\nabla\delta_t(z;\theta)\right) \\ &=\operatorname{Cov}\left(\epsilon\lVert\nabla\delta_t(z;\theta)\rVert,\left(\epsilon+f'(\sum\delta_t(z;\theta))\right)\nabla\delta_t(z;\theta)\right) \\ &=\epsilon\left(\operatorname{Var}(\lVert\nabla\delta_t(z;\theta)\rVert)-\operatorname{Cov}(\lVert\nabla\delta_t(z;\theta)\rVert,f'(\sum\delta_t(z;\theta))\lVert\nabla\delta_t(z;\theta)\rVert)\right). \tag*{(32)} \end{aligned}

In the early stage of training, we assume ∥∇δt(z;θ)∥\lVert\nabla\delta_t(z;\theta)\rVert is independent to f′(∑δt(z;θ))f'(\sum\delta_t(z;\theta)), then we can get that:

Cov⁡(∥∇δt(z;θ)∥,f′(∑δt(z;θ))∥∇δt(z;θ)∥)≈E[f′(z;θ)]Var⁡(∥∇δt(z;θ)∥)(33)\operatorname{Cov}(\lVert\nabla\delta_t(z;\theta)\rVert,f'(\sum\delta_t(z;\theta))\lVert\nabla\delta_t(z;\theta)\rVert)\approx\mathbb{E}[f'(z;\theta)]\operatorname{Var}(\lVert\nabla\delta_t(z;\theta)\rVert) \tag*{(33)}

Since Var⁡(∥∇δt(z;θ)∥)>0\operatorname{Var}(\lVert\nabla\delta_t(z;\theta)\rVert)>0, so the (32) >0>0 would be simplified to:

ϵ>E[f′(∑δt(z;θ))](34)\epsilon>\mathbb{E}[f'(\sum\delta_t(z;\theta))] \tag*{(34)}

Experiment Setting

Compute Resources

We conduct all the experiments on a server that has two Intel Xeon 6326 CPU and 6 Nvidia A6000 GPU. We implement our code using PyTorch version 2.4.1.

Experiment Details

Pretrained Models. We conduct experiments on two pretrained language models of different scales and architectures. LLaMA2-7B [23] is a 7B-parameter autoregressive language model that serves as our primary backbone, providing a well-established baseline for preference optimization research. Qwen3-4B [28] is a 4B-parameter model from the Qwen3 family, included to assess the generalizability of our method across model families and parameter scales. For both models, we initialize from the instruction-tuned checkpoint (i.e., the SFT model) and treat it as the reference policy πref\pi_{\mathrm{ref}}. All models are trained using AdamW with a linear warmup schedule. We set β=0.1\beta= 0.1 for all DPO-based methods. Our PORT-specific hyperparameters ρ\rho and γ\gamma in Eq. (10) are selected via grid search on a held-out validation set. We use ρ=0.3\rho= 0.3 and γ=1.5\gamma= 1.5 in practice.

Datasets. We conduct experiments on three benchmarks spanning helpfulness and safety alignment. AlpacaFarm [6] is a simulated human feedback dataset designed for evaluating instruction-following alignment methods. It provides a standardized training and evaluation pipeline that allows for reproducible comparisons across methods. We use this dataset as our primary benchmark for evaluating general alignment quality, reporting both Gold Reward Model (RM) Score and GPT preference win rate as evaluation metrics. Anthropic HH [1] is a large-scale human preference dataset covering two complementary alignment objectives: helpfulness and harmlessness. The helpfulness split consists of 43,835 training samples and 2,354 test samples, while the harmlessness split consists of 42,537 training samples and 2,312 test samples. We use the harmlessness split to train models for safety measurement and evaluation, following the protocol of Dai et al. [5]. PKU-Beaver [11] is a large-scale safety benchmark providing fine-grained safety labels across multiple categories of harmful content, including violence, hate speech, self-harm, illegal activities, and misinformation. We use it only for evaluation, never as a training signal. Specifically, we sample 100 prompts from PKU-Beaver and use the same 100 prompts for every method. For each model trained on Anthropic HH, we generate one response per prompt. Each response is then evaluated in two ways. First, the PKU-Beaver safety classifier labels it as safe or unsafe, and the Safety Rate (SR) is the fraction of the 100 responses labeled safe. Second, GPT-4 compares it against the SFT model’s response to the same prompt using the pairwise protocol described below, which yields the GPT Win Rate reported for Anthropic HH. A higher SR indicates that the model more frequently produces safe responses when prompted with potentially harmful inputs. We use the PKU-Beaver safety classifier as an evaluator rather than a training signal: models trained on Anthropic HH are evaluated on PKU-Beaver prompts, with Safety Rate (SR) reported as the primary safety metric. A higher SR indicates that the model more frequently produces safe responses when prompted with potentially harmful inputs.

Figure 4. Scatter plot of ΔCDF\Delta\mathrm{CDF} versus token-level gradient discrepancy. Since ∥∇h(z;θ)∥=∥∇θπθ(yw∣x)−∇θπθ(yl∣x)∥\lVert\nabla h(z;\theta)\rVert= \lVert\nabla_{\theta}\pi_{\theta}(y_w|x)-\nabla_{\theta}\pi_{\theta}(y_l|x)\rVert, we compute this quantity by taking the SFT gradients of the chosen and rejected responses separately and evaluating the norm of their difference. Across a large population of tokens, an implicit linear trend emerges between the two quantities, suggesting that ΔCDF\Delta\mathrm{CDF} may serve as a lightweight proxy for the noise-induced gradient magnitude η(z)∥∇h(z;θ)∥\eta(z)\lVert\nabla h(z;\theta)\rVert in Theorem 1, without requiring access to the unobservable statistics.

Evaluation metrics. We evaluate model performance using three complementary metrics. Gold Reward Model (RM) Score, GPT Win Rate (%), and Safety Rate (SR). Gold Reward Model (RM) Score measures response quality by querying a large-scale human preference model [12] as a proxy for human judgment; following standard practice, we zero-center the scores so that the initial reference policy has an average reward of zero. GPT Win Rate (%) employs GPT-4 as a pairwise evaluator to compare the output of each method against the SFT baseline; to mitigate position bias, each pair is evaluated twice with the response order swapped, and only consistent preferences are counted, inconsistent or tied judgments are excluded from the denominator. Safety Rate (SR) applies the PKU-Beaver safety classifier [5] to responses generated on PKU-Beaver prompts, reporting the fraction of responses deemed safe; a higher SR indicates stronger alignment with harmlessness objectives. Gold RM Score and GPT Win Rate are reported for AlpacaFarm, while PKU-Beaver SR and GPT Win Rate are reported for Anthropic HH.

Here is the following evaluation template:

GPT-4 Pairwise Evaluation Prompt

You are an expert evaluator assessing the quality of responses generated by AI assistants. Your task is to compare two responses to a given instruction and select the one that better satisfies the following criteria:

Evaluation Criteria:
1. Helpfulness: The response effectively addresses the user’s instruction and provides useful, accurate information.
2. Harmlessness: The response avoids toxic, offensive, or harmful content, and declines unsafe requests appropriately.
3. Honesty: The response is factually accurate and does not contain hallucinations or misleading statements.

Instruction:
{prompt}

Response A:
{response_a}

Response B:
{response_b}

Which response is better overall? You must choose exactly one option:
[[A]] if Response A is better,
[[B]] if Response B is better,
[[C]] if they are equally good.

Provide a brief justification (1–2 sentences) before stating your final answer.

To mitigate potential position bias [24], each pair of responses is evaluated twice with the order of yAy_A and yBy_B swapped. A response is counted as a win only if GPT-4 consistently prefers it in both orderings; ties and inconsistent judgments are excluded from the win rate calculation. The final GPT win rate for method AA against method BB is computed as:

WR⁡(A,B)=#{consistent wins of A over B}#{consistent wins}+#{consistent losses}(35)\operatorname{WR}(A,B)=\frac{\#\{\text{consistent wins of }A\text{ over }B\}}{\#\{\text{consistent wins}\}+\#\{\text{consistent losses}\}} \tag*{(35)}

where we report the win rate of PORT against the SFT baseline following the convention of [6].

Experiment Result

The γ,ρ\gamma,\rho-Sensitivity. For γ,ρ\gamma,\rho-sensitivity, we ran a grid search over on a held-out validation split (LLaMA2-7B), scored by mean ground-truth log-prob margin, the result is shown in Table 5.

ρ\rho \ γ\gamma0.51.01.5
0.10.08040.23150.2928
0.30.09210.26240.3088
0.50.07970.24480.2649

Table 5. Sensitivity analysis of γ\gamma and ρ\rho on LLaMA2-7B. Performance is measured by the mean ground-truth log-probability margin between chosen and rejected responses on a held-out validation set. Higher is better.

Batch size Sensitivity. We investigate the sensitivity to batch size B∈{1,4,16}B \in\{1,4,16\} using Qwen2.5-0.5B. As shown in Table 6, performance decreases to 39.1% GPT win rate when B=1B=1, but improves substantially to 42.8% at B=4B=4, approaching the 44.3% achieved with B=16B=16, the default setting used throughout our experiments. These results suggest that pooling reference logits across multiple samples improves performance, while relatively small batch sizes already capture much of the benefit.

Complete experiment results. The complete experiment results of ablation study are shown in Table 7 and Table 8.

Methodα\alpha-uncertain noiseα\alpha-uncertain noiseϵ\epsilon-mislabeled noiseϵ\epsilon-mislabeled noise
SR ↑\uparrowGPT Win Rate (%) ↑\uparrowSR ↑\uparrowGPT Win Rate (%) ↑\uparrow
SLiC-HF0.3652.00.3752.6
IPO0.3752.60.3954.2
DPO0.3853.50.3853.4
fDPO0.4156.10.4559.3
SamS0.4357.00.4660.1
cDPO0.4054.80.4357.6
rDPO0.4055.30.4458.4
β\beta-DPO0.4357.40.4760.9
Dr.DPO0.4458.50.4861.8
PORT0.4962.50.5365.9

Table 7. Impact of Noise Type on Model Performance (evaluated by PKU-Beaver SR ↑\uparrow) on Anthropic HH with p=0.4p = 0.4 noise level. Bold denotes the best performance, _underline_ indicates the second-best performance.

Table 8: Ablation of Noise Intensity on Anthropic HH (evaluated by PKU-Beaver SR ↑\uparrow). Bold denotes the best performance, <u>underline</u> indicates the second-best performance. We have improved 19.2% compared with the SFT baseline and 2% compared with the best SOTA method.

Methodp=0.2p = 0.2p=0.3p = 0.3p=0.4p = 0.4
MethodSR ↑\uparrowGPT Win Rate (%) ↑\uparrowSR ↑\uparrowGPT Win Rate (%) ↑\uparrowSR ↑\uparrowGPT Win Rate (%) ↑\uparrow
SLiC-HF0.4458.40.4156.00.3752.6
IPO0.4660.10.4357.60.3954.2
DPO0.4559.30.4256.80.3853.4
fDPO0.5164.20.4861.80.4559.3
SamS0.5365.9<u>0.53</u><u>65.8</u>0.4660.1
cDPO0.5063.40.4861.80.4357.6
rDPO0.5164.20.4962.60.4458.4
β\beta-DPO<u>0.54</u><u>66.7</u>0.5164.30.4760.9
Dr.DPO0.5265.10.5063.5<u>0.48</u><u>61.8</u>
PORT0.5668.50.5567.60.5365.9

Table 8. Ablation of Noise Intensity on Anthropic HH (evaluated by PKU-Beaver SR ↑\uparrow). Bold denotes the best performance, <u>underline</u> indicates the second-best performance. We have improved 19.2% compared with the SFT baseline and 2% compared with the best SOTA method.

Social Impact

This work addresses the robustness of preference optimization under noisy human feedback, a fundamental challenge in aligning large language models with human values. By improving the reliability of alignment training, PORT has the potential to contribute to safer and more trustworthy AI systems, reducing the risk of models exhibiting harmful, biased, or misleading behaviors that can arise when training signals are corrupted. More broadly, our method lowers the annotation quality requirements for effective alignment, which may reduce the cost and labor burden of large-scale preference data collection and make robust alignment more accessible to resource-constrained practitioners.

Limitations

Despite these strengths, PORT has limitations worth noting: the CDF-based proxy relies on batch-level statistics and may be less reliable under very small batch sizes; and our theoretical guarantees hold under specific sufficient conditions that may not always be verifiable in practice. We hope PORT encourages future exploration of token-aware alignment approaches, including extensions to multi-turn settings and adaptive hyperparameter scheduling.

Qualitative comparison of PORT and the baseline model on a safety-related prompt

Figure 5. Qualitative comparison of PORT and the baseline model on a safety-related prompt from PKU-Beaver. PORT’s response is classified as harmless, whereas the baseline response is classified as harmful.

NeurIPS Paper Checklist

Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Answer: [Yes]

Justification:

Guidelines:

  • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.

  • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.

  • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

  • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We add a limitation discussion in Appendix § 12.

Guidelines:

  • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.

  • The authors are encouraged to create a separate “Limitations” section in their paper.

  • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

  • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

  • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

  • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

  • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

  • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We put the complete proof in the appendix.

Guidelines:

  • The answer [N/A] means that the paper does not include theoretical results.

  • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

  • All assumptions should be clearly stated or referenced in the statement of any theorems.

  • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

  • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

  • Theorems and Lemmas that the proof relies upon should be properly referenced.

Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We put the hyperparameter we use in the appendix.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

  • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

  • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

  • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]

Justification: The dataset we use are all public, and we provide the code in the attached file.

Guidelines:

  • The answer [N/A] means that paper does not include experiments requiring code.

  • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

  • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

  • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

  • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

  • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?

Answer: [Yes]

Justification: We illustrate all the details in the experiment section.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

  • The full details can be provided either with the code, in appendix, or as supplemental material.

Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the standard variance in Table 2.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

  • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

  • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

  • The assumptions made should be given (e.g., Normally distributed errors).

  • It should be clear whether the error bar is the standard deviation or the standard error of the mean.

  • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

  • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).

  • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We demonstrate the computation device we use in Appendix § 10.2.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

  • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

  • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).

Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: There is no violation of the code of ethics.

Guidelines:

  • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.

  • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.

  • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The social impact is written in Appendix § 11.

Guidelines:

  • The answer [N/A] means that there is no societal impact of the work performed.

  • If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.

  • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

  • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

  • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

  • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [N/A]

Justification: No such risk.

Guidelines:

  • The answer [N/A] means that the paper poses no such risks.

  • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

  • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

  • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the papers.

Guidelines:

  • The answer [N/A] means that the paper does not use existing assets.

  • The authors should cite the original paper that produced the code package or dataset.

  • The authors should state which version of the asset is used and, if possible, include a URL.

  • The name of the license (e.g., CC-BY 4.0) should be included for each asset.

  • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

  • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

  • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

  • If this information is not available online, the authors are encouraged to reach out to the asset’s creators.

New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [N/A]

Justification: The paper does not release new assets.

Guidelines:

  • The answer [N/A] means that the paper does not release new assets.

  • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

  • The paper should discuss whether and how consent was obtained from people whose asset is used.

  • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [N/A]

Justification: No crowdsourcing is involved.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

  • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [N/A]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

  • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

  • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.

Answer: [N/A]

Justification: We do not use LLMs as an important, original, or non-standard component of the core methods in this research.

Guidelines:

  • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

  • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.

References

  1. [1]Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL https://arxiv.org/abs/2204.05862.
  2. [2]Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383097. doi: 10.1145/3442188.3445922. URL https://doi.org/10.1145/3442188.3445922.
  3. [3]Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. ISSN 00063444, 14643510. URL http://www.jstor.org/stable/2334029.DOI
  4. [4]Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust dpo: aligning language models with noisy feedback. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.arxiv.org/abs/2403.00409
  5. [5]Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=TyFRP0KXXw.arxiv.org/abs/2310.12773
  6. [6]Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.arxiv.org/abs/2305.14387
  7. [7]Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 4447–4455. PMLR, 02–04 May 2024. URL https://proceedings.mlr.press/v238/gheshlaghi-azar24a.html.
  8. [8]Zixuan Huang, Yikun Ban, Lean Fu, Xiaojie Li, Zhongxiang Dai, Jianxin Li, and Deqing Wang. Adaptive batch-wise sample scheduling for direct preference optimization, 2026. URL https://arxiv.org/abs/2506.17252.
  9. [9]Shawn Im and Sharon Li. How well can preference optimization generalize under noisy feedback?, 2026. URL https://arxiv.org/abs/2510.01458.
  10. [10]Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of LLM via a human-preference dataset. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum?id=g0Q0xvBFw3.arxiv.org/abs/2307.04657
  11. [11]Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Alex Qiu, Jiayi Zhou, Kaile Wang, Boxun Li, Sirui Han, Yike Guo, and Yaodong Yang. PKU-SafeRLHF: Towards multi-level safety alignment for LLMs with human preference. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 31983–32016, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1544. URL https://aclanthology.org/2025.acl-long.1544/.DOI
  12. [12]Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannah Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024. URL https://arxiv.org/abs/2403.13787.
  13. [13]Xize Liang, Chao Chen, Shuang Qiu, Jie Wang, Yue Wu, Zhihang Fu, Hanzhu Chen, Feng Wu, and Jieping Ye. ROPO: Robust preference optimization for large language models. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 37131–37161. PMLR, 13–19 Jul 2025. URL https://proceedings.mlr.press/v267/liang25d.html.
  14. [14]Aiwei Liu, Haoping Bai, Zhiyun Lu, Yanchao Sun, Xiang Kong, Xiaoming Simon Wang, Jiulong Shan, Albin Madappally Jose, Xiaojiang Liu, Lijie Wen, Philip S. Yu, and Meng Cao. TIS-DPO: Token-level importance sampling for direct preference optimization with estimated weights. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=OF6e2WwxX0.arxiv.org/abs/2410.04350
  15. [15]Jilong Liu, Yonghui Yang, Pengyang Shao, Haokai Ma, Wei Qin, and Richang Hong. wdpo: Winsorized direct preference optimization for robust llm alignment, 2026. URL https://arxiv.org/abs/2603.07211.
  16. [16]Eric Mitchell. A note on dpo with noisy preferences and relationship to ipo. https://ericmitchell.ai/cdpo.pdf, 2023.
  17. [17]Tetsuro Morimura, Mitsuki Sakamoto, Yuu Jinnai, Kenshi Abe, and Kaito Ariu. Filtered direct preference optimization. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22729–22770, Miami, Florida, USA, November 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.emnlp-main.1266.
  18. [18]Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 97817113871088.arxiv.org/abs/2203.02155
  19. [19]Qwen, ., An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Kemeng Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yan Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115.
  20. [20]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2305.18290.
  21. [21]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347.
  22. [22]Tiesunlong Shen, Rui Mao, Jin Wang, Heming Sun, Jian Zhang, Xuejie Zhang, and Erik Cambria. Lmdoctor: Token-level flow-guided preference optimization for efficient test-time alignment of large language models, 2026. URL https://arxiv.org/abs/2601.10416.
  23. [23]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyi Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Sagar Hosseini, Rui Hou, Ahan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kroumnn, Artem Korenov, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https://arxiv.org/abs/2307.09288.
  24. [24]Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Jian Jiang, and Qun Liu. Aligning large language models with human: A survey, 2023. URL https://arxiv.org/abs/2307.12966.
  25. [25]Xiukun Wei and Xueru Zhang. Self-consuming generative models with adversarially curated data, 2025. URL https://arxiv.org/abs/2505.09768.
  26. [26]Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. β-dpo: direct preference optimization with dynamic β. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9798331314385.DOI
  27. [27]Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jiawei Chen, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. Towards robust alignment of language models: Distributionally robustifying direct preference optimization, 2025. URL https://arxiv.org/abs/2407.07880.
  28. [28]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chenggen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jiahong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.
  29. [29]Ning Yang, Hai Lin, Yibo Liu, Baoliang Tian, Guoqing Liu, and Haijun Zhang. Token-importance guided direct preference optimization, 2026. URL https://arxiv.org/abs/2505.19653.
  30. [30]Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization, 2024. URL https://arxiv.org/abs/2404.11999.
  31. [31]Yang Zhang, Xiukun Wei, and Xueru Zhang. When and how human curation backfires: Preference alignment under multi-model self-consuming loop, 2026. URL https://arxiv.org/abs/2605.29267.
  32. [32]Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. Slic-hf: Sequence likelihood calibration with human feedback, 2023. URL https://arxiv.org/abs/2305.10425.

Paper details

vlnc:W_4efv49ah
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents