Stochastic Pruning of the DPO Computation Graph: 1.4x Faster Training and Lower Peak Memory

Introduction

Direct preference optimization (DPO) formulates alignment as a logistic comparison of policy–reference likelihood ratios, with gradient contributions from both chosen and rejected responses Rafailov et al., 2023. In large-vocabulary language-model training, materialized logits can dominate memory consumption Wijmans et al., 2025, and attention I/O and operator fusion also affect memory use and training throughput Dao et al., [9]; Hsu et al., [13]. These costs motivate our question of which computations a preference-training step can omit.

Prior work changes token supervision through confidence-based token selection or weighting Yoon et al., 2025; Jiang et al., [18], reward-based selection Yang et al., 2025b, and weighting from contrastive model predictions Liu et al., 2025, while memory-efficient kernels change output-score computation Wijmans et al., 2025; Hsu et al., [13]. This distinction motivates our question: which parts of a DPO step can be omitted, and how do their resource savings relate to changes in the optimization objective?

Preference-objective structure affects optimization Chen et al., [4], while shared-parameter updates can couple response likelihoods Yuan et al., 2025 and reduce preferred-response probability Razin et al., 2025. Our token-level analysis examines this coupling (Appendix § 8.§ 1), and Proposition 1 separates reward preservation in expectation from the nonlinear loss.

We introduce a framework that samples the DPO computation graph to control training cost (Figure 1). Each training step jointly selects the executed response prefixes, the positions used to estimate rewards, and the branches that carry gradients. These selections precede sequence execution, vocabulary projection, and backpropagation, allowing computation to be omitted throughout the step. This view unifies position, sequence, and branch pruning through their effects on the sampled objective and its gradient. Our analysis characterizes the objective changes induced by reward sampling, while inverse-probability weighting makes the branch gradient conditionally unbiased for that objective.

Our contributions are:

• a unified framework for sampling DPO computation across response execution, token scoring, and backpropagation, with separately configurable sampling decisions.

• a characterization of the position and prefix objective changes, tolerance conditions that relate their optimal gaps to the retention rate and the truncation level, and a conditionally unbiased branch estimator with a stated convergence rate.

• controlled comparisons showing that full supervision retains most output-memory savings, that branch pruning reduces training time across model families, and that mean accuracy is slightly lower while open-ended score changes vary by benchmark (§ 4.2, § 4.4 and § 4.5).

Related Work

Token-level credit in preference optimization. A growing family of methods changes where DPO assigns credit through confidence-based token selection (Yoon et al., 2025), confidence-dependent weighting of varied tokens [18], trained oracles for token rewards (Yang et al., 2025b), and contrastive token weights (Liu et al., 2025). Other approaches use optimal-transport reweighting (Li et al., 2025a), gradient-attribution priors (Yang et al., 2026), prefix-level ratio matching (Nguyen et al., 2026), implicit-reward credit (Zhao et al., 2026), attention-derived weights [15], minimal-edit emphasis in clinical dialogue [20], or token-level KL control (Zeng et al., 2024). Our framework places supervision selection within the computation of a complete DPO step. Matched selection controls motivate uniform membership (Sec. E.1), and full-retention controls distinguish supervision deletion from implementation savings (Table 1).

Configuration	Retention	Step (s)	HBM (MB)
Full-concat	1.0	17.081	23,627
Selected head	1.0	16.827	14,722
Selected head	0.6	16.791	14,080

Efficient preference training. Cut Cross-Entropy avoids materializing the token-by-vocabulary logit matrix (Wijmans et al., 2025), and Liger-Kernel uses fusion and chunked cross-entropy computation to reduce memory and improve training throughput [13]. FlashAttention relates attention efficiency to memory accesses rather than FLOP counts alone [9], while activation checkpointing trades recomputation for lower activation storage [2]. Related efficiency work covers flow training [10] and diffusion inference (Li et al., 2025b). SePO reports reduced GPU training hours through offline token selection (Yang et al., 2025b). Our framework considers how sampling throughout a DPO step changes both executed computation and the training objective. Controlled profiles separate implementation savings from the effects of reduced supervision and execution (§ 4.2).

Stochastic computation and optimization dynamics. Dropout randomly removes units (Srivastava et al., 2014) and stochastic depth bypasses sampled layers [16], while importance sampling can reduce stochastic-gradient variance using gradient-dependent probabilities [19]. Preference-optimization analyses study gradient entanglement (Yuan et al., 2025), likelihood displacement (Razin et al., 2025), logit-space squeezing (Luo et al., 2026), and gated rejected contributions (Mouiche, 2026). Density-ratio estimation provides a distribution-comparison perspective [3] for preference objectives such as DDRO [12], whose update structure is analyzed alongside margin-based losses [4]. Our analysis separates objective changes from gradient-estimation noise under composed sampling (Proposition 2 and § 3.5). Shared-parameter response changes are analyzed in Appendix § 8.§ 1.

Stochastic Graph Pruning

Formulation

Let $\mathcal{D} = \{(x, y_w, y_l)\}$ contain prompts $x$ with chosen and rejected responses $y_w, y_l$. Write $\pi_\theta$ for the policy with trainable parameters $\theta$ and $\pi_{\mathrm{ref}}$ for the fixed reference (Rafailov et al., 2023). For side $b \in\{w,l\}$, let $T_b \ge1$ count response tokens, excluding prompt and padding. At position $t$, define the target-token log-ratio

Here $y_{b,t}$ is the target token and $y_{b,<t}$ its preceding response prefix. With mean rewards $R_b = T_b^{-1}\sum_{t=1}^{T_b}u_{b,t}$, margin $\Delta= R_w-R_l$, and scale $\beta> 0$, DPO minimizes the loss (Rafailov et al., 2023):

This length normalization parallels SimPO (Meng et al., 2024), but retains policy–reference log-ratios.

Figure 1 separates three costs: dense vocabulary projection materializes $T_b \times V$ logits for vocabulary size $V$ (Wijmans et al., 2025), sequence execution computes attention and stores or recomputes activations [9, 2], and backpropagation includes both responses (Rafailov et al., 2023).

Write $\mathbb{I}_M = \{1,\ldots,M\}$. On side $b$, sequence pruning selects length $M_b \in\mathbb{I}_{T_b}$ through cutoff fraction $\rho_b \in(0,1]$, retaining $\tilde{y}_b = y_{b,1:M_b}$. Position pruning selects $S_b \subseteq\mathbb{I}_{M_b}$ with $k_b = |S_b|$ controlled by retention $\alpha\in(0,1]$. Branch pruning selects $b^\star\in\{w,l\}$ with probabilities $q_w,q_l > 0$, $q_w+q_l=1$. All sampling laws are parameter-independent, and rewards average over selected positions. We use $\sigma$ for the sigmoid and $\operatorname{sg}$ for stop-gradient, which preserves values and blocks derivatives. Appendix § 6 summarizes the notation.

Stochastic Position Pruning

Position pruning retains a uniform subset of response positions for scoring while preserving the executed response as context. Draw $S_b \sim\operatorname{Select}(M_b,\alpha)$ by

Standalone position pruning uses $M_b=T_b$. Define the selected-position reward

with masked margin $\Delta=R_w(S_w)-R_l(S_l)$. The subset is drawn before the language-model head, without computing policy probabilities. Applying the head only at $S_b$ evaluates Eq. (5) exactly, skipping $M_b-k_b$ vocabulary projections, their logits, and their backward traffic.

Uniform sampling preserves the mean reward, but the nonlinear loss introduces a gap.

At $\alpha=1$, selected-head execution preserves the full-response objective while avoiding projection outside valid response positions (Table 1).

To bound the sampled optimum, assume throughout this subsection and Secs. B.5 and B.6 that $|u_{b,t}| \le U$, $\lVert\nabla_{\theta}u_{b,t}\rVert\le G$, $v_b \le\bar{v}$, and $T_b \ge T_0 \ge2$ on the analyzed region. The objectives are twice differentiable there and attain their minima.

The variance is evaluated at $\theta^*$ over masks, then averaged over data. Since $k_b \ge\alpha T_b$, Proposition 1 gives $\mathbb{E}_{D}[\operatorname{Var}_{S}(\Delta_U)] \le2\bar{v}\,r_A$, where

Thus the bound decays as $1/T_0$ at fixed retention and vanishes at $\alpha=1$. A sufficient condition for loss tolerance $\varepsilon_F>0$ is

A second-order expansion also gives $\lVert\nabla F_A-\nabla F\rVert=O(r_A)$ (Appendix § 7.6). If $\nabla^2F\succeq\mu I$, $\mu>0$, along the segment joining interior stationary points of $F_A$ and $F$ within the analyzed region, their displacement is $O(r_A)$ rather than $O(\sqrt{r_A})$. The loss bound requires no curvature assumption. Sufficient curvature conditions appear in Appendix § 7.7.

Stochastic Sequence Pruning

Sequence pruning truncates each response before policy and reference execution. For minimum retained fraction $\rho_{\min}\in(0,1)$, draw $(\widetilde{y}_b,M_b)\sim\operatorname{Prefix}(y_b,\rho_{\min})$ by

Both models repack the prompt with $\widetilde{y}_b$, removing suffix attention, activations, and recomputation. The reward is $R_b=M_b^{-1}\sum_{t=1}^{M_b}u_{b,t}$, and $\rho_b\equiv1$ recovers the full-response objective. § 4.3 evaluates $\rho_{\min}$.

Prefix inclusion and length normalization favor earlier positions:

Appendix § 7.§ 2 proves this identity and characterizes the cutoff laws. The nonlinear loss adds a Jensen gap:

Unlike position pruning, sequence pruning can shift the mean margin. The shift $\ell(\mu_{\mathrm{pref}})-\ell(\Delta_{\mathrm{full}})$ has no fixed sign, whereas $J_{\mathrm{pref}}\ge0$ vanishes for deterministic cutoffs or $\beta=0$ (Appendix § 7.§ 3). Parameter-independent cutoffs make $\nabla_{\theta}\ell(\Delta_M)$ unbiased for $\nabla_{\theta}L_{\mathrm{pref}}$, generally different from the full-response gradient. § 4.4 evaluates the resulting performance.

Assume causal execution with the prompt retained, so prefix token scores and gradients match full execution. For truncation fraction $\rho_b=M_b/T_b$, Eq. (11) gives $\mathbb{E}[h_b]\le(1-\rho_{\min})/2$. Prefix and full mean rewards differ by at most $2U h_b$, and their gradients by $2Gh_b$. The $\beta$-Lipschitz loss therefore yields

where $F_P=\mathbb{E}[L_{\mathrm{pref}}]$. Consequently, $F(\theta_P^*)-F(\theta^*)\le4\beta U(1-\rho_{\min})$. Unlike Eq. (8), this bound has no response-length decay. At the tested $\rho_{\min}=1/2$, the optimality bound equals the full loss range over $|\Delta|\le2U$, making it vacuous. This comparison concerns theoretical bounds, not measured deviations. Appendix § 7.8 also derives gradient and displacement bounds.

Stochastic Branch Pruning

Branch pruning removes one backward graph while preserving the expected gradient conditional on the sampled positions and prefixes. With $\lambda= \beta\sigma(-\beta\Delta)$ and $\sigma(z) = (1 + e^{-z})^{-1}$, the DPO gradient decomposes as Rafailov et al. (2023)

Independently of positions and prefixes, draw the active backward-branch set and its probability, $(\mathbb{B},q^\star) \sim\operatorname{Branch}(q_w)$, with $q_l = 1-q_w$ and

Only $b^\star\in\mathbb{B}$ retains a policy backward graph. The opposite side runs under no_grad, providing the other reward needed for $\lambda$. Disabling branch pruning sets $\mathbb{B} = \{w,l\}$ and $q^\star= 1$. Inverse-probability weighting corrects each selected contribution. With $\zeta(z) = \log(1+e^z)$, use

The sampled loss $\widehat{\mathcal{L}}=\widehat{\mathcal{L}}_{b^\star}$ preserves the numerical margin and differentiates only the selected reward.

We default to $q_w=q_l=0.5$ and sweep $q_w \in\{0.3,0.5,0.7\}$ in § 4.3. Gradient norms govern importance-sampling variance Katharopoulos and Fleuret [19], as specialized in Appendix § 7.§ 4.

At $q_w=q_l=1/2$, the variance is $\lVert c_w-c_l\rVert^2 \le4\beta^2G^2$ under the bounds of § 3.2. Averaging $m$ estimators independent in preference pairs, positions, prefixes, and branches divides the variance by $m$. Re-drawing only the branch reduces only its variance component.

Branch pruning adds gradient noise without objective bias. For the optimized objective $\Phi$, assume $\mu$-strong convexity and $L$-smoothness on a closed convex region $\Theta$ with minimizer $\theta^\star$ in its interior. Starting from $\theta_0\in\Theta$, projected SGD with $m$ independent estimators per step and the schedule in Appendix § 7.9 gives $\mathbb{E}\lVert\theta_N-\theta^\star\rVert^2=\mathcal{O}(1/N)$. That appendix supplies the constants and a high-probability bound. Gradient unbiasedness does not imply unbiased finite-step parameters.

Composition

Alg. 1 composes the interventions in execution order: truncate the response, select scoring positions, then select the backward branch and apply its probability correction. Each mode is optional. Full disables all three, using full responses, all response scores, and both backward branches. Tables abbreviate the modes as Position, Sequence, and Branch, with “+” denoting composition.

Algorithm 1 Stochastic DPO step

Require: Pair $(\boldsymbol{x},y_w,y_l)$, policy $\pi_\theta$, frozen $\pi_{\mathrm{ref}}$

Require: Rates $\alpha,\rho_{\min},q_w$, enabled pruning modes

1. for $b\in\{w,l\}$ do

2. $\left(\tilde{y}_b,M_b\right)\leftarrow(y_b,T_b)$

3. if sequence pruning then

4. $\left(\tilde{y}_b,M_b\right)\sim\operatorname{Prefix}(y_b,\rho_{\min})$ $\triangleright$ Eq. (11)

5. $S_b\leftarrow\mathbb{I}_{M_b}$

6. if position pruning then

7. $S_b\sim\operatorname{Select}(M_b,\alpha)$ $\triangleright$ Eq. (3)

8. $\mathbb{B}\leftarrow\{w,l\}$

9. $q^\star\leftarrow1$

10. if branch pruning then

11. $(\mathbb{B},q^\star)\sim\operatorname{Branch}(q_w)$ $\triangleright$ Eq. (16)

12. for $b\in\{w,l\}$ do

13. $z_b\leftarrow\operatorname{LogProbs}(\pi_\theta,\boldsymbol{x},\tilde{y}_b,S_b,b\in\mathbb{B})$

14. $z_b^{\mathrm{ref}}\leftarrow\operatorname{LogProbs}(\pi_{\mathrm{ref}},\boldsymbol{x},\tilde{y}_b,S_b,\mathrm{false})$

15. $R_b\leftarrow\operatorname{mean}(z_b-z_b^{\mathrm{ref}})$ $\triangleright$ Eq. (5)

16. $\mathcal{L}\leftarrow\ell(R_w-R_l)/q^\star$ $\triangleright$ Eq. (17)

17. $\mathcal{L}.\operatorname{backward}()$

18. Update $\theta$

For a policy or reference model $\pi$, the scored log-probability vector is

Both models score the same prefixes and positions, with entries in response order. Position pruning applies the head only at $S$; otherwise, the full head is evaluated before gathering. The argument $g$ controls gradient recording through the backbone and head: only policy branches in $\mathbb{B}$ retain a backward graph, while the reference always runs without gradient. dients. Numerical values are unchanged. The outputs $z_b$, $z_b^{\mathrm{ref}}$ give $R_b$ by the mean difference in Eq. (5). The step loss is $\mathcal{L} = \ell(R_w - R_l)/q^\star$ (Eq. (17)), recovering the per-pair loss in Eq. (2)` on the selected prefixes and positions when $q^\star= 1$.

The minimizer $\theta^\star_{AP}$ of the composed objective $F_{AP}$ satisfies

where $M_0 = \max(2,\lceil\rho_{\min}T_0\rceil)$. The position-pruning bias bound decays as $1/T_0$ at fixed retention. The prefix bound scales with $1-\rho_{\min}$ without length decay, while branch pruning adds no objective bias and has the $O(1/N)$ mean-squared optimization error above. Appendix B.10 proves the composed loss bound and Eqs. (44) and (46), which allocate a total tolerance across $\alpha$, $\rho_{\min}$, and $N$.

Experiments

Experimental Setup

We use Qwen3-4B, Qwen3-8B (Yang et al., 2025a), and Mistral-7B-Instruct-v0.3 (Jiang et al., [17]). The core resource profiles, eight-configuration factorial, and parameter sweeps use Qwen3-4B on HelpSteer3-Preference (Wang et al., 2025). LoRA (Hu et al., [14]) has rank 64, scaling 128, and dropout 0.05. Training uses FlashAttention-2 (Dao, [8]), gradient checkpointing (Chen et al., [2]), AdamW (Loshchilov and Hutter, 2019), DPO $\beta= 1$. Global batch is 64 with per-device batch 2, accumulation 4, and eight Ascend 910B accelerators with 32 GiB each. Optimizer and corpus details are in Appendix § 9.§ 2; matched token-selection controls are in Table 11 and Table 12. Defaults are $\alpha= 0.6$, $\rho_b \sim U(0.5,1)$, and $q_w = q_l = 0.5$. The factorial and sweeps use training and selection seed 42. A second Qwen seed (43) and Mistral seed 42 test the Full–Branch comparison.

Selection	Retention	ARC	GSM8K	TruthfulQA	BBH	MATH-Hard	MMLU-Pro	MuSR	Mean accuracy (%)
Low/Low	0.1	63.48	85.82	54.57	55.54	62.39	42.26	42.86	58.13
Random/Random	0.1	62.88	86.73	53.11	55.41	63.36	42.00	42.59	58.01
Low/Low	0.2	63.82	86.58	55.19	55.32	60.29	42.33	42.59	58.02
Random/Random	0.2	63.05	86.35	53.62	55.06	63.14	42.02	42.46	57.96
Low/Low	0.3	63.82	87.26	55.49	55.47	58.47	42.35	42.59	57.92
Random/Random	0.3	63.23	86.50	54.46	55.51	63.20	42.04	42.72	58.24
Low/Low	0.4	63.99	88.02	55.70	55.43	57.67	42.32	42.86	58.00
Random/Random	0.4	63.40	86.88	54.85	55.64	61.92	42.10	42.59	58.20
Low/Low	0.5	63.14	87.19	53.81	55.63	63.66	42.25	43.12	58.40
Random/Random	0.5	63.05	86.50	53.38	55.63	63.92	42.22	43.12	58.26
Low/Low	0.6	63.40	87.26	53.85	55.60	63.33	42.22	43.12	58.40
Random/Random	0.6	63.05	86.81	53.52	55.71	64.11	42.20	42.99	58.34
Low/Low	0.7	63.31	87.26	53.88	55.53	63.39	42.25	42.99	58.37
Random/Random	0.7	63.23	86.81	53.67	55.77	64.17	42.20	42.86	58.39
Low/Low	0.8	63.14	87.34	53.91	55.60	63.28	42.27	42.99	58.36
Random/Random	0.8	63.05	87.49	53.76	55.78	64.01	42.24	42.59	58.42
Low/Low	0.9	63.23	87.26	53.92	55.57	63.43	42.30	42.99	57.55*
Random/Random	0.9	63.14	87.11	53.86	55.61	–	42.26	43.12	57.52*
Qwen3-4B	–	62.80	85.90	52.27	55.59	62.80	41.98	43.39	57.82
Unmasked DPO	–	62.63	87.57	53.91	55.61	63.33	42.30	43.12	58.35

$\alpha$	Low/Low	Random/Random	Difference
Batch 32
0.1	58.13	58.01	+0.12
0.2	58.02	57.96	+0.06
0.3	57.92	58.24	−0.32
0.4	58.00	58.20	−0.20
Batch 64
0.5	58.40	58.26	+0.14
0.6	58.40	58.34	+0.06
0.7	58.37	58.39	−0.02
0.8	58.36	58.42	−0.06
0.9	57.55	57.52	+0.03*

Metrics. We group benchmarks by evaluation protocol. Accuracy-based benchmarks comprise ARC-Challenge (Clark et al., [5]), GSM8K (Cobbe et al., [6]), TruthfulQA (Lin et al., 2022), BBH (Suzgun et al., 2023), MATH-Hard (Hendrycks et al., [11]), MMLU-Pro (Wang et al., 2024), and MuSR (Sprague et al., 2024). For open-ended generation, MiniMax-M3 (Lai et al., [20]) judges MT-Bench (MT, 0–10) (Zheng et al., 2023), Arena-Hard (AH, tie-adjusted win rate in %) (Li et al., 2024), and WildBench (WB, 1–10) (Lin et al., 2024). Mean Accuracy averages the seven accuracy-based benchmark percentages. Mean Generation Score equally averages 10 MT, AH, and 10 WB on a 100-point scale. Task-level means are retained in Table 9, Table 10 and Table 13.

Method	Wall	Speed	ARC	GSM8K	TQA	BBH	MATH	MMLU	MuSR	Mean accuracy (%)	MT	AH	WB	Mean generation score
Full	187.8	1.000×	62.54	87.41	53.88	55.64	63.59	42.28	42.99	58.33	6.649	50.12	3.139	_49.33_
Position	185.9	1.010×	62.46	86.43	53.51	55.58	63.26	42.18	42.99	58.06	6.829	44.85	3.200	48.38
Sequence	186.4	1.008×	62.37	86.28	53.40	55.92	63.27	42.20	42.86	58.04	6.432	45.80	3.186	47.33
Branch	128.4	1.463×	62.03	86.66	53.06	55.73	63.65	42.16	43.25	58.08	6.800	50.11	3.263	50.25
Position+Sequence	185.0	1.015×	62.20	85.97	53.06	_55.74_	_63.63_	42.07	43.12	57.97	6.893	45.54	3.208	48.85
Position+Branch	127.8	1.469×	61.95	86.50	52.92	55.64	63.31	42.12	_43.12_	57.94	6.676	45.80	3.196	48.17
Sequence+Branch	126.3	1.487×	62.03	86.20	52.84	55.69	63.06	42.05	_43.12_	57.86	6.600	45.05	3.197	47.67
Position+Sequence +Branch	_127.0_	1.479×	62.03	85.97	52.75	55.65	62.24	42.05	_43.12_	57.69	6.693	45.86	_3.225_	48.35

Setting	ARC	GSM8K	TQA	BBH	MATH	MMLU	MuSR	Mean accuracy (%)	MT	AH	WB	Mean generation score
Position $\alpha=0.3$	62.29	86.20	53.15	55.61	62.78	42.15	43.12	57.90	6.568	45.56	3.195	47.73
Position $\alpha=0.6$	62.46	86.43	53.51	55.58	63.26	42.18	42.99	58.06	6.829	44.85	3.200	48.38
Position $\alpha=0.9$	62.54	87.34	53.83	55.64	63.54	42.26	43.12	58.32	6.707	48.00	3.081	48.63
Sequence $U(0.25,1)$	62.37	86.35	53.30	55.80	63.53	42.15	43.12	58.09	6.552	47.05	3.117	47.91
Sequence $U(0.5,1)$	62.37	86.28	53.40	55.92	63.27	42.20	42.86	58.04	6.432	45.80	3.186	47.33
Sequence $U(0.75,1)$	62.46	86.28	53.44	56.03	63.16	42.16	42.72	58.04	6.874	47.92	3.193	49.53
Sequence fixed 0.75	62.29	86.73	53.38	55.95	63.23	42.13	42.86	58.08	6.562	43.33	3.214	47.03
Branch $q_w=0.3$	62.37	87.34	53.03	55.44	62.70	42.16	43.39	58.06	6.669	47.82	3.189	48.80
Branch $q_w=0.5$	62.03	86.66	53.06	55.73	63.65	42.16	43.25	58.08	6.800	50.11	3.263	50.25
Branch $q_w=0.7$	62.03	86.58	52.97	55.75	62.92	42.15	43.39	57.97	6.753	46.64	3.205	48.74

Model / seed	Config.	ARC	GSM8K	TQA	BBH	MATH	MMLU	MuSR	Mean accuracy (%)	MT	AH	WB	Mean generation score
Qwen3-4B / 42	Full	62.54	87.41	53.88	55.64	63.59	42.28	42.99	58.33	6.649	50.12	3.139	49.33
	Branch	62.03	86.66	53.06	55.73	63.65	42.16	43.25	58.08	6.800	50.11	3.263	50.25
Qwen3-4B / 43	Full	62.54	87.04	53.88	55.55	63.45	42.34	42.99	58.26	6.623	44.36	3.233	47.64
	Branch	62.03	86.20	53.05	55.55	62.57	42.14	43.12	57.81	6.750	45.03	3.295	48.49
Mistral-7B / 42	Full	67.06	52.46	64.82	46.37	12.50	30.47	40.34	44.86	5.130	24.60	2.361	33.17
	Branch	65.27	51.18	62.87	46.51	12.96	30.32	42.72	44.55	5.373	24.00	2.405	33.93

Resource Effects and Attribution

Baselines and measurement. Full denotes the unpruned DPO baseline using the original padded stack. Full-concat batches chosen and rejected responses together to reduce padding; both retain all response supervision and both backward graphs. Position, Branch, and Position+Branch profiles use Full-concat; Sequence and end-to-end trainings use Full. The concatenated profiler measures Full-concat at 17.343 s and 23,627 MB, vs. Full at 18.204 s and 18,571 MB: 1.049× faster with 27% more memory. The separate prefix profiler measures Full at 18.826 s. Timings are compared within a profiler. The concatenated profile uses 16 paired measured rounds. Peak high-bandwidth memory (HBM) is reset per step and maximized over 8 ranks.

Output head versus supervision. Position combines selected-head execution with random position pruning. In the separate attribution profile (Table 1), retaining all supervision already saves 8,905 MB; sampling adds only 642 MB. Thus selected-head execution accounts for 93.3% of the saving, vs. 6.7% from supervision deletion. Full-retention rewards match Full-concat exactly, and selected-head and dense-reduced rewards match under identical masks (Appendix § 9.7). The implementation supplies most memory savings, while deletion introduces the loss gap of Proposition 1.

Backward computation. Against Full-concat, Branch reduces step time by 29.9% (1.426× speedup) and peak HBM by 26.7%. Position+Branch reaches 1.443× speedup and 48.6% lower HBM (Figure 2). Its 11,471 MB saving is below the 15,855 MB sum of the separate savings; the cause of this sub-additivity remains unresolved.

Sequence execution. Sequence pruning executes 0.85–0.94 of Full’s input tokens and saves 8–11% peak HBM (Table 2). Across random and fixed cutoffs, steps take 18.430–18.482 s, at most 2.1% below Full. Differences between cutoffs are smaller than the 0.18–0.44 s spread across repeated measurements of one configuration. Token reduction therefore has a much smaller timing effect than branch pruning at this geometry; its physical cause remains unresolved.

Sequence setting	Tokens	HBM (MB)	$\Delta$HBM	Step (s)
Full	1.000×	18,571	—	18.826
$\mathcal{U}(0.25, 1)$	0.854×	16,718	−10.0%	18.480
$\mathcal{U}(0.5, 1)$	0.893×	16,827	−9.4%	18.430
$\mathcal{U}(0.75, 1)$	0.943×	17,072	−8.1%	18.482
fixed 0.75	0.894×	16,561	−10.8%	18.471

Parameter Sensitivity

Table 3 varies one axis at a time using the factorial recipe. Realized pruning fractions are reported in Appendix § 9.§ 4. Sequence input counts include prompt and padding, so fractions are not comparable across axes or directly to $\rho_b$.

Axis	Setting	Resource	Mean accuracy (%)	Mean generation score
Position	$\alpha=0.30$	11,211 MB	57.90	47.73
Position	$\alpha=0.60$	11,320 MB	58.06	48.38
Position	$\alpha=0.90$	11,580 MB	58.32	48.63
Sequence	$\mathcal{U}(0.25, 1)$	16,718 MB	58.09	47.91
Sequence	$\mathcal{U}(0.50, 1)$	16,827 MB	58.04	47.33
Sequence	$\mathcal{U}(0.75, 1)$	17,072 MB	58.04	49.53
Sequence	fixed 0.75	16,561 MB	58.08	47.03
Branch	$q_w=0.30$	127.7 min	58.06	48.80
Branch	$q_w=0.50$	128.4 min	58.08	50.25
Branch	$q_w=0.70$	128.4 min	57.97	48.74

For Position, increasing $\alpha$ from 0.3 to 0.9 raises mean accuracy from 57.90 to 58.32, while peak HBM changes little (11,211–11,580 MB) and AH is non-monotone (Table 10). This weak memory dependence agrees with the implementation-dominated attribution above. Sequence cutoffs track input work and HBM but barely affect step time. Fixed $\rho_b = 0.75$ and $\mathcal{U}(0.5, 1)$ execute similar token fractions yet differ in temporal weights and randomness, so their score difference cannot isolate the Jensen gap. Branch probabilities yield similar training times (127.7–128.4 min) and varying open-ended point estimates. We retain $q_w = 0.5$. HBM was not measured across $q_w$. Position step times are 3.29–3.41 s in its separate paired, interleaved retention profiler; resource comparisons remain within each axis.

The Eight-Configuration Factorial

In Table 4, all four Branch configurations finish in 126–128 min vs. 185–188 min without Branch, preserving a 31–32% time saving across matched contrasts. Position and Sequence retain that benefit when composed with Branch. All 12 single-axis contrasts lower mean accuracy (unrounded contrasts in Appendix § 9.§ 4); Figure 3 shows the resulting time–accuracy tradeoff.

Method	Wall (min)	Mean accuracy (%)	Mean generation score
Full	187.8	58.33	<u>49.33</u>
Position	185.9	58.06	48.38
Sequence	186.4	58.04	47.33
Branch	128.4	<u>58.08</u>	50.25
Position+Sequence	185.0	57.97	48.85
Position+Branch	127.8	57.94	48.17
Sequence+Branch	126.3	57.86	47.67
Position+Sequence +Branch	<u>127.0</u>	57.69	48.35

Open-ended scores move differently. Branch raises MT and WB point estimates with AH close to Full. Configurations containing Position or Sequence fall below Full on AH, while their MT changes depend on the base configuration. Generation means are descriptive single-seed summaries and do not establish score superiority.

Branch Pruning Across Seeds and Models

The matched Full–Branch trainings in Table 5 save 29.9–32.7% wall-clock across two Qwen seeds and Mistral, with mean accuracy lower by 0.25–0.45 pp. In the Mistral profile in Table 6, Branch gives 1.507× speedup and saves 1,347 MB (6.4%) against Full. The time benefit transfers across families.

Config.	Wall (min)	Speedup	Mean acc. (%)	Mean gen. score
Qwen3-4B, seed 42
Full	187.8	1.000×	58.33	49.33
Branch	128.4	1.463×	58.08	50.25
Qwen3-4B, seed 43
Full	187.0	1.000×	58.26	47.64
Branch	131.0	1.428×	57.81	48.49
Mistral-7B, seed 42
Full	211.0	1.000×	44.86	33.17
Branch	142.0	1.486×	44.55	33.93

Configuration	Step (s)	HBM (MB)
Qwen3-4B
Full	18.204	18,571
Branch	12.161	17,319
Mistral-7B
Full	22.438	21,185
Branch	14.888	19,838

MT and WB checkpoint means rise in all three comparisons, while Mistral AH falls. These checkpoint scores are descriptive and do not establish superiority or characterize training-seed variability. Full task scores and scoring coverage are in Appendix D.7. Cross-family validation covers Branch only; Position, Sequence, and their compositions are evaluated on Qwen.

Branch-gradient consistency. We check the conditional expectation in Proposition 2 by enumerating both backward branches. On a small Mistral model in Float64, the probability-weighted average of both branch gradients matches Full to relative L2 error at most $1.7 \times10^{-7}$. On Mistral-7B in BFloat16, the gradient cosine is 0.9997 and the relative L2 error is 0.0236, against 0.0071 for repeated computation of the same configuration, which sets the numerical noise floor for this check. These checks disable LoRA dropout and retain gradient checkpointing using dropout 0.05.

Conclusion

We separate output scoring, sequence execution, and backward computation to locate where stochastic DPO computation saves resources and how it changes optimization. Most output-memory savings come from the implementation rather than from supervision deletion, the branch-pruning time benefit reproduces across Qwen seeds and Mistral, and both accompany lower mean accuracy with mixed open-ended results, so configuration choice should follow the resource constraint and the acceptable objective change.

Limitations

Evaluation covers parameter-efficient text preference training with an online reference on one eight-accelerator setup. Savings may vary with hardware, batch geometry, and reference execution.

Notation

We group the main notation below. Symbols specific to individual proofs are defined where introduced. In single-response results, the side index $b$ is suppressed, so $T$, $k$, $R$, $u_t$ denote the corresponding side quantities.

Preference pairs and loss. $D$ is the preference dataset of prompts $x$, chosen responses $y_w$, and rejected responses $y_l$. The policy $\pi_\theta$ has trainable parameters $\theta$, and $\pi_{\mathrm{ref}}$ is the fixed reference. For side $b \in\{w,l\}$, $T_b$ is the response length, $y_{b,t}$ the target token at position $t$, and $y_{b,<t}$ its response prefix. The token log-ratio $u_{b,t}$ gives the mean reward $R_b$ and margin $\Delta= R_w - R_l$. The scale $\beta$ defines $\ell(d) = \log(1 + e^{-\beta d})$, with per-pair loss $\mathcal{L}_{\mathrm{DPO}} = \ell(\Delta)$ (§ 3.1).

Three pruning axes. Sequence pruning uses cut-off fraction $\rho_b$ to retain prefix $\tilde{y}_b$ of length $M_b$, with minimum fraction $\rho_{\min}$. Position pruning selects $S_b \subseteq\mathbb{I}_{M_b} = \{1,\ldots,M_b\}$ with count $k_b$ controlled by retention $\alpha$. Branch pruning samples backward branch $b^\star$ with probability $q_b$ (§ 3.2 to § 3.4).

Sampling and execution. Prefix, Select, and Branch denote the prefix, position, and branch sampling laws. The active backward branches form $\mathbb{B}$, and $q^\star$ supplies the probability correction. The operator $\operatorname{LogProbs}$ returns scored log-probabilities, with Boolean flag $q$ controlling gradient recording. Its policy and reference outputs are $z_b$ and $z_b^{\mathrm{ref}}$, and $\mathcal{L}$ is the step loss. We write $\sigma$ for sigmoid, $\operatorname{sg}$ for stop-gradient, and $V$ for vocabulary size (§ 3.1, § 3.4 and § 3.5).

Analysis. $\mathbb{E}$ and $\operatorname{Var}$ refer to the indicated sampling. $U_k$ is a uniform size-$k$ position subset, $\phi_t$ a token log-probability gradient, and $\Delta_U$ and $\Delta_{\mathrm{full}}$ the sampled and full margins (Proposition 1). The objectives $F$, $F_A$, $F_P$, $F_{AP}$ correspond to full responses, position pruning, sequence pruning, and their composition, with minimizers $\theta^\star$, $\theta_A^\star$, $\theta_P^\star$, $\theta_{AP}^\star$. The analysis uses token-score and gradient bounds $U$, $G$, variance bound $\bar{v}$, and length bounds $T_0$, $M_0$ (§ 3.2 and § 3.5).

Proofs

This appendix collects the proofs of every stated result: Proposition 1 (Sec. B.1), Lemma 1 (Sec. B.2), Lemma 2 (Sec. B.3), Proposition 2 (Sec. B.4).

Proof of Proposition 1

Proof of Proposition 1. Writing $\mathbb{I}\{E\}$ for the indicator of event $E$, Eq. (5) follows from $\mathbb{E}[\mathbb{I}\{t \in U_k\}] = k/T$ and sampling without replacement. Jensen’s inequality gives the lower bound in Eq. (7). Since $0 \le\ell''(d) \le\beta^2/4$, a second-order upper bound around $\Delta_{\mathrm{full}}$ gives the upper bound. The variance is the finite-population variance of a without-replacement sample, summed over independent sides.

The finite-population identities behind the proposition are as follows. Let $I_t$ indicate membership in a uniform size-$k$ subset of $T$ positions. Then $\mathbb{E}I_t = k/T$, and for $i \ne j$, $\mathbb{E}[I_iI_j] = k(k-1)/[T(T-1)]$. These identities establish Eq. (6) and the variance in Proposition 1. For vector gradients and $T>1$, let $\bar{\phi}=T^{-1}\sum_t\phi_t$ be their population mean and $\Sigma_\phi$ their population covariance. Cov denotes covariance. Then

For $T=1$, the covariance is zero. These identities concern the side-reward gradient. The sampled loss gradient is $-\beta\sigma(-\beta\Delta_U)\nabla_\theta\Delta_U$. Its mask-dependent multiplier means that reward-gradient unbiasedness alone does not imply full-loss-gradient unbiasedness. Equation (6) quantifies the expected loss difference.

Remark (target-logit diagnostics). Let $p_t$ be the current-policy target-token probability at position $t$, and $S$ a selected set. The target-logit diagnostic $\mathcal{E}_z$ (subscript $z$ denotes the target logit) is $\mathcal{E}_z(S)=\frac{1}{|S|^2}\sum_{t\in S}(1-p_t)^2$ on a fixed response. For the full position set $\mathbb{I}_T$, linearity gives

Their ratio is $T/k$, not $k/T$. If a diagnostic instead reports the retained numerator fraction $f(S)=\sum_{t\in S}(1-p_t)^2/\sum_{t=1}^{T}(1-p_t)^2$, its normalized reward-derivative ratio is $(T/k)^2f(S)$ for that response. With varying response lengths and rounded counts, conversion must occur per response before aggregation. Neither quantity measures conditional KL or downstream quality.

Remark (alternative denominator). Keeping the $1/T_b$ denominator would scale the selected mean by the realized fraction $k_b/T_b$. At a common realized fraction $\gamma=k_b/T_b$ on both sides, this equals selected-mean normalization at temperature $\gamma\beta$. We use the $1/k_b$ mean throughout.

Proof of Lemma 1

Proof of Lemma 1. Exchange the order of summation:

For $1\le t<T$, $w_t-w_{t+1}=q(t)/t\ge0$, and $w_T=q(T)/T\ge0$. Also, $\sum_{t=1}^{T}w_t=\sum_{m=1}^{T}q(m)\sum_{t=1}^{m}1/m=1$. Conversely, for a nonnegative, normalized, non-increasing vector $\boldsymbol{w}=(w_1,\ldots,w_T)$, set $w_{T+1}:=0$ and $q(t)=t(w_t-w_{t+1})$. Then $q(t)\ge0$ and $\sum_{t=1}^{T}q(t)=\sum_{t=1}^{T}t(w_t-w_{t+1})=\sum_{t=1}^{T}w_t=1$. Substitution recovers $w_t=\sum_{m=t}^{T}q(m)/m$. Uniform weights $w_t=1/T$ give $q(t)=0$ for $t<T$ and $q(T)=1$, so $M=T$ almost surely, recovering the fixed-cutoff case $\rho_b\equiv1$.

Proof of Lemma 2

Proof of Lemma 2. $\ell(d)=\log(1+e^{-\beta d})$ satisfies $\ell'(d)=-\beta\sigma(-\beta d)$ and $\ell''(d)=\beta^2\sigma(\beta d)\sigma(-\beta d)$. For $\beta>0$, $0<\ell''\le\beta^2/4$ and $\ell'<0$, so $\ell$ is strictly convex and strictly decreasing. For $\beta=0$, $\ell\equiv\log2$ with $\ell'=\ell''=0$. Assume $\beta>0$ unless a degenerate case is named. Jensen’s inequality gives $\mathbb{E}[\ell(\Delta_M)]\ge\ell(\mu_{\mathrm{pref}})$, hence $J_{\mathrm{pref}}:=\mathcal{L}_{\mathrm{pref}}-\ell(\mu_{\mathrm{pref}})\ge0$. For the upper bound, expand $\ell$ to second order around $\mu_{\mathrm{pref}}$: for each scalar margin $x$, $\ell(x)=\ell(\mu_{\mathrm{pref}})+\ell'(\mu_{\mathrm{pref}})(x-\mu_{\mathrm{pref}})+\frac{1}{2}\ell''(\xi)(x-\mu_{\mathrm{pref}})^2$ for some $\xi$ between $x$ and $\mu_{\mathrm{pref}}$. Taking the expectation removes the linear term, and $\ell''\le\beta^2/4$ gives

Subtracting the full-response loss from Eq. (13) gives

Because $\ell$ is strictly decreasing, $\ell(\mu_{\mathrm{pref}})-\ell(\Delta_{\mathrm{full}})$ is positive precisely when the temporally reweighted mean margin $\mu_{\mathrm{pref}}$ is below the full-response margin, which is a data- and distribution-dependent event and therefore carries no fixed sign. Equation (11) gives $\mu_{\mathrm{pref}}=\sum_t w_t^{(w)}u_{w,t}-\sum_t w_t^{(l)}u_{l,t}$, with the side-specific weights $w_t^{(b)}$. Two degeneracies: a deterministic cutoff makes $\Delta_M$ constant, so $\operatorname{Var}(\Delta_M)=0$ and $J_{\mathrm{pref}}=0$. At $\beta=0$ both terms of Eq. (25) vanish.

Finally, the cutoff variables $M_w$, $M_l$ are drawn independently of $\theta$ and have finite support, so the expectation in $\mathcal{L}_{\mathrm{pref}}(\theta)=\mathbb{E}[\ell(\Delta_M(\theta))]$ is a finite sum and differentiation passes through it exactly:

Thus $\nabla_\theta\ell(\Delta_M)$ is an unbiased stochastic gradient of $\mathcal{L}_{\mathrm{pref}}$ and need not coincide in expectation with the full-response gradient. The temporal shift and Jensen gap in Eq. (25) can cancel at a fixed policy. Equality of loss values at one point does not establish equality of gradients.

Proof of Proposition 2

Proof of Proposition 2. Write $\mathcal{L}^{\mathrm{sg}}_w=\zeta(-\beta(R_w-\operatorname{sg}(R_l)))$. Because the stop-gradient blocks the flow into $R_l$, and because $\lambda=\beta\sigma(-\beta(R_w-R_l))$ is evaluated at the current parameters as a chain-rule coefficient (it is differentiated no further),

and symmetrically $\nabla_\theta\mathcal{L}^{\mathrm{sg}}_l=+\lambda\nabla_\theta R_l$. With the $1/q_b$ importance factors, $\mathbb{E}_{b^\star}[\nabla\mathcal{L}_b]=q_w\cdot(-\lambda/q_w)\nabla R_w+q_l\cdot(\lambda/q_l)\nabla R_l=-\lambda\nabla R_w+\lambda\nabla R_l=\nabla_\theta\log(1+e^{-\beta(R_w-R_l)})$. All quantities are those of the sampled response positions and prefixes: with sequence pruning active, $R_b$ is the prefix mean over $M_b$. With position pruning active, it is the masked mean over $S_b$, and the gradients flow only through the selected positions. When position and sequence pruning are disabled, the sampled computation is the full-response objective and the claim recovers the length-normalized loss in Eq. (2).

Remark (branch-estimator variance). With $\lVert\cdot\rVert$ denoting the Euclidean norm, for fixed sampled rewards the single-branch estimator’s second moment is $\lambda^2(\lVert\nabla R_w\rVert^2/q_w+\lVert\nabla R_l\rVert^2/q_l)$. Its variance-optimal probability therefore balances the branch gradient norms, the usual importance-sampling criterion [19]. We use $q_w=0.5$ as a symmetric default. The $q_w\in\{0.3,0.5,0.7\}$ sweep (Table 3) has similar wall-clock times and differing quality point estimates.

Proof of Corollary 1 and the retention rule

Proof of Corollary 1. Since $\ell$ is convex, Jensen’s inequality gives $\mathbb{E}_S[\ell(\Delta_U)]\ge\ell(\mathbb{E}_S[\Delta_U])=\ell(\Delta_{\mathrm{full}})$ at every $\theta$, so $F_A\ge F$ pointwise. Hence

where the first step uses $F\le F_A$, the second the optimality of $\theta_A^\ast$ for $F_A$, and the third Eq. (7) evaluated at $\theta^\ast$. Subtracting $F(\theta^\ast)$ gives Eq. (8).

For Eq. (9), $k_b\ge\alpha T_b$ implies $1/k_b\le1/(\alpha T_b)$ and $T_b-k_b\le(1-\alpha)T_b$, while $v_b\le\bar v$ and $T_b\ge T_0$ give

The display is per response, so averaging over the dataset gives $\mathbb{E}_D[\operatorname{Var}_S(\Delta_U)]\le2\bar v r_A$. Requiring $\frac{\beta^2}{8}\mathbb{E}_D[\operatorname{Var}_S(\Delta_U)]\le\frac{\beta^2\bar v}{4}r_A\le\varepsilon_F$ and solving for $\alpha$ yields Eq. (10).

Proof of Eq. (31) and the stationary displacement

Proof. Fix one preference pair and abbreviate $d=\Delta_{\mathrm{full}}$, $g=\nabla_\theta\Delta_{\mathrm{full}}$, $\delta=\Delta_U-d$, and $e=\nabla_\theta\Delta_U-g$. Unbiasedness gives $\mathbb{E}\delta=0$ and $\mathbb{E}e=0$, so $V_u=\mathbb{E}\delta^2$ and $V_g=\mathbb{E}\lVert e\rVert^2$. Since $\nabla_\theta\ell(\Delta_U)=\ell'(d+\delta)(g+e)$ and $\nabla_\theta\ell(d)=\ell'(d)g$,

The linear term in the first bracket has zero mean, so only $\frac{1}{2}\ell'''(\xi)\delta^2$ survives, and its norm is at most $\frac{1}{2}\sup|\ell'''|V_u\lVert g\rVert$. Because $\ell'''=\beta^3s(1-s)(1-2s)$ with $s=\sigma(\beta d)$ satisfies $\max_s|s(1-s)(1-2s)|=1/(6\sqrt{3})$, we have $\sup|\ell'''|=\beta^3/(6\sqrt{3})$. For the second term, centering with $\mathbb{E}e=0$ and applying Cauchy–Schwarz with $\sup|\ell''|=\beta^2/4$ bounds it by $\frac{\beta^2}{4}\sqrt{V_uV_g}$. Therefore

Averaging over pairs replaces the single-pair quantities by their population counterparts. Substituting $V_u \le2\bar{v}r_A$, $V_g \le2G^2r_A$, and $\|g\| \le2G$ bounds the two terms by $\frac{\beta^3G\bar{v}}{3\sqrt{3}}r_A$ and $\frac{\beta^2G\sqrt{\bar{v}}}{2}r_A$, so that

Now let $\theta_A$ be an interior stationary point of $F_A$ and $\theta^*$ an interior stationary point of $F$, both in the analyzed region, and suppose $\nabla^2F \succeq\mu I$ with $\mu>0$ along the segment joining them. Then $\|\nabla F(\theta_A)\|=\|\nabla F(\theta_A)-\nabla F_A(\theta_A)\|\le C_A r_A$. Integrating the Hessian along that segment gives $\langle\nabla F(\theta_A),\theta_A-\theta^*\rangle\ge\mu\|\theta_A-\theta^*\|^2$; Cauchy–Schwarz together with the previous display yields $\|\theta_A-\theta^*\|\le C_A r_A/\mu$. □

Curvature constants from the margin geometry

Suppose $|\beta\Delta_\theta|\le C$ on the analyzed convex region, $\mathbb{E}_D[\nabla_\theta\Delta\nabla_\theta\Delta^\top]\succeq\nu I$, and $\|\nabla_\theta^2\Delta\|_{\mathrm{op}}\le H$. With $m_C=\sigma(C)\sigma(-C)$, the chain rule gives

because $\ell^{\prime\prime}\ge\beta^2m_C$ on the region and $|\ell'|\le\beta$. Hence $\beta^2m_C\nu>\beta H$ is a sufficient stability condition and one may take $\mu=\beta^2m_C\nu-\beta H$. In a linear-margin model $H=0$, so $\mu>0$ as soon as the comparison matrix has positive smallest eigenvalue $\nu$.

Proof of (14)

Proof. Let $h_b=1-M_b/T_b$. Since $M_b=\max(1,\lceil\rho_bT_b\rceil)$ and $\rho_b\sim\mathcal{U}(\rho_{\min},1)$,

For one realization, split the reward difference into its retained and removed parts:

so $|R_{b,M_b}-R_b|\le(1-\frac{M_b}{T_b})U+(1-\frac{M_b}{T_b})U=2Uh_b$. The same argument with $\|\nabla_\theta u_{b,t}\|\le G$ gives $\|\nabla_\theta R_{b,M_b}-\nabla_\theta R_b\|\le2Gh_b$. The margin difference is the difference of the two sides, so $|\Delta_M-\Delta_{\mathrm{full}}|\le2U(h_w+h_l)$ and $\|\nabla_\theta\Delta_M-\nabla_\theta\Delta_{\mathrm{full}}\|\le2G(h_w+h_l)$.

The loss is $\beta$-Lipschitz, so $|\ell(\Delta_M)-\ell(\Delta_{\mathrm{full}})|\le2\beta U(h_w+h_l)$; taking expectations gives the loss bound of Eq. (14). For the gradient, decompose

Each side’s mean-reward gradient is an average of token gradients of norm at most $G$, so $\|\nabla_\theta\Delta_M\|\le2G$ at every truncation, with no truncation-dependent factor. The first term is therefore at most $\frac{\beta^2}{4}\cdot2U(h_w+h_l)\cdot2G=\beta^2UG(h_w+h_l)$, using $\sup|\ell^{\prime\prime}|=\beta^2/4$, and the second at most $\beta\cdot2G(h_w+h_l)$. Taking expectations and using $\mathbb{E}[h_w+h_l]\le1-\rho_{\min}$ from above gives an expected gradient bias of at most $G(\beta^2U+2\beta)(1-\rho_{\min})$, which is the constant $C_P$ used in Sec. B.10.

For the optimality statement, $F\le F_P+\Delta$ with $\Delta=2\beta U(1-\rho_{\min})$, so $F(\theta_P^*)\le F_P(\theta_P^*)+\Delta\le F_P(\theta^*)+\Delta\le F(\theta^*)+2\Delta$. The displacement bound follows from the segment argument of Sec. B.6 applied to the gradient bound above, and inherits the same requirements: an interior stationary point of each objective lying in the analyzed region, and $\nabla^2F\succeq\mu I$ with $\mu>0$ along the joining segment.

The loss bound is a scaling statement, not a guarantee at the tested setting. Since $|R_b|\le U$ we have $|\Delta|\le2U$, so the total range of the loss on the relevant interval is $\ell(-2U)-\ell(2U)=2\beta U$. At $\rho_{\min}=1/2$ the optimality-gap bound $4\beta U(1-\rho_{\min})$ reaches $2\beta U$, exactly that range, so it is vacuous at the stated cutoff; the loss bound itself is $2\beta U(1-\rho_{\min})=\beta U$, half that range. Both become restrictive only for cutoffs closer to one. □

Proof of Proposition 3 and (42)

Proof. With $c_w,c_l$ as defined in Proposition 3, $\mathbb{E}[\hat{g}]=c_w+c_l$ follows from Proposition 2, and

so $\operatorname{Var}(\hat{g})=\mathbb{E}\lVert\hat{g}\rVert^2-\lVert c_w+c_l\rVert^2$ is Eq. (18). At $q_w=q_l=1/2$ it equals $2\lVert c_w\rVert^2+2\lVert c_l\rVert^2-\lVert c_w+c_l\rVert^2=\lVert c_w-c_l\rVert^2$. Because $|\lambda|=\beta\sigma(-\beta\Delta)\le\beta$ and $\lVert\nabla_\theta R_b\rVert\le G$, we have $\lVert c_w\rVert,\lVert c_l\rVert\le\beta G$, hence $\operatorname{Var}(\hat{g})\le4\beta^2G^2$, and at the symmetric default also $\lVert\hat{g}\rVert=\lVert c_w\rVert/q_w\le2\beta G$. Averaging $m$ estimators divides the variance by $m$ when the $m$ draws are independent in their preference pairs, response positions, prefixes and branches; if only the branch is re-drawn, only the branch component is reduced.

For the rate, take the symmetric default $q_w=q_l=1/2$ so that each step’s noise has second moment at most $4\beta^2G^2/m$. Let $\theta_\Phi^*$ minimize $\Phi$ and $D_n=\mathbb{E}\lVert\theta_n-\theta_\Phi^*\rVert^2$. Projection is non-expansive and $\theta_\Phi^*\in\Theta$, so

Let $A$ be the average Hessian along the segment from $\theta_\Phi^*$ to $\theta_n$. Strong convexity gives $A\succeq\mu I$ and smoothness gives $A\preceq LI$, so $A^2\preceq LA$. Since $\nabla\Phi(\theta_n)=A(\theta_n-\theta_\Phi^*)$,

Substituting this and using $\langle\nabla\Phi,\theta-\theta_\Phi^*\rangle\ge\mu D_n$ together with $\eta_nL\le1$, which $t_0\ge2L/\mu$ ensures,

Insert $\eta_n=2/[\mu(n+t_0)]$, write $s=n+t_0$, and suppose $D_n\le A_B/s$. Then

where the middle step uses $A_B\ge16\beta^2G^2/(\mu^2m)$ and the last uses $(s-1)(s+1)\le s^2$. Since $D_0\le A_B/t_0$, induction gives

for $\eta_n=2/[\mu(n+t_0)]$ and $t_0\ge\max\{2,2L/\mu\}$. Markov’s inequality applied to $\lVert\theta_N-\theta_\Phi^*\rVert^2$ gives $\Pr(\lVert\theta_N-\theta_\Phi^*\rVert>\varepsilon_\theta)\le A_B/[(N+t_0)\varepsilon_\theta^2]$.

Proof of the composed bias and error budget

Proof. Write $r_{A\mid P}=(1-\alpha)/[\alpha(M_0-1)]$. Write $\nabla F_{AP}-\nabla F=(\nabla F_{AP}-\nabla F_P)+(\nabla F_P-\nabla F)$. Conditioned on the executed lengths $(M_w,M_l)$, the first difference is the position-pruning bias of Eq. (32) evaluated at those lengths. A fixed prefix can have a larger token-score variance than the whole response, so this difference must be averaged over the cutoff law rather than bounded prefix by prefix. Two facts supply that average. First, the within-prefix sum of squares cannot exceed the whole-response sum of squares: splitting the total sum of squares into the prefix, the suffix and the part between their means gives

so $Mv_M\le Tv_T$ and hence $v_M\le(T/M)v_T$, with the suffix terms read as zero when $M=T$. Second, for $M_b=\max(1,\lceil\rho_bT_b\rceil)$ with $\rho_b\sim\mathcal{U}(\rho_{\min},1)$, prefixes of length one carry zero variance and the remaining lengths satisfy $\mathbb{E}[\mathbf{1}_{\{M_b>1\}}T_b/(M_b(M_b-1))]\le1/(M_0-1)$ with $M_0=\max(2,\lceil\rho_{\min}T_0\rceil)$. Combining the two bounds the prefix-averaged position-pruning bias by $C_A r_{A\mid P}$, the factor two accounting for the two sides. The second difference is the prefix bias of Eq. (14), contributing $C_P(1-\rho_{\min})$ with $C_P=G(\beta^2U+2\beta)$. Adding the two gives

If the composed objective has interior minimizer $\theta^*_{AP}$ and $F$ has curvature at least $\mu_F$ on the relevant segment, the segment argument of Sec. B.6 gives

The $L^2$ triangle inequality $\lVert\theta_N-\theta^*\rVert\le\lVert\theta_N-\theta^*_{AP}\rVert+\lVert\theta^*_{AP}-\theta^*\rVert$, combined with Eq. (42) applied to the composed objective, yields

where $\mu_F$ bounds the curvature of $F$ on the relevant segment and $A_{B,A_P},t_{0,A_P}$ are the constants of Eq. (42) evaluated for the composed objective. For the composed optimal loss, $F_{A_P}\geq F_P$ by Jensen’s inequality in the position masks and $F\leq F_P+2\beta U(1-\rho_{\min})$. Chaining these as in Sec. B.5 gives

□

Shared-Parameter Analysis

This appendix expands the shared-parameter coupling discussed in the introduction.

Shared-Parameter Updates and Signed Token Movement

Let $i$ index token positions across a pair, $p_i$ be the current-policy target-token probability, and $\phi_i=\nabla_\theta\log p_i$ its log-probability gradient. Set $c_i=1/k_w$ for a selected chosen token, $c_i=-1/k_l$ for a selected rejected token, and $c_i=0$ otherwise. Within a region of fixed mask membership,

For a small gradient-descent step of size $\eta>0$ on the pair loss, the parameter change is $\eta\beta\sigma(-\beta\Delta)g$. Writing $\delta$ for the resulting change, to first order

The pair margin increases to first order, yet an individual chosen probability can decrease if $\phi_i^{\mathsf{T}}g<0$. Token confidence alone does not determine these inner products (Yuan et al., 2025; Razin et al., 2025). Minibatching and AdamW further change the actual update. The single-pair gradient calculation identifies the missing interaction rather than modeling that optimizer exactly.

Experimental Settings and Results

This appendix documents the experimental settings and additional results: the UltraFeedback motivation experiments (Sec. D.1), the HelpSteer3 corpus and recipe (Sec. D.2), the retention-sweep geometry (Sec. D.3), the factorial and parameter sweeps (Sec. D.4), the full per-task tables (Sec. D.5), the detailed HelpSteer3 tables (Sec. D.6), and the supplemental attribution and Full–Branch validation (Sec. D.7).

The UltraFeedback Experiments

The UltraFeedback collection uses sequence length 512, prompt length 256, one training epoch, AdamW, cosine decay, BFloat16 computation, 4-bit loading, gradient checkpointing, and LoRA on attention and MLP projections. The reference is the frozen policy initialization. Qwen thinking mode is disabled for evaluation. Experiments run on Ubuntu 22.04 with two NVIDIA RTX 5090 GPUs (32 GiB each), Python 3.12, PyTorch 2.8.0, and CUDA 12.8. Masks are selected independently on the two sides each step, with the chosen side ranked by current-policy confidence (Low) or uniformly (Random), and gradients bypass the discrete selection.

All three models use LoRA dropout 0.05. The principal comparison uses retention 0.8 and training seed 42 for each rule. Pythia-2.8B uses learning rate $5\times10^{-7}$, DPO $\beta=0.1$, LoRA rank 16 and scaling alpha 32, per-device batch size 2, 16 gradient accumulation steps, and warmup fraction 0.10. Qwen3-4B uses learning rate $1.5\times10^{-6}$, $\beta=0.5$, LoRA rank 64 and scaling alpha 128, per-device batch size 2, 16 gradient accumulation steps, and warmup fraction 0.08. Qwen3-8B uses learning rate $7.5\times10^{-7}$, $\beta=0.5$, LoRA rank 32 and scaling alpha 64, per-device batch size 1, 32 gradient accumulation steps, and warmup fraction 0.10.

The tasks include ARC Challenge (Clark et al., [5]), BBH (Suzgun et al., 2023), MMLU-Pro (Wang et al., 2024), MuSR (Sprague et al., 2024), TruthfulQA MCI (Lin et al., 2022), IFEval prompt-level strict accuracy (Zhou et al., 2023), GSM8K (Cobbe et al., [6]), and MATH-Hard drawn from the MATH dataset (Hendrycks et al., [11]). ARC uses 25-shot normalized accuracy, and GSM8K uses 5-shot exact-answer evaluation. The per-task tables (Table 7 and Table 8) preserve evaluator-reported standard errors, which do not measure training-seed variability. Mean accuracy is a point summary. Pythia’s mean accuracy excludes GSM8K and MATH-Hard because all its variants score zero on both. The summary averages cited in Sec. E.1 are: Low−All +1.95/ + 0.51/ + 0.32, Random−All +1.58/ + 0.00/ + 0.06, and Low−Random +0.37/ + 0.51/ + 0.26 points for Pythia-2.8B, Qwen3-4B, and Qwen3-8B, respectively. The High/High row and the cross-side rules (Low/High, High/Low) are shown for completeness only. They are not part of the matched Low–Random comparison and are not interpreted in the main text. The UltraFeedback and HelpSteer3 collections differ in recipe and evaluation protocol, so absolute scores are not comparable across them.

Benchmark	All/All	Random/Random	High/High	Low/Low
Qwen3-4B
ARC	$62.80 \pm1.41$	$62.97 \pm1.41$	$62.46 \pm1.42$	$62.97 \pm1.41$
BBH	$56.16 \pm3.01$	$56.16 \pm3.01$	$56.08 \pm3.01$	$56.33 \pm3.01$
MMLU-Pro	$42.46 \pm0.45$	$42.25 \pm0.45$	$42.24 \pm0.45$	$42.50 \pm0.45$
MuSR	$42.46 \pm1.73$	$43.39 \pm1.74$	$42.72 \pm1.74$	$43.25 \pm1.74$
TruthfulQA	$37.94 \pm1.70$	$38.07 \pm1.70$	$36.11 \pm1.68$	$38.56 \pm1.70$
IFEval	$43.99 \pm2.14$	$40.67 \pm2.11$	$44.36 \pm2.14$	$43.56 \pm2.10$
GSM8K	$74.15 \pm1.21$	$74.98 \pm1.19$	$68.76 \pm1.28$	$75.36 \pm1.19$
MATH-Hard	$37.94 \pm3.44$	$39.39 \pm3.45$	$37.25 \pm3.36$	$39.50 \pm3.46$
Mean accuracy (%)	49.74	49.74	48.75	50.25
Qwen3-8B
ARC	$68.17 \pm1.36$	$68.09 \pm1.36$	$68.34 \pm1.36$	$68.29 \pm1.36$
BBH	$60.84 \pm2.98$	$60.80 \pm2.98$	$60.55 \pm2.99$	$61.34 \pm2.98$
MMLU-Pro	$47.63 \pm0.46$	$47.70 \pm0.46$	$47.61 \pm0.46$	$47.68 \pm0.46$
MuSR	$43.39 \pm1.77$	$43.78 \pm1.77$	$42.86 \pm1.77$	$43.39 \pm1.77$
TruthfulQA	$36.72 \pm1.69$	$36.96 \pm1.69$	$36.84 \pm1.69$	$36.84 \pm1.69$
IFEval	$25.88 \pm1.88$	$25.32 \pm1.87$	$24.03 \pm1.84$	$26.06 \pm1.89$
GSM8K	$78.09 \pm1.14$	$77.63 \pm1.15$	$77.86 \pm1.14$	$78.62 \pm1.13$
MATH-Hard	$46.25 \pm3.45$	$47.13 \pm3.44$	$46.38 \pm3.49$	$47.29 \pm3.46$
Mean accuracy (%)	50.87	50.93	50.56	51.19

Method	ARC	BBH	MMLU-Pro	MuSR	IFEval	TruthfulQA	Mean accuracy (%)
All/All	$25.51 \pm1.27$	$28.46 \pm2.64$	$10.59 \pm0.28$	$35.85 \pm1.71$	$11.46 \pm1.37$	$22.40 \pm1.46$	22.38
Random/Random	$28.16 \pm1.31$	$27.47 \pm2.55$	$10.86 \pm0.28$	$36.64 \pm1.72$	$15.53 \pm1.56$	$25.09 \pm1.52$	23.96
Low/Low	$27.73 \pm1.31$	$28.47 \pm2.69$	$10.84 \pm0.28$	$39.95 \pm1.73$	$11.46 \pm1.37$	$27.54 \pm1.56$	24.33
Low/High	$24.91 \pm1.26$	$28.92 \pm2.69$	$11.66 \pm0.29$	$35.58 \pm1.70$	$18.30 \pm1.66$	$22.64 \pm1.47$	23.67
High/Low	$26.11 \pm1.28$	$29.47 \pm2.72$	$11.66 \pm0.29$	$34.79 \pm1.69$	$12.57 \pm1.43$	$23.01 \pm1.47$	22.93
High/High	$25.51 \pm1.27$	$29.99 \pm2.76$	$11.16 \pm0.29$	$34.39 \pm1.68$	$12.20 \pm1.41$	$23.13 \pm1.48$	22.73

The HelpSteer3 Corpus and Recipe

Preference conversion and recipe. The preference conversion retains 36,264 of 38,459 training pairs and 1,917 of 2,017 validation pairs, removing ties and empty responses. The main experiments use Qwen3-4B with DPO $\beta= 1$, LoRA rank 64 and alpha 128, dropout 0.05 on attention and MLP projections, BFloat16 computation, FlashAttention2, gradient checkpointing, AdamW at learning rate $10^{-6}$, cosine decay, warmup fraction 0.05, and maximum gradient norm 0.5. Sequence and prompt limits are 2,048 and 1,024. Per-device batch is 2. The factorial and parameter sweeps use training and selection seed 42. The reference is computed online. The supplemental Full–Branch comparisons retain this recipe at Qwen3-4B seed 43 and Mistral-7B seed 42 (Sec. D.7).

Retention-Sweep Geometry

The sweep in Table 12 compares Low/Low and Random/Random at retentions 0.1–0.9, with masks recomputed every step. Gradient accumulation is 16 for retentions 0.1–0.4 and 4 for 0.5–0.9, giving effective batches of 32 (1,133 updates) and 64 (566 updates) for one epoch. Each Low–Random pair within a block shares its geometry and data order. Comparisons are therefore interpreted within each retention level, and the two blocks do not form one controlled dose-response curve. Table 11 gives the full per-task scores.

The Factorial and Parameter Sweeps

Factorial. The factorial of § 4.1 uses the same corpus and recipe with a fixed batch geometry: per-device batch 2, gradient accumulation 4, eight workers, global batch 64 pairs, 566 optimizer steps for one epoch. Position pruning uses $\alpha= 0.6$ (uniform response-position sub-sets, selected-token mean reward). Sequence pruning uses $\rho_b \sim\mathcal{U}(0.5, 1.0)$ (prefix truncation with repacking and a matching reference cutoff). Branch pruning uses $q_w = q_l = 0.5$ (one backward branch per pair, opposite branch forwarded without gradient, $1/q_b$ importance correction and stop-gradient surrogate). Composition follows Alg. 1: truncation first, then response-position pruning inside the retained prefix, then branch pruning. Policy and reference see identical truncations and masks. All eight configurations in Table 4 were trained end-to-end under this recipe and evaluated on all ten benchmarks.

Realized sparsity matches the nominal targets. With all three modes enabled, the selected fraction is 0.6013, the mean $\rho_b$ is 0.7545, and the response retention is 0.7662.

Parameter sweeps. The within-axis sweeps of Table 3 are ten trainings that share the factorial’s recipe and differ from it only in the ablated knob. The sweep midpoints ($\alpha$=0.6, $\mathcal{U}(0.5, 1)$, $q_w$=0.5) coincide with the factorial’s Position, Sequence, and Branch rows. Table 2 gives the sequence-pruning resource profile in full. Realized sparsity matches nominal for position pruning (selected fraction 0.302/0.601/0.902) and branch pruning (global chosen fractions 0.298/0.498/0.699 for $q_w$ = 0.3/0.5/0.7). For sequence pruning, the sampled cutoff matches its distribution (mean $\rho_b$ 0.632/0.755/0.877 against theory 0.625/0.750/0.875, and $\rho_b$ = 0.750 exactly for the fixed-cutoff configuration), with corresponding response-token retention 0.65/0.77/0.88. The Tokens column in Table 2 (0.854/0.893/0.943/0.894) is a third quantity: the executed-sequence fraction, the ratio of executed sequence tokens, prompt and padding included, to the stock baseline. It lies above the response-token retention and must not be compared with the nominal $\rho_b$. Peak HBM was not measured across $q_w$, so branch pruning is compared on training wall-clock only. The per-task scores of all ten runs are in Table 10.

Mean generation score in every table uses the reported task means and the scaling in § 4.1.

Descriptive factorial contrasts. Raw differences of aggregate columns across the factorial are descriptive contrasts on single-seed runs. Contrasts use unrounded aggregates and can differ from differences of the displayed columns by 0.01. Adding position pruning changes mean accuracy by $-0.27$, $-0.07$, $-0.14$, and $-0.17$ pp over Full, Sequence, Branch, and Sequence+Branch, respectively. Adding sequence pruning changes it by $-0.29$, $-0.09$, $-0.22$, and $-0.25$ pp over Full, Position, Branch, and Position+Branch. Adding branch pruning changes it by $-0.26$, $-0.12$, $-0.19$, and $-0.28$ pp over Full, Position, Sequence, and Position+Sequence.

Contrasts in mean generation score use the scaling in § 4.1 and have mixed signs. Position pruning gives $-0.95$, $+1.52$, $-2.07$, and $+0.68$ in the four pairings above. Sequence pruning gives $-2.01$, $+0.47$, $-2.58$, and $+0.17$. Branch pruning gives $+0.92$, $-0.21$, $+0.35$, and $-0.50$. All twelve mean accuracy contrasts are negative. The branch-pruning effect on training wall-clock is consistent in sign: $-59.4$, $-58.1$, $-60.1$, and $-58.0$ min in the four pairings.

Full Task Tables

Table 11 reports per-task scores for Low/Low and Random/Random at each retention level on HelpSteer3-Preference.

Detailed HelpSteer3 Tables

The following tables retain the count-matched motivation results and the per-axis parameter settings summarized in the main text; the completed factorial is in Table 4. All factorial and sweep configurations use seed 42.

Supplemental Attribution and Full–Branch Validation

The attribution and Full–Branch validation add four complete trainings and two resource profiles on the same eight-accelerator hardware. Results are in Table 1, Table 5 and Table 6; this section gives reproduction and scoring details.

Output implementation control. One Qwen profiler compares Full-concat, selected-head execution at retention 1.0, and selected-head execution at 0.6. The configurations share weights, inputs, reference computation, precision, and the $2 \times4 \times8$ training geometry (per-device batch 2, gradient accumulation 4, eight workers). Four warm-up rounds precede 16 measured rounds. Configuration order rotates within rounds, memory counters reset before each measurement, and peak HBM is the maximum over ranks. Full-retention selected-head rewards match Full-concat exactly, as do dense-reduced and selected-head rewards under identical masks. Numerical checks also compare LoRA gradients with a row-order control. Mean step times and peak HBM are reported in Table 1. Memory differences are computed within this profile: 8,905 MB for full-supervision selected-head execution and a further 642 MB for random deletion.

Matched training pairs. Qwen Full and Branch at seed 43 use the same initialization, dataset version, data order, training budget, optimizer, schedule, precision, attention backend, checkpointing, and online reference. Mistral-7B-Instruct-v0.3 uses the same recipe and 566-step budget at seed 42. Both models fit the $2 \times4 \times8$ geometry without a batch-size change. Branch pruning uses $q_w=q_l=0.5$, with position and sequence pruning disabled. The global chosen-branch fractions are 0.5088 for Qwen seed 43 and 0.4956 for Mistral, averaged over all optimizer steps.

The Mistral setup is verified on chat template, prompt masking, EOS padding, the online reference, and the selected/no-gradient branches. The gradient checks in § 4.5 disable LoRA dropout while retaining checkpointing; the trainings use dropout 0.05.

Per-task results. Table 13 retains every benchmark score for the repeated Qwen pair and the Mistral pair. Table 13 retains the native open-ended scores; Table 5 reports category means. The conventional Branch-minus-Full differences are −0.25 and −0.45 pp for the two Qwen seeds (a descriptive mean of −0.35 pp), and −0.31 pp for Mistral.

Open-ended scoring. All supplemental generations are judged by MiniMax-M3. The evaluator reports valid-item means separately for each checkpoint. The supplemental WildBench suite has 260 prompts, four more than the main evaluation’s 256. The Qwen seed-43 Full Arena-Hard generation has 499 judged entries and 443 valid scores. Table 13 reports per-checkpoint scores. These summaries do not characterize training-seed variability.

Additional Token-Selection Analysis

This section reports the matched token-selection comparisons referenced in the main text.

Supervision Membership Is a Limited Lever

We compare selecting positions with the lowest current-policy target-token probabilities (Low) against uniform random selection (Random). Low/Low and Random/Random use the same selection rule on both chosen and rejected responses. At equal retained ratios, Low/Low exceeds Random/Random by 0.37, 0.51, and 0.26 mean-accuracy points on UltraFeedback ([7]) for Pythia-2.8B ([1]), Qwen3-4B, and Qwen3-8B. On HelpSteer3-Preference, the difference changes sign across retention levels and stays within 0.32 points (Table 12). Task differences are larger: at low retention, Low trails Random on MATH-Hard while leading TruthfulQA (Table 11). These comparisons motivate uniform position pruning before vocabulary projection. We compare selection rules at a fixed number of scored positions. The factorial experiment separately assesses the model-quality cost of reduced supervision relative to the Full baseline.

LLM Usage Disclosure

In answer to the submission questions on the use of large language models, a large language model was used to aid and polish the writing of this paper, for retrieval and discovery such as finding related work, and to draft and revise sections of the manuscript.