betaFeedback

WorksComputer Science

Robust Domain-Mixture Scheduling for Multi-Teacher On-Policy Distillation

Abstract

Multi-teacher on-policy distillation (MOPD) merges the capabilities of multiple domain-expert teachers, such as mathematical reasoning, code generation, and instruction following, into a single compact student. Since all domains share a limited training budget, how to allocate it across domains directly determines the quality of distillation. Existing methods allocate the budget by each domain's learning progress, but offer no guarantee for lagging domains. We therefore introduce distributionally robust optimization (DRO) into MOPD to prioritize the worst-performing domain. However, the per-domain training signals are estimated from a few rollouts and are inherently noisy, so directly applying DRO mistakes noise for the worst case, lets occasional spikes hijack the budget, and protects inherently hard domains rather than truly regressing ones. To address these issues, we propose RDS-MOPD, which keeps the DRO objective and redesigns its three parts: it shifts the allocation only when differences clearly exceed the noise, bounds the influence of any single anomaly, and defines the worst case as the regression from each domain's own best level, all with theoretical guarantees. Experiments show that RDS-MOPD outperforms recent competing methods and ranks first on most mathematical reasoning, code generation, and instruction-following benchmarks.

Introduction

As large language model (LLM) services become deeply embedded in various real-world applications, users often issue requests of vastly different types within one continuous session, and expect the same model to both handle every task well and keep the conversation coherent [44]. Deploying a dedicated model for each type of task multiplies the inference and maintenance cost, and breaks the coherence of the session whenever the task switches. Consequently, merging multiple expert capabilities into a single compact student model has become a fundamental requirement for large-scale deployment [9]. Multi-teacher on-policy distillation (MOPD) is a promising post-training paradigm toward this goal [7, 32] (Ma et al. 2026), where the student learns from multiple domain-expert teachers simultaneously on its own sampled responses. Compared to offline sequence-level distillation, on-policy distillation provides dense supervision directly on the student’s own distribution, which alleviates the distribution shift between training and inference [1] (Gu et al. 2024).

In MOPD, all domains (e.g., mathematics, code, and instruction following (IF)) share a limited training budget. Hence, the domain mixture, i.e., how much the student learns from each teacher at every step, directly determines the quality of distillation. Open-MOPD [7] diagnoses the capability imbalance in multi-teacher distillation, and proposes to balance the token shares and follow the per-domain KL gaps. More recently, D3^{3}-MOPD [32] dynamically allocates the budget according to the learning progress of each domain (i.e., the remaining KL gap and its descent velocity), steering the budget toward the domains that benefit the most. However, these methods all aim to accelerate the overall progress: they only favor “which domain improves the fastest” and offer no guarantee for lagging or regressing domains. In multi-capability merging, what users perceive is often the weakest capability, and the lag of any single capability directly hurts the service quality.

To this end, we introduce distributionally robust optimization (DRO) into domain-mixture scheduling for MOPD. DRO makes decisions against the most unfavorable case within an uncertainty set, and thereby prioritizes the worst-performing group [29, 16]. This idea naturally matches the goal of “leaving no capability behind”. Different from prior work that determines the pre-training data mixture offline with a proxy model [37], to the best of our knowledge, this is the first work that applies DRO to online mixture scheduling in MOPD. Nevertheless, directly applying DRO to MOPD (referred to as Vanilla DRO-MOPD) is confronted with a fundamental issue: every decision of DRO relies on the estimated per-domain training signals, which are computed from a limited number of rollouts per domain and are inherently noisy. As shown in Figure 1(a), on the training trajectory of our Vanilla DRO-MOPD implementation, the weight of the code domain surges after a KL burst and stays high until the end of training, even though code is no longer the worst-performing domain. Therefore, it is non-trivial to make reliable robust decisions under noisy training signals, and Vanilla DRO-MOPD mainly suffers from the following three instabilities (diagnosed in § 3.1).

Figure 1. Domain weights on the training trajectory. (a) Vanilla DRO-MOPD: a code KL burst (shaded) lifts the code weight from 0.30 to 0.56, which locks in (hatched); (b) RDS-MOPD keeps re-allocating the budget without lock-in.

  • Worst-Case Misjudgment. Vanilla DRO-MOPD moves its weights at the same rate whether the evidence is strong or weak, so it systematically up-weights the domain whose noisy estimate happens to be high. On the above trajectory, the current worst domain changes 52 times within 145 training steps, and on more than half (54%) of the non-spike steps, the gap between the worst and runner-up domains is below one noise standard deviation.

  • Spike Hijacking. Every observation enters the weights of Vanilla DRO-MOPD in proportion to its magnitude and is carried forward by all later updates, so an abnormal KL spike can seize a large share of the budget. On the above trajectory, the KL of the code domain rises to about 30× its typical level within a dozen steps, and its weight nearly doubles and stays high until the end.

  • Misplaced Protection. DRO is intended to protect the weak domains, but it compares the magnitude of losses, while the KL values given by different teachers naturally differ in scale. As a result, the budget flows to the domains with inherently large KL rather than the truly regressing ones: with the raw KL as the loss, Vanilla DRO-MOPD eventually allocates the whole budget to the code domain, whereas the IF domain, which regresses the most from its own best level, receives almost no protection.

To tackle these challenges, we propose a novel framework, namely Robust Domain-Mixture Scheduling for Multi-Teacher On-Policy Distillation (RDS-MOPD). Our key observation is that the three instabilities do not stem from the idea of DRO itself; instead, Vanilla DRO-MOPD builds all three of its parts (i.e., the deviation radius, the nominal allocation, and the definition of the “worst case”) upon unreliable signals. Hence, RDS-MOPD keeps the DRO objective and redesigns its three parts with three modules. Specifically, (1) to handle worst-case misjudgment, we propose Noise-aware Radius Calibration (NRC), which calibrates the deviation radius to the noise level, so that the evidence strength decides how far the allocation may deviate. It lower-bounds the temperature of the nominal allocation via the online-estimated noise floor, so that differences within the noise range only cause bounded share changes, while truly significant ones still receive a decisive response. (2) To handle spike hijacking, we design Spike-robust Allocation Estimation (SAE), which rebuilds the nominal allocation from KL observations clipped by a log-domain Huber rule, bounding the influence of spikes while still following persistent real changes. Meanwhile, SAE re-estimates the nominal allocation at every update rather than accumulating it from previous decisions, so an anomaly no longer persists in the budget. (3) To handle misplaced protection, we propose Worst-case Domain Reweighting (WDR), which redefines the “worst case” as each domain’s regression from its own best level, and exactly solves DRO around the progress-driven nominal allocation. It thus reweights the domains toward the truly regressing ones with a bounded amount of extra budget. As shown in Figure 1(b), under the same setting, RDS-MOPD keeps re-allocating the budget throughout training, and no domain is locked in. Theoretically, we prove that NRC and SAE guarantee bounded share changes and bounded influence of anomalies, respectively, and that WDR exactly solves a KL-ball DRO problem.

We distill a 3B student model from three domain-expert teachers for mathematics, code, and instruction following, and conduct extensive experiments on six benchmarks. The results show that RDS-MOPD outperforms recent competing methods, including Open-MOPD [7] and D3D^3-MOPD (Sun et al., [32]), and ranks first on the majority of benchmarks. RDS-MOPD also makes no large share reversal in training, and in a stress test that injects noise into the code teacher’s signal, it still keeps a stable mixture and protects the learning progress of the other domains. These results demonstrate both the superiority of RDS-MOPD and the effectiveness of its modules. To summarize, our main contributions are threefold:

  • To the best of our knowledge, we are the first to introduce DRO into domain-mixture scheduling for MOPD, and formally diagnose three instabilities of Vanilla DRO-MOPD under noise.

  • We propose RDS-MOPD, a single DRO structure whose three modules fix these instabilities with theoretical guarantees.

  • Extensive experiments show that RDS-MOPD ranks first on most benchmarks, and validate the effectiveness of its modules.

Preliminaries

We introduce MOPD and its domain-mixture scheduling problem. Bold lowercase letters denote vectors (e.g., q\mathbf{q}) and calligraphic letters denote sets (e.g., D\mathcal{D}). ΔK\Delta^K denotes the probability simplex over KK domains, u=1/K\mathbf{u} = 1/K the uniform allocation (1\mathbf{1} is the all-ones vector), and KL⁡(⋅∥⋅)\operatorname{KL}(\cdot\parallel\cdot) the Kullback–Leibler divergence. We index training steps by ss and scheduling updates by tt (omitted when clear from the context), and summarize the notations in Table 3 of Appendix § 8.

NotationMeaning
ΔK\Delta^{K}, uu, 1\mathbf{1}, ⊙\odotprobability simplex; uniform allocation 1/K1/K; all-ones vector; element-wise product
D\mathcal{D}, dd, KKdomain set, a domain, number of domains (K=∣D∣K = |\mathcal{D}|)
πθ\pi_{\theta}, πϕd\pi_{\phi_d}, Pd\mathcal{P}_dstudent; RL teacher routed to domain dd; prompt pool of dd
xx, yy, ∣y∣|y|, jjprompt; on-policy student response; its length; token position
DjD_j, kk, Ld(θ)\mathcal{L}_d(\theta)top-kk reverse KL at position jj (reward −Dj-D_j); top-kk size; distillation loss of dd
ss, tt, Δ\Delta, BB, SStraining step; scheduling update; update interval; prompts per batch; total steps
m~d(s)\widetilde{m}_d^{(s)}, m~d(s)\widetilde{m}_d^{(s)}, BdB_dper-domain KL observation of dd at step ss; its clipped value in SAE; domain-dd responses in the batch
m‾d(s)\overline{m}_d^{(s)}, m‾d\overline{m}_d, ξd(s)\xi_d^{(s)}true KL level of dd at step ss; best true level so far; log-noise
q(t)q^{(t)}, ppdomain mixture at update tt; nominal allocation (progress-driven in RDS-MOPD)
ℓ^d(s)\widehat{\ell}_d^{(s)}loss estimate in Eqn. (2) (md/Idm_d/I_d in Vanilla DRO-MOPD)
IdI_d, ℓd\ell_d, ℓd⋆\ell_d^{\star}initial KL level; current EMA level; best (lowest) level so far
gdg_d, vdv_d, rdr_d, rmax⁡r_{\max}remaining gap; descent velocity; progress score (rd=gdvdr_d = g_dv_d); largest score
WW, n0n_0, KpK_p, JJEMA window; points for IdI_d; max./used rolling deltas
ωd(j)\omega_d^{(j)}, ω\omega, ϱd,j\varrho_{d,j}EMA of window jj; EMA weights; ratio ℓd[j]/ℓd[j+1]\ell_d^{[j]}/\ell_d^{[j+1]}
TT, τ(t)\tau^{(t)}, ff, α\alpha, zdz_dbase/effective temperature; share floor; jitter range/value
σd\sigma_d, σ^\widehat{\sigma}, NσN_{\sigma}, ϵ\epsilonlog-noise scale; SE of the leading score; noise window; noise-floor multiplier
μ~(s)\widetilde{\mu}^{(s)}, HH, ccrunning median; median window; Huber clipping threshold
hdh_d, κ\kappa, η\eta, λlo\lambda_{\mathrm{lo}}excess loss; excess cap; DRO (tilt) strength; coverage floor
U\mathcal{U}, λd\lambda_d, ν\nufeasible set; per-domain floor; normalizing constant
ζ\zeta, s0s_0, MMscore bound in SEs; step of an anomaly or jump; anomaly size
ρ\rho, γ\gammaKL radius; dual variable (proof of Theorem 3.3)

Table 3. Notations and meanings.

Multi-Teacher On-Policy Distillation

Let D\mathcal{D} be the set of domains (here, D={math,code,IF}\mathcal{D} = \{\text{math}, \text{code}, \text{IF}\}) with K=∣D∣K = |\mathcal{D}|. Each domain dd has a prompt pool Pd\mathcal{P}_d and an RL-trained expert teacher πϕd\pi_{\phi_d}. MOPD distills all teachers into one student policy πθ\pi_\theta, and routes each prompt xx to the teacher of its domain d(x)d(x). Given xx, the student samples a response y∼πθ(⋅∣x)y \sim\pi_\theta(\cdot\mid x) on-policy, and the routed teacher scores it with a single prefill. At each position jj of yy, the teacher computes the reverse KL DjD_j from the next-token distribution of the student to its own, restricted to the student’s top-kk tokens (Gao et al., [7]). The dense token-level reward −Dj-D_j serves as the advantage of a critic-free policy-gradient update (Schulman et al., [30]). The distillation loss Ld(θ)\mathcal{L}_d(\theta) is the expected per-token DjD_j over the domain-dd prompts and their on-policy responses. At training step ss, the per-domain KL signal is computed as:

md(s)=1∣Bd∣∑y∈Bd1∣y∣∑j=1∣y∣Dj(1)m_d^{(s)} = \frac{1}{|\mathcal{B}_d|}\sum_{y\in\mathcal{B}_d}\frac{1}{|y|}\sum_{j=1}^{|y|}D_j \tag*{(1)}

where Bd\mathcal{B}_d holds the domain-dd responses of step ss, and ∣y∣|y| is the length of yy. As a one-batch estimate of Ld\mathcal{L}_d, md(s)m_d^{(s)} reuses the rewards already computed by the trainer (no extra forward pass), but is noisy, as each batch holds a limited number of rollouts per domain.

Domain-Mixture Scheduling

Every few training steps, a scheduler updates the domain mixture q(t)∈ΔK\mathbf{q}^{(t)} \in\Delta^K from the observed KL signals. Each batch then samples prompts in proportion to q(t)\mathbf{q}^{(t)}, with a random jitter (Appendix § 11). Between two scheduling updates, the student thus minimizes ∑dqdLd(θ)\sum_d q_d\mathcal{L}_d(\theta); q\mathbf{q} only decides how many prompts each domain receives, and the per-token loss is unchanged. Some schedulers fix the mixture before training, e.g., by prompt-pool size (Ma et al., 2026), or determine it offline with a proxy model (Xie et al., [37]). Others adapt to each domain’s learning progress online, e.g., Open-MOPD (Gao et al., [7]) reweights the per-domain rewards by token-share balancing and KL-gap following, and D3D^3-MOPD (Sun et al., [32]) resamples domains by the remaining KL gap times its descent velocity.

Our RDS-MOPD Methodology

We first formulate DRO for MOPD and diagnose Vanilla DRO-MOPD (§ 3.1), then present RDS-MOPD with its three modules (§ 3.2, Figure 2) and its overall procedure (§ 3.6).

Figure 2. Illustration of our RDS-MOPD framework (left) and its robust scheduler (right).

DRO for MOPD

DRO FORMULATION FOR MOPD

DRO decides against the most unfavorable distribution within an uncertainty set around a nominal one [2, 4]. In MOPD, where the student minimizes ∑dqdLd\sum_d q_d\mathcal{L}_d (§ 2.2), a DRO scheduler picks the most unfavorable mixture, up-weighting domains with a higher estimated loss. With a KL penalty around a nominal allocation p\mathbf{p}, the DRO decision is:

q=arg⁡max⁡q′∈U{⟨q′,ℓ^⟩−1ηKL⁡(q′∥p)}(2)\mathbf{q} = \mathop{\arg\max}_{\mathbf{q}'\in\mathcal{U}}\left\{\langle\mathbf{q}',\hat{\boldsymbol{\ell}}\rangle-\frac{1}{\eta}\operatorname{KL}(\mathbf{q}'\mathbin\Vert\mathbf{p})\right\} \tag*{(2)}

where U⊆ΔK\mathcal{U}\subseteq\Delta^K is the feasible set. Eqn. (2) has three parts: (i) the deviation radius, set by η\eta (a larger η\eta lets q\mathbf{q} move farther); (ii) the nominal allocation p\mathbf{p}, the anchor that q\mathbf{q} moves from; and (iii) the definition of the worst case, i.e., the loss estimate ℓ^\hat{\boldsymbol{\ell}} of (Ld)d∈D(\mathcal{L}_d)_{d\in\mathcal{D}}.

Vanilla DRO-MOPD applies online Group DRO [29] to MOPD with each domain as a group, where q\mathbf{q} may reweight either the losses or the sampling. At every training step ss (so t=st=s), it solves Eqn. (2) with U=ΔK\mathcal{U}=\Delta^K, whose closed form is qd∝pdeηℓ^dq_d\propto p_d e^{\eta\hat{\ell}_d}. Its nominal allocation is the previous weights, p=q(s−1)\mathbf{p}=\mathbf{q}^{(s-1)} with q(0)=u\mathbf{q}^{(0)}=\mathbf{u}, and its loss is the KL signal normalized by its initial level IdI_d (early mean of mdm_d), ℓ^d(s)=md(s)/Id\hat{\ell}^{(s)}_d=m^{(s)}_d/I_d, giving the exponentiated-gradient update:

qd(s)∝qd(s−1)eηℓ^d(s).(3)q_d^{(s)}\propto q_d^{(s-1)}e^{\eta\hat{\ell}^{(s)}_d}. \tag*{(3)}

INSUFFICIENCY OF VANILLA DRO-MOPD

We model each observation as its true level times a noise factor, i.e., md(s)=mˉd(s)eξd(s)m_d^{(s)}=\bar{m}_d^{(s)}e^{\xi_d^{(s)}}, where the zero-mean log-noise ξd\xi_d may be heavy-tailed and differ in scale across domains. Unrolling Eqn. (3) from q(0)=u\mathbf{q}^{(0)}=\mathbf{u}, taking the log-ratio of two domains dd and d′d' (which cancels the normalization), and writing eξd=1+(eξd−1)e^{\xi_d}=1+(e^{\xi_d}-1) splits it into a signal term and a noise term summed over steps s′≤ss'\leq s:

log⁡qd(s)qd′(s)=η∑s′≤s(mˉd(s′)Id−mˉd′(s′)Id′)+η∑s′≤s(mˉd(s′)Id(eξd(s′)−1)−mˉd′(s′)Id′(eξd′(s′)−1)).(4)\log\frac{q_d^{(s)}}{q_{d'}^{(s)}}= \eta\sum_{s'\leq s}\left(\frac{\bar{m}_d^{(s')}}{I_d}-\frac{\bar{m}_{d'}^{(s')}}{I_{d'}}\right) +\eta\sum_{s'\leq s}\left(\frac{\bar{m}_d^{(s')}}{I_d}\left(e^{\xi_d^{(s')}}-1\right)-\frac{\bar{m}_{d'}^{(s')}}{I_{d'}}\left(e^{\xi_{d'}^{(s')}}-1\right)\right). \tag*{(4)}

Eqn. (4) shows how each part breaks under the noise. (1) Worst-case misjudgment (deviation radius). The radius, fixed by η\eta, ignores the noise level, so the noise term accumulates like a random walk whose spread grows with the number of steps, and even drifts toward the noisier domain since E[eξ]≥1\mathbb{E}[e^{\xi}]\geq1. (2) Spike hijacking (nominal allocation). Anchoring on the previous weights accumulates every observation, so a single observation md(s0)=Mm_d^{(s_0)}=M leaves a term ηM/Id\eta M/I_d in the log-weight ratio of domain dd to any other domain at all later steps, which grows linearly in MM and never fades (e.g., the code burst in Figure 1(a)). (3) Misplaced protection (definition of the worst case). Let mˉd∗\bar{m}_d^* be the lowest true level of domain dd so far; the true level then splits into three factors:

mˉd=Id⏟teacher scale⋅mˉd∗Id⏟descent⋅mˉdmˉd∗⏟regression.(5)\bar{m}_d= \underbrace{I_d}_{\text{teacher scale}}\cdot\underbrace{\frac{\bar{m}_d^*}{I_d}}_{\text{descent}}\cdot\underbrace{\frac{\bar{m}_d}{\bar{m}_d^*}}_{\text{regression}}. \tag*{(5)}

Normalizing by IdI_d removes only the teacher scale, so a domain that descended far but then regresses still ranks below a barely descended one. Thus, the problem lies not in the robust objective but in its three parts, calling for a noise-aware radius, a spike-robust nominal allocation, and a worst case defined by regression alone.

RDS-MOPD

RDS-MOPD keeps the robust objective and redesigns its three parts. At its tt-th update, it solves:

q(t)=arg⁡max⁡q∈U{⟨q,h⟩−1ηKL⁡(q∥p)}(6)\mathbf{q}^{(t)}=\mathop{\arg\max}_{\mathbf{q}\in\mathcal{U}}\left\{\langle\mathbf{q},\mathbf{h}\rangle-\frac{1}{\eta}\operatorname{KL}(\mathbf{q}\mathbin\Vert\mathbf{p})\right\} \tag*{(6)}

where p\mathbf{p} is the nominal allocation, h\mathbf{h} the excess loss, η\eta the tilt strength, and U\mathcal{U} the feasible set, i.e., Eqn. (2) with h\mathbf{h} as the loss.

The nominal allocation is recomputed at every update from the progress score [32] rd=vdℓd/Idr_d=v_d\ell_d/I_d of each domain, where ℓd\ell_d is the exponential moving average (EMA) of the clipped KL of domain dd (§ 3.4), ℓd/Id\ell_d/I_d is the remaining gap, and the descent velocity vd≥0v_d\geq0 is the relative decrease of ℓd\ell_d over recent steps (Appendix § 11):

p=f1+(1−Kf)softmax⁡(r/τ),(7)\mathbf{p}=f\mathbf{1}+(1-Kf)\operatorname{softmax}(\mathbf{r}/\tau), \tag*{(7)}

where ff is the share floor and τ\tau the temperature, so a domain with a large remaining gap that still descends fast receives a larger share. The softmax solves Eqn. (2) with r\mathbf{r}, u\mathbf{u}, and τ\tau in place of ℓ^\hat{\boldsymbol{\ell}}, p\mathbf{p}, and 1/η1/\eta. Hence, τ\tau sets the radius that turns noisy scores into shares (a larger τ\tau keeps p\mathbf{p} closer to u\mathbf{u}), while η\eta only bounds the step from p\mathbf{p} to q\mathbf{q}.

Accordingly, three modules redesign the three parts: (i) NRC calibrates the radius τ\tau to the noise level (§ 3.3); (ii) SAE re-estimates p\mathbf{p} at every update from clipped observations instead of the previous weights (§ 3.4); (iii) WDR defines the worst case as each domain’s excess loss h over its best level and solves Eqn. (6) exactly (§ 3.5).

Noise-Aware Radius Calibration (NRC)

NRC fixes worst-case misjudgment: a difference that does not clearly exceed the noise should not cause a large change in the allocation. A common choice is the max-normalized temperature τ=T⋅rmax⁡\tau= T \cdot r_{\max} ([32]), where TT is the base temperature and rmax⁡=max⁡drdr_{\max} = \max_d r_d is the score of the leading domain (the leader). The softmax input is then (r/rmax⁡)/T(r/r_{\max})/T, so the leader always has a normalized score of 1, whether its lead is far beyond the noise or only a noise artifact. Near a plateau, the velocities and thus rmax⁡r_{\max} approach 0, so the leader switches with the noise, and each switch moves a large share. NRC therefore calibrates the radius to the uncertainty of the leading score.

Noise-Calibrated Radius. We estimate the scale of the log-noise ξd\xi_d (§ 3.1.2) of each domain from the recent log-differences of its clipped observations (§ 3.4). The delta method then carries this scale to the standard error (SE) of each score (Appendix § 11). Let σ^\hat{\sigma} be the SE of the leading score, and ϵ\epsilon the noise-floor multiplier. NRC lower-bounds rmax⁡r_{\max} by the noise floor ϵσ^\epsilon\hat{\sigma}:

τ=T⋅max⁡(rmax⁡,ϵσ^).(8)\tau= T \cdot\max\left(r_{\max}, \epsilon\hat{\sigma}\right). \tag*{(8)}

Thus, as the evidence weakens, τ\tau stays large, and p moves smoothly toward u instead of following a noisy leader (Figure 3(a)).

Figure 3. Three modules on the mixture simplex. (a) NRC sizes the KL ball around u by evidence; without it, p flips between vertices. (b) SAE keeps p bounded after a code spike; without it, p is dragged to the code vertex. (c) WDR’s solution q lies where the highest iso-line of ⟨q,h⟩\langle q, h\rangle touches the KL ball of p.

Proposition 3.1 (Bounded Share Change below the Noise Level). Let p be given by Eqns. (7) and (8) with ϵ>0\epsilon> 0. Suppose that, at two consecutive updates, all scores satisfy 0≤rd≤ζσ^0 \le r_d \le\zeta\hat{\sigma} for some 0<ζ≤ϵ0 < \zeta\le\epsilon, with σ^>0\hat{\sigma} > 0 taken at the same update. In this case, no score exceeds ζ\zeta SEs, and the noise floor is active. Then every nominal share changes by at most (1−Kf)ζ2ϵT\frac{(1-K_f)\zeta}{2\epsilon T}.

Proposition 3.1 (proved in Appendix § 15.§ 1) guarantees that differences within the noise level only cause bounded share changes, and the weaker the evidence, the smaller the change.

Spike-Robust Allocation Estimation (SAE)

SAE fixes spike hijacking: one abnormal observation should not dominate the allocation. Re-estimating the nominal allocation at every update stops a spike from accumulating, but the spike still enters every statistic built on linear averages (e.g., IdI_d, ℓd\ell_d, and vdv_d) in proportion to its size. SAE therefore first clips every new observation into a range around the recent median.

Log-Domain Huber Clipping. Since the noise is multiplicative (§ 3.1.2), we clip each new observation md(s)m_d^{(s)} in the log domain, i.e., by a ratio to the median of the raw observations over the previous HH steps, μd(s)=median⁡{md(s−H),…,md(s−1)}\mu_d^{(s)} = \operatorname{median}\{m_d^{(s-H)},\ldots,m_d^{(s-1)}\} (fewer in warm-up):

m~d(s)=clip⁡(md(s),e−cμd(s),ecμd(s)),(9)\tilde{m}_d^{(s)} = \operatorname{clip}\left(m_d^{(s)}, e^{-c\mu_d^{(s)}}, e^{c\mu_d^{(s)}}\right), \tag*{(9)}

where clip⁡(x,a,b)\operatorname{clip}(x,a,b) restricts xx to [a,b][a,b], and cc is the clipping threshold. Normal fluctuations pass unchanged, and a spike keeps its direction while its size is bounded. This is the Huber ψ\psi-function ([10]) in the log domain, and all other statistics of RDS-MOPD are computed on the clipped sequence. Since the median uses the raw observations, a single anomaly cannot drag the center.

Proposition 3.2 (Bounded Influence and Trend Preservation). Let H≥3H \ge3 and c>0c > 0. (i) Every clipped observation lies in [e−cμd(s),ecμd(s)][e^{-c\mu_d^{(s)}}, e^{c\mu_d^{(s)}}] for s>1s > 1. (ii) Replacing one raw observation md(s0)m_d^{(s_0)} with s0>Hs_0 > H by any M>0M > 0 changes the clipped sequence only at steps s0,…,s0+Hs_0,\ldots,s_0+H, and only within a range independent of MM. Hence any EMA level of the clipped sequence stays in a range independent of MM, whereas without clipping it grows linearly in MM. (iii) If the observations are noise-free and jump at step s0s_0 from one constant level to another, the clipped sequence equals the raw one from step s0+⌊H/2⌋+1s_0 + \lfloor H/2\rfloor+ 1 on.

Hence, a single spike, however large, only has a bounded and short-lived effect, keeping p in a bounded region (Figure 3(b)), while a persistent real change still passes through after a short delay (proofs in Appendix § 15.§ 2). A sustained burst is thus treated as a real change, whose effect is bounded by WDR (Theorem 3.3(ii)) and fades once it ends.

Worst-Case Domain Reweighting (WDR)

WDR fixes misplaced protection: the worst case should be a regressing domain, not an inherently hard one. The nominal allocation alone does not protect such a domain either: a rising KL clips its vdv_d to 0, giving it the smallest share. WDR thus redefines the worst case and solves the DRO problem around p.

Excess Loss. We measure whether a domain is regressing by its excess loss over its own best level:

hd=min⁡(log⁡ℓdℓd∗,log⁡κ)∈[0,log⁡κ],(10)h_d=\min\left(\log\frac{\ell_d}{\ell_d^*},\log\kappa\right)\in[0,\log\kappa], \tag*{(10)}

where ℓd∗\ell_d^* is the lowest EMA level of domain dd so far (current update included) and κ≥1\kappa\ge1 is the excess cap. Here, hdh_d is the log regression factor of Eqn. (5) on the EMA, capped to bound the tilt in Eqn. (11). It is 0 at the best level and grows as dd regresses; as a ratio within one domain, it depends on neither the KL scale nor IdI_d.

Robust Best Response in a KL Ball. Given p\mathbf{p}, WDR solves Eqn. (6) with the feasible set U={q∈ΔK:qd≥λd}\mathcal{U}=\{\mathbf{q}\in\Delta^K:q_d\ge\lambda_d\}, where λd=min⁡(pd,λlo)\lambda_d=\min(p_d,\lambda_{\mathrm{lo}}) and λlo\lambda_{\mathrm{lo}} is the coverage floor: the tilt never pushes a domain below min⁡(pd,λlo)\min(p_d,\lambda_{\mathrm{lo}}), and p\mathbf{p} itself stays feasible. Without the floor, the solution is the closed form of §3.1.1 with ℓ^d\hat{\ell}_d replaced by hdh_d (∝pdeηhd\propto p_d e^{\eta h_d}); the floor lifts any domain below λd\lambda_d to λd\lambda_d:

qd=max⁡(λd,νpdeηhd),(11)q_d=\max\left(\lambda_d,\nu p_d e^{\eta h_d}\right), \tag*{(11)}

where ν>0\nu>0 is the unique normalizer with ∑dqd=1\sum_d q_d=1 (computed exactly in Appendix § 11). Each share is thus multiplied by eηhd∈[1,κη]e^{\eta h_d}\in[1,\kappa^\eta]: the tilt is idle when all domains are at their best, and a more regressing domain gets more budget, up to κη\kappa^\eta times.

Theorem 3.3 (Exact KL-Ball DRO Solution). Let p∈ΔK\mathbf{p}\in\Delta^K with p>0\mathbf{p}>\mathbf{0}, h∈[0,log⁡κ]K\mathbf{h}\in[0,\log\kappa]^K, η>0\eta>0, κ≥1\kappa\ge1, and ∑dλd<1\sum_d\lambda_d<1.

(i) Exact KL-ball DRO solution. Eqn. (6) has a unique maximizer, given by Eqn. (11), which also maximizes ⟨q′,h⟩\langle\mathbf{q}',\mathbf{h}\rangle over the KL ball {q′∈U:KL⁡(q′∥p)≤KL⁡(q∥p)}\{\mathbf{q}'\in\mathcal{U}:\operatorname{KL}(\mathbf{q}'\mathbin\Vert\mathbf{p})\le\operatorname{KL}(\mathbf{q}\mathbin\Vert\mathbf{p})\} around p\mathbf{p}. (ii) Bounded deviation without starvation. κ−η≤qd/pd≤κη\kappa^{-\eta}\le q_d/p_d\le\kappa^\eta for all dd, and qd≥min⁡(f,λlo)q_d\ge\min(f,\lambda_{\mathrm{lo}}) for p\mathbf{p} of Eqn. (7) with Kf≤1Kf\le1. (iii) Targeted protection. If all hdh_d are equal, q=p\mathbf{q}=\mathbf{p}; otherwise, every domain with the largest hdh_d has qd≥pdq_d\ge p_d.

Intuitively, WDR is an exactly solved DRO decision (proofs in Appendix § 15.§ 3): within a KL ball around p\mathbf{p}, q\mathbf{q} attains the largest expected excess loss, where the highest iso-line of ⟨q,h⟩\langle\mathbf{q},\mathbf{h}\rangle touches the ball (Figure 3(c)).

Overall Procedure

SAE clips each new KL observation at every training step (Eqn. (9)); at every scheduling update, RDS-MOPD computes r\mathbf{r}, σ^\hat{\boldsymbol{\sigma}}, and h\mathbf{h} on the clipped sequences, estimates p\mathbf{p} (Eqns. (7) and (8)), and tilts it into q\mathbf{q} (Eqn. (11); Algorithm 1 in Appendix § 11). Setting ϵ=0\epsilon=0, c=∞c=\infty, or η=0\eta=0 (then q=p\mathbf{q}=\mathbf{p}) disables NRC, SAE, or WDR, respectively (§4.3.1). An update only reads the KK KL sequences already computed by the trainer (Eqn. (1)) and costs O(K2)O(K^2), with no extra forward pass or GPU memory (§4.5).

Experiments

We evaluate RDS-MOPD by answering the following questions:

  • RQ1: How does RDS-MOPD perform on math, code, and instruction following, compared with MOPD methods with a fixed or adaptive domain mixture and training-free model merging?

  • RQ2: How does each module of RDS-MOPD contribute to the performance, and does it resolve its corresponding instability?

  • RQ3: Can RDS-MOPD keep a robust mixture and protect the other domains when one domain teacher is corrupted in the middle of training?

  • RQ4: Does RDS-MOPD incur a large time overhead?

Experimental Setups

Benchmarks and Evaluation Metrics

We evaluate all methods on six benchmarks, two per domain teacher: AIME24 and AIME25 (Mathematical Association of America, [22]) for mathematical reasoning, LiveCodeBench v5 and v6 (LCB v5/v6) (Jain et al., [13]) for code generation, and IFEval (Zhou et al., [45]) and IFBench (Pyatkin et al., [25]) for instruction following (IF). Each domain pairs an earlier benchmark with a more recent one, so that both in-domain fitting and generalization to newer distributions are examined. We report avg@64 on AIME24/25, avg@10 on LCB v5/v6, and avg@4 of the strict prompt-level accuracy on IFEval/IFBench (higher is better for all). Benchmark statistics and evaluation configurations are given in Appendix § 9.

Training Data and Models

We build our testbed upon Open-MOPD (Gao et al., [7]), to our knowledge the only recent MOPD work that releases its complete training resources. We distill the SmolLM3-3B (Hugging Face, [11]) student, initialized from the released MixSFT checkpoint (Student-SFT), from the released RL-Math, RL-Code, and RL-IF teachers. The prompt pool contains 86,931 prompts, i.e., 17,917 math prompts from DAPO (Yu et al., 2025), 23,667 code prompts from PrimeIntellect (Prime Intellect, [24]) and TACO (Li et al., 2023), and 45,347 IF prompts from Nemotron-IF-RL (Gao et al., [7]), where each prompt is routed to the teacher of its domain. Per-domain statistics are given in Appendix § 9.

Competing Methods

We compare RDS-MOPD with two groups of competing methods: (1) state-of-the-art MOPD methods with a fixed or adaptive domain mixture, i.e., Vanilla MOPD (Ma et al., 2026), Open-MOPD (Gao et al., [7]), D3^3-MOPD (Sun et al., [32]Sagawa et al., 2020`) (§3.1); and (2) training-free model merging, i.e.,

ParamMerge-Avg (Wortsman et al., [36]) and ParamMerge-TA (Ilharco et al., [12]). We also report RouteRL (Gao et al., [7]) as a routed-teacher reference, which is not a single deployable model and is excluded from the ranking. Details are given in Appendix § 10.

Implementation and Evaluation

All MOPD methods share the same training code and sampler, and differ mainly in the mixture rule. They use the dense top-16 teacher reverse KL as the per-token reward, and are trained for 150 steps with 1,024 prompts per step; the scheduling methods start from q(0)=(0.4,0.4,0.2)q^{(0)} = (0.4, 0.4, 0.2) and update it every 10 steps, except Open-MOPD and Vanilla DRO-MOPD, which keep this sampling mixture and reweight the per-domain rewards and losses, respectively. Following D3^3-MOPD, we report the mean and standard deviation over three seeds under two checkpoint selection criteria. Best-Overall Performance (BO) takes the single checkpoint with the best mean domain score of each run, and Best-per-Benchmark Performance (BB) takes the best checkpoint on each benchmark. As a scale-free summary, we also report the average per-benchmark rank (Avg. Rank; details in Appendix § 12).

Overall Performance (RQ1)

As shown in Table 1, we have the following threefold observations.

MethodMath AIME24Math AIME25Code LCB v5Code LCB v6Instruction Following IFEvalInstruction Following IFBenchAvg. Rank (↓)Recovery (%, ↑)pp-value
Student-SFT15.1015.1615.9915.4966.8218.50–0.0–
RouteRL24.3824.3222.9921.9473.5722.50–100.0–
Best-Overall (BO)
ParamMerge-Avg21.2021.6720.0019.4370.5220.007.0060.43.8×10−63.8 \times10^{-6}
ParamMerge-TA23.3324.1122.4621.2673.2022.835.0094.17.6×10−67.6 \times10^{-6}
Vanilla MOPD23.25 ± 0.2923.30 ± 0.2421.42 ± 0.1221.30 ± 0.2472.41 ± 0.2921.03 ± 0.135.8383.63.8×10−63.8 \times10^{-6}
Open-MOPD24.64 ± 0.1424.70 ± 0.2422.46 ± 0.0622.31 ± 0.1374.26 ± 0.1723.72 ± 0.092.67105.64.2×10−44.2 \times10^{-4}
D3^3-MOPD24.77 ± 0.8124.90 ± 0.3722.91 ± 0.7422.29 ± 0.4674.18 ± 0.2623.42 ± 0.802.58106.59.7×10−49.7 \times10^{-4}
Vanilla DRO-MOPD23.85 ± 0.2624.69 ± 0.2723.37 ± 0.2322.07 ± 0.1273.97 ± 0.0723.42 ± 0.143.42103.91.6×10−41.6 \times10^{-4}
RDS-MOPD26.72 ± 0.1626.09 ± 0.1923.79 ± 0.1923.01 ± 0.0774.74 ± 0.1023.08 ± 0.301.50118.1–
Best-per-Benchmark (BB)
ParamMerge-Avg21.2021.6720.0019.4370.5220.007.0060.43.8×10−63.8 \times10^{-6}
ParamMerge-TA23.3324.1122.4621.2673.2022.835.3394.13.8×10−63.8 \times10^{-6}
Vanilla MOPD23.34 ± 0.1423.65 ± 0.0921.74 ± 0.0621.35 ± 0.1572.63 ± 0.2121.25 ± 0.145.6786.53.8×10−63.8 \times10^{-6}
Open-MOPD24.81 ± 0.0324.90 ± 0.0622.59 ± 0.0722.38 ± 0.0774.55 ± 0.1423.81 ± 0.172.58107.89.7×10−49.7 \times10^{-4}
D3^3-MOPD24.95 ± 0.5625.11 ± 0.5522.91 ± 0.7422.38 ± 0.3274.54 ± 0.4723.50 ± 0.822.58108.79.7×10−49.7 \times10^{-4}
Vanilla DRO-MOPD24.17 ± 0.0624.86 ± 0.0623.51 ± 0.0322.19 ± 0.0974.03 ± 0.0023.56 ± 0.103.33106.11.6×10−41.6 \times10^{-4}
RDS-MOPD26.72 ± 0.1626.09 ± 0.1923.95 ± 0.2223.20 ± 0.1174.88 ± 0.0323.17 ± 0.171.50119.5–

Table 1. Overall performance (%, mean ± std over 3 seeds; training-free methods and reference rows have no std) under the BO (top) and BB (bottom) checkpoint selection criteria. Bold/underline: best/second-best among the ranked methods (gray reference rows excluded); “Avg. Rank” (↓): mean per-benchmark rank over the six benchmarks; “Recovery” (↑): (Total − SFT)/(RouteRL − SFT) following Open-MOPD, where Total is the mean of the three domain averages; pp-value: one-sided Wilcoxon test of RDS-MOPD vs. each method over 18 benchmark-seed pairs, treating pairs as independent (Appendix § 12).

(1) Training-free model merging can hardly fuse the capabilities of multiple teachers. ParamMerge-Avg ranks last on all six benchmarks, and ParamMerge-TA still falls behind all the adaptive MOPD methods (Avg. Rank of 5.00 and 5.33 under BO and BB). This is reasonable as the domain-specific RL updates of the teachers conflict and partly cancel out when merged in the parameter space, whereas MOPD aligns the student with each teacher token by token on its own responses. Yet MOPD alone does not guarantee this advantage: Vanilla MOPD with a fixed mixture even ranks below ParamMerge-TA (5.83 vs. 5.00 under BO), whereas the adaptive methods (Open-MOPD, D3^3-MOPD, and RDS-MOPD) outperform Vanilla MOPD on all six benchmarks under both criteria. This indicates that the key lies in allocating the domain budget.

(2) Vanilla DRO-MOPD is unstable under noisy training signals. Although it up-weights the worst domain, Vanilla DRO-MOPD falls behind the progress-driven Open-MOPD and D3^3-MOPD on four of six benchmarks (AIME24, AIME25, LCB v6, and IFEval), with an Avg. Rank of 3.42 and 3.33 under BO and BB. This is mainly because, besides the three instabilities of § 3.1, reweighting the losses under a fixed sampling mixture only rescales the gradients and amplifies the KL noise of the up-weighted domain. This indicates that robust scheduling should protect the domain regressing from its own best level rather than the one with the largest loss.

(3) Our RDS-MOPD consistently outperforms the existing methods. RDS-MOPD ranks first on five of six benchmarks with the lowest Avg. Rank of 1.50 under both BO and BB, clearly ahead of D3^3-MOPD and Open-MOPD (2.58–2.67). The gains are largest on math, i.e., 1.95 and 1.19 points over the best baseline on AIME24 and AIME25 under BO, and also hold on code (e.g., 0.70 points on LCB v6) and IFEval. The only exception is IFBench, whose 58 constraint types are designed to be out-of-distribution (Pyatkin et al., 2025) and are all absent from our IF training prompts, which may limit the adaptability of the distilled student to them. Even so, all adaptive MOPD methods, including RDS-MOPD, exceed the IF teacher (22.50) on IFBench, whereas on the in-distribution IFEval, RDS-MOPD achieves the best result. RDS-MOPD also surpasses the domain teachers (RouteRL) on all six benchmarks, with all seed standard deviations within 0.30, whereas those of D3^3-MOPD reach 0.81 and 0.74 on AIME24 and LCB v5 under BO. This is mainly because the existing progress-driven methods act on progress scores estimated from noisy KL curves (e.g., D3^3-MOPD normalizes its softmax by the maximum score). Hence, noise-induced leader switches (Appendix § 13) and transient KL spikes can pull a large share of prompts toward one domain. The three modules of RDS-MOPD bound exactly these effects. This indicates that the bottleneck of adaptive scheduling is robustness to noisy signals rather than adaptivity itself. RDS-MOPD outperforms every competing method on at least 15 of the 18 benchmark-seed pairs.

Empirical Module Analyses (RQ2)

Ablation Study

To verify each module, we build three variants of RDS-MOPD, each disabling exactly one module by its hyper-parameter (§3.3) and keeping all other hyper-parameters unchanged. As shown in Table 2, disabling any module degrades RDS-MOPD on most benchmarks.

BenchmarkRDS-MOPDw/o NRCw/o SAEw/o WDR
Best-Overall (BO)
AIME2426.7225.47 (↓4.68%)26.04 (↓2.54%)26.20 (↓1.95%)
AIME2526.0925.62 (↓1.80%)25.36 (↓2.80%)25.89 (↓0.77%)
LCB v523.7922.93 (↓3.61%)23.47 (↓1.35%)23.65 (↓0.59%)
LCB v623.0122.51 (↓2.17%)23.37 (↑1.56%)22.86 (↓0.65%)
IFEval74.7474.58 (↓0.21%)74.82 (↑0.11%)75.74 (↑1.34%)
IFBench23.0822.75 (↓1.43%)22.50 (↓2.51%)23.00 (↓0.35%)
Best-per-Benchmark (BB)
AIME2426.7226.30 (↓1.57%)26.04 (↓2.54%)26.20 (↓1.95%)
AIME2526.0925.62 (↓1.80%)25.36 (↓2.80%)25.89 (↓0.77%)
LCB v523.9523.59 (↓1.50%)23.47 (↓2.00%)23.65 (↓1.25%)
LCB v623.2023.20 (0.00%)23.37 (↑0.73%)22.86 (↓1.47%)
IFEval74.8874.58 (↓0.40%)74.82 (↓0.08%)75.74 (↑1.15%)
IFBench23.1722.75 (↓1.81%)23.42 (↑1.08%)23.08 (↓0.39%)

Table 2. Ablation study of RDS-MOPD (%), with the relative change compared to RDS-MOPD in parentheses (variants are single runs with seed 0).

  • w/o NRC drops on all six benchmarks under BO, most notably by 4.68% and 3.61% on AIME24 and LCB v5, as the scheduler without the noise floor pushes the budget toward a domain that leads by chance, i.e., the worst-case misjudgment. This implies the criticalness of deciding when to respond under noisy signals.

  • w/o SAE drops on four of six benchmarks under both criteria, most notably by 2.54% and 2.80% on AIME24 and AIME25. Without clipping, transient KL spikes of the code domain enter the statistics of the nominal allocation in proportion to their size, distorting the allocation and squeezing the other domains. This implies the criticalness of bounding the influence of any single anomaly.

  • w/o WDR drops on all benchmarks except IFEval under both criteria, e.g., by 1.95% on AIME24 and 1.47% on LCB v6 under BB, as the nominal allocation gives no extra protection to a domain regressing from its best level. This implies the criticalness of each domain’s own best level as the reference.

Module Stability Analyses

To understand why each module works, we examine whether it resolves its instability of Vanilla DRO-MOPD (§3.1). We replay RDS-MOPD and its variants without each module on the same recorded trajectory, so that any difference in decisions is attributable to that module alone. As shown in Figure 4, we have the following threefold observations.

RDS-MOPD versus its variants without the corresponding module

Figure 4. RDS-MOPD vs. its variants without the corresponding module (“w/o module”), replayed on the same recorded trajectory. (a) Share of each domain vs. its progress gap in noise standard deviations (shaded: within one std); (b) share shift remaining 40 steps after a KL spike (lower is better); (c) share of the most regressing domain (higher is better).

(1) NRC resolves the worst-case misjudgment caused by noise. As shown in Figure 4(a), without NRC, the shares swing between 0.15 and 0.65 even when the gap between domains is below one noise standard deviation, i.e., the worst-case misjudgment. With NRC, they stay between 0.17 and 0.40 within the noise range, yet still rise to 0.57 on average once the gap exceeds two noise standard deviations. As a result, the share follows the gap much more closely (Spearman correlation of 0.83 vs. 0.55). This indicates that RDS-MOPD responds only to real learning gaps, while still adjusting the mixture decisively.

(2) **SAE prevents spikes from hijacking the training bud-_ get. As shown in Figure 4(b), without SAE, the share shift caused by an injected one-step KL spike grows with the spike size and lasts long after the spike disappears, i.e., the spike hijacking. The shift is still 0.027 after 40 steps under a 64× spike. With SAE, it stays at 0.002 regardless of the spike size, and its maximum right after the spike is bounded at 0.033, compared with 0.172 without SAE. This indicates that the influence of any single anomaly on RDS-MOPD is both bounded and short-lived.

(3) WDR protects the truly regressing domains. As shown in Figure 4(c), without WDR, the domain regressing the most from its own best level receives only 0.277 of the budget on average, even less than the 0.398 of the domains close to their best levels, i.e., the misplaced protection. With WDR, it receives more budget at 12 of the 13 updates from step 30, and its share rises to 0.382 on average. This indicates that RDS-MOPD protects the truly regressing domains.

Robustness under a Corrupted Teacher (RQ3)

In practice, the signal of a domain teacher may become unreliable during training, e.g., due to a faulty scoring service. To test whether RDS-MOPD stays reliable in such cases, we corrupt the code teacher from step 30 on. Specifically, we add zero-mean noise (re-sampled at every call) to its token log-probabilities within the student’s top-16 candidates, which inflates its KL signal by about 2.5×. We then compare RDS-MOPD with D3^{3}-MOPD under the same seed and data stream for 60 steps, which cover the first three scheduling updates after the corruption. As shown in Figure 5, we have the following twofold observations.

Figure 5. RDS-MOPD vs. D3^{3}-MOPD with the code teacher corrupted from step 30 (shaded). (a) Share of code prompts (dashed: RDS-MOPD without corruption); (b) math KL relative to its level at step 30 (lower is better).

(1) RDS-MOPD keeps a stable mixture under the corrupted teacher. As shown in Figure 5(a), the mixture of RDS-MOPD is almost unaffected by the corruption. Its code share deviates from that of its uncorrupted run by only 0.011 on average over steps 30–60, and averages 0.325 over steps 41–60. In contrast, D3^{3}-MOPD allocates only 0.179 to code and moves the released budget almost entirely to math (+0.145 vs. −0.006 for RDS-MOPD). This is mainly because the inflated KL raises the excess loss of code, so WDR keeps a bounded protection on it. Meanwhile, NRC treats the remaining score differences within the noise level as no evidence for re-allocation, so RDS-MOPD is not dragged by an unreliable teacher.

(2) RDS-MOPD preserves the learning progress of all domains. RDS-MOPD allocates about 1.8× as many prompts to the corrupted code domain as D3^{3}-MOPD, yet ends with a lower code KL (0.0164 vs. 0.0225 over steps 56–60). Meanwhile, as shown in Figure 5(b), its math KL decreases to 0.66× its level at step 30, compared with 0.79× for D3^{3}-MOPD. Thus, RDS-MOPD neither abandons the corrupted domain nor sacrifices clean ones.

Efficiency Analysis (RQ4)

We measure the per-step wall-clock time of the MOPD methods on the same hardware (16 A800 GPUs; stages in Figure 6) and the time of the scheduling decision, with the following twofold observations.

Figure 6. Wall-clock time per training step of MOPD methods.

(1) The scheduling decision of RDS-MOPD introduces negligible overhead. The RDS-MOPD scheduler takes only 0.039 milliseconds per step on average, more than seven orders of magnitude below a training step of about 750 seconds. This is mainly because all three modules operate on a few per-domain scalars, with no extra forward pass or GPU memory. Thus, the robustness of RDS-MOPD comes from its decision rule, not extra computation.

(2) The end-to-end training cost of RDS-MOPD is on par with or even lower than that of existing methods. Over steps 2–84, RDS-MOPD takes 747.5 seconds per step on average, vs. 758.2, 764.5, and 762.6 seconds for Open-MOPD, D3^{3}-MOPD, and Vanilla DRO-MOPD. Over the full 150 steps, it is 5.6% faster per step than D3^{3}-MOPD, as rollout takes about 45% of each step and the student of RDS-MOPD produces shorter responses (7,130 vs. 8,127 tokens on average). Thus, RDS-MOPD distills better at no extra cost.

Related Work

Multi-Teacher and On-Policy Distillation

Knowledge distillation fits a student to the teacher’s output distribution [9], and early multi-teacher methods fuse the soft labels or intermediate representations of multiple teachers [39], or select the teachers for each sample by reinforcement learning [41]. For LLMs, sequence-level distillation [15] suffers from the distribution shift between training and inference, which on-policy distillation (OPD) alleviates by supervising the student on its own sampled responses [1] (Gu et al. 2024), e.g., with the reverse KL divergence to avoid overestimating the low-probability regions of the teacher (Gu et al., 2024), and has become a standard post-training stage [19, 38], as its per-token supervision is denser than the sequence-level rewards of RL. Extending it to multiple domain-expert teachers leads to MOPD (Ma et al., 2026), which merges the capabilities of domain-specific RL teachers into one student. Subsequent work improves MOPD by balancing the capabilities of different teachers via token-share balancing and KL-gap following [7], weighting or gating teacher supervision by its reliability [31, 43], resolving the conflicts among teachers [33, 3], and distilling at the representation level [6]. Most of them keep the domain mixture fixed and refine how the teacher supervision is weighted or combined. Closer to our work, D3-MOPD [32] schedules the domain mixture by the learning progress of each domain, i.e., its remaining KL gap times its descent velocity, MT-GRPO [27] reweights tasks online toward the worst-performing task in multi-task RL, and concurrent work [18] sets model-merging coefficients before training by minimizing the worst per-domain residual. Different from these works, RDS-MOPD introduces DRO into the online domain-mixture scheduling of MOPD and makes it robust to noise.

Distributionally Robust Optimization

DRO optimizes the worst-case loss within an uncertainty set around the nominal distribution [2, 4], where the uncertainty set can be defined by, e.g., the KL divergence, whose worst case admits a closed-form exponential tilting [16], the Wasserstein distance [5], or a C-VaR-type tail risk [28]. Group DRO [29] instantiates this idea over predefined groups by up-weighting high-loss groups online, and has been widely adopted, e.g., in topic-robust language modeling [23], data mixture optimization for pre-training [37], group-robust preference alignment [26], and worst-group knowledge distillation [14, 35, 34]. Among them, DoReMi [37] runs Group DRO on a small proxy model to determine the pre-training domain weights offline. However, DRO is known to be sensitive to outliers, since the worst case tends to be dominated by them, and DORO [42] thus discards a fraction of the highest-loss samples. This motivates combining DRO with robust statistics against heavy-tailed noise, e.g., the Huber estimator [10] and median-of-means [20]. Following this line, RDS-MOPD fixes three concrete instabilities of DRO under the noisy signals of MOPD within a single DRO structure.

Conclusion and Future Work

In this paper, we propose RDS-MOPD, a robust domain-mixture scheduling framework for multi-teacher on-policy distillation. It resolves the instabilities of Vanilla DRO-MOPD under noisy signals with three modules, backed by theoretical guarantees. Extensive experiments on mathematical reasoning, code generation, and instruction-following benchmarks demonstrate the effectiveness and robustness of RDS-MOPD with negligible computational overhead. As for future work, we plan to extend the robust allocation from the domain level to finer granularities, e.g., the prompt level.

Notations

Benchmark and Training Data Details

Table 4 lists the six benchmarks: AIME24/25 (Mathematical Association of America, [22] contain the 2024/2025 AIME problems with integer answers judged by exact match. LiveCodeBench v5/v6 (Jain et al., [13]) collect competitive programming problems by release window and judge each solution by hidden test cases, which alleviates contamination. IFEval (Zhou et al., [45]) contains programmatically verifiable constraints (e.g., length, format, and keywords), and IFBench (Pyatkin et al., [25]) adds novel out-of-distribution constraints; a response is correct only if it satisfies all constraints under the strict prompt-level criterion. We score each IF sample against its own response, unlike the released script of Open-MOPD (Gao et al., [7]), which gives all samples of a prompt one verdict. All metrics are pass@1 estimated by avg@#Samp., with thinking mode, top-p=0.95p = 0.95, and at most 30k generated tokens, and each run is summarized over its checkpoints by BO and BB (Appendix § 12).

We build our testbed on Open-MOPD (Gao et al., [7]), to our knowledge the only recent MOPD work that releases its complete training resources (SFT student, domain teachers, and prompts), so all methods share the same student, teachers, and data. The student is SmolLM3-3B (Hugging Face, [11]) initialized from the released MixSFT checkpoint (Student-SFT), and the teachers RL-Math, RL-Code, and RL-IF are obtained by domain-specific RL on top of Student-SFT. The prompt pool (Table 4) draws math from DAPO (Yu et al., 2025), code from PrimeIntellect (Prime Intellect, [24]) and TACO (Li et al., 2023), and IF from Nemotron-IF-RL (Gao et al., [7]). All trained methods share this pool and differ mainly in the domain mixture q(t)q^{(t)} (except Vanilla DRO-MOPD, Appendix § 11).

DomainEvaluation Benchmark#Prob.#Samp.Temp.SourceTraining #PromptsTeacher
MathAIME2430640.6DAPO17,917RL-Math
MathAIME2530640.6DAPO17,917RL-Math
CodeLCB v5167101.0PrimeIntellect + TACO23,667RL-Code
CodeLCB v6175101.0PrimeIntellect + TACO23,667RL-Code
IFIFEval54141.0Nemotron-IF-RL45,347RL-IF
IFIFBench30041.0Nemotron-IF-RL45,347RL-IF

Table 4. Benchmarks and training data per domain (#Prob.: problems; #Samp.: samples per problem, scored by avg@#Samp.; Temp.: sampling temperature).

Baseline Details

Vanilla MOPD (Ma et al., 2026; Sun et al., [32]) fixes the mixture to each domain’s share of the prompt pool. Open-MOPD (Gao et al., [7]) adds token-share balancing, KL-gap following, and a reward refresh within every inner update. Vanilla DRO-MOPD (Sagawа et al., 2020) normalizes each per-domain loss by its initial level; its strength η\eta is set as in Appendix § 11. D³-MOPD (Sun et al., [32]) samples each domain by the softmax of its remaining KL gap times its descent velocity, with a share floor. ParamMerge-Avg Wortsman et al., [36] averages the parameters of the three teachers. ParamMerge-TA Ilharco et al., [12] adds the scaled sum of the teachers’ task vectors (differences from the SFT student) to the student; we report the better of scaling factors 0.5 and 1.0. The reference rows of Table 1 are excluded from the ranking: Student-SFT is the starting point of all trained methods, and RouteRL Gao et al., [7] answers each prompt with its domain teacher (RL-Math for AIME24/25, RL-Code for LCB v5/v6, and RL-IF for IFEval/IFBench), which needs an oracle router and all teachers.

Implementation Details

Training. All methods use the dense top-16 reverse KL of the routed teacher as the per-token reward, B=1,024B = 1{,}024 prompts per step, a mini-batch size of 256, 150 steps, a learning rate of 1.5×10−61.5 \times10^{-6}, and at most 16k response tokens. Trained methods start from q(0)=(0.4,0.4,0.2)\mathbf{q}^{(0)} = (0.4, 0.4, 0.2) (math : code : IF), except Vanilla MOPD (pool-proportional), and the scheduling methods update the mixture every Δ=10\Delta= 10 steps; Open-MOPD and Vanilla DRO-MOPD reweight the per-domain rewards and losses instead (§ 3.1.1). Each batch jitters the mixture as q~d∝qd(1+zd)\widetilde{q}_d \propto q_d(1+z_d) with zd∼Unif⁡[−α,α]z_d \sim\operatorname{Unif}[-\alpha,\alpha] and draws Bq~dB\widetilde{q}_d prompts by the largest-remainder rule. All MOPD methods, including the ablation variants, share one training code and sampler and differ mainly in the mixture rule.

Hyper-parameters. RDS-MOPD uses ϵ=4\epsilon= 4, Nσ=40N_\sigma= 40 (NRC), c=0.5c = 0.5, H=5H = 5 (SAE), η=1\eta= 1, κ=2\kappa= 2, λlo=0.2\lambda_{\mathrm{lo}} = 0.2 (WDR), W=10W = 10, n0=5n_0 = 5, Kp=3K_p = 3, T=0.5T = 0.5, f=0.10f = 0.10, and α=0.30\alpha= 0.30 (sensitivity in Appendix § 14). Baselines use their official settings; Vanilla DRO-MOPD updates the domain weights by exponentiated gradient on the per-domain loss normalized by its initial level, with strength η=0.03\eta= 0.03.

Scheduler Formulas. The progress score rd=gdvdr_d = g_d v_d follows D3^3-MOPD Sun et al., [32]: gd=ℓd/Idg_d = \ell_d/I_d, and vdv_d averages the relative decrease of ℓd[j]\ell_d^{[j]}, the EMA over the jj-th most recent window of WW clipped observations (ℓd=ℓd[0]\ell_d = \ell_d^{[0]}), for j<J≤Kpj < J \le K_p, clipped at 0. NRC estimates the log-noise scale over the last NσN_\sigma observations as

σd=1.48262median⁡s′∣log⁡m~d(s′)−log⁡m~d(s′−1)∣,\sigma_d = \frac{1.4826}{\sqrt{2}}\operatorname{median}_{s'}\left|\log\widetilde{m}_d^{(s')} - \log\widetilde{m}_d^{(s'-1)}\right|,

and the delta method gives

SE⁡(vd)=2σd∥ω∥2J(∑j=0J−1ϱd,j2)1/2,(12)\operatorname{SE}(v_d) = \frac{\sqrt{2}\sigma_d\lVert\omega\rVert_2}{J}\left(\sum_{j=0}^{J-1}\varrho_{d,j}^{2}\right)^{1/2}, \tag*{(12)}
SE⁡(rd)=((vdσd∥ω∥2gd)2+(gdSE⁡(vd))2)1/2,\operatorname{SE}(r_d) = \left((v_d\sigma_d\lVert\omega\rVert_2g_d)^2 + (g_d\operatorname{SE}(v_d))^2\right)^{1/2},

where ω\omega are the EMA weights over one window and ϱd,j=ℓd[j]/ℓd[j+1]\varrho_{d,j} = \ell_d^{[j]}/\ell_d^{[j+1]}; σ^=SE⁡(rdmax⁡)\widehat{\sigma} = \operatorname{SE}(r_{d_{\max}}) with dmax⁡=arg⁡max⁡drdd_{\max} = \arg\max_d r_d. During warm-up (s≤Hs \le H), the median of SAE uses the available observations. WDR tilts and normalizes p\mathbf{p}, then iteratively clamps the coordinates below their floor and rescales the others, which returns Eqn. (11) exactly within KK rounds (Appendix § 15.§ 3).

Algorithm 1: Training procedure of RDS-MOPD

Input: Student πθ\pi_\theta; teachers {πϕd}\{\pi_{\phi_d}\}; pools {Pd}\{\mathcal{P}_d\}; batch size BB; steps SS; interval Δ\Delta; hyper-parameters (Appendix § 11).

Output: Distilled student πθ\pi_\theta.

  1. q←q(0)\mathbf{q} \leftarrow\mathbf{q}^{(0)};

  2. for s=1,…,Ss = 1,\ldots,S do Sample BB prompts from {Pd}\{\mathcal{P}_d\} by the jittered q\mathbf{q}; Roll out, score with the routed teachers, and update πθ\pi_\theta with the reward −Dj-D_j (§ 2.1);

  3. foreach d∈Dd \in D do Read m~d(s)\widetilde{m}_d^{(s)} by Eqn. (1) and clip it into m~d(s)\widetilde{m}_d^{(s)} by Eqn. (9); // SAE

  4. if ss is a scheduling step (every Δ\Delta steps after 2W2W observations per domain) then Get ℓd\ell_d, vdv_d, rdr_d, and σ^\widehat{\sigma} on {m~d}\{\widetilde{m}_d\} (Appendix § 11); Get τ\tau by Eqn. (8) and p\mathbf{p} by Eqn. (7); // NRC Update ℓd∗\ell_d^* and hdh_d by Eqn. (10); q←\mathbf{q} \leftarrow the solution of Eqn. (11); // WDR

  5. return πθ\pi_\theta;

Evaluation Setup. For each of the three seeds, we evaluate the checkpoints at steps 90, 100, 120, and 150, summarize them by the BO and BB criteria (Appendix § 12), and apply the same rule to all methods. The main runs use two nodes of eight NVIDIA A800-80GB GPUs, the ablation variants one such node, and the stress test four GPUs, with Python 3.12, PyTorch 2.8.0 (CUDA 12.8), verl 0.7.0, vLLM 0.11.0, and Transformers 4.57.

Evaluation Details

Each benchmark is scored at every evaluated checkpoint (steps 90, 100, 120, and 150), and each run is summarized by two criteria. Best-Overall (BO) selects the single checkpoint with the best mean of the three domain scores (each averaging its two benchmarks) and reports all six benchmarks there; it reflects one deployable model. Best-per-Benchmark (BB) takes the best score of each benchmark over all checkpoints separately. We report the mean and standard deviation over three seeds; training-free methods and reference rows have identical results under both criteria. As a scale-free summary, Avg. Rank (↓) ranks all ranked methods on each benchmark by their 3-seed means (1 is best, ties in the unrounded means averaged) and averages over the six benchmarks, per criterion. For significance, as three seeds are too few, we run the one-sided Wilcoxon signed-rank test between RDS-MOPD and each competitor over all 18 benchmark-seed pairs (six benchmarks × three

/* page continues */ seeds), per criterion, treating pairs as independent. RDS-MOPD wins on at least 15 out of the 18 pairs against every competitor.

Mixture Dynamics

We compare the real training runs of RDS-MOPD and D3^{3}-MOPD (same seed and hardware). An update is a large share reversal if any domain share changes by more than 0.3, and the mean share change of a run averages 12∥q(t)−q(t−1)∥1\frac{1}{2}\lVert\mathbf{q}^{(t)}-\mathbf{q}^{(t-1)}\rVert_{1}. A large reversal is noise-driven if the same scheduler reproduces it (same update, domain, and direction) in fewer than half of 100 bootstrap replicas of the KL trajectory, resampling residuals around a 21-step running median.

(1) D3^{3}-MOPD oscillates the mixture with noise. It makes 7 large share reversals in 14 updates, 5 of which are noise-driven (reappearing in only 13–28% of the replicas), e.g., from step 100 to 140 the math share alternates between 0.175 and 0.651 (Figure 7(a)) while the math KL stays within 0.002–0.006. This is mainly because its temperature shrinks with the progress scores (from 0.086 at step 20 to about 0.006 after step 90), so batch-level KL fluctuations alone move most of the budget, i.e., the worst-case misjudgment of § 3.1.

Figure 7. Domain shares during training (seed 0). Filled/open triangles: noise-driven/other large reversals of D3^{3}-MOPD, marked at the most-changed domain; RDS-MOPD has none.

(2) RDS-MOPD keeps the mixture responsive yet stable. It makes no large share reversal and cuts the mean share change from 0.258 to 0.118 (54%) and the maximum from 0.478 to 0.285 (Figure 7), while still following real gaps, e.g., the IF share rises to 0.643 at step 140 as only IF still descends. This is mainly because NRC calibrates the radius to the estimated noise level.

Hyper-parameter Sensitivity

We vary ϵ∈{0,1,2,4,8,16}\epsilon\in\{0,1,2,4,8,16\} (NRC) and (η,κ)∈{0,0.5,1,2}×{1.5,2,3}(\eta,\kappa) \in\{0,0.5,1,2\} \times\{1.5,2,3\} (WDR) with the others at their defaults (Appendix § 11); ϵ=0\epsilon= 0 and η=0\eta= 0 are w/o NRC and w/o WDR. We replay the scheduler on the recorded KL trajectory of RDS-MOPD (150 steps, 14 updates; Figure 8) to isolate the scheduling decisions, with reversals as in Appendix § 13. The lagging domain at an update is the one whose KL regresses most from its own best level.

Figure 8. Sensitivity to ϵ\epsilon of NRC (a, b) and η,κ\eta,\kappa of WDR (c, d) (▲: default ϵ=4\epsilon= 4, η=1\eta= 1, κ=2):(a)\kappa= 2): (a) noise-driven reversals per bootstrap replica (mean ±\pm std); (b) mean share change per update; (c) lagging-domain share; (d) max final share (dashed: 1−2λlo1-2\lambda_{\mathrm{lo}}, i.e., two domains at λlo\lambda_{\mathrm{lo}}).

(1) A moderate noise floor reduces noise-driven reversals without freezing the mixture. Raising ϵ\epsilon from 0 to 4 cuts noise-driven reversals on the recorded trajectory from 3 to 0 and the mean share change from 0.200 to 0.118 (41%); ϵ=4\epsilon= 4 is the smallest value removing all large reversals (5, 4, and 3 for ϵ=0\epsilon= 0, 1, and 2) while keeping a largest single-update change of 0.285, whereas ϵ=8\epsilon= 8 and 16 (0.04 and 0.00 noise-driven reversals) suppress it to 0.158 and 0.161, making the scheduler sluggish.

(2) The tilt strength η\eta controls how much budget goes to the lagging domain. Its share grows monotonically with η\eta under every κ\kappa, e.g., 0.277, 0.327, 0.382, and 0.499 for η=0\eta= 0, 0.5, 1, and 2 under κ=2\kappa= 2, so the default η=1\eta= 1 adds 0.105 as no extra reversals.

(3) The excess cap κ\kappa prevents over-concentration. Under η=1\eta= 1, the largest final share grows from 0.393 to 0.464 and 0.565 for κ=1.5,2\kappa= 1.5, 2, and 3, and η=2\eta= 2, κ=3\kappa= 3 ends at (0.200, 0.200, 0.600) with two domains on the floor λlo=0.2\lambda_{\mathrm{lo}} = 0.2 and 1 noise-driven reversal, whereas the default κ=2\kappa= 2 ends at (0.288, 0.249, 0.464).

Proofs

Proof of Proposition 3.1

Proposition 3.1 is part (i); the full statement adds (ii) for a given σ^>0\hat{\sigma} > 0 and any r,r′≥0\mathbf{r},\mathbf{r}' \ge0, ∥p(r)−p(r′)∥1≤2(1−Kf)Tϵ∥r−r′∥∞\lVert\mathbf{p}(\mathbf{r})-\mathbf{p}(\mathbf{r}')\rVert_{1} \le\frac{2(1-Kf)}{T\epsilon}\lVert\mathbf{r}-\mathbf{r}'\rVert_{\infty}; and (iii) with ϵ=0\epsilon= 0, Eqn. (8) reduces to the max-normalized temperature. Under the max-normalized temperature, such scores can instead change a share by (1−Kf)e1/T−1e1/T+K−1(1-Kf)\frac{e^{1/T}-1}{e^{1/T}+K-1} for any ζ>0\zeta> 0.

Lemma H.1 (ℓ1\ell_1 Sensitivity of Softmax). For any x,x′∈RK\mathbf{x},\mathbf{x}' \in\mathbb{R}^{K}, ∥softmax⁡(x)−softmax⁡(x′)∥1≤12osc⁡(x−x′)\|\operatorname{softmax}(\mathbf{x})-\operatorname{softmax}(\mathbf{x}')\|_1 \le\frac{1}{2}\operatorname{osc}(\mathbf{x}-\mathbf{x}'), where osc⁡(y)=max⁡dyd−min⁡dyd\operatorname{osc}(\mathbf{y})=\max_d y_d-\min_d y_d. Moreover, any Δ∈RK\boldsymbol{\Delta}\in\mathbb{R}^{K} with ∑dΔd=0\sum_d\Delta_d=0 satisfies max⁡d∣Δd∣≤12∥Δ∥1\max_d|\Delta_d|\le\frac{1}{2}\|\boldsymbol{\Delta}\|_1.

Proof. Let δ=x′−x\boldsymbol{\delta}=\mathbf{x}'-\mathbf{x} and π(β)=softmax⁡(x+βδ)\boldsymbol{\pi}(\beta)=\operatorname{softmax}(\mathbf{x}+\beta\boldsymbol{\delta}) for β∈[0,1]\beta\in[0,1]; then ddβπd(β)=πd(β)(δd−⟨π(β),δ⟩)\frac{d}{d\beta}\pi_d(\beta)=\pi_d(\beta)(\delta_d-\langle\boldsymbol{\pi}(\beta),\boldsymbol{\delta}\rangle), so ∥ddβπ(β)∥1=Ed∼π(β)∣δd−δˉ∣\left\|\frac{d}{d\beta}\boldsymbol{\pi}(\beta)\right\|_1=\mathbb{E}_{d\sim\boldsymbol{\pi}(\beta)}|\delta_d-\bar{\delta}| with δˉ=⟨π(β),δ⟩\bar{\delta}=\langle\boldsymbol{\pi}(\beta),\boldsymbol{\delta}\rangle, the mean absolute deviation of X=δdX=\delta_d supported on [δmin⁡,δmax⁡][\delta_{\min},\delta_{\max}] with δmax⁡−δmin⁡=osc⁡(δ)\delta_{\max}-\delta_{\min}=\operatorname{osc}(\boldsymbol{\delta}) (δmax⁡=δmin⁡\delta_{\max}=\delta_{\min} is trivial). By convexity, (z−δˉ)+≤(δmax⁡−δˉ)(z−δmin⁡)/(δmax⁡−δmin⁡)(z-\bar{\delta})_+\le(\delta_{\max}-\bar{\delta})(z-\delta_{\min})/(\delta_{\max}-\delta_{\min}) on [δmin⁡,δmax⁡][\delta_{\min},\delta_{\max}], so E∣X−δˉ∣=2E(X−δˉ)+≤2(δmax⁡−δˉ)(δˉ−δmin⁡)/(δmax⁡−δmin⁡)≤(δmax⁡−δmin⁡)/2\mathbb{E}|X-\bar{\delta}|=2\mathbb{E}(X-\bar{\delta})_+\le2(\delta_{\max}-\bar{\delta})(\bar{\delta}-\delta_{\min})/(\delta_{\max}-\delta_{\min})\le(\delta_{\max}-\delta_{\min})/2. Integrating over β\beta gives the first claim; for the second, the positive and negative entries of Δ\boldsymbol{\Delta} each sum to 12∥Δ∥1\frac{1}{2}\|\boldsymbol{\Delta}\|_1 in absolute value.

(i)

Let b(t′)=ϵσ^(t′)>0b^{(t')}=\epsilon\hat{\sigma}^{(t')}>0 for t′∈{t−1,t}t'\in\{t-1,t\}. Since rmax⁡(t′)≤ζσ^(t′)≤b(t′)r_{\max}^{(t')}\le\zeta\hat{\sigma}^{(t')}\le b^{(t')}, Eqn. (8) gives τ(t′)=Tb(t′)\tau^{(t')}=Tb^{(t')}, so the softmax input x(t)=r(t)/(Tϵσ^(t))\mathbf{x}^{(t)}=\mathbf{r}^{(t)}/(T\epsilon\hat{\sigma}^{(t)}) lies in [0,ζ/(ϵT)]K[0,\zeta/(\epsilon T)]^K and osc⁡(x(t)−x(t−1))≤2ζ/(ϵT)\operatorname{osc}(\mathbf{x}^{(t)}-\mathbf{x}^{(t-1)})\le2\zeta/(\epsilon T). By Lemma H.1, ∥p(t)−p(t−1)∥1≤(1−Kf)ζ/(ϵT)\|\mathbf{p}^{(t)}-\mathbf{p}^{(t-1)}\|_1\le(1-Kf)\zeta/(\epsilon T), so (both allocations sum to one) ∣pd(t)−pd(t−1)∣≤(1−Kf)ζ/(2ϵT)|p_d^{(t)}-p_d^{(t-1)}|\le(1-Kf)\zeta/(2\epsilon T), i.e., 0.7ζ/4=0.175ζ0.7\zeta/4=0.175\zeta for K=3K=3, f=0.1f=0.1, T=0.5T=0.5, and ϵ=4\epsilon=4. For the max-normalized response, r(t−1)=(δ,0,…,0)\mathbf{r}^{(t-1)}=(\delta,0,\ldots,0) and r(t)=(0,δ,0,…,0)\mathbf{r}^{(t)}=(0,\delta,0,\ldots,0) with 0<δ≤ζσ^0<\delta\le\zeta\hat{\sigma} normalize to e1\mathbf{e}_1 and e2\mathbf{e}_2 for any δ\delta, so the first share drops from f+(1−Kf)e1/Te1/T+K−1=0.651f+(1-Kf)\frac{e^{1/T}}{e^{1/T}+K-1}=0.651 to f+(1−Kf)1e1/T+K−1=0.175f+(1-Kf)\frac{1}{e^{1/T}+K-1}=0.175 by default, i.e., by (1−Kf)e1/T−1e1/T+K−1=0.476(1-Kf)\frac{e^{1/T}-1}{e^{1/T}+K-1}=0.476.

(ii)

Let b=ϵσ^>0b=\epsilon\hat{\sigma}>0, n(r)=max⁡(∥r∥∞,b)≥bn(\mathbf{r})=\max(\|\mathbf{r}\|_\infty,b)\ge b, and χ(r)=r/n(r)\boldsymbol{\chi}(\mathbf{r})=\mathbf{r}/n(\mathbf{r}), so that the softmax input is χ(r)/T\boldsymbol{\chi}(\mathbf{r})/T (note ∥r∥∞=rmax⁡\|\mathbf{r}\|_\infty=r_{\max} for r≥0\mathbf{r}\ge0). Then χ(r)−χ(r′)=(r−r′)/n(r)+r′(n(r′)−n(r))/(n(r)n(r′))\boldsymbol{\chi}(\mathbf{r})-\boldsymbol{\chi}(\mathbf{r}')=(\mathbf{r}-\mathbf{r}')/n(\mathbf{r})+\mathbf{r}'(n(\mathbf{r}')-n(\mathbf{r}))/(n(\mathbf{r})n(\mathbf{r}')). Both terms are bounded by ∥r−r′∥∞/b\|\mathbf{r}-\mathbf{r}'\|_\infty/b in ℓ∞\ell_\infty, the second since ∥r′∥∞/n(r′)≤1\|\mathbf{r}'\|_\infty/n(\mathbf{r}')\le1 and n(⋅)n(\cdot) is 1-Lipschitz in ℓ∞\ell_\infty. Hence the oscillation of the softmax input difference is at most 4∥r−r′∥∞/(bT)4\|\mathbf{r}-\mathbf{r}'\|_\infty/(bT), and Lemma H.1 with the factor (1−Kf)(1-Kf) gives the claim. For the max-normalized response, replacing bb by min⁡(rmax⁡,rmax⁡′)\min(r_{\max},r'_{\max}) gives a constant that diverges as the maximum score tends to zero, as the example in (i) realizes.

(iii)

With ϵ=0\epsilon=0, max⁡(rmax⁡(t),0)=rmax⁡(t)\max(r_{\max}^{(t)},0)=r_{\max}^{(t)} for r(t)≥0\mathbf{r}^{(t)}\ge0, i.e., the max-normalized temperature.

Proof of Proposition 3.2

(i) Directly from Eqn. (9). (ii) Primes denote contaminated quantities; LL and UU are the smallest and largest other observations in [s0−H,s0+H][s_0-H,s_0+H]. As the median uses raw observations, μd′(s)≠μd(s)\mu_d'^{(s)}\ne\mu_d^{(s)} only if s∈(s0,s0+H]s\in(s_0,s_0+H], and for s>s0+Hs>s_0+H the window and current observation are clean, so m~d′(s)=m~d(s)\tilde{m}_d'^{(s)}=\tilde{m}_d^{(s)}. For s=s0s=s_0, the window {s0−H,…,s0−1}\{s_0-H,\ldots,s_0-1\} is clean (as s0>Hs_0>H) and lies in [s0−H,s0+H][s_0-H,s_0+H], so μd′(s0)∈[L,U]\mu_d'^{(s_0)}\in[L,U], and (i) gives m~d′(s0)∈[e−cL,ecU]\tilde{m}_d'^{(s_0)}\in[e^{-c}L,e^{c}U] for any dd. For s∈(s0,s0+H]s\in(s_0,s_0+H], the window contains the contaminated value and H−1≥2H-1\ge2 clean values in [L,U][L,U]. The median involves only ranks in [2,H−1][2,H-1] (as H≥3H\ge3): the kk-th order statistic with k=(H+1)/2k=(H+1)/2 for odd HH, or the mean of the kk-th and (k+1)(k+1)-th with k=H/2k=H/2 for even HH. Each such order statistic lies between two order statistics of the remaining H−1H-1 values, so μd′(s)∈[L,U]\mu_d'^{(s)}\in[L,U]. The clean m~d(s)∈[L,U]\tilde{m}_d^{(s)}\in[L,U], and clipping moves it toward μd′(s)\mu_d'^{(s)}, so m~d′(s)∈[L,U]⊆[e−cL,ecU]\tilde{m}_d'^{(s)}\in[L,U]\subseteq[e^{-c}L,e^{c}U]. An EMA level is a convex combination of clipped observations, each equal to its clean counterpart or in [e−cL,ecU][e^{-c}L,e^{c}U], so it lies in an interval independent of MM; without clipping, the EMA weight ωi>0\omega_i>0 of step s0s_0 makes it at least ωiM\omega_iM.

(iii) Let md(s)=Am_d^{(s)}=A for s<s0s<s_0 and md(s)=A′m_d^{(s)}=A' for s≥s0s\ge s_0. For s≥s0+⌈H/2⌉+1s\ge s_0+\lceil H/2\rceil+1, the window of μd(s)\mu_d^{(s)} (of size at most HH) contains at least ⌈H/2⌉+1\lceil H/2\rceil+1 values equal to A′A' (a strict majority), so μd(s)=A′=md(s)\mu_d^{(s)}=A'=m_d^{(s)} lies in the clipping range and m~d(s)=md(s)\tilde{m}_d^{(s)}=m_d^{(s)}.

Proof of Theorem 3.3

The full statement adds: in (i), the maximizer is the KL projection of p⊙eηh\mathbf{p}\odot e^{\eta\mathbf{h}} (normalized) onto U\mathcal{U}; (ii) also gives qd≥max⁡(λd,κ−1ηpd)q_d\ge\max(\lambda_d,\kappa^{-1}\eta p_d) and KL⁡(q∥p)≥ηlog⁡κ\operatorname{KL}(\mathbf{q}\Vert\mathbf{p})\ge\eta\log\kappa; in (i) with ρ=KL⁡(q∥p)\rho=\operatorname{KL}(\mathbf{q}\Vert\mathbf{p}), when the floor is inactive, sup⁡KL⁡(q′∥p)≤ρ⟨q′,h⟩=inf⁡γ>0{γρ+γlog⁡Ed∼p[ehd/γ]}\sup_{\operatorname{KL}(\mathbf{q}'\Vert\mathbf{p})\le\rho}\langle\mathbf{q}',\mathbf{h}\rangle=\inf_{\gamma>0}\{\gamma\rho+\gamma\log\mathbb{E}_{d\sim\mathbf{p}}[e^{h_d/\gamma}]\}, attained at γ=1/η\gamma=1/\eta. Let Z=⟨p,eηh⟩≥1Z=\langle\mathbf{p},e^{\eta\mathbf{h}}\rangle\ge1 (as hd≥0h_d\ge0) and w=p⊙eηh/Z∈ΔK\mathbf{w}=\mathbf{p}\odot e^{\eta\mathbf{h}}/Z\in\Delta^K.

(i), unique maximizer. For any q∈ΔK\mathbf{q}\in\Delta^K, expanding KL⁡(q∥w)\operatorname{KL}(\mathbf{q}\Vert\mathbf{w}) gives

⟨q,h⟩−1ηKL⁡(q∥p)=1ηlog⁡Z−1ηKL⁡(q∥w).(13)\langle\mathbf{q},\mathbf{h}\rangle-\frac{1}{\eta}\operatorname{KL}(\mathbf{q}\Vert\mathbf{p})=\frac{1}{\eta}\log Z-\frac{1}{\eta}\operatorname{KL}(\mathbf{q}\Vert\mathbf{w}). \tag*{(13)}

Hence Eqn. (6) is the KL projection min⁡q∈UKL⁡(q∥w)\min_{\mathbf{q}\in\mathcal{U}}\operatorname{KL}(\mathbf{q}\Vert\mathbf{w}) onto the convex U\mathcal{U}, unique by strict convexity. With multipliers χ\chi and βd\beta_d of the simplex and floor constraints, the KKT conditions log⁡(qd/wd)+1+χ−βd=0\log(q_d/w_d)+1+\chi-\beta_d=0, βd≥0\beta_d\ge0, and βd(qd−λd)=0\beta_d(q_d-\lambda_d)=0 give qd=awdq_d=aw_d if qd>λdq_d>\lambda_d, and qd=λd≥awdq_d=\lambda_d\ge aw_d otherwise, with a=e−1−χa=e^{-1-\chi}, i.e., qd=max⁡(λd,awd)q_d=\max(\lambda_d,aw_d), which is Eqn. (11) with ν=1/a\nu=1/a. G(a)=∑dmax⁡(λd,awd)G(a)=\sum_d\max(\lambda_d,aw_d) is continuous and nondecreasing with G(0)=∑dλd<1≤G(1)G(0)=\sum_d\lambda_d<1\le G(1), so G(a)=1G(a)=1 has a root in (0,1](0,1], and it is unique: at any root, some dd has awd>λdaw_d>\lambda_d (otherwise G(a)=∑dλd<1G(a)=\sum_d\lambda_d<1), and as wd>0w_d>0, G′(a′)≥G(a)+(a′−a)wd>a′G'(a')\ge G(a)+(a'-a)w_d>a' for all a′>aa'>a.

Exactness of the implementation. From w\mathbf{w} (Appendix D), the unclamped coordinates after round kk are bkwdb_kw_d, b0=1b_0=1. Clamping only raises coordinates, so bk+1<bkb_{k+1}<b_k; the clamped set never shrinks, and the unclamped set is never empty as ∑dλd<1\sum_d\lambda_d<1, so the procedure stops within KK rounds, where every unclamped dd satisfies bkwd≥λdb_kw_d\ge\lambda_d, and @references♀♀♀♀♀♀ every clamped dd, clamped at some round j≤kj \le k, satisfies bkwd≤bjwd<λdb_k w_d \le b_j w_d < \lambda_d. Hence q=max⁡(λ,bkw)q = \max(\lambda, b_k w) with a unit sum, and bk=ab_k = a by uniqueness.

(i), KL ball. For any q′∈Uq' \in\mathcal{U} with KL⁡(q′∥p)≤ρ\operatorname{KL}(q' \mathbin{\|} p) \le\rho, the optimality of qq in Eqn. (6) gives ⟨q′,h⟩≤⟨q,h⟩+1η(KL⁡(q′∥p)−ρ)≤⟨q,h⟩\langle q', h \rangle\le\langle q, h \rangle+ \frac{1}{\eta}\left(\operatorname{KL}(q' \mathbin{\|} p) - \rho\right) \le\langle q, h \rangle, and qq is feasible, so it solves the constrained problem. For the dual form, Eqn. (13) with 1/γ1/\gamma in place of η\eta gives, for any γ>0\gamma> 0 and KL⁡(q′∥p)≤ρ\operatorname{KL}(q' \mathbin{\|} p) \le\rho, ⟨q′,h⟩≤γKL⁡(q′∥p)+γlog⁡Ep[ehd/γ]≤γρ+γlog⁡Ep[ehd/γ]\langle q', h \rangle\le\gamma\operatorname{KL}(q' \mathbin{\|} p) + \gamma\log\mathbb{E}_{p}[e^{h_d/\gamma}] \le\gamma\rho+ \gamma\log\mathbb{E}_{p}[e^{h_d/\gamma}], i.e., weak duality [16]. When the floor is inactive, q=wq = w and Eqn. (13) gives ⟨q,h⟩=1ηρ+1ηlog⁡Z\langle q, h \rangle= \frac{1}{\eta}\rho+ \frac{1}{\eta}\log Z, i.e., equality at γ=1/η\gamma= 1/\eta.

(ii), bounded deviation. Let F={d:awd≥λd}F = \{d : a w_d \ge\lambda_d\} and CC its complement, F≠∅F \ne\varnothing as ∑dλd<1\sum_d \lambda_d < 1. Let ψd=aeηhd/Z\psi_d = a e^{\eta h_d}/Z, so that qd/pd=ψdq_d/p_d = \psi_d for d∈Fd \in F, and qd/pd=λd/pd∈(ψd,1]q_d/p_d = \lambda_d/p_d \in(\psi_d, 1] for d∈Cd \in C. Upper bound: for d∈Fd \in F, qd/pd=ψd≤eηhd≤κηq_d/p_d = \psi_d \le e^{\eta h_d} \le\kappa^\eta as a≤1≤Za \le1 \le Z; for d∈Cd \in C, qd/pd≤1q_d/p_d \le1. Lower bound: if max⁡d∈Fψd<1\max_{d \in F} \psi_d < 1, all ratios are below one on FF and at most one on CC, contradicting ∑dpd(qd/pd)=1\sum_d p_d(q_d/p_d) = 1; hence some d′∈Fd' \in F has ψd′≥1\psi_{d'} \ge1, and for every dd, qd/pd≥ψd=ψd′eη(hd−hd′)≥κ−ηq_d/p_d \ge\psi_d = \psi_{d'}e^{\eta(h_d-h_{d'})} \ge\kappa^{-\eta}, since hd−hd′≥−log⁡κh_d - h_{d'} \ge-\log\kappa. Finally, KL⁡(q∥p)=∑dqdlog⁡(qd/pd)≤max⁡dlog⁡(qd/pd)≤ηlog⁡κ\operatorname{KL}(q \mathbin{\|} p) = \sum_d q_d \log(q_d/p_d) \le\max_d \log(q_d/p_d) \le\eta\log\kappa. (ii), no starvation. qd≥λd=min⁡(pd,λlo)q_d \ge\lambda_d = \min(p_d, \lambda_{\mathrm{lo}}) by feasibility, and pd≥fp_d \ge f by Eqn. (7) as Kf≤1Kf \le1, so qd≥min⁡(f,λlo)q_d \ge\min(f, \lambda_{\mathrm{lo}}).

(iii) If hd≡h0h_d \equiv h_0, then w=p∈Uw = p \in\mathcal{U}, so q=pq = p. Let d†∈arg max⁡dhdd^\dagger\in\operatorname*{arg\,max}_d h_d, so that ψd†=max⁡dψd\psi_{d^\dagger} = \max_d \psi_d. If d†∈Cd^\dagger\in C, then every d∈Fd \in F has qd/pd=ψd≤ψd†<λd†/pd†≤1q_d/p_d = \psi_d \le\psi_{d^\dagger} < \lambda_{d^\dagger}/p_{d^\dagger} \le1, and every d∈Cd \in C has qd/pd≤1q_d/p_d \le1, contradicting ∑dpd(qd/pd)=1\sum_d p_d(q_d/p_d) = 1. Hence d†∈Fd^\dagger\in F, and by (ii), $q_{d^\dagger}/p_{d^\dagger} = \max_{d \in F} \psi_d \ge 1. ∎

References

  1. [1]Agarwal, R., Vieillard, N., Zhou, Y., et al. On-policy distillation of language models: Learning from self-generated mistakes. In ICLR, 2024.arxiv.org/abs/2306.13649
  2. [2]Ben-Tal, A., den Hertog, D., Waegenaere, A. D., Melenberg, B., and Rennen, G. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 2013.
  3. [3]Chen, T., Ou, J., Liu, Z., Tang, R., Liang, J., and Li, H. Counteraction-aware multi-teacher on-policy distillation for general capability recovery with domain preservation. arXiv preprint arXiv:2605.27115, 2026.DOI
  4. [4]Duchi, J. C. and Namkoong, H. Learning models with uniform performance via distributionally robust optimization. Ann. Stat., 2021.arxiv.org/abs/1810.08750
  5. [5]Esfahani, P. M. and Kuhn, D. Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations. Math. Program., 2018.arxiv.org/abs/1505.05116
  6. [6]Fang, Z., Hong, S., Yang, J., et al. Latent-MOPD: Latent multi-teacher on-policy distillation. arXiv preprint arXiv:2610.02381, 2026.arxiv.org/abs/2610.02381
  7. [7]Gao, H., Chi, H., Yan, Y., et al. Open-MOPD: Diagnosing and fixing capability imbalance in multi-teacher on-policy distillation. arXiv preprint arXiv:2608.19098, 2026.arxiv.org/abs/2608.19098
  8. [9]Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.arxiv.org/abs/1503.02531
  9. [10]Huber, P. J. Robust estimation of a location parameter. Ann. Math. Stat., 1964.
  10. [11]Hugging Face. SmolLM3: Smol, multilingual, long-context reasoner. Hugging Face blog, 2025.
  11. [12]Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. In ICLR, 2023.arxiv.org/abs/2212.04089
  12. [13]Jain, N., Han, K., Gu, A., et al. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In ICLR, 2025.arxiv.org/abs/2403.07974
  13. [14]Kenfack, P., Aïvodji, U., and Kahou, S. E. Adaptive group robust ensemble knowledge distillation. TMLR, 2025.arxiv.org/abs/2411.14984
  14. [15]Kim, Y. and Rush, A. M. Sequence-level knowledge distillation. In EMNLP, 2016.arxiv.org/abs/1606.07947
  15. [16]Lam, H. Robust sensitivity analysis for stochastic systems. Math. Oper. Res., 2016.arxiv.org/abs/1303.0326
  16. [18]Liu, K., Zhuang, Z., Ma, H., et al. A fisher-geometric view of multi-teacher on-policy distillation. arXiv preprint arXiv:2610.26525, 2026.arxiv.org/abs/2610.26525
  17. [19]Lu, K. and Thinking Machines Lab. On-policy distillation. Thinking Machines Lab: Connectionism (blog post), 2025.
  18. [20]Lugosi, G. and Mendelson, S. Mean estimation and regression under heavy-tailed distributions: A survey. Found. Comput. Math., 2019.arxiv.org/abs/1906.04280
  19. [22]Mathematical Association of America. American invitational mathematics examination (AIME) 2024 and 2025, 2025.
  20. [23]Oren, Y., Sagawa, S., Hashimoto, T. B., and Liang, P. Distributionally robust language modeling. In EMNLP, 2019.
  21. [24]Prime Intellect. SYNTHETIC-1: Two million collaboratively generated reasoning traces from DeepSeek-R1. Prime Intellect blog, 2025.
  22. [25]Pyatkin, V., Malik, S., Graf, V., et al. Generalizing verifiable instruction following. arXiv preprint arXiv:2507.02833, 2025.arxiv.org/abs/2507.02833
  23. [26]Ramesh, S. S., Hu, Y., Chaimalas, I., et al. Group robust preference optimization in reward-free RLHF. In NeurIPS, 2024.arxiv.org/abs/2405.20304
  24. [27]Ramesh, S. S., Ji, X., Zimmer, M., et al. Multi-task GRPO: Reliable LLM reasoning across tasks. In ICML, 2026.arxiv.org/abs/2602.05547
  25. [28]Rockafellar, R. T. and Uryasev, S. Optimization of conditional value-at-risk. J. Risk, 2000.DOI
  26. [29]Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In ICLR, 2020.arxiv.org/abs/1911.08731
  27. [30]Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL https://arxiv.org/abs/1707.06347.
  28. [31]Sun, J., Zheng, M., Song, M., et al. Distill what you trust: Reliability-aware multi-teacher on-policy distillation. arXiv preprint arXiv:2609.23697, 2026a.arxiv.org/abs/2609.23697
  29. [32]Sun, Z., Zhang, Z., Zhao, F., et al. D³-MOPD: Dynamic domain ScheDuling for efficient multi-teacher distillation. arXiv preprint arXiv:2608.24987, 2026b.arxiv.org/abs/2608.24987
  30. [33]Tian, S., Liu, Y., Liu, X., et al. Beyond domain-routed supervision: Cross-teacher guarding for multi-teacher on-policy distillation. arXiv preprint arXiv:2610.13521, 2026.arxiv.org/abs/2610.13521
  31. [34]Vilouras, K., Liu, X., Sanchez, P., O’Neil, A. Q., and Tsaf-taris, S. A. Group distributionally robust knowledge distillation. In MLMI@MICCAI, 2023.arxiv.org/abs/2311.00476
  32. [35]Wang, S., Narasimhan, H., Zhou, Y., Hooker, S., Lukasik, M., and Menon, A. K. Robust distillation for worst-class performance: on the interplay between teacher and student objectives. In UAI, 2023.DOI
  33. [36]Wortsman, M., Ilharco, G., Gadre, S. Y., et al. Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, 2022.
  34. [37]Xie, S. M., Pham, H., Dong, X., et al. DoReMi: Optimizing data mixtures speeds up language model pretraining. In NeurIPS, 2023.arxiv.org/abs/2305.10429
  35. [38]Yang, A., Li, A., Yang, B., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
  36. [39]You, S., Xu, C., Xu, C., and Tao, D. Learning from multiple teacher networks. In KDD, 2017.DOI
  37. [41]Yuan, F., Shou, L., Pei, J., et al. Reinforced multi-teacher selection for knowledge distillation. In AAAI, 2021.arxiv.org/abs/2012.06048
  38. [42]Zhai, R., Dan, C., Kolter, Z., and Ravikumar, P. DORO: distributional and outlier robust optimization. In ICML, 2021.arxiv.org/abs/2106.06142
  39. [43]Zhang, Z., Sun, Z., Zhao, F., et al. Verify before you distill: Prompt-level teacher gating for on-policy distillation. arXiv preprint arXiv:2609.02998, 2026.arxiv.org/abs/2609.02998
  40. [44]Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. WildChat: 1m ChatGPT interaction logs in the wild. In ICLR, 2024.arxiv.org/abs/2405.01470
  41. [45]Zhou, J., Lu, T., Mishra, S., et al. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.arxiv.org/abs/2311.07911

Paper details

vlnc:W_2smrq5tx
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents