WorksComputer Science
Robust Domain-Mixture Scheduling for Multi-Teacher On-Policy Distillation
Abstract
Multi-teacher on-policy distillation (MOPD) merges the capabilities of multiple domain-expert teachers, such as mathematical reasoning, code generation, and instruction following, into a single compact student. Since all domains share a limited training budget, how to allocate it across domains directly determines the quality of distillation. Existing methods allocate the budget by each domain's learning progress, but offer no guarantee for lagging domains. We therefore introduce distributionally robust optimization (DRO) into MOPD to prioritize the worst-performing domain. However, the per-domain training signals are estimated from a few rollouts and are inherently noisy, so directly applying DRO mistakes noise for the worst case, lets occasional spikes hijack the budget, and protects inherently hard domains rather than truly regressing ones. To address these issues, we propose RDS-MOPD, which keeps the DRO objective and redesigns its three parts: it shifts the allocation only when differences clearly exceed the noise, bounds the influence of any single anomaly, and defines the worst case as the regression from each domain's own best level, all with theoretical guarantees. Experiments show that RDS-MOPD outperforms recent competing methods and ranks first on most mathematical reasoning, code generation, and instruction-following benchmarks.
Introduction
As large language model (LLM) services become deeply embedded in various real-world applications, users often issue requests of vastly different types within one continuous session, and expect the same model to both handle every task well and keep the conversation coherent [44]. Deploying a dedicated model for each type of task multiplies the inference and maintenance cost, and breaks the coherence of the session whenever the task switches. Consequently, merging multiple expert capabilities into a single compact student model has become a fundamental requirement for large-scale deployment [9]. Multi-teacher on-policy distillation (MOPD) is a promising post-training paradigm toward this goal [7, 32] (Ma et al. 2026), where the student learns from multiple domain-expert teachers simultaneously on its own sampled responses. Compared to offline sequence-level distillation, on-policy distillation provides dense supervision directly on the student’s own distribution, which alleviates the distribution shift between training and inference [1] (Gu et al. 2024).
In MOPD, all domains (e.g., mathematics, code, and instruction following (IF)) share a limited training budget. Hence, the domain mixture, i.e., how much the student learns from each teacher at every step, directly determines the quality of distillation. Open-MOPD [7] diagnoses the capability imbalance in multi-teacher distillation, and proposes to balance the token shares and follow the per-domain KL gaps. More recently, D-MOPD [32] dynamically allocates the budget according to the learning progress of each domain (i.e., the remaining KL gap and its descent velocity), steering the budget toward the domains that benefit the most. However, these methods all aim to accelerate the overall progress: they only favor “which domain improves the fastest” and offer no guarantee for lagging or regressing domains. In multi-capability merging, what users perceive is often the weakest capability, and the lag of any single capability directly hurts the service quality.
To this end, we introduce distributionally robust optimization (DRO) into domain-mixture scheduling for MOPD. DRO makes decisions against the most unfavorable case within an uncertainty set, and thereby prioritizes the worst-performing group [29, 16]. This idea naturally matches the goal of “leaving no capability behind”. Different from prior work that determines the pre-training data mixture offline with a proxy model [37], to the best of our knowledge, this is the first work that applies DRO to online mixture scheduling in MOPD. Nevertheless, directly applying DRO to MOPD (referred to as Vanilla DRO-MOPD) is confronted with a fundamental issue: every decision of DRO relies on the estimated per-domain training signals, which are computed from a limited number of rollouts per domain and are inherently noisy. As shown in Figure 1(a), on the training trajectory of our Vanilla DRO-MOPD implementation, the weight of the code domain surges after a KL burst and stays high until the end of training, even though code is no longer the worst-performing domain. Therefore, it is non-trivial to make reliable robust decisions under noisy training signals, and Vanilla DRO-MOPD mainly suffers from the following three instabilities (diagnosed in § 3.1).
Figure 1. Domain weights on the training trajectory. (a) Vanilla DRO-MOPD: a code KL burst (shaded) lifts the code weight from 0.30 to 0.56, which locks in (hatched); (b) RDS-MOPD keeps re-allocating the budget without lock-in.
Worst-Case Misjudgment. Vanilla DRO-MOPD moves its weights at the same rate whether the evidence is strong or weak, so it systematically up-weights the domain whose noisy estimate happens to be high. On the above trajectory, the current worst domain changes 52 times within 145 training steps, and on more than half (54%) of the non-spike steps, the gap between the worst and runner-up domains is below one noise standard deviation.
Spike Hijacking. Every observation enters the weights of Vanilla DRO-MOPD in proportion to its magnitude and is carried forward by all later updates, so an abnormal KL spike can seize a large share of the budget. On the above trajectory, the KL of the code domain rises to about 30× its typical level within a dozen steps, and its weight nearly doubles and stays high until the end.
Misplaced Protection. DRO is intended to protect the weak domains, but it compares the magnitude of losses, while the KL values given by different teachers naturally differ in scale. As a result, the budget flows to the domains with inherently large KL rather than the truly regressing ones: with the raw KL as the loss, Vanilla DRO-MOPD eventually allocates the whole budget to the code domain, whereas the IF domain, which regresses the most from its own best level, receives almost no protection.
To tackle these challenges, we propose a novel framework, namely Robust Domain-Mixture Scheduling for Multi-Teacher On-Policy Distillation (RDS-MOPD). Our key observation is that the three instabilities do not stem from the idea of DRO itself; instead, Vanilla DRO-MOPD builds all three of its parts (i.e., the deviation radius, the nominal allocation, and the definition of the “worst case”) upon unreliable signals. Hence, RDS-MOPD keeps the DRO objective and redesigns its three parts with three modules. Specifically, (1) to handle worst-case misjudgment, we propose Noise-aware Radius Calibration (NRC), which calibrates the deviation radius to the noise level, so that the evidence strength decides how far the allocation may deviate. It lower-bounds the temperature of the nominal allocation via the online-estimated noise floor, so that differences within the noise range only cause bounded share changes, while truly significant ones still receive a decisive response. (2) To handle spike hijacking, we design Spike-robust Allocation Estimation (SAE), which rebuilds the nominal allocation from KL observations clipped by a log-domain Huber rule, bounding the influence of spikes while still following persistent real changes. Meanwhile, SAE re-estimates the nominal allocation at every update rather than accumulating it from previous decisions, so an anomaly no longer persists in the budget. (3) To handle misplaced protection, we propose Worst-case Domain Reweighting (WDR), which redefines the “worst case” as each domain’s regression from its own best level, and exactly solves DRO around the progress-driven nominal allocation. It thus reweights the domains toward the truly regressing ones with a bounded amount of extra budget. As shown in Figure 1(b), under the same setting, RDS-MOPD keeps re-allocating the budget throughout training, and no domain is locked in. Theoretically, we prove that NRC and SAE guarantee bounded share changes and bounded influence of anomalies, respectively, and that WDR exactly solves a KL-ball DRO problem.
We distill a 3B student model from three domain-expert teachers for mathematics, code, and instruction following, and conduct extensive experiments on six benchmarks. The results show that RDS-MOPD outperforms recent competing methods, including Open-MOPD [7] and -MOPD (Sun et al., [32]), and ranks first on the majority of benchmarks. RDS-MOPD also makes no large share reversal in training, and in a stress test that injects noise into the code teacher’s signal, it still keeps a stable mixture and protects the learning progress of the other domains. These results demonstrate both the superiority of RDS-MOPD and the effectiveness of its modules. To summarize, our main contributions are threefold:
To the best of our knowledge, we are the first to introduce DRO into domain-mixture scheduling for MOPD, and formally diagnose three instabilities of Vanilla DRO-MOPD under noise.
We propose RDS-MOPD, a single DRO structure whose three modules fix these instabilities with theoretical guarantees.
Extensive experiments show that RDS-MOPD ranks first on most benchmarks, and validate the effectiveness of its modules.
Preliminaries
We introduce MOPD and its domain-mixture scheduling problem. Bold lowercase letters denote vectors (e.g., ) and calligraphic letters denote sets (e.g., ). denotes the probability simplex over domains, the uniform allocation ( is the all-ones vector), and the Kullback–Leibler divergence. We index training steps by and scheduling updates by (omitted when clear from the context), and summarize the notations in Table 3 of Appendix § 8.
| Notation | Meaning |
| , , , | probability simplex; uniform allocation ; all-ones vector; element-wise product |
| , , | domain set, a domain, number of domains () |
| , , | student; RL teacher routed to domain ; prompt pool of |
| , , , | prompt; on-policy student response; its length; token position |
| , , | top- reverse KL at position (reward ); top- size; distillation loss of |
| , , , , | training step; scheduling update; update interval; prompts per batch; total steps |
| , , | per-domain KL observation of at step ; its clipped value in SAE; domain- responses in the batch |
| , , | true KL level of at step ; best true level so far; log-noise |
| , | domain mixture at update ; nominal allocation (progress-driven in RDS-MOPD) |
| loss estimate in Eqn. (2) ( in Vanilla DRO-MOPD) | |
| , , | initial KL level; current EMA level; best (lowest) level so far |
| , , , | remaining gap; descent velocity; progress score (); largest score |
| , , , | EMA window; points for ; max./used rolling deltas |
| , , | EMA of window ; EMA weights; ratio |
| , , , , | base/effective temperature; share floor; jitter range/value |
| , , , | log-noise scale; SE of the leading score; noise window; noise-floor multiplier |
| , , | running median; median window; Huber clipping threshold |
| , , , | excess loss; excess cap; DRO (tilt) strength; coverage floor |
| , , | feasible set; per-domain floor; normalizing constant |
| , , | score bound in SEs; step of an anomaly or jump; anomaly size |
| , | KL radius; dual variable (proof of Theorem 3.3) |
Table 3. Notations and meanings.
Multi-Teacher On-Policy Distillation
Let be the set of domains (here, ) with . Each domain has a prompt pool and an RL-trained expert teacher . MOPD distills all teachers into one student policy , and routes each prompt to the teacher of its domain . Given , the student samples a response on-policy, and the routed teacher scores it with a single prefill. At each position of , the teacher computes the reverse KL from the next-token distribution of the student to its own, restricted to the student’s top- tokens (Gao et al., [7]). The dense token-level reward serves as the advantage of a critic-free policy-gradient update (Schulman et al., [30]). The distillation loss is the expected per-token over the domain- prompts and their on-policy responses. At training step , the per-domain KL signal is computed as:
where holds the domain- responses of step , and is the length of . As a one-batch estimate of , reuses the rewards already computed by the trainer (no extra forward pass), but is noisy, as each batch holds a limited number of rollouts per domain.
Domain-Mixture Scheduling
Every few training steps, a scheduler updates the domain mixture from the observed KL signals. Each batch then samples prompts in proportion to , with a random jitter (Appendix § 11). Between two scheduling updates, the student thus minimizes ; only decides how many prompts each domain receives, and the per-token loss is unchanged. Some schedulers fix the mixture before training, e.g., by prompt-pool size (Ma et al., 2026), or determine it offline with a proxy model (Xie et al., [37]). Others adapt to each domain’s learning progress online, e.g., Open-MOPD (Gao et al., [7]) reweights the per-domain rewards by token-share balancing and KL-gap following, and -MOPD (Sun et al., [32]) resamples domains by the remaining KL gap times its descent velocity.
Our RDS-MOPD Methodology
We first formulate DRO for MOPD and diagnose Vanilla DRO-MOPD (§ 3.1), then present RDS-MOPD with its three modules (§ 3.2, Figure 2) and its overall procedure (§ 3.6).
Figure 2. Illustration of our RDS-MOPD framework (left) and its robust scheduler (right).
DRO for MOPD
DRO FORMULATION FOR MOPD
DRO decides against the most unfavorable distribution within an uncertainty set around a nominal one [2, 4]. In MOPD, where the student minimizes (§ 2.2), a DRO scheduler picks the most unfavorable mixture, up-weighting domains with a higher estimated loss. With a KL penalty around a nominal allocation , the DRO decision is:
where is the feasible set. Eqn. (2) has three parts: (i) the deviation radius, set by (a larger lets move farther); (ii) the nominal allocation , the anchor that moves from; and (iii) the definition of the worst case, i.e., the loss estimate of .
Vanilla DRO-MOPD applies online Group DRO [29] to MOPD with each domain as a group, where may reweight either the losses or the sampling. At every training step (so ), it solves Eqn. (2) with , whose closed form is . Its nominal allocation is the previous weights, with , and its loss is the KL signal normalized by its initial level (early mean of ), , giving the exponentiated-gradient update:
INSUFFICIENCY OF VANILLA DRO-MOPD
We model each observation as its true level times a noise factor, i.e., , where the zero-mean log-noise may be heavy-tailed and differ in scale across domains. Unrolling Eqn. (3) from , taking the log-ratio of two domains and (which cancels the normalization), and writing splits it into a signal term and a noise term summed over steps :
Eqn. (4) shows how each part breaks under the noise. (1) Worst-case misjudgment (deviation radius). The radius, fixed by , ignores the noise level, so the noise term accumulates like a random walk whose spread grows with the number of steps, and even drifts toward the noisier domain since . (2) Spike hijacking (nominal allocation). Anchoring on the previous weights accumulates every observation, so a single observation leaves a term in the log-weight ratio of domain to any other domain at all later steps, which grows linearly in and never fades (e.g., the code burst in Figure 1(a)). (3) Misplaced protection (definition of the worst case). Let be the lowest true level of domain so far; the true level then splits into three factors:
Normalizing by removes only the teacher scale, so a domain that descended far but then regresses still ranks below a barely descended one. Thus, the problem lies not in the robust objective but in its three parts, calling for a noise-aware radius, a spike-robust nominal allocation, and a worst case defined by regression alone.
RDS-MOPD
RDS-MOPD keeps the robust objective and redesigns its three parts. At its -th update, it solves:
where is the nominal allocation, the excess loss, the tilt strength, and the feasible set, i.e., Eqn. (2) with as the loss.
The nominal allocation is recomputed at every update from the progress score [32] of each domain, where is the exponential moving average (EMA) of the clipped KL of domain (§ 3.4), is the remaining gap, and the descent velocity is the relative decrease of over recent steps (Appendix § 11):
where is the share floor and the temperature, so a domain with a large remaining gap that still descends fast receives a larger share. The softmax solves Eqn. (2) with , , and in place of , , and . Hence, sets the radius that turns noisy scores into shares (a larger keeps closer to ), while only bounds the step from to .
Accordingly, three modules redesign the three parts: (i) NRC calibrates the radius to the noise level (§ 3.3); (ii) SAE re-estimates at every update from clipped observations instead of the previous weights (§ 3.4); (iii) WDR defines the worst case as each domain’s excess loss h over its best level and solves Eqn. (6) exactly (§ 3.5).
Noise-Aware Radius Calibration (NRC)
NRC fixes worst-case misjudgment: a difference that does not clearly exceed the noise should not cause a large change in the allocation. A common choice is the max-normalized temperature ([32]), where is the base temperature and is the score of the leading domain (the leader). The softmax input is then , so the leader always has a normalized score of 1, whether its lead is far beyond the noise or only a noise artifact. Near a plateau, the velocities and thus approach 0, so the leader switches with the noise, and each switch moves a large share. NRC therefore calibrates the radius to the uncertainty of the leading score.
Noise-Calibrated Radius. We estimate the scale of the log-noise (§ 3.1.2) of each domain from the recent log-differences of its clipped observations (§ 3.4). The delta method then carries this scale to the standard error (SE) of each score (Appendix § 11). Let be the SE of the leading score, and the noise-floor multiplier. NRC lower-bounds by the noise floor :
Thus, as the evidence weakens, stays large, and p moves smoothly toward u instead of following a noisy leader (Figure 3(a)).
Figure 3. Three modules on the mixture simplex. (a) NRC sizes the KL ball around u by evidence; without it, p flips between vertices. (b) SAE keeps p bounded after a code spike; without it, p is dragged to the code vertex. (c) WDR’s solution q lies where the highest iso-line of touches the KL ball of p.
Proposition 3.1 (Bounded Share Change below the Noise Level). Let p be given by Eqns. (7) and (8) with . Suppose that, at two consecutive updates, all scores satisfy for some , with taken at the same update. In this case, no score exceeds SEs, and the noise floor is active. Then every nominal share changes by at most .
Proposition 3.1 (proved in Appendix § 15.§ 1) guarantees that differences within the noise level only cause bounded share changes, and the weaker the evidence, the smaller the change.
Spike-Robust Allocation Estimation (SAE)
SAE fixes spike hijacking: one abnormal observation should not dominate the allocation. Re-estimating the nominal allocation at every update stops a spike from accumulating, but the spike still enters every statistic built on linear averages (e.g., , , and ) in proportion to its size. SAE therefore first clips every new observation into a range around the recent median.
Log-Domain Huber Clipping. Since the noise is multiplicative (§ 3.1.2), we clip each new observation in the log domain, i.e., by a ratio to the median of the raw observations over the previous steps, (fewer in warm-up):
where restricts to , and is the clipping threshold. Normal fluctuations pass unchanged, and a spike keeps its direction while its size is bounded. This is the Huber -function ([10]) in the log domain, and all other statistics of RDS-MOPD are computed on the clipped sequence. Since the median uses the raw observations, a single anomaly cannot drag the center.
Proposition 3.2 (Bounded Influence and Trend Preservation). Let and . (i) Every clipped observation lies in for . (ii) Replacing one raw observation with by any changes the clipped sequence only at steps , and only within a range independent of . Hence any EMA level of the clipped sequence stays in a range independent of , whereas without clipping it grows linearly in . (iii) If the observations are noise-free and jump at step from one constant level to another, the clipped sequence equals the raw one from step on.
Hence, a single spike, however large, only has a bounded and short-lived effect, keeping p in a bounded region (Figure 3(b)), while a persistent real change still passes through after a short delay (proofs in Appendix § 15.§ 2). A sustained burst is thus treated as a real change, whose effect is bounded by WDR (Theorem 3.3(ii)) and fades once it ends.
Worst-Case Domain Reweighting (WDR)
WDR fixes misplaced protection: the worst case should be a regressing domain, not an inherently hard one. The nominal allocation alone does not protect such a domain either: a rising KL clips its to 0, giving it the smallest share. WDR thus redefines the worst case and solves the DRO problem around p.
Excess Loss. We measure whether a domain is regressing by its excess loss over its own best level:
where is the lowest EMA level of domain so far (current update included) and is the excess cap. Here, is the log regression factor of Eqn. (5) on the EMA, capped to bound the tilt in Eqn. (11). It is 0 at the best level and grows as regresses; as a ratio within one domain, it depends on neither the KL scale nor .
Robust Best Response in a KL Ball. Given , WDR solves Eqn. (6) with the feasible set , where and is the coverage floor: the tilt never pushes a domain below , and itself stays feasible. Without the floor, the solution is the closed form of §3.1.1 with replaced by (); the floor lifts any domain below to :
where is the unique normalizer with (computed exactly in Appendix § 11). Each share is thus multiplied by : the tilt is idle when all domains are at their best, and a more regressing domain gets more budget, up to times.
Theorem 3.3 (Exact KL-Ball DRO Solution). Let with , , , , and .
(i) Exact KL-ball DRO solution. Eqn. (6) has a unique maximizer, given by Eqn. (11), which also maximizes over the KL ball around . (ii) Bounded deviation without starvation. for all , and for of Eqn. (7) with . (iii) Targeted protection. If all are equal, ; otherwise, every domain with the largest has .
Intuitively, WDR is an exactly solved DRO decision (proofs in Appendix § 15.§ 3): within a KL ball around , attains the largest expected excess loss, where the highest iso-line of touches the ball (Figure 3(c)).
Overall Procedure
SAE clips each new KL observation at every training step (Eqn. (9)); at every scheduling update, RDS-MOPD computes , , and on the clipped sequences, estimates (Eqns. (7) and (8)), and tilts it into (Eqn. (11); Algorithm 1 in Appendix § 11). Setting , , or (then ) disables NRC, SAE, or WDR, respectively (§4.3.1). An update only reads the KL sequences already computed by the trainer (Eqn. (1)) and costs , with no extra forward pass or GPU memory (§4.5).
Experiments
We evaluate RDS-MOPD by answering the following questions:
RQ1: How does RDS-MOPD perform on math, code, and instruction following, compared with MOPD methods with a fixed or adaptive domain mixture and training-free model merging?
RQ2: How does each module of RDS-MOPD contribute to the performance, and does it resolve its corresponding instability?
RQ3: Can RDS-MOPD keep a robust mixture and protect the other domains when one domain teacher is corrupted in the middle of training?
RQ4: Does RDS-MOPD incur a large time overhead?
Experimental Setups
Benchmarks and Evaluation Metrics
We evaluate all methods on six benchmarks, two per domain teacher: AIME24 and AIME25 (Mathematical Association of America, [22]) for mathematical reasoning, LiveCodeBench v5 and v6 (LCB v5/v6) (Jain et al., [13]) for code generation, and IFEval (Zhou et al., [45]) and IFBench (Pyatkin et al., [25]) for instruction following (IF). Each domain pairs an earlier benchmark with a more recent one, so that both in-domain fitting and generalization to newer distributions are examined. We report avg@64 on AIME24/25, avg@10 on LCB v5/v6, and avg@4 of the strict prompt-level accuracy on IFEval/IFBench (higher is better for all). Benchmark statistics and evaluation configurations are given in Appendix § 9.
Training Data and Models
We build our testbed upon Open-MOPD (Gao et al., [7]), to our knowledge the only recent MOPD work that releases its complete training resources. We distill the SmolLM3-3B (Hugging Face, [11]) student, initialized from the released MixSFT checkpoint (Student-SFT), from the released RL-Math, RL-Code, and RL-IF teachers. The prompt pool contains 86,931 prompts, i.e., 17,917 math prompts from DAPO (Yu et al., 2025), 23,667 code prompts from PrimeIntellect (Prime Intellect, [24]) and TACO (Li et al., 2023), and 45,347 IF prompts from Nemotron-IF-RL (Gao et al., [7]), where each prompt is routed to the teacher of its domain. Per-domain statistics are given in Appendix § 9.
Competing Methods
We compare RDS-MOPD with two groups of competing methods: (1) state-of-the-art MOPD methods with a fixed or adaptive domain mixture, i.e., Vanilla MOPD (Ma et al., 2026), Open-MOPD (Gao et al., [7]), D-MOPD (Sun et al., [32]Sagawa et al., 2020`) (§3.1); and (2) training-free model merging, i.e.,
ParamMerge-Avg (Wortsman et al., [36]) and ParamMerge-TA (Ilharco et al., [12]). We also report RouteRL (Gao et al., [7]) as a routed-teacher reference, which is not a single deployable model and is excluded from the ranking. Details are given in Appendix § 10.
Implementation and Evaluation
All MOPD methods share the same training code and sampler, and differ mainly in the mixture rule. They use the dense top-16 teacher reverse KL as the per-token reward, and are trained for 150 steps with 1,024 prompts per step; the scheduling methods start from and update it every 10 steps, except Open-MOPD and Vanilla DRO-MOPD, which keep this sampling mixture and reweight the per-domain rewards and losses, respectively. Following D-MOPD, we report the mean and standard deviation over three seeds under two checkpoint selection criteria. Best-Overall Performance (BO) takes the single checkpoint with the best mean domain score of each run, and Best-per-Benchmark Performance (BB) takes the best checkpoint on each benchmark. As a scale-free summary, we also report the average per-benchmark rank (Avg. Rank; details in Appendix § 12).
Overall Performance (RQ1)
As shown in Table 1, we have the following threefold observations.
| Method | Math AIME24 | Math AIME25 | Code LCB v5 | Code LCB v6 | Instruction Following IFEval | Instruction Following IFBench | Avg. Rank (↓) | Recovery (%, ↑) | -value |
| Student-SFT | 15.10 | 15.16 | 15.99 | 15.49 | 66.82 | 18.50 | – | 0.0 | – |
| RouteRL | 24.38 | 24.32 | 22.99 | 21.94 | 73.57 | 22.50 | – | 100.0 | – |
| Best-Overall (BO) | |||||||||
| ParamMerge-Avg | 21.20 | 21.67 | 20.00 | 19.43 | 70.52 | 20.00 | 7.00 | 60.4 | |
| ParamMerge-TA | 23.33 | 24.11 | 22.46 | 21.26 | 73.20 | 22.83 | 5.00 | 94.1 | |
| Vanilla MOPD | 23.25 ± 0.29 | 23.30 ± 0.24 | 21.42 ± 0.12 | 21.30 ± 0.24 | 72.41 ± 0.29 | 21.03 ± 0.13 | 5.83 | 83.6 | |
| Open-MOPD | 24.64 ± 0.14 | 24.70 ± 0.24 | 22.46 ± 0.06 | 22.31 ± 0.13 | 74.26 ± 0.17 | 23.72 ± 0.09 | 2.67 | 105.6 | |
| D-MOPD | 24.77 ± 0.81 | 24.90 ± 0.37 | 22.91 ± 0.74 | 22.29 ± 0.46 | 74.18 ± 0.26 | 23.42 ± 0.80 | 2.58 | 106.5 | |
| Vanilla DRO-MOPD | 23.85 ± 0.26 | 24.69 ± 0.27 | 23.37 ± 0.23 | 22.07 ± 0.12 | 73.97 ± 0.07 | 23.42 ± 0.14 | 3.42 | 103.9 | |
| RDS-MOPD | 26.72 ± 0.16 | 26.09 ± 0.19 | 23.79 ± 0.19 | 23.01 ± 0.07 | 74.74 ± 0.10 | 23.08 ± 0.30 | 1.50 | 118.1 | – |
| Best-per-Benchmark (BB) | |||||||||
| ParamMerge-Avg | 21.20 | 21.67 | 20.00 | 19.43 | 70.52 | 20.00 | 7.00 | 60.4 | |
| ParamMerge-TA | 23.33 | 24.11 | 22.46 | 21.26 | 73.20 | 22.83 | 5.33 | 94.1 | |
| Vanilla MOPD | 23.34 ± 0.14 | 23.65 ± 0.09 | 21.74 ± 0.06 | 21.35 ± 0.15 | 72.63 ± 0.21 | 21.25 ± 0.14 | 5.67 | 86.5 | |
| Open-MOPD | 24.81 ± 0.03 | 24.90 ± 0.06 | 22.59 ± 0.07 | 22.38 ± 0.07 | 74.55 ± 0.14 | 23.81 ± 0.17 | 2.58 | 107.8 | |
| D-MOPD | 24.95 ± 0.56 | 25.11 ± 0.55 | 22.91 ± 0.74 | 22.38 ± 0.32 | 74.54 ± 0.47 | 23.50 ± 0.82 | 2.58 | 108.7 | |
| Vanilla DRO-MOPD | 24.17 ± 0.06 | 24.86 ± 0.06 | 23.51 ± 0.03 | 22.19 ± 0.09 | 74.03 ± 0.00 | 23.56 ± 0.10 | 3.33 | 106.1 | |
| RDS-MOPD | 26.72 ± 0.16 | 26.09 ± 0.19 | 23.95 ± 0.22 | 23.20 ± 0.11 | 74.88 ± 0.03 | 23.17 ± 0.17 | 1.50 | 119.5 | – |
Table 1. Overall performance (%, mean ± std over 3 seeds; training-free methods and reference rows have no std) under the BO (top) and BB (bottom) checkpoint selection criteria. Bold/underline: best/second-best among the ranked methods (gray reference rows excluded); “Avg. Rank” (↓): mean per-benchmark rank over the six benchmarks; “Recovery” (↑): (Total − SFT)/(RouteRL − SFT) following Open-MOPD, where Total is the mean of the three domain averages; -value: one-sided Wilcoxon test of RDS-MOPD vs. each method over 18 benchmark-seed pairs, treating pairs as independent (Appendix § 12).
(1) Training-free model merging can hardly fuse the capabilities of multiple teachers. ParamMerge-Avg ranks last on all six benchmarks, and ParamMerge-TA still falls behind all the adaptive MOPD methods (Avg. Rank of 5.00 and 5.33 under BO and BB). This is reasonable as the domain-specific RL updates of the teachers conflict and partly cancel out when merged in the parameter space, whereas MOPD aligns the student with each teacher token by token on its own responses. Yet MOPD alone does not guarantee this advantage: Vanilla MOPD with a fixed mixture even ranks below ParamMerge-TA (5.83 vs. 5.00 under BO), whereas the adaptive methods (Open-MOPD, D-MOPD, and RDS-MOPD) outperform Vanilla MOPD on all six benchmarks under both criteria. This indicates that the key lies in allocating the domain budget.
(2) Vanilla DRO-MOPD is unstable under noisy training signals. Although it up-weights the worst domain, Vanilla DRO-MOPD falls behind the progress-driven Open-MOPD and D-MOPD on four of six benchmarks (AIME24, AIME25, LCB v6, and IFEval), with an Avg. Rank of 3.42 and 3.33 under BO and BB. This is mainly because, besides the three instabilities of § 3.1, reweighting the losses under a fixed sampling mixture only rescales the gradients and amplifies the KL noise of the up-weighted domain. This indicates that robust scheduling should protect the domain regressing from its own best level rather than the one with the largest loss.
(3) Our RDS-MOPD consistently outperforms the existing methods. RDS-MOPD ranks first on five of six benchmarks with the lowest Avg. Rank of 1.50 under both BO and BB, clearly ahead of D-MOPD and Open-MOPD (2.58–2.67). The gains are largest on math, i.e., 1.95 and 1.19 points over the best baseline on AIME24 and AIME25 under BO, and also hold on code (e.g., 0.70 points on LCB v6) and IFEval. The only exception is IFBench, whose 58 constraint types are designed to be out-of-distribution (Pyatkin et al., 2025) and are all absent from our IF training prompts, which may limit the adaptability of the distilled student to them. Even so, all adaptive MOPD methods, including RDS-MOPD, exceed the IF teacher (22.50) on IFBench, whereas on the in-distribution IFEval, RDS-MOPD achieves the best result. RDS-MOPD also surpasses the domain teachers (RouteRL) on all six benchmarks, with all seed standard deviations within 0.30, whereas those of D-MOPD reach 0.81 and 0.74 on AIME24 and LCB v5 under BO. This is mainly because the existing progress-driven methods act on progress scores estimated from noisy KL curves (e.g., D-MOPD normalizes its softmax by the maximum score). Hence, noise-induced leader switches (Appendix § 13) and transient KL spikes can pull a large share of prompts toward one domain. The three modules of RDS-MOPD bound exactly these effects. This indicates that the bottleneck of adaptive scheduling is robustness to noisy signals rather than adaptivity itself. RDS-MOPD outperforms every competing method on at least 15 of the 18 benchmark-seed pairs.
Empirical Module Analyses (RQ2)
Ablation Study
To verify each module, we build three variants of RDS-MOPD, each disabling exactly one module by its hyper-parameter (§3.3) and keeping all other hyper-parameters unchanged. As shown in Table 2, disabling any module degrades RDS-MOPD on most benchmarks.
| Benchmark | RDS-MOPD | w/o NRC | w/o SAE | w/o WDR |
| Best-Overall (BO) | ||||
| AIME24 | 26.72 | 25.47 (↓4.68%) | 26.04 (↓2.54%) | 26.20 (↓1.95%) |
| AIME25 | 26.09 | 25.62 (↓1.80%) | 25.36 (↓2.80%) | 25.89 (↓0.77%) |
| LCB v5 | 23.79 | 22.93 (↓3.61%) | 23.47 (↓1.35%) | 23.65 (↓0.59%) |
| LCB v6 | 23.01 | 22.51 (↓2.17%) | 23.37 (↑1.56%) | 22.86 (↓0.65%) |
| IFEval | 74.74 | 74.58 (↓0.21%) | 74.82 (↑0.11%) | 75.74 (↑1.34%) |
| IFBench | 23.08 | 22.75 (↓1.43%) | 22.50 (↓2.51%) | 23.00 (↓0.35%) |
| Best-per-Benchmark (BB) | ||||
| AIME24 | 26.72 | 26.30 (↓1.57%) | 26.04 (↓2.54%) | 26.20 (↓1.95%) |
| AIME25 | 26.09 | 25.62 (↓1.80%) | 25.36 (↓2.80%) | 25.89 (↓0.77%) |
| LCB v5 | 23.95 | 23.59 (↓1.50%) | 23.47 (↓2.00%) | 23.65 (↓1.25%) |
| LCB v6 | 23.20 | 23.20 (0.00%) | 23.37 (↑0.73%) | 22.86 (↓1.47%) |
| IFEval | 74.88 | 74.58 (↓0.40%) | 74.82 (↓0.08%) | 75.74 (↑1.15%) |
| IFBench | 23.17 | 22.75 (↓1.81%) | 23.42 (↑1.08%) | 23.08 (↓0.39%) |
Table 2. Ablation study of RDS-MOPD (%), with the relative change compared to RDS-MOPD in parentheses (variants are single runs with seed 0).
w/o NRC drops on all six benchmarks under BO, most notably by 4.68% and 3.61% on AIME24 and LCB v5, as the scheduler without the noise floor pushes the budget toward a domain that leads by chance, i.e., the worst-case misjudgment. This implies the criticalness of deciding when to respond under noisy signals.
w/o SAE drops on four of six benchmarks under both criteria, most notably by 2.54% and 2.80% on AIME24 and AIME25. Without clipping, transient KL spikes of the code domain enter the statistics of the nominal allocation in proportion to their size, distorting the allocation and squeezing the other domains. This implies the criticalness of bounding the influence of any single anomaly.
w/o WDR drops on all benchmarks except IFEval under both criteria, e.g., by 1.95% on AIME24 and 1.47% on LCB v6 under BB, as the nominal allocation gives no extra protection to a domain regressing from its best level. This implies the criticalness of each domain’s own best level as the reference.
Module Stability Analyses
To understand why each module works, we examine whether it resolves its instability of Vanilla DRO-MOPD (§3.1). We replay RDS-MOPD and its variants without each module on the same recorded trajectory, so that any difference in decisions is attributable to that module alone. As shown in Figure 4, we have the following threefold observations.

Figure 4. RDS-MOPD vs. its variants without the corresponding module (“w/o module”), replayed on the same recorded trajectory. (a) Share of each domain vs. its progress gap in noise standard deviations (shaded: within one std); (b) share shift remaining 40 steps after a KL spike (lower is better); (c) share of the most regressing domain (higher is better).
(1) NRC resolves the worst-case misjudgment caused by noise. As shown in Figure 4(a), without NRC, the shares swing between 0.15 and 0.65 even when the gap between domains is below one noise standard deviation, i.e., the worst-case misjudgment. With NRC, they stay between 0.17 and 0.40 within the noise range, yet still rise to 0.57 on average once the gap exceeds two noise standard deviations. As a result, the share follows the gap much more closely (Spearman correlation of 0.83 vs. 0.55). This indicates that RDS-MOPD responds only to real learning gaps, while still adjusting the mixture decisively.
(2) **SAE prevents spikes from hijacking the training bud-_ get. As shown in Figure 4(b), without SAE, the share shift caused by an injected one-step KL spike grows with the spike size and lasts long after the spike disappears, i.e., the spike hijacking. The shift is still 0.027 after 40 steps under a 64× spike. With SAE, it stays at 0.002 regardless of the spike size, and its maximum right after the spike is bounded at 0.033, compared with 0.172 without SAE. This indicates that the influence of any single anomaly on RDS-MOPD is both bounded and short-lived.
(3) WDR protects the truly regressing domains. As shown in Figure 4(c), without WDR, the domain regressing the most from its own best level receives only 0.277 of the budget on average, even less than the 0.398 of the domains close to their best levels, i.e., the misplaced protection. With WDR, it receives more budget at 12 of the 13 updates from step 30, and its share rises to 0.382 on average. This indicates that RDS-MOPD protects the truly regressing domains.
Robustness under a Corrupted Teacher (RQ3)
In practice, the signal of a domain teacher may become unreliable during training, e.g., due to a faulty scoring service. To test whether RDS-MOPD stays reliable in such cases, we corrupt the code teacher from step 30 on. Specifically, we add zero-mean noise (re-sampled at every call) to its token log-probabilities within the student’s top-16 candidates, which inflates its KL signal by about 2.5×. We then compare RDS-MOPD with D-MOPD under the same seed and data stream for 60 steps, which cover the first three scheduling updates after the corruption. As shown in Figure 5, we have the following twofold observations.
Figure 5. RDS-MOPD vs. D-MOPD with the code teacher corrupted from step 30 (shaded). (a) Share of code prompts (dashed: RDS-MOPD without corruption); (b) math KL relative to its level at step 30 (lower is better).
(1) RDS-MOPD keeps a stable mixture under the corrupted teacher. As shown in Figure 5(a), the mixture of RDS-MOPD is almost unaffected by the corruption. Its code share deviates from that of its uncorrupted run by only 0.011 on average over steps 30–60, and averages 0.325 over steps 41–60. In contrast, D-MOPD allocates only 0.179 to code and moves the released budget almost entirely to math (+0.145 vs. −0.006 for RDS-MOPD). This is mainly because the inflated KL raises the excess loss of code, so WDR keeps a bounded protection on it. Meanwhile, NRC treats the remaining score differences within the noise level as no evidence for re-allocation, so RDS-MOPD is not dragged by an unreliable teacher.
(2) RDS-MOPD preserves the learning progress of all domains. RDS-MOPD allocates about 1.8× as many prompts to the corrupted code domain as D-MOPD, yet ends with a lower code KL (0.0164 vs. 0.0225 over steps 56–60). Meanwhile, as shown in Figure 5(b), its math KL decreases to 0.66× its level at step 30, compared with 0.79× for D-MOPD. Thus, RDS-MOPD neither abandons the corrupted domain nor sacrifices clean ones.
Efficiency Analysis (RQ4)
We measure the per-step wall-clock time of the MOPD methods on the same hardware (16 A800 GPUs; stages in Figure 6) and the time of the scheduling decision, with the following twofold observations.
Figure 6. Wall-clock time per training step of MOPD methods.
(1) The scheduling decision of RDS-MOPD introduces negligible overhead. The RDS-MOPD scheduler takes only 0.039 milliseconds per step on average, more than seven orders of magnitude below a training step of about 750 seconds. This is mainly because all three modules operate on a few per-domain scalars, with no extra forward pass or GPU memory. Thus, the robustness of RDS-MOPD comes from its decision rule, not extra computation.
(2) The end-to-end training cost of RDS-MOPD is on par with or even lower than that of existing methods. Over steps 2–84, RDS-MOPD takes 747.5 seconds per step on average, vs. 758.2, 764.5, and 762.6 seconds for Open-MOPD, D-MOPD, and Vanilla DRO-MOPD. Over the full 150 steps, it is 5.6% faster per step than D-MOPD, as rollout takes about 45% of each step and the student of RDS-MOPD produces shorter responses (7,130 vs. 8,127 tokens on average). Thus, RDS-MOPD distills better at no extra cost.
Related Work
Multi-Teacher and On-Policy Distillation
Knowledge distillation fits a student to the teacher’s output distribution [9], and early multi-teacher methods fuse the soft labels or intermediate representations of multiple teachers [39], or select the teachers for each sample by reinforcement learning [41]. For LLMs, sequence-level distillation [15] suffers from the distribution shift between training and inference, which on-policy distillation (OPD) alleviates by supervising the student on its own sampled responses [1] (Gu et al. 2024), e.g., with the reverse KL divergence to avoid overestimating the low-probability regions of the teacher (Gu et al., 2024), and has become a standard post-training stage [19, 38], as its per-token supervision is denser than the sequence-level rewards of RL. Extending it to multiple domain-expert teachers leads to MOPD (Ma et al., 2026), which merges the capabilities of domain-specific RL teachers into one student. Subsequent work improves MOPD by balancing the capabilities of different teachers via token-share balancing and KL-gap following [7], weighting or gating teacher supervision by its reliability [31, 43], resolving the conflicts among teachers [33, 3], and distilling at the representation level [6]. Most of them keep the domain mixture fixed and refine how the teacher supervision is weighted or combined. Closer to our work, D3-MOPD [32] schedules the domain mixture by the learning progress of each domain, i.e., its remaining KL gap times its descent velocity, MT-GRPO [27] reweights tasks online toward the worst-performing task in multi-task RL, and concurrent work [18] sets model-merging coefficients before training by minimizing the worst per-domain residual. Different from these works, RDS-MOPD introduces DRO into the online domain-mixture scheduling of MOPD and makes it robust to noise.
Distributionally Robust Optimization
DRO optimizes the worst-case loss within an uncertainty set around the nominal distribution [2, 4], where the uncertainty set can be defined by, e.g., the KL divergence, whose worst case admits a closed-form exponential tilting [16], the Wasserstein distance [5], or a C-VaR-type tail risk [28]. Group DRO [29] instantiates this idea over predefined groups by up-weighting high-loss groups online, and has been widely adopted, e.g., in topic-robust language modeling [23], data mixture optimization for pre-training [37], group-robust preference alignment [26], and worst-group knowledge distillation [14, 35, 34]. Among them, DoReMi [37] runs Group DRO on a small proxy model to determine the pre-training domain weights offline. However, DRO is known to be sensitive to outliers, since the worst case tends to be dominated by them, and DORO [42] thus discards a fraction of the highest-loss samples. This motivates combining DRO with robust statistics against heavy-tailed noise, e.g., the Huber estimator [10] and median-of-means [20]. Following this line, RDS-MOPD fixes three concrete instabilities of DRO under the noisy signals of MOPD within a single DRO structure.
Conclusion and Future Work
In this paper, we propose RDS-MOPD, a robust domain-mixture scheduling framework for multi-teacher on-policy distillation. It resolves the instabilities of Vanilla DRO-MOPD under noisy signals with three modules, backed by theoretical guarantees. Extensive experiments on mathematical reasoning, code generation, and instruction-following benchmarks demonstrate the effectiveness and robustness of RDS-MOPD with negligible computational overhead. As for future work, we plan to extend the robust allocation from the domain level to finer granularities, e.g., the prompt level.
Notations
Benchmark and Training Data Details
Table 4 lists the six benchmarks: AIME24/25 (Mathematical Association of America, [22] contain the 2024/2025 AIME problems with integer answers judged by exact match. LiveCodeBench v5/v6 (Jain et al., [13]) collect competitive programming problems by release window and judge each solution by hidden test cases, which alleviates contamination. IFEval (Zhou et al., [45]) contains programmatically verifiable constraints (e.g., length, format, and keywords), and IFBench (Pyatkin et al., [25]) adds novel out-of-distribution constraints; a response is correct only if it satisfies all constraints under the strict prompt-level criterion. We score each IF sample against its own response, unlike the released script of Open-MOPD (Gao et al., [7]), which gives all samples of a prompt one verdict. All metrics are pass@1 estimated by avg@#Samp., with thinking mode, top-, and at most 30k generated tokens, and each run is summarized over its checkpoints by BO and BB (Appendix § 12).
We build our testbed on Open-MOPD (Gao et al., [7]), to our knowledge the only recent MOPD work that releases its complete training resources (SFT student, domain teachers, and prompts), so all methods share the same student, teachers, and data. The student is SmolLM3-3B (Hugging Face, [11]) initialized from the released MixSFT checkpoint (Student-SFT), and the teachers RL-Math, RL-Code, and RL-IF are obtained by domain-specific RL on top of Student-SFT. The prompt pool (Table 4) draws math from DAPO (Yu et al., 2025), code from PrimeIntellect (Prime Intellect, [24]) and TACO (Li et al., 2023), and IF from Nemotron-IF-RL (Gao et al., [7]). All trained methods share this pool and differ mainly in the domain mixture (except Vanilla DRO-MOPD, Appendix § 11).
| Domain | Evaluation Benchmark | #Prob. | #Samp. | Temp. | Source | Training #Prompts | Teacher |
| Math | AIME24 | 30 | 64 | 0.6 | DAPO | 17,917 | RL-Math |
| Math | AIME25 | 30 | 64 | 0.6 | DAPO | 17,917 | RL-Math |
| Code | LCB v5 | 167 | 10 | 1.0 | PrimeIntellect + TACO | 23,667 | RL-Code |
| Code | LCB v6 | 175 | 10 | 1.0 | PrimeIntellect + TACO | 23,667 | RL-Code |
| IF | IFEval | 541 | 4 | 1.0 | Nemotron-IF-RL | 45,347 | RL-IF |
| IF | IFBench | 300 | 4 | 1.0 | Nemotron-IF-RL | 45,347 | RL-IF |
Table 4. Benchmarks and training data per domain (#Prob.: problems; #Samp.: samples per problem, scored by avg@#Samp.; Temp.: sampling temperature).
Baseline Details
Vanilla MOPD (Ma et al., 2026; Sun et al., [32]) fixes the mixture to each domain’s share of the prompt pool. Open-MOPD (Gao et al., [7]) adds token-share balancing, KL-gap following, and a reward refresh within every inner update. Vanilla DRO-MOPD (Sagawа et al., 2020) normalizes each per-domain loss by its initial level; its strength is set as in Appendix § 11. D³-MOPD (Sun et al., [32]) samples each domain by the softmax of its remaining KL gap times its descent velocity, with a share floor. ParamMerge-Avg Wortsman et al., [36] averages the parameters of the three teachers. ParamMerge-TA Ilharco et al., [12] adds the scaled sum of the teachers’ task vectors (differences from the SFT student) to the student; we report the better of scaling factors 0.5 and 1.0. The reference rows of Table 1 are excluded from the ranking: Student-SFT is the starting point of all trained methods, and RouteRL Gao et al., [7] answers each prompt with its domain teacher (RL-Math for AIME24/25, RL-Code for LCB v5/v6, and RL-IF for IFEval/IFBench), which needs an oracle router and all teachers.
Implementation Details
Training. All methods use the dense top-16 reverse KL of the routed teacher as the per-token reward, prompts per step, a mini-batch size of 256, 150 steps, a learning rate of , and at most 16k response tokens. Trained methods start from (math : code : IF), except Vanilla MOPD (pool-proportional), and the scheduling methods update the mixture every steps; Open-MOPD and Vanilla DRO-MOPD reweight the per-domain rewards and losses instead (§ 3.1.1). Each batch jitters the mixture as with and draws prompts by the largest-remainder rule. All MOPD methods, including the ablation variants, share one training code and sampler and differ mainly in the mixture rule.
Hyper-parameters. RDS-MOPD uses , (NRC), , (SAE), , , (WDR), , , , , , and (sensitivity in Appendix § 14). Baselines use their official settings; Vanilla DRO-MOPD updates the domain weights by exponentiated gradient on the per-domain loss normalized by its initial level, with strength .
Scheduler Formulas. The progress score follows D-MOPD Sun et al., [32]: , and averages the relative decrease of , the EMA over the -th most recent window of clipped observations (), for , clipped at 0. NRC estimates the log-noise scale over the last observations as
and the delta method gives
where are the EMA weights over one window and ; with . During warm-up (), the median of SAE uses the available observations. WDR tilts and normalizes , then iteratively clamps the coordinates below their floor and rescales the others, which returns Eqn. (11) exactly within rounds (Appendix § 15.§ 3).
Algorithm 1: Training procedure of RDS-MOPD
Input: Student ; teachers ; pools ; batch size ; steps ; interval ; hyper-parameters (Appendix § 11).
Output: Distilled student .
;
for do Sample prompts from by the jittered ; Roll out, score with the routed teachers, and update with the reward (§ 2.1);
foreach do Read by Eqn. (1) and clip it into by Eqn. (9); // SAE
if is a scheduling step (every steps after observations per domain) then Get , , , and on (Appendix § 11); Get by Eqn. (8) and by Eqn. (7); // NRC Update and by Eqn. (10); the solution of Eqn. (11); // WDR
return ;
Evaluation Setup. For each of the three seeds, we evaluate the checkpoints at steps 90, 100, 120, and 150, summarize them by the BO and BB criteria (Appendix § 12), and apply the same rule to all methods. The main runs use two nodes of eight NVIDIA A800-80GB GPUs, the ablation variants one such node, and the stress test four GPUs, with Python 3.12, PyTorch 2.8.0 (CUDA 12.8), verl 0.7.0, vLLM 0.11.0, and Transformers 4.57.
Evaluation Details
Each benchmark is scored at every evaluated checkpoint (steps 90, 100, 120, and 150), and each run is summarized by two criteria. Best-Overall (BO) selects the single checkpoint with the best mean of the three domain scores (each averaging its two benchmarks) and reports all six benchmarks there; it reflects one deployable model. Best-per-Benchmark (BB) takes the best score of each benchmark over all checkpoints separately. We report the mean and standard deviation over three seeds; training-free methods and reference rows have identical results under both criteria. As a scale-free summary, Avg. Rank (↓) ranks all ranked methods on each benchmark by their 3-seed means (1 is best, ties in the unrounded means averaged) and averages over the six benchmarks, per criterion. For significance, as three seeds are too few, we run the one-sided Wilcoxon signed-rank test between RDS-MOPD and each competitor over all 18 benchmark-seed pairs (six benchmarks × three
/* page continues */ seeds), per criterion, treating pairs as independent. RDS-MOPD wins on at least 15 out of the 18 pairs against every competitor.
Mixture Dynamics
We compare the real training runs of RDS-MOPD and D-MOPD (same seed and hardware). An update is a large share reversal if any domain share changes by more than 0.3, and the mean share change of a run averages . A large reversal is noise-driven if the same scheduler reproduces it (same update, domain, and direction) in fewer than half of 100 bootstrap replicas of the KL trajectory, resampling residuals around a 21-step running median.
(1) D-MOPD oscillates the mixture with noise. It makes 7 large share reversals in 14 updates, 5 of which are noise-driven (reappearing in only 13–28% of the replicas), e.g., from step 100 to 140 the math share alternates between 0.175 and 0.651 (Figure 7(a)) while the math KL stays within 0.002–0.006. This is mainly because its temperature shrinks with the progress scores (from 0.086 at step 20 to about 0.006 after step 90), so batch-level KL fluctuations alone move most of the budget, i.e., the worst-case misjudgment of § 3.1.
Figure 7. Domain shares during training (seed 0). Filled/open triangles: noise-driven/other large reversals of D-MOPD, marked at the most-changed domain; RDS-MOPD has none.
(2) RDS-MOPD keeps the mixture responsive yet stable. It makes no large share reversal and cuts the mean share change from 0.258 to 0.118 (54%) and the maximum from 0.478 to 0.285 (Figure 7), while still following real gaps, e.g., the IF share rises to 0.643 at step 140 as only IF still descends. This is mainly because NRC calibrates the radius to the estimated noise level.
Hyper-parameter Sensitivity
We vary (NRC) and (WDR) with the others at their defaults (Appendix § 11); and are w/o NRC and w/o WDR. We replay the scheduler on the recorded KL trajectory of RDS-MOPD (150 steps, 14 updates; Figure 8) to isolate the scheduling decisions, with reversals as in Appendix § 13. The lagging domain at an update is the one whose KL regresses most from its own best level.
Figure 8. Sensitivity to of NRC (a, b) and of WDR (c, d) (▲: default , , noise-driven reversals per bootstrap replica (mean std); (b) mean share change per update; (c) lagging-domain share; (d) max final share (dashed: , i.e., two domains at ).
(1) A moderate noise floor reduces noise-driven reversals without freezing the mixture. Raising from 0 to 4 cuts noise-driven reversals on the recorded trajectory from 3 to 0 and the mean share change from 0.200 to 0.118 (41%); is the smallest value removing all large reversals (5, 4, and 3 for , 1, and 2) while keeping a largest single-update change of 0.285, whereas and 16 (0.04 and 0.00 noise-driven reversals) suppress it to 0.158 and 0.161, making the scheduler sluggish.
(2) The tilt strength controls how much budget goes to the lagging domain. Its share grows monotonically with under every , e.g., 0.277, 0.327, 0.382, and 0.499 for , 0.5, 1, and 2 under , so the default adds 0.105 as no extra reversals.
(3) The excess cap prevents over-concentration. Under , the largest final share grows from 0.393 to 0.464 and 0.565 for , and 3, and , ends at (0.200, 0.200, 0.600) with two domains on the floor and 1 noise-driven reversal, whereas the default ends at (0.288, 0.249, 0.464).
Proofs
Proof of Proposition 3.1
Proposition 3.1 is part (i); the full statement adds (ii) for a given and any , ; and (iii) with , Eqn. (8) reduces to the max-normalized temperature. Under the max-normalized temperature, such scores can instead change a share by for any .
Lemma H.1 ( Sensitivity of Softmax). For any , , where . Moreover, any with satisfies .
Proof. Let and for ; then , so with , the mean absolute deviation of supported on with ( is trivial). By convexity, on , so . Integrating over gives the first claim; for the second, the positive and negative entries of each sum to in absolute value.
(i)
Let for . Since , Eqn. (8) gives , so the softmax input lies in and . By Lemma H.1, , so (both allocations sum to one) , i.e., for , , , and . For the max-normalized response, and with normalize to and for any , so the first share drops from to by default, i.e., by .
(ii)
Let , , and , so that the softmax input is (note for ). Then . Both terms are bounded by in , the second since and is 1-Lipschitz in . Hence the oscillation of the softmax input difference is at most , and Lemma H.1 with the factor gives the claim. For the max-normalized response, replacing by gives a constant that diverges as the maximum score tends to zero, as the example in (i) realizes.
(iii)
With , for , i.e., the max-normalized temperature.
Proof of Proposition 3.2
(i) Directly from Eqn. (9). (ii) Primes denote contaminated quantities; and are the smallest and largest other observations in . As the median uses raw observations, only if , and for the window and current observation are clean, so . For , the window is clean (as ) and lies in , so , and (i) gives for any . For , the window contains the contaminated value and clean values in . The median involves only ranks in (as ): the -th order statistic with for odd , or the mean of the -th and -th with for even . Each such order statistic lies between two order statistics of the remaining values, so . The clean , and clipping moves it toward , so . An EMA level is a convex combination of clipped observations, each equal to its clean counterpart or in , so it lies in an interval independent of ; without clipping, the EMA weight of step makes it at least .
(iii) Let for and for . For , the window of (of size at most ) contains at least values equal to (a strict majority), so lies in the clipping range and .
Proof of Theorem 3.3
The full statement adds: in (i), the maximizer is the KL projection of (normalized) onto ; (ii) also gives and ; in (i) with , when the floor is inactive, , attained at . Let (as ) and .
(i), unique maximizer. For any , expanding gives
Hence Eqn. (6) is the KL projection onto the convex , unique by strict convexity. With multipliers and of the simplex and floor constraints, the KKT conditions , , and give if , and otherwise, with , i.e., , which is Eqn. (11) with . is continuous and nondecreasing with , so has a root in , and it is unique: at any root, some has (otherwise ), and as , for all .
Exactness of the implementation. From (Appendix D), the unclamped coordinates after round are , . Clamping only raises coordinates, so ; the clamped set never shrinks, and the unclamped set is never empty as , so the procedure stops within rounds, where every unclamped satisfies , and @references♀♀♀♀♀♀ every clamped , clamped at some round , satisfies . Hence with a unit sum, and by uniqueness.
(i), KL ball. For any with , the optimality of in Eqn. (6) gives , and is feasible, so it solves the constrained problem. For the dual form, Eqn. (13) with in place of gives, for any and , , i.e., weak duality [16]. When the floor is inactive, and Eqn. (13) gives , i.e., equality at .
(ii), bounded deviation. Let and its complement, as . Let , so that for , and for . Upper bound: for , as ; for , . Lower bound: if , all ratios are below one on and at most one on , contradicting ; hence some has , and for every , , since . Finally, . (ii), no starvation. by feasibility, and by Eqn. (7) as , so .
(iii) If , then , so . Let , so that . If , then every has , and every has , contradicting . Hence , and by (ii), $q_{d^\dagger}/p_{d^\dagger} = \max_{d \in F} \psi_d \ge 1. ∎
References
- [1]Agarwal, R., Vieillard, N., Zhou, Y., et al. On-policy distillation of language models: Learning from self-generated mistakes. In ICLR, 2024.arxiv.org/abs/2306.13649
- [2]Ben-Tal, A., den Hertog, D., Waegenaere, A. D., Melenberg, B., and Rennen, G. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 2013.
- [3]Chen, T., Ou, J., Liu, Z., Tang, R., Liang, J., and Li, H. Counteraction-aware multi-teacher on-policy distillation for general capability recovery with domain preservation. arXiv preprint arXiv:2605.27115, 2026.DOI
- [4]Duchi, J. C. and Namkoong, H. Learning models with uniform performance via distributionally robust optimization. Ann. Stat., 2021.arxiv.org/abs/1810.08750
- [5]Esfahani, P. M. and Kuhn, D. Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations. Math. Program., 2018.arxiv.org/abs/1505.05116
- [6]Fang, Z., Hong, S., Yang, J., et al. Latent-MOPD: Latent multi-teacher on-policy distillation. arXiv preprint arXiv:2610.02381, 2026.arxiv.org/abs/2610.02381
- [7]Gao, H., Chi, H., Yan, Y., et al. Open-MOPD: Diagnosing and fixing capability imbalance in multi-teacher on-policy distillation. arXiv preprint arXiv:2608.19098, 2026.arxiv.org/abs/2608.19098
- [9]Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.arxiv.org/abs/1503.02531
- [10]Huber, P. J. Robust estimation of a location parameter. Ann. Math. Stat., 1964.
- [11]Hugging Face. SmolLM3: Smol, multilingual, long-context reasoner. Hugging Face blog, 2025.
- [12]Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. In ICLR, 2023.arxiv.org/abs/2212.04089
- [13]Jain, N., Han, K., Gu, A., et al. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In ICLR, 2025.arxiv.org/abs/2403.07974
- [14]Kenfack, P., Aïvodji, U., and Kahou, S. E. Adaptive group robust ensemble knowledge distillation. TMLR, 2025.arxiv.org/abs/2411.14984
- [15]Kim, Y. and Rush, A. M. Sequence-level knowledge distillation. In EMNLP, 2016.arxiv.org/abs/1606.07947
- [16]Lam, H. Robust sensitivity analysis for stochastic systems. Math. Oper. Res., 2016.arxiv.org/abs/1303.0326
- [18]Liu, K., Zhuang, Z., Ma, H., et al. A fisher-geometric view of multi-teacher on-policy distillation. arXiv preprint arXiv:2610.26525, 2026.arxiv.org/abs/2610.26525
- [19]Lu, K. and Thinking Machines Lab. On-policy distillation. Thinking Machines Lab: Connectionism (blog post), 2025.
- [20]Lugosi, G. and Mendelson, S. Mean estimation and regression under heavy-tailed distributions: A survey. Found. Comput. Math., 2019.arxiv.org/abs/1906.04280
- [22]Mathematical Association of America. American invitational mathematics examination (AIME) 2024 and 2025, 2025.
- [23]Oren, Y., Sagawa, S., Hashimoto, T. B., and Liang, P. Distributionally robust language modeling. In EMNLP, 2019.
- [24]Prime Intellect. SYNTHETIC-1: Two million collaboratively generated reasoning traces from DeepSeek-R1. Prime Intellect blog, 2025.
- [25]Pyatkin, V., Malik, S., Graf, V., et al. Generalizing verifiable instruction following. arXiv preprint arXiv:2507.02833, 2025.arxiv.org/abs/2507.02833
- [26]Ramesh, S. S., Hu, Y., Chaimalas, I., et al. Group robust preference optimization in reward-free RLHF. In NeurIPS, 2024.arxiv.org/abs/2405.20304
- [27]Ramesh, S. S., Ji, X., Zimmer, M., et al. Multi-task GRPO: Reliable LLM reasoning across tasks. In ICML, 2026.arxiv.org/abs/2602.05547
- [28]Rockafellar, R. T. and Uryasev, S. Optimization of conditional value-at-risk. J. Risk, 2000.DOI
- [29]Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In ICLR, 2020.arxiv.org/abs/1911.08731
- [30]Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL https://arxiv.org/abs/1707.06347.
- [31]Sun, J., Zheng, M., Song, M., et al. Distill what you trust: Reliability-aware multi-teacher on-policy distillation. arXiv preprint arXiv:2609.23697, 2026a.arxiv.org/abs/2609.23697
- [32]Sun, Z., Zhang, Z., Zhao, F., et al. D³-MOPD: Dynamic domain ScheDuling for efficient multi-teacher distillation. arXiv preprint arXiv:2608.24987, 2026b.arxiv.org/abs/2608.24987
- [33]Tian, S., Liu, Y., Liu, X., et al. Beyond domain-routed supervision: Cross-teacher guarding for multi-teacher on-policy distillation. arXiv preprint arXiv:2610.13521, 2026.arxiv.org/abs/2610.13521
- [34]Vilouras, K., Liu, X., Sanchez, P., O’Neil, A. Q., and Tsaf-taris, S. A. Group distributionally robust knowledge distillation. In MLMI@MICCAI, 2023.arxiv.org/abs/2311.00476
- [35]Wang, S., Narasimhan, H., Zhou, Y., Hooker, S., Lukasik, M., and Menon, A. K. Robust distillation for worst-class performance: on the interplay between teacher and student objectives. In UAI, 2023.DOI
- [36]Wortsman, M., Ilharco, G., Gadre, S. Y., et al. Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, 2022.
- [37]Xie, S. M., Pham, H., Dong, X., et al. DoReMi: Optimizing data mixtures speeds up language model pretraining. In NeurIPS, 2023.arxiv.org/abs/2305.10429
- [38]Yang, A., Li, A., Yang, B., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
- [39]You, S., Xu, C., Xu, C., and Tao, D. Learning from multiple teacher networks. In KDD, 2017.DOI
- [41]Yuan, F., Shou, L., Pei, J., et al. Reinforced multi-teacher selection for knowledge distillation. In AAAI, 2021.arxiv.org/abs/2012.06048
- [42]Zhai, R., Dan, C., Kolter, Z., and Ravikumar, P. DORO: distributional and outlier robust optimization. In ICML, 2021.arxiv.org/abs/2106.06142
- [43]Zhang, Z., Sun, Z., Zhao, F., et al. Verify before you distill: Prompt-level teacher gating for on-policy distillation. arXiv preprint arXiv:2609.02998, 2026.arxiv.org/abs/2609.02998
- [44]Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. WildChat: 1m ChatGPT interaction logs in the wild. In ICLR, 2024.arxiv.org/abs/2405.01470
- [45]Zhou, J., Lu, T., Mishra, S., et al. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.arxiv.org/abs/2311.07911






