WorksComputer Science
Whitened Advantage Summation Policy Optimization: Correlation-Aware Aggregation for Multi-Reward Reinforcement Learning
Abstract
Reinforcement learning for language models often combines several rewards by normalizing each one and adding them with fixed weights. When the rewards are correlated, the variation they share is counted once for every reward that carries it. In terms of the ZCA-decorrelated components of the reward signal, the weighted sum applies weights rather than the chosen weights , where is the reward correlation matrix. We study WAS-PO, which whitens the normalized advantages with before adding them, so that the chosen weights apply to those components. We derive the scalarized objective whose gradient it follows when normalization statistics are held fixed, show that its weights on the raw rewards can be negative, give an exact two-reward condition under which the lighter-weighted reward is reversed, and give a loose bound on the error from estimating during training. On tool calling with Qwen2.5-1.5B-Instruct, WAS-PO reached a higher mean training reward than GDPO in all seven seeds with three rewards (3.8% higher) and all three seeds with four (3.9% higher). WAS-PO also replaces GDPO's per-prompt normalization with batch-level normalization, and we did not separate the two changes with three or four rewards. The training-reward gains did not carry over to held-out accuracy, which was slightly lower than GDPO's with three rewards (not significant) and 0.03 lower in absolute terms with four (two seeds evaluated). With two rewards, WAS-PO trailed GDPO in training reward, and a three-seed control weakly suggests that the gap comes from batch-level normalization rather than whitening. In a math setting where the two rewards were nearly identical by construction, WAS-PO drove down the lighter-weighted reward, as the sign condition predicts.
Introduction
Reinforcement learning for language models now routinely optimizes several rewards at once, such as correctness, output format and length. Group Relative Policy Optimization (GRPO; Shao et al., 2024) handles this by summing the rewards and normalizing the sum within each group of completions sampled for a prompt. Liu et al. [16] showed that summing first maps distinct combinations of rewards to the same advantage, and proposed Group reward-Decoupled Normalization Policy Optimization (GDPO), which normalizes each reward within the group before taking a weighted sum.
The weights in that sum are meant to say how much each objective matters, but correlated rewards change what they do. When normalized advantages of correlated rewards are added, the variation they share enters the sum once per reward. In terms of the decorrelated components of the reward signal (§ 3.1), a weighted sum with weights applies weights , where is the correlation matrix of the rewards. Correlation of this kind is common in multi-reward training. In our tool-calling setup, following GDPO’s conditional rewards, the format, length and structure rewards are set to zero whenever accuracy is not positive, so they rise and fall together largely by construction. At the batch-level correlations that WAS-PO estimated in our runs, the effective weight vector of the batch-normalized sum points away from by , and with two, three and four rewards. Whether this is a distortion depends on how the weights are read, which we discuss in § 3.1.
Whitening the normalized advantages with (ZCA whitening) before summing applies to the decorrelated components that are closest to the individual rewards. We call the result Whitened Advantage Summation Policy Optimization (WAS-PO). Whitening is not free. WAS-PO is a scalarized policy gradient whose weights on the raw rewards, with the diagonal matrix of reward standard deviations, can be negative (Proposition 2). With two rewards weighted , the lighter-weighted reward receives a negative weight exactly when the correlation exceeds , so the policy is trained to lower that reward, and the negative weight grows without bound as the correlation approaches one (Proposition 3). We call this ZCA-Inversion and observe it in a math setting, where the format reward fell from 0.67 to 0.13 during training.
In our three- and four-reward tool-calling settings, WAS-PO reached higher training reward than GDPO in all ten paired runs (seven at , three at ). The gain did not carry over to held-out accuracy, and with two rewards WAS-PO trailed GDPO even in training reward. WAS-PO also replaces GDPO’s per-prompt normalization with batch-level normalization, so the comparison with GDPO mixes two design changes. With two rewards, a control that keeps batch-level normalization but drops whitening (WAS-PO-Diag) scores slightly below WAS-PO, which weakly suggests that normalization is the source of the two-reward gap. We have not run this control with three or four rewards, so we report the gains there as gains of WAS-PO as a whole.
Our contributions are:
an exact statement of how correlation changes the effective weights of a weighted sum of normalized advantages, with a variance decomposition (§ 3.1);
the objective that whitened aggregation follows, and the corresponding per-prompt objective of GDPO (§ 3.2);
an exact two-reward condition for sign reversal, its extension to near-duplicate rewards among , and an inversion index that can be logged during training (§ 3.3);
an error bound for the estimated whitening matrix (§ 3.4);
experiments on tool calling with two, three and four rewards, component ablations, an observed instance of ZCA-Inversion, and runs in which a nearly singular estimate did not cause collapse (§ 5).
Background
For a prompt , a policy samples completions . Each completion receives a vector of rewards , and the user supplies weights . A training batch contains completions, which we call rollouts. We write and for averages over the completions of one prompt, for a constant that prevents division by zero, and for normalization across the batch. Standard deviations are sample standard deviations, as in the trainer we use.
GRPO forms a scalar reward and normalizes it within the group:
GDPO normalizes each reward within the group, takes the weighted sum, and normalizes the result across the batch:
Within a group, each has zero mean and close to unit variance, or is identically zero if reward is constant on that group. The vector keeps the correlation of the rewards. If is the covariance of and has unit diagonal, with entries , the covariance of the summed advantage with the -th normalized advantage is
so positively correlated rewards raise each other’s covariance with the aggregate. Both methods update the policy with the clipped surrogate objective of PPO ((Schulman et al., 2017)), a variant of the score-function estimator ((Williams, 1992)).
What Weighted Summation Does to Reward Weights
Effective Weights
Let have zero mean and covariance with unit diagonal, and let denote the symmetric positive definite square root and its inverse. Define
The coordinates of are uncorrelated, and we call them the decorrelated components of the reward signal. Any with decorrelates . The symmetric choice is zero-phase component analysis (ZCA) whitening. Because the coordinates of are standardized, it coincides with what Kessy et al. [14] call ZCA-cor whitening of the raw rewards, which among all whitening transforms maximizes the total correlation between each component and its own reward. Principal component analysis (PCA) whitening produces components with no correspondence to individual rewards, and Cholesky whitening depends on the order in which the rewards are listed.
Proposition 1 (Effective weights). For any ,
Weighted summation applies weights to the decorrelated components, and summation after ZCA whitening applies . The two agree for all if and only if .
The proof is in Appendix § 10. We measure the discrepancy by the angle between and .
Proposition 1 does not say which weighting is appropriate, because that depends on what is meant to weight. Let . If weights the raw rewards, the target is and correlation calls for no correction; summing the rewards before normalizing, as GRPO does, follows it, while summing separately normalized advantages follows (Proposition 2). If is meant to weight the decorrelated components that track each reward, whitening is what applies those weights. We adopt the second reading, which matches the intuition that a weight such as 0.3 on length should not be inflated because length moves with accuracy, and we examine its consequences in § 5. A third reading, under which in (1), would call for the regression weights , which we do not consider further.
Corollary 1 (Variance decomposition). Let . Then
so cross-reward terms account for a fraction of the variance of the summed advantage.
We computed and from , the time average of the correlation estimate that WAS-PO maintained during training (Table 13). The angle is , and for , and is , and , so cross-reward terms carry 34%, 49% and 57% of the variance of the unwhitened sum. Because these use batch-level correlations, they describe the unwhitened sum under batch-level normalization (the WAS-PO-Diag control of § 4). For GDPO, Proposition 1 applies with replaced by the pooled covariance of its group-normalized advantages, rescaled to unit diagonal; it reflects within-group correlation only, and we did not log it. Without renormalization, would be the factor by which correlation inflates the scale of the summed advantage. GDPO and WAS-PO both renormalize across the batch, which removes this effect, so here describes only the composition of the aggregate.
| Pair | Mean | Range | |||
| 2 | acc–fmt | 0.645 | [0.530, 0.869] | 4.6 | 0.112 |
| eigenvalues | [0.355, 1.645] | ||||
| 3 | acc–fmt | 0.643 | [0.545, 0.828] | ||
| acc–len | 0.642 | [0.544, 0.827] | 12.2 | 0.251 | |
| fmt–len | 0.804 | [0.698, 1.000] | |||
| eigenvalues | [0.196, 0.408, 2.396] | ||||
| 4 | acc–fmt | 0.652 | [0.561, 0.828] | ||
| acc–len | 0.651 | [0.560, 0.827] | |||
| acc–struct | 0.643 | [0.551, 0.828] | 16.2 | 0.360 | |
| fmt–len | 0.804 | [0.700, 1.000] | |||
| fmt–struct | 0.796 | [0.693, 1.000] | |||
| len–struct | 0.796 | [0.694, 1.000] | |||
| eigenvalues | [0.196, 0.207, 0.420, 3.177] |
Table 13. The correlation estimate maintained by WAS-PO (seed 42 for ; three seeds for ). Mean and range are over the 500 training steps; is the matrix of means, and its eigenvalues, and are listed.
The Objective Whitened Aggregation Follows
Let be the vector that whitened aggregation applies to the normalized advantages, and let hold the reward standard deviations used for normalization.
Proposition 2 (Effective objectives). Consider the sequence-level score-function estimator without a KL penalty or length normalization. Treat all normalization statistics, the matrix and the final batch scale (the denominator of BN) as constants.
(a) Suppose the aggregate advantage is , with computed from batch statistics. Then the expected update equals , where
(b) With per-prompt statistics , and , as in GDPO, the same holds with .
Under these idealizations WAS-PO is a scalarized policy gradient with weights on the raw rewards. The idealizations are standard but not exact. The batch statistics depend on the sampled completions (an effect), the trainer’s token-level loss differs from the sequence-level estimator, and changes during training as the estimate of and the scales change. In our runs each batch is used for a single update, so the clipping in the surrogate is inactive. Part (b) shows that GDPO weights each reward per prompt by the inverse of its within-group standard deviation, so a reward that varies little on a prompt receives a larger weight there (at most ; a reward that is constant on the group contributes nothing). For binary rewards with , the within-group standard deviation is 0.50 or 0.58 when nonzero, so this reweighting matters mainly for continuous rewards. The batch-normalized sum without whitening follows , whose entries are all positive. For two positively correlated rewards, and at all our measured correlation matrices, has negative off-diagonal entries, so can have negative entries. A negative rewards the policy for lowering reward when the other rewards are held fixed.
When Whitening Reverses a Reward
Proposition 3 (Sign reversal with two rewards). Let and with , and let the correlation be . Then , and
If , then and as . If , then and no reversal occurs.
The threshold depends only on the ratio of the weights. For the math setting of § 5.4, gives ; for the two-reward tool-calling setting, gives . As an example, at and , : an accurate but unformatted completion gains advantage relative to an accurate and formatted one. As , the aggregate approaches a positive multiple of . Batch-level renormalization removes the growth in magnitude but not this direction. Corollary 2 extends the result to rewards when two of them become nearly collinear.
Corollary 2 (Near-duplicate rewards). Let be a sequence of correlation matrices whose smallest eigenvalue is simple and tends to zero, with unit eigenvector , while the other eigenvalues stay above some . If , then . In particular, if under the same hypotheses for one pair , then , where is the -th standard basis vector, and the whitened weights approach a multiple of whenever , so the lighter-weighted of the two rewards is reversed.
For we summarize the sign pattern of with an inversion index,
which is zero when all whitened weights are nonnegative and approaches one when two rewards become collinear (the pair case of Corollary 2). The denominator is positive because and force some to be positive. For , because ; for it can exceed one.
In practice we compute and from , which includes the eigenvalue floor of § 4.
Table 1 lists and for our settings. At , WAS-PO places negative weights on length ( at ) and on length and structure ( and at ), because both are largely predicted by accuracy and format. In all three tool-calling settings, at . A negative weight on length means that WAS-PO penalizes length beyond what accuracy and format predict; it does not by itself make the policy shorter. The estimate was sometimes much worse conditioned: its format–length entry reached 1.000 during the three- and four-reward runs (Table 13). If every pair were at its largest logged value, WAS-PO would put a large positive weight on format and a large negative weight on length at (), and on structure and length at (). None of the three- or four-reward runs collapsed, unlike the math runs; § 5.4 discusses why.
| Setting | AUC | ||||
| Tool, , time-averaged | |||||
| Tool, , largest | |||||
| Tool, , time-averaged | |||||
| Tool, , largest | |||||
| Tool, , time-averaged | |||||
| Tool, , largest | |||||
| Math, step 10 () |
Table 1. Diagnostics at the estimated correlations. is the angle between and (Proposition 1); with the eigenvalue floor of § 4; is the inversion index (5). “Time-averaged” rows use (Table 13). “Largest” rows set every pair to its largest logged value; the maxima need not occur at the same step. At these matrices are singular to numerical precision, so there the magnitudes of are set by the floor and only its signs and are meaningful. AUC is WAS-PO minus GDPO.
Estimation Error
In practice, is estimated during training from rollouts per batch.
Assumption 1. An estimate of from rollouts satisfies for a constant .
Under bivariate normality, the asymptotic variance of a sample correlation is , which gives . For skewed binary rewards this variance can be several times larger, and correlation among the completions of a prompt can reduce the effective sample size by up to a factor of . We use as a working value and have not verified it for our rewards.
Theorem 1 (Whitening estimation error). Let be symmetric, let be a random symmetric matrix satisfying Assumption 1, and suppose that almost surely all eigenvalues of and are at least . Then
The proof (Appendix § 11) bounds the Fréchet derivative of with the Daleckii–Krein formula [4 Section V.3] and integrates it along the segment from to . Every matrix on that segment has eigenvalues at least because the smallest eigenvalue is a concave function of a symmetric matrix.
We compare (6) with the size of the correction, , which is 0.112, 0.251 and 0.360 for . WAS-PO estimates from all rollouts in a batch ( in our main runs) and smooths the estimate with an exponential moving average (EMA) that gives weight to the newest batch. If successive batch estimates were independent with mean , the EMA started from the first batch would multiply the variance of each entry at step by
This factor tends to at , but equals 0.38 at , 0.14 at and 0.023 at ; its average over 500 steps is 0.105. Appendix § 12 evaluates the bound under this idealization, with set to the smallest eigenvalue of . At and , the stationary bound is 0.26, about the size of the correction, and at it is 7.3. The bound is therefore too loose to rule out estimation noise comparable to the correction for much of training, and it does not apply when the estimate approaches singularity, as it did at and . Its value is qualitative: the bound grows as , so adding near-redundant rewards (small ) makes the whitening matrix much harder to estimate, and it falls with the effective number of rollouts behind the estimate.
Method
Each training iteration of WAS-PO has three stages (Algorithm 1), plus an optional inversion check. From here on, denotes the batch-normalized advantage defined below.
Batch-level normalization. Each reward is centered and scaled by its mean and standard deviation over all rollouts in the batch. The covariance of on a batch is then the batch correlation matrix , which feeds the EMA estimate used for whitening. Batch-level normalization also changes the baseline and the scale relative to GDPO. Under GDPO a completion is compared with other completions to the same prompt, and each reward is scaled by its within-group standard deviation (Proposition 2(b)). Under batch-level normalization, completions to easy prompts tend to receive positive advantages, much as in the global advantage normalization of REINFORCE++ [10]. The WAS-PO-Diag control below separates this change from whitening.
Correlation estimate. The Pearson correlation matrix of the raw rewards is computed over the batch and replaced by if any reward has zero variance in the batch. It is smoothed as with and .
Whitening and aggregation. With the eigendecomposition , we set and compute
The eigenvalue floor bounds by 100. It has no effect when every eigenvalue of exceeds , and for two rewards it removes a reversal only when and .
Inversion check (optional). Since is available at every step, can be logged at negligible cost, and can be replaced by when needed (§ 6). The experiments reported here did not use this check.
Controls. WAS-PO-Diag zeros the off-diagonal entries of , which sets because the diagonal is one. It runs through the same code path as WAS-PO, keeps batch-level normalization and drops whitening, so it differs from GDPO only in normalization and from WAS-PO only in whitening. WAS-PO-LW applies Ledoit–Wolf shrinkage [15] to before inversion.
Algorithm 1 WAS-PO advantages for one training iteration
Require: rewards , weights , EMA state , decay
1: 2: , or if some column of is constant 3: at the first iteration, else 4: , where 5: ; 6: (optional) if the inversion check fires, recompute with 7: update with the clipped surrogate objective using
Experiments
Setup
Training. We train Qwen2.5-1.5B-Instruct (Qwen Team (2024)) for 500 steps on the single-turn subset of the Berkeley Function Calling Leaderboard v3 data (BFCL; Patil et al. (2025)), excluding the irrelevance category, using the copy distributed as llamastack/bfcl_v3. The last 20% of this subset, in dataset order, is held out from training. Each step samples 8 prompts with completions each, so , with temperature 1.0 and at most 128 new tokens (all hyperparameters are in Table 14). GRPO, GDPO and WAS-PO share one implementation, GDPO’s patched version of TRL 0.18, and differ only in how advantages are computed; rewards are gathered across the eight GPUs before advantages are formed. There is no KL penalty, and each batch is used for one update. Appendix § 14 gives the prompt template and other details, including a reward-wiring pitfall that reduces multi-reward training to a single reward.
| Tool, | Tool, | Tool, | Math | |
| Model | Qwen2.5-1.5B-Instruct | Qwen2.5-1.5B-Instruct | Qwen2.5-1.5B-Instruct | Qwen2.5-1.5B-Instruct |
| Data | BFCL v3, single-turn, irrelevance excluded; first 80% | BFCL v3, single-turn, irrelevance excluded; first 80% | BFCL v3, single-turn, irrelevance excluded; first 80% | GSM8K train |
| Group size | 4 | 4 | 4 | 4 |
| Prompts per step | 8 | 8 | 8 | 8 |
| Rollouts per step | 32 | 32 | 32 | 32 |
| Max new tokens | 128 | 128 | 128 | 384 |
| Max prompt length | 512 tokens | 512 tokens | 512 tokens | 512 tokens |
| Training steps | 500 | 500 | 500 | 500 |
| Seeds | 7 | 7 | 3 | 3 |
| Reward weights | (1.0, 0.5) | (1.0, 0.5, 0.3) | (1.0, 0.5, 0.3, 0.2) | (1.0, 0.3) |
| Conditional (gated) rewards | yes | yes | yes | yes |
| EMA decay | 0.99 | 0.99 | 0.99 | 0.99 |
| Eigenvalue floor; | ||||
| KL coefficient | 0 (no reference model) | 0 (no reference model) | 0 (no reference model) | 0 (no reference model) |
| Learning rate | ||||
| Optimizer | AdamW, linear decay, no warmup | AdamW, linear decay, no warmup | AdamW, linear decay, no warmup | AdamW, linear decay, no warmup |
| Updates per batch | 1 | 1 | 1 | 1 |
| Sampling | temperature 1.0, top- 1.0 | temperature 1.0, top- 1.0 | temperature 1.0, top- 1.0 | temperature 1.0, top- 1.0 |
| Held-out decoding | greedy | greedy | greedy | greedy |
| Precision | bfloat16 | bfloat16 | bfloat16 | bfloat16 |
| Framework | TRL 0.18 with the GDPO patch | TRL 0.18 with the GDPO patch | TRL 0.18 with the GDPO patch | TRL 0.18 with the GDPO patch |
| Hardware | 8H100 | 8H100 | 8H100 | 8H100 |
Table 14. Training hyperparameters. Settings not listed are the defaults of TRL 0.18 and Transformers.
Rewards. The accuracy reward parses the first JSON object in a completion and compares it with the BFCL ground truth. It is 0 if no object parses, 0.1 if the function name is wrong, and otherwise 0.5 times the fraction of ground-truth argument names present plus 0.5 times the fraction of argument values that match. The format reward is 1 if a JSON object parses and 0 otherwise. The length reward is for a completion of whitespace-separated words; with completions capped at 128 tokens, it lies between 0.61 and about 0.76 and increases with length. The structure reward () is 1 for an object with a function name and nonempty arguments, 0.5 for a name alone, and 0 otherwise. We use GDPO’s conditional rewards: every secondary reward is set to zero when the accuracy reward is not positive, and we call such rewards gated. The gated format reward is therefore exactly the indicator that accuracy is positive, and the length and structure rewards are zero whenever it is zero. This shared gate is the main source of the correlations in Table 13.
Metrics. Our main metric is training AUC, the mean over all steps of the weighted batch-mean reward,
where is the batch-mean reward vector at step . Its maximum is (1.5, 1.8 and 2.0 for ), so AUC values are not comparable across . We also report held-out accuracy: the accuracy reward of the final checkpoint under greedy decoding on the first 500 prompts of the held-out 20%, the same prompts for every method, and seed. Seeds are paired across methods. Training comparisons use exact two-sided Wilcoxon signed-rank tests on the paired differences; with seven seeds the smallest attainable -value is , and with three seeds it is 0.25. We report as the mean paired difference and as that mean divided by the standard deviation of the paired differences. For held-out accuracy we kept per-method summaries rather than per-seed pairs, so we use unpaired two-sided Welch -tests.
Three and Four Rewards
Three rewards. Figure 1 shows training curves for two and three rewards. The rewards are accuracy (), format () and length (). Over seven seeds, training AUC was 1.013 0.011 for GRPO, 1.089 0.006 for GDPO and 1.130 0.007 for WAS-PO (mean SD; Table 5, Figure 1b). WAS-PO was ahead of GDPO in every seed, by (, ).
Figure 1. (a) Two rewards. (b) Three rewards. Training reward of GRPO, GDPO and WAS-PO. Curves and SD bands are computed over a subset of the seeds (three at ). The per-seed AUCs in Table 7 and Table 5 cover all seven seeds.
| Seed | GRPO | GDPO | WAS-PO | |
| 42 | 1.0303 | 1.0958 | 1.1252 | +0.0294 |
| 123 | 0.9995 | 1.0862 | 1.1386 | +0.0524 |
| 456 | 1.0028 | 1.0903 | 1.1213 | +0.0310 |
| 789 | 1.0188 | 1.0947 | 1.1391 | +0.0444 |
| 1011 | 1.0059 | 1.0798 | 1.1303 | +0.0505 |
| 2024 | 1.0110 | 1.0863 | 1.1246 | +0.0383 |
| 9999 | 1.0225 | 1.0879 | 1.1315 | +0.0436 |
| Mean ± SD | 1.013 ± 0.011 | 1.089 ± 0.006 | 1.130 ± 0.007 | +0.041 |
Table 5. Training AUC with three rewards, seven seeds. Exact two-sided Wilcoxon ; .
Held-out accuracy was 0.397 0.076 for GRPO, 0.446 0.012 for GDPO and 0.438 0.011 for WAS-PO (Table 9). WAS-PO was 0.008 below GDPO, which is not significant (Welch ) and within the standard error of a single run’s mean accuracy on 500 prompts (at most about 0.022). The exact-match rate, the fraction of completions with accuracy reward 1, was 0.133 for WAS-PO and 0.131 for GDPO, and the format rate was 0.99 for both.
Table 9: Held-out accuracy (accuracy reward of the final checkpoint under greedy decoding on 500 held-out prompts; mean SD over seeds). is from an unpaired two-sided Welch -test computed from these summaries. At , scores were recorded for two of three seeds.
| Setting | GRPO | GDPO | WAS-PO | vs. GDPO | |
| (seven seeds) | 0.479 0.006 | 0.474 0.008 | 0.457 0.022 | 0.09 | |
| (seven seeds) | 0.397 0.076 | 0.446 0.012 | 0.438 0.011 | 0.22 | |
| (two seeds) | 0.521, 0.517 | 0.517, 0.515 | 0.485, 0.487 | — |
Table 9.
Four rewards. With the structure reward added (), training AUC over three seeds was 1.116 0.024 for GRPO, 1.257 0.007 for GDPO and 1.306 0.009 for WAS-PO (Table 6, Figure 4). WAS-PO was ahead in all three seeds, by +0.040, +0.050 and +0.058. Three seeds cannot reach significance in a Wilcoxon test (the smallest two-sided is 0.25), so we read this as consistent with the three-reward result rather than as independent evidence. Averaged over training, WAS-PO was ahead on every reward: accuracy +0.014, format +0.038, length +0.011 and structure +0.032 (Table 12). Weighted by , about two-thirds of this gain comes from the three gated rewards; the format mean, which equals the rate of completions with positive accuracy, was 0.871 under WAS-PO against 0.833 under GDPO. On the held-out set we recorded scores for two of the three seeds: WAS-PO scored 0.485 and 0.487, GDPO 0.517 and 0.515, and GRPO 0.521 and 0.517. With four rewards, WAS-PO was therefore about 0.03 below GDPO on held-out accuracy, and GRPO matched or exceeded GDPO. Relative to GDPO, the gain in training AUC was 3.8% with three rewards and 3.9% with four (Table 2; Figure 2 in Appendix § 13).
| Seeds | Rewards | GDPO | WAS-PO | Relative | |||
| 2 | 7 | acc, fmt | 0.985 | 0.970 | 4.6 | ||
| 3 | 7 | acc, fmt, len | 1.089 | 1.130 | 12.2 | ||
| 4 | 3 | acc, fmt, len, struct | 1.257 | 1.306 | 16.2 |
Table 2. Training AUC by number of rewards. is the mean paired difference, WAS-PO minus GDPO, and Relative is divided by GDPO’s mean, both computed from unrounded per-seed values. is the condition number of the time-averaged correlation estimate. The settings differ in which rewards they contain as well as in how many, and AUC is not comparable across rows. acc, fmt, len and struct are the accuracy, format, length and structure rewards.
Figure 2. Training AUC for two, three and four rewards (mean SD over seeds). Each panel has its own axis because AUC is not comparable across .
| Seed | GRPO | GDPO | WAS-PO | |
| 42 | 1.1049 | 1.2633 | 1.3036 | +0.0403 |
| 123 | 1.0995 | 1.2488 | 1.2987 | +0.0499 |
| 456 | 1.1439 | 1.2579 | 1.3154 | +0.0575 |
| Mean ± SD | 1.116 ± 0.024 | 1.257 ± 0.007 | 1.306 ± 0.009 | +0.049 |
Table 6. Training AUC with four rewards, three seeds. ; the smallest attainable two-sided Wilcoxon with three seeds is 0.25.
Table 12: Per-reward training reward with four rewards, averaged over 500 steps (three seeds). Per-reward values were not logged for GRPO. The weighted sums of these means (1.262 and 1.305) agree with the AUCs in Table 6 to within 0.006.
| Method | Accuracy | Format | Length | Structure |
| GDPO | 0.521 | 0.833 | 0.528 | 0.831 |
| WAS-PO | 0.535 | 0.871 | 0.539 | 0.863 |
| +0.014 | +0.038 | +0.011 | +0.032 |
Table 12.

Figure 4. Training reward with four rewards (three seeds, mean SD).
Two Rewards and Component Ablations
With accuracy and format only, training AUC over seven seeds was 0.986 for GRPO, 0.985 for GDPO and 0.970 for WAS-PO, which trailed GDPO in every seed (, , ; Table 7, Figure 1a). Held-out accuracy was 0.479 0.006 for GRPO, 0.474 0.008 for GDPO and 0.457 0.022 for WAS-PO (Welch against GDPO).
| Seed | GRPO | GDPO | WAS-PO | |
| 42 | 0.9903 | 0.9870 | 0.9721 | −0.0149 |
| 123 | 0.9759 | 0.9826 | 0.9722 | −0.0104 |
| 456 | 0.9918 | 0.9873 | 0.9713 | −0.0160 |
| 789 | 0.9898 | 0.9833 | 0.9727 | −0.0106 |
| 1011 | 0.9825 | 0.9870 | 0.9691 | −0.0179 |
| 2024 | 0.9825 | 0.9814 | 0.9683 | −0.0131 |
| 9999 | 0.9908 | 0.9849 | 0.9668 | −0.0181 |
| Mean ± SD | 0.986 ± 0.006 | 0.985 ± 0.002 | 0.970 ± 0.002 | −0.014 |
Table 7. Training AUC with two rewards, seven seeds. Exact two-sided Wilcoxon ; .
Changing how is estimated did not close the gap (Table 3). EMA decays from to 0.99, Ledoit–Wolf shrinkage (which lowered the largest estimated correlation from 0.87 to 0.37) and larger groups all left WAS-PO below GDPO. At and at , WAS-PO was below GDPO in each of the three seeds (Table 8).
| Variant ( unless stated) | Seeds | GDPO, same | AUC | |
| WAS-PO | 7 | 0.985 | 0.970 | |
| WAS-PO-LW | 7 | 0.985 | 0.968 | |
| WAS-PO-Diag | 3 | 0.986 | 0.965 | |
| WAS-PO, () | 3 | 1.003 | 0.984 | |
| WAS-PO, () | 3 | 1.011 | 0.993 | |
| WAS-PO, | 1 | 0.990 | 0.967 | |
| WAS-PO, | 1 | 0.990 | 0.970 | |
| WAS-PO, | 1 | 0.990 | 0.955 | |
| WAS-PO, | 1 | 0.990 | 0.955 | |
| WAS-PO, | 1 | 0.990 | 0.966 |
Table 3. Ablations with two rewards. Training AUC. is relative to GDPO trained with the same group size on the same seeds (seeds 42, 123 and 456 for the three-seed rows; seed 42 for the decay sweep) and is computed from unrounded values. WAS-PO-Diag keeps batch-level normalization and removes whitening. The decay sweep is a separate set of single-seed runs with its own GDPO baseline (0.990). Its run repeats the default configuration and scored 0.966, against 0.972 for seed 42 in Table 7, so reruns at a fixed seed vary by about 0.006.
| Seed | GRPO | GDPO | WAS-PO | ||
| 8 | 42 | 1.0021 | 1.0061 | 0.9849 | −0.0212 |
| 8 | 123 | 0.9964 | 0.9976 | 0.9837 | −0.0139 |
| 8 | 456 | 1.0020 | 1.0050 | 0.9830 | −0.0220 |
| 16 | 42 | 1.0125 | 1.0111 | 0.9938 | −0.0173 |
| 16 | 123 | 1.0084 | 1.0064 | 0.9905 | −0.0159 |
| 16 | 456 | 1.0169 | 1.0153 | 0.9960 | −0.0193 |
Table 8. Training AUC at larger group sizes, two rewards, three seeds. Eight prompts per step, so at and at .
WAS-PO-Diag scored 0.965 on seeds 42, 123 and 456, while WAS-PO scored 0.972 on the same seeds. The gap (0.007, three seeds) is about the size of the rerun variation in Table 3. It weakly suggests that, with two rewards, the deficit relative to GDPO comes from batch-level normalization rather than from whitening. The theory does not point to whitening as the cause either. At , ; the estimate started at 0.87, above , and stayed above it only for the first few dozen steps (Figure 3, top).
Figure 3. Correlation estimate and reward difference (seed 42, logged every 10 steps). Blue: the EMA estimate of . Red: training reward of WAS-PO minus GDPO, labeled “advantage” in the legend but meaning a difference in reward. Top: tool calling with two rewards; the two series have correlation 0.44, but both trend over training, so we do not interpret this. Bottom: the earlier math run with a constant format reward, where the estimate is the identity and the difference fluctuates around zero.
In an earlier math run, the format reward was a JSON-format check applied to math answers and was therefore always zero. With a constant reward, at every step and WAS-PO reduced to WAS-PO-Diag. Its gap to GDPO was over three seeds (Figure 3, bottom; Table 10), so batch-level normalization does not always cost much training reward.
Table 10: Training AUC in the earlier math run with a constant format reward, where WAS-PO reduces to WAS-PO-Diag (Section 5.3). This is not the run of § 5.4.
| Seed | GRPO | GDPO | WAS-PO | |
| 42 | 0.8077 | 0.8041 | 0.8085 | +0.0044 |
| 123 | 0.7988 | 0.8120 | 0.7943 | |
| 456 | 0.8075 | 0.7994 | 0.7995 | +0.0001 |
| Mean | 0.805 | 0.805 | 0.801 |
Table 10.
ZCA-Inversion
We train the same model on the GSM8K training set (Cobbe et al., [5]), with a system prompt asking it to end with the answer in the form The answer is: X. The accuracy reward is 1 if the extracted answer matches the reference and 0 otherwise; the extractor takes a boxed answer if there is one and otherwise the last number in the completion. The format reward is 1 if the completion contains The answer is: or a boxed answer, and 0 otherwise. Both rewards are gated as in tool calling, with (); , with at most 384 new tokens. Since the gated format reward is 1 exactly when the answer is correct and formatted, the batch correlation of the two binary rewards follows from their means and : (Appendix § 10). From the means in Table 11, under GDPO and 0.98 under WAS-PO at step 10, well above . Under GDPO the means are equal at steps 50 and 500, so . These values are approximate because the means are averaged over batches and seeds. Over three seeds, training AUC was 1.050 for GRPO, 1.049 for GDPO and 0.821 for WAS-PO. Under WAS-PO the format reward fell from 0.666 at step 10 to 0.132 at step 500 while accuracy kept rising; under GDPO both rose. This matches the sign that Proposition 3 assigns to the format reward: whenever , the completions that are correct and unformatted receive the largest advantages. Because the extractor falls back to the last number, such completions are easy to produce, and the policy learned to produce them. As the format reward drops, the two rewards decouple: by the same formula, the batch correlation under WAS-PO was about 0.41 at step 50 and 0.19 at step 500. The EMA forgets slowly (half-life 69 steps at ): starting from 0.98, the estimate cannot fall below before step 58 unless new batches are negatively correlated, so the reversal stayed in force while the format reward fell to 0.277 by step 50.
Table 11: Per-reward batch-mean training reward in the math setting (three seeds).
| Step | WAS-PO acc | WAS-PO fmt | GDPO acc | GDPO fmt |
| 10 | 0.675 | 0.666 | 0.691 | 0.675 |
| 50 | 0.697 | 0.277 | 0.775 | 0.775 |
| 500 | 0.806 | 0.132 | 0.863 | 0.863 |
Table 11.
The three- and four-reward runs show that a nearly singular estimate does not by itself cause collapse. In those runs the format–length entry of the estimate reached 1.000; this probably happened early in training, since the estimate is initialized from the first batch and the two-reward trace (Figure 3) is highest at the start. Had every pair been at its maximum, would have been 0.82–1.01. Among completions that pass the gate, however, the length reward varies only between 0.61 and about 0.76, so a reversal along that direction changes the reward means little. In the math run, by contrast, the reversed reward could be switched off entirely at no cost to accuracy. Whether reward models trained on overlapping data reach correlations like those of the math rewards is untested here; ensembles of such models are known to share errors [8].
Why Does WAS-PO Raise the Weighted Training Reward?
Training AUC weights the raw rewards by . By Proposition 2, neither method follows exactly that objective: GDPO weights reward by on each prompt, and WAS-PO follows , which puts negative weight on length or structure at . That WAS-PO nonetheless scores higher on is therefore not explained by its applying the intended weights. At , the per-reward means show that most of the gain is in the gated rewards, through a higher rate of completions that parse and score above zero (0.871 against 0.833), which also explains why every reward moved up together. Three explanations fit. First, whitening shifts weight toward accuracy relative to ( is 3.6 at and 2.8 at , against ), and every secondary reward is gated on accuracy being positive. Second, GDPO may spend gradient on noise. In a group where all four completions pass the gate, the format reward is constant and drops out, while the small length differences among them (all within 0.61–0.76) are standardized to unit variance, so by Proposition 2(b) GDPO gives them a large weight on that prompt; batch-level normalization scales them by the much larger batch standard deviation. Third, batch-level normalization gives nonzero advantages to groups in which every completion scored the same, which GDPO does not. Comparing against WAS-PO-Diag and against batch-normalized summation with fixed weights , both at , would separate these explanations.
Practical Guidance
Items marked observed are supported by our experiments on one task; items marked proposed have not been tested.
Training reward is not held-out accuracy (observed). With three or four rewards, WAS-PO reached higher training reward than GDPO in every seed but equal or lower held-out accuracy, and GRPO matched or exceeded GDPO on held-out accuracy at and . In these settings the secondary rewards were close to copies of the accuracy gate. On this evidence we do not recommend WAS-PO as a way to improve held-out accuracy; what we have shown is that it changes which weighted training objective is optimized.
Inspect the whitened weights (proposed). Log and from the matrix that is actually inverted. Any positive means some reward is being reversed, but in our runs did not separate the collapsed run (0.69) from the worst cases assembled from per-pair maxima in runs that did not collapse (0.82–1.01), so we give no threshold. What mattered was whether the policy could move along the reversed direction cheaply. Two simple guards are to clip negative entries of to zero, or to fall back to when the batch mean of a reward with falls over a sustained window; neither has been tested. For two rewards, Proposition 3 gives the exact boundary for the correlation being whitened.
Screen before training (proposed). Log rewards for a few dozen steps, estimate from the batches, and compute , and . We give no threshold for the angle; at WAS-PO still differed from GDPO, mainly through normalization (Table 3).
Merge near-duplicate rewards (follows from Corollary 2). Two rewards that measure nearly the same thing, such as a gated format reward that is almost always earned, or two reward models trained on similar data, should be merged or left unwhitened. With unequal weights, whitening reverses the lighter-weighted one; with two equally weighted rewards it reverses neither (Proposition 3). Our own three- and four-reward settings contain such near-duplicates (Table 13), so the gains reported there come from a configuration this item advises against.
Limitations
WAS-PO changes both normalization and aggregation relative to GDPO, and the control that separates them was run only with two rewards and three seeds. The gains at are gains of WAS-PO as a whole, and Section 5.5 lists competing explanations that we have not tested.
The gains are in training reward only. Held-out accuracy did not improve and was lower than GDPO’s with four rewards. Training AUC is measured on temperature-1 samples from the training prompts, while held-out accuracy uses greedy decoding on new prompts, so a method that sharpens the policy faster can raise the first without the second. The held-out prompts come from the same BFCL subset as the training prompts, taken in dataset order, so their category mix may differ from the training portion, and we have no evidence about transfer to other tool-calling distributions or to the official BFCL evaluation.
The tool-calling results come from one task, one 1.5B model and a small batch (); the math runs have three seeds each. Each uses one reward set, and moving from two to three rewards adds the length reward, so the number of rewards is confounded with which rewards are used. The four-reward setting has three seeds and held-out scores for two of them. Conditional reward gating produces much of the correlation we study, and our diagnostics use batch-level correlations, not the within-group correlations that govern GDPO.
The analysis relies on a stated reading of (Proposition 1), on treating normalization statistics as constants (Proposition 2), and on Assumption 1 with an idealized EMA (Theorem 1). The inversion diagnostics rest on one collapsed run and two runs that did not collapse despite a nearly singular estimate.
Related Work
Group-based policy optimization. Shao et al. (2024) introduced GRPO, which normalizes a scalar reward within a group of completions and needs no learned value function. Liu et al. [16] showed that GRPO’s summed reward maps distinct reward combinations to the same advantage, and proposed GDPO. On BFCL with Qwen2.5-1.5B they report 32.81% average accuracy for GDPO against 30.18% for GRPO over five runs, with an evaluation protocol that differs from ours. Liu et al. [17] analyze biases introduced by the length and standard-deviation normalization terms in GRPO, and Hu et al. [10] argue for normalizing advantages over the whole batch. WAS-PO uses batch-level normalization in the spirit of the latter and adds a correction for correlation between rewards, which these methods do not consider.
Multiple rewards in language model alignment. Bai et al. [2] train with a preference model for helpfulness and harmlessness, and Touvron et al. (2023) combine separate helpfulness and safety reward models with a threshold rule rather than a weighted sum. Wu et al. (2023) combine several fine-grained reward models with fixed weights, and Dai et al. [7] treat helpfulness and harmlessness as a constrained problem. Ramé et al. (2023) interpolate the weights of policies fine-tuned on different rewards to trace a Pareto front. Coste et al. [6] and Eisenstein et al. [8] combine reward models in ensembles and find that models trained on similar data share errors, and Moskovitz et al. [18] find that the correlation between component reward models affects where overoptimization sets in. We instead ask how correlation changes what a fixed weight vector does.
Multi-task loss weighting. Multi-task learning faces a similar problem with conflicting or redundant losses. Kendall et al. [13] learn task weights from homoscedastic uncertainty, Sener & Koltun (2018) cast multi-task learning as multi-objective optimization, and Yu et al. (2020) project away conflicting gradient components. These methods operate on losses or parameter gradients, whereas WAS-PO changes per-sample advantages before any gradient is formed.
Whitening and covariance estimation. ZCA whitening was used in early work on the independent components of natural images [3]. Kessy et al. [14] characterize the family of whitening transforms and the optimality of ZCA, and Huang et al. [11] use ZCA whitening inside network layers and find that PCA whitening performs worse there. Estimating covariance from few samples is a classical difficulty [15]; in our two-reward runs, Ledoit–Wolf shrinkage did not close the gap to GDPO. Amari [1] and Kakade [12] precondition parameter updates with the Fisher information; WAS-PO can loosely be viewed as preconditioning in reward space instead.
Reward hacking. Skalse et al. (2022) define reward hacking formally, Gao et al. [9] measure how optimizing a learned reward model overshoots the reward it approximates, and Pan et al. (2022) show that misspecified rewards are exploited more as agents become more capable. ZCA-Inversion differs from reward hacking in that it occurs even when every reward is correctly specified, because it arises from how the rewards are combined.
Conclusion
When normalized rewards are correlated, adding them with fixed weights does not apply those weights to the decorrelated components of the reward signal. We quantified the discrepancy, derived the objectives that GDPO and whitened aggregation follow under fixed normalization statistics, and gave an exact two-reward condition, with an extension to near-duplicate rewards, under which whitening reverses a reward. In tool calling with three or four correlated rewards, WAS-PO reached higher training reward than GDPO in every seed, but its held-out accuracy was not higher, and with four rewards it was about 0.03 lower (two seeds). In our two-reward setting, GDPO was better. With two nearly identical rewards, WAS-PO lowered the lighter-weighted reward, as the sign condition predicts. The open questions are how much of the training-reward gain comes from whitening rather than batch-level normalization, why it does not carry over to held-out accuracy, and whether either finding holds on other tasks and larger models.
Broader Impact Statement
Multi-reward training is used to combine helpfulness, format and safety objectives. ZCA-Inversion is relevant to safety because whitening two highly correlated rewards with unequal weights can train the policy to lower the lighter-weighted one, which could be a safety reward. Proposition 3 gives the exact condition for two rewards in terms of the correlation being whitened, but the estimate actually inverted can differ from the population value (Table 13), and negatively correlated pairs are never reversed. We ran no experiments with reward models or safety rewards. We advise against whitening a safety reward unless its weight in is constrained to be nonnegative. We release no new datasets or models; all resources used are publicly available.
Code Availability
Code is available at https://github.com/lleye/waspo.
Proofs for Section 3
Proposition 1. Since , its symmetric positive definite square root and the inverse of that root exist and are unique. With we have and . Then by symmetry, and . The two agree for every if and only if for every , that is, , which is equivalent to .
Corollary 1. For a fixed vector , . With and unit diagonal this is . With it is .
Proposition 2. (a) With the stated constants, , where and does not depend on . For any that does not depend on , the score-function identity gives , and . Applying both to and to the constant , and averaging over prompts, gives . Setting gives for the batch-normalized sum without whitening. (b) For a fixed prompt , with independent of , and the same argument applied conditionally on , followed by averaging over prompts, gives the stated objective. Using the group mean that includes itself, with still held fixed, adds a factor common to all rewards, because the other completions are independent of ; it does not change the direction of the gradient. □
Proposition 3. For the eigenvectors are with eigenvalues , so and
Both terms of are nonnegative and the first is positive, so . Both terms in the expression for are nonnegative, so if and only if , that is, , which simplifies to , or . If , the second terms vanish and . If , as the first terms stay bounded while the second terms grow as , so and . □
Corollary 2. Write in an orthonormal eigenbasis of , with . The terms with have norm at most , while the first has norm because . Dividing by gives the limit. For the pair case let . Then , while . Hence , so and , which is nonzero when . □
Correlation of gated binary rewards. Let and be binary with , as for an accuracy reward and a format reward gated on it, with means and . Then , so , and dividing by gives . □
Proof of Theorem 1
Step 1. Let on symmetric matrices with all eigenvalues at least , and write with eigenvalues . By the Daleckii–Krein formula [4 Section V.3], the Fréchet derivative is , where is the entrywise product and
with when . Since and , every . The Frobenius norm is invariant under orthogonal conjugation and , so .
Step 2. Let and for . The smallest eigenvalue is concave on symmetric matrices, so . Hence is continuously differentiable along the segment and , which by Step 1 gives .
Step 3. almost surely. Taking expectations and applying Assumption 1 gives (6). □
EMA variance factor. Unrolling the recursion from gives
If the are independent with mean and entrywise variance , each entry of has variance , with as in § 3.4. Bias in the batch estimates, which is for sample correlations, is not reduced by the EMA.
Evaluating the Bound
Table 4 evaluates (6) with and set to the smallest eigenvalue of , for one batch of rollouts and for the EMA under the idealization of § 3.4, both at its stationary factor and at step (). We use in the main runs and in the ablation. Using the time-averaged is itself an idealization: the estimate became nearly singular at times in the three- and four-reward runs, and single-batch estimates can be worse, so for them Theorem 1 holds only with a much smaller , or not usefully at all. At the largest two-reward estimate of , , and the constant rises from to about . Step 3 of the proof is tight only when is close to rank one with in its row space.
| One batch, | One batch, | EMA, stationary, | EMA, stationary, | EMA, , | EMA, , | ||||
| 2 | 0.355 | 0.112 | 111.8 | 3.61 | 0.880 | 0.018 | 0.004 | 0.508 | 0.124 |
| 3 | 0.196 | 0.251 | 1601.7 | 51.7 | 12.6 | 0.260 | 0.063 | 7.29 | 1.78 |
| 4 | 0.196 | 0.360 | 2932.5 | 94.6 | 23.1 | 0.475 | 0.116 | 13.3 | 3.26 |
Table 4. Bound (6) against the size of the correction .
Full Results
Hyperparameters and Implementation Details
Table 14: Training hyperparameters. Settings not listed are the defaults of TRL 0.18 and Transformers.
Prompts. Tool-calling prompts list the names and descriptions (truncated to 80 characters) of the first three available functions, then the user request (truncated to 300 characters), then the instruction to respond only with a JSON function call, followed by the example {"name": "function_name", "arguments": {"param": "value"}}. The held-out evaluation uses the same content with the system message “You are a helpful assistant.” Ground-truth argument values in BFCL can list several acceptable values; the accuracy reward compares against the first, as a case-insensitive string match. The math system prompt asks the model to end its response with The answer is: X, where is the numeric answer only.
Compute. The reported runs used about 120 GPU-hours, and preliminary and failed runs about 40 more. The and ablations also use eight prompts per step.
Reward wiring. A natural way to add several rewards to a GRPO trainer is to wrap all reward functions in one function that returns their weighted sum. The trainer then sees a single reward, and GRPO, GDPO and WAS-PO all run the same single-reward path, so no comparison between them is meaningful. All results here use an implementation that passes each reward function to the trainer separately, so that the reward tensor has one column per reward. We recommend checking that the reward tensor has columns before comparing multi-reward methods.
References
- [1]Shun-ichi Amari. Natural gradient works efficiently in learning. Neural Computation, 10(2):251–276, 1998. doi: 10.1162/089976698300017746.DOI
- [2]Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Noya DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.arxiv.org/abs/2204.05862
- [3]Anthony J. Bell and Terrence J. Sejnowski. The “independent components” of natural scenes are edge filters. Vision Research, 37(23):3327–3338, 1997. doi: 10.1016/S0042-6989(97)00121-1.DOI
- [4]Rajendra Bhatia. Matrix Analysis, volume 169 of Graduate Texts in Mathematics. Springer, New York, 1997.
- [5]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
- [6]Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.02743
- [7]Juntao Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.12773
- [8]Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, DJ Divjotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. Helping or herding? Reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09244, 2023.arxiv.org/abs/2312.09244
- [9]Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 10835–10866, 2023.arxiv.org/abs/2210.10760
- [10]Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. REINFORCE++: Stabilizing critic-free policy optimization with global advantage normalization. arXiv preprint arXiv:2501.03262, 2025.arxiv.org/abs/2501.03262
- [12]Sham M. Kakade. A natural policy gradient. In Advances in Neural Information Processing Systems, volume 14, pp. 1531–1538, 2001.
- [13]Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7482–7491, 2018.arxiv.org/abs/1705.07115
- [14]Agnan Kessy, Alex Lewin, and Korbinian Strimmer. Optimal whitening and decorrelation. The American Statistician, 72(4):309–314, 2018. doi: 10.1080/00031305.2016.1277159.DOI
- [15]Olivier Ledoit and Michael Wolf. A well-conditioned estimator for large-dimensional covariance matrices. Journal of Multivariate Analysis, 88(2):365–411, 2004. doi: 10.1016/S0047-259X(03)00096-4.DOI
- [16]Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Peter Belcak, Mingjie Liu, Min-Hung Chen, Hongxu Yin, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Yejin Choi, Jan Kautz, and Pavlo Molchanov. GDPO: Group reward-decoupled normalization policy optimization for multi-reward RL optimization. arXiv preprint arXiv:2601.05242, 2026.arxiv.org/abs/2601.05242
- [17]Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025.arxiv.org/abs/2503.20783
- [18]Ted Moskovitz, Aaditya K. Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca Dragan, and Stephen Marcus McAleer. Confronting reward model overoptimization with constrained RLHF. In International Conference on Learning Representations, 2024.


