betaFeedback

WorksComputer Science

Whitened Advantage Summation Policy Optimization: Correlation-Aware Aggregation for Multi-Reward Reinforcement Learning

Abstract

Reinforcement learning for language models often combines several rewards by normalizing each one and adding them with fixed weights. When the rewards are correlated, the variation they share is counted once for every reward that carries it. In terms of the ZCA-decorrelated components of the reward signal, the weighted sum applies weights Σ1/2wΣ^{1/2} w rather than the chosen weights ww, where ΣΣ is the reward correlation matrix. We study WAS-PO, which whitens the normalized advantages with Σ−1/2Σ^{-1/2} before adding them, so that the chosen weights apply to those components. We derive the scalarized objective whose gradient it follows when normalization statistics are held fixed, show that its weights on the raw rewards can be negative, give an exact two-reward condition under which the lighter-weighted reward is reversed, and give a loose bound on the error from estimating ΣΣ during training. On tool calling with Qwen2.5-1.5B-Instruct, WAS-PO reached a higher mean training reward than GDPO in all seven seeds with three rewards (3.8% higher) and all three seeds with four (3.9% higher). WAS-PO also replaces GDPO's per-prompt normalization with batch-level normalization, and we did not separate the two changes with three or four rewards. The training-reward gains did not carry over to held-out accuracy, which was slightly lower than GDPO's with three rewards (not significant) and 0.03 lower in absolute terms with four (two seeds evaluated). With two rewards, WAS-PO trailed GDPO in training reward, and a three-seed control weakly suggests that the gap comes from batch-level normalization rather than whitening. In a math setting where the two rewards were nearly identical by construction, WAS-PO drove down the lighter-weighted reward, as the sign condition predicts.

Introduction

Reinforcement learning for language models now routinely optimizes several rewards at once, such as correctness, output format and length. Group Relative Policy Optimization (GRPO; Shao et al., 2024) handles this by summing the rewards and normalizing the sum within each group of completions sampled for a prompt. Liu et al. [16] showed that summing first maps distinct combinations of rewards to the same advantage, and proposed Group reward-Decoupled Normalization Policy Optimization (GDPO), which normalizes each reward within the group before taking a weighted sum.

The weights in that sum are meant to say how much each objective matters, but correlated rewards change what they do. When normalized advantages of correlated rewards are added, the variation they share enters the sum once per reward. In terms of the decorrelated components of the reward signal (§ 3.1), a weighted sum with weights w\mathbf{w} applies weights Σ1/2w\boldsymbol{\Sigma}^{1/2}\mathbf{w}, where Σ\boldsymbol{\Sigma} is the correlation matrix of the rewards. Correlation of this kind is common in multi-reward training. In our tool-calling setup, following GDPO’s conditional rewards, the format, length and structure rewards are set to zero whenever accuracy is not positive, so they rise and fall together largely by construction. At the batch-level correlations that WAS-PO estimated in our runs, the effective weight vector of the batch-normalized sum points away from w\mathbf{w} by 9.6∘9.6^\circ, 16.0∘16.0^\circ and 20.7∘20.7^\circ with two, three and four rewards. Whether this is a distortion depends on how the weights are read, which we discuss in § 3.1.

Whitening the normalized advantages with Σ−1/2\boldsymbol{\Sigma}^{-1/2} (ZCA whitening) before summing applies w\mathbf{w} to the decorrelated components that are closest to the individual rewards. We call the result Whitened Advantage Summation Policy Optimization (WAS-PO). Whitening is not free. WAS-PO is a scalarized policy gradient whose weights on the raw rewards, D−1Σ−1/2w\mathbf{D}^{-1}\boldsymbol{\Sigma}^{-1/2}\mathbf{w} with D\mathbf{D} the diagonal matrix of reward standard deviations, can be negative (Proposition 2). With two rewards weighted a>b>0a>b>0, the lighter-weighted reward receives a negative weight exactly when the correlation exceeds 2ab/(a2+b2)2ab/(a^2+b^2), so the policy is trained to lower that reward, and the negative weight grows without bound as the correlation approaches one (Proposition 3). We call this ZCA-Inversion and observe it in a math setting, where the format reward fell from 0.67 to 0.13 during training.

In our three- and four-reward tool-calling settings, WAS-PO reached higher training reward than GDPO in all ten paired runs (seven at K=3K=3, three at K=4K=4). The gain did not carry over to held-out accuracy, and with two rewards WAS-PO trailed GDPO even in training reward. WAS-PO also replaces GDPO’s per-prompt normalization with batch-level normalization, so the comparison with GDPO mixes two design changes. With two rewards, a control that keeps batch-level normalization but drops whitening (WAS-PO-Diag) scores slightly below WAS-PO, which weakly suggests that normalization is the source of the two-reward gap. We have not run this control with three or four rewards, so we report the gains there as gains of WAS-PO as a whole.

Our contributions are:

  1. an exact statement of how correlation changes the effective weights of a weighted sum of normalized advantages, with a variance decomposition (§ 3.1);

  2. the objective that whitened aggregation follows, and the corresponding per-prompt objective of GDPO (§ 3.2);

  3. an exact two-reward condition for sign reversal, its extension to near-duplicate rewards among KK, and an inversion index that can be logged during training (§ 3.3);

  4. an error bound for the estimated whitening matrix (§ 3.4);

  5. experiments on tool calling with two, three and four rewards, component ablations, an observed instance of ZCA-Inversion, and runs in which a nearly singular estimate did not cause collapse (§ 5).

Background

For a prompt xx, a policy πθ\pi_\theta samples GG completions y1,…,yGy_1,\ldots,y_G. Each completion receives a vector of KK rewards ri∈RK\mathbf{r}_i \in\mathbb{R}^K, and the user supplies weights w∈R>0K\mathbf{w} \in\mathbb{R}^K_{>0}. A training batch contains NN completions, which we call rollouts. We write mean⁡g\operatorname{mean}_g and std⁡g\operatorname{std}_g for averages over the GG completions of one prompt, ϵ=10−4\epsilon=10^{-4} for a constant that prevents division by zero, and BN⁡(u)i=(ui−mean⁡(u))/(std⁡(u)+ϵ)\operatorname{BN}(u)_i=(u_i-\operatorname{mean}(u))/(\operatorname{std}(u)+\epsilon) for normalization across the batch. Standard deviations are sample standard deviations, as in the trainer we use.

GRPO forms a scalar reward and normalizes it within the group:

A^iGRPO=wTri−mean⁡g(wTr)std⁡g(wTr)+ϵ.\widehat{A}^{\mathrm{GRPO}}_i = \frac{\mathbf{w}^{\mathsf{T}}\mathbf{r}_i-\operatorname{mean}_g(\mathbf{w}^{\mathsf{T}}\mathbf{r})}{\operatorname{std}_g(\mathbf{w}^{\mathsf{T}}\mathbf{r})+\epsilon}.

GDPO normalizes each reward within the group, takes the weighted sum, and normalizes the result across the batch:

A^i(k)=ri(k)−mean⁡g(r(k))std⁡g(r(k))+ϵ,A^iGDPO=BN⁡(∑k=1KwkA^(k))i\widehat{A}^{(k)}_i = \frac{r^{(k)}_i-\operatorname{mean}_g(r^{(k)})}{\operatorname{std}_g(r^{(k)})+\epsilon}, \qquad\widehat{A}^{\mathrm{GDPO}}_i = \operatorname{BN}\left(\sum_{k=1}^{K} w_k\widehat{A}^{(k)}\right)_i

Within a group, each A^(k)\widehat{A}^{(k)} has zero mean and close to unit variance, or is identically zero if reward kk is constant on that group. The vector A^i=(A^i(1),…,A^i(K))\widehat{\mathbf{A}}_i = (\widehat{A}^{(1)}_i,\ldots,\widehat{A}^{(K)}_i) keeps the correlation of the rewards. If Σ\boldsymbol{\Sigma} is the covariance of A^\widehat{\mathbf{A}} and has unit diagonal, with entries ρjk\rho_{jk}, the covariance of the summed advantage with the kk-th normalized advantage is

Cov⁡(wTA^,A^(k))=(Σw)k=wk+∑j≠kwjρjk,(1)\operatorname{Cov}(\mathbf{w}^{\mathsf{T}}\widehat{\mathbf{A}},\widehat{A}^{(k)})=(\boldsymbol{\Sigma}\mathbf{w})_k=w_k+\sum_{j\ne k}w_j\rho_{jk}, \tag*{(1)}

so positively correlated rewards raise each other’s covariance with the aggregate. Both methods update the policy with the clipped surrogate objective of PPO ((Schulman et al., 2017)), a variant of the score-function estimator ((Williams, 1992)).

What Weighted Summation Does to Reward Weights

Effective Weights

Let A^∈RK\widehat{\mathbf{A}}\in\mathbb{R}^K have zero mean and covariance Σ≻0\boldsymbol{\Sigma}\succ0 with unit diagonal, and let Σ±1/2\boldsymbol{\Sigma}^{\pm1/2} denote the symmetric positive definite square root and its inverse. Define

z:=Σ−1/2A^,so thatA^=Σ1/2zandCov⁡(z)=I.(2)\mathbf{z}:=\boldsymbol{\Sigma}^{-1/2}\widehat{\mathbf{A}},\qquad\text{so that}\qquad\widehat{\mathbf{A}}=\boldsymbol{\Sigma}^{1/2}\mathbf{z}\qquad\text{and}\qquad\operatorname{Cov}(\mathbf{z})=\mathbf{I}. \tag*{(2)}

The coordinates of z\mathbf{z} are uncorrelated, and we call them the decorrelated components of the reward signal. Any W\mathbf{W} with WTW=Σ−1\mathbf{W}^{\mathsf{T}}\mathbf{W}=\boldsymbol{\Sigma}^{-1} decorrelates A^\widehat{\mathbf{A}}. The symmetric choice W=Σ−1/2\mathbf{W}=\boldsymbol{\Sigma}^{-1/2} is zero-phase component analysis (ZCA) whitening. Because the coordinates of A^\widehat{\mathbf{A}} are standardized, it coincides with what Kessy et al. [14] call ZCA-cor whitening of the raw rewards, which among all whitening transforms maximizes the total correlation ∑kCorr⁡(zk,A^(k))\sum_k\operatorname{Corr}(z_k,\widehat{A}^{(k)}) between each component and its own reward. Principal component analysis (PCA) whitening produces components with no correspondence to individual rewards, and Cholesky whitening depends on the order in which the rewards are listed.

Proposition 1 (Effective weights). For any w∈RK\mathbf{w}\in\mathbb{R}^K,

wTA^=(Σ1/2w)TzandwTΣ−1/2A^=wTz.(3)\mathbf{w}^{\mathsf{T}}\widehat{\mathbf{A}}=(\boldsymbol{\Sigma}^{1/2}\mathbf{w})^{\mathsf{T}}\mathbf{z}\qquad\text{and}\qquad\mathbf{w}^{\mathsf{T}}\boldsymbol{\Sigma}^{-1/2}\widehat{\mathbf{A}}=\mathbf{w}^{\mathsf{T}}\mathbf{z}. \tag*{(3)}

Weighted summation applies weights Σ1/2w\boldsymbol{\Sigma}^{1/2}\mathbf{w} to the decorrelated components, and summation after ZCA whitening applies w\mathbf{w}. The two agree for all w\mathbf{w} if and only if Σ=I\boldsymbol{\Sigma}=\mathbf{I}.

The proof is in Appendix § 10. We measure the discrepancy by the angle φw\varphi_{\mathbf{w}} between Σ1/2w\boldsymbol{\Sigma}^{1/2}\mathbf{w} and w\mathbf{w}.

Proposition 1 does not say which weighting is appropriate, because that depends on what w\mathbf{w} is meant to weight. Let Jk(θ)=Ex,y∼πθ[r(k)]J_k(\theta)=\mathbb{E}_{x,y\sim\pi_\theta}[r^{(k)}]. If w\mathbf{w} weights the raw rewards, the target is ∑kwkJk\sum_k w_kJ_k and correlation calls for no correction; summing the rewards before normalizing, as GRPO does, follows it, while summing separately normalized advantages follows ∑k(wk/σk)Jk\sum_k(w_k/\sigma_k)J_k (Proposition 2). If w\mathbf{w} is meant to weight the decorrelated components that track each reward, whitening is what applies those weights. We adopt the second reading, which matches the intuition that a weight such as 0.3 on length should not be inflated because length moves with accuracy, and we examine its consequences in § 5. A third reading, under which Cov⁡(aggregate⁡,A^(k))=wk\operatorname{Cov}(\operatorname{aggregate},\widehat{A}^{(k)})=w_k in (1), would call for the regression weights Σ−1w\boldsymbol{\Sigma}^{-1}\mathbf{w}, which we do not consider further.

Corollary 1 (Variance decomposition). Let κw:=wTΣw/∥w∥2\kappa_{\mathbf{w}}:=\mathbf{w}^{\mathsf{T}}\boldsymbol{\Sigma}\mathbf{w}/\lVert\mathbf{w}\rVert^2. Then

Var⁡(wTA^)=wTΣw=∥w∥2+∑j≠kwjwkρjk,Var⁡(wTΣ−1/2A^)=∥w∥2,\operatorname{Var}(\mathbf{w}^{\mathsf{T}}\widehat{\mathbf{A}})=\mathbf{w}^{\mathsf{T}}\boldsymbol{\Sigma}\mathbf{w}=\lVert\mathbf{w}\rVert^2+\sum_{j\ne k}w_jw_k\rho_{jk},\qquad\operatorname{Var}(\mathbf{w}^{\mathsf{T}}\boldsymbol{\Sigma}^{-1/2}\widehat{\mathbf{A}})=\lVert\mathbf{w}\rVert^2,

so cross-reward terms account for a fraction 1−1/κw1-1/\kappa_{\mathbf{w}} of the variance of the summed advantage.

We computed φw\varphi_w and κw\kappa_w from Σ‾\overline{\boldsymbol{\Sigma}}, the time average of the correlation estimate that WAS-PO maintained during training (Table 13). The angle is 9.6∘9.6^\circ, 16.0∘16.0^\circ and 20.7∘20.7^\circ for K=2,3,4K=2,3,4, and κw\kappa_w is 1.521.52, 1.951.95 and 2.302.30, so cross-reward terms carry 34%, 49% and 57% of the variance of the unwhitened sum. Because these use batch-level correlations, they describe the unwhitened sum under batch-level normalization (the WAS-PO-Diag control of § 4). For GDPO, Proposition 1 applies with Σ\boldsymbol{\Sigma} replaced by the pooled covariance of its group-normalized advantages, rescaled to unit diagonal; it reflects within-group correlation only, and we did not log it. Without renormalization, κw\sqrt{\kappa_w} would be the factor by which correlation inflates the scale of the summed advantage. GDPO and WAS-PO both renormalize across the batch, which removes this effect, so here κw\kappa_w describes only the composition of the aggregate.

KKPairMean ρ\rhoRangecond⁡(Σˉ)\operatorname{cond}(\bar{\Sigma})E\mathcal{E}
2acc–fmt0.645[0.530, 0.869]4.60.112
eigenvalues[0.355, 1.645]
3acc–fmt0.643[0.545, 0.828]
acc–len0.642[0.544, 0.827]12.20.251
fmt–len0.804[0.698, 1.000]
eigenvalues[0.196, 0.408, 2.396]
4acc–fmt0.652[0.561, 0.828]
acc–len0.651[0.560, 0.827]
acc–struct0.643[0.551, 0.828]16.20.360
fmt–len0.804[0.700, 1.000]
fmt–struct0.796[0.693, 1.000]
len–struct0.796[0.694, 1.000]
eigenvalues[0.196, 0.207, 0.420, 3.177]

Table 13. The correlation estimate Σ^t\hat{\Sigma}_t maintained by WAS-PO (seed 42 for K=2,3K = 2,3; three seeds for K=4K = 4). Mean and range are over the 500 training steps; Σˉ\bar{\Sigma} is the matrix of means, and its eigenvalues, cond⁡(Σˉ)\operatorname{cond}(\bar{\Sigma}) and E\mathcal{E} are listed.

The Objective Whitened Aggregation Follows

Let v:=Σ−1/2w\mathbf{v}:=\boldsymbol{\Sigma}^{-1/2}\mathbf{w} be the vector that whitened aggregation applies to the normalized advantages, and let D=diag⁡(σ1,…,σK)\mathbf{D}=\operatorname{diag}(\sigma_1,\ldots,\sigma_K) hold the reward standard deviations used for normalization.

Proposition 2 (Effective objectives). Consider the sequence-level score-function estimator without a KL penalty or length normalization. Treat all normalization statistics, the matrix Σ\boldsymbol{\Sigma} and the final batch scale s>0s>0 (the denominator of BN) as constants.

(a) Suppose the aggregate advantage is A~i=s−1vTA^i\widetilde{A}_i=s^{-1}\mathbf{v}^{\mathsf{T}}\widehat{\mathbf{A}}_i, with A^i(k)=(ri(k)−μk)/σk\widehat{A}_i^{(k)}=(r_i^{(k)}-\mu_k)/\sigma_k computed from batch statistics. Then the expected update E[A~∇θlog⁡πθ(y∣x)]\mathbb{E}[\widetilde{A}\nabla_\theta\log\pi_\theta(y\mid x)] equals s−1∇θJω(θ)s^{-1}\nabla_\theta J_{\boldsymbol{\omega}}(\theta), where

Jω(θ)=∑k=1KωkJk(θ),ω=D−1Σ−1/2w.(4)J_{\boldsymbol{\omega}}(\theta)=\sum_{k=1}^{K}\omega_kJ_k(\theta),\qquad\boldsymbol{\omega}=\mathbf{D}^{-1}\boldsymbol{\Sigma}^{-1/2}\mathbf{w}. \tag*{(4)}

(b) With per-prompt statistics μk(x)\mu_k(x), σk(x)>0\sigma_k(x)>0 and v=w\mathbf{v}=\mathbf{w}, as in GDPO, the same holds with J=Ex[∑k(wk/σk(x))Ey∼πθ(⋅∣x)[r(k)]]J=\mathbb{E}_x\left[\sum_k(w_k/\sigma_k(x))\mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}[r^{(k)}]\right].

Under these idealizations WAS-PO is a scalarized policy gradient with weights ω\boldsymbol{\omega} on the raw rewards. The idealizations are standard but not exact. The batch statistics depend on the sampled completions (an O(1/N)O(1/N) effect), the trainer’s token-level loss differs from the sequence-level estimator, and ω\boldsymbol{\omega} changes during training as the estimate of Σ\boldsymbol{\Sigma} and the scales D\mathbf{D} change. In our runs each batch is used for a single update, so the clipping in the surrogate is inactive. Part (b) shows that GDPO weights each reward per prompt by the inverse of its within-group standard deviation, so a reward that varies little on a prompt receives a larger weight there (at most wk/ϵw_k/\epsilon; a reward that is constant on the group contributes nothing). For binary rewards with G=4G=4, the within-group standard deviation is 0.50 or 0.58 when nonzero, so this reweighting matters mainly for continuous rewards. The batch-normalized sum without whitening follows D−1w\mathbf{D}^{-1}\mathbf{w}, whose entries are all positive. For two positively correlated rewards, and at all our measured correlation matrices, Σ−1/2\boldsymbol{\Sigma}^{-1/2} has negative off-diagonal entries, so ω\boldsymbol{\omega} can have negative entries. A negative ωk\omega_k rewards the policy for lowering reward kk when the other rewards are held fixed.

When Whitening Reverses a Reward

Proposition 3 (Sign reversal with two rewards). Let K=2K=2 and w=(a,b)\mathbf{w}=(a,b) with a≥b>0a\ge b>0, and let the correlation be ρ∈(−1,1)\rho\in(-1,1). Then v1>0v_1>0, and

v2<0⟺ρ>ρ⋆(a,b):=2aba2+b2.v_2<0\quad\Longleftrightarrow\quad\rho>\rho^\star(a,b):=\frac{2ab}{a^2+b^2}.

If a>ba>b, then ∣v2∣→∞\lvert v_2\rvert\to\infty and v2/v1→−1v_2/v_1\to-1 as ρ→1\rho\to1. If a=ba=b, then ρ⋆=1\rho^\star=1 and no reversal occurs.

The threshold depends only on the ratio of the weights. For the math setting of § 5.4, w=(1.0,0.3)\mathbf{w}=(1.0,0.3) gives ρ⋆=0.55\rho^\star=0.55; for the two-reward tool-calling setting, w=(1.0,0.5)\mathbf{w}=(1.0,0.5) gives ρ⋆=0.80\rho^\star=0.80. As an example, at ρ=0.98\rho=0.98 and w=(1.0,0.3)\mathbf{w}=(1.0,0.3), v=(2.94,−2.01)\mathbf{v}=(2.94,-2.01): an accurate but unformatted completion gains advantage relative to an accurate and formatted one. As ρ→1\rho\to1, the aggregate approaches a positive multiple of A^(1)−A^(2)\widehat{A}^{(1)}-\widehat{A}^{(2)}. Batch-level renormalization removes the growth in magnitude but not this direction. Corollary 2 extends the result to KK rewards when two of them become nearly collinear.

Corollary 2 (Near-duplicate rewards). Let Σn\boldsymbol{\Sigma}_n be a sequence of correlation matrices whose smallest eigenvalue λ1n\lambda_1^n is simple and tends to zero, with unit eigenvector un→u\mathbf{u}_n \to\mathbf{u}, while the other eigenvalues stay above some λ>0\lambda> 0. If uTw≠0\mathbf{u}^{\mathsf{T}}\mathbf{w} \ne0, then vn/∥vn∥→sign⁡(uTw)u\mathbf{v}_n/\lVert\mathbf{v}_n\rVert\to\operatorname{sign}(\mathbf{u}^{\mathsf{T}}\mathbf{w})\mathbf{u}. In particular, if under the same hypotheses ρij→1\rho_{ij} \to1 for one pair i≠ji \ne j, then u=±(ei−ej)/2\mathbf{u} = \pm(\mathbf{e}_i - \mathbf{e}_j)/\sqrt{2}, where ei\mathbf{e}_i is the ii-th standard basis vector, and the whitened weights approach a multiple of ei−ej\mathbf{e}_i - \mathbf{e}_j whenever wi≠wjw_i \ne w_j, so the lighter-weighted of the two rewards is reversed.

For K>2K > 2 we summarize the sign pattern of v\mathbf{v} with an inversion index,

ψ(Σ,w):=max⁡{0,−min⁡kvk}max⁡kvk,(5)\psi(\boldsymbol{\Sigma},\mathbf{w}) := \frac{\max\{0,-\min_k v_k\}}{\max_k v_k}, \tag*{(5)}

which is zero when all whitened weights are nonnegative and approaches one when two rewards become collinear (the pair case of Corollary 2). The denominator is positive because w>0\mathbf{w} > 0 and wTv=wTΣ−1/2w>0\mathbf{w}^{\mathsf{T}}\mathbf{v} = \mathbf{w}^{\mathsf{T}}\boldsymbol{\Sigma}^{-1/2}\mathbf{w} > 0 force some vkv_k to be positive. For K=2K = 2, ψ<1\psi< 1 because v1+v2=(a+b)/1+ρ>0v_1 + v_2 = (a + b)/\sqrt{1 + \rho} > 0; for K>2K > 2 it can exceed one.

In practice we compute v\mathbf{v} and ψ\psi from vt=Wtw\mathbf{v}_t = \mathbf{W}_t\mathbf{w}, which includes the eigenvalue floor of § 4.

Table 1 lists v\mathbf{v} and ψ\psi for our settings. At Σ‾\overline{\boldsymbol{\Sigma}}, WAS-PO places negative weights on length (−0.15-0.15 at K=3K = 3) and on length and structure (−0.06-0.06 and −0.26-0.26 at K=4K = 4), because both are largely predicted by accuracy and format. In all three tool-calling settings, ψ≤0.23\psi\le0.23 at Σ‾\overline{\boldsymbol{\Sigma}}. A negative weight on length means that WAS-PO penalizes length beyond what accuracy and format predict; it does not by itself make the policy shorter. The estimate was sometimes much worse conditioned: its format–length entry reached 1.000 during the three- and four-reward runs (Table 13). If every pair were at its largest logged value, WAS-PO would put a large positive weight on format and a large negative weight on length at K=3K = 3 (ψ=1.01\psi= 1.01), and on structure and length at K=4K = 4 (ψ=0.82\psi= 0.82). None of the three- or four-reward runs collapsed, unlike the math runs; § 5.4 discusses why.

Settingw\mathbf{w}φw\varphi_{\mathbf{w}}v\mathbf{v}ψ\psiΔ\DeltaAUC
Tool, K=2K = 2, time-averaged(1.0,0.5)(1.0, 0.5)9.6∘9.6^\circ(1.00,0.17)(1.00, 0.17)0.000.00−0.014-0.014
Tool, K=2K = 2, largest(1.0,0.5)(1.0, 0.5)(1.24,−0.14)(1.24, -0.14)0.110.11
Tool, K=3K = 3, time-averaged(1.0,0.5,0.3)(1.0, 0.5, 0.3)16.0∘16.0^\circ(1.05,0.30,−0.15)(1.05, 0.30, -0.15)0.150.15+0.041+0.041
Tool, K=3K = 3, largest(1.0,0.5,0.3)(1.0, 0.5, 0.3)(1.22,9.84,−9.95)(1.22, 9.84, -9.95)1.011.01
Tool, K=4K = 4, time-averaged(1.0,0.5,0.3,0.2)(1.0, 0.5, 0.3, 0.2)20.7∘20.7^\circ(1.10,0.39,−0.06,−0.26)(1.10, 0.39, -0.06, -0.26)0.230.23+0.049+0.049
Tool, K=4K = 4, largest(1.0,0.5,0.3,0.2)(1.0, 0.5, 0.3, 0.2)(1.32,16.5,−3.25,−13.5)(1.32, 16.5, -3.25, -13.5)0.820.82
Math, step 10 (ρ≈0.98\rho\approx0.98)(1.0,0.3)(1.0, 0.3)(2.94,−2.01)(2.94, -2.01)0.690.69−0.228-0.228

Table 1. Diagnostics at the estimated correlations. φw\varphi_{\mathbf{w}} is the angle between Σ1/2w\boldsymbol{\Sigma}^{1/2}\mathbf{w} and w\mathbf{w} (Proposition 1); v=Ww\mathbf{v} = \mathbf{W}\mathbf{w} with the eigenvalue floor of § 4; ψ\psi is the inversion index (5). “Time-averaged” rows use Σ‾\overline{\boldsymbol{\Sigma}} (Table 13). “Largest” rows set every pair to its largest logged value; the maxima need not occur at the same step. At K=3,4K = 3, 4 these matrices are singular to numerical precision, so there the magnitudes of v\mathbf{v} are set by the floor and only its signs and ψ\psi are meaningful. Δ\DeltaAUC is WAS-PO minus GDPO.

Estimation Error

In practice, Σ\boldsymbol{\Sigma} is estimated during training from NN rollouts per batch.

Assumption 1. An estimate Σ^\widehat{\boldsymbol{\Sigma}} of Σ\boldsymbol{\Sigma} from NN rollouts satisfies E∥Σ^−Σ∥F2≤cK2/(N−1)\mathbb{E}\lVert\widehat{\boldsymbol{\Sigma}}-\boldsymbol{\Sigma}\rVert_F^2 \le cK^2/(N-1) for a constant cc.

Under bivariate normality, the asymptotic variance of a sample correlation is (1−ρ2)2/N≤1/N(1-\rho^2)^2/N \le1/N, which gives c≈1c \approx1. For skewed binary rewards this variance can be several times larger, and correlation among the GG completions of a prompt can reduce the effective sample size by up to a factor of GG. We use c=4c = 4 as a working value and have not verified it for our rewards.

Theorem 1 (Whitening estimation error). Let Σ\boldsymbol{\Sigma} be symmetric, let Σ^\widehat{\boldsymbol{\Sigma}} be a random symmetric matrix satisfying Assumption 1, and suppose that almost surely all eigenvalues of Σ\boldsymbol{\Sigma} and Σ^\widehat{\boldsymbol{\Sigma}} are at least λ0>0\lambda_0 > 0. Then

E[∥Σ^−1/2−Σ−1/2∥2]≤cK2∥w∥24λ03(N−1).(6)\mathbb{E}\left[\left\lVert\widehat{\boldsymbol{\Sigma}}^{-1/2}-\boldsymbol{\Sigma}^{-1/2}\right\rVert^2\right] \le\frac{cK^2\lVert\mathbf{w}\rVert^2}{4\lambda_0^3(N-1)}. \tag*{(6)}

The proof (Appendix § 11) bounds the Fréchet derivative of Σ↦Σ−1/2\boldsymbol{\Sigma} \mapsto\boldsymbol{\Sigma}^{-1/2} with the Daleckii–Krein formula [4 Section V.3] and integrates it along the segment from Σ\boldsymbol{\Sigma} to Σ^\widehat{\boldsymbol{\Sigma}}. Every matrix on that segment has eigenvalues at least λ0\lambda_0 because the smallest eigenvalue is a concave function of a symmetric matrix.

We compare (6) with the size of the correction, E:=∥Σ−1/2w−w∥2\mathcal{E} := \lVert\boldsymbol{\Sigma}^{-1/2}\mathbf{w} - \mathbf{w}\rVert^2, which is 0.112, 0.251 and 0.360 for K=2,3,4K = 2,3,4. WAS-PO estimates Σ\boldsymbol{\Sigma} from all rollouts in a batch (N=32N = 32 in our main runs) and smooths the estimate with an exponential moving average (EMA) that gives weight 1−β1-\beta to the newest batch. If successive batch estimates were independent with mean Σ\boldsymbol{\Sigma}, the EMA started from the first batch would multiply the variance of each entry at step tt by

ft=β2(t−1)+1−β1+β(1−β2(t−1)).f_t = \beta^{2(t-1)} + \frac{1-\beta}{1+\beta}\left(1-\beta^{2(t-1)}\right).

This factor tends to (1−β)/(1+β)=1/199(1-\beta)/(1+\beta) = 1/199 at β=0.99\beta= 0.99, but equals 0.38 at t=50t = 50, 0.14 at t=100t = 100 and 0.023 at t=200t = 200; its average over 500 steps is 0.105. Appendix § 12 evaluates the bound under this idealization, with λ0\lambda_0 set to the smallest eigenvalue of Σ‾\overline{\boldsymbol{\Sigma}}. At K=3K = 3 and N=32N = 32, the stationary bound is 0.26, about the size of the correction, and at t=100t = 100 it is 7.3. The bound is therefore too loose to rule out estimation noise comparable to the correction for much of training, and it does not apply when the estimate approaches singularity, as it did at K=3K = 3 and K=4K = 4. Its value is qualitative: the bound grows as K2λmin⁡−3K^2\lambda_{\min}^{-3}, so adding near-redundant rewards (small λmin⁡\lambda_{\min}) makes the whitening matrix much harder to estimate, and it falls with the effective number of rollouts behind the estimate.

Method

Each training iteration of WAS-PO has three stages (Algorithm 1), plus an optional inversion check. From here on, A^(k)\widehat{A}^{(k)} denotes the batch-normalized advantage defined below.

Batch-level normalization. Each reward is centered and scaled by its mean and standard deviation over all NN rollouts in the batch. The covariance of A^\widehat{A} on a batch is then the batch correlation matrix CtC_t, which feeds the EMA estimate used for whitening. Batch-level normalization also changes the baseline and the scale relative to GDPO. Under GDPO a completion is compared with other completions to the same prompt, and each reward is scaled by its within-group standard deviation (Proposition 2(b)). Under batch-level normalization, completions to easy prompts tend to receive positive advantages, much as in the global advantage normalization of REINFORCE++ [10]. The WAS-PO-Diag control below separates this change from whitening.

Correlation estimate. The Pearson correlation matrix CtC_t of the raw rewards is computed over the batch and replaced by I\mathbf{I} if any reward has zero variance in the batch. It is smoothed as Σ^t=βΣ^t−1+(1−β)Ct\widehat{\boldsymbol{\Sigma}}_t = \beta\widehat{\boldsymbol{\Sigma}}_{t-1} + (1-\beta)C_t with β=0.99\beta= 0.99 and Σ^1=C1\widehat{\boldsymbol{\Sigma}}_1 = C_1.

Whitening and aggregation. With the eigendecomposition Σ^t=Utdiag⁡(η1,…,ηK)UtT\widehat{\boldsymbol{\Sigma}}_t = \mathbf{U}_t \operatorname{diag}(\eta_1,\ldots,\eta_K)\mathbf{U}_t^{\mathsf{T}}, we set Wt=Utdiag⁡(max⁡(ηi,10−4)−1/2)UtT\mathbf{W}_t = \mathbf{U}_t \operatorname{diag}(\max(\eta_i,10^{-4})^{-1/2})\mathbf{U}_t^{\mathsf{T}} and compute

A^iWAS−PO=BN⁡(wTWtA^)i.(7)\widehat{A}^{\mathrm{WAS-PO}}_i = \operatorname{BN}(\mathbf{w}^{\mathsf{T}}\mathbf{W}_t\widehat{A})_i. \tag*{(7)}

The eigenvalue floor bounds ∥Wt∥2\lVert\mathbf{W}_t\rVert_2 by 100. It has no effect when every eigenvalue of Σ^t\widehat{\boldsymbol{\Sigma}}_t exceeds 10−410^{-4}, and for two rewards it removes a reversal only when ρ>1−10−4\rho> 1-10^{-4} and a/b<1.0142a/b < 1.0142.

Inversion check (optional). Since vt=Wtw\mathbf{v}_t = \mathbf{W}_t\mathbf{w} is available at every step, ψ(Σ^t,w)\psi(\widehat{\boldsymbol{\Sigma}}_t,\mathbf{w}) can be logged at negligible cost, and Wt\mathbf{W}_t can be replaced by I\mathbf{I} when needed (§ 6). The experiments reported here did not use this check.

Controls. WAS-PO-Diag zeros the off-diagonal entries of Σ^t\widehat{\boldsymbol{\Sigma}}_t, which sets Wt=I\mathbf{W}_t = \mathbf{I} because the diagonal is one. It runs through the same code path as WAS-PO, keeps batch-level normalization and drops whitening, so it differs from GDPO only in normalization and from WAS-PO only in whitening. WAS-PO-LW applies Ledoit–Wolf shrinkage [15] to Σ^t\widehat{\boldsymbol{\Sigma}}_t before inversion.

Algorithm 1 WAS-PO advantages for one training iteration

Require: rewards R∈RN×K\mathbf{R} \in\mathbb{R}^{N \times K}, weights w\mathbf{w}, EMA state Σ^\hat{\boldsymbol{\Sigma}}, decay β\beta

1: A^i(k)←(Rik−mean⁡iRik)/(std⁡iRik+ϵ)\hat{A}_i^{(k)} \leftarrow(R_{ik}-\operatorname{mean}_i R_{ik})/(\operatorname{std}_i R_{ik}+\epsilon) 2: C←Corr⁡(R)\mathbf{C} \leftarrow\operatorname{Corr}(\mathbf{R}), or C←I\mathbf{C} \leftarrow\mathbf{I} if some column of R\mathbf{R} is constant 3: Σ^←C\hat{\boldsymbol{\Sigma}} \leftarrow\mathbf{C} at the first iteration, else Σ^←βΣ^+(1−β)C\hat{\boldsymbol{\Sigma}} \leftarrow\beta\hat{\boldsymbol{\Sigma}}+(1-\beta)\mathbf{C} 4: W←Udiag⁡(max⁡(ηi,10−4)−1/2)UT\mathbf{W} \leftarrow\mathbf{U}\operatorname{diag}(\max(\eta_i,10^{-4})^{-1/2})\mathbf{U}^{\mathsf{T}}, where Σ^=Udiag⁡(η)UT\hat{\boldsymbol{\Sigma}}=\mathbf{U}\operatorname{diag}(\boldsymbol{\eta})\mathbf{U}^{\mathsf{T}} 5: A^i←wTWA^i\hat{A}_i \leftarrow\mathbf{w}^{\mathsf{T}}\mathbf{W}\hat{\mathbf{A}}_i; A^iWAS-PO←BN⁡(A~)i\hat{A}_i^{\mathrm{WAS\text{-}PO}} \leftarrow\operatorname{BN}(\tilde{A})_i 6: (optional) if the inversion check fires, recompute A~i\tilde{A}_i with W=I\mathbf{W}=\mathbf{I} 7: update πθ\pi_\theta with the clipped surrogate objective using {A^iWAS-PO}\{\hat{A}_i^{\mathrm{WAS\text{-}PO}}\}

Experiments

Setup

Training. We train Qwen2.5-1.5B-Instruct (Qwen Team (2024)) for 500 steps on the single-turn subset of the Berkeley Function Calling Leaderboard v3 data (BFCL; Patil et al. (2025)), excluding the irrelevance category, using the copy distributed as llamastack/bfcl_v3. The last 20% of this subset, in dataset order, is held out from training. Each step samples 8 prompts with G=4G=4 completions each, so N=32N=32, with temperature 1.0 and at most 128 new tokens (all hyperparameters are in Table 14). GRPO, GDPO and WAS-PO share one implementation, GDPO’s patched version of TRL 0.18, and differ only in how advantages are computed; rewards are gathered across the eight GPUs before advantages are formed. There is no KL penalty, and each batch is used for one update. Appendix § 14 gives the prompt template and other details, including a reward-wiring pitfall that reduces multi-reward training to a single reward.

Tool, K=2K = 2Tool, K=3K = 3Tool, K=4K = 4Math
ModelQwen2.5-1.5B-InstructQwen2.5-1.5B-InstructQwen2.5-1.5B-InstructQwen2.5-1.5B-Instruct
DataBFCL v3, single-turn, irrelevance excluded; first 80%BFCL v3, single-turn, irrelevance excluded; first 80%BFCL v3, single-turn, irrelevance excluded; first 80%GSM8K train
Group size GG4444
Prompts per step8888
Rollouts per step NN32323232
Max new tokens128128128384
Max prompt length512 tokens512 tokens512 tokens512 tokens
Training steps500500500500
Seeds7733
Reward weights(1.0, 0.5)(1.0, 0.5, 0.3)(1.0, 0.5, 0.3, 0.2)(1.0, 0.3)
Conditional (gated) rewardsyesyesyesyes
EMA decay β\beta0.990.990.990.99
Eigenvalue floor; ϵ\epsilon10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}
KL coefficient0 (no reference model)0 (no reference model)0 (no reference model)0 (no reference model)
Learning rate1×10−61 \times10^{-6}1×10−61 \times10^{-6}1×10−61 \times10^{-6}1×10−61 \times10^{-6}
OptimizerAdamW, linear decay, no warmupAdamW, linear decay, no warmupAdamW, linear decay, no warmupAdamW, linear decay, no warmup
Updates per batch1111
Samplingtemperature 1.0, top-pp 1.0temperature 1.0, top-pp 1.0temperature 1.0, top-pp 1.0temperature 1.0, top-pp 1.0
Held-out decodinggreedygreedygreedygreedy
Precisionbfloat16bfloat16bfloat16bfloat16
FrameworkTRL 0.18 with the GDPO patchTRL 0.18 with the GDPO patchTRL 0.18 with the GDPO patchTRL 0.18 with the GDPO patch
Hardware8×\timesH1008×\timesH1008×\timesH1008×\timesH100

Table 14. Training hyperparameters. Settings not listed are the defaults of TRL 0.18 and Transformers.

Rewards. The accuracy reward parses the first JSON object in a completion and compares it with the BFCL ground truth. It is 0 if no object parses, 0.1 if the function name is wrong, and otherwise 0.5 times the fraction of ground-truth argument names present plus 0.5 times the fraction of argument values that match. The format reward is 1 if a JSON object parses and 0 otherwise. The length reward is exp⁡{−12(ℓ/512−1)2}\exp\left\{-\frac{1}{2}(\ell/512-1)^2\right\} for a completion of ℓ\ell whitespace-separated words; with completions capped at 128 tokens, it lies between 0.61 and about 0.76 and increases with length. The structure reward (K=4K=4) is 1 for an object with a function name and nonempty arguments, 0.5 for a name alone, and 0 otherwise. We use GDPO’s conditional rewards: every secondary reward is set to zero when the accuracy reward is not positive, and we call such rewards gated. The gated format reward is therefore exactly the indicator that accuracy is positive, and the length and structure rewards are zero whenever it is zero. This shared gate is the main source of the correlations in Table 13.

Metrics. Our main metric is training AUC, the mean over all T=500T=500 steps of the weighted batch-mean reward,

AUC⁡=1T∑t=1TwTrˉt\operatorname{AUC}=\frac{1}{T}\sum_{t=1}^{T}\mathbf{w}^{\mathsf{T}}\bar{\mathbf{r}}_t

where rˉt\bar{\mathbf{r}}_t is the batch-mean reward vector at step tt. Its maximum is ∑kwk\sum_k w_k (1.5, 1.8 and 2.0 for K=2,3,4K=2,3,4), so AUC values are not comparable across KK. We also report held-out accuracy: the accuracy reward of the final checkpoint under greedy decoding on the first 500 prompts of the held-out 20%, the same prompts for every method, KK and seed. Seeds are paired across methods. Training comparisons use exact two-sided Wilcoxon signed-rank tests on the paired differences; with seven seeds the smallest attainable pp-value is 2/128≈0.0162/128\approx0.016, and with three seeds it is 0.25. We report Δ\Delta as the mean paired difference and dzd_z as that mean divided by the standard deviation of the paired differences. For held-out accuracy we kept per-method summaries rather than per-seed pairs, so we use unpaired two-sided Welch tt-tests.

Three and Four Rewards

Three rewards. Figure 1 shows training curves for two and three rewards. The rewards are accuracy (w1=1.0w_1 = 1.0), format (w2=0.5w_2 = 0.5) and length (w3=0.3w_3 = 0.3). Over seven seeds, training AUC was 1.013 ±\pm 0.011 for GRPO, 1.089 ±\pm 0.006 for GDPO and 1.130 ±\pm 0.007 for WAS-PO (mean ±\pm SD; Table 5, Figure 1b). WAS-PO was ahead of GDPO in every seed, by Δ=+0.041\Delta= +0.041 (p=0.016p = 0.016, dz=4.6d_z = 4.6).

Figure 1. (a) Two rewards. (b) Three rewards. Training reward of GRPO, GDPO and WAS-PO. Curves and ±1\pm1 SD bands are computed over a subset of the seeds (three at K=2K = 2). The per-seed AUCs in Table 7 and Table 5 cover all seven seeds.

SeedGRPOGDPOWAS-POΔ\Delta
421.03031.09581.1252+0.0294
1230.99951.08621.1386+0.0524
4561.00281.09031.1213+0.0310
7891.01881.09471.1391+0.0444
10111.00591.07981.1303+0.0505
20241.01101.08631.1246+0.0383
99991.02251.08791.1315+0.0436
Mean ± SD1.013 ± 0.0111.089 ± 0.0061.130 ± 0.007+0.041

Table 5. Training AUC with three rewards, seven seeds. Exact two-sided Wilcoxon p=0.016p = 0.016; dz=4.6d_z = 4.6.

Held-out accuracy was 0.397 ±\pm 0.076 for GRPO, 0.446 ±\pm 0.012 for GDPO and 0.438 ±\pm 0.011 for WAS-PO (Table 9). WAS-PO was 0.008 below GDPO, which is not significant (Welch p=0.22p = 0.22) and within the standard error of a single run’s mean accuracy on 500 prompts (at most about 0.022). The exact-match rate, the fraction of completions with accuracy reward 1, was 0.133 for WAS-PO and 0.131 for GDPO, and the format rate was 0.99 for both.

Table 9: Held-out accuracy (accuracy reward of the final checkpoint under greedy decoding on 500 held-out prompts; mean ±\pm SD over seeds). pp is from an unpaired two-sided Welch tt-test computed from these summaries. At K=4K = 4, scores were recorded for two of three seeds.

SettingGRPOGDPOWAS-POΔ\Delta vs. GDPOpp
K=2K = 2 (seven seeds)0.479 ±\pm 0.0060.474 ±\pm 0.0080.457 ±\pm 0.022−0.017-0.0170.09
K=3K = 3 (seven seeds)0.397 ±\pm 0.0760.446 ±\pm 0.0120.438 ±\pm 0.011−0.008-0.0080.22
K=4K = 4 (two seeds)0.521, 0.5170.517, 0.5150.485, 0.487−0.030-0.030—

Table 9.

Four rewards. With the structure reward added (w4=0.2w_4 = 0.2), training AUC over three seeds was 1.116 ±\pm 0.024 for GRPO, 1.257 ±\pm 0.007 for GDPO and 1.306 ±\pm 0.009 for WAS-PO (Table 6, Figure 4). WAS-PO was ahead in all three seeds, by +0.040, +0.050 and +0.058. Three seeds cannot reach significance in a Wilcoxon test (the smallest two-sided pp is 0.25), so we read this as consistent with the three-reward result rather than as independent evidence. Averaged over training, WAS-PO was ahead on every reward: accuracy +0.014, format +0.038, length +0.011 and structure +0.032 (Table 12). Weighted by ww, about two-thirds of this gain comes from the three gated rewards; the format mean, which equals the rate of completions with positive accuracy, was 0.871 under WAS-PO against 0.833 under GDPO. On the held-out set we recorded scores for two of the three seeds: WAS-PO scored 0.485 and 0.487, GDPO 0.517 and 0.515, and GRPO 0.521 and 0.517. With four rewards, WAS-PO was therefore about 0.03 below GDPO on held-out accuracy, and GRPO matched or exceeded GDPO. Relative to GDPO, the gain in training AUC was 3.8% with three rewards and 3.9% with four (Table 2; Figure 2 in Appendix § 13).

KKSeedsRewardsGDPOWAS-POΔ\DeltaRelativecond⁡(Σˉ)\operatorname{cond}(\bar{\Sigma})
27acc, fmt0.9850.970−0.014-0.014−1.5%-1.5\%4.6
37acc, fmt, len1.0891.130+0.041+0.041+3.8%+3.8\%12.2
43acc, fmt, len, struct1.2571.306+0.049+0.049+3.9%+3.9\%16.2

Table 2. Training AUC by number of rewards. Δ\Delta is the mean paired difference, WAS-PO minus GDPO, and Relative is Δ\Delta divided by GDPO’s mean, both computed from unrounded per-seed values. cond⁡(Σˉ)\operatorname{cond}(\bar{\Sigma}) is the condition number of the time-averaged correlation estimate. The settings differ in which rewards they contain as well as in how many, and AUC is not comparable across rows. acc, fmt, len and struct are the accuracy, format, length and structure rewards.

Figure 2. Training AUC for two, three and four rewards (mean ±\pm SD over seeds). Each panel has its own axis because AUC is not comparable across KK.

SeedGRPOGDPOWAS-POΔ\Delta
421.10491.26331.3036+0.0403
1231.09951.24881.2987+0.0499
4561.14391.25791.3154+0.0575
Mean ± SD1.116 ± 0.0241.257 ± 0.0071.306 ± 0.009+0.049

Table 6. Training AUC with four rewards, three seeds. dz=5.7d_z = 5.7; the smallest attainable two-sided Wilcoxon pp with three seeds is 0.25.

Table 12: Per-reward training reward with four rewards, averaged over 500 steps (three seeds). Per-reward values were not logged for GRPO. The weighted sums of these means (1.262 and 1.305) agree with the AUCs in Table 6 to within 0.006.

MethodAccuracyFormatLengthStructure
GDPO0.5210.8330.5280.831
WAS-PO0.5350.8710.5390.863
Δ\Delta+0.014+0.038+0.011+0.032

Table 12.

Training reward with four rewards

Figure 4. Training reward with four rewards (three seeds, mean ±\pm SD).

Two Rewards and Component Ablations

With accuracy and format only, training AUC over seven seeds was 0.986 for GRPO, 0.985 for GDPO and 0.970 for WAS-PO, which trailed GDPO in every seed (Δ=−0.014\Delta= -0.014, p=0.016p = 0.016, dz=−4.5d_z = -4.5; Table 7, Figure 1a). Held-out accuracy was 0.479 ±\pm 0.006 for GRPO, 0.474 ±\pm 0.008 for GDPO and 0.457 ±\pm 0.022 for WAS-PO (Welch p=0.09p = 0.09 against GDPO).

SeedGRPOGDPOWAS-POΔ\Delta
420.99030.98700.9721−0.0149
1230.97590.98260.9722−0.0104
4560.99180.98730.9713−0.0160
7890.98980.98330.9727−0.0106
10110.98250.98700.9691−0.0179
20240.98250.98140.9683−0.0131
99990.99080.98490.9668−0.0181
Mean ± SD0.986 ± 0.0060.985 ± 0.0020.970 ± 0.002−0.014

Table 7. Training AUC with two rewards, seven seeds. Exact two-sided Wilcoxon p=0.016p = 0.016; dz=−4.5d_z = -4.5.

Changing how Σ\Sigma is estimated did not close the gap (Table 3). EMA decays from β=0.5\beta= 0.5 to 0.99, Ledoit–Wolf shrinkage (which lowered the largest estimated correlation from 0.87 to 0.37) and larger groups all left WAS-PO below GDPO. At G=8G = 8 and at G=16G = 16, WAS-PO was below GDPO in each of the three seeds (Table 8).

Variant (G=4G = 4 unless stated)SeedsGDPO, same GGAUCΔ\Delta
WAS-PO70.9850.970−0.014-0.014
WAS-PO-LW70.9850.968−0.017-0.017
WAS-PO-Diag30.9860.965−0.021-0.021
WAS-PO, G=8G = 8 (N=64N = 64)31.0030.984−0.019-0.019
WAS-PO, G=16G = 16 (N=128N = 128)31.0110.993−0.018-0.018
WAS-PO, β=0.5\beta= 0.510.9900.967−0.023-0.023
WAS-PO, β=0.7\beta= 0.710.9900.970−0.020-0.020
WAS-PO, β=0.9\beta= 0.910.9900.955−0.035-0.035
WAS-PO, β=0.95\beta= 0.9510.9900.955−0.035-0.035
WAS-PO, β=0.99\beta= 0.9910.9900.966−0.025-0.025

Table 3. Ablations with two rewards. Training AUC. Δ\Delta is relative to GDPO trained with the same group size GG on the same seeds (seeds 42, 123 and 456 for the three-seed rows; seed 42 for the decay sweep) and is computed from unrounded values. WAS-PO-Diag keeps batch-level normalization and removes whitening. The decay sweep is a separate set of single-seed runs with its own GDPO baseline (0.990). Its β=0.99\beta= 0.99 run repeats the default configuration and scored 0.966, against 0.972 for seed 42 in Table 7, so reruns at a fixed seed vary by about 0.006.

GGSeedGRPOGDPOWAS-POΔ\Delta
8421.00211.00610.9849−0.0212
81230.99640.99760.9837−0.0139
84561.00201.00500.9830−0.0220
16421.01251.01110.9938−0.0173
161231.00841.00640.9905−0.0159
164561.01691.01530.9960−0.0193

Table 8. Training AUC at larger group sizes, two rewards, three seeds. Eight prompts per step, so N=64N = 64 at G=8G = 8 and N=128N = 128 at G=16G = 16.

WAS-PO-Diag scored 0.965 on seeds 42, 123 and 456, while WAS-PO scored 0.972 on the same seeds. The gap (0.007, three seeds) is about the size of the rerun variation in Table 3. It weakly suggests that, with two rewards, the deficit relative to GDPO comes from batch-level normalization rather than from whitening. The theory does not point to whitening as the cause either. At Σˉ\bar{\Sigma}, ψ=0\psi= 0; the estimate started at 0.87, above ρ⋆=0.80\rho^{\star} = 0.80, and stayed above it only for the first few dozen steps (Figure 3, top).

Figure 3. Correlation estimate and reward difference (seed 42, logged every 10 steps). Blue: the EMA estimate of ρ(acc,fmt)\rho(\mathrm{acc},\mathrm{fmt}). Red: training reward of WAS-PO minus GDPO, labeled “advantage” in the legend but meaning a difference in reward. Top: tool calling with two rewards; the two series have correlation 0.44, but both trend over training, so we do not interpret this. Bottom: the earlier math run with a constant format reward, where the estimate is the identity and the difference fluctuates around zero.

In an earlier math run, the format reward was a JSON-format check applied to math answers and was therefore always zero. With a constant reward, Ct=IC_t = I at every step and WAS-PO reduced to WAS-PO-Diag. Its gap to GDPO was −0.004-0.004 over three seeds (Figure 3, bottom; Table 10), so batch-level normalization does not always cost much training reward.

Table 10: Training AUC in the earlier math run with a constant format reward, where WAS-PO reduces to WAS-PO-Diag (Section 5.3). This is not the run of § 5.4.

SeedGRPOGDPOWAS-POΔ\Delta
420.80770.80410.8085+0.0044
1230.79880.81200.7943−0.0177-0.0177
4560.80750.79940.7995+0.0001
Mean0.8050.8050.801−0.004-0.004

Table 10.

ZCA-Inversion

We train the same model on the GSM8K training set (Cobbe et al., [5]), with a system prompt asking it to end with the answer in the form The answer is: X. The accuracy reward is 1 if the extracted answer matches the reference and 0 otherwise; the extractor takes a boxed answer if there is one and otherwise the last number in the completion. The format reward is 1 if the completion contains The answer is: or a boxed answer, and 0 otherwise. Both rewards are gated as in tool calling, with w=(1.0,0.3)w = (1.0, 0.3) (ρ⋆=0.55\rho^{\star} = 0.55); N=32N = 32, with at most 384 new tokens. Since the gated format reward is 1 exactly when the answer is correct and formatted, the batch correlation of the two binary rewards follows from their means paccp_{\mathrm{acc}} and pfmtp_{\mathrm{fmt}}: ρ=pfmt(1−pacc)/(pacc(1−pfmt))\rho= \sqrt{p_{\mathrm{fmt}}(1-p_{\mathrm{acc}})/(p_{\mathrm{acc}}(1-p_{\mathrm{fmt}}))} (Appendix § 10). From the means in Table 11, ρ≈0.96\rho\approx0.96 under GDPO and 0.98 under WAS-PO at step 10, well above ρ⋆\rho^{\star}. Under GDPO the means are equal at steps 50 and 500, so ρ=1\rho= 1. These values are approximate because the means are averaged over batches and seeds. Over three seeds, training AUC was 1.050 for GRPO, 1.049 for GDPO and 0.821 for WAS-PO. Under WAS-PO the format reward fell from 0.666 at step 10 to 0.132 at step 500 while accuracy kept rising; under GDPO both rose. This matches the sign that Proposition 3 assigns to the format reward: whenever ρ>ρ⋆\rho> \rho^{\star}, the completions that are correct and unformatted receive the largest advantages. Because the extractor falls back to the last number, such completions are easy to produce, and the policy learned to produce them. As the format reward drops, the two rewards decouple: by the same formula, the batch correlation under WAS-PO was about 0.41 at step 50 and 0.19 at step 500. The EMA forgets slowly (half-life 69 steps at β=0.99\beta= 0.99): starting from 0.98, the estimate cannot fall below ρ⋆\rho^{\star} before step 58 unless new batches are negatively correlated, so the reversal stayed in force while the format reward fell to 0.277 by step 50.

Table 11: Per-reward batch-mean training reward in the math setting (three seeds).

StepWAS-PO accWAS-PO fmtGDPO accGDPO fmt
100.6750.6660.6910.675
500.6970.2770.7750.775
5000.8060.1320.8630.863

Table 11.

The three- and four-reward runs show that a nearly singular estimate does not by itself cause collapse. In those runs the format–length entry of the estimate reached 1.000; this probably happened early in training, since the estimate is initialized from the first batch and the two-reward trace (Figure 3) is highest at the start. Had every pair been at its maximum, ψ\psi would have been 0.82–1.01. Among completions that pass the gate, however, the length reward varies only between 0.61 and about 0.76, so a reversal along that direction changes the reward means little. In the math run, by contrast, the reversed reward could be switched off entirely at no cost to accuracy. Whether reward models trained on overlapping data reach correlations like those of the math rewards is untested here; ensembles of such models are known to share errors [8].

Why Does WAS-PO Raise the Weighted Training Reward?

Training AUC weights the raw rewards by w\mathbf{w}. By Proposition 2, neither method follows exactly that objective: GDPO weights reward kk by wk/σk(x)w_k/\sigma_k(x) on each prompt, and WAS-PO follows ω\boldsymbol{\omega}, which puts negative weight on length or structure at Σˉ\bar{\Sigma}. That WAS-PO nonetheless scores higher on wTr\mathbf{w}^{\mathsf{T}}\mathbf{r} is therefore not explained by its applying the intended weights. At K=4K = 4, the per-reward means show that most of the gain is in the gated rewards, through a higher rate of completions that parse and score above zero (0.871 against 0.833), which also explains why every reward moved up together. Three explanations fit. First, whitening shifts weight toward accuracy relative to w\mathbf{w} (v1/v2v_1/v_2 is 3.6 at K=3K = 3 and 2.8 at K=4K = 4, against w1/w2=2w_1/w_2 = 2), and every secondary reward is gated on accuracy being positive. Second, GDPO may spend gradient on noise. In a group where all four completions pass the gate, the format reward is constant and drops out, while the small length differences among them (all within 0.61–0.76) are standardized to unit variance, so by Proposition 2(b) GDPO gives them a large weight on that prompt; batch-level normalization scales them by the much larger batch standard deviation. Third, batch-level normalization gives nonzero advantages to groups in which every completion scored the same, which GDPO does not. Comparing against WAS-PO-Diag and against batch-normalized summation with fixed weights v\mathbf{v}, both at K≥3K \ge3, would separate these explanations.

Practical Guidance

Items marked observed are supported by our experiments on one task; items marked proposed have not been tested.

Training reward is not held-out accuracy (observed). With three or four rewards, WAS-PO reached higher training reward than GDPO in every seed but equal or lower held-out accuracy, and GRPO matched or exceeded GDPO on held-out accuracy at K=2K = 2 and K=4K = 4. In these settings the secondary rewards were close to copies of the accuracy gate. On this evidence we do not recommend WAS-PO as a way to improve held-out accuracy; what we have shown is that it changes which weighted training objective is optimized.

Inspect the whitened weights (proposed). Log vt\mathbf{v}_t and ψ(Σˉt,w)\psi(\bar{\Sigma}_t,\mathbf{w}) from the matrix that is actually inverted. Any positive ψ\psi means some reward is being reversed, but in our runs ψ\psi did not separate the collapsed run (0.69) from the worst cases assembled from per-pair maxima in runs that did not collapse (0.82–1.01), so we give no threshold. What mattered was whether the policy could move along the reversed direction cheaply. Two simple guards are to clip negative entries of vt\mathbf{v}_t to zero, or to fall back to Wt=I\mathbf{W}_t = \mathbf{I} when the batch mean of a reward with ωk<0\omega_k < 0 falls over a sustained window; neither has been tested. For two rewards, Proposition 3 gives the exact boundary ρ⋆=2ab/(a2+b2)\rho^{\star} = 2ab/(a^2 + b^2) for the correlation being whitened.

Screen before training (proposed). Log rewards for a few dozen steps, estimate Σ\Sigma from the batches, and compute φw\varphi_w, κw\kappa_w and v\mathbf{v}. We give no threshold for the angle; at 9.6∘9.6^\circ WAS-PO still differed from GDPO, mainly through normalization (Table 3).

Merge near-duplicate rewards (follows from Corollary 2). Two rewards that measure nearly the same thing, such as a gated format reward that is almost always earned, or two reward models trained on similar data, should be merged or left unwhitened. With unequal weights, whitening reverses the lighter-weighted one; with two equally weighted rewards it reverses neither (Proposition 3). Our own three- and four-reward settings contain such near-duplicates (Table 13), so the gains reported there come from a configuration this item advises against.

Limitations

WAS-PO changes both normalization and aggregation relative to GDPO, and the control that separates them was run only with two rewards and three seeds. The gains at K≥3K \ge3 are gains of WAS-PO as a whole, and Section 5.5 lists competing explanations that we have not tested.

The gains are in training reward only. Held-out accuracy did not improve and was lower than GDPO’s with four rewards. Training AUC is measured on temperature-1 samples from the training prompts, while held-out accuracy uses greedy decoding on new prompts, so a method that sharpens the policy faster can raise the first without the second. The held-out prompts come from the same BFCL subset as the training prompts, taken in dataset order, so their category mix may differ from the training portion, and we have no evidence about transfer to other tool-calling distributions or to the official BFCL evaluation.

The tool-calling results come from one task, one 1.5B model and a small batch (N=32N = 32); the math runs have three seeds each. Each KK uses one reward set, and moving from two to three rewards adds the length reward, so the number of rewards is confounded with which rewards are used. The four-reward setting has three seeds and held-out scores for two of them. Conditional reward gating produces much of the correlation we study, and our diagnostics use batch-level correlations, not the within-group correlations that govern GDPO.

The analysis relies on a stated reading of w\mathbf{w} (Proposition 1), on treating normalization statistics as constants (Proposition 2), and on Assumption 1 with an idealized EMA (Theorem 1). The inversion diagnostics rest on one collapsed run and two runs that did not collapse despite a nearly singular estimate.

Related Work

Group-based policy optimization. Shao et al. (2024) introduced GRPO, which normalizes a scalar reward within a group of completions and needs no learned value function. Liu et al. [16] showed that GRPO’s summed reward maps distinct reward combinations to the same advantage, and proposed GDPO. On BFCL with Qwen2.5-1.5B they report 32.81% average accuracy for GDPO against 30.18% for GRPO over five runs, with an evaluation protocol that differs from ours. Liu et al. [17] analyze biases introduced by the length and standard-deviation normalization terms in GRPO, and Hu et al. [10] argue for normalizing advantages over the whole batch. WAS-PO uses batch-level normalization in the spirit of the latter and adds a correction for correlation between rewards, which these methods do not consider.

Multiple rewards in language model alignment. Bai et al. [2] train with a preference model for helpfulness and harmlessness, and Touvron et al. (2023) combine separate helpfulness and safety reward models with a threshold rule rather than a weighted sum. Wu et al. (2023) combine several fine-grained reward models with fixed weights, and Dai et al. [7] treat helpfulness and harmlessness as a constrained problem. Ramé et al. (2023) interpolate the weights of policies fine-tuned on different rewards to trace a Pareto front. Coste et al. [6] and Eisenstein et al. [8] combine reward models in ensembles and find that models trained on similar data share errors, and Moskovitz et al. [18] find that the correlation between component reward models affects where overoptimization sets in. We instead ask how correlation changes what a fixed weight vector does.

Multi-task loss weighting. Multi-task learning faces a similar problem with conflicting or redundant losses. Kendall et al. [13] learn task weights from homoscedastic uncertainty, Sener & Koltun (2018) cast multi-task learning as multi-objective optimization, and Yu et al. (2020) project away conflicting gradient components. These methods operate on losses or parameter gradients, whereas WAS-PO changes per-sample advantages before any gradient is formed.

Whitening and covariance estimation. ZCA whitening was used in early work on the independent components of natural images [3]. Kessy et al. [14] characterize the family of whitening transforms and the optimality of ZCA, and Huang et al. [11] use ZCA whitening inside network layers and find that PCA whitening performs worse there. Estimating covariance from few samples is a classical difficulty [15]; in our two-reward runs, Ledoit–Wolf shrinkage did not close the gap to GDPO. Amari [1] and Kakade [12] precondition parameter updates with the Fisher information; WAS-PO can loosely be viewed as preconditioning in reward space instead.

Reward hacking. Skalse et al. (2022) define reward hacking formally, Gao et al. [9] measure how optimizing a learned reward model overshoots the reward it approximates, and Pan et al. (2022) show that misspecified rewards are exploited more as agents become more capable. ZCA-Inversion differs from reward hacking in that it occurs even when every reward is correctly specified, because it arises from how the rewards are combined.

Conclusion

When normalized rewards are correlated, adding them with fixed weights does not apply those weights to the decorrelated components of the reward signal. We quantified the discrepancy, derived the objectives that GDPO and whitened aggregation follow under fixed normalization statistics, and gave an exact two-reward condition, with an extension to near-duplicate rewards, under which whitening reverses a reward. In tool calling with three or four correlated rewards, WAS-PO reached higher training reward than GDPO in every seed, but its held-out accuracy was not higher, and with four rewards it was about 0.03 lower (two seeds). In our two-reward setting, GDPO was better. With two nearly identical rewards, WAS-PO lowered the lighter-weighted reward, as the sign condition predicts. The open questions are how much of the training-reward gain comes from whitening rather than batch-level normalization, why it does not carry over to held-out accuracy, and whether either finding holds on other tasks and larger models.

Broader Impact Statement

Multi-reward training is used to combine helpfulness, format and safety objectives. ZCA-Inversion is relevant to safety because whitening two highly correlated rewards with unequal weights can train the policy to lower the lighter-weighted one, which could be a safety reward. Proposition 3 gives the exact condition for two rewards in terms of the correlation being whitened, but the estimate actually inverted can differ from the population value (Table 13), and negatively correlated pairs are never reversed. We ran no experiments with reward models or safety rewards. We advise against whitening a safety reward unless its weight in ω\boldsymbol{\omega} is constrained to be nonnegative. We release no new datasets or models; all resources used are publicly available.

Code Availability

Code is available at https://github.com/lleye/waspo.

Proofs for Section 3

Proposition 1. Since Σ≻0\boldsymbol{\Sigma} \succ0, its symmetric positive definite square root and the inverse of that root exist and are unique. With z=Σ−1/2A^w\mathbf{z} = \boldsymbol{\Sigma}^{-1/2}\hat{\mathbf{A}}\mathbf{w} we have Cov⁡(z)=Σ−1/2ΣΣ−1/2=I\operatorname{Cov}(\mathbf{z}) = \boldsymbol{\Sigma}^{-1/2}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^{-1/2} = \mathbf{I} and A^=Σ1/2z\hat{\mathbf{A}} = \boldsymbol{\Sigma}^{1/2}\mathbf{z}. Then wTA^=wTΣ1/2z=(Σ1/2w)Tz\mathbf{w}^{\mathsf{T}}\hat{\mathbf{A}} = \mathbf{w}^{\mathsf{T}}\boldsymbol{\Sigma}^{1/2}\mathbf{z} = (\boldsymbol{\Sigma}^{1/2}\mathbf{w})^{\mathsf{T}}\mathbf{z} by symmetry, and wTΣ−1/2A^=wTΣ−1/2Σ1/2z=wTz\mathbf{w}^{\mathsf{T}}\boldsymbol{\Sigma}^{-1/2}\hat{\mathbf{A}} = \mathbf{w}^{\mathsf{T}}\boldsymbol{\Sigma}^{-1/2}\boldsymbol{\Sigma}^{1/2}\mathbf{z} = \mathbf{w}^{\mathsf{T}}\mathbf{z}. The two agree for every w\mathbf{w} if and only if Σ1/2w=w\boldsymbol{\Sigma}^{1/2}\mathbf{w} = \mathbf{w} for every w\mathbf{w}, that is, Σ1/2=I\boldsymbol{\Sigma}^{1/2} = \mathbf{I}, which is equivalent to Σ=I\boldsymbol{\Sigma} = \mathbf{I}. □\square

Corollary 1. For a fixed vector m\mathbf{m}, Var⁡(mTA^)=mTΣm\operatorname{Var}(\mathbf{m}^{\mathsf{T}}\hat{\mathbf{A}}) = \mathbf{m}^{\mathsf{T}}\boldsymbol{\Sigma}\mathbf{m}. With m=w\mathbf{m} = \mathbf{w} and unit diagonal this is ∥w∥2+∑j≠kwjwkρjk\lVert\mathbf{w}\rVert^2 + \sum_{j\ne k} w_jw_k\rho_{jk}. With m=Σ−1/2w\mathbf{m} = \boldsymbol{\Sigma}^{-1/2}\mathbf{w} it is wTΣ−1/2ΣΣ−1/2w=∥w∥2\mathbf{w}^{\mathsf{T}}\boldsymbol{\Sigma}^{-1/2}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^{-1/2}\mathbf{w} = \lVert\mathbf{w}\rVert^2. □\square

Proposition 2. (a) With the stated constants, A~i=s−1∑kvk(ri(k)−μk)/σk=s−1(ωTri−γ0)\widetilde{A}_i=s^{-1}\sum_k v_k(r_i^{(k)}-\mu_k)/\sigma_k=s^{-1}(\boldsymbol{\omega}^{\mathsf{T}}\mathbf{r}_i-\gamma_0), where ω=D−1v\boldsymbol{\omega}=\mathbf{D}^{-1}\mathbf{v} and γ0=∑kvkμk/σk\gamma_0=\sum_k v_k\mu_k/\sigma_k does not depend on yiy_i. For any qq that does not depend on θ\theta, the score-function identity gives Ey∼πθ[q(y)∇θlog⁡πθ(y∣x)]=∇θEy∼πθ[q(y)]\mathbb{E}_{y\sim\pi_\theta}[q(y)\nabla_\theta\log\pi_\theta(y\mid x)]=\nabla_\theta\mathbb{E}_{y\sim\pi_\theta}[q(y)], and Ey∼πθ[∇θlog⁡πθ(y∣x)]=0\mathbb{E}_{y\sim\pi_\theta}[\nabla_\theta\log\pi_\theta(y\mid x)]=0. Applying both to q=ωTrq=\boldsymbol{\omega}^{\mathsf{T}}\mathbf{r} and to the constant γ0\gamma_0, and averaging over prompts, gives E[A~∇θlog⁡πθ]=s−1∇θJω\mathbb{E}[\widetilde{A}\nabla_\theta\log\pi_\theta]=s^{-1}\nabla_\theta J_{\boldsymbol{\omega}}. Setting Σ=I\boldsymbol{\Sigma}=\mathbf{I} gives D−1w\mathbf{D}^{-1}\mathbf{w} for the batch-normalized sum without whitening. (b) For a fixed prompt xx, A~i=s−1(∑k(wk/σk(x))ri(k)−γ0(x))\widetilde{A}_i=s^{-1}(\sum_k(w_k/\sigma_k(x))r_i^{(k)}-\gamma_0(x)) with γ0(x)\gamma_0(x) independent of yiy_i, and the same argument applied conditionally on xx, followed by averaging over prompts, gives the stated objective. Using the group mean that includes yiy_i itself, with σk(x)\sigma_k(x) still held fixed, adds a factor (1−1/G)(1-1/G) common to all rewards, because the other completions are independent of yiy_i; it does not change the direction of the gradient. □

Proposition 3. For Σ=(1ρρ1)\boldsymbol{\Sigma}=\begin{pmatrix}1&\rho\\\rho&1\end{pmatrix} the eigenvectors are u±=(1,±1)/2\mathbf{u}_{\pm}=(1,\pm1)/\sqrt{2} with eigenvalues 1±ρ1\pm\rho, so Σ−1/2=(1+ρ)−1/2u+u+T+(1−ρ)−1/2u−u−T\boldsymbol{\Sigma}^{-1/2}=(1+\rho)^{-1/2}\mathbf{u}_{+}\mathbf{u}_{+}^{\mathsf{T}}+(1-\rho)^{-1/2}\mathbf{u}_{-}\mathbf{u}_{-}^{\mathsf{T}} and

v1=a+b21+ρ+a−b21−ρ,v2=a+b21+ρ−a−b21−ρ.v_1=\frac{a+b}{2\sqrt{1+\rho}}+\frac{a-b}{2\sqrt{1-\rho}},\qquad v_2=\frac{a+b}{2\sqrt{1+\rho}}-\frac{a-b}{2\sqrt{1-\rho}}.

Both terms of v1v_1 are nonnegative and the first is positive, so v1>0v_1>0. Both terms in the expression for v2v_2 are nonnegative, so v2<0v_2<0 if and only if (a−b)2(1+ρ)>(a+b)2(1−ρ)(a-b)^2(1+\rho)>(a+b)^2(1-\rho), that is, ρ[(a−b)2+(a+b)2]>(a+b)2−(a−b)2\rho[(a-b)^2+(a+b)^2]>(a+b)^2-(a-b)^2, which simplifies to 2ρ(a2+b2)>4ab2\rho(a^2+b^2)>4ab, or ρ>2ab/(a2+b2)\rho>2ab/(a^2+b^2). If a=ba=b, the second terms vanish and v1=v2>0v_1=v_2>0. If a>ba>b, as ρ→1\rho\to1 the first terms stay bounded while the second terms grow as (a−b)/(21−ρ)(a-b)/(2\sqrt{1-\rho}), so ∣v2∣→∞\lvert v_2\rvert\to\infty and v2/v1→−1v_2/v_1\to-1. □

Corollary 2. Write vn=∑m(λm(n))−1/2(un,mTw)un,m\mathbf{v}_n=\sum_m(\lambda_m^{(n)})^{-1/2}(\mathbf{u}_{n,m}^{\mathsf{T}}\mathbf{w})\mathbf{u}_{n,m} in an orthonormal eigenbasis of Σn\boldsymbol{\Sigma}_n, with un,1=un\mathbf{u}_{n,1}=\mathbf{u}_n. The terms with m≥2m\ge2 have norm at most λn−1/2∥w∥\lambda_n^{-1/2}\lVert\mathbf{w}\rVert, while the first has norm (λ1(n))−1/2∣unTw∣→∞(\lambda_1^{(n)})^{-1/2}\lvert\mathbf{u}_n^{\mathsf{T}}\mathbf{w}\rvert\to\infty because unTw→uTw≠0\mathbf{u}_n^{\mathsf{T}}\mathbf{w}\to\mathbf{u}^{\mathsf{T}}\mathbf{w}\ne0. Dividing by ∥vn∥\lVert\mathbf{v}_n\rVert gives the limit. For the pair case let d=(ei−ej)/2\mathbf{d}=(\mathbf{e}_i-\mathbf{e}_j)/\sqrt{2}. Then dTΣnd=1−ρij→0\mathbf{d}^{\mathsf{T}}\boldsymbol{\Sigma}_n\mathbf{d}=1-\rho_{ij}\to0, while dTΣnd≥λn∑m≥2(un,mTd)2\mathbf{d}^{\mathsf{T}}\boldsymbol{\Sigma}_n\mathbf{d}\ge\lambda_n\sum_{m\ge2}(\mathbf{u}_{n,m}^{\mathsf{T}}\mathbf{d})^2. Hence (unTd)2→1(\mathbf{u}_n^{\mathsf{T}}\mathbf{d})^2\to1, so u=±d\mathbf{u}=\pm\mathbf{d} and uTw=±(wi−wj)/2\mathbf{u}^{\mathsf{T}}\mathbf{w}=\pm(w_i-w_j)/\sqrt{2}, which is nonzero when wi≠wjw_i\ne w_j. □

Correlation of gated binary rewards. Let AA and FF be binary with F≤AF\le A, as for an accuracy reward and a format reward gated on it, with means paccp_{\mathrm{acc}} and pfmtp_{\mathrm{fmt}}. Then E[AF]=pfmt\mathbb{E}[AF]=p_{\mathrm{fmt}}, so Cov⁡(A,F)=pfmt(1−pacc)\operatorname{Cov}(A,F)=p_{\mathrm{fmt}}(1-p_{\mathrm{acc}}), and dividing by pacc(1−pacc)pfmt(1−pfmt)\sqrt{p_{\mathrm{acc}}(1-p_{\mathrm{acc}})p_{\mathrm{fmt}}(1-p_{\mathrm{fmt}})} gives ρ=pfmt(1−pacc)/(pacc(1−pfmt))\rho=\sqrt{p_{\mathrm{fmt}}(1-p_{\mathrm{acc}})/(p_{\mathrm{acc}}(1-p_{\mathrm{fmt}}))}. □

Proof of Theorem 1

Step 1. Let h(X)=X−1/2h(\mathbf{X})=\mathbf{X}^{-1/2} on symmetric matrices with all eigenvalues at least λ0\lambda_0, and write X=QΛQT\mathbf{X}=\mathbf{Q}\boldsymbol{\Lambda}\mathbf{Q}^{\mathsf{T}} with eigenvalues λ1,…,λK\lambda_1,\ldots,\lambda_K. By the Daleckii–Krein formula [4 Section V.3], the Fréchet derivative is dh(X)[H]=Q(M∘(QTHQ))QTdh(\mathbf{X})[\mathbf{H}]=\mathbf{Q}(\mathbf{M}\circ(\mathbf{Q}^{\mathsf{T}}\mathbf{H}\mathbf{Q}))\mathbf{Q}^{\mathsf{T}}, where ∘\circ is the entrywise product and

mii=−12λi−3/2,mij=λi−1/2−λj−1/2λi−λj=−1λiλj(λi+λj)(λi≠λj).m_{ii}=-\frac{1}{2}\lambda_i^{-3/2},\qquad m_{ij}=\frac{\lambda_i^{-1/2}-\lambda_j^{-1/2}}{\lambda_i-\lambda_j}=-\frac{1}{\sqrt{\lambda_i\lambda_j}(\sqrt{\lambda_i}+\sqrt{\lambda_j})}\quad(\lambda_i\ne\lambda_j).

with mij=−12λi−3/2m_{ij}=-\frac{1}{2}\lambda_i^{-3/2} when λi=λj\lambda_i=\lambda_j. Since λiλj≥λ0\sqrt{\lambda_i\lambda_j}\ge\lambda_0 and λi+λj≥2λ0\sqrt{\lambda_i}+\sqrt{\lambda_j}\ge2\sqrt{\lambda_0}, every ∣mij∣≤1/(2λ03/2)\lvert m_{ij}\rvert\le1/(2\lambda_0^{3/2}). The Frobenius norm is invariant under orthogonal conjugation and ∥M∘B∥F≤max⁡ij∣mij∣∥B∥F\lVert\mathbf{M}\circ\mathbf{B}\rVert_F\le\max_{ij}\lvert m_{ij}\rvert\lVert\mathbf{B}\rVert_F, so ∥dh(X)[H]∥F≤∥H∥F/(2λ03/2)\lVert dh(\mathbf{X})[\mathbf{H}]\rVert_F\le\lVert\mathbf{H}\rVert_F/(2\lambda_0^{3/2}).

Step 2. Let H=Σ^−Σ\mathbf{H}=\widehat{\boldsymbol{\Sigma}}-\boldsymbol{\Sigma} and Xξ=Σ+ξH\mathbf{X}_\xi=\boldsymbol{\Sigma}+\xi\mathbf{H} for ξ∈[0,1]\xi\in[0,1]. The smallest eigenvalue is concave on symmetric matrices, so λmin⁡(Xξ)≥(1−ξ)λmin⁡(Σ)+ξλmin⁡(Σ^)≥λ0\lambda_{\min}(\mathbf{X}_\xi)\ge(1-\xi)\lambda_{\min}(\boldsymbol{\Sigma})+\xi\lambda_{\min}(\widehat{\boldsymbol{\Sigma}})\ge\lambda_0. Hence hh is continuously differentiable along the segment and Σ^−1/2−Σ−1/2=∫01dh(Xξ)[H] dξ\widehat{\boldsymbol{\Sigma}}^{-1/2}-\boldsymbol{\Sigma}^{-1/2}=\int_0^1dh(\mathbf{X}_\xi)[\mathbf{H}]\,d\xi, which by Step 1 gives ∥Σ^−1/2−Σ−1/2∥F≤∥H∥F/(2λ03/2)\lVert\widehat{\boldsymbol{\Sigma}}^{-1/2}-\boldsymbol{\Sigma}^{-1/2}\rVert_F\le\lVert\mathbf{H}\rVert_F/(2\lambda_0^{3/2}).

Step 3. ∥(Σ^−1/2−Σ−1/2)w∥2≤∥Σ^−1/2−Σ−1/2∥F2∥w∥2≤∥H∥F2∥w∥2/(4λ03)\lVert(\widehat{\boldsymbol{\Sigma}}^{-1/2}-\boldsymbol{\Sigma}^{-1/2})\mathbf{w}\rVert^2\le\lVert\widehat{\boldsymbol{\Sigma}}^{-1/2}-\boldsymbol{\Sigma}^{-1/2}\rVert_F^2\lVert\mathbf{w}\rVert^2\le\lVert\mathbf{H}\rVert_F^2\lVert\mathbf{w}\rVert^2/(4\lambda_0^3) almost surely. Taking expectations and applying Assumption 1 gives (6). □

EMA variance factor. Unrolling the recursion Σ^t=βΣ^t−1+(1−β)Ct\widehat{\boldsymbol{\Sigma}}_t = \beta\widehat{\boldsymbol{\Sigma}}_{t-1} + (1-\beta)\mathbf{C}_t from Σ^1=C1\widehat{\boldsymbol{\Sigma}}_1 = \mathbf{C}_1 gives

Σ^t=βt−1C1+(1−β)∑j=0t−2βjCt−j.\widehat{\boldsymbol{\Sigma}}_t = \beta^{t-1}\mathbf{C}_1 + (1-\beta)\sum_{j=0}^{t-2}\beta^j\mathbf{C}_{t-j}.

If the Ct\mathbf{C}_t are independent with mean Σ\boldsymbol{\Sigma} and entrywise variance τ2\tau^2, each entry of Σ^t\widehat{\boldsymbol{\Sigma}}_t has variance τ2[β2(t−1)+(1−β)2∑j=0t−2β2j]=τ2ft\tau^2\left[\beta^{2(t-1)}+(1-\beta)^2\sum_{j=0}^{t-2}\beta^{2j}\right]=\tau^2 f_t, with ftf_t as in § 3.4. Bias in the batch estimates, which is O(1/N)O(1/N) for sample correlations, is not reduced by the EMA.

Evaluating the Bound

Table 4 evaluates (6) with c=4c=4 and λ0\lambda_0 set to the smallest eigenvalue of Σ‾\overline{\boldsymbol{\Sigma}}, for one batch of NN rollouts and for the EMA under the idealization of § 3.4, both at its stationary factor 1/1991/199 and at step t=100t=100 (f100=0.141f_{100}=0.141). We use N=32N=32 in the main runs and N=128N=128 in the G=16G=16 ablation. Using the time-averaged λmin⁡\lambda_{\min} is itself an idealization: the estimate became nearly singular at times in the three- and four-reward runs, and single-batch estimates can be worse, so for them Theorem 1 holds only with a much smaller λ0\lambda_0, or not usefully at all. At the largest two-reward estimate of 0.8690.869, λmin⁡=0.131\lambda_{\min}=0.131, and the constant K2∥w∥2/λmin⁡3K^2\lVert\mathbf{w}\rVert^2/\lambda_{\min}^3 rises from 111.8111.8 to about 22242224. Step 3 of the proof is tight only when Σ^−1/2−Σ−1/2\widehat{\boldsymbol{\Sigma}}^{-1/2}-\boldsymbol{\Sigma}^{-1/2} is close to rank one with w\mathbf{w} in its row space.

KKλmin⁡\lambda_{\min}E\mathcal{E}K2∥w∥2/λmin⁡3K^2\lVert\mathbf{w}\rVert^2/\lambda_{\min}^3One batch, N=32N=32One batch, N=128N=128EMA, stationary, N=32N=32EMA, stationary, N=128N=128EMA, t=100t=100, N=32N=32EMA, t=100t=100, N=128N=128
20.3550.112111.83.610.8800.0180.0040.5080.124
30.1960.2511601.751.712.60.2600.0637.291.78
40.1960.3602932.594.623.10.4750.11613.33.26

Table 4. Bound (6) against the size of the correction E\mathcal{E}.

Full Results

Hyperparameters and Implementation Details

Table 14: Training hyperparameters. Settings not listed are the defaults of TRL 0.18 and Transformers.

Prompts. Tool-calling prompts list the names and descriptions (truncated to 80 characters) of the first three available functions, then the user request (truncated to 300 characters), then the instruction to respond only with a JSON function call, followed by the example {"name": "function_name", "arguments": {"param": "value"}}. The held-out evaluation uses the same content with the system message “You are a helpful assistant.” Ground-truth argument values in BFCL can list several acceptable values; the accuracy reward compares against the first, as a case-insensitive string match. The math system prompt asks the model to end its response with The answer is: X, where XX is the numeric answer only.

Compute. The reported runs used about 120 GPU-hours, and preliminary and failed runs about 40 more. The G=8G = 8 and G=16G = 16 ablations also use eight prompts per step.

Reward wiring. A natural way to add several rewards to a GRPO trainer is to wrap all reward functions in one function that returns their weighted sum. The trainer then sees a single reward, and GRPO, GDPO and WAS-PO all run the same single-reward path, so no comparison between them is meaningful. All results here use an implementation that passes each reward function to the trainer separately, so that the reward tensor has one column per reward. We recommend checking that the reward tensor has KK columns before comparing multi-reward methods.

References

  1. [1]Shun-ichi Amari. Natural gradient works efficiently in learning. Neural Computation, 10(2):251–276, 1998. doi: 10.1162/089976698300017746.DOI
  2. [2]Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Noya DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.arxiv.org/abs/2204.05862
  3. [3]Anthony J. Bell and Terrence J. Sejnowski. The “independent components” of natural scenes are edge filters. Vision Research, 37(23):3327–3338, 1997. doi: 10.1016/S0042-6989(97)00121-1.DOI
  4. [4]Rajendra Bhatia. Matrix Analysis, volume 169 of Graduate Texts in Mathematics. Springer, New York, 1997.
  5. [5]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
  6. [6]Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.02743
  7. [7]Juntao Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.12773
  8. [8]Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, DJ Divjotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. Helping or herding? Reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09244, 2023.arxiv.org/abs/2312.09244
  9. [9]Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 10835–10866, 2023.arxiv.org/abs/2210.10760
  10. [10]Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. REINFORCE++: Stabilizing critic-free policy optimization with global advantage normalization. arXiv preprint arXiv:2501.03262, 2025.arxiv.org/abs/2501.03262
  11. [11]Lei Huang, Dawei Yang, Bo Lang, and Jia Deng. Decorrelated batch normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. doi: 10.1109/CVPR.2018.00089.
  12. [12]Sham M. Kakade. A natural policy gradient. In Advances in Neural Information Processing Systems, volume 14, pp. 1531–1538, 2001.
  13. [13]Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7482–7491, 2018.arxiv.org/abs/1705.07115
  14. [14]Agnan Kessy, Alex Lewin, and Korbinian Strimmer. Optimal whitening and decorrelation. The American Statistician, 72(4):309–314, 2018. doi: 10.1080/00031305.2016.1277159.DOI
  15. [15]Olivier Ledoit and Michael Wolf. A well-conditioned estimator for large-dimensional covariance matrices. Journal of Multivariate Analysis, 88(2):365–411, 2004. doi: 10.1016/S0047-259X(03)00096-4.DOI
  16. [16]Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Peter Belcak, Mingjie Liu, Min-Hung Chen, Hongxu Yin, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Yejin Choi, Jan Kautz, and Pavlo Molchanov. GDPO: Group reward-decoupled normalization policy optimization for multi-reward RL optimization. arXiv preprint arXiv:2601.05242, 2026.arxiv.org/abs/2601.05242
  17. [17]Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025.arxiv.org/abs/2503.20783
  18. [18]Ted Moskovitz, Aaditya K. Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca Dragan, and Stephen Marcus McAleer. Confronting reward model overoptimization with constrained RLHF. In International Conference on Learning Representations, 2024.

Paper details

vlnc:W_e3ra2pkr
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents